2024-09-15T22:55:55.855 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T22:55:55.882 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T22:55:55.968 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906290' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request sha1: 8921c8ef947807b717db041d85f0761b7914d5fc subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3393 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 294/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLcJdXZJH7eFAmDuYpN6oRniIxmjGqJrdeMDI8muwhXgR27rLd6JkJd1g21h7rxi7ZebHSEy/kjrNV678M+n5Rc= smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYgN6oediMfQ0GcCrWhSw9UKhmRybV9w6HPmh1rXKPGn1lzmLCLcD7N+MmcfsLCWk5rDUq/Bb5ypi+ND5KfLbU= smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMSJzxGUwANXNXfNFl4Vi2L0LEgHShuO/OQThHqTOKxedgP7VAc2ANErt/Uo3kP/6ZfFJcqo0kqsOM0qMBbxvrM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T22:55:55.969 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-15T22:55:55.969 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-15T22:55:55.970 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T22:55:55.971 INFO:teuthology.task.internal:Checking packages... 2024-09-15T22:55:55.992 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-15T22:55:55.993 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T22:55:55.993 INFO:teuthology.packaging:ref: None 2024-09-15T22:55:55.993 INFO:teuthology.packaging:tag: None 2024-09-15T22:55:55.993 INFO:teuthology.packaging:branch: squid 2024-09-15T22:55:55.993 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T22:55:55.993 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-15T22:55:56.201 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-328.gad35db06 2024-09-15T22:55:56.202 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T22:55:56.208 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T22:55:56.209 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T22:55:56.267 INFO:teuthology.task.internal:Saving configuration 2024-09-15T22:55:56.287 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T22:55:56.309 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T22:55:56.334 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 22:48:51.353991', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLcJdXZJH7eFAmDuYpN6oRniIxmjGqJrdeMDI8muwhXgR27rLd6JkJd1g21h7rxi7ZebHSEy/kjrNV678M+n5Rc='} 2024-09-15T22:55:56.354 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 22:48:51.352329', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYgN6oediMfQ0GcCrWhSw9UKhmRybV9w6HPmh1rXKPGn1lzmLCLcD7N+MmcfsLCWk5rDUq/Bb5ypi+ND5KfLbU='} 2024-09-15T22:55:56.373 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 22:48:51.355520', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMSJzxGUwANXNXfNFl4Vi2L0LEgHShuO/OQThHqTOKxedgP7VAc2ANErt/Uo3kP/6ZfFJcqo0kqsOM0qMBbxvrM='} 2024-09-15T22:55:56.374 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T22:55:56.380 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-09-15T22:55:56.380 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-09-15T22:55:56.380 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-09-15T22:55:56.380 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T22:55:56.483 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e310b7be0>, signals=[15]) 2024-09-15T22:55:56.483 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T22:55:56.489 INFO:teuthology.task.internal:Opening connections... 2024-09-15T22:55:56.489 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T22:55:56.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T22:55:56.610 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T22:55:56.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T22:55:56.685 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-09-15T22:55:56.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T22:55:56.759 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T22:55:56.767 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-09-15T22:55:56.786 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-09-15T22:55:56.786 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-09-15T22:55:56.844 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T22:55:56.845 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-09-15T22:55:56.845 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T22:55:56.845 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T22:55:56.845 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T22:55:56.845 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-09-15T22:55:56.862 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-09-15T22:55:56.880 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-09-15T22:55:56.880 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-09-15T22:55:56.937 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T22:55:56.938 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T22:55:56.939 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T22:55:56.939 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-09-15T22:55:56.962 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-09-15T22:55:56.980 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-09-15T22:55:56.980 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-09-15T22:55:57.039 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2024-09-15T22:55:57.039 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="9" 2024-09-15T22:55:57.039 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2024-09-15T22:55:57.039 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2024-09-15T22:55:57.039 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="9" 2024-09-15T22:55:57.039 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el9" 2024-09-15T22:55:57.039 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T22:55:57.040 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2024-09-15T22:55:57.040 INFO:teuthology.orchestra.run.smithi148.stdout:LOGO="fedora-logo-icon" 2024-09-15T22:55:57.040 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T22:55:57.040 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2024-09-15T22:55:57.040 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T22:55:57.040 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T22:55:57.040 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T22:55:57.040 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-09-15T22:55:57.061 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T22:55:57.071 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T22:55:57.109 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T22:55:57.109 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-09-15T22:55:57.111 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-09-15T22:55:57.114 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-09-15T22:55:57.131 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T22:55:57.150 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T22:55:57.150 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-09-15T22:55:57.169 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-09-15T22:55:57.171 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-09-15T22:55:57.199 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T22:55:57.266 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T22:55:57.280 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-15T22:55:57.280 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-09-15T22:55:57.281 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-09-15T22:55:57.281 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T22:55:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:55:57.298 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-09-15T22:55:57.357 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-503.el9.x86_64 2024-09-15T22:55:57.358 INFO:teuthology.task.kernel:Running kernel on smithi062: 5.14.0-503.el9.x86_64 2024-09-15T22:55:57.358 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum install -y kernel 2024-09-15T22:56:00.053 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 8.2 MB 00:00 2024-09-15T22:56:01.962 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-15T22:56:06.446 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-15T22:56:07.964 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 19 kB 00:00 2024-09-15T22:56:08.959 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-15T22:56:14.649 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-15T22:56:16.697 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T22:56:16.697 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T22:56:16.799 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-09-15T22:56:16.807 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-09-15T22:56:16.807 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-09-15T22:56:16.807 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout:Install 4 Packages 2024-09-15T22:56:16.808 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T22:56:16.809 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 79 M 2024-09-15T22:56:16.809 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 126 M 2024-09-15T22:56:16.809 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-09-15T22:56:17.400 INFO:teuthology.orchestra.run.smithi062.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 40 kB/s | 23 kB 00:00 2024-09-15T22:56:18.225 INFO:teuthology.orchestra.run.smithi062.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-15T22:56:18.909 INFO:teuthology.orchestra.run.smithi062.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-15T22:56:19.734 INFO:teuthology.orchestra.run.smithi062.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-15T22:56:19.734 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-09-15T22:56:19.734 INFO:teuthology.orchestra.run.smithi062.stdout:Total 27 MB/s | 79 MB 00:02 2024-09-15T22:56:20.440 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-09-15T22:56:20.835 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-09-15T22:56:20.835 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-09-15T22:56:21.826 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-09-15T22:56:21.827 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-09-15T22:56:24.392 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-09-15T22:56:25.052 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T22:56:25.133 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T22:56:27.908 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T22:56:28.037 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T22:56:33.161 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T22:56:33.204 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T22:56:38.416 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T22:57:17.299 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T22:57:17.339 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T22:57:17.616 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T22:57:17.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T22:57:17.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T22:57:17.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T22:57:18.447 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T22:57:18.447 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T22:57:18.447 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-09-15T22:57:18.447 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T22:57:18.447 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T22:57:18.447 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T22:57:18.447 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T22:57:18.447 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T22:57:18.447 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-09-15T22:57:18.792 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T22:57:18.872 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T22:57:18.872 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T22:57:18.872 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T22:57:18.872 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T22:57:18.872 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T22:57:18.873 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-15T22:57:18.873 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T22:57:18.873 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T22:57:18.931 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T22:57:18.931 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T22:57:18.931 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2024-09-15T22:57:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2024-09-15T22:57:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2024-09-15T22:57:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.06 2024-09-15T22:57:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 82.el9 2024-09-15T22:57:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2024-09-15T22:57:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T22:57:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:Group : Unspecified 2024-09-15T22:57:18.972 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 8274670 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : builder@centos.org 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T22:57:18.973 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T22:57:18.974 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2024-09-15T22:57:18.974 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T22:57:18.974 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T22:57:18.974 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T22:57:18.974 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 5.14.0-505.el9.x86_64 2024-09-15T22:57:18.975 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T22:57:19.598 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-09-15T22:57:21.005 INFO:teuthology.orchestra.run.smithi062.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T22:57:21.044 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-09-15T22:57:21.046 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-09-15T22:57:21.061 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.w6zTtCFz5Y 2024-09-15T22:57:21.062 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.w6zTtCFz5Y 2024-09-15T22:57:21.129 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.w6zTtCFz5Y 2024-09-15T22:57:21.260 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.w6zTtCFz5Y is 6KB 2024-09-15T22:57:21.309 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.w6zTtCFz5Y 2024-09-15T22:57:21.324 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T22:57:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T22:57:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T22:57:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T22:57:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T22:57:21.388 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T22:57:21.540 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-09-15T22:57:51.544 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-15T22:57:51.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-09-15T22:57:51.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T22:58:10.109 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-09-15T22:58:19.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-09-15T22:58:19.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T22:58:22.173 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-09-15T22:58:34.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-09-15T22:58:34.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T22:59:05.949 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-09-15T22:59:06.259 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-09-15T22:59:06.259 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-505.el9.x86_64"... 2024-09-15T22:59:06.259 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-09-15T22:59:06.276 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-505.el9.x86_64 2024-09-15T22:59:06.276 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T22:59:06.276 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T22:59:06.277 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T22:59:07.278 DEBUG:teuthology.parallel:result is None 2024-09-15T22:59:07.278 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T22:59:07.549 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T22:59:07.550 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T22:59:07.554 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T22:59:07.557 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T22:59:07.605 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T22:59:07.613 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T22:59:07.656 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T22:59:07.656 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T22:59:07.659 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T22:59:07.661 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T22:59:07.772 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T22:59:07.781 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T22:59:07.782 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T22:59:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:59:07.799 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T22:59:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:59:07.816 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T22:59:07.819 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T22:59:07.843 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T22:59:07.874 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:59:07.893 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:59:07.895 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:59:07.899 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:59:07.912 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:59:07.946 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T22:59:07.947 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T22:59:07.959 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T22:59:07.960 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T22:59:07.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T22:59:07.964 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T22:59:07.991 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T22:59:08.006 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T22:59:08.006 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T22:59:08.029 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T22:59:08.032 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T22:59:08.048 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T22:59:08.120 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T22:59:08.292 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T22:59:08.330 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T22:59:08.378 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T22:59:08.378 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T22:59:08.445 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T22:59:08.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T22:59:08.545 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T22:59:08.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T22:59:08.673 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T22:59:08.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T22:59:08.741 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T22:59:08.786 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T22:59:08.872 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T22:59:08.915 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T22:59:08.960 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T22:59:08.960 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T22:59:09.027 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-09-15T22:59:09.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-09-15T22:59:09.032 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-09-15T22:59:09.067 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T22:59:09.079 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T22:59:09.104 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T22:59:09.579 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T22:59:09.588 INFO:teuthology.task.internal:Starting timer... 2024-09-15T22:59:09.588 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T22:59:09.638 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T22:59:09.669 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-15T22:59:09.670 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2024-09-15T22:59:09.736 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2024-09-15T22:59:09.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-09-15T22:59:09.797 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-09-15T22:59:09.799 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2024-09-15T22:59:09.880 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2024-09-15T22:59:09.882 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T22:59:09.882 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2024-09-15T22:59:09.906 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2024-09-15T22:59:09.906 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-09-15T22:59:09.928 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-09-15T22:59:09.928 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2024-09-15T22:59:09.962 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2024-09-15T22:59:09.963 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi062.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive'} 2024-09-15T22:59:09.963 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T22:59:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:59:09.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T22:59:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:59:10.020 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T22:59:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T22:59:10.050 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T22:59:10.050 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2024-09-15T22:59:10.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-09-15T22:59:10.114 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2024-09-15T22:59:10.151 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T22:59:10.161 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T22:59:10.174 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T22:59:10.175 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi067.front.sepia.ceph.com,smithi148.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T23:03:18.958 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi148.front.sepia.ceph.com')] 2024-09-15T23:03:18.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-09-15T23:03:18.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:03:19.040 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-09-15T23:03:19.128 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-09-15T23:03:19.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T23:03:19.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:03:19.206 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-09-15T23:03:19.285 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-09-15T23:03:19.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-15T23:03:19.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T23:03:19.374 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-09-15T23:03:19.452 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-09-15T23:03:19.453 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T23:03:19.467 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T23:03:19.468 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T23:03:19.468 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:03:19.471 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T23:03:19.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:03:19.474 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T23:03:19.474 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:03:19.500 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T23:03:19.505 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T23:03:19.514 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T23:03:19.521 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T23:03:19.522 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T23:03:19.536 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T23:03:19.540 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2024-09-15T23:03:19.548 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-09-15T23:03:19.551 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2024-09-15T23:03:19.560 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-09-15T23:03:19.560 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2024-09-15T23:03:19.566 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T23:03:19.571 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2024-09-15T23:03:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T23:03:19.580 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T23:03:19.586 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T23:03:19.589 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T23:03:19.599 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T23:03:19.637 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:03:19.640 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:03:19.640 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-09-15T23:03:19.640 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.641 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.641 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.641 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:03:19.653 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:03:19.654 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:03:19.654 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-09-15T23:03:19.654 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.654 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.654 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.654 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.656 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:03:19.657 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-09-15T23:03:19.657 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.657 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.657 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.657 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:03:19.659 INFO:teuthology.run_tasks:Running task install... 2024-09-15T23:03:19.673 DEBUG:teuthology.task.install:project ceph 2024-09-15T23:03:19.673 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-15T23:03:19.673 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-15T23:03:19.673 INFO:teuthology.task.install:Using flavor: default 2024-09-15T23:03:19.679 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T23:03:19.679 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-15T23:03:19.680 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-15T23:03:19.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:03:19.682 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-15T23:03:19.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:03:19.684 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'tag': None, 'wait_for_package': False} 2024-09-15T23:03:19.684 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:03:19.832 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-15T23:03:19.833 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-15T23:03:19.836 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-15T23:03:19.836 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-15T23:03:19.873 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/ 2024-09-15T23:03:19.873 INFO:teuthology.task.install.rpm:Package version is 19.1.1-326.g8921c8ef 2024-09-15T23:03:19.941 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T23:03:19.941 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:03:19.942 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T23:03:19.953 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T23:03:19.953 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:03:19.953 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T23:03:19.968 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-15T23:03:19.968 DEBUG:teuthology.orchestra.run.smithi062:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T23:03:19.979 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-15T23:03:19.979 DEBUG:teuthology.orchestra.run.smithi148:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T23:03:19.983 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T23:03:19.983 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:03:19.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T23:03:20.010 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-15T23:03:20.010 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8921c8ef947807b717db041d85f0761b7914d5fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T23:03:20.036 DEBUG:teuthology.orchestra.run.smithi062:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T23:03:20.046 DEBUG:teuthology.orchestra.run.smithi148:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T23:03:20.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T23:03:20.112 DEBUG:teuthology.orchestra.run.smithi062:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T23:03:20.122 DEBUG:teuthology.orchestra.run.smithi148:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T23:03:20.154 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T23:03:20.177 INFO:teuthology.orchestra.run.smithi062.stdout:check_obsoletes = 1 2024-09-15T23:03:20.178 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum clean all 2024-09-15T23:03:20.185 INFO:teuthology.orchestra.run.smithi148.stdout:check_obsoletes = 1 2024-09-15T23:03:20.187 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum clean all 2024-09-15T23:03:20.218 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-09-15T23:03:20.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-09-15T23:03:20.414 INFO:teuthology.orchestra.run.smithi148.stdout:45 files removed 2024-09-15T23:03:20.415 INFO:teuthology.orchestra.run.smithi062.stdout:45 files removed 2024-09-15T23:03:20.443 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-15T23:03:20.443 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-15T23:03:20.452 INFO:teuthology.orchestra.run.smithi067.stdout:45 files removed 2024-09-15T23:03:20.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-15T23:03:20.978 INFO:teuthology.orchestra.run.smithi148.stdout:ceph packages for x86_64 322 kB/s | 83 kB 00:00 2024-09-15T23:03:21.005 INFO:teuthology.orchestra.run.smithi062.stdout:ceph packages for x86_64 323 kB/s | 83 kB 00:00 2024-09-15T23:03:21.030 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 314 kB/s | 83 kB 00:00 2024-09-15T23:03:21.261 INFO:teuthology.orchestra.run.smithi148.stdout:ceph noarch packages 48 kB/s | 12 kB 00:00 2024-09-15T23:03:21.281 INFO:teuthology.orchestra.run.smithi062.stdout:ceph noarch packages 51 kB/s | 12 kB 00:00 2024-09-15T23:03:21.314 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-15T23:03:21.529 INFO:teuthology.orchestra.run.smithi148.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-15T23:03:21.539 INFO:teuthology.orchestra.run.smithi062.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-15T23:03:21.590 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-15T23:03:22.012 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-15T23:03:22.865 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.2 MB 00:01 2024-09-15T23:03:23.473 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-09-15T23:03:24.730 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - AppStream 13 MB/s | 20 MB 00:01 2024-09-15T23:03:25.433 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-15T23:03:26.108 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 20 MB 00:02 2024-09-15T23:03:29.983 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-15T23:03:30.114 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.5 MB 00:01 2024-09-15T23:03:30.575 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-15T23:03:31.709 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-15T23:03:32.017 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 96 kB/s | 19 kB 00:00 2024-09-15T23:03:32.389 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-15T23:03:32.990 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-15T23:03:33.330 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-15T23:03:33.577 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-15T23:03:38.813 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-15T23:03:38.945 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-09-15T23:03:39.008 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-15T23:03:41.240 INFO:teuthology.orchestra.run.smithi062.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:03:41.241 INFO:teuthology.orchestra.run.smithi062.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:03:41.252 INFO:teuthology.orchestra.run.smithi062.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-15T23:03:41.253 INFO:teuthology.orchestra.run.smithi062.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-15T23:03:41.254 INFO:teuthology.orchestra.run.smithi062.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-15T23:03:41.255 INFO:teuthology.orchestra.run.smithi062.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-15T23:03:41.256 INFO:teuthology.orchestra.run.smithi062.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-15T23:03:41.259 INFO:teuthology.orchestra.run.smithi062.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-15T23:03:41.260 INFO:teuthology.orchestra.run.smithi062.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-15T23:03:41.278 INFO:teuthology.orchestra.run.smithi148.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:03:41.279 INFO:teuthology.orchestra.run.smithi148.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:03:41.293 INFO:teuthology.orchestra.run.smithi148.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-15T23:03:41.293 INFO:teuthology.orchestra.run.smithi148.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-15T23:03:41.294 INFO:teuthology.orchestra.run.smithi148.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-15T23:03:41.295 INFO:teuthology.orchestra.run.smithi148.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-15T23:03:41.297 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:03:41.297 INFO:teuthology.orchestra.run.smithi148.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-15T23:03:41.298 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-15T23:03:41.300 INFO:teuthology.orchestra.run.smithi148.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-15T23:03:41.301 INFO:teuthology.orchestra.run.smithi148.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-15T23:03:41.309 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-15T23:03:41.310 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-15T23:03:41.311 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-15T23:03:41.312 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-15T23:03:41.313 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-15T23:03:41.316 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-15T23:03:41.317 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-15T23:03:41.326 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout:====================================================================================== 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout:====================================================================================== 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-15T23:03:41.344 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-15T23:03:41.345 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-15T23:03:41.346 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-15T23:03:41.347 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-15T23:03:41.348 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T23:03:41.349 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T23:03:41.350 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T23:03:41.351 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T23:03:41.352 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T23:03:41.353 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T23:03:41.354 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T23:03:41.355 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout:Installing weak dependencies: 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout:====================================================================================== 2024-09-15T23:03:41.356 INFO:teuthology.orchestra.run.smithi062.stdout:Install 148 Packages 2024-09-15T23:03:41.357 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 8 Packages 2024-09-15T23:03:41.357 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:03:41.357 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 226 M 2024-09-15T23:03:41.357 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-09-15T23:03:41.363 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-09-15T23:03:41.381 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================== 2024-09-15T23:03:41.381 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-09-15T23:03:41.381 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================== 2024-09-15T23:03:41.381 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-09-15T23:03:41.381 INFO:teuthology.orchestra.run.smithi148.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-15T23:03:41.381 INFO:teuthology.orchestra.run.smithi148.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-15T23:03:41.381 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-15T23:03:41.382 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-15T23:03:41.383 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrading: 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-15T23:03:41.384 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-15T23:03:41.385 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-15T23:03:41.386 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-15T23:03:41.386 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-15T23:03:41.386 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-15T23:03:41.386 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-15T23:03:41.386 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-15T23:03:41.386 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T23:03:41.386 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T23:03:41.386 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T23:03:41.387 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T23:03:41.388 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T23:03:41.389 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T23:03:41.390 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T23:03:41.391 INFO:teuthology.orchestra.run.smithi148.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T23:03:41.392 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T23:03:41.393 INFO:teuthology.orchestra.run.smithi148.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout:Installing weak dependencies: 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout:====================================================================================== 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout:Install 148 Packages 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrade 8 Packages 2024-09-15T23:03:41.394 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:03:41.395 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-09-15T23:03:41.395 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 226 M 2024-09-15T23:03:41.395 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-09-15T23:03:41.402 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-09-15T23:03:41.402 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 6.5 k 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.4 M 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 896 k 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 145 k 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 1.5 M 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 146 k 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 3.6 M 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 7.4 M 2024-09-15T23:03:41.403 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 49 k 2024-09-15T23:03:41.404 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 11 M 2024-09-15T23:03:41.404 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 49 M 2024-09-15T23:03:41.404 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 285 k 2024-09-15T23:03:41.404 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 765 k 2024-09-15T23:03:41.404 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 228 k 2024-09-15T23:03:41.404 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 27 k 2024-09-15T23:03:41.404 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 18 k 2024-09-15T23:03:41.404 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 32 k 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 751 k 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 127 k 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-15T23:03:41.405 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 161 k 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 320 k 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 300 k 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 99 k 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 85 k 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.1 M 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 170 k 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-15T23:03:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.4 M 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 3.2 M 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 21 M 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 32 k 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 2.4 M 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 249 k 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 4.7 M 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 17 M 2024-09-15T23:03:41.407 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.1.1-326.g8921c8ef.el9 ceph-noarch 16 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 25 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 163 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T23:03:41.408 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 506 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 5.3 M 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T23:03:41.409 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 45 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.1.1-326.g8921c8ef.el9 ceph 136 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T23:03:41.410 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T23:03:41.411 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T23:03:41.412 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T23:03:41.413 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T23:03:41.414 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================== 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout:Install 148 Packages 2024-09-15T23:03:41.415 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 8 Packages 2024-09-15T23:03:41.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:03:41.416 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 226 M 2024-09-15T23:03:41.416 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-09-15T23:03:41.908 INFO:teuthology.orchestra.run.smithi062.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 25 kB/s | 6.5 kB 00:00 2024-09-15T23:03:41.921 INFO:teuthology.orchestra.run.smithi067.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 56 kB/s | 6.5 kB 00:00 2024-09-15T23:03:41.951 INFO:teuthology.orchestra.run.smithi148.stdout:(1/156): ceph-19.1.1-326.g8921c8ef.el9.x86_64.r 48 kB/s | 6.5 kB 00:00 2024-09-15T23:03:42.156 INFO:teuthology.orchestra.run.smithi062.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 3.5 MB/s | 896 kB 00:00 2024-09-15T23:03:42.257 INFO:teuthology.orchestra.run.smithi062.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.4 MB/s | 145 kB 00:00 2024-09-15T23:03:42.304 INFO:teuthology.orchestra.run.smithi067.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 2.3 MB/s | 896 kB 00:00 2024-09-15T23:03:42.348 INFO:teuthology.orchestra.run.smithi062.stdout:(4/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 7.7 MB/s | 5.4 MB 00:00 2024-09-15T23:03:42.363 INFO:teuthology.orchestra.run.smithi067.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 2.4 MB/s | 145 kB 00:00 2024-09-15T23:03:42.418 INFO:teuthology.orchestra.run.smithi148.stdout:(2/156): ceph-fuse-19.1.1-326.g8921c8ef.el9.x86 1.9 MB/s | 896 kB 00:00 2024-09-15T23:03:42.510 INFO:teuthology.orchestra.run.smithi148.stdout:(3/156): ceph-immutable-object-cache-19.1.1-326 1.5 MB/s | 145 kB 00:00 2024-09-15T23:03:42.524 INFO:teuthology.orchestra.run.smithi062.stdout:(5/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 8.7 MB/s | 1.5 MB 00:00 2024-09-15T23:03:42.817 INFO:teuthology.orchestra.run.smithi062.stdout:(6/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 4.3 MB/s | 2.4 MB 00:00 2024-09-15T23:03:42.838 INFO:teuthology.orchestra.run.smithi067.stdout:(4/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 5.2 MB/s | 5.4 MB 00:01 2024-09-15T23:03:42.882 INFO:teuthology.orchestra.run.smithi062.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 13 MB/s | 4.7 MB 00:00 2024-09-15T23:03:43.138 INFO:teuthology.orchestra.run.smithi067.stdout:(5/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 5.1 MB/s | 1.5 MB 00:00 2024-09-15T23:03:43.272 INFO:teuthology.orchestra.run.smithi067.stdout:(6/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 2.6 MB/s | 2.4 MB 00:00 2024-09-15T23:03:43.285 INFO:teuthology.orchestra.run.smithi148.stdout:(4/156): ceph-mds-19.1.1-326.g8921c8ef.el9.x86_ 3.1 MB/s | 2.4 MB 00:00 2024-09-15T23:03:43.333 INFO:teuthology.orchestra.run.smithi062.stdout:(8/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 13 MB/s | 21 MB 00:01 2024-09-15T23:03:43.375 INFO:teuthology.orchestra.run.smithi062.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 607 kB/s | 25 kB 00:00 2024-09-15T23:03:43.377 INFO:teuthology.orchestra.run.smithi148.stdout:(5/156): ceph-base-19.1.1-326.g8921c8ef.el9.x86 3.5 MB/s | 5.4 MB 00:01 2024-09-15T23:03:43.659 INFO:teuthology.orchestra.run.smithi062.stdout:(10/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 14 MB/s | 11 MB 00:00 2024-09-15T23:03:43.734 INFO:teuthology.orchestra.run.smithi062.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 3.0 MB/s | 228 kB 00:00 2024-09-15T23:03:43.801 INFO:teuthology.orchestra.run.smithi062.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 405 kB/s | 27 kB 00:00 2024-09-15T23:03:43.867 INFO:teuthology.orchestra.run.smithi062.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 477 kB/s | 32 kB 00:00 2024-09-15T23:03:43.889 INFO:teuthology.orchestra.run.smithi067.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 6.3 MB/s | 4.7 MB 00:00 2024-09-15T23:03:43.919 INFO:teuthology.orchestra.run.smithi148.stdout:(6/156): ceph-mgr-19.1.1-326.g8921c8ef.el9.x86_ 2.4 MB/s | 1.5 MB 00:00 2024-09-15T23:03:43.942 INFO:teuthology.orchestra.run.smithi062.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 9.8 MB/s | 751 kB 00:00 2024-09-15T23:03:44.018 INFO:teuthology.orchestra.run.smithi062.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.1 MB/s | 163 kB 00:00 2024-09-15T23:03:44.093 INFO:teuthology.orchestra.run.smithi062.stdout:(16/156): librados-devel-19.1.1-326.g8921c8ef.e 1.6 MB/s | 127 kB 00:00 2024-09-15T23:03:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:(17/156): libradosstriper1-19.1.1-326.g8921c8ef 5.9 MB/s | 506 kB 00:00 2024-09-15T23:03:44.402 INFO:teuthology.orchestra.run.smithi148.stdout:(7/156): ceph-mon-19.1.1-326.g8921c8ef.el9.x86_ 4.6 MB/s | 4.7 MB 00:01 2024-09-15T23:03:44.569 INFO:teuthology.orchestra.run.smithi062.stdout:(18/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 14 MB/s | 5.3 MB 00:00 2024-09-15T23:03:44.644 INFO:teuthology.orchestra.run.smithi062.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 602 kB/s | 45 kB 00:00 2024-09-15T23:03:44.745 INFO:teuthology.orchestra.run.smithi062.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 1.3 MB/s | 136 kB 00:00 2024-09-15T23:03:44.828 INFO:teuthology.orchestra.run.smithi062.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 1.9 MB/s | 161 kB 00:00 2024-09-15T23:03:44.912 INFO:teuthology.orchestra.run.smithi062.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 3.8 MB/s | 320 kB 00:00 2024-09-15T23:03:44.996 INFO:teuthology.orchestra.run.smithi062.stdout:(23/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 3.5 MB/s | 300 kB 00:00 2024-09-15T23:03:45.071 INFO:teuthology.orchestra.run.smithi062.stdout:(24/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.3 MB/s | 99 kB 00:00 2024-09-15T23:03:45.154 INFO:teuthology.orchestra.run.smithi062.stdout:(25/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.0 MB/s | 85 kB 00:00 2024-09-15T23:03:45.396 INFO:teuthology.orchestra.run.smithi062.stdout:(26/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 13 MB/s | 3.1 MB 00:00 2024-09-15T23:03:45.397 INFO:teuthology.orchestra.run.smithi067.stdout:(8/156): ceph-common-19.1.1-326.g8921c8ef.el9.x 5.9 MB/s | 21 MB 00:03 2024-09-15T23:03:45.464 INFO:teuthology.orchestra.run.smithi067.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 379 kB/s | 25 kB 00:00 2024-09-15T23:03:45.471 INFO:teuthology.orchestra.run.smithi062.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 2.2 MB/s | 170 kB 00:00 2024-09-15T23:03:45.688 INFO:teuthology.orchestra.run.smithi062.stdout:(28/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 5.9 MB/s | 17 MB 00:02 2024-09-15T23:03:45.714 INFO:teuthology.orchestra.run.smithi062.stdout:(29/156): ceph-grafana-dashboards-19.1.1-326.g8 132 kB/s | 32 kB 00:00 2024-09-15T23:03:45.747 INFO:teuthology.orchestra.run.smithi062.stdout:(30/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.5 MB/s | 146 kB 00:00 2024-09-15T23:03:45.781 INFO:teuthology.orchestra.run.smithi067.stdout:(10/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9 5.6 MB/s | 11 MB 00:01 2024-09-15T23:03:45.881 INFO:teuthology.orchestra.run.smithi067.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 2.2 MB/s | 228 kB 00:00 2024-09-15T23:03:45.964 INFO:teuthology.orchestra.run.smithi062.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 14 MB/s | 3.6 MB 00:00 2024-09-15T23:03:45.973 INFO:teuthology.orchestra.run.smithi067.stdout:(12/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 294 kB/s | 27 kB 00:00 2024-09-15T23:03:46.039 INFO:teuthology.orchestra.run.smithi062.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 3.3 MB/s | 249 kB 00:00 2024-09-15T23:03:46.048 INFO:teuthology.orchestra.run.smithi067.stdout:(13/156): libcephfs-devel-19.1.1-326.g8921c8ef. 423 kB/s | 32 kB 00:00 2024-09-15T23:03:46.115 INFO:teuthology.orchestra.run.smithi062.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 657 kB/s | 49 kB 00:00 2024-09-15T23:03:46.198 INFO:teuthology.orchestra.run.smithi062.stdout:(34/156): ceph-prometheus-alerts-19.1.1-326.g89 197 kB/s | 16 kB 00:00 2024-09-15T23:03:46.240 INFO:teuthology.orchestra.run.smithi067.stdout:(14/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 3.8 MB/s | 751 kB 00:00 2024-09-15T23:03:46.273 INFO:teuthology.orchestra.run.smithi062.stdout:(35/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 3.7 MB/s | 285 kB 00:00 2024-09-15T23:03:46.299 INFO:teuthology.orchestra.run.smithi067.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 2.7 MB/s | 163 kB 00:00 2024-09-15T23:03:46.357 INFO:teuthology.orchestra.run.smithi067.stdout:(16/156): librados-devel-19.1.1-326.g8921c8ef.e 2.1 MB/s | 127 kB 00:00 2024-09-15T23:03:46.386 INFO:teuthology.orchestra.run.smithi148.stdout:(8/156): ceph-radosgw-19.1.1-326.g8921c8ef.el9. 5.3 MB/s | 11 MB 00:01 2024-09-15T23:03:46.436 INFO:teuthology.orchestra.run.smithi148.stdout:(9/156): ceph-selinux-19.1.1-326.g8921c8ef.el9. 507 kB/s | 25 kB 00:00 2024-09-15T23:03:46.441 INFO:teuthology.orchestra.run.smithi067.stdout:(17/156): libradosstriper1-19.1.1-326.g8921c8ef 5.9 MB/s | 506 kB 00:00 2024-09-15T23:03:46.692 INFO:teuthology.orchestra.run.smithi062.stdout:(36/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 15 MB/s | 49 MB 00:03 2024-09-15T23:03:46.725 INFO:teuthology.orchestra.run.smithi062.stdout:(37/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 1.7 MB/s | 765 kB 00:00 2024-09-15T23:03:46.750 INFO:teuthology.orchestra.run.smithi062.stdout:(38/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 314 kB/s | 18 kB 00:00 2024-09-15T23:03:46.896 INFO:teuthology.orchestra.run.smithi062.stdout:(39/156): ceph-mgr-diskprediction-local-19.1.1- 6.5 MB/s | 7.4 MB 00:01 2024-09-15T23:03:46.925 INFO:teuthology.orchestra.run.smithi062.stdout:(40/156): fuse-2.9.9-16.el9.x86_64.rpm 398 kB/s | 80 kB 00:00 2024-09-15T23:03:47.033 INFO:teuthology.orchestra.run.smithi067.stdout:(18/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 4.5 MB/s | 17 MB 00:03 2024-09-15T23:03:47.084 INFO:teuthology.orchestra.run.smithi062.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-15T23:03:47.109 INFO:teuthology.orchestra.run.smithi062.stdout:(42/156): libquadmath-11.5.0-2.el9.x86_64.rpm 881 kB/s | 191 kB 00:00 2024-09-15T23:03:47.142 INFO:teuthology.orchestra.run.smithi067.stdout:(19/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 7.6 MB/s | 5.3 MB 00:00 2024-09-15T23:03:47.168 INFO:teuthology.orchestra.run.smithi067.stdout:(20/156): python3-ceph-argparse-19.1.1-326.g892 337 kB/s | 45 kB 00:00 2024-09-15T23:03:47.184 INFO:teuthology.orchestra.run.smithi062.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-15T23:03:47.193 INFO:teuthology.orchestra.run.smithi067.stdout:(21/156): python3-ceph-common-19.1.1-326.g8921c 2.7 MB/s | 136 kB 00:00 2024-09-15T23:03:47.218 INFO:teuthology.orchestra.run.smithi067.stdout:(22/156): python3-cephfs-19.1.1-326.g8921c8ef.e 3.1 MB/s | 161 kB 00:00 2024-09-15T23:03:47.235 INFO:teuthology.orchestra.run.smithi062.stdout:(44/156): libgfortran-11.5.0-2.el9.x86_64.rpm 1.6 MB/s | 801 kB 00:00 2024-09-15T23:03:47.243 INFO:teuthology.orchestra.run.smithi067.stdout:(23/156): python3-rados-19.1.1-326.g8921c8ef.el 6.2 MB/s | 320 kB 00:00 2024-09-15T23:03:47.245 INFO:teuthology.orchestra.run.smithi148.stdout:(10/156): ceph-common-19.1.1-326.g8921c8ef.el9. 3.9 MB/s | 21 MB 00:05 2024-09-15T23:03:47.285 INFO:teuthology.orchestra.run.smithi062.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-09-15T23:03:47.293 INFO:teuthology.orchestra.run.smithi067.stdout:(24/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 3.9 MB/s | 300 kB 00:00 2024-09-15T23:03:47.319 INFO:teuthology.orchestra.run.smithi067.stdout:(25/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 1.3 MB/s | 99 kB 00:00 2024-09-15T23:03:47.319 INFO:teuthology.orchestra.run.smithi062.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-09-15T23:03:47.337 INFO:teuthology.orchestra.run.smithi148.stdout:(11/156): cephfs-mirror-19.1.1-326.g8921c8ef.el 2.4 MB/s | 228 kB 00:00 2024-09-15T23:03:47.344 INFO:teuthology.orchestra.run.smithi067.stdout:(26/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-15T23:03:47.394 INFO:teuthology.orchestra.run.smithi067.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 3.3 MB/s | 170 kB 00:00 2024-09-15T23:03:47.402 INFO:teuthology.orchestra.run.smithi062.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-09-15T23:03:47.444 INFO:teuthology.orchestra.run.smithi062.stdout:(48/156): python3-cryptography-36.0.1-4.el9.x86 3.4 MB/s | 1.2 MB 00:00 2024-09-15T23:03:47.445 INFO:teuthology.orchestra.run.smithi067.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 638 kB/s | 32 kB 00:00 2024-09-15T23:03:47.495 INFO:teuthology.orchestra.run.smithi067.stdout:(29/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.9 MB/s | 146 kB 00:00 2024-09-15T23:03:47.503 INFO:teuthology.orchestra.run.smithi148.stdout:(12/156): ceph-osd-19.1.1-326.g8921c8ef.el9.x86 4.7 MB/s | 17 MB 00:03 2024-09-15T23:03:47.520 INFO:teuthology.orchestra.run.smithi148.stdout:(13/156): cephfs-shell-19.1.1-326.g8921c8ef.el9 147 kB/s | 27 kB 00:00 2024-09-15T23:03:47.571 INFO:teuthology.orchestra.run.smithi148.stdout:(14/156): libcephfs-devel-19.1.1-326.g8921c8ef. 474 kB/s | 32 kB 00:00 2024-09-15T23:03:47.577 INFO:teuthology.orchestra.run.smithi062.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 801 kB/s | 107 kB 00:00 2024-09-15T23:03:47.619 INFO:teuthology.orchestra.run.smithi062.stdout:(50/156): elfutils-libelf-devel-0.191-4.el9.x86 522 kB/s | 22 kB 00:00 2024-09-15T23:03:47.620 INFO:teuthology.orchestra.run.smithi067.stdout:(30/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 10 MB/s | 3.1 MB 00:00 2024-09-15T23:03:47.621 INFO:teuthology.orchestra.run.smithi148.stdout:(15/156): libcephsqlite-19.1.1-326.g8921c8ef.el 3.2 MB/s | 163 kB 00:00 2024-09-15T23:03:47.654 INFO:teuthology.orchestra.run.smithi062.stdout:(51/156): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 689 kB 00:00 2024-09-15T23:03:47.655 INFO:teuthology.orchestra.run.smithi148.stdout:(16/156): libcephfs2-19.1.1-326.g8921c8ef.el9.x 5.5 MB/s | 751 kB 00:00 2024-09-15T23:03:47.680 INFO:teuthology.orchestra.run.smithi148.stdout:(17/156): librados-devel-19.1.1-326.g8921c8ef.e 2.1 MB/s | 127 kB 00:00 2024-09-15T23:03:47.695 INFO:teuthology.orchestra.run.smithi062.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 788 kB/s | 33 kB 00:00 2024-09-15T23:03:47.729 INFO:teuthology.orchestra.run.smithi062.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-09-15T23:03:47.764 INFO:teuthology.orchestra.run.smithi148.stdout:(18/156): libradosstriper1-19.1.1-326.g8921c8ef 4.6 MB/s | 506 kB 00:00 2024-09-15T23:03:47.773 INFO:teuthology.orchestra.run.smithi062.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 438 kB/s | 18 kB 00:00 2024-09-15T23:03:47.814 INFO:teuthology.orchestra.run.smithi148.stdout:(19/156): python3-ceph-argparse-19.1.1-326.g892 901 kB/s | 45 kB 00:00 2024-09-15T23:03:47.821 INFO:teuthology.orchestra.run.smithi062.stdout:(55/156): bison-3.7.4-5.el9.x86_64.rpm 2.2 MB/s | 943 kB 00:00 2024-09-15T23:03:47.846 INFO:teuthology.orchestra.run.smithi062.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-09-15T23:03:47.881 INFO:teuthology.orchestra.run.smithi148.stdout:(20/156): python3-ceph-common-19.1.1-326.g8921c 2.0 MB/s | 136 kB 00:00 2024-09-15T23:03:47.896 INFO:teuthology.orchestra.run.smithi062.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 906 kB/s | 45 kB 00:00 2024-09-15T23:03:47.922 INFO:teuthology.orchestra.run.smithi062.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-09-15T23:03:47.940 INFO:teuthology.orchestra.run.smithi148.stdout:(21/156): python3-cephfs-19.1.1-326.g8921c8ef.e 2.7 MB/s | 161 kB 00:00 2024-09-15T23:03:48.003 INFO:teuthology.orchestra.run.smithi067.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 7.0 MB/s | 3.6 MB 00:00 2024-09-15T23:03:48.015 INFO:teuthology.orchestra.run.smithi148.stdout:(22/156): python3-rados-19.1.1-326.g8921c8ef.el 4.2 MB/s | 320 kB 00:00 2024-09-15T23:03:48.070 INFO:teuthology.orchestra.run.smithi067.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 3.7 MB/s | 249 kB 00:00 2024-09-15T23:03:48.099 INFO:teuthology.orchestra.run.smithi148.stdout:(23/156): python3-rbd-19.1.1-326.g8921c8ef.el9. 3.5 MB/s | 300 kB 00:00 2024-09-15T23:03:48.121 INFO:teuthology.orchestra.run.smithi067.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 988 kB/s | 49 kB 00:00 2024-09-15T23:03:48.147 INFO:teuthology.orchestra.run.smithi062.stdout:(59/156): libtool-2.4.6-46.el9.x86_64.rpm 2.5 MB/s | 581 kB 00:00 2024-09-15T23:03:48.149 INFO:teuthology.orchestra.run.smithi148.stdout:(24/156): python3-rgw-19.1.1-326.g8921c8ef.el9. 2.0 MB/s | 99 kB 00:00 2024-09-15T23:03:48.171 INFO:teuthology.orchestra.run.smithi067.stdout:(34/156): ceph-prometheus-alerts-19.1.1-326.g89 328 kB/s | 16 kB 00:00 2024-09-15T23:03:48.199 INFO:teuthology.orchestra.run.smithi148.stdout:(25/156): rbd-fuse-19.1.1-326.g8921c8ef.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-15T23:03:48.215 INFO:teuthology.orchestra.run.smithi062.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 2.0 MB/s | 662 kB 00:00 2024-09-15T23:03:48.264 INFO:teuthology.orchestra.run.smithi062.stdout:(61/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 976 kB/s | 47 kB 00:00 2024-09-15T23:03:48.289 INFO:teuthology.orchestra.run.smithi062.stdout:(62/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 172 kB/s | 24 kB 00:00 2024-09-15T23:03:48.296 INFO:teuthology.orchestra.run.smithi067.stdout:(35/156): ceph-mgr-diskprediction-local-19.1.1- 11 MB/s | 7.4 MB 00:00 2024-09-15T23:03:48.321 INFO:teuthology.orchestra.run.smithi067.stdout:(36/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 1.9 MB/s | 285 kB 00:00 2024-09-15T23:03:48.372 INFO:teuthology.orchestra.run.smithi067.stdout:(37/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 366 kB/s | 18 kB 00:00 2024-09-15T23:03:48.389 INFO:teuthology.orchestra.run.smithi062.stdout:(63/156): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-09-15T23:03:48.397 INFO:teuthology.orchestra.run.smithi067.stdout:(38/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 7.5 MB/s | 765 kB 00:00 2024-09-15T23:03:48.416 INFO:teuthology.orchestra.run.smithi148.stdout:(26/156): librgw2-19.1.1-326.g8921c8ef.el9.x86_ 7.3 MB/s | 5.3 MB 00:00 2024-09-15T23:03:48.423 INFO:teuthology.orchestra.run.smithi062.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2024-09-15T23:03:48.465 INFO:teuthology.orchestra.run.smithi062.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 512 kB/s | 38 kB 00:00 2024-09-15T23:03:48.467 INFO:teuthology.orchestra.run.smithi148.stdout:(27/156): rbd-nbd-19.1.1-326.g8921c8ef.el9.x86_ 3.3 MB/s | 170 kB 00:00 2024-09-15T23:03:48.517 INFO:teuthology.orchestra.run.smithi148.stdout:(28/156): ceph-grafana-dashboards-19.1.1-326.g8 640 kB/s | 32 kB 00:00 2024-09-15T23:03:48.531 INFO:teuthology.orchestra.run.smithi067.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 503 kB/s | 80 kB 00:00 2024-09-15T23:03:48.567 INFO:teuthology.orchestra.run.smithi148.stdout:(29/156): ceph-mgr-cephadm-19.1.1-326.g8921c8ef 2.9 MB/s | 146 kB 00:00 2024-09-15T23:03:48.606 INFO:teuthology.orchestra.run.smithi067.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-15T23:03:48.626 INFO:teuthology.orchestra.run.smithi148.stdout:(30/156): rbd-mirror-19.1.1-326.g8921c8ef.el9.x 7.3 MB/s | 3.1 MB 00:00 2024-09-15T23:03:48.639 INFO:teuthology.orchestra.run.smithi067.stdout:(41/156): libgfortran-11.5.0-2.el9.x86_64.rpm 3.2 MB/s | 801 kB 00:00 2024-09-15T23:03:48.673 INFO:teuthology.orchestra.run.smithi067.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-09-15T23:03:48.682 INFO:teuthology.orchestra.run.smithi062.stdout:(66/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.0 MB/s | 3.0 MB 00:00 2024-09-15T23:03:48.739 INFO:teuthology.orchestra.run.smithi067.stdout:(43/156): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-09-15T23:03:48.764 INFO:teuthology.orchestra.run.smithi067.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-09-15T23:03:48.789 INFO:teuthology.orchestra.run.smithi067.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-09-15T23:03:48.815 INFO:teuthology.orchestra.run.smithi067.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-15T23:03:48.840 INFO:teuthology.orchestra.run.smithi067.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-09-15T23:03:48.968 INFO:teuthology.orchestra.run.smithi148.stdout:(31/156): ceph-mgr-dashboard-19.1.1-326.g8921c8 8.9 MB/s | 3.6 MB 00:00 2024-09-15T23:03:49.026 INFO:teuthology.orchestra.run.smithi148.stdout:(32/156): ceph-mgr-modules-core-19.1.1-326.g892 4.2 MB/s | 249 kB 00:00 2024-09-15T23:03:49.077 INFO:teuthology.orchestra.run.smithi148.stdout:(33/156): ceph-mgr-rook-19.1.1-326.g8921c8ef.el 990 kB/s | 49 kB 00:00 2024-09-15T23:03:49.124 INFO:teuthology.orchestra.run.smithi067.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 689 kB 00:00 2024-09-15T23:03:49.127 INFO:teuthology.orchestra.run.smithi148.stdout:(34/156): ceph-prometheus-alerts-19.1.1-326.g89 330 kB/s | 16 kB 00:00 2024-09-15T23:03:49.132 INFO:teuthology.orchestra.run.smithi062.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-09-15T23:03:49.194 INFO:teuthology.orchestra.run.smithi148.stdout:(35/156): ceph-volume-19.1.1-326.g8921c8ef.el9. 4.2 MB/s | 285 kB 00:00 2024-09-15T23:03:49.294 INFO:teuthology.orchestra.run.smithi148.stdout:(36/156): cephadm-19.1.1-326.g8921c8ef.el9.noar 7.5 MB/s | 765 kB 00:00 2024-09-15T23:03:49.336 INFO:teuthology.orchestra.run.smithi148.stdout:(37/156): cephfs-top-19.1.1-326.g8921c8ef.el9.n 440 kB/s | 18 kB 00:00 2024-09-15T23:03:49.382 INFO:teuthology.orchestra.run.smithi062.stdout:(68/156): postgresql-contrib-13.16-1.el9.x86_64 3.6 MB/s | 910 kB 00:00 2024-09-15T23:03:49.461 INFO:teuthology.orchestra.run.smithi148.stdout:(38/156): ceph-mgr-diskprediction-local-19.1.1- 8.9 MB/s | 7.4 MB 00:00 2024-09-15T23:03:49.466 INFO:teuthology.orchestra.run.smithi062.stdout:(69/156): postgresql-private-libs-13.16-1.el9.x 1.6 MB/s | 137 kB 00:00 2024-09-15T23:03:49.549 INFO:teuthology.orchestra.run.smithi067.stdout:(49/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 12 MB/s | 49 MB 00:04 2024-09-15T23:03:49.566 INFO:teuthology.orchestra.run.smithi067.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 242 kB/s | 107 kB 00:00 2024-09-15T23:03:49.578 INFO:teuthology.orchestra.run.smithi148.stdout:(39/156): fuse-2.9.9-16.el9.x86_64.rpm 330 kB/s | 80 kB 00:00 2024-09-15T23:03:49.600 INFO:teuthology.orchestra.run.smithi067.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-09-15T23:03:49.650 INFO:teuthology.orchestra.run.smithi067.stdout:(52/156): elfutils-libelf-devel-0.191-4.el9.x86 217 kB/s | 22 kB 00:00 2024-09-15T23:03:49.683 INFO:teuthology.orchestra.run.smithi067.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 392 kB/s | 33 kB 00:00 2024-09-15T23:03:49.697 INFO:teuthology.orchestra.run.smithi148.stdout:(40/156): libquadmath-11.5.0-2.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2024-09-15T23:03:49.709 INFO:teuthology.orchestra.run.smithi067.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 2.2 MB/s | 313 kB 00:00 2024-09-15T23:03:49.734 INFO:teuthology.orchestra.run.smithi067.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-09-15T23:03:49.759 INFO:teuthology.orchestra.run.smithi067.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-09-15T23:03:49.826 INFO:teuthology.orchestra.run.smithi067.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 676 kB/s | 45 kB 00:00 2024-09-15T23:03:49.845 INFO:teuthology.orchestra.run.smithi148.stdout:(41/156): libgfortran-11.5.0-2.el9.x86_64.rpm 2.0 MB/s | 801 kB 00:00 2024-09-15T23:03:49.860 INFO:teuthology.orchestra.run.smithi067.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-15T23:03:49.908 INFO:teuthology.orchestra.run.smithi062.stdout:(70/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-09-15T23:03:49.946 INFO:teuthology.orchestra.run.smithi148.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-15T23:03:50.008 INFO:teuthology.orchestra.run.smithi062.stdout:(71/156): openblas-openmp-0.3.26-2.el9.x86_64.r 3.1 MB/s | 4.9 MB 00:01 2024-09-15T23:03:50.013 INFO:teuthology.orchestra.run.smithi148.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-09-15T23:03:50.058 INFO:teuthology.orchestra.run.smithi062.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-09-15T23:03:50.060 INFO:teuthology.orchestra.run.smithi067.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-09-15T23:03:50.085 INFO:teuthology.orchestra.run.smithi067.stdout:(60/156): libtool-2.4.6-46.el9.x86_64.rpm 2.5 MB/s | 581 kB 00:00 2024-09-15T23:03:50.088 INFO:teuthology.orchestra.run.smithi148.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-09-15T23:03:50.111 INFO:teuthology.orchestra.run.smithi067.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 488 kB/s | 24 kB 00:00 2024-09-15T23:03:50.136 INFO:teuthology.orchestra.run.smithi067.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-09-15T23:03:50.172 INFO:teuthology.orchestra.run.smithi148.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-09-15T23:03:50.219 INFO:teuthology.orchestra.run.smithi067.stdout:(63/156): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-09-15T23:03:50.234 INFO:teuthology.orchestra.run.smithi062.stdout:(73/156): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-09-15T23:03:50.245 INFO:teuthology.orchestra.run.smithi067.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-15T23:03:50.264 INFO:teuthology.orchestra.run.smithi148.stdout:(46/156): python3-cryptography-36.0.1-4.el9.x86 2.9 MB/s | 1.2 MB 00:00 2024-09-15T23:03:50.284 INFO:teuthology.orchestra.run.smithi062.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-09-15T23:03:50.295 INFO:teuthology.orchestra.run.smithi067.stdout:(65/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.6 MB/s | 3.0 MB 00:00 2024-09-15T23:03:50.306 INFO:teuthology.orchestra.run.smithi148.stdout:(47/156): python3-urllib3-1.26.5-6.el9.noarch.r 1.6 MB/s | 215 kB 00:00 2024-09-15T23:03:50.320 INFO:teuthology.orchestra.run.smithi067.stdout:(66/156): openblas-0.3.26-2.el9.x86_64.rpm 383 kB/s | 38 kB 00:00 2024-09-15T23:03:50.367 INFO:teuthology.orchestra.run.smithi062.stdout:(75/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-15T23:03:50.451 INFO:teuthology.orchestra.run.smithi062.stdout:(76/156): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-09-15T23:03:50.502 INFO:teuthology.orchestra.run.smithi062.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-09-15T23:03:50.552 INFO:teuthology.orchestra.run.smithi062.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-09-15T23:03:50.606 INFO:teuthology.orchestra.run.smithi148.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 689 kB 00:00 2024-09-15T23:03:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-15T23:03:50.657 INFO:teuthology.orchestra.run.smithi148.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-09-15T23:03:50.682 INFO:teuthology.orchestra.run.smithi148.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-09-15T23:03:50.712 INFO:teuthology.orchestra.run.smithi067.stdout:(67/156): postgresql-13.16-1.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2024-09-15T23:03:50.724 INFO:teuthology.orchestra.run.smithi148.stdout:(51/156): elfutils-libelf-devel-0.191-4.el9.x86 329 kB/s | 22 kB 00:00 2024-09-15T23:03:50.736 INFO:teuthology.orchestra.run.smithi062.stdout:(80/156): postgresql-server-13.16-1.el9.x86_64. 4.5 MB/s | 5.8 MB 00:01 2024-09-15T23:03:50.749 INFO:teuthology.orchestra.run.smithi148.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-09-15T23:03:50.761 INFO:teuthology.orchestra.run.smithi062.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 277 kB/s | 35 kB 00:00 2024-09-15T23:03:50.791 INFO:teuthology.orchestra.run.smithi148.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 487 kB/s | 33 kB 00:00 2024-09-15T23:03:50.845 INFO:teuthology.orchestra.run.smithi062.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-09-15T23:03:50.858 INFO:teuthology.orchestra.run.smithi148.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 274 kB/s | 18 kB 00:00 2024-09-15T23:03:50.917 INFO:teuthology.orchestra.run.smithi148.stdout:(55/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-09-15T23:03:50.929 INFO:teuthology.orchestra.run.smithi062.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-09-15T23:03:50.942 INFO:teuthology.orchestra.run.smithi148.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 133 kB/s | 11 kB 00:00 2024-09-15T23:03:50.946 INFO:teuthology.orchestra.run.smithi067.stdout:(68/156): postgresql-contrib-13.16-1.el9.x86_64 3.8 MB/s | 910 kB 00:00 2024-09-15T23:03:50.975 INFO:teuthology.orchestra.run.smithi148.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-09-15T23:03:50.987 INFO:teuthology.orchestra.run.smithi062.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-09-15T23:03:51.000 INFO:teuthology.orchestra.run.smithi148.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 775 kB/s | 45 kB 00:00 2024-09-15T23:03:51.014 INFO:teuthology.orchestra.run.smithi067.stdout:(69/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-09-15T23:03:51.037 INFO:teuthology.orchestra.run.smithi062.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-09-15T23:03:51.038 INFO:teuthology.orchestra.run.smithi067.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 1.5 MB/s | 137 kB 00:00 2024-09-15T23:03:51.051 INFO:teuthology.orchestra.run.smithi148.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-09-15T23:03:51.121 INFO:teuthology.orchestra.run.smithi062.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-09-15T23:03:51.180 INFO:teuthology.orchestra.run.smithi062.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-09-15T23:03:51.263 INFO:teuthology.orchestra.run.smithi062.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 3.3 MB/s | 279 kB 00:00 2024-09-15T23:03:51.271 INFO:teuthology.orchestra.run.smithi067.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-09-15T23:03:51.305 INFO:teuthology.orchestra.run.smithi062.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-15T23:03:51.347 INFO:teuthology.orchestra.run.smithi067.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-09-15T23:03:51.348 INFO:teuthology.orchestra.run.smithi062.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 54 kB 00:00 2024-09-15T23:03:51.414 INFO:teuthology.orchestra.run.smithi067.stdout:(73/156): openblas-openmp-0.3.26-2.el9.x86_64.r 4.2 MB/s | 4.9 MB 00:01 2024-09-15T23:03:51.447 INFO:teuthology.orchestra.run.smithi062.stdout:(91/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.2 MB/s | 5.1 MB 00:00 2024-09-15T23:03:51.472 INFO:teuthology.orchestra.run.smithi067.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-09-15T23:03:51.485 INFO:teuthology.orchestra.run.smithi148.stdout:(60/156): ceph-test-19.1.1-326.g8921c8ef.el9.x8 9.6 MB/s | 49 MB 00:05 2024-09-15T23:03:51.489 INFO:teuthology.orchestra.run.smithi062.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 997 kB/s | 42 kB 00:00 2024-09-15T23:03:51.510 INFO:teuthology.orchestra.run.smithi148.stdout:(61/156): libuuid-devel-2.37.4-20.el9.x86_64.rp 53 kB/s | 24 kB 00:00 2024-09-15T23:03:51.535 INFO:teuthology.orchestra.run.smithi148.stdout:(62/156): libtool-2.4.6-46.el9.x86_64.rpm 1.1 MB/s | 581 kB 00:00 2024-09-15T23:03:51.548 INFO:teuthology.orchestra.run.smithi062.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 757 kB/s | 44 kB 00:00 2024-09-15T23:03:51.556 INFO:teuthology.orchestra.run.smithi067.stdout:(75/156): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-15T23:03:51.569 INFO:teuthology.orchestra.run.smithi148.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-15T23:03:51.594 INFO:teuthology.orchestra.run.smithi148.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-09-15T23:03:51.606 INFO:teuthology.orchestra.run.smithi062.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-15T23:03:51.628 INFO:teuthology.orchestra.run.smithi148.stdout:(65/156): openblas-0.3.26-2.el9.x86_64.rpm 658 kB/s | 38 kB 00:00 2024-09-15T23:03:51.639 INFO:teuthology.orchestra.run.smithi067.stdout:(76/156): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-09-15T23:03:51.657 INFO:teuthology.orchestra.run.smithi062.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 719 kB/s | 36 kB 00:00 2024-09-15T23:03:51.698 INFO:teuthology.orchestra.run.smithi067.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 442 kB/s | 26 kB 00:00 2024-09-15T23:03:51.707 INFO:teuthology.orchestra.run.smithi062.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-09-15T23:03:51.748 INFO:teuthology.orchestra.run.smithi067.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-09-15T23:03:51.761 INFO:teuthology.orchestra.run.smithi148.stdout:(66/156): openssl-devel-3.2.2-6.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-09-15T23:03:51.767 INFO:teuthology.orchestra.run.smithi062.stdout:(97/156): socat-1.7.4.1-6.el9.x86_64.rpm 5.1 MB/s | 304 kB 00:00 2024-09-15T23:03:51.787 INFO:teuthology.orchestra.run.smithi148.stdout:(67/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 157 kB/s | 47 kB 00:00 2024-09-15T23:03:51.815 INFO:teuthology.orchestra.run.smithi067.stdout:(79/156): postgresql-server-13.16-1.el9.x86_64. 7.2 MB/s | 5.8 MB 00:00 2024-09-15T23:03:51.824 INFO:teuthology.orchestra.run.smithi062.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 995 kB/s | 58 kB 00:00 2024-09-15T23:03:51.837 INFO:teuthology.orchestra.run.smithi148.stdout:(68/156): postgresql-13.16-1.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-09-15T23:03:51.840 INFO:teuthology.orchestra.run.smithi067.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-09-15T23:03:51.865 INFO:teuthology.orchestra.run.smithi067.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-09-15T23:03:51.891 INFO:teuthology.orchestra.run.smithi062.stdout:(99/156): python3-babel-2.9.1-2.el9.noarch.rpm 3.3 MB/s | 6.0 MB 00:01 2024-09-15T23:03:51.904 INFO:teuthology.orchestra.run.smithi148.stdout:(69/156): openblas-openmp-0.3.26-2.el9.x86_64.r 16 MB/s | 4.9 MB 00:00 2024-09-15T23:03:51.917 INFO:teuthology.orchestra.run.smithi062.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.0 MB/s | 96 kB 00:00 2024-09-15T23:03:51.924 INFO:teuthology.orchestra.run.smithi067.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-09-15T23:03:51.929 INFO:teuthology.orchestra.run.smithi148.stdout:(70/156): postgresql-private-libs-13.16-1.el9.x 1.5 MB/s | 137 kB 00:00 2024-09-15T23:03:51.942 INFO:teuthology.orchestra.run.smithi062.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-15T23:03:51.967 INFO:teuthology.orchestra.run.smithi062.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 895 kB/s | 45 kB 00:00 2024-09-15T23:03:51.974 INFO:teuthology.orchestra.run.smithi067.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-09-15T23:03:51.996 INFO:teuthology.orchestra.run.smithi148.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-09-15T23:03:52.017 INFO:teuthology.orchestra.run.smithi062.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-09-15T23:03:52.025 INFO:teuthology.orchestra.run.smithi067.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-15T23:03:52.068 INFO:teuthology.orchestra.run.smithi062.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 11 MB/s | 548 kB 00:00 2024-09-15T23:03:52.071 INFO:teuthology.orchestra.run.smithi148.stdout:(72/156): postgresql-server-13.16-1.el9.x86_64. 35 MB/s | 5.8 MB 00:00 2024-09-15T23:03:52.075 INFO:teuthology.orchestra.run.smithi067.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 848 kB/s | 42 kB 00:00 2024-09-15T23:03:52.097 INFO:teuthology.orchestra.run.smithi148.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 872 kB/s | 87 kB 00:00 2024-09-15T23:03:52.109 INFO:teuthology.orchestra.run.smithi062.stdout:(105/156): gperftools-libs-2.9.1-3.el9.x86_64.r 7.3 MB/s | 308 kB 00:00 2024-09-15T23:03:52.134 INFO:teuthology.orchestra.run.smithi067.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-09-15T23:03:52.134 INFO:teuthology.orchestra.run.smithi062.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 775 kB/s | 19 kB 00:00 2024-09-15T23:03:52.147 INFO:teuthology.orchestra.run.smithi148.stdout:(74/156): python3-colorama-0.4.6-3.el9.noarch.r 1.2 MB/s | 61 kB 00:00 2024-09-15T23:03:52.177 INFO:teuthology.orchestra.run.smithi062.stdout:(107/156): protobuf-compiler-3.14.0-13.el9.x86_ 4.0 MB/s | 863 kB 00:00 2024-09-15T23:03:52.184 INFO:teuthology.orchestra.run.smithi067.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-09-15T23:03:52.206 INFO:teuthology.orchestra.run.smithi148.stdout:(75/156): python3-devel-3.9.19-8.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-09-15T23:03:52.227 INFO:teuthology.orchestra.run.smithi062.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-09-15T23:03:52.242 INFO:teuthology.orchestra.run.smithi067.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-09-15T23:03:52.252 INFO:teuthology.orchestra.run.smithi062.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-15T23:03:52.281 INFO:teuthology.orchestra.run.smithi148.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 6.0 MB 00:00 2024-09-15T23:03:52.286 INFO:teuthology.orchestra.run.smithi062.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-15T23:03:52.301 INFO:teuthology.orchestra.run.smithi067.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 869 kB/s | 51 kB 00:00 2024-09-15T23:03:52.306 INFO:teuthology.orchestra.run.smithi148.stdout:(77/156): python3-jinja2-2.11.3-6.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-09-15T23:03:52.319 INFO:teuthology.orchestra.run.smithi062.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.4 MB/s | 151 kB 00:00 2024-09-15T23:03:52.348 INFO:teuthology.orchestra.run.smithi148.stdout:(78/156): postgresql-contrib-13.16-1.el9.x86_64 1.6 MB/s | 910 kB 00:00 2024-09-15T23:03:52.351 INFO:teuthology.orchestra.run.smithi067.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-09-15T23:03:52.373 INFO:teuthology.orchestra.run.smithi148.stdout:(79/156): python3-jsonpatch-1.21-16.el9.noarch. 278 kB/s | 26 kB 00:00 2024-09-15T23:03:52.387 INFO:teuthology.orchestra.run.smithi062.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-15T23:03:52.398 INFO:teuthology.orchestra.run.smithi148.stdout:(80/156): python3-jsonpointer-2.0-4.el9.noarch. 208 kB/s | 19 kB 00:00 2024-09-15T23:03:52.453 INFO:teuthology.orchestra.run.smithi062.stdout:(113/156): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-15T23:03:52.457 INFO:teuthology.orchestra.run.smithi148.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 598 kB/s | 35 kB 00:00 2024-09-15T23:03:52.479 INFO:teuthology.orchestra.run.smithi062.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-15T23:03:52.504 INFO:teuthology.orchestra.run.smithi062.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-15T23:03:52.510 INFO:teuthology.orchestra.run.smithi067.stdout:(91/156): python3-babel-2.9.1-2.el9.noarch.rpm 5.1 MB/s | 6.0 MB 00:01 2024-09-15T23:03:52.516 INFO:teuthology.orchestra.run.smithi148.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-09-15T23:03:52.529 INFO:teuthology.orchestra.run.smithi062.stdout:(116/156): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-09-15T23:03:52.541 INFO:teuthology.orchestra.run.smithi148.stdout:(83/156): python3-mako-1.1.4-6.el9.noarch.rpm 895 kB/s | 172 kB 00:00 2024-09-15T23:03:52.554 INFO:teuthology.orchestra.run.smithi062.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 282 kB/s | 14 kB 00:00 2024-09-15T23:03:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 835 kB/s | 42 kB 00:00 2024-09-15T23:03:52.580 INFO:teuthology.orchestra.run.smithi062.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-15T23:03:52.613 INFO:teuthology.orchestra.run.smithi062.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-09-15T23:03:52.616 INFO:teuthology.orchestra.run.smithi148.stdout:(84/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-09-15T23:03:52.627 INFO:teuthology.orchestra.run.smithi067.stdout:(93/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.5 MB/s | 5.1 MB 00:00 2024-09-15T23:03:52.639 INFO:teuthology.orchestra.run.smithi062.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.0 MB/s | 303 kB 00:00 2024-09-15T23:03:52.645 INFO:teuthology.orchestra.run.smithi148.stdout:(85/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-09-15T23:03:52.653 INFO:teuthology.orchestra.run.smithi067.stdout:(94/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 481 kB/s | 44 kB 00:00 2024-09-15T23:03:52.664 INFO:teuthology.orchestra.run.smithi062.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 4.3 MB/s | 220 kB 00:00 2024-09-15T23:03:52.667 INFO:teuthology.orchestra.run.smithi148.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 614 kB/s | 77 kB 00:00 2024-09-15T23:03:52.686 INFO:teuthology.orchestra.run.smithi067.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-15T23:03:52.692 INFO:teuthology.orchestra.run.smithi148.stdout:(87/156): python3-prettytable-0.7.2-27.el9.noar 560 kB/s | 42 kB 00:00 2024-09-15T23:03:52.697 INFO:teuthology.orchestra.run.smithi062.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-09-15T23:03:52.717 INFO:teuthology.orchestra.run.smithi148.stdout:(88/156): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-09-15T23:03:52.719 INFO:teuthology.orchestra.run.smithi067.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 537 kB/s | 36 kB 00:00 2024-09-15T23:03:52.724 INFO:teuthology.orchestra.run.smithi062.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 427 kB/s | 11 kB 00:00 2024-09-15T23:03:52.743 INFO:teuthology.orchestra.run.smithi148.stdout:(89/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-09-15T23:03:52.745 INFO:teuthology.orchestra.run.smithi067.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-09-15T23:03:52.748 INFO:teuthology.orchestra.run.smithi062.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 704 kB/s | 18 kB 00:00 2024-09-15T23:03:52.768 INFO:teuthology.orchestra.run.smithi148.stdout:(90/156): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-09-15T23:03:52.773 INFO:teuthology.orchestra.run.smithi062.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 933 kB/s | 23 kB 00:00 2024-09-15T23:03:52.793 INFO:teuthology.orchestra.run.smithi148.stdout:(91/156): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-15T23:03:52.795 INFO:teuthology.orchestra.run.smithi067.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-15T23:03:52.798 INFO:teuthology.orchestra.run.smithi062.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 777 kB/s | 19 kB 00:00 2024-09-15T23:03:52.816 INFO:teuthology.orchestra.run.smithi062.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 1.1 MB/s | 20 kB 00:00 2024-09-15T23:03:52.818 INFO:teuthology.orchestra.run.smithi148.stdout:(92/156): python3-requests-oauthlib-1.3.0-12.el 713 kB/s | 54 kB 00:00 2024-09-15T23:03:52.820 INFO:teuthology.orchestra.run.smithi067.stdout:(99/156): socat-1.7.4.1-6.el9.x86_64.rpm 3.0 MB/s | 304 kB 00:00 2024-09-15T23:03:52.852 INFO:teuthology.orchestra.run.smithi148.stdout:(93/156): python3-toml-0.10.2-6.el9.noarch.rpm 712 kB/s | 42 kB 00:00 2024-09-15T23:03:52.856 INFO:teuthology.orchestra.run.smithi067.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.6 MB/s | 96 kB 00:00 2024-09-15T23:03:52.860 INFO:teuthology.orchestra.run.smithi062.stdout:(128/156): python3-grpcio-1.46.7-10.el9.x86_64. 9.3 MB/s | 2.0 MB 00:00 2024-09-15T23:03:52.875 INFO:teuthology.orchestra.run.smithi062.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 2.7 MB/s | 46 kB 00:00 2024-09-15T23:03:52.879 INFO:teuthology.orchestra.run.smithi067.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-15T23:03:52.905 INFO:teuthology.orchestra.run.smithi067.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-09-15T23:03:52.910 INFO:teuthology.orchestra.run.smithi148.stdout:(94/156): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-09-15T23:03:52.916 INFO:teuthology.orchestra.run.smithi062.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-09-15T23:03:52.930 INFO:teuthology.orchestra.run.smithi067.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-09-15T23:03:52.943 INFO:teuthology.orchestra.run.smithi062.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-09-15T23:03:52.959 INFO:teuthology.orchestra.run.smithi062.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-09-15T23:03:52.969 INFO:teuthology.orchestra.run.smithi148.stdout:(95/156): qatlib-service-24.02.0-1.el9.x86_64.r 616 kB/s | 36 kB 00:00 2024-09-15T23:03:52.993 INFO:teuthology.orchestra.run.smithi062.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-15T23:03:52.997 INFO:teuthology.orchestra.run.smithi067.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 8.1 MB/s | 548 kB 00:00 2024-09-15T23:03:53.018 INFO:teuthology.orchestra.run.smithi062.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-09-15T23:03:53.022 INFO:teuthology.orchestra.run.smithi067.stdout:(105/156): protobuf-compiler-3.14.0-13.el9.x86_ 7.2 MB/s | 863 kB 00:00 2024-09-15T23:03:53.027 INFO:teuthology.orchestra.run.smithi148.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 817 kB/s | 47 kB 00:00 2024-09-15T23:03:53.043 INFO:teuthology.orchestra.run.smithi062.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-09-15T23:03:53.064 INFO:teuthology.orchestra.run.smithi067.stdout:(106/156): gperftools-libs-2.9.1-3.el9.x86_64.r 4.5 MB/s | 308 kB 00:00 2024-09-15T23:03:53.068 INFO:teuthology.orchestra.run.smithi062.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-09-15T23:03:53.089 INFO:teuthology.orchestra.run.smithi067.stdout:(107/156): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-09-15T23:03:53.094 INFO:teuthology.orchestra.run.smithi062.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 610 kB/s | 31 kB 00:00 2024-09-15T23:03:53.106 INFO:teuthology.orchestra.run.smithi067.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.5 MB/s | 26 kB 00:00 2024-09-15T23:03:53.119 INFO:teuthology.orchestra.run.smithi062.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-09-15T23:03:53.123 INFO:teuthology.orchestra.run.smithi067.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 2.8 MB/s | 49 kB 00:00 2024-09-15T23:03:53.144 INFO:teuthology.orchestra.run.smithi062.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-15T23:03:53.157 INFO:teuthology.orchestra.run.smithi067.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-15T23:03:53.170 INFO:teuthology.orchestra.run.smithi062.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-09-15T23:03:53.190 INFO:teuthology.orchestra.run.smithi067.stdout:(111/156): luarocks-3.9.2-5.el9.noarch.rpm 4.5 MB/s | 151 kB 00:00 2024-09-15T23:03:53.195 INFO:teuthology.orchestra.run.smithi148.stdout:(97/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-09-15T23:03:53.195 INFO:teuthology.orchestra.run.smithi062.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-15T23:03:53.220 INFO:teuthology.orchestra.run.smithi148.stdout:(98/156): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-15T23:03:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:(142/156): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-15T23:03:53.245 INFO:teuthology.orchestra.run.smithi062.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-09-15T23:03:53.254 INFO:teuthology.orchestra.run.smithi148.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 995 kB/s | 58 kB 00:00 2024-09-15T23:03:53.274 INFO:teuthology.orchestra.run.smithi067.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-09-15T23:03:53.279 INFO:teuthology.orchestra.run.smithi148.stdout:(100/156): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.6 MB/s | 96 kB 00:00 2024-09-15T23:03:53.313 INFO:teuthology.orchestra.run.smithi148.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-15T23:03:53.332 INFO:teuthology.orchestra.run.smithi067.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-09-15T23:03:53.338 INFO:teuthology.orchestra.run.smithi148.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 764 kB/s | 45 kB 00:00 2024-09-15T23:03:53.358 INFO:teuthology.orchestra.run.smithi067.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-15T23:03:53.383 INFO:teuthology.orchestra.run.smithi067.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-15T23:03:53.404 INFO:teuthology.orchestra.run.smithi062.stdout:(144/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.9 MB/s | 16 MB 00:02 2024-09-15T23:03:53.408 INFO:teuthology.orchestra.run.smithi067.stdout:(116/156): python3-certifi-2023.05.07-4.el9.noa 562 kB/s | 14 kB 00:00 2024-09-15T23:03:53.429 INFO:teuthology.orchestra.run.smithi062.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 109 kB/s | 20 kB 00:00 2024-09-15T23:03:53.430 INFO:teuthology.orchestra.run.smithi148.stdout:(103/156): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-09-15T23:03:53.455 INFO:teuthology.orchestra.run.smithi148.stdout:(104/156): python3-wcwidth-0.2.5-8.el9.noarch.r 69 kB/s | 44 kB 00:00 2024-09-15T23:03:53.471 INFO:teuthology.orchestra.run.smithi062.stdout:(146/156): python3-werkzeug-2.0.3-3.el9.1.noarc 1.7 MB/s | 427 kB 00:00 2024-09-15T23:03:53.475 INFO:teuthology.orchestra.run.smithi067.stdout:(117/156): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-15T23:03:53.501 INFO:teuthology.orchestra.run.smithi067.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-09-15T23:03:53.521 INFO:teuthology.orchestra.run.smithi062.stdout:(147/156): re2-20211101-20.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2024-09-15T23:03:53.543 INFO:teuthology.orchestra.run.smithi067.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-09-15T23:03:53.547 INFO:teuthology.orchestra.run.smithi148.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 4.6 MB/s | 548 kB 00:00 2024-09-15T23:03:53.564 INFO:teuthology.orchestra.run.smithi062.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-15T23:03:53.577 INFO:teuthology.orchestra.run.smithi067.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-09-15T23:03:53.581 INFO:teuthology.orchestra.run.smithi148.stdout:(106/156): gperftools-libs-2.9.1-3.el9.x86_64.r 2.4 MB/s | 308 kB 00:00 2024-09-15T23:03:53.602 INFO:teuthology.orchestra.run.smithi067.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-15T23:03:53.606 INFO:teuthology.orchestra.run.smithi148.stdout:(107/156): grpc-data-1.46.7-10.el9.noarch.rpm 331 kB/s | 19 kB 00:00 2024-09-15T23:03:53.631 INFO:teuthology.orchestra.run.smithi148.stdout:(108/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.9 MB/s | 863 kB 00:00 2024-09-15T23:03:53.670 INFO:teuthology.orchestra.run.smithi148.stdout:(109/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-09-15T23:03:53.690 INFO:teuthology.orchestra.run.smithi148.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-09-15T23:03:53.715 INFO:teuthology.orchestra.run.smithi148.stdout:(111/156): liboath-2.6.7-2.el9.x86_64.rpm 580 kB/s | 49 kB 00:00 2024-09-15T23:03:53.741 INFO:teuthology.orchestra.run.smithi148.stdout:(112/156): luarocks-3.9.2-5.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-09-15T23:03:53.760 INFO:teuthology.orchestra.run.smithi067.stdout:(122/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 12 MB/s | 16 MB 00:01 2024-09-15T23:03:53.786 INFO:teuthology.orchestra.run.smithi067.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 787 kB/s | 144 kB 00:00 2024-09-15T23:03:53.806 INFO:teuthology.orchestra.run.smithi062.stdout:(149/156): librados2-19.1.1-326.g8921c8ef.el9.x 10 MB/s | 3.4 MB 00:00 2024-09-15T23:03:53.808 INFO:teuthology.orchestra.run.smithi148.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 8.1 MB/s | 548 kB 00:00 2024-09-15T23:03:53.819 INFO:teuthology.orchestra.run.smithi067.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 698 kB/s | 18 kB 00:00 2024-09-15T23:03:53.833 INFO:teuthology.orchestra.run.smithi148.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-15T23:03:53.845 INFO:teuthology.orchestra.run.smithi067.stdout:(125/156): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-09-15T23:03:53.858 INFO:teuthology.orchestra.run.smithi148.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-15T23:03:53.870 INFO:teuthology.orchestra.run.smithi067.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 484 kB/s | 23 kB 00:00 2024-09-15T23:03:53.883 INFO:teuthology.orchestra.run.smithi148.stdout:(116/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.9 MB/s | 839 kB 00:00 2024-09-15T23:03:53.895 INFO:teuthology.orchestra.run.smithi067.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-09-15T23:03:53.909 INFO:teuthology.orchestra.run.smithi148.stdout:(117/156): python3-certifi-2023.05.07-4.el9.noa 281 kB/s | 14 kB 00:00 2024-09-15T23:03:53.921 INFO:teuthology.orchestra.run.smithi067.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-09-15T23:03:53.934 INFO:teuthology.orchestra.run.smithi148.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-15T23:03:53.939 INFO:teuthology.orchestra.run.smithi062.stdout:(150/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 7.6 MB/s | 3.2 MB 00:00 2024-09-15T23:03:53.945 INFO:teuthology.orchestra.run.smithi067.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-15T23:03:53.976 INFO:teuthology.orchestra.run.smithi148.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-09-15T23:03:53.979 INFO:teuthology.orchestra.run.smithi067.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-15T23:03:54.010 INFO:teuthology.orchestra.run.smithi148.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-09-15T23:03:54.035 INFO:teuthology.orchestra.run.smithi148.stdout:(121/156): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-15T23:03:54.037 INFO:teuthology.orchestra.run.smithi067.stdout:(131/156): python3-grpcio-1.46.7-10.el9.x86_64. 4.4 MB/s | 2.0 MB 00:00 2024-09-15T23:03:54.063 INFO:teuthology.orchestra.run.smithi067.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 692 kB/s | 58 kB 00:00 2024-09-15T23:03:54.068 INFO:teuthology.orchestra.run.smithi148.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-09-15T23:03:54.081 INFO:teuthology.orchestra.run.smithi062.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 288 kB/s | 41 kB 00:00 2024-09-15T23:03:54.088 INFO:teuthology.orchestra.run.smithi067.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 660 kB/s | 16 kB 00:00 2024-09-15T23:03:54.113 INFO:teuthology.orchestra.run.smithi067.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-15T23:03:54.123 INFO:teuthology.orchestra.run.smithi062.stdout:(152/156): NetworkManager-1.49.90-1.el9.x86_64. 4.2 MB/s | 2.3 MB 00:00 2024-09-15T23:03:54.144 INFO:teuthology.orchestra.run.smithi148.stdout:(123/156): libarrow-9.0.0-11.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-09-15T23:03:54.147 INFO:teuthology.orchestra.run.smithi067.stdout:(135/156): python3-kubernetes-26.1.0-2.el9.noar 4.1 MB/s | 1.0 MB 00:00 2024-09-15T23:03:54.169 INFO:teuthology.orchestra.run.smithi148.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-09-15T23:03:54.172 INFO:teuthology.orchestra.run.smithi067.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-09-15T23:03:54.190 INFO:teuthology.orchestra.run.smithi062.stdout:(153/156): NetworkManager-tui-1.49.90-1.el9.x86 2.2 MB/s | 249 kB 00:00 2024-09-15T23:03:54.194 INFO:teuthology.orchestra.run.smithi148.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-09-15T23:03:54.197 INFO:teuthology.orchestra.run.smithi067.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 338 kB/s | 28 kB 00:00 2024-09-15T23:03:54.220 INFO:teuthology.orchestra.run.smithi148.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 463 kB/s | 23 kB 00:00 2024-09-15T23:03:54.222 INFO:teuthology.orchestra.run.smithi067.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-09-15T23:03:54.245 INFO:teuthology.orchestra.run.smithi148.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-09-15T23:03:54.248 INFO:teuthology.orchestra.run.smithi067.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-15T23:03:54.270 INFO:teuthology.orchestra.run.smithi148.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 394 kB/s | 20 kB 00:00 2024-09-15T23:03:54.273 INFO:teuthology.orchestra.run.smithi067.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-09-15T23:03:54.295 INFO:teuthology.orchestra.run.smithi148.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-15T23:03:54.298 INFO:teuthology.orchestra.run.smithi067.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-15T23:03:54.315 INFO:teuthology.orchestra.run.smithi062.stdout:(154/156): openssl-3.2.2-6.el9.x86_64.rpm 7.1 MB/s | 1.4 MB 00:00 2024-09-15T23:03:54.324 INFO:teuthology.orchestra.run.smithi067.stdout:(142/156): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-09-15T23:03:54.329 INFO:teuthology.orchestra.run.smithi148.stdout:(130/156): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-15T23:03:54.349 INFO:teuthology.orchestra.run.smithi067.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-15T23:03:54.357 INFO:teuthology.orchestra.run.smithi062.stdout:(155/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-09-15T23:03:54.362 INFO:teuthology.orchestra.run.smithi148.stdout:(131/156): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-09-15T23:03:54.374 INFO:teuthology.orchestra.run.smithi067.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-15T23:03:54.396 INFO:teuthology.orchestra.run.smithi148.stdout:(132/156): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-09-15T23:03:54.399 INFO:teuthology.orchestra.run.smithi067.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-09-15T23:03:54.421 INFO:teuthology.orchestra.run.smithi148.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 664 kB/s | 16 kB 00:00 2024-09-15T23:03:54.425 INFO:teuthology.orchestra.run.smithi067.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-15T23:03:54.447 INFO:teuthology.orchestra.run.smithi148.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-15T23:03:54.450 INFO:teuthology.orchestra.run.smithi067.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-09-15T23:03:54.485 INFO:teuthology.orchestra.run.smithi062.stdout:(156/156): NetworkManager-libnm-1.49.90-1.el9.x 2.7 MB/s | 1.8 MB 00:00 2024-09-15T23:03:54.488 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:03:54.488 INFO:teuthology.orchestra.run.smithi062.stdout:Total 17 MB/s | 226 MB 00:13 2024-09-15T23:03:54.525 INFO:teuthology.orchestra.run.smithi067.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-15T23:03:54.545 INFO:teuthology.orchestra.run.smithi148.stdout:(135/156): python3-grpcio-1.46.7-10.el9.x86_64. 3.8 MB/s | 2.0 MB 00:00 2024-09-15T23:03:54.656 INFO:teuthology.orchestra.run.smithi148.stdout:(136/156): python3-pecan-1.4.2-3.el9.noarch.rpm 929 kB/s | 272 kB 00:00 2024-09-15T23:03:54.692 INFO:teuthology.orchestra.run.smithi148.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 115 kB/s | 28 kB 00:00 2024-09-15T23:03:54.715 INFO:teuthology.orchestra.run.smithi148.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 528 kB/s | 31 kB 00:00 2024-09-15T23:03:54.732 INFO:teuthology.orchestra.run.smithi148.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.5 MB/s | 59 kB 00:00 2024-09-15T23:03:54.749 INFO:teuthology.orchestra.run.smithi148.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-09-15T23:03:54.767 INFO:teuthology.orchestra.run.smithi067.stdout:(149/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 10 MB/s | 3.2 MB 00:00 2024-09-15T23:03:54.774 INFO:teuthology.orchestra.run.smithi148.stdout:(141/156): python3-typing-extensions-4.12.2-2.e 1.7 MB/s | 74 kB 00:00 2024-09-15T23:03:54.799 INFO:teuthology.orchestra.run.smithi148.stdout:(142/156): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-09-15T23:03:54.817 INFO:teuthology.orchestra.run.smithi067.stdout:(150/156): NetworkManager-1.49.90-1.el9.x86_64. 8.0 MB/s | 2.3 MB 00:00 2024-09-15T23:03:54.825 INFO:teuthology.orchestra.run.smithi148.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-15T23:03:54.850 INFO:teuthology.orchestra.run.smithi148.stdout:(144/156): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-15T23:03:54.859 INFO:teuthology.orchestra.run.smithi067.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 980 kB/s | 41 kB 00:00 2024-09-15T23:03:54.867 INFO:teuthology.orchestra.run.smithi148.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 479 kB/s | 20 kB 00:00 2024-09-15T23:03:54.892 INFO:teuthology.orchestra.run.smithi148.stdout:(146/156): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-15T23:03:54.919 INFO:teuthology.orchestra.run.smithi148.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-09-15T23:03:54.926 INFO:teuthology.orchestra.run.smithi067.stdout:(152/156): librados2-19.1.1-326.g8921c8ef.el9.x 6.8 MB/s | 3.4 MB 00:00 2024-09-15T23:03:54.972 INFO:teuthology.orchestra.run.smithi067.stdout:(153/156): NetworkManager-tui-1.49.90-1.el9.x86 2.2 MB/s | 249 kB 00:00 2024-09-15T23:03:55.001 INFO:teuthology.orchestra.run.smithi148.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-15T23:03:55.069 INFO:teuthology.orchestra.run.smithi067.stdout:(154/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 25 MB/s | 2.4 MB 00:00 2024-09-15T23:03:55.136 INFO:teuthology.orchestra.run.smithi067.stdout:(155/156): NetworkManager-libnm-1.49.90-1.el9.x 4.9 MB/s | 1.8 MB 00:00 2024-09-15T23:03:55.243 INFO:teuthology.orchestra.run.smithi148.stdout:(149/156): librados2-19.1.1-326.g8921c8ef.el9.x 9.8 MB/s | 3.4 MB 00:00 2024-09-15T23:03:55.327 INFO:teuthology.orchestra.run.smithi148.stdout:(150/156): librbd1-19.1.1-326.g8921c8ef.el9.x86 7.7 MB/s | 3.2 MB 00:00 2024-09-15T23:03:55.377 INFO:teuthology.orchestra.run.smithi067.stdout:(156/156): openssl-3.2.2-6.el9.x86_64.rpm 3.0 MB/s | 1.4 MB 00:00 2024-09-15T23:03:55.384 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:03:55.384 INFO:teuthology.orchestra.run.smithi067.stdout:Total 16 MB/s | 226 MB 00:13 2024-09-15T23:03:55.435 INFO:teuthology.orchestra.run.smithi148.stdout:(151/156): NetworkManager-team-1.49.90-1.el9.x8 377 kB/s | 41 kB 00:00 2024-09-15T23:03:55.527 INFO:teuthology.orchestra.run.smithi148.stdout:(152/156): NetworkManager-1.49.90-1.el9.x86_64. 4.4 MB/s | 2.3 MB 00:00 2024-09-15T23:03:55.561 INFO:teuthology.orchestra.run.smithi148.stdout:(153/156): NetworkManager-tui-1.49.90-1.el9.x86 1.9 MB/s | 249 kB 00:00 2024-09-15T23:03:55.661 INFO:teuthology.orchestra.run.smithi148.stdout:(154/156): NetworkManager-libnm-1.49.90-1.el9.x 4.3 MB/s | 1.8 MB 00:00 2024-09-15T23:03:55.719 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-09-15T23:03:55.753 INFO:teuthology.orchestra.run.smithi148.stdout:(155/156): openssl-3.2.2-6.el9.x86_64.rpm 6.0 MB/s | 1.4 MB 00:00 2024-09-15T23:03:55.888 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-09-15T23:03:55.888 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-09-15T23:03:55.911 INFO:teuthology.orchestra.run.smithi148.stdout:(156/156): openssl-libs-3.2.2-6.el9.x86_64.rpm 6.8 MB/s | 2.4 MB 00:00 2024-09-15T23:03:55.918 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:03:55.918 INFO:teuthology.orchestra.run.smithi148.stdout:Total 16 MB/s | 226 MB 00:14 2024-09-15T23:03:56.648 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-09-15T23:03:56.815 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-09-15T23:03:56.815 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-09-15T23:03:57.170 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-09-15T23:03:57.322 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-09-15T23:03:57.322 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-09-15T23:03:58.168 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-09-15T23:03:58.169 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-09-15T23:03:58.719 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-09-15T23:03:58.719 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-09-15T23:03:59.120 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-09-15T23:03:59.120 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-09-15T23:04:04.834 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-09-15T23:04:04.930 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-15T23:04:05.022 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-15T23:04:05.089 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-15T23:04:05.130 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-15T23:04:05.155 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-15T23:04:05.207 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-15T23:04:05.403 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-09-15T23:04:05.506 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-15T23:04:05.545 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-15T23:04:05.556 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-15T23:04:05.607 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-15T23:04:05.641 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-15T23:04:05.647 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-09-15T23:04:05.664 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:04:05.665 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-15T23:04:05.709 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-15T23:04:05.723 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/164 2024-09-15T23:04:05.790 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:04:05.823 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:04:05.861 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-09-15T23:04:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:04:05.940 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-09-15T23:04:05.945 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:04:05.974 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-09-15T23:04:05.999 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-09-15T23:04:06.042 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-09-15T23:04:06.057 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-15T23:04:06.121 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-15T23:04:06.166 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:04:06.167 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-15T23:04:06.185 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:04:06.283 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:04:06.291 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:04:06.303 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:04:06.318 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:04:06.366 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:04:06.373 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-09-15T23:04:06.379 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:04:06.401 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:04:06.409 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:04:06.439 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:04:06.484 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:04:06.540 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:04:06.598 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-15T23:04:06.606 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:04:06.635 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:04:06.652 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-15T23:04:06.680 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:04:06.702 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:04:06.725 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:04:06.755 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:04:06.785 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:04:06.814 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:04:06.852 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-15T23:04:06.868 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:04:06.890 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:04:06.913 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:04:06.936 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/164 2024-09-15T23:04:06.958 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:04:06.994 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/164 2024-09-15T23:04:07.022 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:04:07.062 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:04:07.108 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:04:07.118 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:04:07.148 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:04:07.211 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 14/164 2024-09-15T23:04:07.233 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:04:07.233 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:04:07.258 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-09-15T23:04:07.311 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:04:07.354 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-15T23:04:07.424 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-15T23:04:07.449 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:04:07.450 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:04:07.506 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:04:07.519 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-15T23:04:07.551 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-15T23:04:07.585 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-09-15T23:04:07.595 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-15T23:04:07.671 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-15T23:04:07.717 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 17/164 2024-09-15T23:04:07.720 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-15T23:04:07.739 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:04:07.761 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-15T23:04:07.804 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-15T23:04:07.827 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:04:07.837 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-15T23:04:07.857 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-15T23:04:07.864 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-15T23:04:07.902 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:04:07.936 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-15T23:04:07.952 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-15T23:04:07.986 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-15T23:04:07.988 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-15T23:04:08.017 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-15T23:04:08.022 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-15T23:04:08.052 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-15T23:04:08.097 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-15T23:04:08.122 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 18/164 2024-09-15T23:04:08.140 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-15T23:04:08.148 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-15T23:04:08.187 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-15T23:04:08.189 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-15T23:04:08.232 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-15T23:04:08.238 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : re2-1:20211101-20.el9.x86_64 19/164 2024-09-15T23:04:08.239 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-15T23:04:08.260 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-15T23:04:08.272 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-15T23:04:08.299 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-15T23:04:08.335 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-15T23:04:08.340 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-09-15T23:04:08.370 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-15T23:04:08.383 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-15T23:04:08.400 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-09-15T23:04:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-15T23:04:08.435 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-09-15T23:04:08.462 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-15T23:04:08.463 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-15T23:04:08.507 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-15T23:04:08.508 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-09-15T23:04:08.561 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-09-15T23:04:08.586 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-15T23:04:08.600 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/164 2024-09-15T23:04:08.616 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-15T23:04:08.618 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-15T23:04:08.652 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/164 2024-09-15T23:04:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-15T23:04:08.670 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-15T23:04:08.685 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/164 2024-09-15T23:04:08.691 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-15T23:04:08.712 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/164 2024-09-15T23:04:08.763 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-15T23:04:08.782 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/164 2024-09-15T23:04:08.804 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-15T23:04:08.835 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/164 2024-09-15T23:04:08.872 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/164 2024-09-15T23:04:08.882 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-15T23:04:08.900 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/164 2024-09-15T23:04:08.980 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/164 2024-09-15T23:04:09.011 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 34/164 2024-09-15T23:04:09.028 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-15T23:04:09.063 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 35/164 2024-09-15T23:04:09.091 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-15T23:04:09.108 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 36/164 2024-09-15T23:04:09.177 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/164 2024-09-15T23:04:09.224 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/164 2024-09-15T23:04:09.233 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-15T23:04:09.298 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/164 2024-09-15T23:04:09.325 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-15T23:04:09.360 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-15T23:04:09.412 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-15T23:04:09.432 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/164 2024-09-15T23:04:09.449 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-15T23:04:09.484 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/164 2024-09-15T23:04:09.498 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-15T23:04:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-15T23:04:09.673 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-15T23:04:09.730 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-15T23:04:09.735 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-15T23:04:09.765 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-15T23:04:09.792 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-15T23:04:09.808 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-15T23:04:09.845 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-15T23:04:09.867 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-15T23:04:09.888 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-15T23:04:09.903 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-15T23:04:09.939 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-15T23:04:09.988 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-15T23:04:10.019 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/164 2024-09-15T23:04:10.022 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-15T23:04:10.049 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-15T23:04:10.065 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-15T23:04:10.096 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/164 2024-09-15T23:04:10.106 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-15T23:04:10.107 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-15T23:04:10.125 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/164 2024-09-15T23:04:10.163 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-15T23:04:10.168 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/164 2024-09-15T23:04:10.206 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 46/164 2024-09-15T23:04:10.236 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-15T23:04:10.248 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/164 2024-09-15T23:04:10.250 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-15T23:04:10.274 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-15T23:04:10.300 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-15T23:04:10.310 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-15T23:04:10.342 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-15T23:04:10.358 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-15T23:04:10.400 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-15T23:04:10.413 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/164 2024-09-15T23:04:10.445 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-15T23:04:10.450 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-15T23:04:10.476 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/164 2024-09-15T23:04:10.484 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-15T23:04:10.486 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-15T23:04:10.529 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-15T23:04:10.530 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/164 2024-09-15T23:04:10.557 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-15T23:04:10.582 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-15T23:04:10.602 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/164 2024-09-15T23:04:10.628 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-15T23:04:10.635 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 52/164 2024-09-15T23:04:10.671 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 53/164 2024-09-15T23:04:10.672 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-15T23:04:10.698 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-15T23:04:10.734 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 54/164 2024-09-15T23:04:10.770 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 55/164 2024-09-15T23:04:10.796 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-15T23:04:10.814 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 56/164 2024-09-15T23:04:10.830 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-15T23:04:10.864 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 57/164 2024-09-15T23:04:11.029 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-15T23:04:11.059 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-15T23:04:11.096 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-15T23:04:11.135 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-15T23:04:11.161 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-15T23:04:11.185 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-15T23:04:11.191 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-15T23:04:11.206 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 58/164 2024-09-15T23:04:11.237 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-15T23:04:11.265 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-15T23:04:11.280 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-15T23:04:11.282 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 59/164 2024-09-15T23:04:11.319 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-15T23:04:11.323 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/164 2024-09-15T23:04:11.349 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-15T23:04:11.383 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-15T23:04:11.400 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-15T23:04:11.416 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-15T23:04:11.427 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-09-15T23:04:11.442 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-15T23:04:11.459 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-15T23:04:11.466 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 62/164 2024-09-15T23:04:11.470 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-15T23:04:11.505 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-15T23:04:11.505 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-09-15T23:04:11.515 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-15T23:04:11.531 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 64/164 2024-09-15T23:04:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-15T23:04:11.556 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-09-15T23:04:11.584 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-15T23:04:11.613 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-15T23:04:11.614 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-15T23:04:11.630 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-09-15T23:04:11.648 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-15T23:04:11.665 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-15T23:04:11.681 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-15T23:04:11.703 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-15T23:04:11.715 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-15T23:04:11.749 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-15T23:04:11.758 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-09-15T23:04:11.788 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-09-15T23:04:11.813 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-15T23:04:11.815 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-09-15T23:04:11.838 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-15T23:04:11.858 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-09-15T23:04:11.872 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-15T23:04:11.896 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-15T23:04:11.897 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-09-15T23:04:11.927 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-15T23:04:11.929 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-09-15T23:04:11.959 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-09-15T23:04:11.993 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-09-15T23:04:12.027 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-09-15T23:04:12.034 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-15T23:04:12.068 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-09-15T23:04:12.087 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-15T23:04:12.103 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-09-15T23:04:12.207 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-09-15T23:04:12.250 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-09-15T23:04:12.290 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-09-15T23:04:12.393 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-09-15T23:04:12.458 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 82/164 2024-09-15T23:04:12.647 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-15T23:04:12.734 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-15T23:04:12.768 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-15T23:04:12.850 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-15T23:04:12.872 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-15T23:04:12.899 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-15T23:04:12.923 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-15T23:04:12.933 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-15T23:04:12.977 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-15T23:04:13.035 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-15T23:04:13.069 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-15T23:04:13.081 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-15T23:04:13.107 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-15T23:04:13.126 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-15T23:04:13.176 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 83/164 2024-09-15T23:04:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-15T23:04:13.229 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 84/164 2024-09-15T23:04:13.253 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-15T23:04:13.271 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 85/164 2024-09-15T23:04:13.294 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-15T23:04:13.326 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-15T23:04:13.372 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 86/164 2024-09-15T23:04:13.409 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-15T23:04:13.427 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 87/164 2024-09-15T23:04:13.443 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-15T23:04:13.472 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 88/164 2024-09-15T23:04:13.485 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-15T23:04:13.559 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-09-15T23:04:13.570 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-15T23:04:13.641 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-09-15T23:04:13.834 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-0.3.26-2.el9.x86_64 91/164 2024-09-15T23:04:13.872 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 92/164 2024-09-15T23:04:13.949 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-09-15T23:04:14.137 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-15T23:04:14.248 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-15T23:04:14.263 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-15T23:04:14.359 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-15T23:04:14.599 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-09-15T23:04:14.723 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-09-15T23:04:15.606 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-15T23:04:15.649 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-15T23:04:15.696 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-15T23:04:15.721 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-15T23:04:15.724 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-15T23:04:15.750 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-15T23:04:15.757 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-15T23:04:15.782 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-15T23:04:15.944 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-09-15T23:04:16.036 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-09-15T23:04:16.063 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-09-15T23:04:16.086 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-15T23:04:16.097 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 99/164 2024-09-15T23:04:16.108 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-15T23:04:16.137 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:04:16.145 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:04:16.237 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:04:16.244 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:04:16.267 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-15T23:04:16.275 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-15T23:04:16.306 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-15T23:04:16.311 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-15T23:04:16.406 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-09-15T23:04:16.443 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:04:16.542 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 101/164 2024-09-15T23:04:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 102/164 2024-09-15T23:04:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-09-15T23:04:16.813 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-15T23:04:16.820 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-15T23:04:16.848 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:04:16.866 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:04:16.903 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:04:16.927 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:04:16.933 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-15T23:04:16.949 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-15T23:04:17.092 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-09-15T23:04:17.121 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:04:17.192 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 105/164 2024-09-15T23:04:17.231 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 106/164 2024-09-15T23:04:19.348 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:04:19.373 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:04:19.379 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:04:19.405 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:04:19.906 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:04:19.936 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:04:19.981 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:04:19.991 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:04:20.025 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:04:20.025 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:04:20.114 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:04:20.114 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T23:04:20.114 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T23:04:20.114 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:20.115 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:04:20.115 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T23:04:20.116 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T23:04:20.116 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:04:20.153 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:04:20.479 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 107/164 2024-09-15T23:04:20.515 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:04:20.610 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 108/164 2024-09-15T23:04:20.610 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T23:04:20.610 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T23:04:20.611 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:20.641 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:04:32.328 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:04:32.328 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /sys 2024-09-15T23:04:32.328 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /proc 2024-09-15T23:04:32.328 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /mnt 2024-09-15T23:04:32.328 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /var/tmp 2024-09-15T23:04:32.328 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /home 2024-09-15T23:04:32.329 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /root 2024-09-15T23:04:32.329 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /tmp 2024-09-15T23:04:32.329 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:32.374 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:04:32.375 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-09-15T23:04:32.375 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-09-15T23:04:32.375 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-09-15T23:04:32.375 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-09-15T23:04:32.375 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-09-15T23:04:32.375 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-09-15T23:04:32.375 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-09-15T23:04:32.375 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:32.502 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:04:32.518 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:04:32.518 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:32.518 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T23:04:32.519 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:32.554 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:04:32.570 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:04:32.570 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:32.571 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T23:04:32.571 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:32.913 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 109/164 2024-09-15T23:04:32.913 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /sys 2024-09-15T23:04:32.913 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /proc 2024-09-15T23:04:32.913 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /mnt 2024-09-15T23:04:32.913 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /var/tmp 2024-09-15T23:04:32.914 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /home 2024-09-15T23:04:32.914 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /root 2024-09-15T23:04:32.914 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /tmp 2024-09-15T23:04:32.914 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:33.181 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:04:33.198 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 110/164 2024-09-15T23:04:33.198 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:33.198 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T23:04:33.198 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:34.005 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:04:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:04:34.032 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:34.032 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T23:04:34.032 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:04:34.032 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:04:34.032 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:34.098 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:04:34.125 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:04:34.125 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:34.125 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T23:04:34.125 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:04:34.125 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:04:34.125 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:34.255 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:04:34.279 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:04:34.280 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:34.280 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T23:04:34.280 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:04:34.280 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:04:34.280 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:34.365 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:04:34.389 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:04:34.389 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:34.389 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T23:04:34.389 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:04:34.389 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:04:34.389 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:34.714 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:04:34.739 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:04:34.739 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:34.739 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T23:04:34.739 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:04:34.739 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:04:34.739 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:34.829 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:04:34.844 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-15T23:04:34.849 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:04:34.854 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 111/164 2024-09-15T23:04:34.854 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:34.854 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T23:04:34.854 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:04:34.854 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T23:04:34.854 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:34.873 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:04:34.873 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:34.873 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T23:04:34.873 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:04:34.874 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:04:34.874 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:34.901 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-15T23:04:34.958 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-15T23:04:34.980 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-15T23:04:35.039 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-15T23:04:35.101 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:04:35.102 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-15T23:04:35.126 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 112/164 2024-09-15T23:04:35.126 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:35.126 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T23:04:35.126 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:04:35.127 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T23:04:35.127 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:35.142 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-15T23:04:35.190 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-15T23:04:35.268 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-15T23:04:35.296 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-15T23:04:35.342 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-15T23:04:35.358 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-15T23:04:35.432 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-15T23:04:35.504 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-15T23:04:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-15T23:04:35.557 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-15T23:04:35.585 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:04:35.597 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-15T23:04:35.611 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 113/164 2024-09-15T23:04:35.611 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:35.611 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T23:04:35.611 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:04:35.611 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T23:04:35.611 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:35.625 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-15T23:04:35.671 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-15T23:04:35.717 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-15T23:04:35.725 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-09-15T23:04:35.749 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-15T23:04:35.783 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-09-15T23:04:35.784 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-15T23:04:35.847 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-09-15T23:04:36.045 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-09-15T23:04:36.088 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:04:36.104 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-09-15T23:04:36.115 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:04:36.147 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-15T23:04:36.184 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-09-15T23:04:36.209 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-15T23:04:36.238 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:04:36.248 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:04:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:04:36.283 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-09-15T23:04:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-15T23:04:36.372 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-15T23:04:36.397 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:04:36.427 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-09-15T23:04:36.489 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-09-15T23:04:36.527 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-09-15T23:04:36.554 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fuse-2.9.9-16.el9.x86_64 124/164 2024-09-15T23:04:36.944 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:04:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 125/164 2024-09-15T23:04:37.010 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 126/164 2024-09-15T23:04:37.075 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 127/164 2024-09-15T23:04:37.101 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:04:37.285 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:04:37.324 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:04:37.503 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:04:37.541 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:04:38.205 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 128/164 2024-09-15T23:04:38.236 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:04:38.289 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:04:38.316 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:04:38.420 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:04:38.555 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:04:38.574 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:04:38.587 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-15T23:04:38.618 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:04:38.642 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:04:38.642 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:38.642 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T23:04:38.642 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:04:38.642 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:04:38.642 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:38.684 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:04:38.693 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:04:38.706 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:04:38.804 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-15T23:04:38.850 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-15T23:04:38.852 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-15T23:04:38.884 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:04:38.909 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:04:38.909 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:38.909 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T23:04:38.909 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:04:38.909 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:04:38.909 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:38.960 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:04:38.972 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:04:39.073 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-15T23:04:39.126 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-15T23:04:39.251 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 129/164 2024-09-15T23:04:39.278 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:04:39.385 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 130/164 2024-09-15T23:04:39.569 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 131/164 2024-09-15T23:04:39.597 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:04:39.622 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 132/164 2024-09-15T23:04:39.622 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:39.622 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T23:04:39.622 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:04:39.622 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T23:04:39.622 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:39.672 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:04:39.684 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 133/164 2024-09-15T23:04:39.785 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-15T23:04:39.792 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 134/164 2024-09-15T23:04:39.821 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:04:39.845 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:04:39.845 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:39.846 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T23:04:39.846 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:04:39.846 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:04:39.846 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:39.854 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 135/164 2024-09-15T23:04:39.888 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:04:39.911 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:04:39.912 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:39.912 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T23:04:39.912 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:39.956 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:04:39.980 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:04:39.980 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:39.980 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-15T23:04:39.981 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:04:39.981 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:04:39.981 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:40.101 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-15T23:04:40.129 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:04:40.153 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:04:40.154 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:40.154 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T23:04:40.154 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:04:40.154 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:04:40.154 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:40.197 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:04:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:04:40.221 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:40.221 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T23:04:40.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:40.272 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:04:40.282 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:04:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:04:40.296 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:40.296 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-15T23:04:40.296 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:04:40.296 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:04:40.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:40.305 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:04:40.305 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:40.305 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T23:04:40.306 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:04:40.306 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:04:40.306 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:40.631 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:04:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:04:40.655 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:40.655 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T23:04:40.655 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:04:40.655 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:04:40.655 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:40.835 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libtool-2.4.6-46.el9.x86_64 136/164 2024-09-15T23:04:40.867 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:04:40.891 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 137/164 2024-09-15T23:04:40.891 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:40.892 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T23:04:40.892 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:04:40.892 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T23:04:40.892 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:40.942 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:04:40.966 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-326.g8921c8 138/164 2024-09-15T23:04:40.967 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:40.967 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T23:04:40.967 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:41.018 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:04:41.043 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 139/164 2024-09-15T23:04:41.043 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:41.043 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-15T23:04:41.043 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:04:41.043 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-15T23:04:41.043 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:41.360 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:04:41.385 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 140/164 2024-09-15T23:04:41.385 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T23:04:41.385 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T23:04:41.385 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:04:41.386 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T23:04:41.386 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:44.876 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-15T23:04:44.923 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-15T23:04:44.958 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-15T23:04:44.990 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-15T23:04:45.018 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-15T23:04:45.197 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-15T23:04:45.239 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-15T23:04:45.270 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-15T23:04:45.299 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-15T23:04:45.337 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-15T23:04:45.460 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-15T23:04:45.501 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-15T23:04:45.542 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-15T23:04:45.574 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-15T23:04:45.601 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-15T23:04:45.796 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-15T23:04:45.839 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-15T23:04:45.869 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-15T23:04:45.899 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-15T23:04:45.909 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:04:45.937 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-15T23:04:45.952 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:04:45.973 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:04:46.281 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 141/164 2024-09-15T23:04:46.323 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 142/164 2024-09-15T23:04:46.373 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 143/164 2024-09-15T23:04:46.413 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 144/164 2024-09-15T23:04:46.440 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 145/164 2024-09-15T23:04:46.538 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:04:46.576 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:04:46.595 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:04:46.657 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 146/164 2024-09-15T23:04:46.703 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 147/164 2024-09-15T23:04:46.733 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 148/164 2024-09-15T23:04:46.771 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 149/164 2024-09-15T23:04:46.807 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-15T23:04:46.809 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 150/164 2024-09-15T23:04:46.898 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-15T23:04:46.947 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-15T23:04:47.037 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-15T23:04:47.076 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-15T23:04:47.101 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-15T23:04:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-15T23:04:47.128 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:04:47.212 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:04:47.212 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:04:47.296 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:04:47.321 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-15T23:04:47.326 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:04:47.326 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:04:47.465 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:04:47.498 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:04:47.505 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-15T23:04:47.514 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:04:47.515 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:04:47.520 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 151/164 2024-09-15T23:04:47.588 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-15T23:04:47.598 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:04:47.630 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-15T23:04:47.661 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:04:47.714 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-15T23:04:47.741 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-15T23:04:47.766 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-15T23:04:47.792 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-15T23:04:47.792 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:04:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:04:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:04:47.952 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:04:47.977 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-15T23:04:47.983 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:04:47.983 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:04:48.162 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:04:48.162 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:04:48.246 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:04:48.320 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:04:48.430 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 152/164 2024-09-15T23:04:48.519 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 153/164 2024-09-15T23:04:48.566 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-09-15T23:04:48.665 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-09-15T23:04:48.697 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-09-15T23:04:48.723 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 157/164 2024-09-15T23:04:48.757 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 158/164 2024-09-15T23:04:48.757 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:04:48.850 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/164 2024-09-15T23:04:48.850 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:04:48.926 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-09-15T23:04:48.953 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 161/164 2024-09-15T23:04:48.956 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:04:48.956 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:04:49.144 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 162/164 2024-09-15T23:04:49.144 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:04:49.220 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 163/164 2024-09-15T23:04:49.288 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:04:53.134 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:04:53.135 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-15T23:04:53.135 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-15T23:04:53.135 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-15T23:04:53.135 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-15T23:04:53.135 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-15T23:04:53.135 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-15T23:04:53.135 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-15T23:04:53.136 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-15T23:04:53.137 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-15T23:04:53.137 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-15T23:04:53.137 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-15T23:04:53.137 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-15T23:04:53.137 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-15T23:04:53.137 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-15T23:04:53.137 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-15T23:04:53.137 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-15T23:04:53.149 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-15T23:04:53.150 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-15T23:04:53.151 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-15T23:04:53.152 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-15T23:04:53.153 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-15T23:04:53.154 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-15T23:04:53.155 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-15T23:04:53.156 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-15T23:04:53.157 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-15T23:04:53.158 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-15T23:04:53.159 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-15T23:04:53.760 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:04:53.760 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-15T23:04:53.760 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-15T23:04:53.760 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-15T23:04:53.760 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-15T23:04:53.760 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-15T23:04:53.760 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-15T23:04:53.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-15T23:04:53.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:04:53.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:04:53.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:04:53.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-15T23:04:53.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-15T23:04:53.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:04:53.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-15T23:04:53.761 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-15T23:04:53.762 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-15T23:04:53.763 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-15T23:04:53.764 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-15T23:04:53.765 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-15T23:04:53.766 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-15T23:04:53.767 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-15T23:04:53.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-15T23:04:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-15T23:04:53.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-15T23:04:53.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-15T23:04:53.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-15T23:04:53.770 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-15T23:04:53.771 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-15T23:04:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-15T23:04:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-15T23:04:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-15T23:04:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-15T23:04:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-15T23:04:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-15T23:04:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-15T23:04:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-15T23:04:53.772 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-15T23:04:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-15T23:04:53.774 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-15T23:04:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-15T23:04:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-15T23:04:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-15T23:04:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-15T23:04:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-15T23:04:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-15T23:04:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-15T23:04:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-15T23:04:53.775 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-15T23:04:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-15T23:04:53.776 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-15T23:04:54.962 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:04:54.962 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 1/164 2024-09-15T23:04:54.962 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2/164 2024-09-15T23:04:54.962 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 3/164 2024-09-15T23:04:54.962 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 4/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-326.g8921c8 5/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 6/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 7/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 8/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 9/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 10/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 11/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 12/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 13/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 14/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_6 15/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 16/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 17/164 2024-09-15T23:04:54.963 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 18/164 2024-09-15T23:04:54.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_ 19/164 2024-09-15T23:04:54.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 20/164 2024-09-15T23:04:54.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9 21/164 2024-09-15T23:04:54.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x 22/164 2024-09-15T23:04:54.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 23/164 2024-09-15T23:04:54.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 24/164 2024-09-15T23:04:54.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 25/164 2024-09-15T23:04:54.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 26/164 2024-09-15T23:04:54.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 27/164 2024-09-15T23:04:54.977 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 28/164 2024-09-15T23:04:54.977 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 29/164 2024-09-15T23:04:54.977 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.e 30/164 2024-09-15T23:04:54.977 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noar 31/164 2024-09-15T23:04:54.977 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.no 32/164 2024-09-15T23:04:54.977 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-326.g8921 33/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9 34/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 35/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el 36/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 37/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 38/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 39/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-09-15T23:04:54.978 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-09-15T23:04:54.979 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-09-15T23:04:54.980 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-09-15T23:04:54.980 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 62/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 66/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 67/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 68/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-13.16-1.el9.x86_64 69/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 70/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 71/164 2024-09-15T23:04:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 72/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 77/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/164 2024-09-15T23:04:54.982 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/164 2024-09-15T23:04:54.983 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 101/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 107/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/164 2024-09-15T23:04:54.984 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 122/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-09-15T23:04:54.985 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-09-15T23:04:54.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-09-15T23:04:54.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-09-15T23:04:54.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-09-15T23:04:54.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-09-15T23:04:54.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-09-15T23:04:54.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-09-15T23:04:54.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-09-15T23:04:54.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-09-15T23:04:54.986 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 142/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 143/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : re2-1:20211101-20.el9.x86_64 147/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-09-15T23:04:54.987 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 149/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 151/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 153/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 154/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 155/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 156/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 157/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 158/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 159/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 160/164 2024-09-15T23:04:54.988 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 161/164 2024-09-15T23:04:54.989 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 162/164 2024-09-15T23:04:54.989 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 163/164 2024-09-15T23:04:55.971 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout:Upgraded: 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-09-15T23:04:55.972 INFO:teuthology.orchestra.run.smithi148.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: automake-1.16.2-8.el9.noarch 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: bison-3.7.4-5.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.973 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.974 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: flex-2.6.4-9.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.975 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T23:04:55.976 INFO:teuthology.orchestra.run.smithi148.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-13.16-1.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-15T23:04:55.977 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T23:04:55.978 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T23:04:55.979 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T23:04:55.980 INFO:teuthology.orchestra.run.smithi148.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T23:04:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T23:04:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T23:04:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T23:04:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T23:04:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T23:04:55.981 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.982 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T23:04:55.983 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:04:55.984 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-09-15T23:04:56.411 DEBUG:teuthology.parallel:result is None 2024-09-15T23:04:56.621 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-15T23:04:56.622 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.2-8.el9.noarch 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.7.4-5.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.624 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.4-9.el9.x86_64 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T23:04:56.625 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-15T23:04:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T23:04:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T23:04:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T23:04:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T23:04:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T23:04:56.626 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T23:04:56.627 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-13.16-1.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-15T23:04:56.628 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T23:04:56.629 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T23:04:56.629 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T23:04:56.629 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.629 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.629 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.629 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T23:04:56.629 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T23:04:56.629 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T23:04:56.630 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T23:04:56.631 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T23:04:56.631 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T23:04:56.631 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T23:04:56.631 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T23:04:56.631 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T23:04:56.631 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T23:04:56.631 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T23:04:56.632 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T23:04:56.632 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T23:04:56.632 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T23:04:56.632 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T23:04:56.633 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T23:04:56.633 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T23:04:56.633 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T23:04:56.633 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T23:04:56.633 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T23:04:56.633 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T23:04:56.634 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T23:04:56.634 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T23:04:56.634 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T23:04:56.634 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T23:04:56.634 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T23:04:56.634 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T23:04:56.635 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-15T23:04:56.635 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T23:04:56.635 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.635 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.635 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T23:04:56.635 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T23:04:56.636 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T23:04:56.636 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.636 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T23:04:56.636 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T23:04:56.636 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T23:04:56.636 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T23:04:56.637 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T23:04:56.637 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T23:04:56.637 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T23:04:56.637 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-15T23:04:56.637 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T23:04:56.637 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T23:04:56.638 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T23:04:56.638 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T23:04:56.638 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T23:04:56.638 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T23:04:56.638 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T23:04:56.638 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.638 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.639 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:56.639 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T23:04:56.639 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T23:04:56.639 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T23:04:56.639 INFO:teuthology.orchestra.run.smithi067.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-15T23:04:56.639 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-15T23:04:56.640 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T23:04:56.640 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-15T23:04:56.640 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:04:56.641 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-09-15T23:04:57.014 DEBUG:teuthology.parallel:result is None 2024-09-15T23:04:57.399 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 164/164 2024-09-15T23:04:57.399 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:57.399 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2024-09-15T23:04:57.399 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:57.399 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:57.399 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:57.399 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-15T23:04:57.399 INFO:teuthology.orchestra.run.smithi062.stdout: librados2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.399 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: automake-1.16.2-8.el9.noarch 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: bison-3.7.4-5.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.400 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.401 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-shell-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-top-2:19.1.1-326.g8921c8ef.el9.noarch 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: flex-2.6.4-9.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T23:04:57.402 INFO:teuthology.orchestra.run.smithi062.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T23:04:57.403 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T23:04:57.404 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-13.16-1.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.405 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T23:04:57.406 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T23:04:57.407 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-15T23:04:57.408 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T23:04:57.409 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd-2:19.1.1-326.g8921c8ef.el9.x86_64 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T23:04:57.410 INFO:teuthology.orchestra.run.smithi062.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-15T23:04:57.411 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-15T23:04:57.411 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T23:04:57.411 INFO:teuthology.orchestra.run.smithi062.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-15T23:04:57.411 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:04:57.411 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-09-15T23:04:57.832 DEBUG:teuthology.parallel:result is None 2024-09-15T23:04:57.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:04:57.987 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T23:04:58.010 INFO:teuthology.orchestra.run.smithi062.stdout:19.1.1-326.g8921c8ef.el9 2024-09-15T23:04:58.011 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-15T23:04:58.011 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-15T23:04:58.013 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:04:58.157 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T23:04:58.183 INFO:teuthology.orchestra.run.smithi067.stdout:19.1.1-326.g8921c8ef.el9 2024-09-15T23:04:58.183 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-15T23:04:58.183 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-15T23:04:58.185 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:04:58.328 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T23:04:58.355 INFO:teuthology.orchestra.run.smithi148.stdout:19.1.1-326.g8921c8ef.el9 2024-09-15T23:04:58.355 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326.g8921c8ef.el9 2024-09-15T23:04:58.356 INFO:teuthology.task.install:The correct ceph version 19.1.1-326.g8921c8ef is installed. 2024-09-15T23:04:58.357 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T23:04:58.358 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:04:58.358 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T23:04:58.388 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:04:58.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T23:04:58.420 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:04:58.420 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T23:04:58.451 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T23:04:58.452 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:04:58.452 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T23:04:58.479 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T23:04:58.547 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:04:58.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T23:04:58.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T23:04:58.645 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:04:58.645 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T23:04:58.675 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T23:04:58.745 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T23:04:58.746 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:04:58.746 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T23:04:58.773 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T23:04:58.841 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:04:58.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T23:04:58.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T23:04:58.937 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:04:58.938 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T23:04:58.968 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T23:04:59.038 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T23:04:59.038 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:04:59.038 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T23:04:59.066 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T23:04:59.133 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:04:59.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T23:04:59.163 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T23:04:59.233 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:04:59.233 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T23:04:59.262 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T23:04:59.333 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-15T23:04:59.434 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-15T23:04:59.434 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:04:59.435 INFO:tasks.cephadm:Cluster fsid is ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:04:59.435 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-15T23:04:59.435 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.62', 'mon.b': '172.21.15.67', 'mon.c': '172.21.15.148'} 2024-09-15T23:04:59.436 INFO:tasks.cephadm:First mon is mon.a on smithi062 2024-09-15T23:04:59.436 INFO:tasks.cephadm:First mgr is x 2024-09-15T23:04:59.436 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-15T23:04:59.436 DEBUG:teuthology.orchestra.run.smithi062:> sudo hostname $(hostname -s) 2024-09-15T23:04:59.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo hostname $(hostname -s) 2024-09-15T23:04:59.496 DEBUG:teuthology.orchestra.run.smithi148:> sudo hostname $(hostname -s) 2024-09-15T23:04:59.526 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-15T23:04:59.527 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:04:59.663 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56081/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-326-g8921c8ef', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-326.g8921c8ef'}, 'url': 'https://4.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-15 16:06:42.385367', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-15T23:04:59.798 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref squid, sha1 8921c8ef947807b717db041d85f0761b7914d5fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:04:59.800 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-15T23:04:59.800 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm 2024-09-15T23:04:59.800 DEBUG:teuthology.orchestra.run.smithi062:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-15T23:05:00.167 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 15 23:05 /home/ubuntu/cephtest/cephadm 2024-09-15T23:05:00.167 DEBUG:teuthology.orchestra.run.smithi067:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-15T23:05:00.566 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 15 23:05 /home/ubuntu/cephtest/cephadm 2024-09-15T23:05:00.566 DEBUG:teuthology.orchestra.run.smithi148:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-15T23:05:00.975 INFO:teuthology.orchestra.run.smithi148.stdout:-rw-r--r--. 1 ubuntu ubuntu 784481 Sep 15 23:05 /home/ubuntu/cephtest/cephadm 2024-09-15T23:05:00.976 DEBUG:teuthology.orchestra.run.smithi062:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-15T23:05:00.996 DEBUG:teuthology.orchestra.run.smithi067:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-15T23:05:01.016 DEBUG:teuthology.orchestra.run.smithi148:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-15T23:05:01.054 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc on all hosts... 2024-09-15T23:05:01.054 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-15T23:05:01.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-15T23:05:01.059 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc pull 2024-09-15T23:05:02.600 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:05:02.736 INFO:teuthology.orchestra.run.smithi148.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:05:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:05:29.958 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-15T23:05:29.959 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T23:05:29.959 INFO:teuthology.orchestra.run.smithi148.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-15T23:05:29.959 INFO:teuthology.orchestra.run.smithi148.stdout: "repo_digests": [ 2024-09-15T23:05:29.959 INFO:teuthology.orchestra.run.smithi148.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-15T23:05:29.959 INFO:teuthology.orchestra.run.smithi148.stdout: ] 2024-09-15T23:05:29.959 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-15T23:05:30.066 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:05:30.066 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T23:05:30.066 INFO:teuthology.orchestra.run.smithi062.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-15T23:05:30.066 INFO:teuthology.orchestra.run.smithi062.stdout: "repo_digests": [ 2024-09-15T23:05:30.066 INFO:teuthology.orchestra.run.smithi062.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-15T23:05:30.067 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:05:30.067 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:05:33.192 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T23:05:33.193 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T23:05:33.193 INFO:teuthology.orchestra.run.smithi067.stdout: "image_id": "f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee", 2024-09-15T23:05:33.193 INFO:teuthology.orchestra.run.smithi067.stdout: "repo_digests": [ 2024-09-15T23:05:33.193 INFO:teuthology.orchestra.run.smithi067.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80" 2024-09-15T23:05:33.193 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-09-15T23:05:33.193 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T23:05:33.218 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph 2024-09-15T23:05:33.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2024-09-15T23:05:33.283 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph 2024-09-15T23:05:33.317 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /etc/ceph 2024-09-15T23:05:33.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /etc/ceph 2024-09-15T23:05:33.373 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /etc/ceph 2024-09-15T23:05:33.401 INFO:tasks.cephadm:Writing seed config... 2024-09-15T23:05:33.402 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [client] debug client = 20 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-09-15T23:05:33.403 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-15T23:05:33.404 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-09-15T23:05:33.405 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-09-15T23:05:33.406 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-09-15T23:05:33.406 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-09-15T23:05:33.406 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-09-15T23:05:33.406 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-09-15T23:05:33.407 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-15T23:05:33.407 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-09-15T23:05:33.407 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:05:33.408 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-15T23:05:33.426 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ee7f888e-73b6-11ef-bceb-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-09-15T23:05:33.427 DEBUG:teuthology.orchestra.run.smithi062:mon.a> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.a.service 2024-09-15T23:05:33.469 DEBUG:teuthology.orchestra.run.smithi062:mgr.x> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.x.service 2024-09-15T23:05:33.512 INFO:tasks.cephadm:Bootstrapping... 2024-09-15T23:05:33.512 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc -v bootstrap --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.62 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:05:33.727 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-09-15T23:05:33.727 INFO:teuthology.orchestra.run.smithi062.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc', '-v', 'bootstrap', '--fsid', 'ee7f888e-73b6-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.62', '--skip-admin-label'] 2024-09-15T23:05:33.727 INFO:teuthology.orchestra.run.smithi062.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-15T23:05:33.728 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying podman|docker is present... 2024-09-15T23:05:33.752 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stdout 5.2.2 2024-09-15T23:05:33.752 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying lvm2 is present... 2024-09-15T23:05:33.752 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying time synchronization is in place... 2024-09-15T23:05:33.760 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-15T23:05:33.760 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-15T23:05:33.766 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-15T23:05:33.766 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-09-15T23:05:33.773 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout enabled 2024-09-15T23:05:33.780 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout active 2024-09-15T23:05:33.780 INFO:teuthology.orchestra.run.smithi062.stdout:Unit chronyd.service is enabled and running 2024-09-15T23:05:33.780 INFO:teuthology.orchestra.run.smithi062.stdout:Repeating the final host check... 2024-09-15T23:05:33.804 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stdout 5.2.2 2024-09-15T23:05:33.804 INFO:teuthology.orchestra.run.smithi062.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-15T23:05:33.804 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl is present 2024-09-15T23:05:33.804 INFO:teuthology.orchestra.run.smithi062.stdout:lvcreate is present 2024-09-15T23:05:33.810 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-15T23:05:33.810 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-15T23:05:33.820 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-15T23:05:33.820 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-09-15T23:05:33.824 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout enabled 2024-09-15T23:05:33.830 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout active 2024-09-15T23:05:33.831 INFO:teuthology.orchestra.run.smithi062.stdout:Unit chronyd.service is enabled and running 2024-09-15T23:05:33.831 INFO:teuthology.orchestra.run.smithi062.stdout:Host looks OK 2024-09-15T23:05:33.831 INFO:teuthology.orchestra.run.smithi062.stdout:Cluster fsid: ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:05:33.831 INFO:teuthology.orchestra.run.smithi062.stdout:Acquiring lock 140326058427392 on /run/cephadm/ee7f888e-73b6-11ef-bceb-c7b262605968.lock 2024-09-15T23:05:33.831 INFO:teuthology.orchestra.run.smithi062.stdout:Lock 140326058427392 acquired on /run/cephadm/ee7f888e-73b6-11ef-bceb-c7b262605968.lock 2024-09-15T23:05:33.831 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying IP 172.21.15.62 port 3300 ... 2024-09-15T23:05:33.832 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying IP 172.21.15.62 port 6789 ... 2024-09-15T23:05:33.832 INFO:teuthology.orchestra.run.smithi062.stdout:Base mon IP(s) is [172.21.15.62:3300, 172.21.15.62:6789], mon addrv is [v2:172.21.15.62:3300,v1:172.21.15.62:6789] 2024-09-15T23:05:33.835 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.62 metric 100 2024-09-15T23:05:33.835 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.62 metric 100 2024-09-15T23:05:33.838 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-15T23:05:33.838 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-15T23:05:33.838 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-15T23:05:33.840 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-15T23:05:33.840 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-15T23:05:33.840 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-15T23:05:33.840 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-15T23:05:33.841 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febb:d1c3/64 scope link noprefixroute 2024-09-15T23:05:33.841 INFO:teuthology.orchestra.run.smithi062.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-15T23:05:33.842 INFO:teuthology.orchestra.run.smithi062.stdout:Mon IP `172.21.15.62` is in CIDR network `172.21.0.0/20` 2024-09-15T23:05:33.842 INFO:teuthology.orchestra.run.smithi062.stdout:Mon IP `172.21.15.62` is in CIDR network `172.21.0.0/20` 2024-09-15T23:05:33.842 INFO:teuthology.orchestra.run.smithi062.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-15T23:05:33.842 INFO:teuthology.orchestra.run.smithi062.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-15T23:05:33.842 INFO:teuthology.orchestra.run.smithi062.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:05:34.251 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stdout f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-15T23:05:34.252 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc... 2024-09-15T23:05:34.252 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Getting image source signatures 2024-09-15T23:05:34.252 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Copying blob sha256:c0574da0c3174f3dc49daa274908f97d06456bd6a3cfab87e94022c1369a83a6 2024-09-15T23:05:34.252 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-15T23:05:34.252 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Copying config sha256:f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee 2024-09-15T23:05:34.252 INFO:teuthology.orchestra.run.smithi062.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-15T23:05:35.148 INFO:teuthology.orchestra.run.smithi062.stdout:ceph: stdout ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-15T23:05:35.148 INFO:teuthology.orchestra.run.smithi062.stdout:Ceph version: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable) 2024-09-15T23:05:35.148 INFO:teuthology.orchestra.run.smithi062.stdout:Extracting ceph user uid/gid from container image... 2024-09-15T23:05:36.079 INFO:teuthology.orchestra.run.smithi062.stdout:stat: stdout 167 167 2024-09-15T23:05:36.079 INFO:teuthology.orchestra.run.smithi062.stdout:Creating initial keys... 2024-09-15T23:05:36.993 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph-authtool: stdout AQBAaOdmc9OlFxAANRPdEvc9K17lWuJplOLKzA== 2024-09-15T23:05:37.890 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph-authtool: stdout AQBBaOdmwtH3FBAAVEMPqgAHQhXLrbY4StFgBA== 2024-09-15T23:05:38.754 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph-authtool: stdout AQBCaOdmW9FfCxAATGHMj5gcCsm/b3Nse1ZhTA== 2024-09-15T23:05:38.755 INFO:teuthology.orchestra.run.smithi062.stdout:Creating initial monmap... 2024-09-15T23:05:39.744 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-15T23:05:39.744 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-15T23:05:39.744 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:05:39.744 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-15T23:05:39.744 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool for a [v2:172.21.15.62:3300,v1:172.21.15.62:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-15T23:05:39.745 INFO:teuthology.orchestra.run.smithi062.stdout:setting min_mon_release = quincy 2024-09-15T23:05:39.745 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: set fsid to ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:05:39.745 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-15T23:05:39.745 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:05:39.745 INFO:teuthology.orchestra.run.smithi062.stdout:Creating mon... 2024-09-15T23:05:47.606 INFO:teuthology.orchestra.run.smithi062.stdout:create mon.a on 2024-09-15T23:05:48.139 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-15T23:05:48.401 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-15T23:05:48.927 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968.target. 2024-09-15T23:05:48.927 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968.target. 2024-09-15T23:05:49.943 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.a 2024-09-15T23:05:49.943 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-15T23:05:50.278 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968.target.wants/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@.service. 2024-09-15T23:05:50.778 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T23:05:50.778 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout disabled 2024-09-15T23:05:50.785 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T23:05:50.785 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-09-15T23:05:50.785 INFO:teuthology.orchestra.run.smithi062.stdout:firewalld.service is not enabled 2024-09-15T23:05:50.785 INFO:teuthology.orchestra.run.smithi062.stdout:Not possible to enable service . firewalld.service is not available 2024-09-15T23:05:50.785 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mon to start... 2024-09-15T23:05:50.785 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mon... 2024-09-15T23:05:51.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:51 smithi062 ceph-mon[30586]: from='client.? 172.21.15.62:0/3309832102' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-15T23:05:51.988 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout cluster: 2024-09-15T23:05:51.988 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout id: ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:05:51.988 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-15T23:05:51.988 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:05:51.988 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout services: 2024-09-15T23:05:51.988 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.676101s) 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout data: 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout pgs: 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.350+0000 7f857d361640 1 Processor -- start 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.350+0000 7f857d361640 1 -- start start 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.350+0000 7f857d361640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857810a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:05:51.989 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.350+0000 7f857d361640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f857807be50 con 0x7f8578107fa0 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.351+0000 7f8576ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857810a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.351+0000 7f8576ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857810a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48364/0 (socket says 172.21.15.62:48364) 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.351+0000 7f8576ffd640 1 -- 172.21.15.62:0/1755962877 learned_addr learned my addr 172.21.15.62:0/1755962877 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.351+0000 7f8576ffd640 1 -- 172.21.15.62:0/1755962877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857810a8d0 con 0x7f8578107fa0 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.353+0000 7f8576ffd640 1 --2- 172.21.15.62:0/1755962877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857810a390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8570009e80 tx=0x7f857002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32364b484c6d254 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.353+0000 7f8575ffb640 1 -- 172.21.15.62:0/1755962877 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8570036070 con 0x7f8578107fa0 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.353+0000 7f8575ffb640 1 -- 172.21.15.62:0/1755962877 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8570031400 con 0x7f8578107fa0 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.353+0000 7f8575ffb640 1 -- 172.21.15.62:0/1755962877 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8570031700 con 0x7f8578107fa0 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.354+0000 7f857d361640 1 -- 172.21.15.62:0/1755962877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 msgr2=0x7f857810a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.354+0000 7f857d361640 1 --2- 172.21.15.62:0/1755962877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857810a390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8570009e80 tx=0x7f857002a7a0 comp rx=0 tx=0).stop 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.354+0000 7f857d361640 1 -- 172.21.15.62:0/1755962877 shutdown_connections 2024-09-15T23:05:51.990 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.354+0000 7f857d361640 1 --2- 172.21.15.62:0/1755962877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857810a390 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.354+0000 7f857d361640 1 -- 172.21.15.62:0/1755962877 >> 172.21.15.62:0/1755962877 conn(0x7f8578101bc0 msgr2=0x7f8578104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.354+0000 7f857d361640 1 -- 172.21.15.62:0/1755962877 shutdown_connections 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.354+0000 7f857d361640 1 -- 172.21.15.62:0/1755962877 wait complete. 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.355+0000 7f857d361640 1 Processor -- start 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.355+0000 7f857d361640 1 -- start start 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.355+0000 7f857d361640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857819acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.355+0000 7f857d361640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f857810ae10 con 0x7f8578107fa0 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.355+0000 7f8576ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857819acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.355+0000 7f8576ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857819acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48372/0 (socket says 172.21.15.62:48372) 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.355+0000 7f8576ffd640 1 -- 172.21.15.62:0/3309832102 learned_addr learned my addr 172.21.15.62:0/3309832102 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.355+0000 7f8576ffd640 1 -- 172.21.15.62:0/3309832102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857819b200 con 0x7f8578107fa0 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.356+0000 7f8576ffd640 1 --2- 172.21.15.62:0/3309832102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857819acc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8570034040 tx=0x7f8570004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.356+0000 7f856ffff640 1 -- 172.21.15.62:0/3309832102 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8570041070 con 0x7f8578107fa0 2024-09-15T23:05:51.991 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.356+0000 7f856ffff640 1 -- 172.21.15.62:0/3309832102 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f857003c650 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.356+0000 7f857d361640 1 -- 172.21.15.62:0/3309832102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f857819b490 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.356+0000 7f856ffff640 1 -- 172.21.15.62:0/3309832102 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8570036040 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.356+0000 7f857d361640 1 -- 172.21.15.62:0/3309832102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f857819e180 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.356+0000 7f856ffff640 1 -- 172.21.15.62:0/3309832102 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8570044070 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.356+0000 7f856ffff640 1 -- 172.21.15.62:0/3309832102 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8570046c20 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.357+0000 7f857d361640 1 -- 172.21.15.62:0/3309832102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8578108400 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.359+0000 7f856ffff640 1 -- 172.21.15.62:0/3309832102 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f857003c950 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.421+0000 7f857d361640 1 -- 172.21.15.62:0/3309832102 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f857802da30 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.421+0000 7f856ffff640 1 -- 172.21.15.62:0/3309832102 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f857003cb30 con 0x7f8578107fa0 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.422+0000 7f857d361640 1 -- 172.21.15.62:0/3309832102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 msgr2=0x7f857819acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.422+0000 7f857d361640 1 --2- 172.21.15.62:0/3309832102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857819acc0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8570034040 tx=0x7f8570004a40 comp rx=0 tx=0).stop 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.422+0000 7f857d361640 1 -- 172.21.15.62:0/3309832102 shutdown_connections 2024-09-15T23:05:51.992 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.422+0000 7f857d361640 1 --2- 172.21.15.62:0/3309832102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8578107fa0 0x7f857819acc0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:05:51.993 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.422+0000 7f857d361640 1 -- 172.21.15.62:0/3309832102 >> 172.21.15.62:0/3309832102 conn(0x7f8578101bc0 msgr2=0x7f8578102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:05:51.993 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.422+0000 7f857d361640 1 -- 172.21.15.62:0/3309832102 shutdown_connections 2024-09-15T23:05:51.993 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:51.422+0000 7f857d361640 1 -- 172.21.15.62:0/3309832102 wait complete. 2024-09-15T23:05:51.993 INFO:teuthology.orchestra.run.smithi062.stdout:mon is available 2024-09-15T23:05:51.993 INFO:teuthology.orchestra.run.smithi062.stdout:Assimilating anything we can from ceph.conf... 2024-09-15T23:05:53.212 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:52 smithi062 ceph-mon[30586]: from='client.? 172.21.15.62:0/2436627675' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-15T23:05:53.212 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:52 smithi062 ceph-mon[30586]: from='client.? 172.21.15.62:0/2436627675' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [global] 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout fsid = ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.62:3300,v1:172.21.15.62:6789] 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-15T23:05:53.220 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [osd] 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.393+0000 7f646452a640 1 Processor -- start 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.393+0000 7f646452a640 1 -- start start 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.393+0000 7f646452a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c106870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.393+0000 7f646452a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f645c106e40 con 0x7f645c106470 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.394+0000 7f64622c6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c106870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.394+0000 7f64622c6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c106870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38076/0 (socket says 172.21.15.62:38076) 2024-09-15T23:05:53.221 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.394+0000 7f64622c6640 1 -- 172.21.15.62:0/3208543420 learned_addr learned my addr 172.21.15.62:0/3208543420 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.394+0000 7f64622c6640 1 -- 172.21.15.62:0/3208543420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645c107620 con 0x7f645c106470 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f64622c6640 1 --2- 172.21.15.62:0/3208543420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c106870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6454009820 tx=0x7f645402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f8707b6962cafb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f64612c4640 1 -- 172.21.15.62:0/3208543420 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6454037070 con 0x7f645c106470 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f64612c4640 1 -- 172.21.15.62:0/3208543420 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6454032440 con 0x7f645c106470 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f64612c4640 1 -- 172.21.15.62:0/3208543420 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6454032740 con 0x7f645c106470 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f646452a640 1 -- 172.21.15.62:0/3208543420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 msgr2=0x7f645c106870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f646452a640 1 --2- 172.21.15.62:0/3208543420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c106870 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6454009820 tx=0x7f645402a270 comp rx=0 tx=0).stop 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f646452a640 1 -- 172.21.15.62:0/3208543420 shutdown_connections 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f646452a640 1 --2- 172.21.15.62:0/3208543420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c106870 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f646452a640 1 -- 172.21.15.62:0/3208543420 >> 172.21.15.62:0/3208543420 conn(0x7f645c101be0 msgr2=0x7f645c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f646452a640 1 -- 172.21.15.62:0/3208543420 shutdown_connections 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.395+0000 7f646452a640 1 -- 172.21.15.62:0/3208543420 wait complete. 2024-09-15T23:05:53.222 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.396+0000 7f646452a640 1 Processor -- start 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.396+0000 7f646452a640 1 -- start start 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.396+0000 7f646452a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c07c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.396+0000 7f646452a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f645c107bc0 con 0x7f645c106470 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.396+0000 7f64622c6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c07c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.396+0000 7f64622c6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c07c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38088/0 (socket says 172.21.15.62:38088) 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.396+0000 7f64622c6640 1 -- 172.21.15.62:0/2436627675 learned_addr learned my addr 172.21.15.62:0/2436627675 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.397+0000 7f64622c6640 1 -- 172.21.15.62:0/2436627675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645c19da10 con 0x7f645c106470 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.397+0000 7f64622c6640 1 --2- 172.21.15.62:0/2436627675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c07c210 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f64540090b0 tx=0x7f64540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.397+0000 7f645b7fe640 1 -- 172.21.15.62:0/2436627675 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6454041070 con 0x7f645c106470 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.397+0000 7f645b7fe640 1 -- 172.21.15.62:0/2436627675 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f645403c7a0 con 0x7f645c106470 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.397+0000 7f646452a640 1 -- 172.21.15.62:0/2436627675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f645c19dbe0 con 0x7f645c106470 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.397+0000 7f645b7fe640 1 -- 172.21.15.62:0/2436627675 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6454037040 con 0x7f645c106470 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.397+0000 7f646452a640 1 -- 172.21.15.62:0/2436627675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f645c19df80 con 0x7f645c106470 2024-09-15T23:05:53.223 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.397+0000 7f645b7fe640 1 -- 172.21.15.62:0/2436627675 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6454044070 con 0x7f645c106470 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.397+0000 7f645b7fe640 1 -- 172.21.15.62:0/2436627675 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f645403d730 con 0x7f645c106470 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.398+0000 7f646452a640 1 -- 172.21.15.62:0/2436627675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f645c108210 con 0x7f645c106470 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.400+0000 7f645b7fe640 1 -- 172.21.15.62:0/2436627675 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f645403cbe0 con 0x7f645c106470 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.459+0000 7f646452a640 1 -- 172.21.15.62:0/2436627675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f645c19e290 con 0x7f645c106470 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.463+0000 7f645b7fe640 1 -- 172.21.15.62:0/2436627675 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f645403cdc0 con 0x7f645c106470 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.463+0000 7f646452a640 1 -- 172.21.15.62:0/2436627675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 msgr2=0x7f645c07c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.463+0000 7f646452a640 1 --2- 172.21.15.62:0/2436627675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c07c210 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f64540090b0 tx=0x7f64540046c0 comp rx=0 tx=0).stop 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.464+0000 7f646452a640 1 -- 172.21.15.62:0/2436627675 shutdown_connections 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.464+0000 7f646452a640 1 --2- 172.21.15.62:0/2436627675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f645c106470 0x7f645c07c210 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.464+0000 7f646452a640 1 -- 172.21.15.62:0/2436627675 >> 172.21.15.62:0/2436627675 conn(0x7f645c101be0 msgr2=0x7f645c18dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.464+0000 7f646452a640 1 -- 172.21.15.62:0/2436627675 shutdown_connections 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:52.464+0000 7f646452a640 1 -- 172.21.15.62:0/2436627675 wait complete. 2024-09-15T23:05:53.224 INFO:teuthology.orchestra.run.smithi062.stdout:Generating new minimal ceph.conf... 2024-09-15T23:05:53.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:53 smithi062 ceph-mon[30586]: from='client.? 172.21.15.62:0/1470383832' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.643+0000 7fde157a1640 1 Processor -- start 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.643+0000 7fde157a1640 1 -- start start 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.644+0000 7fde157a1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1010a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.644+0000 7fde157a1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fde1007bdf0 con 0x7fde10107e80 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.644+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1010a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.644+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1010a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38092/0 (socket says 172.21.15.62:38092) 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.644+0000 7fde0ffff640 1 -- 172.21.15.62:0/407878696 learned_addr learned my addr 172.21.15.62:0/407878696 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.644+0000 7fde0ffff640 1 -- 172.21.15.62:0/407878696 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1010a7b0 con 0x7fde10107e80 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.645+0000 7fde0ffff640 1 --2- 172.21.15.62:0/407878696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1010a270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fde00009820 tx=0x7fde0002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1f037955bfe3bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:05:54.250 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.645+0000 7fde0effd640 1 -- 172.21.15.62:0/407878696 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde00037070 con 0x7fde10107e80 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.645+0000 7fde0effd640 1 -- 172.21.15.62:0/407878696 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fde00032900 con 0x7fde10107e80 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.645+0000 7fde0effd640 1 -- 172.21.15.62:0/407878696 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde00030850 con 0x7fde10107e80 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.646+0000 7fde157a1640 1 -- 172.21.15.62:0/407878696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 msgr2=0x7fde1010a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.646+0000 7fde157a1640 1 --2- 172.21.15.62:0/407878696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1010a270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fde00009820 tx=0x7fde0002a270 comp rx=0 tx=0).stop 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.646+0000 7fde157a1640 1 -- 172.21.15.62:0/407878696 shutdown_connections 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.646+0000 7fde157a1640 1 --2- 172.21.15.62:0/407878696 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1010a270 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.646+0000 7fde157a1640 1 -- 172.21.15.62:0/407878696 >> 172.21.15.62:0/407878696 conn(0x7fde10101a20 msgr2=0x7fde10103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.646+0000 7fde157a1640 1 -- 172.21.15.62:0/407878696 shutdown_connections 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.646+0000 7fde157a1640 1 -- 172.21.15.62:0/407878696 wait complete. 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.647+0000 7fde157a1640 1 Processor -- start 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.647+0000 7fde157a1640 1 -- start start 2024-09-15T23:05:54.251 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.647+0000 7fde157a1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1019b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.647+0000 7fde157a1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fde1010b7f0 con 0x7fde10107e80 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.647+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1019b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.647+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1019b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38096/0 (socket says 172.21.15.62:38096) 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.647+0000 7fde0ffff640 1 -- 172.21.15.62:0/1470383832 learned_addr learned my addr 172.21.15.62:0/1470383832 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.647+0000 7fde0ffff640 1 -- 172.21.15.62:0/1470383832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1019b910 con 0x7fde10107e80 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.648+0000 7fde0ffff640 1 --2- 172.21.15.62:0/1470383832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1019b3d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fde000097f0 tx=0x7fde00030ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.648+0000 7fde0d7fa640 1 -- 172.21.15.62:0/1470383832 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde00042070 con 0x7fde10107e80 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.648+0000 7fde157a1640 1 -- 172.21.15.62:0/1470383832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1019bba0 con 0x7fde10107e80 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.648+0000 7fde0d7fa640 1 -- 172.21.15.62:0/1470383832 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fde00032ca0 con 0x7fde10107e80 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.648+0000 7fde0d7fa640 1 -- 172.21.15.62:0/1470383832 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde00037040 con 0x7fde10107e80 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.648+0000 7fde157a1640 1 -- 172.21.15.62:0/1470383832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde1019e890 con 0x7fde10107e80 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.648+0000 7fde0d7fa640 1 -- 172.21.15.62:0/1470383832 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fde00045070 con 0x7fde10107e80 2024-09-15T23:05:54.252 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.648+0000 7fde0d7fa640 1 -- 172.21.15.62:0/1470383832 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fde00047af0 con 0x7fde10107e80 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.649+0000 7fde157a1640 1 -- 172.21.15.62:0/1470383832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde101082e0 con 0x7fde10107e80 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.651+0000 7fde0d7fa640 1 -- 172.21.15.62:0/1470383832 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fde0003db10 con 0x7fde10107e80 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.710+0000 7fde157a1640 1 -- 172.21.15.62:0/1470383832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fde1019ee10 con 0x7fde10107e80 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.711+0000 7fde0d7fa640 1 -- 172.21.15.62:0/1470383832 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fde0003d430 con 0x7fde10107e80 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.712+0000 7fde157a1640 1 -- 172.21.15.62:0/1470383832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 msgr2=0x7fde1019b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.712+0000 7fde157a1640 1 --2- 172.21.15.62:0/1470383832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1019b3d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fde000097f0 tx=0x7fde00030ec0 comp rx=0 tx=0).stop 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.712+0000 7fde157a1640 1 -- 172.21.15.62:0/1470383832 shutdown_connections 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.712+0000 7fde157a1640 1 --2- 172.21.15.62:0/1470383832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde10107e80 0x7fde1019b3d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.712+0000 7fde157a1640 1 -- 172.21.15.62:0/1470383832 >> 172.21.15.62:0/1470383832 conn(0x7fde10101a20 msgr2=0x7fde10102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.712+0000 7fde157a1640 1 -- 172.21.15.62:0/1470383832 shutdown_connections 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:53.712+0000 7fde157a1640 1 -- 172.21.15.62:0/1470383832 wait complete. 2024-09-15T23:05:54.253 INFO:teuthology.orchestra.run.smithi062.stdout:Restarting the monitor... 2024-09-15T23:05:54.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:54 smithi062 systemd[1]: Stopping Ceph mon.a for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:05:54.929 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:54 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a[30582]: 2024-09-15T23:05:54.520+0000 7ff6a14dc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:05:54.929 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:54 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a[30582]: 2024-09-15T23:05:54.520+0000 7ff6a14dc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-15T23:05:54.929 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:54 smithi062 podman[30826]: 2024-09-15 23:05:54.70508149 +0000 UTC m=+0.283675909 container died 924df3822e4b0c6da57a188eaf3990ed35ef12929a13027c1b37aa02dfe8947a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e) 2024-09-15T23:05:55.188 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:55 smithi062 podman[30826]: 2024-09-15 23:05:55.028787136 +0000 UTC m=+0.607381555 container cleanup 924df3822e4b0c6da57a188eaf3990ed35ef12929a13027c1b37aa02dfe8947a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:05:55.188 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:55 smithi062 bash[30826]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a 2024-09-15T23:05:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:55 smithi062 podman[30840]: 2024-09-15 23:05:55.188036274 +0000 UTC m=+0.485847991 container remove 924df3822e4b0c6da57a188eaf3990ed35ef12929a13027c1b37aa02dfe8947a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-15T23:05:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:55 smithi062 systemd[1]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-15T23:05:55.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:55 smithi062 systemd[1]: Stopped Ceph mon.a for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:05:55.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:55 smithi062 systemd[1]: Starting Ceph mon.a for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:05:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:55 smithi062 podman[30930]: 2024-09-15 23:05:55.594113207 +0000 UTC m=+0.016484753 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-15T23:05:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:56 smithi062 podman[30930]: 2024-09-15 23:05:56.483738617 +0000 UTC m=+0.906110152 container create 293eb3210083980e5630f0dfe4069d6ff633cdc31fd5c2673f86edd4071ceba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e) 2024-09-15T23:05:58.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 podman[30930]: 2024-09-15 23:05:58.341594728 +0000 UTC m=+2.763966263 container init 293eb3210083980e5630f0dfe4069d6ff633cdc31fd5c2673f86edd4071ceba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-15T23:05:58.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 podman[30930]: 2024-09-15 23:05:58.346064533 +0000 UTC m=+2.768436068 container start 293eb3210083980e5630f0dfe4069d6ff633cdc31fd5c2673f86edd4071ceba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e) 2024-09-15T23:05:58.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: set uid:gid to 167:167 (ceph:ceph) 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable), process ceph-mon, pid 2 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: pidfile_write: ignore empty --pid-file 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: load: jerasure load: lrc 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: RocksDB version: 7.9.2 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Git sha 0 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Compile date 2024-09-15 15:04:35 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: DB SUMMARY 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: DB Session ID: KSOBCUT47B0M2PQB7ZIA 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: CURRENT file: CURRENT 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: IDENTITY file: IDENTITY 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-15T23:05:58.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104800 ; 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.error_if_exists: 0 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.create_if_missing: 0 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.paranoid_checks: 1 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.env: 0x56213a3dc940 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.fs: PosixFileSystem 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.info_log: 0x56213bef56e0 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_file_opening_threads: 16 2024-09-15T23:05:58.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.statistics: (nil) 2024-09-15T23:05:58.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.use_fsync: 0 2024-09-15T23:05:58.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_log_file_size: 0 2024-09-15T23:05:58.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-15T23:05:58.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-15T23:05:58.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.keep_log_file_num: 1000 2024-09-15T23:05:58.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.recycle_log_file_num: 0 2024-09-15T23:05:58.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.allow_fallocate: 1 2024-09-15T23:05:58.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.allow_mmap_reads: 0 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.allow_mmap_writes: 0 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.use_direct_reads: 0 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.create_missing_column_families: 0 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.db_log_dir: 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.wal_dir: 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-15T23:05:58.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.advise_random_on_open: 1 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.db_write_buffer_size: 0 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.write_buffer_manager: 0x56213bf04dc0 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.rate_limiter: (nil) 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.wal_recovery_mode: 2 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.enable_thread_tracking: 0 2024-09-15T23:05:58.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.enable_pipelined_write: 0 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.unordered_write: 0 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.row_cache: None 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.wal_filter: None 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.allow_ingest_behind: 0 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.two_write_queues: 0 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.manual_wal_flush: 0 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.wal_compression: 0 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.atomic_flush: 0 2024-09-15T23:05:58.719 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.log_readahead_size: 0 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.best_efforts_recovery: 0 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.allow_data_in_errors: 0 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.db_host_id: __hostname__ 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_background_jobs: 2 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_background_compactions: -1 2024-09-15T23:05:58.720 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_subcompactions: 1 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_total_wal_size: 0 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_open_files: -1 2024-09-15T23:05:58.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bytes_per_sync: 0 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_readahead_size: 0 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_background_flushes: -1 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Compression algorithms supported: 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: kZSTD supported: 0 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: kXpressCompression supported: 0 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: kBZip2Compression supported: 0 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: kLZ4Compression supported: 1 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: kZlibCompression supported: 1 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: kLZ4HCCompression supported: 1 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: kSnappyCompression supported: 1 2024-09-15T23:05:58.722 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.merge_operator: 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_filter: None 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_filter_factory: None 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.sst_partitioner_factory: None 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-15T23:05:58.723 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56213bef5320) 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout: cache_index_and_filter_blocks: 1 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout: pin_top_level_index_and_filter: 1 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout: index_type: 0 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout: data_block_index_type: 0 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout: index_shortening: 1 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-15T23:05:58.724 INFO:journalctl@ceph.mon.a.smithi062.stdout: checksum: 4 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: no_block_cache: 0 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache: 0x56213bee91f0 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache_name: BinnedLRUCache 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache_options: 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: capacity : 536870912 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: num_shard_bits : 4 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: strict_capacity_limit : 0 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: high_pri_pool_ratio: 0.000 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache_compressed: (nil) 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: persistent_cache: (nil) 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_size: 4096 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_size_deviation: 10 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_restart_interval: 16 2024-09-15T23:05:58.725 INFO:journalctl@ceph.mon.a.smithi062.stdout: index_block_restart_interval: 1 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: metadata_block_size: 4096 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: partition_filters: 0 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: use_delta_encoding: 1 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: filter_policy: bloomfilter 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: whole_key_filtering: 1 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: verify_compression: 0 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: read_amp_bytes_per_bit: 0 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: format_version: 5 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: enable_index_compression: 1 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_align: 0 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: max_auto_readahead_size: 262144 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: prepopulate_block_cache: 0 2024-09-15T23:05:58.726 INFO:journalctl@ceph.mon.a.smithi062.stdout: initial_auto_readahead_size: 8192 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout: num_file_reads_for_auto_readahead: 2 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.write_buffer_size: 33554432 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_write_buffer_number: 2 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression: NoCompression 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression: Disabled 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.prefix_extractor: nullptr 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.num_levels: 7 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-15T23:05:58.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression_opts.level: 32767 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression_opts.strategy: 0 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-15T23:05:58.728 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression_opts.enabled: false 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.target_file_size_base: 67108864 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-15T23:05:58.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.arena_block_size: 1048576 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.disable_auto_compactions: 0 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-15T23:05:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.inplace_update_support: 0 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.bloom_locality: 0 2024-09-15T23:05:58.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.max_successive_merges: 0 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.paranoid_file_checks: 0 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.force_consistency_checks: 1 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.report_bg_io_stats: 0 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.ttl: 2592000 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.enable_blob_files: false 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.min_blob_size: 0 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.blob_file_size: 268435456 2024-09-15T23:05:58.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.blob_file_starting_level: 0 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5692ef96-a0b8-49a6-9770-fefa6f4ca60f 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726441558377884, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726441558379194, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99904, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 97790, "index_size": 306, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14179, "raw_average_key_size": 49, "raw_value_size": 90126, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726441558, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5692ef96-a0b8-49a6-9770-fefa6f4ca60f", "db_session_id": "KSOBCUT47B0M2PQB7ZIA", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726441558379317, "job": 1, "event": "recovery_finished"} 2024-09-15T23:05:58.733 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56213bf16e00 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: DB pointer 0x56213c012000 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** DB Stats ** 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-15T23:05:58.734 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-09-15T23:05:58.735 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** Compaction Stats [default] ** 2024-09-15T23:05:58.735 INFO:journalctl@ceph.mon.a.smithi062.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-15T23:05:58.735 INFO:journalctl@ceph.mon.a.smithi062.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-15T23:05:58.735 INFO:journalctl@ceph.mon.a.smithi062.stdout: L0 2/0 99.42 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 110.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T23:05:58.735 INFO:journalctl@ceph.mon.a.smithi062.stdout: Sum 2/0 99.42 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 110.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T23:05:58.735 INFO:journalctl@ceph.mon.a.smithi062.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 110.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** Compaction Stats [default] ** 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 110.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-15T23:05:58.736 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative compaction: 0.00 GB write, 24.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval compaction: 0.00 GB write, 24.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: Block cache BinnedLRUCache@0x56213bee91f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: starting mon.a rank 0 at public addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] at bind addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mon.a@-1(???) e1 preinit fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mon.a@-1(???).mds e1 new map 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mon.a@-1(???).mds e1 print_map 2024-09-15T23:05:58.737 INFO:journalctl@ceph.mon.a.smithi062.stdout: e1 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout: btime 2024-09-15T23:05:50:746836+0000 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout: legacy client fscid: -1 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout: No filesystems configured 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-15T23:05:58.738 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: monmap epoch 1 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: last_changed 2024-09-15T23:05:39.118425+0000 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: created 2024-09-15T23:05:39.118425+0000 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: min_mon_release 19 (squid) 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: election_strategy: 1 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: fsmap 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: osdmap e1: 0 total, 0 up, 0 in 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 ceph-mon[30949]: mgrmap e1: no daemons active 2024-09-15T23:05:58.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 bash[30930]: 293eb3210083980e5630f0dfe4069d6ff633cdc31fd5c2673f86edd4071ceba0 2024-09-15T23:05:58.740 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:05:58 smithi062 systemd[1]: Started Ceph mon.a for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:05:58.740 INFO:teuthology.orchestra.run.smithi062.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-15T23:05:59.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.167+0000 7ff609be3640 1 Processor -- start 2024-09-15T23:05:59.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.167+0000 7ff609be3640 1 -- start start 2024-09-15T23:05:59.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.167+0000 7ff609be3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff604107950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.167+0000 7ff609be3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff604107f20 con 0x7ff604107550 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.168+0000 7ff6037fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff604107950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.168+0000 7ff6037fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff604107950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38104/0 (socket says 172.21.15.62:38104) 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.168+0000 7ff6037fe640 1 -- 172.21.15.62:0/4196879597 learned_addr learned my addr 172.21.15.62:0/4196879597 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.168+0000 7ff6037fe640 1 -- 172.21.15.62:0/4196879597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff604108750 con 0x7ff604107550 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.170+0000 7ff6037fe640 1 --2- 172.21.15.62:0/4196879597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff604107950 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009820 tx=0x7ff5f402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e7586e0b219d08 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.170+0000 7ff6027fc640 1 -- 172.21.15.62:0/4196879597 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5f4037070 con 0x7ff604107550 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.170+0000 7ff6027fc640 1 -- 172.21.15.62:0/4196879597 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff5f4032900 con 0x7ff604107550 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.170+0000 7ff6027fc640 1 -- 172.21.15.62:0/4196879597 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5f4030850 con 0x7ff604107550 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.171+0000 7ff609be3640 1 -- 172.21.15.62:0/4196879597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 msgr2=0x7ff604107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.171+0000 7ff609be3640 1 --2- 172.21.15.62:0/4196879597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff604107950 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009820 tx=0x7ff5f402a270 comp rx=0 tx=0).stop 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.171+0000 7ff609be3640 1 -- 172.21.15.62:0/4196879597 shutdown_connections 2024-09-15T23:05:59.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.171+0000 7ff609be3640 1 --2- 172.21.15.62:0/4196879597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff604107950 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.171+0000 7ff609be3640 1 -- 172.21.15.62:0/4196879597 >> 172.21.15.62:0/4196879597 conn(0x7ff604102ce0 msgr2=0x7ff604105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.171+0000 7ff609be3640 1 -- 172.21.15.62:0/4196879597 shutdown_connections 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.171+0000 7ff609be3640 1 -- 172.21.15.62:0/4196879597 wait complete. 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.172+0000 7ff609be3640 1 Processor -- start 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.172+0000 7ff609be3640 1 -- start start 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.172+0000 7ff609be3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff60419b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.172+0000 7ff609be3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff60410bb00 con 0x7ff604107550 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.172+0000 7ff6037fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff60419b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.172+0000 7ff6037fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff60419b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38116/0 (socket says 172.21.15.62:38116) 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.172+0000 7ff6037fe640 1 -- 172.21.15.62:0/3899205290 learned_addr learned my addr 172.21.15.62:0/3899205290 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.173+0000 7ff6037fe640 1 -- 172.21.15.62:0/3899205290 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60419ba80 con 0x7ff604107550 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.173+0000 7ff6037fe640 1 --2- 172.21.15.62:0/3899205290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff60419b540 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009940 tx=0x7ff5f4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:05:59.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.173+0000 7ff600ff9640 1 -- 172.21.15.62:0/3899205290 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5f4042070 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.173+0000 7ff600ff9640 1 -- 172.21.15.62:0/3899205290 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff5f40308c0 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.173+0000 7ff609be3640 1 -- 172.21.15.62:0/3899205290 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60419bd10 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.173+0000 7ff600ff9640 1 -- 172.21.15.62:0/3899205290 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5f4037040 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.173+0000 7ff609be3640 1 -- 172.21.15.62:0/3899205290 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff60419ea00 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.174+0000 7ff600ff9640 1 -- 172.21.15.62:0/3899205290 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff5f4045070 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.174+0000 7ff600ff9640 1 -- 172.21.15.62:0/3899205290 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff5f4047b70 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.174+0000 7ff609be3640 1 -- 172.21.15.62:0/3899205290 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff60410bc80 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.177+0000 7ff600ff9640 1 -- 172.21.15.62:0/3899205290 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ff5f40327b0 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.237+0000 7ff609be3640 1 -- 172.21.15.62:0/3899205290 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7ff6041079d0 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.239+0000 7ff600ff9640 1 -- 172.21.15.62:0/3899205290 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7ff5f403c760 con 0x7ff604107550 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.239+0000 7ff609be3640 1 -- 172.21.15.62:0/3899205290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 msgr2=0x7ff60419b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.239+0000 7ff609be3640 1 --2- 172.21.15.62:0/3899205290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff60419b540 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff5f4009940 tx=0x7ff5f4032ce0 comp rx=0 tx=0).stop 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.239+0000 7ff609be3640 1 -- 172.21.15.62:0/3899205290 shutdown_connections 2024-09-15T23:05:59.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.240+0000 7ff609be3640 1 --2- 172.21.15.62:0/3899205290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff604107550 0x7ff60419b540 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:05:59.762 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.240+0000 7ff609be3640 1 -- 172.21.15.62:0/3899205290 >> 172.21.15.62:0/3899205290 conn(0x7ff604102ce0 msgr2=0x7ff604103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:05:59.762 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.240+0000 7ff609be3640 1 -- 172.21.15.62:0/3899205290 shutdown_connections 2024-09-15T23:05:59.762 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:05:59.240+0000 7ff609be3640 1 -- 172.21.15.62:0/3899205290 wait complete. 2024-09-15T23:05:59.762 INFO:teuthology.orchestra.run.smithi062.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-15T23:05:59.784 INFO:teuthology.orchestra.run.smithi062.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:05:59.784 INFO:teuthology.orchestra.run.smithi062.stdout:Creating mgr... 2024-09-15T23:05:59.784 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-15T23:05:59.785 INFO:teuthology.orchestra.run.smithi062.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-15T23:06:00.037 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.x 2024-09-15T23:06:00.037 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.x.service: Unit ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.x.service not loaded. 2024-09-15T23:06:00.189 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968.target.wants/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@.service. 2024-09-15T23:06:00.424 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:00 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3899205290' entity='client.admin' 2024-09-15T23:06:00.722 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:00 smithi062 podman[31159]: 2024-09-15 23:06:00.649245349 +0000 UTC m=+0.339025606 container start 460390df780a825c934ea1c594a0d6aeaeddabff3663df6b2926bbedb1006eb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-15T23:06:00.736 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T23:06:00.736 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout disabled 2024-09-15T23:06:00.744 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T23:06:00.744 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-09-15T23:06:00.744 INFO:teuthology.orchestra.run.smithi062.stdout:firewalld.service is not enabled 2024-09-15T23:06:00.744 INFO:teuthology.orchestra.run.smithi062.stdout:Not possible to enable service . firewalld.service is not available 2024-09-15T23:06:00.753 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T23:06:00.753 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout disabled 2024-09-15T23:06:00.761 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T23:06:00.761 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-09-15T23:06:00.762 INFO:teuthology.orchestra.run.smithi062.stdout:firewalld.service is not enabled 2024-09-15T23:06:00.762 INFO:teuthology.orchestra.run.smithi062.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-15T23:06:00.762 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mgr to start... 2024-09-15T23:06:00.762 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mgr... 2024-09-15T23:06:00.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:00 smithi062 bash[31159]: 460390df780a825c934ea1c594a0d6aeaeddabff3663df6b2926bbedb1006eb3 2024-09-15T23:06:00.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:00 smithi062 systemd[1]: Started Ceph mgr.x for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:06:00.981 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:00.775+0000 7f2f6ba01040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:06:00.981 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:00.837+0000 7f2f6ba01040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:06:01.282 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:00.995+0000 7f2f6ba01040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:06:01.282 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:01.067+0000 7f2f6ba01040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:06:01.282 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:01.239+0000 7f2f6ba01040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:06:01.636 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:01 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2312079096' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-15T23:06:01.637 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:01.393+0000 7f2f6ba01040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:06:01.903 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:01.728+0000 7f2f6ba01040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:06:01.904 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:01.791+0000 7f2f6ba01040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:06:01.906 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "health": { 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 0 2024-09-15T23:06:01.909 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "a" 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-15T23:06:01.910 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-15T23:06:01.918 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-15T23:06:01.918 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-15T23:06:01.918 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-15T23:06:01.918 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:01.918 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-15T23:06:01.918 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-15T23:06:01.918 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-15T23:06:01.918 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "btime": "2024-09-15T23:05:50:746836+0000", 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-15T23:06:01.919 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "restful" 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modified": "2024-09-15T23:05:50.747500+0000", 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:06:01.920 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.196+0000 7f095f3a7640 1 Processor -- start 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.196+0000 7f095f3a7640 1 -- start start 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.197+0000 7f095f3a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f095810a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.197+0000 7f095f3a7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f095807be50 con 0x7f0958107fa0 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.197+0000 7f095d143640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f095810a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.197+0000 7f095d143640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f095810a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38154/0 (socket says 172.21.15.62:38154) 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.197+0000 7f095d143640 1 -- 172.21.15.62:0/4156714699 learned_addr learned my addr 172.21.15.62:0/4156714699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.197+0000 7f095d143640 1 -- 172.21.15.62:0/4156714699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095810a8d0 con 0x7f0958107fa0 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.198+0000 7f095d143640 1 --2- 172.21.15.62:0/4156714699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f095810a390 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0954009820 tx=0x7f095402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32130c1038aea4ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.198+0000 7f094ffff640 1 -- 172.21.15.62:0/4156714699 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0954037070 con 0x7f0958107fa0 2024-09-15T23:06:01.921 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.198+0000 7f094ffff640 1 -- 172.21.15.62:0/4156714699 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0954032900 con 0x7f0958107fa0 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.198+0000 7f094ffff640 1 -- 172.21.15.62:0/4156714699 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0954030850 con 0x7f0958107fa0 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.199+0000 7f095f3a7640 1 -- 172.21.15.62:0/4156714699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 msgr2=0x7f095810a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.199+0000 7f095f3a7640 1 --2- 172.21.15.62:0/4156714699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f095810a390 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0954009820 tx=0x7f095402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.199+0000 7f095f3a7640 1 -- 172.21.15.62:0/4156714699 shutdown_connections 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.199+0000 7f095f3a7640 1 --2- 172.21.15.62:0/4156714699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f095810a390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.199+0000 7f095f3a7640 1 -- 172.21.15.62:0/4156714699 >> 172.21.15.62:0/4156714699 conn(0x7f0958101bc0 msgr2=0x7f0958104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.199+0000 7f095f3a7640 1 -- 172.21.15.62:0/4156714699 shutdown_connections 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.199+0000 7f095f3a7640 1 -- 172.21.15.62:0/4156714699 wait complete. 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.200+0000 7f095f3a7640 1 Processor -- start 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.200+0000 7f095f3a7640 1 -- start start 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.200+0000 7f095f3a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f0958107820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.200+0000 7f095f3a7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f095810b970 con 0x7f0958107fa0 2024-09-15T23:06:01.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.200+0000 7f095d143640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f0958107820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.200+0000 7f095d143640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f0958107820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38156/0 (socket says 172.21.15.62:38156) 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.200+0000 7f095d143640 1 -- 172.21.15.62:0/2312079096 learned_addr learned my addr 172.21.15.62:0/2312079096 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.200+0000 7f095d143640 1 -- 172.21.15.62:0/2312079096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0958107d60 con 0x7f0958107fa0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.201+0000 7f095d143640 1 --2- 172.21.15.62:0/2312079096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f0958107820 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0954032ce0 tx=0x7f0954032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.201+0000 7f094e7fc640 1 -- 172.21.15.62:0/2312079096 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0954042070 con 0x7f0958107fa0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.201+0000 7f095f3a7640 1 -- 172.21.15.62:0/2312079096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0958105f50 con 0x7f0958107fa0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.201+0000 7f094e7fc640 1 -- 172.21.15.62:0/2312079096 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f09540308c0 con 0x7f0958107fa0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.201+0000 7f094e7fc640 1 -- 172.21.15.62:0/2312079096 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0954037040 con 0x7f0958107fa0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.201+0000 7f095f3a7640 1 -- 172.21.15.62:0/2312079096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0958106490 con 0x7f0958107fa0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.201+0000 7f094e7fc640 1 -- 172.21.15.62:0/2312079096 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0954045070 con 0x7f0958107fa0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.201+0000 7f094e7fc640 1 -- 172.21.15.62:0/2312079096 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0954047bd0 con 0x7f0958107fa0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.202+0000 7f095f3a7640 1 -- 172.21.15.62:0/2312079096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0958108400 con 0x7f0958107fa0 2024-09-15T23:06:01.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.204+0000 7f094e7fc640 1 -- 172.21.15.62:0/2312079096 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f0954047e20 con 0x7f0958107fa0 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.280+0000 7f095f3a7640 1 -- 172.21.15.62:0/2312079096 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f09581069c0 con 0x7f0958107fa0 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.281+0000 7f094e7fc640 1 -- 172.21.15.62:0/2312079096 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f095403c870 con 0x7f0958107fa0 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.282+0000 7f095f3a7640 1 -- 172.21.15.62:0/2312079096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 msgr2=0x7f0958107820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.282+0000 7f095f3a7640 1 --2- 172.21.15.62:0/2312079096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f0958107820 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0954032ce0 tx=0x7f0954032d10 comp rx=0 tx=0).stop 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.282+0000 7f095f3a7640 1 -- 172.21.15.62:0/2312079096 shutdown_connections 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.282+0000 7f095f3a7640 1 --2- 172.21.15.62:0/2312079096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0958107fa0 0x7f0958107820 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.282+0000 7f095f3a7640 1 -- 172.21.15.62:0/2312079096 >> 172.21.15.62:0/2312079096 conn(0x7f0958101bc0 msgr2=0x7f0958102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.282+0000 7f095f3a7640 1 -- 172.21.15.62:0/2312079096 shutdown_connections 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:01.282+0000 7f095f3a7640 1 -- 172.21.15.62:0/2312079096 wait complete. 2024-09-15T23:06:01.924 INFO:teuthology.orchestra.run.smithi062.stdout:mgr not available, waiting (1/15)... 2024-09-15T23:06:02.230 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:01.987+0000 7f2f6ba01040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:06:03.230 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:02 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:02.913+0000 7f2f6ba01040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:06:03.230 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:02 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:02.984+0000 7f2f6ba01040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:06:04.018 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:03 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:03.744+0000 7f2f6ba01040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:06:04.018 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:03 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:03.865+0000 7f2f6ba01040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:06:04.309 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.017+0000 7f2f6ba01040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:06:04.309 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.081+0000 7f2f6ba01040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:06:04.310 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.237+0000 7f2f6ba01040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:06:04.310 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.308+0000 7f2f6ba01040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:06:04.607 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.371+0000 7f2f6ba01040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:06:04.607 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.537+0000 7f2f6ba01040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:06:04.607 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.605+0000 7f2f6ba01040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:06:04.886 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.675+0000 7f2f6ba01040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:06:04.887 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.765+0000 7f2f6ba01040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:06:04.887 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.828+0000 7f2f6ba01040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:06:05.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:05 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3004911447' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-15T23:06:05.230 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:04.894+0000 7f2f6ba01040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:06:05.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:06:05.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:06:05.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:06:05.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "health": { 2024-09-15T23:06:05.648 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 0 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "a" 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:05.649 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:05.650 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "btime": "2024-09-15T23:05:50:746836+0000", 2024-09-15T23:06:05.651 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-15T23:06:05.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-15T23:06:05.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:05.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-15T23:06:05.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-15T23:06:05.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-15T23:06:05.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-15T23:06:05.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-15T23:06:05.652 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "restful" 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modified": "2024-09-15T23:05:50.747500+0000", 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:06:05.653 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.884+0000 7f0a544e5640 1 Processor -- start 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.884+0000 7f0a544e5640 1 -- start start 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.885+0000 7f0a544e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c10a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.885+0000 7f0a544e5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c07be50 con 0x7f0a4c107fa0 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.885+0000 7f0a52281640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c10a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.885+0000 7f0a52281640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c10a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54428/0 (socket says 172.21.15.62:54428) 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.885+0000 7f0a52281640 1 -- 172.21.15.62:0/1335040694 learned_addr learned my addr 172.21.15.62:0/1335040694 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.885+0000 7f0a52281640 1 -- 172.21.15.62:0/1335040694 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c10a8d0 con 0x7f0a4c107fa0 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.886+0000 7f0a52281640 1 --2- 172.21.15.62:0/1335040694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c10a390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009820 tx=0x7f0a4802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8125dbadb45d98ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.887+0000 7f0a5127f640 1 -- 172.21.15.62:0/1335040694 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a48037070 con 0x7f0a4c107fa0 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.887+0000 7f0a5127f640 1 -- 172.21.15.62:0/1335040694 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0a48032900 con 0x7f0a4c107fa0 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.887+0000 7f0a5127f640 1 -- 172.21.15.62:0/1335040694 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a48030850 con 0x7f0a4c107fa0 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.888+0000 7f0a544e5640 1 -- 172.21.15.62:0/1335040694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 msgr2=0x7f0a4c10a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:05.654 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.888+0000 7f0a544e5640 1 --2- 172.21.15.62:0/1335040694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c10a390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009820 tx=0x7f0a4802a270 comp rx=0 tx=0).stop 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.888+0000 7f0a544e5640 1 -- 172.21.15.62:0/1335040694 shutdown_connections 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.888+0000 7f0a544e5640 1 --2- 172.21.15.62:0/1335040694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c10a390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.888+0000 7f0a544e5640 1 -- 172.21.15.62:0/1335040694 >> 172.21.15.62:0/1335040694 conn(0x7f0a4c101bc0 msgr2=0x7f0a4c104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.888+0000 7f0a544e5640 1 -- 172.21.15.62:0/1335040694 shutdown_connections 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.888+0000 7f0a544e5640 1 -- 172.21.15.62:0/1335040694 wait complete. 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.889+0000 7f0a544e5640 1 Processor -- start 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.889+0000 7f0a544e5640 1 -- start start 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.889+0000 7f0a544e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c19b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.889+0000 7f0a544e5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c10b970 con 0x7f0a4c107fa0 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.889+0000 7f0a52281640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c19b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.889+0000 7f0a52281640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c19b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54430/0 (socket says 172.21.15.62:54430) 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.889+0000 7f0a52281640 1 -- 172.21.15.62:0/3004911447 learned_addr learned my addr 172.21.15.62:0/3004911447 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:05.655 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.889+0000 7f0a52281640 1 -- 172.21.15.62:0/3004911447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c19ba70 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.889+0000 7f0a52281640 1 --2- 172.21.15.62:0/3004911447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c19b530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0a480097f0 tx=0x7f0a48030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.890+0000 7f0a437fe640 1 -- 172.21.15.62:0/3004911447 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a48037040 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.890+0000 7f0a437fe640 1 -- 172.21.15.62:0/3004911447 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0a48040070 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.890+0000 7f0a544e5640 1 -- 172.21.15.62:0/3004911447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4c19bd00 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.890+0000 7f0a437fe640 1 -- 172.21.15.62:0/3004911447 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a4803a390 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.890+0000 7f0a544e5640 1 -- 172.21.15.62:0/3004911447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4c19e9f0 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.890+0000 7f0a437fe640 1 -- 172.21.15.62:0/3004911447 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0a4803a6b0 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.891+0000 7f0a437fe640 1 -- 172.21.15.62:0/3004911447 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0a4803b430 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.891+0000 7f0a544e5640 1 -- 172.21.15.62:0/3004911447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a4c108400 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.893+0000 7f0a437fe640 1 -- 172.21.15.62:0/3004911447 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f0a4803b630 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.956+0000 7f0a544e5640 1 -- 172.21.15.62:0/3004911447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0a4c19ec40 con 0x7f0a4c107fa0 2024-09-15T23:06:05.656 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.957+0000 7f0a437fe640 1 -- 172.21.15.62:0/3004911447 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1291 (secure 0 0 0) 0x7f0a4803b810 con 0x7f0a4c107fa0 2024-09-15T23:06:05.657 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.958+0000 7f0a544e5640 1 -- 172.21.15.62:0/3004911447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 msgr2=0x7f0a4c19b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:05.657 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.958+0000 7f0a544e5640 1 --2- 172.21.15.62:0/3004911447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c19b530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0a480097f0 tx=0x7f0a48030d80 comp rx=0 tx=0).stop 2024-09-15T23:06:05.657 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.958+0000 7f0a544e5640 1 -- 172.21.15.62:0/3004911447 shutdown_connections 2024-09-15T23:06:05.657 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.958+0000 7f0a544e5640 1 --2- 172.21.15.62:0/3004911447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a4c107fa0 0x7f0a4c19b530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:05.657 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.958+0000 7f0a544e5640 1 -- 172.21.15.62:0/3004911447 >> 172.21.15.62:0/3004911447 conn(0x7f0a4c101bc0 msgr2=0x7f0a4c102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:05.657 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.958+0000 7f0a544e5640 1 -- 172.21.15.62:0/3004911447 shutdown_connections 2024-09-15T23:06:05.657 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:04.958+0000 7f0a544e5640 1 -- 172.21.15.62:0/3004911447 wait complete. 2024-09-15T23:06:05.657 INFO:teuthology.orchestra.run.smithi062.stdout:mgr not available, waiting (2/15)... 2024-09-15T23:06:05.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:05 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:05.742+0000 7f2f6ba01040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:06:06.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: Activating manager daemon x 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: mgrmap e2: x(active, starting, since 0.00400238s) 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: Manager daemon x is now available 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' 2024-09-15T23:06:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:06 smithi062 ceph-mon[30949]: from='mgr.14100 172.21.15.62:0/2943806534' entity='mgr.x' 2024-09-15T23:06:08.047 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:07 smithi062 ceph-mon[30949]: mgrmap e3: x(active, since 1.00758s) 2024-09-15T23:06:08.917 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:08 smithi062 ceph-mon[30949]: mgrmap e4: x(active, since 2s) 2024-09-15T23:06:08.918 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:08 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2620259373' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-15T23:06:08.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:06:08.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:06:08.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:06:08.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "health": { 2024-09-15T23:06:08.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-15T23:06:08.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-15T23:06:08.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-15T23:06:08.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:08.922 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 0 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "a" 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-15T23:06:08.923 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:08.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-15T23:06:08.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-15T23:06:08.924 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-15T23:06:08.933 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-15T23:06:08.933 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-15T23:06:08.933 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-15T23:06:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "btime": "2024-09-15T23:05:50:746836+0000", 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "restful" 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ], 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:06:08.935 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "modified": "2024-09-15T23:05:50.747500+0000", 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout }, 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.058+0000 7f12aefea640 1 Processor -- start 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.058+0000 7f12aefea640 1 -- start start 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.059+0000 7f12aefea640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8106830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.059+0000 7f12aefea640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f12a8106e00 con 0x7f12a8106430 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.059+0000 7f12acd86640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.059+0000 7f12acd86640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54502/0 (socket says 172.21.15.62:54502) 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.059+0000 7f12acd86640 1 -- 172.21.15.62:0/301578163 learned_addr learned my addr 172.21.15.62:0/301578163 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.059+0000 7f12acd86640 1 -- 172.21.15.62:0/301578163 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a8107630 con 0x7f12a8106430 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.060+0000 7f12acd86640 1 --2- 172.21.15.62:0/301578163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8106830 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1290009820 tx=0x7f129002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc3fa910fe0681c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.060+0000 7f12a77fe640 1 -- 172.21.15.62:0/301578163 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1290037070 con 0x7f12a8106430 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.060+0000 7f12a77fe640 1 -- 172.21.15.62:0/301578163 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1290032900 con 0x7f12a8106430 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.060+0000 7f12a77fe640 1 -- 172.21.15.62:0/301578163 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1290030850 con 0x7f12a8106430 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.061+0000 7f12aefea640 1 -- 172.21.15.62:0/301578163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 msgr2=0x7f12a8106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.061+0000 7f12aefea640 1 --2- 172.21.15.62:0/301578163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8106830 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1290009820 tx=0x7f129002a270 comp rx=0 tx=0).stop 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.061+0000 7f12aefea640 1 -- 172.21.15.62:0/301578163 shutdown_connections 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.061+0000 7f12aefea640 1 --2- 172.21.15.62:0/301578163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8106830 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.061+0000 7f12aefea640 1 -- 172.21.15.62:0/301578163 >> 172.21.15.62:0/301578163 conn(0x7f12a8101bc0 msgr2=0x7f12a8104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.061+0000 7f12aefea640 1 -- 172.21.15.62:0/301578163 shutdown_connections 2024-09-15T23:06:08.937 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.061+0000 7f12aefea640 1 -- 172.21.15.62:0/301578163 wait complete. 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.062+0000 7f12aefea640 1 Processor -- start 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.062+0000 7f12aefea640 1 -- start start 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.062+0000 7f12aefea640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8191660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.062+0000 7f12aefea640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f12a810a9e0 con 0x7f12a8106430 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.062+0000 7f12acd86640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8191660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.062+0000 7f12acd86640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8191660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54504/0 (socket says 172.21.15.62:54504) 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.062+0000 7f12acd86640 1 -- 172.21.15.62:0/2620259373 learned_addr learned my addr 172.21.15.62:0/2620259373 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.063+0000 7f12acd86640 1 -- 172.21.15.62:0/2620259373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a8191ba0 con 0x7f12a8106430 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.063+0000 7f12acd86640 1 --2- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8191660 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f12900097f0 tx=0x7f1290030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.063+0000 7f12a5ffb640 1 -- 172.21.15.62:0/2620259373 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1290037040 con 0x7f12a8106430 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.063+0000 7f12a5ffb640 1 -- 172.21.15.62:0/2620259373 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1290040070 con 0x7f12a8106430 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.063+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a818fd90 con 0x7f12a8106430 2024-09-15T23:06:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.063+0000 7f12a5ffb640 1 -- 172.21.15.62:0/2620259373 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f129003a390 con 0x7f12a8106430 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.063+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a8190270 con 0x7f12a8106430 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.064+0000 7f12a5ffb640 1 -- 172.21.15.62:0/2620259373 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7f129003a630 con 0x7f12a8106430 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.064+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12a810ab60 con 0x7f12a8106430 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.064+0000 7f12a5ffb640 1 --2- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f1294039160 0x7f129403b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.064+0000 7f12a5ffb640 1 -- 172.21.15.62:0/2620259373 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f129006c4f0 con 0x7f12a8106430 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.065+0000 7f12a7fff640 1 --2- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f1294039160 0x7f129403b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.065+0000 7f12a7fff640 1 --2- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f1294039160 0x7f129403b620 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f129800ad30 tx=0x7f12980093f0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.070+0000 7f12a5ffb640 1 -- 172.21.15.62:0/2620259373 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f129006c250 con 0x7f12a8106430 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.312+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f12a81953c0 con 0x7f12a8106430 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.313+0000 7f12a5ffb640 1 -- 172.21.15.62:0/2620259373 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1290 (secure 0 0 0) 0x7f1290043760 con 0x7f12a8106430 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.315+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f1294039160 msgr2=0x7f129403b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.315+0000 7f12aefea640 1 --2- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f1294039160 0x7f129403b620 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f129800ad30 tx=0x7f12980093f0 comp rx=0 tx=0).stop 2024-09-15T23:06:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.315+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 msgr2=0x7f12a8191660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:08.940 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.315+0000 7f12aefea640 1 --2- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8191660 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f12900097f0 tx=0x7f1290030d80 comp rx=0 tx=0).stop 2024-09-15T23:06:08.940 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.315+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 shutdown_connections 2024-09-15T23:06:08.940 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.315+0000 7f12aefea640 1 --2- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f1294039160 0x7f129403b620 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:08.940 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.315+0000 7f12aefea640 1 --2- 172.21.15.62:0/2620259373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12a8106430 0x7f12a8191660 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:08.940 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.315+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 >> 172.21.15.62:0/2620259373 conn(0x7f12a8101bc0 msgr2=0x7f12a8102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:08.940 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.316+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 shutdown_connections 2024-09-15T23:06:08.940 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:08.316+0000 7f12aefea640 1 -- 172.21.15.62:0/2620259373 wait complete. 2024-09-15T23:06:08.940 INFO:teuthology.orchestra.run.smithi062.stdout:mgr is available 2024-09-15T23:06:10.024 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:09 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3364921000' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [global] 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout fsid = ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.62:3300,v1:172.21.15.62:6789] 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:06:10.033 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout [osd] 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.283+0000 7fbe296d8640 1 Processor -- start 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.283+0000 7fbe296d8640 1 -- start start 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.283+0000 7fbe296d8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe24106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.283+0000 7fbe296d8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbe24106e20 con 0x7fbe24106450 2024-09-15T23:06:10.034 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.283+0000 7fbe22ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe24106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.283+0000 7fbe22ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe24106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54516/0 (socket says 172.21.15.62:54516) 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.283+0000 7fbe22ffd640 1 -- 172.21.15.62:0/1701271090 learned_addr learned my addr 172.21.15.62:0/1701271090 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.284+0000 7fbe22ffd640 1 -- 172.21.15.62:0/1701271090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe24107650 con 0x7fbe24106450 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.284+0000 7fbe22ffd640 1 --2- 172.21.15.62:0/1701271090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe24106850 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009820 tx=0x7fbe1402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af304d085b7693f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.285+0000 7fbe227fc640 1 -- 172.21.15.62:0/1701271090 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe14037070 con 0x7fbe24106450 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.285+0000 7fbe227fc640 1 -- 172.21.15.62:0/1701271090 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbe14032900 con 0x7fbe24106450 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.285+0000 7fbe227fc640 1 -- 172.21.15.62:0/1701271090 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe14030850 con 0x7fbe24106450 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 -- 172.21.15.62:0/1701271090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 msgr2=0x7fbe24106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 --2- 172.21.15.62:0/1701271090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe24106850 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009820 tx=0x7fbe1402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 -- 172.21.15.62:0/1701271090 shutdown_connections 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 --2- 172.21.15.62:0/1701271090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe24106850 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 -- 172.21.15.62:0/1701271090 >> 172.21.15.62:0/1701271090 conn(0x7fbe24101be0 msgr2=0x7fbe24104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:10.035 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 -- 172.21.15.62:0/1701271090 shutdown_connections 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 -- 172.21.15.62:0/1701271090 wait complete. 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 Processor -- start 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 -- start start 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.286+0000 7fbe296d8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe2419b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe296d8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2410aa00 con 0x7fbe24106450 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe22ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe2419b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe22ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe2419b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54522/0 (socket says 172.21.15.62:54522) 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe22ffd640 1 -- 172.21.15.62:0/3364921000 learned_addr learned my addr 172.21.15.62:0/3364921000 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe22ffd640 1 -- 172.21.15.62:0/3364921000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2419bb10 con 0x7fbe24106450 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe22ffd640 1 --2- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe2419b5d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009940 tx=0x7fbe14032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe20ff9640 1 -- 172.21.15.62:0/3364921000 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe14042070 con 0x7fbe24106450 2024-09-15T23:06:10.036 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe20ff9640 1 -- 172.21.15.62:0/3364921000 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbe140308c0 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe2419bda0 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe20ff9640 1 -- 172.21.15.62:0/3364921000 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe14037040 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.287+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe2419ea90 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.288+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe2410ab80 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.288+0000 7fbe20ff9640 1 -- 172.21.15.62:0/3364921000 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7fbe14045070 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.289+0000 7fbe20ff9640 1 --2- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7fbe100391b0 0x7fbe1003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.289+0000 7fbe20ff9640 1 -- 172.21.15.62:0/3364921000 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbe1406c1c0 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.289+0000 7fbe1bfff640 1 --2- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7fbe100391b0 0x7fbe1003b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.289+0000 7fbe1bfff640 1 --2- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7fbe100391b0 0x7fbe1003b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0098c0 tx=0x7fbe0c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.298+0000 7fbe20ff9640 1 -- 172.21.15.62:0/3364921000 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbe14045360 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.472+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fbe241068d0 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.473+0000 7fbe20ff9640 1 -- 172.21.15.62:0/3364921000 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+429 (secure 0 0 0) 0x7fbe1402ed90 con 0x7fbe24106450 2024-09-15T23:06:10.037 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7fbe100391b0 msgr2=0x7fbe1003b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 --2- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7fbe100391b0 0x7fbe1003b670 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0098c0 tx=0x7fbe0c006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 msgr2=0x7fbe2419b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 --2- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe2419b5d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009940 tx=0x7fbe14032ce0 comp rx=0 tx=0).stop 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 shutdown_connections 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 --2- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7fbe100391b0 0x7fbe1003b670 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 --2- 172.21.15.62:0/3364921000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe24106450 0x7fbe2419b5d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 >> 172.21.15.62:0/3364921000 conn(0x7fbe24101be0 msgr2=0x7fbe24102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 shutdown_connections 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:09.475+0000 7fbe296d8640 1 -- 172.21.15.62:0/3364921000 wait complete. 2024-09-15T23:06:10.038 INFO:teuthology.orchestra.run.smithi062.stdout:Enabling cephadm module... 2024-09-15T23:06:10.959 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:10 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2049782677' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-15T23:06:10.959 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:10 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: ignoring --setuser ceph since I am not root 2024-09-15T23:06:10.959 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:10 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: ignoring --setgroup ceph since I am not root 2024-09-15T23:06:10.959 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:10 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:10.898+0000 7f6942b2c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:06:10.959 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:10 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:10.957+0000 7f6942b2c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:06:11.232 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:11 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:11.114+0000 7f6942b2c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:06:11.233 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:11 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:11.183+0000 7f6942b2c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:06:11.296 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.432+0000 7f12d25a4640 1 Processor -- start 2024-09-15T23:06:11.296 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.432+0000 7f12d25a4640 1 -- start start 2024-09-15T23:06:11.296 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.433+0000 7f12d25a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:11.296 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.433+0000 7f12d25a4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc07c890 con 0x7f12cc07de60 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.433+0000 7f12cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.433+0000 7f12cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54536/0 (socket says 172.21.15.62:54536) 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.433+0000 7f12cbfff640 1 -- 172.21.15.62:0/4040197329 learned_addr learned my addr 172.21.15.62:0/4040197329 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.433+0000 7f12cbfff640 1 -- 172.21.15.62:0/4040197329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12cc07ca10 con 0x7f12cc07de60 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.434+0000 7f12cbfff640 1 --2- 172.21.15.62:0/4040197329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc07c2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f12bc009820 tx=0x7f12bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fae2251c3e643da server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.434+0000 7f12caffd640 1 -- 172.21.15.62:0/4040197329 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12bc037070 con 0x7f12cc07de60 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.434+0000 7f12caffd640 1 -- 172.21.15.62:0/4040197329 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f12bc032900 con 0x7f12cc07de60 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.434+0000 7f12caffd640 1 -- 172.21.15.62:0/4040197329 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12bc030850 con 0x7f12cc07de60 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.435+0000 7f12d25a4640 1 -- 172.21.15.62:0/4040197329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 msgr2=0x7f12cc07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.435+0000 7f12d25a4640 1 --2- 172.21.15.62:0/4040197329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc07c2c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f12bc009820 tx=0x7f12bc02a270 comp rx=0 tx=0).stop 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.435+0000 7f12d25a4640 1 -- 172.21.15.62:0/4040197329 shutdown_connections 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.435+0000 7f12d25a4640 1 --2- 172.21.15.62:0/4040197329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc07c2c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:11.297 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.435+0000 7f12d25a4640 1 -- 172.21.15.62:0/4040197329 >> 172.21.15.62:0/4040197329 conn(0x7f12cc102cb0 msgr2=0x7f12cc105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.435+0000 7f12d25a4640 1 -- 172.21.15.62:0/4040197329 shutdown_connections 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.435+0000 7f12d25a4640 1 -- 172.21.15.62:0/4040197329 wait complete. 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.436+0000 7f12d25a4640 1 Processor -- start 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.436+0000 7f12d25a4640 1 -- start start 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.436+0000 7f12d25a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc19d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.436+0000 7f12d25a4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f12cc10dbb0 con 0x7f12cc07de60 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.436+0000 7f12cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc19d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.436+0000 7f12cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc19d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54538/0 (socket says 172.21.15.62:54538) 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.436+0000 7f12cbfff640 1 -- 172.21.15.62:0/2049782677 learned_addr learned my addr 172.21.15.62:0/2049782677 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.436+0000 7f12cbfff640 1 -- 172.21.15.62:0/2049782677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12cc19dce0 con 0x7f12cc07de60 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.437+0000 7f12cbfff640 1 --2- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc19d7a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f12bc009940 tx=0x7f12bc032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:11.298 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.437+0000 7f12c97fa640 1 -- 172.21.15.62:0/2049782677 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12bc042070 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.437+0000 7f12c97fa640 1 -- 172.21.15.62:0/2049782677 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f12bc0308c0 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.437+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12cc19df70 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.437+0000 7f12c97fa640 1 -- 172.21.15.62:0/2049782677 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12bc037040 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.437+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12cc1a0c60 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.438+0000 7f12c97fa640 1 -- 172.21.15.62:0/2049782677 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) ==== 46450+0+0 (secure 0 0 0) 0x7f12bc045070 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.438+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12cc07c340 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.438+0000 7f12c97fa640 1 --2- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f12b40391b0 0x7f12b403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.438+0000 7f12c97fa640 1 -- 172.21.15.62:0/2049782677 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f12bc06c1b0 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.438+0000 7f12cb7fe640 1 --2- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f12b40391b0 0x7f12b403b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.438+0000 7f12cb7fe640 1 --2- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f12b40391b0 0x7f12b403b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f12c00098c0 tx=0x7f12c0006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.443+0000 7f12c97fa640 1 -- 172.21.15.62:0/2049782677 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f12bc045360 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.668+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f12cc1a12e0 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.755+0000 7f12c97fa640 1 -- 172.21.15.62:0/2049782677 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f12bc02ed90 con 0x7f12cc07de60 2024-09-15T23:06:11.299 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.757+0000 7f12c97fa640 1 -- 172.21.15.62:0/2049782677 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f12bc03c760 con 0x7f12cc07de60 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f12b40391b0 msgr2=0x7f12b403b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 --2- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f12b40391b0 0x7f12b403b670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f12c00098c0 tx=0x7f12c0006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 msgr2=0x7f12cc19d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 --2- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc19d7a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f12bc009940 tx=0x7f12bc032ce0 comp rx=0 tx=0).stop 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 shutdown_connections 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 --2- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f12b40391b0 0x7f12b403b670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 --2- 172.21.15.62:0/2049782677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12cc07de60 0x7f12cc19d7a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 >> 172.21.15.62:0/2049782677 conn(0x7f12cc102cb0 msgr2=0x7f12cc1038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 shutdown_connections 2024-09-15T23:06:11.300 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:10.758+0000 7f12d25a4640 1 -- 172.21.15.62:0/2049782677 wait complete. 2024-09-15T23:06:11.539 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:11 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:11.345+0000 7f6942b2c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:06:11.540 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:11 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:11.494+0000 7f6942b2c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:06:11.832 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:11 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2049782677' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-15T23:06:11.832 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:11 smithi062 ceph-mon[30949]: mgrmap e5: x(active, since 5s) 2024-09-15T23:06:12.094 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:11 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:11.831+0000 7f6942b2c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:06:12.094 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:11 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:11.895+0000 7f6942b2c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:06:12.094 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:12 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:12.092+0000 7f6942b2c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:06:12.539 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:06:12.539 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-15T23:06:12.539 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-15T23:06:12.539 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-15T23:06:12.539 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-15T23:06:12.539 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:06:12.539 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.702+0000 7f9f28c5c640 1 Processor -- start 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.702+0000 7f9f28c5c640 1 -- start start 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.703+0000 7f9f28c5c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2410a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.703+0000 7f9f28c5c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2407be50 con 0x7f9f24107fa0 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.703+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2410a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.703+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2410a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54568/0 (socket says 172.21.15.62:54568) 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.703+0000 7f9f2259c640 1 -- 172.21.15.62:0/1686307545 learned_addr learned my addr 172.21.15.62:0/1686307545 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.703+0000 7f9f2259c640 1 -- 172.21.15.62:0/1686307545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2410a8d0 con 0x7f9f24107fa0 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.704+0000 7f9f2259c640 1 --2- 172.21.15.62:0/1686307545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2410a390 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009820 tx=0x7f9f1402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18179ee70d247e23 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.704+0000 7f9f21d9b640 1 -- 172.21.15.62:0/1686307545 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f14037070 con 0x7f9f24107fa0 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.704+0000 7f9f21d9b640 1 -- 172.21.15.62:0/1686307545 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f14032900 con 0x7f9f24107fa0 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.704+0000 7f9f21d9b640 1 -- 172.21.15.62:0/1686307545 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f14030850 con 0x7f9f24107fa0 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.705+0000 7f9f28c5c640 1 -- 172.21.15.62:0/1686307545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 msgr2=0x7f9f2410a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:12.540 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.705+0000 7f9f28c5c640 1 --2- 172.21.15.62:0/1686307545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2410a390 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009820 tx=0x7f9f1402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.705+0000 7f9f28c5c640 1 -- 172.21.15.62:0/1686307545 shutdown_connections 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.705+0000 7f9f28c5c640 1 --2- 172.21.15.62:0/1686307545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2410a390 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.705+0000 7f9f28c5c640 1 -- 172.21.15.62:0/1686307545 >> 172.21.15.62:0/1686307545 conn(0x7f9f24101bc0 msgr2=0x7f9f24104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.705+0000 7f9f28c5c640 1 -- 172.21.15.62:0/1686307545 shutdown_connections 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.705+0000 7f9f28c5c640 1 -- 172.21.15.62:0/1686307545 wait complete. 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.706+0000 7f9f28c5c640 1 Processor -- start 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.706+0000 7f9f28c5c640 1 -- start start 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.706+0000 7f9f28c5c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2419b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.706+0000 7f9f28c5c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2410b970 con 0x7f9f24107fa0 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.706+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2419b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.706+0000 7f9f2259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2419b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54576/0 (socket says 172.21.15.62:54576) 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.706+0000 7f9f2259c640 1 -- 172.21.15.62:0/3372415273 learned_addr learned my addr 172.21.15.62:0/3372415273 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.706+0000 7f9f2259c640 1 -- 172.21.15.62:0/3372415273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2419ba70 con 0x7f9f24107fa0 2024-09-15T23:06:12.541 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.706+0000 7f9f2259c640 1 --2- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2419b530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9f14032ce0 tx=0x7f9f14032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.707+0000 7f9f1b7fe640 1 -- 172.21.15.62:0/3372415273 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f14042070 con 0x7f9f24107fa0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.707+0000 7f9f1b7fe640 1 -- 172.21.15.62:0/3372415273 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f140308c0 con 0x7f9f24107fa0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.707+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2419bd00 con 0x7f9f24107fa0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.707+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f2419e9f0 con 0x7f9f24107fa0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.707+0000 7f9f1b7fe640 1 -- 172.21.15.62:0/3372415273 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f14037040 con 0x7f9f24107fa0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.708+0000 7f9f1b7fe640 1 -- 172.21.15.62:0/3372415273 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f9f14045070 con 0x7f9f24107fa0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.708+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f040052c0 con 0x7f9f24107fa0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.708+0000 7f9f1b7fe640 1 --2- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f9f100391b0 0x7f9f1003b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.708+0000 7f9f1b7fe640 1 -- 172.21.15.62:0/3372415273 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9f1406c260 con 0x7f9f24107fa0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.708+0000 7f9f1bfff640 1 -- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f9f100391b0 msgr2=0x7f9f1003b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1277095060 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.708+0000 7f9f1bfff640 1 --2- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f9f100391b0 0x7f9f1003b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.713+0000 7f9f1b7fe640 1 -- 172.21.15.62:0/3372415273 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9f1403cdd0 con 0x7f9f24107fa0 2024-09-15T23:06:12.542 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.909+0000 7f9f1bfff640 1 -- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f9f100391b0 msgr2=0x7f9f1003b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1277095060 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.909+0000 7f9f1bfff640 1 --2- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f9f100391b0 0x7f9f1003b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.938+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f9f04005dc0 con 0x7f9f24107fa0 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.938+0000 7f9f1b7fe640 1 -- 172.21.15.62:0/3372415273 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f9f1403c870 con 0x7f9f24107fa0 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f9f100391b0 msgr2=0x7f9f1003b670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 --2- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f9f100391b0 0x7f9f1003b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 msgr2=0x7f9f2419b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 --2- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2419b530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9f14032ce0 tx=0x7f9f14032d10 comp rx=0 tx=0).stop 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 shutdown_connections 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 --2- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f9f100391b0 0x7f9f1003b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 --2- 172.21.15.62:0/3372415273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f24107fa0 0x7f9f2419b530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 >> 172.21.15.62:0/3372415273 conn(0x7f9f24101bc0 msgr2=0x7f9f24102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:12.543 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 shutdown_connections 2024-09-15T23:06:12.544 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:11.940+0000 7f9f28c5c640 1 -- 172.21.15.62:0/3372415273 wait complete. 2024-09-15T23:06:12.544 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for the mgr to restart... 2024-09-15T23:06:12.544 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mgr epoch 5... 2024-09-15T23:06:12.849 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:12 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3372415273' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-15T23:06:13.107 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:13 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:13.035+0000 7f6942b2c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:06:13.107 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:13 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:13.105+0000 7f6942b2c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:06:14.121 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:13 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:13.850+0000 7f6942b2c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:06:14.121 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:13 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:13.971+0000 7f6942b2c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:06:14.412 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.119+0000 7f6942b2c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:06:14.413 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.183+0000 7f6942b2c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:06:14.413 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.340+0000 7f6942b2c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:06:14.413 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.411+0000 7f6942b2c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:06:14.709 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.474+0000 7f6942b2c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:06:14.709 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.639+0000 7f6942b2c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:06:14.709 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.707+0000 7f6942b2c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:06:14.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.777+0000 7f6942b2c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:06:14.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.863+0000 7f6942b2c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:06:14.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.925+0000 7f6942b2c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:06:15.480 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:14.987+0000 7f6942b2c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:06:16.230 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:15.813+0000 7f6942b2c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:06:16.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: Active manager daemon x restarted 2024-09-15T23:06:16.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: Activating manager daemon x 2024-09-15T23:06:16.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: osdmap e2: 0 total, 0 up, 0 in 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: mgrmap e6: x(active, starting, since 0.00379863s) 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: Manager daemon x is now available 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T23:06:16.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:15 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T23:06:17.393 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.927+0000 7f5c8cf5a640 1 Processor -- start 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.927+0000 7f5c8cf5a640 1 -- start start 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.927+0000 7f5c8cf5a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c8810a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.927+0000 7f5c8cf5a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8807be50 con 0x7f5c88108010 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.928+0000 7f5c8659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c8810a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.928+0000 7f5c8659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c8810a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38914/0 (socket says 172.21.15.62:38914) 2024-09-15T23:06:17.394 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.928+0000 7f5c8659c640 1 -- 172.21.15.62:0/3861232963 learned_addr learned my addr 172.21.15.62:0/3861232963 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.928+0000 7f5c8659c640 1 -- 172.21.15.62:0/3861232963 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8810a940 con 0x7f5c88108010 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.929+0000 7f5c8659c640 1 --2- 172.21.15.62:0/3861232963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c8810a400 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009820 tx=0x7f5c7802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c84c3cd2e942767 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.929+0000 7f5c85d9b640 1 -- 172.21.15.62:0/3861232963 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c78037070 con 0x7f5c88108010 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.929+0000 7f5c85d9b640 1 -- 172.21.15.62:0/3861232963 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5c78032900 con 0x7f5c88108010 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.929+0000 7f5c85d9b640 1 -- 172.21.15.62:0/3861232963 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c78030850 con 0x7f5c88108010 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.930+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/3861232963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 msgr2=0x7f5c8810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.930+0000 7f5c8cf5a640 1 --2- 172.21.15.62:0/3861232963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c8810a400 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009820 tx=0x7f5c7802a270 comp rx=0 tx=0).stop 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.930+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/3861232963 shutdown_connections 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.930+0000 7f5c8cf5a640 1 --2- 172.21.15.62:0/3861232963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c8810a400 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.930+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/3861232963 >> 172.21.15.62:0/3861232963 conn(0x7f5c88101c10 msgr2=0x7f5c88104070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.930+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/3861232963 shutdown_connections 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.930+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/3861232963 wait complete. 2024-09-15T23:06:17.395 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.930+0000 7f5c8cf5a640 1 Processor -- start 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.931+0000 7f5c8cf5a640 1 -- start start 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.931+0000 7f5c8cf5a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c88107890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.931+0000 7f5c8cf5a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8810b9e0 con 0x7f5c88108010 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.931+0000 7f5c8659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c88107890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.931+0000 7f5c8659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c88107890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38924/0 (socket says 172.21.15.62:38924) 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.931+0000 7f5c8659c640 1 -- 172.21.15.62:0/1741355509 learned_addr learned my addr 172.21.15.62:0/1741355509 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.931+0000 7f5c8659c640 1 -- 172.21.15.62:0/1741355509 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c88107dd0 con 0x7f5c88108010 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.931+0000 7f5c8659c640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c88107890 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5c78031000 tx=0x7f5c780326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.931+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c78042070 con 0x7f5c88108010 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.932+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5c78032be0 con 0x7f5c88108010 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.932+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/1741355509 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c88105fc0 con 0x7f5c88108010 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.932+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/1741355509 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c88106500 con 0x7f5c88108010 2024-09-15T23:06:17.396 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.932+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5c78037040 con 0x7f5c88108010 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.933+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 46461+0+0 (secure 0 0 0) 0x7f5c78045070 con 0x7f5c88108010 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.933+0000 7f5c7f7fe640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 0x7f5c7403b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.933+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 --> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5c7403bd80 con 0x7f5c740391b0 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.933+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5c7806c110 con 0x7f5c88108010 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.933+0000 7f5c7ffff640 1 -- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 msgr2=0x7f5c7403b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1277095060 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:12.933+0000 7f5c7ffff640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 0x7f5c7403b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:13.134+0000 7f5c7ffff640 1 -- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 msgr2=0x7f5c7403b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1277095060 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:13.134+0000 7f5c7ffff640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 0x7f5c7403b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:13.535+0000 7f5c7ffff640 1 -- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 msgr2=0x7f5c7403b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1277095060 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:13.535+0000 7f5c7ffff640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 0x7f5c7403b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:14.336+0000 7f5c7ffff640 1 -- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 msgr2=0x7f5c7403b670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/1277095060 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:14.336+0000 7f5c7ffff640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 0x7f5c7403b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T23:06:17.397 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:15.817+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mgrmap(e 6) ==== 46228+0+0 (secure 0 0 0) 0x7f5c7802ed70 con 0x7f5c88108010 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:15.817+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 msgr2=0x7f5c7403b670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:15.817+0000 7f5c7f7fe640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 0x7f5c7403b670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.817+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7f5c7803ad40 con 0x7f5c88108010 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.817+0000 7f5c7f7fe640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f5c7403cae0 0x7f5c7403eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.817+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5c7802bba0 con 0x7f5c7403cae0 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.818+0000 7f5c7ffff640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f5c7403cae0 0x7f5c7403eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.818+0000 7f5c7ffff640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f5c7403cae0 0x7f5c7403eed0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5c70003980 tx=0x7f5c70006e10 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.819+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 <== mgr.14118 v2:172.21.15.62:6800/2614061737 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f5c7802bba0 con 0x7f5c7403cae0 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.826+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/1741355509 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f5c680025a0 con 0x7f5c7403cae0 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.826+0000 7f5c7f7fe640 1 -- 172.21.15.62:0/1741355509 <== mgr.14118 v2:172.21.15.62:6800/2614061737 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f5c680025a0 con 0x7f5c7403cae0 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f5c7403cae0 msgr2=0x7f5c7403eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f5c7403cae0 0x7f5c7403eed0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5c70003980 tx=0x7f5c70006e10 comp rx=0 tx=0).stop 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 msgr2=0x7f5c88107890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:17.398 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c88107890 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5c78031000 tx=0x7f5c780326e0 comp rx=0 tx=0).stop 2024-09-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/1741355509 shutdown_connections 2024-09-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f5c7403cae0 0x7f5c7403eed0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:6800/1277095060,v1:172.21.15.62:6801/1277095060] conn(0x7f5c740391b0 0x7f5c7403b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 --2- 172.21.15.62:0/1741355509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c88108010 0x7f5c88107890 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/1741355509 >> 172.21.15.62:0/1741355509 conn(0x7f5c88101c10 msgr2=0x7f5c88102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/1741355509 shutdown_connections 2024-09-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:16.827+0000 7f5c8cf5a640 1 -- 172.21.15.62:0/1741355509 wait complete. 2024-09-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi062.stdout:mgr epoch 5 is available 2024-09-15T23:06:17.399 INFO:teuthology.orchestra.run.smithi062.stdout:Setting orchestrator backend to cephadm... 2024-09-15T23:06:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:17 smithi062 ceph-mon[30949]: Found migration_current of "None". Setting to last migration. 2024-09-15T23:06:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:17 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:17 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:17 smithi062 ceph-mon[30949]: mgrmap e7: x(active, since 1.00514s) 2024-09-15T23:06:18.558 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:18.299+0000 7f68a9e46640 -1 log_channel(cephadm) log [ERR] : [15/Sep/2024:23:06:18] ENGINE Error in HTTPServer.serve 2024-09-15T23:06:18.558 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: Traceback (most recent call last): 2024-09-15T23:06:18.558 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T23:06:18.558 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: self._connections.run(self.expiration_interval) 2024-09-15T23:06:18.558 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T23:06:18.558 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: self._run(expiration_interval) 2024-09-15T23:06:18.558 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T23:06:18.558 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: s = self.context.wrap_socket( 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: return self.sslsocket_class._create( 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: self.do_handshake() 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: self._sslobj.do_handshake() 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T23:06:18.559 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.791+0000 7fa4da037640 1 Processor -- start 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.791+0000 7fa4da037640 1 -- start start 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.791+0000 7fa4da037640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d410a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.791+0000 7fa4da037640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d407bdd0 con 0x7fa4d4108020 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.791+0000 7fa4d37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d410a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.791+0000 7fa4d37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d410a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38998/0 (socket says 172.21.15.62:38998) 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.791+0000 7fa4d37fe640 1 -- 172.21.15.62:0/3768890378 learned_addr learned my addr 172.21.15.62:0/3768890378 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.791+0000 7fa4d37fe640 1 -- 172.21.15.62:0/3768890378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d410a950 con 0x7fa4d4108020 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.792+0000 7fa4d37fe640 1 --2- 172.21.15.62:0/3768890378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d410a410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4009820 tx=0x7fa4c402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0054665b48f9f4b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:18.565 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.792+0000 7fa4d27fc640 1 -- 172.21.15.62:0/3768890378 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4c4037070 con 0x7fa4d4108020 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.792+0000 7fa4d27fc640 1 -- 172.21.15.62:0/3768890378 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa4c4032900 con 0x7fa4d4108020 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.792+0000 7fa4d27fc640 1 -- 172.21.15.62:0/3768890378 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4c4030850 con 0x7fa4d4108020 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.793+0000 7fa4da037640 1 -- 172.21.15.62:0/3768890378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 msgr2=0x7fa4d410a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.793+0000 7fa4da037640 1 --2- 172.21.15.62:0/3768890378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d410a410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4009820 tx=0x7fa4c402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.793+0000 7fa4da037640 1 -- 172.21.15.62:0/3768890378 shutdown_connections 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.793+0000 7fa4da037640 1 --2- 172.21.15.62:0/3768890378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d410a410 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.793+0000 7fa4da037640 1 -- 172.21.15.62:0/3768890378 >> 172.21.15.62:0/3768890378 conn(0x7fa4d4101bc0 msgr2=0x7fa4d4104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.793+0000 7fa4da037640 1 -- 172.21.15.62:0/3768890378 shutdown_connections 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.793+0000 7fa4da037640 1 -- 172.21.15.62:0/3768890378 wait complete. 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.794+0000 7fa4da037640 1 Processor -- start 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.794+0000 7fa4da037640 1 -- start start 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.794+0000 7fa4da037640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d41078a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:18.566 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.794+0000 7fa4da037640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d410b930 con 0x7fa4d4108020 2024-09-15T23:06:18.567 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.794+0000 7fa4d37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d41078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:18.567 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.794+0000 7fa4d37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d41078a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39000/0 (socket says 172.21.15.62:39000) 2024-09-15T23:06:18.567 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.794+0000 7fa4d37fe640 1 -- 172.21.15.62:0/685448230 learned_addr learned my addr 172.21.15.62:0/685448230 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:18.567 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.795+0000 7fa4d37fe640 1 -- 172.21.15.62:0/685448230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d4107de0 con 0x7fa4d4108020 2024-09-15T23:06:18.567 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.795+0000 7fa4d37fe640 1 --2- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d41078a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40097f0 tx=0x7fa4c4030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:18.567 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.795+0000 7fa4d0ff9640 1 -- 172.21.15.62:0/685448230 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4c4037040 con 0x7fa4d4108020 2024-09-15T23:06:18.567 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.795+0000 7fa4d0ff9640 1 -- 172.21.15.62:0/685448230 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa4c4040070 con 0x7fa4d4108020 2024-09-15T23:06:18.567 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.795+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d4105fd0 con 0x7fa4d4108020 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.795+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d41064b0 con 0x7fa4d4108020 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.795+0000 7fa4d0ff9640 1 -- 172.21.15.62:0/685448230 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4c403a390 con 0x7fa4d4108020 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.796+0000 7fa4d0ff9640 1 -- 172.21.15.62:0/685448230 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 7) ==== 46355+0+0 (secure 0 0 0) 0x7fa4c403a6b0 con 0x7fa4d4108020 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.796+0000 7fa4d0ff9640 1 --2- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fa4bc0390a0 0x7fa4bc03b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.796+0000 7fa4d0ff9640 1 -- 172.21.15.62:0/685448230 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa4c406c620 con 0x7fa4d4108020 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.796+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d4108480 con 0x7fa4d4108020 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.796+0000 7fa4d2ffd640 1 --2- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fa4bc0390a0 0x7fa4bc03b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.797+0000 7fa4d2ffd640 1 --2- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fa4bc0390a0 0x7fa4bc03b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80098c0 tx=0x7fa4c8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:17.801+0000 7fa4d0ff9640 1 -- 172.21.15.62:0/685448230 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa4c404a200 con 0x7fa4d4108020 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.007+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fa4d40008d0 con 0x7fa4bc0390a0 2024-09-15T23:06:18.568 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.013+0000 7fa4d0ff9640 1 -- 172.21.15.62:0/685448230 <== mgr.14118 v2:172.21.15.62:6800/2614061737 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa4d40008d0 con 0x7fa4bc0390a0 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fa4bc0390a0 msgr2=0x7fa4bc03b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 --2- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fa4bc0390a0 0x7fa4bc03b560 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80098c0 tx=0x7fa4c8006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 msgr2=0x7fa4d41078a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 --2- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d41078a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40097f0 tx=0x7fa4c4030d80 comp rx=0 tx=0).stop 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 shutdown_connections 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 --2- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fa4bc0390a0 0x7fa4bc03b560 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 --2- 172.21.15.62:0/685448230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d4108020 0x7fa4d41078a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 >> 172.21.15.62:0/685448230 conn(0x7fa4d4101bc0 msgr2=0x7fa4d4102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 shutdown_connections 2024-09-15T23:06:18.569 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.015+0000 7fa4da037640 1 -- 172.21.15.62:0/685448230 wait complete. 2024-09-15T23:06:18.817 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-mon[30949]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-15T23:06:18.817 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-mon[30949]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-15T23:06:18.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:18.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:06:18.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:18 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:06:19.697 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:19 smithi062 ceph-mon[30949]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:19.697 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:19 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:18] ENGINE Bus STARTING 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:19 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:18] ENGINE Serving on http://172.21.15.62:8765 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:19 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:18] ENGINE Serving on https://172.21.15.62:7150 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:19 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:18] ENGINE Bus STARTED 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:19 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:18] ENGINE Error in HTTPServer.serve 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout: Traceback (most recent call last): 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout: self._connections.run(self.expiration_interval) 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout: self._run(expiration_interval) 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T23:06:19.698 INFO:journalctl@ceph.mon.a.smithi062.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout: s = self.context.wrap_socket( 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout: return self.sslsocket_class._create( 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout: self.do_handshake() 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout: self._sslobj.do_handshake() 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T23:06:19.699 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:19 smithi062 ceph-mon[30949]: mgrmap e8: x(active, since 2s) 2024-09-15T23:06:19.705 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-15T23:06:19.705 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.980+0000 7f0240091640 1 Processor -- start 2024-09-15T23:06:19.705 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.980+0000 7f0240091640 1 -- start start 2024-09-15T23:06:19.705 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.980+0000 7f0240091640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f02381079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:19.705 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.980+0000 7f0240091640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0238107f80 con 0x7f02381075b0 2024-09-15T23:06:19.705 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.980+0000 7f023de2d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f02381079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:19.705 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.980+0000 7f023de2d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f02381079b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39014/0 (socket says 172.21.15.62:39014) 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.980+0000 7f023de2d640 1 -- 172.21.15.62:0/2265827862 learned_addr learned my addr 172.21.15.62:0/2265827862 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.981+0000 7f023de2d640 1 -- 172.21.15.62:0/2265827862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02381087b0 con 0x7f02381075b0 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.981+0000 7f023de2d640 1 --2- 172.21.15.62:0/2265827862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f02381079b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0234009820 tx=0x7f023402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db34d7496780569 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f023ce2b640 1 -- 172.21.15.62:0/2265827862 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0234037070 con 0x7f02381075b0 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f023ce2b640 1 -- 172.21.15.62:0/2265827862 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0234032900 con 0x7f02381075b0 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f023ce2b640 1 -- 172.21.15.62:0/2265827862 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0234030850 con 0x7f02381075b0 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f0240091640 1 -- 172.21.15.62:0/2265827862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 msgr2=0x7f02381079b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f0240091640 1 --2- 172.21.15.62:0/2265827862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f02381079b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0234009820 tx=0x7f023402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f0240091640 1 -- 172.21.15.62:0/2265827862 shutdown_connections 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f0240091640 1 --2- 172.21.15.62:0/2265827862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f02381079b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f0240091640 1 -- 172.21.15.62:0/2265827862 >> 172.21.15.62:0/2265827862 conn(0x7f0238102e50 msgr2=0x7f0238105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f0240091640 1 -- 172.21.15.62:0/2265827862 shutdown_connections 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.982+0000 7f0240091640 1 -- 172.21.15.62:0/2265827862 wait complete. 2024-09-15T23:06:19.706 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.983+0000 7f0240091640 1 Processor -- start 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.983+0000 7f0240091640 1 -- start start 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.983+0000 7f0240091640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f023819b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.983+0000 7f0240091640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f023810bb60 con 0x7f02381075b0 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.983+0000 7f023de2d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f023819b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.983+0000 7f023de2d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f023819b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39026/0 (socket says 172.21.15.62:39026) 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.983+0000 7f023de2d640 1 -- 172.21.15.62:0/2419180651 learned_addr learned my addr 172.21.15.62:0/2419180651 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.984+0000 7f023de2d640 1 -- 172.21.15.62:0/2419180651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023819bb60 con 0x7f02381075b0 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.984+0000 7f023de2d640 1 --2- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f023819b620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0234009940 tx=0x7f02340326e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.984+0000 7f022effd640 1 -- 172.21.15.62:0/2419180651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0234042070 con 0x7f02381075b0 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.984+0000 7f022effd640 1 -- 172.21.15.62:0/2419180651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0234032be0 con 0x7f02381075b0 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.984+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023819bdf0 con 0x7f02381075b0 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.984+0000 7f022effd640 1 -- 172.21.15.62:0/2419180651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0234037040 con 0x7f02381075b0 2024-09-15T23:06:19.707 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.984+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023819eae0 con 0x7f02381075b0 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.985+0000 7f022effd640 1 -- 172.21.15.62:0/2419180651 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f0234045070 con 0x7f02381075b0 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.985+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f023810bce0 con 0x7f02381075b0 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.985+0000 7f022effd640 1 --2- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0224039200 0x7f022403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.985+0000 7f022effd640 1 -- 172.21.15.62:0/2419180651 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f023406c1d0 con 0x7f02381075b0 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.985+0000 7f023d62c640 1 --2- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0224039200 0x7f022403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.986+0000 7f023d62c640 1 --2- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0224039200 0x7f022403b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f02280098c0 tx=0x7f0228006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:18.990+0000 7f022effd640 1 -- 172.21.15.62:0/2419180651 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f023404b170 con 0x7f02381075b0 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.167+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f0238105150 con 0x7f0224039200 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.167+0000 7f022effd640 1 -- 172.21.15.62:0/2419180651 <== mgr.14118 v2:172.21.15.62:6800/2614061737 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f0238105150 con 0x7f0224039200 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.169+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0224039200 msgr2=0x7f022403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.169+0000 7f0240091640 1 --2- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0224039200 0x7f022403b6c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f02280098c0 tx=0x7f0228006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.169+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 msgr2=0x7f023819b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:19.708 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.169+0000 7f0240091640 1 --2- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f023819b620 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0234009940 tx=0x7f02340326e0 comp rx=0 tx=0).stop 2024-09-15T23:06:19.709 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.169+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 shutdown_connections 2024-09-15T23:06:19.709 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.169+0000 7f0240091640 1 --2- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0224039200 0x7f022403b6c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:19.709 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.169+0000 7f0240091640 1 --2- 172.21.15.62:0/2419180651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02381075b0 0x7f023819b620 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:19.709 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.169+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 >> 172.21.15.62:0/2419180651 conn(0x7f0238102e50 msgr2=0x7f0238104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:19.709 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.170+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 shutdown_connections 2024-09-15T23:06:19.709 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:19.170+0000 7f0240091640 1 -- 172.21.15.62:0/2419180651 wait complete. 2024-09-15T23:06:19.709 INFO:teuthology.orchestra.run.smithi062.stdout:Generating ssh key... 2024-09-15T23:06:20.665 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: Generating public/private rsa key pair. 2024-09-15T23:06:20.665 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: Your identification has been saved in /tmp/tmpgz6k7tyx/key 2024-09-15T23:06:20.665 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: Your public key has been saved in /tmp/tmpgz6k7tyx/key.pub 2024-09-15T23:06:20.665 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: The key fingerprint is: 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: SHA256:Qtrj+54KpQvkQzVPHru6w1irwiKeBOhhT5IzKxlquLM ceph-ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: The key's randomart image is: 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: +---[RSA 3072]----+ 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: | | 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: | | 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: | o + | 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: |. .. O o | 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: |+Bo.. O S | 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: |**O .+ + | 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: |*=+=o.o | 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: |Oooo++ . . | 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: |E*..+oo++ | 2024-09-15T23:06:20.666 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: +----[SHA256]-----+ 2024-09-15T23:06:20.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-mon[30949]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:20.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:20.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:20 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:21.077 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.121+0000 7fcceae52640 1 Processor -- start 2024-09-15T23:06:21.077 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.121+0000 7fcceae52640 1 -- start start 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.121+0000 7fcceae52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce4106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.121+0000 7fcceae52640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcce4106e20 con 0x7fcce4106450 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.121+0000 7fcce8bee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce4106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.122+0000 7fcce8bee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce4106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39034/0 (socket says 172.21.15.62:39034) 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.122+0000 7fcce8bee640 1 -- 172.21.15.62:0/4163723131 learned_addr learned my addr 172.21.15.62:0/4163723131 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.122+0000 7fcce8bee640 1 -- 172.21.15.62:0/4163723131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce4107650 con 0x7fcce4106450 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.122+0000 7fcce8bee640 1 --2- 172.21.15.62:0/4163723131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce4106850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009820 tx=0x7fccd802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3525823d8ea597a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcce37fe640 1 -- 172.21.15.62:0/4163723131 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd8037070 con 0x7fcce4106450 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcce37fe640 1 -- 172.21.15.62:0/4163723131 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fccd8032900 con 0x7fcce4106450 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcce37fe640 1 -- 172.21.15.62:0/4163723131 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd8030850 con 0x7fcce4106450 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcceae52640 1 -- 172.21.15.62:0/4163723131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 msgr2=0x7fcce4106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcceae52640 1 --2- 172.21.15.62:0/4163723131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce4106850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009820 tx=0x7fccd802a270 comp rx=0 tx=0).stop 2024-09-15T23:06:21.078 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcceae52640 1 -- 172.21.15.62:0/4163723131 shutdown_connections 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcceae52640 1 --2- 172.21.15.62:0/4163723131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce4106850 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcceae52640 1 -- 172.21.15.62:0/4163723131 >> 172.21.15.62:0/4163723131 conn(0x7fcce4101be0 msgr2=0x7fcce4104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcceae52640 1 -- 172.21.15.62:0/4163723131 shutdown_connections 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.123+0000 7fcceae52640 1 -- 172.21.15.62:0/4163723131 wait complete. 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.124+0000 7fcceae52640 1 Processor -- start 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.124+0000 7fcceae52640 1 -- start start 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.124+0000 7fcceae52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce419b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.124+0000 7fcceae52640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcce410aa00 con 0x7fcce4106450 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.124+0000 7fcce8bee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce419b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.124+0000 7fcce8bee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce419b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39036/0 (socket says 172.21.15.62:39036) 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.124+0000 7fcce8bee640 1 -- 172.21.15.62:0/1328237960 learned_addr learned my addr 172.21.15.62:0/1328237960 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.125+0000 7fcce8bee640 1 -- 172.21.15.62:0/1328237960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce419bb10 con 0x7fcce4106450 2024-09-15T23:06:21.079 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.125+0000 7fcce8bee640 1 --2- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce419b5d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fccd80097f0 tx=0x7fccd8030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.125+0000 7fcce1ffb640 1 -- 172.21.15.62:0/1328237960 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd8037040 con 0x7fcce4106450 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.125+0000 7fcce1ffb640 1 -- 172.21.15.62:0/1328237960 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fccd8040070 con 0x7fcce4106450 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.125+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce419bda0 con 0x7fcce4106450 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.125+0000 7fcce1ffb640 1 -- 172.21.15.62:0/1328237960 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd803a390 con 0x7fcce4106450 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.125+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce419ea90 con 0x7fcce4106450 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.126+0000 7fcce1ffb640 1 -- 172.21.15.62:0/1328237960 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fccd803a5e0 con 0x7fcce4106450 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.126+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcce41068d0 con 0x7fcce4106450 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.126+0000 7fcce1ffb640 1 --2- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fccd4039200 0x7fccd403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.126+0000 7fcce1ffb640 1 -- 172.21.15.62:0/1328237960 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fccd806c870 con 0x7fcce4106450 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.127+0000 7fcce3fff640 1 --2- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fccd4039200 0x7fccd403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.127+0000 7fcce3fff640 1 --2- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fccd4039200 0x7fccd403b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fccd00098c0 tx=0x7fccd0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.131+0000 7fcce1ffb640 1 -- 172.21.15.62:0/1328237960 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fccd8043760 con 0x7fcce4106450 2024-09-15T23:06:21.080 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.310+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fcce402dab0 con 0x7fccd4039200 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.519+0000 7fcce1ffb640 1 -- 172.21.15.62:0/1328237960 <== mgr.14118 v2:172.21.15.62:6800/2614061737 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcce402dab0 con 0x7fccd4039200 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fccd4039200 msgr2=0x7fccd403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 --2- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fccd4039200 0x7fccd403b6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fccd00098c0 tx=0x7fccd0006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 msgr2=0x7fcce419b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 --2- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce419b5d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fccd80097f0 tx=0x7fccd8030d80 comp rx=0 tx=0).stop 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 shutdown_connections 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 --2- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fccd4039200 0x7fccd403b6c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 --2- 172.21.15.62:0/1328237960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce4106450 0x7fcce419b5d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 >> 172.21.15.62:0/1328237960 conn(0x7fcce4101be0 msgr2=0x7fcce4102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 shutdown_connections 2024-09-15T23:06:21.081 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:20.521+0000 7fcceae52640 1 -- 172.21.15.62:0/1328237960 wait complete. 2024-09-15T23:06:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:21 smithi062 ceph-mon[30949]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:21 smithi062 ceph-mon[30949]: Generating ssh key... 2024-09-15T23:06:22.435 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC00w1d4wi6mqNy9KjXmf2HBgM7p6nxu0bccUKwZhdQC4TRV+QuBeyDPGW/JOkGEsKjwSGg8MsziDkySOy2OzFkvPPMiec8IFYprcll7tlE8kT2ndkgjz9BT9JVsHHCXcxgcwyMFRLlGPoneoYYMM8RbwHwsacvUfrHjb3FbTG5/uGz/rJwAC2bhEhGCsZFWmiHCwcmxCb9rmA/b6kZSut9xYP+tXVDBocO25P60mvbcZyVpPsTSpx1tvFgJNVAKivMCxFRHCxyjuCM+Kkpwh/0IwfDkiWLw/7cUUe87dBLFDaQtFTQAaVT2GO6BqSy6Hwe8peCB3FluZDoO6J6cR/u9HITnZv92suwd/LaBrhkTD49W939xFua3PUZY5KiC1lypkosmPP6pzj+9iTxV7Uu/qDFdhWFT9yB2xSz2s7fPzy8lul4t8iX5ppY0YQs8TKJEYfIEIkPiSZtWtx7y8Zj5DH4o3GV/hYhNmJ1DLBXZ/hd22WpBcrCSdz21s+gY80= ceph-ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:06:22.435 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.477+0000 7f6446bfc640 1 Processor -- start 2024-09-15T23:06:22.435 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.477+0000 7f6446bfc640 1 -- start start 2024-09-15T23:06:22.435 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.477+0000 7f6446bfc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440106d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:22.435 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.477+0000 7f6446bfc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6440107340 con 0x7f6440106970 2024-09-15T23:06:22.435 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.477+0000 7f6444998640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440106d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:22.435 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.477+0000 7f6444998640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440106d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39052/0 (socket says 172.21.15.62:39052) 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.477+0000 7f6444998640 1 -- 172.21.15.62:0/2054443024 learned_addr learned my addr 172.21.15.62:0/2054443024 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.477+0000 7f6444998640 1 -- 172.21.15.62:0/2054443024 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6440107690 con 0x7f6440106970 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.478+0000 7f6444998640 1 --2- 172.21.15.62:0/2054443024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440106d70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6434009820 tx=0x7f643402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a85260f2cf8005f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.478+0000 7f643f7fe640 1 -- 172.21.15.62:0/2054443024 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6434037070 con 0x7f6440106970 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.478+0000 7f643f7fe640 1 -- 172.21.15.62:0/2054443024 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6434032900 con 0x7f6440106970 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.478+0000 7f643f7fe640 1 -- 172.21.15.62:0/2054443024 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6434030850 con 0x7f6440106970 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.479+0000 7f6446bfc640 1 -- 172.21.15.62:0/2054443024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 msgr2=0x7f6440106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.479+0000 7f6446bfc640 1 --2- 172.21.15.62:0/2054443024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440106d70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6434009820 tx=0x7f643402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.479+0000 7f6446bfc640 1 -- 172.21.15.62:0/2054443024 shutdown_connections 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.479+0000 7f6446bfc640 1 --2- 172.21.15.62:0/2054443024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440106d70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.479+0000 7f6446bfc640 1 -- 172.21.15.62:0/2054443024 >> 172.21.15.62:0/2054443024 conn(0x7f6440101360 msgr2=0x7f64401037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.479+0000 7f6446bfc640 1 -- 172.21.15.62:0/2054443024 shutdown_connections 2024-09-15T23:06:22.436 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.479+0000 7f6446bfc640 1 -- 172.21.15.62:0/2054443024 wait complete. 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.479+0000 7f6446bfc640 1 Processor -- start 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.479+0000 7f6446bfc640 1 -- start start 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f6446bfc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440191690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f6446bfc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f644010aa40 con 0x7f6440106970 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f6444998640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440191690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f6444998640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440191690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39056/0 (socket says 172.21.15.62:39056) 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f6444998640 1 -- 172.21.15.62:0/1603694725 learned_addr learned my addr 172.21.15.62:0/1603694725 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f6444998640 1 -- 172.21.15.62:0/1603694725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6440191bd0 con 0x7f6440106970 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f6444998640 1 --2- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440191690 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f64340097f0 tx=0x7f6434030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:22.437 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f643dffb640 1 -- 172.21.15.62:0/1603694725 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6434037040 con 0x7f6440106970 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f643dffb640 1 -- 172.21.15.62:0/1603694725 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6434040070 con 0x7f6440106970 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644018fdc0 con 0x7f6440106970 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.480+0000 7f643dffb640 1 -- 172.21.15.62:0/1603694725 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f643403a390 con 0x7f6440106970 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.481+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64401902a0 con 0x7f6440106970 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.481+0000 7f643dffb640 1 -- 172.21.15.62:0/1603694725 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f643403b430 con 0x7f6440106970 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.481+0000 7f643dffb640 1 --2- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f6430039200 0x7f643003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.481+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6440106df0 con 0x7f6440106970 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.482+0000 7f643dffb640 1 -- 172.21.15.62:0/1603694725 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f643406c710 con 0x7f6440106970 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.482+0000 7f643ffff640 1 --2- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f6430039200 0x7f643003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.482+0000 7f643ffff640 1 --2- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f6430039200 0x7f643003b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f642c0098c0 tx=0x7f642c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.486+0000 7f643dffb640 1 -- 172.21.15.62:0/1603694725 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f643403a590 con 0x7f6440106970 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.663+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f6440195440 con 0x7f6430039200 2024-09-15T23:06:22.438 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.664+0000 7f643dffb640 1 -- 172.21.15.62:0/1603694725 <== mgr.14118 v2:172.21.15.62:6800/2614061737 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f6440195440 con 0x7f6430039200 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.666+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f6430039200 msgr2=0x7f643003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.666+0000 7f6446bfc640 1 --2- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f6430039200 0x7f643003b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f642c0098c0 tx=0x7f642c006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.666+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 msgr2=0x7f6440191690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.666+0000 7f6446bfc640 1 --2- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440191690 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f64340097f0 tx=0x7f6434030d80 comp rx=0 tx=0).stop 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.667+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 shutdown_connections 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.667+0000 7f6446bfc640 1 --2- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f6430039200 0x7f643003b6c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.667+0000 7f6446bfc640 1 --2- 172.21.15.62:0/1603694725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6440106970 0x7f6440191690 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.667+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 >> 172.21.15.62:0/1603694725 conn(0x7f6440101360 msgr2=0x7f6440102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.667+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 shutdown_connections 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:21.667+0000 7f6446bfc640 1 -- 172.21.15.62:0/1603694725 wait complete. 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-15T23:06:22.439 INFO:teuthology.orchestra.run.smithi062.stdout:Adding key to root@localhost authorized_keys... 2024-09-15T23:06:22.443 INFO:teuthology.orchestra.run.smithi062.stdout:Adding host smithi062... 2024-09-15T23:06:22.846 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:22 smithi062 ceph-mon[30949]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:24.097 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:23 smithi062 ceph-mon[30949]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi062", "addr": "172.21.15.62", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:24.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:24 smithi062 ceph-mon[30949]: Deploying cephadm binary to smithi062 2024-09-15T23:06:27.525 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Added host 'smithi062' with addr '172.21.15.62' 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.849+0000 7f87bfd04640 1 Processor -- start 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.849+0000 7f87bfd04640 1 -- start start 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.849+0000 7f87bfd04640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b8107950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.849+0000 7f87bfd04640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f87b8107f20 con 0x7f87b8107550 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.850+0000 7f87bdaa0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b8107950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.850+0000 7f87bdaa0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b8107950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56522/0 (socket says 172.21.15.62:56522) 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.850+0000 7f87bdaa0640 1 -- 172.21.15.62:0/1846185692 learned_addr learned my addr 172.21.15.62:0/1846185692 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.850+0000 7f87bdaa0640 1 -- 172.21.15.62:0/1846185692 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b8108750 con 0x7f87b8107550 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.850+0000 7f87bdaa0640 1 --2- 172.21.15.62:0/1846185692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b8107950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009820 tx=0x7f87b402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a94da77bad4a6bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.851+0000 7f87bca9e640 1 -- 172.21.15.62:0/1846185692 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87b4037070 con 0x7f87b8107550 2024-09-15T23:06:27.526 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.851+0000 7f87bca9e640 1 -- 172.21.15.62:0/1846185692 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f87b4032900 con 0x7f87b8107550 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.851+0000 7f87bca9e640 1 -- 172.21.15.62:0/1846185692 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87b4030850 con 0x7f87b8107550 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.851+0000 7f87bfd04640 1 -- 172.21.15.62:0/1846185692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 msgr2=0x7f87b8107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.851+0000 7f87bfd04640 1 --2- 172.21.15.62:0/1846185692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b8107950 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009820 tx=0x7f87b402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.851+0000 7f87bfd04640 1 -- 172.21.15.62:0/1846185692 shutdown_connections 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.851+0000 7f87bfd04640 1 --2- 172.21.15.62:0/1846185692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b8107950 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.851+0000 7f87bfd04640 1 -- 172.21.15.62:0/1846185692 >> 172.21.15.62:0/1846185692 conn(0x7f87b8102ce0 msgr2=0x7f87b8105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.851+0000 7f87bfd04640 1 -- 172.21.15.62:0/1846185692 shutdown_connections 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.852+0000 7f87bfd04640 1 -- 172.21.15.62:0/1846185692 wait complete. 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.852+0000 7f87bfd04640 1 Processor -- start 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.852+0000 7f87bfd04640 1 -- start start 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.852+0000 7f87bfd04640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b819b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.852+0000 7f87bfd04640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f87b810bb00 con 0x7f87b8107550 2024-09-15T23:06:27.527 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.852+0000 7f87bdaa0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b819b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.852+0000 7f87bdaa0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b819b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56526/0 (socket says 172.21.15.62:56526) 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.852+0000 7f87bdaa0640 1 -- 172.21.15.62:0/2794822293 learned_addr learned my addr 172.21.15.62:0/2794822293 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.853+0000 7f87bdaa0640 1 -- 172.21.15.62:0/2794822293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b819ba80 con 0x7f87b8107550 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.853+0000 7f87bdaa0640 1 --2- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b819b540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009940 tx=0x7f87b4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.853+0000 7f87aeffd640 1 -- 172.21.15.62:0/2794822293 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87b4042070 con 0x7f87b8107550 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.853+0000 7f87aeffd640 1 -- 172.21.15.62:0/2794822293 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f87b40308c0 con 0x7f87b8107550 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.853+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b819bd10 con 0x7f87b8107550 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.853+0000 7f87aeffd640 1 -- 172.21.15.62:0/2794822293 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87b4037040 con 0x7f87b8107550 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.853+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b819ea00 con 0x7f87b8107550 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.854+0000 7f87aeffd640 1 -- 172.21.15.62:0/2794822293 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f87b4045070 con 0x7f87b8107550 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.854+0000 7f87aeffd640 1 --2- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f87a4039200 0x7f87a403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:27.528 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.854+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87b810bc80 con 0x7f87b8107550 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.854+0000 7f87aeffd640 1 -- 172.21.15.62:0/2794822293 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f87b406c340 con 0x7f87b8107550 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.854+0000 7f87bd29f640 1 --2- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f87a4039200 0x7f87a403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.855+0000 7f87bd29f640 1 --2- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f87a4039200 0x7f87a403b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87a80098c0 tx=0x7f87a8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:22.859+0000 7f87aeffd640 1 -- 172.21.15.62:0/2794822293 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f87b404b170 con 0x7f87b8107550 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:23.040+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi062", "addr": "172.21.15.62", "target": ["mon-mgr", ""]}) -- 0x7f87b8103f70 con 0x7f87a4039200 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.974+0000 7f87aeffd640 1 -- 172.21.15.62:0/2794822293 <== mgr.14118 v2:172.21.15.62:6800/2614061737 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f87b8103f70 con 0x7f87a4039200 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f87a4039200 msgr2=0x7f87a403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 --2- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f87a4039200 0x7f87a403b6c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87a80098c0 tx=0x7f87a8006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 msgr2=0x7f87b819b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 --2- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b819b540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009940 tx=0x7f87b4032ce0 comp rx=0 tx=0).stop 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 shutdown_connections 2024-09-15T23:06:27.529 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 --2- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f87a4039200 0x7f87a403b6c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:27.530 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 --2- 172.21.15.62:0/2794822293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b8107550 0x7f87b819b540 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:27.530 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 >> 172.21.15.62:0/2794822293 conn(0x7f87b8102ce0 msgr2=0x7f87b8103860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:27.530 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 shutdown_connections 2024-09-15T23:06:27.530 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:26.977+0000 7f87bfd04640 1 -- 172.21.15.62:0/2794822293 wait complete. 2024-09-15T23:06:27.530 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying unmanaged mon service... 2024-09-15T23:06:28.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:27 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:28.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:27 smithi062 ceph-mon[30949]: Added host smithi062 2024-09-15T23:06:28.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:27 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:06:28.949 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-15T23:06:28.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.878+0000 7f16ca945640 1 Processor -- start 2024-09-15T23:06:28.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.878+0000 7f16ca945640 1 -- start start 2024-09-15T23:06:28.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.878+0000 7f16ca945640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c4107950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:28.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.878+0000 7f16ca945640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f16c4107f20 con 0x7f16c4107550 2024-09-15T23:06:28.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.879+0000 7f16c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c4107950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:28.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.879+0000 7f16c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c4107950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56532/0 (socket says 172.21.15.62:56532) 2024-09-15T23:06:28.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.879+0000 7f16c3fff640 1 -- 172.21.15.62:0/2684999677 learned_addr learned my addr 172.21.15.62:0/2684999677 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:28.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.879+0000 7f16c3fff640 1 -- 172.21.15.62:0/2684999677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c4108750 con 0x7f16c4107550 2024-09-15T23:06:28.955 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.880+0000 7f16c3fff640 1 --2- 172.21.15.62:0/2684999677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c4107950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009820 tx=0x7f16b402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b56403ee9d02316 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.880+0000 7f16c2ffd640 1 -- 172.21.15.62:0/2684999677 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b4037070 con 0x7f16c4107550 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.880+0000 7f16c2ffd640 1 -- 172.21.15.62:0/2684999677 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f16b4032900 con 0x7f16c4107550 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.880+0000 7f16c2ffd640 1 -- 172.21.15.62:0/2684999677 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b4030850 con 0x7f16c4107550 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.881+0000 7f16ca945640 1 -- 172.21.15.62:0/2684999677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 msgr2=0x7f16c4107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.881+0000 7f16ca945640 1 --2- 172.21.15.62:0/2684999677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c4107950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009820 tx=0x7f16b402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.881+0000 7f16ca945640 1 -- 172.21.15.62:0/2684999677 shutdown_connections 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.881+0000 7f16ca945640 1 --2- 172.21.15.62:0/2684999677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c4107950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.881+0000 7f16ca945640 1 -- 172.21.15.62:0/2684999677 >> 172.21.15.62:0/2684999677 conn(0x7f16c4102ce0 msgr2=0x7f16c4105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.881+0000 7f16ca945640 1 -- 172.21.15.62:0/2684999677 shutdown_connections 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.881+0000 7f16ca945640 1 -- 172.21.15.62:0/2684999677 wait complete. 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.881+0000 7f16ca945640 1 Processor -- start 2024-09-15T23:06:28.956 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.881+0000 7f16ca945640 1 -- start start 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16ca945640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c419b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16ca945640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f16c410bb00 con 0x7f16c4107550 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c419b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c419b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56544/0 (socket says 172.21.15.62:56544) 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16c3fff640 1 -- 172.21.15.62:0/3696864329 learned_addr learned my addr 172.21.15.62:0/3696864329 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16c3fff640 1 -- 172.21.15.62:0/3696864329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c419bab0 con 0x7f16c4107550 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16c3fff640 1 --2- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c419b570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009940 tx=0x7f16b4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16c17fa640 1 -- 172.21.15.62:0/3696864329 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b4042070 con 0x7f16c4107550 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16c17fa640 1 -- 172.21.15.62:0/3696864329 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f16b40308c0 con 0x7f16c4107550 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16c17fa640 1 -- 172.21.15.62:0/3696864329 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16b4037040 con 0x7f16c4107550 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.882+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c419bd40 con 0x7f16c4107550 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.883+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c419ea30 con 0x7f16c4107550 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.883+0000 7f16c17fa640 1 -- 172.21.15.62:0/3696864329 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f16b4045070 con 0x7f16c4107550 2024-09-15T23:06:28.957 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.884+0000 7f16c17fa640 1 --2- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f16ac0391b0 0x7f16ac03b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.884+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c410bc80 con 0x7f16c4107550 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.884+0000 7f16c17fa640 1 -- 172.21.15.62:0/3696864329 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f16b406c320 con 0x7f16c4107550 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.884+0000 7f16c37fe640 1 --2- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f16ac0391b0 0x7f16ac03b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.884+0000 7f16c37fe640 1 --2- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f16ac0391b0 0x7f16ac03b670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f16b80098c0 tx=0x7f16b8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:27.889+0000 7f16c17fa640 1 -- 172.21.15.62:0/3696864329 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f16b403cd90 con 0x7f16c4107550 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.078+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f16c4103f90 con 0x7f16ac0391b0 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.083+0000 7f16c17fa640 1 -- 172.21.15.62:0/3696864329 <== mgr.14118 v2:172.21.15.62:6800/2614061737 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f16c4103f90 con 0x7f16ac0391b0 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f16ac0391b0 msgr2=0x7f16ac03b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 --2- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f16ac0391b0 0x7f16ac03b670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f16b80098c0 tx=0x7f16b8006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 msgr2=0x7f16c419b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 --2- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c419b570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009940 tx=0x7f16b4032ce0 comp rx=0 tx=0).stop 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 shutdown_connections 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 --2- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f16ac0391b0 0x7f16ac03b670 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:28.958 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 --2- 172.21.15.62:0/3696864329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16c4107550 0x7f16c419b570 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:28.959 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 >> 172.21.15.62:0/3696864329 conn(0x7f16c4102ce0 msgr2=0x7f16c4103880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:28.959 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 shutdown_connections 2024-09-15T23:06:28.959 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:28.085+0000 7f16ca945640 1 -- 172.21.15.62:0/3696864329 wait complete. 2024-09-15T23:06:28.959 INFO:teuthology.orchestra.run.smithi062.stdout:Deploying unmanaged mgr service... 2024-09-15T23:06:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:29 smithi062 ceph-mon[30949]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:29 smithi062 ceph-mon[30949]: Saving service mon spec with placement count:5 2024-09-15T23:06:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:29 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:30.723 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-15T23:06:30.723 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.648+0000 7f2996051640 1 Processor -- start 2024-09-15T23:06:30.723 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.648+0000 7f2996051640 1 -- start start 2024-09-15T23:06:30.723 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.648+0000 7f2996051640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f2990106850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:30.723 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.648+0000 7f2996051640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2990106e20 con 0x7f2990106450 2024-09-15T23:06:30.723 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.649+0000 7f298f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f2990106850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:30.723 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.649+0000 7f298f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f2990106850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56560/0 (socket says 172.21.15.62:56560) 2024-09-15T23:06:30.723 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.649+0000 7f298f7fe640 1 -- 172.21.15.62:0/1148996088 learned_addr learned my addr 172.21.15.62:0/1148996088 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.649+0000 7f298f7fe640 1 -- 172.21.15.62:0/1148996088 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2990107650 con 0x7f2990106450 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f298f7fe640 1 --2- 172.21.15.62:0/1148996088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f2990106850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2980009820 tx=0x7f298002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc362ec1219a5d47 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f298e7fc640 1 -- 172.21.15.62:0/1148996088 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2980037070 con 0x7f2990106450 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f298e7fc640 1 -- 172.21.15.62:0/1148996088 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2980032900 con 0x7f2990106450 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f298e7fc640 1 -- 172.21.15.62:0/1148996088 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2980030850 con 0x7f2990106450 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f2996051640 1 -- 172.21.15.62:0/1148996088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 msgr2=0x7f2990106850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f2996051640 1 --2- 172.21.15.62:0/1148996088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f2990106850 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2980009820 tx=0x7f298002a270 comp rx=0 tx=0).stop 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f2996051640 1 -- 172.21.15.62:0/1148996088 shutdown_connections 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f2996051640 1 --2- 172.21.15.62:0/1148996088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f2990106850 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f2996051640 1 -- 172.21.15.62:0/1148996088 >> 172.21.15.62:0/1148996088 conn(0x7f2990101be0 msgr2=0x7f2990104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.650+0000 7f2996051640 1 -- 172.21.15.62:0/1148996088 shutdown_connections 2024-09-15T23:06:30.724 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.651+0000 7f2996051640 1 -- 172.21.15.62:0/1148996088 wait complete. 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.651+0000 7f2996051640 1 Processor -- start 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.651+0000 7f2996051640 1 -- start start 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.651+0000 7f2996051640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f299019b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.651+0000 7f2996051640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f299007b980 con 0x7f2990106450 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.651+0000 7f298f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f299019b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.651+0000 7f298f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f299019b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56564/0 (socket says 172.21.15.62:56564) 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.651+0000 7f298f7fe640 1 -- 172.21.15.62:0/3878880631 learned_addr learned my addr 172.21.15.62:0/3878880631 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.652+0000 7f298f7fe640 1 -- 172.21.15.62:0/3878880631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f299019ba70 con 0x7f2990106450 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.652+0000 7f298f7fe640 1 --2- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f299019b530 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29800097f0 tx=0x7f2980030d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.652+0000 7f298cff9640 1 -- 172.21.15.62:0/3878880631 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2980037040 con 0x7f2990106450 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.652+0000 7f298cff9640 1 -- 172.21.15.62:0/3878880631 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2980040070 con 0x7f2990106450 2024-09-15T23:06:30.725 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.652+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f299019bd00 con 0x7f2990106450 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.652+0000 7f298cff9640 1 -- 172.21.15.62:0/3878880631 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f298003a390 con 0x7f2990106450 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.652+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f299019e9f0 con 0x7f2990106450 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.653+0000 7f298cff9640 1 -- 172.21.15.62:0/3878880631 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f298003a5e0 con 0x7f2990106450 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.653+0000 7f298cff9640 1 --2- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2978039200 0x7f297803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.653+0000 7f298cff9640 1 -- 172.21.15.62:0/3878880631 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f298006c990 con 0x7f2990106450 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.653+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2990106850 con 0x7f2990106450 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.653+0000 7f298effd640 1 --2- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2978039200 0x7f297803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.654+0000 7f298effd640 1 --2- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2978039200 0x7f297803b6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f29840098c0 tx=0x7f2984006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.658+0000 7f298cff9640 1 -- 172.21.15.62:0/3878880631 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2980043760 con 0x7f2990106450 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.848+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f299002db30 con 0x7f2978039200 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.852+0000 7f298cff9640 1 -- 172.21.15.62:0/3878880631 <== mgr.14118 v2:172.21.15.62:6800/2614061737 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f299002db30 con 0x7f2978039200 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.854+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2978039200 msgr2=0x7f297803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.854+0000 7f2996051640 1 --2- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2978039200 0x7f297803b6c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f29840098c0 tx=0x7f2984006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:30.727 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.854+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 msgr2=0x7f299019b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:30.727 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.854+0000 7f2996051640 1 --2- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f299019b530 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29800097f0 tx=0x7f2980030d80 comp rx=0 tx=0).stop 2024-09-15T23:06:30.727 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.854+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 shutdown_connections 2024-09-15T23:06:30.727 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.854+0000 7f2996051640 1 --2- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2978039200 0x7f297803b6c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:30.727 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.854+0000 7f2996051640 1 --2- 172.21.15.62:0/3878880631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2990106450 0x7f299019b530 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:30.727 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.854+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 >> 172.21.15.62:0/3878880631 conn(0x7f2990101be0 msgr2=0x7f2990102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:30.727 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.855+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 shutdown_connections 2024-09-15T23:06:30.727 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:29.855+0000 7f2996051640 1 -- 172.21.15.62:0/3878880631 wait complete. 2024-09-15T23:06:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:30 smithi062 ceph-mon[30949]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:30 smithi062 ceph-mon[30949]: Saving service mgr spec with placement count:2 2024-09-15T23:06:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:30 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:30 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:31.879 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.101+0000 7f2641571640 1 Processor -- start 2024-09-15T23:06:31.879 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.101+0000 7f2641571640 1 -- start start 2024-09-15T23:06:31.879 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.102+0000 7f2641571640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c10a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:31.879 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.102+0000 7f2641571640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f263c07c5a0 con 0x7f263c108060 2024-09-15T23:06:31.879 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.102+0000 7f263affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c10a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:31.879 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.102+0000 7f263affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c10a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56574/0 (socket says 172.21.15.62:56574) 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.102+0000 7f263affd640 1 -- 172.21.15.62:0/1852609757 learned_addr learned my addr 172.21.15.62:0/1852609757 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.102+0000 7f263affd640 1 -- 172.21.15.62:0/1852609757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263c10a990 con 0x7f263c108060 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.103+0000 7f263affd640 1 --2- 172.21.15.62:0/1852609757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c10a450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f262c009820 tx=0x7f262c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e511969337267c98 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.103+0000 7f2639ffb640 1 -- 172.21.15.62:0/1852609757 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f262c037070 con 0x7f263c108060 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.103+0000 7f2639ffb640 1 -- 172.21.15.62:0/1852609757 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f262c032920 con 0x7f263c108060 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.103+0000 7f2639ffb640 1 -- 172.21.15.62:0/1852609757 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f262c030810 con 0x7f263c108060 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.104+0000 7f2641571640 1 -- 172.21.15.62:0/1852609757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 msgr2=0x7f263c10a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.104+0000 7f2641571640 1 --2- 172.21.15.62:0/1852609757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c10a450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f262c009820 tx=0x7f262c02a270 comp rx=0 tx=0).stop 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.104+0000 7f2641571640 1 -- 172.21.15.62:0/1852609757 shutdown_connections 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.104+0000 7f2641571640 1 --2- 172.21.15.62:0/1852609757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c10a450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.104+0000 7f2641571640 1 -- 172.21.15.62:0/1852609757 >> 172.21.15.62:0/1852609757 conn(0x7f263c101be0 msgr2=0x7f263c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:31.880 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.104+0000 7f2641571640 1 -- 172.21.15.62:0/1852609757 shutdown_connections 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.104+0000 7f2641571640 1 -- 172.21.15.62:0/1852609757 wait complete. 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.105+0000 7f2641571640 1 Processor -- start 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.105+0000 7f2641571640 1 -- start start 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.105+0000 7f2641571640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c19b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.105+0000 7f2641571640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f263c10b950 con 0x7f263c108060 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.105+0000 7f263affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c19b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.105+0000 7f263affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c19b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56580/0 (socket says 172.21.15.62:56580) 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.105+0000 7f263affd640 1 -- 172.21.15.62:0/1730904643 learned_addr learned my addr 172.21.15.62:0/1730904643 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.106+0000 7f263affd640 1 -- 172.21.15.62:0/1730904643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f263c19ba10 con 0x7f263c108060 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.106+0000 7f263affd640 1 --2- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c19b4d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f262c030380 tx=0x7f262c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.106+0000 7f2623fff640 1 -- 172.21.15.62:0/1730904643 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f262c037040 con 0x7f263c108060 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.106+0000 7f2623fff640 1 -- 172.21.15.62:0/1730904643 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f262c040070 con 0x7f263c108060 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.106+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f263c19bca0 con 0x7f263c108060 2024-09-15T23:06:31.881 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.106+0000 7f2623fff640 1 -- 172.21.15.62:0/1730904643 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f262c03a7b0 con 0x7f263c108060 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.106+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f263c19e990 con 0x7f263c108060 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.107+0000 7f2623fff640 1 -- 172.21.15.62:0/1730904643 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7f262c03a9e0 con 0x7f263c108060 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.107+0000 7f2623fff640 1 --2- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2624039200 0x7f262403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.107+0000 7f2623fff640 1 -- 172.21.15.62:0/1730904643 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f262c06d910 con 0x7f263c108060 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.107+0000 7f263a7fc640 1 --2- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2624039200 0x7f262403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.107+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f263c1084c0 con 0x7f263c108060 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.108+0000 7f263a7fc640 1 --2- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2624039200 0x7f262403b6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f26300098c0 tx=0x7f2630006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.112+0000 7f2623fff640 1 -- 172.21.15.62:0/1730904643 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f262c044c70 con 0x7f263c108060 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.293+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f263c19ee50 con 0x7f263c108060 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.298+0000 7f2623fff640 1 -- 172.21.15.62:0/1730904643 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f262c041270 con 0x7f263c108060 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2624039200 msgr2=0x7f262403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 --2- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2624039200 0x7f262403b6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f26300098c0 tx=0x7f2630006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:31.882 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 msgr2=0x7f263c19b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:31.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 --2- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c19b4d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f262c030380 tx=0x7f262c032670 comp rx=0 tx=0).stop 2024-09-15T23:06:31.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 shutdown_connections 2024-09-15T23:06:31.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 --2- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f2624039200 0x7f262403b6c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:31.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 --2- 172.21.15.62:0/1730904643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f263c108060 0x7f263c19b4d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:31.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 >> 172.21.15.62:0/1730904643 conn(0x7f263c101be0 msgr2=0x7f263c1027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:31.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 shutdown_connections 2024-09-15T23:06:31.883 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:31.301+0000 7f2641571640 1 -- 172.21.15.62:0/1730904643 wait complete. 2024-09-15T23:06:32.469 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:32 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:32.469 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:32 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1730904643' entity='client.admin' 2024-09-15T23:06:33.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.274+0000 7ff154238640 1 Processor -- start 2024-09-15T23:06:33.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.274+0000 7ff154238640 1 -- start start 2024-09-15T23:06:33.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.275+0000 7ff154238640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c10a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:33.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.275+0000 7ff154238640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c07be50 con 0x7ff14c107fa0 2024-09-15T23:06:33.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.275+0000 7ff151fd4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c10a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:33.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.275+0000 7ff151fd4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c10a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56588/0 (socket says 172.21.15.62:56588) 2024-09-15T23:06:33.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.275+0000 7ff151fd4640 1 -- 172.21.15.62:0/1683709928 learned_addr learned my addr 172.21.15.62:0/1683709928 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:33.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.275+0000 7ff151fd4640 1 -- 172.21.15.62:0/1683709928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14c10a8d0 con 0x7ff14c107fa0 2024-09-15T23:06:33.305 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.276+0000 7ff151fd4640 1 --2- 172.21.15.62:0/1683709928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c10a390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff148009820 tx=0x7ff14802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f8ae7cc72d1fbe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.276+0000 7ff150fd2640 1 -- 172.21.15.62:0/1683709928 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff148037070 con 0x7ff14c107fa0 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.276+0000 7ff150fd2640 1 -- 172.21.15.62:0/1683709928 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff148032920 con 0x7ff14c107fa0 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.276+0000 7ff150fd2640 1 -- 172.21.15.62:0/1683709928 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff148030810 con 0x7ff14c107fa0 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.277+0000 7ff154238640 1 -- 172.21.15.62:0/1683709928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 msgr2=0x7ff14c10a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.277+0000 7ff154238640 1 --2- 172.21.15.62:0/1683709928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c10a390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff148009820 tx=0x7ff14802a270 comp rx=0 tx=0).stop 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.277+0000 7ff154238640 1 -- 172.21.15.62:0/1683709928 shutdown_connections 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.277+0000 7ff154238640 1 --2- 172.21.15.62:0/1683709928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c10a390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.277+0000 7ff154238640 1 -- 172.21.15.62:0/1683709928 >> 172.21.15.62:0/1683709928 conn(0x7ff14c101bc0 msgr2=0x7ff14c104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.277+0000 7ff154238640 1 -- 172.21.15.62:0/1683709928 shutdown_connections 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.277+0000 7ff154238640 1 -- 172.21.15.62:0/1683709928 wait complete. 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.277+0000 7ff154238640 1 Processor -- start 2024-09-15T23:06:33.306 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.277+0000 7ff154238640 1 -- start start 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff154238640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c19b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff154238640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff14c10b9b0 con 0x7ff14c107fa0 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff151fd4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c19b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff151fd4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c19b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56600/0 (socket says 172.21.15.62:56600) 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff151fd4640 1 -- 172.21.15.62:0/109396451 learned_addr learned my addr 172.21.15.62:0/109396451 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff151fd4640 1 -- 172.21.15.62:0/109396451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14c19ba90 con 0x7ff14c107fa0 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff151fd4640 1 --2- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c19b550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff148030380 tx=0x7ff148032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff142ffd640 1 -- 172.21.15.62:0/109396451 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff148037040 con 0x7ff14c107fa0 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff142ffd640 1 -- 172.21.15.62:0/109396451 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff148040070 con 0x7ff14c107fa0 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14c19bd20 con 0x7ff14c107fa0 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff142ffd640 1 -- 172.21.15.62:0/109396451 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff14803a7b0 con 0x7ff14c107fa0 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.278+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff14c19ea10 con 0x7ff14c107fa0 2024-09-15T23:06:33.307 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.279+0000 7ff142ffd640 1 -- 172.21.15.62:0/109396451 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7ff14803aa30 con 0x7ff14c107fa0 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.279+0000 7ff142ffd640 1 --2- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7ff1380391b0 0x7ff13803b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.279+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff14c108400 con 0x7ff14c107fa0 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.280+0000 7ff142ffd640 1 -- 172.21.15.62:0/109396451 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff14806cae0 con 0x7ff14c107fa0 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.280+0000 7ff1517d3640 1 --2- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7ff1380391b0 0x7ff13803b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.280+0000 7ff1517d3640 1 --2- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7ff1380391b0 0x7ff13803b670 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0098c0 tx=0x7ff13c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.284+0000 7ff142ffd640 1 -- 172.21.15.62:0/109396451 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff14803ad20 con 0x7ff14c107fa0 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.468+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7ff14c19f040 con 0x7ff14c107fa0 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.472+0000 7ff142ffd640 1 -- 172.21.15.62:0/109396451 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7ff14805b0c0 con 0x7ff14c107fa0 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7ff1380391b0 msgr2=0x7ff13803b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 --2- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7ff1380391b0 0x7ff13803b670 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0098c0 tx=0x7ff13c006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 msgr2=0x7ff14c19b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 --2- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c19b550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff148030380 tx=0x7ff148032670 comp rx=0 tx=0).stop 2024-09-15T23:06:33.308 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 shutdown_connections 2024-09-15T23:06:33.309 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 --2- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7ff1380391b0 0x7ff13803b670 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:33.309 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 --2- 172.21.15.62:0/109396451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff14c107fa0 0x7ff14c19b550 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:33.309 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 >> 172.21.15.62:0/109396451 conn(0x7ff14c101bc0 msgr2=0x7ff14c102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:33.309 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 shutdown_connections 2024-09-15T23:06:33.309 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:32.475+0000 7ff154238640 1 -- 172.21.15.62:0/109396451 wait complete. 2024-09-15T23:06:33.309 INFO:teuthology.orchestra.run.smithi062.stdout:Enabling the dashboard module... 2024-09-15T23:06:33.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:33 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/109396451' entity='client.admin' 2024-09-15T23:06:34.504 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:34 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/573896216' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-15T23:06:34.504 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:34 smithi062 ceph-mon[30949]: from='mgr.14118 172.21.15.62:0/501367482' entity='mgr.x' 2024-09-15T23:06:34.803 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:34 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: ignoring --setuser ceph since I am not root 2024-09-15T23:06:34.803 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:34 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: ignoring --setgroup ceph since I am not root 2024-09-15T23:06:34.803 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:34 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:34.642+0000 7f90ee3cb040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:06:34.804 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:34 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:34.703+0000 7f90ee3cb040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:06:35.088 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:34 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:34.861+0000 7f90ee3cb040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:06:35.088 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:34 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:34.931+0000 7f90ee3cb040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:06:35.088 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:35 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:35.085+0000 7f90ee3cb040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:06:35.103 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.681+0000 7fde8873a640 1 Processor -- start 2024-09-15T23:06:35.103 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.681+0000 7fde8873a640 1 -- start start 2024-09-15T23:06:35.103 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.681+0000 7fde8873a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde80107990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:35.103 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.681+0000 7fde8873a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fde80107f60 con 0x7fde80107590 2024-09-15T23:06:35.103 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.682+0000 7fde864d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde80107990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.682+0000 7fde864d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde80107990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51726/0 (socket says 172.21.15.62:51726) 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.682+0000 7fde864d6640 1 -- 172.21.15.62:0/2211162208 learned_addr learned my addr 172.21.15.62:0/2211162208 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.682+0000 7fde864d6640 1 -- 172.21.15.62:0/2211162208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde80108740 con 0x7fde80107590 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.683+0000 7fde864d6640 1 --2- 172.21.15.62:0/2211162208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde80107990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009820 tx=0x7fde7c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aeedbe9e395a31a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.683+0000 7fde854d4640 1 -- 172.21.15.62:0/2211162208 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde7c037070 con 0x7fde80107590 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.683+0000 7fde854d4640 1 -- 172.21.15.62:0/2211162208 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde7c032920 con 0x7fde80107590 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.683+0000 7fde854d4640 1 -- 172.21.15.62:0/2211162208 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde7c030810 con 0x7fde80107590 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.683+0000 7fde8873a640 1 -- 172.21.15.62:0/2211162208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 msgr2=0x7fde80107990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.683+0000 7fde8873a640 1 --2- 172.21.15.62:0/2211162208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde80107990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009820 tx=0x7fde7c02a270 comp rx=0 tx=0).stop 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.684+0000 7fde8873a640 1 -- 172.21.15.62:0/2211162208 shutdown_connections 2024-09-15T23:06:35.104 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.684+0000 7fde8873a640 1 --2- 172.21.15.62:0/2211162208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde80107990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.684+0000 7fde8873a640 1 -- 172.21.15.62:0/2211162208 >> 172.21.15.62:0/2211162208 conn(0x7fde80102de0 msgr2=0x7fde801051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.684+0000 7fde8873a640 1 -- 172.21.15.62:0/2211162208 shutdown_connections 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.684+0000 7fde8873a640 1 -- 172.21.15.62:0/2211162208 wait complete. 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.684+0000 7fde8873a640 1 Processor -- start 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.684+0000 7fde8873a640 1 -- start start 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.684+0000 7fde8873a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde8019b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.684+0000 7fde8873a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fde8010baf0 con 0x7fde80107590 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde864d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde8019b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde864d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde8019b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51728/0 (socket says 172.21.15.62:51728) 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde864d6640 1 -- 172.21.15.62:0/573896216 learned_addr learned my addr 172.21.15.62:0/573896216 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde864d6640 1 -- 172.21.15.62:0/573896216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8019baf0 con 0x7fde80107590 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde864d6640 1 --2- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde8019b5b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0097f0 tx=0x7fde7c0327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:35.105 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde777fe640 1 -- 172.21.15.62:0/573896216 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde7c037040 con 0x7fde80107590 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde777fe640 1 -- 172.21.15.62:0/573896216 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde7c030cc0 con 0x7fde80107590 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde777fe640 1 -- 172.21.15.62:0/573896216 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde7c03a640 con 0x7fde80107590 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8019bd80 con 0x7fde80107590 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.685+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8019ea70 con 0x7fde80107590 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.686+0000 7fde777fe640 1 -- 172.21.15.62:0/573896216 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 8) ==== 46461+0+0 (secure 0 0 0) 0x7fde7c03b3f0 con 0x7fde80107590 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.686+0000 7fde777fe640 1 --2- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fde6c039200 0x7fde6c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.686+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8010bc70 con 0x7fde80107590 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.687+0000 7fde777fe640 1 -- 172.21.15.62:0/573896216 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fde7c06c7a0 con 0x7fde80107590 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.687+0000 7fde85cd5640 1 --2- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fde6c039200 0x7fde6c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.687+0000 7fde85cd5640 1 --2- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fde6c039200 0x7fde6c03b6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fde700098c0 tx=0x7fde70006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.692+0000 7fde777fe640 1 -- 172.21.15.62:0/573896216 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fde7c042e10 con 0x7fde80107590 2024-09-15T23:06:35.106 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:33.919+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fde8019ec40 con 0x7fde80107590 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.474+0000 7fde777fe640 1 -- 172.21.15.62:0/573896216 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fde7c03b6e0 con 0x7fde80107590 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.476+0000 7fde777fe640 1 -- 172.21.15.62:0/573896216 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7fde7c049ca0 con 0x7fde80107590 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.477+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fde6c039200 msgr2=0x7fde6c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.477+0000 7fde8873a640 1 --2- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fde6c039200 0x7fde6c03b6c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fde700098c0 tx=0x7fde70006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.477+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 msgr2=0x7fde8019b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.477+0000 7fde8873a640 1 --2- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde8019b5b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0097f0 tx=0x7fde7c0327b0 comp rx=0 tx=0).stop 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.477+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 shutdown_connections 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.477+0000 7fde8873a640 1 --2- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fde6c039200 0x7fde6c03b6c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.477+0000 7fde8873a640 1 --2- 172.21.15.62:0/573896216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde80107590 0x7fde8019b5b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.477+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 >> 172.21.15.62:0/573896216 conn(0x7fde80102de0 msgr2=0x7fde80103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.478+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 shutdown_connections 2024-09-15T23:06:35.107 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:34.478+0000 7fde8873a640 1 -- 172.21.15.62:0/573896216 wait complete. 2024-09-15T23:06:35.345 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:35 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:35.237+0000 7f90ee3cb040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:06:35.640 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:35 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/573896216' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-15T23:06:35.640 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:35 smithi062 ceph-mon[30949]: mgrmap e9: x(active, since 18s) 2024-09-15T23:06:35.640 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:35 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:35.573+0000 7f90ee3cb040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:06:35.640 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:35 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:35.637+0000 7f90ee3cb040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:06:35.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:35 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:35.848+0000 7f90ee3cb040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:06:36.376 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.535+0000 7f0a5bfb2640 1 Processor -- start 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.536+0000 7f0a5bfb2640 1 -- start start 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.536+0000 7f0a5bfb2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5410a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.536+0000 7f0a5bfb2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5407be50 con 0x7f0a54107fa0 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.536+0000 7f0a59d4e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5410a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.536+0000 7f0a59d4e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5410a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51762/0 (socket says 172.21.15.62:51762) 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.536+0000 7f0a59d4e640 1 -- 172.21.15.62:0/3251658512 learned_addr learned my addr 172.21.15.62:0/3251658512 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:36.377 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.537+0000 7f0a59d4e640 1 -- 172.21.15.62:0/3251658512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5410a8d0 con 0x7f0a54107fa0 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.537+0000 7f0a59d4e640 1 --2- 172.21.15.62:0/3251658512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5410a390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009820 tx=0x7f0a5002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9fe39695b194ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.538+0000 7f0a58d4c640 1 -- 172.21.15.62:0/3251658512 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a50037070 con 0x7f0a54107fa0 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.538+0000 7f0a58d4c640 1 -- 172.21.15.62:0/3251658512 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a50032920 con 0x7f0a54107fa0 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.538+0000 7f0a58d4c640 1 -- 172.21.15.62:0/3251658512 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a50030810 con 0x7f0a54107fa0 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.538+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/3251658512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 msgr2=0x7f0a5410a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.538+0000 7f0a5bfb2640 1 --2- 172.21.15.62:0/3251658512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5410a390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009820 tx=0x7f0a5002a270 comp rx=0 tx=0).stop 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.538+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/3251658512 shutdown_connections 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.538+0000 7f0a5bfb2640 1 --2- 172.21.15.62:0/3251658512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5410a390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.538+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/3251658512 >> 172.21.15.62:0/3251658512 conn(0x7f0a54101bc0 msgr2=0x7f0a54104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.539+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/3251658512 shutdown_connections 2024-09-15T23:06:36.378 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.539+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/3251658512 wait complete. 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.539+0000 7f0a5bfb2640 1 Processor -- start 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.539+0000 7f0a5bfb2640 1 -- start start 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.539+0000 7f0a5bfb2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5419d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.539+0000 7f0a5bfb2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0a5410b9b0 con 0x7f0a54107fa0 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.539+0000 7f0a59d4e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5419d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.539+0000 7f0a59d4e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5419d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51776/0 (socket says 172.21.15.62:51776) 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.539+0000 7f0a59d4e640 1 -- 172.21.15.62:0/2874933143 learned_addr learned my addr 172.21.15.62:0/2874933143 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.540+0000 7f0a59d4e640 1 -- 172.21.15.62:0/2874933143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5419dc90 con 0x7f0a54107fa0 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.540+0000 7f0a59d4e640 1 --2- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5419d750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0a50030380 tx=0x7f0a50032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.540+0000 7f0a4affd640 1 -- 172.21.15.62:0/2874933143 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a50037040 con 0x7f0a54107fa0 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.540+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5419df20 con 0x7f0a54107fa0 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.540+0000 7f0a4affd640 1 -- 172.21.15.62:0/2874933143 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a50040070 con 0x7f0a54107fa0 2024-09-15T23:06:36.379 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.540+0000 7f0a4affd640 1 -- 172.21.15.62:0/2874933143 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a5003a7b0 con 0x7f0a54107fa0 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.540+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a541a0c10 con 0x7f0a54107fa0 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.541+0000 7f0a4affd640 1 -- 172.21.15.62:0/2874933143 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7f0a5003bc00 con 0x7f0a54107fa0 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.541+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a54108400 con 0x7f0a54107fa0 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.542+0000 7f0a4affd640 1 --2- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0a40039200 0x7f0a4003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.542+0000 7f0a4affd640 1 -- 172.21.15.62:0/2874933143 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0a5006c960 con 0x7f0a54107fa0 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.542+0000 7f0a5954d640 1 -- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0a40039200 msgr2=0x7f0a4003b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/2614061737 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.542+0000 7f0a5954d640 1 --2- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0a40039200 0x7f0a4003b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.546+0000 7f0a4affd640 1 -- 172.21.15.62:0/2874933143 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0a50040220 con 0x7f0a54107fa0 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.743+0000 7f0a5954d640 1 -- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0a40039200 msgr2=0x7f0a4003b6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/2614061737 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.743+0000 7f0a5954d640 1 --2- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0a40039200 0x7f0a4003b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.770+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f0a541a1200 con 0x7f0a54107fa0 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.771+0000 7f0a4affd640 1 -- 172.21.15.62:0/2874933143 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f0a500313a0 con 0x7f0a54107fa0 2024-09-15T23:06:36.380 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0a40039200 msgr2=0x7f0a4003b6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 --2- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0a40039200 0x7f0a4003b6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 msgr2=0x7f0a5419d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 --2- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5419d750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0a50030380 tx=0x7f0a50032670 comp rx=0 tx=0).stop 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 shutdown_connections 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 --2- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7f0a40039200 0x7f0a4003b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 --2- 172.21.15.62:0/2874933143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a54107fa0 0x7f0a5419d750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 >> 172.21.15.62:0/2874933143 conn(0x7f0a54101bc0 msgr2=0x7f0a541026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 shutdown_connections 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:35.774+0000 7f0a5bfb2640 1 -- 172.21.15.62:0/2874933143 wait complete. 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for the mgr to restart... 2024-09-15T23:06:36.381 INFO:teuthology.orchestra.run.smithi062.stdout:Waiting for mgr epoch 9... 2024-09-15T23:06:36.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:36 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2874933143' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-15T23:06:36.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:36 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:36.793+0000 7f90ee3cb040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:06:36.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:36 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:36.863+0000 7f90ee3cb040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:06:37.885 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:37.613+0000 7f90ee3cb040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:06:37.886 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:37.733+0000 7f90ee3cb040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:06:38.177 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:37.883+0000 7f90ee3cb040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:06:38.177 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:37.947+0000 7f90ee3cb040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:06:38.177 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:38.103+0000 7f90ee3cb040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:06:38.177 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:38.174+0000 7f90ee3cb040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:06:38.458 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:38.239+0000 7f90ee3cb040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:06:38.458 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:38.408+0000 7f90ee3cb040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:06:38.730 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:38.479+0000 7f90ee3cb040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:06:38.730 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:38.549+0000 7f90ee3cb040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:06:38.730 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:38.638+0000 7f90ee3cb040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:06:38.730 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:38.702+0000 7f90ee3cb040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:06:39.230 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:38.766+0000 7f90ee3cb040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:06:39.730 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:39.604+0000 7f90ee3cb040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:06:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: Active manager daemon x restarted 2024-09-15T23:06:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: Activating manager daemon x 2024-09-15T23:06:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: osdmap e3: 0 total, 0 up, 0 in 2024-09-15T23:06:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: mgrmap e10: x(active, starting, since 0.00360488s) 2024-09-15T23:06:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:06:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-15T23:06:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-15T23:06:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-15T23:06:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-15T23:06:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: Manager daemon x is now available 2024-09-15T23:06:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:06:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-15T23:06:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-15T23:06:41.503 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout { 2024-09-15T23:06:41.503 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-15T23:06:41.503 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-15T23:06:41.503 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout } 2024-09-15T23:06:41.503 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.723+0000 7fd8391d8640 1 Processor -- start 2024-09-15T23:06:41.503 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.723+0000 7fd8391d8640 1 -- start start 2024-09-15T23:06:41.503 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.724+0000 7fd8391d8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83410a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.724+0000 7fd8391d8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd83407c680 con 0x7fd834108070 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.724+0000 7fd832d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83410a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.724+0000 7fd832d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83410a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51792/0 (socket says 172.21.15.62:51792) 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.724+0000 7fd832d9d640 1 -- 172.21.15.62:0/3348095531 learned_addr learned my addr 172.21.15.62:0/3348095531 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.724+0000 7fd832d9d640 1 -- 172.21.15.62:0/3348095531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83410a9a0 con 0x7fd834108070 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.725+0000 7fd832d9d640 1 --2- 172.21.15.62:0/3348095531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83410a460 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd824009820 tx=0x7fd82402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbecba66f5fd317 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.725+0000 7fd83259c640 1 -- 172.21.15.62:0/3348095531 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd824037070 con 0x7fd834108070 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.725+0000 7fd83259c640 1 -- 172.21.15.62:0/3348095531 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd824032920 con 0x7fd834108070 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.725+0000 7fd83259c640 1 -- 172.21.15.62:0/3348095531 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd824030810 con 0x7fd834108070 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.726+0000 7fd8391d8640 1 -- 172.21.15.62:0/3348095531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 msgr2=0x7fd83410a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.726+0000 7fd8391d8640 1 --2- 172.21.15.62:0/3348095531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83410a460 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd824009820 tx=0x7fd82402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:41.504 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.726+0000 7fd8391d8640 1 -- 172.21.15.62:0/3348095531 shutdown_connections 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.726+0000 7fd8391d8640 1 --2- 172.21.15.62:0/3348095531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83410a460 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.726+0000 7fd8391d8640 1 -- 172.21.15.62:0/3348095531 >> 172.21.15.62:0/3348095531 conn(0x7fd834101c70 msgr2=0x7fd834104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.726+0000 7fd8391d8640 1 -- 172.21.15.62:0/3348095531 shutdown_connections 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.726+0000 7fd8391d8640 1 -- 172.21.15.62:0/3348095531 wait complete. 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.727+0000 7fd8391d8640 1 Processor -- start 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.727+0000 7fd8391d8640 1 -- start start 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.727+0000 7fd8391d8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83419b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.727+0000 7fd8391d8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd83410ba20 con 0x7fd834108070 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.727+0000 7fd832d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83419b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.727+0000 7fd832d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83419b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51804/0 (socket says 172.21.15.62:51804) 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.727+0000 7fd832d9d640 1 -- 172.21.15.62:0/2998237825 learned_addr learned my addr 172.21.15.62:0/2998237825 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.727+0000 7fd832d9d640 1 -- 172.21.15.62:0/2998237825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd83419bb20 con 0x7fd834108070 2024-09-15T23:06:41.505 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.728+0000 7fd832d9d640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83419b5e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097f0 tx=0x7fd8240327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.728+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd824037040 con 0x7fd834108070 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.728+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd824030c40 con 0x7fd834108070 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.728+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd82403a5d0 con 0x7fd834108070 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.728+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd83419bdb0 con 0x7fd834108070 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.728+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd83419eaa0 con 0x7fd834108070 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.729+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 9) ==== 46474+0+0 (secure 0 0 0) 0x7fd82403a7c0 con 0x7fd834108070 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.729+0000 7fd830d99640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 0x7fd82003b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.729+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 --> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd8341084d0 con 0x7fd820039260 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.729+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd82406c8c0 con 0x7fd834108070 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.729+0000 7fd82bfff640 1 -- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 msgr2=0x7fd82003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/2614061737 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.729+0000 7fd82bfff640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 0x7fd82003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.930+0000 7fd82bfff640 1 -- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 msgr2=0x7fd82003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/2614061737 2024-09-15T23:06:41.506 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:36.930+0000 7fd82bfff640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 0x7fd82003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:37.331+0000 7fd82bfff640 1 -- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 msgr2=0x7fd82003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/2614061737 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:37.331+0000 7fd82bfff640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 0x7fd82003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:38.132+0000 7fd82bfff640 1 -- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 msgr2=0x7fd82003b720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.62:6800/2614061737 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:38.132+0000 7fd82bfff640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 0x7fd82003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:39.608+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mgrmap(e 10) ==== 46241+0+0 (secure 0 0 0) 0x7fd82406b7e0 con 0x7fd834108070 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:39.608+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 msgr2=0x7fd82003b720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:39.608+0000 7fd830d99640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 0x7fd82003b720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.609+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7fd824044400 con 0x7fd834108070 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.610+0000 7fd830d99640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd82003cb30 0x7fd82003ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.610+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd82403eec0 con 0x7fd82003cb30 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.610+0000 7fd82bfff640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd82003cb30 0x7fd82003ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.611+0000 7fd82bfff640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd82003cb30 0x7fd82003ef20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd81c003980 tx=0x7fd81c0092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:41.507 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.612+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fd82403eec0 con 0x7fd82003cb30 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd834102f30 con 0x7fd82003cb30 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd830d99640 1 -- 172.21.15.62:0/2998237825 <== mgr.14150 v2:172.21.15.62:6800/4035090192 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fd834102f30 con 0x7fd82003cb30 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd82003cb30 msgr2=0x7fd82003ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd82003cb30 0x7fd82003ef20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd81c003980 tx=0x7fd81c0092b0 comp rx=0 tx=0).stop 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 msgr2=0x7fd83419b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83419b5e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097f0 tx=0x7fd8240327b0 comp rx=0 tx=0).stop 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 shutdown_connections 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd82003cb30 0x7fd82003ef20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:6800/2614061737,v1:172.21.15.62:6801/2614061737] conn(0x7fd820039260 0x7fd82003b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 --2- 172.21.15.62:0/2998237825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd834108070 0x7fd83419b5e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.619+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 >> 172.21.15.62:0/2998237825 conn(0x7fd834101c70 msgr2=0x7fd834102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.620+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 shutdown_connections 2024-09-15T23:06:41.508 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:40.620+0000 7fd8391d8640 1 -- 172.21.15.62:0/2998237825 wait complete. 2024-09-15T23:06:41.509 INFO:teuthology.orchestra.run.smithi062.stdout:mgr epoch 9 is available 2024-09-15T23:06:41.509 INFO:teuthology.orchestra.run.smithi062.stdout:Generating a dashboard self-signed certificate... 2024-09-15T23:06:41.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-mon[30949]: mgrmap e11: x(active, since 1.00613s) 2024-09-15T23:06:41.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:41.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:41] ENGINE Bus STARTING 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:41.671+0000 7f9053e94640 -1 log_channel(cephadm) log [ERR] : [15/Sep/2024:23:06:41] ENGINE Error in HTTPServer.serve 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: Traceback (most recent call last): 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: self._connections.run(self.expiration_interval) 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: self._run(expiration_interval) 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: s = self.context.wrap_socket( 2024-09-15T23:06:41.765 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T23:06:41.766 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: return self.sslsocket_class._create( 2024-09-15T23:06:41.766 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T23:06:41.766 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: self.do_handshake() 2024-09-15T23:06:41.766 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T23:06:41.766 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: self._sslobj.do_handshake() 2024-09-15T23:06:41.766 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T23:06:41.766 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:06:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:06:42.757 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-15T23:06:42.757 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.942+0000 7fc958ed2640 1 Processor -- start 2024-09-15T23:06:42.757 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.942+0000 7fc958ed2640 1 -- start start 2024-09-15T23:06:42.757 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.943+0000 7fc958ed2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954106830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.943+0000 7fc958ed2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc954106e00 con 0x7fc954106430 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.943+0000 7fc95259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.943+0000 7fc95259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51872/0 (socket says 172.21.15.62:51872) 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.943+0000 7fc95259c640 1 -- 172.21.15.62:0/117446086 learned_addr learned my addr 172.21.15.62:0/117446086 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.943+0000 7fc95259c640 1 -- 172.21.15.62:0/117446086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc954107630 con 0x7fc954106430 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.944+0000 7fc95259c640 1 --2- 172.21.15.62:0/117446086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954106830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc944009820 tx=0x7fc94402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909ac63b002fdded server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.944+0000 7fc95159a640 1 -- 172.21.15.62:0/117446086 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc944037070 con 0x7fc954106430 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.944+0000 7fc95159a640 1 -- 172.21.15.62:0/117446086 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc944032920 con 0x7fc954106430 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.944+0000 7fc95159a640 1 -- 172.21.15.62:0/117446086 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc944030810 con 0x7fc954106430 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.944+0000 7fc958ed2640 1 -- 172.21.15.62:0/117446086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 msgr2=0x7fc954106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.944+0000 7fc958ed2640 1 --2- 172.21.15.62:0/117446086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954106830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc944009820 tx=0x7fc94402a270 comp rx=0 tx=0).stop 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.945+0000 7fc958ed2640 1 -- 172.21.15.62:0/117446086 shutdown_connections 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.945+0000 7fc958ed2640 1 --2- 172.21.15.62:0/117446086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954106830 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:42.758 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.945+0000 7fc958ed2640 1 -- 172.21.15.62:0/117446086 >> 172.21.15.62:0/117446086 conn(0x7fc954101bc0 msgr2=0x7fc954104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.945+0000 7fc958ed2640 1 -- 172.21.15.62:0/117446086 shutdown_connections 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.945+0000 7fc958ed2640 1 -- 172.21.15.62:0/117446086 wait complete. 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.945+0000 7fc958ed2640 1 Processor -- start 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.945+0000 7fc958ed2640 1 -- start start 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc958ed2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954191660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc958ed2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc95410a9e0 con 0x7fc954106430 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc95259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954191660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc95259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954191660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51886/0 (socket says 172.21.15.62:51886) 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc95259c640 1 -- 172.21.15.62:0/4025563266 learned_addr learned my addr 172.21.15.62:0/4025563266 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc95259c640 1 -- 172.21.15.62:0/4025563266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc954191ba0 con 0x7fc954106430 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc95259c640 1 --2- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954191660 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc9440097f0 tx=0x7fc9440327b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc94b7fe640 1 -- 172.21.15.62:0/4025563266 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc944037040 con 0x7fc954106430 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc94b7fe640 1 -- 172.21.15.62:0/4025563266 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc944030cc0 con 0x7fc954106430 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95418fd90 con 0x7fc954106430 2024-09-15T23:06:42.759 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc94b7fe640 1 -- 172.21.15.62:0/4025563266 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc94403a640 con 0x7fc954106430 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.946+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9541902d0 con 0x7fc954106430 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.947+0000 7fc94b7fe640 1 -- 172.21.15.62:0/4025563266 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 11) ==== 46368+0+0 (secure 0 0 0) 0x7fc94403a940 con 0x7fc954106430 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.947+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc95410ab60 con 0x7fc954106430 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.947+0000 7fc94b7fe640 1 --2- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9400390f0 0x7fc94003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.947+0000 7fc94b7fe640 1 -- 172.21.15.62:0/4025563266 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc94406c730 con 0x7fc954106430 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.948+0000 7fc951d9b640 1 --2- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9400390f0 0x7fc94003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.948+0000 7fc951d9b640 1 --2- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9400390f0 0x7fc94003b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0098c0 tx=0x7fc93c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.952+0000 7fc94b7fe640 1 -- 172.21.15.62:0/4025563266 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc94403ba40 con 0x7fc954106430 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:41.956+0000 7fc94b7fe640 1 -- 172.21.15.62:0/4025563266 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7fc94406ecc0 con 0x7fc954106430 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.145+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fc954195400 con 0x7fc9400390f0 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.212+0000 7fc94b7fe640 1 -- 172.21.15.62:0/4025563266 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fc954195400 con 0x7fc9400390f0 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9400390f0 msgr2=0x7fc94003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 --2- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9400390f0 0x7fc94003b5b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0098c0 tx=0x7fc93c006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:42.760 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 msgr2=0x7fc954191660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 --2- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954191660 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc9440097f0 tx=0x7fc9440327b0 comp rx=0 tx=0).stop 2024-09-15T23:06:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 shutdown_connections 2024-09-15T23:06:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 --2- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9400390f0 0x7fc94003b5b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 --2- 172.21.15.62:0/4025563266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc954106430 0x7fc954191660 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 >> 172.21.15.62:0/4025563266 conn(0x7fc954101bc0 msgr2=0x7fc954102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 shutdown_connections 2024-09-15T23:06:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:42.214+0000 7fc958ed2640 1 -- 172.21.15.62:0/4025563266 wait complete. 2024-09-15T23:06:42.761 INFO:teuthology.orchestra.run.smithi062.stdout:Creating initial admin user... 2024-09-15T23:06:43.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:42 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:41] ENGINE Serving on http://172.21.15.62:8765 2024-09-15T23:06:43.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:42 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:41] ENGINE Serving on https://172.21.15.62:7150 2024-09-15T23:06:43.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:42 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:41] ENGINE Bus STARTED 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:42 smithi062 ceph-mon[30949]: [15/Sep/2024:23:06:41] ENGINE Error in HTTPServer.serve 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: Traceback (most recent call last): 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: self._connections.run(self.expiration_interval) 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: self._run(expiration_interval) 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: s = self.context.wrap_socket( 2024-09-15T23:06:43.231 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout: return self.sslsocket_class._create( 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout: self.do_handshake() 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout: self._sslobj.do_handshake() 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:42 smithi062 ceph-mon[30949]: mgrmap e12: x(active, since 2s) 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:42 smithi062 ceph-mon[30949]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:43.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:44.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:43 smithi062 ceph-mon[30949]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:44.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:44.786 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$G5dBoPOcY/XovJeDNJKGM.o9pXJM5y7w.VzGJHVhW5BoYrSyjaeyq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726441603, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.155+0000 7f8922327640 1 Processor -- start 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.155+0000 7f8922327640 1 -- start start 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.155+0000 7f8922327640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c07c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.155+0000 7f8922327640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f891c07c870 con 0x7f891c07de40 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.156+0000 7f891bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c07c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.156+0000 7f891bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c07c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58194/0 (socket says 172.21.15.62:58194) 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.156+0000 7f891bfff640 1 -- 172.21.15.62:0/336428357 learned_addr learned my addr 172.21.15.62:0/336428357 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.156+0000 7f891bfff640 1 -- 172.21.15.62:0/336428357 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891c07c9f0 con 0x7f891c07de40 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.157+0000 7f891bfff640 1 --2- 172.21.15.62:0/336428357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c07c2a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f890c009820 tx=0x7f890c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cca062ab39035cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.157+0000 7f891affd640 1 -- 172.21.15.62:0/336428357 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f890c037070 con 0x7f891c07de40 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.157+0000 7f891affd640 1 -- 172.21.15.62:0/336428357 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f890c032920 con 0x7f891c07de40 2024-09-15T23:06:44.787 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.157+0000 7f891affd640 1 -- 172.21.15.62:0/336428357 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f890c030810 con 0x7f891c07de40 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.157+0000 7f8922327640 1 -- 172.21.15.62:0/336428357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 msgr2=0x7f891c07c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.157+0000 7f8922327640 1 --2- 172.21.15.62:0/336428357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c07c2a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f890c009820 tx=0x7f890c02a270 comp rx=0 tx=0).stop 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.158+0000 7f8922327640 1 -- 172.21.15.62:0/336428357 shutdown_connections 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.158+0000 7f8922327640 1 --2- 172.21.15.62:0/336428357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c07c2a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.158+0000 7f8922327640 1 -- 172.21.15.62:0/336428357 >> 172.21.15.62:0/336428357 conn(0x7f891c102cb0 msgr2=0x7f891c105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.158+0000 7f8922327640 1 -- 172.21.15.62:0/336428357 shutdown_connections 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.158+0000 7f8922327640 1 -- 172.21.15.62:0/336428357 wait complete. 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.158+0000 7f8922327640 1 Processor -- start 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.158+0000 7f8922327640 1 -- start start 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f8922327640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c19d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f8922327640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f891c10dc30 con 0x7f891c07de40 2024-09-15T23:06:44.788 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f891bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c19d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f891bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c19d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58204/0 (socket says 172.21.15.62:58204) 2024-09-15T23:06:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f891bfff640 1 -- 172.21.15.62:0/1091042291 learned_addr learned my addr 172.21.15.62:0/1091042291 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f891bfff640 1 -- 172.21.15.62:0/1091042291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891c19dc80 con 0x7f891c07de40 2024-09-15T23:06:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f891bfff640 1 --2- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c19d740 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f890c030380 tx=0x7f890c032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f89197fa640 1 -- 172.21.15.62:0/1091042291 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f890c037040 con 0x7f891c07de40 2024-09-15T23:06:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f89197fa640 1 -- 172.21.15.62:0/1091042291 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f890c040070 con 0x7f891c07de40 2024-09-15T23:06:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f891c19df10 con 0x7f891c07de40 2024-09-15T23:06:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f89197fa640 1 -- 172.21.15.62:0/1091042291 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f890c03a7b0 con 0x7f891c07de40 2024-09-15T23:06:44.789 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.159+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f891c1a0c00 con 0x7f891c07de40 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.160+0000 7f89197fa640 1 -- 172.21.15.62:0/1091042291 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f890c03a9e0 con 0x7f891c07de40 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.160+0000 7f89197fa640 1 --2- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8904039260 0x7f890403b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.160+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f891c07c320 con 0x7f891c07de40 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.161+0000 7f89197fa640 1 -- 172.21.15.62:0/1091042291 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f890c06c9d0 con 0x7f891c07de40 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.161+0000 7f891b7fe640 1 --2- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8904039260 0x7f890403b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.161+0000 7f891b7fe640 1 --2- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8904039260 0x7f890403b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f89100098c0 tx=0x7f8910006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.165+0000 7f89197fa640 1 -- 172.21.15.62:0/1091042291 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f890c044c70 con 0x7f891c07de40 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.359+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f891c19e3f0 con 0x7f8904039260 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.608+0000 7f89197fa640 1 -- 172.21.15.62:0/1091042291 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f891c19e3f0 con 0x7f8904039260 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.610+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8904039260 msgr2=0x7f890403b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.610+0000 7f8922327640 1 --2- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8904039260 0x7f890403b720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f89100098c0 tx=0x7f8910006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.610+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 msgr2=0x7f891c19d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:44.790 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.610+0000 7f8922327640 1 --2- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c19d740 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f890c030380 tx=0x7f890c032670 comp rx=0 tx=0).stop 2024-09-15T23:06:44.791 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.611+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 shutdown_connections 2024-09-15T23:06:44.791 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.611+0000 7f8922327640 1 --2- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8904039260 0x7f890403b720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:44.791 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.611+0000 7f8922327640 1 --2- 172.21.15.62:0/1091042291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f891c07de40 0x7f891c19d740 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:44.791 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.611+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 >> 172.21.15.62:0/1091042291 conn(0x7f891c102cb0 msgr2=0x7f891c103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:44.791 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.611+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 shutdown_connections 2024-09-15T23:06:44.791 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:43.611+0000 7f8922327640 1 -- 172.21.15.62:0/1091042291 wait complete. 2024-09-15T23:06:44.791 INFO:teuthology.orchestra.run.smithi062.stdout:Fetching dashboard port number... 2024-09-15T23:06:45.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:45 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3041302314' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-15T23:06:45.950 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stdout 8443 2024-09-15T23:06:45.950 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.184+0000 7fa5f6e29640 1 Processor -- start 2024-09-15T23:06:45.950 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.184+0000 7fa5f6e29640 1 -- start start 2024-09-15T23:06:45.950 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.185+0000 7fa5f6e29640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f010a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.185+0000 7fa5f6e29640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f007c5a0 con 0x7fa5f0108060 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.185+0000 7fa5f4bc5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f010a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.185+0000 7fa5f4bc5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f010a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58206/0 (socket says 172.21.15.62:58206) 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.185+0000 7fa5f4bc5640 1 -- 172.21.15.62:0/790486864 learned_addr learned my addr 172.21.15.62:0/790486864 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.185+0000 7fa5f4bc5640 1 -- 172.21.15.62:0/790486864 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f010a990 con 0x7fa5f0108060 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.186+0000 7fa5f4bc5640 1 --2- 172.21.15.62:0/790486864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f010a450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009820 tx=0x7fa5e802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f18f05a805e18c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.186+0000 7fa5ef7fe640 1 -- 172.21.15.62:0/790486864 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5e8037070 con 0x7fa5f0108060 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.186+0000 7fa5ef7fe640 1 -- 172.21.15.62:0/790486864 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e8032920 con 0x7fa5f0108060 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.186+0000 7fa5ef7fe640 1 -- 172.21.15.62:0/790486864 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5e8030810 con 0x7fa5f0108060 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 -- 172.21.15.62:0/790486864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 msgr2=0x7fa5f010a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 --2- 172.21.15.62:0/790486864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f010a450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009820 tx=0x7fa5e802a270 comp rx=0 tx=0).stop 2024-09-15T23:06:45.951 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 -- 172.21.15.62:0/790486864 shutdown_connections 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 --2- 172.21.15.62:0/790486864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f010a450 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 -- 172.21.15.62:0/790486864 >> 172.21.15.62:0/790486864 conn(0x7fa5f0101be0 msgr2=0x7fa5f0104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 -- 172.21.15.62:0/790486864 shutdown_connections 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 -- 172.21.15.62:0/790486864 wait complete. 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 Processor -- start 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 -- start start 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f019b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.187+0000 7fa5f6e29640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f010b950 con 0x7fa5f0108060 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5f4bc5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f019b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5f4bc5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f019b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58212/0 (socket says 172.21.15.62:58212) 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5f4bc5640 1 -- 172.21.15.62:0/3041302314 learned_addr learned my addr 172.21.15.62:0/3041302314 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5f4bc5640 1 -- 172.21.15.62:0/3041302314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f019ba50 con 0x7fa5f0108060 2024-09-15T23:06:45.952 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5f4bc5640 1 --2- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f019b510 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8030380 tx=0x7fa5e8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5edffb640 1 -- 172.21.15.62:0/3041302314 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5e8037040 con 0x7fa5f0108060 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5edffb640 1 -- 172.21.15.62:0/3041302314 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e8040070 con 0x7fa5f0108060 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f019bce0 con 0x7fa5f0108060 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5edffb640 1 -- 172.21.15.62:0/3041302314 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa5e803a7b0 con 0x7fa5f0108060 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.188+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f019e9d0 con 0x7fa5f0108060 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.189+0000 7fa5edffb640 1 -- 172.21.15.62:0/3041302314 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7fa5e803a9e0 con 0x7fa5f0108060 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.190+0000 7fa5edffb640 1 --2- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5dc039260 0x7fa5dc03b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.190+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f01084c0 con 0x7fa5f0108060 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.190+0000 7fa5edffb640 1 -- 172.21.15.62:0/3041302314 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa5e806c9d0 con 0x7fa5f0108060 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.190+0000 7fa5effff640 1 --2- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5dc039260 0x7fa5dc03b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.190+0000 7fa5effff640 1 --2- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5dc039260 0x7fa5dc03b720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80098c0 tx=0x7fa5d8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.195+0000 7fa5edffb640 1 -- 172.21.15.62:0/3041302314 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa5e804a200 con 0x7fa5f0108060 2024-09-15T23:06:45.953 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.371+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fa5f019ef90 con 0x7fa5f0108060 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.371+0000 7fa5edffb640 1 -- 172.21.15.62:0/3041302314 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fa5e8031e80 con 0x7fa5f0108060 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5dc039260 msgr2=0x7fa5dc03b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 --2- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5dc039260 0x7fa5dc03b720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80098c0 tx=0x7fa5d8006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 msgr2=0x7fa5f019b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 --2- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f019b510 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8030380 tx=0x7fa5e8032670 comp rx=0 tx=0).stop 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 shutdown_connections 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 --2- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5dc039260 0x7fa5dc03b720 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 --2- 172.21.15.62:0/3041302314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5f0108060 0x7fa5f019b510 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 >> 172.21.15.62:0/3041302314 conn(0x7fa5f0101be0 msgr2=0x7fa5f0102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 shutdown_connections 2024-09-15T23:06:45.954 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:45.374+0000 7fa5f6e29640 1 -- 172.21.15.62:0/3041302314 wait complete. 2024-09-15T23:06:45.958 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-15T23:06:45.958 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout disabled 2024-09-15T23:06:45.965 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-15T23:06:45.965 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stdout inactive 2024-09-15T23:06:45.966 INFO:teuthology.orchestra.run.smithi062.stdout:firewalld.service is not enabled 2024-09-15T23:06:45.966 INFO:teuthology.orchestra.run.smithi062.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-15T23:06:45.967 INFO:teuthology.orchestra.run.smithi062.stdout:Ceph Dashboard is now available at: 2024-09-15T23:06:45.968 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:45.968 INFO:teuthology.orchestra.run.smithi062.stdout: URL: https://smithi062.front.sepia.ceph.com:8443/ 2024-09-15T23:06:45.968 INFO:teuthology.orchestra.run.smithi062.stdout: User: admin 2024-09-15T23:06:45.968 INFO:teuthology.orchestra.run.smithi062.stdout: Password: 4cuw9ikqpl 2024-09-15T23:06:45.968 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:45.968 INFO:teuthology.orchestra.run.smithi062.stdout:Saving cluster configuration to /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config directory 2024-09-15T23:06:47.275 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.393+0000 7f3cfcfcc640 1 Processor -- start 2024-09-15T23:06:47.275 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.393+0000 7f3cfcfcc640 1 -- start start 2024-09-15T23:06:47.275 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.393+0000 7f3cfcfcc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf807cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:47.275 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.393+0000 7f3cfcfcc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf807d100 con 0x7f3cf807e6d0 2024-09-15T23:06:47.275 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.394+0000 7f3cf659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf807cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:47.275 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.394+0000 7f3cf659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf807cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58220/0 (socket says 172.21.15.62:58220) 2024-09-15T23:06:47.275 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.394+0000 7f3cf659c640 1 -- 172.21.15.62:0/1392213589 learned_addr learned my addr 172.21.15.62:0/1392213589 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:47.275 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.394+0000 7f3cf659c640 1 -- 172.21.15.62:0/1392213589 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf807d280 con 0x7f3cf807e6d0 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.395+0000 7f3cf659c640 1 --2- 172.21.15.62:0/1392213589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf807cb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009820 tx=0x7f3ce802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cce49632330e987 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.395+0000 7f3cf5d9b640 1 -- 172.21.15.62:0/1392213589 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce8037070 con 0x7f3cf807e6d0 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.395+0000 7f3cf5d9b640 1 -- 172.21.15.62:0/1392213589 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ce8032920 con 0x7f3cf807e6d0 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.395+0000 7f3cf5d9b640 1 -- 172.21.15.62:0/1392213589 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce8030810 con 0x7f3cf807e6d0 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.396+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1392213589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 msgr2=0x7f3cf807cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.396+0000 7f3cfcfcc640 1 --2- 172.21.15.62:0/1392213589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf807cb30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009820 tx=0x7f3ce802a270 comp rx=0 tx=0).stop 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.396+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1392213589 shutdown_connections 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.396+0000 7f3cfcfcc640 1 --2- 172.21.15.62:0/1392213589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf807cb30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.396+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1392213589 >> 172.21.15.62:0/1392213589 conn(0x7f3cf8102ce0 msgr2=0x7f3cf8105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.396+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1392213589 shutdown_connections 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.396+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1392213589 wait complete. 2024-09-15T23:06:47.276 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.396+0000 7f3cfcfcc640 1 Processor -- start 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.396+0000 7f3cfcfcc640 1 -- start start 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cfcfcc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf819d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cfcfcc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf810dc20 con 0x7f3cf807e6d0 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cf659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf819d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cf659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf819d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58226/0 (socket says 172.21.15.62:58226) 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cf659c640 1 -- 172.21.15.62:0/1937404036 learned_addr learned my addr 172.21.15.62:0/1937404036 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cf659c640 1 -- 172.21.15.62:0/1937404036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf819dcb0 con 0x7f3cf807e6d0 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cf659c640 1 --2- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf819d770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8030380 tx=0x7f3ce8032670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cef7fe640 1 -- 172.21.15.62:0/1937404036 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce8037040 con 0x7f3cf807e6d0 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf819df40 con 0x7f3cf807e6d0 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cef7fe640 1 -- 172.21.15.62:0/1937404036 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ce8040070 con 0x7f3cf807e6d0 2024-09-15T23:06:47.277 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cef7fe640 1 -- 172.21.15.62:0/1937404036 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ce803a7b0 con 0x7f3cf807e6d0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.397+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf81a0c30 con 0x7f3cf807e6d0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.398+0000 7f3cef7fe640 1 -- 172.21.15.62:0/1937404036 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 12) ==== 46474+0+0 (secure 0 0 0) 0x7f3ce803aa30 con 0x7f3cf807e6d0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.398+0000 7f3cef7fe640 1 --2- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3ce4039200 0x7f3ce403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.398+0000 7f3cef7fe640 1 -- 172.21.15.62:0/1937404036 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3ce803b910 con 0x7f3cf807e6d0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.399+0000 7f3ceffff640 1 --2- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3ce4039200 0x7f3ce403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.399+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf807cbb0 con 0x7f3cf807e6d0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.399+0000 7f3ceffff640 1 --2- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3ce4039200 0x7f3ce403b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00098c0 tx=0x7f3ce0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.403+0000 7f3cef7fe640 1 -- 172.21.15.62:0/1937404036 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3ce8044c70 con 0x7f3cf807e6d0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.425+0000 7f3cef7fe640 1 -- 172.21.15.62:0/1937404036 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f3ce803bc80 con 0x7f3cf807e6d0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.672+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f3cf81a10f0 con 0x7f3cf807e6d0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.674+0000 7f3cef7fe640 1 -- 172.21.15.62:0/1937404036 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f3ce80313a0 con 0x7f3cf807e6d0 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-15T23:06:47.278 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.676+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3ce4039200 msgr2=0x7f3ce403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.676+0000 7f3cfcfcc640 1 --2- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3ce4039200 0x7f3ce403b6c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3ce00098c0 tx=0x7f3ce0006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.676+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 msgr2=0x7f3cf819d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.676+0000 7f3cfcfcc640 1 --2- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf819d770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8030380 tx=0x7f3ce8032670 comp rx=0 tx=0).stop 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.676+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 shutdown_connections 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.676+0000 7f3cfcfcc640 1 --2- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3ce4039200 0x7f3ce403b6c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.676+0000 7f3cfcfcc640 1 --2- 172.21.15.62:0/1937404036 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cf807e6d0 0x7f3cf819d770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.676+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 >> 172.21.15.62:0/1937404036 conn(0x7f3cf8102ce0 msgr2=0x7f3cf81038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.676+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 shutdown_connections 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/bin/ceph: stderr 2024-09-15T23:06:46.677+0000 7f3cfcfcc640 1 -- 172.21.15.62:0/1937404036 wait complete. 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:47.279 INFO:teuthology.orchestra.run.smithi062.stdout:Or, if you are only running a single cluster on this host: 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout: ceph telemetry on 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout:For more information see: 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:47.280 INFO:teuthology.orchestra.run.smithi062.stdout:Bootstrap complete. 2024-09-15T23:06:47.317 INFO:tasks.cephadm:Fetching config... 2024-09-15T23:06:47.317 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:06:47.317 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-15T23:06:47.335 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-15T23:06:47.335 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:06:47.335 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-15T23:06:47.392 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-15T23:06:47.392 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:06:47.393 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-15T23:06:47.459 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-15T23:06:47.459 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:06:47.459 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-15T23:06:47.515 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-15T23:06:47.516 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC00w1d4wi6mqNy9KjXmf2HBgM7p6nxu0bccUKwZhdQC4TRV+QuBeyDPGW/JOkGEsKjwSGg8MsziDkySOy2OzFkvPPMiec8IFYprcll7tlE8kT2ndkgjz9BT9JVsHHCXcxgcwyMFRLlGPoneoYYMM8RbwHwsacvUfrHjb3FbTG5/uGz/rJwAC2bhEhGCsZFWmiHCwcmxCb9rmA/b6kZSut9xYP+tXVDBocO25P60mvbcZyVpPsTSpx1tvFgJNVAKivMCxFRHCxyjuCM+Kkpwh/0IwfDkiWLw/7cUUe87dBLFDaQtFTQAaVT2GO6BqSy6Hwe8peCB3FluZDoO6J6cR/u9HITnZv92suwd/LaBrhkTD49W939xFua3PUZY5KiC1lypkosmPP6pzj+9iTxV7Uu/qDFdhWFT9yB2xSz2s7fPzy8lul4t8iX5ppY0YQs8TKJEYfIEIkPiSZtWtx7y8Zj5DH4o3GV/hYhNmJ1DLBXZ/hd22WpBcrCSdz21s+gY80= ceph-ee7f888e-73b6-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-15T23:06:47.553 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:47 smithi062 ceph-mon[30949]: mgrmap e13: x(active, since 6s) 2024-09-15T23:06:47.553 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:47 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1937404036' entity='client.admin' 2024-09-15T23:06:47.589 INFO:teuthology.orchestra.run.smithi062.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC00w1d4wi6mqNy9KjXmf2HBgM7p6nxu0bccUKwZhdQC4TRV+QuBeyDPGW/JOkGEsKjwSGg8MsziDkySOy2OzFkvPPMiec8IFYprcll7tlE8kT2ndkgjz9BT9JVsHHCXcxgcwyMFRLlGPoneoYYMM8RbwHwsacvUfrHjb3FbTG5/uGz/rJwAC2bhEhGCsZFWmiHCwcmxCb9rmA/b6kZSut9xYP+tXVDBocO25P60mvbcZyVpPsTSpx1tvFgJNVAKivMCxFRHCxyjuCM+Kkpwh/0IwfDkiWLw/7cUUe87dBLFDaQtFTQAaVT2GO6BqSy6Hwe8peCB3FluZDoO6J6cR/u9HITnZv92suwd/LaBrhkTD49W939xFua3PUZY5KiC1lypkosmPP6pzj+9iTxV7Uu/qDFdhWFT9yB2xSz2s7fPzy8lul4t8iX5ppY0YQs8TKJEYfIEIkPiSZtWtx7y8Zj5DH4o3GV/hYhNmJ1DLBXZ/hd22WpBcrCSdz21s+gY80= ceph-ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:06:47.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC00w1d4wi6mqNy9KjXmf2HBgM7p6nxu0bccUKwZhdQC4TRV+QuBeyDPGW/JOkGEsKjwSGg8MsziDkySOy2OzFkvPPMiec8IFYprcll7tlE8kT2ndkgjz9BT9JVsHHCXcxgcwyMFRLlGPoneoYYMM8RbwHwsacvUfrHjb3FbTG5/uGz/rJwAC2bhEhGCsZFWmiHCwcmxCb9rmA/b6kZSut9xYP+tXVDBocO25P60mvbcZyVpPsTSpx1tvFgJNVAKivMCxFRHCxyjuCM+Kkpwh/0IwfDkiWLw/7cUUe87dBLFDaQtFTQAaVT2GO6BqSy6Hwe8peCB3FluZDoO6J6cR/u9HITnZv92suwd/LaBrhkTD49W939xFua3PUZY5KiC1lypkosmPP6pzj+9iTxV7Uu/qDFdhWFT9yB2xSz2s7fPzy8lul4t8iX5ppY0YQs8TKJEYfIEIkPiSZtWtx7y8Zj5DH4o3GV/hYhNmJ1DLBXZ/hd22WpBcrCSdz21s+gY80= ceph-ee7f888e-73b6-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-15T23:06:47.643 INFO:teuthology.orchestra.run.smithi067.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC00w1d4wi6mqNy9KjXmf2HBgM7p6nxu0bccUKwZhdQC4TRV+QuBeyDPGW/JOkGEsKjwSGg8MsziDkySOy2OzFkvPPMiec8IFYprcll7tlE8kT2ndkgjz9BT9JVsHHCXcxgcwyMFRLlGPoneoYYMM8RbwHwsacvUfrHjb3FbTG5/uGz/rJwAC2bhEhGCsZFWmiHCwcmxCb9rmA/b6kZSut9xYP+tXVDBocO25P60mvbcZyVpPsTSpx1tvFgJNVAKivMCxFRHCxyjuCM+Kkpwh/0IwfDkiWLw/7cUUe87dBLFDaQtFTQAaVT2GO6BqSy6Hwe8peCB3FluZDoO6J6cR/u9HITnZv92suwd/LaBrhkTD49W939xFua3PUZY5KiC1lypkosmPP6pzj+9iTxV7Uu/qDFdhWFT9yB2xSz2s7fPzy8lul4t8iX5ppY0YQs8TKJEYfIEIkPiSZtWtx7y8Zj5DH4o3GV/hYhNmJ1DLBXZ/hd22WpBcrCSdz21s+gY80= ceph-ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:06:47.656 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC00w1d4wi6mqNy9KjXmf2HBgM7p6nxu0bccUKwZhdQC4TRV+QuBeyDPGW/JOkGEsKjwSGg8MsziDkySOy2OzFkvPPMiec8IFYprcll7tlE8kT2ndkgjz9BT9JVsHHCXcxgcwyMFRLlGPoneoYYMM8RbwHwsacvUfrHjb3FbTG5/uGz/rJwAC2bhEhGCsZFWmiHCwcmxCb9rmA/b6kZSut9xYP+tXVDBocO25P60mvbcZyVpPsTSpx1tvFgJNVAKivMCxFRHCxyjuCM+Kkpwh/0IwfDkiWLw/7cUUe87dBLFDaQtFTQAaVT2GO6BqSy6Hwe8peCB3FluZDoO6J6cR/u9HITnZv92suwd/LaBrhkTD49W939xFua3PUZY5KiC1lypkosmPP6pzj+9iTxV7Uu/qDFdhWFT9yB2xSz2s7fPzy8lul4t8iX5ppY0YQs8TKJEYfIEIkPiSZtWtx7y8Zj5DH4o3GV/hYhNmJ1DLBXZ/hd22WpBcrCSdz21s+gY80= ceph-ee7f888e-73b6-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-15T23:06:47.697 INFO:teuthology.orchestra.run.smithi148.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC00w1d4wi6mqNy9KjXmf2HBgM7p6nxu0bccUKwZhdQC4TRV+QuBeyDPGW/JOkGEsKjwSGg8MsziDkySOy2OzFkvPPMiec8IFYprcll7tlE8kT2ndkgjz9BT9JVsHHCXcxgcwyMFRLlGPoneoYYMM8RbwHwsacvUfrHjb3FbTG5/uGz/rJwAC2bhEhGCsZFWmiHCwcmxCb9rmA/b6kZSut9xYP+tXVDBocO25P60mvbcZyVpPsTSpx1tvFgJNVAKivMCxFRHCxyjuCM+Kkpwh/0IwfDkiWLw/7cUUe87dBLFDaQtFTQAaVT2GO6BqSy6Hwe8peCB3FluZDoO6J6cR/u9HITnZv92suwd/LaBrhkTD49W939xFua3PUZY5KiC1lypkosmPP6pzj+9iTxV7Uu/qDFdhWFT9yB2xSz2s7fPzy8lul4t8iX5ppY0YQs8TKJEYfIEIkPiSZtWtx7y8Zj5DH4o3GV/hYhNmJ1DLBXZ/hd22WpBcrCSdz21s+gY80= ceph-ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:06:47.710 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-15T23:06:47.965 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:06:48.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.472+0000 7f666b436640 1 -- 172.21.15.62:0/2003607607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 msgr2=0x7f6664101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:48.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.472+0000 7f666b436640 1 --2- 172.21.15.62:0/2003607607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 0x7f6664101a00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6660009880 tx=0x7f666002a5d0 comp rx=0 tx=0).stop 2024-09-15T23:06:48.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.473+0000 7f666b436640 1 -- 172.21.15.62:0/2003607607 shutdown_connections 2024-09-15T23:06:48.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.473+0000 7f666b436640 1 --2- 172.21.15.62:0/2003607607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 0x7f6664101a00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:48.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.473+0000 7f666b436640 1 -- 172.21.15.62:0/2003607607 >> 172.21.15.62:0/2003607607 conn(0x7f66640fcdb0 msgr2=0x7f66640ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:48.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.473+0000 7f666b436640 1 -- 172.21.15.62:0/2003607607 shutdown_connections 2024-09-15T23:06:48.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.473+0000 7f666b436640 1 -- 172.21.15.62:0/2003607607 wait complete. 2024-09-15T23:06:48.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.473+0000 7f666b436640 1 Processor -- start 2024-09-15T23:06:48.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.473+0000 7f666b436640 1 -- start start 2024-09-15T23:06:48.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.473+0000 7f666b436640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 0x7f6664196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.473+0000 7f666b436640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6664107710 con 0x7f6664101600 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f66691d2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 0x7f6664196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f66691d2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 0x7f6664196db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58246/0 (socket says 172.21.15.62:58246) 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f66691d2640 1 -- 172.21.15.62:0/371167951 learned_addr learned my addr 172.21.15.62:0/371167951 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f66691d2640 1 -- 172.21.15.62:0/371167951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66641972f0 con 0x7f6664101600 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f66691d2640 1 --2- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 0x7f6664196db0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6660004f60 tx=0x7f66600048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f665a7fc640 1 -- 172.21.15.62:0/371167951 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f666003f070 con 0x7f6664101600 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f665a7fc640 1 -- 172.21.15.62:0/371167951 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6660004d40 con 0x7f6664101600 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6664197580 con 0x7f6664101600 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f665a7fc640 1 -- 172.21.15.62:0/371167951 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6660035030 con 0x7f6664101600 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.474+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f666419a230 con 0x7f6664101600 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.475+0000 7f665a7fc640 1 -- 172.21.15.62:0/371167951 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f6660042070 con 0x7f6664101600 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.475+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6664107e90 con 0x7f6664101600 2024-09-15T23:06:48.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.475+0000 7f665a7fc640 1 --2- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66500392a0 0x7f665003b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.476+0000 7f665a7fc640 1 -- 172.21.15.62:0/371167951 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f666006d000 con 0x7f6664101600 2024-09-15T23:06:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.476+0000 7f66689d1640 1 --2- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66500392a0 0x7f665003b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:48.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.476+0000 7f66689d1640 1 --2- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66500392a0 0x7f665003b760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f66540098c0 tx=0x7f6654006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:48.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.481+0000 7f665a7fc640 1 -- 172.21.15.62:0/371167951 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f666004a260 con 0x7f6664101600 2024-09-15T23:06:48.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.660+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f666402d940 con 0x7f6664101600 2024-09-15T23:06:48.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.663+0000 7f665a7fc640 1 -- 172.21.15.62:0/371167951 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f6660003e10 con 0x7f6664101600 2024-09-15T23:06:48.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.665+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66500392a0 msgr2=0x7f665003b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:48.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.665+0000 7f666b436640 1 --2- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66500392a0 0x7f665003b760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f66540098c0 tx=0x7f6654006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:48.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.665+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 msgr2=0x7f6664196db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:48.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.665+0000 7f666b436640 1 --2- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 0x7f6664196db0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6660004f60 tx=0x7f66600048f0 comp rx=0 tx=0).stop 2024-09-15T23:06:48.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.666+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 shutdown_connections 2024-09-15T23:06:48.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.666+0000 7f666b436640 1 --2- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66500392a0 0x7f665003b760 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:48.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.666+0000 7f666b436640 1 --2- 172.21.15.62:0/371167951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6664101600 0x7f6664196db0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.666+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 >> 172.21.15.62:0/371167951 conn(0x7f66640fcdb0 msgr2=0x7f66640fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.666+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 shutdown_connections 2024-09-15T23:06:48.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:48.666+0000 7f666b436640 1 -- 172.21.15.62:0/371167951 wait complete. 2024-09-15T23:06:49.716 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-15T23:06:49.717 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-15T23:06:49.967 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:06:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:49 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/371167951' entity='client.admin' 2024-09-15T23:06:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi062", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:06:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:06:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.385+0000 7fe8ade2c640 1 -- 172.21.15.62:0/363678964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 msgr2=0x7fe8a8100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.385+0000 7fe8ade2c640 1 --2- 172.21.15.62:0/363678964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 0x7fe8a8100910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe890009880 tx=0x7fe89002a5d0 comp rx=0 tx=0).stop 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.385+0000 7fe8ade2c640 1 -- 172.21.15.62:0/363678964 shutdown_connections 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.385+0000 7fe8ade2c640 1 --2- 172.21.15.62:0/363678964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 0x7fe8a8100910 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.385+0000 7fe8ade2c640 1 -- 172.21.15.62:0/363678964 >> 172.21.15.62:0/363678964 conn(0x7fe8a80fbc80 msgr2=0x7fe8a80fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.385+0000 7fe8ade2c640 1 -- 172.21.15.62:0/363678964 shutdown_connections 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.385+0000 7fe8ade2c640 1 -- 172.21.15.62:0/363678964 wait complete. 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.385+0000 7fe8ade2c640 1 Processor -- start 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.386+0000 7fe8ade2c640 1 -- start start 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.386+0000 7fe8ade2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 0x7fe8a8198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:50.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.386+0000 7fe8ade2c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a8106620 con 0x7fe8a8100510 2024-09-15T23:06:50.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.386+0000 7fe8a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 0x7fe8a8198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:50.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.386+0000 7fe8a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 0x7fe8a8198fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58274/0 (socket says 172.21.15.62:58274) 2024-09-15T23:06:50.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.386+0000 7fe8a77fe640 1 -- 172.21.15.62:0/2175767229 learned_addr learned my addr 172.21.15.62:0/2175767229 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:50.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.386+0000 7fe8a77fe640 1 -- 172.21.15.62:0/2175767229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a81994f0 con 0x7fe8a8100510 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.386+0000 7fe8a77fe640 1 --2- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 0x7fe8a8198fb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe890004f60 tx=0x7fe8900048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.387+0000 7fe8a4ff9640 1 -- 172.21.15.62:0/2175767229 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe89003f070 con 0x7fe8a8100510 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.387+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a8199780 con 0x7fe8a8100510 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.387+0000 7fe8a4ff9640 1 -- 172.21.15.62:0/2175767229 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe890004d40 con 0x7fe8a8100510 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.387+0000 7fe8a4ff9640 1 -- 172.21.15.62:0/2175767229 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe890035030 con 0x7fe8a8100510 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.387+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a819c470 con 0x7fe8a8100510 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.388+0000 7fe8a4ff9640 1 -- 172.21.15.62:0/2175767229 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fe890042070 con 0x7fe8a8100510 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.388+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a8106da0 con 0x7fe8a8100510 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.388+0000 7fe8a4ff9640 1 --2- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8940392a0 0x7fe89403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.388+0000 7fe8a4ff9640 1 -- 172.21.15.62:0/2175767229 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe89006d0e0 con 0x7fe8a8100510 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.388+0000 7fe8a6ffd640 1 --2- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8940392a0 0x7fe89403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:50.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.388+0000 7fe8a6ffd640 1 --2- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8940392a0 0x7fe89403b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe8980098c0 tx=0x7fe898006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:50.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.393+0000 7fe8a4ff9640 1 -- 172.21.15.62:0/2175767229 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe8900303a0 con 0x7fe8a8100510 2024-09-15T23:06:50.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.572+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fe8a802d830 con 0x7fe8940392a0 2024-09-15T23:06:50.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.576+0000 7fe8a4ff9640 1 -- 172.21.15.62:0/2175767229 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe8a802d830 con 0x7fe8940392a0 2024-09-15T23:06:50.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8940392a0 msgr2=0x7fe89403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:50.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 --2- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8940392a0 0x7fe89403b760 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe8980098c0 tx=0x7fe898006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:50.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 msgr2=0x7fe8a8198fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:50.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 --2- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 0x7fe8a8198fb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe890004f60 tx=0x7fe8900048f0 comp rx=0 tx=0).stop 2024-09-15T23:06:50.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 shutdown_connections 2024-09-15T23:06:50.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 --2- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8940392a0 0x7fe89403b760 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:50.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 --2- 172.21.15.62:0/2175767229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8100510 0x7fe8a8198fb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:50.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 >> 172.21.15.62:0/2175767229 conn(0x7fe8a80fbc80 msgr2=0x7fe8a806a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:50.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 shutdown_connections 2024-09-15T23:06:50.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:50.578+0000 7fe8ade2c640 1 -- 172.21.15.62:0/2175767229 wait complete. 2024-09-15T23:06:51.294 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi067 2024-09-15T23:06:51.294 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:06:51.294 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/etc/ceph/ceph.conf 2024-09-15T23:06:51.313 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:06:51.313 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:06:51.371 INFO:tasks.cephadm:Adding host smithi067 to orchestrator... 2024-09-15T23:06:51.372 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch host add smithi067 2024-09-15T23:06:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:06:51.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:51.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:51.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:06:51.845 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:06:51.845 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:06:51.845 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: Updating smithi062:/etc/ceph/ceph.conf 2024-09-15T23:06:51.845 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: Updating smithi062:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:06:51.845 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:06:51.845 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: Updating smithi062:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T23:06:51.845 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:51.845 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:51.845 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:52.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.031+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1348911944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 msgr2=0x7f1d44106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:52.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.031+0000 7f1d4abb2640 1 --2- 172.21.15.62:0/1348911944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 0x7f1d44106950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009880 tx=0x7f1d4002a5d0 comp rx=0 tx=0).stop 2024-09-15T23:06:52.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.031+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1348911944 shutdown_connections 2024-09-15T23:06:52.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.031+0000 7f1d4abb2640 1 --2- 172.21.15.62:0/1348911944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 0x7f1d44106950 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:52.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.031+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1348911944 >> 172.21.15.62:0/1348911944 conn(0x7f1d440fc1a0 msgr2=0x7f1d440fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:52.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.031+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1348911944 shutdown_connections 2024-09-15T23:06:52.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.031+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1348911944 wait complete. 2024-09-15T23:06:52.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.031+0000 7f1d4abb2640 1 Processor -- start 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.032+0000 7f1d4abb2640 1 -- start start 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.032+0000 7f1d4abb2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 0x7f1d440fc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.032+0000 7f1d4abb2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4410aba0 con 0x7f1d44106570 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.032+0000 7f1d4894e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 0x7f1d440fc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.032+0000 7f1d4894e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 0x7f1d440fc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:58290/0 (socket says 172.21.15.62:58290) 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.032+0000 7f1d4894e640 1 -- 172.21.15.62:0/1031595549 learned_addr learned my addr 172.21.15.62:0/1031595549 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.032+0000 7f1d4894e640 1 -- 172.21.15.62:0/1031595549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d440fe540 con 0x7f1d44106570 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.032+0000 7f1d4894e640 1 --2- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 0x7f1d440fc650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009230 tx=0x7f1d40003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.032+0000 7f1d39ffb640 1 -- 172.21.15.62:0/1031595549 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d4003f070 con 0x7f1d44106570 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.033+0000 7f1d39ffb640 1 -- 172.21.15.62:0/1031595549 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d4003bd40 con 0x7f1d44106570 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.033+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d440fcb90 con 0x7f1d44106570 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.033+0000 7f1d39ffb640 1 -- 172.21.15.62:0/1031595549 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d40035030 con 0x7f1d44106570 2024-09-15T23:06:52.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.033+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d440fd050 con 0x7f1d44106570 2024-09-15T23:06:52.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.033+0000 7f1d39ffb640 1 -- 172.21.15.62:0/1031595549 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f1d40042070 con 0x7f1d44106570 2024-09-15T23:06:52.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.034+0000 7f1d39ffb640 1 --2- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1d2c039300 0x7f1d2c03b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:52.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.034+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d44101be0 con 0x7f1d44106570 2024-09-15T23:06:52.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.034+0000 7f1d39ffb640 1 -- 172.21.15.62:0/1031595549 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1d4006cf10 con 0x7f1d44106570 2024-09-15T23:06:52.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.034+0000 7f1d3bfff640 1 --2- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1d2c039300 0x7f1d2c03b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:52.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.034+0000 7f1d3bfff640 1 --2- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1d2c039300 0x7f1d2c03b7c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1d340098c0 tx=0x7f1d34006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:52.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.039+0000 7f1d39ffb640 1 -- 172.21.15.62:0/1031595549 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1d400314c0 con 0x7f1d44106570 2024-09-15T23:06:52.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:52.219+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi067", "target": ["mon-mgr", ""]}) -- 0x7f1d440fdb80 con 0x7f1d2c039300 2024-09-15T23:06:52.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:52 smithi062 ceph-mon[30949]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:06:53.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:53 smithi062 ceph-mon[30949]: Deploying cephadm binary to smithi067 2024-09-15T23:06:56.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.082+0000 7f1d39ffb640 1 -- 172.21.15.62:0/1031595549 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f1d440fdb80 con 0x7f1d2c039300 2024-09-15T23:06:56.084 INFO:teuthology.orchestra.run.smithi062.stdout:Added host 'smithi067' with addr '172.21.15.67' 2024-09-15T23:06:56.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1d2c039300 msgr2=0x7f1d2c03b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 --2- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1d2c039300 0x7f1d2c03b7c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1d340098c0 tx=0x7f1d34006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 msgr2=0x7f1d440fc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 --2- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 0x7f1d440fc650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009230 tx=0x7f1d40003d80 comp rx=0 tx=0).stop 2024-09-15T23:06:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 shutdown_connections 2024-09-15T23:06:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 --2- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1d2c039300 0x7f1d2c03b7c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 --2- 172.21.15.62:0/1031595549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d44106570 0x7f1d440fc650 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 >> 172.21.15.62:0/1031595549 conn(0x7f1d440fc1a0 msgr2=0x7f1d44069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 shutdown_connections 2024-09-15T23:06:56.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:56.085+0000 7f1d4abb2640 1 -- 172.21.15.62:0/1031595549 wait complete. 2024-09-15T23:06:56.698 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-15T23:06:56.948 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:06:57.347 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:57.347 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:57 smithi062 ceph-mon[30949]: Added host smithi067 2024-09-15T23:06:57.348 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:06:57.348 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:57.348 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:57.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.430+0000 7fbfd1184640 1 -- 172.21.15.62:0/3299921491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 msgr2=0x7fbfcc102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:57.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.430+0000 7fbfd1184640 1 --2- 172.21.15.62:0/3299921491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 0x7fbfcc102b10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009810 tx=0x7fbfbc02a540 comp rx=0 tx=0).stop 2024-09-15T23:06:57.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.430+0000 7fbfd1184640 1 -- 172.21.15.62:0/3299921491 shutdown_connections 2024-09-15T23:06:57.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.430+0000 7fbfd1184640 1 --2- 172.21.15.62:0/3299921491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 0x7fbfcc102b10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:57.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.430+0000 7fbfd1184640 1 -- 172.21.15.62:0/3299921491 >> 172.21.15.62:0/3299921491 conn(0x7fbfcc077310 msgr2=0x7fbfcc0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:57.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.430+0000 7fbfd1184640 1 -- 172.21.15.62:0/3299921491 shutdown_connections 2024-09-15T23:06:57.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.430+0000 7fbfd1184640 1 -- 172.21.15.62:0/3299921491 wait complete. 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.431+0000 7fbfd1184640 1 Processor -- start 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.431+0000 7fbfd1184640 1 -- start start 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.431+0000 7fbfd1184640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 0x7fbfcc199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.431+0000 7fbfd1184640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbfcc10bc80 con 0x7fbfcc102730 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.431+0000 7fbfcad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 0x7fbfcc199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.431+0000 7fbfcad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 0x7fbfcc199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49408/0 (socket says 172.21.15.62:49408) 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.431+0000 7fbfcad9d640 1 -- 172.21.15.62:0/4110624718 learned_addr learned my addr 172.21.15.62:0/4110624718 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.431+0000 7fbfcad9d640 1 -- 172.21.15.62:0/4110624718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfcc199960 con 0x7fbfcc102730 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.432+0000 7fbfcad9d640 1 --2- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 0x7fbfcc199420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0045e0 tx=0x7fbfbc004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:57.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.432+0000 7fbfb3fff640 1 -- 172.21.15.62:0/4110624718 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfbc030e70 con 0x7fbfcc102730 2024-09-15T23:06:57.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.432+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfcc199bf0 con 0x7fbfcc102730 2024-09-15T23:06:57.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.432+0000 7fbfb3fff640 1 -- 172.21.15.62:0/4110624718 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfbc0318e0 con 0x7fbfcc102730 2024-09-15T23:06:57.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.432+0000 7fbfb3fff640 1 -- 172.21.15.62:0/4110624718 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbfbc0049c0 con 0x7fbfcc102730 2024-09-15T23:06:57.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.432+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfcc19a050 con 0x7fbfcc102730 2024-09-15T23:06:57.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.433+0000 7fbfb3fff640 1 -- 172.21.15.62:0/4110624718 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fbfbc039040 con 0x7fbfcc102730 2024-09-15T23:06:57.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.433+0000 7fbfb3fff640 1 --2- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfb40392a0 0x7fbfb403b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:57.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.433+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfcc069f90 con 0x7fbfcc102730 2024-09-15T23:06:57.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.433+0000 7fbfb3fff640 1 -- 172.21.15.62:0/4110624718 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbfbc06d5c0 con 0x7fbfcc102730 2024-09-15T23:06:57.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.433+0000 7fbfca59c640 1 --2- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfb40392a0 0x7fbfb403b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:57.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.433+0000 7fbfca59c640 1 --2- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfb40392a0 0x7fbfb403b760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00098c0 tx=0x7fbfc0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:57.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.438+0000 7fbfb3fff640 1 -- 172.21.15.62:0/4110624718 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbfbc040030 con 0x7fbfcc102730 2024-09-15T23:06:57.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.621+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbfcc19a780 con 0x7fbfb40392a0 2024-09-15T23:06:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.622+0000 7fbfb3fff640 1 -- 172.21.15.62:0/4110624718 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7fbfcc19a780 con 0x7fbfb40392a0 2024-09-15T23:06:57.623 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:06:57.624 INFO:teuthology.orchestra.run.smithi062.stdout:[{"addr": "172.21.15.62", "hostname": "smithi062", "labels": [], "status": ""}, {"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}] 2024-09-15T23:06:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfb40392a0 msgr2=0x7fbfb403b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 --2- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfb40392a0 0x7fbfb403b760 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbfc00098c0 tx=0x7fbfc0006db0 comp rx=0 tx=0).stop 2024-09-15T23:06:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 msgr2=0x7fbfcc199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:57.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 --2- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 0x7fbfcc199420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0045e0 tx=0x7fbfbc004610 comp rx=0 tx=0).stop 2024-09-15T23:06:57.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 shutdown_connections 2024-09-15T23:06:57.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 --2- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfb40392a0 0x7fbfb403b760 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:57.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 --2- 172.21.15.62:0/4110624718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfcc102730 0x7fbfcc199420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:57.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 >> 172.21.15.62:0/4110624718 conn(0x7fbfcc077310 msgr2=0x7fbfcc1078c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:57.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 shutdown_connections 2024-09-15T23:06:57.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:57.624+0000 7fbfd1184640 1 -- 172.21.15.62:0/4110624718 wait complete. 2024-09-15T23:06:58.238 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi148 2024-09-15T23:06:58.238 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:06:58.238 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.conf 2024-09-15T23:06:58.257 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:06:58.257 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:06:58.314 INFO:tasks.cephadm:Adding host smithi148 to orchestrator... 2024-09-15T23:06:58.315 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch host add smithi148 2024-09-15T23:06:58.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:06:58.565 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:06:59.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.012+0000 7f3d31623640 1 -- 172.21.15.62:0/2593020677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 msgr2=0x7f3d2c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:06:59.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.012+0000 7f3d31623640 1 --2- 172.21.15.62:0/2593020677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 0x7f3d2c107a40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009880 tx=0x7f3d2402a5d0 comp rx=0 tx=0).stop 2024-09-15T23:06:59.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.012+0000 7f3d31623640 1 -- 172.21.15.62:0/2593020677 shutdown_connections 2024-09-15T23:06:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.012+0000 7f3d31623640 1 --2- 172.21.15.62:0/2593020677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 0x7f3d2c107a40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:06:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.012+0000 7f3d31623640 1 -- 172.21.15.62:0/2593020677 >> 172.21.15.62:0/2593020677 conn(0x7f3d2c0fd290 msgr2=0x7f3d2c0ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:06:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.012+0000 7f3d31623640 1 -- 172.21.15.62:0/2593020677 shutdown_connections 2024-09-15T23:06:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.012+0000 7f3d31623640 1 -- 172.21.15.62:0/2593020677 wait complete. 2024-09-15T23:06:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.012+0000 7f3d31623640 1 Processor -- start 2024-09-15T23:06:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.012+0000 7f3d31623640 1 -- start start 2024-09-15T23:06:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d31623640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 0x7f3d2c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d31623640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c10bc90 con 0x7f3d2c107660 2024-09-15T23:06:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d2affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 0x7f3d2c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d2affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 0x7f3d2c0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49424/0 (socket says 172.21.15.62:49424) 2024-09-15T23:06:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d2affd640 1 -- 172.21.15.62:0/3321974601 learned_addr learned my addr 172.21.15.62:0/3321974601 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:06:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d2affd640 1 -- 172.21.15.62:0/3321974601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c0763e0 con 0x7f3d2c107660 2024-09-15T23:06:59.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d2affd640 1 --2- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 0x7f3d2c0744f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009230 tx=0x7f3d24003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d23fff640 1 -- 172.21.15.62:0/3321974601 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d2403f070 con 0x7f3d2c107660 2024-09-15T23:06:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d23fff640 1 -- 172.21.15.62:0/3321974601 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d2403bc80 con 0x7f3d2c107660 2024-09-15T23:06:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2c074a30 con 0x7f3d2c107660 2024-09-15T23:06:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.013+0000 7f3d23fff640 1 -- 172.21.15.62:0/3321974601 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d24035030 con 0x7f3d2c107660 2024-09-15T23:06:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.014+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2c074ef0 con 0x7f3d2c107660 2024-09-15T23:06:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.014+0000 7f3d23fff640 1 -- 172.21.15.62:0/3321974601 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f3d24042070 con 0x7f3d2c107660 2024-09-15T23:06:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.014+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d2c102cd0 con 0x7f3d2c107660 2024-09-15T23:06:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.015+0000 7f3d23fff640 1 --2- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d180392a0 0x7f3d1803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:06:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.015+0000 7f3d23fff640 1 -- 172.21.15.62:0/3321974601 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3d2406d0e0 con 0x7f3d2c107660 2024-09-15T23:06:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.015+0000 7f3d2a7fc640 1 --2- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d180392a0 0x7f3d1803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:06:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.015+0000 7f3d2a7fc640 1 --2- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d180392a0 0x7f3d1803b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3d140098c0 tx=0x7f3d14006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:06:59.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.020+0000 7f3d23fff640 1 -- 172.21.15.62:0/3321974601 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3d2404c350 con 0x7f3d2c107660 2024-09-15T23:06:59.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:06:59.197+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi148", "target": ["mon-mgr", ""]}) -- 0x7f3d2c075a20 con 0x7f3d180392a0 2024-09-15T23:06:59.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:06:59 smithi062 ceph-mon[30949]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:07:00.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:00 smithi062 ceph-mon[30949]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi148", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:07:00.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:00.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:00.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:00.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:00.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:07:00.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:00.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:01 smithi062 ceph-mon[30949]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:01 smithi062 ceph-mon[30949]: Updating smithi067:/etc/ceph/ceph.conf 2024-09-15T23:07:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:01 smithi062 ceph-mon[30949]: Deploying cephadm binary to smithi148 2024-09-15T23:07:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:01 smithi062 ceph-mon[30949]: Updating smithi067:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:01 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:01 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:01 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:02.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:02 smithi062 ceph-mon[30949]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:07:02.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:02 smithi062 ceph-mon[30949]: Updating smithi067:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T23:07:03.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.098+0000 7f3d23fff640 1 -- 172.21.15.62:0/3321974601 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f3d2c075a20 con 0x7f3d180392a0 2024-09-15T23:07:03.100 INFO:teuthology.orchestra.run.smithi062.stdout:Added host 'smithi148' with addr '172.21.15.148' 2024-09-15T23:07:03.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d180392a0 msgr2=0x7f3d1803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:03.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 --2- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d180392a0 0x7f3d1803b760 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3d140098c0 tx=0x7f3d14006db0 comp rx=0 tx=0).stop 2024-09-15T23:07:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 msgr2=0x7f3d2c0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 --2- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 0x7f3d2c0744f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009230 tx=0x7f3d24003d40 comp rx=0 tx=0).stop 2024-09-15T23:07:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 shutdown_connections 2024-09-15T23:07:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 --2- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d180392a0 0x7f3d1803b760 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 --2- 172.21.15.62:0/3321974601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3d2c107660 0x7f3d2c0744f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 >> 172.21.15.62:0/3321974601 conn(0x7f3d2c0fd290 msgr2=0x7f3d2c0febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 shutdown_connections 2024-09-15T23:07:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:03.101+0000 7f3d31623640 1 -- 172.21.15.62:0/3321974601 wait complete. 2024-09-15T23:07:03.665 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:03 smithi062 ceph-mon[30949]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:03.665 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:03.665 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:03.665 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:03.698 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-15T23:07:03.947 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:07:04.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.398+0000 7fb414ce3640 1 -- 172.21.15.62:0/315121032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 msgr2=0x7fb410102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.398+0000 7fb414ce3640 1 --2- 172.21.15.62:0/315121032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 0x7fb410102b10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb408009810 tx=0x7fb40802a540 comp rx=0 tx=0).stop 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.398+0000 7fb414ce3640 1 -- 172.21.15.62:0/315121032 shutdown_connections 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.398+0000 7fb414ce3640 1 --2- 172.21.15.62:0/315121032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 0x7fb410102b10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.398+0000 7fb414ce3640 1 -- 172.21.15.62:0/315121032 >> 172.21.15.62:0/315121032 conn(0x7fb410077310 msgr2=0x7fb4100fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.398+0000 7fb414ce3640 1 -- 172.21.15.62:0/315121032 shutdown_connections 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.398+0000 7fb414ce3640 1 -- 172.21.15.62:0/315121032 wait complete. 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.399+0000 7fb414ce3640 1 Processor -- start 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.399+0000 7fb414ce3640 1 -- start start 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.399+0000 7fb414ce3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 0x7fb410100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:04.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.399+0000 7fb414ce3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb41010bc80 con 0x7fb410102730 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.399+0000 7fb40e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 0x7fb410100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.400+0000 7fb40e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 0x7fb410100390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47222/0 (socket says 172.21.15.62:47222) 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.400+0000 7fb40e59c640 1 -- 172.21.15.62:0/733235543 learned_addr learned my addr 172.21.15.62:0/733235543 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.400+0000 7fb40e59c640 1 -- 172.21.15.62:0/733235543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb410102280 con 0x7fb410102730 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.400+0000 7fb40e59c640 1 --2- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 0x7fb410100390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb4080045e0 tx=0x7fb408004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.400+0000 7fb4077fe640 1 -- 172.21.15.62:0/733235543 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb408030e70 con 0x7fb410102730 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.400+0000 7fb4077fe640 1 -- 172.21.15.62:0/733235543 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4080318e0 con 0x7fb410102730 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.400+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4101008d0 con 0x7fb410102730 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.400+0000 7fb4077fe640 1 -- 172.21.15.62:0/733235543 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb4080049c0 con 0x7fb410102730 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.400+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb410100d90 con 0x7fb410102730 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.401+0000 7fb4077fe640 1 -- 172.21.15.62:0/733235543 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fb408039040 con 0x7fb410102730 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.401+0000 7fb4077fe640 1 --2- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3fc0392a0 0x7fb3fc03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:04.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.401+0000 7fb4077fe640 1 -- 172.21.15.62:0/733235543 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb40806d740 con 0x7fb410102730 2024-09-15T23:07:04.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.401+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb410069f90 con 0x7fb410102730 2024-09-15T23:07:04.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.401+0000 7fb40dd9b640 1 --2- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3fc0392a0 0x7fb3fc03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:04.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.402+0000 7fb40dd9b640 1 --2- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3fc0392a0 0x7fb3fc03b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80098c0 tx=0x7fb3f8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:04.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.406+0000 7fb4077fe640 1 -- 172.21.15.62:0/733235543 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb408040030 con 0x7fb410102730 2024-09-15T23:07:04.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.590+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb4101018c0 con 0x7fb3fc0392a0 2024-09-15T23:07:04.591 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:04 smithi062 ceph-mon[30949]: Added host smithi148 2024-09-15T23:07:04.592 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:04 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:04.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.591+0000 7fb4077fe640 1 -- 172.21.15.62:0/733235543 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fb4101018c0 con 0x7fb3fc0392a0 2024-09-15T23:07:04.592 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:07:04.592 INFO:teuthology.orchestra.run.smithi062.stdout:[{"addr": "172.21.15.62", "hostname": "smithi062", "labels": [], "status": ""}, {"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.148", "hostname": "smithi148", "labels": [], "status": ""}] 2024-09-15T23:07:04.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3fc0392a0 msgr2=0x7fb3fc03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:04.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 --2- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3fc0392a0 0x7fb3fc03b760 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80098c0 tx=0x7fb3f8006db0 comp rx=0 tx=0).stop 2024-09-15T23:07:04.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 msgr2=0x7fb410100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:04.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 --2- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 0x7fb410100390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb4080045e0 tx=0x7fb408004610 comp rx=0 tx=0).stop 2024-09-15T23:07:04.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 shutdown_connections 2024-09-15T23:07:04.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 --2- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3fc0392a0 0x7fb3fc03b760 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:04.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 --2- 172.21.15.62:0/733235543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb410102730 0x7fb410100390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:04.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 >> 172.21.15.62:0/733235543 conn(0x7fb410077310 msgr2=0x7fb4101034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:04.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 shutdown_connections 2024-09-15T23:07:04.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:04.593+0000 7fb414ce3640 1 -- 172.21.15.62:0/733235543 wait complete. 2024-09-15T23:07:05.158 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-15T23:07:05.158 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-15T23:07:05.408 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:07:05.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:05 smithi062 ceph-mon[30949]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:05.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:05 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.797+0000 7f58668cc640 1 -- 172.21.15.62:0/2565551467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 msgr2=0x7f5860102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.797+0000 7f58668cc640 1 --2- 172.21.15.62:0/2565551467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 0x7f5860102b10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5850009810 tx=0x7f585002a540 comp rx=0 tx=0).stop 2024-09-15T23:07:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.798+0000 7f58668cc640 1 -- 172.21.15.62:0/2565551467 shutdown_connections 2024-09-15T23:07:05.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.798+0000 7f58668cc640 1 --2- 172.21.15.62:0/2565551467 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 0x7f5860102b10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.798+0000 7f58668cc640 1 -- 172.21.15.62:0/2565551467 >> 172.21.15.62:0/2565551467 conn(0x7f5860077310 msgr2=0x7f58600fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.798+0000 7f58668cc640 1 -- 172.21.15.62:0/2565551467 shutdown_connections 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.798+0000 7f58668cc640 1 -- 172.21.15.62:0/2565551467 wait complete. 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.798+0000 7f58668cc640 1 Processor -- start 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.798+0000 7f58668cc640 1 -- start start 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f58668cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 0x7f58601971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f58668cc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f586010bc80 con 0x7f5860102730 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f585ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 0x7f58601971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f585ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 0x7f58601971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47248/0 (socket says 172.21.15.62:47248) 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f585ffff640 1 -- 172.21.15.62:0/3213484489 learned_addr learned my addr 172.21.15.62:0/3213484489 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:07:05.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f585ffff640 1 -- 172.21.15.62:0/3213484489 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58601976f0 con 0x7f5860102730 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f585ffff640 1 --2- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 0x7f58601971b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58500045e0 tx=0x7f5850004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f585d7fa640 1 -- 172.21.15.62:0/3213484489 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5850030e70 con 0x7f5860102730 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f585d7fa640 1 -- 172.21.15.62:0/3213484489 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58500318e0 con 0x7f5860102730 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5860197980 con 0x7f5860102730 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f585d7fa640 1 -- 172.21.15.62:0/3213484489 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58500049c0 con 0x7f5860102730 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.799+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5860197de0 con 0x7f5860102730 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.800+0000 7f585d7fa640 1 -- 172.21.15.62:0/3213484489 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f5850039040 con 0x7f5860102730 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.800+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5860069f90 con 0x7f5860102730 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.800+0000 7f585d7fa640 1 --2- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f58480392a0 0x7f584803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.800+0000 7f585d7fa640 1 -- 172.21.15.62:0/3213484489 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f585006d590 con 0x7f5860102730 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.801+0000 7f585f7fe640 1 --2- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f58480392a0 0x7f584803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:05.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.801+0000 7f585f7fe640 1 --2- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f58480392a0 0x7f584803b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f58540098c0 tx=0x7f5854006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:05.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.805+0000 7f585d7fa640 1 -- 172.21.15.62:0/3213484489 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5850040030 con 0x7f5860102730 2024-09-15T23:07:05.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:05.984+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f5860198570 con 0x7f5860102730 2024-09-15T23:07:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.444+0000 7f585d7fa640 1 -- 172.21.15.62:0/3213484489 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f5850037d70 con 0x7f5860102730 2024-09-15T23:07:06.445 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2024-09-15T23:07:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.446+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f58480392a0 msgr2=0x7f584803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.447+0000 7f58668cc640 1 --2- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f58480392a0 0x7f584803b760 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f58540098c0 tx=0x7f5854006db0 comp rx=0 tx=0).stop 2024-09-15T23:07:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.447+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 msgr2=0x7f58601971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.447+0000 7f58668cc640 1 --2- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 0x7f58601971b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58500045e0 tx=0x7f5850004610 comp rx=0 tx=0).stop 2024-09-15T23:07:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.447+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 shutdown_connections 2024-09-15T23:07:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.447+0000 7f58668cc640 1 --2- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f58480392a0 0x7f584803b760 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:06.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.447+0000 7f58668cc640 1 --2- 172.21.15.62:0/3213484489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5860102730 0x7f58601971b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.447+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 >> 172.21.15.62:0/3213484489 conn(0x7f5860077310 msgr2=0x7f58601054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.447+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 shutdown_connections 2024-09-15T23:07:06.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:06.447+0000 7f58668cc640 1 -- 172.21.15.62:0/3213484489 wait complete. 2024-09-15T23:07:06.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:06 smithi062 ceph-mon[30949]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:07:06.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:06 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3213484489' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-15T23:07:07.034 INFO:tasks.cephadm:Adding mon.a on smithi062 2024-09-15T23:07:07.034 INFO:tasks.cephadm:Adding mon.b on smithi067 2024-09-15T23:07:07.034 INFO:tasks.cephadm:Adding mon.c on smithi148 2024-09-15T23:07:07.035 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi062:172.21.15.62=a;smithi067:172.21.15.67=b;smithi148:172.21.15.148=c' 2024-09-15T23:07:07.308 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-15T23:07:07.354 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-15T23:07:07.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:07.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3213484489' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-15T23:07:07.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:07:07.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:07.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:07.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:07.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:07.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi148", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:07:07.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:07.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:07 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.866+0000 7f4663b42640 1 -- 172.21.15.148:0/703691772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 msgr2=0x7f465c1044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.866+0000 7f4663b42640 1 --2- 172.21.15.148:0/703691772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 0x7f465c1044c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4654009880 tx=0x7f465402a5d0 comp rx=0 tx=0).stop 2024-09-15T23:07:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.866+0000 7f4663b42640 1 -- 172.21.15.148:0/703691772 shutdown_connections 2024-09-15T23:07:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.866+0000 7f4663b42640 1 --2- 172.21.15.148:0/703691772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 0x7f465c1044c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.866+0000 7f4663b42640 1 -- 172.21.15.148:0/703691772 >> 172.21.15.148:0/703691772 conn(0x7f465c0fbcd0 msgr2=0x7f465c0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.866+0000 7f4663b42640 1 -- 172.21.15.148:0/703691772 shutdown_connections 2024-09-15T23:07:07.874 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.866+0000 7f4663b42640 1 -- 172.21.15.148:0/703691772 wait complete. 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.867+0000 7f4663b42640 1 Processor -- start 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.867+0000 7f4663b42640 1 -- start start 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.867+0000 7f4663b42640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 0x7f465c101950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.867+0000 7f4663b42640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f465c107670 con 0x7f465c1020d0 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.867+0000 7f46618de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 0x7f465c101950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.867+0000 7f46618de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 0x7f465c101950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.148:37666/0 (socket says 172.21.15.148:37666) 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.867+0000 7f46618de640 1 -- 172.21.15.148:0/2808062426 learned_addr learned my addr 172.21.15.148:0/2808062426 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.868+0000 7f46618de640 1 -- 172.21.15.148:0/2808062426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465c101e90 con 0x7f465c1020d0 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.868+0000 7f46618de640 1 --2- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 0x7f465c101950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4654004f60 tx=0x7f46540048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.868+0000 7f4652ffd640 1 -- 172.21.15.148:0/2808062426 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f465403f070 con 0x7f465c1020d0 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.868+0000 7f4652ffd640 1 -- 172.21.15.148:0/2808062426 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4654003ad0 con 0x7f465c1020d0 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.868+0000 7f4652ffd640 1 -- 172.21.15.148:0/2808062426 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4654035030 con 0x7f465c1020d0 2024-09-15T23:07:07.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.868+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465c100080 con 0x7f465c1020d0 2024-09-15T23:07:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.868+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f465c1005c0 con 0x7f465c1020d0 2024-09-15T23:07:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.869+0000 7f4652ffd640 1 -- 172.21.15.148:0/2808062426 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f4654042070 con 0x7f465c1020d0 2024-09-15T23:07:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.869+0000 7f4652ffd640 1 --2- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46480392a0 0x7f464803b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.869+0000 7f4652ffd640 1 -- 172.21.15.148:0/2808062426 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f465406d0f0 con 0x7f465c1020d0 2024-09-15T23:07:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.869+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f465c107df0 con 0x7f465c1020d0 2024-09-15T23:07:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.869+0000 7f46610dd640 1 --2- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46480392a0 0x7f464803b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.870+0000 7f46610dd640 1 --2- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46480392a0 0x7f464803b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f464c0098c0 tx=0x7f464c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:07.876 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:07.874+0000 7f4652ffd640 1 -- 172.21.15.148:0/2808062426 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f465404c350 con 0x7f465c1020d0 2024-09-15T23:07:08.068 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.067+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi062:172.21.15.62=a;smithi067:172.21.15.67=b;smithi148:172.21.15.148=c", "target": ["mon-mgr", ""]}) -- 0x7f465c190410 con 0x7f46480392a0 2024-09-15T23:07:08.073 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.073+0000 7f4652ffd640 1 -- 172.21.15.148:0/2808062426 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f465c190410 con 0x7f46480392a0 2024-09-15T23:07:08.074 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled mon update... 2024-09-15T23:07:08.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46480392a0 msgr2=0x7f464803b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:08.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 --2- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46480392a0 0x7f464803b760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f464c0098c0 tx=0x7f464c006db0 comp rx=0 tx=0).stop 2024-09-15T23:07:08.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 msgr2=0x7f465c101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:08.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 --2- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 0x7f465c101950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4654004f60 tx=0x7f46540048f0 comp rx=0 tx=0).stop 2024-09-15T23:07:08.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 shutdown_connections 2024-09-15T23:07:08.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 --2- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46480392a0 0x7f464803b760 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:08.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 --2- 172.21.15.148:0/2808062426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f465c1020d0 0x7f465c101950 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:08.076 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 >> 172.21.15.148:0/2808062426 conn(0x7f465c0fbcd0 msgr2=0x7f465c0fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:08.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 shutdown_connections 2024-09-15T23:07:08.077 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:08.075+0000 7f4663b42640 1 -- 172.21.15.148:0/2808062426 wait complete. 2024-09-15T23:07:08.715 DEBUG:teuthology.orchestra.run.smithi067:mon.b> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.b.service 2024-09-15T23:07:08.718 DEBUG:teuthology.orchestra.run.smithi148:mon.c> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.c.service 2024-09-15T23:07:08.721 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T23:07:08.721 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T23:07:08.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:08 smithi062 ceph-mon[30949]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-15T23:07:08.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:08 smithi062 ceph-mon[30949]: Updating smithi148:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:08.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:08.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:09.469 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:09.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:09 smithi062 ceph-mon[30949]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:09.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:09 smithi062 ceph-mon[30949]: Updating smithi148:/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:07:09.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:09 smithi062 ceph-mon[30949]: Updating smithi148:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-15T23:07:09.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:09 smithi062 ceph-mon[30949]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi062:172.21.15.62=a;smithi067:172.21.15.67=b;smithi148:172.21.15.148=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:07:09.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:09 smithi062 ceph-mon[30949]: Saving service mon spec with placement smithi062:172.21.15.62=a;smithi067:172.21.15.67=b;smithi148:172.21.15.148=c;count:3 2024-09-15T23:07:09.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:09 smithi062 ceph-mon[30949]: Deploying daemon mon.c on smithi148 2024-09-15T23:07:11.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.421+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/295930642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 msgr2=0x7f5d441006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:11.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.421+0000 7f5d4a1c7640 1 --2- 172.21.15.148:0/295930642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 0x7f5d441006a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009810 tx=0x7f5d3402a540 comp rx=0 tx=0).stop 2024-09-15T23:07:11.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.421+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/295930642 shutdown_connections 2024-09-15T23:07:11.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.421+0000 7f5d4a1c7640 1 --2- 172.21.15.148:0/295930642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 0x7f5d441006a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:11.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.421+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/295930642 >> 172.21.15.148:0/295930642 conn(0x7f5d44077310 msgr2=0x7f5d440fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:11.454 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.421+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/295930642 shutdown_connections 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.421+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/295930642 wait complete. 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.422+0000 7f5d4a1c7640 1 Processor -- start 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.422+0000 7f5d4a1c7640 1 -- start start 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.422+0000 7f5d4a1c7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 0x7f5d441993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.422+0000 7f5d4a1c7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4410de80 con 0x7f5d441002c0 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.423+0000 7f5d437fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 0x7f5d441993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.423+0000 7f5d437fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 0x7f5d441993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.148:58042/0 (socket says 172.21.15.148:58042) 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.423+0000 7f5d437fe640 1 -- 172.21.15.148:0/1159082417 learned_addr learned my addr 172.21.15.148:0/1159082417 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.423+0000 7f5d437fe640 1 -- 172.21.15.148:0/1159082417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d441998f0 con 0x7f5d441002c0 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.423+0000 7f5d437fe640 1 --2- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 0x7f5d441993b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009930 tx=0x7f5d34004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.424+0000 7f5d40ff9640 1 -- 172.21.15.148:0/1159082417 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d34031400 con 0x7f5d441002c0 2024-09-15T23:07:11.455 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.424+0000 7f5d40ff9640 1 -- 172.21.15.148:0/1159082417 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d340035f0 con 0x7f5d441002c0 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.424+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d44199b80 con 0x7f5d441002c0 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.424+0000 7f5d40ff9640 1 -- 172.21.15.148:0/1159082417 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d3403c440 con 0x7f5d441002c0 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.424+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d44199fe0 con 0x7f5d441002c0 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.425+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d44069f90 con 0x7f5d441002c0 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.426+0000 7f5d40ff9640 1 -- 172.21.15.148:0/1159082417 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f5d34039040 con 0x7f5d441002c0 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.426+0000 7f5d40ff9640 1 --2- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d2c0392a0 0x7f5d2c03b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.426+0000 7f5d40ff9640 1 -- 172.21.15.148:0/1159082417 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5d3406da20 con 0x7f5d441002c0 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.426+0000 7f5d42ffd640 1 --2- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d2c0392a0 0x7f5d2c03b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.427+0000 7f5d42ffd640 1 --2- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d2c0392a0 0x7f5d2c03b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5d380098c0 tx=0x7f5d38006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:11.456 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.430+0000 7f5d40ff9640 1 -- 172.21.15.148:0/1159082417 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5d34040030 con 0x7f5d441002c0 2024-09-15T23:07:11.693 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.692+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f5d4419a710 con 0x7f5d441002c0 2024-09-15T23:07:11.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.693+0000 7f5d40ff9640 1 -- 172.21.15.148:0/1159082417 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f5d34003b10 con 0x7f5d441002c0 2024-09-15T23:07:11.694 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:07:11.694 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":1,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","modified":"2024-09-15T23:05:39.118425Z","created":"2024-09-15T23:05:39.118425Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-15T23:07:11.695 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 1 2024-09-15T23:07:11.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d2c0392a0 msgr2=0x7f5d2c03b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:11.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 --2- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d2c0392a0 0x7f5d2c03b760 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5d380098c0 tx=0x7f5d38006db0 comp rx=0 tx=0).stop 2024-09-15T23:07:11.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 msgr2=0x7f5d441993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:11.697 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 --2- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 0x7f5d441993b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009930 tx=0x7f5d34004570 comp rx=0 tx=0).stop 2024-09-15T23:07:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 shutdown_connections 2024-09-15T23:07:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 --2- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d2c0392a0 0x7f5d2c03b760 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 --2- 172.21.15.148:0/1159082417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d441002c0 0x7f5d441993b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 >> 172.21.15.148:0/1159082417 conn(0x7f5d44077310 msgr2=0x7f5d440fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 shutdown_connections 2024-09-15T23:07:11.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:11.696+0000 7f5d4a1c7640 1 -- 172.21.15.148:0/1159082417 wait complete. 2024-09-15T23:07:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:11 smithi062 ceph-mon[30949]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:12.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:12 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/1159082417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:07:12.739 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:12 smithi148 ceph-mon[35610]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-15T23:07:13.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:12 smithi148 bash[35590]: 2e8816ac8b2ae38766f783fff8ce673e48e7d6f390918202b6935c722e5ef53d 2024-09-15T23:07:13.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:12 smithi148 systemd[1]: Started Ceph mon.c for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:07:13.264 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T23:07:13.265 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T23:07:13.521 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:07:18.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: mon.a calling monitor election 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: mon.c calling monitor election 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:07:18.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: monmap epoch 2 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: last_changed 2024-09-15T23:07:12.744124+0000 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: created 2024-09-15T23:05:39.118425+0000 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: min_mon_release 19 (squid) 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: election_strategy: 1 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: 1: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.c 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: fsmap 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: mgrmap e13: x(active, since 38s) 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: overall HEALTH_OK 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:18.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:18.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:18.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:18.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:18.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: mon.a calling monitor election 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: mon.c calling monitor election 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: monmap epoch 2 2024-09-15T23:07:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: last_changed 2024-09-15T23:07:12.744124+0000 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: created 2024-09-15T23:05:39.118425+0000 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: min_mon_release 19 (squid) 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: election_strategy: 1 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: 1: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.c 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: fsmap 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: mgrmap e13: x(active, since 38s) 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: overall HEALTH_OK 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:18.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:18.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.365+0000 7f30b20ea640 1 -- 172.21.15.148:0/3861928437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3098003530 msgr2=0x7f30980059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:18.366 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.365+0000 7f30b20ea640 1 --2- 172.21.15.148:0/3861928437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3098003530 0x7f30980059c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f30ac069e40 tx=0x7f309c02c490 comp rx=0 tx=0).stop 2024-09-15T23:07:18.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.365+0000 7f30b20ea640 1 -- 172.21.15.148:0/3861928437 shutdown_connections 2024-09-15T23:07:18.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.365+0000 7f30b20ea640 1 --2- 172.21.15.148:0/3861928437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3098003530 0x7f30980059c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:18.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.365+0000 7f30b20ea640 1 -- 172.21.15.148:0/3861928437 >> 172.21.15.148:0/3861928437 conn(0x7f30ac0fb9a0 msgr2=0x7f30ac0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:18.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.365+0000 7f30b20ea640 1 -- 172.21.15.148:0/3861928437 shutdown_connections 2024-09-15T23:07:18.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.365+0000 7f30b20ea640 1 -- 172.21.15.148:0/3861928437 wait complete. 2024-09-15T23:07:18.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.366+0000 7f30b20ea640 1 Processor -- start 2024-09-15T23:07:18.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.366+0000 7f30b20ea640 1 -- start start 2024-09-15T23:07:18.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30b20ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 0x7f30ac1972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30b20ea640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30ac1977e0 0x7f30ac19bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30b20ea640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac109760 con 0x7f30ac1977e0 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30b20ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1095e0 con 0x7f3098003530 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30ab7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 0x7f30ac1972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30ab7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 0x7f30ac1972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43088/0 (socket says 172.21.15.148:43088) 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30ab7fe640 1 -- 172.21.15.148:0/906958452 learned_addr learned my addr 172.21.15.148:0/906958452 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30a3fff640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30ac1977e0 0x7f30ac19bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30ab7fe640 1 -- 172.21.15.148:0/906958452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 msgr2=0x7f30ac1972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30ab7fe640 1 -- 172.21.15.148:0/906958452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 msgr2=0x7f30ac1972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30ab7fe640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 0x7f30ac1972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30ab7fe640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 0x7f30ac1972a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30a3fff640 1 -- 172.21.15.148:0/906958452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 msgr2=0x7f30ac1972a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:07:18.369 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30a3fff640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 0x7f30ac1972a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:18.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30a3fff640 1 -- 172.21.15.148:0/906958452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ac197e00 con 0x7f30ac1977e0 2024-09-15T23:07:18.370 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.367+0000 7f30a3fff640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30ac1977e0 0x7f30ac19bb70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f309400b3d0 tx=0x7f309400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.368+0000 7f30a97fa640 1 -- 172.21.15.148:0/906958452 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f3094011630 con 0x7f30ac1977e0 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.368+0000 7f30a97fa640 1 -- 172.21.15.148:0/906958452 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3094011cb0 con 0x7f30ac1977e0 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.368+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ac19c0b0 con 0x7f30ac1977e0 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.368+0000 7f30a97fa640 1 -- 172.21.15.148:0/906958452 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f3094010df0 con 0x7f30ac1977e0 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.368+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ac19c670 con 0x7f30ac1977e0 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.369+0000 7f30a97fa640 1 -- 172.21.15.148:0/906958452 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f3094018020 con 0x7f30ac1977e0 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.369+0000 7f30a97fa640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f308c039090 0x7f308c03b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.369+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30ac19c980 con 0x7f30ac1977e0 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.369+0000 7f30a97fa640 1 -- 172.21.15.148:0/906958452 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f309404bed0 con 0x7f30ac1977e0 2024-09-15T23:07:18.371 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.370+0000 7f30ab7fe640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f308c039090 0x7f308c03b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:18.372 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.370+0000 7f30ab7fe640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f308c039090 0x7f308c03b550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f309c012c10 tx=0x7f309c0323a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:18.376 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.375+0000 7f30a97fa640 1 -- 172.21.15.148:0/906958452 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f309401f020 con 0x7f30ac1977e0 2024-09-15T23:07:18.607 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.606+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f30ac190f40 con 0x7f30ac1977e0 2024-09-15T23:07:18.608 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.607+0000 7f30a97fa640 1 -- 172.21.15.148:0/906958452 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f3094018310 con 0x7f30ac1977e0 2024-09-15T23:07:18.608 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:07:18.608 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":2,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","modified":"2024-09-15T23:07:12.744124Z","created":"2024-09-15T23:05:39.118425Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:3300","nonce":0},{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-15T23:07:18.608 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 2 2024-09-15T23:07:18.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f308c039090 msgr2=0x7f308c03b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:18.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f308c039090 0x7f308c03b550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f309c012c10 tx=0x7f309c0323a0 comp rx=0 tx=0).stop 2024-09-15T23:07:18.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30ac1977e0 msgr2=0x7f30ac19bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:18.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30ac1977e0 0x7f30ac19bb70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f309400b3d0 tx=0x7f309400b890 comp rx=0 tx=0).stop 2024-09-15T23:07:18.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 shutdown_connections 2024-09-15T23:07:18.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f308c039090 0x7f308c03b550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:18.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30ac1977e0 0x7f30ac19bb70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:18.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 --2- 172.21.15.148:0/906958452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3098003530 0x7f30ac1972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:18.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 >> 172.21.15.148:0/906958452 conn(0x7f30ac0fb9a0 msgr2=0x7f30ac0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:18.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 shutdown_connections 2024-09-15T23:07:18.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:18.609+0000 7f30b20ea640 1 -- 172.21.15.148:0/906958452 wait complete. 2024-09-15T23:07:18.908 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:18 smithi148 ceph-mon[35610]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:18.909 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:18 smithi148 ceph-mon[35610]: Deploying daemon mon.b on smithi067 2024-09-15T23:07:18.909 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:18 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/906958452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:07:18.909 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:19.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:18 smithi062 ceph-mon[30949]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:19.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:18 smithi062 ceph-mon[30949]: Deploying daemon mon.b on smithi067 2024-09-15T23:07:19.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:18 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/906958452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:07:19.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:20.303 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T23:07:20.303 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T23:07:20.556 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:07:21.028 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:20 smithi148 ceph-mon[35610]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.107+0000 7fda76de1640 1 -- 172.21.15.148:0/1173064844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda70106630 msgr2=0x7fda70106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.107+0000 7fda76de1640 1 --2- 172.21.15.148:0/1173064844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda70106630 0x7fda70106a10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fda64009880 tx=0x7fda6402a5d0 comp rx=0 tx=0).stop 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.107+0000 7fda76de1640 1 -- 172.21.15.148:0/1173064844 shutdown_connections 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.107+0000 7fda76de1640 1 --2- 172.21.15.148:0/1173064844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda70106630 0x7fda70106a10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.107+0000 7fda76de1640 1 -- 172.21.15.148:0/1173064844 >> 172.21.15.148:0/1173064844 conn(0x7fda700fc220 msgr2=0x7fda700fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.107+0000 7fda76de1640 1 -- 172.21.15.148:0/1173064844 shutdown_connections 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.107+0000 7fda76de1640 1 -- 172.21.15.148:0/1173064844 wait complete. 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.108+0000 7fda76de1640 1 Processor -- start 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.108+0000 7fda76de1640 1 -- start start 2024-09-15T23:07:21.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.108+0000 7fda76de1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 0x7fda700fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.108+0000 7fda76de1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda700fcc10 0x7fda700fd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.108+0000 7fda76de1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fda7010ac60 con 0x7fda700fcc10 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.108+0000 7fda76de1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fda7010aae0 con 0x7fda70106630 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.108+0000 7fda74b7d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 0x7fda700fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.108+0000 7fda74b7d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 0x7fda700fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:43096/0 (socket says 172.21.15.148:43096) 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda74b7d640 1 -- 172.21.15.148:0/4021494834 learned_addr learned my addr 172.21.15.148:0/4021494834 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda6ffff640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda700fcc10 0x7fda700fd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda74b7d640 1 -- 172.21.15.148:0/4021494834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 msgr2=0x7fda700fc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda74b7d640 1 -- 172.21.15.148:0/4021494834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 msgr2=0x7fda700fc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda74b7d640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 0x7fda700fc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda74b7d640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 0x7fda700fc6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda6ffff640 1 -- 172.21.15.148:0/4021494834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 msgr2=0x7fda700fc6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:07:21.111 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda6ffff640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 0x7fda700fc6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:21.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda6ffff640 1 -- 172.21.15.148:0/4021494834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda700fe5c0 con 0x7fda700fcc10 2024-09-15T23:07:21.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda6ffff640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda700fcc10 0x7fda700fd070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00b400 tx=0x7fda5c00b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:21.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda6dffb640 1 -- 172.21.15.148:0/4021494834 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fda5c011630 con 0x7fda700fcc10 2024-09-15T23:07:21.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda701a35c0 con 0x7fda700fcc10 2024-09-15T23:07:21.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda6dffb640 1 -- 172.21.15.148:0/4021494834 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda5c011cb0 con 0x7fda700fcc10 2024-09-15T23:07:21.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.109+0000 7fda6dffb640 1 -- 172.21.15.148:0/4021494834 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fda5c010d80 con 0x7fda700fcc10 2024-09-15T23:07:21.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.110+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda701a3b50 con 0x7fda700fcc10 2024-09-15T23:07:21.113 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.110+0000 7fda6dffb640 1 -- 172.21.15.148:0/4021494834 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fda5c018020 con 0x7fda700fcc10 2024-09-15T23:07:21.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.110+0000 7fda6dffb640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda60039030 0x7fda6003b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:21.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.111+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda70101ca0 con 0x7fda700fcc10 2024-09-15T23:07:21.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.111+0000 7fda6dffb640 1 -- 172.21.15.148:0/4021494834 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fda5c04bf00 con 0x7fda700fcc10 2024-09-15T23:07:21.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.111+0000 7fda74b7d640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda60039030 0x7fda6003b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:21.114 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.111+0000 7fda74b7d640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda60039030 0x7fda6003b4f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fda640096c0 tx=0x7fda64005a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:21.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.116+0000 7fda6dffb640 1 -- 172.21.15.148:0/4021494834 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fda5c01f020 con 0x7fda700fcc10 2024-09-15T23:07:21.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:20 smithi062 ceph-mon[30949]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:21.350 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.349+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fda701a3e60 con 0x7fda700fcc10 2024-09-15T23:07:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.350+0000 7fda6dffb640 1 -- 172.21.15.148:0/4021494834 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fda5c00c1b0 con 0x7fda700fcc10 2024-09-15T23:07:21.351 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:07:21.351 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":2,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","modified":"2024-09-15T23:07:12.744124Z","created":"2024-09-15T23:05:39.118425Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:3300","nonce":0},{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-15T23:07:21.351 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 2 2024-09-15T23:07:21.353 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda60039030 msgr2=0x7fda6003b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda60039030 0x7fda6003b4f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fda640096c0 tx=0x7fda64005a40 comp rx=0 tx=0).stop 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda700fcc10 msgr2=0x7fda700fd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda700fcc10 0x7fda700fd070 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00b400 tx=0x7fda5c00b8c0 comp rx=0 tx=0).stop 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 shutdown_connections 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda60039030 0x7fda6003b4f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda700fcc10 0x7fda700fd070 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 --2- 172.21.15.148:0/4021494834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda70106630 0x7fda700fc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 >> 172.21.15.148:0/4021494834 conn(0x7fda700fc220 msgr2=0x7fda70104130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 shutdown_connections 2024-09-15T23:07:21.354 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:21.352+0000 7fda76de1640 1 -- 172.21.15.148:0/4021494834 wait complete. 2024-09-15T23:07:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:21 smithi067 systemd[1]: Started Ceph mon.b for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:07:22.994 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-15T23:07:22.994 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-15T23:07:23.246 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:07:26.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.874+0000 7f2f07eae640 1 -- 172.21.15.148:0/3628827186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2eec003530 msgr2=0x7f2eec0059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:26.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.874+0000 7f2f07eae640 1 --2- 172.21.15.148:0/3628827186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2eec003530 0x7f2eec0059c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f00102af0 tx=0x7f2efc02c150 comp rx=0 tx=0).stop 2024-09-15T23:07:26.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.874+0000 7f2f07eae640 1 -- 172.21.15.148:0/3628827186 shutdown_connections 2024-09-15T23:07:26.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.874+0000 7f2f07eae640 1 --2- 172.21.15.148:0/3628827186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2eec003530 0x7f2eec0059c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:26.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.874+0000 7f2f07eae640 1 -- 172.21.15.148:0/3628827186 >> 172.21.15.148:0/3628827186 conn(0x7f2f000fd320 msgr2=0x7f2f000ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:26.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.874+0000 7f2f07eae640 1 -- 172.21.15.148:0/3628827186 shutdown_connections 2024-09-15T23:07:26.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.874+0000 7f2f07eae640 1 -- 172.21.15.148:0/3628827186 wait complete. 2024-09-15T23:07:26.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f07eae640 1 Processor -- start 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f07eae640 1 -- start start 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f07eae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2eec003530 0x7f2f00199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f07eae640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f00199a60 0x7f2f0019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f07eae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0019e330 0x7f2f0019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f07eae640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2f00109780 con 0x7f2f0019e330 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f07eae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f00109600 con 0x7f2eec003530 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f07eae640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2f00109900 con 0x7f2f00199a60 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f0644b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0019e330 0x7f2f0019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f05449640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f00199a60 0x7f2f0019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f0644b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0019e330 0x7f2f0019e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.148:54504/0 (socket says 172.21.15.148:54504) 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f0644b640 1 -- 172.21.15.148:0/3819060257 learned_addr learned my addr 172.21.15.148:0/3819060257 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:07:26.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f05c4a640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2eec003530 0x7f2f00199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f05449640 1 -- 172.21.15.148:0/3819060257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f00199a60 msgr2=0x7f2f0019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f05449640 1 -- 172.21.15.148:0/3819060257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f00199a60 msgr2=0x7f2f0019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.875+0000 7f2f05449640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f00199a60 0x7f2f0019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f05c4a640 1 -- 172.21.15.148:0/3819060257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2eec003530 msgr2=0x7f2f00199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f05c4a640 1 -- 172.21.15.148:0/3819060257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2eec003530 msgr2=0x7f2f00199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f05c4a640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2eec003530 0x7f2f00199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f05449640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f00199a60 0x7f2f0019ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f05c4a640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2eec003530 0x7f2f00199520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f0644b640 1 -- 172.21.15.148:0/3819060257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f00199a60 msgr2=0x7f2f0019ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f0644b640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f00199a60 0x7f2f0019ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f0644b640 1 -- 172.21.15.148:0/3819060257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2eec003530 msgr2=0x7f2f00199520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:07:26.879 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f0644b640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2eec003530 0x7f2f00199520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:26.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f0644b640 1 -- 172.21.15.148:0/3819060257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f001a60a0 con 0x7f2f0019e330 2024-09-15T23:07:26.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f0644b640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0019e330 0x7f2f0019e7e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800ba50 tx=0x7f2ef800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2ef6ffd640 1 -- 172.21.15.148:0/3819060257 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef8019070 con 0x7f2f0019e330 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2ef6ffd640 1 -- 172.21.15.148:0/3819060257 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ef800cbe0 con 0x7f2f0019e330 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f001a62d0 con 0x7f2f0019e330 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2ef6ffd640 1 -- 172.21.15.148:0/3819060257 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ef80076a0 con 0x7f2f0019e330 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.876+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f001a6790 con 0x7f2f0019e330 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.877+0000 7f2ef6ffd640 1 -- 172.21.15.148:0/3819060257 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f2ef801c070 con 0x7f2f0019e330 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.877+0000 7f2ef6ffd640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ee4039730 0x7f2ee403bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.878+0000 7f2ef6ffd640 1 -- 172.21.15.148:0/3819060257 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2ef804c080 con 0x7f2f0019e330 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.878+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f0010a8b0 con 0x7f2f0019e330 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.878+0000 7f2f05c4a640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ee4039730 0x7f2ee403bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:26.881 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.878+0000 7f2f05c4a640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ee4039730 0x7f2ee403bbf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2efc005dc0 tx=0x7f2efc005d10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:26.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:26.883+0000 7f2ef6ffd640 1 -- 172.21.15.148:0/3819060257 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2ef80112a0 con 0x7f2f0019e330 2024-09-15T23:07:27.127 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.126+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2f0002d7c0 con 0x7f2f0019e330 2024-09-15T23:07:27.128 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:27.128 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:07:27.128 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: mon.c calling monitor election 2024-09-15T23:07:27.128 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: mon.a calling monitor election 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: monmap epoch 3 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: last_changed 2024-09-15T23:07:21.777305+0000 2024-09-15T23:07:27.129 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: created 2024-09-15T23:05:39.118425+0000 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: min_mon_release 19 (squid) 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: election_strategy: 1 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: 1: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.c 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: 2: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: fsmap 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: mgrmap e13: x(active, since 47s) 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: overall HEALTH_OK 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:27.130 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:27.131 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.127+0000 7f2ef6ffd640 1 -- 172.21.15.148:0/3819060257 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f2ef800ce00 con 0x7f2f0019e330 2024-09-15T23:07:27.131 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:07:27.131 INFO:teuthology.orchestra.run.smithi148.stdout:{"epoch":3,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","modified":"2024-09-15T23:07:21.777305Z","created":"2024-09-15T23:05:39.118425Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:3300","nonce":0},{"type":"v1","addr":"172.21.15.148:6789","nonce":0}]},"addr":"172.21.15.148:6789/0","public_addr":"172.21.15.148:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-15T23:07:27.132 INFO:teuthology.orchestra.run.smithi148.stderr:dumped monmap epoch 3 2024-09-15T23:07:27.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.129+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ee4039730 msgr2=0x7f2ee403bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:27.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ee4039730 0x7f2ee403bbf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2efc005dc0 tx=0x7f2efc005d10 comp rx=0 tx=0).stop 2024-09-15T23:07:27.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0019e330 msgr2=0x7f2f0019e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:27.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0019e330 0x7f2f0019e7e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2ef800ba50 tx=0x7f2ef800bf10 comp rx=0 tx=0).stop 2024-09-15T23:07:27.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 shutdown_connections 2024-09-15T23:07:27.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ee4039730 0x7f2ee403bbf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:27.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f0019e330 0x7f2f0019e7e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:27.132 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f00199a60 0x7f2f0019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:27.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 --2- 172.21.15.148:0/3819060257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2eec003530 0x7f2f00199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:27.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 >> 172.21.15.148:0/3819060257 conn(0x7f2f000fd320 msgr2=0x7f2f000ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:27.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 shutdown_connections 2024-09-15T23:07:27.133 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:27.130+0000 7f2f07eae640 1 -- 172.21.15.148:0/3819060257 wait complete. 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: mon.c calling monitor election 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: mon.a calling monitor election 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: monmap epoch 3 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: last_changed 2024-09-15T23:07:21.777305+0000 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: created 2024-09-15T23:05:39.118425+0000 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: min_mon_release 19 (squid) 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: election_strategy: 1 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: 1: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.c 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: 2: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T23:07:27.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: fsmap 2024-09-15T23:07:27.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:07:27.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: mgrmap e13: x(active, since 47s) 2024-09-15T23:07:27.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: overall HEALTH_OK 2024-09-15T23:07:27.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:27.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:27.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:27.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:27.859 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-15T23:07:27.859 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-15T23:07:28.122 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:07:28.165 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:28.166 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:27 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/3819060257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:07:28.166 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:28.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:28.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:27 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/3819060257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:07:28.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:28.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.591+0000 7fb5dae14640 1 -- 172.21.15.62:0/1520424536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 msgr2=0x7fb5d4102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:28.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.591+0000 7fb5dae14640 1 --2- 172.21.15.62:0/1520424536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 0x7fb5d4102b10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009880 tx=0x7fb5c802a670 comp rx=0 tx=0).stop 2024-09-15T23:07:28.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.591+0000 7fb5dae14640 1 -- 172.21.15.62:0/1520424536 shutdown_connections 2024-09-15T23:07:28.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.591+0000 7fb5dae14640 1 --2- 172.21.15.62:0/1520424536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 0x7fb5d4102b10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:28.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.591+0000 7fb5dae14640 1 -- 172.21.15.62:0/1520424536 >> 172.21.15.62:0/1520424536 conn(0x7fb5d40fc130 msgr2=0x7fb5d40fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:28.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.591+0000 7fb5dae14640 1 -- 172.21.15.62:0/1520424536 shutdown_connections 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.591+0000 7fb5dae14640 1 -- 172.21.15.62:0/1520424536 wait complete. 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.592+0000 7fb5dae14640 1 Processor -- start 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.592+0000 7fb5dae14640 1 -- start start 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.592+0000 7fb5dae14640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 0x7fb5d41996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.592+0000 7fb5dae14640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5d4199c20 0x7fb5d419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5dae14640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5d419e4f0 0x7fb5d419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5dae14640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d410bc50 con 0x7fb5d4102730 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5dae14640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d410bad0 con 0x7fb5d419e4f0 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5dae14640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d410bdd0 con 0x7fb5d4199c20 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d8bb0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 0x7fb5d41996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d8bb0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 0x7fb5d41996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55834/0 (socket says 172.21.15.62:55834) 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d8bb0640 1 -- 172.21.15.62:0/4002656050 learned_addr learned my addr 172.21.15.62:0/4002656050 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:07:28.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d3fff640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5d4199c20 0x7fb5d419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d93b1640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5d419e4f0 0x7fb5d419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d8bb0640 1 -- 172.21.15.62:0/4002656050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5d4199c20 msgr2=0x7fb5d419dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d8bb0640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5d4199c20 0x7fb5d419dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d8bb0640 1 -- 172.21.15.62:0/4002656050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5d419e4f0 msgr2=0x7fb5d419e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d8bb0640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5d419e4f0 0x7fb5d419e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d8bb0640 1 -- 172.21.15.62:0/4002656050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d41a6260 con 0x7fb5d4102730 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.593+0000 7fb5d3fff640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5d4199c20 0x7fb5d419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.594+0000 7fb5d8bb0640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 0x7fb5d41996e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8031550 tx=0x7fb5c80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.594+0000 7fb5d1ffb640 1 -- 172.21.15.62:0/4002656050 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c803f030 con 0x7fb5d4102730 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.594+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d41a6430 con 0x7fb5d4102730 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.594+0000 7fb5d1ffb640 1 -- 172.21.15.62:0/4002656050 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5c8003ce0 con 0x7fb5d4102730 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.594+0000 7fb5d1ffb640 1 -- 172.21.15.62:0/4002656050 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8035070 con 0x7fb5d4102730 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.594+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d41a68e0 con 0x7fb5d4102730 2024-09-15T23:07:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.595+0000 7fb5d1ffb640 1 -- 172.21.15.62:0/4002656050 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7fb5c8042030 con 0x7fb5d4102730 2024-09-15T23:07:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.595+0000 7fb5d1ffb640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb5c4039730 0x7fb5c403bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.595+0000 7fb5d1ffb640 1 -- 172.21.15.62:0/4002656050 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb5c8003800 con 0x7fb5d4102730 2024-09-15T23:07:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.595+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d4102b90 con 0x7fb5d4102730 2024-09-15T23:07:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.596+0000 7fb5d3fff640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb5c4039730 0x7fb5c403bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.596+0000 7fb5d3fff640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb5c4039730 0x7fb5c403bbf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009e20 tx=0x7fb5c0009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.601+0000 7fb5d1ffb640 1 -- 172.21.15.62:0/4002656050 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb5c8003b00 con 0x7fb5d4102730 2024-09-15T23:07:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.790+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb5d41a6bd0 con 0x7fb5d4102730 2024-09-15T23:07:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.794+0000 7fb5d1ffb640 1 -- 172.21.15.62:0/4002656050 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fb5c8036260 con 0x7fb5d4102730 2024-09-15T23:07:28.796 INFO:teuthology.orchestra.run.smithi062.stdout:# minimal ceph.conf for ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:28.796 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-09-15T23:07:28.796 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:28.796 INFO:teuthology.orchestra.run.smithi062.stdout: mon_host = [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] 2024-09-15T23:07:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb5c4039730 msgr2=0x7fb5c403bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb5c4039730 0x7fb5c403bbf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009e20 tx=0x7fb5c0009190 comp rx=0 tx=0).stop 2024-09-15T23:07:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 msgr2=0x7fb5d41996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 0x7fb5d41996e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8031550 tx=0x7fb5c80042f0 comp rx=0 tx=0).stop 2024-09-15T23:07:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 shutdown_connections 2024-09-15T23:07:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb5c4039730 0x7fb5c403bbf0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5d419e4f0 0x7fb5d419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb5d4199c20 0x7fb5d419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 --2- 172.21.15.62:0/4002656050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5d4102730 0x7fb5d41996e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 >> 172.21.15.62:0/4002656050 conn(0x7fb5d40fc130 msgr2=0x7fb5d4107b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 shutdown_connections 2024-09-15T23:07:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:28.797+0000 7fb5dae14640 1 -- 172.21.15.62:0/4002656050 wait complete. 2024-09-15T23:07:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:07:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: mon.c calling monitor election 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: mon.a calling monitor election 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: monmap epoch 3 2024-09-15T23:07:29.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: last_changed 2024-09-15T23:07:21.777305+0000 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: created 2024-09-15T23:05:39.118425+0000 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: min_mon_release 19 (squid) 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: election_strategy: 1 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: 1: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.c 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: 2: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: fsmap 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: mgrmap e13: x(active, since 47s) 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: overall HEALTH_OK 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:29.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='client.? 172.21.15.148:0/3819060257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-15T23:07:29.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:29.450 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-15T23:07:29.450 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:07:29.450 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.conf 2024-09-15T23:07:29.481 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:07:29.481 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:07:29.548 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:07:29.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.conf 2024-09-15T23:07:29.575 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:07:29.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:07:29.640 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:07:29.640 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.conf 2024-09-15T23:07:29.669 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:07:29.669 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-15T23:07:29.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: mon.b calling monitor election 2024-09-15T23:07:29.714 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: Updating smithi062:/etc/ceph/ceph.conf 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: Updating smithi067:/etc/ceph/ceph.conf 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: mon.b calling monitor election 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: mon.c calling monitor election 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: mon.a calling monitor election 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: monmap epoch 3 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: last_changed 2024-09-15T23:07:21.777305+0000 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: created 2024-09-15T23:05:39.118425+0000 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: min_mon_release 19 (squid) 2024-09-15T23:07:29.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: election_strategy: 1 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: 1: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.c 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: 2: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: fsmap 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: mgrmap e13: x(active, since 49s) 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: overall HEALTH_OK 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4002656050' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: Updating smithi148:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: Updating smithi067:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: Updating smithi062:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.716 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: Reconfiguring mon.a (unknown last config time)... 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:07:29.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:29.718 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:29 smithi062 ceph-mon[30949]: Reconfiguring daemon mon.a on smithi062 2024-09-15T23:07:29.734 INFO:tasks.cephadm:Adding mgr.x on smithi062 2024-09-15T23:07:29.734 INFO:tasks.cephadm:Adding mgr.y on smithi067 2024-09-15T23:07:29.734 INFO:tasks.cephadm:Adding mgr.z on smithi148 2024-09-15T23:07:29.735 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch apply mgr '3;smithi062=x;smithi067=y;smithi148=z' 2024-09-15T23:07:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: mon.b calling monitor election 2024-09-15T23:07:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: Updating smithi062:/etc/ceph/ceph.conf 2024-09-15T23:07:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: Updating smithi067:/etc/ceph/ceph.conf 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: mon.b calling monitor election 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: mon.c calling monitor election 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: mon.a calling monitor election 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: monmap epoch 3 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: last_changed 2024-09-15T23:07:21.777305+0000 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: created 2024-09-15T23:05:39.118425+0000 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: min_mon_release 19 (squid) 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: election_strategy: 1 2024-09-15T23:07:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: 1: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.c 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: 2: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: fsmap 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: mgrmap e13: x(active, since 49s) 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: overall HEALTH_OK 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4002656050' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: Updating smithi148:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: Updating smithi067:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: Updating smithi062:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: Reconfiguring mon.a (unknown last config time)... 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:29.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:29 smithi067 ceph-mon[35665]: Reconfiguring daemon mon.a on smithi062 2024-09-15T23:07:30.017 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:07:30.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: mon.b calling monitor election 2024-09-15T23:07:30.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: Updating smithi062:/etc/ceph/ceph.conf 2024-09-15T23:07:30.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: Updating smithi067:/etc/ceph/ceph.conf 2024-09-15T23:07:30.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: Updating smithi148:/etc/ceph/ceph.conf 2024-09-15T23:07:30.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: mon.b calling monitor election 2024-09-15T23:07:30.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: mon.c calling monitor election 2024-09-15T23:07:30.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: mon.a calling monitor election 2024-09-15T23:07:30.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-15T23:07:30.041 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: monmap epoch 3 2024-09-15T23:07:30.041 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:07:30.041 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: last_changed 2024-09-15T23:07:21.777305+0000 2024-09-15T23:07:30.041 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: created 2024-09-15T23:05:39.118425+0000 2024-09-15T23:07:30.041 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: min_mon_release 19 (squid) 2024-09-15T23:07:30.041 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: election_strategy: 1 2024-09-15T23:07:30.042 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-09-15T23:07:30.042 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: 1: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.c 2024-09-15T23:07:30.042 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: 2: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.b 2024-09-15T23:07:30.042 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: fsmap 2024-09-15T23:07:30.042 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: osdmap e4: 0 total, 0 up, 0 in 2024-09-15T23:07:30.042 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: mgrmap e13: x(active, since 49s) 2024-09-15T23:07:30.042 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: overall HEALTH_OK 2024-09-15T23:07:30.043 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4002656050' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:30.043 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: Updating smithi148:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:30.043 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: Updating smithi067:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:30.043 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: Updating smithi062:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-15T23:07:30.043 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.043 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.043 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.043 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.043 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.044 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.044 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.044 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.044 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.044 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.044 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.044 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: Reconfiguring mon.a (unknown last config time)... 2024-09-15T23:07:30.044 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:30.045 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:07:30.045 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:30.045 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:29 smithi148 ceph-mon[35610]: Reconfiguring daemon mon.a on smithi062 2024-09-15T23:07:30.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.508+0000 7f1b7d55b640 1 -- 172.21.15.148:0/661326806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 msgr2=0x7f1b781009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:30.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.508+0000 7f1b7d55b640 1 --2- 172.21.15.148:0/661326806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 0x7f1b781009f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009880 tx=0x7f1b6802a5d0 comp rx=0 tx=0).stop 2024-09-15T23:07:30.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 -- 172.21.15.148:0/661326806 shutdown_connections 2024-09-15T23:07:30.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 --2- 172.21.15.148:0/661326806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 0x7f1b781009f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:30.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 -- 172.21.15.148:0/661326806 >> 172.21.15.148:0/661326806 conn(0x7f1b780fbda0 msgr2=0x7f1b780fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 -- 172.21.15.148:0/661326806 shutdown_connections 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 -- 172.21.15.148:0/661326806 wait complete. 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 Processor -- start 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 -- start start 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 0x7f1b7806af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b78199d10 0x7f1b7819c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b7819c640 0x7f1b7819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1b78106700 con 0x7f1b781005f0 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1b78106580 con 0x7f1b78199d10 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b7d55b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1b78106880 con 0x7f1b7819c640 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b76ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 0x7f1b7806af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.509+0000 7f1b777fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b7819c640 0x7f1b7819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:30.513 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b76ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 0x7f1b7806af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.148:54518/0 (socket says 172.21.15.148:54518) 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b76ffd640 1 -- 172.21.15.148:0/2269946624 learned_addr learned my addr 172.21.15.148:0/2269946624 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b777fe640 1 -- 172.21.15.148:0/2269946624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b78199d10 msgr2=0x7f1b7819c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b6ffff640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b78199d10 0x7f1b7819c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b777fe640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b78199d10 0x7f1b7819c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b777fe640 1 -- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 msgr2=0x7f1b7806af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b777fe640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 0x7f1b7806af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b777fe640 1 -- 172.21.15.148:0/2269946624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7819f410 con 0x7f1b7819c640 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b76ffd640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 0x7f1b7806af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b777fe640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b7819c640 0x7f1b7819eb00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000bd50 tx=0x7f1b7000be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b74ff9640 1 -- 172.21.15.148:0/2269946624 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70019070 con 0x7f1b7819c640 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7819f6a0 con 0x7f1b7819c640 2024-09-15T23:07:30.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b74ff9640 1 -- 172.21.15.148:0/2269946624 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b70014920 con 0x7f1b7819c640 2024-09-15T23:07:30.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b74ff9640 1 -- 172.21.15.148:0/2269946624 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b70012990 con 0x7f1b7819c640 2024-09-15T23:07:30.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.510+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b7819fc60 con 0x7f1b7819c640 2024-09-15T23:07:30.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.511+0000 7f1b74ff9640 1 -- 172.21.15.148:0/2269946624 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f1b7001c070 con 0x7f1b7819c640 2024-09-15T23:07:30.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.512+0000 7f1b74ff9640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b64039730 0x7f1b6403bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:30.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.512+0000 7f1b74ff9640 1 -- 172.21.15.148:0/2269946624 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1b7004c0a0 con 0x7f1b7819c640 2024-09-15T23:07:30.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.512+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b78100a70 con 0x7f1b7819c640 2024-09-15T23:07:30.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.512+0000 7f1b76ffd640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b64039730 0x7f1b6403bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:30.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.512+0000 7f1b76ffd640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b64039730 0x7f1b6403bbf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1b680096c0 tx=0x7f1b680315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:30.518 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.517+0000 7f1b74ff9640 1 -- 172.21.15.148:0/2269946624 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1b700139f0 con 0x7f1b7819c640 2024-09-15T23:07:30.698 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.697+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi062=x;smithi067=y;smithi148=z", "target": ["mon-mgr", ""]}) -- 0x7f1b780626c0 con 0x7f1b64039730 2024-09-15T23:07:30.698 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:30 smithi148 ceph-mon[35610]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:30.699 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.699 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:30.699 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.699 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.699 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:30.699 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:07:30.699 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:30.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.702+0000 7f1b74ff9640 1 -- 172.21.15.148:0/2269946624 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1b780626c0 con 0x7f1b64039730 2024-09-15T23:07:30.703 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled mgr update... 2024-09-15T23:07:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b64039730 msgr2=0x7f1b6403bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b64039730 0x7f1b6403bbf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1b680096c0 tx=0x7f1b680315d0 comp rx=0 tx=0).stop 2024-09-15T23:07:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b7819c640 msgr2=0x7f1b7819eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b7819c640 0x7f1b7819eb00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000bd50 tx=0x7f1b7000be40 comp rx=0 tx=0).stop 2024-09-15T23:07:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 shutdown_connections 2024-09-15T23:07:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b64039730 0x7f1b6403bbf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b7819c640 0x7f1b7819eb00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b78199d10 0x7f1b7819c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 --2- 172.21.15.148:0/2269946624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b781005f0 0x7f1b7806af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 >> 172.21.15.148:0/2269946624 conn(0x7f1b780fbda0 msgr2=0x7f1b780fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 shutdown_connections 2024-09-15T23:07:30.706 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:07:30.704+0000 7f1b7d55b640 1 -- 172.21.15.148:0/2269946624 wait complete. 2024-09-15T23:07:30.974 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:30 smithi067 ceph-mon[35665]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:30.974 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:30.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:30.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:07:30.975 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:30 smithi062 ceph-mon[30949]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:07:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:07:30.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:31.350 DEBUG:teuthology.orchestra.run.smithi067:mgr.y> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.y.service 2024-09-15T23:07:31.353 DEBUG:teuthology.orchestra.run.smithi148:mgr.z> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.z.service 2024-09-15T23:07:31.355 INFO:tasks.cephadm:Deploying OSDs... 2024-09-15T23:07:31.356 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:07:31.356 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T23:07:31.376 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T23:07:31.377 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-09-15T23:07:31.436 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T23:07:31.436 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:31.436 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-15T23:07:31.436 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:31.436 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:31.436 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-09-15 23:06:48.142794284 +0000 2024-09-15T23:07:31.436 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-09-15 23:02:41.720868255 +0000 2024-09-15T23:07:31.436 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-09-15 23:02:41.720868255 +0000 2024-09-15T23:07:31.437 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-09-15 23:02:41.720868255 +0000 2024-09-15T23:07:31.437 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T23:07:31.505 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-09-15T23:07:31.505 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-09-15T23:07:31.505 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000245063 s, 2.1 MB/s 2024-09-15T23:07:31.507 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T23:07:31.566 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-09-15T23:07:31.624 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T23:07:31.625 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:31.625 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-15T23:07:31.625 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:31.625 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:31.625 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-09-15 23:06:48.175793779 +0000 2024-09-15T23:07:31.626 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-09-15 23:02:42.048863128 +0000 2024-09-15T23:07:31.626 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-09-15 23:02:42.048863128 +0000 2024-09-15T23:07:31.626 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-09-15 23:02:42.048863128 +0000 2024-09-15T23:07:31.626 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T23:07:31.692 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-09-15T23:07:31.692 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-09-15T23:07:31.692 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000195467 s, 2.6 MB/s 2024-09-15T23:07:31.693 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T23:07:31.752 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-09-15T23:07:31.810 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T23:07:31.810 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:31.810 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T23:07:31.810 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:31.810 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:31.810 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-09-15 23:06:48.208793274 +0000 2024-09-15T23:07:31.810 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-09-15 23:02:42.394857721 +0000 2024-09-15T23:07:31.810 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-09-15 23:02:42.394857721 +0000 2024-09-15T23:07:31.811 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-09-15 23:02:42.394857721 +0000 2024-09-15T23:07:31.811 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T23:07:31.873 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:31 smithi062 ceph-mon[30949]: Reconfiguring mon.b (monmap changed)... 2024-09-15T23:07:31.873 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:31 smithi062 ceph-mon[30949]: Reconfiguring daemon mon.b on smithi067 2024-09-15T23:07:31.873 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:31 smithi062 ceph-mon[30949]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi062=x;smithi067=y;smithi148=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:07:31.873 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:31 smithi062 ceph-mon[30949]: Saving service mgr spec with placement smithi062=x;smithi067=y;smithi148=z;count:3 2024-09-15T23:07:31.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:31.876 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-09-15T23:07:31.876 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-09-15T23:07:31.876 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000218234 s, 2.3 MB/s 2024-09-15T23:07:31.877 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T23:07:31.935 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-09-15T23:07:31.978 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:31 smithi067 ceph-mon[35665]: Reconfiguring mon.b (monmap changed)... 2024-09-15T23:07:31.978 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:31 smithi067 ceph-mon[35665]: Reconfiguring daemon mon.b on smithi067 2024-09-15T23:07:31.978 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:31 smithi067 ceph-mon[35665]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi062=x;smithi067=y;smithi148=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:07:31.978 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:31 smithi067 ceph-mon[35665]: Saving service mgr spec with placement smithi062=x;smithi067=y;smithi148=z;count:3 2024-09-15T23:07:31.978 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:31.991 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T23:07:31.991 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:31.991 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T23:07:31.991 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:31.991 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:31.991 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-09-15 23:06:48.243792739 +0000 2024-09-15T23:07:31.991 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-09-15 23:02:42.744852250 +0000 2024-09-15T23:07:31.992 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-09-15 23:02:42.744852250 +0000 2024-09-15T23:07:31.992 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-09-15 23:02:42.744852250 +0000 2024-09-15T23:07:31.992 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T23:07:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:31 smithi148 ceph-mon[35610]: Reconfiguring mon.b (monmap changed)... 2024-09-15T23:07:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:31 smithi148 ceph-mon[35610]: Reconfiguring daemon mon.b on smithi067 2024-09-15T23:07:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:31 smithi148 ceph-mon[35610]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi062=x;smithi067=y;smithi148=z", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:07:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:31 smithi148 ceph-mon[35610]: Saving service mgr spec with placement smithi062=x;smithi067=y;smithi148=z;count:3 2024-09-15T23:07:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:32.056 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-09-15T23:07:32.056 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-09-15T23:07:32.057 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000200329 s, 2.6 MB/s 2024-09-15T23:07:32.058 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T23:07:32.115 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-09-15T23:07:32.115 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T23:07:32.132 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T23:07:32.132 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-09-15T23:07:32.188 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T23:07:32.188 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:32.188 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-15T23:07:32.188 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:32.188 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:32.188 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 23:07:21.277942388 +0000 2024-09-15T23:07:32.188 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 23:03:00.006965696 +0000 2024-09-15T23:07:32.188 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 23:03:00.006965696 +0000 2024-09-15T23:07:32.188 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 23:03:00.006965696 +0000 2024-09-15T23:07:32.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T23:07:32.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T23:07:32.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T23:07:32.253 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000195705 s, 2.6 MB/s 2024-09-15T23:07:32.254 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T23:07:32.310 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-09-15T23:07:32.367 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T23:07:32.367 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:32.367 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-15T23:07:32.367 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:32.367 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:32.367 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 23:07:21.277942388 +0000 2024-09-15T23:07:32.367 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 23:03:00.330958819 +0000 2024-09-15T23:07:32.367 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 23:03:00.330958819 +0000 2024-09-15T23:07:32.367 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 23:03:00.330958819 +0000 2024-09-15T23:07:32.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T23:07:32.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T23:07:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T23:07:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000186555 s, 2.7 MB/s 2024-09-15T23:07:32.432 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T23:07:32.488 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-09-15T23:07:32.543 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T23:07:32.543 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:32.543 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-09-15T23:07:32.543 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:32.544 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:32.544 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 23:07:21.277942388 +0000 2024-09-15T23:07:32.544 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 23:03:00.674951518 +0000 2024-09-15T23:07:32.544 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 23:03:00.674951518 +0000 2024-09-15T23:07:32.544 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 23:03:00.674951518 +0000 2024-09-15T23:07:32.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T23:07:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T23:07:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T23:07:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000201529 s, 2.5 MB/s 2024-09-15T23:07:32.609 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T23:07:32.665 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-09-15T23:07:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T23:07:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:32.720 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-15T23:07:32.720 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:32.720 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:32.720 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-09-15 23:07:21.278942367 +0000 2024-09-15T23:07:32.720 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-09-15 23:03:00.993944747 +0000 2024-09-15T23:07:32.720 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-09-15 23:03:00.993944747 +0000 2024-09-15T23:07:32.720 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-09-15 23:03:00.993944747 +0000 2024-09-15T23:07:32.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T23:07:32.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-09-15T23:07:32.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-09-15T23:07:32.783 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000215041 s, 2.4 MB/s 2024-09-15T23:07:32.784 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T23:07:32.840 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-09-15T23:07:32.840 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T23:07:32.857 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T23:07:32.857 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2024-09-15T23:07:32.914 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T23:07:32.914 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:32.914 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-15T23:07:32.914 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:32.914 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:32.914 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-15 23:07:11.832495288 +0000 2024-09-15T23:07:32.914 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-15 23:02:53.485589337 +0000 2024-09-15T23:07:32.914 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-15 23:02:53.485589337 +0000 2024-09-15T23:07:32.914 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-15 23:02:53.485589337 +0000 2024-09-15T23:07:32.914 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T23:07:32.978 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-15T23:07:32.978 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-15T23:07:32.978 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000246698 s, 2.1 MB/s 2024-09-15T23:07:32.979 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T23:07:33.036 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2024-09-15T23:07:33.091 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T23:07:33.091 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:33.091 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-15T23:07:33.091 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:33.091 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:33.091 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-15 23:07:11.832495288 +0000 2024-09-15T23:07:33.092 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-15 23:02:53.830581490 +0000 2024-09-15T23:07:33.092 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-15 23:02:53.830581490 +0000 2024-09-15T23:07:33.092 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-15 23:02:53.830581490 +0000 2024-09-15T23:07:33.092 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T23:07:33.153 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:32 smithi148 ceph-mon[35610]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:33.153 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:32 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:33.153 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:32 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:33.153 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:32 smithi148 ceph-mon[35610]: Reconfiguring mon.c (monmap changed)... 2024-09-15T23:07:33.153 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:32 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:33.153 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:32 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:07:33.153 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:32 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:33.153 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:32 smithi148 ceph-mon[35610]: Reconfiguring daemon mon.c on smithi148 2024-09-15T23:07:33.155 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-15T23:07:33.155 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-15T23:07:33.155 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000198588 s, 2.6 MB/s 2024-09-15T23:07:33.156 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T23:07:33.214 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2024-09-15T23:07:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:32 smithi062 ceph-mon[30949]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:32 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:32 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:32 smithi062 ceph-mon[30949]: Reconfiguring mon.c (monmap changed)... 2024-09-15T23:07:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:32 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:32 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:07:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:32 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:32 smithi062 ceph-mon[30949]: Reconfiguring daemon mon.c on smithi148 2024-09-15T23:07:33.269 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T23:07:33.269 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:33.270 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-15T23:07:33.270 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:33.270 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:33.270 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-15 23:07:11.833495267 +0000 2024-09-15T23:07:33.270 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-15 23:02:54.154574120 +0000 2024-09-15T23:07:33.270 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-15 23:02:54.154574120 +0000 2024-09-15T23:07:33.270 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-15 23:02:54.154574120 +0000 2024-09-15T23:07:33.270 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T23:07:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:32 smithi067 ceph-mon[35665]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:32 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:32 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:32 smithi067 ceph-mon[35665]: Reconfiguring mon.c (monmap changed)... 2024-09-15T23:07:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:32 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-15T23:07:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:32 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-15T23:07:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:32 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:32 smithi067 ceph-mon[35665]: Reconfiguring daemon mon.c on smithi148 2024-09-15T23:07:33.335 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-15T23:07:33.336 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-15T23:07:33.336 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000180782 s, 2.8 MB/s 2024-09-15T23:07:33.337 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T23:07:33.393 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2024-09-15T23:07:33.450 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T23:07:33.450 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T23:07:33.450 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-15T23:07:33.451 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T23:07:33.451 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T23:07:33.451 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-09-15 23:07:11.833495267 +0000 2024-09-15T23:07:33.451 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-09-15 23:02:54.485566591 +0000 2024-09-15T23:07:33.451 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-09-15 23:02:54.485566591 +0000 2024-09-15T23:07:33.451 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-09-15 23:02:54.485566591 +0000 2024-09-15T23:07:33.451 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T23:07:33.514 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-09-15T23:07:33.514 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-09-15T23:07:33.515 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000218409 s, 2.3 MB/s 2024-09-15T23:07:33.516 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T23:07:33.573 INFO:tasks.cephadm:Deploying osd.0 on smithi062 with /dev/vg_nvme/lv_4... 2024-09-15T23:07:33.574 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-15T23:07:33.839 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:07:34.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:34.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:34.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:34.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:34.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:34.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:34.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:07:34.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:07:34.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:07:34.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:34.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:34 smithi062 ceph-mon[30949]: Deploying daemon mgr.z on smithi148 2024-09-15T23:07:34.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:34.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:34.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:34.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:34.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:34.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:34.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:07:34.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:07:34.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:07:34.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:34.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:34 smithi067 ceph-mon[35665]: Deploying daemon mgr.z on smithi148 2024-09-15T23:07:34.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:34.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:34.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:34.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:34.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:34.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:34.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:07:34.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:07:34.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:07:34.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:34.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:34 smithi148 ceph-mon[35610]: Deploying daemon mgr.z on smithi148 2024-09-15T23:07:35.249 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 systemd[1]: Starting Ceph mgr.z for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:07:35.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:35 smithi062 ceph-mon[30949]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:35.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:35 smithi067 ceph-mon[35665]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:35.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:35 smithi148 ceph-mon[35610]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:35.539 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 podman[37408]: 2024-09-15 23:07:35.151022695 +0000 UTC m=+0.015950012 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:07:35.540 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 podman[37408]: 2024-09-15 23:07:35.27104206 +0000 UTC m=+0.135969351 container create 39d719a09112c909a3b4523a64d1eca55b79e56b18602266f01eb529b5bff67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-09-15T23:07:35.540 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 podman[37408]: 2024-09-15 23:07:35.465185648 +0000 UTC m=+0.330112950 container init 39d719a09112c909a3b4523a64d1eca55b79e56b18602266f01eb529b5bff67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-15T23:07:35.540 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 podman[37408]: 2024-09-15 23:07:35.468367689 +0000 UTC m=+0.333294982 container start 39d719a09112c909a3b4523a64d1eca55b79e56b18602266f01eb529b5bff67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:07:35.540 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 ceph-mgr[37427]: -- 172.21.15.148:0/3539152009 <== mon.1 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f4558310e0 con 0x55f45580e800 2024-09-15T23:07:35.803 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 bash[37408]: 39d719a09112c909a3b4523a64d1eca55b79e56b18602266f01eb529b5bff67b 2024-09-15T23:07:35.803 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 systemd[1]: Started Ceph mgr.z for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:07:35.803 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:35.590+0000 7f771c609040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:07:35.804 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:35.654+0000 7f771c609040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:07:35.804 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:35.801+0000 7f771c609040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:07:36.140 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:07:36.157 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:35.868+0000 7f771c609040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:07:36.157 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:36.016+0000 7f771c609040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:07:36.157 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:36.155+0000 7f771c609040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:07:36.166 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_4 2024-09-15T23:07:36.419 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:07:36.789 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:36.472+0000 7f771c609040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:07:36.789 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:36.533+0000 7f771c609040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:36.719+0000 7f771c609040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-mon[35610]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:36 smithi148 ceph-mon[35610]: Deploying daemon mgr.y on smithi067 2024-09-15T23:07:36.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:36.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:36.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:36 smithi062 ceph-mon[30949]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:36.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:36.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:07:36.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:07:36.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:07:36.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:36.818 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:36 smithi062 ceph-mon[30949]: Deploying daemon mgr.y on smithi067 2024-09-15T23:07:36.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.816+0000 7f17508dc640 1 -- 172.21.15.62:0/352175281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c102690 msgr2=0x7f174c108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:36.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.816+0000 7f17508dc640 1 --2- 172.21.15.62:0/352175281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c102690 0x7f174c108f20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1744009ca0 tx=0x7f174402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:07:36.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.816+0000 7f17508dc640 1 -- 172.21.15.62:0/352175281 shutdown_connections 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.816+0000 7f17508dc640 1 --2- 172.21.15.62:0/352175281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c102690 0x7f174c108f20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.816+0000 7f17508dc640 1 --2- 172.21.15.62:0/352175281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c101cd0 0x7f174c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.816+0000 7f17508dc640 1 --2- 172.21.15.62:0/352175281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f174c100ad0 0x7f174c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.816+0000 7f17508dc640 1 -- 172.21.15.62:0/352175281 >> 172.21.15.62:0/352175281 conn(0x7f174c0fc280 msgr2=0x7f174c0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.816+0000 7f17508dc640 1 -- 172.21.15.62:0/352175281 shutdown_connections 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.816+0000 7f17508dc640 1 -- 172.21.15.62:0/352175281 wait complete. 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f17508dc640 1 Processor -- start 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f17508dc640 1 -- start start 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f17508dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c100ad0 0x7f174c197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f17508dc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f174c101cd0 0x7f174c1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f17508dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c102690 0x7f174c19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f17508dc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f174c10c750 con 0x7f174c100ad0 2024-09-15T23:07:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f17508dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f174c10c5d0 con 0x7f174c102690 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f17508dc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f174c10c8d0 con 0x7f174c101cd0 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f174a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c100ad0 0x7f174c197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f174a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c100ad0 0x7f174c197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43280/0 (socket says 172.21.15.62:43280) 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f174a59c640 1 -- 172.21.15.62:0/2169940261 learned_addr learned my addr 172.21.15.62:0/2169940261 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f1749d9b640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f174c101cd0 0x7f174c1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f174ad9d640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c102690 0x7f174c19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f174a59c640 1 -- 172.21.15.62:0/2169940261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f174c101cd0 msgr2=0x7f174c1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f174a59c640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f174c101cd0 0x7f174c1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f174a59c640 1 -- 172.21.15.62:0/2169940261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c102690 msgr2=0x7f174c19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f174a59c640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c102690 0x7f174c19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.817+0000 7f174a59c640 1 -- 172.21.15.62:0/2169940261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c19fad0 con 0x7f174c100ad0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.818+0000 7f174a59c640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c100ad0 0x7f174c197ea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f173c00e9c0 tx=0x7f173c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.818+0000 7f17437fe640 1 -- 172.21.15.62:0/2169940261 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c00cd80 con 0x7f174c100ad0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.818+0000 7f17437fe640 1 -- 172.21.15.62:0/2169940261 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f173c0035f0 con 0x7f174c100ad0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.818+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174c19fdc0 con 0x7f174c100ad0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.818+0000 7f17437fe640 1 -- 172.21.15.62:0/2169940261 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c010690 con 0x7f174c100ad0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.818+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174c1a0220 con 0x7f174c100ad0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.819+0000 7f17437fe640 1 -- 172.21.15.62:0/2169940261 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) ==== 46517+0+0 (secure 0 0 0) 0x7f173c010830 con 0x7f174c100ad0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.819+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f174c1a04b0 con 0x7f174c100ad0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.819+0000 7f17437fe640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1738039500 0x7f173803b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.819+0000 7f17437fe640 1 -- 172.21.15.62:0/2169940261 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f173c04cb70 con 0x7f174c100ad0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.819+0000 7f1749d9b640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1738039500 0x7f173803b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:07:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.819+0000 7f1749d9b640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1738039500 0x7f173803b9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f174c1993c0 tx=0x7f1734005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:07:36.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:36.825+0000 7f17437fe640 1 -- 172.21.15.62:0/2169940261 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f173c0203a0 con 0x7f174c100ad0 2024-09-15T23:07:36.884 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:36.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:36.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:36 smithi067 ceph-mon[35665]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:36.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:36.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:07:36.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-15T23:07:36.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:07:36.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:36.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:36 smithi067 ceph-mon[35665]: Deploying daemon mgr.y on smithi067 2024-09-15T23:07:37.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:37.010+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f174c06a680 con 0x7f1738039500 2024-09-15T23:07:37.524 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:37 smithi067 podman[37018]: 2024-09-15 23:07:37.478857283 +0000 UTC m=+0.131731925 container create 73e4e6a40fa40f2a4dd3080d97afd9af0031b933725fb998cd180f9c8b057d26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y, RELEASE=squid-8921c8e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-15T23:07:37.684 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:37 smithi062 ceph-mon[30949]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:07:37.684 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:37 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:07:37.684 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:37 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:07:37.684 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:37 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:37.889 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:37 smithi067 ceph-mon[35665]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:07:37.889 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:37 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:07:37.889 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:37 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:07:37.889 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:37 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:37.889 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:37 smithi067 podman[37018]: 2024-09-15 23:07:37.698145238 +0000 UTC m=+0.351019889 container init 73e4e6a40fa40f2a4dd3080d97afd9af0031b933725fb998cd180f9c8b057d26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:07:37.889 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:37 smithi067 podman[37018]: 2024-09-15 23:07:37.70100098 +0000 UTC m=+0.353875621 container start 73e4e6a40fa40f2a4dd3080d97afd9af0031b933725fb998cd180f9c8b057d26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-15T23:07:37.889 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:37 smithi067 ceph-mgr[37039]: -- 172.21.15.67:0/3856077759 <== mon.2 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b2403e10e0 con 0x55b2403bf400 2024-09-15T23:07:37.889 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:37 smithi067 bash[37018]: 73e4e6a40fa40f2a4dd3080d97afd9af0031b933725fb998cd180f9c8b057d26 2024-09-15T23:07:37.890 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:37 smithi067 systemd[1]: Started Ceph mgr.y for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:07:37.890 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:37 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:37.824+0000 7f9263e44040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T23:07:37.890 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:37 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:37.887+0000 7f9263e44040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T23:07:38.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:37 smithi148 ceph-mon[35610]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:07:38.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:37 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:07:38.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:37 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:07:38.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:37 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:38.040 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:37 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:37.609+0000 7f771c609040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:07:38.040 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:37 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:37.677+0000 7f771c609040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:07:38.274 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:38.044+0000 7f9263e44040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-15T23:07:38.274 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:38.113+0000 7f9263e44040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T23:07:38.554 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:38.276+0000 7f9263e44040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:07:38.554 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:38.426+0000 7f9263e44040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T23:07:38.726 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:38.448+0000 7f771c609040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:07:38.726 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:38.565+0000 7f771c609040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:07:38.726 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:38.724+0000 7f771c609040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:07:38.827 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:38.762+0000 7f9263e44040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T23:07:39.021 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-mon[35610]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:39.021 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.022 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.022 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.022 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.022 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:39.022 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:38.787+0000 7f771c609040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:07:39.022 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:38.941+0000 7f771c609040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:07:39.022 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:39.019+0000 7f771c609040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:07:39.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:38 smithi062 ceph-mon[30949]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:39.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:39.274 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:38.825+0000 7f9263e44040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T23:07:39.274 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:39.027+0000 7f9263e44040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T23:07:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-mon[35665]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:39.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:39.289 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:39.081+0000 7f771c609040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:07:39.289 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:39.245+0000 7f771c609040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:07:39.579 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:39.311+0000 7f771c609040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:07:39.579 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:39.377+0000 7f771c609040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:07:39.579 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:39.460+0000 7f771c609040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:07:39.579 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:39.518+0000 7f771c609040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:07:39.579 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:39.577+0000 7f771c609040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:07:40.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: Reconfiguring mgr.x (unknown last config time)... 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: Reconfiguring daemon mgr.x on smithi062 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3033324519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2a0a912-0888-41cb-a20e-745c2c97e6df"}]: dispatch 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3033324519' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2a0a912-0888-41cb-a20e-745c2c97e6df"}]': finished 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: osdmap e5: 1 total, 0 up, 1 in 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:39 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/246904551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:07:40.274 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:39.957+0000 7f9263e44040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T23:07:40.274 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:40 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:40.027+0000 7f9263e44040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T23:07:40.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: Reconfiguring mgr.x (unknown last config time)... 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: Reconfiguring daemon mgr.x on smithi062 2024-09-15T23:07:40.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3033324519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2a0a912-0888-41cb-a20e-745c2c97e6df"}]: dispatch 2024-09-15T23:07:40.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3033324519' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2a0a912-0888-41cb-a20e-745c2c97e6df"}]': finished 2024-09-15T23:07:40.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: osdmap e5: 1 total, 0 up, 1 in 2024-09-15T23:07:40.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:07:40.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:39 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/246904551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:07:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: Reconfiguring mgr.x (unknown last config time)... 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: Reconfiguring daemon mgr.x on smithi062 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3033324519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2a0a912-0888-41cb-a20e-745c2c97e6df"}]: dispatch 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3033324519' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2a0a912-0888-41cb-a20e-745c2c97e6df"}]': finished 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: osdmap e5: 1 total, 0 up, 1 in 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:40.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:39 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/246904551' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:07:40.789 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:07:40 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z[37423]: 2024-09-15T23:07:40.366+0000 7f771c609040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:07:41.042 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:40 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:40.773+0000 7f9263e44040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-15T23:07:41.042 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:40 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:40.892+0000 7f9263e44040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T23:07:41.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:41.292+0000 7f17437fe640 1 -- 172.21.15.62:0/2169940261 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 14) ==== 92408+0+0 (secure 0 0 0) 0x7f173c021440 con 0x7f174c100ad0 2024-09-15T23:07:41.330 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-mon[35665]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:41.330 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-mon[35665]: Standby manager daemon z started 2024-09-15T23:07:41.330 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-mon[35665]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T23:07:41.331 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-mon[35665]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:07:41.331 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-mon[35665]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T23:07:41.331 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-mon[35665]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:07:41.331 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.040+0000 7f9263e44040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T23:07:41.331 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.103+0000 7f9263e44040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T23:07:41.331 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.258+0000 7f9263e44040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-15T23:07:41.331 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.329+0000 7f9263e44040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T23:07:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:41 smithi148 ceph-mon[35610]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:41 smithi148 ceph-mon[35610]: Standby manager daemon z started 2024-09-15T23:07:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:41 smithi148 ceph-mon[35610]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T23:07:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:41 smithi148 ceph-mon[35610]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:07:41.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:41 smithi148 ceph-mon[35610]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T23:07:41.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:41 smithi148 ceph-mon[35610]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:07:41.623 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.391+0000 7f9263e44040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T23:07:41.624 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.555+0000 7f9263e44040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T23:07:41.624 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.622+0000 7f9263e44040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T23:07:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:41 smithi062 ceph-mon[30949]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:41 smithi062 ceph-mon[30949]: Standby manager daemon z started 2024-09-15T23:07:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:41 smithi062 ceph-mon[30949]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-09-15T23:07:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:41 smithi062 ceph-mon[30949]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:07:41.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:41 smithi062 ceph-mon[30949]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-09-15T23:07:41.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:41 smithi062 ceph-mon[30949]: from='mgr.? 172.21.15.148:0/1650141225' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:07:41.900 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.690+0000 7f9263e44040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T23:07:41.900 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.776+0000 7f9263e44040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T23:07:41.900 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.837+0000 7f9263e44040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T23:07:41.900 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:41.898+0000 7f9263e44040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T23:07:42.723 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:42 smithi067 ceph-mon[35665]: mgrmap e14: x(active, since 61s), standbys: z 2024-09-15T23:07:42.723 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T23:07:42.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:42 smithi062 ceph-mon[30949]: mgrmap e14: x(active, since 61s), standbys: z 2024-09-15T23:07:42.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T23:07:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:42 smithi148 ceph-mon[35610]: mgrmap e14: x(active, since 61s), standbys: z 2024-09-15T23:07:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-09-15T23:07:43.024 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:07:42 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y[37035]: 2024-09-15T23:07:42.722+0000 7f9263e44040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T23:07:43.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:43.304+0000 7f17437fe640 1 -- 172.21.15.62:0/2169940261 <== mon.0 v2:172.21.15.62:3300/0 8 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f173c05fd90 con 0x7f174c100ad0 2024-09-15T23:07:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:43 smithi062 ceph-mon[30949]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:43 smithi062 ceph-mon[30949]: Standby manager daemon y started 2024-09-15T23:07:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:43 smithi062 ceph-mon[30949]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T23:07:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:43 smithi062 ceph-mon[30949]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:07:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:43 smithi062 ceph-mon[30949]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T23:07:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:43 smithi062 ceph-mon[30949]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:07:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:43 smithi067 ceph-mon[35665]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:43 smithi067 ceph-mon[35665]: Standby manager daemon y started 2024-09-15T23:07:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:43 smithi067 ceph-mon[35665]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T23:07:43.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:43 smithi067 ceph-mon[35665]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:07:43.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:43 smithi067 ceph-mon[35665]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T23:07:43.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:43 smithi067 ceph-mon[35665]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:07:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:43 smithi148 ceph-mon[35610]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:43 smithi148 ceph-mon[35610]: Standby manager daemon y started 2024-09-15T23:07:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:43 smithi148 ceph-mon[35610]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-15T23:07:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:43 smithi148 ceph-mon[35610]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-15T23:07:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:43 smithi148 ceph-mon[35610]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-15T23:07:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:43 smithi148 ceph-mon[35610]: from='mgr.? 172.21.15.67:0/400802374' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-15T23:07:44.695 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:44 smithi062 ceph-mon[30949]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-09-15T23:07:44.695 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T23:07:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:44 smithi067 ceph-mon[35665]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-09-15T23:07:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T23:07:44.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:44 smithi148 ceph-mon[35610]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-09-15T23:07:44.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-15T23:07:45.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:45 smithi062 ceph-mon[30949]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:45 smithi067 ceph-mon[35665]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:45.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:45 smithi148 ceph-mon[35610]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:47.625 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:47 smithi062 ceph-mon[30949]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:47.625 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:47.625 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:47.625 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:47 smithi067 ceph-mon[35665]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:47.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:47.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:47.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:47 smithi148 ceph-mon[35610]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:47.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-15T23:07:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:48.675 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:48.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:48.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:48.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-15T23:07:48.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:48.690 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:48.691 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:48.691 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:48.691 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-15T23:07:48.691 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:49.657 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:49 smithi062 ceph-mon[30949]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:49.657 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:49 smithi062 ceph-mon[30949]: Deploying daemon osd.0 on smithi062 2024-09-15T23:07:49.706 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:49 smithi148 ceph-mon[35610]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:49.706 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:49 smithi148 ceph-mon[35610]: Deploying daemon osd.0 on smithi062 2024-09-15T23:07:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:49 smithi067 ceph-mon[35665]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:49 smithi067 ceph-mon[35665]: Deploying daemon osd.0 on smithi062 2024-09-15T23:07:51.729 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:51 smithi062 ceph-mon[30949]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:51 smithi067 ceph-mon[35665]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:51 smithi148 ceph-mon[35610]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:53.665 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:53 smithi062 ceph-mon[30949]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:53.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:53 smithi067 ceph-mon[35665]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:53 smithi148 ceph-mon[35610]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:55.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:55 smithi062 ceph-mon[30949]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:55.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:55.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:55.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:55 smithi067 ceph-mon[35665]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:55 smithi148 ceph-mon[35610]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:57 smithi062 ceph-mon[30949]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:57.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:57.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:57.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:57 smithi067 ceph-mon[35665]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:57.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:57.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:57.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:57.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:57.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:57 smithi148 ceph-mon[35610]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:07:57.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:07:57.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:58.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.662+0000 7f17437fe640 1 -- 172.21.15.62:0/2169940261 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f174c06a680 con 0x7f1738039500 2024-09-15T23:07:58.663 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 0 on host 'smithi062' 2024-09-15T23:07:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1738039500 msgr2=0x7f173803b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1738039500 0x7f173803b9c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f174c1993c0 tx=0x7f1734005f90 comp rx=0 tx=0).stop 2024-09-15T23:07:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c100ad0 msgr2=0x7f174c197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:07:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c100ad0 0x7f174c197ea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f173c00e9c0 tx=0x7f173c00ee80 comp rx=0 tx=0).stop 2024-09-15T23:07:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 shutdown_connections 2024-09-15T23:07:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1738039500 0x7f173803b9c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f174c102690 0x7f174c19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f174c101cd0 0x7f174c1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 --2- 172.21.15.62:0/2169940261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c100ad0 0x7f174c197ea0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:07:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 >> 172.21.15.62:0/2169940261 conn(0x7f174c0fc280 msgr2=0x7f174c069f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:07:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 shutdown_connections 2024-09-15T23:07:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:07:58.664+0000 7f17508dc640 1 -- 172.21.15.62:0/2169940261 wait complete. 2024-09-15T23:07:59.257 DEBUG:teuthology.orchestra.run.smithi062:osd.0> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.0.service 2024-09-15T23:07:59.259 INFO:tasks.cephadm:Deploying osd.1 on smithi067 with /dev/vg_nvme/lv_4... 2024-09-15T23:07:59.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-15T23:07:59.506 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:07:59.553 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:59 smithi062 ceph-mon[30949]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:59.553 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:59.553 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:59.553 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:07:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:59.627 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:59 smithi067 ceph-mon[35665]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:59.627 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:59.628 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:59.628 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:07:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:59.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:59 smithi148 ceph-mon[35610]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:07:59.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:07:59.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:07:59.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:07:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:00.230 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0[42523]: 2024-09-15T23:08:00.036+0000 7fcfefc00640 -1 osd.0 0 log_to_monitors true 2024-09-15T23:08:00.669 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:00 smithi062 ceph-mon[30949]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-15T23:08:00.747 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:00 smithi067 ceph-mon[35665]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-15T23:08:00.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:00 smithi148 ceph-mon[35610]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-15T23:08:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:01 smithi062 ceph-mon[30949]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:08:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:01 smithi062 ceph-mon[30949]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-15T23:08:01.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:01 smithi062 ceph-mon[30949]: osdmap e6: 1 total, 0 up, 1 in 2024-09-15T23:08:01.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:01 smithi062 ceph-mon[30949]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:08:01.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:01 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:01.744 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:01 smithi067 ceph-mon[35665]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:08:01.745 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:01 smithi067 ceph-mon[35665]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-15T23:08:01.745 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:01 smithi067 ceph-mon[35665]: osdmap e6: 1 total, 0 up, 1 in 2024-09-15T23:08:01.745 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:01 smithi067 ceph-mon[35665]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:08:01.745 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:01 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:01.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:01 smithi148 ceph-mon[35610]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:08:01.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:01 smithi148 ceph-mon[35610]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-15T23:08:01.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:01 smithi148 ceph-mon[35610]: osdmap e6: 1 total, 0 up, 1 in 2024-09-15T23:08:01.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:01 smithi148 ceph-mon[35610]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:08:01.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:01 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:01.968 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:08:01.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_4 2024-09-15T23:08:02.230 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0[42523]: 2024-09-15T23:08:01.909+0000 7fcfebb83640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T23:08:02.230 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0[42523]: 2024-09-15T23:08:01.912+0000 7fcfe39a3640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:08:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:08:02.651 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:08:02.651 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: osdmap e7: 1 total, 0 up, 1 in 2024-09-15T23:08:02.651 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:02.651 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:02.651 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' 2024-09-15T23:08:02.651 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.652 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.652 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:08:02.652 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.652 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:02.652 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:02.652 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:08:02.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: osdmap e7: 1 total, 0 up, 1 in 2024-09-15T23:08:02.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:02.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:02.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' 2024-09-15T23:08:02.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:08:02.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:02.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:02.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 -- 172.21.15.67:0/341649505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab41037c0 msgr2=0x7faab410a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 --2- 172.21.15.67:0/341649505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab41037c0 0x7faab410a050 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faaac009ca0 tx=0x7faaac02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 -- 172.21.15.67:0/341649505 shutdown_connections 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 --2- 172.21.15.67:0/341649505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab41037c0 0x7faab410a050 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 --2- 172.21.15.67:0/341649505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4102e00 0x7faab4103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 --2- 172.21.15.67:0/341649505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faab4101c00 0x7faab4102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 -- 172.21.15.67:0/341649505 >> 172.21.15.67:0/341649505 conn(0x7faab40fd3b0 msgr2=0x7faab40ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 -- 172.21.15.67:0/341649505 shutdown_connections 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 -- 172.21.15.67:0/341649505 wait complete. 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.735+0000 7faab9ae9640 1 Processor -- start 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab9ae9640 1 -- start start 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab9ae9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4101c00 0x7faab4197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:02.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab9ae9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4102e00 0x7faab41983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab9ae9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faab41037c0 0x7faab419f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab9ae9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7faab410d8d0 con 0x7faab4102e00 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab9ae9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faab410d750 con 0x7faab4101c00 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab9ae9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7faab410da50 con 0x7faab41037c0 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4101c00 0x7faab4197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4101c00 0x7faab4197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58710/0 (socket says 172.21.15.67:58710) 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab37fe640 1 -- 172.21.15.67:0/395071563 learned_addr learned my addr 172.21.15.67:0/395071563 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab2ffd640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4102e00 0x7faab41983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab3fff640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faab41037c0 0x7faab419f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab37fe640 1 -- 172.21.15.67:0/395071563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faab41037c0 msgr2=0x7faab419f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab37fe640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faab41037c0 0x7faab419f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab37fe640 1 -- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4102e00 msgr2=0x7faab41983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:02.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab37fe640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4102e00 0x7faab41983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab37fe640 1 -- 172.21.15.67:0/395071563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab419fad0 con 0x7faab4101c00 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab3fff640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faab41037c0 0x7faab419f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.736+0000 7faab37fe640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4101c00 0x7faab4197ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faaa400e9c0 tx=0x7faaa400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.737+0000 7faab0ff9640 1 -- 172.21.15.67:0/395071563 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa400cd80 con 0x7faab4101c00 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.737+0000 7faab0ff9640 1 -- 172.21.15.67:0/395071563 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaa40035f0 con 0x7faab4101c00 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.737+0000 7faab0ff9640 1 -- 172.21.15.67:0/395071563 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4010690 con 0x7faab4101c00 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.737+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab419fdc0 con 0x7faab4101c00 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.737+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab41a0220 con 0x7faab4101c00 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.738+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab4108f30 con 0x7faab4101c00 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.739+0000 7faab0ff9640 1 -- 172.21.15.67:0/395071563 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7faaa40037a0 con 0x7faab4101c00 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.740+0000 7faab0ff9640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7faa9c0a31e0 0x7faa9c0a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.740+0000 7faab0ff9640 1 -- 172.21.15.67:0/395071563 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1487+0+0 (secure 0 0 0) 0x7faaa40ce1c0 con 0x7faab4101c00 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.740+0000 7faab2ffd640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7faa9c0a31e0 0x7faa9c0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:02.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.740+0000 7faab2ffd640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7faa9c0a31e0 0x7faa9c0a56a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faab41993c0 tx=0x7faaa8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:08:02.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.747+0000 7faab0ff9640 1 -- 172.21.15.67:0/395071563 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7faaa409ca70 con 0x7faab4101c00 2024-09-15T23:08:02.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:08:02.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: osdmap e7: 1 total, 0 up, 1 in 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506]' entity='osd.0' 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:02.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:02.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:02.931+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7faab4062630 con 0x7faa9c0a31e0 2024-09-15T23:08:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:08:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:08:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:08:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi062 2024-09-15T23:08:03.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi062 to 17088M 2024-09-15T23:08:03.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:03.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] boot 2024-09-15T23:08:03.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: osdmap e8: 1 total, 1 up, 1 in 2024-09-15T23:08:03.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:03.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:08:03.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:08:03.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:03.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:08:03.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi062 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi062 to 17088M 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] boot 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: osdmap e8: 1 total, 1 up, 1 in 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:08:03.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:03 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:08:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:08:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-15T23:08:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi062 2024-09-15T23:08:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi062 to 17088M 2024-09-15T23:08:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:03.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: osd.0 [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] boot 2024-09-15T23:08:03.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: osdmap e8: 1 total, 1 up, 1 in 2024-09-15T23:08:03.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-15T23:08:03.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:08:03.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:08:03.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:04.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:04 smithi062 ceph-mon[30949]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:08:04.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:04 smithi062 ceph-mon[30949]: osdmap e9: 1 total, 1 up, 1 in 2024-09-15T23:08:04.745 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:04 smithi067 ceph-mon[35665]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:08:04.745 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:04 smithi067 ceph-mon[35665]: osdmap e9: 1 total, 1 up, 1 in 2024-09-15T23:08:04.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:04 smithi148 ceph-mon[35610]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:08:04.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:04 smithi148 ceph-mon[35610]: osdmap e9: 1 total, 1 up, 1 in 2024-09-15T23:08:05.633 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:05 smithi067 ceph-mon[35665]: pgmap v40: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:05.633 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:05 smithi067 ceph-mon[35665]: from='client.? 172.21.15.67:0/3248663550' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebfe65d3-c394-48c5-8063-078760acf36c"}]: dispatch 2024-09-15T23:08:05.633 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:05 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebfe65d3-c394-48c5-8063-078760acf36c"}]: dispatch 2024-09-15T23:08:05.633 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:05 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebfe65d3-c394-48c5-8063-078760acf36c"}]': finished 2024-09-15T23:08:05.633 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:05 smithi067 ceph-mon[35665]: osdmap e10: 2 total, 1 up, 2 in 2024-09-15T23:08:05.633 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:05 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:05 smithi062 ceph-mon[30949]: pgmap v40: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:05 smithi062 ceph-mon[30949]: from='client.? 172.21.15.67:0/3248663550' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebfe65d3-c394-48c5-8063-078760acf36c"}]: dispatch 2024-09-15T23:08:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:05 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebfe65d3-c394-48c5-8063-078760acf36c"}]: dispatch 2024-09-15T23:08:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:05 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebfe65d3-c394-48c5-8063-078760acf36c"}]': finished 2024-09-15T23:08:05.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:05 smithi062 ceph-mon[30949]: osdmap e10: 2 total, 1 up, 2 in 2024-09-15T23:08:05.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:05 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:05.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:05 smithi148 ceph-mon[35610]: pgmap v40: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:05.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:05 smithi148 ceph-mon[35610]: from='client.? 172.21.15.67:0/3248663550' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebfe65d3-c394-48c5-8063-078760acf36c"}]: dispatch 2024-09-15T23:08:05.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:05 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebfe65d3-c394-48c5-8063-078760acf36c"}]: dispatch 2024-09-15T23:08:05.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:05 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebfe65d3-c394-48c5-8063-078760acf36c"}]': finished 2024-09-15T23:08:05.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:05 smithi148 ceph-mon[35610]: osdmap e10: 2 total, 1 up, 2 in 2024-09-15T23:08:05.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:05 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:06.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:06 smithi062 ceph-mon[30949]: from='client.? 172.21.15.67:0/2621029084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:08:06.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:06 smithi067 ceph-mon[35665]: from='client.? 172.21.15.67:0/2621029084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:08:06.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:06 smithi148 ceph-mon[35610]: from='client.? 172.21.15.67:0/2621029084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:08:07.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:07 smithi062 ceph-mon[30949]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:07.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:07 smithi067 ceph-mon[35665]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:07.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:07 smithi148 ceph-mon[35610]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:09.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:09 smithi062 ceph-mon[30949]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:09.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:09 smithi067 ceph-mon[35665]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:09.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:09 smithi148 ceph-mon[35610]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:11.618 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:11 smithi067 ceph-mon[35665]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:11 smithi062 ceph-mon[30949]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:11.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:11 smithi148 ceph-mon[35610]: pgmap v45: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:13 smithi067 ceph-mon[35665]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:13.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:13 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-15T23:08:13.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:13 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:13.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:13 smithi062 ceph-mon[30949]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:13.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-15T23:08:13.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:13.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:13 smithi148 ceph-mon[35610]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:13.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:13 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-15T23:08:13.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:13 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:14.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:14 smithi062 ceph-mon[30949]: Deploying daemon osd.1 on smithi067 2024-09-15T23:08:14.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:14 smithi067 ceph-mon[35665]: Deploying daemon osd.1 on smithi067 2024-09-15T23:08:14.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:14 smithi148 ceph-mon[35610]: Deploying daemon osd.1 on smithi067 2024-09-15T23:08:15.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:15 smithi062 ceph-mon[30949]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:15.748 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:15 smithi067 ceph-mon[35665]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:15.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:15 smithi148 ceph-mon[35610]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:17.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:17 smithi062 ceph-mon[30949]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:17 smithi067 ceph-mon[35665]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:17.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:17 smithi148 ceph-mon[35610]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:19.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:19 smithi067 ceph-mon[35665]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:19.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:19 smithi062 ceph-mon[30949]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:19.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:19 smithi148 ceph-mon[35610]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:20.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:20.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:20.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:20.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:20.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:20.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:20.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:20.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:20.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:21 smithi062 ceph-mon[30949]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:21.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:21 smithi067 ceph-mon[35665]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:21 smithi148 ceph-mon[35610]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.948+0000 7faab0ff9640 1 -- 172.21.15.67:0/395071563 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7faab4062630 con 0x7faa9c0a31e0 2024-09-15T23:08:22.950 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 1 on host 'smithi067' 2024-09-15T23:08:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7faa9c0a31e0 msgr2=0x7faa9c0a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7faa9c0a31e0 0x7faa9c0a56a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faab41993c0 tx=0x7faaa8005f90 comp rx=0 tx=0).stop 2024-09-15T23:08:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4101c00 msgr2=0x7faab4197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4101c00 0x7faab4197ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faaa400e9c0 tx=0x7faaa400ee80 comp rx=0 tx=0).stop 2024-09-15T23:08:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 shutdown_connections 2024-09-15T23:08:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7faa9c0a31e0 0x7faa9c0a56a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7faab41037c0 0x7faab419f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab4102e00 0x7faab41983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 --2- 172.21.15.67:0/395071563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faab4101c00 0x7faab4197ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 >> 172.21.15.67:0/395071563 conn(0x7faab40fd3b0 msgr2=0x7faab4106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:08:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 shutdown_connections 2024-09-15T23:08:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:08:22.951+0000 7faab9ae9640 1 -- 172.21.15.67:0/395071563 wait complete. 2024-09-15T23:08:23.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:23 smithi067 ceph-mon[35665]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:23.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:23.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:23.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:23.245 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.246 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:23 smithi148 ceph-mon[35610]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:23.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:23.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:23.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:23.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:23 smithi062 ceph-mon[30949]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:23.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:23.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:23.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:23.514 DEBUG:teuthology.orchestra.run.smithi067:osd.1> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.1.service 2024-09-15T23:08:23.516 INFO:tasks.cephadm:Deploying osd.2 on smithi148 with /dev/vg_nvme/lv_4... 2024-09-15T23:08:23.516 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-15T23:08:23.770 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:08:25.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:25 smithi067 ceph-mon[35665]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:25.275 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:25 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1[40138]: 2024-09-15T23:08:25.166+0000 7ff97af6f640 -1 osd.1 0 log_to_monitors true 2024-09-15T23:08:25.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:25 smithi148 ceph-mon[35610]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:25 smithi062 ceph-mon[30949]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:26.132 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:08:26.157 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:26 smithi148 ceph-mon[35610]: from='osd.1 [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:08:26.157 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:26 smithi148 ceph-mon[35610]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:08:26.159 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_4 2024-09-15T23:08:26.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:26 smithi067 ceph-mon[35665]: from='osd.1 [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:08:26.242 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:26 smithi067 ceph-mon[35665]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:08:26.407 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:08:26.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:26 smithi062 ceph-mon[30949]: from='osd.1 [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:08:26.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:26 smithi062 ceph-mon[30949]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-15T23:08:26.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.854+0000 7fb865deb640 1 -- 172.21.15.148:0/3099090522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb860101c00 msgr2=0x7fb860102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:26.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.854+0000 7fb865deb640 1 --2- 172.21.15.148:0/3099090522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb860101c00 0x7fb860102000 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb850009900 tx=0x7fb85002a410 comp rx=0 tx=0).stop 2024-09-15T23:08:26.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.854+0000 7fb865deb640 1 -- 172.21.15.148:0/3099090522 shutdown_connections 2024-09-15T23:08:26.882 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.854+0000 7fb865deb640 1 --2- 172.21.15.148:0/3099090522 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8601037c0 0x7fb86010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:26.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.854+0000 7fb865deb640 1 --2- 172.21.15.148:0/3099090522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb860102e00 0x7fb860103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:26.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.854+0000 7fb865deb640 1 --2- 172.21.15.148:0/3099090522 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb860101c00 0x7fb860102000 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:26.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.854+0000 7fb865deb640 1 -- 172.21.15.148:0/3099090522 >> 172.21.15.148:0/3099090522 conn(0x7fb8600fd390 msgr2=0x7fb8600ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:08:26.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 -- 172.21.15.148:0/3099090522 shutdown_connections 2024-09-15T23:08:26.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 -- 172.21.15.148:0/3099090522 wait complete. 2024-09-15T23:08:26.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 Processor -- start 2024-09-15T23:08:26.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 -- start start 2024-09-15T23:08:26.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb860101c00 0x7fb860197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:26.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb860102e00 0x7fb860198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:26.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8601037c0 0x7fb86019f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:26.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb86010d930 con 0x7fb860102e00 2024-09-15T23:08:26.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb86010d7b0 con 0x7fb860101c00 2024-09-15T23:08:26.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb865deb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb86010dab0 con 0x7fb8601037c0 2024-09-15T23:08:26.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.855+0000 7fb85ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8601037c0 0x7fb86019f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:26.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8601037c0 0x7fb86019f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:38338/0 (socket says 172.21.15.148:38338) 2024-09-15T23:08:26.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85ffff640 1 -- 172.21.15.148:0/3884602938 learned_addr learned my addr 172.21.15.148:0/3884602938 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:08:26.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85ffff640 1 -- 172.21.15.148:0/3884602938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb860101c00 msgr2=0x7fb860197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:26.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85ffff640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb860101c00 0x7fb860197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:26.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85ffff640 1 -- 172.21.15.148:0/3884602938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb860102e00 msgr2=0x7fb860198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:26.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85ffff640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb860102e00 0x7fb860198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:26.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85ffff640 1 -- 172.21.15.148:0/3884602938 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86019fb00 con 0x7fb8601037c0 2024-09-15T23:08:26.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85ffff640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8601037c0 0x7fb86019f490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb85800b970 tx=0x7fb85800be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:08:26.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85d7fa640 1 -- 172.21.15.148:0/3884602938 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb858019070 con 0x7fb8601037c0 2024-09-15T23:08:26.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb86019fdf0 con 0x7fb8601037c0 2024-09-15T23:08:26.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85d7fa640 1 -- 172.21.15.148:0/3884602938 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb85800cb40 con 0x7fb8601037c0 2024-09-15T23:08:26.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb85d7fa640 1 -- 172.21.15.148:0/3884602938 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb858007650 con 0x7fb8601037c0 2024-09-15T23:08:26.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.856+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8601a0330 con 0x7fb8601037c0 2024-09-15T23:08:26.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.857+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb860108f30 con 0x7fb8601037c0 2024-09-15T23:08:26.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.858+0000 7fb85d7fa640 1 -- 172.21.15.148:0/3884602938 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7fb85801c070 con 0x7fb8601037c0 2024-09-15T23:08:26.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.859+0000 7fb85d7fa640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb84c0a31e0 0x7fb84c0a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:26.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.859+0000 7fb85d7fa640 1 -- 172.21.15.148:0/3884602938 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 1903+0+0 (secure 0 0 0) 0x7fb8580cd980 con 0x7fb8601037c0 2024-09-15T23:08:26.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.860+0000 7fb85f7fe640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb84c0a31e0 0x7fb84c0a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:26.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.861+0000 7fb85f7fe640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb84c0a31e0 0x7fb84c0a56a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb8500057c0 tx=0x7fb850031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:08:26.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:26.862+0000 7fb85d7fa640 1 -- 172.21.15.148:0/3884602938 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb85809c090 con 0x7fb8601037c0 2024-09-15T23:08:27.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:27.036+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb860062630 con 0x7fb84c0a31e0 2024-09-15T23:08:27.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:27.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-15T23:08:27.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='osd.1 [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: osdmap e11: 2 total, 1 up, 2 in 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi067 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi067 to 17088M 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:27.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='osd.1 [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: osdmap e11: 2 total, 1 up, 2 in 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi067 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi067 to 17088M 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:27.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.524 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1[40138]: 2024-09-15T23:08:27.033+0000 7ff976ef2640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T23:08:27.524 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1[40138]: 2024-09-15T23:08:27.037+0000 7ff972d1a640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='osd.1 [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: osdmap e11: 2 total, 1 up, 2 in 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi067 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi067 to 17088M 2024-09-15T23:08:27.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:27.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:27.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:27.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:28.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:28 smithi148 ceph-mon[35610]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:08:28.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:28 smithi148 ceph-mon[35610]: osdmap e12: 2 total, 1 up, 2 in 2024-09-15T23:08:28.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:28.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:28.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:28 smithi148 ceph-mon[35610]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:08:28.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:08:28.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:08:28.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:28.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:28 smithi062 ceph-mon[30949]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:08:28.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:28 smithi062 ceph-mon[30949]: osdmap e12: 2 total, 1 up, 2 in 2024-09-15T23:08:28.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:28.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:28.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:28 smithi062 ceph-mon[30949]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:08:28.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:08:28.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:08:28.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:28.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:28 smithi067 ceph-mon[35665]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:08:28.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:28 smithi067 ceph-mon[35665]: osdmap e12: 2 total, 1 up, 2 in 2024-09-15T23:08:28.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:28.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:28.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:28 smithi067 ceph-mon[35665]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:08:28.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:08:28.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:08:28.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:29 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:08:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:29 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:08:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:29 smithi148 ceph-mon[35610]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:29.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:29.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:29 smithi148 ceph-mon[35610]: osd.1 [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] boot 2024-09-15T23:08:29.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:29 smithi148 ceph-mon[35610]: osdmap e13: 2 total, 2 up, 2 in 2024-09-15T23:08:29.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:29 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:08:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:29 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:08:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:29 smithi062 ceph-mon[30949]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:29 smithi062 ceph-mon[30949]: osd.1 [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] boot 2024-09-15T23:08:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:29 smithi062 ceph-mon[30949]: osdmap e13: 2 total, 2 up, 2 in 2024-09-15T23:08:29.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:29 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:08:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:29 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:08:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:29 smithi067 ceph-mon[35665]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-09-15T23:08:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:29.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:29 smithi067 ceph-mon[35665]: osd.1 [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] boot 2024-09-15T23:08:29.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:29 smithi067 ceph-mon[35665]: osdmap e13: 2 total, 2 up, 2 in 2024-09-15T23:08:29.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-15T23:08:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:30 smithi148 ceph-mon[35610]: osdmap e14: 2 total, 2 up, 2 in 2024-09-15T23:08:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:30 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/2970341183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51482605-4595-4d0b-b982-8613d1e73969"}]: dispatch 2024-09-15T23:08:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:30 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51482605-4595-4d0b-b982-8613d1e73969"}]: dispatch 2024-09-15T23:08:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:30 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51482605-4595-4d0b-b982-8613d1e73969"}]': finished 2024-09-15T23:08:30.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:30 smithi148 ceph-mon[35610]: osdmap e15: 3 total, 2 up, 3 in 2024-09-15T23:08:30.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:30.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:30 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/1392423898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:08:30.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:30 smithi062 ceph-mon[30949]: osdmap e14: 2 total, 2 up, 2 in 2024-09-15T23:08:30.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:30 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/2970341183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51482605-4595-4d0b-b982-8613d1e73969"}]: dispatch 2024-09-15T23:08:30.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:30 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51482605-4595-4d0b-b982-8613d1e73969"}]: dispatch 2024-09-15T23:08:30.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:30 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51482605-4595-4d0b-b982-8613d1e73969"}]': finished 2024-09-15T23:08:30.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:30 smithi062 ceph-mon[30949]: osdmap e15: 3 total, 2 up, 3 in 2024-09-15T23:08:30.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:30.482 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:30 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/1392423898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:08:30.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:30 smithi067 ceph-mon[35665]: osdmap e14: 2 total, 2 up, 2 in 2024-09-15T23:08:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:30 smithi067 ceph-mon[35665]: from='client.? 172.21.15.148:0/2970341183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51482605-4595-4d0b-b982-8613d1e73969"}]: dispatch 2024-09-15T23:08:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:30 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51482605-4595-4d0b-b982-8613d1e73969"}]: dispatch 2024-09-15T23:08:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:30 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51482605-4595-4d0b-b982-8613d1e73969"}]': finished 2024-09-15T23:08:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:30 smithi067 ceph-mon[35665]: osdmap e15: 3 total, 2 up, 3 in 2024-09-15T23:08:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:30.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:30 smithi067 ceph-mon[35665]: from='client.? 172.21.15.148:0/1392423898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:08:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:31 smithi067 ceph-mon[35665]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:31.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:31 smithi148 ceph-mon[35610]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:31 smithi062 ceph-mon[30949]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:33.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:33 smithi067 ceph-mon[35665]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:33 smithi148 ceph-mon[35610]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:33.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:33 smithi062 ceph-mon[30949]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:35.505 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:35 smithi148 ceph-mon[35610]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:35.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:35 smithi067 ceph-mon[35665]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:35 smithi062 ceph-mon[30949]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:37 smithi067 ceph-mon[35665]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:37 smithi148 ceph-mon[35610]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:37.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:37 smithi062 ceph-mon[30949]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-15T23:08:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:38 smithi148 ceph-mon[35610]: Deploying daemon osd.2 on smithi148 2024-09-15T23:08:38.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-15T23:08:38.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:38.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:38 smithi062 ceph-mon[30949]: Deploying daemon osd.2 on smithi148 2024-09-15T23:08:38.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-15T23:08:38.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:38.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:38 smithi067 ceph-mon[35665]: Deploying daemon osd.2 on smithi148 2024-09-15T23:08:39.456 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:39 smithi148 ceph-mon[35610]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:39.631 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:39 smithi062 ceph-mon[30949]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:39 smithi067 ceph-mon[35665]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:41 smithi148 ceph-mon[35610]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:41 smithi062 ceph-mon[30949]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:41 smithi067 ceph-mon[35665]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:43 smithi148 ceph-mon[35610]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:43 smithi062 ceph-mon[30949]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:43.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:43 smithi067 ceph-mon[35665]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:44.626 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:44.626 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:44.626 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:44.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:44.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:44.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:45.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:45 smithi148 ceph-mon[35610]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:45.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:45 smithi062 ceph-mon[30949]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:45 smithi067 ceph-mon[35665]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:47.505 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:47 smithi148 ceph-mon[35610]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:47.506 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:47.506 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:47.506 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:47.506 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:47.506 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:47 smithi062 ceph-mon[30949]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:47.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:47 smithi067 ceph-mon[35665]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:47.835 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.833+0000 7fb85d7fa640 1 -- 172.21.15.148:0/3884602938 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb860062630 con 0x7fb84c0a31e0 2024-09-15T23:08:47.835 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 2 on host 'smithi148' 2024-09-15T23:08:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb84c0a31e0 msgr2=0x7fb84c0a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb84c0a31e0 0x7fb84c0a56a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb8500057c0 tx=0x7fb850031400 comp rx=0 tx=0).stop 2024-09-15T23:08:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8601037c0 msgr2=0x7fb86019f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8601037c0 0x7fb86019f490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb85800b970 tx=0x7fb85800be30 comp rx=0 tx=0).stop 2024-09-15T23:08:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 shutdown_connections 2024-09-15T23:08:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb84c0a31e0 0x7fb84c0a56a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:47.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8601037c0 0x7fb86019f490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb860102e00 0x7fb860198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 --2- 172.21.15.148:0/3884602938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb860101c00 0x7fb860197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 >> 172.21.15.148:0/3884602938 conn(0x7fb8600fd390 msgr2=0x7fb860106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:08:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 shutdown_connections 2024-09-15T23:08:47.838 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:08:47.836+0000 7fb865deb640 1 -- 172.21.15.148:0/3884602938 wait complete. 2024-09-15T23:08:48.409 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:48.409 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:48.409 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:48.436 DEBUG:teuthology.orchestra.run.smithi148:osd.2> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.2.service 2024-09-15T23:08:48.438 INFO:tasks.cephadm:Deploying osd.3 on smithi062 with /dev/vg_nvme/lv_3... 2024-09-15T23:08:48.438 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-15T23:08:48.675 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:48.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:48.676 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:48.689 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:08:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:08:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:49.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:49 smithi062 ceph-mon[30949]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:49.623 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:49 smithi148 ceph-mon[35610]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:49 smithi067 ceph-mon[35665]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:50.177 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:49 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2[40560]: 2024-09-15T23:08:49.984+0000 7f845a7b1640 -1 osd.2 0 log_to_monitors true 2024-09-15T23:08:50.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:50 smithi062 ceph-mon[30949]: from='osd.2 [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:08:50.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:50 smithi062 ceph-mon[30949]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:08:50.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:50 smithi148 ceph-mon[35610]: from='osd.2 [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:08:50.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:50 smithi148 ceph-mon[35610]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:08:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:50 smithi067 ceph-mon[35665]: from='osd.2 [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:08:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:50 smithi067 ceph-mon[35665]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-15T23:08:51.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:51 smithi062 ceph-mon[30949]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:51.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:51 smithi062 ceph-mon[30949]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-15T23:08:51.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:51 smithi062 ceph-mon[30949]: osdmap e16: 3 total, 2 up, 3 in 2024-09-15T23:08:51.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:51.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:51 smithi062 ceph-mon[30949]: from='osd.2 [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:08:51.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:51 smithi062 ceph-mon[30949]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:08:51.534 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:08:51.561 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_3 2024-09-15T23:08:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:51 smithi067 ceph-mon[35665]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:51 smithi067 ceph-mon[35665]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-15T23:08:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:51 smithi067 ceph-mon[35665]: osdmap e16: 3 total, 2 up, 3 in 2024-09-15T23:08:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:51.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:51 smithi067 ceph-mon[35665]: from='osd.2 [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:08:51.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:51 smithi067 ceph-mon[35665]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:08:51.789 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2[40560]: 2024-09-15T23:08:51.321+0000 7f8456734640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T23:08:51.789 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2[40560]: 2024-09-15T23:08:51.325+0000 7f845255c640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:08:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:51 smithi148 ceph-mon[35610]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:51 smithi148 ceph-mon[35610]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-15T23:08:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:51 smithi148 ceph-mon[35610]: osdmap e16: 3 total, 2 up, 3 in 2024-09-15T23:08:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:51 smithi148 ceph-mon[35610]: from='osd.2 [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:08:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:51 smithi148 ceph-mon[35610]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:08:51.824 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:08:52.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 -- 172.21.15.62:0/1786510702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 msgr2=0x7f6d60108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:52.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 --2- 172.21.15.62:0/1786510702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 0x7f6d60108f20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009ca0 tx=0x7f6d5802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:08:52.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 -- 172.21.15.62:0/1786510702 shutdown_connections 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 --2- 172.21.15.62:0/1786510702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 0x7f6d60108f20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 --2- 172.21.15.62:0/1786510702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d60101cd0 0x7f6d60102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 --2- 172.21.15.62:0/1786510702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d60100ad0 0x7f6d60100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 -- 172.21.15.62:0/1786510702 >> 172.21.15.62:0/1786510702 conn(0x7f6d600fc280 msgr2=0x7f6d600fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 -- 172.21.15.62:0/1786510702 shutdown_connections 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 -- 172.21.15.62:0/1786510702 wait complete. 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.310+0000 7f6d6579b640 1 Processor -- start 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d6579b640 1 -- start start 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d6579b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d60100ad0 0x7f6d60104a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d6579b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d60101cd0 0x7f6d601030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d6579b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 0x7f6d601035f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:52.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d6579b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6010c750 con 0x7f6d60102690 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d6579b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6010c5d0 con 0x7f6d60101cd0 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d6579b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6010c8d0 con 0x7f6d60100ad0 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 0x7f6d601035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d60100ad0 0x7f6d60104a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d60100ad0 0x7f6d60104a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48414/0 (socket says 172.21.15.62:48414) 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 0x7f6d601035f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51956/0 (socket says 172.21.15.62:51956) 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5effd640 1 -- 172.21.15.62:0/2840624854 learned_addr learned my addr 172.21.15.62:0/2840624854 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5e7fc640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d60101cd0 0x7f6d601030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5effd640 1 -- 172.21.15.62:0/2840624854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d60101cd0 msgr2=0x7f6d601030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5effd640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d60101cd0 0x7f6d601030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5effd640 1 -- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 msgr2=0x7f6d601035f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5effd640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 0x7f6d601035f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:08:52.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5effd640 1 -- 172.21.15.62:0/2840624854 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d60103eb0 con 0x7f6d60100ad0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5f7fe640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 0x7f6d601035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.311+0000 7f6d5effd640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d60100ad0 0x7f6d60104a60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000d950 tx=0x7f6d5000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.312+0000 7f6d47fff640 1 -- 172.21.15.62:0/2840624854 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50017070 con 0x7f6d60100ad0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.312+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d601a1fc0 con 0x7f6d60100ad0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.312+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d601a2420 con 0x7f6d60100ad0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.312+0000 7f6d47fff640 1 -- 172.21.15.62:0/2840624854 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d50009c30 con 0x7f6d60100ad0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.312+0000 7f6d47fff640 1 -- 172.21.15.62:0/2840624854 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50021060 con 0x7f6d60100ad0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.313+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d601a26a0 con 0x7f6d60100ad0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.314+0000 7f6d47fff640 1 -- 172.21.15.62:0/2840624854 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 15) ==== 138299+0+0 (secure 0 0 0) 0x7f6d5001a070 con 0x7f6d60100ad0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.315+0000 7f6d47fff640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6d480a31b0 0x7f6d480a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.315+0000 7f6d5e7fc640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6d480a31b0 0x7f6d480a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:08:52.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.315+0000 7f6d47fff640 1 -- 172.21.15.62:0/2840624854 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 2571+0+0 (secure 0 0 0) 0x7f6d500cdbb0 con 0x7f6d60100ad0 2024-09-15T23:08:52.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.315+0000 7f6d5e7fc640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6d480a31b0 0x7f6d480a5670 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6d601047e0 tx=0x7f6d5400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:08:52.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.318+0000 7f6d47fff640 1 -- 172.21.15.62:0/2840624854 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6d5009b930 con 0x7f6d60100ad0 2024-09-15T23:08:52.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:08:52.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: osdmap e17: 3 total, 2 up, 3 in 2024-09-15T23:08:52.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi148 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi148 to 17088M 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:52.506+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6d6006a6b0 con 0x7f6d480a31b0 2024-09-15T23:08:52.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:08:52.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: osdmap e17: 3 total, 2 up, 3 in 2024-09-15T23:08:52.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi148 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi148 to 17088M 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:52.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: osdmap e17: 3 total, 2 up, 3 in 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi148 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi148 to 17088M 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:52.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:08:52.791 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:08:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:08:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: osd.2 [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] boot 2024-09-15T23:08:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: osdmap e18: 3 total, 3 up, 3 in 2024-09-15T23:08:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:08:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:08:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:08:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:53.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:08:53.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:08:53.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:53.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: osd.2 [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] boot 2024-09-15T23:08:53.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: osdmap e18: 3 total, 3 up, 3 in 2024-09-15T23:08:53.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:53.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:08:53.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:08:53.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:08:53.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:08:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:08:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-09-15T23:08:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: osd.2 [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] boot 2024-09-15T23:08:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: osdmap e18: 3 total, 3 up, 3 in 2024-09-15T23:08:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-15T23:08:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:08:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:08:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:08:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:08:54.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:54 smithi062 ceph-mon[30949]: osdmap e19: 3 total, 3 up, 3 in 2024-09-15T23:08:54.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:54 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:08:54.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:54 smithi067 ceph-mon[35665]: osdmap e19: 3 total, 3 up, 3 in 2024-09-15T23:08:54.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:54 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:08:54.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:54 smithi148 ceph-mon[35610]: osdmap e19: 3 total, 3 up, 3 in 2024-09-15T23:08:54.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:54 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:08:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:55 smithi062 ceph-mon[30949]: pgmap v76: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:08:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-15T23:08:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:55 smithi062 ceph-mon[30949]: osdmap e20: 3 total, 3 up, 3 in 2024-09-15T23:08:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:08:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:55 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1638988855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cbd02ce-8ea0-479b-812d-5540bfd4fbd0"}]: dispatch 2024-09-15T23:08:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:08:55.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:55 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1638988855' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cbd02ce-8ea0-479b-812d-5540bfd4fbd0"}]': finished 2024-09-15T23:08:55.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:55 smithi062 ceph-mon[30949]: osdmap e21: 4 total, 3 up, 4 in 2024-09-15T23:08:55.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:08:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:55 smithi067 ceph-mon[35665]: pgmap v76: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:08:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-15T23:08:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:55 smithi067 ceph-mon[35665]: osdmap e20: 3 total, 3 up, 3 in 2024-09-15T23:08:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:08:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:55 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1638988855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cbd02ce-8ea0-479b-812d-5540bfd4fbd0"}]: dispatch 2024-09-15T23:08:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:08:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:55 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1638988855' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cbd02ce-8ea0-479b-812d-5540bfd4fbd0"}]': finished 2024-09-15T23:08:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:55 smithi067 ceph-mon[35665]: osdmap e21: 4 total, 3 up, 4 in 2024-09-15T23:08:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:08:55.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:55 smithi148 ceph-mon[35610]: pgmap v76: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:08:55.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-15T23:08:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:55 smithi148 ceph-mon[35610]: osdmap e20: 3 total, 3 up, 3 in 2024-09-15T23:08:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:08:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:55 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1638988855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cbd02ce-8ea0-479b-812d-5540bfd4fbd0"}]: dispatch 2024-09-15T23:08:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:08:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:55 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1638988855' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cbd02ce-8ea0-479b-812d-5540bfd4fbd0"}]': finished 2024-09-15T23:08:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:55 smithi148 ceph-mon[35610]: osdmap e21: 4 total, 3 up, 4 in 2024-09-15T23:08:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:08:55.945 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:55 smithi062 sudo[47394]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:08:55.945 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:55 smithi062 sudo[47394]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:55.946 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:55 smithi062 sudo[47394]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:55.946 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:55 smithi062 sudo[47394]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:56.230 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:55 smithi062 sudo[47443]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:08:56.230 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:55 smithi062 sudo[47443]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:56.230 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:55 smithi062 sudo[47443]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:56.230 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:08:55 smithi062 sudo[47443]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:56.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:08:56.449+0000 7f6d47fff640 1 -- 172.21.15.62:0/2840624854 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6d50097fa0 con 0x7f6d60100ad0 2024-09-15T23:08:56.501 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:56 smithi067 sudo[44269]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:08:56.501 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:56 smithi067 sudo[44269]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:56.501 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:56 smithi067 sudo[44269]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:56.501 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:56 smithi067 sudo[44269]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:56.501 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:56 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2236378135' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:08:56.501 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:56 smithi067 ceph-mon[35665]: osdmap e22: 4 total, 3 up, 4 in 2024-09-15T23:08:56.501 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:08:56.502 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:56.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:56 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2236378135' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:08:56.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:56 smithi062 ceph-mon[30949]: osdmap e22: 4 total, 3 up, 4 in 2024-09-15T23:08:56.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:08:56.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:56.771 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:56 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2236378135' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:08:56.771 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:56 smithi148 ceph-mon[35610]: osdmap e22: 4 total, 3 up, 4 in 2024-09-15T23:08:56.771 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:08:56.772 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:56.774 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:56 smithi067 sudo[44272]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:08:56.774 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:56 smithi067 sudo[44272]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:56.774 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:56 smithi067 sudo[44272]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:56.775 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:08:56 smithi067 sudo[44272]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:57.026 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:56 smithi148 sudo[44602]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:08:57.027 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:56 smithi148 sudo[44602]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:57.027 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:56 smithi148 sudo[44602]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:57.027 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:56 smithi148 sudo[44602]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:57.289 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:57 smithi148 sudo[44605]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:08:57.289 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:57 smithi148 sudo[44605]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:57.289 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:57 smithi148 sudo[44605]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:57.290 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:08:57 smithi148 sudo[44605]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:57.945 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 sudo[47819]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:08:57.945 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 sudo[47819]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:57.946 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 sudo[47819]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:57.946 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 sudo[47819]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:58.212 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:57 smithi148 ceph-mon[35610]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:08:58.212 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:57 smithi148 ceph-mon[35610]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-15T23:08:58.212 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:58.212 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:58.212 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:57 smithi148 ceph-mon[35610]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:08:58.212 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 sudo[44608]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:08:58.213 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 sudo[44608]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:58.213 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 sudo[44608]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:58.213 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 sudo[44608]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:58.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 sudo[47824]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:08:58.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 sudo[47824]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:58.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 sudo[47824]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:58.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 sudo[47824]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:58.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 ceph-mon[30949]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:08:58.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 ceph-mon[30949]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-15T23:08:58.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:58.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:58.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:57 smithi062 ceph-mon[30949]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:08:58.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:57 smithi067 ceph-mon[35665]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:08:58.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:57 smithi067 ceph-mon[35665]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-09-15T23:08:58.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:58.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:08:58.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:57 smithi067 ceph-mon[35665]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:08:58.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 sudo[44611]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:08:58.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 sudo[44611]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:58.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 sudo[44611]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:58.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 sudo[44611]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:58.977 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 sudo[44275]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-15T23:08:58.977 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 sudo[44275]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:58.977 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 sudo[44275]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:58.977 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 sudo[44275]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 sudo[44278]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-15T23:08:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 sudo[44278]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-15T23:08:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 sudo[44278]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 sudo[44278]: pam_unix(sudo:session): session closed for user root 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:08:59.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:08:59.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:08:59.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:08:59.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:08:59.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:08:59.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:08:59.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:08:59.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:08:59.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:08:59.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:08:59.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:08:59.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:08:59.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:08:59.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:08:59.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:08:59.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:08:59.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:08:59.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-15T23:08:59.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-15T23:08:59.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-15T23:09:00.128 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:59 smithi062 ceph-mon[30949]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:09:00.128 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:08:59 smithi062 ceph-mon[30949]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:09:00.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:59 smithi067 ceph-mon[35665]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:09:00.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:08:59 smithi067 ceph-mon[35665]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:09:00.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:59 smithi148 ceph-mon[35610]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-15T23:09:00.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:08:59 smithi148 ceph-mon[35610]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-15T23:09:01.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:00 smithi062 ceph-mon[30949]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:01.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:00 smithi067 ceph-mon[35665]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:00 smithi148 ceph-mon[35610]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:03.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:02 smithi062 ceph-mon[30949]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:02 smithi067 ceph-mon[35665]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:02 smithi148 ceph-mon[35610]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:04.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-15T23:09:04.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:04 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-15T23:09:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:04 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:04.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-15T23:09:04.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:05.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:05 smithi067 ceph-mon[35665]: Deploying daemon osd.3 on smithi062 2024-09-15T23:09:05.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:05 smithi067 ceph-mon[35665]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:05.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:05 smithi148 ceph-mon[35610]: Deploying daemon osd.3 on smithi062 2024-09-15T23:09:05.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:05 smithi148 ceph-mon[35610]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:05.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:05 smithi062 ceph-mon[30949]: Deploying daemon osd.3 on smithi062 2024-09-15T23:09:05.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:05 smithi062 ceph-mon[30949]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:07.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:07 smithi067 ceph-mon[35665]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:07.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:07 smithi148 ceph-mon[35610]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:07.311 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:07 smithi062 ceph-mon[30949]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:09.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:09 smithi067 ceph-mon[35665]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:09.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:09 smithi148 ceph-mon[35610]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:09.357 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:09 smithi062 ceph-mon[30949]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:10.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:10.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:10.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:10.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:10.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:10.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:11.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:11 smithi067 ceph-mon[35665]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:11.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:11 smithi148 ceph-mon[35610]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:11.340 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:11 smithi062 ceph-mon[30949]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:13.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:13 smithi067 ceph-mon[35665]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:13.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:13 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:13.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:13 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:13.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:13 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:13.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:13 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:13.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:13 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:13.299 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:13 smithi148 ceph-mon[35610]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:13.300 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:13 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:13.300 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:13 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:13.300 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:13 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:13.300 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:13 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:13.300 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:13 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:13.351 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:13 smithi062 ceph-mon[30949]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:13.351 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:13.352 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:13.352 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:13.352 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:13.352 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:14.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.034+0000 7f6d47fff640 1 -- 172.21.15.62:0/2840624854 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6d6006a6b0 con 0x7f6d480a31b0 2024-09-15T23:09:14.035 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 3 on host 'smithi062' 2024-09-15T23:09:14.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6d480a31b0 msgr2=0x7f6d480a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:14.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6d480a31b0 0x7f6d480a5670 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6d601047e0 tx=0x7f6d5400b390 comp rx=0 tx=0).stop 2024-09-15T23:09:14.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d60100ad0 msgr2=0x7f6d60104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:14.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d60100ad0 0x7f6d60104a60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000d950 tx=0x7f6d5000de10 comp rx=0 tx=0).stop 2024-09-15T23:09:14.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 shutdown_connections 2024-09-15T23:09:14.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6d480a31b0 0x7f6d480a5670 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:14.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d60102690 0x7f6d601035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:14.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d60101cd0 0x7f6d601030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:14.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 --2- 172.21.15.62:0/2840624854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6d60100ad0 0x7f6d60104a60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:14.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 >> 172.21.15.62:0/2840624854 conn(0x7f6d600fc280 msgr2=0x7f6d60069fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:09:14.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 shutdown_connections 2024-09-15T23:09:14.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:09:14.036+0000 7f6d6579b640 1 -- 172.21.15.62:0/2840624854 wait complete. 2024-09-15T23:09:14.644 DEBUG:teuthology.orchestra.run.smithi062:osd.3> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.3.service 2024-09-15T23:09:14.646 INFO:tasks.cephadm:Deploying osd.4 on smithi067 with /dev/vg_nvme/lv_3... 2024-09-15T23:09:14.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-15T23:09:14.908 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:09:15.216 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:15 smithi062 ceph-mon[30949]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:15.216 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:15 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:15.216 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:15 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:15.217 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:15 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:15.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:15 smithi067 ceph-mon[35665]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:15.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:15 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:15.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:15 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:15.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:15 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:15 smithi148 ceph-mon[35610]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:15 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:15 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:15 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:16.043 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:16 smithi062 ceph-mon[30949]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-15T23:09:16.043 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:09:15 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3[49391]: 2024-09-15T23:09:15.780+0000 7f4aff7c2640 -1 osd.3 0 log_to_monitors true 2024-09-15T23:09:16.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:16 smithi148 ceph-mon[35610]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-15T23:09:16.317 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:16 smithi067 ceph-mon[35665]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-15T23:09:17.230 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:09:17 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3[49391]: 2024-09-15T23:09:17.030+0000 7f4afb745640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T23:09:17.230 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:09:17 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3[49391]: 2024-09-15T23:09:17.036+0000 7f4af756d640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:09:17.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:17 smithi062 ceph-mon[30949]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:17.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:17 smithi062 ceph-mon[30949]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-15T23:09:17.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:17 smithi062 ceph-mon[30949]: osdmap e23: 4 total, 3 up, 4 in 2024-09-15T23:09:17.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:17 smithi062 ceph-mon[30949]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:09:17.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:17 smithi067 ceph-mon[35665]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:17 smithi067 ceph-mon[35665]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-15T23:09:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:17 smithi067 ceph-mon[35665]: osdmap e23: 4 total, 3 up, 4 in 2024-09-15T23:09:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:17 smithi067 ceph-mon[35665]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:09:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:17.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:17 smithi148 ceph-mon[35610]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:17.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:17 smithi148 ceph-mon[35610]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-15T23:09:17.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:17 smithi148 ceph-mon[35610]: osdmap e23: 4 total, 3 up, 4 in 2024-09-15T23:09:17.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:17 smithi148 ceph-mon[35610]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:09:17.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:17.727 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:09:17.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_3 2024-09-15T23:09:18.002 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:09:18.281 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:09:18.281 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: osdmap e24: 4 total, 3 up, 4 in 2024-09-15T23:09:18.281 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:18.281 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:18.282 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.282 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.282 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:18.282 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:18.282 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.282 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:18.282 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:18.282 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:18 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:09:18.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: osdmap e24: 4 total, 3 up, 4 in 2024-09-15T23:09:18.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:18.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:18.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:18 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:09:18.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: osdmap e24: 4 total, 3 up, 4 in 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:18 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.618+0000 7f3076fb9640 1 -- 172.21.15.67:0/3424511074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3070102e00 msgr2=0x7f3070103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.618+0000 7f3076fb9640 1 --2- 172.21.15.67:0/3424511074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3070102e00 0x7f3070103280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f305c009900 tx=0x7f305c02a410 comp rx=0 tx=0).stop 2024-09-15T23:09:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.618+0000 7f3076fb9640 1 -- 172.21.15.67:0/3424511074 shutdown_connections 2024-09-15T23:09:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.618+0000 7f3076fb9640 1 --2- 172.21.15.67:0/3424511074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30701037c0 0x7f307010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.618+0000 7f3076fb9640 1 --2- 172.21.15.67:0/3424511074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3070102e00 0x7f3070103280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.618+0000 7f3076fb9640 1 --2- 172.21.15.67:0/3424511074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3070101c00 0x7f3070102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.618+0000 7f3076fb9640 1 -- 172.21.15.67:0/3424511074 >> 172.21.15.67:0/3424511074 conn(0x7f30700fd390 msgr2=0x7f30700ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:09:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.618+0000 7f3076fb9640 1 -- 172.21.15.67:0/3424511074 shutdown_connections 2024-09-15T23:09:18.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.618+0000 7f3076fb9640 1 -- 172.21.15.67:0/3424511074 wait complete. 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3076fb9640 1 Processor -- start 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3076fb9640 1 -- start start 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3076fb9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3070101c00 0x7f3070197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3076fb9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3070102e00 0x7f3070198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3076fb9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30701037c0 0x7f307019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3076fb9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f307010d8b0 con 0x7f3070102e00 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3076fb9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f307010d730 con 0x7f30701037c0 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3076fb9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f307010da30 con 0x7f3070101c00 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3075556640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30701037c0 0x7f307019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3075556640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30701037c0 0x7f307019f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54582/0 (socket says 172.21.15.67:54582) 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.619+0000 7f3075556640 1 -- 172.21.15.67:0/2353963003 learned_addr learned my addr 172.21.15.67:0/2353963003 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3074d55640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3070101c00 0x7f3070197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:09:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f306ffff640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3070102e00 0x7f3070198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3075556640 1 -- 172.21.15.67:0/2353963003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3070101c00 msgr2=0x7f3070197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3075556640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3070101c00 0x7f3070197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3075556640 1 -- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3070102e00 msgr2=0x7f3070198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3075556640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3070102e00 0x7f3070198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3075556640 1 -- 172.21.15.67:0/2353963003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307019fb30 con 0x7f30701037c0 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3074d55640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3070101c00 0x7f3070197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3075556640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30701037c0 0x7f307019f4c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f306800eb10 tx=0x7f306800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f306dffb640 1 -- 172.21.15.67:0/2353963003 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068004b20 con 0x7f30701037c0 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f306dffb640 1 -- 172.21.15.67:0/2353963003 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3068004cc0 con 0x7f30701037c0 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307019fe20 con 0x7f30701037c0 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f306dffb640 1 -- 172.21.15.67:0/2353963003 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068010760 con 0x7f30701037c0 2024-09-15T23:09:18.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.620+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30701a02d0 con 0x7f30701037c0 2024-09-15T23:09:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.621+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3070108fb0 con 0x7f30701037c0 2024-09-15T23:09:18.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.623+0000 7f306dffb640 1 -- 172.21.15.67:0/2353963003 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3068010900 con 0x7f30701037c0 2024-09-15T23:09:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.623+0000 7f306dffb640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30600a32f0 0x7f30600a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:09:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.624+0000 7f3074d55640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30600a32f0 0x7f30600a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:09:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.624+0000 7f306dffb640 1 -- 172.21.15.67:0/2353963003 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 3698+0+0 (secure 0 0 0) 0x7f30680cef20 con 0x7f30701037c0 2024-09-15T23:09:18.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.624+0000 7f3074d55640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30600a32f0 0x7f30600a57b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f30640058c0 tx=0x7f3064005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:09:18.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.627+0000 7f306dffb640 1 -- 172.21.15.67:0/2353963003 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f30680541b0 con 0x7f30701037c0 2024-09-15T23:09:18.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:18.811+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3070062630 con 0x7f30600a32f0 2024-09-15T23:09:19.225 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:09:19.225 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:09:19.225 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:19.225 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi062 2024-09-15T23:09:19.225 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi062 to 8544M 2024-09-15T23:09:19.226 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] boot 2024-09-15T23:09:19.226 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: osdmap e25: 4 total, 4 up, 4 in 2024-09-15T23:09:19.226 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:19.226 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: osdmap e26: 4 total, 4 up, 4 in 2024-09-15T23:09:19.226 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:09:19.226 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:09:19.226 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:19 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:19.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:09:19.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:09:19.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi062 2024-09-15T23:09:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi062 to 8544M 2024-09-15T23:09:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] boot 2024-09-15T23:09:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: osdmap e25: 4 total, 4 up, 4 in 2024-09-15T23:09:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: osdmap e26: 4 total, 4 up, 4 in 2024-09-15T23:09:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:09:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:09:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:19 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:19.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:09:19.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi062 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi062 to 8544M 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: osd.3 [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] boot 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: osdmap e25: 4 total, 4 up, 4 in 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: osdmap e26: 4 total, 4 up, 4 in 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:09:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:19 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:20.464 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:20 smithi148 ceph-mon[35610]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:09:20.464 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:20 smithi148 ceph-mon[35610]: osdmap e27: 4 total, 4 up, 4 in 2024-09-15T23:09:20.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:20 smithi062 ceph-mon[30949]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:09:20.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:20 smithi062 ceph-mon[30949]: osdmap e27: 4 total, 4 up, 4 in 2024-09-15T23:09:20.745 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:20 smithi067 ceph-mon[35665]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:09:20.745 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:20 smithi067 ceph-mon[35665]: osdmap e27: 4 total, 4 up, 4 in 2024-09-15T23:09:21.678 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:21 smithi067 ceph-mon[35665]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:21.678 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:21 smithi067 ceph-mon[35665]: osdmap e28: 4 total, 4 up, 4 in 2024-09-15T23:09:21.678 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:21 smithi067 ceph-mon[35665]: from='client.? 172.21.15.67:0/1024918521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6441578-9a31-46f2-a84e-0bfa41cd1c49"}]: dispatch 2024-09-15T23:09:21.678 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:21 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6441578-9a31-46f2-a84e-0bfa41cd1c49"}]: dispatch 2024-09-15T23:09:21.678 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:21 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6441578-9a31-46f2-a84e-0bfa41cd1c49"}]': finished 2024-09-15T23:09:21.678 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:21 smithi067 ceph-mon[35665]: osdmap e29: 5 total, 4 up, 5 in 2024-09-15T23:09:21.678 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:21 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:21 smithi062 ceph-mon[30949]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:21.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:21 smithi062 ceph-mon[30949]: osdmap e28: 4 total, 4 up, 4 in 2024-09-15T23:09:21.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:21 smithi062 ceph-mon[30949]: from='client.? 172.21.15.67:0/1024918521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6441578-9a31-46f2-a84e-0bfa41cd1c49"}]: dispatch 2024-09-15T23:09:21.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:21 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6441578-9a31-46f2-a84e-0bfa41cd1c49"}]: dispatch 2024-09-15T23:09:21.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:21 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6441578-9a31-46f2-a84e-0bfa41cd1c49"}]': finished 2024-09-15T23:09:21.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:21 smithi062 ceph-mon[30949]: osdmap e29: 5 total, 4 up, 5 in 2024-09-15T23:09:21.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:21 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:21 smithi148 ceph-mon[35610]: pgmap v97: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:21 smithi148 ceph-mon[35610]: osdmap e28: 4 total, 4 up, 4 in 2024-09-15T23:09:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:21 smithi148 ceph-mon[35610]: from='client.? 172.21.15.67:0/1024918521' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6441578-9a31-46f2-a84e-0bfa41cd1c49"}]: dispatch 2024-09-15T23:09:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:21 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6441578-9a31-46f2-a84e-0bfa41cd1c49"}]: dispatch 2024-09-15T23:09:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:21 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6441578-9a31-46f2-a84e-0bfa41cd1c49"}]': finished 2024-09-15T23:09:21.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:21 smithi148 ceph-mon[35610]: osdmap e29: 5 total, 4 up, 5 in 2024-09-15T23:09:21.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:21 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:22.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:22 smithi062 ceph-mon[30949]: from='client.? 172.21.15.67:0/2535478677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:09:22.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:22 smithi067 ceph-mon[35665]: from='client.? 172.21.15.67:0/2535478677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:09:22.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:22 smithi148 ceph-mon[35610]: from='client.? 172.21.15.67:0/2535478677' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:09:23.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:23 smithi062 ceph-mon[30949]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:23.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:23 smithi067 ceph-mon[35665]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:23 smithi148 ceph-mon[35610]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:25.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:25 smithi062 ceph-mon[30949]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:25.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:25 smithi067 ceph-mon[35665]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:25.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:25 smithi148 ceph-mon[35610]: pgmap v101: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:27 smithi067 ceph-mon[35665]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-15T23:09:27.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:27 smithi062 ceph-mon[30949]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-15T23:09:27.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:27 smithi148 ceph-mon[35610]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-15T23:09:29.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:29 smithi062 ceph-mon[30949]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:09:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:29 smithi067 ceph-mon[35665]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:09:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:29 smithi148 ceph-mon[35610]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:09:30.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-15T23:09:30.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:30.750 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-15T23:09:30.750 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-15T23:09:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:31.677 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:31 smithi067 ceph-mon[35665]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-09-15T23:09:31.677 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:31 smithi067 ceph-mon[35665]: Deploying daemon osd.4 on smithi067 2024-09-15T23:09:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:31 smithi062 ceph-mon[30949]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-09-15T23:09:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:31 smithi062 ceph-mon[30949]: Deploying daemon osd.4 on smithi067 2024-09-15T23:09:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:31 smithi148 ceph-mon[35610]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-09-15T23:09:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:31 smithi148 ceph-mon[35610]: Deploying daemon osd.4 on smithi067 2024-09-15T23:09:33.666 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:33 smithi067 ceph-mon[35665]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-15T23:09:33.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:33 smithi062 ceph-mon[30949]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-15T23:09:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:33 smithi148 ceph-mon[35610]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-15T23:09:35.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:35 smithi067 ceph-mon[35665]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:09:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:35 smithi062 ceph-mon[30949]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:09:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:35 smithi148 ceph-mon[35610]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:09:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:36.736 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:36.736 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:36.736 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:37.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:37 smithi062 ceph-mon[30949]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:09:37.747 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:37 smithi067 ceph-mon[35665]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:09:37.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:37 smithi148 ceph-mon[35610]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:09:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:39 smithi062 ceph-mon[30949]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:39.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:39 smithi067 ceph-mon[35665]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:39.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:39.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:39.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:39 smithi148 ceph-mon[35610]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:39.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:39.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:39.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:39.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:39.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:40.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.378+0000 7f306dffb640 1 -- 172.21.15.67:0/2353963003 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3070062630 con 0x7f30600a32f0 2024-09-15T23:09:40.379 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 4 on host 'smithi067' 2024-09-15T23:09:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30600a32f0 msgr2=0x7f30600a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30600a32f0 0x7f30600a57b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f30640058c0 tx=0x7f3064005850 comp rx=0 tx=0).stop 2024-09-15T23:09:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30701037c0 msgr2=0x7f307019f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:40.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30701037c0 0x7f307019f4c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f306800eb10 tx=0x7f306800c2d0 comp rx=0 tx=0).stop 2024-09-15T23:09:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 shutdown_connections 2024-09-15T23:09:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30600a32f0 0x7f30600a57b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30701037c0 0x7f307019f4c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3070102e00 0x7f3070198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 --2- 172.21.15.67:0/2353963003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3070101c00 0x7f3070197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 >> 172.21.15.67:0/2353963003 conn(0x7f30700fd390 msgr2=0x7f3070106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:09:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 shutdown_connections 2024-09-15T23:09:40.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:09:40.380+0000 7f3076fb9640 1 -- 172.21.15.67:0/2353963003 wait complete. 2024-09-15T23:09:40.709 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:40.709 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:40.710 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:40.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:09:40.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:40.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:41.018 DEBUG:teuthology.orchestra.run.smithi067:osd.4> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.4.service 2024-09-15T23:09:41.020 INFO:tasks.cephadm:Deploying osd.5 on smithi148 with /dev/vg_nvme/lv_3... 2024-09-15T23:09:41.020 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-15T23:09:41.275 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:09:41.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:41 smithi067 ceph-mon[35665]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:41.687 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:41 smithi148 ceph-mon[35610]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:41 smithi062 ceph-mon[30949]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:42.393 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:09:42 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4[47037]: 2024-09-15T23:09:42.258+0000 7f78f1899640 -1 osd.4 0 log_to_monitors true 2024-09-15T23:09:42.677 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:42 smithi148 ceph-mon[35610]: from='osd.4 [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:09:42.677 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:42 smithi148 ceph-mon[35610]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:09:42.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:42 smithi062 ceph-mon[30949]: from='osd.4 [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:09:42.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:42 smithi062 ceph-mon[30949]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:09:42.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:42 smithi067 ceph-mon[35665]: from='osd.4 [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:09:42.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:42 smithi067 ceph-mon[35665]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-15T23:09:43.745 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:43 smithi148 ceph-mon[35610]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:43.745 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:43 smithi148 ceph-mon[35610]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-15T23:09:43.745 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:43 smithi148 ceph-mon[35610]: from='osd.4 [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:09:43.745 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:43 smithi148 ceph-mon[35610]: osdmap e30: 5 total, 4 up, 5 in 2024-09-15T23:09:43.746 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:43.746 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:43 smithi148 ceph-mon[35610]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:09:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:43 smithi067 ceph-mon[35665]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:43 smithi067 ceph-mon[35665]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-15T23:09:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:43 smithi067 ceph-mon[35665]: from='osd.4 [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:09:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:43 smithi067 ceph-mon[35665]: osdmap e30: 5 total, 4 up, 5 in 2024-09-15T23:09:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:43.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:43 smithi067 ceph-mon[35665]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:09:43.929 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:09:43.956 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_3 2024-09-15T23:09:43.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:43 smithi062 ceph-mon[30949]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:43.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:43 smithi062 ceph-mon[30949]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-15T23:09:43.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:43 smithi062 ceph-mon[30949]: from='osd.4 [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:09:43.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:43 smithi062 ceph-mon[30949]: osdmap e30: 5 total, 4 up, 5 in 2024-09-15T23:09:43.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:43.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:43 smithi062 ceph-mon[30949]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:09:44.218 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:09:44.274 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:09:43 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4[47037]: 2024-09-15T23:09:43.903+0000 7f78ed81c640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T23:09:44.274 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:09:43 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4[47037]: 2024-09-15T23:09:43.908+0000 7f78e563c640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:09:44.662 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: osdmap e31: 5 total, 4 up, 5 in 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.663 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:44 smithi148 ceph-mon[35610]: from='osd.4 ' entity='osd.4' 2024-09-15T23:09:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:09:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: osdmap e31: 5 total, 4 up, 5 in 2024-09-15T23:09:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:44.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:44.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:44.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:44.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:44 smithi067 ceph-mon[35665]: from='osd.4 ' entity='osd.4' 2024-09-15T23:09:44.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.745+0000 7fba31f2a640 1 -- 172.21.15.148:0/1365574153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c102e00 msgr2=0x7fba2c103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:44.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.745+0000 7fba31f2a640 1 --2- 172.21.15.148:0/1365574153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c102e00 0x7fba2c103280 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fba20009900 tx=0x7fba2002a410 comp rx=0 tx=0).stop 2024-09-15T23:09:44.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.745+0000 7fba31f2a640 1 -- 172.21.15.148:0/1365574153 shutdown_connections 2024-09-15T23:09:44.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.745+0000 7fba31f2a640 1 --2- 172.21.15.148:0/1365574153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c1037c0 0x7fba2c10a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:44.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.745+0000 7fba31f2a640 1 --2- 172.21.15.148:0/1365574153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c102e00 0x7fba2c103280 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:44.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.745+0000 7fba31f2a640 1 --2- 172.21.15.148:0/1365574153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba2c101c00 0x7fba2c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:44.778 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.745+0000 7fba31f2a640 1 -- 172.21.15.148:0/1365574153 >> 172.21.15.148:0/1365574153 conn(0x7fba2c0fd390 msgr2=0x7fba2c0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.745+0000 7fba31f2a640 1 -- 172.21.15.148:0/1365574153 shutdown_connections 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.745+0000 7fba31f2a640 1 -- 172.21.15.148:0/1365574153 wait complete. 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba31f2a640 1 Processor -- start 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba31f2a640 1 -- start start 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba31f2a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba2c101c00 0x7fba2c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba31f2a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c102e00 0x7fba2c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba31f2a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c1037c0 0x7fba2c19f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba31f2a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c10d880 con 0x7fba2c101c00 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba31f2a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c10d700 con 0x7fba2c102e00 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba31f2a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fba2c10da00 con 0x7fba2c1037c0 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c1037c0 0x7fba2c19f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:09:44.779 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c1037c0 0x7fba2c19f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:52510/0 (socket says 172.21.15.148:52510) 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2bfff640 1 -- 172.21.15.148:0/746387920 learned_addr learned my addr 172.21.15.148:0/746387920 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2affd640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c102e00 0x7fba2c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2bfff640 1 -- 172.21.15.148:0/746387920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c102e00 msgr2=0x7fba2c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2b7fe640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba2c101c00 0x7fba2c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2bfff640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c102e00 0x7fba2c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2bfff640 1 -- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba2c101c00 msgr2=0x7fba2c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2bfff640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba2c101c00 0x7fba2c197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2bfff640 1 -- 172.21.15.148:0/746387920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba2c19fb00 con 0x7fba2c1037c0 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.746+0000 7fba2affd640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c102e00 0x7fba2c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.747+0000 7fba2bfff640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c1037c0 0x7fba2c19f490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba2400eaa0 tx=0x7fba2400ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.747+0000 7fba28ff9640 1 -- 172.21.15.148:0/746387920 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2400cdb0 con 0x7fba2c1037c0 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.747+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba2c19fdf0 con 0x7fba2c1037c0 2024-09-15T23:09:44.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.747+0000 7fba28ff9640 1 -- 172.21.15.148:0/746387920 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba24005020 con 0x7fba2c1037c0 2024-09-15T23:09:44.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.747+0000 7fba28ff9640 1 -- 172.21.15.148:0/746387920 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba240116d0 con 0x7fba2c1037c0 2024-09-15T23:09:44.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.747+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba2c1a02a0 con 0x7fba2c1037c0 2024-09-15T23:09:44.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.748+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba2c108fb0 con 0x7fba2c1037c0 2024-09-15T23:09:44.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.749+0000 7fba28ff9640 1 -- 172.21.15.148:0/746387920 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fba240054d0 con 0x7fba2c1037c0 2024-09-15T23:09:44.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.749+0000 7fba28ff9640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fba140a3270 0x7fba140a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:09:44.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.750+0000 7fba28ff9640 1 -- 172.21.15.148:0/746387920 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 3821+0+0 (secure 0 0 0) 0x7fba240cea30 con 0x7fba2c1037c0 2024-09-15T23:09:44.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.750+0000 7fba2b7fe640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fba140a3270 0x7fba140a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:09:44.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.750+0000 7fba2b7fe640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fba140a3270 0x7fba140a5730 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba2c102c60 tx=0x7fba1c005c60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:09:44.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.754+0000 7fba28ff9640 1 -- 172.21.15.148:0/746387920 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fba24016030 con 0x7fba2c1037c0 2024-09-15T23:09:44.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:09:44.929+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fba2c062630 con 0x7fba140a3270 2024-09-15T23:09:44.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:09:44.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: osdmap e31: 5 total, 4 up, 5 in 2024-09-15T23:09:44.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:44.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:44.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:44.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:09:44.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:44.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:09:44.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:09:44.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:44 smithi062 ceph-mon[30949]: from='osd.4 ' entity='osd.4' 2024-09-15T23:09:45.615 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:09:45.615 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:09:45.615 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:45.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi067 2024-09-15T23:09:45.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi067 to 8544M 2024-09-15T23:09:45.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:45.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: osd.4 [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] boot 2024-09-15T23:09:45.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: osdmap e32: 5 total, 5 up, 5 in 2024-09-15T23:09:45.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:45.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:09:45.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:09:45.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:45 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:09:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:09:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:45.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi067 2024-09-15T23:09:45.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi067 to 8544M 2024-09-15T23:09:45.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:45.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: osd.4 [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] boot 2024-09-15T23:09:45.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: osdmap e32: 5 total, 5 up, 5 in 2024-09-15T23:09:45.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:45.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:09:45.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:09:45.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:45 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:45.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:09:45.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:09:45.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-09-15T23:09:45.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi067 2024-09-15T23:09:45.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi067 to 8544M 2024-09-15T23:09:45.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:45.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: osd.4 [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] boot 2024-09-15T23:09:45.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: osdmap e32: 5 total, 5 up, 5 in 2024-09-15T23:09:45.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-15T23:09:45.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:09:45.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:09:45.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:45 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:46.686 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:46 smithi148 ceph-mon[35610]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:09:46.686 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:46 smithi148 ceph-mon[35610]: osdmap e33: 5 total, 5 up, 5 in 2024-09-15T23:09:46.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:46 smithi067 ceph-mon[35665]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:09:46.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:46 smithi067 ceph-mon[35665]: osdmap e33: 5 total, 5 up, 5 in 2024-09-15T23:09:46.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:46 smithi062 ceph-mon[30949]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:09:46.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:46 smithi062 ceph-mon[30949]: osdmap e33: 5 total, 5 up, 5 in 2024-09-15T23:09:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:47 smithi067 ceph-mon[35665]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:47 smithi067 ceph-mon[35665]: from='client.? 172.21.15.148:0/3303285265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d4f2bbc-07fb-40de-88f0-ecf68cdb2579"}]: dispatch 2024-09-15T23:09:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:47 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d4f2bbc-07fb-40de-88f0-ecf68cdb2579"}]: dispatch 2024-09-15T23:09:47.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:47 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d4f2bbc-07fb-40de-88f0-ecf68cdb2579"}]': finished 2024-09-15T23:09:47.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:47 smithi067 ceph-mon[35665]: osdmap e34: 6 total, 5 up, 6 in 2024-09-15T23:09:47.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:09:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:47 smithi148 ceph-mon[35610]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:47 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/3303285265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d4f2bbc-07fb-40de-88f0-ecf68cdb2579"}]: dispatch 2024-09-15T23:09:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:47 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d4f2bbc-07fb-40de-88f0-ecf68cdb2579"}]: dispatch 2024-09-15T23:09:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:47 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d4f2bbc-07fb-40de-88f0-ecf68cdb2579"}]': finished 2024-09-15T23:09:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:47 smithi148 ceph-mon[35610]: osdmap e34: 6 total, 5 up, 6 in 2024-09-15T23:09:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:09:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:47 smithi062 ceph-mon[30949]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:47 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/3303285265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d4f2bbc-07fb-40de-88f0-ecf68cdb2579"}]: dispatch 2024-09-15T23:09:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:47 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d4f2bbc-07fb-40de-88f0-ecf68cdb2579"}]: dispatch 2024-09-15T23:09:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:47 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d4f2bbc-07fb-40de-88f0-ecf68cdb2579"}]': finished 2024-09-15T23:09:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:47 smithi062 ceph-mon[30949]: osdmap e34: 6 total, 5 up, 6 in 2024-09-15T23:09:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:09:48.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:48 smithi067 ceph-mon[35665]: from='client.? 172.21.15.148:0/618835510' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:09:48.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:48 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/618835510' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:09:48.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:48 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/618835510' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:09:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:49 smithi067 ceph-mon[35665]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:49.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:49 smithi148 ceph-mon[35610]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:49 smithi062 ceph-mon[30949]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:51 smithi067 ceph-mon[35665]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:51 smithi148 ceph-mon[35610]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:51 smithi062 ceph-mon[30949]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:53.730 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:53 smithi148 ceph-mon[35610]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:53.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:53 smithi067 ceph-mon[35665]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:53.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:53 smithi062 ceph-mon[30949]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:55.616 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:55 smithi148 ceph-mon[35610]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:55 smithi067 ceph-mon[35665]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:55 smithi062 ceph-mon[30949]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:56.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-15T23:09:56.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-15T23:09:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:57.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-15T23:09:57.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:09:57.642 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:57 smithi148 ceph-mon[35610]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:57.642 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:57 smithi148 ceph-mon[35610]: Deploying daemon osd.5 on smithi148 2024-09-15T23:09:57.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:57 smithi062 ceph-mon[30949]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:57.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:57 smithi062 ceph-mon[30949]: Deploying daemon osd.5 on smithi148 2024-09-15T23:09:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:57 smithi067 ceph-mon[35665]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:57 smithi067 ceph-mon[35665]: Deploying daemon osd.5 on smithi148 2024-09-15T23:09:59.732 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:09:59 smithi148 ceph-mon[35610]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:59.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:09:59 smithi067 ceph-mon[35665]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:09:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:09:59 smithi062 ceph-mon[30949]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:00.689 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:00 smithi148 ceph-mon[35610]: overall HEALTH_OK 2024-09-15T23:10:00.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:00 smithi062 ceph-mon[30949]: overall HEALTH_OK 2024-09-15T23:10:01.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:00 smithi067 ceph-mon[35665]: overall HEALTH_OK 2024-09-15T23:10:01.578 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:01 smithi148 ceph-mon[35610]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:01.945 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:01 smithi062 ceph-mon[30949]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:01 smithi067 ceph-mon[35665]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:02.632 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:02.632 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:02.633 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:02.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:02.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:02.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:03.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:03.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:03.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:03.648 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:03 smithi148 ceph-mon[35610]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:03.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:03 smithi067 ceph-mon[35665]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:03 smithi062 ceph-mon[30949]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:05.378 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.376+0000 7fba28ff9640 1 -- 172.21.15.148:0/746387920 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fba2c062630 con 0x7fba140a3270 2024-09-15T23:10:05.379 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 5 on host 'smithi148' 2024-09-15T23:10:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.378+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fba140a3270 msgr2=0x7fba140a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.378+0000 7fba31f2a640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fba140a3270 0x7fba140a5730 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba2c102c60 tx=0x7fba1c005c60 comp rx=0 tx=0).stop 2024-09-15T23:10:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c1037c0 msgr2=0x7fba2c19f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c1037c0 0x7fba2c19f490 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba2400eaa0 tx=0x7fba2400ef60 comp rx=0 tx=0).stop 2024-09-15T23:10:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 shutdown_connections 2024-09-15T23:10:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fba140a3270 0x7fba140a5730 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba2c1037c0 0x7fba2c19f490 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:05.381 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba2c102e00 0x7fba2c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:05.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 --2- 172.21.15.148:0/746387920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba2c101c00 0x7fba2c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:05.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 >> 172.21.15.148:0/746387920 conn(0x7fba2c0fd390 msgr2=0x7fba2c106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:10:05.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 shutdown_connections 2024-09-15T23:10:05.382 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:05.379+0000 7fba31f2a640 1 -- 172.21.15.148:0/746387920 wait complete. 2024-09-15T23:10:05.702 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:05 smithi148 ceph-mon[35610]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:05.702 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:05 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:05.702 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:05 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:05.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:05 smithi062 ceph-mon[30949]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:05.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:05 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:05.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:05 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:05 smithi067 ceph-mon[35665]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:05 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:05 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:06.055 DEBUG:teuthology.orchestra.run.smithi148:osd.5> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.5.service 2024-09-15T23:10:06.058 INFO:tasks.cephadm:Deploying osd.6 on smithi062 with /dev/vg_nvme/lv_2... 2024-09-15T23:10:06.058 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-15T23:10:06.325 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:10:07.469 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:07 smithi062 ceph-mon[30949]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:07.469 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:07 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:07.469 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:07 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:07.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:07 smithi067 ceph-mon[35665]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:07.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:07 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:07.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:07 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:07.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:07 smithi148 ceph-mon[35610]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:07.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:07 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:07.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:07 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:08.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:08 smithi148 ceph-mon[35610]: from='osd.5 [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:10:08.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:08 smithi148 ceph-mon[35610]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:10:08.290 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:10:07 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:10:07.939+0000 7f26e4cae640 -1 osd.5 0 log_to_monitors true 2024-09-15T23:10:08.323 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:08 smithi062 ceph-mon[30949]: from='osd.5 [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:10:08.323 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:08 smithi062 ceph-mon[30949]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:10:08.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:08 smithi067 ceph-mon[35665]: from='osd.5 [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:10:08.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:08 smithi067 ceph-mon[35665]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-15T23:10:08.999 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:10:09.025 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_2 2024-09-15T23:10:09.287 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:10:09.393 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:09 smithi148 ceph-mon[35610]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:09.393 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:09 smithi148 ceph-mon[35610]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-15T23:10:09.394 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:09 smithi148 ceph-mon[35610]: from='osd.5 [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:10:09.394 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:09 smithi148 ceph-mon[35610]: osdmap e35: 6 total, 5 up, 6 in 2024-09-15T23:10:09.394 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:09 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:09.394 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:09 smithi148 ceph-mon[35610]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:10:09.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:09 smithi062 ceph-mon[30949]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:09.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:09 smithi062 ceph-mon[30949]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-15T23:10:09.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:09 smithi062 ceph-mon[30949]: from='osd.5 [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:10:09.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:09 smithi062 ceph-mon[30949]: osdmap e35: 6 total, 5 up, 6 in 2024-09-15T23:10:09.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:09 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:09.410 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:09 smithi062 ceph-mon[30949]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:10:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:09 smithi067 ceph-mon[35665]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:09 smithi067 ceph-mon[35665]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-15T23:10:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:09 smithi067 ceph-mon[35665]: from='osd.5 [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:10:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:09 smithi067 ceph-mon[35665]: osdmap e35: 6 total, 5 up, 6 in 2024-09-15T23:10:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:09 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:09 smithi067 ceph-mon[35665]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:10:09.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.760+0000 7f67d17f0640 1 -- 172.21.15.62:0/1008464849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc1037c0 msgr2=0x7f67cc10a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.760+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1008464849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc1037c0 0x7f67cc10a050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009ca0 tx=0x7f67c402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:10:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.760+0000 7f67d17f0640 1 -- 172.21.15.62:0/1008464849 shutdown_connections 2024-09-15T23:10:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.760+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1008464849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc1037c0 0x7f67cc10a050 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.760+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1008464849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67cc102e00 0x7f67cc103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.760+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1008464849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67cc101c00 0x7f67cc102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.760+0000 7f67d17f0640 1 -- 172.21.15.62:0/1008464849 >> 172.21.15.62:0/1008464849 conn(0x7f67cc0fd390 msgr2=0x7f67cc0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:10:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.760+0000 7f67d17f0640 1 -- 172.21.15.62:0/1008464849 shutdown_connections 2024-09-15T23:10:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.760+0000 7f67d17f0640 1 -- 172.21.15.62:0/1008464849 wait complete. 2024-09-15T23:10:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67d17f0640 1 Processor -- start 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67d17f0640 1 -- start start 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67d17f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67cc101c00 0x7f67cc075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67d17f0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc102e00 0x7f67cc0745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67d17f0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67cc1037c0 0x7f67cc074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67d17f0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc10d880 con 0x7f67cc102e00 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67d17f0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc10d700 con 0x7f67cc101c00 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67d17f0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc10da00 con 0x7f67cc1037c0 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67ca7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc102e00 0x7f67cc0745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67ca7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc102e00 0x7f67cc0745e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56388/0 (socket says 172.21.15.62:56388) 2024-09-15T23:10:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67ca7fc640 1 -- 172.21.15.62:0/1754345323 learned_addr learned my addr 172.21.15.62:0/1754345323 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:10:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.761+0000 7f67caffd640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67cc101c00 0x7f67cc075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67cb7fe640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67cc1037c0 0x7f67cc074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67ca7fc640 1 -- 172.21.15.62:0/1754345323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67cc1037c0 msgr2=0x7f67cc074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67ca7fc640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67cc1037c0 0x7f67cc074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67ca7fc640 1 -- 172.21.15.62:0/1754345323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67cc101c00 msgr2=0x7f67cc075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67ca7fc640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67cc101c00 0x7f67cc075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67ca7fc640 1 -- 172.21.15.62:0/1754345323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67cc0753e0 con 0x7f67cc102e00 2024-09-15T23:10:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67ca7fc640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc102e00 0x7f67cc0745e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f67c000c890 tx=0x7f67c000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:10:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67b3fff640 1 -- 172.21.15.62:0/1754345323 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0004220 con 0x7f67cc102e00 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67cc1a1fd0 con 0x7f67cc102e00 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67b3fff640 1 -- 172.21.15.62:0/1754345323 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c00043c0 con 0x7f67cc102e00 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67b3fff640 1 -- 172.21.15.62:0/1754345323 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0004a00 con 0x7f67cc102e00 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.762+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67cc1a24d0 con 0x7f67cc102e00 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.763+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67cc109000 con 0x7f67cc102e00 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.764+0000 7f67b3fff640 1 -- 172.21.15.62:0/1754345323 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f67c00079b0 con 0x7f67cc102e00 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.765+0000 7f67b3fff640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f67b40a3240 0x7f67b40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.765+0000 7f67caffd640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f67b40a3240 0x7f67b40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.765+0000 7f67b3fff640 1 -- 172.21.15.62:0/1754345323 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4253+0+0 (secure 0 0 0) 0x7f67c00ce910 con 0x7f67cc102e00 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.765+0000 7f67caffd640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f67b40a3240 0x7f67b40a5700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67bc005fd0 tx=0x7f67bc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:10:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.768+0000 7f67b3fff640 1 -- 172.21.15.62:0/1754345323 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f67c0014030 con 0x7f67cc102e00 2024-09-15T23:10:09.793 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:10:09 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:10:09.487+0000 7f26e0c31640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T23:10:09.793 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:10:09 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:10:09.494+0000 7f26d8a51640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:10:09.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:09.956+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f67cc062630 con 0x7f67b40a3240 2024-09-15T23:10:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:10:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: osdmap e36: 6 total, 5 up, 6 in 2024-09-15T23:10:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi148 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi148 to 8544M 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='osd.5 ' entity='osd.5' 2024-09-15T23:10:10.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:10.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:10.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:10.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:10:10.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:10:10.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:10.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: osdmap e36: 6 total, 5 up, 6 in 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi148 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi148 to 8544M 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:10.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='osd.5 ' entity='osd.5' 2024-09-15T23:10:10.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:10.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:10.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:10.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:10:10.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:10:10.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:10.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: osdmap e36: 6 total, 5 up, 6 in 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi148 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi148 to 8544M 2024-09-15T23:10:10.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='osd.5 ' entity='osd.5' 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:10:10.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:10:10.542 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:11.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:11 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:10:11.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:11 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:10:11.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:11 smithi062 ceph-mon[30949]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:11.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:11 smithi062 ceph-mon[30949]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:10:11.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:11 smithi062 ceph-mon[30949]: osd.5 [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] boot 2024-09-15T23:10:11.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:11 smithi062 ceph-mon[30949]: osdmap e37: 6 total, 6 up, 6 in 2024-09-15T23:10:11.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:11 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:11.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:11 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:10:11.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:11 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:10:11.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:11 smithi067 ceph-mon[35665]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:11.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:11 smithi067 ceph-mon[35665]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:10:11.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:11 smithi067 ceph-mon[35665]: osd.5 [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] boot 2024-09-15T23:10:11.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:11 smithi067 ceph-mon[35665]: osdmap e37: 6 total, 6 up, 6 in 2024-09-15T23:10:11.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:11 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:11 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:10:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:11 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:10:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:11 smithi148 ceph-mon[35610]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-09-15T23:10:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:11 smithi148 ceph-mon[35610]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:10:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:11 smithi148 ceph-mon[35610]: osd.5 [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] boot 2024-09-15T23:10:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:11 smithi148 ceph-mon[35610]: osdmap e37: 6 total, 6 up, 6 in 2024-09-15T23:10:11.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:11 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-15T23:10:12.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:12 smithi062 ceph-mon[30949]: osdmap e38: 6 total, 6 up, 6 in 2024-09-15T23:10:12.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:12 smithi067 ceph-mon[35665]: osdmap e38: 6 total, 6 up, 6 in 2024-09-15T23:10:12.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:12 smithi148 ceph-mon[35610]: osdmap e38: 6 total, 6 up, 6 in 2024-09-15T23:10:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:13 smithi062 ceph-mon[30949]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:13 smithi062 ceph-mon[30949]: osdmap e39: 6 total, 6 up, 6 in 2024-09-15T23:10:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:13 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2459792965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75629f41-6236-4d44-82fd-0f1d5e0df841"}]: dispatch 2024-09-15T23:10:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:13 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2459792965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75629f41-6236-4d44-82fd-0f1d5e0df841"}]': finished 2024-09-15T23:10:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:13 smithi062 ceph-mon[30949]: osdmap e40: 7 total, 6 up, 7 in 2024-09-15T23:10:13.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:13.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:13 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3039332580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:10:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:13 smithi067 ceph-mon[35665]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:13 smithi067 ceph-mon[35665]: osdmap e39: 6 total, 6 up, 6 in 2024-09-15T23:10:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:13 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2459792965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75629f41-6236-4d44-82fd-0f1d5e0df841"}]: dispatch 2024-09-15T23:10:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:13 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2459792965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75629f41-6236-4d44-82fd-0f1d5e0df841"}]': finished 2024-09-15T23:10:13.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:13 smithi067 ceph-mon[35665]: osdmap e40: 7 total, 6 up, 7 in 2024-09-15T23:10:13.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:13 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:13.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:13 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3039332580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:10:13.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:13 smithi148 ceph-mon[35610]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:13.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:13 smithi148 ceph-mon[35610]: osdmap e39: 6 total, 6 up, 6 in 2024-09-15T23:10:13.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:13 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2459792965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75629f41-6236-4d44-82fd-0f1d5e0df841"}]: dispatch 2024-09-15T23:10:13.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:13 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2459792965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75629f41-6236-4d44-82fd-0f1d5e0df841"}]': finished 2024-09-15T23:10:13.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:13 smithi148 ceph-mon[35610]: osdmap e40: 7 total, 6 up, 7 in 2024-09-15T23:10:13.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:13 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:13.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:13 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3039332580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:10:14.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:14 smithi062 ceph-mon[30949]: osdmap e41: 7 total, 6 up, 7 in 2024-09-15T23:10:14.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:14 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:14.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:14 smithi067 ceph-mon[35665]: osdmap e41: 7 total, 6 up, 7 in 2024-09-15T23:10:14.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:14 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:14.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:14 smithi148 ceph-mon[35610]: osdmap e41: 7 total, 6 up, 7 in 2024-09-15T23:10:14.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:14 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:15.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:15 smithi062 ceph-mon[30949]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:15.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:15 smithi062 ceph-mon[30949]: osdmap e42: 7 total, 6 up, 7 in 2024-09-15T23:10:15.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:15 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:15.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:15 smithi067 ceph-mon[35665]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:15.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:15 smithi067 ceph-mon[35665]: osdmap e42: 7 total, 6 up, 7 in 2024-09-15T23:10:15.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:15 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:15.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:15 smithi148 ceph-mon[35610]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:15.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:15 smithi148 ceph-mon[35610]: osdmap e42: 7 total, 6 up, 7 in 2024-09-15T23:10:15.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:15 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:17.697 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:17 smithi062 ceph-mon[30949]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-15T23:10:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:17 smithi067 ceph-mon[35665]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-15T23:10:17.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:17 smithi148 ceph-mon[35610]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-15T23:10:19.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:19 smithi062 ceph-mon[30949]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-15T23:10:19.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:19 smithi067 ceph-mon[35665]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-15T23:10:19.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:19 smithi148 ceph-mon[35610]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-15T23:10:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:21 smithi062 ceph-mon[30949]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-15T23:10:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:21 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-15T23:10:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:21 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:21 smithi062 ceph-mon[30949]: Deploying daemon osd.6 on smithi062 2024-09-15T23:10:21.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:21 smithi067 ceph-mon[35665]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-15T23:10:21.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:21 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-15T23:10:21.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:21 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:21.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:21 smithi067 ceph-mon[35665]: Deploying daemon osd.6 on smithi062 2024-09-15T23:10:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:21 smithi148 ceph-mon[35610]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-15T23:10:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:21 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-15T23:10:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:21 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:21 smithi148 ceph-mon[35610]: Deploying daemon osd.6 on smithi062 2024-09-15T23:10:23.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:23 smithi062 ceph-mon[30949]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-15T23:10:23.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:23 smithi067 ceph-mon[35665]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-15T23:10:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:23 smithi148 ceph-mon[35610]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-15T23:10:25.476 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:25 smithi062 ceph-mon[30949]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T23:10:25.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:25 smithi067 ceph-mon[35665]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T23:10:25.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:25 smithi148 ceph-mon[35610]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T23:10:27.470 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:27 smithi062 ceph-mon[30949]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:27.470 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:27.470 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:27.470 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:27.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:27 smithi067 ceph-mon[35665]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:27.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:27.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:27.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:27.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:27 smithi148 ceph-mon[35610]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:27.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:27.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:27.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:29 smithi062 ceph-mon[30949]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:29.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:29 smithi067 ceph-mon[35665]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:29.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:29 smithi148 ceph-mon[35610]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:30.783 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 6 on host 'smithi062' 2024-09-15T23:10:30.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.781+0000 7f67b3fff640 1 -- 172.21.15.62:0/1754345323 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f67cc062630 con 0x7f67b40a3240 2024-09-15T23:10:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f67b40a3240 msgr2=0x7f67b40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f67b40a3240 0x7f67b40a5700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67bc005fd0 tx=0x7f67bc00b390 comp rx=0 tx=0).stop 2024-09-15T23:10:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc102e00 msgr2=0x7f67cc0745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:30.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc102e00 0x7f67cc0745e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f67c000c890 tx=0x7f67c000cd50 comp rx=0 tx=0).stop 2024-09-15T23:10:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 shutdown_connections 2024-09-15T23:10:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f67b40a3240 0x7f67b40a5700 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67cc1037c0 0x7f67cc074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67cc102e00 0x7f67cc0745e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 --2- 172.21.15.62:0/1754345323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67cc101c00 0x7f67cc075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 >> 172.21.15.62:0/1754345323 conn(0x7f67cc0fd390 msgr2=0x7f67cc1061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:10:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 shutdown_connections 2024-09-15T23:10:30.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:10:30.784+0000 7f67d17f0640 1 -- 172.21.15.62:0/1754345323 wait complete. 2024-09-15T23:10:31.426 DEBUG:teuthology.orchestra.run.smithi062:osd.6> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.6.service 2024-09-15T23:10:31.428 INFO:tasks.cephadm:Deploying osd.7 on smithi067 with /dev/vg_nvme/lv_2... 2024-09-15T23:10:31.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-15T23:10:31.687 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:10:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:31 smithi062 ceph-mon[30949]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:31.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:31 smithi067 ceph-mon[35665]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:31.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:31.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:31.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:31 smithi148 ceph-mon[35610]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:32.611 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:32 smithi062 ceph-mon[30949]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-15T23:10:32.612 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:10:32 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:10:32.267+0000 7fa99bdc2640 -1 osd.6 0 log_to_monitors true 2024-09-15T23:10:32.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:32 smithi067 ceph-mon[35665]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-15T23:10:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:32 smithi148 ceph-mon[35610]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-15T23:10:33.710 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:33 smithi062 ceph-mon[30949]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:33.711 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:33 smithi062 ceph-mon[30949]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-15T23:10:33.711 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:33 smithi062 ceph-mon[30949]: osdmap e43: 7 total, 6 up, 7 in 2024-09-15T23:10:33.711 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:33 smithi062 ceph-mon[30949]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:10:33.711 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:33 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:33 smithi067 ceph-mon[35665]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:33.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:33 smithi067 ceph-mon[35665]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-15T23:10:33.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:33 smithi067 ceph-mon[35665]: osdmap e43: 7 total, 6 up, 7 in 2024-09-15T23:10:33.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:33 smithi067 ceph-mon[35665]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:10:33.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:33 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:33 smithi148 ceph-mon[35610]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:33 smithi148 ceph-mon[35610]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-15T23:10:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:33 smithi148 ceph-mon[35610]: osdmap e43: 7 total, 6 up, 7 in 2024-09-15T23:10:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:33 smithi148 ceph-mon[35610]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:10:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:33 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:33.980 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:10:33 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:10:33.900+0000 7fa997d45640 -1 osd.6 0 waiting for initial osdmap 2024-09-15T23:10:33.980 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:10:33 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:10:33.909+0000 7fa98fb65640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:10:34.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:34 smithi067 ceph-mon[35665]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:10:34.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:34 smithi067 ceph-mon[35665]: osdmap e44: 7 total, 6 up, 7 in 2024-09-15T23:10:34.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:34.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:34.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:34 smithi067 ceph-mon[35665]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' 2024-09-15T23:10:34.534 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:10:34.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_2 2024-09-15T23:10:34.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:34 smithi062 ceph-mon[30949]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:10:34.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:34 smithi062 ceph-mon[30949]: osdmap e44: 7 total, 6 up, 7 in 2024-09-15T23:10:34.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:34.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:34.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:34 smithi062 ceph-mon[30949]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' 2024-09-15T23:10:34.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:34 smithi148 ceph-mon[35610]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:10:34.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:34 smithi148 ceph-mon[35610]: osdmap e44: 7 total, 6 up, 7 in 2024-09-15T23:10:34.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:34.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:34.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:34 smithi148 ceph-mon[35610]: from='osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035]' entity='osd.6' 2024-09-15T23:10:34.816 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:10:35.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.376+0000 7f73da953640 1 -- 172.21.15.67:0/1228640411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4100ad0 msgr2=0x7f73d4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:35.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.376+0000 7f73da953640 1 --2- 172.21.15.67:0/1228640411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4100ad0 0x7f73d4100ed0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009900 tx=0x7f73c402a410 comp rx=0 tx=0).stop 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.376+0000 7f73da953640 1 -- 172.21.15.67:0/1228640411 shutdown_connections 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.376+0000 7f73da953640 1 --2- 172.21.15.67:0/1228640411 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73d4102690 0x7f73d4108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.376+0000 7f73da953640 1 --2- 172.21.15.67:0/1228640411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d4101cd0 0x7f73d4102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.376+0000 7f73da953640 1 --2- 172.21.15.67:0/1228640411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4100ad0 0x7f73d4100ed0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.376+0000 7f73da953640 1 -- 172.21.15.67:0/1228640411 >> 172.21.15.67:0/1228640411 conn(0x7f73d40fc280 msgr2=0x7f73d40fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.376+0000 7f73da953640 1 -- 172.21.15.67:0/1228640411 shutdown_connections 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.376+0000 7f73da953640 1 -- 172.21.15.67:0/1228640411 wait complete. 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73da953640 1 Processor -- start 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73da953640 1 -- start start 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73da953640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d4100ad0 0x7f73d4104a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73da953640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73d4101cd0 0x7f73d41030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:35.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73da953640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4102690 0x7f73d41035f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73da953640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f73d410c7b0 con 0x7f73d4100ad0 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73da953640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f73d410c630 con 0x7f73d4102690 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73da953640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f73d410c930 con 0x7f73d4101cd0 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73d8ef0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4102690 0x7f73d41035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73d8ef0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4102690 0x7f73d41035f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46484/0 (socket says 172.21.15.67:46484) 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.377+0000 7f73d8ef0640 1 -- 172.21.15.67:0/2323279064 learned_addr learned my addr 172.21.15.67:0/2323279064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d3fff640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d4100ad0 0x7f73d4104a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d37fe640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73d4101cd0 0x7f73d41030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d8ef0640 1 -- 172.21.15.67:0/2323279064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73d4101cd0 msgr2=0x7f73d41030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d8ef0640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73d4101cd0 0x7f73d41030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d8ef0640 1 -- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d4100ad0 msgr2=0x7f73d4104a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d8ef0640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d4100ad0 0x7f73d4104a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:35.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d8ef0640 1 -- 172.21.15.67:0/2323279064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d4103eb0 con 0x7f73d4102690 2024-09-15T23:10:35.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d8ef0640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4102690 0x7f73d41035f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00b970 tx=0x7f73cc00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:10:35.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d17fa640 1 -- 172.21.15.67:0/2323279064 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc019070 con 0x7f73d4102690 2024-09-15T23:10:35.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73d41a2130 con 0x7f73d4102690 2024-09-15T23:10:35.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d17fa640 1 -- 172.21.15.67:0/2323279064 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73cc00cb40 con 0x7f73d4102690 2024-09-15T23:10:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73d17fa640 1 -- 172.21.15.67:0/2323279064 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc007650 con 0x7f73d4102690 2024-09-15T23:10:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.378+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73d41a25f0 con 0x7f73d4102690 2024-09-15T23:10:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.379+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73d41a2880 con 0x7f73d4102690 2024-09-15T23:10:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.380+0000 7f73d17fa640 1 -- 172.21.15.67:0/2323279064 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f73cc01c070 con 0x7f73d4102690 2024-09-15T23:10:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.381+0000 7f73d17fa640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73bc0a3240 0x7f73bc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.381+0000 7f73d17fa640 1 -- 172.21.15.67:0/2323279064 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 4965+0+0 (secure 0 0 0) 0x7f73cc0ce1e0 con 0x7f73d4102690 2024-09-15T23:10:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.381+0000 7f73d3fff640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73bc0a3240 0x7f73bc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.382+0000 7f73d3fff640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73bc0a3240 0x7f73bc0a5700 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f73c4005ab0 tx=0x7f73c4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:10:35.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.385+0000 7f73d17fa640 1 -- 172.21.15.67:0/2323279064 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f73cc09b5c0 con 0x7f73d4102690 2024-09-15T23:10:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:35.571+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f73d41a2a50 con 0x7f73bc0a3240 2024-09-15T23:10:35.573 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:10:35.573 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:10:35.573 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi062 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi062 to 5696M 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.574 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] boot 2024-09-15T23:10:35.575 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: osdmap e45: 7 total, 7 up, 7 in 2024-09-15T23:10:35.575 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:35 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:10:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:10:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi062 2024-09-15T23:10:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi062 to 5696M 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] boot 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: osdmap e45: 7 total, 7 up, 7 in 2024-09-15T23:10:35.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:35 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:10:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:10:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi062 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi062 to 5696M 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: osd.6 [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] boot 2024-09-15T23:10:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: osdmap e45: 7 total, 7 up, 7 in 2024-09-15T23:10:35.791 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:35 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-15T23:10:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:36 smithi062 ceph-mon[30949]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:10:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:10:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:10:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:36 smithi062 ceph-mon[30949]: osdmap e46: 7 total, 7 up, 7 in 2024-09-15T23:10:36.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:36 smithi067 ceph-mon[35665]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:10:36.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:10:36.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:10:36.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:36.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:36 smithi067 ceph-mon[35665]: osdmap e46: 7 total, 7 up, 7 in 2024-09-15T23:10:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:36 smithi148 ceph-mon[35610]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:10:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:10:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:10:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:36.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:36 smithi148 ceph-mon[35610]: osdmap e46: 7 total, 7 up, 7 in 2024-09-15T23:10:37.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:37 smithi062 ceph-mon[30949]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:37.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:37 smithi067 ceph-mon[35665]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:37.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:37 smithi148 ceph-mon[35610]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:38.716 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:38 smithi067 ceph-mon[35665]: from='client.? 172.21.15.67:0/1969594462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f605b7f7-4cee-43c8-aae4-f197c5664f49"}]: dispatch 2024-09-15T23:10:38.716 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:38 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f605b7f7-4cee-43c8-aae4-f197c5664f49"}]: dispatch 2024-09-15T23:10:38.716 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:38 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f605b7f7-4cee-43c8-aae4-f197c5664f49"}]': finished 2024-09-15T23:10:38.716 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:38 smithi067 ceph-mon[35665]: osdmap e47: 8 total, 7 up, 8 in 2024-09-15T23:10:38.717 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:10:38.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:38 smithi062 ceph-mon[30949]: from='client.? 172.21.15.67:0/1969594462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f605b7f7-4cee-43c8-aae4-f197c5664f49"}]: dispatch 2024-09-15T23:10:38.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:38 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f605b7f7-4cee-43c8-aae4-f197c5664f49"}]: dispatch 2024-09-15T23:10:38.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:38 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f605b7f7-4cee-43c8-aae4-f197c5664f49"}]': finished 2024-09-15T23:10:38.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:38 smithi062 ceph-mon[30949]: osdmap e47: 8 total, 7 up, 8 in 2024-09-15T23:10:38.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:10:38.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:38 smithi148 ceph-mon[35610]: from='client.? 172.21.15.67:0/1969594462' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f605b7f7-4cee-43c8-aae4-f197c5664f49"}]: dispatch 2024-09-15T23:10:38.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:38 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f605b7f7-4cee-43c8-aae4-f197c5664f49"}]: dispatch 2024-09-15T23:10:38.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:38 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f605b7f7-4cee-43c8-aae4-f197c5664f49"}]': finished 2024-09-15T23:10:38.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:38 smithi148 ceph-mon[35610]: osdmap e47: 8 total, 7 up, 8 in 2024-09-15T23:10:38.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:10:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:39 smithi062 ceph-mon[30949]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:39 smithi062 ceph-mon[30949]: from='client.? 172.21.15.67:0/4049197243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:10:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:39 smithi067 ceph-mon[35665]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:39 smithi067 ceph-mon[35665]: from='client.? 172.21.15.67:0/4049197243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:10:39.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:39 smithi148 ceph-mon[35610]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:39.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:39 smithi148 ceph-mon[35610]: from='client.? 172.21.15.67:0/4049197243' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:10:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:41 smithi062 ceph-mon[30949]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:41 smithi067 ceph-mon[35665]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:41 smithi148 ceph-mon[35610]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:43 smithi067 ceph-mon[35665]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:43 smithi062 ceph-mon[30949]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:43 smithi148 ceph-mon[35610]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:45.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:45 smithi062 ceph-mon[30949]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:45 smithi067 ceph-mon[35665]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:45.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:45 smithi148 ceph-mon[35610]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:46.711 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:46 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-15T23:10:46.711 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:46 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:46.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:46 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-15T23:10:46.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:46 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:46.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:46 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-15T23:10:46.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:46 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:47.697 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:47 smithi067 ceph-mon[35665]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:47.697 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:47 smithi067 ceph-mon[35665]: Deploying daemon osd.7 on smithi067 2024-09-15T23:10:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:47 smithi062 ceph-mon[30949]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:47 smithi062 ceph-mon[30949]: Deploying daemon osd.7 on smithi067 2024-09-15T23:10:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:47 smithi148 ceph-mon[35610]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:47 smithi148 ceph-mon[35610]: Deploying daemon osd.7 on smithi067 2024-09-15T23:10:49.644 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:49 smithi067 ceph-mon[35665]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:49.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:49 smithi148 ceph-mon[35610]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:49 smithi062 ceph-mon[30949]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:51 smithi067 ceph-mon[35665]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:51 smithi148 ceph-mon[35610]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:51 smithi062 ceph-mon[30949]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:52.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:52.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:52.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:52.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:52.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:52.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:52.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:52.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:52.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:53.568 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:53 smithi067 ceph-mon[35665]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:53 smithi148 ceph-mon[35610]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:53.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:53 smithi062 ceph-mon[30949]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:55.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.536+0000 7f73d17fa640 1 -- 172.21.15.67:0/2323279064 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f73d41a2a50 con 0x7f73bc0a3240 2024-09-15T23:10:55.538 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 7 on host 'smithi067' 2024-09-15T23:10:55.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73bc0a3240 msgr2=0x7f73bc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73bc0a3240 0x7f73bc0a5700 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f73c4005ab0 tx=0x7f73c4000bc0 comp rx=0 tx=0).stop 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4102690 msgr2=0x7f73d41035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4102690 0x7f73d41035f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00b970 tx=0x7f73cc00be30 comp rx=0 tx=0).stop 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 shutdown_connections 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73bc0a3240 0x7f73bc0a5700 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73d4102690 0x7f73d41035f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f73d4101cd0 0x7f73d41030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 --2- 172.21.15.67:0/2323279064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73d4100ad0 0x7f73d4104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 >> 172.21.15.67:0/2323279064 conn(0x7f73d40fc280 msgr2=0x7f73d4069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 shutdown_connections 2024-09-15T23:10:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:10:55.538+0000 7f73da953640 1 -- 172.21.15.67:0/2323279064 wait complete. 2024-09-15T23:10:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:55 smithi067 ceph-mon[35665]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:55.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:55 smithi148 ceph-mon[35610]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:55 smithi062 ceph-mon[30949]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:56.171 DEBUG:teuthology.orchestra.run.smithi067:osd.7> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.7.service 2024-09-15T23:10:56.174 INFO:tasks.cephadm:Deploying osd.8 on smithi148 with /dev/vg_nvme/lv_2... 2024-09-15T23:10:56.174 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-15T23:10:56.434 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:10:56.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.801 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.801 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.801 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.801 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:57.838 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:57 smithi067 ceph-mon[35665]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:57.884 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:57 smithi148 ceph-mon[35610]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:57.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:57 smithi062 ceph-mon[30949]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:58.153 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:10:57 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:10:57.911+0000 7fbbf45a9640 -1 osd.7 0 log_to_monitors true 2024-09-15T23:10:58.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:58 smithi062 ceph-mon[30949]: from='osd.7 [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:10:58.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:58 smithi062 ceph-mon[30949]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:10:59.023 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:58 smithi148 ceph-mon[35610]: from='osd.7 [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:10:59.023 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:58 smithi148 ceph-mon[35610]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:10:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:58 smithi067 ceph-mon[35665]: from='osd.7 [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:10:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:58 smithi067 ceph-mon[35665]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-15T23:10:59.026 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:10:59.052 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_2 2024-09-15T23:10:59.312 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:10:59.726 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:59.726 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-15T23:10:59.726 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: osdmap e48: 8 total, 7 up, 8 in 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='osd.7 [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:59.727 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.728 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:59.728 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:59.728 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:59.728 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:10:59 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:59.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-15T23:10:59.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: osdmap e48: 8 total, 7 up, 8 in 2024-09-15T23:10:59.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='osd.7 [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:59.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:59.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:59.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.809+0000 7f181cd41640 1 -- 172.21.15.148:0/3772126032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18181037c0 msgr2=0x7f181810a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:59.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.809+0000 7f181cd41640 1 --2- 172.21.15.148:0/3772126032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18181037c0 0x7f181810a050 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1810009ca0 tx=0x7f181002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:10:59.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.809+0000 7f181cd41640 1 -- 172.21.15.148:0/3772126032 shutdown_connections 2024-09-15T23:10:59.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.809+0000 7f181cd41640 1 --2- 172.21.15.148:0/3772126032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18181037c0 0x7f181810a050 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:59.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.809+0000 7f181cd41640 1 --2- 172.21.15.148:0/3772126032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1818102e00 0x7f1818103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:59.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.809+0000 7f181cd41640 1 --2- 172.21.15.148:0/3772126032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1818101c00 0x7f1818102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:59.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.809+0000 7f181cd41640 1 -- 172.21.15.148:0/3772126032 >> 172.21.15.148:0/3772126032 conn(0x7f18180fd390 msgr2=0x7f18180ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:10:59.843 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.810+0000 7f181cd41640 1 -- 172.21.15.148:0/3772126032 shutdown_connections 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.810+0000 7f181cd41640 1 -- 172.21.15.148:0/3772126032 wait complete. 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.810+0000 7f181cd41640 1 Processor -- start 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.810+0000 7f181cd41640 1 -- start start 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181cd41640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1818101c00 0x7f1818197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181cd41640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1818102e00 0x7f1818198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181cd41640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18181037c0 0x7f181819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181cd41640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f181810d900 con 0x7f18181037c0 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181cd41640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f181810d780 con 0x7f1818102e00 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181cd41640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f181810da80 con 0x7f1818101c00 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1818101c00 0x7f1818197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:59.844 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1818101c00 0x7f1818197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42604/0 (socket says 172.21.15.148:42604) 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181659c640 1 -- 172.21.15.148:0/1079789686 learned_addr learned my addr 172.21.15.148:0/1079789686 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f180ffff640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1818102e00 0x7f1818198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f1816d9d640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18181037c0 0x7f181819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181659c640 1 -- 172.21.15.148:0/1079789686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1818102e00 msgr2=0x7f1818198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181659c640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1818102e00 0x7f1818198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181659c640 1 -- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18181037c0 msgr2=0x7f181819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181659c640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18181037c0 0x7f181819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181659c640 1 -- 172.21.15.148:0/1079789686 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181819fb00 con 0x7f1818101c00 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181659c640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1818101c00 0x7f1818197ed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f180800b450 tx=0x7f180800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f180f7fe640 1 -- 172.21.15.148:0/1079789686 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1808015020 con 0x7f1818101c00 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f181819fdf0 con 0x7f1818101c00 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f180f7fe640 1 -- 172.21.15.148:0/1079789686 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1808011d70 con 0x7f1818101c00 2024-09-15T23:10:59.845 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f180f7fe640 1 -- 172.21.15.148:0/1079789686 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180800f340 con 0x7f1818101c00 2024-09-15T23:10:59.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.811+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18181a02c0 con 0x7f1818101c00 2024-09-15T23:10:59.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.812+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1818108fb0 con 0x7f1818101c00 2024-09-15T23:10:59.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.814+0000 7f180f7fe640 1 -- 172.21.15.148:0/1079789686 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f1808018020 con 0x7f1818101c00 2024-09-15T23:10:59.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.815+0000 7f180f7fe640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18040a3270 0x7f18040a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:10:59.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.815+0000 7f180f7fe640 1 -- 172.21.15.148:0/1079789686 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5117+0+0 (secure 0 0 0) 0x7f18080ceb50 con 0x7f1818101c00 2024-09-15T23:10:59.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.815+0000 7f180ffff640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18040a3270 0x7f18040a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:10:59.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.816+0000 7f180ffff640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18040a3270 0x7f18040a5730 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f18181993f0 tx=0x7f180000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:10:59.846 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.818+0000 7f180f7fe640 1 -- 172.21.15.148:0/1079789686 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f180801f020 con 0x7f1818101c00 2024-09-15T23:10:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:10:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-15T23:10:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: osdmap e48: 8 total, 7 up, 8 in 2024-09-15T23:10:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:10:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='osd.7 [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:10:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:10:59.982 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:10:59 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:00.000 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:10:59.999+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1818062630 con 0x7f18040a3270 2024-09-15T23:11:00.274 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:10:59.950+0000 7fbbf052c640 -1 osd.7 0 waiting for initial osdmap 2024-09-15T23:11:00.274 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:10:59 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:10:59.959+0000 7fbbe834c640 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:11:00.679 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi067 2024-09-15T23:11:00.679 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi067 to 5696M 2024-09-15T23:11:00.679 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:11:00.680 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: osdmap e49: 8 total, 7 up, 8 in 2024-09-15T23:11:00.680 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:11:00.680 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:11:00.680 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: from='osd.7 ' entity='osd.7' 2024-09-15T23:11:00.680 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:11:00.680 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:11:00.680 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:00.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi067 2024-09-15T23:11:00.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi067 to 5696M 2024-09-15T23:11:00.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:11:00.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: osdmap e49: 8 total, 7 up, 8 in 2024-09-15T23:11:00.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:11:00.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:11:00.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: from='osd.7 ' entity='osd.7' 2024-09-15T23:11:00.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:11:00.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:11:00.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi067 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi067 to 5696M 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: osdmap e49: 8 total, 7 up, 8 in 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: from='osd.7 ' entity='osd.7' 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:11:01.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:01.739 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:01 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:11:01.740 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:01 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:11:01.740 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:01 smithi148 ceph-mon[35610]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:11:01.740 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:01 smithi148 ceph-mon[35610]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:11:01.740 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:01 smithi148 ceph-mon[35610]: osd.7 [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] boot 2024-09-15T23:11:01.740 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:01 smithi148 ceph-mon[35610]: osdmap e50: 8 total, 8 up, 8 in 2024-09-15T23:11:01.740 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:01 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:11:01.740 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:01 smithi148 ceph-mon[35610]: osdmap e51: 8 total, 8 up, 8 in 2024-09-15T23:11:01.895 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:01 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:11:01.895 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:01 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:11:01.895 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:01 smithi067 ceph-mon[35665]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:11:01.895 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:01 smithi067 ceph-mon[35665]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:11:01.895 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:01 smithi067 ceph-mon[35665]: osd.7 [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] boot 2024-09-15T23:11:01.895 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:01 smithi067 ceph-mon[35665]: osdmap e50: 8 total, 8 up, 8 in 2024-09-15T23:11:01.895 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:01 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:11:01.896 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:01 smithi067 ceph-mon[35665]: osdmap e51: 8 total, 8 up, 8 in 2024-09-15T23:11:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:01 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:11:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:01 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:11:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:01 smithi062 ceph-mon[30949]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-09-15T23:11:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:01 smithi062 ceph-mon[30949]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:11:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:01 smithi062 ceph-mon[30949]: osd.7 [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] boot 2024-09-15T23:11:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:01 smithi062 ceph-mon[30949]: osdmap e50: 8 total, 8 up, 8 in 2024-09-15T23:11:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:01 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-15T23:11:01.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:01 smithi062 ceph-mon[30949]: osdmap e51: 8 total, 8 up, 8 in 2024-09-15T23:11:02.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:02 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/3233313296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a81c2c5-9444-460b-b7a4-403598e61ca3"}]: dispatch 2024-09-15T23:11:02.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:02 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a81c2c5-9444-460b-b7a4-403598e61ca3"}]: dispatch 2024-09-15T23:11:02.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:02 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a81c2c5-9444-460b-b7a4-403598e61ca3"}]': finished 2024-09-15T23:11:02.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:02 smithi062 ceph-mon[30949]: osdmap e52: 9 total, 8 up, 9 in 2024-09-15T23:11:02.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:03.015 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:02 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/3233313296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a81c2c5-9444-460b-b7a4-403598e61ca3"}]: dispatch 2024-09-15T23:11:03.015 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:02 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a81c2c5-9444-460b-b7a4-403598e61ca3"}]: dispatch 2024-09-15T23:11:03.016 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:02 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a81c2c5-9444-460b-b7a4-403598e61ca3"}]': finished 2024-09-15T23:11:03.016 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:02 smithi148 ceph-mon[35610]: osdmap e52: 9 total, 8 up, 9 in 2024-09-15T23:11:03.016 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:03.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:02 smithi067 ceph-mon[35665]: from='client.? 172.21.15.148:0/3233313296' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a81c2c5-9444-460b-b7a4-403598e61ca3"}]: dispatch 2024-09-15T23:11:03.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:02 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a81c2c5-9444-460b-b7a4-403598e61ca3"}]: dispatch 2024-09-15T23:11:03.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:02 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a81c2c5-9444-460b-b7a4-403598e61ca3"}]': finished 2024-09-15T23:11:03.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:02 smithi067 ceph-mon[35665]: osdmap e52: 9 total, 8 up, 9 in 2024-09-15T23:11:03.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:03 smithi062 ceph-mon[30949]: pgmap v172: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:03 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/3572332609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:11:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:03 smithi062 ceph-mon[30949]: osdmap e53: 9 total, 8 up, 9 in 2024-09-15T23:11:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:03 smithi067 ceph-mon[35665]: pgmap v172: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:03 smithi067 ceph-mon[35665]: from='client.? 172.21.15.148:0/3572332609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:11:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:03 smithi067 ceph-mon[35665]: osdmap e53: 9 total, 8 up, 9 in 2024-09-15T23:11:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:03 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:03 smithi148 ceph-mon[35610]: pgmap v172: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:03 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/3572332609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:11:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:03 smithi148 ceph-mon[35610]: osdmap e53: 9 total, 8 up, 9 in 2024-09-15T23:11:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:05 smithi062 ceph-mon[30949]: pgmap v175: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:05 smithi062 ceph-mon[30949]: osdmap e54: 9 total, 8 up, 9 in 2024-09-15T23:11:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:05 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:05.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:05 smithi067 ceph-mon[35665]: pgmap v175: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:05.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:05 smithi067 ceph-mon[35665]: osdmap e54: 9 total, 8 up, 9 in 2024-09-15T23:11:05.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:05 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:05.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:05 smithi148 ceph-mon[35610]: pgmap v175: 1 pgs: 1 remapped; 577 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:05.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:05 smithi148 ceph-mon[35610]: osdmap e54: 9 total, 8 up, 9 in 2024-09-15T23:11:05.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:05 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:07.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:07 smithi148 ceph-mon[35610]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:07.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:07 smithi062 ceph-mon[30949]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:07.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:07 smithi067 ceph-mon[35665]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:09.699 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:09 smithi148 ceph-mon[35610]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:09.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:09 smithi062 ceph-mon[30949]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:09.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:09 smithi067 ceph-mon[35665]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:11.702 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:11 smithi148 ceph-mon[35610]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:11.702 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:11 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-15T23:11:11.702 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:11 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:11 smithi062 ceph-mon[30949]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:11 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-15T23:11:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:11 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:11 smithi067 ceph-mon[35665]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:11 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-09-15T23:11:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:11 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:12.707 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:12 smithi148 ceph-mon[35610]: Deploying daemon osd.8 on smithi148 2024-09-15T23:11:12.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:12 smithi062 ceph-mon[30949]: Deploying daemon osd.8 on smithi148 2024-09-15T23:11:12.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:12 smithi067 ceph-mon[35665]: Deploying daemon osd.8 on smithi148 2024-09-15T23:11:13.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:13 smithi062 ceph-mon[30949]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:13.755 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:13 smithi148 ceph-mon[35610]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:13.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:13 smithi067 ceph-mon[35665]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:15.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:15 smithi062 ceph-mon[30949]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:15.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:15 smithi067 ceph-mon[35665]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:15.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:15 smithi148 ceph-mon[35610]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:17.728 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:17 smithi148 ceph-mon[35610]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:17.728 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:17.728 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:17.728 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:17.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:17 smithi062 ceph-mon[30949]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:17.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:17.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:17.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:17 smithi067 ceph-mon[35665]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:19.559 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:19 smithi148 ceph-mon[35610]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:19.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:19 smithi062 ceph-mon[30949]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:19.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:19 smithi067 ceph-mon[35665]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:21.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:20 smithi148 ceph-mon[35610]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:21.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:21.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:21.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:21.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:21.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:21.088 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.087+0000 7f180f7fe640 1 -- 172.21.15.148:0/1079789686 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1818062630 con 0x7f18040a3270 2024-09-15T23:11:21.088 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 8 on host 'smithi148' 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.089+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18040a3270 msgr2=0x7f18040a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.089+0000 7f181cd41640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18040a3270 0x7f18040a5730 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f18181993f0 tx=0x7f180000b390 comp rx=0 tx=0).stop 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1818101c00 msgr2=0x7f1818197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1818101c00 0x7f1818197ed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f180800b450 tx=0x7f180800b910 comp rx=0 tx=0).stop 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 shutdown_connections 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18040a3270 0x7f18040a5730 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18181037c0 0x7f181819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1818102e00 0x7f1818198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 --2- 172.21.15.148:0/1079789686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1818101c00 0x7f1818197ed0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 >> 172.21.15.148:0/1079789686 conn(0x7f18180fd390 msgr2=0x7f1818106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 shutdown_connections 2024-09-15T23:11:21.091 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:11:21.090+0000 7f181cd41640 1 -- 172.21.15.148:0/1079789686 wait complete. 2024-09-15T23:11:21.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:20 smithi062 ceph-mon[30949]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:21.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:21.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:21.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:21.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:21.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:20 smithi067 ceph-mon[35665]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:21.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:21.786 DEBUG:teuthology.orchestra.run.smithi148:osd.8> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.8.service 2024-09-15T23:11:21.789 INFO:tasks.cephadm:Deploying osd.9 on smithi062 with /dev/vg_nvme/lv_1... 2024-09-15T23:11:21.789 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-15T23:11:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:21 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:21 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:21 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:22.060 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:11:22.181 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:21 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:22.182 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:21 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:22.182 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:21 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:22.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:21 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:22.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:21 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:22.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:21 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:23.246 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:23 smithi148 ceph-mon[35610]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:23.246 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:23 smithi148 ceph-mon[35610]: from='osd.8 [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:11:23.246 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:23 smithi148 ceph-mon[35610]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:11:23.247 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:11:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:11:22.970+0000 7f5971d4b640 -1 osd.8 0 log_to_monitors true 2024-09-15T23:11:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:23 smithi062 ceph-mon[30949]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:23 smithi062 ceph-mon[30949]: from='osd.8 [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:11:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:23 smithi062 ceph-mon[30949]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:11:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:23 smithi067 ceph-mon[35665]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:23 smithi067 ceph-mon[35665]: from='osd.8 [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:11:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:23 smithi067 ceph-mon[35665]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-09-15T23:11:24.289 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:11:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:11:24.106+0000 7f596dcce640 -1 osd.8 0 waiting for initial osdmap 2024-09-15T23:11:24.289 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:11:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:11:24.115+0000 7f5969af6640 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:11:24.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:24 smithi148 ceph-mon[35610]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-15T23:11:24.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:24 smithi148 ceph-mon[35610]: osdmap e55: 9 total, 8 up, 9 in 2024-09-15T23:11:24.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:24 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:24.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:24 smithi148 ceph-mon[35610]: from='osd.8 [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:11:24.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:24 smithi148 ceph-mon[35610]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:11:24.385 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:24 smithi062 ceph-mon[30949]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-15T23:11:24.385 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:24 smithi062 ceph-mon[30949]: osdmap e55: 9 total, 8 up, 9 in 2024-09-15T23:11:24.385 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:24 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:24.386 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:24 smithi062 ceph-mon[30949]: from='osd.8 [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:11:24.386 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:24 smithi062 ceph-mon[30949]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:11:24.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:24 smithi067 ceph-mon[35665]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-15T23:11:24.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:24 smithi067 ceph-mon[35665]: osdmap e55: 9 total, 8 up, 9 in 2024-09-15T23:11:24.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:24 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:24.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:24 smithi067 ceph-mon[35665]: from='osd.8 [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:11:24.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:24 smithi067 ceph-mon[35665]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:11:24.996 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:11:25.030 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_1 2024-09-15T23:11:25.289 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:11:25.411 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: osdmap e56: 9 total, 8 up, 9 in 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.412 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:25.413 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:25.413 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:25.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:11:25.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: osdmap e56: 9 total, 8 up, 9 in 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:25.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: osdmap e56: 9 total, 8 up, 9 in 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:25.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:25.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:25.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.764+0000 7ff25912e640 1 -- 172.21.15.62:0/242905616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff254101c00 msgr2=0x7ff254102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:25.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.764+0000 7ff25912e640 1 --2- 172.21.15.62:0/242905616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff254101c00 0x7ff254102000 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff244009900 tx=0x7ff24402a410 comp rx=0 tx=0).stop 2024-09-15T23:11:25.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.764+0000 7ff25912e640 1 -- 172.21.15.62:0/242905616 shutdown_connections 2024-09-15T23:11:25.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.764+0000 7ff25912e640 1 --2- 172.21.15.62:0/242905616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff2541037c0 0x7ff25410a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:25.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.764+0000 7ff25912e640 1 --2- 172.21.15.62:0/242905616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff254102e00 0x7ff254103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.764+0000 7ff25912e640 1 --2- 172.21.15.62:0/242905616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff254101c00 0x7ff254102000 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.764+0000 7ff25912e640 1 -- 172.21.15.62:0/242905616 >> 172.21.15.62:0/242905616 conn(0x7ff2540fd390 msgr2=0x7ff2540ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.764+0000 7ff25912e640 1 -- 172.21.15.62:0/242905616 shutdown_connections 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.764+0000 7ff25912e640 1 -- 172.21.15.62:0/242905616 wait complete. 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.765+0000 7ff25912e640 1 Processor -- start 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.765+0000 7ff25912e640 1 -- start start 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25912e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff254101c00 0x7ff254075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25912e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff254102e00 0x7ff2540745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25912e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2541037c0 0x7ff254074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25912e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff25410d890 con 0x7ff2541037c0 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25912e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff25410d710 con 0x7ff254102e00 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25912e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff25410da10 con 0x7ff254101c00 2024-09-15T23:11:25.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25359e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2541037c0 0x7ff254074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25359e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2541037c0 0x7ff254074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45578/0 (socket says 172.21.15.62:45578) 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25359e640 1 -- 172.21.15.62:0/1566487054 learned_addr learned my addr 172.21.15.62:0/1566487054 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff252d9d640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff254101c00 0x7ff254075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25259c640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff254102e00 0x7ff2540745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25359e640 1 -- 172.21.15.62:0/1566487054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff254101c00 msgr2=0x7ff254075f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25359e640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff254101c00 0x7ff254075f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25359e640 1 -- 172.21.15.62:0/1566487054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff254102e00 msgr2=0x7ff2540745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25359e640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff254102e00 0x7ff2540745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25359e640 1 -- 172.21.15.62:0/1566487054 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2540753e0 con 0x7ff2541037c0 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25259c640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff254102e00 0x7ff2540745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25359e640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2541037c0 0x7ff254074b20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00cad0 tx=0x7ff24c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff23bfff640 1 -- 172.21.15.62:0/1566487054 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c00ae20 con 0x7ff2541037c0 2024-09-15T23:11:25.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff23bfff640 1 -- 172.21.15.62:0/1566487054 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff24c005020 con 0x7ff2541037c0 2024-09-15T23:11:25.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.766+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2541a20a0 con 0x7ff2541037c0 2024-09-15T23:11:25.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.767+0000 7ff23bfff640 1 -- 172.21.15.62:0/1566487054 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c00f6f0 con 0x7ff2541037c0 2024-09-15T23:11:25.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.767+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2541a25b0 con 0x7ff2541037c0 2024-09-15T23:11:25.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.768+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff254109020 con 0x7ff2541037c0 2024-09-15T23:11:25.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.769+0000 7ff23bfff640 1 -- 172.21.15.62:0/1566487054 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff24c0051d0 con 0x7ff2541037c0 2024-09-15T23:11:25.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.769+0000 7ff23bfff640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff23c0a3240 0x7ff23c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:11:25.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.770+0000 7ff252d9d640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff23c0a3240 0x7ff23c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:11:25.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.770+0000 7ff23bfff640 1 -- 172.21.15.62:0/1566487054 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5862+0+0 (secure 0 0 0) 0x7ff24c0cf240 con 0x7ff2541037c0 2024-09-15T23:11:25.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.770+0000 7ff252d9d640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff23c0a3240 0x7ff23c0a5700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff244005ab0 tx=0x7ff244000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:11:25.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.774+0000 7ff23bfff640 1 -- 172.21.15.62:0/1566487054 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff24c014030 con 0x7ff2541037c0 2024-09-15T23:11:25.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:25.958+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff254062630 con 0x7ff23c0a3240 2024-09-15T23:11:26.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:11:26.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:11:26.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi148 2024-09-15T23:11:26.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi148 to 5696M 2024-09-15T23:11:26.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:26.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: osd.8 [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] boot 2024-09-15T23:11:26.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: osdmap e57: 9 total, 9 up, 9 in 2024-09-15T23:11:26.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:26.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:11:26.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:11:26.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:26 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:26.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:11:26.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:11:26.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi148 2024-09-15T23:11:26.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi148 to 5696M 2024-09-15T23:11:26.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:26.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: osd.8 [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] boot 2024-09-15T23:11:26.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: osdmap e57: 9 total, 9 up, 9 in 2024-09-15T23:11:26.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:26.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:11:26.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:11:26.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:26 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:26.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:11:26.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:11:26.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi148 2024-09-15T23:11:26.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi148 to 5696M 2024-09-15T23:11:26.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:26.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: osd.8 [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] boot 2024-09-15T23:11:26.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: osdmap e57: 9 total, 9 up, 9 in 2024-09-15T23:11:26.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-09-15T23:11:26.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:11:26.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:11:26.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:26 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:27 smithi062 ceph-mon[30949]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:27 smithi062 ceph-mon[30949]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:11:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:27 smithi062 ceph-mon[30949]: osdmap e58: 9 total, 9 up, 9 in 2024-09-15T23:11:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:27 smithi067 ceph-mon[35665]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:27 smithi067 ceph-mon[35665]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:11:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:27 smithi067 ceph-mon[35665]: osdmap e58: 9 total, 9 up, 9 in 2024-09-15T23:11:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:27 smithi148 ceph-mon[35610]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-09-15T23:11:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:27 smithi148 ceph-mon[35610]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:11:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:27 smithi148 ceph-mon[35610]: osdmap e58: 9 total, 9 up, 9 in 2024-09-15T23:11:28.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:28 smithi062 ceph-mon[30949]: osdmap e59: 9 total, 9 up, 9 in 2024-09-15T23:11:28.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:28 smithi067 ceph-mon[35665]: osdmap e59: 9 total, 9 up, 9 in 2024-09-15T23:11:28.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:28 smithi148 ceph-mon[35610]: osdmap e59: 9 total, 9 up, 9 in 2024-09-15T23:11:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:29 smithi062 ceph-mon[30949]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:29 smithi062 ceph-mon[30949]: osdmap e60: 9 total, 9 up, 9 in 2024-09-15T23:11:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:29 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/164916993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2220ce09-96e7-4fcf-af2a-2c7fa4081679"}]: dispatch 2024-09-15T23:11:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:29 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/164916993' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2220ce09-96e7-4fcf-af2a-2c7fa4081679"}]': finished 2024-09-15T23:11:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:29 smithi062 ceph-mon[30949]: osdmap e61: 10 total, 9 up, 10 in 2024-09-15T23:11:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:29.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:29 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1304473872' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:11:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:29 smithi067 ceph-mon[35665]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:29 smithi067 ceph-mon[35665]: osdmap e60: 9 total, 9 up, 9 in 2024-09-15T23:11:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:29 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/164916993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2220ce09-96e7-4fcf-af2a-2c7fa4081679"}]: dispatch 2024-09-15T23:11:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:29 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/164916993' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2220ce09-96e7-4fcf-af2a-2c7fa4081679"}]': finished 2024-09-15T23:11:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:29 smithi067 ceph-mon[35665]: osdmap e61: 10 total, 9 up, 10 in 2024-09-15T23:11:29.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:29.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:29 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1304473872' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:11:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:29 smithi148 ceph-mon[35610]: pgmap v193: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:29 smithi148 ceph-mon[35610]: osdmap e60: 9 total, 9 up, 9 in 2024-09-15T23:11:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:29 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/164916993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2220ce09-96e7-4fcf-af2a-2c7fa4081679"}]: dispatch 2024-09-15T23:11:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:29 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/164916993' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2220ce09-96e7-4fcf-af2a-2c7fa4081679"}]': finished 2024-09-15T23:11:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:29 smithi148 ceph-mon[35610]: osdmap e61: 10 total, 9 up, 10 in 2024-09-15T23:11:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:29 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1304473872' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:11:31.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:31 smithi067 ceph-mon[35665]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:31.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:31 smithi148 ceph-mon[35610]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:31.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:31 smithi062 ceph-mon[30949]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:33.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:33 smithi062 ceph-mon[30949]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-15T23:11:33.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:33 smithi067 ceph-mon[35665]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-15T23:11:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:33 smithi148 ceph-mon[35610]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-15T23:11:35.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:35 smithi062 ceph-mon[30949]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-15T23:11:35.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:35 smithi067 ceph-mon[35665]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-15T23:11:35.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:35 smithi148 ceph-mon[35610]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-15T23:11:37.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:37 smithi062 ceph-mon[30949]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:11:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:37 smithi067 ceph-mon[35665]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:11:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:37 smithi148 ceph-mon[35610]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-15T23:11:38.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-15T23:11:38.225 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:38.225 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:38 smithi062 ceph-mon[30949]: Deploying daemon osd.9 on smithi062 2024-09-15T23:11:38.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-15T23:11:38.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:38.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:38 smithi067 ceph-mon[35665]: Deploying daemon osd.9 on smithi062 2024-09-15T23:11:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-09-15T23:11:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:38 smithi148 ceph-mon[35610]: Deploying daemon osd.9 on smithi062 2024-09-15T23:11:39.332 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:39 smithi062 ceph-mon[30949]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-15T23:11:39.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:39 smithi067 ceph-mon[35665]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-15T23:11:39.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:39 smithi148 ceph-mon[35610]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-15T23:11:41.321 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:41 smithi062 ceph-mon[30949]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-15T23:11:41.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:41 smithi067 ceph-mon[35665]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-15T23:11:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:41 smithi148 ceph-mon[35610]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-15T23:11:43.444 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:43 smithi062 ceph-mon[30949]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:11:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:43 smithi067 ceph-mon[35665]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:11:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:43 smithi148 ceph-mon[35610]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-15T23:11:44.453 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:44.453 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:44.453 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:44.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:44.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:44.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:44.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:44.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:44.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:45.446 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:45 smithi062 ceph-mon[30949]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:45.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:45 smithi067 ceph-mon[35665]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:45.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:45 smithi148 ceph-mon[35610]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:47 smithi062 ceph-mon[30949]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:47 smithi067 ceph-mon[35665]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:47.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:47.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.525+0000 7ff23bfff640 1 -- 172.21.15.62:0/1566487054 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff254062630 con 0x7ff23c0a3240 2024-09-15T23:11:47.526 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 9 on host 'smithi062' 2024-09-15T23:11:47.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff23c0a3240 msgr2=0x7ff23c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff23c0a3240 0x7ff23c0a5700 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff244005ab0 tx=0x7ff244000bc0 comp rx=0 tx=0).stop 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2541037c0 msgr2=0x7ff254074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2541037c0 0x7ff254074b20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff24c00cad0 tx=0x7ff24c00cf90 comp rx=0 tx=0).stop 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 shutdown_connections 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff23c0a3240 0x7ff23c0a5700 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2541037c0 0x7ff254074b20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff254102e00 0x7ff2540745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 --2- 172.21.15.62:0/1566487054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff254101c00 0x7ff254075f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.528+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 >> 172.21.15.62:0/1566487054 conn(0x7ff2540fd390 msgr2=0x7ff2541061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.529+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 shutdown_connections 2024-09-15T23:11:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:11:47.529+0000 7ff25912e640 1 -- 172.21.15.62:0/1566487054 wait complete. 2024-09-15T23:11:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:47 smithi148 ceph-mon[35610]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:48.255 DEBUG:teuthology.orchestra.run.smithi062:osd.9> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.9.service 2024-09-15T23:11:48.257 INFO:tasks.cephadm:Deploying osd.10 on smithi067 with /dev/vg_nvme/lv_1... 2024-09-15T23:11:48.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-15T23:11:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:48.522 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:11:48.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:48.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:48.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:48.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:11:48.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:48.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:49.480 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:11:49 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:11:49.277+0000 7fcfb7987640 -1 osd.9 0 log_to_monitors true 2024-09-15T23:11:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:49 smithi067 ceph-mon[35665]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:49 smithi067 ceph-mon[35665]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-15T23:11:49.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:49 smithi062 ceph-mon[30949]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:49.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:49 smithi062 ceph-mon[30949]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-15T23:11:49.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:49 smithi148 ceph-mon[35610]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:49.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:49 smithi148 ceph-mon[35610]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-09-15T23:11:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:50 smithi067 ceph-mon[35665]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-15T23:11:50.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:50 smithi067 ceph-mon[35665]: osdmap e62: 10 total, 9 up, 10 in 2024-09-15T23:11:50.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:50 smithi067 ceph-mon[35665]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:11:50.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:50 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:50.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:50 smithi067 ceph-mon[35665]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:11:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:50 smithi148 ceph-mon[35610]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-15T23:11:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:50 smithi148 ceph-mon[35610]: osdmap e62: 10 total, 9 up, 10 in 2024-09-15T23:11:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:50 smithi148 ceph-mon[35610]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:11:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:50 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:50 smithi148 ceph-mon[35610]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:11:50.864 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:50 smithi062 ceph-mon[30949]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-15T23:11:50.865 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:50 smithi062 ceph-mon[30949]: osdmap e62: 10 total, 9 up, 10 in 2024-09-15T23:11:50.865 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:50 smithi062 ceph-mon[30949]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-09-15T23:11:50.865 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:50 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:50.865 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:50 smithi062 ceph-mon[30949]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-09-15T23:11:51.230 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:11:50 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:11:50.921+0000 7fcfb390a640 -1 osd.9 0 waiting for initial osdmap 2024-09-15T23:11:51.230 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:11:50 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:11:50.933+0000 7fcfab72a640 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:11:51.275 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-09-15T23:11:51.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_1 2024-09-15T23:11:51.559 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:11:51.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:51.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: osdmap e63: 10 total, 9 up, 10 in 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:51.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:51.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: osdmap e63: 10 total, 9 up, 10 in 2024-09-15T23:11:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.797 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:51.797 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: osdmap e63: 10 total, 9 up, 10 in 2024-09-15T23:11:51.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:51.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:51.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977]' entity='osd.9' 2024-09-15T23:11:51.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.798 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:11:51.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:51.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:51.799 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:11:51.800 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:11:52.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.069+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1681221180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e81037c0 msgr2=0x7fe8e810a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:52.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.069+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1681221180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e81037c0 0x7fe8e810a050 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0009900 tx=0x7fe8e002a410 comp rx=0 tx=0).stop 2024-09-15T23:11:52.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.069+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1681221180 shutdown_connections 2024-09-15T23:11:52.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.069+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1681221180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e81037c0 0x7fe8e810a050 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:52.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.069+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1681221180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e8102e00 0x7fe8e8103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:52.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.069+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1681221180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8e8101c00 0x7fe8e8102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.069+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1681221180 >> 172.21.15.67:0/1681221180 conn(0x7fe8e80fd390 msgr2=0x7fe8e80ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.069+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1681221180 shutdown_connections 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.069+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1681221180 wait complete. 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8efcb9640 1 Processor -- start 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8efcb9640 1 -- start start 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8efcb9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e8101c00 0x7fe8e8197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8efcb9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e8102e00 0x7fe8e8198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8efcb9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8e81037c0 0x7fe8e819f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8efcb9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e810d860 con 0x7fe8e8101c00 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8efcb9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e810d6e0 con 0x7fe8e8102e00 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8efcb9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e810d9e0 con 0x7fe8e81037c0 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8ed254640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e8102e00 0x7fe8e8198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:11:52.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8ed254640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e8102e00 0x7fe8e8198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48592/0 (socket says 172.21.15.67:48592) 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8ed254640 1 -- 172.21.15.67:0/1753505101 learned_addr learned my addr 172.21.15.67:0/1753505101 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8eda55640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e8101c00 0x7fe8e8197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.070+0000 7fe8ee256640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8e81037c0 0x7fe8e819f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8ed254640 1 -- 172.21.15.67:0/1753505101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8e81037c0 msgr2=0x7fe8e819f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8ed254640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8e81037c0 0x7fe8e819f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8ed254640 1 -- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e8101c00 msgr2=0x7fe8e8197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8ed254640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e8101c00 0x7fe8e8197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8ed254640 1 -- 172.21.15.67:0/1753505101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e819fa00 con 0x7fe8e8102e00 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8ed254640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e8102e00 0x7fe8e8198310 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00d950 tx=0x7fe8dc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8e6ffd640 1 -- 172.21.15.67:0/1753505101 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc017030 con 0x7fe8e8102e00 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8e6ffd640 1 -- 172.21.15.67:0/1753505101 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8dc004020 con 0x7fe8e8102e00 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e819fcf0 con 0x7fe8e8102e00 2024-09-15T23:11:52.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8e6ffd640 1 -- 172.21.15.67:0/1753505101 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc0047a0 con 0x7fe8e8102e00 2024-09-15T23:11:52.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.071+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e81a0220 con 0x7fe8e8102e00 2024-09-15T23:11:52.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.072+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e8108f30 con 0x7fe8e8102e00 2024-09-15T23:11:52.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.073+0000 7fe8e6ffd640 1 -- 172.21.15.67:0/1753505101 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe8dc0041d0 con 0x7fe8e8102e00 2024-09-15T23:11:52.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.074+0000 7fe8e6ffd640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8d00a32f0 0x7fe8d00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:11:52.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.074+0000 7fe8e6ffd640 1 -- 172.21.15.67:0/1753505101 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 6261+0+0 (secure 0 0 0) 0x7fe8dc0cf0a0 con 0x7fe8e8102e00 2024-09-15T23:11:52.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.074+0000 7fe8eda55640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8d00a32f0 0x7fe8d00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:11:52.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.074+0000 7fe8eda55640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8d00a32f0 0x7fe8d00a57b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8102c60 tx=0x7fe8d8007600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:11:52.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.077+0000 7fe8e6ffd640 1 -- 172.21.15.67:0/1753505101 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe8dc09c640 con 0x7fe8e8102e00 2024-09-15T23:11:52.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:11:52.261+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe8e8062630 con 0x7fe8d00a32f0 2024-09-15T23:11:52.842 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:11:52.842 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:11:52.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi062 2024-09-15T23:11:52.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi062 to 4272M 2024-09-15T23:11:52.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:52.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] boot 2024-09-15T23:11:52.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: osdmap e64: 10 total, 10 up, 10 in 2024-09-15T23:11:52.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:52.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:11:52.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:11:52.843 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:52 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:52.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:11:52.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:11:52.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi062 2024-09-15T23:11:52.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi062 to 4272M 2024-09-15T23:11:52.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:52.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] boot 2024-09-15T23:11:52.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: osdmap e64: 10 total, 10 up, 10 in 2024-09-15T23:11:52.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:52.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:11:52.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:11:52.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:52 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:53.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:11:53.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:11:53.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi062 2024-09-15T23:11:53.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi062 to 4272M 2024-09-15T23:11:53.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:53.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: osd.9 [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] boot 2024-09-15T23:11:53.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: osdmap e64: 10 total, 10 up, 10 in 2024-09-15T23:11:53.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-09-15T23:11:53.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:11:53.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:11:53.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:52 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:11:53.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:53 smithi067 ceph-mon[35665]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:53.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:53 smithi067 ceph-mon[35665]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:11:53.885 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:53 smithi067 ceph-mon[35665]: osdmap e65: 10 total, 10 up, 10 in 2024-09-15T23:11:53.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:53 smithi062 ceph-mon[30949]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:53.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:53 smithi062 ceph-mon[30949]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:11:53.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:53 smithi062 ceph-mon[30949]: osdmap e65: 10 total, 10 up, 10 in 2024-09-15T23:11:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:53 smithi148 ceph-mon[35610]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-09-15T23:11:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:53 smithi148 ceph-mon[35610]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:11:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:53 smithi148 ceph-mon[35610]: osdmap e65: 10 total, 10 up, 10 in 2024-09-15T23:11:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:55 smithi067 ceph-mon[35665]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:11:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:55 smithi067 ceph-mon[35665]: from='client.? 172.21.15.67:0/994580185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bb9d76c-f64e-4792-b7c4-9e873a5837c5"}]: dispatch 2024-09-15T23:11:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:55 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bb9d76c-f64e-4792-b7c4-9e873a5837c5"}]: dispatch 2024-09-15T23:11:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:55 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bb9d76c-f64e-4792-b7c4-9e873a5837c5"}]': finished 2024-09-15T23:11:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:55 smithi067 ceph-mon[35665]: osdmap e66: 11 total, 10 up, 11 in 2024-09-15T23:11:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:11:55.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:55 smithi067 ceph-mon[35665]: from='client.? 172.21.15.67:0/2424959124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:11:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:55 smithi062 ceph-mon[30949]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:11:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:55 smithi062 ceph-mon[30949]: from='client.? 172.21.15.67:0/994580185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bb9d76c-f64e-4792-b7c4-9e873a5837c5"}]: dispatch 2024-09-15T23:11:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:55 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bb9d76c-f64e-4792-b7c4-9e873a5837c5"}]: dispatch 2024-09-15T23:11:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:55 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bb9d76c-f64e-4792-b7c4-9e873a5837c5"}]': finished 2024-09-15T23:11:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:55 smithi062 ceph-mon[30949]: osdmap e66: 11 total, 10 up, 11 in 2024-09-15T23:11:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:11:55.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:55 smithi062 ceph-mon[30949]: from='client.? 172.21.15.67:0/2424959124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:11:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:55 smithi148 ceph-mon[35610]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:11:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:55 smithi148 ceph-mon[35610]: from='client.? 172.21.15.67:0/994580185' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bb9d76c-f64e-4792-b7c4-9e873a5837c5"}]: dispatch 2024-09-15T23:11:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:55 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bb9d76c-f64e-4792-b7c4-9e873a5837c5"}]: dispatch 2024-09-15T23:11:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:55 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bb9d76c-f64e-4792-b7c4-9e873a5837c5"}]': finished 2024-09-15T23:11:56.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:55 smithi148 ceph-mon[35610]: osdmap e66: 11 total, 10 up, 11 in 2024-09-15T23:11:56.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:11:56.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:55 smithi148 ceph-mon[35610]: from='client.? 172.21.15.67:0/2424959124' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:11:57.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:57 smithi062 ceph-mon[30949]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:11:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:57 smithi067 ceph-mon[35665]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:11:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:57 smithi148 ceph-mon[35610]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:11:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:11:59 smithi062 ceph-mon[30949]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:00.021 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:11:59 smithi067 ceph-mon[35665]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:11:59 smithi148 ceph-mon[35610]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:01.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:01 smithi067 ceph-mon[35665]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:01.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:01 smithi148 ceph-mon[35610]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:01 smithi062 ceph-mon[30949]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:03.918 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:03 smithi067 ceph-mon[35665]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:03.919 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:03 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-15T23:12:03.919 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:03 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:03 smithi062 ceph-mon[30949]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-15T23:12:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:03 smithi148 ceph-mon[35610]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-09-15T23:12:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:04.823 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:04 smithi067 ceph-mon[35665]: Deploying daemon osd.10 on smithi067 2024-09-15T23:12:04.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:04 smithi062 ceph-mon[30949]: Deploying daemon osd.10 on smithi067 2024-09-15T23:12:05.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:04 smithi148 ceph-mon[35610]: Deploying daemon osd.10 on smithi067 2024-09-15T23:12:05.897 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:05 smithi067 ceph-mon[35665]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:05.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:05 smithi062 ceph-mon[30949]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:05 smithi148 ceph-mon[35610]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:07.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:07 smithi067 ceph-mon[35665]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:07.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:07 smithi062 ceph-mon[30949]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:07 smithi148 ceph-mon[35610]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:09.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:09 smithi067 ceph-mon[35665]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:09.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:09 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:09.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:09 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:09.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:09 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:09.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:09 smithi062 ceph-mon[30949]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:09.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:09 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:09.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:09 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:09.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:09 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:10.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:09 smithi148 ceph-mon[35610]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:10.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:09 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:10.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:09 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:10.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:09 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:11 smithi067 ceph-mon[35665]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:11 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:11 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:11 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:11 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:11.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:11 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:11.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:11 smithi062 ceph-mon[30949]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:11.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:11 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:11.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:11 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:11.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:11 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:11.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:11 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:11.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:11 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:11 smithi148 ceph-mon[35610]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:11 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:11 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:12.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:11 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:12.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:11 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:12.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:11 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:12.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.133+0000 7fe8e6ffd640 1 -- 172.21.15.67:0/1753505101 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe8e8062630 con 0x7fe8d00a32f0 2024-09-15T23:12:12.134 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 10 on host 'smithi067' 2024-09-15T23:12:12.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8d00a32f0 msgr2=0x7fe8d00a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8d00a32f0 0x7fe8d00a57b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8102c60 tx=0x7fe8d8007600 comp rx=0 tx=0).stop 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e8102e00 msgr2=0x7fe8e8198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e8102e00 0x7fe8e8198310 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00d950 tx=0x7fe8dc00de10 comp rx=0 tx=0).stop 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 shutdown_connections 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8d00a32f0 0x7fe8d00a57b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe8e81037c0 0x7fe8e819f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8e8102e00 0x7fe8e8198310 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 --2- 172.21.15.67:0/1753505101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8e8101c00 0x7fe8e8197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.135+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 >> 172.21.15.67:0/1753505101 conn(0x7fe8e80fd390 msgr2=0x7fe8e8106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.136+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 shutdown_connections 2024-09-15T23:12:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-09-15T23:12:12.136+0000 7fe8efcb9640 1 -- 172.21.15.67:0/1753505101 wait complete. 2024-09-15T23:12:12.748 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:12 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:12.748 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:12 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:12.748 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:12 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:12.776 DEBUG:teuthology.orchestra.run.smithi067:osd.10> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.10.service 2024-09-15T23:12:12.778 INFO:tasks.cephadm:Deploying osd.11 on smithi148 with /dev/vg_nvme/lv_1... 2024-09-15T23:12:12.778 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-15T23:12:12.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:12 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:12.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:12 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:12.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:12 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:13.034 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:12:13.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:12 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:13.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:12 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:13.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:12 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:13.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:13 smithi148 ceph-mon[35610]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:13.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:13 smithi062 ceph-mon[30949]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:13.990 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:13 smithi067 ceph-mon[35665]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:14.255 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:12:14 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:12:14.226+0000 7f8fc55b3640 -1 osd.10 0 log_to_monitors true 2024-09-15T23:12:14.851 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:14 smithi148 ceph-mon[35610]: from='osd.10 [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:12:14.851 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:14 smithi148 ceph-mon[35610]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:12:14.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:14 smithi062 ceph-mon[30949]: from='osd.10 [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:12:14.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:14 smithi062 ceph-mon[30949]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:12:15.005 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:14 smithi067 ceph-mon[35665]: from='osd.10 [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:12:15.005 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:14 smithi067 ceph-mon[35665]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-09-15T23:12:15.715 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:15 smithi067 ceph-mon[35665]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:15.715 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:15 smithi067 ceph-mon[35665]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-15T23:12:15.715 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:15 smithi067 ceph-mon[35665]: from='osd.10 [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:12:15.715 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:15 smithi067 ceph-mon[35665]: osdmap e67: 11 total, 10 up, 11 in 2024-09-15T23:12:15.715 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:15 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:15.716 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:15 smithi067 ceph-mon[35665]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:12:15.772 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:15 smithi148 ceph-mon[35610]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:15.772 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:15 smithi148 ceph-mon[35610]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-15T23:12:15.772 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:15 smithi148 ceph-mon[35610]: from='osd.10 [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:12:15.773 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:15 smithi148 ceph-mon[35610]: osdmap e67: 11 total, 10 up, 11 in 2024-09-15T23:12:15.773 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:15 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:15.773 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:15 smithi148 ceph-mon[35610]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:12:15.776 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-09-15T23:12:15.805 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi148:vg_nvme/lv_1 2024-09-15T23:12:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:15 smithi062 ceph-mon[30949]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:15 smithi062 ceph-mon[30949]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-15T23:12:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:15 smithi062 ceph-mon[30949]: from='osd.10 [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:12:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:15 smithi062 ceph-mon[30949]: osdmap e67: 11 total, 10 up, 11 in 2024-09-15T23:12:15.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:15 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:15.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:15 smithi062 ceph-mon[30949]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-09-15T23:12:16.063 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:12:16.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:12:16.005+0000 7f8fc1536640 -1 osd.10 0 waiting for initial osdmap 2024-09-15T23:12:16.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:12:16.015+0000 7f8fb9356640 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:12:16.609 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.575+0000 7fe204f9d640 1 -- 172.21.15.148:0/3232768936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200100950 msgr2=0x7fe200100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:16.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.575+0000 7fe204f9d640 1 --2- 172.21.15.148:0/3232768936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200100950 0x7fe200100d50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009900 tx=0x7fe1f002a410 comp rx=0 tx=0).stop 2024-09-15T23:12:16.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.575+0000 7fe204f9d640 1 -- 172.21.15.148:0/3232768936 shutdown_connections 2024-09-15T23:12:16.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.575+0000 7fe204f9d640 1 --2- 172.21.15.148:0/3232768936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200102510 0x7fe200108da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:16.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.575+0000 7fe204f9d640 1 --2- 172.21.15.148:0/3232768936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe200101b50 0x7fe200101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:16.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.575+0000 7fe204f9d640 1 --2- 172.21.15.148:0/3232768936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200100950 0x7fe200100d50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:16.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.575+0000 7fe204f9d640 1 -- 172.21.15.148:0/3232768936 >> 172.21.15.148:0/3232768936 conn(0x7fe2000fc0e0 msgr2=0x7fe2000fe520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:16.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.575+0000 7fe204f9d640 1 -- 172.21.15.148:0/3232768936 shutdown_connections 2024-09-15T23:12:16.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.575+0000 7fe204f9d640 1 -- 172.21.15.148:0/3232768936 wait complete. 2024-09-15T23:12:16.610 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe204f9d640 1 Processor -- start 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe204f9d640 1 -- start start 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe204f9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200100950 0x7fe20010b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe204f9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe200101b50 0x7fe200109920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe204f9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200102510 0x7fe200109e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe204f9d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe20010d700 con 0x7fe200100950 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe204f9d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe20010d580 con 0x7fe200101b50 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe204f9d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe20010d880 con 0x7fe200102510 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1fffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200102510 0x7fe200109e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1fffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200102510 0x7fe200109e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:41196/0 (socket says 172.21.15.148:41196) 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1fffff640 1 -- 172.21.15.148:0/2484029148 learned_addr learned my addr 172.21.15.148:0/2484029148 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1f7fff640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe200101b50 0x7fe200109920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:16.611 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1ff7fe640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200100950 0x7fe20010b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1fffff640 1 -- 172.21.15.148:0/2484029148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe200101b50 msgr2=0x7fe200109920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1fffff640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe200101b50 0x7fe200109920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1fffff640 1 -- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200100950 msgr2=0x7fe20010b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1fffff640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200100950 0x7fe20010b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1fffff640 1 -- 172.21.15.148:0/2484029148 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20010a720 con 0x7fe200102510 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.576+0000 7fe1ff7fe640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200100950 0x7fe20010b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.577+0000 7fe1fffff640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200102510 0x7fe200109e60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800cce0 tx=0x7fe1f800a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.577+0000 7fe1fd7fa640 1 -- 172.21.15.148:0/2484029148 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800f450 con 0x7fe200102510 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.577+0000 7fe1fd7fa640 1 -- 172.21.15.148:0/2484029148 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f800fad0 con 0x7fe200102510 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.577+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2001a1e20 con 0x7fe200102510 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.577+0000 7fe1fd7fa640 1 -- 172.21.15.148:0/2484029148 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8018c10 con 0x7fe200102510 2024-09-15T23:12:16.612 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.577+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2001a22d0 con 0x7fe200102510 2024-09-15T23:12:16.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.578+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe200107cd0 con 0x7fe200102510 2024-09-15T23:12:16.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.580+0000 7fe1fd7fa640 1 -- 172.21.15.148:0/2484029148 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe1f800f5f0 con 0x7fe200102510 2024-09-15T23:12:16.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.580+0000 7fe1fd7fa640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1ec0a32c0 0x7fe1ec0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:16.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.581+0000 7fe1fd7fa640 1 -- 172.21.15.148:0/2484029148 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6414+0+0 (secure 0 0 0) 0x7fe1f80cf8a0 con 0x7fe200102510 2024-09-15T23:12:16.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.581+0000 7fe1ff7fe640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1ec0a32c0 0x7fe1ec0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:16.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.581+0000 7fe1ff7fe640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1ec0a32c0 0x7fe1ec0a5780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0000c00 tx=0x7fe1f0009560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:16.613 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.584+0000 7fe1fd7fa640 1 -- 172.21.15.148:0/2484029148 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe1f8014030 con 0x7fe200102510 2024-09-15T23:12:16.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:16.761+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe200062630 con 0x7fe1ec0a32c0 2024-09-15T23:12:16.763 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: osdmap e68: 11 total, 10 up, 11 in 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:16.764 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:16.765 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:16.765 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:16.765 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:16 smithi148 ceph-mon[35610]: from='osd.10 ' entity='osd.10' 2024-09-15T23:12:16.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:12:16.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: osdmap e68: 11 total, 10 up, 11 in 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:16.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:16.982 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:16.982 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:16.982 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:16 smithi062 ceph-mon[30949]: from='osd.10 ' entity='osd.10' 2024-09-15T23:12:17.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: osdmap e68: 11 total, 10 up, 11 in 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:17.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:17.026 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:17.026 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:17.026 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:17.026 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:16 smithi067 ceph-mon[35665]: from='osd.10 ' entity='osd.10' 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi067 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi067 to 4272M 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: osd.10 [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] boot 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: osdmap e69: 11 total, 11 up, 11 in 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:12:17.793 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:17.794 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:17 smithi148 ceph-mon[35610]: osdmap e70: 11 total, 11 up, 11 in 2024-09-15T23:12:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:12:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:12:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi067 2024-09-15T23:12:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi067 to 4272M 2024-09-15T23:12:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: osd.10 [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] boot 2024-09-15T23:12:17.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: osdmap e69: 11 total, 11 up, 11 in 2024-09-15T23:12:17.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:17.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:12:17.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:12:17.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:17.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:17 smithi062 ceph-mon[30949]: osdmap e70: 11 total, 11 up, 11 in 2024-09-15T23:12:18.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:12:18.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi067 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi067 to 4272M 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: osd.10 [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] boot 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: osdmap e69: 11 total, 11 up, 11 in 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:18.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:17 smithi067 ceph-mon[35665]: osdmap e70: 11 total, 11 up, 11 in 2024-09-15T23:12:18.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:18 smithi148 ceph-mon[35610]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:12:18.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:18 smithi148 ceph-mon[35610]: osdmap e71: 11 total, 11 up, 11 in 2024-09-15T23:12:18.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:18 smithi062 ceph-mon[30949]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:12:18.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:18 smithi062 ceph-mon[30949]: osdmap e71: 11 total, 11 up, 11 in 2024-09-15T23:12:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:18 smithi067 ceph-mon[35665]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi148:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:12:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:18 smithi067 ceph-mon[35665]: osdmap e71: 11 total, 11 up, 11 in 2024-09-15T23:12:19.861 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:19 smithi148 ceph-mon[35610]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:19.861 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:19 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/3719545471' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08c325c-1728-416e-893e-35954d5d8cd7"}]: dispatch 2024-09-15T23:12:19.861 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:19 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08c325c-1728-416e-893e-35954d5d8cd7"}]: dispatch 2024-09-15T23:12:19.861 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:19 smithi148 ceph-mon[35610]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c08c325c-1728-416e-893e-35954d5d8cd7"}]': finished 2024-09-15T23:12:19.861 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:19 smithi148 ceph-mon[35610]: osdmap e72: 12 total, 11 up, 12 in 2024-09-15T23:12:19.861 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:19 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:19 smithi067 ceph-mon[35665]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:19 smithi067 ceph-mon[35665]: from='client.? 172.21.15.148:0/3719545471' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08c325c-1728-416e-893e-35954d5d8cd7"}]: dispatch 2024-09-15T23:12:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:19 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08c325c-1728-416e-893e-35954d5d8cd7"}]: dispatch 2024-09-15T23:12:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:19 smithi067 ceph-mon[35665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c08c325c-1728-416e-893e-35954d5d8cd7"}]': finished 2024-09-15T23:12:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:19 smithi067 ceph-mon[35665]: osdmap e72: 12 total, 11 up, 12 in 2024-09-15T23:12:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:19 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:19 smithi062 ceph-mon[30949]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:19 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/3719545471' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08c325c-1728-416e-893e-35954d5d8cd7"}]: dispatch 2024-09-15T23:12:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:19 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c08c325c-1728-416e-893e-35954d5d8cd7"}]: dispatch 2024-09-15T23:12:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:19 smithi062 ceph-mon[30949]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c08c325c-1728-416e-893e-35954d5d8cd7"}]': finished 2024-09-15T23:12:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:19 smithi062 ceph-mon[30949]: osdmap e72: 12 total, 11 up, 12 in 2024-09-15T23:12:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:19 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:20.941 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:20 smithi067 ceph-mon[35665]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:20.941 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:20 smithi067 ceph-mon[35665]: from='client.? 172.21.15.148:0/1680649171' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:12:20.941 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:20 smithi067 ceph-mon[35665]: osdmap e73: 12 total, 11 up, 12 in 2024-09-15T23:12:20.941 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:20.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:20 smithi062 ceph-mon[30949]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:20.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:20 smithi062 ceph-mon[30949]: from='client.? 172.21.15.148:0/1680649171' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:12:20.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:20 smithi062 ceph-mon[30949]: osdmap e73: 12 total, 11 up, 12 in 2024-09-15T23:12:20.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:21.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:20 smithi148 ceph-mon[35610]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:21.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:20 smithi148 ceph-mon[35610]: from='client.? 172.21.15.148:0/1680649171' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-15T23:12:21.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:20 smithi148 ceph-mon[35610]: osdmap e73: 12 total, 11 up, 12 in 2024-09-15T23:12:21.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:22.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:22 smithi062 ceph-mon[30949]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-09-15T23:12:23.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:22 smithi067 ceph-mon[35665]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-09-15T23:12:23.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:22 smithi148 ceph-mon[35610]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-09-15T23:12:25.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:24 smithi067 ceph-mon[35665]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T23:12:25.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:24 smithi148 ceph-mon[35610]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T23:12:25.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:24 smithi062 ceph-mon[30949]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-15T23:12:27.015 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:26 smithi148 ceph-mon[35610]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-15T23:12:27.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:26 smithi067 ceph-mon[35665]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-15T23:12:27.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:26 smithi062 ceph-mon[30949]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-09-15T23:12:28.880 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:28 smithi148 ceph-mon[35610]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-15T23:12:28.880 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-15T23:12:28.880 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:28.881 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:28 smithi148 ceph-mon[35610]: Deploying daemon osd.11 on smithi148 2024-09-15T23:12:29.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:28 smithi067 ceph-mon[35665]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-15T23:12:29.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-15T23:12:29.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:29.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:28 smithi067 ceph-mon[35665]: Deploying daemon osd.11 on smithi148 2024-09-15T23:12:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:28 smithi062 ceph-mon[30949]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-15T23:12:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-09-15T23:12:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:28 smithi062 ceph-mon[30949]: Deploying daemon osd.11 on smithi148 2024-09-15T23:12:30.943 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:30 smithi148 ceph-mon[35610]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T23:12:31.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:30 smithi067 ceph-mon[35665]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T23:12:31.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:30 smithi062 ceph-mon[30949]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-15T23:12:32.852 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:32 smithi148 ceph-mon[35610]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-15T23:12:33.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:32 smithi067 ceph-mon[35665]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-15T23:12:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:32 smithi062 ceph-mon[30949]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-09-15T23:12:34.951 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:34 smithi148 ceph-mon[35610]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:34.951 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:34.951 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:34.951 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:35.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:34 smithi067 ceph-mon[35665]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:35.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:35.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:35.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:35.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:34 smithi062 ceph-mon[30949]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:35.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:35.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:35.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:36.973 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:36 smithi148 ceph-mon[35610]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:37.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:36 smithi067 ceph-mon[35665]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:37.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:36 smithi062 ceph-mon[30949]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:37.954 INFO:teuthology.orchestra.run.smithi148.stdout:Created osd(s) 11 on host 'smithi148' 2024-09-15T23:12:37.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.950+0000 7fe1fd7fa640 1 -- 172.21.15.148:0/2484029148 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe200062630 con 0x7fe1ec0a32c0 2024-09-15T23:12:37.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1ec0a32c0 msgr2=0x7fe1ec0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:37.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1ec0a32c0 0x7fe1ec0a5780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0000c00 tx=0x7fe1f0009560 comp rx=0 tx=0).stop 2024-09-15T23:12:37.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200102510 msgr2=0x7fe200109e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:37.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200102510 0x7fe200109e60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800cce0 tx=0x7fe1f800a620 comp rx=0 tx=0).stop 2024-09-15T23:12:37.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 shutdown_connections 2024-09-15T23:12:37.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1ec0a32c0 0x7fe1ec0a5780 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:37.955 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe200102510 0x7fe200109e60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:37.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe200101b50 0x7fe200109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:37.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 --2- 172.21.15.148:0/2484029148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe200100950 0x7fe20010b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:37.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.952+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 >> 172.21.15.148:0/2484029148 conn(0x7fe2000fc0e0 msgr2=0x7fe200104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:37.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.953+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 shutdown_connections 2024-09-15T23:12:37.956 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:37.953+0000 7fe204f9d640 1 -- 172.21.15.148:0/2484029148 wait complete. 2024-09-15T23:12:38.648 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.648 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.649 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:38.649 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:38.649 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.649 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:38.649 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.649 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.680 DEBUG:teuthology.orchestra.run.smithi148:osd.11> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.11.service 2024-09-15T23:12:38.682 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-15T23:12:38.682 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-15T23:12:38.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:38.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:38.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:38.712 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.713 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:38.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:38.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:38.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:38.951 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:12:39.334 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:39 smithi062 ceph-mon[30949]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:39.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 -- 172.21.15.62:0/3382558384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 msgr2=0x7fefd4102010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:39.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 --2- 172.21.15.62:0/3382558384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 0x7fefd4102010 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009ca0 tx=0x7fefc402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 -- 172.21.15.62:0/3382558384 shutdown_connections 2024-09-15T23:12:39.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 --2- 172.21.15.62:0/3382558384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd41037d0 0x7fefd410a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 --2- 172.21.15.62:0/3382558384 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefd4102e10 0x7fefd4103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 --2- 172.21.15.62:0/3382558384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 0x7fefd4102010 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 -- 172.21.15.62:0/3382558384 >> 172.21.15.62:0/3382558384 conn(0x7fefd40fd3c0 msgr2=0x7fefd40ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 -- 172.21.15.62:0/3382558384 shutdown_connections 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 -- 172.21.15.62:0/3382558384 wait complete. 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 Processor -- start 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 -- start start 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 0x7fefd4075f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefd4102e10 0x7fefd40745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd41037d0 0x7fefd4074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.416+0000 7fefdac5f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fefd410d870 con 0x7fefd4101c10 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefdac5f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fefd410d6f0 con 0x7fefd41037d0 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefdac5f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fefd410d9f0 con 0x7fefd4102e10 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd89fb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 0x7fefd4075f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:39.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd89fb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 0x7fefd4075f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45012/0 (socket says 172.21.15.62:45012) 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd89fb640 1 -- 172.21.15.62:0/2832563069 learned_addr learned my addr 172.21.15.62:0/2832563069 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd3fff640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefd4102e10 0x7fefd40745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd91fc640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd41037d0 0x7fefd4074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd89fb640 1 -- 172.21.15.62:0/2832563069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefd4102e10 msgr2=0x7fefd40745e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd89fb640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefd4102e10 0x7fefd40745e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd89fb640 1 -- 172.21.15.62:0/2832563069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd41037d0 msgr2=0x7fefd4074b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd89fb640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd41037d0 0x7fefd4074b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd89fb640 1 -- 172.21.15.62:0/2832563069 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd40753e0 con 0x7fefd4101c10 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd89fb640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 0x7fefd4075f90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009bf0 tx=0x7fefc4031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd1ffb640 1 -- 172.21.15.62:0/2832563069 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc402fae0 con 0x7fefd4101c10 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd41a2030 con 0x7fefd4101c10 2024-09-15T23:12:39.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd1ffb640 1 -- 172.21.15.62:0/2832563069 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc402fc80 con 0x7fefd4101c10 2024-09-15T23:12:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.417+0000 7fefd1ffb640 1 -- 172.21.15.62:0/2832563069 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc40467e0 con 0x7fefd4101c10 2024-09-15T23:12:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.418+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd41a2540 con 0x7fefd4101c10 2024-09-15T23:12:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.419+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd4102010 con 0x7fefd4101c10 2024-09-15T23:12:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.420+0000 7fefd1ffb640 1 -- 172.21.15.62:0/2832563069 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fefc4030b30 con 0x7fefd4101c10 2024-09-15T23:12:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.420+0000 7fefd1ffb640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefbc0a3240 0x7fefbc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.420+0000 7fefd3fff640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefbc0a3240 0x7fefbc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.420+0000 7fefd1ffb640 1 -- 172.21.15.62:0/2832563069 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 6809+0+0 (secure 0 0 0) 0x7fefc40f0770 con 0x7fefd4101c10 2024-09-15T23:12:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.421+0000 7fefd3fff640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefbc0a3240 0x7fefbc0a5700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fefd4075d10 tx=0x7fefc8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:39.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.425+0000 7fefd1ffb640 1 -- 172.21.15.62:0/2832563069 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fefc40bda70 con 0x7fefd4101c10 2024-09-15T23:12:39.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.604+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fefd402db00 con 0x7fefd4101c10 2024-09-15T23:12:39.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.604+0000 7fefd1ffb640 1 -- 172.21.15.62:0/2832563069 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) ==== 74+0+133 (secure 0 0 0) 0x7fefc40bd370 con 0x7fefd4101c10 2024-09-15T23:12:39.606 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.606+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefbc0a3240 msgr2=0x7fefbc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.606+0000 7fefdac5f640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefbc0a3240 0x7fefbc0a5700 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fefd4075d10 tx=0x7fefc8005f90 comp rx=0 tx=0).stop 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.606+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 msgr2=0x7fefd4075f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.606+0000 7fefdac5f640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 0x7fefd4075f90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009bf0 tx=0x7fefc4031ee0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.607+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 shutdown_connections 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.607+0000 7fefdac5f640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefbc0a3240 0x7fefbc0a5700 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.607+0000 7fefdac5f640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefd41037d0 0x7fefd4074b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.607+0000 7fefdac5f640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefd4102e10 0x7fefd40745e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.607+0000 7fefdac5f640 1 --2- 172.21.15.62:0/2832563069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefd4101c10 0x7fefd4075f90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.607+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 >> 172.21.15.62:0/2832563069 conn(0x7fefd40fd3c0 msgr2=0x7fefd4106230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:39.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.607+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 shutdown_connections 2024-09-15T23:12:39.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:39.607+0000 7fefdac5f640 1 -- 172.21.15.62:0/2832563069 wait complete. 2024-09-15T23:12:39.765 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:39 smithi148 ceph-mon[35610]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:39 smithi067 ceph-mon[35665]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:40.441 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":73,"num_osds":12,"num_up_osds":11,"osd_up_since":1726441936,"num_in_osds":12,"osd_in_since":1726441939,"num_remapped_pgs":0} 2024-09-15T23:12:40.474 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:40 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2832563069' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:12:40.475 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:40 smithi148 ceph-mon[35610]: from='osd.11 [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:12:40.475 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:40 smithi148 ceph-mon[35610]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:12:40.475 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:12:40 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:12:40.274+0000 7f4cac4a5640 -1 osd.11 0 log_to_monitors true 2024-09-15T23:12:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:40 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2832563069' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:12:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:40 smithi062 ceph-mon[30949]: from='osd.11 [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:12:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:40 smithi062 ceph-mon[30949]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:12:40.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:40 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2832563069' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:12:40.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:40 smithi067 ceph-mon[35665]: from='osd.11 [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:12:40.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:40 smithi067 ceph-mon[35665]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-09-15T23:12:41.443 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-15T23:12:41.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:12:41 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:12:41.336+0000 7f4ca8428640 -1 osd.11 0 waiting for initial osdmap 2024-09-15T23:12:41.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:12:41 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:12:41.348+0000 7f4ca4250640 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T23:12:41.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:41 smithi148 ceph-mon[35610]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:41.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:41 smithi148 ceph-mon[35610]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-15T23:12:41.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:41 smithi148 ceph-mon[35610]: osdmap e74: 12 total, 11 up, 12 in 2024-09-15T23:12:41.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:41.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:41 smithi148 ceph-mon[35610]: from='osd.11 [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:12:41.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:41 smithi148 ceph-mon[35610]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:12:41.711 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:12:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:41 smithi062 ceph-mon[30949]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:41 smithi062 ceph-mon[30949]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-15T23:12:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:41 smithi062 ceph-mon[30949]: osdmap e74: 12 total, 11 up, 12 in 2024-09-15T23:12:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:41 smithi062 ceph-mon[30949]: from='osd.11 [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:12:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:41 smithi062 ceph-mon[30949]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:12:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:41 smithi067 ceph-mon[35665]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:41 smithi067 ceph-mon[35665]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-15T23:12:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:41 smithi067 ceph-mon[35665]: osdmap e74: 12 total, 11 up, 12 in 2024-09-15T23:12:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:41.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:41 smithi067 ceph-mon[35665]: from='osd.11 [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:12:41.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:41 smithi067 ceph-mon[35665]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]: dispatch 2024-09-15T23:12:42.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 -- 172.21.15.62:0/481795567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c1026a0 msgr2=0x7fa61c104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:42.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 --2- 172.21.15.62:0/481795567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c1026a0 0x7fa61c104a90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa618009900 tx=0x7fa61802a410 comp rx=0 tx=0).stop 2024-09-15T23:12:42.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 -- 172.21.15.62:0/481795567 shutdown_connections 2024-09-15T23:12:42.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 --2- 172.21.15.62:0/481795567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61c107900 0x7fa61c109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 --2- 172.21.15.62:0/481795567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61c104fd0 0x7fa61c1073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 --2- 172.21.15.62:0/481795567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c1026a0 0x7fa61c104a90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 -- 172.21.15.62:0/481795567 >> 172.21.15.62:0/481795567 conn(0x7fa61c0fc2a0 msgr2=0x7fa61c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 -- 172.21.15.62:0/481795567 shutdown_connections 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 -- 172.21.15.62:0/481795567 wait complete. 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 Processor -- start 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 -- start start 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.182+0000 7fa623486640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61c1026a0 0x7fa61c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa623486640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61c104fd0 0x7fa61c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa623486640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c107900 0x7fa61c19f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa623486640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa61c10d8d0 con 0x7fa61c107900 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa623486640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa61c10d750 con 0x7fa61c1026a0 2024-09-15T23:12:42.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa623486640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa61c10da50 con 0x7fa61c104fd0 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621a23640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c107900 0x7fa61c19f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621a23640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c107900 0x7fa61c19f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45028/0 (socket says 172.21.15.62:45028) 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621a23640 1 -- 172.21.15.62:0/2365542973 learned_addr learned my addr 172.21.15.62:0/2365542973 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa620a21640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61c104fd0 0x7fa61c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621222640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61c1026a0 0x7fa61c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621a23640 1 -- 172.21.15.62:0/2365542973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61c104fd0 msgr2=0x7fa61c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621a23640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61c104fd0 0x7fa61c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621a23640 1 -- 172.21.15.62:0/2365542973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61c1026a0 msgr2=0x7fa61c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621a23640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61c1026a0 0x7fa61c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621a23640 1 -- 172.21.15.62:0/2365542973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61c19fb80 con 0x7fa61c107900 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa621a23640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c107900 0x7fa61c19f510 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa61400eaa0 tx=0x7fa61400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa6127fc640 1 -- 172.21.15.62:0/2365542973 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61400cdb0 con 0x7fa61c107900 2024-09-15T23:12:42.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa6127fc640 1 -- 172.21.15.62:0/2365542973 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa614005020 con 0x7fa61c107900 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.183+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa61c19fe70 con 0x7fa61c107900 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.184+0000 7fa6127fc640 1 -- 172.21.15.62:0/2365542973 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6140116b0 con 0x7fa61c107900 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.184+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa61c1a0320 con 0x7fa61c107900 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.185+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa61c0fdd90 con 0x7fa61c107900 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.186+0000 7fa6127fc640 1 -- 172.21.15.62:0/2365542973 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa614005560 con 0x7fa61c107900 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.186+0000 7fa6127fc640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa6080a32f0 0x7fa6080a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.187+0000 7fa621222640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa6080a32f0 0x7fa6080a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.187+0000 7fa6127fc640 1 -- 172.21.15.62:0/2365542973 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6847+0+0 (secure 0 0 0) 0x7fa6140cf810 con 0x7fa61c107900 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.187+0000 7fa621222640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa6080a32f0 0x7fa6080a57b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa6180314e0 tx=0x7fa618000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:42.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.190+0000 7fa6127fc640 1 -- 172.21.15.62:0/2365542973 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa614016030 con 0x7fa61c107900 2024-09-15T23:12:42.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.367+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fa61c02da10 con 0x7fa61c107900 2024-09-15T23:12:42.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.368+0000 7fa6127fc640 1 -- 172.21.15.62:0/2365542973 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) ==== 74+0+133 (secure 0 0 0) 0x7fa61409c520 con 0x7fa61c107900 2024-09-15T23:12:42.369 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: osdmap e75: 12 total, 11 up, 12 in 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: Detected new or changed devices on smithi148 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.371 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.371 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:42.371 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:42.371 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa6080a32f0 msgr2=0x7fa6080a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa6080a32f0 0x7fa6080a57b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa6180314e0 tx=0x7fa618000bc0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c107900 msgr2=0x7fa61c19f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c107900 0x7fa61c19f510 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa61400eaa0 tx=0x7fa61400ef60 comp rx=0 tx=0).stop 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 shutdown_connections 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa6080a32f0 0x7fa6080a57b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa61c107900 0x7fa61c19f510 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa61c104fd0 0x7fa61c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 --2- 172.21.15.62:0/2365542973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa61c1026a0 0x7fa61c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:42.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 >> 172.21.15.62:0/2365542973 conn(0x7fa61c0fc2a0 msgr2=0x7fa61c103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:42.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 shutdown_connections 2024-09-15T23:12:42.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:42.371+0000 7fa623486640 1 -- 172.21.15.62:0/2365542973 wait complete. 2024-09-15T23:12:42.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:12:42.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: osdmap e75: 12 total, 11 up, 12 in 2024-09-15T23:12:42.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: Detected new or changed devices on smithi148 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:42.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi148", "root=default"]}]': finished 2024-09-15T23:12:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: osdmap e75: 12 total, 11 up, 12 in 2024-09-15T23:12:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: Detected new or changed devices on smithi148 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:42.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:43.053 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":76,"num_osds":12,"num_up_osds":12,"osd_up_since":1726441962,"num_in_osds":12,"osd_in_since":1726441939,"num_remapped_pgs":0} 2024-09-15T23:12:43.054 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:12:43.311 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:12:43.601 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:43 smithi062 ceph-mon[30949]: purged_snaps scrub starts 2024-09-15T23:12:43.601 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:43 smithi062 ceph-mon[30949]: purged_snaps scrub ok 2024-09-15T23:12:43.601 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:43 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi148 to 4272M 2024-09-15T23:12:43.601 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:43 smithi062 ceph-mon[30949]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:43.601 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:43.601 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:43 smithi062 ceph-mon[30949]: osd.11 [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] boot 2024-09-15T23:12:43.601 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:43 smithi062 ceph-mon[30949]: osdmap e76: 12 total, 12 up, 12 in 2024-09-15T23:12:43.602 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:43.602 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:43 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2365542973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:12:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:43 smithi067 ceph-mon[35665]: purged_snaps scrub starts 2024-09-15T23:12:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:43 smithi067 ceph-mon[35665]: purged_snaps scrub ok 2024-09-15T23:12:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:43 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi148 to 4272M 2024-09-15T23:12:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:43 smithi067 ceph-mon[35665]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:43.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:43.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:43 smithi067 ceph-mon[35665]: osd.11 [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] boot 2024-09-15T23:12:43.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:43 smithi067 ceph-mon[35665]: osdmap e76: 12 total, 12 up, 12 in 2024-09-15T23:12:43.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:43.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:43 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2365542973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:12:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:43 smithi148 ceph-mon[35610]: purged_snaps scrub starts 2024-09-15T23:12:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:43 smithi148 ceph-mon[35610]: purged_snaps scrub ok 2024-09-15T23:12:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:43 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi148 to 4272M 2024-09-15T23:12:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:43 smithi148 ceph-mon[35610]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-09-15T23:12:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:43 smithi148 ceph-mon[35610]: osd.11 [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] boot 2024-09-15T23:12:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:43 smithi148 ceph-mon[35610]: osdmap e76: 12 total, 12 up, 12 in 2024-09-15T23:12:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-09-15T23:12:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:43 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2365542973' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-15T23:12:43.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.806+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/3696548854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 msgr2=0x7f1cc0107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:43.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.806+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/3696548854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 0x7f1cc0107b80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009930 tx=0x7f1cb402a420 comp rx=0 tx=0).stop 2024-09-15T23:12:43.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.806+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/3696548854 shutdown_connections 2024-09-15T23:12:43.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.806+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/3696548854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc0108290 0x7f1cc010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.806+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/3696548854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 0x7f1cc0107b80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.806+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/3696548854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cc0102d90 0x7f1cc0103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.806+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/3696548854 >> 172.21.15.62:0/3696548854 conn(0x7f1cc00fc7d0 msgr2=0x7f1cc00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.806+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/3696548854 shutdown_connections 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.806+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/3696548854 wait complete. 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.807+0000 7f1cc6e8c640 1 Processor -- start 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.807+0000 7f1cc6e8c640 1 -- start start 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.807+0000 7f1cc6e8c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc0102d90 0x7f1cc0198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.807+0000 7f1cc6e8c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 0x7f1cc0198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.807+0000 7f1cc6e8c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cc0108290 0x7f1cc019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.807+0000 7f1cc6e8c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc01120b0 con 0x7f1cc0103740 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.807+0000 7f1cc6e8c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc0111f30 con 0x7f1cc0102d90 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.807+0000 7f1cc6e8c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc0112230 con 0x7f1cc0108290 2024-09-15T23:12:43.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.807+0000 7f1cbffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 0x7f1cc0198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 0x7f1cc0198b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33060/0 (socket says 172.21.15.62:33060) 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbffff640 1 -- 172.21.15.62:0/1223066830 learned_addr learned my addr 172.21.15.62:0/1223066830 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cc5429640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cc0108290 0x7f1cc019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cc4c28640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc0102d90 0x7f1cc0198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbffff640 1 -- 172.21.15.62:0/1223066830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cc0108290 msgr2=0x7f1cc019cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbffff640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cc0108290 0x7f1cc019cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbffff640 1 -- 172.21.15.62:0/1223066830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc0102d90 msgr2=0x7f1cc0198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbffff640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc0102d90 0x7f1cc0198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbffff640 1 -- 172.21.15.62:0/1223066830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc019d5c0 con 0x7f1cc0103740 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cc4c28640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc0102d90 0x7f1cc0198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbffff640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 0x7f1cc0198b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009230 tx=0x7f1cb40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:43.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbdffb640 1 -- 172.21.15.62:0/1223066830 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb403f030 con 0x7f1cc0103740 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbdffb640 1 -- 172.21.15.62:0/1223066830 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cb4003ce0 con 0x7f1cc0103740 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc019d850 con 0x7f1cc0103740 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cbdffb640 1 -- 172.21.15.62:0/1223066830 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4035070 con 0x7f1cc0103740 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.808+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc01a5190 con 0x7f1cc0103740 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.810+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc0103170 con 0x7f1cc0103740 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.811+0000 7f1cbdffb640 1 -- 172.21.15.62:0/1223066830 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f1cb4042030 con 0x7f1cc0103740 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.811+0000 7f1cbdffb640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ca80a32c0 0x7f1ca80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.812+0000 7f1cc4c28640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ca80a32c0 0x7f1ca80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.812+0000 7f1cbdffb640 1 -- 172.21.15.62:0/1223066830 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7127+0+0 (secure 0 0 0) 0x7f1cb40efe50 con 0x7f1cc0103740 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.812+0000 7f1cc4c28640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ca80a32c0 0x7f1ca80a5780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00058c0 tx=0x7f1cb0005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:43.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.815+0000 7f1cbdffb640 1 -- 172.21.15.62:0/1223066830 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1cb40bd010 con 0x7f1cc0103740 2024-09-15T23:12:43.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.991+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1cc010a2a0 con 0x7f1cc0103740 2024-09-15T23:12:43.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.992+0000 7f1cbdffb640 1 -- 172.21.15.62:0/1223066830 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) ==== 74+0+18420 (secure 0 0 0) 0x7f1cb40bc910 con 0x7f1cc0103740 2024-09-15T23:12:43.993 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:12:43.994 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":76,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:12:42.334694+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:12:43.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ca80a32c0 msgr2=0x7f1ca80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:43.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ca80a32c0 0x7f1ca80a5780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00058c0 tx=0x7f1cb0005d30 comp rx=0 tx=0).stop 2024-09-15T23:12:43.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 msgr2=0x7f1cc0198b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:43.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 0x7f1cc0198b50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009230 tx=0x7f1cb40042f0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 shutdown_connections 2024-09-15T23:12:43.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ca80a32c0 0x7f1ca80a5780 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1cc0108290 0x7f1cc019cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1cc0103740 0x7f1cc0198b50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 --2- 172.21.15.62:0/1223066830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc0102d90 0x7f1cc0198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:43.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 >> 172.21.15.62:0/1223066830 conn(0x7f1cc00fc7d0 msgr2=0x7f1cc0106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:43.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 shutdown_connections 2024-09-15T23:12:43.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:43.995+0000 7f1cc6e8c640 1 -- 172.21.15.62:0/1223066830 wait complete. 2024-09-15T23:12:44.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:44 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1223066830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:12:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:44 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1223066830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:12:44.788 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T23:08:54.129326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-15T23:12:44.788 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-15T23:12:44.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:44 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1223066830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:12:45.045 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:12:45.458 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:45 smithi062 ceph-mon[30949]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:45.458 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:45 smithi062 ceph-mon[30949]: osdmap e77: 12 total, 12 up, 12 in 2024-09-15T23:12:45.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.543+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3099864553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8106f10 msgr2=0x7ff8f810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:45.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.543+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3099864553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8106f10 0x7ff8f810ea90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0009ca0 tx=0x7ff8f002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.543+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3099864553 shutdown_connections 2024-09-15T23:12:45.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.543+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3099864553 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8106f10 0x7ff8f810ea90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.543+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3099864553 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8f8102790 0x7ff8f81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.543+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3099864553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f8101de0 0x7ff8f81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.543+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3099864553 >> 172.21.15.62:0/3099864553 conn(0x7ff8f80fd8c0 msgr2=0x7ff8f80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.543+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3099864553 shutdown_connections 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.543+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3099864553 wait complete. 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8ff04b640 1 Processor -- start 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8ff04b640 1 -- start start 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8ff04b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8101de0 0x7ff8f8198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8ff04b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8f8102790 0x7ff8f8198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8ff04b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f8106f10 0x7ff8f819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8ff04b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f8111f70 con 0x7ff8f8101de0 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8ff04b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f8111df0 con 0x7ff8f8106f10 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8ff04b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff8f81120f0 con 0x7ff8f8102790 2024-09-15T23:12:45.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8fcde7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8101de0 0x7ff8f8198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8fcde7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8101de0 0x7ff8f8198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33072/0 (socket says 172.21.15.62:33072) 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8fcde7640 1 -- 172.21.15.62:0/3858804191 learned_addr learned my addr 172.21.15.62:0/3858804191 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.544+0000 7ff8fd5e8640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f8106f10 0x7ff8f819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8fcde7640 1 -- 172.21.15.62:0/3858804191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8f8102790 msgr2=0x7ff8f8198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8f7fff640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8f8102790 0x7ff8f8198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8fcde7640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8f8102790 0x7ff8f8198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8fcde7640 1 -- 172.21.15.62:0/3858804191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f8106f10 msgr2=0x7ff8f819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8fcde7640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f8106f10 0x7ff8f819cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8fcde7640 1 -- 172.21.15.62:0/3858804191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f819d2f0 con 0x7ff8f8101de0 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8fd5e8640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f8106f10 0x7ff8f819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:12:45.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8f7fff640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8f8102790 0x7ff8f8198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8fcde7640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8101de0 0x7ff8f8198340 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00e9f0 tx=0x7ff8ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8f5ffb640 1 -- 172.21.15.62:0/3858804191 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec00cd80 con 0x7ff8f8101de0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8f5ffb640 1 -- 172.21.15.62:0/3858804191 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ec010910 con 0x7ff8f8101de0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8f819d5e0 con 0x7ff8f8101de0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8f5ffb640 1 -- 172.21.15.62:0/3858804191 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec018990 con 0x7ff8f8101de0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.545+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8f81a4ec0 con 0x7ff8f8101de0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.546+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8f81021c0 con 0x7ff8f8101de0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.548+0000 7ff8f5ffb640 1 -- 172.21.15.62:0/3858804191 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7ff8ec010ac0 con 0x7ff8f8101de0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.548+0000 7ff8f5ffb640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff8e80a3240 0x7ff8e80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.548+0000 7ff8f7fff640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff8e80a3240 0x7ff8e80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.548+0000 7ff8f5ffb640 1 -- 172.21.15.62:0/3858804191 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7ff8ec0cfd20 con 0x7ff8f8101de0 2024-09-15T23:12:45.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.549+0000 7ff8f7fff640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff8e80a3240 0x7ff8e80a5700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8108880 tx=0x7ff8e4005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:45.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.551+0000 7ff8f5ffb640 1 -- 172.21.15.62:0/3858804191 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff8ec09cee0 con 0x7ff8f8101de0 2024-09-15T23:12:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.726+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff8f802d7c0 con 0x7ff8f8101de0 2024-09-15T23:12:45.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.726+0000 7ff8f5ffb640 1 -- 172.21.15.62:0/3858804191 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7ff8ec09c7e0 con 0x7ff8f8101de0 2024-09-15T23:12:45.727 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-09-15T23:12:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff8e80a3240 msgr2=0x7ff8e80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:45.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff8e80a3240 0x7ff8e80a5700 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8108880 tx=0x7ff8e4005d20 comp rx=0 tx=0).stop 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8101de0 msgr2=0x7ff8f8198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8101de0 0x7ff8f8198340 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00e9f0 tx=0x7ff8ec00eeb0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 shutdown_connections 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff8e80a3240 0x7ff8e80a5700 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8f8106f10 0x7ff8f819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff8f8102790 0x7ff8f8198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 --2- 172.21.15.62:0/3858804191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8f8101de0 0x7ff8f8198340 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 >> 172.21.15.62:0/3858804191 conn(0x7ff8f80fd8c0 msgr2=0x7ff8f80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 shutdown_connections 2024-09-15T23:12:45.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:45.728+0000 7ff8ff04b640 1 -- 172.21.15.62:0/3858804191 wait complete. 2024-09-15T23:12:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:45 smithi067 ceph-mon[35665]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:45 smithi067 ceph-mon[35665]: osdmap e77: 12 total, 12 up, 12 in 2024-09-15T23:12:45.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:45 smithi148 ceph-mon[35610]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:45.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:45 smithi148 ceph-mon[35610]: osdmap e77: 12 total, 12 up, 12 in 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.a on smithi062 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.d on smithi062 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.g on smithi062 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.j on smithi062 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.b on smithi067 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.e on smithi067 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.h on smithi067 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.k on smithi067 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.c on smithi148 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.f on smithi148 2024-09-15T23:12:46.558 INFO:tasks.cephadm:Adding mds.i on smithi148 2024-09-15T23:12:46.559 INFO:tasks.cephadm:Adding mds.l on smithi148 2024-09-15T23:12:46.559 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph orch apply mds all '12;smithi062=a;smithi062=d;smithi062=g;smithi062=j;smithi067=b;smithi067=e;smithi067=h;smithi067=k;smithi148=c;smithi148=f;smithi148=i;smithi148=l' 2024-09-15T23:12:46.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:46 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3858804191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:12:46.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:46 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3858804191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:12:46.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:46 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3858804191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:12:46.820 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.294+0000 7f00be3e3640 1 -- 172.21.15.148:0/662937203 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b81037b0 msgr2=0x7f00b810a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.294+0000 7f00be3e3640 1 --2- 172.21.15.148:0/662937203 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b81037b0 0x7f00b810a040 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f00b0009ca0 tx=0x7f00b002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.294+0000 7f00be3e3640 1 -- 172.21.15.148:0/662937203 shutdown_connections 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.294+0000 7f00be3e3640 1 --2- 172.21.15.148:0/662937203 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b81037b0 0x7f00b810a040 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.294+0000 7f00be3e3640 1 --2- 172.21.15.148:0/662937203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8102df0 0x7f00b8103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.294+0000 7f00be3e3640 1 --2- 172.21.15.148:0/662937203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00b8101bf0 0x7f00b8101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.294+0000 7f00be3e3640 1 -- 172.21.15.148:0/662937203 >> 172.21.15.148:0/662937203 conn(0x7f00b80fd3c0 msgr2=0x7f00b80ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.294+0000 7f00be3e3640 1 -- 172.21.15.148:0/662937203 shutdown_connections 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.294+0000 7f00be3e3640 1 -- 172.21.15.148:0/662937203 wait complete. 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00be3e3640 1 Processor -- start 2024-09-15T23:12:47.318 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00be3e3640 1 -- start start 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00be3e3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8101bf0 0x7f00b8197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00be3e3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b8102df0 0x7f00b81983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00be3e3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00b81037b0 0x7f00b819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00be3e3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f00b810d8d0 con 0x7f00b81037b0 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00be3e3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00b810d750 con 0x7f00b8101bf0 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00be3e3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f00b810da50 con 0x7f00b8102df0 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00affff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b8102df0 0x7f00b81983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00affff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b8102df0 0x7f00b81983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:39488/0 (socket says 172.21.15.148:39488) 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00affff640 1 -- 172.21.15.148:0/1916388745 learned_addr learned my addr 172.21.15.148:0/1916388745 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00b7fff640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8101bf0 0x7f00b8197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00bc980640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00b81037b0 0x7f00b819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00affff640 1 -- 172.21.15.148:0/1916388745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8101bf0 msgr2=0x7f00b8197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00affff640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8101bf0 0x7f00b8197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00affff640 1 -- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00b81037b0 msgr2=0x7f00b819f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:47.319 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00affff640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00b81037b0 0x7f00b819f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.295+0000 7f00affff640 1 -- 172.21.15.148:0/1916388745 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b819fad0 con 0x7f00b8102df0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.296+0000 7f00affff640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b8102df0 0x7f00b81983e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f00a000b4a0 tx=0x7f00a000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.296+0000 7f00b5ffb640 1 -- 172.21.15.148:0/1916388745 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0014070 con 0x7f00b8102df0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.296+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00b819fdc0 con 0x7f00b8102df0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.296+0000 7f00b5ffb640 1 -- 172.21.15.148:0/1916388745 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00a0010af0 con 0x7f00b8102df0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.296+0000 7f00b5ffb640 1 -- 172.21.15.148:0/1916388745 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0017d10 con 0x7f00b8102df0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.296+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00b81a02f0 con 0x7f00b8102df0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.297+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00b8108fa0 con 0x7f00b8102df0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.299+0000 7f00b5ffb640 1 -- 172.21.15.148:0/1916388745 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f00a0017070 con 0x7f00b8102df0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.299+0000 7f00b5ffb640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a40a3270 0x7f00a40a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.300+0000 7f00b5ffb640 1 -- 172.21.15.148:0/1916388745 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f00a00ceb40 con 0x7f00b8102df0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.300+0000 7f00b7fff640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a40a3270 0x7f00a40a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.300+0000 7f00b7fff640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a40a3270 0x7f00a40a5730 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f00b8102c50 tx=0x7f00a800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:47.320 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.302+0000 7f00b5ffb640 1 -- 172.21.15.148:0/1916388745 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f00a009bd00 con 0x7f00b8102df0 2024-09-15T23:12:47.483 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:47 smithi148 ceph-mon[35610]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:47.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.486+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi062=a;smithi062=d;smithi062=g;smithi062=j;smithi067=b;smithi067=e;smithi067=h;smithi067=k;smithi148=c;smithi148=f;smithi148=i;smithi148=l", "target": ["mon-mgr", ""]}) -- 0x7f00b8062730 con 0x7f00a40a3270 2024-09-15T23:12:47.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.493+0000 7f00b5ffb640 1 -- 172.21.15.148:0/1916388745 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f00b8062730 con 0x7f00a40a3270 2024-09-15T23:12:47.494 INFO:teuthology.orchestra.run.smithi148.stdout:Scheduled mds.all update... 2024-09-15T23:12:47.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a40a3270 msgr2=0x7f00a40a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:47.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a40a3270 0x7f00a40a5730 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f00b8102c50 tx=0x7f00a800b390 comp rx=0 tx=0).stop 2024-09-15T23:12:47.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b8102df0 msgr2=0x7f00b81983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:47.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b8102df0 0x7f00b81983e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f00a000b4a0 tx=0x7f00a000b960 comp rx=0 tx=0).stop 2024-09-15T23:12:47.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 shutdown_connections 2024-09-15T23:12:47.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a40a3270 0x7f00a40a5730 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00b81037b0 0x7f00b819f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00b8102df0 0x7f00b81983e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 --2- 172.21.15.148:0/1916388745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00b8101bf0 0x7f00b8197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:47.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 >> 172.21.15.148:0/1916388745 conn(0x7f00b80fd3c0 msgr2=0x7f00b8106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:47.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 shutdown_connections 2024-09-15T23:12:47.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-09-15T23:12:47.495+0000 7f00be3e3640 1 -- 172.21.15.148:0/1916388745 wait complete. 2024-09-15T23:12:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:47 smithi062 ceph-mon[30949]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:47 smithi067 ceph-mon[35665]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:48.252 DEBUG:teuthology.orchestra.run.smithi062:mds.a> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.a.service 2024-09-15T23:12:48.255 DEBUG:teuthology.orchestra.run.smithi062:mds.d> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.d.service 2024-09-15T23:12:48.298 DEBUG:teuthology.orchestra.run.smithi062:mds.g> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.g.service 2024-09-15T23:12:48.341 DEBUG:teuthology.orchestra.run.smithi062:mds.j> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.j.service 2024-09-15T23:12:48.384 DEBUG:teuthology.orchestra.run.smithi067:mds.b> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.b.service 2024-09-15T23:12:48.386 DEBUG:teuthology.orchestra.run.smithi067:mds.e> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.e.service 2024-09-15T23:12:48.429 DEBUG:teuthology.orchestra.run.smithi067:mds.h> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.h.service 2024-09-15T23:12:48.472 DEBUG:teuthology.orchestra.run.smithi067:mds.k> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.k.service 2024-09-15T23:12:48.515 DEBUG:teuthology.orchestra.run.smithi148:mds.c> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.c.service 2024-09-15T23:12:48.518 DEBUG:teuthology.orchestra.run.smithi148:mds.f> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.f.service 2024-09-15T23:12:48.560 DEBUG:teuthology.orchestra.run.smithi148:mds.i> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.i.service 2024-09-15T23:12:48.605 DEBUG:teuthology.orchestra.run.smithi148:mds.l> sudo journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.l.service 2024-09-15T23:12:48.647 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-09-15T23:12:48.648 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:12:48.671 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi062=a;smithi062=d;smithi062=g;smithi062=j;smithi067=b;smithi067=e;smithi067=h;smithi067=k;smithi148=c;smithi148=f;smithi148=i;smithi148=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:12:48.672 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: Saving service mds.all spec with placement smithi062=a;smithi062=d;smithi062=g;smithi062=j;smithi067=b;smithi067=e;smithi067=h;smithi067=k;smithi148=c;smithi148=f;smithi148=i;smithi148=l;count:12 2024-09-15T23:12:48.672 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:48.672 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:48.672 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:48.672 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:48.672 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:48.672 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:48.672 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:48.672 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:48.768 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi062=a;smithi062=d;smithi062=g;smithi062=j;smithi067=b;smithi067=e;smithi067=h;smithi067=k;smithi148=c;smithi148=f;smithi148=i;smithi148=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:12:48.768 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: Saving service mds.all spec with placement smithi062=a;smithi062=d;smithi062=g;smithi062=j;smithi067=b;smithi067=e;smithi067=h;smithi067=k;smithi148=c;smithi148=f;smithi148=i;smithi148=l;count:12 2024-09-15T23:12:48.768 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:48.769 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:48.769 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:48.769 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:48.769 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:48.769 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:48.769 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:48.769 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:48.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi062=a;smithi062=d;smithi062=g;smithi062=j;smithi067=b;smithi067=e;smithi067=h;smithi067=k;smithi148=c;smithi148=f;smithi148=i;smithi148=l", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:12:48.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: Saving service mds.all spec with placement smithi062=a;smithi062=d;smithi062=g;smithi062=j;smithi067=b;smithi067=e;smithi067=h;smithi067=k;smithi148=c;smithi148=f;smithi148=i;smithi148=l;count:12 2024-09-15T23:12:48.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:48.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:12:48.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:48.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:12:48.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:48.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:48.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:48.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:48.900 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:12:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 -- 172.21.15.62:0/143865610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8106a80 msgr2=0x7f16a8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 --2- 172.21.15.62:0/143865610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8106a80 0x7f16a8106e60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1698009930 tx=0x7f169802a420 comp rx=0 tx=0).stop 2024-09-15T23:12:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 -- 172.21.15.62:0/143865610 shutdown_connections 2024-09-15T23:12:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 --2- 172.21.15.62:0/143865610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a8101420 0x7f16a810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 --2- 172.21.15.62:0/143865610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a8100a80 0x7f16a8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 --2- 172.21.15.62:0/143865610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8106a80 0x7f16a8106e60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 -- 172.21.15.62:0/143865610 >> 172.21.15.62:0/143865610 conn(0x7f16a80fc7a0 msgr2=0x7f16a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 -- 172.21.15.62:0/143865610 shutdown_connections 2024-09-15T23:12:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 -- 172.21.15.62:0/143865610 wait complete. 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 Processor -- start 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.357+0000 7f16acb3e640 1 -- start start 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f16acb3e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a8100a80 0x7f16a81992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f16acb3e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8101420 0x7f16a8199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f16acb3e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a8106a80 0x7f16a81a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f16acb3e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f16a8110e40 con 0x7f16a8101420 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f16acb3e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f16a8110cc0 con 0x7f16a8100a80 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f16acb3e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f16a8110fc0 con 0x7f16a8106a80 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8101420 0x7f16a8199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8101420 0x7f16a8199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33094/0 (socket says 172.21.15.62:33094) 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169ffff640 1 -- 172.21.15.62:0/3905861267 learned_addr learned my addr 172.21.15.62:0/3905861267 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f16a659c640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a8100a80 0x7f16a81992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:49.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f16a6d9d640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a8106a80 0x7f16a81a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169ffff640 1 -- 172.21.15.62:0/3905861267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a8106a80 msgr2=0x7f16a81a4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169ffff640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a8106a80 0x7f16a81a4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169ffff640 1 -- 172.21.15.62:0/3905861267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a8100a80 msgr2=0x7f16a81992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169ffff640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a8100a80 0x7f16a81992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169ffff640 1 -- 172.21.15.62:0/3905861267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a81a4890 con 0x7f16a8101420 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f16a6d9d640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a8106a80 0x7f16a81a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169ffff640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8101420 0x7f16a8199810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f169000d950 tx=0x7f169000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169f7fe640 1 -- 172.21.15.62:0/3905861267 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690017030 con 0x7f16a8101420 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.358+0000 7f169f7fe640 1 -- 172.21.15.62:0/3905861267 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1690004020 con 0x7f16a8101420 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.359+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a81a4a60 con 0x7f16a8101420 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.359+0000 7f169f7fe640 1 -- 172.21.15.62:0/3905861267 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16900047a0 con 0x7f16a8101420 2024-09-15T23:12:49.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.359+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a81a4f70 con 0x7f16a8101420 2024-09-15T23:12:49.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.360+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a8100ee0 con 0x7f16a8101420 2024-09-15T23:12:49.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.361+0000 7f169f7fe640 1 -- 172.21.15.62:0/3905861267 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f16900041d0 con 0x7f16a8101420 2024-09-15T23:12:49.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.361+0000 7f169f7fe640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16940a3240 0x7f16940a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:49.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.362+0000 7f169f7fe640 1 -- 172.21.15.62:0/3905861267 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f16900cf4f0 con 0x7f16a8101420 2024-09-15T23:12:49.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.362+0000 7f16a659c640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16940a3240 0x7f16940a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:49.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.362+0000 7f16a659c640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16940a3240 0x7f16940a5700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1698009900 tx=0x7f16980058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:49.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.365+0000 7f169f7fe640 1 -- 172.21.15.62:0/3905861267 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f169009c6b0 con 0x7f16a8101420 2024-09-15T23:12:49.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.579+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16a806ad80 con 0x7f16a8101420 2024-09-15T23:12:49.580 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:49 smithi062 ceph-mon[30949]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:49.580 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:49 smithi062 ceph-mon[30949]: Deploying daemon mds.l on smithi148 2024-09-15T23:12:49.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.579+0000 7f169f7fe640 1 -- 172.21.15.62:0/3905861267 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+611 (secure 0 0 0) 0x7f169009bfb0 con 0x7f16a8101420 2024-09-15T23:12:49.581 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:12:49.581 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":1,"btime":"2024-09-15T23:05:50:746836+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-09-15T23:12:49.581 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 1 2024-09-15T23:12:49.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16940a3240 msgr2=0x7f16940a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16940a3240 0x7f16940a5700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1698009900 tx=0x7f16980058d0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8101420 msgr2=0x7f16a8199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8101420 0x7f16a8199810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f169000d950 tx=0x7f169000de10 comp rx=0 tx=0).stop 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 shutdown_connections 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16940a3240 0x7f16940a5700 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a8106a80 0x7f16a81a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a8101420 0x7f16a8199810 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 --2- 172.21.15.62:0/3905861267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a8100a80 0x7f16a81992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 >> 172.21.15.62:0/3905861267 conn(0x7f16a80fc7a0 msgr2=0x7f16a80fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 shutdown_connections 2024-09-15T23:12:49.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:49.582+0000 7f16acb3e640 1 -- 172.21.15.62:0/3905861267 wait complete. 2024-09-15T23:12:49.705 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:49 smithi148 ceph-mon[35610]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:49.706 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:49 smithi148 ceph-mon[35610]: Deploying daemon mds.l on smithi148 2024-09-15T23:12:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:49 smithi067 ceph-mon[35665]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:49 smithi067 ceph-mon[35665]: Deploying daemon mds.l on smithi148 2024-09-15T23:12:50.039 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:12:49 smithi148 systemd[1]: Starting Ceph mds.l for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:12:50.361 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:12:50 smithi148 podman[65929]: 2024-09-15 23:12:49.998757133 +0000 UTC m=+0.016177859 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:12:50.362 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:12:50 smithi148 podman[65929]: 2024-09-15 23:12:50.125230291 +0000 UTC m=+0.142651065 container create c31cf6fdb3874838cdb3633717e5c1654fef00f2ce9eeedea4b6b7df16bd460f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e) 2024-09-15T23:12:50.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:50 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3905861267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:12:50.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:50 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.148:6832/2970459675,v1:172.21.15.148:6833/2970459675] up:boot 2024-09-15T23:12:50.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:50 smithi062 ceph-mon[30949]: fsmap 1 up:standby 2024-09-15T23:12:50.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:50 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T23:12:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:50 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3905861267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:12:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:50 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.148:6832/2970459675,v1:172.21.15.148:6833/2970459675] up:boot 2024-09-15T23:12:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:50 smithi067 ceph-mon[35665]: fsmap 1 up:standby 2024-09-15T23:12:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:50 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T23:12:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:50 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3905861267' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:12:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:50 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.148:6832/2970459675,v1:172.21.15.148:6833/2970459675] up:boot 2024-09-15T23:12:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:50 smithi148 ceph-mon[35610]: fsmap 1 up:standby 2024-09-15T23:12:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:50 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-09-15T23:12:50.789 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:12:50 smithi148 podman[65929]: 2024-09-15 23:12:50.361334636 +0000 UTC m=+0.378755346 container init c31cf6fdb3874838cdb3633717e5c1654fef00f2ce9eeedea4b6b7df16bd460f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-l, RELEASE=squid-8921c8e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:12:50.790 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:12:50 smithi148 podman[65929]: 2024-09-15 23:12:50.364210508 +0000 UTC m=+0.381631223 container start c31cf6fdb3874838cdb3633717e5c1654fef00f2ce9eeedea4b6b7df16bd460f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-l, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e) 2024-09-15T23:12:50.790 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:12:50 smithi148 ceph-mds[65948]: -- 172.21.15.148:0/729822796 <== mon.1 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555ffb883c20 con 0x555ffb861400 2024-09-15T23:12:50.790 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:12:50 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-l[65944]: starting mds.l at 2024-09-15T23:12:50.790 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:12:50 smithi148 bash[65929]: c31cf6fdb3874838cdb3633717e5c1654fef00f2ce9eeedea4b6b7df16bd460f 2024-09-15T23:12:50.790 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:12:50 smithi148 systemd[1]: Started Ceph mds.l for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:12:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:51 smithi067 ceph-mon[35665]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:51.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:51.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:51.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:51.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:51.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:51.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:51.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:51 smithi067 ceph-mon[35665]: Deploying daemon mds.k on smithi067 2024-09-15T23:12:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:51 smithi148 ceph-mon[35610]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:51.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:51 smithi148 ceph-mon[35610]: Deploying daemon mds.k on smithi067 2024-09-15T23:12:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:51 smithi062 ceph-mon[30949]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:51.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:51.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:51.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:51.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:51 smithi062 ceph-mon[30949]: Deploying daemon mds.k on smithi067 2024-09-15T23:12:52.411 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:12:52.676 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:12:52 smithi067 systemd[1]: Starting Ceph mds.k for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:12:52.678 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:12:53.024 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:12:52 smithi067 podman[64784]: 2024-09-15 23:12:52.789519641 +0000 UTC m=+0.015967037 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:12:53.025 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:12:52 smithi067 podman[64784]: 2024-09-15 23:12:52.909032961 +0000 UTC m=+0.135480341 container create b7512f6e00c8dbc543cfffb16ce085feb9a1f4213442f58912d99f43904b5f78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-k, ceph=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-15T23:12:53.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.155+0000 7f1545b28640 1 -- 172.21.15.62:0/1322143837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 msgr2=0x7f15401067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:53.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.155+0000 7f1545b28640 1 --2- 172.21.15.62:0/1322143837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 0x7f15401067e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1534009900 tx=0x7f153402a410 comp rx=0 tx=0).stop 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.155+0000 7f1545b28640 1 -- 172.21.15.62:0/1322143837 shutdown_connections 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.155+0000 7f1545b28640 1 --2- 172.21.15.62:0/1322143837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1540106f10 0x7f154010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.155+0000 7f1545b28640 1 --2- 172.21.15.62:0/1322143837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 0x7f15401067e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.155+0000 7f1545b28640 1 --2- 172.21.15.62:0/1322143837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1540101de0 0x7f15401021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.155+0000 7f1545b28640 1 -- 172.21.15.62:0/1322143837 >> 172.21.15.62:0/1322143837 conn(0x7f15400fd8c0 msgr2=0x7f15400ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.155+0000 7f1545b28640 1 -- 172.21.15.62:0/1322143837 shutdown_connections 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.155+0000 7f1545b28640 1 -- 172.21.15.62:0/1322143837 wait complete. 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f1545b28640 1 Processor -- start 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f1545b28640 1 -- start start 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f1545b28640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1540101de0 0x7f15401984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f1545b28640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 0x7f15401989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:53.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f1545b28640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1540106f10 0x7f154019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f1545b28640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1540111f70 con 0x7f1540102790 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f1545b28640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1540111df0 con 0x7f1540101de0 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f1545b28640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f15401120f0 con 0x7f1540106f10 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f153effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 0x7f15401989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f153effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 0x7f15401989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53986/0 (socket says 172.21.15.62:53986) 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.156+0000 7f153effd640 1 -- 172.21.15.62:0/3078099667 learned_addr learned my addr 172.21.15.62:0/3078099667 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153f7fe640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1540101de0 0x7f15401984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153ffff640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1540106f10 0x7f154019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153effd640 1 -- 172.21.15.62:0/3078099667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1540106f10 msgr2=0x7f154019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153effd640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1540106f10 0x7f154019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153effd640 1 -- 172.21.15.62:0/3078099667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1540101de0 msgr2=0x7f15401984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:53.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153effd640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1540101de0 0x7f15401984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153effd640 1 -- 172.21.15.62:0/3078099667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154019d460 con 0x7f1540102790 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153f7fe640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1540101de0 0x7f15401984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153effd640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 0x7f15401989f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1534006010 tx=0x7f15340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153cff9640 1 -- 172.21.15.62:0/3078099667 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153403f030 con 0x7f1540102790 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153cff9640 1 -- 172.21.15.62:0/3078099667 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1534003ce0 con 0x7f1540102790 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154019d6f0 con 0x7f1540102790 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f153cff9640 1 -- 172.21.15.62:0/3078099667 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1534035070 con 0x7f1540102790 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.157+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15401a5030 con 0x7f1540102790 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.158+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15401032c0 con 0x7f1540102790 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.159+0000 7f153cff9640 1 -- 172.21.15.62:0/3078099667 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f1534042030 con 0x7f1540102790 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.160+0000 7f153cff9640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15280a3240 0x7f15280a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.160+0000 7f153f7fe640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15280a3240 0x7f15280a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.160+0000 7f153cff9640 1 -- 172.21.15.62:0/3078099667 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f15340efb00 con 0x7f1540102790 2024-09-15T23:12:53.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.161+0000 7f153f7fe640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15280a3240 0x7f15280a5700 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15300058c0 tx=0x7f1530005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:53.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.164+0000 7f153cff9640 1 -- 172.21.15.62:0/3078099667 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f15340bcd80 con 0x7f1540102790 2024-09-15T23:12:53.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.384+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f154010a150 con 0x7f1540102790 2024-09-15T23:12:53.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.384+0000 7f153cff9640 1 -- 172.21.15.62:0/3078099667 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7f15340bc680 con 0x7f1540102790 2024-09-15T23:12:53.385 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:12:53.386 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":2,"btime":"2024-09-15T23:12:50:392915+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-15T23:12:53.386 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 2 2024-09-15T23:12:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.386+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15280a3240 msgr2=0x7f15280a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.386+0000 7f1545b28640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15280a3240 0x7f15280a5700 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f15300058c0 tx=0x7f1530005850 comp rx=0 tx=0).stop 2024-09-15T23:12:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.386+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 msgr2=0x7f15401989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.386+0000 7f1545b28640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 0x7f15401989f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1534006010 tx=0x7f15340042f0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.387+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 shutdown_connections 2024-09-15T23:12:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.387+0000 7f1545b28640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15280a3240 0x7f15280a5700 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.387+0000 7f1545b28640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1540106f10 0x7f154019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.387+0000 7f1545b28640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1540102790 0x7f15401989f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.387+0000 7f1545b28640 1 --2- 172.21.15.62:0/3078099667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1540101de0 0x7f15401984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:53.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.387+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 >> 172.21.15.62:0/3078099667 conn(0x7f15400fd8c0 msgr2=0x7f15400fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:53.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.387+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 shutdown_connections 2024-09-15T23:12:53.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:53.387+0000 7f1545b28640 1 -- 172.21.15.62:0/3078099667 wait complete. 2024-09-15T23:12:53.524 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:12:53 smithi067 podman[64784]: 2024-09-15 23:12:53.170207864 +0000 UTC m=+0.396655245 container init b7512f6e00c8dbc543cfffb16ce085feb9a1f4213442f58912d99f43904b5f78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-09-15T23:12:53.524 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:12:53 smithi067 podman[64784]: 2024-09-15 23:12:53.172956049 +0000 UTC m=+0.399403421 container start b7512f6e00c8dbc543cfffb16ce085feb9a1f4213442f58912d99f43904b5f78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-15T23:12:53.524 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-mds[64803]: -- 172.21.15.67:0/2053774734 <== mon.2 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56165f46d860 con 0x56165f44b000 2024-09-15T23:12:53.524 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-k[64799]: starting mds.k at 2024-09-15T23:12:53.524 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:12:53 smithi067 bash[64784]: b7512f6e00c8dbc543cfffb16ce085feb9a1f4213442f58912d99f43904b5f78 2024-09-15T23:12:53.524 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:12:53 smithi067 systemd[1]: Started Ceph mds.k for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:12:53.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:53 smithi062 ceph-mon[30949]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:53.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:53.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:53.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:53.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:53.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:53.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:53 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:53.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:53 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3078099667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:12:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:53 smithi148 ceph-mon[35610]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:53 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:53.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:53 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3078099667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:12:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-mon[35665]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:54.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:54.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:54.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:53 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3078099667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:12:54.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:54 smithi062 ceph-mon[30949]: Deploying daemon mds.j on smithi062 2024-09-15T23:12:54.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:54 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.67:6832/190864103,v1:172.21.15.67:6833/190864103] up:boot 2024-09-15T23:12:54.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:54 smithi062 ceph-mon[30949]: fsmap 2 up:standby 2024-09-15T23:12:54.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:54 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T23:12:54.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:54 smithi148 ceph-mon[35610]: Deploying daemon mds.j on smithi062 2024-09-15T23:12:54.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:54 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.67:6832/190864103,v1:172.21.15.67:6833/190864103] up:boot 2024-09-15T23:12:54.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:54 smithi148 ceph-mon[35610]: fsmap 2 up:standby 2024-09-15T23:12:54.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:54 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T23:12:55.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:54 smithi067 ceph-mon[35665]: Deploying daemon mds.j on smithi062 2024-09-15T23:12:55.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:54 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.67:6832/190864103,v1:172.21.15.67:6833/190864103] up:boot 2024-09-15T23:12:55.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:54 smithi067 ceph-mon[35665]: fsmap 2 up:standby 2024-09-15T23:12:55.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:54 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-09-15T23:12:55.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:55 smithi148 ceph-mon[35610]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:55.849 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:55 smithi062 ceph-mon[30949]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:56.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:55 smithi067 ceph-mon[35665]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:56.224 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:12:56.229 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:12:56 smithi062 systemd[1]: Starting Ceph mds.j for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:12:56.510 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:12:56 smithi062 podman[67863]: 2024-09-15 23:12:56.15332322 +0000 UTC m=+0.016532474 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:12:56.510 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:12:56 smithi062 podman[67863]: 2024-09-15 23:12:56.281626354 +0000 UTC m=+0.144835592 container create 51f3dd19b664cc728dc92fb6e16be36049123b9baba2320c4eaedf468562df6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-j, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-15T23:12:56.510 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:12:56 smithi062 podman[67863]: 2024-09-15 23:12:56.50930577 +0000 UTC m=+0.372515009 container init 51f3dd19b664cc728dc92fb6e16be36049123b9baba2320c4eaedf468562df6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-j, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-15T23:12:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:12:56.845 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:12:56 smithi062 podman[67863]: 2024-09-15 23:12:56.512578052 +0000 UTC m=+0.375787291 container start 51f3dd19b664cc728dc92fb6e16be36049123b9baba2320c4eaedf468562df6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-15T23:12:56.845 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:12:56 smithi062 ceph-mds[67912]: -- 172.21.15.62:0/1507274381 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5643be579860 con 0x5643be557000 2024-09-15T23:12:56.845 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:12:56 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-j[67896]: starting mds.j at 2024-09-15T23:12:56.845 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:12:56 smithi062 bash[67863]: 51f3dd19b664cc728dc92fb6e16be36049123b9baba2320c4eaedf468562df6c 2024-09-15T23:12:56.845 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:12:56 smithi062 systemd[1]: Started Ceph mds.j for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:12:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.237+0000 7f1561cc2640 1 -- 172.21.15.62:0/3969676515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c102790 msgr2=0x7f155c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.237+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3969676515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c102790 0x7f155c1067e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1550009900 tx=0x7f155002a410 comp rx=0 tx=0).stop 2024-09-15T23:12:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.237+0000 7f1561cc2640 1 -- 172.21.15.62:0/3969676515 shutdown_connections 2024-09-15T23:12:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.237+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3969676515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f155c106f10 0x7f155c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.237+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3969676515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c102790 0x7f155c1067e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.237+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3969676515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c101de0 0x7f155c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.237+0000 7f1561cc2640 1 -- 172.21.15.62:0/3969676515 >> 172.21.15.62:0/3969676515 conn(0x7f155c0fd8c0 msgr2=0x7f155c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:57.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.237+0000 7f1561cc2640 1 -- 172.21.15.62:0/3969676515 shutdown_connections 2024-09-15T23:12:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.237+0000 7f1561cc2640 1 -- 172.21.15.62:0/3969676515 wait complete. 2024-09-15T23:12:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f1561cc2640 1 Processor -- start 2024-09-15T23:12:57.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f1561cc2640 1 -- start start 2024-09-15T23:12:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f1561cc2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c101de0 0x7f155c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f1561cc2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f155c102790 0x7f155c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:57.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f1561cc2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c106f10 0x7f155c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f1561cc2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f155c111f90 con 0x7f155c101de0 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f1561cc2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f155c111e10 con 0x7f155c106f10 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f1561cc2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f155c112110 con 0x7f155c102790 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f155b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c101de0 0x7f155c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f155b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c101de0 0x7f155c0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54034/0 (socket says 172.21.15.62:54034) 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.238+0000 7f155b7fe640 1 -- 172.21.15.62:0/3849475728 learned_addr learned my addr 172.21.15.62:0/3849475728 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155bfff640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c106f10 0x7f155c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155affd640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f155c102790 0x7f155c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155b7fe640 1 -- 172.21.15.62:0/3849475728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f155c102790 msgr2=0x7f155c074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155b7fe640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f155c102790 0x7f155c074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155b7fe640 1 -- 172.21.15.62:0/3849475728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c106f10 msgr2=0x7f155c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155b7fe640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c106f10 0x7f155c074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:57.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155b7fe640 1 -- 172.21.15.62:0/3849475728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155c1a6930 con 0x7f155c101de0 2024-09-15T23:12:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155bfff640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c106f10 0x7f155c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:12:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155affd640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f155c102790 0x7f155c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:12:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f155b7fe640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c101de0 0x7f155c0744f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f154c00d950 tx=0x7f154c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f1558ff9640 1 -- 172.21.15.62:0/3849475728 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c017070 con 0x7f155c101de0 2024-09-15T23:12:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155c1a6c20 con 0x7f155c101de0 2024-09-15T23:12:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f1558ff9640 1 -- 172.21.15.62:0/3849475728 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f154c009c30 con 0x7f155c101de0 2024-09-15T23:12:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f1558ff9640 1 -- 172.21.15.62:0/3849475728 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c00f6b0 con 0x7f155c101de0 2024-09-15T23:12:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.239+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f155c1a7160 con 0x7f155c101de0 2024-09-15T23:12:57.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.240+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f155c1021c0 con 0x7f155c101de0 2024-09-15T23:12:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.242+0000 7f1558ff9640 1 -- 172.21.15.62:0/3849475728 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f154c0109d0 con 0x7f155c101de0 2024-09-15T23:12:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.242+0000 7f1558ff9640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15440a3240 0x7f15440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:12:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.242+0000 7f155affd640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15440a3240 0x7f15440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:12:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.242+0000 7f1558ff9640 1 -- 172.21.15.62:0/3849475728 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f154c0cf130 con 0x7f155c101de0 2024-09-15T23:12:57.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.243+0000 7f155affd640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15440a3240 0x7f15440a5700 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1550000c00 tx=0x7f1550032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:12:57.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.247+0000 7f1558ff9640 1 -- 172.21.15.62:0/3849475728 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f154c009de0 con 0x7f155c101de0 2024-09-15T23:12:57.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.469+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f155c075a20 con 0x7f155c101de0 2024-09-15T23:12:57.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.469+0000 7f1558ff9640 1 -- 172.21.15.62:0/3849475728 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2260 (secure 0 0 0) 0x7f154c09c2f0 con 0x7f155c101de0 2024-09-15T23:12:57.470 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:12:57.470 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":3,"btime":"2024-09-15T23:12:53:516072+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-15T23:12:57.471 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 3 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15440a3240 msgr2=0x7f15440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15440a3240 0x7f15440a5700 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1550000c00 tx=0x7f1550032c70 comp rx=0 tx=0).stop 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c101de0 msgr2=0x7f155c0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c101de0 0x7f155c0744f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f154c00d950 tx=0x7f154c00de10 comp rx=0 tx=0).stop 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 shutdown_connections 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f15440a3240 0x7f15440a5700 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f155c106f10 0x7f155c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f155c102790 0x7f155c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 --2- 172.21.15.62:0/3849475728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f155c101de0 0x7f155c0744f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.471+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 >> 172.21.15.62:0/3849475728 conn(0x7f155c0fd8c0 msgr2=0x7f155c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.472+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 shutdown_connections 2024-09-15T23:12:57.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:12:57.472+0000 7f1561cc2640 1 -- 172.21.15.62:0/3849475728 wait complete. 2024-09-15T23:12:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:57 smithi062 ceph-mon[30949]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:57.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:57 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:57.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:57 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3849475728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:12:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:57 smithi148 ceph-mon[35610]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:57 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:57.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:57 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3849475728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:12:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:57 smithi067 ceph-mon[35665]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:58.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:58.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:12:58.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:12:58.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:12:58.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:57 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:12:58.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:57 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3849475728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:12:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:58 smithi062 ceph-mon[30949]: Deploying daemon mds.d on smithi062 2024-09-15T23:12:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:58 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.62:6834/3727539953,v1:172.21.15.62:6835/3727539953] up:boot 2024-09-15T23:12:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:58 smithi062 ceph-mon[30949]: fsmap 3 up:standby 2024-09-15T23:12:58.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:58 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T23:12:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:58 smithi067 ceph-mon[35665]: Deploying daemon mds.d on smithi062 2024-09-15T23:12:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:58 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.62:6834/3727539953,v1:172.21.15.62:6835/3727539953] up:boot 2024-09-15T23:12:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:58 smithi067 ceph-mon[35665]: fsmap 3 up:standby 2024-09-15T23:12:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:58 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T23:12:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:58 smithi148 ceph-mon[35610]: Deploying daemon mds.d on smithi062 2024-09-15T23:12:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:58 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.62:6834/3727539953,v1:172.21.15.62:6835/3727539953] up:boot 2024-09-15T23:12:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:58 smithi148 ceph-mon[35610]: fsmap 3 up:standby 2024-09-15T23:12:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:58 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-09-15T23:12:59.354 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:12:59 smithi062 systemd[1]: Starting Ceph mds.d for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:12:59.709 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:12:59 smithi062 ceph-mon[30949]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:12:59.709 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:12:59 smithi062 podman[68274]: 2024-09-15 23:12:59.453958713 +0000 UTC m=+0.016503404 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:12:59.709 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:12:59 smithi062 podman[68274]: 2024-09-15 23:12:59.578180233 +0000 UTC m=+0.140724905 container create 46315ff11dd3ad86d6e11aaf0d8cacbb95ce7fd03cc19c478aca59f3736d834d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-d, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-15T23:12:59.980 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:12:59 smithi062 podman[68274]: 2024-09-15 23:12:59.831498987 +0000 UTC m=+0.394043666 container init 46315ff11dd3ad86d6e11aaf0d8cacbb95ce7fd03cc19c478aca59f3736d834d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-d, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-15T23:12:59.980 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:12:59 smithi062 podman[68274]: 2024-09-15 23:12:59.834658086 +0000 UTC m=+0.397202759 container start 46315ff11dd3ad86d6e11aaf0d8cacbb95ce7fd03cc19c478aca59f3736d834d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-d, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-15T23:12:59.980 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:12:59 smithi062 ceph-mds[68293]: -- 172.21.15.62:0/2466437202 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564cd6bcb860 con 0x564cd6ba9000 2024-09-15T23:12:59.980 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:12:59 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-d[68289]: starting mds.d at 2024-09-15T23:12:59.980 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:12:59 smithi062 bash[68274]: 46315ff11dd3ad86d6e11aaf0d8cacbb95ce7fd03cc19c478aca59f3736d834d 2024-09-15T23:12:59.980 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:12:59 smithi062 systemd[1]: Started Ceph mds.d for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:13:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:12:59 smithi067 ceph-mon[35665]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:12:59 smithi148 ceph-mon[35610]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:01.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:01.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:01.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:01.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:01.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:01.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:01.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:01.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: Deploying daemon mds.e on smithi067 2024-09-15T23:13:01.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.62:6836/862835307,v1:172.21.15.62:6837/862835307] up:boot 2024-09-15T23:13:01.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: fsmap 4 up:standby 2024-09-15T23:13:01.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:00 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T23:13:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:01.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:01.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:01.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:01.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: Deploying daemon mds.e on smithi067 2024-09-15T23:13:01.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.62:6836/862835307,v1:172.21.15.62:6837/862835307] up:boot 2024-09-15T23:13:01.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: fsmap 4 up:standby 2024-09-15T23:13:01.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:00 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T23:13:01.387 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: Deploying daemon mds.e on smithi067 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.62:6836/862835307,v1:172.21.15.62:6837/862835307] up:boot 2024-09-15T23:13:01.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: fsmap 4 up:standby 2024-09-15T23:13:01.418 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:00 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-09-15T23:13:01.659 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:02.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.125+0000 7f6cebe45640 1 -- 172.21.15.62:0/3913687524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41082a0 msgr2=0x7f6ce410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:02.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.125+0000 7f6cebe45640 1 --2- 172.21.15.62:0/3913687524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41082a0 0x7f6ce410ec40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4009cd0 tx=0x7f6cd402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.125+0000 7f6cebe45640 1 -- 172.21.15.62:0/3913687524 shutdown_connections 2024-09-15T23:13:02.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.125+0000 7f6cebe45640 1 --2- 172.21.15.62:0/3913687524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41082a0 0x7f6ce410ec40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.125+0000 7f6cebe45640 1 --2- 172.21.15.62:0/3913687524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce41036e0 0x7f6ce4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.125+0000 7f6cebe45640 1 --2- 172.21.15.62:0/3913687524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ce4102d30 0x7f6ce4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.125+0000 7f6cebe45640 1 -- 172.21.15.62:0/3913687524 >> 172.21.15.62:0/3913687524 conn(0x7f6ce40fc7d0 msgr2=0x7f6ce40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.125+0000 7f6cebe45640 1 -- 172.21.15.62:0/3913687524 shutdown_connections 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.125+0000 7f6cebe45640 1 -- 172.21.15.62:0/3913687524 wait complete. 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cebe45640 1 Processor -- start 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cebe45640 1 -- start start 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cebe45640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ce4102d30 0x7f6ce4100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cebe45640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41036e0 0x7f6ce4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cebe45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce41082a0 0x7f6ce4101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:02.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cebe45640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce4112100 con 0x7f6ce41036e0 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cebe45640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce4111f80 con 0x7f6ce41082a0 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cebe45640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce4112280 con 0x7f6ce4102d30 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce93e0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41036e0 0x7f6ce4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce93e0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41036e0 0x7f6ce4100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54074/0 (socket says 172.21.15.62:54074) 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce93e0640 1 -- 172.21.15.62:0/4000216321 learned_addr learned my addr 172.21.15.62:0/4000216321 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cea3e2640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce41082a0 0x7f6ce4101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce9be1640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ce4102d30 0x7f6ce4100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce93e0640 1 -- 172.21.15.62:0/4000216321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ce4102d30 msgr2=0x7f6ce4100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce93e0640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ce4102d30 0x7f6ce4100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce93e0640 1 -- 172.21.15.62:0/4000216321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce41082a0 msgr2=0x7f6ce4101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce93e0640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce41082a0 0x7f6ce4101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce93e0640 1 -- 172.21.15.62:0/4000216321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce41a6a10 con 0x7f6ce41036e0 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6cea3e2640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce41082a0 0x7f6ce4101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:02.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.126+0000 7f6ce9be1640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ce4102d30 0x7f6ce4100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.127+0000 7f6ce93e0640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41036e0 0x7f6ce4100ed0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800c8c0 tx=0x7f6cd800cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.127+0000 7f6ce2ffd640 1 -- 172.21.15.62:0/4000216321 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8007bb0 con 0x7f6ce41036e0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.127+0000 7f6ce2ffd640 1 -- 172.21.15.62:0/4000216321 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cd8007d50 con 0x7f6ce41036e0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.127+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce41a6d00 con 0x7f6ce41036e0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.127+0000 7f6ce2ffd640 1 -- 172.21.15.62:0/4000216321 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8018830 con 0x7f6ce41036e0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.127+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce41a7230 con 0x7f6ce41036e0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.128+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce4103110 con 0x7f6ce41036e0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.129+0000 7f6ce2ffd640 1 -- 172.21.15.62:0/4000216321 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6cd8018a10 con 0x7f6ce41036e0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.130+0000 7f6ce2ffd640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6ccc0a32c0 0x7f6ccc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.130+0000 7f6ce9be1640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6ccc0a32c0 0x7f6ccc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.130+0000 7f6ce2ffd640 1 -- 172.21.15.62:0/4000216321 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f6cd80cf5b0 con 0x7f6ce41036e0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.130+0000 7f6ce9be1640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6ccc0a32c0 0x7f6ccc0a5780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009930 tx=0x7f6cdc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:02.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.134+0000 7f6ce2ffd640 1 -- 172.21.15.62:0/4000216321 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6cd8014030 con 0x7f6ce41036e0 2024-09-15T23:13:02.224 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:13:02 smithi067 systemd[1]: Starting Ceph mds.e for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:13:02.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.345+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ce410a2b0 con 0x7f6ce41036e0 2024-09-15T23:13:02.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.345+0000 7f6ce2ffd640 1 -- 172.21.15.62:0/4000216321 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3907 (secure 0 0 0) 0x7f6cd801f300 con 0x7f6ce41036e0 2024-09-15T23:13:02.347 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:02.347 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":5,"btime":"2024-09-15T23:13:00:546555+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-15T23:13:02.347 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 5 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.347+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6ccc0a32c0 msgr2=0x7f6ccc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.347+0000 7f6cebe45640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6ccc0a32c0 0x7f6ccc0a5780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009930 tx=0x7f6cdc008040 comp rx=0 tx=0).stop 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.347+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41036e0 msgr2=0x7f6ce4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.347+0000 7f6cebe45640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41036e0 0x7f6ce4100ed0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800c8c0 tx=0x7f6cd800cd80 comp rx=0 tx=0).stop 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.348+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 shutdown_connections 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.348+0000 7f6cebe45640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6ccc0a32c0 0x7f6ccc0a5780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.348+0000 7f6cebe45640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ce41082a0 0x7f6ce4101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.348+0000 7f6cebe45640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ce41036e0 0x7f6ce4100ed0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.348+0000 7f6cebe45640 1 --2- 172.21.15.62:0/4000216321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6ce4102d30 0x7f6ce4100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.348+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 >> 172.21.15.62:0/4000216321 conn(0x7f6ce40fc7d0 msgr2=0x7f6ce4106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:02.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.348+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 shutdown_connections 2024-09-15T23:13:02.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:02.348+0000 7f6cebe45640 1 -- 172.21.15.62:0/4000216321 wait complete. 2024-09-15T23:13:02.486 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:13:02 smithi067 podman[65083]: 2024-09-15 23:13:02.223141608 +0000 UTC m=+0.093761419 container create 0e3456702d7a859fd56fcc5d3c1884c94dc223658c07e49040d0f46f03186a0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-15T23:13:02.486 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:13:02 smithi067 podman[65083]: 2024-09-15 23:13:02.145353934 +0000 UTC m=+0.015973718 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:13:02.486 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:13:02 smithi067 podman[65083]: 2024-09-15 23:13:02.459470043 +0000 UTC m=+0.330089813 container init 0e3456702d7a859fd56fcc5d3c1884c94dc223658c07e49040d0f46f03186a0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-15T23:13:02.486 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:13:02 smithi067 podman[65083]: 2024-09-15 23:13:02.462352083 +0000 UTC m=+0.332971860 container start 0e3456702d7a859fd56fcc5d3c1884c94dc223658c07e49040d0f46f03186a0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-15T23:13:02.774 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-mds[65103]: -- 172.21.15.67:0/3174304778 <== mon.2 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559414b3d860 con 0x559414b1b000 2024-09-15T23:13:02.774 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e[65099]: starting mds.e at 2024-09-15T23:13:02.774 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:13:02 smithi067 bash[65083]: 0e3456702d7a859fd56fcc5d3c1884c94dc223658c07e49040d0f46f03186a0b 2024-09-15T23:13:02.774 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:13:02 smithi067 systemd[1]: Started Ceph mds.e for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:13:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-mon[35665]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4000216321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:03.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:02 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:02 smithi148 ceph-mon[35610]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:02 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4000216321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:02 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:02 smithi062 ceph-mon[30949]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:02 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4000216321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:03.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:03.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:03.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:02 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:03 smithi148 ceph-mon[35610]: Deploying daemon mds.f on smithi148 2024-09-15T23:13:04.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:03 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.67:6834/3132447542,v1:172.21.15.67:6835/3132447542] up:boot 2024-09-15T23:13:04.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:03 smithi148 ceph-mon[35610]: fsmap 5 up:standby 2024-09-15T23:13:04.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:03 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:13:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:03 smithi067 ceph-mon[35665]: Deploying daemon mds.f on smithi148 2024-09-15T23:13:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:03 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.67:6834/3132447542,v1:172.21.15.67:6835/3132447542] up:boot 2024-09-15T23:13:04.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:03 smithi067 ceph-mon[35665]: fsmap 5 up:standby 2024-09-15T23:13:04.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:03 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:13:04.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:03 smithi062 ceph-mon[30949]: Deploying daemon mds.f on smithi148 2024-09-15T23:13:04.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:03 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.67:6834/3132447542,v1:172.21.15.67:6835/3132447542] up:boot 2024-09-15T23:13:04.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:03 smithi062 ceph-mon[30949]: fsmap 5 up:standby 2024-09-15T23:13:04.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:03 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:13:04.748 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:13:04 smithi148 systemd[1]: Starting Ceph mds.f for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:13:05.016 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:04 smithi148 ceph-mon[35610]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:05.016 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:13:04 smithi148 podman[66233]: 2024-09-15 23:13:04.649922951 +0000 UTC m=+0.016020736 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:13:05.017 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:13:04 smithi148 podman[66233]: 2024-09-15 23:13:04.780204725 +0000 UTC m=+0.146302537 container create 7463f5940306b8e9ec9f697b31654ddd546baa4d0824325dd29c32e76685389b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True) 2024-09-15T23:13:05.017 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:13:05 smithi148 podman[66233]: 2024-09-15 23:13:05.0161556 +0000 UTC m=+0.382253371 container init 7463f5940306b8e9ec9f697b31654ddd546baa4d0824325dd29c32e76685389b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-15T23:13:05.159 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:13:05.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:04 smithi067 ceph-mon[35665]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:05.289 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:13:05 smithi148 podman[66233]: 2024-09-15 23:13:05.019234178 +0000 UTC m=+0.385331950 container start 7463f5940306b8e9ec9f697b31654ddd546baa4d0824325dd29c32e76685389b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , ceph=True) 2024-09-15T23:13:05.289 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:13:05 smithi148 ceph-mds[66252]: -- 172.21.15.148:0/1413806982 <== mon.1 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a9d4687860 con 0x55a9d4664800 2024-09-15T23:13:05.289 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:13:05 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f[66248]: starting mds.f at 2024-09-15T23:13:05.289 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:13:05 smithi148 bash[66233]: 7463f5940306b8e9ec9f697b31654ddd546baa4d0824325dd29c32e76685389b 2024-09-15T23:13:05.289 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:13:05 smithi148 systemd[1]: Started Ceph mds.f for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:13:05.433 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:05.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:04 smithi062 ceph-mon[30949]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:05.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.905+0000 7f9327199640 1 -- 172.21.15.62:0/132025171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320101de0 msgr2=0x7f93201021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.905+0000 7f9327199640 1 --2- 172.21.15.62:0/132025171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320101de0 0x7f93201021c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9308009900 tx=0x7f930802a410 comp rx=0 tx=0).stop 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.905+0000 7f9327199640 1 -- 172.21.15.62:0/132025171 shutdown_connections 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.905+0000 7f9327199640 1 --2- 172.21.15.62:0/132025171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9320106f10 0x7f932010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.905+0000 7f9327199640 1 --2- 172.21.15.62:0/132025171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9320102790 0x7f93201067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.905+0000 7f9327199640 1 --2- 172.21.15.62:0/132025171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320101de0 0x7f93201021c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.905+0000 7f9327199640 1 -- 172.21.15.62:0/132025171 >> 172.21.15.62:0/132025171 conn(0x7f93200fd8c0 msgr2=0x7f93200ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.905+0000 7f9327199640 1 -- 172.21.15.62:0/132025171 shutdown_connections 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.905+0000 7f9327199640 1 -- 172.21.15.62:0/132025171 wait complete. 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9327199640 1 Processor -- start 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9327199640 1 -- start start 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9327199640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9320101de0 0x7f9320198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:05.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9327199640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320102790 0x7f9320198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9327199640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9320106f10 0x7f932019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9327199640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9320111f70 con 0x7f9320102790 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9327199640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9320111df0 con 0x7f9320101de0 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9327199640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f93201120f0 con 0x7f9320106f10 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f931ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320102790 0x7f9320198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f931ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320102790 0x7f9320198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55446/0 (socket says 172.21.15.62:55446) 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f931ffff640 1 -- 172.21.15.62:0/773306103 learned_addr learned my addr 172.21.15.62:0/773306103 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9324f35640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9320101de0 0x7f9320198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f9325736640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9320106f10 0x7f932019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f931ffff640 1 -- 172.21.15.62:0/773306103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9320106f10 msgr2=0x7f932019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f931ffff640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9320106f10 0x7f932019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f931ffff640 1 -- 172.21.15.62:0/773306103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9320101de0 msgr2=0x7f9320198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:05.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f931ffff640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9320101de0 0x7f9320198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.906+0000 7f931ffff640 1 -- 172.21.15.62:0/773306103 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932019d3e0 con 0x7f9320102790 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.907+0000 7f931ffff640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320102790 0x7f9320198970 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f931000eab0 tx=0x7f931000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.907+0000 7f931dffb640 1 -- 172.21.15.62:0/773306103 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931000ce40 con 0x7f9320102790 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.907+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932019d6d0 con 0x7f9320102790 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.907+0000 7f931dffb640 1 -- 172.21.15.62:0/773306103 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9310011950 con 0x7f9320102790 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.907+0000 7f931dffb640 1 -- 172.21.15.62:0/773306103 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9310010ae0 con 0x7f9320102790 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.907+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93201a4fb0 con 0x7f9320102790 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.908+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93201021c0 con 0x7f9320102790 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.909+0000 7f931dffb640 1 -- 172.21.15.62:0/773306103 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f9310011b00 con 0x7f9320102790 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.909+0000 7f931dffb640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f930c0a3240 0x7f930c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.910+0000 7f9324f35640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f930c0a3240 0x7f930c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:05.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.910+0000 7f931dffb640 1 -- 172.21.15.62:0/773306103 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f93100cfb60 con 0x7f9320102790 2024-09-15T23:13:05.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.910+0000 7f9324f35640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f930c0a3240 0x7f930c0a5700 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9308009230 tx=0x7f93080058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:05.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:05.914+0000 7f931dffb640 1 -- 172.21.15.62:0/773306103 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9310016030 con 0x7f9320102790 2024-09-15T23:13:06.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.124+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f932002d7c0 con 0x7f9320102790 2024-09-15T23:13:06.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.125+0000 7f931dffb640 1 -- 172.21.15.62:0/773306103 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5560 (secure 0 0 0) 0x7f931009cd20 con 0x7f9320102790 2024-09-15T23:13:06.127 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:06.127 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":7,"btime":"2024-09-15T23:13:05:997218+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-09-15T23:13:06.128 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 7 2024-09-15T23:13:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f930c0a3240 msgr2=0x7f930c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f930c0a3240 0x7f930c0a5700 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9308009230 tx=0x7f93080058d0 comp rx=0 tx=0).stop 2024-09-15T23:13:06.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320102790 msgr2=0x7f9320198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320102790 0x7f9320198970 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f931000eab0 tx=0x7f931000ef70 comp rx=0 tx=0).stop 2024-09-15T23:13:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 shutdown_connections 2024-09-15T23:13:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f930c0a3240 0x7f930c0a5700 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9320106f10 0x7f932019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9320102790 0x7f9320198970 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 --2- 172.21.15.62:0/773306103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9320101de0 0x7f9320198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 >> 172.21.15.62:0/773306103 conn(0x7f93200fd8c0 msgr2=0x7f93200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 shutdown_connections 2024-09-15T23:13:06.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:06.127+0000 7f9327199640 1 -- 172.21.15.62:0/773306103 wait complete. 2024-09-15T23:13:06.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:06.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:06.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:06.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:06.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:06.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:06.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: Deploying daemon mds.c on smithi148 2024-09-15T23:13:06.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.148:6834/1239577031,v1:172.21.15.148:6835/1239577031] up:boot 2024-09-15T23:13:06.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: fsmap 6 up:standby 2024-09-15T23:13:06.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:13:06.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:06 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/773306103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:06.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:06.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:06.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:06.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: Deploying daemon mds.c on smithi148 2024-09-15T23:13:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.148:6834/1239577031,v1:172.21.15.148:6835/1239577031] up:boot 2024-09-15T23:13:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: fsmap 6 up:standby 2024-09-15T23:13:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:13:06.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:06 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/773306103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:06.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:06.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:06.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:06.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:06.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:06.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:06.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: Deploying daemon mds.c on smithi148 2024-09-15T23:13:06.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.148:6834/1239577031,v1:172.21.15.148:6835/1239577031] up:boot 2024-09-15T23:13:06.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: fsmap 6 up:standby 2024-09-15T23:13:06.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:13:06.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:06 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/773306103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:07.185 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:13:07 smithi148 systemd[1]: Starting Ceph mds.c for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:13:07.449 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:07 smithi148 ceph-mon[35610]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:07.449 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:13:07 smithi148 podman[66535]: 2024-09-15 23:13:07.230375152 +0000 UTC m=+0.016296233 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:13:07.449 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:13:07 smithi148 podman[66535]: 2024-09-15 23:13:07.335652985 +0000 UTC m=+0.121574064 container create adb39dd76ca19f009b1925b2b31b5cd8a70dc9d7314877dcd95862285d80a975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:13:07.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:07 smithi062 ceph-mon[30949]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:07.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:07 smithi067 ceph-mon[35665]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:08.039 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:13:07 smithi148 podman[66535]: 2024-09-15 23:13:07.58015329 +0000 UTC m=+0.366074370 container init adb39dd76ca19f009b1925b2b31b5cd8a70dc9d7314877dcd95862285d80a975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-15T23:13:08.040 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:13:07 smithi148 podman[66535]: 2024-09-15 23:13:07.582890696 +0000 UTC m=+0.368811784 container start adb39dd76ca19f009b1925b2b31b5cd8a70dc9d7314877dcd95862285d80a975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-15T23:13:08.040 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:13:07 smithi148 ceph-mds[66554]: -- 172.21.15.148:0/1931679235 <== mon.1 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56275772d860 con 0x56275770b400 2024-09-15T23:13:08.040 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:13:07 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c[66550]: starting mds.c at 2024-09-15T23:13:08.040 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:13:07 smithi148 bash[66535]: adb39dd76ca19f009b1925b2b31b5cd8a70dc9d7314877dcd95862285d80a975 2024-09-15T23:13:08.040 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:13:07 smithi148 systemd[1]: Started Ceph mds.c for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:13:08.750 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:13:09.021 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: Deploying daemon mds.h on smithi067 2024-09-15T23:13:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.148:6836/870913016,v1:172.21.15.148:6837/870913016] up:boot 2024-09-15T23:13:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: fsmap 7 up:standby 2024-09-15T23:13:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:08 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:13:09.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:09.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:09.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:09.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: Deploying daemon mds.h on smithi067 2024-09-15T23:13:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.148:6836/870913016,v1:172.21.15.148:6837/870913016] up:boot 2024-09-15T23:13:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: fsmap 7 up:standby 2024-09-15T23:13:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:08 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:13:09.142 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:09.142 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:09.142 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:09.142 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:09.142 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:09.142 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:09.142 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:09.143 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: Deploying daemon mds.h on smithi067 2024-09-15T23:13:09.143 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.148:6836/870913016,v1:172.21.15.148:6837/870913016] up:boot 2024-09-15T23:13:09.143 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: fsmap 7 up:standby 2024-09-15T23:13:09.143 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:08 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:13:09.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.549+0000 7fbe06725640 1 -- 172.21.15.62:0/2116498361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00102570 msgr2=0x7fbe001029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:09.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.549+0000 7fbe06725640 1 --2- 172.21.15.62:0/2116498361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00102570 0x7fbe001029b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009900 tx=0x7fbdf402a480 comp rx=0 tx=0).stop 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.549+0000 7fbe06725640 1 -- 172.21.15.62:0/2116498361 shutdown_connections 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.549+0000 7fbe06725640 1 --2- 172.21.15.62:0/2116498361 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe00102ef0 0x7fbe0010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.549+0000 7fbe06725640 1 --2- 172.21.15.62:0/2116498361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00102570 0x7fbe001029b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.549+0000 7fbe06725640 1 --2- 172.21.15.62:0/2116498361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe00100e20 0x7fbe00101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.549+0000 7fbe06725640 1 -- 172.21.15.62:0/2116498361 >> 172.21.15.62:0/2116498361 conn(0x7fbe000fc7d0 msgr2=0x7fbe000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.549+0000 7fbe06725640 1 -- 172.21.15.62:0/2116498361 shutdown_connections 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.549+0000 7fbe06725640 1 -- 172.21.15.62:0/2116498361 wait complete. 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbe06725640 1 Processor -- start 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbe06725640 1 -- start start 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbe06725640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00100e20 0x7fbe001a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbe06725640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe00102570 0x7fbe001a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:09.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbe06725640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe00102ef0 0x7fbe0019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbe06725640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbe00110ff0 con 0x7fbe00100e20 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbe06725640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe00110e70 con 0x7fbe00102570 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbe06725640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbe00111170 con 0x7fbe00102ef0 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbdfffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00100e20 0x7fbe001a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbdfffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00100e20 0x7fbe001a0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55468/0 (socket says 172.21.15.62:55468) 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbdfffff640 1 -- 172.21.15.62:0/880876832 learned_addr learned my addr 172.21.15.62:0/880876832 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbdff7fe640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe00102570 0x7fbe001a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.550+0000 7fbe04cc2640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe00102ef0 0x7fbe0019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdfffff640 1 -- 172.21.15.62:0/880876832 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe00102ef0 msgr2=0x7fbe0019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdfffff640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe00102ef0 0x7fbe0019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdfffff640 1 -- 172.21.15.62:0/880876832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe00102570 msgr2=0x7fbe001a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdfffff640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe00102570 0x7fbe001a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdfffff640 1 -- 172.21.15.62:0/880876832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0019ad30 con 0x7fbe00100e20 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbe04cc2640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe00102ef0 0x7fbe0019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdff7fe640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe00102570 0x7fbe001a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdfffff640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00100e20 0x7fbe001a0440 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000e9f0 tx=0x7fbdf000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdfd7fa640 1 -- 172.21.15.62:0/880876832 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf000cd80 con 0x7fbe00100e20 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdfd7fa640 1 -- 172.21.15.62:0/880876832 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdf0010910 con 0x7fbe00100e20 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbdfd7fa640 1 -- 172.21.15.62:0/880876832 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0018990 con 0x7fbe00100e20 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0019b020 con 0x7fbe00100e20 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.551+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe001a7310 con 0x7fbe00100e20 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.552+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe00101200 con 0x7fbe00100e20 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.553+0000 7fbdfd7fa640 1 -- 172.21.15.62:0/880876832 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fbdf0010ac0 con 0x7fbe00100e20 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.554+0000 7fbdfd7fa640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbde80a3240 0x7fbde80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.555+0000 7fbdff7fe640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbde80a3240 0x7fbde80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.555+0000 7fbdfd7fa640 1 -- 172.21.15.62:0/880876832 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7fbdf00cf850 con 0x7fbe00100e20 2024-09-15T23:13:09.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.555+0000 7fbdff7fe640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbde80a3240 0x7fbde80a5700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4000c00 tx=0x7fbdf4005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:09.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.559+0000 7fbdfd7fa640 1 -- 172.21.15.62:0/880876832 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbdf009ca10 con 0x7fbe00100e20 2024-09-15T23:13:09.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.772+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe001a74e0 con 0x7fbe00100e20 2024-09-15T23:13:09.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.773+0000 7fbdfd7fa640 1 -- 172.21.15.62:0/880876832 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6385 (secure 0 0 0) 0x7fbdf009c310 con 0x7fbe00100e20 2024-09-15T23:13:09.775 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:09.775 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"btime":"2024-09-15T23:13:08:184882+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-09-15T23:13:09.776 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-09-15T23:13:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbde80a3240 msgr2=0x7fbde80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbde80a3240 0x7fbde80a5700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4000c00 tx=0x7fbdf4005b50 comp rx=0 tx=0).stop 2024-09-15T23:13:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00100e20 msgr2=0x7fbe001a0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:09.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00100e20 0x7fbe001a0440 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbdf000e9f0 tx=0x7fbdf000eeb0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 shutdown_connections 2024-09-15T23:13:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbde80a3240 0x7fbde80a5700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe00102ef0 0x7fbe0019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe00102570 0x7fbe001a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 --2- 172.21.15.62:0/880876832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00100e20 0x7fbe001a0440 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 >> 172.21.15.62:0/880876832 conn(0x7fbe000fc7d0 msgr2=0x7fbe000ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 shutdown_connections 2024-09-15T23:13:09.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:09.775+0000 7fbe06725640 1 -- 172.21.15.62:0/880876832 wait complete. 2024-09-15T23:13:09.989 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:13:09 smithi067 systemd[1]: Starting Ceph mds.h for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:13:10.249 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:13:09 smithi067 podman[65383]: 2024-09-15 23:13:09.890998764 +0000 UTC m=+0.015942359 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:13:10.249 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:13:10 smithi067 podman[65383]: 2024-09-15 23:13:10.004297773 +0000 UTC m=+0.129241351 container create 4273c63666aea844efcb20200d6f0b47eeaf6a16c29451e4a983c9a6b2d80ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-15T23:13:10.249 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:13:10 smithi067 podman[65383]: 2024-09-15 23:13:10.248852074 +0000 UTC m=+0.373795654 container init 4273c63666aea844efcb20200d6f0b47eeaf6a16c29451e4a983c9a6b2d80ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-15T23:13:10.524 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:13:10 smithi067 podman[65383]: 2024-09-15 23:13:10.251767215 +0000 UTC m=+0.376710795 container start 4273c63666aea844efcb20200d6f0b47eeaf6a16c29451e4a983c9a6b2d80ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-15T23:13:10.524 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mds[65402]: -- 172.21.15.67:0/801974147 <== mon.2 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55947bc7bc20 con 0x55947bc59000 2024-09-15T23:13:10.524 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h[65398]: starting mds.h at 2024-09-15T23:13:10.524 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:13:10 smithi067 bash[65383]: 4273c63666aea844efcb20200d6f0b47eeaf6a16c29451e4a983c9a6b2d80ad4 2024-09-15T23:13:10.524 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:13:10 smithi067 systemd[1]: Started Ceph mds.h for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:13:10.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:10 smithi062 ceph-mon[30949]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:10.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:10 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/880876832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:10.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:10.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:10.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:10.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:10.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:10.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:10 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:10.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:10 smithi062 ceph-mon[30949]: Deploying daemon mds.i on smithi148 2024-09-15T23:13:11.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mon[35665]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:11.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/880876832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:11.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:11.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:11.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:11.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:11.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:11.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:11.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:10 smithi067 ceph-mon[35665]: Deploying daemon mds.i on smithi148 2024-09-15T23:13:11.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:10 smithi148 ceph-mon[35610]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:11.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:10 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/880876832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:11.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:11.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:11.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:11.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:11.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:11.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:10 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:11.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:10 smithi148 ceph-mon[35610]: Deploying daemon mds.i on smithi148 2024-09-15T23:13:11.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:11 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] up:boot 2024-09-15T23:13:11.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:11 smithi148 ceph-mon[35610]: fsmap 8 up:standby 2024-09-15T23:13:11.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:11 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:13:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:11 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] up:boot 2024-09-15T23:13:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:11 smithi067 ceph-mon[35665]: fsmap 8 up:standby 2024-09-15T23:13:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:11 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:13:12.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:11 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] up:boot 2024-09-15T23:13:12.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:11 smithi062 ceph-mon[30949]: fsmap 8 up:standby 2024-09-15T23:13:12.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:11 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:13:12.429 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:13:12.526 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:13:12 smithi148 systemd[1]: Starting Ceph mds.i for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:13:12.701 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:12.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:12 smithi148 ceph-mon[35610]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:12.789 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:13:12 smithi148 podman[66833]: 2024-09-15 23:13:12.428115731 +0000 UTC m=+0.015958925 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:13:12.789 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:13:12 smithi148 podman[66833]: 2024-09-15 23:13:12.54995867 +0000 UTC m=+0.137801856 container create bc6ff9a3ba66ad6f927eaca78f4dcf86a0b221af58ec48b3c58c431f11aed998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-15T23:13:13.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:12 smithi067 ceph-mon[35665]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:13.100 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:12 smithi062 ceph-mon[30949]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:13.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.183+0000 7f6bd453e640 1 -- 172.21.15.62:0/1324823853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 msgr2=0x7f6bcc103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:13.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.183+0000 7f6bd453e640 1 --2- 172.21.15.62:0/1324823853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 0x7f6bcc103110 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009930 tx=0x7f6bc802a420 comp rx=0 tx=0).stop 2024-09-15T23:13:13.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.183+0000 7f6bd453e640 1 -- 172.21.15.62:0/1324823853 shutdown_connections 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.183+0000 7f6bd453e640 1 --2- 172.21.15.62:0/1324823853 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6bcc1082a0 0x7f6bcc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.183+0000 7f6bd453e640 1 --2- 172.21.15.62:0/1324823853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bcc1036e0 0x7f6bcc107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.183+0000 7f6bd453e640 1 --2- 172.21.15.62:0/1324823853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 0x7f6bcc103110 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.183+0000 7f6bd453e640 1 -- 172.21.15.62:0/1324823853 >> 172.21.15.62:0/1324823853 conn(0x7f6bcc0fc7d0 msgr2=0x7f6bcc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 -- 172.21.15.62:0/1324823853 shutdown_connections 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 -- 172.21.15.62:0/1324823853 wait complete. 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 Processor -- start 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 -- start start 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 0x7f6bcc1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6bcc1036e0 0x7f6bcc198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bcc1082a0 0x7f6bcc19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc1120a0 con 0x7f6bcc102d30 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc111f20 con 0x7f6bcc1082a0 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.184+0000 7f6bd453e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc112220 con 0x7f6bcc1036e0 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd22da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 0x7f6bcc1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd22da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 0x7f6bcc1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57728/0 (socket says 172.21.15.62:57728) 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd22da640 1 -- 172.21.15.62:0/3740223134 learned_addr learned my addr 172.21.15.62:0/3740223134 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd1ad9640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6bcc1036e0 0x7f6bcc198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd2adb640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bcc1082a0 0x7f6bcc19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd22da640 1 -- 172.21.15.62:0/3740223134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6bcc1036e0 msgr2=0x7f6bcc198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd22da640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6bcc1036e0 0x7f6bcc198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd22da640 1 -- 172.21.15.62:0/3740223134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bcc1082a0 msgr2=0x7f6bcc19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd22da640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bcc1082a0 0x7f6bcc19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd22da640 1 -- 172.21.15.62:0/3740223134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc19d570 con 0x7f6bcc102d30 2024-09-15T23:13:13.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd2adb640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bcc1082a0 0x7f6bcc19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd22da640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 0x7f6bcc1985c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009a50 tx=0x7f6bc8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bc37fe640 1 -- 172.21.15.62:0/3740223134 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc803f030 con 0x7f6bcc102d30 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bcc19d800 con 0x7f6bcc102d30 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bc37fe640 1 -- 172.21.15.62:0/3740223134 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bc8003ad0 con 0x7f6bcc102d30 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bc37fe640 1 -- 172.21.15.62:0/3740223134 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8035070 con 0x7f6bcc102d30 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.185+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bcc1a5140 con 0x7f6bcc102d30 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.187+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bcc103110 con 0x7f6bcc102d30 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.188+0000 7f6bc37fe640 1 -- 172.21.15.62:0/3740223134 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6bc8042030 con 0x7f6bcc102d30 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.189+0000 7f6bc37fe640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6bb80a3240 0x7f6bb80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.189+0000 7f6bd1ad9640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6bb80a3240 0x7f6bb80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.189+0000 7f6bc37fe640 1 -- 172.21.15.62:0/3740223134 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f6bc80efc00 con 0x7f6bcc102d30 2024-09-15T23:13:13.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.189+0000 7f6bd1ad9640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6bb80a3240 0x7f6bb80a5700 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0fee30 tx=0x7f6bbc00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:13.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.193+0000 7f6bc37fe640 1 -- 172.21.15.62:0/3740223134 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6bc80f6080 con 0x7f6bcc102d30 2024-09-15T23:13:13.289 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:13:12 smithi148 podman[66833]: 2024-09-15 23:13:12.836215876 +0000 UTC m=+0.424059070 container init bc6ff9a3ba66ad6f927eaca78f4dcf86a0b221af58ec48b3c58c431f11aed998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-15T23:13:13.289 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:13:12 smithi148 podman[66833]: 2024-09-15 23:13:12.839087646 +0000 UTC m=+0.426930831 container start bc6ff9a3ba66ad6f927eaca78f4dcf86a0b221af58ec48b3c58c431f11aed998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-15T23:13:13.290 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:13:12 smithi148 ceph-mds[66852]: -- 172.21.15.148:0/2145030420 <== mon.1 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e48b60d860 con 0x55e48b5eb000 2024-09-15T23:13:13.290 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:13:12 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i[66848]: starting mds.i at 2024-09-15T23:13:13.290 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:13:12 smithi148 bash[66833]: bc6ff9a3ba66ad6f927eaca78f4dcf86a0b221af58ec48b3c58c431f11aed998 2024-09-15T23:13:13.290 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:13:12 smithi148 systemd[1]: Started Ceph mds.i for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:13:13.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.432+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bcc10a2b0 con 0x7f6bcc102d30 2024-09-15T23:13:13.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.432+0000 7f6bc37fe640 1 -- 172.21.15.62:0/3740223134 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7207 (secure 0 0 0) 0x7f6bc80bcdc0 con 0x7f6bcc102d30 2024-09-15T23:13:13.435 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:13.435 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":9,"btime":"2024-09-15T23:13:10:732351+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-15T23:13:13.435 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 9 2024-09-15T23:13:13.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6bb80a3240 msgr2=0x7f6bb80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:13.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6bb80a3240 0x7f6bb80a5700 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc0fee30 tx=0x7f6bbc00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 msgr2=0x7f6bcc1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:13.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 0x7f6bcc1985c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009a50 tx=0x7f6bc8004270 comp rx=0 tx=0).stop 2024-09-15T23:13:13.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 shutdown_connections 2024-09-15T23:13:13.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6bb80a3240 0x7f6bb80a5700 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6bcc1082a0 0x7f6bcc19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6bcc1036e0 0x7f6bcc198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 --2- 172.21.15.62:0/3740223134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bcc102d30 0x7f6bcc1985c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:13.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 >> 172.21.15.62:0/3740223134 conn(0x7f6bcc0fc7d0 msgr2=0x7f6bcc105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:13.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 shutdown_connections 2024-09-15T23:13:13.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:13.435+0000 7f6bd453e640 1 -- 172.21.15.62:0/3740223134 wait complete. 2024-09-15T23:13:14.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:14.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:14.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:14.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:14.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:14.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:14.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: Deploying daemon mds.g on smithi062 2024-09-15T23:13:14.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3740223134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:14.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] up:boot 2024-09-15T23:13:14.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: fsmap 9 up:standby 2024-09-15T23:13:14.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:13 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:13:14.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:13 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:14.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:14.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:14.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:14.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:14.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:14.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: Deploying daemon mds.g on smithi062 2024-09-15T23:13:14.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3740223134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:14.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] up:boot 2024-09-15T23:13:14.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: fsmap 9 up:standby 2024-09-15T23:13:14.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:14 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:13:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: Deploying daemon mds.g on smithi062 2024-09-15T23:13:14.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3740223134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:14.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] up:boot 2024-09-15T23:13:14.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: fsmap 9 up:standby 2024-09-15T23:13:14.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:14 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:13:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:15 smithi148 ceph-mon[35610]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:15.303 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:15 smithi062 ceph-mon[30949]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:15.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:15 smithi067 ceph-mon[35665]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:15.980 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:13:15 smithi062 systemd[1]: Starting Ceph mds.g for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:13:16.258 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:13:15 smithi062 podman[69045]: 2024-09-15 23:13:15.884471097 +0000 UTC m=+0.016362197 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:13:16.258 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:13:16 smithi062 podman[69045]: 2024-09-15 23:13:16.021637008 +0000 UTC m=+0.153528103 container create 7c00931df3a67b1c2f94ca312e13f3e94274d6cd44b3d0eba404506a00e2c9a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-g, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-15T23:13:16.258 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:13:16 smithi062 podman[69045]: 2024-09-15 23:13:16.257380556 +0000 UTC m=+0.389271647 container init 7c00931df3a67b1c2f94ca312e13f3e94274d6cd44b3d0eba404506a00e2c9a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-g, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-15T23:13:16.316 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:13:16.585 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:16.708 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:13:16 smithi062 podman[69045]: 2024-09-15 23:13:16.260635916 +0000 UTC m=+0.392527014 container start 7c00931df3a67b1c2f94ca312e13f3e94274d6cd44b3d0eba404506a00e2c9a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-g, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-15T23:13:16.708 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:13:16 smithi062 ceph-mds[69064]: -- 172.21.15.62:0/3441237109 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56509a2ff860 con 0x56509a2dc800 2024-09-15T23:13:16.708 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:13:16 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-g[69060]: starting mds.g at 2024-09-15T23:13:16.708 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:13:16 smithi062 bash[69045]: 7c00931df3a67b1c2f94ca312e13f3e94274d6cd44b3d0eba404506a00e2c9a1 2024-09-15T23:13:16.708 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:13:16 smithi062 systemd[1]: Started Ceph mds.g for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:13:17.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 -- 172.21.15.62:0/1881593425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 msgr2=0x7fe880112650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:17.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 --2- 172.21.15.62:0/1881593425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 0x7fe880112650 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe86c009900 tx=0x7fe86c02a480 comp rx=0 tx=0).stop 2024-09-15T23:13:17.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 -- 172.21.15.62:0/1881593425 shutdown_connections 2024-09-15T23:13:17.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 --2- 172.21.15.62:0/1881593425 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe880112b90 0x7fe880114f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 --2- 172.21.15.62:0/1881593425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 0x7fe880112650 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 --2- 172.21.15.62:0/1881593425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe880074590 0x7fe880074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 -- 172.21.15.62:0/1881593425 >> 172.21.15.62:0/1881593425 conn(0x7fe8800fd8c0 msgr2=0x7fe8800ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 -- 172.21.15.62:0/1881593425 shutdown_connections 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 -- 172.21.15.62:0/1881593425 wait complete. 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 Processor -- start 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.072+0000 7fe886d55640 1 -- start start 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe886d55640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe880074590 0x7fe8801a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe886d55640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 0x7fe8801a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe886d55640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe880112b90 0x7fe88019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe886d55640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe8801185d0 con 0x7fe880074eb0 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe886d55640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe880118450 con 0x7fe880074590 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe886d55640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe880118750 con 0x7fe880112b90 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 0x7fe8801a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:17.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 0x7fe8801a4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57750/0 (socket says 172.21.15.62:57750) 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87ffff640 1 -- 172.21.15.62:0/657807386 learned_addr learned my addr 172.21.15.62:0/657807386 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe884af1640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe880074590 0x7fe8801a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe8852f2640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe880112b90 0x7fe88019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87ffff640 1 -- 172.21.15.62:0/657807386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe880112b90 msgr2=0x7fe88019e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87ffff640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe880112b90 0x7fe88019e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87ffff640 1 -- 172.21.15.62:0/657807386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe880074590 msgr2=0x7fe8801a4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87ffff640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe880074590 0x7fe8801a4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87ffff640 1 -- 172.21.15.62:0/657807386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88019f060 con 0x7fe880074eb0 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87ffff640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 0x7fe8801a4cb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe86c006010 tx=0x7fe86c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.073+0000 7fe87dffb640 1 -- 172.21.15.62:0/657807386 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c03f030 con 0x7fe880074eb0 2024-09-15T23:13:17.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.074+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88019f2f0 con 0x7fe880074eb0 2024-09-15T23:13:17.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.074+0000 7fe87dffb640 1 -- 172.21.15.62:0/657807386 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe86c003ce0 con 0x7fe880074eb0 2024-09-15T23:13:17.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.074+0000 7fe87dffb640 1 -- 172.21.15.62:0/657807386 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c035070 con 0x7fe880074eb0 2024-09-15T23:13:17.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.074+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8801ab640 con 0x7fe880074eb0 2024-09-15T23:13:17.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.075+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe880074970 con 0x7fe880074eb0 2024-09-15T23:13:17.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.076+0000 7fe87dffb640 1 -- 172.21.15.62:0/657807386 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe86c042030 con 0x7fe880074eb0 2024-09-15T23:13:17.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.076+0000 7fe87dffb640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8700a3240 0x7fe8700a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:17.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.077+0000 7fe884af1640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8700a3240 0x7fe8700a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:17.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.077+0000 7fe87dffb640 1 -- 172.21.15.62:0/657807386 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7fe86c0efb30 con 0x7fe880074eb0 2024-09-15T23:13:17.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.077+0000 7fe884af1640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8700a3240 0x7fe8700a5700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe88019fa70 tx=0x7fe87400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:17.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.081+0000 7fe87dffb640 1 -- 172.21.15.62:0/657807386 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe86c0bcdb0 con 0x7fe880074eb0 2024-09-15T23:13:17.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.298+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe880105000 con 0x7fe880074eb0 2024-09-15T23:13:17.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.299+0000 7fe87dffb640 1 -- 172.21.15.62:0/657807386 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8863 (secure 0 0 0) 0x7fe86c0bc6b0 con 0x7fe880074eb0 2024-09-15T23:13:17.300 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:17.300 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":11,"btime":"2024-09-15T23:13:17:025004+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24466,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-15T23:13:17.300 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 11 2024-09-15T23:13:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8700a3240 msgr2=0x7fe8700a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8700a3240 0x7fe8700a5700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe88019fa70 tx=0x7fe87400b390 comp rx=0 tx=0).stop 2024-09-15T23:13:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 msgr2=0x7fe8801a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 0x7fe8801a4cb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe86c006010 tx=0x7fe86c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 shutdown_connections 2024-09-15T23:13:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe8700a3240 0x7fe8700a5700 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe880112b90 0x7fe88019e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe880074eb0 0x7fe8801a4cb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 --2- 172.21.15.62:0/657807386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe880074590 0x7fe8801a4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 >> 172.21.15.62:0/657807386 conn(0x7fe8800fd8c0 msgr2=0x7fe8800ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 shutdown_connections 2024-09-15T23:13:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:17.301+0000 7fe886d55640 1 -- 172.21.15.62:0/657807386 wait complete. 2024-09-15T23:13:17.641 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:17.641 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:17.641 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:17.641 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:17.641 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:17.642 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:17.642 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:17.642 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: Deploying daemon mds.a on smithi062 2024-09-15T23:13:17.642 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.62:6838/2389718073,v1:172.21.15.62:6839/2389718073] up:boot 2024-09-15T23:13:17.642 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: fsmap 10 up:standby 2024-09-15T23:13:17.642 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T23:13:17.642 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:17 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/657807386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:17.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:17.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:17.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:17.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: Deploying daemon mds.a on smithi062 2024-09-15T23:13:17.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.62:6838/2389718073,v1:172.21.15.62:6839/2389718073] up:boot 2024-09-15T23:13:17.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: fsmap 10 up:standby 2024-09-15T23:13:17.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T23:13:17.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:17 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/657807386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:17.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:17.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:17.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:17.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:17.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:17.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:17.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:17.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: Deploying daemon mds.a on smithi062 2024-09-15T23:13:17.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.62:6838/2389718073,v1:172.21.15.62:6839/2389718073] up:boot 2024-09-15T23:13:17.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: fsmap 10 up:standby 2024-09-15T23:13:17.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-09-15T23:13:17.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:17 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/657807386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:19.427 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:13:19 smithi062 systemd[1]: Starting Ceph mds.a for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:13:19.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:19 smithi062 ceph-mon[30949]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:19.730 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:13:19 smithi062 podman[69459]: 2024-09-15 23:13:19.397572435 +0000 UTC m=+0.016631986 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:13:19.730 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:13:19 smithi062 podman[69459]: 2024-09-15 23:13:19.518994629 +0000 UTC m=+0.138054171 container create 334aab8fbdd1e62e8d3e52599cfdfb40536d9fb44372308f03d72535c42e66aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-a, RELEASE=squid-8921c8e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-15T23:13:19.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:19 smithi067 ceph-mon[35665]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:19.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:19 smithi148 ceph-mon[35610]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:20.230 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:13:19 smithi062 podman[69459]: 2024-09-15 23:13:19.738095049 +0000 UTC m=+0.357154593 container init 334aab8fbdd1e62e8d3e52599cfdfb40536d9fb44372308f03d72535c42e66aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:13:20.230 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:13:19 smithi062 podman[69459]: 2024-09-15 23:13:19.741054034 +0000 UTC m=+0.360113584 container start 334aab8fbdd1e62e8d3e52599cfdfb40536d9fb44372308f03d72535c42e66aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:13:20.230 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:13:19 smithi062 ceph-mds[69478]: -- 172.21.15.62:0/8629010 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56336ef57860 con 0x56336ef34800 2024-09-15T23:13:20.230 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:13:19 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-a[69474]: starting mds.a at 2024-09-15T23:13:20.230 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:13:19 smithi062 bash[69459]: 334aab8fbdd1e62e8d3e52599cfdfb40536d9fb44372308f03d72535c42e66aa 2024-09-15T23:13:20.230 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:13:19 smithi062 systemd[1]: Started Ceph mds.a for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:13:20.429 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:13:20.704 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: Deploying daemon mds.b on smithi067 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.62:6840/3484718619,v1:172.21.15.62:6841/3484718619] up:boot 2024-09-15T23:13:21.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: fsmap 11 up:standby 2024-09-15T23:13:21.085 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:20 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T23:13:21.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.165+0000 7f65be999640 1 -- 172.21.15.62:0/872921359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 msgr2=0x7f65b81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:21.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.165+0000 7f65be999640 1 --2- 172.21.15.62:0/872921359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 0x7f65b81067e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f65ac009900 tx=0x7f65ac02a410 comp rx=0 tx=0).stop 2024-09-15T23:13:21.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.165+0000 7f65be999640 1 -- 172.21.15.62:0/872921359 shutdown_connections 2024-09-15T23:13:21.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.165+0000 7f65be999640 1 --2- 172.21.15.62:0/872921359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65b8106f10 0x7f65b810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.165+0000 7f65be999640 1 --2- 172.21.15.62:0/872921359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 0x7f65b81067e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.165+0000 7f65be999640 1 --2- 172.21.15.62:0/872921359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b8101de0 0x7f65b81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.165+0000 7f65be999640 1 -- 172.21.15.62:0/872921359 >> 172.21.15.62:0/872921359 conn(0x7f65b80fd8c0 msgr2=0x7f65b80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.165+0000 7f65be999640 1 -- 172.21.15.62:0/872921359 shutdown_connections 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.165+0000 7f65be999640 1 -- 172.21.15.62:0/872921359 wait complete. 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.166+0000 7f65be999640 1 Processor -- start 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.166+0000 7f65be999640 1 -- start start 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.166+0000 7f65be999640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b8101de0 0x7f65b8198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.166+0000 7f65be999640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 0x7f65b81989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.166+0000 7f65be999640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65b8106f10 0x7f65b819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.166+0000 7f65be999640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f65b8111f70 con 0x7f65b8102790 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.166+0000 7f65be999640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65b8111df0 con 0x7f65b8101de0 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.166+0000 7f65be999640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f65b81120f0 con 0x7f65b8106f10 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 0x7f65b81989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:21.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 0x7f65b81989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57782/0 (socket says 172.21.15.62:57782) 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b77fe640 1 -- 172.21.15.62:0/3922917164 learned_addr learned my addr 172.21.15.62:0/3922917164 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65bcf36640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65b8106f10 0x7f65b819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b7fff640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b8101de0 0x7f65b8198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b77fe640 1 -- 172.21.15.62:0/3922917164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65b8106f10 msgr2=0x7f65b819cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b77fe640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65b8106f10 0x7f65b819cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b77fe640 1 -- 172.21.15.62:0/3922917164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b8101de0 msgr2=0x7f65b8198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b77fe640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b8101de0 0x7f65b8198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b77fe640 1 -- 172.21.15.62:0/3922917164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b819d420 con 0x7f65b8102790 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b7fff640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b8101de0 0x7f65b8198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b77fe640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 0x7f65b81989b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006010 tx=0x7f65ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b57fa640 1 -- 172.21.15.62:0/3922917164 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac03f030 con 0x7f65b8102790 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b57fa640 1 -- 172.21.15.62:0/3922917164 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65ac003ce0 con 0x7f65b8102790 2024-09-15T23:13:21.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65b57fa640 1 -- 172.21.15.62:0/3922917164 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ac035070 con 0x7f65b8102790 2024-09-15T23:13:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b819d6b0 con 0x7f65b8102790 2024-09-15T23:13:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.167+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b81a4ff0 con 0x7f65b8102790 2024-09-15T23:13:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.169+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b81032c0 con 0x7f65b8102790 2024-09-15T23:13:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.170+0000 7f65b57fa640 1 -- 172.21.15.62:0/3922917164 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f65ac042030 con 0x7f65b8102790 2024-09-15T23:13:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.171+0000 7f65b57fa640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65a00a3240 0x7f65a00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.171+0000 7f65b7fff640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65a00a3240 0x7f65a00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.171+0000 7f65b57fa640 1 -- 172.21.15.62:0/3922917164 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f65ac0efc20 con 0x7f65b8102790 2024-09-15T23:13:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.172+0000 7f65b7fff640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65a00a3240 0x7f65a00a5700 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009c20 tx=0x7f65a8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:21.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.175+0000 7f65b57fa640 1 -- 172.21.15.62:0/3922917164 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f65ac0bcea0 con 0x7f65b8102790 2024-09-15T23:13:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:21.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:21.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: Deploying daemon mds.b on smithi067 2024-09-15T23:13:21.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.62:6840/3484718619,v1:172.21.15.62:6841/3484718619] up:boot 2024-09-15T23:13:21.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: fsmap 11 up:standby 2024-09-15T23:13:21.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:20 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T23:13:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-15T23:13:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-15T23:13:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:21.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: Deploying daemon mds.b on smithi067 2024-09-15T23:13:21.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.62:6840/3484718619,v1:172.21.15.62:6841/3484718619] up:boot 2024-09-15T23:13:21.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: fsmap 11 up:standby 2024-09-15T23:13:21.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:20 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-09-15T23:13:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.384+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65b81021c0 con 0x7f65b8102790 2024-09-15T23:13:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.385+0000 7f65b57fa640 1 -- 172.21.15.62:0/3922917164 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9689 (secure 0 0 0) 0x7f65ac0bc7a0 con 0x7f65b8102790 2024-09-15T23:13:21.389 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:21.389 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":12,"btime":"2024-09-15T23:13:20:427504+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24466,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 12 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.387+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65a00a3240 msgr2=0x7f65a00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.387+0000 7f65be999640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65a00a3240 0x7f65a00a5700 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009c20 tx=0x7f65a8005f90 comp rx=0 tx=0).stop 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.387+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 msgr2=0x7f65b81989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.387+0000 7f65be999640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 0x7f65b81989b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f65ac006010 tx=0x7f65ac0042f0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.388+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 shutdown_connections 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.388+0000 7f65be999640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65a00a3240 0x7f65a00a5700 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.388+0000 7f65be999640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65b8106f10 0x7f65b819cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.388+0000 7f65be999640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65b8102790 0x7f65b81989b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.388+0000 7f65be999640 1 --2- 172.21.15.62:0/3922917164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65b8101de0 0x7f65b8198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.388+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 >> 172.21.15.62:0/3922917164 conn(0x7f65b80fd8c0 msgr2=0x7f65b80fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.388+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 shutdown_connections 2024-09-15T23:13:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:21.388+0000 7f65be999640 1 -- 172.21.15.62:0/3922917164 wait complete. 2024-09-15T23:13:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:21 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3922917164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:22.025 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:21 smithi067 systemd[1]: Starting Ceph mds.b for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:13:22.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:21 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3922917164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:22.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:21 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3922917164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:22.355 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:22 smithi067 podman[65695]: 2024-09-15 23:13:21.994113741 +0000 UTC m=+0.016581240 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:13:22.356 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:22 smithi067 podman[65695]: 2024-09-15 23:13:22.093173662 +0000 UTC m=+0.115641202 container create c2da18fa89c1be1359be7ec9d2da06472591065ddab16b18dbf7caec8a633b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e) 2024-09-15T23:13:22.356 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:22 smithi067 podman[65695]: 2024-09-15 23:13:22.329215553 +0000 UTC m=+0.351683048 container init c2da18fa89c1be1359be7ec9d2da06472591065ddab16b18dbf7caec8a633b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:13:22.356 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:22 smithi067 podman[65695]: 2024-09-15 23:13:22.332235653 +0000 UTC m=+0.354703141 container start c2da18fa89c1be1359be7ec9d2da06472591065ddab16b18dbf7caec8a633b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e) 2024-09-15T23:13:22.774 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:22 smithi067 ceph-mds[65714]: -- 172.21.15.67:0/3294814120 <== mon.2 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55598e18b860 con 0x55598e169000 2024-09-15T23:13:22.774 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:22 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: starting mds.b at 2024-09-15T23:13:22.774 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:22 smithi067 bash[65695]: c2da18fa89c1be1359be7ec9d2da06472591065ddab16b18dbf7caec8a633b9d 2024-09-15T23:13:22.774 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:22 smithi067 systemd[1]: Started Ceph mds.b for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:13:23.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:23.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:13:23.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] up:boot 2024-09-15T23:13:23.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: fsmap 12 up:standby 2024-09-15T23:13:23.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:23 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:13:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:13:23.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] up:boot 2024-09-15T23:13:23.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: fsmap 12 up:standby 2024-09-15T23:13:23.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:23 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:13:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:13:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] up:boot 2024-09-15T23:13:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: fsmap 12 up:standby 2024-09-15T23:13:23.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:23 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:13:24.027 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:13:24.298 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:24.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.742+0000 7fabf872a640 1 -- 172.21.15.62:0/1712526404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0102d90 msgr2=0x7fabf0103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:24.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.742+0000 7fabf872a640 1 --2- 172.21.15.62:0/1712526404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0102d90 0x7fabf0103170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fabec009890 tx=0x7fabec02a380 comp rx=0 tx=0).stop 2024-09-15T23:13:24.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.742+0000 7fabf872a640 1 -- 172.21.15.62:0/1712526404 shutdown_connections 2024-09-15T23:13:24.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.742+0000 7fabf872a640 1 --2- 172.21.15.62:0/1712526404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabf0108290 0x7fabf010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.742+0000 7fabf872a640 1 --2- 172.21.15.62:0/1712526404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf0103740 0x7fabf0107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.742+0000 7fabf872a640 1 --2- 172.21.15.62:0/1712526404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0102d90 0x7fabf0103170 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.742+0000 7fabf872a640 1 -- 172.21.15.62:0/1712526404 >> 172.21.15.62:0/1712526404 conn(0x7fabf00fc7d0 msgr2=0x7fabf00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:24.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.742+0000 7fabf872a640 1 -- 172.21.15.62:0/1712526404 shutdown_connections 2024-09-15T23:13:24.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.742+0000 7fabf872a640 1 -- 172.21.15.62:0/1712526404 wait complete. 2024-09-15T23:13:24.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf872a640 1 Processor -- start 2024-09-15T23:13:24.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf872a640 1 -- start start 2024-09-15T23:13:24.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf872a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf0102d90 0x7fabf01009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:24.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf872a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabf0103740 0x7fabf0100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:24.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf872a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0108290 0x7fabf0101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:24.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf872a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fabf0112090 con 0x7fabf0102d90 2024-09-15T23:13:24.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf872a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fabf0111f10 con 0x7fabf0108290 2024-09-15T23:13:24.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf872a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fabf0112210 con 0x7fabf0103740 2024-09-15T23:13:24.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf6cc7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0108290 0x7fabf0101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:24.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf5cc5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabf0103740 0x7fabf0100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:24.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf6cc7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0108290 0x7fabf0101470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:55964/0 (socket says 172.21.15.62:55964) 2024-09-15T23:13:24.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf6cc7640 1 -- 172.21.15.62:0/1612139595 learned_addr learned my addr 172.21.15.62:0/1612139595 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:24.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf64c6640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf0102d90 0x7fabf01009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:24.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf6cc7640 1 -- 172.21.15.62:0/1612139595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabf0103740 msgr2=0x7fabf0100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:24.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf6cc7640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabf0103740 0x7fabf0100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf6cc7640 1 -- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf0102d90 msgr2=0x7fabf01009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:24.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf6cc7640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf0102d90 0x7fabf01009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.743+0000 7fabf6cc7640 1 -- 172.21.15.62:0/1612139595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf01a6a70 con 0x7fabf0108290 2024-09-15T23:13:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.744+0000 7fabf6cc7640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0108290 0x7fabf0101470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fabe800eb10 tx=0x7fabe800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.744+0000 7fabe77fe640 1 -- 172.21.15.62:0/1612139595 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe8004b20 con 0x7fabf0108290 2024-09-15T23:13:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.744+0000 7fabe77fe640 1 -- 172.21.15.62:0/1612139595 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabe8004cc0 con 0x7fabf0108290 2024-09-15T23:13:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.744+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf01a6d60 con 0x7fabf0108290 2024-09-15T23:13:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.744+0000 7fabe77fe640 1 -- 172.21.15.62:0/1612139595 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe8010760 con 0x7fabf0108290 2024-09-15T23:13:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.744+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf01a7250 con 0x7fabf0108290 2024-09-15T23:13:24.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.745+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf006a7f0 con 0x7fabf0108290 2024-09-15T23:13:24.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.746+0000 7fabe77fe640 1 -- 172.21.15.62:0/1612139595 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fabe8010900 con 0x7fabf0108290 2024-09-15T23:13:24.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.747+0000 7fabe77fe640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fabdc0a3240 0x7fabdc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:24.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.747+0000 7fabf64c6640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fabdc0a3240 0x7fabdc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:24.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.747+0000 7fabe77fe640 1 -- 172.21.15.62:0/1612139595 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7fabe80cfcd0 con 0x7fabf0108290 2024-09-15T23:13:24.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.748+0000 7fabf64c6640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fabdc0a3240 0x7fabdc0a5700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fabec0097b0 tx=0x7fabec0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:24.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.750+0000 7fabe77fe640 1 -- 172.21.15.62:0/1612139595 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fabe809cdf0 con 0x7fabf0108290 2024-09-15T23:13:24.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.965+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabf0103170 con 0x7fabf0108290 2024-09-15T23:13:24.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.966+0000 7fabe77fe640 1 -- 172.21.15.62:0/1612139595 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10512 (secure 0 0 0) 0x7fabe809c6f0 con 0x7fabf0108290 2024-09-15T23:13:24.968 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:24.968 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":13,"btime":"2024-09-15T23:13:22:875772+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24466,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24497,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-09-15T23:13:24.968 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 13 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.968+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fabdc0a3240 msgr2=0x7fabdc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.968+0000 7fabf872a640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fabdc0a3240 0x7fabdc0a5700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fabec0097b0 tx=0x7fabec0058d0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0108290 msgr2=0x7fabf0101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0108290 0x7fabf0101470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fabe800eb10 tx=0x7fabe800c2d0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 shutdown_connections 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fabdc0a3240 0x7fabdc0a5700 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabf0108290 0x7fabf0101470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fabf0103740 0x7fabf0100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 --2- 172.21.15.62:0/1612139595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fabf0102d90 0x7fabf01009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 >> 172.21.15.62:0/1612139595 conn(0x7fabf00fc7d0 msgr2=0x7fabf0105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 shutdown_connections 2024-09-15T23:13:24.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:24.969+0000 7fabf872a640 1 -- 172.21.15.62:0/1612139595 wait complete. 2024-09-15T23:13:25.399 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:25.399 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.399 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.399 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.399 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.399 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.399 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.400 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:25.400 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:13:25.400 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.400 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:13:25.400 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.400 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:25 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1612139595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:25.425 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:25.425 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.425 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.425 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.425 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.425 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.425 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.425 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:25.425 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:13:25.426 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.426 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:13:25.426 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.426 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:25 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1612139595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:25.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:13:25.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:13:25.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:25.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:25 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1612139595' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:25.660 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-09-15T23:13:26.362 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.812+0000 7f13db643640 1 -- 172.21.15.62:0/3882264291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4106a80 msgr2=0x7f13d4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.812+0000 7f13db643640 1 --2- 172.21.15.62:0/3882264291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4106a80 0x7f13d4106e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009930 tx=0x7f13d002a420 comp rx=0 tx=0).stop 2024-09-15T23:13:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.812+0000 7f13db643640 1 -- 172.21.15.62:0/3882264291 shutdown_connections 2024-09-15T23:13:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.812+0000 7f13db643640 1 --2- 172.21.15.62:0/3882264291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4101420 0x7f13d410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.812+0000 7f13db643640 1 --2- 172.21.15.62:0/3882264291 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13d4100a80 0x7f13d4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.812+0000 7f13db643640 1 --2- 172.21.15.62:0/3882264291 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4106a80 0x7f13d4106e60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.812+0000 7f13db643640 1 -- 172.21.15.62:0/3882264291 >> 172.21.15.62:0/3882264291 conn(0x7f13d40fc7a0 msgr2=0x7f13d40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.812+0000 7f13db643640 1 -- 172.21.15.62:0/3882264291 shutdown_connections 2024-09-15T23:13:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.812+0000 7f13db643640 1 -- 172.21.15.62:0/3882264291 wait complete. 2024-09-15T23:13:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13db643640 1 Processor -- start 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13db643640 1 -- start start 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13db643640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4100a80 0x7f13d419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13db643640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4101420 0x7f13d419abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13db643640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13d4106a80 0x7f13d419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13db643640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f13d410fd70 con 0x7f13d4100a80 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13db643640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13d410fbf0 con 0x7f13d4101420 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13db643640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f13d410fef0 con 0x7f13d4106a80 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13d93df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4100a80 0x7f13d419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13d93df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4100a80 0x7f13d419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50684/0 (socket says 172.21.15.62:50684) 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13d93df640 1 -- 172.21.15.62:0/2535128651 learned_addr learned my addr 172.21.15.62:0/2535128651 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.813+0000 7f13d8bde640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4101420 0x7f13d419abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13d9be0640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13d4106a80 0x7f13d419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13d93df640 1 -- 172.21.15.62:0/2535128651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13d4106a80 msgr2=0x7f13d419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13d93df640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13d4106a80 0x7f13d419ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13d93df640 1 -- 172.21.15.62:0/2535128651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4101420 msgr2=0x7f13d419abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13d93df640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4101420 0x7f13d419abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13d93df640 1 -- 172.21.15.62:0/2535128651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d419f650 con 0x7f13d4100a80 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13d8bde640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4101420 0x7f13d419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13d93df640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4100a80 0x7f13d419a6a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009230 tx=0x7f13d0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13ca7fc640 1 -- 172.21.15.62:0/2535128651 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d003f030 con 0x7f13d4100a80 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d419f8e0 con 0x7f13d4100a80 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13ca7fc640 1 -- 172.21.15.62:0/2535128651 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d0003ad0 con 0x7f13d4100a80 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13ca7fc640 1 -- 172.21.15.62:0/2535128651 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0035070 con 0x7f13d4100a80 2024-09-15T23:13:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.814+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d41a7220 con 0x7f13d4100a80 2024-09-15T23:13:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.815+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d4100ee0 con 0x7f13d4100a80 2024-09-15T23:13:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.816+0000 7f13ca7fc640 1 -- 172.21.15.62:0/2535128651 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f13d0042030 con 0x7f13d4100a80 2024-09-15T23:13:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.817+0000 7f13ca7fc640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13c00a3230 0x7f13c00a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:26.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.817+0000 7f13d8bde640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13c00a3230 0x7f13c00a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:26.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.817+0000 7f13ca7fc640 1 -- 172.21.15.62:0/2535128651 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f13d00efe60 con 0x7f13d4100a80 2024-09-15T23:13:26.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.818+0000 7f13d8bde640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13c00a3230 0x7f13c00a56f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f13c40058c0 tx=0x7f13c4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:26.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:26.821+0000 7f13ca7fc640 1 -- 172.21.15.62:0/2535128651 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f13d00bd020 con 0x7f13d4100a80 2024-09-15T23:13:27.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.019+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f13d4108ff0 con 0x7f13d4100a80 2024-09-15T23:13:27.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.020+0000 7f13ca7fc640 1 -- 172.21.15.62:0/2535128651 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) ==== 84+0+65 (secure 0 0 0) 0x7f13d00bc920 con 0x7f13d4100a80 2024-09-15T23:13:27.021 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:27.021 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-09-15T23:13:27.021 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:13:27.022 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-09-15T23:13:27.022 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-09-15T23:13:27.022 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:13:27.022 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-09-15T23:13:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13c00a3230 msgr2=0x7f13c00a56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13c00a3230 0x7f13c00a56f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f13c40058c0 tx=0x7f13c4005850 comp rx=0 tx=0).stop 2024-09-15T23:13:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4100a80 msgr2=0x7f13d419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:27.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4100a80 0x7f13d419a6a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f13d0009230 tx=0x7f13d0004270 comp rx=0 tx=0).stop 2024-09-15T23:13:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 shutdown_connections 2024-09-15T23:13:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13c00a3230 0x7f13c00a56f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13d4106a80 0x7f13d419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13d4101420 0x7f13d419abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 --2- 172.21.15.62:0/2535128651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13d4100a80 0x7f13d419a6a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 >> 172.21.15.62:0/2535128651 conn(0x7f13d40fc7a0 msgr2=0x7f13d40fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 shutdown_connections 2024-09-15T23:13:27.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:27.022+0000 7f13db643640 1 -- 172.21.15.62:0/2535128651 wait complete. 2024-09-15T23:13:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:27 smithi062 ceph-mon[30949]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:27 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2535128651' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-15T23:13:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:27 smithi067 ceph-mon[35665]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:27 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2535128651' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-15T23:13:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:27 smithi148 ceph-mon[35610]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:27 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2535128651' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-09-15T23:13:27.813 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-15T23:13:27.813 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-15T23:13:28.068 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:28.464 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:28.464 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:28.464 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:28.464 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:13:28.464 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:28 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:28.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.552+0000 7f3dfefaa640 1 -- 172.21.15.62:0/212100731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 msgr2=0x7f3df8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.552+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/212100731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 0x7f3df8101200 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3dec009900 tx=0x7f3dec02a480 comp rx=0 tx=0).stop 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.552+0000 7f3dfefaa640 1 -- 172.21.15.62:0/212100731 shutdown_connections 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.552+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/212100731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df8102ef0 0x7f3df810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.552+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/212100731 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3df8102570 0x7f3df81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.552+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/212100731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 0x7f3df8101200 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.552+0000 7f3dfefaa640 1 -- 172.21.15.62:0/212100731 >> 172.21.15.62:0/212100731 conn(0x7f3df80fc7d0 msgr2=0x7f3df80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.552+0000 7f3dfefaa640 1 -- 172.21.15.62:0/212100731 shutdown_connections 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.552+0000 7f3dfefaa640 1 -- 172.21.15.62:0/212100731 wait complete. 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfefaa640 1 Processor -- start 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfefaa640 1 -- start start 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfefaa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 0x7f3df8197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:28.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfefaa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df8102570 0x7f3df8197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfefaa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3df8102ef0 0x7f3df819bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfefaa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3df8111020 con 0x7f3df8100e20 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfefaa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3df8110ea0 con 0x7f3df8102570 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfefaa640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3df81111a0 con 0x7f3df8102ef0 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfcd46640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 0x7f3df8197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfcd46640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 0x7f3df8197500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50704/0 (socket says 172.21.15.62:50704) 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.553+0000 7f3dfcd46640 1 -- 172.21.15.62:0/1398924843 learned_addr learned my addr 172.21.15.62:0/1398924843 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3df7fff640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df8102570 0x7f3df8197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfd547640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3df8102ef0 0x7f3df819bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfcd46640 1 -- 172.21.15.62:0/1398924843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3df8102ef0 msgr2=0x7f3df819bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfcd46640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3df8102ef0 0x7f3df819bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:28.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfcd46640 1 -- 172.21.15.62:0/1398924843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df8102570 msgr2=0x7f3df8197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfcd46640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df8102570 0x7f3df8197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfcd46640 1 -- 172.21.15.62:0/1398924843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df819c4b0 con 0x7f3df8100e20 2024-09-15T23:13:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3df7fff640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df8102570 0x7f3df8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:13:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfd547640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3df8102ef0 0x7f3df819bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:13:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfcd46640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 0x7f3df8197500 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3dec006010 tx=0x7f3dec003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3df5ffb640 1 -- 172.21.15.62:0/1398924843 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec03f070 con 0x7f3df8100e20 2024-09-15T23:13:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3df5ffb640 1 -- 172.21.15.62:0/1398924843 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dec031e30 con 0x7f3df8100e20 2024-09-15T23:13:28.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df819c740 con 0x7f3df8100e20 2024-09-15T23:13:28.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3df5ffb640 1 -- 172.21.15.62:0/1398924843 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dec035030 con 0x7f3df8100e20 2024-09-15T23:13:28.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.554+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df81a4080 con 0x7f3df8100e20 2024-09-15T23:13:28.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.555+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3df8101280 con 0x7f3df8100e20 2024-09-15T23:13:28.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.557+0000 7f3df5ffb640 1 -- 172.21.15.62:0/1398924843 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3dec042070 con 0x7f3df8100e20 2024-09-15T23:13:28.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.557+0000 7f3df5ffb640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3de80a3240 0x7f3de80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:28.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.557+0000 7f3df7fff640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3de80a3240 0x7f3de80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:28.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.557+0000 7f3df5ffb640 1 -- 172.21.15.62:0/1398924843 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7127+0+0 (secure 0 0 0) 0x7f3dec0efe70 con 0x7f3df8100e20 2024-09-15T23:13:28.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.558+0000 7f3df7fff640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3de80a3240 0x7f3de80a5700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3de4006fd0 tx=0x7f3de4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:28.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.561+0000 7f3df5ffb640 1 -- 172.21.15.62:0/1398924843 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3dec0bd030 con 0x7f3df8100e20 2024-09-15T23:13:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:28.738+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3df8109090 con 0x7f3df8100e20 2024-09-15T23:13:28.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:28.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:28.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:28.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:13:28.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:28 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:28.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:28.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:28.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:13:28.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:13:28.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:28 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:13:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.471+0000 7f3df5ffb640 1 -- 172.21.15.62:0/1398924843 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) ==== 138+0+0 (secure 0 0 0) 0x7f3dec0bc930 con 0x7f3df8100e20 2024-09-15T23:13:29.472 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-09-15T23:13:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.473+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3de80a3240 msgr2=0x7f3de80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.473+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3de80a3240 0x7f3de80a5700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3de4006fd0 tx=0x7f3de4008040 comp rx=0 tx=0).stop 2024-09-15T23:13:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.473+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 msgr2=0x7f3df8197500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.473+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 0x7f3df8197500 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3dec006010 tx=0x7f3dec003c80 comp rx=0 tx=0).stop 2024-09-15T23:13:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.474+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 shutdown_connections 2024-09-15T23:13:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.474+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3de80a3240 0x7f3de80a5700 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:29.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.474+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3df8102ef0 0x7f3df819bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.474+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3df8102570 0x7f3df8197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.474+0000 7f3dfefaa640 1 --2- 172.21.15.62:0/1398924843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3df8100e20 0x7f3df8197500 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.474+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 >> 172.21.15.62:0/1398924843 conn(0x7f3df80fc7d0 msgr2=0x7f3df80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.474+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 shutdown_connections 2024-09-15T23:13:29.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:29.474+0000 7f3dfefaa640 1 -- 172.21.15.62:0/1398924843 wait complete. 2024-09-15T23:13:29.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:29 smithi062 ceph-mon[30949]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:29.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:29 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1398924843' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-15T23:13:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:29 smithi067 ceph-mon[35665]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:29 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1398924843' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-15T23:13:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:29 smithi148 ceph-mon[35610]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:29 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1398924843' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-09-15T23:13:30.056 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-15T23:13:30.315 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:30.703 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:30 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1398924843' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-15T23:13:30.704 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:30 smithi062 ceph-mon[30949]: osdmap e78: 12 total, 12 up, 12 in 2024-09-15T23:13:30.704 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:30 smithi062 ceph-mon[30949]: osdmap e79: 12 total, 12 up, 12 in 2024-09-15T23:13:30.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:30 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1398924843' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-15T23:13:30.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:30 smithi067 ceph-mon[35665]: osdmap e78: 12 total, 12 up, 12 in 2024-09-15T23:13:30.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:30 smithi067 ceph-mon[35665]: osdmap e79: 12 total, 12 up, 12 in 2024-09-15T23:13:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:30 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1398924843' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-09-15T23:13:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:30 smithi148 ceph-mon[35610]: osdmap e78: 12 total, 12 up, 12 in 2024-09-15T23:13:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:30 smithi148 ceph-mon[35610]: osdmap e79: 12 total, 12 up, 12 in 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 -- 172.21.15.62:0/2641672751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 msgr2=0x7f34381067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 --2- 172.21.15.62:0/2641672751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 0x7f34381067e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3428009900 tx=0x7f342802a3f0 comp rx=0 tx=0).stop 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 -- 172.21.15.62:0/2641672751 shutdown_connections 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 --2- 172.21.15.62:0/2641672751 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3438106f10 0x7f343810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 --2- 172.21.15.62:0/2641672751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 0x7f34381067e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 --2- 172.21.15.62:0/2641672751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3438101de0 0x7f34381021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 -- 172.21.15.62:0/2641672751 >> 172.21.15.62:0/2641672751 conn(0x7f34380fd8c0 msgr2=0x7f34380ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 -- 172.21.15.62:0/2641672751 shutdown_connections 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 -- 172.21.15.62:0/2641672751 wait complete. 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 Processor -- start 2024-09-15T23:13:30.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.791+0000 7f343fddc640 1 -- start start 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343fddc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3438101de0 0x7f3438198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343fddc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 0x7f3438198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343fddc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3438106f10 0x7f343819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343fddc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3438111f30 con 0x7f3438102790 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343fddc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3438111db0 con 0x7f3438101de0 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343fddc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f34381120b0 con 0x7f3438106f10 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343d377640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 0x7f3438198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343d377640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 0x7f3438198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50720/0 (socket says 172.21.15.62:50720) 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343d377640 1 -- 172.21.15.62:0/3906519616 learned_addr learned my addr 172.21.15.62:0/3906519616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343e379640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3438106f10 0x7f343819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343db78640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3438101de0 0x7f3438198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:30.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343d377640 1 -- 172.21.15.62:0/3906519616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3438106f10 msgr2=0x7f343819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343d377640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3438106f10 0x7f343819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343d377640 1 -- 172.21.15.62:0/3906519616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3438101de0 msgr2=0x7f3438198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343d377640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3438101de0 0x7f3438198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343d377640 1 -- 172.21.15.62:0/3906519616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343819d3c0 con 0x7f3438102790 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343db78640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3438101de0 0x7f3438198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343d377640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 0x7f3438198950 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3428006010 tx=0x7f34280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f342effd640 1 -- 172.21.15.62:0/3906519616 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342803f030 con 0x7f3438102790 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f342effd640 1 -- 172.21.15.62:0/3906519616 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3428003ce0 con 0x7f3438102790 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343819d650 con 0x7f3438102790 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f342effd640 1 -- 172.21.15.62:0/3906519616 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428035070 con 0x7f3438102790 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.792+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34381a4f90 con 0x7f3438102790 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.793+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34381032c0 con 0x7f3438102790 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.795+0000 7f342effd640 1 -- 172.21.15.62:0/3906519616 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f3428042030 con 0x7f3438102790 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.795+0000 7f342effd640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f34240a3240 0x7f34240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:30.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.796+0000 7f343db78640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f34240a3240 0x7f34240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:30.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.796+0000 7f342effd640 1 -- 172.21.15.62:0/3906519616 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7497+0+0 (secure 0 0 0) 0x7f34280eff40 con 0x7f3438102790 2024-09-15T23:13:30.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.796+0000 7f343db78640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f34240a3240 0x7f34240a5700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f343400abe0 tx=0x7f343400a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:30.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.799+0000 7f342effd640 1 -- 172.21.15.62:0/3906519616 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f34280bd050 con 0x7f3438102790 2024-09-15T23:13:30.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:30.976+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f343810a0f0 con 0x7f3438102790 2024-09-15T23:13:31.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.485+0000 7f342effd640 1 -- 172.21.15.62:0/3906519616 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v80) ==== 171+0+0 (secure 0 0 0) 0x7f34280bc950 con 0x7f3438102790 2024-09-15T23:13:31.486 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f34240a3240 msgr2=0x7f34240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f34240a3240 0x7f34240a5700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f343400abe0 tx=0x7f343400a5c0 comp rx=0 tx=0).stop 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 msgr2=0x7f3438198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 0x7f3438198950 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3428006010 tx=0x7f34280042f0 comp rx=0 tx=0).stop 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 shutdown_connections 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f34240a3240 0x7f34240a5700 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3438106f10 0x7f343819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3438102790 0x7f3438198950 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 --2- 172.21.15.62:0/3906519616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3438101de0 0x7f3438198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 >> 172.21.15.62:0/3906519616 conn(0x7f34380fd8c0 msgr2=0x7f34380febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 shutdown_connections 2024-09-15T23:13:31.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:31.489+0000 7f343fddc640 1 -- 172.21.15.62:0/3906519616 wait complete. 2024-09-15T23:13:31.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:31 smithi067 ceph-mon[35665]: pgmap v273: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:31.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:31 smithi067 ceph-mon[35665]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:13:31.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:31 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3906519616' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-15T23:13:31.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:31 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3906519616' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-15T23:13:31.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:31 smithi067 ceph-mon[35665]: osdmap e80: 12 total, 12 up, 12 in 2024-09-15T23:13:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:31 smithi148 ceph-mon[35610]: pgmap v273: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:31 smithi148 ceph-mon[35610]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:13:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:31 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3906519616' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-15T23:13:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:31 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3906519616' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-15T23:13:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:31 smithi148 ceph-mon[35610]: osdmap e80: 12 total, 12 up, 12 in 2024-09-15T23:13:31.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:31 smithi062 ceph-mon[30949]: pgmap v273: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:31.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:31 smithi062 ceph-mon[30949]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:13:31.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:31 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3906519616' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-09-15T23:13:31.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:31 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3906519616' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-09-15T23:13:31.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:31 smithi062 ceph-mon[30949]: osdmap e80: 12 total, 12 up, 12 in 2024-09-15T23:13:32.221 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-15T23:13:32.477 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:32.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 -- 172.21.15.62:0/2538296078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 msgr2=0x7fef9c10ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:32.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 --2- 172.21.15.62:0/2538296078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 0x7fef9c10ec40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fef94009930 tx=0x7fef9402a420 comp rx=0 tx=0).stop 2024-09-15T23:13:32.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 -- 172.21.15.62:0/2538296078 shutdown_connections 2024-09-15T23:13:32.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 --2- 172.21.15.62:0/2538296078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 0x7fef9c10ec40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:32.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 --2- 172.21.15.62:0/2538296078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c1036e0 0x7fef9c107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:32.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 --2- 172.21.15.62:0/2538296078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef9c102d30 0x7fef9c103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:32.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 -- 172.21.15.62:0/2538296078 >> 172.21.15.62:0/2538296078 conn(0x7fef9c0fc7d0 msgr2=0x7fef9c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 -- 172.21.15.62:0/2538296078 shutdown_connections 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 -- 172.21.15.62:0/2538296078 wait complete. 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 Processor -- start 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.979+0000 7fefa1b90640 1 -- start start 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fefa1b90640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef9c102d30 0x7fef9c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fefa1b90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c1036e0 0x7fef9c198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fefa1b90640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 0x7fef9c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fefa1b90640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c1120a0 con 0x7fef9c1082a0 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fefa1b90640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c111f20 con 0x7fef9c1036e0 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fefa1b90640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fef9c112220 con 0x7fef9c102d30 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 0x7fef9c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 0x7fef9c19cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48530/0 (socket says 172.21.15.62:48530) 2024-09-15T23:13:32.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9bfff640 1 -- 172.21.15.62:0/3241625202 learned_addr learned my addr 172.21.15.62:0/3241625202 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9affd640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c1036e0 0x7fef9c198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9b7fe640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef9c102d30 0x7fef9c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9bfff640 1 -- 172.21.15.62:0/3241625202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef9c102d30 msgr2=0x7fef9c1984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9bfff640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef9c102d30 0x7fef9c1984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9bfff640 1 -- 172.21.15.62:0/3241625202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c1036e0 msgr2=0x7fef9c198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9bfff640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c1036e0 0x7fef9c198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9bfff640 1 -- 172.21.15.62:0/3241625202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9c19d480 con 0x7fef9c1082a0 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9affd640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c1036e0 0x7fef9c198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef9bfff640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 0x7fef9c19cda0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fef94005ab0 tx=0x7fef940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef98ff9640 1 -- 172.21.15.62:0/3241625202 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9403f030 con 0x7fef9c1082a0 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9c19d710 con 0x7fef9c1082a0 2024-09-15T23:13:32.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef98ff9640 1 -- 172.21.15.62:0/3241625202 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef94003ce0 con 0x7fef9c1082a0 2024-09-15T23:13:32.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.980+0000 7fef98ff9640 1 -- 172.21.15.62:0/3241625202 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94035070 con 0x7fef9c1082a0 2024-09-15T23:13:32.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.981+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef9c1a5050 con 0x7fef9c1082a0 2024-09-15T23:13:32.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.981+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef9c06a850 con 0x7fef9c1082a0 2024-09-15T23:13:32.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.983+0000 7fef98ff9640 1 -- 172.21.15.62:0/3241625202 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fef94042030 con 0x7fef9c1082a0 2024-09-15T23:13:32.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.984+0000 7fef98ff9640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef880a3240 0x7fef880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:32.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.984+0000 7fef9b7fe640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef880a3240 0x7fef880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:32.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.984+0000 7fef98ff9640 1 -- 172.21.15.62:0/3241625202 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7879+0+0 (secure 0 0 0) 0x7fef940f0190 con 0x7fef9c1082a0 2024-09-15T23:13:32.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.984+0000 7fef9b7fe640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef880a3240 0x7fef880a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fef8c007350 tx=0x7fef8c0072e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:32.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:32.987+0000 7fef98ff9640 1 -- 172.21.15.62:0/3241625202 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fef940bd060 con 0x7fef9c1082a0 2024-09-15T23:13:33.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.199+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fef9c10a2b0 con 0x7fef9c1082a0 2024-09-15T23:13:33.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.504+0000 7fef98ff9640 1 -- 172.21.15.62:0/3241625202 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fef940bc960 con 0x7fef9c1082a0 2024-09-15T23:13:33.505 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 2 and data pool 3 2024-09-15T23:13:33.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef880a3240 msgr2=0x7fef880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:33.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef880a3240 0x7fef880a5700 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fef8c007350 tx=0x7fef8c0072e0 comp rx=0 tx=0).stop 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 msgr2=0x7fef9c19cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 0x7fef9c19cda0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fef94005ab0 tx=0x7fef940042f0 comp rx=0 tx=0).stop 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 shutdown_connections 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef880a3240 0x7fef880a5700 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef9c1082a0 0x7fef9c19cda0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef9c1036e0 0x7fef9c198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 --2- 172.21.15.62:0/3241625202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef9c102d30 0x7fef9c1984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.506+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 >> 172.21.15.62:0/3241625202 conn(0x7fef9c0fc7d0 msgr2=0x7fef9c106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.507+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 shutdown_connections 2024-09-15T23:13:33.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:33.507+0000 7fefa1b90640 1 -- 172.21.15.62:0/3241625202 wait complete. 2024-09-15T23:13:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:33 smithi067 ceph-mon[35665]: pgmap v276: 97 pgs: 20 active+clean, 1 creating+activating, 7 creating+peering, 69 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:33 smithi067 ceph-mon[35665]: osdmap e81: 12 total, 12 up, 12 in 2024-09-15T23:13:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:33 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3241625202' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-15T23:13:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:33 smithi067 ceph-mon[35665]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-15T23:13:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:33 smithi148 ceph-mon[35610]: pgmap v276: 97 pgs: 20 active+clean, 1 creating+activating, 7 creating+peering, 69 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:33 smithi148 ceph-mon[35610]: osdmap e81: 12 total, 12 up, 12 in 2024-09-15T23:13:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:33 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3241625202' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-15T23:13:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:33 smithi148 ceph-mon[35610]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-15T23:13:33.873 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:33 smithi062 ceph-mon[30949]: pgmap v276: 97 pgs: 20 active+clean, 1 creating+activating, 7 creating+peering, 69 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:33.873 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:33 smithi062 ceph-mon[30949]: osdmap e81: 12 total, 12 up, 12 in 2024-09-15T23:13:33.873 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:33 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3241625202' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-09-15T23:13:33.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:33 smithi062 ceph-mon[30949]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-09-15T23:13:34.256 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-15T23:13:34.256 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-15T23:13:34.518 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:34.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:34 smithi062 ceph-mon[30949]: osdmap e82: 12 total, 12 up, 12 in 2024-09-15T23:13:34.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:34 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3241625202' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-15T23:13:34.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:34 smithi062 ceph-mon[30949]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-15T23:13:34.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:34 smithi062 ceph-mon[30949]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-15T23:13:34.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:34 smithi067 ceph-mon[35665]: osdmap e82: 12 total, 12 up, 12 in 2024-09-15T23:13:34.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:34 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3241625202' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-15T23:13:34.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:34 smithi067 ceph-mon[35665]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-15T23:13:34.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:34 smithi067 ceph-mon[35665]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-15T23:13:34.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:34 smithi148 ceph-mon[35610]: osdmap e82: 12 total, 12 up, 12 in 2024-09-15T23:13:34.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:34 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3241625202' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-09-15T23:13:34.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:34 smithi148 ceph-mon[35610]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-09-15T23:13:34.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:34 smithi148 ceph-mon[35610]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-09-15T23:13:34.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.989+0000 7fdedd8c2640 1 -- 172.21.15.62:0/3504934540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8100a80 msgr2=0x7fded8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:34.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.989+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/3504934540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8100a80 0x7fded8100ee0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009930 tx=0x7fdecc02a420 comp rx=0 tx=0).stop 2024-09-15T23:13:34.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.989+0000 7fdedd8c2640 1 -- 172.21.15.62:0/3504934540 shutdown_connections 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.989+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/3504934540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded8101420 0x7fded810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.989+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/3504934540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8100a80 0x7fded8100ee0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.989+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/3504934540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded8106a80 0x7fded8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.989+0000 7fdedd8c2640 1 -- 172.21.15.62:0/3504934540 >> 172.21.15.62:0/3504934540 conn(0x7fded80fc7a0 msgr2=0x7fded80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.989+0000 7fdedd8c2640 1 -- 172.21.15.62:0/3504934540 shutdown_connections 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.989+0000 7fdedd8c2640 1 -- 172.21.15.62:0/3504934540 wait complete. 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fdedd8c2640 1 Processor -- start 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fdedd8c2640 1 -- start start 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fdedd8c2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded8100a80 0x7fded810e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:34.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fdedd8c2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8101420 0x7fded810e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fdedd8c2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded8106a80 0x7fded810ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fdedd8c2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fded8111fa0 con 0x7fded8101420 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fdedd8c2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fded8111e20 con 0x7fded8100a80 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fdedd8c2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fded8112120 con 0x7fded8106a80 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fded67fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8101420 0x7fded810e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fded67fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8101420 0x7fded810e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48554/0 (socket says 172.21.15.62:48554) 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fded67fc640 1 -- 172.21.15.62:0/744702860 learned_addr learned my addr 172.21.15.62:0/744702860 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.990+0000 7fded77fe640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded8106a80 0x7fded810ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fded6ffd640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded8100a80 0x7fded810e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fded67fc640 1 -- 172.21.15.62:0/744702860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded8106a80 msgr2=0x7fded810ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fded67fc640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded8106a80 0x7fded810ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fded67fc640 1 -- 172.21.15.62:0/744702860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded8100a80 msgr2=0x7fded810e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:34.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fded67fc640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded8100a80 0x7fded810e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fded67fc640 1 -- 172.21.15.62:0/744702860 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded81a6940 con 0x7fded8101420 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fded6ffd640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded8100a80 0x7fded810e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fded67fc640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8101420 0x7fded810e920 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009230 tx=0x7fdecc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fdedc8c0640 1 -- 172.21.15.62:0/744702860 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc03f030 con 0x7fded8101420 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fdedc8c0640 1 -- 172.21.15.62:0/744702860 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdecc003ad0 con 0x7fded8101420 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded81a6bd0 con 0x7fded8101420 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fdedc8c0640 1 -- 172.21.15.62:0/744702860 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc035070 con 0x7fded8101420 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.991+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded81a7100 con 0x7fded8101420 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.992+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded8100ee0 con 0x7fded8101420 2024-09-15T23:13:34.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.993+0000 7fdedc8c0640 1 -- 172.21.15.62:0/744702860 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fdecc042030 con 0x7fded8101420 2024-09-15T23:13:34.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.994+0000 7fdedc8c0640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdec00a32c0 0x7fdec00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:34.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.994+0000 7fded6ffd640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdec00a32c0 0x7fdec00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:34.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.994+0000 7fdedc8c0640 1 -- 172.21.15.62:0/744702860 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7979+0+0 (secure 0 0 0) 0x7fdecc0efc10 con 0x7fded8101420 2024-09-15T23:13:34.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.995+0000 7fded6ffd640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdec00a32c0 0x7fdec00a5780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdec8005fd0 tx=0x7fdec800b5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:34.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:34.998+0000 7fdedc8c0640 1 -- 172.21.15.62:0/744702860 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdecc0bca80 con 0x7fded8101420 2024-09-15T23:13:35.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.183+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fded8109010 con 0x7fded8101420 2024-09-15T23:13:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.517+0000 7fdedc8c0640 1 -- 172.21.15.62:0/744702860 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v83) ==== 157+0+0 (secure 0 0 0) 0x7fdecc0bc380 con 0x7fded8101420 2024-09-15T23:13:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdec00a32c0 msgr2=0x7fdec00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdec00a32c0 0x7fdec00a5780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdec8005fd0 tx=0x7fdec800b5d0 comp rx=0 tx=0).stop 2024-09-15T23:13:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8101420 msgr2=0x7fded810e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8101420 0x7fded810e920 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009230 tx=0x7fdecc004270 comp rx=0 tx=0).stop 2024-09-15T23:13:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 shutdown_connections 2024-09-15T23:13:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdec00a32c0 0x7fdec00a5780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fded8106a80 0x7fded810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fded8101420 0x7fded810e920 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 --2- 172.21.15.62:0/744702860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fded8100a80 0x7fded810e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 >> 172.21.15.62:0/744702860 conn(0x7fded80fc7a0 msgr2=0x7fded80fcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 shutdown_connections 2024-09-15T23:13:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:35.520+0000 7fdedd8c2640 1 -- 172.21.15.62:0/744702860 wait complete. 2024-09-15T23:13:35.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:35 smithi067 ceph-mon[35665]: pgmap v279: 97 pgs: 57 active+clean, 1 creating+activating, 7 creating+peering, 32 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:35.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:35 smithi067 ceph-mon[35665]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-15T23:13:35.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:35 smithi067 ceph-mon[35665]: Cluster is now healthy 2024-09-15T23:13:35.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:35 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] up:active 2024-09-15T23:13:35.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:35 smithi067 ceph-mon[35665]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:13:35.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:35 smithi067 ceph-mon[35665]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:13:35.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:35 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/744702860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T23:13:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:35 smithi148 ceph-mon[35610]: pgmap v279: 97 pgs: 57 active+clean, 1 creating+activating, 7 creating+peering, 32 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:35 smithi148 ceph-mon[35610]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-15T23:13:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:35 smithi148 ceph-mon[35610]: Cluster is now healthy 2024-09-15T23:13:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:35 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] up:active 2024-09-15T23:13:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:35 smithi148 ceph-mon[35610]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:13:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:35 smithi148 ceph-mon[35610]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:13:35.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:35 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/744702860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T23:13:35.889 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:35 smithi062 ceph-mon[30949]: pgmap v279: 97 pgs: 57 active+clean, 1 creating+activating, 7 creating+peering, 32 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:35.889 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:35 smithi062 ceph-mon[30949]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-09-15T23:13:35.889 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:35 smithi062 ceph-mon[30949]: Cluster is now healthy 2024-09-15T23:13:35.890 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:35 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] up:active 2024-09-15T23:13:35.890 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:35 smithi062 ceph-mon[30949]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:13:35.890 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:35 smithi062 ceph-mon[30949]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:13:35.890 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:35 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/744702860' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T23:13:36.214 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-15T23:13:36.473 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:36 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/744702860' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-15T23:13:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:36 smithi148 ceph-mon[35610]: osdmap e83: 12 total, 12 up, 12 in 2024-09-15T23:13:36.868 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:36 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/744702860' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-15T23:13:36.868 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:36 smithi062 ceph-mon[30949]: osdmap e83: 12 total, 12 up, 12 in 2024-09-15T23:13:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.956+0000 7fa59e6cd640 1 -- 172.21.15.62:0/2739483013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598103700 msgr2=0x7fa59810bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.956+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/2739483013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598103700 0x7fa59810bbf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009900 tx=0x7fa58c02a480 comp rx=0 tx=0).stop 2024-09-15T23:13:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.956+0000 7fa59e6cd640 1 -- 172.21.15.62:0/2739483013 shutdown_connections 2024-09-15T23:13:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.956+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/2739483013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa59810c180 0x7fa59810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.956+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/2739483013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa598103700 0x7fa59810bbf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.956+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/2739483013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa598102de0 0x7fa5981031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.956+0000 7fa59e6cd640 1 -- 172.21.15.62:0/2739483013 >> 172.21.15.62:0/2739483013 conn(0x7fa5980fc7a0 msgr2=0x7fa5980febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:36.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 -- 172.21.15.62:0/2739483013 shutdown_connections 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 -- 172.21.15.62:0/2739483013 wait complete. 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 Processor -- start 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 -- start start 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa598102de0 0x7fa598198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa598103700 0x7fa5981989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa59810c180 0x7fa59819cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa598111fc0 con 0x7fa59810c180 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa598111e40 con 0x7fa598102de0 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.957+0000 7fa59e6cd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa598112140 con 0x7fa598103700 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59cc6a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa59810c180 0x7fa59819cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59cc6a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa59810c180 0x7fa59819cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48562/0 (socket says 172.21.15.62:48562) 2024-09-15T23:13:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa5977fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa598103700 0x7fa5981989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59cc6a640 1 -- 172.21.15.62:0/3105053671 learned_addr learned my addr 172.21.15.62:0/3105053671 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa597fff640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa598102de0 0x7fa598198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59cc6a640 1 -- 172.21.15.62:0/3105053671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa598103700 msgr2=0x7fa5981989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59cc6a640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa598103700 0x7fa5981989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59cc6a640 1 -- 172.21.15.62:0/3105053671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa598102de0 msgr2=0x7fa598198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59cc6a640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa598102de0 0x7fa598198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59cc6a640 1 -- 172.21.15.62:0/3105053671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59819d430 con 0x7fa59810c180 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59cc6a640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa59810c180 0x7fa59819cd50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa59000eaa0 tx=0x7fa59000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa5957fa640 1 -- 172.21.15.62:0/3105053671 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59000cdb0 con 0x7fa59810c180 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa59819d720 con 0x7fa59810c180 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5981a5000 con 0x7fa59810c180 2024-09-15T23:13:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa5957fa640 1 -- 172.21.15.62:0/3105053671 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa590005020 con 0x7fa59810c180 2024-09-15T23:13:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.958+0000 7fa5957fa640 1 -- 172.21.15.62:0/3105053671 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5900116b0 con 0x7fa59810c180 2024-09-15T23:13:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.959+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa59806afd0 con 0x7fa59810c180 2024-09-15T23:13:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.961+0000 7fa5957fa640 1 -- 172.21.15.62:0/3105053671 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa590005560 con 0x7fa59810c180 2024-09-15T23:13:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.962+0000 7fa5957fa640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5800a3240 0x7fa5800a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.962+0000 7fa597fff640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5800a3240 0x7fa5800a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.962+0000 7fa5957fa640 1 -- 172.21.15.62:0/3105053671 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fa5900cfeb0 con 0x7fa59810c180 2024-09-15T23:13:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.962+0000 7fa597fff640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5800a3240 0x7fa5800a5700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa588005fd0 tx=0x7fa58800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:36.965+0000 7fa5957fa640 1 -- 172.21.15.62:0/3105053671 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa590016030 con 0x7fa59810c180 2024-09-15T23:13:37.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:36 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/744702860' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-09-15T23:13:37.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:36 smithi067 ceph-mon[35665]: osdmap e83: 12 total, 12 up, 12 in 2024-09-15T23:13:37.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:37.144+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa5981031c0 con 0x7fa59810c180 2024-09-15T23:13:38.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:37 smithi062 ceph-mon[30949]: pgmap v281: 97 pgs: 84 active+clean, 4 creating+peering, 9 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1 op/s 2024-09-15T23:13:38.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:37 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3105053671' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-15T23:13:38.274 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:13:37 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: 2024-09-15T23:13:37.883+0000 7ff291122640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-15T23:13:38.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:37 smithi067 ceph-mon[35665]: pgmap v281: 97 pgs: 84 active+clean, 4 creating+peering, 9 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1 op/s 2024-09-15T23:13:38.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:37 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3105053671' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-15T23:13:38.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:37 smithi148 ceph-mon[35610]: pgmap v281: 97 pgs: 84 active+clean, 4 creating+peering, 9 unknown; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1 op/s 2024-09-15T23:13:38.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:37 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3105053671' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-09-15T23:13:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.529+0000 7fa5957fa640 1 -- 172.21.15.62:0/3105053671 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v85) ==== 228+0+0 (secure 0 0 0) 0x7fa5981031c0 con 0x7fa59810c180 2024-09-15T23:13:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data_ec' created 2024-09-15T23:13:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5800a3240 msgr2=0x7fa5800a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5800a3240 0x7fa5800a5700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa588005fd0 tx=0x7fa58800b390 comp rx=0 tx=0).stop 2024-09-15T23:13:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa59810c180 msgr2=0x7fa59819cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa59810c180 0x7fa59819cd50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa59000eaa0 tx=0x7fa59000ef60 comp rx=0 tx=0).stop 2024-09-15T23:13:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 shutdown_connections 2024-09-15T23:13:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa5800a3240 0x7fa5800a5700 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa59810c180 0x7fa59819cd50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa598103700 0x7fa5981989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 --2- 172.21.15.62:0/3105053671 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa598102de0 0x7fa598198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 >> 172.21.15.62:0/3105053671 conn(0x7fa5980fc7a0 msgr2=0x7fa5980fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 shutdown_connections 2024-09-15T23:13:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:38.533+0000 7fa59e6cd640 1 -- 172.21.15.62:0/3105053671 wait complete. 2024-09-15T23:13:39.164 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:38 smithi062 ceph-mon[30949]: pgmap v283: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 8 op/s 2024-09-15T23:13:39.164 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:38 smithi062 ceph-mon[30949]: osdmap e84: 12 total, 12 up, 12 in 2024-09-15T23:13:39.164 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:38 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3105053671' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-15T23:13:39.164 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:38 smithi062 ceph-mon[30949]: osdmap e85: 12 total, 12 up, 12 in 2024-09-15T23:13:39.197 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-15T23:13:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:38 smithi067 ceph-mon[35665]: pgmap v283: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 8 op/s 2024-09-15T23:13:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:38 smithi067 ceph-mon[35665]: osdmap e84: 12 total, 12 up, 12 in 2024-09-15T23:13:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:38 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3105053671' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-15T23:13:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:38 smithi067 ceph-mon[35665]: osdmap e85: 12 total, 12 up, 12 in 2024-09-15T23:13:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:38 smithi148 ceph-mon[35610]: pgmap v283: 97 pgs: 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 8 op/s 2024-09-15T23:13:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:38 smithi148 ceph-mon[35610]: osdmap e84: 12 total, 12 up, 12 in 2024-09-15T23:13:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:38 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3105053671' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-09-15T23:13:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:38 smithi148 ceph-mon[35610]: osdmap e85: 12 total, 12 up, 12 in 2024-09-15T23:13:39.459 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:39.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.945+0000 7f112dc09640 1 -- 172.21.15.62:0/4287908133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 msgr2=0x7f11281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.945+0000 7f112dc09640 1 --2- 172.21.15.62:0/4287908133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 0x7f11281067e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f111c009900 tx=0x7f111c02a410 comp rx=0 tx=0).stop 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.945+0000 7f112dc09640 1 -- 172.21.15.62:0/4287908133 shutdown_connections 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.945+0000 7f112dc09640 1 --2- 172.21.15.62:0/4287908133 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1128106f10 0x7f112810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.945+0000 7f112dc09640 1 --2- 172.21.15.62:0/4287908133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 0x7f11281067e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.945+0000 7f112dc09640 1 --2- 172.21.15.62:0/4287908133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128101de0 0x7f11281021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.945+0000 7f112dc09640 1 -- 172.21.15.62:0/4287908133 >> 172.21.15.62:0/4287908133 conn(0x7f11280fd8c0 msgr2=0x7f11280ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.945+0000 7f112dc09640 1 -- 172.21.15.62:0/4287908133 shutdown_connections 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.946+0000 7f112dc09640 1 -- 172.21.15.62:0/4287908133 wait complete. 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.946+0000 7f112dc09640 1 Processor -- start 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.946+0000 7f112dc09640 1 -- start start 2024-09-15T23:13:39.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f112dc09640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128101de0 0x7f11281983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f112dc09640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 0x7f11281988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f112dc09640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1128106f10 0x7f112819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f112dc09640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1128111f70 con 0x7f1128102790 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f112dc09640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1128111df0 con 0x7f1128101de0 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f112dc09640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f11281120f0 con 0x7f1128106f10 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1126ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 0x7f11281988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1126ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 0x7f11281988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48578/0 (socket says 172.21.15.62:48578) 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1126ffd640 1 -- 172.21.15.62:0/2911444725 learned_addr learned my addr 172.21.15.62:0/2911444725 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1127fff640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1128106f10 0x7f112819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f11277fe640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128101de0 0x7f11281983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1126ffd640 1 -- 172.21.15.62:0/2911444725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1128106f10 msgr2=0x7f112819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1126ffd640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1128106f10 0x7f112819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:39.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1126ffd640 1 -- 172.21.15.62:0/2911444725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128101de0 msgr2=0x7f11281983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1126ffd640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128101de0 0x7f11281983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1126ffd640 1 -- 172.21.15.62:0/2911444725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112819d360 con 0x7f1128102790 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1126ffd640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 0x7f11281988f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f111c006010 tx=0x7f111c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1124ff9640 1 -- 172.21.15.62:0/2911444725 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c03f030 con 0x7f1128102790 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f1124ff9640 1 -- 172.21.15.62:0/2911444725 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f111c003ce0 con 0x7f1128102790 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.947+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f112819d5f0 con 0x7f1128102790 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.948+0000 7f1124ff9640 1 -- 172.21.15.62:0/2911444725 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c035070 con 0x7f1128102790 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.948+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11281a4f30 con 0x7f1128102790 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.949+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11281021c0 con 0x7f1128102790 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.950+0000 7f1124ff9640 1 -- 172.21.15.62:0/2911444725 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f111c042030 con 0x7f1128102790 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.950+0000 7f1124ff9640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f11100a32c0 0x7f11100a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.951+0000 7f11277fe640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f11100a32c0 0x7f11100a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:39.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.951+0000 7f1124ff9640 1 -- 172.21.15.62:0/2911444725 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8770+0+0 (secure 0 0 0) 0x7f111c0f0170 con 0x7f1128102790 2024-09-15T23:13:39.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.951+0000 7f11277fe640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f11100a32c0 0x7f11100a5780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1118005fd0 tx=0x7f1118005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:39.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:39.955+0000 7f1124ff9640 1 -- 172.21.15.62:0/2911444725 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f111c0bcd30 con 0x7f1128102790 2024-09-15T23:13:40.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.129+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f112810a180 con 0x7f1128102790 2024-09-15T23:13:40.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.536+0000 7f1124ff9640 1 -- 172.21.15.62:0/2911444725 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v87) ==== 169+0+0 (secure 0 0 0) 0x7f111c0dc0c0 con 0x7f1128102790 2024-09-15T23:13:40.538 INFO:teuthology.orchestra.run.smithi062.stderr:set pool 4 allow_ec_overwrites to true 2024-09-15T23:13:40.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f11100a32c0 msgr2=0x7f11100a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:40.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f11100a32c0 0x7f11100a5780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1118005fd0 tx=0x7f1118005850 comp rx=0 tx=0).stop 2024-09-15T23:13:40.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 msgr2=0x7f11281988f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:40.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 0x7f11281988f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f111c006010 tx=0x7f111c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:13:40.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 shutdown_connections 2024-09-15T23:13:40.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f11100a32c0 0x7f11100a5780 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:40.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1128106f10 0x7f112819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:40.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1128102790 0x7f11281988f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:40.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 --2- 172.21.15.62:0/2911444725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1128101de0 0x7f11281983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:40.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.539+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 >> 172.21.15.62:0/2911444725 conn(0x7f11280fd8c0 msgr2=0x7f11280fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:40.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.540+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 shutdown_connections 2024-09-15T23:13:40.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:40.540+0000 7f112dc09640 1 -- 172.21.15.62:0/2911444725 wait complete. 2024-09-15T23:13:40.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:40 smithi148 ceph-mon[35610]: osdmap e86: 12 total, 12 up, 12 in 2024-09-15T23:13:40.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:40 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2911444725' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-15T23:13:40.837 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:40 smithi062 ceph-mon[30949]: osdmap e86: 12 total, 12 up, 12 in 2024-09-15T23:13:40.837 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:40 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2911444725' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-15T23:13:41.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:40 smithi067 ceph-mon[35665]: osdmap e86: 12 total, 12 up, 12 in 2024-09-15T23:13:41.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:40 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2911444725' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-09-15T23:13:41.251 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-15T23:13:41.520 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:41 smithi148 ceph-mon[35610]: pgmap v286: 129 pgs: 1 creating+peering, 31 unknown, 97 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 9 op/s 2024-09-15T23:13:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:41 smithi148 ceph-mon[35610]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:13:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:41 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2911444725' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T23:13:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:41 smithi148 ceph-mon[35610]: osdmap e87: 12 total, 12 up, 12 in 2024-09-15T23:13:41.791 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:41 smithi062 ceph-mon[30949]: pgmap v286: 129 pgs: 1 creating+peering, 31 unknown, 97 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 9 op/s 2024-09-15T23:13:41.791 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:41 smithi062 ceph-mon[30949]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:13:41.791 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:41 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2911444725' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T23:13:41.791 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:41 smithi062 ceph-mon[30949]: osdmap e87: 12 total, 12 up, 12 in 2024-09-15T23:13:42.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.010+0000 7fe4b957d640 1 -- 172.21.15.62:0/3610774933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b41027a0 msgr2=0x7fe4b41067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:42.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.010+0000 7fe4b957d640 1 --2- 172.21.15.62:0/3610774933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b41027a0 0x7fe4b41067f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8009900 tx=0x7fe4a802a410 comp rx=0 tx=0).stop 2024-09-15T23:13:42.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.010+0000 7fe4b957d640 1 -- 172.21.15.62:0/3610774933 shutdown_connections 2024-09-15T23:13:42.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.010+0000 7fe4b957d640 1 --2- 172.21.15.62:0/3610774933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b4106f20 0x7fe4b410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.010+0000 7fe4b957d640 1 --2- 172.21.15.62:0/3610774933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b41027a0 0x7fe4b41067f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.010+0000 7fe4b957d640 1 --2- 172.21.15.62:0/3610774933 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4b4101df0 0x7fe4b41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.010+0000 7fe4b957d640 1 -- 172.21.15.62:0/3610774933 >> 172.21.15.62:0/3610774933 conn(0x7fe4b40fd8b0 msgr2=0x7fe4b40ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:42.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.010+0000 7fe4b957d640 1 -- 172.21.15.62:0/3610774933 shutdown_connections 2024-09-15T23:13:42.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.010+0000 7fe4b957d640 1 -- 172.21.15.62:0/3610774933 wait complete. 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b957d640 1 Processor -- start 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b957d640 1 -- start start 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b957d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b4101df0 0x7fe4b4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b957d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4b41027a0 0x7fe4b4198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b957d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b4106f20 0x7fe4b419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b957d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b4111fa0 con 0x7fe4b4106f20 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b957d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b4111e20 con 0x7fe4b4101df0 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b957d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b4112120 con 0x7fe4b41027a0 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b4106f20 0x7fe4b419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b4106f20 0x7fe4b419ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48592/0 (socket says 172.21.15.62:48592) 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.011+0000 7fe4b37fe640 1 -- 172.21.15.62:0/1242044284 learned_addr learned my addr 172.21.15.62:0/1242044284 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:42.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b2ffd640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b4101df0 0x7fe4b4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b27fc640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4b41027a0 0x7fe4b4198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b37fe640 1 -- 172.21.15.62:0/1242044284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4b41027a0 msgr2=0x7fe4b4198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b37fe640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4b41027a0 0x7fe4b4198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b37fe640 1 -- 172.21.15.62:0/1242044284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b4101df0 msgr2=0x7fe4b4198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b37fe640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b4101df0 0x7fe4b4198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b37fe640 1 -- 172.21.15.62:0/1242044284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b419d4f0 con 0x7fe4b4106f20 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b27fc640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4b41027a0 0x7fe4b4198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b37fe640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b4106f20 0x7fe4b419ce10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00cad0 tx=0x7fe4ac00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe49bfff640 1 -- 172.21.15.62:0/1242044284 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac00ad60 con 0x7fe4b4106f20 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b419d7e0 con 0x7fe4b4106f20 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe49bfff640 1 -- 172.21.15.62:0/1242044284 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4ac005020 con 0x7fe4b4106f20 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe49bfff640 1 -- 172.21.15.62:0/1242044284 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac00f6b0 con 0x7fe4b4106f20 2024-09-15T23:13:42.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.012+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b41a50c0 con 0x7fe4b4106f20 2024-09-15T23:13:42.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.013+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b41021d0 con 0x7fe4b4106f20 2024-09-15T23:13:42.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.015+0000 7fe49bfff640 1 -- 172.21.15.62:0/1242044284 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fe4ac005560 con 0x7fe4b4106f20 2024-09-15T23:13:42.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.016+0000 7fe49bfff640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe49c0a3240 0x7fe49c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:42.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.016+0000 7fe4b2ffd640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe49c0a3240 0x7fe49c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:42.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.016+0000 7fe49bfff640 1 -- 172.21.15.62:0/1242044284 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fe4ac0cffa0 con 0x7fe4b4106f20 2024-09-15T23:13:42.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.017+0000 7fe4b2ffd640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe49c0a3240 0x7fe49c0a5700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009990 tx=0x7fe4a4009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:42.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.019+0000 7fe49bfff640 1 -- 172.21.15.62:0/1242044284 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe4ac014030 con 0x7fe4b4106f20 2024-09-15T23:13:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:41 smithi067 ceph-mon[35665]: pgmap v286: 129 pgs: 1 creating+peering, 31 unknown, 97 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s wr, 9 op/s 2024-09-15T23:13:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:41 smithi067 ceph-mon[35665]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T23:13:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:41 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2911444725' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T23:13:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:41 smithi067 ceph-mon[35665]: osdmap e87: 12 total, 12 up, 12 in 2024-09-15T23:13:42.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.232+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fe4b410a160 con 0x7fe4b4106f20 2024-09-15T23:13:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.544+0000 7fe49bfff640 1 -- 172.21.15.62:0/1242044284 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fe4ac09caa0 con 0x7fe4b4106f20 2024-09-15T23:13:42.546 INFO:teuthology.orchestra.run.smithi062.stderr:added data pool 4 to fsmap 2024-09-15T23:13:42.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe49c0a3240 msgr2=0x7fe49c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:42.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe49c0a3240 0x7fe49c0a5700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009990 tx=0x7fe4a4009110 comp rx=0 tx=0).stop 2024-09-15T23:13:42.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b4106f20 msgr2=0x7fe4b419ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:42.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b4106f20 0x7fe4b419ce10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac00cad0 tx=0x7fe4ac00ced0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 shutdown_connections 2024-09-15T23:13:42.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe49c0a3240 0x7fe49c0a5700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4b4106f20 0x7fe4b419ce10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4b41027a0 0x7fe4b4198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 --2- 172.21.15.62:0/1242044284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4b4101df0 0x7fe4b4198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:42.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 >> 172.21.15.62:0/1242044284 conn(0x7fe4b40fd8b0 msgr2=0x7fe4b40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:42.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 shutdown_connections 2024-09-15T23:13:42.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:42.548+0000 7fe4b957d640 1 -- 172.21.15.62:0/1242044284 wait complete. 2024-09-15T23:13:42.870 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:42 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1242044284' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-15T23:13:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:42 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1242044284' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-15T23:13:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:42 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1242044284' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-09-15T23:13:43.215 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:13:43.483 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:43.882 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:43 smithi062 ceph-mon[30949]: pgmap v288: 129 pgs: 1 creating+peering, 15 unknown, 113 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 1 op/s 2024-09-15T23:13:43.882 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:43 smithi062 ceph-mon[30949]: osdmap e88: 12 total, 12 up, 12 in 2024-09-15T23:13:43.882 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:43 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1242044284' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-15T23:13:43.882 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:43 smithi062 ceph-mon[30949]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.964+0000 7f916f8da640 1 -- 172.21.15.62:0/458553331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168101de0 msgr2=0x7f91681021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.964+0000 7f916f8da640 1 --2- 172.21.15.62:0/458553331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168101de0 0x7f91681021c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f915c009900 tx=0x7f915c02a410 comp rx=0 tx=0).stop 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.964+0000 7f916f8da640 1 -- 172.21.15.62:0/458553331 shutdown_connections 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.964+0000 7f916f8da640 1 --2- 172.21.15.62:0/458553331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9168106f10 0x7f916810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.964+0000 7f916f8da640 1 --2- 172.21.15.62:0/458553331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9168102790 0x7f91681067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.964+0000 7f916f8da640 1 --2- 172.21.15.62:0/458553331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168101de0 0x7f91681021c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.964+0000 7f916f8da640 1 -- 172.21.15.62:0/458553331 >> 172.21.15.62:0/458553331 conn(0x7f91680fd8c0 msgr2=0x7f91680ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.964+0000 7f916f8da640 1 -- 172.21.15.62:0/458553331 shutdown_connections 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.964+0000 7f916f8da640 1 -- 172.21.15.62:0/458553331 wait complete. 2024-09-15T23:13:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916f8da640 1 Processor -- start 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916f8da640 1 -- start start 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916f8da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9168101de0 0x7f91680744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916f8da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168102790 0x7f9168074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916f8da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9168106f10 0x7f9168074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916f8da640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9168112060 con 0x7f9168102790 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916f8da640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9168111ee0 con 0x7f9168106f10 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916f8da640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91681121e0 con 0x7f9168101de0 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916ce75640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168102790 0x7f9168074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916ce75640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168102790 0x7f9168074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34228/0 (socket says 172.21.15.62:34228) 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916ce75640 1 -- 172.21.15.62:0/101721315 learned_addr learned my addr 172.21.15.62:0/101721315 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916de77640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9168106f10 0x7f9168074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916d676640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9168101de0 0x7f91680744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916ce75640 1 -- 172.21.15.62:0/101721315 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9168101de0 msgr2=0x7f91680744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916ce75640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9168101de0 0x7f91680744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916ce75640 1 -- 172.21.15.62:0/101721315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9168106f10 msgr2=0x7f9168074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916ce75640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9168106f10 0x7f9168074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916ce75640 1 -- 172.21.15.62:0/101721315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91681a6a80 con 0x7f9168102790 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.965+0000 7f916ce75640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168102790 0x7f9168074a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f916400b4a0 tx=0x7f916400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.966+0000 7f915a7fc640 1 -- 172.21.15.62:0/101721315 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164014070 con 0x7f9168102790 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.966+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91681a6d70 con 0x7f9168102790 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.966+0000 7f915a7fc640 1 -- 172.21.15.62:0/101721315 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9164010af0 con 0x7f9168102790 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.966+0000 7f915a7fc640 1 -- 172.21.15.62:0/101721315 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9164017d10 con 0x7f9168102790 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.966+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91681a7250 con 0x7f9168102790 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.967+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9168102240 con 0x7f9168102790 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.968+0000 7f915a7fc640 1 -- 172.21.15.62:0/101721315 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f9164017070 con 0x7f9168102790 2024-09-15T23:13:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.969+0000 7f915a7fc640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91540a3240 0x7f91540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:43.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.969+0000 7f916d676640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91540a3240 0x7f91540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:43.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.969+0000 7f915a7fc640 1 -- 172.21.15.62:0/101721315 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f91640ceec0 con 0x7f9168102790 2024-09-15T23:13:43.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.969+0000 7f916d676640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91540a3240 0x7f91540a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f915c02a930 tx=0x7f915c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:43.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:43.973+0000 7f915a7fc640 1 -- 172.21.15.62:0/101721315 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f916409b9a0 con 0x7f9168102790 2024-09-15T23:13:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:43 smithi067 ceph-mon[35665]: pgmap v288: 129 pgs: 1 creating+peering, 15 unknown, 113 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 1 op/s 2024-09-15T23:13:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:43 smithi067 ceph-mon[35665]: osdmap e88: 12 total, 12 up, 12 in 2024-09-15T23:13:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:43 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1242044284' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-15T23:13:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:43 smithi067 ceph-mon[35665]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:13:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:43 smithi148 ceph-mon[35610]: pgmap v288: 129 pgs: 1 creating+peering, 15 unknown, 113 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 1 op/s 2024-09-15T23:13:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:43 smithi148 ceph-mon[35610]: osdmap e88: 12 total, 12 up, 12 in 2024-09-15T23:13:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:43 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1242044284' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-09-15T23:13:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:43 smithi148 ceph-mon[35610]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:13:44.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.188+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9168109b60 con 0x7f9168102790 2024-09-15T23:13:44.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.189+0000 7f915a7fc640 1 -- 172.21.15.62:0/101721315 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11830 (secure 0 0 0) 0x7f916409b2a0 con 0x7f9168102790 2024-09-15T23:13:44.190 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:44.190 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":17,"btime":"2024-09-15T23:13:42:539016+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24466,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:13:42.234096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24497},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24497,"qdb_cluster":[24497]},"id":1}]} 2024-09-15T23:13:44.191 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 17 2024-09-15T23:13:44.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91540a3240 msgr2=0x7f91540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:44.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91540a3240 0x7f91540a5700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f915c02a930 tx=0x7f915c005fd0 comp rx=0 tx=0).stop 2024-09-15T23:13:44.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168102790 msgr2=0x7f9168074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:44.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168102790 0x7f9168074a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f916400b4a0 tx=0x7f916400b960 comp rx=0 tx=0).stop 2024-09-15T23:13:44.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 shutdown_connections 2024-09-15T23:13:44.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91540a3240 0x7f91540a5700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:44.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9168106f10 0x7f9168074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:44.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9168102790 0x7f9168074a30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:44.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 --2- 172.21.15.62:0/101721315 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9168101de0 0x7f91680744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:44.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 >> 172.21.15.62:0/101721315 conn(0x7f91680fd8c0 msgr2=0x7f91680fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:44.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 shutdown_connections 2024-09-15T23:13:44.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:44.191+0000 7f916f8da640 1 -- 172.21.15.62:0/101721315 wait complete. 2024-09-15T23:13:44.839 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:44 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/101721315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:44.875 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:13:45.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:44 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/101721315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:45.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:44 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/101721315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:45.140 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:45.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.629+0000 7f32e5cbf640 1 -- 172.21.15.62:0/994334986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e010c170 msgr2=0x7f32e010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.629+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/994334986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e010c170 0x7f32e010e610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009ca0 tx=0x7f32d802a950 comp rx=0 tx=0).stop 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.629+0000 7f32e5cbf640 1 -- 172.21.15.62:0/994334986 shutdown_connections 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.629+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/994334986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e010c170 0x7f32e010e610 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.629+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/994334986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f32e0103660 0x7f32e010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.629+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/994334986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e0102d40 0x7f32e0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.629+0000 7f32e5cbf640 1 -- 172.21.15.62:0/994334986 >> 172.21.15.62:0/994334986 conn(0x7f32e00fc7a0 msgr2=0x7f32e00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.629+0000 7f32e5cbf640 1 -- 172.21.15.62:0/994334986 shutdown_connections 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.629+0000 7f32e5cbf640 1 -- 172.21.15.62:0/994334986 wait complete. 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.630+0000 7f32e5cbf640 1 Processor -- start 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.630+0000 7f32e5cbf640 1 -- start start 2024-09-15T23:13:45.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.630+0000 7f32e5cbf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e0102d40 0x7f32e0198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.630+0000 7f32e5cbf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e0103660 0x7f32e0198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32e5cbf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f32e010c170 0x7f32e019ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32e5cbf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f32e0112080 con 0x7f32e0103660 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32e5cbf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f32e0111f00 con 0x7f32e0102d40 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32e5cbf640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f32e0112200 con 0x7f32e010c170 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32deffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e0103660 0x7f32e0198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32deffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e0103660 0x7f32e0198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34246/0 (socket says 172.21.15.62:34246) 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32deffd640 1 -- 172.21.15.62:0/1954791298 learned_addr learned my addr 172.21.15.62:0/1954791298 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32dffff640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f32e010c170 0x7f32e019ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32df7fe640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e0102d40 0x7f32e0198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:45.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32deffd640 1 -- 172.21.15.62:0/1954791298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f32e010c170 msgr2=0x7f32e019ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32deffd640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f32e010c170 0x7f32e019ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32deffd640 1 -- 172.21.15.62:0/1954791298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e0102d40 msgr2=0x7f32e0198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32deffd640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e0102d40 0x7f32e0198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32deffd640 1 -- 172.21.15.62:0/1954791298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e019d4f0 con 0x7f32e0103660 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32df7fe640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e0102d40 0x7f32e0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32deffd640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e0103660 0x7f32e0198a80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f32d400b3f0 tx=0x7f32d400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32dcff9640 1 -- 172.21.15.62:0/1954791298 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d4015020 con 0x7f32e0103660 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32dcff9640 1 -- 172.21.15.62:0/1954791298 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32d4011d10 con 0x7f32e0103660 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e019d7e0 con 0x7f32e0103660 2024-09-15T23:13:45.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32dcff9640 1 -- 172.21.15.62:0/1954791298 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d400f2e0 con 0x7f32e0103660 2024-09-15T23:13:45.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.631+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e01a50c0 con 0x7f32e0103660 2024-09-15T23:13:45.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.632+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e01031a0 con 0x7f32e0103660 2024-09-15T23:13:45.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.633+0000 7f32dcff9640 1 -- 172.21.15.62:0/1954791298 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f32d4018020 con 0x7f32e0103660 2024-09-15T23:13:45.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.634+0000 7f32dcff9640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32c80a32c0 0x7f32c80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:45.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.634+0000 7f32dcff9640 1 -- 172.21.15.62:0/1954791298 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f32d40cf300 con 0x7f32e0103660 2024-09-15T23:13:45.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.634+0000 7f32df7fe640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32c80a32c0 0x7f32c80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:45.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.635+0000 7f32df7fe640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32c80a32c0 0x7f32c80a5780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009930 tx=0x7f32d0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:45.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.638+0000 7f32dcff9640 1 -- 172.21.15.62:0/1954791298 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f32d401f020 con 0x7f32e0103660 2024-09-15T23:13:45.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.818+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32e0106c50 con 0x7f32e0103660 2024-09-15T23:13:45.819 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:45 smithi062 ceph-mon[30949]: pgmap v290: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:45.819 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:45 smithi062 ceph-mon[30949]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-15T23:13:45.819 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:45 smithi062 ceph-mon[30949]: Cluster is now healthy 2024-09-15T23:13:45.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.819+0000 7f32dcff9640 1 -- 172.21.15.62:0/1954791298 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f32d409bde0 con 0x7f32e0103660 2024-09-15T23:13:45.821 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:45.821 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:13:42.233963+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:13:45.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32c80a32c0 msgr2=0x7f32c80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:45.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32c80a32c0 0x7f32c80a5780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009930 tx=0x7f32d0008040 comp rx=0 tx=0).stop 2024-09-15T23:13:45.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e0103660 msgr2=0x7f32e0198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:45.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e0103660 0x7f32e0198a80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f32d400b3f0 tx=0x7f32d400b8b0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 shutdown_connections 2024-09-15T23:13:45.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32c80a32c0 0x7f32c80a5780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f32e010c170 0x7f32e019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32e0103660 0x7f32e0198a80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 --2- 172.21.15.62:0/1954791298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32e0102d40 0x7f32e0198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:45.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 >> 172.21.15.62:0/1954791298 conn(0x7f32e00fc7a0 msgr2=0x7f32e00fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:45.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 shutdown_connections 2024-09-15T23:13:45.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:45.822+0000 7f32e5cbf640 1 -- 172.21.15.62:0/1954791298 wait complete. 2024-09-15T23:13:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:45 smithi067 ceph-mon[35665]: pgmap v290: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:45 smithi067 ceph-mon[35665]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-15T23:13:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:45 smithi067 ceph-mon[35665]: Cluster is now healthy 2024-09-15T23:13:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:45 smithi148 ceph-mon[35610]: pgmap v290: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:45 smithi148 ceph-mon[35610]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-15T23:13:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:45 smithi148 ceph-mon[35610]: Cluster is now healthy 2024-09-15T23:13:46.413 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:13:46.674 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:46.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:46 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1954791298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:13:47.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:46 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1954791298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:13:47.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:46 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1954791298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:13:47.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 -- 172.21.15.62:0/267741096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80108290 msgr2=0x7f4a8010ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:47.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 --2- 172.21.15.62:0/267741096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80108290 0x7f4a8010ec30 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4a70009930 tx=0x7f4a7002a420 comp rx=0 tx=0).stop 2024-09-15T23:13:47.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 -- 172.21.15.62:0/267741096 shutdown_connections 2024-09-15T23:13:47.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 --2- 172.21.15.62:0/267741096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80108290 0x7f4a8010ec30 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:47.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 --2- 172.21.15.62:0/267741096 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a80103740 0x7f4a80107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:47.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 --2- 172.21.15.62:0/267741096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a80102d90 0x7f4a80103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:47.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 -- 172.21.15.62:0/267741096 >> 172.21.15.62:0/267741096 conn(0x7f4a800fc7d0 msgr2=0x7f4a800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:47.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 -- 172.21.15.62:0/267741096 shutdown_connections 2024-09-15T23:13:47.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 -- 172.21.15.62:0/267741096 wait complete. 2024-09-15T23:13:47.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 Processor -- start 2024-09-15T23:13:47.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.170+0000 7f4a87055640 1 -- start start 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a87055640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a80102d90 0x7f4a801985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a87055640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80103740 0x7f4a80198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a87055640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a80108290 0x7f4a8019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a87055640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4a801120d0 con 0x7f4a80103740 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a87055640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a80111f50 con 0x7f4a80102d90 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a87055640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4a80112250 con 0x7f4a80108290 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80103740 0x7f4a80198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80103740 0x7f4a80198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34258/0 (socket says 172.21.15.62:34258) 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7ffff640 1 -- 172.21.15.62:0/3380782207 learned_addr learned my addr 172.21.15.62:0/3380782207 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a84df1640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a80102d90 0x7f4a801985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:47.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a855f2640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a80108290 0x7f4a8019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7ffff640 1 -- 172.21.15.62:0/3380782207 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a80108290 msgr2=0x7f4a8019ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7ffff640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a80108290 0x7f4a8019ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7ffff640 1 -- 172.21.15.62:0/3380782207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a80102d90 msgr2=0x7f4a801985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7ffff640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a80102d90 0x7f4a801985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7ffff640 1 -- 172.21.15.62:0/3380782207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a8019d560 con 0x7f4a80103740 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a855f2640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a80108290 0x7f4a8019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7ffff640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80103740 0x7f4a80198af0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800b450 tx=0x7f4a6800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7dffb640 1 -- 172.21.15.62:0/3380782207 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a68015020 con 0x7f4a80103740 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7dffb640 1 -- 172.21.15.62:0/3380782207 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a68011d70 con 0x7f4a80103740 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8019d850 con 0x7f4a80103740 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.171+0000 7f4a7dffb640 1 -- 172.21.15.62:0/3380782207 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6800f340 con 0x7f4a80103740 2024-09-15T23:13:47.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.172+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a801a5130 con 0x7f4a80103740 2024-09-15T23:13:47.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.173+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a80103170 con 0x7f4a80103740 2024-09-15T23:13:47.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.174+0000 7f4a7dffb640 1 -- 172.21.15.62:0/3380782207 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f4a68018020 con 0x7f4a80103740 2024-09-15T23:13:47.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.174+0000 7f4a7dffb640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4a6c0a32c0 0x7f4a6c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:47.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.175+0000 7f4a84df1640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4a6c0a32c0 0x7f4a6c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:47.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.175+0000 7f4a7dffb640 1 -- 172.21.15.62:0/3380782207 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f4a680cf650 con 0x7f4a80103740 2024-09-15T23:13:47.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.175+0000 7f4a84df1640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4a6c0a32c0 0x7f4a6c0a5780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4a78005fd0 tx=0x7f4a78009870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:47.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.179+0000 7f4a7dffb640 1 -- 172.21.15.62:0/3380782207 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4a6801f020 con 0x7f4a80103740 2024-09-15T23:13:47.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.357+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a8010a2d0 con 0x7f4a80103740 2024-09-15T23:13:47.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.359+0000 7f4a7dffb640 1 -- 172.21.15.62:0/3380782207 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f4a6800d040 con 0x7f4a80103740 2024-09-15T23:13:47.360 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:47.360 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:13:42.233963+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:13:47.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4a6c0a32c0 msgr2=0x7f4a6c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:47.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4a6c0a32c0 0x7f4a6c0a5780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4a78005fd0 tx=0x7f4a78009870 comp rx=0 tx=0).stop 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80103740 msgr2=0x7f4a80198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80103740 0x7f4a80198af0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4a6800b450 tx=0x7f4a6800b910 comp rx=0 tx=0).stop 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 shutdown_connections 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4a6c0a32c0 0x7f4a6c0a5780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4a80108290 0x7f4a8019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4a80103740 0x7f4a80198af0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 --2- 172.21.15.62:0/3380782207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a80102d90 0x7f4a801985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 >> 172.21.15.62:0/3380782207 conn(0x7f4a800fc7d0 msgr2=0x7f4a80105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 shutdown_connections 2024-09-15T23:13:47.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:47.361+0000 7f4a87055640 1 -- 172.21.15.62:0/3380782207 wait complete. 2024-09-15T23:13:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:47 smithi062 ceph-mon[30949]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:47 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3380782207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:13:47.995 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:13:48.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:47 smithi067 ceph-mon[35665]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:48.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:47 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3380782207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:13:48.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:47 smithi148 ceph-mon[35610]: pgmap v291: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:48.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:47 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3380782207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:13:48.248 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:48.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.793+0000 7f76e0742640 1 -- 172.21.15.62:0/3833524947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d8100b20 msgr2=0x7f76d8100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.793+0000 7f76e0742640 1 --2- 172.21.15.62:0/3833524947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d8100b20 0x7f76d8100f00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009ca0 tx=0x7f76c802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.793+0000 7f76e0742640 1 -- 172.21.15.62:0/3833524947 shutdown_connections 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.793+0000 7f76e0742640 1 --2- 172.21.15.62:0/3833524947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d810e500 0x7f76d8110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.793+0000 7f76e0742640 1 --2- 172.21.15.62:0/3833524947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76d8101440 0x7f76d810df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.793+0000 7f76e0742640 1 --2- 172.21.15.62:0/3833524947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d8100b20 0x7f76d8100f00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.793+0000 7f76e0742640 1 -- 172.21.15.62:0/3833524947 >> 172.21.15.62:0/3833524947 conn(0x7f76d80fc7d0 msgr2=0x7f76d80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.793+0000 7f76e0742640 1 -- 172.21.15.62:0/3833524947 shutdown_connections 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.793+0000 7f76e0742640 1 -- 172.21.15.62:0/3833524947 wait complete. 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76e0742640 1 Processor -- start 2024-09-15T23:13:48.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76e0742640 1 -- start start 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76e0742640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d8100b20 0x7f76d81a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76e0742640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76d8101440 0x7f76d81a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76e0742640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d810e500 0x7f76d819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76e0742640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8114260 con 0x7f76d810e500 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76e0742640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81140e0 con 0x7f76d8100b20 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76e0742640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81143e0 con 0x7f76d8101440 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76decdf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d810e500 0x7f76d819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76decdf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d810e500 0x7f76d819a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34280/0 (socket says 172.21.15.62:34280) 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76decdf640 1 -- 172.21.15.62:0/3502433140 learned_addr learned my addr 172.21.15.62:0/3502433140 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76de4de640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d8100b20 0x7f76d81a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.794+0000 7f76ddcdd640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76d8101440 0x7f76d81a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76decdf640 1 -- 172.21.15.62:0/3502433140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76d8101440 msgr2=0x7f76d81a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:48.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76decdf640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76d8101440 0x7f76d81a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76decdf640 1 -- 172.21.15.62:0/3502433140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d8100b20 msgr2=0x7f76d81a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76decdf640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d8100b20 0x7f76d81a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76decdf640 1 -- 172.21.15.62:0/3502433140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d819ace0 con 0x7f76d810e500 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76decdf640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d810e500 0x7f76d819a5b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f76d000e970 tx=0x7f76d000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76d77fe640 1 -- 172.21.15.62:0/3502433140 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d000ccc0 con 0x7f76d810e500 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76d77fe640 1 -- 172.21.15.62:0/3502433140 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d0005500 con 0x7f76d810e500 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d819afd0 con 0x7f76d810e500 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76d77fe640 1 -- 172.21.15.62:0/3502433140 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0010690 con 0x7f76d810e500 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.795+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d81a72c0 con 0x7f76d810e500 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.796+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d806a850 con 0x7f76d810e500 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.797+0000 7f76d77fe640 1 -- 172.21.15.62:0/3502433140 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f76d00046c0 con 0x7f76d810e500 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.798+0000 7f76d77fe640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76c00a32c0 0x7f76c00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:48.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.798+0000 7f76de4de640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76c00a32c0 0x7f76c00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:48.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.798+0000 7f76d77fe640 1 -- 172.21.15.62:0/3502433140 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f76d00cff80 con 0x7f76d810e500 2024-09-15T23:13:48.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.798+0000 7f76de4de640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76c00a32c0 0x7f76c00a5780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76c8000f80 tx=0x7f76c80093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:48.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:48.802+0000 7f76d77fe640 1 -- 172.21.15.62:0/3502433140 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f76d009ca60 con 0x7f76d810e500 2024-09-15T23:13:49.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.014+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76d8109600 con 0x7f76d810e500 2024-09-15T23:13:49.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.015+0000 7f76d77fe640 1 -- 172.21.15.62:0/3502433140 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11830 (secure 0 0 0) 0x7f76d009c360 con 0x7f76d810e500 2024-09-15T23:13:49.016 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:49.016 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":17,"btime":"2024-09-15T23:13:42:539016+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24466,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:13:42.234096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24497},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24497,"qdb_cluster":[24497]},"id":1}]} 2024-09-15T23:13:49.016 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 17 2024-09-15T23:13:49.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76c00a32c0 msgr2=0x7f76c00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:49.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76c00a32c0 0x7f76c00a5780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f76c8000f80 tx=0x7f76c80093f0 comp rx=0 tx=0).stop 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d810e500 msgr2=0x7f76d819a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d810e500 0x7f76d819a5b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f76d000e970 tx=0x7f76d000ee30 comp rx=0 tx=0).stop 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 shutdown_connections 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76c00a32c0 0x7f76c00a5780 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76d810e500 0x7f76d819a5b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f76d8101440 0x7f76d81a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 --2- 172.21.15.62:0/3502433140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76d8100b20 0x7f76d81a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 >> 172.21.15.62:0/3502433140 conn(0x7f76d80fc7d0 msgr2=0x7f76d80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 shutdown_connections 2024-09-15T23:13:49.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:49.017+0000 7f76e0742640 1 -- 172.21.15.62:0/3502433140 wait complete. 2024-09-15T23:13:49.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:49 smithi062 ceph-mon[30949]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:49.717 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:49 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3502433140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:49.746 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:13:49.826 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:49 smithi067 ceph-mon[35665]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:49.826 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:49 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3502433140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:49.842 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:49 smithi148 ceph-mon[35610]: pgmap v292: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:49.842 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:49 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3502433140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:13:50.005 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:50.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 -- 172.21.15.62:0/1019613368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 msgr2=0x7f26781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 --2- 172.21.15.62:0/1019613368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 0x7f26781067e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2660009900 tx=0x7f266002a410 comp rx=0 tx=0).stop 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 -- 172.21.15.62:0/1019613368 shutdown_connections 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 --2- 172.21.15.62:0/1019613368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2678106f10 0x7f267810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 --2- 172.21.15.62:0/1019613368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 0x7f26781067e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 --2- 172.21.15.62:0/1019613368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2678101de0 0x7f26781021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 -- 172.21.15.62:0/1019613368 >> 172.21.15.62:0/1019613368 conn(0x7f26780fd8c0 msgr2=0x7f26780ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 -- 172.21.15.62:0/1019613368 shutdown_connections 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 -- 172.21.15.62:0/1019613368 wait complete. 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 Processor -- start 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.476+0000 7f267de42640 1 -- start start 2024-09-15T23:13:50.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f267de42640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2678101de0 0x7f2678198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f267de42640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 0x7f2678198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f267de42640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2678106f10 0x7f267819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f267de42640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2678111f70 con 0x7f2678102790 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f267de42640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2678111df0 con 0x7f2678101de0 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f267de42640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f26781120f0 con 0x7f2678106f10 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2676ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 0x7f2678198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2676ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 0x7f2678198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34302/0 (socket says 172.21.15.62:34302) 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2676ffd640 1 -- 172.21.15.62:0/2427396209 learned_addr learned my addr 172.21.15.62:0/2427396209 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f26777fe640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2678101de0 0x7f2678198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2677fff640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2678106f10 0x7f267819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2676ffd640 1 -- 172.21.15.62:0/2427396209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2678106f10 msgr2=0x7f267819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:50.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2676ffd640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2678106f10 0x7f267819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2676ffd640 1 -- 172.21.15.62:0/2427396209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2678101de0 msgr2=0x7f2678198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2676ffd640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2678101de0 0x7f2678198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2676ffd640 1 -- 172.21.15.62:0/2427396209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f267819d400 con 0x7f2678102790 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2677fff640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2678106f10 0x7f267819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.477+0000 7f2676ffd640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 0x7f2678198990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2660006010 tx=0x7f26600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.478+0000 7f2674ff9640 1 -- 172.21.15.62:0/2427396209 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266003f030 con 0x7f2678102790 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.478+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f267819d690 con 0x7f2678102790 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.478+0000 7f2674ff9640 1 -- 172.21.15.62:0/2427396209 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2660003ce0 con 0x7f2678102790 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.478+0000 7f2674ff9640 1 -- 172.21.15.62:0/2427396209 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660035070 con 0x7f2678102790 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.478+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26781a4fd0 con 0x7f2678102790 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.479+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26781032c0 con 0x7f2678102790 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.480+0000 7f2674ff9640 1 -- 172.21.15.62:0/2427396209 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2660042030 con 0x7f2678102790 2024-09-15T23:13:50.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.481+0000 7f2674ff9640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26640a3240 0x7f26640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:50.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.481+0000 7f26777fe640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26640a3240 0x7f26640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:50.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.481+0000 7f2674ff9640 1 -- 172.21.15.62:0/2427396209 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f26600f0210 con 0x7f2678102790 2024-09-15T23:13:50.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.481+0000 7f26777fe640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26640a3240 0x7f26640a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f266800bbe0 tx=0x7f266800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:50.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.485+0000 7f2674ff9640 1 -- 172.21.15.62:0/2427396209 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f26600bcdb0 con 0x7f2678102790 2024-09-15T23:13:50.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.665+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f267810a180 con 0x7f2678102790 2024-09-15T23:13:50.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.666+0000 7f2674ff9640 1 -- 172.21.15.62:0/2427396209 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f26600bc6b0 con 0x7f2678102790 2024-09-15T23:13:50.667 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:50.667 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:13:42.233963+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:13:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.668+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26640a3240 msgr2=0x7f26640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.668+0000 7f267de42640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26640a3240 0x7f26640a5700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f266800bbe0 tx=0x7f266800b390 comp rx=0 tx=0).stop 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.668+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 msgr2=0x7f2678198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.668+0000 7f267de42640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 0x7f2678198990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2660006010 tx=0x7f26600042f0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.669+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 shutdown_connections 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.669+0000 7f267de42640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26640a3240 0x7f26640a5700 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.669+0000 7f267de42640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2678106f10 0x7f267819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.669+0000 7f267de42640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2678102790 0x7f2678198990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.669+0000 7f267de42640 1 --2- 172.21.15.62:0/2427396209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2678101de0 0x7f2678198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.669+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 >> 172.21.15.62:0/2427396209 conn(0x7f26780fd8c0 msgr2=0x7f26780fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.669+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 shutdown_connections 2024-09-15T23:13:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:50.669+0000 7f267de42640 1 -- 172.21.15.62:0/2427396209 wait complete. 2024-09-15T23:13:51.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-15T23:13:33.200855+0000', 'modified': '2024-09-15T23:13:42.234096+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24497}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24497': {'gid': 24497, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6839/818760439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6838', 'nonce': 818760439}, {'type': 'v1', 'addr': '172.21.15.67:6839', 'nonce': 818760439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24497, 'qdb_cluster': [24497]} 2024-09-15T23:13:51.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-15T23:13:51.498 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T23:13:51.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:51 smithi062 ceph-mon[30949]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:51.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:51 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2427396209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:13:51.758 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:52.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:51 smithi067 ceph-mon[35665]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:52.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:51 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2427396209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:13:52.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:51 smithi148 ceph-mon[35610]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:52.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:51 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2427396209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:13:52.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 -- 172.21.15.62:0/1590012588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 msgr2=0x7f2d941029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 --2- 172.21.15.62:0/1590012588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 0x7f2d941029b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2d88009900 tx=0x7f2d8802a480 comp rx=0 tx=0).stop 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 -- 172.21.15.62:0/1590012588 shutdown_connections 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 --2- 172.21.15.62:0/1590012588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d94102ef0 0x7f2d9410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 --2- 172.21.15.62:0/1590012588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 0x7f2d941029b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 --2- 172.21.15.62:0/1590012588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d94100e20 0x7f2d94101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 -- 172.21.15.62:0/1590012588 >> 172.21.15.62:0/1590012588 conn(0x7f2d940fc7d0 msgr2=0x7f2d940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 -- 172.21.15.62:0/1590012588 shutdown_connections 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 -- 172.21.15.62:0/1590012588 wait complete. 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.211+0000 7f2d99689640 1 Processor -- start 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d99689640 1 -- start start 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d99689640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d94100e20 0x7f2d941993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d99689640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 0x7f2d94199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d99689640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d94102ef0 0x7f2d941a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d99689640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2d94110fa0 con 0x7f2d94102570 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d99689640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2d94110e20 con 0x7f2d94100e20 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d99689640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2d94111120 con 0x7f2d94102ef0 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d927fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 0x7f2d94199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d927fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 0x7f2d94199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34310/0 (socket says 172.21.15.62:34310) 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d927fc640 1 -- 172.21.15.62:0/679177545 learned_addr learned my addr 172.21.15.62:0/679177545 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d94100e20 0x7f2d941993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d937fe640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d94102ef0 0x7f2d941a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d927fc640 1 -- 172.21.15.62:0/679177545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d94102ef0 msgr2=0x7f2d941a4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d927fc640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d94102ef0 0x7f2d941a4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d927fc640 1 -- 172.21.15.62:0/679177545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d94100e20 msgr2=0x7f2d941993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:52.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d927fc640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d94100e20 0x7f2d941993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d927fc640 1 -- 172.21.15.62:0/679177545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d941a4990 con 0x7f2d94102570 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d937fe640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d94102ef0 0x7f2d941a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.212+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d94100e20 0x7f2d941993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.213+0000 7f2d927fc640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 0x7f2d94199910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2d88006010 tx=0x7f2d88003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.213+0000 7f2d7bfff640 1 -- 172.21.15.62:0/679177545 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8803f070 con 0x7f2d94102570 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.213+0000 7f2d7bfff640 1 -- 172.21.15.62:0/679177545 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d88031c30 con 0x7f2d94102570 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.213+0000 7f2d99689640 1 -- 172.21.15.62:0/679177545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d941a4b60 con 0x7f2d94102570 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.213+0000 7f2d7bfff640 1 -- 172.21.15.62:0/679177545 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d88035030 con 0x7f2d94102570 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.213+0000 7f2d99689640 1 -- 172.21.15.62:0/679177545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d941a5010 con 0x7f2d94102570 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.215+0000 7f2d7bfff640 1 -- 172.21.15.62:0/679177545 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f2d88042070 con 0x7f2d94102570 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.215+0000 7f2d9ba62740 1 Processor -- start 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.215+0000 7f2d9ba62740 1 -- start start 2024-09-15T23:13:52.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630bc9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564630b30090 con 0x564630bd4bb0 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564630ad7a70 con 0x564630bc99d0 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564630b2c050 con 0x564630bca2f0 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d92ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d937fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630bc9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d927fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d7bfff640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2d7c0a32c0 0x7f2d7c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d92ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:55398/0 (socket says 172.21.15.62:55398) 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d92ffd640 1 -- 172.21.15.62:0/1811030470 learned_addr learned my addr 172.21.15.62:0/1811030470 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d937fe640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2d7c0a32c0 0x7f2d7c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d7bfff640 1 -- 172.21.15.62:0/679177545 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f2d880f0200 con 0x7f2d94102570 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d92ffd640 1 -- 172.21.15.62:0/1811030470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 msgr2=0x564630bc9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/1811030470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630bc9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d92ffd640 1 -- 172.21.15.62:0/1811030470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 msgr2=0x564630bd6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/1811030470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d92ffd640 1 -- 172.21.15.62:0/1811030470 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564630ae4130 con 0x564630bca2f0 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d937fe640 1 --2- 172.21.15.62:0/1811030470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630bc9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.216+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/1811030470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd4670 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2d840115f0 tx=0x7f2d84011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58a674c84de1bcfd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d937fe640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2d7c0a32c0 0x7f2d7c0a5780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0139c0 tx=0x7f2d8c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d797fa640 1 -- 172.21.15.62:0/1811030470 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8401e070 con 0x564630bca2f0 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d797fa640 1 -- 172.21.15.62:0/1811030470 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8400fb20 con 0x564630bca2f0 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d797fa640 1 -- 172.21.15.62:0/1811030470 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d84019bf0 con 0x564630bca2f0 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1811030470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 msgr2=0x564630bd4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1811030470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd4670 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2d840115f0 tx=0x7f2d84011ab0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1811030470 shutdown_connections 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1811030470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1811030470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd4670 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1811030470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630bc9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1811030470 >> 172.21.15.62:0/1811030470 conn(0x564630bc5f60 msgr2=0x564630bc6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1811030470 shutdown_connections 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.217+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1811030470 wait complete. 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d9ba62740 1 -- start start 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d9ba62740 1 Processor -- start 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630bd43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564630bd7780 con 0x564630bd4bb0 2024-09-15T23:13:52.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564630aeaf20 con 0x564630bc99d0 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564630bd7900 con 0x564630bca2f0 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d9ba62740 10 client.0 _ll_register_callbacks cb 0x564630ab4030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34324/0 (socket says 172.21.15.62:34324) 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d937fe640 1 -- 172.21.15.62:0/452899906 learned_addr learned my addr 172.21.15.62:0/452899906 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630bd43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.218+0000 7f2d927fc640 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d937fe640 1 -- 172.21.15.62:0/452899906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 msgr2=0x564630bd1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d937fe640 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d937fe640 1 -- 172.21.15.62:0/452899906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 msgr2=0x564630bd43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d937fe640 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630bd43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d937fe640 1 -- 172.21.15.62:0/452899906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646308a12b0 con 0x564630bd4bb0 2024-09-15T23:13:52.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d937fe640 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd1c50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c010e80 tx=0x7f2d8c004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f290e990c56c81dd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d76ff5640 1 -- 172.21.15.62:0/452899906 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c022020 con 0x564630bd4bb0 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d9ba62740 1 -- 172.21.15.62:0/452899906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d8400fb20 con 0x564630bd4bb0 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d76ff5640 1 -- 172.21.15.62:0/452899906 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f2d8c01e440 con 0x564630bd4bb0 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d76ff5640 1 -- 172.21.15.62:0/452899906 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f2d8c02d050 con 0x564630bd4bb0 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d9ba62740 10 client.14721 fetch_fsmap learned FSMap version 17 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d9ba62740 1 -- 172.21.15.62:0/452899906 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x564630be30f0 con 0x564630bd4bb0 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d76ff5640 1 -- 172.21.15.62:0/452899906 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8400fb20 con 0x564630bd4bb0 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.219+0000 7f2d76ff5640 1 -- 172.21.15.62:0/452899906 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f2d8c01d6d0 con 0x564630bd4bb0 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.220+0000 7f2d9ba62740 10 client.14721 fetch_fsmap finished waiting for FSMap version 17 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.220+0000 7f2d9ba62740 10 client.14721 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.220+0000 7f2d9ba62740 20 client.14721 populate_metadata read hostname 'smithi062' 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.220+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564630be57a0 0x564630be6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.220+0000 7f2d9ba62740 4 client.14721 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.220+0000 7f2d9ba62740 1 -- 172.21.15.62:0/452899906 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564630b3d600 con 0x564630be57a0 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.220+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564630be57a0 0x564630be6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.220+0000 7f2d76ff5640 10 client.14721 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.220+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564630be57a0 0x564630be6c10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77a7d8107a16a940 server_cookie=bc3b227d2bc0ff0c in_seq=0 out_seq=0 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d76ff5640 1 -- 172.21.15.62:0/452899906 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564630b3d600 con 0x564630be57a0 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d76ff5640 10 client.14721 handle_command_reply: tid=0 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 client.14721 shutdown 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 -- 172.21.15.62:0/452899906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 msgr2=0x564630bd1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd1c50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c010e80 tx=0x7f2d8c004b50 comp rx=0 tx=0).stop 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 -- 172.21.15.62:0/452899906 shutdown_connections 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564630be57a0 0x564630be6c10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bd1c50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d76ff5640 0 client.14721 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:13:52.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630bd1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/452899906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630bd43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 -- 172.21.15.62:0/452899906 >> 172.21.15.62:0/452899906 conn(0x564630bc5f60 msgr2=0x564630bcbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 -- 172.21.15.62:0/452899906 shutdown_connections 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 1 -- 172.21.15.62:0/452899906 wait complete. 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.221+0000 7f2d9ba62740 20 client.14721 trim_cache size 0 max 16384 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d9ba62740 1 Processor -- start 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d9ba62740 1 -- start start 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630be50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630be6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bcffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564630be4f50 con 0x564630bd4bb0 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564630be5620 con 0x564630bc99d0 2024-09-15T23:13:52.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564630be5cf0 con 0x564630bca2f0 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bcffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bcffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34332/0 (socket says 172.21.15.62:34332) 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.239+0000 7f2d937fe640 1 -- 172.21.15.62:0/2278803208 learned_addr learned my addr 172.21.15.62:0/2278803208 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d927fc640 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630be6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d937fe640 1 -- 172.21.15.62:0/2278803208 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 msgr2=0x564630be6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d937fe640 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630be6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d937fe640 1 -- 172.21.15.62:0/2278803208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 msgr2=0x564630be50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630be50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d937fe640 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630be50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d937fe640 1 -- 172.21.15.62:0/2278803208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564630bd4730 con 0x564630bd4bb0 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d927fc640 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630be6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d937fe640 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bcffe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c01daf0 tx=0x7f2d8c01df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb49f4ef5bbb1f37 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d78ff9640 1 -- 172.21.15.62:0/2278803208 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c035030 con 0x564630bd4bb0 2024-09-15T23:13:52.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d78ff9640 1 -- 172.21.15.62:0/2278803208 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c017030 con 0x564630bd4bb0 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d78ff9640 1 -- 172.21.15.62:0/2278803208 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c022040 con 0x564630bd4bb0 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d9ba62740 1 -- 172.21.15.62:0/2278803208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 msgr2=0x564630bcffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bcffe0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c01daf0 tx=0x7f2d8c01df70 comp rx=0 tx=0).stop 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d9ba62740 1 -- 172.21.15.62:0/2278803208 shutdown_connections 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bd4bb0 0x564630bcffe0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bca2f0 0x564630be6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/2278803208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630be50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.240+0000 7f2d9ba62740 1 -- 172.21.15.62:0/2278803208 >> 172.21.15.62:0/2278803208 conn(0x564630bc5f60 msgr2=0x564630bd16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 -- 172.21.15.62:0/2278803208 shutdown_connections 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 -- 172.21.15.62:0/2278803208 wait complete. 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 -- start start 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 Processor -- start 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630c48130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bca2f0 0x564630c48670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bd4bb0 0x564630bea440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564630be6a90 con 0x564630bca2f0 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564630be63c0 con 0x564630bc99d0 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564630be7160 con 0x564630bd4bb0 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d9ba62740 10 client.0 _ll_register_callbacks cb 0x7f2d8c007f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d927fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bca2f0 0x564630c48670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d927fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bca2f0 0x564630c48670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34340/0 (socket says 172.21.15.62:34340) 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.241+0000 7f2d927fc640 1 -- 172.21.15.62:0/1334533988 learned_addr learned my addr 172.21.15.62:0/1334533988 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d927fc640 1 -- 172.21.15.62:0/1334533988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bd4bb0 msgr2=0x564630bea440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d927fc640 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bd4bb0 0x564630bea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d927fc640 1 -- 172.21.15.62:0/1334533988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 msgr2=0x564630c48130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630c48130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d927fc640 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630c48130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d927fc640 1 -- 172.21.15.62:0/1334533988 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564630beab30 con 0x564630bca2f0 2024-09-15T23:13:52.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630c48130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d927fc640 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bca2f0 0x564630c48670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2d880edac0 tx=0x7f2d880bc120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220282c79da68a5a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d76ff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d76ff5640 1 -- 172.21.15.62:0/1334533988 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d880d5060 con 0x564630bca2f0 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d76ff5640 1 -- 172.21.15.62:0/1334533988 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f2d8803b460 con 0x564630bca2f0 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1334533988 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d8c017030 con 0x564630bca2f0 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d76ff5640 1 -- 172.21.15.62:0/1334533988 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f2d88049040 con 0x564630bca2f0 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d9ba62740 10 client.14727 fetch_fsmap learned FSMap version 17 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1334533988 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x564630c4f900 con 0x564630bca2f0 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d76ff5640 1 -- 172.21.15.62:0/1334533988 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d8c017030 con 0x564630bca2f0 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d76ff5640 1 -- 172.21.15.62:0/1334533988 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f2d880b8c50 con 0x564630bca2f0 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d9ba62740 10 client.14727 fetch_fsmap finished waiting for FSMap version 17 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d9ba62740 10 client.14727 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:13:52.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d9ba62740 20 client.14727 populate_metadata read hostname 'smithi062' 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564630be57a0 0x564630c53270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d9ba62740 4 client.14727 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.242+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1334533988 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "status"}) -- 0x564630b37b10 con 0x564630be57a0 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564630be57a0 0x564630c53270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d92ffd640 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564630be57a0 0x564630c53270 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=227181b40a2f40a9 server_cookie=921c365d2bfbaf4c in_seq=0 out_seq=0 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d76ff5640 10 client.14727 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d76ff5640 1 -- 172.21.15.62:0/1334533988 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564630b37b10 con 0x564630be57a0 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d76ff5640 10 client.14727 handle_command_reply: tid=0 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d9ba62740 1 client.14727 shutdown 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1334533988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bca2f0 msgr2=0x564630c48670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bca2f0 0x564630c48670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2d880edac0 tx=0x7f2d880bc120 comp rx=0 tx=0).stop 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1334533988 shutdown_connections 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564630be57a0 0x564630c53270 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.243+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564630bd4bb0 0x564630bea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564630bca2f0 0x564630c48670 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d9ba62740 1 --2- 172.21.15.62:0/1334533988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564630bc99d0 0x564630c48130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1334533988 >> 172.21.15.62:0/1334533988 conn(0x564630bc5f60 msgr2=0x564630bcffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d76ff5640 0 client.14727 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1334533988 shutdown_connections 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d9ba62740 1 -- 172.21.15.62:0/1334533988 wait complete. 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d9ba62740 20 client.14727 trim_cache size 0 max 16384 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-09-15T23:13:52.262 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24497, 2024-09-15T23:13:52.263 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:13:52.263 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:13:52.263 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:13:52.263 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 18.737760348999998, 2024-09-15T23:13:52.263 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 17, 2024-09-15T23:13:52.263 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 88, 2024-09-15T23:13:52.263 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T23:13:52.263 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 29.886422233000001 2024-09-15T23:13:52.263 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d99689640 1 -- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2d7c0a32c0 msgr2=0x7f2d7c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d99689640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2d7c0a32c0 0x7f2d7c0a5780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0139c0 tx=0x7f2d8c004a40 comp rx=0 tx=0).stop 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d99689640 1 -- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 msgr2=0x7f2d94199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d99689640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 0x7f2d94199910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2d88006010 tx=0x7f2d88003c10 comp rx=0 tx=0).stop 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d99689640 1 -- 172.21.15.62:0/679177545 shutdown_connections 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d99689640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2d7c0a32c0 0x7f2d7c0a5780 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d99689640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d94102ef0 0x7f2d941a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d99689640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d94102570 0x7f2d94199910 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.244+0000 7f2d99689640 1 --2- 172.21.15.62:0/679177545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d94100e20 0x7f2d941993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:52.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.245+0000 7f2d99689640 1 -- 172.21.15.62:0/679177545 >> 172.21.15.62:0/679177545 conn(0x7f2d940fc7d0 msgr2=0x7f2d940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:52.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.245+0000 7f2d99689640 1 -- 172.21.15.62:0/679177545 shutdown_connections 2024-09-15T23:13:52.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:52.245+0000 7f2d99689640 1 -- 172.21.15.62:0/679177545 wait complete. 2024-09-15T23:13:52.930 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-15T23:13:52.930 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:13:52.930 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-15T23:13:53.188 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.670+0000 7f9dc9497640 1 -- 172.21.15.62:0/1089142524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc4103650 msgr2=0x7f9dc410bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.670+0000 7f9dc9497640 1 --2- 172.21.15.62:0/1089142524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc4103650 0x7f9dc410bb40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9dac009900 tx=0x7f9dac02a480 comp rx=0 tx=0).stop 2024-09-15T23:13:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.670+0000 7f9dc9497640 1 -- 172.21.15.62:0/1089142524 shutdown_connections 2024-09-15T23:13:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.670+0000 7f9dc9497640 1 --2- 172.21.15.62:0/1089142524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc410c160 0x7f9dc410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.670+0000 7f9dc9497640 1 --2- 172.21.15.62:0/1089142524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc4103650 0x7f9dc410bb40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.670+0000 7f9dc9497640 1 --2- 172.21.15.62:0/1089142524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dc4102d30 0x7f9dc4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.670+0000 7f9dc9497640 1 -- 172.21.15.62:0/1089142524 >> 172.21.15.62:0/1089142524 conn(0x7f9dc40fc790 msgr2=0x7f9dc40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.670+0000 7f9dc9497640 1 -- 172.21.15.62:0/1089142524 shutdown_connections 2024-09-15T23:13:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.670+0000 7f9dc9497640 1 -- 172.21.15.62:0/1089142524 wait complete. 2024-09-15T23:13:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.671+0000 7f9dc9497640 1 Processor -- start 2024-09-15T23:13:53.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.671+0000 7f9dc9497640 1 -- start start 2024-09-15T23:13:53.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.671+0000 7f9dc9497640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dc4102d30 0x7f9dc4100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:53.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.671+0000 7f9dc9497640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc4103650 0x7f9dc4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.671+0000 7f9dc9497640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc410c160 0x7f9dc4101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.671+0000 7f9dc9497640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc4111f60 con 0x7f9dc410c160 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.671+0000 7f9dc9497640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc4111de0 con 0x7f9dc4103650 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.671+0000 7f9dc9497640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc41120e0 con 0x7f9dc4102d30 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc410c160 0x7f9dc4101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dc4102d30 0x7f9dc4100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dc4102d30 0x7f9dc4100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52618/0 (socket says 172.21.15.62:52618) 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc410c160 0x7f9dc4101410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33158/0 (socket says 172.21.15.62:33158) 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc2ffd640 1 -- 172.21.15.62:0/3213743095 learned_addr learned my addr 172.21.15.62:0/3213743095 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:53.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc27fc640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc4103650 0x7f9dc4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc37fe640 1 -- 172.21.15.62:0/3213743095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dc4102d30 msgr2=0x7f9dc4100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc37fe640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dc4102d30 0x7f9dc4100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc37fe640 1 -- 172.21.15.62:0/3213743095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc4103650 msgr2=0x7f9dc4100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc37fe640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc4103650 0x7f9dc4100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc37fe640 1 -- 172.21.15.62:0/3213743095 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc41a68e0 con 0x7f9dc410c160 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc27fc640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc4103650 0x7f9dc4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc37fe640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc410c160 0x7f9dc4101410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9db400c710 tx=0x7f9db400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dbbfff640 1 -- 172.21.15.62:0/3213743095 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db4004220 con 0x7f9dc410c160 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dbbfff640 1 -- 172.21.15.62:0/3213743095 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9db400ce20 con 0x7f9dc410c160 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dbbfff640 1 -- 172.21.15.62:0/3213743095 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db4003e20 con 0x7f9dc410c160 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc41a6bd0 con 0x7f9dc410c160 2024-09-15T23:13:53.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.672+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc41a70c0 con 0x7f9dc410c160 2024-09-15T23:13:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.673+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc4103110 con 0x7f9dc410c160 2024-09-15T23:13:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.675+0000 7f9dbbfff640 1 -- 172.21.15.62:0/3213743095 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f9db40079b0 con 0x7f9dc410c160 2024-09-15T23:13:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.675+0000 7f9dbbfff640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9db00a32c0 0x7f9db00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.675+0000 7f9dc2ffd640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9db00a32c0 0x7f9db00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:53.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.675+0000 7f9dbbfff640 1 -- 172.21.15.62:0/3213743095 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f9db40cf9f0 con 0x7f9dc410c160 2024-09-15T23:13:53.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.676+0000 7f9dc2ffd640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9db00a32c0 0x7f9db00a5780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc007f80 tx=0x7f9dbc007670 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:53.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.679+0000 7f9dbbfff640 1 -- 172.21.15.62:0/3213743095 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9db4014030 con 0x7f9dc410c160 2024-09-15T23:13:53.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.891+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9dc4101ec0 con 0x7f9dc410c160 2024-09-15T23:13:53.893 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:53 smithi062 ceph-mon[30949]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:53.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.892+0000 7f9dbbfff640 1 -- 172.21.15.62:0/3213743095 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f9db409c550 con 0x7f9dc410c160 2024-09-15T23:13:53.893 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-15T23:13:53.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9db00a32c0 msgr2=0x7f9db00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:53.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9db00a32c0 0x7f9db00a5780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc007f80 tx=0x7f9dbc007670 comp rx=0 tx=0).stop 2024-09-15T23:13:53.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc410c160 msgr2=0x7f9dc4101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:53.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc410c160 0x7f9dc4101410 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9db400c710 tx=0x7f9db400cbd0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 shutdown_connections 2024-09-15T23:13:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9db00a32c0 0x7f9db00a5780 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dc410c160 0x7f9dc4101410 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc4103650 0x7f9dc4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 --2- 172.21.15.62:0/3213743095 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dc4102d30 0x7f9dc4100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 >> 172.21.15.62:0/3213743095 conn(0x7f9dc40fc790 msgr2=0x7f9dc40fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 shutdown_connections 2024-09-15T23:13:53.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:53.894+0000 7f9dc9497640 1 -- 172.21.15.62:0/3213743095 wait complete. 2024-09-15T23:13:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:53 smithi067 ceph-mon[35665]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:53 smithi148 ceph-mon[35610]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:54.698 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-15T23:13:54.698 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-15T23:13:54.698 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-09-15T23:13:54.698 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-15T23:13:54.698 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:13:54.699 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-15T23:13:54.699 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-15T23:13:54.699 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-15T23:13:54.699 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-15T23:13:54.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:54.699 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-09-15T23:13:54.717 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-15T23:13:54.717 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-15T23:13:54.717 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-09-15T23:13:54.717 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:13:54.717 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:13:54.718 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-15T23:13:54.719 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42298sec preferred_lft 42298sec 2024-09-15T23:13:54.719 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link noprefixroute 2024-09-15T23:13:54.719 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:13:54.719 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-15T23:13:54.720 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:13:54.720 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:13:54.720 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add name ceph-brx type bridge 2024-09-15T23:13:54.720 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr flush dev ceph-brx 2024-09-15T23:13:54.720 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set ceph-brx up 2024-09-15T23:13:54.720 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-15T23:13:54.720 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:13:54.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:13:54.820 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:54 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3213743095' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:13:54.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:13:54.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:54.937 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-15T23:13:54.963 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-09-15T23:13:54.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:54.964 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-09-15T23:13:55.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:54 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3213743095' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:13:55.036 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-09-15T23:13:55.036 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-15T23:13:55.036 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-15T23:13:55.037 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-15T23:13:55.037 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-15T23:13:55.037 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:13:55.037 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:13:55.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-15T23:13:55.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-15T23:13:55.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-15T23:13:55.037 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:13:55.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:54 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3213743095' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:13:55.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:13:55.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:13:55.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:55.327 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-09-15T23:13:55.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:55.344 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list-id 2024-09-15T23:13:55.399 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:13:55.399 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:13:55.399 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:13:55.399 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-15T23:13:55.399 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:13:55.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:13:55.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:13:55.521 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-15T23:13:55.521 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:13:55.521 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:13:55.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-15T23:13:55.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-15T23:13:55.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-15T23:13:55.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-15T23:13:55.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-15T23:13:55.522 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:13:55.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:13:55.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:13:55.650 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:13:55.650 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:13:55.650 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-09-15T23:13:55.650 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set dev brx.0 master ceph-brx 2024-09-15T23:13:55.650 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:13:55.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:13:55.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:13:55.730 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-15T23:13:55.730 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T23:13:55.730 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:13:55.746 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-15T23:13:55.746 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T23:13:55.746 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:13:55.804 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-09-15T23:13:55.867 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:55 smithi062 ceph-mon[30949]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:55.872 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-09-15T23:13:55.928 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-09-15T23:13:55.928 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/selinux 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-09-15T23:13:55.929 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysctl.service 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/nfs/rpc_pipefs 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/dc7738e5ff900b7eb25f1fac77d4c217b0bb28bae57705436a402f3eecc9e23e/merged 2024-09-15T23:13:55.930 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/4de7ca23a4c57dde895731a6b7e8f7513b748c78cd35fcdb2cea45a2f5e1298c/merged 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/0 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/3cae88d9a6c1c753e978ca8b4c252d8410d4f2152239a3d2d9de0162fa632b1f/merged 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/f0cbf74885f8a2b84ca0579fdb4dc66540ee5e9af627cc61ba12a1f858eb447b/merged 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/3905572d89bbab1b435dbea50a158cc7c1a5dbf1d2cf69c588e1cda54b261ef0/merged 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/182cb9d681d31f93331a66553b1bfe20cf7a3fae4fb7cc730f04916a2a42a953/merged 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/87b52e9980c08f25b8cd2269fb1c6c3435f46932eda6f7c3349d1ee853e4ffab/merged 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/84eddfd6fa799187da49fd6cd0393093b41bedb139d2c007d6c5affe3c268973/merged 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/d11e93e2455bd35eda88bf114e5794e2a76c1fc945b4fbd25d852a534640cfa8/merged 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/ef0d0f1cebea9960ae2de9c0ab68fbd68b85cd2e657cebc447e711e414fcf071/merged 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-09-15T23:13:55.931 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:13:55.932 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:13:55.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:55.932 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-09-15T23:13:55.984 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-15T23:13:55.984 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-15T23:13:56.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:55 smithi067 ceph-mon[35665]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:56.027 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-09-15T23:13:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:55 smithi148 ceph-mon[35610]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:56.052 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-09-15T23:13:56.109 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-09-15T23:13:56.109 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-09-15T23:13:56.110 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/selinux 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysctl.service 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/nfs/rpc_pipefs 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-09-15T23:13:56.111 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/dc7738e5ff900b7eb25f1fac77d4c217b0bb28bae57705436a402f3eecc9e23e/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/4de7ca23a4c57dde895731a6b7e8f7513b748c78cd35fcdb2cea45a2f5e1298c/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/0 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/3cae88d9a6c1c753e978ca8b4c252d8410d4f2152239a3d2d9de0162fa632b1f/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/f0cbf74885f8a2b84ca0579fdb4dc66540ee5e9af627cc61ba12a1f858eb447b/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/3905572d89bbab1b435dbea50a158cc7c1a5dbf1d2cf69c588e1cda54b261ef0/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/182cb9d681d31f93331a66553b1bfe20cf7a3fae4fb7cc730f04916a2a42a953/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/87b52e9980c08f25b8cd2269fb1c6c3435f46932eda6f7c3349d1ee853e4ffab/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/84eddfd6fa799187da49fd6cd0393093b41bedb139d2c007d6c5affe3c268973/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/d11e93e2455bd35eda88bf114e5794e2a76c1fc945b4fbd25d852a534640cfa8/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/containers/storage/overlay/ef0d0f1cebea9960ae2de9c0ab68fbd68b85cd2e657cebc447e711e414fcf071/merged 2024-09-15T23:13:56.112 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-09-15T23:13:56.113 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:13:56.113 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:13:56.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:56.113 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-09-15T23:13:56.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi062.stderr:2024-09-15T23:13:56.152+0000 7fc29d140480 -1 init, newargv = 0x5611d6ef7b30 newargc=15 2024-09-15T23:13:56.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi062.stderr:ceph-fuse[72676]: starting ceph client 2024-09-15T23:13:56.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi062.stderr:ceph-fuse[72676]: starting fuse 2024-09-15T23:13:56.167 INFO:teuthology.orchestra.run.smithi062.stdout:120 2024-09-15T23:13:56.167 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [120] 2024-09-15T23:13:56.167 DEBUG:teuthology.orchestra.run.smithi062:> sudo stdin-killer -- python3 -c ' 2024-09-15T23:13:56.167 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-09-15T23:13:56.167 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-09-15T23:13:56.167 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-09-15T23:13:56.167 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-15T23:13:56.168 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-15T23:13:56.169 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-15T23:13:56.169 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-09-15T23:13:56.169 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-09-15T23:13:56.169 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-09-15T23:13:56.169 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-15T23:13:56.169 DEBUG:teuthology.orchestra.run.smithi062:> 2024-09-15T23:13:56.169 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.admin")) 2024-09-15T23:13:56.169 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-09-15T23:13:56.287 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.admin.72676.asok 2024-09-15T23:13:56.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:13:56.298 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72676.asok 2024-09-15T23:13:56.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:56.298 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72676.asok status 2024-09-15T23:13:56.435 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:13:56.435 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-09-15T23:13:56.436 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "8921c8ef947807b717db041d85f0761b7914d5fc", 2024-09-15T23:13:56.436 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 19.1.1-326-g8921c8ef (8921c8ef947807b717db041d85f0761b7914d5fc) squid (stable)", 2024-09-15T23:13:56.436 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "admin", 2024-09-15T23:13:56.436 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-09-15T23:13:56.436 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-15T23:13:56.436 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "72676", 2024-09-15T23:13:56.436 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-09-15T23:13:56.436 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:13:56.436 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 14739, 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 14739 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2221925522 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:13:56.437 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2221925522 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.14739 192.168.0.1:0/2221925522", 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/2221925522", 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 17, 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 88, 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-09-15T23:13:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-09-15T23:13:56.439 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:13:56.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:56.444 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-09-15T23:13:56.444 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:13:56.460 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-09-15T23:13:56.460 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:13:56.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:56.461 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:13:56.529 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-15T23:13:56.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:56 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T23:13:56.640 INFO:teuthology.orchestra.run.smithi062.stdout:# file: . 2024-09-15T23:13:56.641 INFO:teuthology.orchestra.run.smithi062.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-15T23:13:56.641 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:13:56.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:13:56.646 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:13:56.646 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:13:56.663 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-15T23:13:56.961 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:57.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.470+0000 7f44ad3b4640 1 -- 172.21.15.62:0/725991563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8101de0 msgr2=0x7f44a81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:57.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.470+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/725991563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8101de0 0x7f44a81021c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4498009900 tx=0x7f449802a410 comp rx=0 tx=0).stop 2024-09-15T23:13:57.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.470+0000 7f44ad3b4640 1 -- 172.21.15.62:0/725991563 shutdown_connections 2024-09-15T23:13:57.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.470+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/725991563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44a8106f10 0x7f44a810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:57.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.470+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/725991563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44a8102790 0x7f44a81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:57.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.470+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/725991563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8101de0 0x7f44a81021c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:57.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.470+0000 7f44ad3b4640 1 -- 172.21.15.62:0/725991563 >> 172.21.15.62:0/725991563 conn(0x7f44a80fd8c0 msgr2=0x7f44a80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:57.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.470+0000 7f44ad3b4640 1 -- 172.21.15.62:0/725991563 shutdown_connections 2024-09-15T23:13:57.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.470+0000 7f44ad3b4640 1 -- 172.21.15.62:0/725991563 wait complete. 2024-09-15T23:13:57.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.471+0000 7f44ad3b4640 1 Processor -- start 2024-09-15T23:13:57.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.471+0000 7f44ad3b4640 1 -- start start 2024-09-15T23:13:57.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.471+0000 7f44ad3b4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44a8101de0 0x7f44a8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:57.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.471+0000 7f44ad3b4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44a8102790 0x7f44a8198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:57.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.471+0000 7f44ad3b4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8106f10 0x7f44a819cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:57.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.471+0000 7f44ad3b4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f44a8111f90 con 0x7f44a8106f10 2024-09-15T23:13:57.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.471+0000 7f44ad3b4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f44a8111e10 con 0x7f44a8101de0 2024-09-15T23:13:57.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.471+0000 7f44ad3b4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f44a8112110 con 0x7f44a8102790 2024-09-15T23:13:57.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8106f10 0x7f44a819cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:57.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8106f10 0x7f44a819cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33188/0 (socket says 172.21.15.62:33188) 2024-09-15T23:13:57.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a77fe640 1 -- 172.21.15.62:0/1154081187 learned_addr learned my addr 172.21.15.62:0/1154081187 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:57.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f449e7fc640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44a8102790 0x7f44a8198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:57.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a6ffd640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44a8101de0 0x7f44a8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:57.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a77fe640 1 -- 172.21.15.62:0/1154081187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44a8102790 msgr2=0x7f44a8198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:57.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a77fe640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44a8102790 0x7f44a8198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:57.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a77fe640 1 -- 172.21.15.62:0/1154081187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44a8101de0 msgr2=0x7f44a8198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:57.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a77fe640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44a8101de0 0x7f44a8198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:57.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a77fe640 1 -- 172.21.15.62:0/1154081187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a819d4c0 con 0x7f44a8106f10 2024-09-15T23:13:57.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a6ffd640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44a8101de0 0x7f44a8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:13:57.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f449e7fc640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44a8102790 0x7f44a8198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:13:57.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a77fe640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8106f10 0x7f44a819cde0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f44a000eaa0 tx=0x7f44a000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:57.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a4ff9640 1 -- 172.21.15.62:0/1154081187 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a000cdb0 con 0x7f44a8106f10 2024-09-15T23:13:57.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a4ff9640 1 -- 172.21.15.62:0/1154081187 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44a0005020 con 0x7f44a8106f10 2024-09-15T23:13:57.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a819d7b0 con 0x7f44a8106f10 2024-09-15T23:13:57.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44a4ff9640 1 -- 172.21.15.62:0/1154081187 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a00116b0 con 0x7f44a8106f10 2024-09-15T23:13:57.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.472+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a81a5090 con 0x7f44a8106f10 2024-09-15T23:13:57.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.473+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a81021c0 con 0x7f44a8106f10 2024-09-15T23:13:57.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.475+0000 7f44a4ff9640 1 -- 172.21.15.62:0/1154081187 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f44a0005560 con 0x7f44a8106f10 2024-09-15T23:13:57.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.475+0000 7f44a4ff9640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44940a3240 0x7f44940a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:57.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.475+0000 7f44a6ffd640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44940a3240 0x7f44940a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:57.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.475+0000 7f44a4ff9640 1 -- 172.21.15.62:0/1154081187 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f44a00d0240 con 0x7f44a8106f10 2024-09-15T23:13:57.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.476+0000 7f44a6ffd640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44940a3240 0x7f44940a5700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4498000c00 tx=0x7f4498032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:57.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.478+0000 7f44a4ff9640 1 -- 172.21.15.62:0/1154081187 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f44a0016030 con 0x7f44a8106f10 2024-09-15T23:13:57.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.665+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f44a802d7c0 con 0x7f44a8106f10 2024-09-15T23:13:57.666 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:57 smithi062 ceph-mon[30949]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:57.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.665+0000 7f44a4ff9640 1 -- 172.21.15.62:0/1154081187 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+589 (secure 0 0 0) 0x7f44a009cd20 con 0x7f44a8106f10 2024-09-15T23:13:57.667 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/1303697831 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:57.667 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/2614061737 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:57.667 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/574363632 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:57.667 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/1277095060 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:57.667 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/2614061737 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:57.667 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/1277095060 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:57.668 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/50968825 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:57.668 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/2637169275 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:57.668 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3526091028 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:57.668 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3293859590 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:57.668 INFO:teuthology.orchestra.run.smithi062.stderr:listed 10 entries 2024-09-15T23:13:57.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44940a3240 msgr2=0x7f44940a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:57.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44940a3240 0x7f44940a5700 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4498000c00 tx=0x7f4498032c70 comp rx=0 tx=0).stop 2024-09-15T23:13:57.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8106f10 msgr2=0x7f44a819cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:57.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8106f10 0x7f44a819cde0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f44a000eaa0 tx=0x7f44a000ef60 comp rx=0 tx=0).stop 2024-09-15T23:13:57.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 shutdown_connections 2024-09-15T23:13:57.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44940a3240 0x7f44940a5700 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:57.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8106f10 0x7f44a819cde0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:57.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f44a8102790 0x7f44a8198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:57.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 --2- 172.21.15.62:0/1154081187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44a8101de0 0x7f44a8198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:57.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 >> 172.21.15.62:0/1154081187 conn(0x7f44a80fd8c0 msgr2=0x7f44a80fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:57.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 shutdown_connections 2024-09-15T23:13:57.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:57.668+0000 7f44ad3b4640 1 -- 172.21.15.62:0/1154081187 wait complete. 2024-09-15T23:13:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:57 smithi067 ceph-mon[35665]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:57 smithi148 ceph-mon[35610]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:13:58.288 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:13:58.288 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:13:58.306 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-15T23:13:58.606 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:13:58.852 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:58 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1154081187' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:13:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:58 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1154081187' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:13:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:58 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1154081187' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:13:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.061+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/2792444889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 msgr2=0x7f1e44110910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.061+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/2792444889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 0x7f1e44110910 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c009ca0 tx=0x7f1e3c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.061+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/2792444889 shutdown_connections 2024-09-15T23:13:59.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.061+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/2792444889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 0x7f1e44110910 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.061+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/2792444889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44101440 0x7f1e4410df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.061+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/2792444889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e44100b20 0x7f1e44100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.061+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/2792444889 >> 172.21.15.62:0/2792444889 conn(0x7f1e440fc7d0 msgr2=0x7f1e440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.061+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/2792444889 shutdown_connections 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.061+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/2792444889 wait complete. 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.062+0000 7f1e4c7d6640 1 Processor -- start 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.062+0000 7f1e4c7d6640 1 -- start start 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.062+0000 7f1e4c7d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44100b20 0x7f1e4419a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.062+0000 7f1e4c7d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e44101440 0x7f1e4419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.062+0000 7f1e4c7d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 0x7f1e4419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.062+0000 7f1e4c7d6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1e44114230 con 0x7f1e4410e500 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.062+0000 7f1e4c7d6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e441140b0 con 0x7f1e44100b20 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.062+0000 7f1e4c7d6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1e441143b0 con 0x7f1e44101440 2024-09-15T23:13:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.062+0000 7f1e4ad73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 0x7f1e4419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4ad73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 0x7f1e4419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33220/0 (socket says 172.21.15.62:33220) 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4ad73640 1 -- 172.21.15.62:0/1140500363 learned_addr learned my addr 172.21.15.62:0/1140500363 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4a572640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44100b20 0x7f1e4419a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e49d71640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e44101440 0x7f1e4419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4ad73640 1 -- 172.21.15.62:0/1140500363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e44101440 msgr2=0x7f1e4419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4ad73640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e44101440 0x7f1e4419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4ad73640 1 -- 172.21.15.62:0/1140500363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44100b20 msgr2=0x7f1e4419a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4ad73640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44100b20 0x7f1e4419a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4ad73640 1 -- 172.21.15.62:0/1140500363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4419f6a0 con 0x7f1e4410e500 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4a572640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44100b20 0x7f1e4419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4ad73640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 0x7f1e4419efc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0319f0 tx=0x7f1e3c031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e3b7fe640 1 -- 172.21.15.62:0/1140500363 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c003b70 con 0x7f1e4410e500 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e3b7fe640 1 -- 172.21.15.62:0/1140500363 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e3c003d10 con 0x7f1e4410e500 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4419f930 con 0x7f1e4410e500 2024-09-15T23:13:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e3b7fe640 1 -- 172.21.15.62:0/1140500363 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3c02fd60 con 0x7f1e4410e500 2024-09-15T23:13:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.063+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e441a7270 con 0x7f1e4410e500 2024-09-15T23:13:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.064+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e44100f00 con 0x7f1e4410e500 2024-09-15T23:13:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.066+0000 7f1e3b7fe640 1 -- 172.21.15.62:0/1140500363 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f1e3c004050 con 0x7f1e4410e500 2024-09-15T23:13:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.066+0000 7f1e3b7fe640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1e300a3240 0x7f1e300a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:13:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.066+0000 7f1e4a572640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1e300a3240 0x7f1e300a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:13:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.066+0000 7f1e3b7fe640 1 -- 172.21.15.62:0/1140500363 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f1e3c0f0d40 con 0x7f1e4410e500 2024-09-15T23:13:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.067+0000 7f1e4a572640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1e300a3240 0x7f1e300a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000afb0 tx=0x7f1e4000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:13:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.071+0000 7f1e3b7fe640 1 -- 172.21.15.62:0/1140500363 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1e3c0bd850 con 0x7f1e4410e500 2024-09-15T23:13:59.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.258+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1e441a7440 con 0x7f1e4410e500 2024-09-15T23:13:59.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.258+0000 7f1e3b7fe640 1 -- 172.21.15.62:0/1140500363 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+589 (secure 0 0 0) 0x7f1e3c0bd150 con 0x7f1e4410e500 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/1303697831 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/2614061737 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/574363632 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/1277095060 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/2614061737 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/1277095060 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/50968825 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/2637169275 2024-09-16T23:06:39.607848+0000 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3526091028 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:59.260 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3293859590 2024-09-16T23:06:15.815154+0000 2024-09-15T23:13:59.261 INFO:teuthology.orchestra.run.smithi062.stderr:listed 10 entries 2024-09-15T23:13:59.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1e300a3240 msgr2=0x7f1e300a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1e300a3240 0x7f1e300a5700 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000afb0 tx=0x7f1e4000a2d0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 msgr2=0x7f1e4419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 0x7f1e4419efc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c0319f0 tx=0x7f1e3c031a20 comp rx=0 tx=0).stop 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 shutdown_connections 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1e300a3240 0x7f1e300a5700 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e4410e500 0x7f1e4419efc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e44101440 0x7f1e4419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 --2- 172.21.15.62:0/1140500363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e44100b20 0x7f1e4419a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 >> 172.21.15.62:0/1140500363 conn(0x7f1e440fc7d0 msgr2=0x7f1e440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 shutdown_connections 2024-09-15T23:13:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:13:59.261+0000 7f1e4c7d6640 1 -- 172.21.15.62:0/1140500363 wait complete. 2024-09-15T23:13:59.848 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:59 smithi062 ceph-mon[30949]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-15T23:13:59.849 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:13:59 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1140500363' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:13:59.876 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-09-15T23:13:59.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:59.877 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-15T23:13:59.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi062.stderr:ceph-fuse[72676]: fuse finished with error 0 and tester_r 0 2024-09-15T23:13:59.904 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T23:13:59.927 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T23:13:59.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:13:59.928 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-15T23:13:59.994 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:14:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:59 smithi067 ceph-mon[35665]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-15T23:14:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:13:59 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1140500363' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:14:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:59 smithi148 ceph-mon[35610]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-15T23:14:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:13:59 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1140500363' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:14:00.289 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:00.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.715+0000 7f6a362e7640 1 -- 172.21.15.62:0/703977190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30102d40 msgr2=0x7f6a30103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:00.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.715+0000 7f6a362e7640 1 --2- 172.21.15.62:0/703977190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30102d40 0x7f6a30103120 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009900 tx=0x7f6a2002a480 comp rx=0 tx=0).stop 2024-09-15T23:14:00.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.715+0000 7f6a362e7640 1 -- 172.21.15.62:0/703977190 shutdown_connections 2024-09-15T23:14:00.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.715+0000 7f6a362e7640 1 --2- 172.21.15.62:0/703977190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a3010c170 0x7f6a3010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:00.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.715+0000 7f6a362e7640 1 --2- 172.21.15.62:0/703977190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a30103660 0x7f6a3010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:00.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.715+0000 7f6a362e7640 1 --2- 172.21.15.62:0/703977190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30102d40 0x7f6a30103120 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:00.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.715+0000 7f6a362e7640 1 -- 172.21.15.62:0/703977190 >> 172.21.15.62:0/703977190 conn(0x7f6a300fc7a0 msgr2=0x7f6a300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:00.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.715+0000 7f6a362e7640 1 -- 172.21.15.62:0/703977190 shutdown_connections 2024-09-15T23:14:00.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.715+0000 7f6a362e7640 1 -- 172.21.15.62:0/703977190 wait complete. 2024-09-15T23:14:00.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.716+0000 7f6a362e7640 1 Processor -- start 2024-09-15T23:14:00.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.716+0000 7f6a362e7640 1 -- start start 2024-09-15T23:14:00.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a362e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a30102d40 0x7f6a30198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:00.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a362e7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30103660 0x7f6a30198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:00.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a362e7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a3010c170 0x7f6a3019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:00.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a362e7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6a30111fc0 con 0x7f6a30103660 2024-09-15T23:14:00.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a362e7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6a30111e40 con 0x7f6a30102d40 2024-09-15T23:14:00.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a362e7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6a30112140 con 0x7f6a3010c170 2024-09-15T23:14:00.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30103660 0x7f6a30198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:00.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30103660 0x7f6a30198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33236/0 (socket says 172.21.15.62:33236) 2024-09-15T23:14:00.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2f7fe640 1 -- 172.21.15.62:0/255937213 learned_addr learned my addr 172.21.15.62:0/255937213 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:00.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2ffff640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a30102d40 0x7f6a30198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:00.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a34884640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a3010c170 0x7f6a3019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:00.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2f7fe640 1 -- 172.21.15.62:0/255937213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a3010c170 msgr2=0x7f6a3019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:00.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2f7fe640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a3010c170 0x7f6a3019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:00.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2f7fe640 1 -- 172.21.15.62:0/255937213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a30102d40 msgr2=0x7f6a30198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:00.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2f7fe640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a30102d40 0x7f6a30198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:00.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2f7fe640 1 -- 172.21.15.62:0/255937213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3019d400 con 0x7f6a30103660 2024-09-15T23:14:00.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.717+0000 7f6a2f7fe640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30103660 0x7f6a30198990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6a2400c970 tx=0x7f6a2400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:00.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.718+0000 7f6a2d7fa640 1 -- 172.21.15.62:0/255937213 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24007bb0 con 0x7f6a30103660 2024-09-15T23:14:00.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.718+0000 7f6a2d7fa640 1 -- 172.21.15.62:0/255937213 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a24007d50 con 0x7f6a30103660 2024-09-15T23:14:00.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.718+0000 7f6a2d7fa640 1 -- 172.21.15.62:0/255937213 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a24018820 con 0x7f6a30103660 2024-09-15T23:14:00.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.718+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3019d6f0 con 0x7f6a30103660 2024-09-15T23:14:00.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.718+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a301a4fd0 con 0x7f6a30103660 2024-09-15T23:14:00.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.719+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a30103120 con 0x7f6a30103660 2024-09-15T23:14:00.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.720+0000 7f6a2d7fa640 1 -- 172.21.15.62:0/255937213 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6a240189c0 con 0x7f6a30103660 2024-09-15T23:14:00.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.720+0000 7f6a2d7fa640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6a180a32c0 0x7f6a180a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:00.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.721+0000 7f6a2ffff640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6a180a32c0 0x7f6a180a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:00.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.721+0000 7f6a2d7fa640 1 -- 172.21.15.62:0/255937213 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f6a240cfbd0 con 0x7f6a30103660 2024-09-15T23:14:00.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.721+0000 7f6a2ffff640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6a180a32c0 0x7f6a180a5780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6a20000c00 tx=0x7f6a20005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:00.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.725+0000 7f6a2d7fa640 1 -- 172.21.15.62:0/255937213 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6a24014030 con 0x7f6a30103660 2024-09-15T23:14:00.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.905+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a30107270 con 0x7f6a30103660 2024-09-15T23:14:00.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.907+0000 7f6a2d7fa640 1 -- 172.21.15.62:0/255937213 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f6a2409c650 con 0x7f6a30103660 2024-09-15T23:14:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:14:00.908 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:13:42.233963+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:14:00.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.909+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6a180a32c0 msgr2=0x7f6a180a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:00.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.909+0000 7f6a362e7640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6a180a32c0 0x7f6a180a5780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6a20000c00 tx=0x7f6a20005b50 comp rx=0 tx=0).stop 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.909+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30103660 msgr2=0x7f6a30198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.909+0000 7f6a362e7640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30103660 0x7f6a30198990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6a2400c970 tx=0x7f6a2400ce30 comp rx=0 tx=0).stop 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.910+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 shutdown_connections 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.910+0000 7f6a362e7640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6a180a32c0 0x7f6a180a5780 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.910+0000 7f6a362e7640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6a3010c170 0x7f6a3019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.910+0000 7f6a362e7640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a30103660 0x7f6a30198990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.910+0000 7f6a362e7640 1 --2- 172.21.15.62:0/255937213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a30102d40 0x7f6a30198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.910+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 >> 172.21.15.62:0/255937213 conn(0x7f6a300fc7a0 msgr2=0x7f6a300fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.910+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 shutdown_connections 2024-09-15T23:14:00.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:00.910+0000 7f6a362e7640 1 -- 172.21.15.62:0/255937213 wait complete. 2024-09-15T23:14:01.745 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:14:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:01 smithi062 ceph-mon[30949]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-15T23:14:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:01 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/255937213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:02.003 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:01 smithi067 ceph-mon[35665]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-15T23:14:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:01 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/255937213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:01 smithi148 ceph-mon[35610]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-09-15T23:14:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:01 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/255937213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 -- 172.21.15.62:0/954522610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b841036e0 msgr2=0x7f1b84107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/954522610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b841036e0 0x7f1b84107b90 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009930 tx=0x7f1b7402a420 comp rx=0 tx=0).stop 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 -- 172.21.15.62:0/954522610 shutdown_connections 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/954522610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b841082a0 0x7f1b8410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/954522610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b841036e0 0x7f1b84107b90 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/954522610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b84102d30 0x7f1b84103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 -- 172.21.15.62:0/954522610 >> 172.21.15.62:0/954522610 conn(0x7f1b840fc7d0 msgr2=0x7f1b840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 -- 172.21.15.62:0/954522610 shutdown_connections 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 -- 172.21.15.62:0/954522610 wait complete. 2024-09-15T23:14:02.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.466+0000 7f1b8b87b640 1 Processor -- start 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b8b87b640 1 -- start start 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b8b87b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b84102d30 0x7f1b84100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b8b87b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b841036e0 0x7f1b84100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b8b87b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b841082a0 0x7f1b84101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b8b87b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1b841120a0 con 0x7f1b84102d30 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b8b87b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1b84111f20 con 0x7f1b841082a0 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b8b87b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1b84112220 con 0x7f1b841036e0 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89617640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b84102d30 0x7f1b84100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89617640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b84102d30 0x7f1b84100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55286/0 (socket says 172.21.15.62:55286) 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89617640 1 -- 172.21.15.62:0/1011563895 learned_addr learned my addr 172.21.15.62:0/1011563895 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89e18640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b841082a0 0x7f1b84101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b88e16640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b841036e0 0x7f1b84100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:02.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89617640 1 -- 172.21.15.62:0/1011563895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b841036e0 msgr2=0x7f1b84100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89617640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b841036e0 0x7f1b84100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89617640 1 -- 172.21.15.62:0/1011563895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b841082a0 msgr2=0x7f1b84101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89617640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b841082a0 0x7f1b84101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89617640 1 -- 172.21.15.62:0/1011563895 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b841a6a80 con 0x7f1b84102d30 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b88e16640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b841036e0 0x7f1b84100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.467+0000 7f1b89617640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b84102d30 0x7f1b84100990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000d950 tx=0x7f1b8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.468+0000 7f1b7a7fc640 1 -- 172.21.15.62:0/1011563895 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b80017070 con 0x7f1b84102d30 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.468+0000 7f1b7a7fc640 1 -- 172.21.15.62:0/1011563895 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b80009c30 con 0x7f1b84102d30 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.468+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b841a6d70 con 0x7f1b84102d30 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.468+0000 7f1b7a7fc640 1 -- 172.21.15.62:0/1011563895 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8000f6b0 con 0x7f1b84102d30 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.468+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b841a7220 con 0x7f1b84102d30 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.469+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b84103110 con 0x7f1b84102d30 2024-09-15T23:14:02.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.470+0000 7f1b7a7fc640 1 -- 172.21.15.62:0/1011563895 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f1b800109d0 con 0x7f1b84102d30 2024-09-15T23:14:02.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.471+0000 7f1b7a7fc640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b700a3240 0x7f1b700a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:02.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.471+0000 7f1b88e16640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b700a3240 0x7f1b700a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:02.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.471+0000 7f1b7a7fc640 1 -- 172.21.15.62:0/1011563895 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f1b800cf760 con 0x7f1b84102d30 2024-09-15T23:14:02.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.471+0000 7f1b88e16640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b700a3240 0x7f1b700a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009900 tx=0x7f1b74005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:02.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.474+0000 7f1b7a7fc640 1 -- 172.21.15.62:0/1011563895 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1b80009de0 con 0x7f1b84102d30 2024-09-15T23:14:02.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.653+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b8410a2b0 con 0x7f1b84102d30 2024-09-15T23:14:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.655+0000 7f1b7a7fc640 1 -- 172.21.15.62:0/1011563895 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f1b8009c240 con 0x7f1b84102d30 2024-09-15T23:14:02.656 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:14:02.657 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:13:42.233963+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:14:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b700a3240 msgr2=0x7f1b700a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b700a3240 0x7f1b700a5700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009900 tx=0x7f1b74005a20 comp rx=0 tx=0).stop 2024-09-15T23:14:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b84102d30 msgr2=0x7f1b84100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b84102d30 0x7f1b84100990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1b8000d950 tx=0x7f1b8000de10 comp rx=0 tx=0).stop 2024-09-15T23:14:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 shutdown_connections 2024-09-15T23:14:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1b700a3240 0x7f1b700a5700 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:02.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b841082a0 0x7f1b84101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1b841036e0 0x7f1b84100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 --2- 172.21.15.62:0/1011563895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b84102d30 0x7f1b84100990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 >> 172.21.15.62:0/1011563895 conn(0x7f1b840fc7d0 msgr2=0x7f1b84105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 shutdown_connections 2024-09-15T23:14:02.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:02.657+0000 7f1b8b87b640 1 -- 172.21.15.62:0/1011563895 wait complete. 2024-09-15T23:14:03.472 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-09-15T23:14:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:03 smithi062 ceph-mon[30949]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:03 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1011563895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:03.736 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:03 smithi067 ceph-mon[35665]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:03 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1011563895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:03 smithi148 ceph-mon[35610]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:03 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1011563895' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 -- 172.21.15.62:0/3059943923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8106f10 msgr2=0x7f54a810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 --2- 172.21.15.62:0/3059943923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8106f10 0x7f54a810ea90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009ca0 tx=0x7f54a002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 -- 172.21.15.62:0/3059943923 shutdown_connections 2024-09-15T23:14:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 --2- 172.21.15.62:0/3059943923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8106f10 0x7f54a810ea90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 --2- 172.21.15.62:0/3059943923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8102790 0x7f54a81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 --2- 172.21.15.62:0/3059943923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54a8101de0 0x7f54a81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 -- 172.21.15.62:0/3059943923 >> 172.21.15.62:0/3059943923 conn(0x7f54a80fd8c0 msgr2=0x7f54a80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 -- 172.21.15.62:0/3059943923 shutdown_connections 2024-09-15T23:14:04.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 -- 172.21.15.62:0/3059943923 wait complete. 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 Processor -- start 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.209+0000 7f54ae615640 1 -- start start 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54ae615640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101de0 0x7f54a8198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54ae615640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8102790 0x7f54a8198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54ae615640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54a8106f10 0x7f54a819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54ae615640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f54a8112060 con 0x7f54a8102790 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54ae615640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54a8111ee0 con 0x7f54a8101de0 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54ae615640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f54a81121e0 con 0x7f54a8106f10 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8102790 0x7f54a8198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8102790 0x7f54a8198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55306/0 (socket says 172.21.15.62:55306) 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a77fe640 1 -- 172.21.15.62:0/2847430824 learned_addr learned my addr 172.21.15.62:0/2847430824 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a7fff640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101de0 0x7f54a8198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:04.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a77fe640 1 -- 172.21.15.62:0/2847430824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54a8106f10 msgr2=0x7f54a819ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a77fe640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54a8106f10 0x7f54a819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a77fe640 1 -- 172.21.15.62:0/2847430824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101de0 msgr2=0x7f54a8198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a77fe640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101de0 0x7f54a8198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a77fe640 1 -- 172.21.15.62:0/2847430824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a819d510 con 0x7f54a8102790 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a7fff640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101de0 0x7f54a8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.210+0000 7f54a77fe640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8102790 0x7f54a8198aa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f549c00b4a0 tx=0x7f549c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.211+0000 7f54a57fa640 1 -- 172.21.15.62:0/2847430824 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c014070 con 0x7f54a8102790 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.211+0000 7f54a57fa640 1 -- 172.21.15.62:0/2847430824 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f549c010af0 con 0x7f54a8102790 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.211+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a819d800 con 0x7f54a8102790 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.211+0000 7f54a57fa640 1 -- 172.21.15.62:0/2847430824 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c017d10 con 0x7f54a8102790 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.211+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a81a50e0 con 0x7f54a8102790 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.212+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a8102240 con 0x7f54a8102790 2024-09-15T23:14:04.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.213+0000 7f54a57fa640 1 -- 172.21.15.62:0/2847430824 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f549c017070 con 0x7f54a8102790 2024-09-15T23:14:04.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.213+0000 7f54a57fa640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54900a32c0 0x7f54900a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:04.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.214+0000 7f54a7fff640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54900a32c0 0x7f54900a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:04.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.214+0000 7f54a57fa640 1 -- 172.21.15.62:0/2847430824 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f549c0ceef0 con 0x7f54a8102790 2024-09-15T23:14:04.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.214+0000 7f54a7fff640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54900a32c0 0x7f54900a5780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5498009930 tx=0x7f5498008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:04.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.218+0000 7f54a57fa640 1 -- 172.21.15.62:0/2847430824 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f549c09b9d0 con 0x7f54a8102790 2024-09-15T23:14:04.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.430+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f54a802d7c0 con 0x7f54a8102790 2024-09-15T23:14:04.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.602+0000 7f54a57fa640 1 -- 172.21.15.62:0/2847430824 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f549c09b2d0 con 0x7f54a8102790 2024-09-15T23:14:04.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.605+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54900a32c0 msgr2=0x7f54900a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:04.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.605+0000 7f54ae615640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54900a32c0 0x7f54900a5780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5498009930 tx=0x7f5498008040 comp rx=0 tx=0).stop 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.605+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8102790 msgr2=0x7f54a8198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.605+0000 7f54ae615640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8102790 0x7f54a8198aa0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f549c00b4a0 tx=0x7f549c00b960 comp rx=0 tx=0).stop 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.606+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 shutdown_connections 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.606+0000 7f54ae615640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54900a32c0 0x7f54900a5780 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.606+0000 7f54ae615640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54a8106f10 0x7f54a819ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.606+0000 7f54ae615640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a8102790 0x7f54a8198aa0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.606+0000 7f54ae615640 1 --2- 172.21.15.62:0/2847430824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54a8101de0 0x7f54a8198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.606+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 >> 172.21.15.62:0/2847430824 conn(0x7f54a80fd8c0 msgr2=0x7f54a80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.606+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 shutdown_connections 2024-09-15T23:14:04.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:04.606+0000 7f54ae615640 1 -- 172.21.15.62:0/2847430824 wait complete. 2024-09-15T23:14:04.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:04 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2847430824' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-15T23:14:05.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:04 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2847430824' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-15T23:14:05.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:04 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2847430824' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-09-15T23:14:05.410 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-09-15T23:14:05.411 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-09-15T23:14:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:05 smithi062 ceph-mon[30949]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:05 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2847430824' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-15T23:14:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:05 smithi062 ceph-mon[30949]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:14:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:05 smithi067 ceph-mon[35665]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:05 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2847430824' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-15T23:14:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:05 smithi067 ceph-mon[35665]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:14:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:05 smithi148 ceph-mon[35610]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:05 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2847430824' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-09-15T23:14:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:05 smithi148 ceph-mon[35610]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:14:06.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.109+0000 7f5d54a1e640 1 -- 172.21.15.62:0/2142479155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50101400 msgr2=0x7f5d5010d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:06.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.109+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/2142479155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50101400 0x7f5d5010d930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009cd0 tx=0x7f5d4802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.109+0000 7f5d54a1e640 1 -- 172.21.15.62:0/2142479155 shutdown_connections 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.109+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/2142479155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50101400 0x7f5d5010d930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.109+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/2142479155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5d50100a60 0x7f5d50100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.109+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/2142479155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d50106a60 0x7f5d50106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.109+0000 7f5d54a1e640 1 -- 172.21.15.62:0/2142479155 >> 172.21.15.62:0/2142479155 conn(0x7f5d500fc780 msgr2=0x7f5d500feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.109+0000 7f5d54a1e640 1 -- 172.21.15.62:0/2142479155 shutdown_connections 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.109+0000 7f5d54a1e640 1 -- 172.21.15.62:0/2142479155 wait complete. 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d54a1e640 1 Processor -- start 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d54a1e640 1 -- start start 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d54a1e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d50100a60 0x7f5d50199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d54a1e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5d50101400 0x7f5d501997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:06.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d54a1e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50106a60 0x7f5d501a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d54a1e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5d50110e60 con 0x7f5d50106a60 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d54a1e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5d50110ce0 con 0x7f5d50100a60 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d54a1e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5d50110fe0 con 0x7f5d50101400 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d4ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50106a60 0x7f5d501a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d4ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50106a60 0x7f5d501a42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55312/0 (socket says 172.21.15.62:55312) 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d4e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d50100a60 0x7f5d50199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d4ed9d640 1 -- 172.21.15.62:0/4076480153 learned_addr learned my addr 172.21.15.62:0/4076480153 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.110+0000 7f5d4dd9b640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5d50101400 0x7f5d501997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d4ed9d640 1 -- 172.21.15.62:0/4076480153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5d50101400 msgr2=0x7f5d501997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d4ed9d640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5d50101400 0x7f5d501997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d4ed9d640 1 -- 172.21.15.62:0/4076480153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d50100a60 msgr2=0x7f5d50199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d4ed9d640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d50100a60 0x7f5d50199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d4ed9d640 1 -- 172.21.15.62:0/4076480153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d50197060 con 0x7f5d50106a60 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d4e59c640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d50100a60 0x7f5d50199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d4ed9d640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50106a60 0x7f5d501a42f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5d48005cc0 tx=0x7f5d48003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d3f7fe640 1 -- 172.21.15.62:0/4076480153 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48004040 con 0x7f5d50106a60 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d3f7fe640 1 -- 172.21.15.62:0/4076480153 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d480046c0 con 0x7f5d50106a60 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d501972f0 con 0x7f5d50106a60 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d3f7fe640 1 -- 172.21.15.62:0/4076480153 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48046480 con 0x7f5d50106a60 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.111+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d50197880 con 0x7f5d50106a60 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.112+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d501021a0 con 0x7f5d50106a60 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.113+0000 7f5d3f7fe640 1 -- 172.21.15.62:0/4076480153 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f5d480305a0 con 0x7f5d50106a60 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.114+0000 7f5d3f7fe640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d380a32c0 0x7f5d380a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.114+0000 7f5d4e59c640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d380a32c0 0x7f5d380a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.114+0000 7f5d3f7fe640 1 -- 172.21.15.62:0/4076480153 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f5d4802e030 con 0x7f5d50106a60 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.115+0000 7f5d4e59c640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d380a32c0 0x7f5d380a5780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5d40005d00 tx=0x7f5d4000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:06.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.118+0000 7f5d3f7fe640 1 -- 172.21.15.62:0/4076480153 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5d4804b060 con 0x7f5d50106a60 2024-09-15T23:14:06.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.330+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f5d5006ad80 con 0x7f5d50106a60 2024-09-15T23:14:06.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.609+0000 7f5d3f7fe640 1 -- 172.21.15.62:0/4076480153 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f5d480bdd60 con 0x7f5d50106a60 2024-09-15T23:14:06.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d380a32c0 msgr2=0x7f5d380a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:06.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d380a32c0 0x7f5d380a5780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5d40005d00 tx=0x7f5d4000b410 comp rx=0 tx=0).stop 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50106a60 msgr2=0x7f5d501a42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50106a60 0x7f5d501a42f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5d48005cc0 tx=0x7f5d48003970 comp rx=0 tx=0).stop 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 shutdown_connections 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5d380a32c0 0x7f5d380a5780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d50106a60 0x7f5d501a42f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5d50101400 0x7f5d501997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 --2- 172.21.15.62:0/4076480153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5d50100a60 0x7f5d50199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 >> 172.21.15.62:0/4076480153 conn(0x7f5d500fc780 msgr2=0x7f5d500fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 shutdown_connections 2024-09-15T23:14:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:06.611+0000 7f5d54a1e640 1 -- 172.21.15.62:0/4076480153 wait complete. 2024-09-15T23:14:06.928 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:06 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4076480153' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-09-15T23:14:07.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:06 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4076480153' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-09-15T23:14:07.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:06 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4076480153' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-09-15T23:14:07.243 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-09-15T23:14:07.500 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:07.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:07 smithi062 ceph-mon[30949]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:07.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:07 smithi062 ceph-mon[30949]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-15T23:14:07.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:07 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4076480153' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-15T23:14:07.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:07 smithi062 ceph-mon[30949]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-15T23:14:07.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:07 smithi062 ceph-mon[30949]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:14:07.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:07 smithi062 ceph-mon[30949]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-15T23:14:07.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:07 smithi062 ceph-mon[30949]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-15T23:14:07.887 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:07 smithi067 ceph-mon[35665]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:07.888 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:07 smithi067 ceph-mon[35665]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-15T23:14:07.888 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:07 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4076480153' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-15T23:14:07.888 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:07 smithi067 ceph-mon[35665]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-15T23:14:07.888 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:07 smithi067 ceph-mon[35665]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:14:07.888 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:07 smithi067 ceph-mon[35665]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-15T23:14:07.888 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:07 smithi067 ceph-mon[35665]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-15T23:14:07.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.957+0000 7fa915e11640 1 -- 172.21.15.62:0/3912261258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910102d40 msgr2=0x7fa910103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:07.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.957+0000 7fa915e11640 1 --2- 172.21.15.62:0/3912261258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910102d40 0x7fa910103120 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa900009900 tx=0x7fa90002a480 comp rx=0 tx=0).stop 2024-09-15T23:14:07.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.957+0000 7fa915e11640 1 -- 172.21.15.62:0/3912261258 shutdown_connections 2024-09-15T23:14:07.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.957+0000 7fa915e11640 1 --2- 172.21.15.62:0/3912261258 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa91010c170 0x7fa91010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:07.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.957+0000 7fa915e11640 1 --2- 172.21.15.62:0/3912261258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa910103660 0x7fa91010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:07.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.957+0000 7fa915e11640 1 --2- 172.21.15.62:0/3912261258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910102d40 0x7fa910103120 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:07.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.957+0000 7fa915e11640 1 -- 172.21.15.62:0/3912261258 >> 172.21.15.62:0/3912261258 conn(0x7fa9100fc7a0 msgr2=0x7fa9100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:07.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.957+0000 7fa915e11640 1 -- 172.21.15.62:0/3912261258 shutdown_connections 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.957+0000 7fa915e11640 1 -- 172.21.15.62:0/3912261258 wait complete. 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.958+0000 7fa915e11640 1 Processor -- start 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.958+0000 7fa915e11640 1 -- start start 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa915e11640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa910102d40 0x7fa910198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa915e11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910103660 0x7fa910198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa915e11640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa91010c170 0x7fa91019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa915e11640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa910111f70 con 0x7fa910103660 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa915e11640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa910111df0 con 0x7fa910102d40 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa915e11640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa9101120f0 con 0x7fa91010c170 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910103660 0x7fa910198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910103660 0x7fa910198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55324/0 (socket says 172.21.15.62:55324) 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90effd640 1 -- 172.21.15.62:0/3878748027 learned_addr learned my addr 172.21.15.62:0/3878748027 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90ffff640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa91010c170 0x7fa91019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90f7fe640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa910102d40 0x7fa910198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:07.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90effd640 1 -- 172.21.15.62:0/3878748027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa91010c170 msgr2=0x7fa91019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90effd640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa91010c170 0x7fa91019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90effd640 1 -- 172.21.15.62:0/3878748027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa910102d40 msgr2=0x7fa910198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90effd640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa910102d40 0x7fa910198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90effd640 1 -- 172.21.15.62:0/3878748027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91019d3e0 con 0x7fa910103660 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.959+0000 7fa90effd640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910103660 0x7fa910198970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c940 tx=0x7fa90400ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.960+0000 7fa90cff9640 1 -- 172.21.15.62:0/3878748027 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa904004220 con 0x7fa910103660 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.960+0000 7fa90cff9640 1 -- 172.21.15.62:0/3878748027 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9040043c0 con 0x7fa910103660 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.960+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91019d6d0 con 0x7fa910103660 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.960+0000 7fa90cff9640 1 -- 172.21.15.62:0/3878748027 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa904004a00 con 0x7fa910103660 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.960+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9101a4fb0 con 0x7fa910103660 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.961+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa910103120 con 0x7fa910103660 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.962+0000 7fa90cff9640 1 -- 172.21.15.62:0/3878748027 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fa9040079b0 con 0x7fa910103660 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.962+0000 7fa90cff9640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8f80a32c0 0x7fa8f80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:07.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.963+0000 7fa90f7fe640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8f80a32c0 0x7fa8f80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:07.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.963+0000 7fa90cff9640 1 -- 172.21.15.62:0/3878748027 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fa9040cfaa0 con 0x7fa910103660 2024-09-15T23:14:07.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.963+0000 7fa90f7fe640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8f80a32c0 0x7fa8f80a5780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa900005ab0 tx=0x7fa900005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:07.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:07.967+0000 7fa90cff9640 1 -- 172.21.15.62:0/3878748027 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa904014030 con 0x7fa910103660 2024-09-15T23:14:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:07 smithi148 ceph-mon[35610]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:07 smithi148 ceph-mon[35610]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-15T23:14:08.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:07 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4076480153' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-09-15T23:14:08.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:07 smithi148 ceph-mon[35610]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-09-15T23:14:08.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:07 smithi148 ceph-mon[35610]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-09-15T23:14:08.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:07 smithi148 ceph-mon[35610]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-09-15T23:14:08.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:07 smithi148 ceph-mon[35610]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-09-15T23:14:08.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.178+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fa910107270 con 0x7fa910103660 2024-09-15T23:14:08.274 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:14:07 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: 2024-09-15T23:14:07.885+0000 7ff291122640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-15T23:14:08.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.477+0000 7fa90cff9640 1 -- 172.21.15.62:0/3878748027 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7fa90409c580 con 0x7fa910103660 2024-09-15T23:14:08.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8f80a32c0 msgr2=0x7fa8f80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:08.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8f80a32c0 0x7fa8f80a5780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa900005ab0 tx=0x7fa900005a20 comp rx=0 tx=0).stop 2024-09-15T23:14:08.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910103660 msgr2=0x7fa910198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:08.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910103660 0x7fa910198970 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c940 tx=0x7fa90400ce00 comp rx=0 tx=0).stop 2024-09-15T23:14:08.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 shutdown_connections 2024-09-15T23:14:08.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8f80a32c0 0x7fa8f80a5780 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:08.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa91010c170 0x7fa91019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:08.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa910103660 0x7fa910198970 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:08.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 --2- 172.21.15.62:0/3878748027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa910102d40 0x7fa910198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:08.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 >> 172.21.15.62:0/3878748027 conn(0x7fa9100fc7a0 msgr2=0x7fa9100fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:08.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 shutdown_connections 2024-09-15T23:14:08.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:08.479+0000 7fa915e11640 1 -- 172.21.15.62:0/3878748027 wait complete. 2024-09-15T23:14:08.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] up:active 2024-09-15T23:14:08.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-15T23:14:08.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-15T23:14:08.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: Cluster is now healthy 2024-09-15T23:14:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-15T23:14:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-09-15T23:14:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-15T23:14:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3878748027' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-15T23:14:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3878748027' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-15T23:14:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] up:active 2024-09-15T23:14:08.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:08 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:14:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] up:active 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: Cluster is now healthy 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3878748027' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3878748027' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] up:active 2024-09-15T23:14:09.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:08 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:14:09.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] up:active 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: Cluster is now healthy 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3878748027' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3878748027' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] up:active 2024-09-15T23:14:09.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:08 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:14:09.269 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-09-15T23:14:09.531 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:09.881 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:09 smithi062 ceph-mon[30949]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:09.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.965+0000 7fd352b56640 1 -- 172.21.15.62:0/873585632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 msgr2=0x7fd34c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.965+0000 7fd352b56640 1 --2- 172.21.15.62:0/873585632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 0x7fd34c106e60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd340009930 tx=0x7fd34002a420 comp rx=0 tx=0).stop 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.965+0000 7fd352b56640 1 -- 172.21.15.62:0/873585632 shutdown_connections 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.965+0000 7fd352b56640 1 --2- 172.21.15.62:0/873585632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd34c101420 0x7fd34c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.965+0000 7fd352b56640 1 --2- 172.21.15.62:0/873585632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34c100a80 0x7fd34c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.965+0000 7fd352b56640 1 --2- 172.21.15.62:0/873585632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 0x7fd34c106e60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.965+0000 7fd352b56640 1 -- 172.21.15.62:0/873585632 >> 172.21.15.62:0/873585632 conn(0x7fd34c0fc7a0 msgr2=0x7fd34c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.965+0000 7fd352b56640 1 -- 172.21.15.62:0/873585632 shutdown_connections 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.965+0000 7fd352b56640 1 -- 172.21.15.62:0/873585632 wait complete. 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd352b56640 1 Processor -- start 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd352b56640 1 -- start start 2024-09-15T23:14:09.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd352b56640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd34c100a80 0x7fd34c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd352b56640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34c101420 0x7fd34c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd352b56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 0x7fd34c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd352b56640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c112020 con 0x7fd34c106a80 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd352b56640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c111ea0 con 0x7fd34c100a80 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd352b56640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c1121a0 con 0x7fd34c101420 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd3510f3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 0x7fd34c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd3510f3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 0x7fd34c19ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55336/0 (socket says 172.21.15.62:55336) 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd3510f3640 1 -- 172.21.15.62:0/3051791343 learned_addr learned my addr 172.21.15.62:0/3051791343 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd3508f2640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd34c100a80 0x7fd34c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd34bfff640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34c101420 0x7fd34c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:09.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd3510f3640 1 -- 172.21.15.62:0/3051791343 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34c101420 msgr2=0x7fd34c198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.966+0000 7fd3510f3640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34c101420 0x7fd34c198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd3510f3640 1 -- 172.21.15.62:0/3051791343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd34c100a80 msgr2=0x7fd34c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd3510f3640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd34c100a80 0x7fd34c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd3510f3640 1 -- 172.21.15.62:0/3051791343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34c19d590 con 0x7fd34c106a80 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd34bfff640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34c101420 0x7fd34c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd3510f3640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 0x7fd34c19ceb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd34400b970 tx=0x7fd34400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd349ffb640 1 -- 172.21.15.62:0/3051791343 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd344019070 con 0x7fd34c106a80 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd349ffb640 1 -- 172.21.15.62:0/3051791343 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd34400cb40 con 0x7fd34c106a80 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd34c19d880 con 0x7fd34c106a80 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd349ffb640 1 -- 172.21.15.62:0/3051791343 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd344007650 con 0x7fd34c106a80 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.967+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd34c1a5160 con 0x7fd34c106a80 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.968+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd34c100f60 con 0x7fd34c106a80 2024-09-15T23:14:09.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.969+0000 7fd349ffb640 1 -- 172.21.15.62:0/3051791343 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7fd34401c070 con 0x7fd34c106a80 2024-09-15T23:14:09.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.970+0000 7fd349ffb640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd33c0a3240 0x7fd33c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:09.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.970+0000 7fd3508f2640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd33c0a3240 0x7fd33c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:09.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.970+0000 7fd349ffb640 1 -- 172.21.15.62:0/3051791343 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fd3440cf140 con 0x7fd34c106a80 2024-09-15T23:14:09.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.970+0000 7fd3508f2640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd33c0a3240 0x7fd33c0a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd340009900 tx=0x7fd3400058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:09.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:09.974+0000 7fd349ffb640 1 -- 172.21.15.62:0/3051791343 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd34409b5d0 con 0x7fd34c106a80 2024-09-15T23:14:10.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:09 smithi067 ceph-mon[35665]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:10.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:09 smithi148 ceph-mon[35610]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-09-15T23:14:10.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.186+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd34c109050 con 0x7fd34c106a80 2024-09-15T23:14:10.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.629+0000 7fd349ffb640 1 -- 172.21.15.62:0/3051791343 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v24) ==== 115+0+0 (secure 0 0 0) 0x7fd34409f8d0 con 0x7fd34c106a80 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.631+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd33c0a3240 msgr2=0x7fd33c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.631+0000 7fd352b56640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd33c0a3240 0x7fd33c0a5700 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd340009900 tx=0x7fd3400058d0 comp rx=0 tx=0).stop 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.631+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 msgr2=0x7fd34c19ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.631+0000 7fd352b56640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 0x7fd34c19ceb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd34400b970 tx=0x7fd34400be30 comp rx=0 tx=0).stop 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.631+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 shutdown_connections 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.631+0000 7fd352b56640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd33c0a3240 0x7fd33c0a5700 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.631+0000 7fd352b56640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd34c106a80 0x7fd34c19ceb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.631+0000 7fd352b56640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd34c101420 0x7fd34c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.631+0000 7fd352b56640 1 --2- 172.21.15.62:0/3051791343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd34c100a80 0x7fd34c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.632+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 >> 172.21.15.62:0/3051791343 conn(0x7fd34c0fc7a0 msgr2=0x7fd34c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.632+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 shutdown_connections 2024-09-15T23:14:10.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:10.632+0000 7fd352b56640 1 -- 172.21.15.62:0/3051791343 wait complete. 2024-09-15T23:14:10.931 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:10 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3051791343' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-15T23:14:11.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:10 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3051791343' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-15T23:14:11.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:10 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3051791343' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-09-15T23:14:11.484 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-09-15T23:14:11.484 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-09-15T23:14:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:11 smithi062 ceph-mon[30949]: pgmap v303: 129 pgs: 129 active+clean; 582 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 1 op/s 2024-09-15T23:14:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:11 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3051791343' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-15T23:14:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:11 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:14:11.749 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:11 smithi067 ceph-mon[35665]: pgmap v303: 129 pgs: 129 active+clean; 582 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 1 op/s 2024-09-15T23:14:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:11 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3051791343' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-15T23:14:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:11 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:14:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:11 smithi148 ceph-mon[35610]: pgmap v303: 129 pgs: 129 active+clean; 582 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 1 op/s 2024-09-15T23:14:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:11 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3051791343' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-09-15T23:14:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:11 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:14:12.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.217+0000 7f6681b6f640 1 -- 172.21.15.62:0/2798228568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 msgr2=0x7f667c10e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.217+0000 7f6681b6f640 1 --2- 172.21.15.62:0/2798228568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 0x7f667c10e610 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6674009ca0 tx=0x7f667402a950 comp rx=0 tx=0).stop 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.217+0000 7f6681b6f640 1 -- 172.21.15.62:0/2798228568 shutdown_connections 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.217+0000 7f6681b6f640 1 --2- 172.21.15.62:0/2798228568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 0x7f667c10e610 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.217+0000 7f6681b6f640 1 --2- 172.21.15.62:0/2798228568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f667c103660 0x7f667c10bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.217+0000 7f6681b6f640 1 --2- 172.21.15.62:0/2798228568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f667c102d40 0x7f667c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.217+0000 7f6681b6f640 1 -- 172.21.15.62:0/2798228568 >> 172.21.15.62:0/2798228568 conn(0x7f667c0fc7a0 msgr2=0x7f667c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.217+0000 7f6681b6f640 1 -- 172.21.15.62:0/2798228568 shutdown_connections 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.217+0000 7f6681b6f640 1 -- 172.21.15.62:0/2798228568 wait complete. 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f6681b6f640 1 Processor -- start 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f6681b6f640 1 -- start start 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f6681b6f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f667c102d40 0x7f667c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:12.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f6681b6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f667c103660 0x7f667c1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f6681b6f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 0x7f667c19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f6681b6f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f667c111f50 con 0x7f667c10c170 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f6681b6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f667c111dd0 con 0x7f667c103660 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f6681b6f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f667c1120d0 con 0x7f667c102d40 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f667bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 0x7f667c19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.218+0000 7f667bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 0x7f667c19cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55354/0 (socket says 172.21.15.62:55354) 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667bfff640 1 -- 172.21.15.62:0/3075562687 learned_addr learned my addr 172.21.15.62:0/3075562687 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667b7fe640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f667c102d40 0x7f667c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667affd640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f667c103660 0x7f667c1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667bfff640 1 -- 172.21.15.62:0/3075562687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f667c102d40 msgr2=0x7f667c198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667bfff640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f667c102d40 0x7f667c198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667bfff640 1 -- 172.21.15.62:0/3075562687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f667c103660 msgr2=0x7f667c1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:12.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667bfff640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f667c103660 0x7f667c1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667bfff640 1 -- 172.21.15.62:0/3075562687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c19d320 con 0x7f667c10c170 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667affd640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f667c103660 0x7f667c1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f667bfff640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 0x7f667c19cc40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6674004af0 tx=0x7f667402f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f6678ff9640 1 -- 172.21.15.62:0/3075562687 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674034040 con 0x7f667c10c170 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667c19d5b0 con 0x7f667c10c170 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f6678ff9640 1 -- 172.21.15.62:0/3075562687 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6674030920 con 0x7f667c10c170 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f6678ff9640 1 -- 172.21.15.62:0/3075562687 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6674004740 con 0x7f667c10c170 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.219+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f667c1a4ef0 con 0x7f667c10c170 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.220+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667c103120 con 0x7f667c10c170 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.221+0000 7f6678ff9640 1 -- 172.21.15.62:0/3075562687 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f6674037020 con 0x7f667c10c170 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.222+0000 7f6678ff9640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66640a3240 0x7f66640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.222+0000 7f667b7fe640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66640a3240 0x7f66640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:12.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.222+0000 7f6678ff9640 1 -- 172.21.15.62:0/3075562687 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f66740f0b50 con 0x7f667c10c170 2024-09-15T23:14:12.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.223+0000 7f667b7fe640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66640a3240 0x7f66640a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f666c00abe0 tx=0x7f666c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:12.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.225+0000 7f6678ff9640 1 -- 172.21.15.62:0/3075562687 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f66740811b0 con 0x7f667c10c170 2024-09-15T23:14:12.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.426+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f667c0feac0 con 0x7f66640a3240 2024-09-15T23:14:12.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.451+0000 7f6678ff9640 1 -- 172.21.15.62:0/3075562687 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f667c0feac0 con 0x7f66640a3240 2024-09-15T23:14:12.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66640a3240 msgr2=0x7f66640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:12.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66640a3240 0x7f66640a5700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f666c00abe0 tx=0x7f666c00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 msgr2=0x7f667c19cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 0x7f667c19cc40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6674004af0 tx=0x7f667402f5b0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 shutdown_connections 2024-09-15T23:14:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66640a3240 0x7f66640a5700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f667c10c170 0x7f667c19cc40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f667c103660 0x7f667c1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 --2- 172.21.15.62:0/3075562687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f667c102d40 0x7f667c198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 >> 172.21.15.62:0/3075562687 conn(0x7f667c0fc7a0 msgr2=0x7f667c0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 shutdown_connections 2024-09-15T23:14:12.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:12.453+0000 7f6681b6f640 1 -- 172.21.15.62:0/3075562687 wait complete. 2024-09-15T23:14:12.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:12 smithi062 ceph-mon[30949]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-09-15T23:14:12.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:12 smithi062 ceph-mon[30949]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:12.731 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:14:12 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:14:12.432+0000 7f9059f60640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:14:12.731 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:14:12 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:14:12.432+0000 7f9059f60640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:14:12.731 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:14:12 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:14:12.432+0000 7f9059f60640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:14:12.731 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:14:12 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:14:12.432+0000 7f9059f60640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:14:12.731 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:14:12 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:14:12.432+0000 7f9059f60640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-15T23:14:13.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:12 smithi067 ceph-mon[35665]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-09-15T23:14:13.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:12 smithi067 ceph-mon[35665]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:13.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:12 smithi148 ceph-mon[35610]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-09-15T23:14:13.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:12 smithi148 ceph-mon[35610]: from='client.14793 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:13.149 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-09-15T23:14:13.408 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.901+0000 7f4649674640 1 -- 172.21.15.62:0/536150574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 msgr2=0x7f464410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.901+0000 7f4649674640 1 --2- 172.21.15.62:0/536150574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 0x7f464410ea90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4634009ca0 tx=0x7f463402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:14:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.902+0000 7f4649674640 1 -- 172.21.15.62:0/536150574 shutdown_connections 2024-09-15T23:14:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.902+0000 7f4649674640 1 --2- 172.21.15.62:0/536150574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 0x7f464410ea90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.902+0000 7f4649674640 1 --2- 172.21.15.62:0/536150574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4644102790 0x7f46441067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.902+0000 7f4649674640 1 --2- 172.21.15.62:0/536150574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4644101de0 0x7f46441021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.902+0000 7f4649674640 1 -- 172.21.15.62:0/536150574 >> 172.21.15.62:0/536150574 conn(0x7f46440fd8c0 msgr2=0x7f46440ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.902+0000 7f4649674640 1 -- 172.21.15.62:0/536150574 shutdown_connections 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.902+0000 7f4649674640 1 -- 172.21.15.62:0/536150574 wait complete. 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.902+0000 7f4649674640 1 Processor -- start 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.902+0000 7f4649674640 1 -- start start 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f4649674640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4644101de0 0x7f46440744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f4649674640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4644102790 0x7f4644074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f4649674640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 0x7f4644074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f4649674640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4644111f70 con 0x7f4644106f10 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f4649674640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4644111df0 con 0x7f4644101de0 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f4649674640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f46441120f0 con 0x7f4644102790 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46437fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 0x7f4644074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:13.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f4642ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4644101de0 0x7f46440744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46437fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 0x7f4644074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45046/0 (socket says 172.21.15.62:45046) 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46437fe640 1 -- 172.21.15.62:0/4188291862 learned_addr learned my addr 172.21.15.62:0/4188291862 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46427fc640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4644102790 0x7f4644074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46437fe640 1 -- 172.21.15.62:0/4188291862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4644102790 msgr2=0x7f4644074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46437fe640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4644102790 0x7f4644074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46437fe640 1 -- 172.21.15.62:0/4188291862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4644101de0 msgr2=0x7f46440744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46437fe640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4644101de0 0x7f46440744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46437fe640 1 -- 172.21.15.62:0/4188291862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46441a68b0 con 0x7f4644106f10 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f4642ffd640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4644101de0 0x7f46440744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.903+0000 7f46437fe640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 0x7f4644074f70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4634031a20 tx=0x7f4634031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.904+0000 7f463bfff640 1 -- 172.21.15.62:0/4188291862 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634003b70 con 0x7f4644106f10 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.904+0000 7f463bfff640 1 -- 172.21.15.62:0/4188291862 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4634003d10 con 0x7f4644106f10 2024-09-15T23:14:13.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.904+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46441a6b40 con 0x7f4644106f10 2024-09-15T23:14:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.904+0000 7f463bfff640 1 -- 172.21.15.62:0/4188291862 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463402fc40 con 0x7f4644106f10 2024-09-15T23:14:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.904+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46441a70d0 con 0x7f4644106f10 2024-09-15T23:14:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.905+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46441021c0 con 0x7f4644106f10 2024-09-15T23:14:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.907+0000 7f463bfff640 1 -- 172.21.15.62:0/4188291862 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) ==== 138355+0+0 (secure 0 0 0) 0x7f4634004b40 con 0x7f4644106f10 2024-09-15T23:14:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.907+0000 7f463bfff640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46300a32c0 0x7f46300a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.908+0000 7f4642ffd640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46300a32c0 0x7f46300a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.908+0000 7f463bfff640 1 -- 172.21.15.62:0/4188291862 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f46340f0d70 con 0x7f4644106f10 2024-09-15T23:14:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.908+0000 7f4642ffd640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46300a32c0 0x7f46300a5780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f463c009fc0 tx=0x7f463c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:13.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:13.911+0000 7f463bfff640 1 -- 172.21.15.62:0/4188291862 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f46340bd8e0 con 0x7f4644106f10 2024-09-15T23:14:14.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.112+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4644062630 con 0x7f46300a32c0 2024-09-15T23:14:14.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.137+0000 7f463bfff640 1 -- 172.21.15.62:0/4188291862 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4644062630 con 0x7f46300a32c0 2024-09-15T23:14:14.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46300a32c0 msgr2=0x7f46300a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:14.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46300a32c0 0x7f46300a5780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f463c009fc0 tx=0x7f463c005f90 comp rx=0 tx=0).stop 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 msgr2=0x7f4644074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 0x7f4644074f70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4634031a20 tx=0x7f4634031a50 comp rx=0 tx=0).stop 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 shutdown_connections 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f46300a32c0 0x7f46300a5780 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4644106f10 0x7f4644074f70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4644102790 0x7f4644074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 --2- 172.21.15.62:0/4188291862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4644101de0 0x7f46440744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 >> 172.21.15.62:0/4188291862 conn(0x7f46440fd8c0 msgr2=0x7f46440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 shutdown_connections 2024-09-15T23:14:14.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:14.139+0000 7f4649674640 1 -- 172.21.15.62:0/4188291862 wait complete. 2024-09-15T23:14:14.901 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-09-15T23:14:15.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:14 smithi067 ceph-mon[35665]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-15T23:14:15.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:14 smithi067 ceph-mon[35665]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:15.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:14 smithi148 ceph-mon[35610]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-15T23:14:15.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:14 smithi148 ceph-mon[35610]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:15.157 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:15.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:14 smithi062 ceph-mon[30949]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-15T23:14:15.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:14 smithi062 ceph-mon[30949]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.593+0000 7fdd46e4a640 1 -- 172.21.15.62:0/2625442290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 msgr2=0x7fdd4010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.593+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/2625442290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 0x7fdd4010d950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009930 tx=0x7fdd3802a420 comp rx=0 tx=0).stop 2024-09-15T23:14:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.593+0000 7fdd46e4a640 1 -- 172.21.15.62:0/2625442290 shutdown_connections 2024-09-15T23:14:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.593+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/2625442290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 0x7fdd4010d950 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.593+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/2625442290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40100a80 0x7fdd40100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.593+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/2625442290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd40106a80 0x7fdd40106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:15.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.593+0000 7fdd46e4a640 1 -- 172.21.15.62:0/2625442290 >> 172.21.15.62:0/2625442290 conn(0x7fdd400fc7a0 msgr2=0x7fdd400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.593+0000 7fdd46e4a640 1 -- 172.21.15.62:0/2625442290 shutdown_connections 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.593+0000 7fdd46e4a640 1 -- 172.21.15.62:0/2625442290 wait complete. 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd46e4a640 1 Processor -- start 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd46e4a640 1 -- start start 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd46e4a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40100a80 0x7fdd401a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd46e4a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 0x7fdd401a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd46e4a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd40106a80 0x7fdd4019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd46e4a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40110f70 con 0x7fdd40101420 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd46e4a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40110df0 con 0x7fdd40100a80 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd46e4a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdd401110f0 con 0x7fdd40106a80 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd3ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 0x7fdd401a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd3ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 0x7fdd401a0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45076/0 (socket says 172.21.15.62:45076) 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd3ffff640 1 -- 172.21.15.62:0/4093713572 learned_addr learned my addr 172.21.15.62:0/4093713572 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:15.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.594+0000 7fdd44be6640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40100a80 0x7fdd401a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd453e7640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd40106a80 0x7fdd4019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd3ffff640 1 -- 172.21.15.62:0/4093713572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd40106a80 msgr2=0x7fdd4019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd3ffff640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd40106a80 0x7fdd4019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd3ffff640 1 -- 172.21.15.62:0/4093713572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40100a80 msgr2=0x7fdd401a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd3ffff640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40100a80 0x7fdd401a0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd3ffff640 1 -- 172.21.15.62:0/4093713572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4019aee0 con 0x7fdd40101420 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd453e7640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd40106a80 0x7fdd4019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd3ffff640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 0x7fdd401a0a90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000ba50 tx=0x7fdd3000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd3dffb640 1 -- 172.21.15.62:0/4093713572 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30019070 con 0x7fdd40101420 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd3dffb640 1 -- 172.21.15.62:0/4093713572 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd30014440 con 0x7fdd40101420 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4019b1d0 con 0x7fdd40101420 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd3dffb640 1 -- 172.21.15.62:0/4093713572 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd300075d0 con 0x7fdd40101420 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.595+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd401a7540 con 0x7fdd40101420 2024-09-15T23:14:15.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.596+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd401021c0 con 0x7fdd40101420 2024-09-15T23:14:15.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.597+0000 7fdd3dffb640 1 -- 172.21.15.62:0/4093713572 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fdd3001c070 con 0x7fdd40101420 2024-09-15T23:14:15.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.598+0000 7fdd3dffb640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdd2c0a32d0 0x7fdd2c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:15.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.598+0000 7fdd44be6640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdd2c0a32d0 0x7fdd2c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:15.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.598+0000 7fdd3dffb640 1 -- 172.21.15.62:0/4093713572 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fdd300cf1d0 con 0x7fdd40101420 2024-09-15T23:14:15.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.598+0000 7fdd44be6640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdd2c0a32d0 0x7fdd2c0a5790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009990 tx=0x7fdd28009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:15.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.602+0000 7fdd3dffb640 1 -- 172.21.15.62:0/4093713572 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fdd3009bc60 con 0x7fdd40101420 2024-09-15T23:14:15.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.809+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fdd4019bbc0 con 0x7fdd2c0a32d0 2024-09-15T23:14:15.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.814+0000 7fdd3dffb640 1 -- 172.21.15.62:0/4093713572 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fdd4019bbc0 con 0x7fdd2c0a32d0 2024-09-15T23:14:15.816 INFO:teuthology.orchestra.run.smithi062.stdout:/volumes/qa/sv_0/ca4d943c-f142-4efe-870f-4affc909ab16 2024-09-15T23:14:15.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdd2c0a32d0 msgr2=0x7fdd2c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:15.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdd2c0a32d0 0x7fdd2c0a5790 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009990 tx=0x7fdd28009110 comp rx=0 tx=0).stop 2024-09-15T23:14:15.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 msgr2=0x7fdd401a0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:15.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 0x7fdd401a0a90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000ba50 tx=0x7fdd3000be50 comp rx=0 tx=0).stop 2024-09-15T23:14:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 shutdown_connections 2024-09-15T23:14:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdd2c0a32d0 0x7fdd2c0a5790 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdd40106a80 0x7fdd4019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd40101420 0x7fdd401a0a90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 --2- 172.21.15.62:0/4093713572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd40100a80 0x7fdd401a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 >> 172.21.15.62:0/4093713572 conn(0x7fdd400fc7a0 msgr2=0x7fdd4010c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 shutdown_connections 2024-09-15T23:14:15.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:15.816+0000 7fdd46e4a640 1 -- 172.21.15.62:0/4093713572 wait complete. 2024-09-15T23:14:15.852 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:15 smithi062 ceph-mon[30949]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-15T23:14:16.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:15 smithi067 ceph-mon[35665]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-15T23:14:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:15 smithi148 ceph-mon[35610]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-09-15T23:14:16.721 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-09-15T23:14:16.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:16 smithi062 ceph-mon[30949]: pgmap v306: 129 pgs: 129 active+clean; 584 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-09-15T23:14:16.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:16 smithi062 ceph-mon[30949]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:16.987 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:17.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:16 smithi148 ceph-mon[35610]: pgmap v306: 129 pgs: 129 active+clean; 584 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-09-15T23:14:17.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:16 smithi148 ceph-mon[35610]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:16 smithi067 ceph-mon[35665]: pgmap v306: 129 pgs: 129 active+clean; 584 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-09-15T23:14:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:16 smithi067 ceph-mon[35665]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:17.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.434+0000 7f4fa2957640 1 -- 172.21.15.62:0/1529434605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 msgr2=0x7f4f9c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:17.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.434+0000 7f4fa2957640 1 --2- 172.21.15.62:0/1529434605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 0x7f4f9c1021c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009900 tx=0x7f4f8c02a410 comp rx=0 tx=0).stop 2024-09-15T23:14:17.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.434+0000 7f4fa2957640 1 -- 172.21.15.62:0/1529434605 shutdown_connections 2024-09-15T23:14:17.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.434+0000 7f4fa2957640 1 --2- 172.21.15.62:0/1529434605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f9c106f10 0x7f4f9c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.434+0000 7f4fa2957640 1 --2- 172.21.15.62:0/1529434605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c102790 0x7f4f9c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.434+0000 7f4fa2957640 1 --2- 172.21.15.62:0/1529434605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 0x7f4f9c1021c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.434+0000 7f4fa2957640 1 -- 172.21.15.62:0/1529434605 >> 172.21.15.62:0/1529434605 conn(0x7f4f9c0fd8c0 msgr2=0x7f4f9c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.434+0000 7f4fa2957640 1 -- 172.21.15.62:0/1529434605 shutdown_connections 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.434+0000 7f4fa2957640 1 -- 172.21.15.62:0/1529434605 wait complete. 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.435+0000 7f4fa2957640 1 Processor -- start 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.435+0000 7f4fa2957640 1 -- start start 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4fa2957640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 0x7f4f9c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4fa2957640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c102790 0x7f4f9c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4fa2957640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f9c106f10 0x7f4f9c19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4fa2957640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c111f70 con 0x7f4f9c101de0 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4fa2957640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c111df0 con 0x7f4f9c102790 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4fa2957640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c1120f0 con 0x7f4f9c106f10 2024-09-15T23:14:17.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 0x7f4f9c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 0x7f4f9c1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45102/0 (socket says 172.21.15.62:45102) 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9bfff640 1 -- 172.21.15.62:0/377097779 learned_addr learned my addr 172.21.15.62:0/377097779 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9b7fe640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c102790 0x7f4f9c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4fa0ef4640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f9c106f10 0x7f4f9c19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9bfff640 1 -- 172.21.15.62:0/377097779 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f9c106f10 msgr2=0x7f4f9c19cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9bfff640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f9c106f10 0x7f4f9c19cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9bfff640 1 -- 172.21.15.62:0/377097779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c102790 msgr2=0x7f4f9c198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9bfff640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c102790 0x7f4f9c198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9bfff640 1 -- 172.21.15.62:0/377097779 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9c19d380 con 0x7f4f9c101de0 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.436+0000 7f4f9bfff640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 0x7f4f9c1983d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c006010 tx=0x7f4f8c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.437+0000 7f4f997fa640 1 -- 172.21.15.62:0/377097779 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c03f030 con 0x7f4f9c101de0 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.437+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f9c19d610 con 0x7f4f9c101de0 2024-09-15T23:14:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.437+0000 7f4f997fa640 1 -- 172.21.15.62:0/377097779 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f8c003ce0 con 0x7f4f9c101de0 2024-09-15T23:14:17.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.437+0000 7f4f997fa640 1 -- 172.21.15.62:0/377097779 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c035070 con 0x7f4f9c101de0 2024-09-15T23:14:17.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.437+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f9c1a4f50 con 0x7f4f9c101de0 2024-09-15T23:14:17.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.438+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f9c1032c0 con 0x7f4f9c101de0 2024-09-15T23:14:17.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.439+0000 7f4f997fa640 1 -- 172.21.15.62:0/377097779 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4f8c042030 con 0x7f4f9c101de0 2024-09-15T23:14:17.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.440+0000 7f4f997fa640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f840a32d0 0x7f4f840a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:17.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.440+0000 7f4f9b7fe640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f840a32d0 0x7f4f840a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:17.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.440+0000 7f4f997fa640 1 -- 172.21.15.62:0/377097779 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f4f8c0f0590 con 0x7f4f9c101de0 2024-09-15T23:14:17.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.440+0000 7f4f9b7fe640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f840a32d0 0x7f4f840a5790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0fdce0 tx=0x7f4f9000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:17.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.443+0000 7f4f997fa640 1 -- 172.21.15.62:0/377097779 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4f8c0bc980 con 0x7f4f9c101de0 2024-09-15T23:14:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.646+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f4f9c062630 con 0x7f4f840a32d0 2024-09-15T23:14:17.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.670+0000 7f4f997fa640 1 -- 172.21.15.62:0/377097779 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4f9c062630 con 0x7f4f840a32d0 2024-09-15T23:14:17.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f840a32d0 msgr2=0x7f4f840a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:17.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f840a32d0 0x7f4f840a5790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0fdce0 tx=0x7f4f9000b390 comp rx=0 tx=0).stop 2024-09-15T23:14:17.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 msgr2=0x7f4f9c1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:17.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 0x7f4f9c1983d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c006010 tx=0x7f4f8c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 shutdown_connections 2024-09-15T23:14:17.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f840a32d0 0x7f4f840a5790 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f9c106f10 0x7f4f9c19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f9c102790 0x7f4f9c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 --2- 172.21.15.62:0/377097779 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f9c101de0 0x7f4f9c1983d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:17.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 >> 172.21.15.62:0/377097779 conn(0x7f4f9c0fd8c0 msgr2=0x7f4f9c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:17.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 shutdown_connections 2024-09-15T23:14:17.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:17.672+0000 7f4fa2957640 1 -- 172.21.15.62:0/377097779 wait complete. 2024-09-15T23:14:18.315 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-09-15T23:14:18.571 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:18 smithi062 ceph-mon[30949]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:18.943 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:18 smithi062 ceph-mon[30949]: pgmap v307: 129 pgs: 129 active+clean; 605 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-15T23:14:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:18 smithi148 ceph-mon[35610]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:18 smithi148 ceph-mon[35610]: pgmap v307: 129 pgs: 129 active+clean; 605 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-15T23:14:19.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1924233405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 msgr2=0x7fc9c810df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:19.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1924233405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 0x7fc9c810df40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8009900 tx=0x7fc9b802a410 comp rx=0 tx=0).stop 2024-09-15T23:14:19.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1924233405 shutdown_connections 2024-09-15T23:14:19.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1924233405 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9c810e480 0x7fc9c8110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1924233405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 0x7fc9c810df40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1924233405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c81008d0 0x7fc9c8100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1924233405 >> 172.21.15.62:0/1924233405 conn(0x7fc9c80fc7a0 msgr2=0x7fc9c80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1924233405 shutdown_connections 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1924233405 wait complete. 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 Processor -- start 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.024+0000 7fc9cf3f0640 1 -- start start 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cf3f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c81008d0 0x7fc9c819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cf3f0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 0x7fc9c819ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cf3f0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9c810e480 0x7fc9c819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cf3f0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c8114190 con 0x7fc9c8101280 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cf3f0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c8114010 con 0x7fc9c81008d0 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cf3f0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c8114310 con 0x7fc9c810e480 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cc98b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 0x7fc9c819ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:19.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cc98b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 0x7fc9c819ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45116/0 (socket says 172.21.15.62:45116) 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cc98b640 1 -- 172.21.15.62:0/1264204529 learned_addr learned my addr 172.21.15.62:0/1264204529 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cd18c640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c81008d0 0x7fc9c819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cd98d640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9c810e480 0x7fc9c819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cc98b640 1 -- 172.21.15.62:0/1264204529 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9c810e480 msgr2=0x7fc9c819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cc98b640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9c810e480 0x7fc9c819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cc98b640 1 -- 172.21.15.62:0/1264204529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c81008d0 msgr2=0x7fc9c819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cc98b640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c81008d0 0x7fc9c819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cc98b640 1 -- 172.21.15.62:0/1264204529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c819f680 con 0x7fc9c8101280 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cd98d640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9c810e480 0x7fc9c819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cd18c640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c81008d0 0x7fc9c819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cc98b640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 0x7fc9c819ac10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8006010 tx=0x7fc9b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9be7fc640 1 -- 172.21.15.62:0/1264204529 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b803f030 con 0x7fc9c8101280 2024-09-15T23:14:19.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9be7fc640 1 -- 172.21.15.62:0/1264204529 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9b8003ce0 con 0x7fc9c8101280 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c819f910 con 0x7fc9c8101280 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.025+0000 7fc9be7fc640 1 -- 172.21.15.62:0/1264204529 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8035070 con 0x7fc9c8101280 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.026+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c81a7250 con 0x7fc9c8101280 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.026+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c806b070 con 0x7fc9c8101280 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.028+0000 7fc9be7fc640 1 -- 172.21.15.62:0/1264204529 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc9b8042030 con 0x7fc9c8101280 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.028+0000 7fc9be7fc640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9b40a32d0 0x7fc9b40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.028+0000 7fc9cd18c640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9b40a32d0 0x7fc9b40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.028+0000 7fc9be7fc640 1 -- 172.21.15.62:0/1264204529 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fc9b80f0540 con 0x7fc9c8101280 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.029+0000 7fc9cd18c640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9b40a32d0 0x7fc9b40a5790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4005fd0 tx=0x7fc9c4009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:19.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.032+0000 7fc9be7fc640 1 -- 172.21.15.62:0/1264204529 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc9b80bcf80 con 0x7fc9c8101280 2024-09-15T23:14:19.076 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:18 smithi067 ceph-mon[35665]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:19.076 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:18 smithi067 ceph-mon[35665]: pgmap v307: 129 pgs: 129 active+clean; 605 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-15T23:14:19.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.235+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fc9c8062630 con 0x7fc9b40a32d0 2024-09-15T23:14:19.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.240+0000 7fc9be7fc640 1 -- 172.21.15.62:0/1264204529 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fc9c8062630 con 0x7fc9b40a32d0 2024-09-15T23:14:19.242 INFO:teuthology.orchestra.run.smithi062.stdout:/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e 2024-09-15T23:14:19.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9b40a32d0 msgr2=0x7fc9b40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:19.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9b40a32d0 0x7fc9b40a5790 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc9c4005fd0 tx=0x7fc9c4009500 comp rx=0 tx=0).stop 2024-09-15T23:14:19.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 msgr2=0x7fc9c819ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:19.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 0x7fc9c819ac10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8006010 tx=0x7fc9b80042f0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 shutdown_connections 2024-09-15T23:14:19.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9b40a32d0 0x7fc9b40a5790 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9c810e480 0x7fc9c819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9c8101280 0x7fc9c819ac10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 --2- 172.21.15.62:0/1264204529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c81008d0 0x7fc9c819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:19.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 >> 172.21.15.62:0/1264204529 conn(0x7fc9c80fc7a0 msgr2=0x7fc9c80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:19.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 shutdown_connections 2024-09-15T23:14:19.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:19.243+0000 7fc9cf3f0640 1 -- 172.21.15.62:0/1264204529 wait complete. 2024-09-15T23:14:19.888 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:14:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:19 smithi062 ceph-mon[30949]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:20.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:19 smithi148 ceph-mon[35610]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:20.145 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:20.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:19 smithi067 ceph-mon[35665]: from='client.14823 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:14:20.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 -- 172.21.15.62:0/1336080203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c1008d0 msgr2=0x7fd47c100cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:20.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 --2- 172.21.15.62:0/1336080203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c1008d0 0x7fd47c100cb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd478009900 tx=0x7fd47802a410 comp rx=0 tx=0).stop 2024-09-15T23:14:20.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 -- 172.21.15.62:0/1336080203 shutdown_connections 2024-09-15T23:14:20.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 --2- 172.21.15.62:0/1336080203 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd47c10e480 0x7fd47c110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 --2- 172.21.15.62:0/1336080203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47c101280 0x7fd47c10df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 --2- 172.21.15.62:0/1336080203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c1008d0 0x7fd47c100cb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 -- 172.21.15.62:0/1336080203 >> 172.21.15.62:0/1336080203 conn(0x7fd47c0fc7a0 msgr2=0x7fd47c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 -- 172.21.15.62:0/1336080203 shutdown_connections 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 -- 172.21.15.62:0/1336080203 wait complete. 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 Processor -- start 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.612+0000 7fd4840d0640 1 -- start start 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd4840d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47c1008d0 0x7fd47c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd4840d0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c101280 0x7fd47c19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd4840d0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd47c10e480 0x7fd47c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd4840d0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c1141b0 con 0x7fd47c101280 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd4840d0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c114030 con 0x7fd47c1008d0 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd4840d0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd47c114330 con 0x7fd47c10e480 2024-09-15T23:14:20.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48166b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c101280 0x7fd47c19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48166b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c101280 0x7fd47c19ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45130/0 (socket says 172.21.15.62:45130) 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48166b640 1 -- 172.21.15.62:0/3768854753 learned_addr learned my addr 172.21.15.62:0/3768854753 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd481e6c640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47c1008d0 0x7fd47c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48266d640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd47c10e480 0x7fd47c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48166b640 1 -- 172.21.15.62:0/3768854753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd47c10e480 msgr2=0x7fd47c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48166b640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd47c10e480 0x7fd47c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48166b640 1 -- 172.21.15.62:0/3768854753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47c1008d0 msgr2=0x7fd47c19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48166b640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47c1008d0 0x7fd47c19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48166b640 1 -- 172.21.15.62:0/3768854753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd47c19f680 con 0x7fd47c101280 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd48166b640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c101280 0x7fd47c19ac10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00e9f0 tx=0x7fd46c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.613+0000 7fd472ffd640 1 -- 172.21.15.62:0/3768854753 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c00cd80 con 0x7fd47c101280 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.614+0000 7fd472ffd640 1 -- 172.21.15.62:0/3768854753 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd46c010910 con 0x7fd47c101280 2024-09-15T23:14:20.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.614+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd47c19f970 con 0x7fd47c101280 2024-09-15T23:14:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.614+0000 7fd472ffd640 1 -- 172.21.15.62:0/3768854753 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd46c018990 con 0x7fd47c101280 2024-09-15T23:14:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.614+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd47c1a7250 con 0x7fd47c101280 2024-09-15T23:14:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.615+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd47c100cb0 con 0x7fd47c101280 2024-09-15T23:14:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.616+0000 7fd472ffd640 1 -- 172.21.15.62:0/3768854753 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd46c010ac0 con 0x7fd47c101280 2024-09-15T23:14:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.616+0000 7fd472ffd640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4680a3350 0x7fd4680a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.617+0000 7fd481e6c640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4680a3350 0x7fd4680a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.617+0000 7fd472ffd640 1 -- 172.21.15.62:0/3768854753 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fd46c0cff50 con 0x7fd47c101280 2024-09-15T23:14:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.617+0000 7fd481e6c640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4680a3350 0x7fd4680a5810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd478009230 tx=0x7fd4780058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:20.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.621+0000 7fd472ffd640 1 -- 172.21.15.62:0/3768854753 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd46c09c280 con 0x7fd47c101280 2024-09-15T23:14:20.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.832+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd47c109080 con 0x7fd47c101280 2024-09-15T23:14:20.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.833+0000 7fd472ffd640 1 -- 172.21.15.62:0/3768854753 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11903 (secure 0 0 0) 0x7fd46c0a0580 con 0x7fd47c101280 2024-09-15T23:14:20.834 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:14:20.834 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"btime":"2024-09-15T23:14:10:623582+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:14:10.188556+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:14:20.834 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 24 2024-09-15T23:14:20.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4680a3350 msgr2=0x7fd4680a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:20.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4680a3350 0x7fd4680a5810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd478009230 tx=0x7fd4780058d0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c101280 msgr2=0x7fd47c19ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:20.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c101280 0x7fd47c19ac10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd46c00e9f0 tx=0x7fd46c00eeb0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 shutdown_connections 2024-09-15T23:14:20.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4680a3350 0x7fd4680a5810 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd47c10e480 0x7fd47c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd47c101280 0x7fd47c19ac10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 --2- 172.21.15.62:0/3768854753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd47c1008d0 0x7fd47c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:20.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 >> 172.21.15.62:0/3768854753 conn(0x7fd47c0fc7a0 msgr2=0x7fd47c0feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:20.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 shutdown_connections 2024-09-15T23:14:20.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:20.835+0000 7fd4840d0640 1 -- 172.21.15.62:0/3768854753 wait complete. 2024-09-15T23:14:21.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:20 smithi148 ceph-mon[35610]: pgmap v308: 129 pgs: 129 active+clean; 605 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-15T23:14:21.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:20 smithi062 ceph-mon[30949]: pgmap v308: 129 pgs: 129 active+clean; 605 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-15T23:14:21.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:20 smithi067 ceph-mon[35665]: pgmap v308: 129 pgs: 129 active+clean; 605 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 4.4 KiB/s wr, 14 op/s 2024-09-15T23:14:21.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:14:21.779 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:21 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3768854753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:14:22.042 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:21 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3768854753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:14:22.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.244+0000 7f31e668e640 1 -- 172.21.15.62:0/3678932900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e010c170 msgr2=0x7f31e010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.244+0000 7f31e668e640 1 --2- 172.21.15.62:0/3678932900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e010c170 0x7f31e010e610 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f31d8009ca0 tx=0x7f31d802a950 comp rx=0 tx=0).stop 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.244+0000 7f31e668e640 1 -- 172.21.15.62:0/3678932900 shutdown_connections 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.244+0000 7f31e668e640 1 --2- 172.21.15.62:0/3678932900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e010c170 0x7f31e010e610 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.244+0000 7f31e668e640 1 --2- 172.21.15.62:0/3678932900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e0103660 0x7f31e010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.244+0000 7f31e668e640 1 --2- 172.21.15.62:0/3678932900 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31e0102d40 0x7f31e0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.244+0000 7f31e668e640 1 -- 172.21.15.62:0/3678932900 >> 172.21.15.62:0/3678932900 conn(0x7f31e00fc7a0 msgr2=0x7f31e00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.244+0000 7f31e668e640 1 -- 172.21.15.62:0/3678932900 shutdown_connections 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.244+0000 7f31e668e640 1 -- 172.21.15.62:0/3678932900 wait complete. 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31e668e640 1 Processor -- start 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31e668e640 1 -- start start 2024-09-15T23:14:22.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31e668e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e0102d40 0x7f31e01009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31e668e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e0103660 0x7f31e0100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31e668e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31e010c170 0x7f31e0101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31e668e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f31e0111f70 con 0x7f31e0102d40 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31e668e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f31e0111df0 con 0x7f31e0103660 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31e668e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f31e01120f0 con 0x7f31e010c170 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31dffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e0102d40 0x7f31e01009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31dffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e0102d40 0x7f31e01009a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45158/0 (socket says 172.21.15.62:45158) 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31dffff640 1 -- 172.21.15.62:0/4233961559 learned_addr learned my addr 172.21.15.62:0/4233961559 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31d7fff640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e0103660 0x7f31e0100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31e4c2b640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31e010c170 0x7f31e0101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31dffff640 1 -- 172.21.15.62:0/4233961559 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31e010c170 msgr2=0x7f31e0101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:22.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31dffff640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31e010c170 0x7f31e0101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31dffff640 1 -- 172.21.15.62:0/4233961559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e0103660 msgr2=0x7f31e0100ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31dffff640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e0103660 0x7f31e0100ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.245+0000 7f31dffff640 1 -- 172.21.15.62:0/4233961559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e01a68d0 con 0x7f31e0102d40 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.246+0000 7f31dffff640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e0102d40 0x7f31e01009a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f31d000d770 tx=0x7f31d000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.246+0000 7f31ddffb640 1 -- 172.21.15.62:0/4233961559 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d0017070 con 0x7f31e0102d40 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.246+0000 7f31ddffb640 1 -- 172.21.15.62:0/4233961559 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31d0005500 con 0x7f31e0102d40 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.246+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e01a6bc0 con 0x7f31e0102d40 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.246+0000 7f31ddffb640 1 -- 172.21.15.62:0/4233961559 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d0004e00 con 0x7f31e0102d40 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.246+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e01a7100 con 0x7f31e0102d40 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.247+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e0103120 con 0x7f31e0102d40 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.248+0000 7f31ddffb640 1 -- 172.21.15.62:0/4233961559 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f31d0004fa0 con 0x7f31e0102d40 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.249+0000 7f31ddffb640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31cc0a3350 0x7f31cc0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.249+0000 7f31ddffb640 1 -- 172.21.15.62:0/4233961559 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f31d00cf640 con 0x7f31e0102d40 2024-09-15T23:14:22.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.249+0000 7f31d7fff640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31cc0a3350 0x7f31cc0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:22.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.249+0000 7f31d7fff640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31cc0a3350 0x7f31cc0a5810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f31e0102430 tx=0x7f31c800b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:22.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.253+0000 7f31ddffb640 1 -- 172.21.15.62:0/4233961559 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f31d009b970 con 0x7f31e0102d40 2024-09-15T23:14:22.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:21 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3768854753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:14:22.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.427+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31e01a72d0 con 0x7f31e0102d40 2024-09-15T23:14:22.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.429+0000 7f31ddffb640 1 -- 172.21.15.62:0/4233961559 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f31d009fc70 con 0x7f31e0102d40 2024-09-15T23:14:22.430 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:14:22.430 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:13:42.233963+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:14:22.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31cc0a3350 msgr2=0x7f31cc0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:22.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31cc0a3350 0x7f31cc0a5810 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f31e0102430 tx=0x7f31c800b2e0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e0102d40 msgr2=0x7f31e01009a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e0102d40 0x7f31e01009a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f31d000d770 tx=0x7f31d000dc30 comp rx=0 tx=0).stop 2024-09-15T23:14:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 shutdown_connections 2024-09-15T23:14:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31cc0a3350 0x7f31cc0a5810 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f31e010c170 0x7f31e0101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31e0103660 0x7f31e0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 --2- 172.21.15.62:0/4233961559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f31e0102d40 0x7f31e01009a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 >> 172.21.15.62:0/4233961559 conn(0x7f31e00fc7a0 msgr2=0x7f31e00fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.431+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 shutdown_connections 2024-09-15T23:14:22.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:22.432+0000 7f31e668e640 1 -- 172.21.15.62:0/4233961559 wait complete. 2024-09-15T23:14:23.034 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:22 smithi062 ceph-mon[30949]: pgmap v309: 129 pgs: 129 active+clean; 623 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.0 KiB/s wr, 15 op/s 2024-09-15T23:14:23.034 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:22 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4233961559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:23.062 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-15T23:14:23.062 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:14:23.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:22 smithi067 ceph-mon[35665]: pgmap v309: 129 pgs: 129 active+clean; 623 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.0 KiB/s wr, 15 op/s 2024-09-15T23:14:23.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:22 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4233961559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:23.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:22 smithi148 ceph-mon[35610]: pgmap v309: 129 pgs: 129 active+clean; 623 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.0 KiB/s wr, 15 op/s 2024-09-15T23:14:23.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:22 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4233961559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:23.319 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:23.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.776+0000 7efdc03c7640 1 -- 172.21.15.62:0/3712389625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 msgr2=0x7efdb810e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.776+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3712389625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 0x7efdb810e600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efdb0009ca0 tx=0x7efdb002a950 comp rx=0 tx=0).stop 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.776+0000 7efdc03c7640 1 -- 172.21.15.62:0/3712389625 shutdown_connections 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.776+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3712389625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 0x7efdb810e600 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.776+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3712389625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb81036c0 0x7efdb810bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.776+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3712389625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdb8102da0 0x7efdb8103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.776+0000 7efdc03c7640 1 -- 172.21.15.62:0/3712389625 >> 172.21.15.62:0/3712389625 conn(0x7efdb80fc7a0 msgr2=0x7efdb80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.776+0000 7efdc03c7640 1 -- 172.21.15.62:0/3712389625 shutdown_connections 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.776+0000 7efdc03c7640 1 -- 172.21.15.62:0/3712389625 wait complete. 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdc03c7640 1 Processor -- start 2024-09-15T23:14:23.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdc03c7640 1 -- start start 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdc03c7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb8102da0 0x7efdb81983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdc03c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdb81036c0 0x7efdb8198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdc03c7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 0x7efdb819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdc03c7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efdb8111f40 con 0x7efdb810c180 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdc03c7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efdb8111dc0 con 0x7efdb81036c0 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdc03c7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efdb81120c0 con 0x7efdb8102da0 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdbe964640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 0x7efdb819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdbe964640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 0x7efdb819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60700/0 (socket says 172.21.15.62:60700) 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdbe964640 1 -- 172.21.15.62:0/3579384959 learned_addr learned my addr 172.21.15.62:0/3579384959 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdbe163640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb8102da0 0x7efdb81983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.777+0000 7efdbd962640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdb81036c0 0x7efdb8198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:23.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdbe964640 1 -- 172.21.15.62:0/3579384959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb8102da0 msgr2=0x7efdb81983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdbe964640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb8102da0 0x7efdb81983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdbe964640 1 -- 172.21.15.62:0/3579384959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdb81036c0 msgr2=0x7efdb8198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdbe964640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdb81036c0 0x7efdb8198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdbe964640 1 -- 172.21.15.62:0/3579384959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb819d380 con 0x7efdb810c180 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdbd962640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdb81036c0 0x7efdb8198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdbe964640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 0x7efdb819cca0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efdb002ae70 tx=0x7efdb002f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdab7fe640 1 -- 172.21.15.62:0/3579384959 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0034040 con 0x7efdb810c180 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdab7fe640 1 -- 172.21.15.62:0/3579384959 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdb0030920 con 0x7efdb810c180 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb819d610 con 0x7efdb810c180 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdab7fe640 1 -- 172.21.15.62:0/3579384959 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0045480 con 0x7efdb810c180 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.778+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb81a4f50 con 0x7efdb810c180 2024-09-15T23:14:23.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.779+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdb8103180 con 0x7efdb810c180 2024-09-15T23:14:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.780+0000 7efdab7fe640 1 -- 172.21.15.62:0/3579384959 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7efdb0037020 con 0x7efdb810c180 2024-09-15T23:14:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.781+0000 7efdab7fe640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a32d0 0x7efda40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.781+0000 7efdbe163640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a32d0 0x7efda40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.781+0000 7efdab7fe640 1 -- 172.21.15.62:0/3579384959 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7efdb00f0d80 con 0x7efdb810c180 2024-09-15T23:14:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.781+0000 7efdbe163640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a32d0 0x7efda40a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efdb4005fd0 tx=0x7efdb4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:23.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.784+0000 7efdab7fe640 1 -- 172.21.15.62:0/3579384959 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7efdb00811b0 con 0x7efdb810c180 2024-09-15T23:14:23.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.955+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efdb8107290 con 0x7efda40a32d0 2024-09-15T23:14:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.964+0000 7efdab7fe640 1 -- 172.21.15.62:0/3579384959 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340590 (secure 0 0 0) 0x7efdb400b360 con 0x7efda40a32d0 2024-09-15T23:14:23.969 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:14:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:14:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.967+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a32d0 msgr2=0x7efda40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.967+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a32d0 0x7efda40a5790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efdb4005fd0 tx=0x7efdb4005850 comp rx=0 tx=0).stop 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.967+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 msgr2=0x7efdb819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.967+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 0x7efdb819cca0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efdb002ae70 tx=0x7efdb002f5b0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.968+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 shutdown_connections 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.968+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a32d0 0x7efda40a5790 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.968+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdb810c180 0x7efdb819cca0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.968+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdb81036c0 0x7efdb8198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.968+0000 7efdc03c7640 1 --2- 172.21.15.62:0/3579384959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdb8102da0 0x7efdb81983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.968+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 >> 172.21.15.62:0/3579384959 conn(0x7efdb80fc7a0 msgr2=0x7efdb80fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.968+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 shutdown_connections 2024-09-15T23:14:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:23.968+0000 7efdc03c7640 1 -- 172.21.15.62:0/3579384959 wait complete. 2024-09-15T23:14:24.663 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-15T23:14:23.702719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637950,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":231,"num_read_kb":213,"num_write":340,"num_write_kb":3033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365140,"kb_used_data":11732,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576684,"statfs":{"total":1151940427776,"available":1151566524416,"internally_reserved":0,"allocated":12013568,"data_stored":7717533,"data_compressed":318288,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40128,"num_objects":5,"num_object_clones":0,"num_object_copies":17,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":68,"num_read_kb":74,"num_write":39,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002934"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.634446+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:14:07.634446+0000","last_peered":"2024-09-15T23:14:07.634446+0000","last_clean":"2024-09-15T23:14:07.634446+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:14:07.634446+0000","last_undegraded":"2024-09-15T23:14:07.634446+0000","last_fullsized":"2024-09-15T23:14:07.634446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.323257+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:13:43.323257+0000","last_peered":"2024-09-15T23:13:43.323257+0000","last_clean":"2024-09-15T23:13:43.323257+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:13:43.323257+0000","last_undegraded":"2024-09-15T23:13:43.323257+0000","last_fullsized":"2024-09-15T23:13:43.323257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189806+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:13:44.189806+0000","last_peered":"2024-09-15T23:13:44.189806+0000","last_clean":"2024-09-15T23:13:44.189806+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:13:44.189806+0000","last_undegraded":"2024-09-15T23:13:44.189806+0000","last_fullsized":"2024-09-15T23:13:44.189806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.323448+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:13:43.323448+0000","last_peered":"2024-09-15T23:13:43.323448+0000","last_clean":"2024-09-15T23:13:43.323448+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:13:43.323448+0000","last_undegraded":"2024-09-15T23:13:43.323448+0000","last_fullsized":"2024-09-15T23:13:43.323448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480717+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:13:43.480717+0000","last_peered":"2024-09-15T23:13:43.480717+0000","last_clean":"2024-09-15T23:13:43.480717+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:13:43.480717+0000","last_undegraded":"2024-09-15T23:13:43.480717+0000","last_fullsized":"2024-09-15T23:13:43.480717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189125+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:13:44.189125+0000","last_peered":"2024-09-15T23:13:44.189125+0000","last_clean":"2024-09-15T23:13:44.189125+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:13:44.189125+0000","last_undegraded":"2024-09-15T23:13:44.189125+0000","last_fullsized":"2024-09-15T23:13:44.189125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.634120+0000","last_change":"2024-09-15T23:13:30.491541+0000","last_active":"2024-09-15T23:14:07.634120+0000","last_peered":"2024-09-15T23:14:07.634120+0000","last_clean":"2024-09-15T23:14:07.634120+0000","last_became_active":"2024-09-15T23:13:30.491195+0000","last_became_peered":"2024-09-15T23:13:30.491195+0000","last_unstale":"2024-09-15T23:14:07.634120+0000","last_undegraded":"2024-09-15T23:14:07.634120+0000","last_fullsized":"2024-09-15T23:14:07.634120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:39.714183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408118+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:13:43.408118+0000","last_peered":"2024-09-15T23:13:43.408118+0000","last_clean":"2024-09-15T23:13:43.408118+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:13:43.408118+0000","last_undegraded":"2024-09-15T23:13:43.408118+0000","last_fullsized":"2024-09-15T23:13:43.408118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189434+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:13:44.189434+0000","last_peered":"2024-09-15T23:13:44.189434+0000","last_clean":"2024-09-15T23:13:44.189434+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:13:44.189434+0000","last_undegraded":"2024-09-15T23:13:44.189434+0000","last_fullsized":"2024-09-15T23:13:44.189434+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597884+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:13:42.597884+0000","last_peered":"2024-09-15T23:13:42.597884+0000","last_clean":"2024-09-15T23:13:42.597884+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:13:42.597884+0000","last_undegraded":"2024-09-15T23:13:42.597884+0000","last_fullsized":"2024-09-15T23:13:42.597884+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190605+0000","last_change":"2024-09-15T23:13:32.511232+0000","last_active":"2024-09-15T23:13:44.190605+0000","last_peered":"2024-09-15T23:13:44.190605+0000","last_clean":"2024-09-15T23:13:44.190605+0000","last_became_active":"2024-09-15T23:13:32.510875+0000","last_became_peered":"2024-09-15T23:13:32.510875+0000","last_unstale":"2024-09-15T23:13:44.190605+0000","last_undegraded":"2024-09-15T23:13:44.190605+0000","last_fullsized":"2024-09-15T23:13:44.190605+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:54:29.918573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189046+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:13:44.189046+0000","last_peered":"2024-09-15T23:13:44.189046+0000","last_clean":"2024-09-15T23:13:44.189046+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:13:44.189046+0000","last_undegraded":"2024-09-15T23:13:44.189046+0000","last_fullsized":"2024-09-15T23:13:44.189046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341615+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:13:43.341615+0000","last_peered":"2024-09-15T23:13:43.341615+0000","last_clean":"2024-09-15T23:13:43.341615+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:13:43.341615+0000","last_undegraded":"2024-09-15T23:13:43.341615+0000","last_fullsized":"2024-09-15T23:13:43.341615+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.627763+0000","last_change":"2024-09-15T23:13:30.490411+0000","last_active":"2024-09-15T23:14:06.627763+0000","last_peered":"2024-09-15T23:14:06.627763+0000","last_clean":"2024-09-15T23:14:06.627763+0000","last_became_active":"2024-09-15T23:13:30.490081+0000","last_became_peered":"2024-09-15T23:13:30.490081+0000","last_unstale":"2024-09-15T23:14:06.627763+0000","last_undegraded":"2024-09-15T23:14:06.627763+0000","last_fullsized":"2024-09-15T23:14:06.627763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:31:26.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189195+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:13:44.189195+0000","last_peered":"2024-09-15T23:13:44.189195+0000","last_clean":"2024-09-15T23:13:44.189195+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:13:44.189195+0000","last_undegraded":"2024-09-15T23:13:44.189195+0000","last_fullsized":"2024-09-15T23:13:44.189195+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322897+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:13:43.322897+0000","last_peered":"2024-09-15T23:13:43.322897+0000","last_clean":"2024-09-15T23:13:43.322897+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:13:43.322897+0000","last_undegraded":"2024-09-15T23:13:43.322897+0000","last_fullsized":"2024-09-15T23:13:43.322897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190031+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:13:44.190031+0000","last_peered":"2024-09-15T23:13:44.190031+0000","last_clean":"2024-09-15T23:13:44.190031+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:13:44.190031+0000","last_undegraded":"2024-09-15T23:13:44.190031+0000","last_fullsized":"2024-09-15T23:13:44.190031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189484+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:13:44.189484+0000","last_peered":"2024-09-15T23:13:44.189484+0000","last_clean":"2024-09-15T23:13:44.189484+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:13:44.189484+0000","last_undegraded":"2024-09-15T23:13:44.189484+0000","last_fullsized":"2024-09-15T23:13:44.189484+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542334+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:13:42.542334+0000","last_peered":"2024-09-15T23:13:42.542334+0000","last_clean":"2024-09-15T23:13:42.542334+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:13:42.542334+0000","last_undegraded":"2024-09-15T23:13:42.542334+0000","last_fullsized":"2024-09-15T23:13:42.542334+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.626505+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:14:06.626505+0000","last_peered":"2024-09-15T23:14:06.626505+0000","last_clean":"2024-09-15T23:14:06.626505+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:14:06.626505+0000","last_undegraded":"2024-09-15T23:14:06.626505+0000","last_fullsized":"2024-09-15T23:14:06.626505+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542121+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:13:42.542121+0000","last_peered":"2024-09-15T23:13:42.542121+0000","last_clean":"2024-09-15T23:13:42.542121+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:13:42.542121+0000","last_undegraded":"2024-09-15T23:13:42.542121+0000","last_fullsized":"2024-09-15T23:13:42.542121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597377+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:13:42.597377+0000","last_peered":"2024-09-15T23:13:42.597377+0000","last_clean":"2024-09-15T23:13:42.597377+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:13:42.597377+0000","last_undegraded":"2024-09-15T23:13:42.597377+0000","last_fullsized":"2024-09-15T23:13:42.597377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.625088+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:14:06.625088+0000","last_peered":"2024-09-15T23:14:06.625088+0000","last_clean":"2024-09-15T23:14:06.625088+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:14:06.625088+0000","last_undegraded":"2024-09-15T23:14:06.625088+0000","last_fullsized":"2024-09-15T23:14:06.625088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189391+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:13:44.189391+0000","last_peered":"2024-09-15T23:13:44.189391+0000","last_clean":"2024-09-15T23:13:44.189391+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:13:44.189391+0000","last_undegraded":"2024-09-15T23:13:44.189391+0000","last_fullsized":"2024-09-15T23:13:44.189391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480567+0000","last_change":"2024-09-15T23:13:32.512270+0000","last_active":"2024-09-15T23:13:43.480567+0000","last_peered":"2024-09-15T23:13:43.480567+0000","last_clean":"2024-09-15T23:13:43.480567+0000","last_became_active":"2024-09-15T23:13:32.511975+0000","last_became_peered":"2024-09-15T23:13:32.511975+0000","last_unstale":"2024-09-15T23:13:43.480567+0000","last_undegraded":"2024-09-15T23:13:43.480567+0000","last_fullsized":"2024-09-15T23:13:43.480567+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:32.183443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.625181+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:14:06.625181+0000","last_peered":"2024-09-15T23:14:06.625181+0000","last_clean":"2024-09-15T23:14:06.625181+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:14:06.625181+0000","last_undegraded":"2024-09-15T23:14:06.625181+0000","last_fullsized":"2024-09-15T23:14:06.625181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189270+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:13:44.189270+0000","last_peered":"2024-09-15T23:13:44.189270+0000","last_clean":"2024-09-15T23:13:44.189270+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:13:44.189270+0000","last_undegraded":"2024-09-15T23:13:44.189270+0000","last_fullsized":"2024-09-15T23:13:44.189270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.323360+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:13:43.323360+0000","last_peered":"2024-09-15T23:13:43.323360+0000","last_clean":"2024-09-15T23:13:43.323360+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:13:43.323360+0000","last_undegraded":"2024-09-15T23:13:43.323360+0000","last_fullsized":"2024-09-15T23:13:43.323360+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:21.519105+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:14:21.519105+0000","last_peered":"2024-09-15T23:14:21.519105+0000","last_clean":"2024-09-15T23:14:21.519105+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:14:21.519105+0000","last_undegraded":"2024-09-15T23:14:21.519105+0000","last_fullsized":"2024-09-15T23:14:21.519105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322661+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:13:43.322661+0000","last_peered":"2024-09-15T23:13:43.322661+0000","last_clean":"2024-09-15T23:13:43.322661+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:13:43.322661+0000","last_undegraded":"2024-09-15T23:13:43.322661+0000","last_fullsized":"2024-09-15T23:13:43.322661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189262+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:13:44.189262+0000","last_peered":"2024-09-15T23:13:44.189262+0000","last_clean":"2024-09-15T23:13:44.189262+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:13:44.189262+0000","last_undegraded":"2024-09-15T23:13:44.189262+0000","last_fullsized":"2024-09-15T23:13:44.189262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.625884+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:14:06.625884+0000","last_peered":"2024-09-15T23:14:06.625884+0000","last_clean":"2024-09-15T23:14:06.625884+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:14:06.625884+0000","last_undegraded":"2024-09-15T23:14:06.625884+0000","last_fullsized":"2024-09-15T23:14:06.625884+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.543078+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:13:42.543078+0000","last_peered":"2024-09-15T23:13:42.543078+0000","last_clean":"2024-09-15T23:13:42.543078+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:13:42.543078+0000","last_undegraded":"2024-09-15T23:13:42.543078+0000","last_fullsized":"2024-09-15T23:13:42.543078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542747+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:13:42.542747+0000","last_peered":"2024-09-15T23:13:42.542747+0000","last_clean":"2024-09-15T23:13:42.542747+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:13:42.542747+0000","last_undegraded":"2024-09-15T23:13:42.542747+0000","last_fullsized":"2024-09-15T23:13:42.542747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542321+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:13:42.542321+0000","last_peered":"2024-09-15T23:13:42.542321+0000","last_clean":"2024-09-15T23:13:42.542321+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:13:42.542321+0000","last_undegraded":"2024-09-15T23:13:42.542321+0000","last_fullsized":"2024-09-15T23:13:42.542321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597514+0000","last_change":"2024-09-15T23:13:39.748083+0000","last_active":"2024-09-15T23:13:42.597514+0000","last_peered":"2024-09-15T23:13:42.597514+0000","last_clean":"2024-09-15T23:13:42.597514+0000","last_became_active":"2024-09-15T23:13:39.747584+0000","last_became_peered":"2024-09-15T23:13:39.747584+0000","last_unstale":"2024-09-15T23:13:42.597514+0000","last_undegraded":"2024-09-15T23:13:42.597514+0000","last_fullsized":"2024-09-15T23:13:42.597514+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:35:53.649800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542708+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:13:42.542708+0000","last_peered":"2024-09-15T23:13:42.542708+0000","last_clean":"2024-09-15T23:13:42.542708+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:13:42.542708+0000","last_undegraded":"2024-09-15T23:13:42.542708+0000","last_fullsized":"2024-09-15T23:13:42.542708+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341797+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:13:43.341797+0000","last_peered":"2024-09-15T23:13:43.341797+0000","last_clean":"2024-09-15T23:13:43.341797+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:13:43.341797+0000","last_undegraded":"2024-09-15T23:13:43.341797+0000","last_fullsized":"2024-09-15T23:13:43.341797+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189139+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:13:44.189139+0000","last_peered":"2024-09-15T23:13:44.189139+0000","last_clean":"2024-09-15T23:13:44.189139+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:13:44.189139+0000","last_undegraded":"2024-09-15T23:13:44.189139+0000","last_fullsized":"2024-09-15T23:13:44.189139+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189503+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:13:44.189503+0000","last_peered":"2024-09-15T23:13:44.189503+0000","last_clean":"2024-09-15T23:13:44.189503+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:13:44.189503+0000","last_undegraded":"2024-09-15T23:13:44.189503+0000","last_fullsized":"2024-09-15T23:13:44.189503+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:18.503504+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:14:18.503504+0000","last_peered":"2024-09-15T23:14:18.503504+0000","last_clean":"2024-09-15T23:14:18.503504+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:14:18.503504+0000","last_undegraded":"2024-09-15T23:14:18.503504+0000","last_fullsized":"2024-09-15T23:14:18.503504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189333+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:13:44.189333+0000","last_peered":"2024-09-15T23:13:44.189333+0000","last_clean":"2024-09-15T23:13:44.189333+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:13:44.189333+0000","last_undegraded":"2024-09-15T23:13:44.189333+0000","last_fullsized":"2024-09-15T23:13:44.189333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.188705+0000","last_change":"2024-09-15T23:13:32.513091+0000","last_active":"2024-09-15T23:13:44.188705+0000","last_peered":"2024-09-15T23:13:44.188705+0000","last_clean":"2024-09-15T23:13:44.188705+0000","last_became_active":"2024-09-15T23:13:32.512752+0000","last_became_peered":"2024-09-15T23:13:32.512752+0000","last_unstale":"2024-09-15T23:13:44.188705+0000","last_undegraded":"2024-09-15T23:13:44.188705+0000","last_fullsized":"2024-09-15T23:13:44.188705+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:48:58.131659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.633384+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:14:07.633384+0000","last_peered":"2024-09-15T23:14:07.633384+0000","last_clean":"2024-09-15T23:14:07.633384+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:14:07.633384+0000","last_undegraded":"2024-09-15T23:14:07.633384+0000","last_fullsized":"2024-09-15T23:14:07.633384+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189986+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:13:44.189986+0000","last_peered":"2024-09-15T23:13:44.189986+0000","last_clean":"2024-09-15T23:13:44.189986+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:13:44.189986+0000","last_undegraded":"2024-09-15T23:13:44.189986+0000","last_fullsized":"2024-09-15T23:13:44.189986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322913+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:13:43.322913+0000","last_peered":"2024-09-15T23:13:43.322913+0000","last_clean":"2024-09-15T23:13:43.322913+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:13:43.322913+0000","last_undegraded":"2024-09-15T23:13:43.322913+0000","last_fullsized":"2024-09-15T23:13:43.322913+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:08.484346+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:14:08.484346+0000","last_peered":"2024-09-15T23:14:08.484346+0000","last_clean":"2024-09-15T23:14:08.484346+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:14:08.484346+0000","last_undegraded":"2024-09-15T23:14:08.484346+0000","last_fullsized":"2024-09-15T23:14:08.484346+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":344,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341618+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:13:43.341618+0000","last_peered":"2024-09-15T23:13:43.341618+0000","last_clean":"2024-09-15T23:13:43.341618+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:13:43.341618+0000","last_undegraded":"2024-09-15T23:13:43.341618+0000","last_fullsized":"2024-09-15T23:13:43.341618+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.342025+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:13:43.342025+0000","last_peered":"2024-09-15T23:13:43.342025+0000","last_clean":"2024-09-15T23:13:43.342025+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:13:43.342025+0000","last_undegraded":"2024-09-15T23:13:43.342025+0000","last_fullsized":"2024-09-15T23:13:43.342025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.342085+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:13:43.342085+0000","last_peered":"2024-09-15T23:13:43.342085+0000","last_clean":"2024-09-15T23:13:43.342085+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:13:43.342085+0000","last_undegraded":"2024-09-15T23:13:43.342085+0000","last_fullsized":"2024-09-15T23:13:43.342085+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.634874+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:14:07.634874+0000","last_peered":"2024-09-15T23:14:07.634874+0000","last_clean":"2024-09-15T23:14:07.634874+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:14:07.634874+0000","last_undegraded":"2024-09-15T23:14:07.634874+0000","last_fullsized":"2024-09-15T23:14:07.634874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.407983+0000","last_change":"2024-09-15T23:13:39.545839+0000","last_active":"2024-09-15T23:13:43.407983+0000","last_peered":"2024-09-15T23:13:43.407983+0000","last_clean":"2024-09-15T23:13:43.407983+0000","last_became_active":"2024-09-15T23:13:39.545374+0000","last_became_peered":"2024-09-15T23:13:39.545374+0000","last_unstale":"2024-09-15T23:13:43.407983+0000","last_undegraded":"2024-09-15T23:13:43.407983+0000","last_fullsized":"2024-09-15T23:13:43.407983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:03.634809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341846+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:13:43.341846+0000","last_peered":"2024-09-15T23:13:43.341846+0000","last_clean":"2024-09-15T23:13:43.341846+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:13:43.341846+0000","last_undegraded":"2024-09-15T23:13:43.341846+0000","last_fullsized":"2024-09-15T23:13:43.341846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190884+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:13:44.190884+0000","last_peered":"2024-09-15T23:13:44.190884+0000","last_clean":"2024-09-15T23:13:44.190884+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:13:44.190884+0000","last_undegraded":"2024-09-15T23:13:44.190884+0000","last_fullsized":"2024-09-15T23:13:44.190884+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480267+0000","last_change":"2024-09-15T23:13:39.544890+0000","last_active":"2024-09-15T23:13:43.480267+0000","last_peered":"2024-09-15T23:13:43.480267+0000","last_clean":"2024-09-15T23:13:43.480267+0000","last_became_active":"2024-09-15T23:13:39.544481+0000","last_became_peered":"2024-09-15T23:13:39.544481+0000","last_unstale":"2024-09-15T23:13:43.480267+0000","last_undegraded":"2024-09-15T23:13:43.480267+0000","last_fullsized":"2024-09-15T23:13:43.480267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:47.961187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.342011+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:13:43.342011+0000","last_peered":"2024-09-15T23:13:43.342011+0000","last_clean":"2024-09-15T23:13:43.342011+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:13:43.342011+0000","last_undegraded":"2024-09-15T23:13:43.342011+0000","last_fullsized":"2024-09-15T23:13:43.342011+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189244+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:13:44.189244+0000","last_peered":"2024-09-15T23:13:44.189244+0000","last_clean":"2024-09-15T23:13:44.189244+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:13:44.189244+0000","last_undegraded":"2024-09-15T23:13:44.189244+0000","last_fullsized":"2024-09-15T23:13:44.189244+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189331+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:13:44.189331+0000","last_peered":"2024-09-15T23:13:44.189331+0000","last_clean":"2024-09-15T23:13:44.189331+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:13:44.189331+0000","last_undegraded":"2024-09-15T23:13:44.189331+0000","last_fullsized":"2024-09-15T23:13:44.189331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542736+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:13:42.542736+0000","last_peered":"2024-09-15T23:13:42.542736+0000","last_clean":"2024-09-15T23:13:42.542736+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:13:42.542736+0000","last_undegraded":"2024-09-15T23:13:42.542736+0000","last_fullsized":"2024-09-15T23:13:42.542736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:08.485838+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:14:08.485838+0000","last_peered":"2024-09-15T23:14:08.485838+0000","last_clean":"2024-09-15T23:14:08.485838+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:14:08.485838+0000","last_undegraded":"2024-09-15T23:14:08.485838+0000","last_fullsized":"2024-09-15T23:14:08.485838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189429+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:13:44.189429+0000","last_peered":"2024-09-15T23:13:44.189429+0000","last_clean":"2024-09-15T23:13:44.189429+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:13:44.189429+0000","last_undegraded":"2024-09-15T23:13:44.189429+0000","last_fullsized":"2024-09-15T23:13:44.189429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190324+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:13:44.190324+0000","last_peered":"2024-09-15T23:13:44.190324+0000","last_clean":"2024-09-15T23:13:44.190324+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:13:44.190324+0000","last_undegraded":"2024-09-15T23:13:44.190324+0000","last_fullsized":"2024-09-15T23:13:44.190324+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.635176+0000","last_change":"2024-09-15T23:13:30.597432+0000","last_active":"2024-09-15T23:14:07.635176+0000","last_peered":"2024-09-15T23:14:07.635176+0000","last_clean":"2024-09-15T23:14:07.635176+0000","last_became_active":"2024-09-15T23:13:30.597086+0000","last_became_peered":"2024-09-15T23:13:30.597086+0000","last_unstale":"2024-09-15T23:14:07.635176+0000","last_undegraded":"2024-09-15T23:14:07.635176+0000","last_fullsized":"2024-09-15T23:14:07.635176+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:02.616381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189286+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:13:44.189286+0000","last_peered":"2024-09-15T23:13:44.189286+0000","last_clean":"2024-09-15T23:13:44.189286+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:13:44.189286+0000","last_undegraded":"2024-09-15T23:13:44.189286+0000","last_fullsized":"2024-09-15T23:13:44.189286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341368+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:13:43.341368+0000","last_peered":"2024-09-15T23:13:43.341368+0000","last_clean":"2024-09-15T23:13:43.341368+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:13:43.341368+0000","last_undegraded":"2024-09-15T23:13:43.341368+0000","last_fullsized":"2024-09-15T23:13:43.341368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:19.505332+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:14:19.505332+0000","last_peered":"2024-09-15T23:14:19.505332+0000","last_clean":"2024-09-15T23:14:19.505332+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:14:19.505332+0000","last_undegraded":"2024-09-15T23:14:19.505332+0000","last_fullsized":"2024-09-15T23:14:19.505332+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542313+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:13:42.542313+0000","last_peered":"2024-09-15T23:13:42.542313+0000","last_clean":"2024-09-15T23:13:42.542313+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:13:42.542313+0000","last_undegraded":"2024-09-15T23:13:42.542313+0000","last_fullsized":"2024-09-15T23:13:42.542313+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341739+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:13:43.341739+0000","last_peered":"2024-09-15T23:13:43.341739+0000","last_clean":"2024-09-15T23:13:43.341739+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:13:43.341739+0000","last_undegraded":"2024-09-15T23:13:43.341739+0000","last_fullsized":"2024-09-15T23:13:43.341739+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.627366+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:14:06.627366+0000","last_peered":"2024-09-15T23:14:06.627366+0000","last_clean":"2024-09-15T23:14:06.627366+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:14:06.627366+0000","last_undegraded":"2024-09-15T23:14:06.627366+0000","last_fullsized":"2024-09-15T23:14:06.627366+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542489+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:13:42.542489+0000","last_peered":"2024-09-15T23:13:42.542489+0000","last_clean":"2024-09-15T23:13:42.542489+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:13:42.542489+0000","last_undegraded":"2024-09-15T23:13:42.542489+0000","last_fullsized":"2024-09-15T23:13:42.542489+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.188909+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:13:44.188909+0000","last_peered":"2024-09-15T23:13:44.188909+0000","last_clean":"2024-09-15T23:13:44.188909+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:13:44.188909+0000","last_undegraded":"2024-09-15T23:13:44.188909+0000","last_fullsized":"2024-09-15T23:13:44.188909+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.634639+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:14:07.634639+0000","last_peered":"2024-09-15T23:14:07.634639+0000","last_clean":"2024-09-15T23:14:07.634639+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:14:07.634639+0000","last_undegraded":"2024-09-15T23:14:07.634639+0000","last_fullsized":"2024-09-15T23:14:07.634639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190624+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:13:44.190624+0000","last_peered":"2024-09-15T23:13:44.190624+0000","last_clean":"2024-09-15T23:13:44.190624+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:13:44.190624+0000","last_undegraded":"2024-09-15T23:13:44.190624+0000","last_fullsized":"2024-09-15T23:13:44.190624+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597800+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:13:42.597800+0000","last_peered":"2024-09-15T23:13:42.597800+0000","last_clean":"2024-09-15T23:13:42.597800+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:13:42.597800+0000","last_undegraded":"2024-09-15T23:13:42.597800+0000","last_fullsized":"2024-09-15T23:13:42.597800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'25","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:21.508517+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:14:21.508517+0000","last_peered":"2024-09-15T23:14:21.508517+0000","last_clean":"2024-09-15T23:14:21.508517+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:14:21.508517+0000","last_undegraded":"2024-09-15T23:14:21.508517+0000","last_fullsized":"2024-09-15T23:14:21.508517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43345,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":59,"num_write":26,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542140+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:13:42.542140+0000","last_peered":"2024-09-15T23:13:42.542140+0000","last_clean":"2024-09-15T23:13:42.542140+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:13:42.542140+0000","last_undegraded":"2024-09-15T23:13:42.542140+0000","last_fullsized":"2024-09-15T23:13:42.542140+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408125+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:13:43.408125+0000","last_peered":"2024-09-15T23:13:43.408125+0000","last_clean":"2024-09-15T23:13:43.408125+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:13:43.408125+0000","last_undegraded":"2024-09-15T23:13:43.408125+0000","last_fullsized":"2024-09-15T23:13:43.408125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.624091+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:14:06.624091+0000","last_peered":"2024-09-15T23:14:06.624091+0000","last_clean":"2024-09-15T23:14:06.624091+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:14:06.624091+0000","last_undegraded":"2024-09-15T23:14:06.624091+0000","last_fullsized":"2024-09-15T23:14:06.624091+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189226+0000","last_change":"2024-09-15T23:13:39.554205+0000","last_active":"2024-09-15T23:13:44.189226+0000","last_peered":"2024-09-15T23:13:44.189226+0000","last_clean":"2024-09-15T23:13:44.189226+0000","last_became_active":"2024-09-15T23:13:39.553784+0000","last_became_peered":"2024-09-15T23:13:39.553784+0000","last_unstale":"2024-09-15T23:13:44.189226+0000","last_undegraded":"2024-09-15T23:13:44.189226+0000","last_fullsized":"2024-09-15T23:13:44.189226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:08.325422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480699+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:13:43.480699+0000","last_peered":"2024-09-15T23:13:43.480699+0000","last_clean":"2024-09-15T23:13:43.480699+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:13:43.480699+0000","last_undegraded":"2024-09-15T23:13:43.480699+0000","last_fullsized":"2024-09-15T23:13:43.480699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.627042+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:14:06.627042+0000","last_peered":"2024-09-15T23:14:06.627042+0000","last_clean":"2024-09-15T23:14:06.627042+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:14:06.627042+0000","last_undegraded":"2024-09-15T23:14:06.627042+0000","last_fullsized":"2024-09-15T23:14:06.627042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.541706+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:13:42.541706+0000","last_peered":"2024-09-15T23:13:42.541706+0000","last_clean":"2024-09-15T23:13:42.541706+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:13:42.541706+0000","last_undegraded":"2024-09-15T23:13:42.541706+0000","last_fullsized":"2024-09-15T23:13:42.541706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408349+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:13:43.408349+0000","last_peered":"2024-09-15T23:13:43.408349+0000","last_clean":"2024-09-15T23:13:43.408349+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:13:43.408349+0000","last_undegraded":"2024-09-15T23:13:43.408349+0000","last_fullsized":"2024-09-15T23:13:43.408349+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:20.505465+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:14:20.505465+0000","last_peered":"2024-09-15T23:14:20.505465+0000","last_clean":"2024-09-15T23:14:20.505465+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:14:20.505465+0000","last_undegraded":"2024-09-15T23:14:20.505465+0000","last_fullsized":"2024-09-15T23:14:20.505465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322877+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:13:43.322877+0000","last_peered":"2024-09-15T23:13:43.322877+0000","last_clean":"2024-09-15T23:13:43.322877+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:13:43.322877+0000","last_undegraded":"2024-09-15T23:13:43.322877+0000","last_fullsized":"2024-09-15T23:13:43.322877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480582+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:13:43.480582+0000","last_peered":"2024-09-15T23:13:43.480582+0000","last_clean":"2024-09-15T23:13:43.480582+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:13:43.480582+0000","last_undegraded":"2024-09-15T23:13:43.480582+0000","last_fullsized":"2024-09-15T23:13:43.480582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.541717+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:13:42.541717+0000","last_peered":"2024-09-15T23:13:42.541717+0000","last_clean":"2024-09-15T23:13:42.541717+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:13:42.541717+0000","last_undegraded":"2024-09-15T23:13:42.541717+0000","last_fullsized":"2024-09-15T23:13:42.541717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189732+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:13:44.189732+0000","last_peered":"2024-09-15T23:13:44.189732+0000","last_clean":"2024-09-15T23:13:44.189732+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:13:44.189732+0000","last_undegraded":"2024-09-15T23:13:44.189732+0000","last_fullsized":"2024-09-15T23:13:44.189732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542765+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:13:42.542765+0000","last_peered":"2024-09-15T23:13:42.542765+0000","last_clean":"2024-09-15T23:13:42.542765+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:13:42.542765+0000","last_undegraded":"2024-09-15T23:13:42.542765+0000","last_fullsized":"2024-09-15T23:13:42.542765+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322766+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:13:43.322766+0000","last_peered":"2024-09-15T23:13:43.322766+0000","last_clean":"2024-09-15T23:13:43.322766+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:13:43.322766+0000","last_undegraded":"2024-09-15T23:13:43.322766+0000","last_fullsized":"2024-09-15T23:13:43.322766+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.633777+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:14:07.633777+0000","last_peered":"2024-09-15T23:14:07.633777+0000","last_clean":"2024-09-15T23:14:07.633777+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:14:07.633777+0000","last_undegraded":"2024-09-15T23:14:07.633777+0000","last_fullsized":"2024-09-15T23:14:07.633777+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.633253+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:14:07.633253+0000","last_peered":"2024-09-15T23:14:07.633253+0000","last_clean":"2024-09-15T23:14:07.633253+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:14:07.633253+0000","last_undegraded":"2024-09-15T23:14:07.633253+0000","last_fullsized":"2024-09-15T23:14:07.633253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190475+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:13:44.190475+0000","last_peered":"2024-09-15T23:13:44.190475+0000","last_clean":"2024-09-15T23:13:44.190475+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:13:44.190475+0000","last_undegraded":"2024-09-15T23:13:44.190475+0000","last_fullsized":"2024-09-15T23:13:44.190475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189527+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:13:44.189527+0000","last_peered":"2024-09-15T23:13:44.189527+0000","last_clean":"2024-09-15T23:13:44.189527+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:13:44.189527+0000","last_undegraded":"2024-09-15T23:13:44.189527+0000","last_fullsized":"2024-09-15T23:13:44.189527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":52,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:19.504802+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:14:19.504802+0000","last_peered":"2024-09-15T23:14:19.504802+0000","last_clean":"2024-09-15T23:14:19.504802+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:14:19.504802+0000","last_undegraded":"2024-09-15T23:14:19.504802+0000","last_fullsized":"2024-09-15T23:14:19.504802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597489+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:13:42.597489+0000","last_peered":"2024-09-15T23:13:42.597489+0000","last_clean":"2024-09-15T23:13:42.597489+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:13:42.597489+0000","last_undegraded":"2024-09-15T23:13:42.597489+0000","last_fullsized":"2024-09-15T23:13:42.597489+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190108+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:13:44.190108+0000","last_peered":"2024-09-15T23:13:44.190108+0000","last_clean":"2024-09-15T23:13:44.190108+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:13:44.190108+0000","last_undegraded":"2024-09-15T23:13:44.190108+0000","last_fullsized":"2024-09-15T23:13:44.190108+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542249+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:13:42.542249+0000","last_peered":"2024-09-15T23:13:42.542249+0000","last_clean":"2024-09-15T23:13:42.542249+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:13:42.542249+0000","last_undegraded":"2024-09-15T23:13:42.542249+0000","last_fullsized":"2024-09-15T23:13:42.542249+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189645+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:13:44.189645+0000","last_peered":"2024-09-15T23:13:44.189645+0000","last_clean":"2024-09-15T23:13:44.189645+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:13:44.189645+0000","last_undegraded":"2024-09-15T23:13:44.189645+0000","last_fullsized":"2024-09-15T23:13:44.189645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189399+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:13:44.189399+0000","last_peered":"2024-09-15T23:13:44.189399+0000","last_clean":"2024-09-15T23:13:44.189399+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:13:44.189399+0000","last_undegraded":"2024-09-15T23:13:44.189399+0000","last_fullsized":"2024-09-15T23:13:44.189399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480240+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:13:43.480240+0000","last_peered":"2024-09-15T23:13:43.480240+0000","last_clean":"2024-09-15T23:13:43.480240+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:13:43.480240+0000","last_undegraded":"2024-09-15T23:13:43.480240+0000","last_fullsized":"2024-09-15T23:13:43.480240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189186+0000","last_change":"2024-09-15T23:13:32.508145+0000","last_active":"2024-09-15T23:13:44.189186+0000","last_peered":"2024-09-15T23:13:44.189186+0000","last_clean":"2024-09-15T23:13:44.189186+0000","last_became_active":"2024-09-15T23:13:32.507776+0000","last_became_peered":"2024-09-15T23:13:32.507776+0000","last_unstale":"2024-09-15T23:13:44.189186+0000","last_undegraded":"2024-09-15T23:13:44.189186+0000","last_fullsized":"2024-09-15T23:13:44.189186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:03:59.287209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542723+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:13:42.542723+0000","last_peered":"2024-09-15T23:13:42.542723+0000","last_clean":"2024-09-15T23:13:42.542723+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:13:42.542723+0000","last_undegraded":"2024-09-15T23:13:42.542723+0000","last_fullsized":"2024-09-15T23:13:42.542723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189910+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:13:44.189910+0000","last_peered":"2024-09-15T23:13:44.189910+0000","last_clean":"2024-09-15T23:13:44.189910+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:13:44.189910+0000","last_undegraded":"2024-09-15T23:13:44.189910+0000","last_fullsized":"2024-09-15T23:13:44.189910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.323355+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:13:43.323355+0000","last_peered":"2024-09-15T23:13:43.323355+0000","last_clean":"2024-09-15T23:13:43.323355+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:13:43.323355+0000","last_undegraded":"2024-09-15T23:13:43.323355+0000","last_fullsized":"2024-09-15T23:13:43.323355+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189398+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:13:44.189398+0000","last_peered":"2024-09-15T23:13:44.189398+0000","last_clean":"2024-09-15T23:13:44.189398+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:13:44.189398+0000","last_undegraded":"2024-09-15T23:13:44.189398+0000","last_fullsized":"2024-09-15T23:13:44.189398+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322992+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:13:43.322992+0000","last_peered":"2024-09-15T23:13:43.322992+0000","last_clean":"2024-09-15T23:13:43.322992+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:13:43.322992+0000","last_undegraded":"2024-09-15T23:13:43.322992+0000","last_fullsized":"2024-09-15T23:13:43.322992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408109+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:13:43.408109+0000","last_peered":"2024-09-15T23:13:43.408109+0000","last_clean":"2024-09-15T23:13:43.408109+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:13:43.408109+0000","last_undegraded":"2024-09-15T23:13:43.408109+0000","last_fullsized":"2024-09-15T23:13:43.408109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189014+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:13:44.189014+0000","last_peered":"2024-09-15T23:13:44.189014+0000","last_clean":"2024-09-15T23:13:44.189014+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:13:44.189014+0000","last_undegraded":"2024-09-15T23:13:44.189014+0000","last_fullsized":"2024-09-15T23:13:44.189014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542497+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:13:42.542497+0000","last_peered":"2024-09-15T23:13:42.542497+0000","last_clean":"2024-09-15T23:13:42.542497+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:13:42.542497+0000","last_undegraded":"2024-09-15T23:13:42.542497+0000","last_fullsized":"2024-09-15T23:13:42.542497+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190102+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:13:44.190102+0000","last_peered":"2024-09-15T23:13:44.190102+0000","last_clean":"2024-09-15T23:13:44.190102+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:13:44.190102+0000","last_undegraded":"2024-09-15T23:13:44.190102+0000","last_fullsized":"2024-09-15T23:13:44.190102+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542224+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:13:42.542224+0000","last_peered":"2024-09-15T23:13:42.542224+0000","last_clean":"2024-09-15T23:13:42.542224+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:13:42.542224+0000","last_undegraded":"2024-09-15T23:13:42.542224+0000","last_fullsized":"2024-09-15T23:13:42.542224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542906+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:13:42.542906+0000","last_peered":"2024-09-15T23:13:42.542906+0000","last_clean":"2024-09-15T23:13:42.542906+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:13:42.542906+0000","last_undegraded":"2024-09-15T23:13:42.542906+0000","last_fullsized":"2024-09-15T23:13:42.542906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.407738+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:13:43.407738+0000","last_peered":"2024-09-15T23:13:43.407738+0000","last_clean":"2024-09-15T23:13:43.407738+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:13:43.407738+0000","last_undegraded":"2024-09-15T23:13:43.407738+0000","last_fullsized":"2024-09-15T23:13:43.407738+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.479985+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:13:43.479985+0000","last_peered":"2024-09-15T23:13:43.479985+0000","last_clean":"2024-09-15T23:13:43.479985+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:13:43.479985+0000","last_undegraded":"2024-09-15T23:13:43.479985+0000","last_fullsized":"2024-09-15T23:13:43.479985+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480136+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:13:43.480136+0000","last_peered":"2024-09-15T23:13:43.480136+0000","last_clean":"2024-09-15T23:13:43.480136+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:13:43.480136+0000","last_undegraded":"2024-09-15T23:13:43.480136+0000","last_fullsized":"2024-09-15T23:13:43.480136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.188976+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:13:44.188976+0000","last_peered":"2024-09-15T23:13:44.188976+0000","last_clean":"2024-09-15T23:13:44.188976+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:13:44.188976+0000","last_undegraded":"2024-09-15T23:13:44.188976+0000","last_fullsized":"2024-09-15T23:13:44.188976+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189560+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:13:44.189560+0000","last_peered":"2024-09-15T23:13:44.189560+0000","last_clean":"2024-09-15T23:13:44.189560+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:13:44.189560+0000","last_undegraded":"2024-09-15T23:13:44.189560+0000","last_fullsized":"2024-09-15T23:13:44.189560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.342023+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:13:43.342023+0000","last_peered":"2024-09-15T23:13:43.342023+0000","last_clean":"2024-09-15T23:13:43.342023+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:13:43.342023+0000","last_undegraded":"2024-09-15T23:13:43.342023+0000","last_fullsized":"2024-09-15T23:13:43.342023+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.543117+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:13:42.543117+0000","last_peered":"2024-09-15T23:13:42.543117+0000","last_clean":"2024-09-15T23:13:42.543117+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:13:42.543117+0000","last_undegraded":"2024-09-15T23:13:42.543117+0000","last_fullsized":"2024-09-15T23:13:42.543117+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408467+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:13:43.408467+0000","last_peered":"2024-09-15T23:13:43.408467+0000","last_clean":"2024-09-15T23:13:43.408467+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:13:43.408467+0000","last_undegraded":"2024-09-15T23:13:43.408467+0000","last_fullsized":"2024-09-15T23:13:43.408467+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322760+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:13:43.322760+0000","last_peered":"2024-09-15T23:13:43.322760+0000","last_clean":"2024-09-15T23:13:43.322760+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:13:43.322760+0000","last_undegraded":"2024-09-15T23:13:43.322760+0000","last_fullsized":"2024-09-15T23:13:43.322760+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341479+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:13:43.341479+0000","last_peered":"2024-09-15T23:13:43.341479+0000","last_clean":"2024-09-15T23:13:43.341479+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:13:43.341479+0000","last_undegraded":"2024-09-15T23:13:43.341479+0000","last_fullsized":"2024-09-15T23:13:43.341479+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542582+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:13:42.542582+0000","last_peered":"2024-09-15T23:13:42.542582+0000","last_clean":"2024-09-15T23:13:42.542582+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:13:42.542582+0000","last_undegraded":"2024-09-15T23:13:42.542582+0000","last_fullsized":"2024-09-15T23:13:42.542582+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597455+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:13:42.597455+0000","last_peered":"2024-09-15T23:13:42.597455+0000","last_clean":"2024-09-15T23:13:42.597455+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:13:42.597455+0000","last_undegraded":"2024-09-15T23:13:42.597455+0000","last_fullsized":"2024-09-15T23:13:42.597455+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189521+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:13:44.189521+0000","last_peered":"2024-09-15T23:13:44.189521+0000","last_clean":"2024-09-15T23:13:44.189521+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:13:44.189521+0000","last_undegraded":"2024-09-15T23:13:44.189521+0000","last_fullsized":"2024-09-15T23:13:44.189521+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480335+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:13:43.480335+0000","last_peered":"2024-09-15T23:13:43.480335+0000","last_clean":"2024-09-15T23:13:43.480335+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:13:43.480335+0000","last_undegraded":"2024-09-15T23:13:43.480335+0000","last_fullsized":"2024-09-15T23:13:43.480335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480510+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:13:43.480510+0000","last_peered":"2024-09-15T23:13:43.480510+0000","last_clean":"2024-09-15T23:13:43.480510+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:13:43.480510+0000","last_undegraded":"2024-09-15T23:13:43.480510+0000","last_fullsized":"2024-09-15T23:13:43.480510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542851+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:13:42.542851+0000","last_peered":"2024-09-15T23:13:42.542851+0000","last_clean":"2024-09-15T23:13:42.542851+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:13:42.542851+0000","last_undegraded":"2024-09-15T23:13:42.542851+0000","last_fullsized":"2024-09-15T23:13:42.542851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":47348,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":105,"num_read_kb":104,"num_write":157,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":155377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514519,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":484535,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743451,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":1115148,"data_compressed":53347,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906977,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":488867,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135910,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":484691,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364842,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":1080602,"data_compressed":53347,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528368,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":479947,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790005,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":1076190,"data_compressed":53347,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953530,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":484597,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":480141,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":538278,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":523202,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":481335,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":43345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":58421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":44923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:14:24.669 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:14:24.930 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:24.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:24 smithi062 ceph-mon[30949]: pgmap v310: 129 pgs: 129 active+clean; 623 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2024-09-15T23:14:24.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:24 smithi062 ceph-mon[30949]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:14:25.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:24 smithi067 ceph-mon[35665]: pgmap v310: 129 pgs: 129 active+clean; 623 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2024-09-15T23:14:25.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:24 smithi067 ceph-mon[35665]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:14:25.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:24 smithi148 ceph-mon[35610]: pgmap v310: 129 pgs: 129 active+clean; 623 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 4.3 KiB/s wr, 8 op/s 2024-09-15T23:14:25.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:24 smithi148 ceph-mon[35610]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:14:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 -- 172.21.15.62:0/4008669274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470102790 msgr2=0x7fa4701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 --2- 172.21.15.62:0/4008669274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470102790 0x7fa4701067e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009900 tx=0x7fa45c02a410 comp rx=0 tx=0).stop 2024-09-15T23:14:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 -- 172.21.15.62:0/4008669274 shutdown_connections 2024-09-15T23:14:25.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 --2- 172.21.15.62:0/4008669274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa470106f10 0x7fa47010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 --2- 172.21.15.62:0/4008669274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470102790 0x7fa4701067e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 --2- 172.21.15.62:0/4008669274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa470101de0 0x7fa4701021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 -- 172.21.15.62:0/4008669274 >> 172.21.15.62:0/4008669274 conn(0x7fa4700fd8c0 msgr2=0x7fa4700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 -- 172.21.15.62:0/4008669274 shutdown_connections 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 -- 172.21.15.62:0/4008669274 wait complete. 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 Processor -- start 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.416+0000 7fa476c4a640 1 -- start start 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa476c4a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa470101de0 0x7fa470198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa476c4a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa470102790 0x7fa470198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa476c4a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470106f10 0x7fa47019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa476c4a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa470111f70 con 0x7fa470106f10 2024-09-15T23:14:25.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa476c4a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa470111df0 con 0x7fa470102790 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa476c4a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa4701120f0 con 0x7fa470101de0 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4751e7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470106f10 0x7fa47019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4751e7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470106f10 0x7fa47019cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60732/0 (socket says 172.21.15.62:60732) 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4751e7640 1 -- 172.21.15.62:0/567826422 learned_addr learned my addr 172.21.15.62:0/567826422 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4749e6640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa470101de0 0x7fa470198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa46ffff640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa470102790 0x7fa470198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4751e7640 1 -- 172.21.15.62:0/567826422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa470101de0 msgr2=0x7fa470198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4751e7640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa470101de0 0x7fa470198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4751e7640 1 -- 172.21.15.62:0/567826422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa470102790 msgr2=0x7fa470198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4751e7640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa470102790 0x7fa470198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4751e7640 1 -- 172.21.15.62:0/567826422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47019d4c0 con 0x7fa470106f10 2024-09-15T23:14:25.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa4751e7640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470106f10 0x7fa47019cde0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa46800cad0 tx=0x7fa46800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa46dffb640 1 -- 172.21.15.62:0/567826422 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46800ae20 con 0x7fa470106f10 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.417+0000 7fa46dffb640 1 -- 172.21.15.62:0/567826422 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa468005020 con 0x7fa470106f10 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.418+0000 7fa46dffb640 1 -- 172.21.15.62:0/567826422 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46800f6f0 con 0x7fa470106f10 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.418+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47019d7b0 con 0x7fa470106f10 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.418+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4701a5090 con 0x7fa470106f10 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.419+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4701021c0 con 0x7fa470106f10 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.420+0000 7fa46dffb640 1 -- 172.21.15.62:0/567826422 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa4680051d0 con 0x7fa470106f10 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.420+0000 7fa46dffb640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4600a32d0 0x7fa4600a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.421+0000 7fa4749e6640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4600a32d0 0x7fa4600a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.421+0000 7fa46dffb640 1 -- 172.21.15.62:0/567826422 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fa4680cfd10 con 0x7fa470106f10 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.421+0000 7fa4749e6640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4600a32d0 0x7fa4600a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa464005fd0 tx=0x7fa46400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:25.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.424+0000 7fa46dffb640 1 -- 172.21.15.62:0/567826422 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa468014030 con 0x7fa470106f10 2024-09-15T23:14:25.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.597+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa470062630 con 0x7fa4600a32d0 2024-09-15T23:14:25.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.605+0000 7fa46dffb640 1 -- 172.21.15.62:0/567826422 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340590 (secure 0 0 0) 0x7fa470062630 con 0x7fa4600a32d0 2024-09-15T23:14:25.609 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:14:25.611 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:14:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4600a32d0 msgr2=0x7fa4600a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4600a32d0 0x7fa4600a5790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa464005fd0 tx=0x7fa46400b390 comp rx=0 tx=0).stop 2024-09-15T23:14:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470106f10 msgr2=0x7fa47019cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470106f10 0x7fa47019cde0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa46800cad0 tx=0x7fa46800cf90 comp rx=0 tx=0).stop 2024-09-15T23:14:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 shutdown_connections 2024-09-15T23:14:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4600a32d0 0x7fa4600a5790 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa470106f10 0x7fa47019cde0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa470102790 0x7fa470198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:25.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 --2- 172.21.15.62:0/567826422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa470101de0 0x7fa470198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 >> 172.21.15.62:0/567826422 conn(0x7fa4700fd8c0 msgr2=0x7fa4700fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 shutdown_connections 2024-09-15T23:14:25.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:25.609+0000 7fa476c4a640 1 -- 172.21.15.62:0/567826422 wait complete. 2024-09-15T23:14:25.972 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:25 smithi062 ceph-mon[30949]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:14:26.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:25 smithi067 ceph-mon[35665]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:14:26.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:25 smithi148 ceph-mon[35610]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:14:26.326 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-15T23:14:23.702719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637950,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":231,"num_read_kb":213,"num_write":340,"num_write_kb":3033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365140,"kb_used_data":11732,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576684,"statfs":{"total":1151940427776,"available":1151566524416,"internally_reserved":0,"allocated":12013568,"data_stored":7717533,"data_compressed":318288,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40128,"num_objects":5,"num_object_clones":0,"num_object_copies":17,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":68,"num_read_kb":74,"num_write":39,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002934"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.634446+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:14:07.634446+0000","last_peered":"2024-09-15T23:14:07.634446+0000","last_clean":"2024-09-15T23:14:07.634446+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:14:07.634446+0000","last_undegraded":"2024-09-15T23:14:07.634446+0000","last_fullsized":"2024-09-15T23:14:07.634446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.323257+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:13:43.323257+0000","last_peered":"2024-09-15T23:13:43.323257+0000","last_clean":"2024-09-15T23:13:43.323257+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:13:43.323257+0000","last_undegraded":"2024-09-15T23:13:43.323257+0000","last_fullsized":"2024-09-15T23:13:43.323257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189806+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:13:44.189806+0000","last_peered":"2024-09-15T23:13:44.189806+0000","last_clean":"2024-09-15T23:13:44.189806+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:13:44.189806+0000","last_undegraded":"2024-09-15T23:13:44.189806+0000","last_fullsized":"2024-09-15T23:13:44.189806+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.323448+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:13:43.323448+0000","last_peered":"2024-09-15T23:13:43.323448+0000","last_clean":"2024-09-15T23:13:43.323448+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:13:43.323448+0000","last_undegraded":"2024-09-15T23:13:43.323448+0000","last_fullsized":"2024-09-15T23:13:43.323448+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480717+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:13:43.480717+0000","last_peered":"2024-09-15T23:13:43.480717+0000","last_clean":"2024-09-15T23:13:43.480717+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:13:43.480717+0000","last_undegraded":"2024-09-15T23:13:43.480717+0000","last_fullsized":"2024-09-15T23:13:43.480717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189125+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:13:44.189125+0000","last_peered":"2024-09-15T23:13:44.189125+0000","last_clean":"2024-09-15T23:13:44.189125+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:13:44.189125+0000","last_undegraded":"2024-09-15T23:13:44.189125+0000","last_fullsized":"2024-09-15T23:13:44.189125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.634120+0000","last_change":"2024-09-15T23:13:30.491541+0000","last_active":"2024-09-15T23:14:07.634120+0000","last_peered":"2024-09-15T23:14:07.634120+0000","last_clean":"2024-09-15T23:14:07.634120+0000","last_became_active":"2024-09-15T23:13:30.491195+0000","last_became_peered":"2024-09-15T23:13:30.491195+0000","last_unstale":"2024-09-15T23:14:07.634120+0000","last_undegraded":"2024-09-15T23:14:07.634120+0000","last_fullsized":"2024-09-15T23:14:07.634120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:12:39.714183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408118+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:13:43.408118+0000","last_peered":"2024-09-15T23:13:43.408118+0000","last_clean":"2024-09-15T23:13:43.408118+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:13:43.408118+0000","last_undegraded":"2024-09-15T23:13:43.408118+0000","last_fullsized":"2024-09-15T23:13:43.408118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189434+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:13:44.189434+0000","last_peered":"2024-09-15T23:13:44.189434+0000","last_clean":"2024-09-15T23:13:44.189434+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:13:44.189434+0000","last_undegraded":"2024-09-15T23:13:44.189434+0000","last_fullsized":"2024-09-15T23:13:44.189434+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597884+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:13:42.597884+0000","last_peered":"2024-09-15T23:13:42.597884+0000","last_clean":"2024-09-15T23:13:42.597884+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:13:42.597884+0000","last_undegraded":"2024-09-15T23:13:42.597884+0000","last_fullsized":"2024-09-15T23:13:42.597884+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190605+0000","last_change":"2024-09-15T23:13:32.511232+0000","last_active":"2024-09-15T23:13:44.190605+0000","last_peered":"2024-09-15T23:13:44.190605+0000","last_clean":"2024-09-15T23:13:44.190605+0000","last_became_active":"2024-09-15T23:13:32.510875+0000","last_became_peered":"2024-09-15T23:13:32.510875+0000","last_unstale":"2024-09-15T23:13:44.190605+0000","last_undegraded":"2024-09-15T23:13:44.190605+0000","last_fullsized":"2024-09-15T23:13:44.190605+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:54:29.918573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189046+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:13:44.189046+0000","last_peered":"2024-09-15T23:13:44.189046+0000","last_clean":"2024-09-15T23:13:44.189046+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:13:44.189046+0000","last_undegraded":"2024-09-15T23:13:44.189046+0000","last_fullsized":"2024-09-15T23:13:44.189046+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341615+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:13:43.341615+0000","last_peered":"2024-09-15T23:13:43.341615+0000","last_clean":"2024-09-15T23:13:43.341615+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:13:43.341615+0000","last_undegraded":"2024-09-15T23:13:43.341615+0000","last_fullsized":"2024-09-15T23:13:43.341615+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.627763+0000","last_change":"2024-09-15T23:13:30.490411+0000","last_active":"2024-09-15T23:14:06.627763+0000","last_peered":"2024-09-15T23:14:06.627763+0000","last_clean":"2024-09-15T23:14:06.627763+0000","last_became_active":"2024-09-15T23:13:30.490081+0000","last_became_peered":"2024-09-15T23:13:30.490081+0000","last_unstale":"2024-09-15T23:14:06.627763+0000","last_undegraded":"2024-09-15T23:14:06.627763+0000","last_fullsized":"2024-09-15T23:14:06.627763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:31:26.275565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189195+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:13:44.189195+0000","last_peered":"2024-09-15T23:13:44.189195+0000","last_clean":"2024-09-15T23:13:44.189195+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:13:44.189195+0000","last_undegraded":"2024-09-15T23:13:44.189195+0000","last_fullsized":"2024-09-15T23:13:44.189195+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322897+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:13:43.322897+0000","last_peered":"2024-09-15T23:13:43.322897+0000","last_clean":"2024-09-15T23:13:43.322897+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:13:43.322897+0000","last_undegraded":"2024-09-15T23:13:43.322897+0000","last_fullsized":"2024-09-15T23:13:43.322897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190031+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:13:44.190031+0000","last_peered":"2024-09-15T23:13:44.190031+0000","last_clean":"2024-09-15T23:13:44.190031+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:13:44.190031+0000","last_undegraded":"2024-09-15T23:13:44.190031+0000","last_fullsized":"2024-09-15T23:13:44.190031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189484+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:13:44.189484+0000","last_peered":"2024-09-15T23:13:44.189484+0000","last_clean":"2024-09-15T23:13:44.189484+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:13:44.189484+0000","last_undegraded":"2024-09-15T23:13:44.189484+0000","last_fullsized":"2024-09-15T23:13:44.189484+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542334+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:13:42.542334+0000","last_peered":"2024-09-15T23:13:42.542334+0000","last_clean":"2024-09-15T23:13:42.542334+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:13:42.542334+0000","last_undegraded":"2024-09-15T23:13:42.542334+0000","last_fullsized":"2024-09-15T23:13:42.542334+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.626505+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:14:06.626505+0000","last_peered":"2024-09-15T23:14:06.626505+0000","last_clean":"2024-09-15T23:14:06.626505+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:14:06.626505+0000","last_undegraded":"2024-09-15T23:14:06.626505+0000","last_fullsized":"2024-09-15T23:14:06.626505+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542121+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:13:42.542121+0000","last_peered":"2024-09-15T23:13:42.542121+0000","last_clean":"2024-09-15T23:13:42.542121+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:13:42.542121+0000","last_undegraded":"2024-09-15T23:13:42.542121+0000","last_fullsized":"2024-09-15T23:13:42.542121+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597377+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:13:42.597377+0000","last_peered":"2024-09-15T23:13:42.597377+0000","last_clean":"2024-09-15T23:13:42.597377+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:13:42.597377+0000","last_undegraded":"2024-09-15T23:13:42.597377+0000","last_fullsized":"2024-09-15T23:13:42.597377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.625088+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:14:06.625088+0000","last_peered":"2024-09-15T23:14:06.625088+0000","last_clean":"2024-09-15T23:14:06.625088+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:14:06.625088+0000","last_undegraded":"2024-09-15T23:14:06.625088+0000","last_fullsized":"2024-09-15T23:14:06.625088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189391+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:13:44.189391+0000","last_peered":"2024-09-15T23:13:44.189391+0000","last_clean":"2024-09-15T23:13:44.189391+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:13:44.189391+0000","last_undegraded":"2024-09-15T23:13:44.189391+0000","last_fullsized":"2024-09-15T23:13:44.189391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480567+0000","last_change":"2024-09-15T23:13:32.512270+0000","last_active":"2024-09-15T23:13:43.480567+0000","last_peered":"2024-09-15T23:13:43.480567+0000","last_clean":"2024-09-15T23:13:43.480567+0000","last_became_active":"2024-09-15T23:13:32.511975+0000","last_became_peered":"2024-09-15T23:13:32.511975+0000","last_unstale":"2024-09-15T23:13:43.480567+0000","last_undegraded":"2024-09-15T23:13:43.480567+0000","last_fullsized":"2024-09-15T23:13:43.480567+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:19:32.183443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.625181+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:14:06.625181+0000","last_peered":"2024-09-15T23:14:06.625181+0000","last_clean":"2024-09-15T23:14:06.625181+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:14:06.625181+0000","last_undegraded":"2024-09-15T23:14:06.625181+0000","last_fullsized":"2024-09-15T23:14:06.625181+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189270+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:13:44.189270+0000","last_peered":"2024-09-15T23:13:44.189270+0000","last_clean":"2024-09-15T23:13:44.189270+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:13:44.189270+0000","last_undegraded":"2024-09-15T23:13:44.189270+0000","last_fullsized":"2024-09-15T23:13:44.189270+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.323360+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:13:43.323360+0000","last_peered":"2024-09-15T23:13:43.323360+0000","last_clean":"2024-09-15T23:13:43.323360+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:13:43.323360+0000","last_undegraded":"2024-09-15T23:13:43.323360+0000","last_fullsized":"2024-09-15T23:13:43.323360+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:21.519105+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:14:21.519105+0000","last_peered":"2024-09-15T23:14:21.519105+0000","last_clean":"2024-09-15T23:14:21.519105+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:14:21.519105+0000","last_undegraded":"2024-09-15T23:14:21.519105+0000","last_fullsized":"2024-09-15T23:14:21.519105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322661+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:13:43.322661+0000","last_peered":"2024-09-15T23:13:43.322661+0000","last_clean":"2024-09-15T23:13:43.322661+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:13:43.322661+0000","last_undegraded":"2024-09-15T23:13:43.322661+0000","last_fullsized":"2024-09-15T23:13:43.322661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189262+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:13:44.189262+0000","last_peered":"2024-09-15T23:13:44.189262+0000","last_clean":"2024-09-15T23:13:44.189262+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:13:44.189262+0000","last_undegraded":"2024-09-15T23:13:44.189262+0000","last_fullsized":"2024-09-15T23:13:44.189262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.625884+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:14:06.625884+0000","last_peered":"2024-09-15T23:14:06.625884+0000","last_clean":"2024-09-15T23:14:06.625884+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:14:06.625884+0000","last_undegraded":"2024-09-15T23:14:06.625884+0000","last_fullsized":"2024-09-15T23:14:06.625884+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.543078+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:13:42.543078+0000","last_peered":"2024-09-15T23:13:42.543078+0000","last_clean":"2024-09-15T23:13:42.543078+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:13:42.543078+0000","last_undegraded":"2024-09-15T23:13:42.543078+0000","last_fullsized":"2024-09-15T23:13:42.543078+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542747+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:13:42.542747+0000","last_peered":"2024-09-15T23:13:42.542747+0000","last_clean":"2024-09-15T23:13:42.542747+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:13:42.542747+0000","last_undegraded":"2024-09-15T23:13:42.542747+0000","last_fullsized":"2024-09-15T23:13:42.542747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542321+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:13:42.542321+0000","last_peered":"2024-09-15T23:13:42.542321+0000","last_clean":"2024-09-15T23:13:42.542321+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:13:42.542321+0000","last_undegraded":"2024-09-15T23:13:42.542321+0000","last_fullsized":"2024-09-15T23:13:42.542321+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597514+0000","last_change":"2024-09-15T23:13:39.748083+0000","last_active":"2024-09-15T23:13:42.597514+0000","last_peered":"2024-09-15T23:13:42.597514+0000","last_clean":"2024-09-15T23:13:42.597514+0000","last_became_active":"2024-09-15T23:13:39.747584+0000","last_became_peered":"2024-09-15T23:13:39.747584+0000","last_unstale":"2024-09-15T23:13:42.597514+0000","last_undegraded":"2024-09-15T23:13:42.597514+0000","last_fullsized":"2024-09-15T23:13:42.597514+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:35:53.649800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542708+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:13:42.542708+0000","last_peered":"2024-09-15T23:13:42.542708+0000","last_clean":"2024-09-15T23:13:42.542708+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:13:42.542708+0000","last_undegraded":"2024-09-15T23:13:42.542708+0000","last_fullsized":"2024-09-15T23:13:42.542708+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341797+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:13:43.341797+0000","last_peered":"2024-09-15T23:13:43.341797+0000","last_clean":"2024-09-15T23:13:43.341797+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:13:43.341797+0000","last_undegraded":"2024-09-15T23:13:43.341797+0000","last_fullsized":"2024-09-15T23:13:43.341797+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189139+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:13:44.189139+0000","last_peered":"2024-09-15T23:13:44.189139+0000","last_clean":"2024-09-15T23:13:44.189139+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:13:44.189139+0000","last_undegraded":"2024-09-15T23:13:44.189139+0000","last_fullsized":"2024-09-15T23:13:44.189139+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189503+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:13:44.189503+0000","last_peered":"2024-09-15T23:13:44.189503+0000","last_clean":"2024-09-15T23:13:44.189503+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:13:44.189503+0000","last_undegraded":"2024-09-15T23:13:44.189503+0000","last_fullsized":"2024-09-15T23:13:44.189503+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:18.503504+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:14:18.503504+0000","last_peered":"2024-09-15T23:14:18.503504+0000","last_clean":"2024-09-15T23:14:18.503504+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:14:18.503504+0000","last_undegraded":"2024-09-15T23:14:18.503504+0000","last_fullsized":"2024-09-15T23:14:18.503504+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189333+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:13:44.189333+0000","last_peered":"2024-09-15T23:13:44.189333+0000","last_clean":"2024-09-15T23:13:44.189333+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:13:44.189333+0000","last_undegraded":"2024-09-15T23:13:44.189333+0000","last_fullsized":"2024-09-15T23:13:44.189333+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.188705+0000","last_change":"2024-09-15T23:13:32.513091+0000","last_active":"2024-09-15T23:13:44.188705+0000","last_peered":"2024-09-15T23:13:44.188705+0000","last_clean":"2024-09-15T23:13:44.188705+0000","last_became_active":"2024-09-15T23:13:32.512752+0000","last_became_peered":"2024-09-15T23:13:32.512752+0000","last_unstale":"2024-09-15T23:13:44.188705+0000","last_undegraded":"2024-09-15T23:13:44.188705+0000","last_fullsized":"2024-09-15T23:13:44.188705+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:48:58.131659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.633384+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:14:07.633384+0000","last_peered":"2024-09-15T23:14:07.633384+0000","last_clean":"2024-09-15T23:14:07.633384+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:14:07.633384+0000","last_undegraded":"2024-09-15T23:14:07.633384+0000","last_fullsized":"2024-09-15T23:14:07.633384+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189986+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:13:44.189986+0000","last_peered":"2024-09-15T23:13:44.189986+0000","last_clean":"2024-09-15T23:13:44.189986+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:13:44.189986+0000","last_undegraded":"2024-09-15T23:13:44.189986+0000","last_fullsized":"2024-09-15T23:13:44.189986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322913+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:13:43.322913+0000","last_peered":"2024-09-15T23:13:43.322913+0000","last_clean":"2024-09-15T23:13:43.322913+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:13:43.322913+0000","last_undegraded":"2024-09-15T23:13:43.322913+0000","last_fullsized":"2024-09-15T23:13:43.322913+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:08.484346+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:14:08.484346+0000","last_peered":"2024-09-15T23:14:08.484346+0000","last_clean":"2024-09-15T23:14:08.484346+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:14:08.484346+0000","last_undegraded":"2024-09-15T23:14:08.484346+0000","last_fullsized":"2024-09-15T23:14:08.484346+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":344,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341618+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:13:43.341618+0000","last_peered":"2024-09-15T23:13:43.341618+0000","last_clean":"2024-09-15T23:13:43.341618+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:13:43.341618+0000","last_undegraded":"2024-09-15T23:13:43.341618+0000","last_fullsized":"2024-09-15T23:13:43.341618+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.342025+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:13:43.342025+0000","last_peered":"2024-09-15T23:13:43.342025+0000","last_clean":"2024-09-15T23:13:43.342025+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:13:43.342025+0000","last_undegraded":"2024-09-15T23:13:43.342025+0000","last_fullsized":"2024-09-15T23:13:43.342025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.342085+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:13:43.342085+0000","last_peered":"2024-09-15T23:13:43.342085+0000","last_clean":"2024-09-15T23:13:43.342085+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:13:43.342085+0000","last_undegraded":"2024-09-15T23:13:43.342085+0000","last_fullsized":"2024-09-15T23:13:43.342085+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.634874+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:14:07.634874+0000","last_peered":"2024-09-15T23:14:07.634874+0000","last_clean":"2024-09-15T23:14:07.634874+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:14:07.634874+0000","last_undegraded":"2024-09-15T23:14:07.634874+0000","last_fullsized":"2024-09-15T23:14:07.634874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.407983+0000","last_change":"2024-09-15T23:13:39.545839+0000","last_active":"2024-09-15T23:13:43.407983+0000","last_peered":"2024-09-15T23:13:43.407983+0000","last_clean":"2024-09-15T23:13:43.407983+0000","last_became_active":"2024-09-15T23:13:39.545374+0000","last_became_peered":"2024-09-15T23:13:39.545374+0000","last_unstale":"2024-09-15T23:13:43.407983+0000","last_undegraded":"2024-09-15T23:13:43.407983+0000","last_fullsized":"2024-09-15T23:13:43.407983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:00:03.634809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341846+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:13:43.341846+0000","last_peered":"2024-09-15T23:13:43.341846+0000","last_clean":"2024-09-15T23:13:43.341846+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:13:43.341846+0000","last_undegraded":"2024-09-15T23:13:43.341846+0000","last_fullsized":"2024-09-15T23:13:43.341846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190884+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:13:44.190884+0000","last_peered":"2024-09-15T23:13:44.190884+0000","last_clean":"2024-09-15T23:13:44.190884+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:13:44.190884+0000","last_undegraded":"2024-09-15T23:13:44.190884+0000","last_fullsized":"2024-09-15T23:13:44.190884+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480267+0000","last_change":"2024-09-15T23:13:39.544890+0000","last_active":"2024-09-15T23:13:43.480267+0000","last_peered":"2024-09-15T23:13:43.480267+0000","last_clean":"2024-09-15T23:13:43.480267+0000","last_became_active":"2024-09-15T23:13:39.544481+0000","last_became_peered":"2024-09-15T23:13:39.544481+0000","last_unstale":"2024-09-15T23:13:43.480267+0000","last_undegraded":"2024-09-15T23:13:43.480267+0000","last_fullsized":"2024-09-15T23:13:43.480267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:47.961187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.342011+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:13:43.342011+0000","last_peered":"2024-09-15T23:13:43.342011+0000","last_clean":"2024-09-15T23:13:43.342011+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:13:43.342011+0000","last_undegraded":"2024-09-15T23:13:43.342011+0000","last_fullsized":"2024-09-15T23:13:43.342011+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189244+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:13:44.189244+0000","last_peered":"2024-09-15T23:13:44.189244+0000","last_clean":"2024-09-15T23:13:44.189244+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:13:44.189244+0000","last_undegraded":"2024-09-15T23:13:44.189244+0000","last_fullsized":"2024-09-15T23:13:44.189244+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189331+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:13:44.189331+0000","last_peered":"2024-09-15T23:13:44.189331+0000","last_clean":"2024-09-15T23:13:44.189331+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:13:44.189331+0000","last_undegraded":"2024-09-15T23:13:44.189331+0000","last_fullsized":"2024-09-15T23:13:44.189331+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542736+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:13:42.542736+0000","last_peered":"2024-09-15T23:13:42.542736+0000","last_clean":"2024-09-15T23:13:42.542736+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:13:42.542736+0000","last_undegraded":"2024-09-15T23:13:42.542736+0000","last_fullsized":"2024-09-15T23:13:42.542736+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:08.485838+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:14:08.485838+0000","last_peered":"2024-09-15T23:14:08.485838+0000","last_clean":"2024-09-15T23:14:08.485838+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:14:08.485838+0000","last_undegraded":"2024-09-15T23:14:08.485838+0000","last_fullsized":"2024-09-15T23:14:08.485838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189429+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:13:44.189429+0000","last_peered":"2024-09-15T23:13:44.189429+0000","last_clean":"2024-09-15T23:13:44.189429+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:13:44.189429+0000","last_undegraded":"2024-09-15T23:13:44.189429+0000","last_fullsized":"2024-09-15T23:13:44.189429+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190324+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:13:44.190324+0000","last_peered":"2024-09-15T23:13:44.190324+0000","last_clean":"2024-09-15T23:13:44.190324+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:13:44.190324+0000","last_undegraded":"2024-09-15T23:13:44.190324+0000","last_fullsized":"2024-09-15T23:13:44.190324+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.635176+0000","last_change":"2024-09-15T23:13:30.597432+0000","last_active":"2024-09-15T23:14:07.635176+0000","last_peered":"2024-09-15T23:14:07.635176+0000","last_clean":"2024-09-15T23:14:07.635176+0000","last_became_active":"2024-09-15T23:13:30.597086+0000","last_became_peered":"2024-09-15T23:13:30.597086+0000","last_unstale":"2024-09-15T23:14:07.635176+0000","last_undegraded":"2024-09-15T23:14:07.635176+0000","last_fullsized":"2024-09-15T23:14:07.635176+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:12:02.616381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189286+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:13:44.189286+0000","last_peered":"2024-09-15T23:13:44.189286+0000","last_clean":"2024-09-15T23:13:44.189286+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:13:44.189286+0000","last_undegraded":"2024-09-15T23:13:44.189286+0000","last_fullsized":"2024-09-15T23:13:44.189286+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341368+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:13:43.341368+0000","last_peered":"2024-09-15T23:13:43.341368+0000","last_clean":"2024-09-15T23:13:43.341368+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:13:43.341368+0000","last_undegraded":"2024-09-15T23:13:43.341368+0000","last_fullsized":"2024-09-15T23:13:43.341368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":48,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:19.505332+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:14:19.505332+0000","last_peered":"2024-09-15T23:14:19.505332+0000","last_clean":"2024-09-15T23:14:19.505332+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:14:19.505332+0000","last_undegraded":"2024-09-15T23:14:19.505332+0000","last_fullsized":"2024-09-15T23:14:19.505332+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542313+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:13:42.542313+0000","last_peered":"2024-09-15T23:13:42.542313+0000","last_clean":"2024-09-15T23:13:42.542313+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:13:42.542313+0000","last_undegraded":"2024-09-15T23:13:42.542313+0000","last_fullsized":"2024-09-15T23:13:42.542313+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341739+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:13:43.341739+0000","last_peered":"2024-09-15T23:13:43.341739+0000","last_clean":"2024-09-15T23:13:43.341739+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:13:43.341739+0000","last_undegraded":"2024-09-15T23:13:43.341739+0000","last_fullsized":"2024-09-15T23:13:43.341739+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.627366+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:14:06.627366+0000","last_peered":"2024-09-15T23:14:06.627366+0000","last_clean":"2024-09-15T23:14:06.627366+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:14:06.627366+0000","last_undegraded":"2024-09-15T23:14:06.627366+0000","last_fullsized":"2024-09-15T23:14:06.627366+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542489+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:13:42.542489+0000","last_peered":"2024-09-15T23:13:42.542489+0000","last_clean":"2024-09-15T23:13:42.542489+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:13:42.542489+0000","last_undegraded":"2024-09-15T23:13:42.542489+0000","last_fullsized":"2024-09-15T23:13:42.542489+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.188909+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:13:44.188909+0000","last_peered":"2024-09-15T23:13:44.188909+0000","last_clean":"2024-09-15T23:13:44.188909+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:13:44.188909+0000","last_undegraded":"2024-09-15T23:13:44.188909+0000","last_fullsized":"2024-09-15T23:13:44.188909+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.634639+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:14:07.634639+0000","last_peered":"2024-09-15T23:14:07.634639+0000","last_clean":"2024-09-15T23:14:07.634639+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:14:07.634639+0000","last_undegraded":"2024-09-15T23:14:07.634639+0000","last_fullsized":"2024-09-15T23:14:07.634639+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190624+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:13:44.190624+0000","last_peered":"2024-09-15T23:13:44.190624+0000","last_clean":"2024-09-15T23:13:44.190624+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:13:44.190624+0000","last_undegraded":"2024-09-15T23:13:44.190624+0000","last_fullsized":"2024-09-15T23:13:44.190624+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597800+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:13:42.597800+0000","last_peered":"2024-09-15T23:13:42.597800+0000","last_clean":"2024-09-15T23:13:42.597800+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:13:42.597800+0000","last_undegraded":"2024-09-15T23:13:42.597800+0000","last_fullsized":"2024-09-15T23:13:42.597800+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'25","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:21.508517+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:14:21.508517+0000","last_peered":"2024-09-15T23:14:21.508517+0000","last_clean":"2024-09-15T23:14:21.508517+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:14:21.508517+0000","last_undegraded":"2024-09-15T23:14:21.508517+0000","last_fullsized":"2024-09-15T23:14:21.508517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43345,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":59,"num_write":26,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542140+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:13:42.542140+0000","last_peered":"2024-09-15T23:13:42.542140+0000","last_clean":"2024-09-15T23:13:42.542140+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:13:42.542140+0000","last_undegraded":"2024-09-15T23:13:42.542140+0000","last_fullsized":"2024-09-15T23:13:42.542140+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408125+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:13:43.408125+0000","last_peered":"2024-09-15T23:13:43.408125+0000","last_clean":"2024-09-15T23:13:43.408125+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:13:43.408125+0000","last_undegraded":"2024-09-15T23:13:43.408125+0000","last_fullsized":"2024-09-15T23:13:43.408125+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.624091+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:14:06.624091+0000","last_peered":"2024-09-15T23:14:06.624091+0000","last_clean":"2024-09-15T23:14:06.624091+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:14:06.624091+0000","last_undegraded":"2024-09-15T23:14:06.624091+0000","last_fullsized":"2024-09-15T23:14:06.624091+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189226+0000","last_change":"2024-09-15T23:13:39.554205+0000","last_active":"2024-09-15T23:13:44.189226+0000","last_peered":"2024-09-15T23:13:44.189226+0000","last_clean":"2024-09-15T23:13:44.189226+0000","last_became_active":"2024-09-15T23:13:39.553784+0000","last_became_peered":"2024-09-15T23:13:39.553784+0000","last_unstale":"2024-09-15T23:13:44.189226+0000","last_undegraded":"2024-09-15T23:13:44.189226+0000","last_fullsized":"2024-09-15T23:13:44.189226+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:11:08.325422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480699+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:13:43.480699+0000","last_peered":"2024-09-15T23:13:43.480699+0000","last_clean":"2024-09-15T23:13:43.480699+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:13:43.480699+0000","last_undegraded":"2024-09-15T23:13:43.480699+0000","last_fullsized":"2024-09-15T23:13:43.480699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:06.627042+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:14:06.627042+0000","last_peered":"2024-09-15T23:14:06.627042+0000","last_clean":"2024-09-15T23:14:06.627042+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:14:06.627042+0000","last_undegraded":"2024-09-15T23:14:06.627042+0000","last_fullsized":"2024-09-15T23:14:06.627042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.541706+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:13:42.541706+0000","last_peered":"2024-09-15T23:13:42.541706+0000","last_clean":"2024-09-15T23:13:42.541706+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:13:42.541706+0000","last_undegraded":"2024-09-15T23:13:42.541706+0000","last_fullsized":"2024-09-15T23:13:42.541706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408349+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:13:43.408349+0000","last_peered":"2024-09-15T23:13:43.408349+0000","last_clean":"2024-09-15T23:13:43.408349+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:13:43.408349+0000","last_undegraded":"2024-09-15T23:13:43.408349+0000","last_fullsized":"2024-09-15T23:13:43.408349+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:20.505465+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:14:20.505465+0000","last_peered":"2024-09-15T23:14:20.505465+0000","last_clean":"2024-09-15T23:14:20.505465+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:14:20.505465+0000","last_undegraded":"2024-09-15T23:14:20.505465+0000","last_fullsized":"2024-09-15T23:14:20.505465+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322877+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:13:43.322877+0000","last_peered":"2024-09-15T23:13:43.322877+0000","last_clean":"2024-09-15T23:13:43.322877+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:13:43.322877+0000","last_undegraded":"2024-09-15T23:13:43.322877+0000","last_fullsized":"2024-09-15T23:13:43.322877+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480582+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:13:43.480582+0000","last_peered":"2024-09-15T23:13:43.480582+0000","last_clean":"2024-09-15T23:13:43.480582+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:13:43.480582+0000","last_undegraded":"2024-09-15T23:13:43.480582+0000","last_fullsized":"2024-09-15T23:13:43.480582+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.541717+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:13:42.541717+0000","last_peered":"2024-09-15T23:13:42.541717+0000","last_clean":"2024-09-15T23:13:42.541717+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:13:42.541717+0000","last_undegraded":"2024-09-15T23:13:42.541717+0000","last_fullsized":"2024-09-15T23:13:42.541717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189732+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:13:44.189732+0000","last_peered":"2024-09-15T23:13:44.189732+0000","last_clean":"2024-09-15T23:13:44.189732+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:13:44.189732+0000","last_undegraded":"2024-09-15T23:13:44.189732+0000","last_fullsized":"2024-09-15T23:13:44.189732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542765+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:13:42.542765+0000","last_peered":"2024-09-15T23:13:42.542765+0000","last_clean":"2024-09-15T23:13:42.542765+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:13:42.542765+0000","last_undegraded":"2024-09-15T23:13:42.542765+0000","last_fullsized":"2024-09-15T23:13:42.542765+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322766+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:13:43.322766+0000","last_peered":"2024-09-15T23:13:43.322766+0000","last_clean":"2024-09-15T23:13:43.322766+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:13:43.322766+0000","last_undegraded":"2024-09-15T23:13:43.322766+0000","last_fullsized":"2024-09-15T23:13:43.322766+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.633777+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:14:07.633777+0000","last_peered":"2024-09-15T23:14:07.633777+0000","last_clean":"2024-09-15T23:14:07.633777+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:14:07.633777+0000","last_undegraded":"2024-09-15T23:14:07.633777+0000","last_fullsized":"2024-09-15T23:14:07.633777+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:07.633253+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:14:07.633253+0000","last_peered":"2024-09-15T23:14:07.633253+0000","last_clean":"2024-09-15T23:14:07.633253+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:14:07.633253+0000","last_undegraded":"2024-09-15T23:14:07.633253+0000","last_fullsized":"2024-09-15T23:14:07.633253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190475+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:13:44.190475+0000","last_peered":"2024-09-15T23:13:44.190475+0000","last_clean":"2024-09-15T23:13:44.190475+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:13:44.190475+0000","last_undegraded":"2024-09-15T23:13:44.190475+0000","last_fullsized":"2024-09-15T23:13:44.190475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189527+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:13:44.189527+0000","last_peered":"2024-09-15T23:13:44.189527+0000","last_clean":"2024-09-15T23:13:44.189527+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:13:44.189527+0000","last_undegraded":"2024-09-15T23:13:44.189527+0000","last_fullsized":"2024-09-15T23:13:44.189527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":52,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:14:19.504802+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:14:19.504802+0000","last_peered":"2024-09-15T23:14:19.504802+0000","last_clean":"2024-09-15T23:14:19.504802+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:14:19.504802+0000","last_undegraded":"2024-09-15T23:14:19.504802+0000","last_fullsized":"2024-09-15T23:14:19.504802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597489+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:13:42.597489+0000","last_peered":"2024-09-15T23:13:42.597489+0000","last_clean":"2024-09-15T23:13:42.597489+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:13:42.597489+0000","last_undegraded":"2024-09-15T23:13:42.597489+0000","last_fullsized":"2024-09-15T23:13:42.597489+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190108+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:13:44.190108+0000","last_peered":"2024-09-15T23:13:44.190108+0000","last_clean":"2024-09-15T23:13:44.190108+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:13:44.190108+0000","last_undegraded":"2024-09-15T23:13:44.190108+0000","last_fullsized":"2024-09-15T23:13:44.190108+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542249+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:13:42.542249+0000","last_peered":"2024-09-15T23:13:42.542249+0000","last_clean":"2024-09-15T23:13:42.542249+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:13:42.542249+0000","last_undegraded":"2024-09-15T23:13:42.542249+0000","last_fullsized":"2024-09-15T23:13:42.542249+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189645+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:13:44.189645+0000","last_peered":"2024-09-15T23:13:44.189645+0000","last_clean":"2024-09-15T23:13:44.189645+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:13:44.189645+0000","last_undegraded":"2024-09-15T23:13:44.189645+0000","last_fullsized":"2024-09-15T23:13:44.189645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189399+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:13:44.189399+0000","last_peered":"2024-09-15T23:13:44.189399+0000","last_clean":"2024-09-15T23:13:44.189399+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:13:44.189399+0000","last_undegraded":"2024-09-15T23:13:44.189399+0000","last_fullsized":"2024-09-15T23:13:44.189399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480240+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:13:43.480240+0000","last_peered":"2024-09-15T23:13:43.480240+0000","last_clean":"2024-09-15T23:13:43.480240+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:13:43.480240+0000","last_undegraded":"2024-09-15T23:13:43.480240+0000","last_fullsized":"2024-09-15T23:13:43.480240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189186+0000","last_change":"2024-09-15T23:13:32.508145+0000","last_active":"2024-09-15T23:13:44.189186+0000","last_peered":"2024-09-15T23:13:44.189186+0000","last_clean":"2024-09-15T23:13:44.189186+0000","last_became_active":"2024-09-15T23:13:32.507776+0000","last_became_peered":"2024-09-15T23:13:32.507776+0000","last_unstale":"2024-09-15T23:13:44.189186+0000","last_undegraded":"2024-09-15T23:13:44.189186+0000","last_fullsized":"2024-09-15T23:13:44.189186+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:03:59.287209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542723+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:13:42.542723+0000","last_peered":"2024-09-15T23:13:42.542723+0000","last_clean":"2024-09-15T23:13:42.542723+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:13:42.542723+0000","last_undegraded":"2024-09-15T23:13:42.542723+0000","last_fullsized":"2024-09-15T23:13:42.542723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189910+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:13:44.189910+0000","last_peered":"2024-09-15T23:13:44.189910+0000","last_clean":"2024-09-15T23:13:44.189910+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:13:44.189910+0000","last_undegraded":"2024-09-15T23:13:44.189910+0000","last_fullsized":"2024-09-15T23:13:44.189910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.323355+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:13:43.323355+0000","last_peered":"2024-09-15T23:13:43.323355+0000","last_clean":"2024-09-15T23:13:43.323355+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:13:43.323355+0000","last_undegraded":"2024-09-15T23:13:43.323355+0000","last_fullsized":"2024-09-15T23:13:43.323355+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189398+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:13:44.189398+0000","last_peered":"2024-09-15T23:13:44.189398+0000","last_clean":"2024-09-15T23:13:44.189398+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:13:44.189398+0000","last_undegraded":"2024-09-15T23:13:44.189398+0000","last_fullsized":"2024-09-15T23:13:44.189398+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322992+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:13:43.322992+0000","last_peered":"2024-09-15T23:13:43.322992+0000","last_clean":"2024-09-15T23:13:43.322992+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:13:43.322992+0000","last_undegraded":"2024-09-15T23:13:43.322992+0000","last_fullsized":"2024-09-15T23:13:43.322992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408109+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:13:43.408109+0000","last_peered":"2024-09-15T23:13:43.408109+0000","last_clean":"2024-09-15T23:13:43.408109+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:13:43.408109+0000","last_undegraded":"2024-09-15T23:13:43.408109+0000","last_fullsized":"2024-09-15T23:13:43.408109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189014+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:13:44.189014+0000","last_peered":"2024-09-15T23:13:44.189014+0000","last_clean":"2024-09-15T23:13:44.189014+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:13:44.189014+0000","last_undegraded":"2024-09-15T23:13:44.189014+0000","last_fullsized":"2024-09-15T23:13:44.189014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542497+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:13:42.542497+0000","last_peered":"2024-09-15T23:13:42.542497+0000","last_clean":"2024-09-15T23:13:42.542497+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:13:42.542497+0000","last_undegraded":"2024-09-15T23:13:42.542497+0000","last_fullsized":"2024-09-15T23:13:42.542497+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.190102+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:13:44.190102+0000","last_peered":"2024-09-15T23:13:44.190102+0000","last_clean":"2024-09-15T23:13:44.190102+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:13:44.190102+0000","last_undegraded":"2024-09-15T23:13:44.190102+0000","last_fullsized":"2024-09-15T23:13:44.190102+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542224+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:13:42.542224+0000","last_peered":"2024-09-15T23:13:42.542224+0000","last_clean":"2024-09-15T23:13:42.542224+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:13:42.542224+0000","last_undegraded":"2024-09-15T23:13:42.542224+0000","last_fullsized":"2024-09-15T23:13:42.542224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542906+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:13:42.542906+0000","last_peered":"2024-09-15T23:13:42.542906+0000","last_clean":"2024-09-15T23:13:42.542906+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:13:42.542906+0000","last_undegraded":"2024-09-15T23:13:42.542906+0000","last_fullsized":"2024-09-15T23:13:42.542906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.407738+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:13:43.407738+0000","last_peered":"2024-09-15T23:13:43.407738+0000","last_clean":"2024-09-15T23:13:43.407738+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:13:43.407738+0000","last_undegraded":"2024-09-15T23:13:43.407738+0000","last_fullsized":"2024-09-15T23:13:43.407738+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.479985+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:13:43.479985+0000","last_peered":"2024-09-15T23:13:43.479985+0000","last_clean":"2024-09-15T23:13:43.479985+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:13:43.479985+0000","last_undegraded":"2024-09-15T23:13:43.479985+0000","last_fullsized":"2024-09-15T23:13:43.479985+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480136+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:13:43.480136+0000","last_peered":"2024-09-15T23:13:43.480136+0000","last_clean":"2024-09-15T23:13:43.480136+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:13:43.480136+0000","last_undegraded":"2024-09-15T23:13:43.480136+0000","last_fullsized":"2024-09-15T23:13:43.480136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.188976+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:13:44.188976+0000","last_peered":"2024-09-15T23:13:44.188976+0000","last_clean":"2024-09-15T23:13:44.188976+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:13:44.188976+0000","last_undegraded":"2024-09-15T23:13:44.188976+0000","last_fullsized":"2024-09-15T23:13:44.188976+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189560+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:13:44.189560+0000","last_peered":"2024-09-15T23:13:44.189560+0000","last_clean":"2024-09-15T23:13:44.189560+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:13:44.189560+0000","last_undegraded":"2024-09-15T23:13:44.189560+0000","last_fullsized":"2024-09-15T23:13:44.189560+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.342023+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:13:43.342023+0000","last_peered":"2024-09-15T23:13:43.342023+0000","last_clean":"2024-09-15T23:13:43.342023+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:13:43.342023+0000","last_undegraded":"2024-09-15T23:13:43.342023+0000","last_fullsized":"2024-09-15T23:13:43.342023+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.543117+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:13:42.543117+0000","last_peered":"2024-09-15T23:13:42.543117+0000","last_clean":"2024-09-15T23:13:42.543117+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:13:42.543117+0000","last_undegraded":"2024-09-15T23:13:42.543117+0000","last_fullsized":"2024-09-15T23:13:42.543117+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.408467+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:13:43.408467+0000","last_peered":"2024-09-15T23:13:43.408467+0000","last_clean":"2024-09-15T23:13:43.408467+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:13:43.408467+0000","last_undegraded":"2024-09-15T23:13:43.408467+0000","last_fullsized":"2024-09-15T23:13:43.408467+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.322760+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:13:43.322760+0000","last_peered":"2024-09-15T23:13:43.322760+0000","last_clean":"2024-09-15T23:13:43.322760+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:13:43.322760+0000","last_undegraded":"2024-09-15T23:13:43.322760+0000","last_fullsized":"2024-09-15T23:13:43.322760+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.341479+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:13:43.341479+0000","last_peered":"2024-09-15T23:13:43.341479+0000","last_clean":"2024-09-15T23:13:43.341479+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:13:43.341479+0000","last_undegraded":"2024-09-15T23:13:43.341479+0000","last_fullsized":"2024-09-15T23:13:43.341479+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542582+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:13:42.542582+0000","last_peered":"2024-09-15T23:13:42.542582+0000","last_clean":"2024-09-15T23:13:42.542582+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:13:42.542582+0000","last_undegraded":"2024-09-15T23:13:42.542582+0000","last_fullsized":"2024-09-15T23:13:42.542582+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.597455+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:13:42.597455+0000","last_peered":"2024-09-15T23:13:42.597455+0000","last_clean":"2024-09-15T23:13:42.597455+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:13:42.597455+0000","last_undegraded":"2024-09-15T23:13:42.597455+0000","last_fullsized":"2024-09-15T23:13:42.597455+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:44.189521+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:13:44.189521+0000","last_peered":"2024-09-15T23:13:44.189521+0000","last_clean":"2024-09-15T23:13:44.189521+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:13:44.189521+0000","last_undegraded":"2024-09-15T23:13:44.189521+0000","last_fullsized":"2024-09-15T23:13:44.189521+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480335+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:13:43.480335+0000","last_peered":"2024-09-15T23:13:43.480335+0000","last_clean":"2024-09-15T23:13:43.480335+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:13:43.480335+0000","last_undegraded":"2024-09-15T23:13:43.480335+0000","last_fullsized":"2024-09-15T23:13:43.480335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:43.480510+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:13:43.480510+0000","last_peered":"2024-09-15T23:13:43.480510+0000","last_clean":"2024-09-15T23:13:43.480510+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:13:43.480510+0000","last_undegraded":"2024-09-15T23:13:43.480510+0000","last_fullsized":"2024-09-15T23:13:43.480510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-15T23:13:42.542851+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:13:42.542851+0000","last_peered":"2024-09-15T23:13:42.542851+0000","last_clean":"2024-09-15T23:13:42.542851+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:13:42.542851+0000","last_undegraded":"2024-09-15T23:13:42.542851+0000","last_fullsized":"2024-09-15T23:13:42.542851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":47348,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":105,"num_read_kb":104,"num_write":157,"num_write_kb":93,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":155377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514519,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":484535,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743451,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":1115148,"data_compressed":53347,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906977,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":488867,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135910,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":484691,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364842,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":1080602,"data_compressed":53347,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528368,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":479947,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790005,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":1076190,"data_compressed":53347,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953530,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":484597,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":480141,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1044480,"data_stored":538278,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":523202,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":983040,"data_stored":481335,"data_compressed":17583,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":43345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":58421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":44923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:14:26.331 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-15T23:14:26.331 INFO:tasks.cephadm:Setting up client nodes... 2024-09-15T23:14:26.332 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-15T23:14:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:26.968 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:26 smithi062 ceph-mon[30949]: pgmap v311: 129 pgs: 129 active+clean; 623 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-09-15T23:14:27.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.058+0000 7fbfe64fa640 1 -- 172.21.15.62:0/125134191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0102700 msgr2=0x7fbfe0104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.058+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/125134191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0102700 0x7fbfe0104af0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0009900 tx=0x7fbfd002a410 comp rx=0 tx=0).stop 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.058+0000 7fbfe64fa640 1 -- 172.21.15.62:0/125134191 shutdown_connections 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.058+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/125134191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe0107960 0x7fbfe0109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.058+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/125134191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbfe0105030 0x7fbfe0107420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.058+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/125134191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0102700 0x7fbfe0104af0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.058+0000 7fbfe64fa640 1 -- 172.21.15.62:0/125134191 >> 172.21.15.62:0/125134191 conn(0x7fbfe00fc2a0 msgr2=0x7fbfe00fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.058+0000 7fbfe64fa640 1 -- 172.21.15.62:0/125134191 shutdown_connections 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.058+0000 7fbfe64fa640 1 -- 172.21.15.62:0/125134191 wait complete. 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe64fa640 1 Processor -- start 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe64fa640 1 -- start start 2024-09-15T23:14:27.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe64fa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe0102700 0x7fbfe0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe64fa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbfe0105030 0x7fbfe0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe64fa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0107960 0x7fbfe019f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe64fa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe010d910 con 0x7fbfe0107960 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe64fa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe010d790 con 0x7fbfe0102700 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe64fa640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe010da90 con 0x7fbfe0105030 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe4a97640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0107960 0x7fbfe019f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe4a97640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0107960 0x7fbfe019f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60762/0 (socket says 172.21.15.62:60762) 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe4a97640 1 -- 172.21.15.62:0/966306650 learned_addr learned my addr 172.21.15.62:0/966306650 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfdffff640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe0102700 0x7fbfe0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe4a97640 1 -- 172.21.15.62:0/966306650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbfe0105030 msgr2=0x7fbfe0198500 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe4a97640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbfe0105030 0x7fbfe0198500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe4a97640 1 -- 172.21.15.62:0/966306650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe0102700 msgr2=0x7fbfe0197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:27.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe4a97640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe0102700 0x7fbfe0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.059+0000 7fbfe4a97640 1 -- 172.21.15.62:0/966306650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe019fbf0 con 0x7fbfe0107960 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.060+0000 7fbfe4a97640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0107960 0x7fbfe019f580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800cad0 tx=0x7fbfd800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.060+0000 7fbfdd7fa640 1 -- 172.21.15.62:0/966306650 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd800ae20 con 0x7fbfe0107960 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.060+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe019fee0 con 0x7fbfe0107960 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.060+0000 7fbfdd7fa640 1 -- 172.21.15.62:0/966306650 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfd8005020 con 0x7fbfe0107960 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.060+0000 7fbfdd7fa640 1 -- 172.21.15.62:0/966306650 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd800f6f0 con 0x7fbfe0107960 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.060+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe01a03c0 con 0x7fbfe0107960 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.061+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe00fdd90 con 0x7fbfe0107960 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.062+0000 7fbfdd7fa640 1 -- 172.21.15.62:0/966306650 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fbfd80051d0 con 0x7fbfe0107960 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.063+0000 7fbfdd7fa640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfc80a3300 0x7fbfc80a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.063+0000 7fbfdffff640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfc80a3300 0x7fbfc80a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.063+0000 7fbfdd7fa640 1 -- 172.21.15.62:0/966306650 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fbfd80d0180 con 0x7fbfe0107960 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.064+0000 7fbfdffff640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfc80a3300 0x7fbfc80a57c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0031610 tx=0x7fbfd0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:27.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.066+0000 7fbfdd7fa640 1 -- 172.21.15.62:0/966306650 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbfd8014030 con 0x7fbfe0107960 2024-09-15T23:14:27.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:26 smithi067 ceph-mon[35665]: pgmap v311: 129 pgs: 129 active+clean; 623 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-09-15T23:14:27.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:26 smithi148 ceph-mon[35610]: pgmap v311: 129 pgs: 129 active+clean; 623 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-09-15T23:14:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.312+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fbfe002d7c0 con 0x7fbfe0107960 2024-09-15T23:14:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.315+0000 7fbfdd7fa640 1 -- 172.21.15.62:0/966306650 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fbfd809cb00 con 0x7fbfe0107960 2024-09-15T23:14:27.316 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2024-09-15T23:14:27.316 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBTaudm5x22EhAA/5Pv4sYiiBY2WjJ4iB/DAQ== 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfc80a3300 msgr2=0x7fbfc80a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfc80a3300 0x7fbfc80a57c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0031610 tx=0x7fbfd0000bc0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0107960 msgr2=0x7fbfe019f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0107960 0x7fbfe019f580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbfd800cad0 tx=0x7fbfd800cf90 comp rx=0 tx=0).stop 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 shutdown_connections 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbfc80a3300 0x7fbfc80a57c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbfe0107960 0x7fbfe019f580 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbfe0105030 0x7fbfe0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 --2- 172.21.15.62:0/966306650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfe0102700 0x7fbfe0197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 >> 172.21.15.62:0/966306650 conn(0x7fbfe00fc2a0 msgr2=0x7fbfe01030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 shutdown_connections 2024-09-15T23:14:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:27.317+0000 7fbfe64fa640 1 -- 172.21.15.62:0/966306650 wait complete. 2024-09-15T23:14:27.940 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:27 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/966306650' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-15T23:14:27.940 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:27 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/966306650' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-15T23:14:27.940 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:27 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:14:27.971 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:14:27.971 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-15T23:14:27.971 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T23:14:28.009 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T23:14:28.010 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-15T23:14:28.010 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-15T23:14:28.124 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:27 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/966306650' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-15T23:14:28.124 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:27 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/966306650' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-15T23:14:28.124 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:27 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:14:28.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:27 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/966306650' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-15T23:14:28.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:27 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/966306650' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-15T23:14:28.141 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:27 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:14:28.819 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.315+0000 7f022b98a640 1 -- 172.21.15.62:0/1137289589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102ef0 msgr2=0x7f022410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:29.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.315+0000 7f022b98a640 1 --2- 172.21.15.62:0/1137289589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102ef0 0x7f022410da40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f021c009ca0 tx=0x7f021c02a950 comp rx=0 tx=0).stop 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.315+0000 7f022b98a640 1 -- 172.21.15.62:0/1137289589 shutdown_connections 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.315+0000 7f022b98a640 1 --2- 172.21.15.62:0/1137289589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102ef0 0x7f022410da40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.315+0000 7f022b98a640 1 --2- 172.21.15.62:0/1137289589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0224102570 0x7f02241029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.315+0000 7f022b98a640 1 --2- 172.21.15.62:0/1137289589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224100e20 0x7f0224101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.315+0000 7f022b98a640 1 -- 172.21.15.62:0/1137289589 >> 172.21.15.62:0/1137289589 conn(0x7f02240fc7d0 msgr2=0x7f02240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.315+0000 7f022b98a640 1 -- 172.21.15.62:0/1137289589 shutdown_connections 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.315+0000 7f022b98a640 1 -- 172.21.15.62:0/1137289589 wait complete. 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f022b98a640 1 Processor -- start 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f022b98a640 1 -- start start 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f022b98a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224100e20 0x7f0224197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f022b98a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102570 0x7f02241978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f022b98a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0224102ef0 0x7f022419bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:29.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f022b98a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0224110f80 con 0x7f0224102570 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f022b98a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0224110e00 con 0x7f0224100e20 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f022b98a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0224111100 con 0x7f0224102ef0 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f0229726640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224100e20 0x7f0224197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f0228f25640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102570 0x7f02241978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f0229726640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224100e20 0x7f0224197360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36070/0 (socket says 172.21.15.62:36070) 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f0228f25640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102570 0x7f02241978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60776/0 (socket says 172.21.15.62:60776) 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f0229726640 1 -- 172.21.15.62:0/3847796742 learned_addr learned my addr 172.21.15.62:0/3847796742 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.316+0000 7f0229f27640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0224102ef0 0x7f022419bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.317+0000 7f0229726640 1 -- 172.21.15.62:0/3847796742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0224102ef0 msgr2=0x7f022419bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.317+0000 7f0229726640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0224102ef0 0x7f022419bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.317+0000 7f0229726640 1 -- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102570 msgr2=0x7f02241978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.317+0000 7f0229726640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102570 0x7f02241978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.317+0000 7f0229726640 1 -- 172.21.15.62:0/3847796742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f022419c310 con 0x7f0224100e20 2024-09-15T23:14:29.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.317+0000 7f0228f25640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102570 0x7f02241978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.317+0000 7f0229726640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224100e20 0x7f0224197360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f022000e9f0 tx=0x7f022000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.320+0000 7f021a7fc640 1 -- 172.21.15.62:0/3847796742 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022000cd80 con 0x7f0224100e20 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.320+0000 7f021a7fc640 1 -- 172.21.15.62:0/3847796742 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0220010910 con 0x7f0224100e20 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.320+0000 7f021a7fc640 1 -- 172.21.15.62:0/3847796742 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0220018990 con 0x7f0224100e20 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.320+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f022419c600 con 0x7f0224100e20 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.320+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0224069f70 con 0x7f0224100e20 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.321+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0224101200 con 0x7f0224100e20 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.322+0000 7f021a7fc640 1 -- 172.21.15.62:0/3847796742 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0220010ac0 con 0x7f0224100e20 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.323+0000 7f021a7fc640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f02100a3350 0x7f02100a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.323+0000 7f0228f25640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f02100a3350 0x7f02100a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.323+0000 7f021a7fc640 1 -- 172.21.15.62:0/3847796742 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f02200cff50 con 0x7f0224100e20 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.323+0000 7f0228f25640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f02100a3350 0x7f02100a5810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f02240fec10 tx=0x7f021400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:29.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.327+0000 7f021a7fc640 1 -- 172.21.15.62:0/3847796742 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f022009c280 con 0x7f0224100e20 2024-09-15T23:14:29.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.580+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f022406a1d0 con 0x7f0224100e20 2024-09-15T23:14:29.582 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:29 smithi062 ceph-mon[30949]: pgmap v312: 129 pgs: 129 active+clean; 636 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-09-15T23:14:29.582 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.582 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.582 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.582 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:29 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.585+0000 7f021a7fc640 1 -- 172.21.15.62:0/3847796742 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264046 (secure 0 0 0) 0x7f02200a0580 con 0x7f0224100e20 2024-09-15T23:14:29.589 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f02100a3350 msgr2=0x7f02100a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f02100a3350 0x7f02100a5810 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f02240fec10 tx=0x7f021400b410 comp rx=0 tx=0).stop 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224100e20 msgr2=0x7f0224197360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224100e20 0x7f0224197360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f022000e9f0 tx=0x7f022000eeb0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 shutdown_connections 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f02100a3350 0x7f02100a5810 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0224102ef0 0x7f022419bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0224102570 0x7f02241978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 --2- 172.21.15.62:0/3847796742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0224100e20 0x7f0224197360 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:29.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 >> 172.21.15.62:0/3847796742 conn(0x7f02240fc7d0 msgr2=0x7f0224106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:29.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 shutdown_connections 2024-09-15T23:14:29.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:29.588+0000 7f022b98a640 1 -- 172.21.15.62:0/3847796742 wait complete. 2024-09-15T23:14:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:29 smithi067 ceph-mon[35665]: pgmap v312: 129 pgs: 129 active+clean; 636 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-09-15T23:14:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:29 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:29 smithi148 ceph-mon[35610]: pgmap v312: 129 pgs: 129 active+clean; 636 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-09-15T23:14:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:29 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:30.290 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4035090192},{"type":"v1","addr":"172.21.15.62:6801","nonce":4035090192}]},"active_addr":"172.21.15.62:6801/4035090192","active_change":"2024-09-15T23:06:39.607997+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.62:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":467758453}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":1420105809}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":1108354}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2629754523}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":2422057310}]}]} 2024-09-15T23:14:30.296 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-15T23:14:30.296 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-15T23:14:30.296 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:14:30.555 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:30.677 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:30 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3847796742' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-15T23:14:30.677 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:30.677 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:30.677 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:14:30.677 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:14:30.677 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:30.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:30 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3847796742' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-15T23:14:30.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:30.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:30.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:14:30.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:14:30.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:30 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3847796742' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-15T23:14:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:14:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:14:30.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.002+0000 7f4e4f09a640 1 -- 172.21.15.62:0/105562828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 msgr2=0x7f4e4810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.002+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/105562828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 0x7f4e4810d950 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009cd0 tx=0x7f4e4002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.002+0000 7f4e4f09a640 1 -- 172.21.15.62:0/105562828 shutdown_connections 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.002+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/105562828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 0x7f4e4810d950 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.002+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/105562828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e48100a80 0x7f4e48100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.002+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/105562828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48106a80 0x7f4e48106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.002+0000 7f4e4f09a640 1 -- 172.21.15.62:0/105562828 >> 172.21.15.62:0/105562828 conn(0x7f4e480fc7a0 msgr2=0x7f4e480febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.002+0000 7f4e4f09a640 1 -- 172.21.15.62:0/105562828 shutdown_connections 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.002+0000 7f4e4f09a640 1 -- 172.21.15.62:0/105562828 wait complete. 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e4f09a640 1 Processor -- start 2024-09-15T23:14:31.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e4f09a640 1 -- start start 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e4f09a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48100a80 0x7f4e48198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e4f09a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 0x7f4e48198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e4f09a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e48106a80 0x7f4e4819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e4f09a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4e48110e80 con 0x7f4e48101420 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e4f09a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4e48110d00 con 0x7f4e48106a80 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e4f09a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4e48111000 con 0x7f4e48100a80 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e47fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 0x7f4e48198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e47fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 0x7f4e48198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60796/0 (socket says 172.21.15.62:60796) 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e47fff640 1 -- 172.21.15.62:0/277298002 learned_addr learned my addr 172.21.15.62:0/277298002 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.003+0000 7f4e4ce36640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48100a80 0x7f4e48198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e4d637640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e48106a80 0x7f4e4819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e47fff640 1 -- 172.21.15.62:0/277298002 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48100a80 msgr2=0x7f4e48198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:31.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e47fff640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48100a80 0x7f4e48198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e47fff640 1 -- 172.21.15.62:0/277298002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e48106a80 msgr2=0x7f4e4819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e47fff640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e48106a80 0x7f4e4819cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e47fff640 1 -- 172.21.15.62:0/277298002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4819d2c0 con 0x7f4e48101420 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e4ce36640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48100a80 0x7f4e48198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e47fff640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 0x7f4e48198850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800c920 tx=0x7f4e3800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e45ffb640 1 -- 172.21.15.62:0/277298002 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e38007c70 con 0x7f4e48101420 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e4819d5b0 con 0x7f4e48101420 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e45ffb640 1 -- 172.21.15.62:0/277298002 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e38007e10 con 0x7f4e48101420 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e45ffb640 1 -- 172.21.15.62:0/277298002 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e38018890 con 0x7f4e48101420 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.004+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e481a4e90 con 0x7f4e48101420 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.005+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e48100ee0 con 0x7f4e48101420 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.006+0000 7f4e45ffb640 1 -- 172.21.15.62:0/277298002 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4e38018a70 con 0x7f4e48101420 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.007+0000 7f4e45ffb640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4e340a32d0 0x7f4e340a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:31.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.007+0000 7f4e45ffb640 1 -- 172.21.15.62:0/277298002 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f4e380cfee0 con 0x7f4e48101420 2024-09-15T23:14:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.007+0000 7f4e4ce36640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4e340a32d0 0x7f4e340a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.007+0000 7f4e4ce36640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4e340a32d0 0x7f4e340a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4e30005e10 tx=0x7f4e30005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:31.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.011+0000 7f4e45ffb640 1 -- 172.21.15.62:0/277298002 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4e38014030 con 0x7f4e48101420 2024-09-15T23:14:31.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.189+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e481a5130 con 0x7f4e48101420 2024-09-15T23:14:31.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.191+0000 7f4e45ffb640 1 -- 172.21.15.62:0/277298002 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f4e3800de20 con 0x7f4e48101420 2024-09-15T23:14:31.192 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:14:31.192 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:13:42.233963+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4e340a32d0 msgr2=0x7f4e340a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4e340a32d0 0x7f4e340a5790 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4e30005e10 tx=0x7f4e30005da0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 msgr2=0x7f4e48198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 0x7f4e48198850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800c920 tx=0x7f4e3800cde0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 shutdown_connections 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4e340a32d0 0x7f4e340a5790 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e48106a80 0x7f4e4819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4e48101420 0x7f4e48198850 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 --2- 172.21.15.62:0/277298002 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4e48100a80 0x7f4e48198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:31.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 >> 172.21.15.62:0/277298002 conn(0x7f4e480fc7a0 msgr2=0x7f4e480fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:31.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 shutdown_connections 2024-09-15T23:14:31.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:31.193+0000 7f4e4f09a640 1 -- 172.21.15.62:0/277298002 wait complete. 2024-09-15T23:14:31.836 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:31 smithi062 ceph-mon[30949]: pgmap v313: 129 pgs: 129 active+clean; 636 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-09-15T23:14:31.836 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:31 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/277298002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:31.867 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-15T23:14:31.867 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:14:32.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:31 smithi067 ceph-mon[35665]: pgmap v313: 129 pgs: 129 active+clean; 636 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-09-15T23:14:32.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:31 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/277298002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:31 smithi148 ceph-mon[35610]: pgmap v313: 129 pgs: 129 active+clean; 636 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-09-15T23:14:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:31 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/277298002' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:32.127 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:32.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.604+0000 7f4f386f7640 1 -- 172.21.15.62:0/2019538206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30101de0 msgr2=0x7f4f301021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.604+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2019538206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30101de0 0x7f4f301021c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009ca0 tx=0x7f4f2402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.604+0000 7f4f386f7640 1 -- 172.21.15.62:0/2019538206 shutdown_connections 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.604+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2019538206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f30106f10 0x7f4f3010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.604+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2019538206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f30102790 0x7f4f301067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.604+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2019538206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30101de0 0x7f4f301021c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.604+0000 7f4f386f7640 1 -- 172.21.15.62:0/2019538206 >> 172.21.15.62:0/2019538206 conn(0x7f4f300fd8c0 msgr2=0x7f4f300ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.604+0000 7f4f386f7640 1 -- 172.21.15.62:0/2019538206 shutdown_connections 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.604+0000 7f4f386f7640 1 -- 172.21.15.62:0/2019538206 wait complete. 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f386f7640 1 Processor -- start 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f386f7640 1 -- start start 2024-09-15T23:14:32.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f386f7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f30101de0 0x7f4f30198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:32.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f386f7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f30102790 0x7f4f30198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:32.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f386f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30106f10 0x7f4f3019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:32.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f386f7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4f30111f70 con 0x7f4f30106f10 2024-09-15T23:14:32.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f386f7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f30111df0 con 0x7f4f30102790 2024-09-15T23:14:32.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f386f7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4f301120f0 con 0x7f4f30101de0 2024-09-15T23:14:32.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f36c94640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30106f10 0x7f4f3019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:32.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f36c94640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30106f10 0x7f4f3019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51570/0 (socket says 172.21.15.62:51570) 2024-09-15T23:14:32.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f36c94640 1 -- 172.21.15.62:0/2822375576 learned_addr learned my addr 172.21.15.62:0/2822375576 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:32.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f36493640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f30101de0 0x7f4f30198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:32.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.605+0000 7f4f35c92640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f30102790 0x7f4f30198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:32.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f36c94640 1 -- 172.21.15.62:0/2822375576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f30101de0 msgr2=0x7f4f30198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:32.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f36c94640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f30101de0 0x7f4f30198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f36c94640 1 -- 172.21.15.62:0/2822375576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f30102790 msgr2=0x7f4f30198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:32.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f36c94640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f30102790 0x7f4f30198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f36c94640 1 -- 172.21.15.62:0/2822375576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3019d3e0 con 0x7f4f30106f10 2024-09-15T23:14:32.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f36c94640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30106f10 0x7f4f3019cd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800c970 tx=0x7f4f2800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:32.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f237fe640 1 -- 172.21.15.62:0/2822375576 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28007bb0 con 0x7f4f30106f10 2024-09-15T23:14:32.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f237fe640 1 -- 172.21.15.62:0/2822375576 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f28007d50 con 0x7f4f30106f10 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3019d6d0 con 0x7f4f30106f10 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f237fe640 1 -- 172.21.15.62:0/2822375576 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2800f450 con 0x7f4f30106f10 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.606+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f301a4fb0 con 0x7f4f30106f10 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.607+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f301021c0 con 0x7f4f30106f10 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.608+0000 7f4f237fe640 1 -- 172.21.15.62:0/2822375576 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4f28004f20 con 0x7f4f30106f10 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.609+0000 7f4f237fe640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f1c0a32d0 0x7f4f1c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.609+0000 7f4f36493640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f1c0a32d0 0x7f4f1c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.609+0000 7f4f237fe640 1 -- 172.21.15.62:0/2822375576 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f4f280cf8a0 con 0x7f4f30106f10 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.610+0000 7f4f36493640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f1c0a32d0 0x7f4f1c0a5790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4f24006010 tx=0x7f4f24009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.613+0000 7f4f237fe640 1 -- 172.21.15.62:0/2822375576 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4f28014030 con 0x7f4f30106f10 2024-09-15T23:14:32.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.789+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f30109b60 con 0x7f4f30106f10 2024-09-15T23:14:32.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.790+0000 7f4f237fe640 1 -- 172.21.15.62:0/2822375576 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24187 (secure 0 0 0) 0x7f4f2809bb70 con 0x7f4f30106f10 2024-09-15T23:14:32.792 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:14:32.792 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":88,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:13:42.233963+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f1c0a32d0 msgr2=0x7f4f1c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f1c0a32d0 0x7f4f1c0a5790 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4f24006010 tx=0x7f4f24009360 comp rx=0 tx=0).stop 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30106f10 msgr2=0x7f4f3019cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30106f10 0x7f4f3019cd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800c970 tx=0x7f4f2800ce30 comp rx=0 tx=0).stop 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 shutdown_connections 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f1c0a32d0 0x7f4f1c0a5790 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f30106f10 0x7f4f3019cd00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f30102790 0x7f4f30198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 --2- 172.21.15.62:0/2822375576 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f30101de0 0x7f4f30198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.793+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 >> 172.21.15.62:0/2822375576 conn(0x7f4f300fd8c0 msgr2=0x7f4f300fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.794+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 shutdown_connections 2024-09-15T23:14:32.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:32.794+0000 7f4f386f7640 1 -- 172.21.15.62:0/2822375576 wait complete. 2024-09-15T23:14:33.502 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-15T23:14:33.503 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-15T23:14:33.503 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-15T23:14:33.504 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-15T23:14:33.504 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-15T23:14:33.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-15T23:14:33.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-15T23:14:33.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-15T23:14:33.506 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-15T23:14:33.506 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-15T23:14:33.507 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-15T23:14:33.507 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-15T23:14:33.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:33 smithi062 ceph-mon[30949]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-15T23:14:33.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:33 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2822375576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:33 smithi067 ceph-mon[35665]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-15T23:14:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:33 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2822375576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:33 smithi148 ceph-mon[35610]: pgmap v314: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.6 KiB/s wr, 9 op/s 2024-09-15T23:14:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:33 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2822375576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:14:34.305 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.305 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.305 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.309 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.310 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.310 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.312 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.312 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.324 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:34.932 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.479+0000 7f4927fc5640 1 -- 172.21.15.62:0/1452370426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4920105f40 msgr2=0x7f49201063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.479+0000 7f4927fc5640 1 --2- 172.21.15.62:0/1452370426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4920105f40 0x7f49201063d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4918009ca0 tx=0x7f491802a940 comp rx=0 tx=0).stop 2024-09-15T23:14:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.479+0000 7f4927fc5640 1 -- 172.21.15.62:0/1452370426 shutdown_connections 2024-09-15T23:14:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.479+0000 7f4927fc5640 1 --2- 172.21.15.62:0/1452370426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4920105f40 0x7f49201063d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.479+0000 7f4927fc5640 1 --2- 172.21.15.62:0/1452370426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4920101450 0x7f4920105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.479+0000 7f4927fc5640 1 --2- 172.21.15.62:0/1452370426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920100b30 0x7f4920100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.479+0000 7f4927fc5640 1 -- 172.21.15.62:0/1452370426 >> 172.21.15.62:0/1452370426 conn(0x7f49200fc7a0 msgr2=0x7f49200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.480+0000 7f4927fc5640 1 -- 172.21.15.62:0/1452370426 shutdown_connections 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.480+0000 7f4927fc5640 1 -- 172.21.15.62:0/1452370426 wait complete. 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.480+0000 7f4927fc5640 1 Processor -- start 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.480+0000 7f4927fc5640 1 -- start start 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.480+0000 7f4927fc5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920100b30 0x7f49201a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4927fc5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4920101450 0x7f49201a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4927fc5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4920105f40 0x7f492019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4927fc5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4920118580 con 0x7f4920101450 2024-09-15T23:14:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4927fc5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4920118400 con 0x7f4920100b30 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4927fc5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4920118700 con 0x7f4920105f40 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925560640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4920101450 0x7f49201a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925560640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4920101450 0x7f49201a4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51618/0 (socket says 172.21.15.62:51618) 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925560640 1 -- 172.21.15.62:0/323487772 learned_addr learned my addr 172.21.15.62:0/323487772 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925d61640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920100b30 0x7f49201a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4926562640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4920105f40 0x7f492019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925560640 1 -- 172.21.15.62:0/323487772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4920105f40 msgr2=0x7f492019e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925560640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4920105f40 0x7f492019e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925560640 1 -- 172.21.15.62:0/323487772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920100b30 msgr2=0x7f49201a4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925560640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4920100b30 0x7f49201a4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925560640 1 -- 172.21.15.62:0/323487772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492019f0a0 con 0x7f4920101450 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4925560640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4920101450 0x7f49201a4c50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f491000e9f0 tx=0x7f491000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4916ffd640 1 -- 172.21.15.62:0/323487772 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491000cab0 con 0x7f4920101450 2024-09-15T23:14:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492019f390 con 0x7f4920101450 2024-09-15T23:14:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4916ffd640 1 -- 172.21.15.62:0/323487772 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4910005500 con 0x7f4920101450 2024-09-15T23:14:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4916ffd640 1 -- 172.21.15.62:0/323487772 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4910010430 con 0x7f4920101450 2024-09-15T23:14:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.481+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49201ab530 con 0x7f4920101450 2024-09-15T23:14:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.482+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4920105270 con 0x7f4920101450 2024-09-15T23:14:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.484+0000 7f4916ffd640 1 -- 172.21.15.62:0/323487772 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4910004f90 con 0x7f4920101450 2024-09-15T23:14:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.484+0000 7f4916ffd640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f490c0a32d0 0x7f490c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.485+0000 7f4925d61640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f490c0a32d0 0x7f490c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.485+0000 7f4916ffd640 1 -- 172.21.15.62:0/323487772 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f49100d0630 con 0x7f4920101450 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.485+0000 7f4916ffd640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] conn(0x7f490c0ae000 0x7f490c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.485+0000 7f4916ffd640 1 -- 172.21.15.62:0/323487772 --> [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f490c0b0b10 con 0x7f490c0ae000 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.485+0000 7f4916ffd640 1 -- 172.21.15.62:0/323487772 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f49100d0a80 con 0x7f4920101450 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.485+0000 7f4926562640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] conn(0x7f490c0ae000 0x7f490c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.485+0000 7f4925d61640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f490c0a32d0 0x7f490c0a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f491c00abe0 tx=0x7f491c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.485+0000 7f4926562640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] conn(0x7f490c0ae000 0x7f490c0b0440 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.486+0000 7f4916ffd640 1 -- 172.21.15.62:0/323487772 <== osd.5 v2:172.21.15.148:6808/1859303200 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f490c0b0b10 con 0x7f490c0ae000 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.500+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 --> [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f492006b070 con 0x7f490c0ae000 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.500+0000 7f4916ffd640 1 -- 172.21.15.62:0/323487772 <== osd.5 v2:172.21.15.148:6808/1859303200 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f492006b070 con 0x7f490c0ae000 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.500+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 >> [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] conn(0x7f490c0ae000 msgr2=0x7f490c0b0440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.500+0000 7f4927fc5640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] conn(0x7f490c0ae000 0x7f490c0b0440 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.501+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f490c0a32d0 msgr2=0x7f490c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.501+0000 7f4927fc5640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f490c0a32d0 0x7f490c0a5790 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f491c00abe0 tx=0x7f491c00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.501+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4920101450 msgr2=0x7f49201a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.501+0000 7f4927fc5640 1 --2- 172.21.15.62:0/323487772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4920101450 0x7f49201a4c50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f491000e9f0 tx=0x7f491000edf0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.501+0000 7f4926562640 1 -- 172.21.15.62:0/323487772 reap_dead start 2024-09-15T23:14:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.501+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 shutdown_connections 2024-09-15T23:14:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.501+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 >> 172.21.15.62:0/323487772 conn(0x7f49200fc7a0 msgr2=0x7f49201038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.501+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 shutdown_connections 2024-09-15T23:14:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.501+0000 7f4927fc5640 1 -- 172.21.15.62:0/323487772 wait complete. 2024-09-15T23:14:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:35 smithi062 ceph-mon[30949]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-09-15T23:14:35.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.378+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1309830373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0106f10 msgr2=0x7fe9f010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.378+0000 7fe9f49d9640 1 --2- 172.21.15.62:0/1309830373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0106f10 0x7fe9f010ea90 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8009ca0 tx=0x7fe9e802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.378+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1309830373 shutdown_connections 2024-09-15T23:14:35.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.378+0000 7fe9f49d9640 1 --2- 172.21.15.62:0/1309830373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0106f10 0x7fe9f010ea90 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.378+0000 7fe9f49d9640 1 --2- 172.21.15.62:0/1309830373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9f0102790 0x7fe9f01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.378+0000 7fe9f49d9640 1 --2- 172.21.15.62:0/1309830373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0101de0 0x7fe9f01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.378+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1309830373 >> 172.21.15.62:0/1309830373 conn(0x7fe9f00fd8c0 msgr2=0x7fe9f00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.378+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1309830373 shutdown_connections 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.378+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1309830373 wait complete. 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9f49d9640 1 Processor -- start 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9f49d9640 1 -- start start 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9f49d9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0101de0 0x7fe9f00744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9f49d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9f0102790 0x7fe9f0074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9f49d9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0106f10 0x7fe9f0074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9f49d9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f0112010 con 0x7fe9f0101de0 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9f49d9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f0111e90 con 0x7fe9f0106f10 2024-09-15T23:14:35.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9f49d9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f0112190 con 0x7fe9f0102790 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9ee59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0101de0 0x7fe9f00744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9ee59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0101de0 0x7fe9f00744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51588/0 (socket says 172.21.15.62:51588) 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9ee59c640 1 -- 172.21.15.62:0/1074375131 learned_addr learned my addr 172.21.15.62:0/1074375131 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.379+0000 7fe9edd9b640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9f0102790 0x7fe9f0074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9eed9d640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0106f10 0x7fe9f0074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9ee59c640 1 -- 172.21.15.62:0/1074375131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9f0102790 msgr2=0x7fe9f0074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9ee59c640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9f0102790 0x7fe9f0074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9ee59c640 1 -- 172.21.15.62:0/1074375131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0106f10 msgr2=0x7fe9f0074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9ee59c640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0106f10 0x7fe9f0074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9ee59c640 1 -- 172.21.15.62:0/1074375131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f01a6a50 con 0x7fe9f0101de0 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9edd9b640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9f0102790 0x7fe9f0074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9eed9d640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9f0106f10 0x7fe9f0074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:14:35.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9ee59c640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0101de0 0x7fe9f00744f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000b4a0 tx=0x7fe9e000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9df7fe640 1 -- 172.21.15.62:0/1074375131 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0014070 con 0x7fe9f0101de0 2024-09-15T23:14:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f01a6d40 con 0x7fe9f0101de0 2024-09-15T23:14:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9df7fe640 1 -- 172.21.15.62:0/1074375131 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9e0010af0 con 0x7fe9f0101de0 2024-09-15T23:14:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9df7fe640 1 -- 172.21.15.62:0/1074375131 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0017d10 con 0x7fe9f0101de0 2024-09-15T23:14:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.380+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f01a7270 con 0x7fe9f0101de0 2024-09-15T23:14:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.381+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe9f0062880 con 0x7fe9f0101de0 2024-09-15T23:14:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.382+0000 7fe9df7fe640 1 -- 172.21.15.62:0/1074375131 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fe9e0017070 con 0x7fe9f0101de0 2024-09-15T23:14:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.383+0000 7fe9df7fe640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9d80a32d0 0x7fe9d80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.383+0000 7fe9edd9b640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9d80a32d0 0x7fe9d80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.383+0000 7fe9df7fe640 1 -- 172.21.15.62:0/1074375131 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fe9e00cf3e0 con 0x7fe9f0101de0 2024-09-15T23:14:35.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.383+0000 7fe9df7fe640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] conn(0x7fe9d80ae000 0x7fe9d80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:35.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.383+0000 7fe9df7fe640 1 -- 172.21.15.62:0/1074375131 --> [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9e000dec0 con 0x7fe9d80ae000 2024-09-15T23:14:35.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.383+0000 7fe9df7fe640 1 -- 172.21.15.62:0/1074375131 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fe9e00cf830 con 0x7fe9f0101de0 2024-09-15T23:14:35.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.383+0000 7fe9edd9b640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9d80a32d0 0x7fe9d80a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0075f80 tx=0x7fe9e4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:35.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.384+0000 7fe9eed9d640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] conn(0x7fe9d80ae000 0x7fe9d80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.384+0000 7fe9eed9d640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] conn(0x7fe9d80ae000 0x7fe9d80b0440 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.385+0000 7fe9df7fe640 1 -- 172.21.15.62:0/1074375131 <== osd.4 v2:172.21.15.67:6808/3506283519 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe9e000dec0 con 0x7fe9d80ae000 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.400+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 --> [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe9f0062c80 con 0x7fe9d80ae000 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.400+0000 7fe9df7fe640 1 -- 172.21.15.62:0/1074375131 <== osd.4 v2:172.21.15.67:6808/3506283519 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe9f0062c80 con 0x7fe9d80ae000 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 >> [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] conn(0x7fe9d80ae000 msgr2=0x7fe9d80b0440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] conn(0x7fe9d80ae000 0x7fe9d80b0440 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9d80a32d0 msgr2=0x7fe9d80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9d80a32d0 0x7fe9d80a5790 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0075f80 tx=0x7fe9e4008040 comp rx=0 tx=0).stop 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0101de0 msgr2=0x7fe9f00744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 --2- 172.21.15.62:0/1074375131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9f0101de0 0x7fe9f00744f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe9e000b4a0 tx=0x7fe9e000b960 comp rx=0 tx=0).stop 2024-09-15T23:14:35.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9eed9d640 1 -- 172.21.15.62:0/1074375131 reap_dead start 2024-09-15T23:14:35.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 shutdown_connections 2024-09-15T23:14:35.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 >> 172.21.15.62:0/1074375131 conn(0x7fe9f00fd8c0 msgr2=0x7fe9f00fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:35.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 shutdown_connections 2024-09-15T23:14:35.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:35.401+0000 7fe9f49d9640 1 -- 172.21.15.62:0/1074375131 wait complete. 2024-09-15T23:14:36.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:35 smithi067 ceph-mon[35665]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-09-15T23:14:36.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:35 smithi148 ceph-mon[35610]: pgmap v315: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-09-15T23:14:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.690+0000 7ff5aece0640 1 -- 172.21.15.62:0/2143357546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5a8100a80 msgr2=0x7ff5a8100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.690+0000 7ff5aece0640 1 --2- 172.21.15.62:0/2143357546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5a8100a80 0x7ff5a8100ee0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff594009930 tx=0x7ff59402a420 comp rx=0 tx=0).stop 2024-09-15T23:14:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.690+0000 7ff5aece0640 1 -- 172.21.15.62:0/2143357546 shutdown_connections 2024-09-15T23:14:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.690+0000 7ff5aece0640 1 --2- 172.21.15.62:0/2143357546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5a8101420 0x7ff5a810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.690+0000 7ff5aece0640 1 --2- 172.21.15.62:0/2143357546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5a8100a80 0x7ff5a8100ee0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.690+0000 7ff5aece0640 1 --2- 172.21.15.62:0/2143357546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8106a80 0x7ff5a8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 -- 172.21.15.62:0/2143357546 >> 172.21.15.62:0/2143357546 conn(0x7ff5a80fc7a0 msgr2=0x7ff5a80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 -- 172.21.15.62:0/2143357546 shutdown_connections 2024-09-15T23:14:36.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 -- 172.21.15.62:0/2143357546 wait complete. 2024-09-15T23:14:36.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 Processor -- start 2024-09-15T23:14:36.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 -- start start 2024-09-15T23:14:36.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100a80 0x7ff5a81962d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5a8101420 0x7ff5a8196810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5a8106a80 0x7ff5a819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a810fd90 con 0x7ff5a8101420 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a810fc10 con 0x7ff5a8100a80 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.691+0000 7ff5aece0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a810ff10 con 0x7ff5a8106a80 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5a8101420 0x7ff5a8196810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5a8101420 0x7ff5a8196810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51634/0 (socket says 172.21.15.62:51634) 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a7fff640 1 -- 172.21.15.62:0/1220699574 learned_addr learned my addr 172.21.15.62:0/1220699574 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5ad27d640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5a8106a80 0x7ff5a819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5aca7c640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100a80 0x7ff5a81962d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a7fff640 1 -- 172.21.15.62:0/1220699574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5a8106a80 msgr2=0x7ff5a819aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a7fff640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff5a8106a80 0x7ff5a819aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a7fff640 1 -- 172.21.15.62:0/1220699574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100a80 msgr2=0x7ff5a81962d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a7fff640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100a80 0x7ff5a81962d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a7fff640 1 -- 172.21.15.62:0/1220699574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a819b280 con 0x7ff5a8101420 2024-09-15T23:14:36.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5aca7c640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5a8100a80 0x7ff5a81962d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:36.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a7fff640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5a8101420 0x7ff5a8196810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff594009a50 tx=0x7ff594004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:36.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a5ffb640 1 -- 172.21.15.62:0/1220699574 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59403f030 con 0x7ff5a8101420 2024-09-15T23:14:36.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a5ffb640 1 -- 172.21.15.62:0/1220699574 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff594003ce0 con 0x7ff5a8101420 2024-09-15T23:14:36.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a819b510 con 0x7ff5a8101420 2024-09-15T23:14:36.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.692+0000 7ff5a5ffb640 1 -- 172.21.15.62:0/1220699574 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594035070 con 0x7ff5a8101420 2024-09-15T23:14:36.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.693+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a81a2e50 con 0x7ff5a8101420 2024-09-15T23:14:36.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.693+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff5a8062980 con 0x7ff5a8101420 2024-09-15T23:14:36.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.695+0000 7ff5a5ffb640 1 -- 172.21.15.62:0/1220699574 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff594042030 con 0x7ff5a8101420 2024-09-15T23:14:36.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.695+0000 7ff5a5ffb640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff5980a54e0 0x7ff5980a79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.696+0000 7ff5aca7c640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff5980a54e0 0x7ff5980a79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.696+0000 7ff5a5ffb640 1 -- 172.21.15.62:0/1220699574 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7ff5940f06e0 con 0x7ff5a8101420 2024-09-15T23:14:36.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.696+0000 7ff5a5ffb640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] conn(0x7ff5980b0210 0x7ff5980b2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.696+0000 7ff5a5ffb640 1 -- 172.21.15.62:0/1220699574 --> [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff594046e50 con 0x7ff5980b0210 2024-09-15T23:14:36.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.696+0000 7ff5a5ffb640 1 -- 172.21.15.62:0/1220699574 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7ff5940f0b30 con 0x7ff5a8101420 2024-09-15T23:14:36.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.696+0000 7ff5ad27d640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] conn(0x7ff5980b0210 0x7ff5980b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.696+0000 7ff5aca7c640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff5980a54e0 0x7ff5980a79a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff59c0058c0 tx=0x7ff59c005ec0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:36.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.696+0000 7ff5ad27d640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] conn(0x7ff5980b0210 0x7ff5980b2650 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:36.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.697+0000 7ff5a5ffb640 1 -- 172.21.15.62:0/1220699574 <== osd.6 v2:172.21.15.62:6818/2615321035 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff594046e50 con 0x7ff5980b0210 2024-09-15T23:14:36.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.711+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 --> [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff5a8062e00 con 0x7ff5980b0210 2024-09-15T23:14:36.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.711+0000 7ff5a5ffb640 1 -- 172.21.15.62:0/1220699574 <== osd.6 v2:172.21.15.62:6818/2615321035 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff5a8062e00 con 0x7ff5980b0210 2024-09-15T23:14:36.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] conn(0x7ff5980b0210 msgr2=0x7ff5980b2650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] conn(0x7ff5980b0210 0x7ff5980b2650 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff5980a54e0 msgr2=0x7ff5980a79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff5980a54e0 0x7ff5980a79a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff59c0058c0 tx=0x7ff59c005ec0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5a8101420 msgr2=0x7ff5a8196810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 --2- 172.21.15.62:0/1220699574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff5a8101420 0x7ff5a8196810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff594009a50 tx=0x7ff594004270 comp rx=0 tx=0).stop 2024-09-15T23:14:36.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5ad27d640 1 -- 172.21.15.62:0/1220699574 reap_dead start 2024-09-15T23:14:36.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 shutdown_connections 2024-09-15T23:14:36.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 >> 172.21.15.62:0/1220699574 conn(0x7ff5a80fc7a0 msgr2=0x7ff5a80fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:36.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 shutdown_connections 2024-09-15T23:14:36.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.712+0000 7ff5aece0640 1 -- 172.21.15.62:0/1220699574 wait complete. 2024-09-15T23:14:36.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 -- 172.21.15.62:0/2157717814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef48102d90 msgr2=0x7fef48103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 --2- 172.21.15.62:0/2157717814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef48102d90 0x7fef48103170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fef44009930 tx=0x7fef4402a420 comp rx=0 tx=0).stop 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 -- 172.21.15.62:0/2157717814 shutdown_connections 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 --2- 172.21.15.62:0/2157717814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48108290 0x7fef4810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 --2- 172.21.15.62:0/2157717814 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef48103740 0x7fef48107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 --2- 172.21.15.62:0/2157717814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef48102d90 0x7fef48103170 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 -- 172.21.15.62:0/2157717814 >> 172.21.15.62:0/2157717814 conn(0x7fef480fc7d0 msgr2=0x7fef480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 -- 172.21.15.62:0/2157717814 shutdown_connections 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 -- 172.21.15.62:0/2157717814 wait complete. 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.874+0000 7fef4fe47640 1 Processor -- start 2024-09-15T23:14:36.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4fe47640 1 -- start start 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4fe47640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef48102d90 0x7fef481985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4fe47640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48103740 0x7fef48198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4fe47640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef48108290 0x7fef4819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4fe47640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef48112090 con 0x7fef48102d90 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4fe47640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef48111f10 con 0x7fef48103740 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4fe47640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fef48112210 con 0x7fef48108290 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4dbe3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef48102d90 0x7fef481985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4dbe3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef48102d90 0x7fef481985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51648/0 (socket says 172.21.15.62:51648) 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4dbe3640 1 -- 172.21.15.62:0/3836730389 learned_addr learned my addr 172.21.15.62:0/3836730389 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4d3e2640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48103740 0x7fef48198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4e3e4640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef48108290 0x7fef4819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4dbe3640 1 -- 172.21.15.62:0/3836730389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef48108290 msgr2=0x7fef4819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4dbe3640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef48108290 0x7fef4819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4dbe3640 1 -- 172.21.15.62:0/3836730389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48103740 msgr2=0x7fef48198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4dbe3640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48103740 0x7fef48198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4dbe3640 1 -- 172.21.15.62:0/3836730389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4819d5a0 con 0x7fef48102d90 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4d3e2640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef48103740 0x7fef48198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.875+0000 7fef4dbe3640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef48102d90 0x7fef481985f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fef44009230 tx=0x7fef44004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.876+0000 7fef3affd640 1 -- 172.21.15.62:0/3836730389 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4403f030 con 0x7fef48102d90 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.876+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4819d830 con 0x7fef48102d90 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.876+0000 7fef3affd640 1 -- 172.21.15.62:0/3836730389 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef44003ad0 con 0x7fef48102d90 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.876+0000 7fef3affd640 1 -- 172.21.15.62:0/3836730389 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44035070 con 0x7fef48102d90 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.876+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef481a5170 con 0x7fef48102d90 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.876+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fef48062980 con 0x7fef48102d90 2024-09-15T23:14:36.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.878+0000 7fef3affd640 1 -- 172.21.15.62:0/3836730389 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fef44042030 con 0x7fef48102d90 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.878+0000 7fef3affd640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef340a32d0 0x7fef340a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.878+0000 7fef4d3e2640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef340a32d0 0x7fef340a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.879+0000 7fef3affd640 1 -- 172.21.15.62:0/3836730389 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7fef440f06b0 con 0x7fef48102d90 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.879+0000 7fef3affd640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] conn(0x7fef340ae000 0x7fef340b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.879+0000 7fef3affd640 1 -- 172.21.15.62:0/3836730389 --> [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fef340b0b10 con 0x7fef340ae000 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.879+0000 7fef3affd640 1 -- 172.21.15.62:0/3836730389 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7fef440f0b00 con 0x7fef48102d90 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.879+0000 7fef4d3e2640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef340a32d0 0x7fef340a5790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0058c0 tx=0x7fef3c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.879+0000 7fef4e3e4640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] conn(0x7fef340ae000 0x7fef340b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.879+0000 7fef4e3e4640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] conn(0x7fef340ae000 0x7fef340b0440 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.880+0000 7fef3affd640 1 -- 172.21.15.62:0/3836730389 <== osd.2 v2:172.21.15.148:6800/1833942254 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fef340b0b10 con 0x7fef340ae000 2024-09-15T23:14:36.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.894+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 --> [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fef48062d80 con 0x7fef340ae000 2024-09-15T23:14:36.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef3affd640 1 -- 172.21.15.62:0/3836730389 <== osd.2 v2:172.21.15.148:6800/1833942254 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fef48062d80 con 0x7fef340ae000 2024-09-15T23:14:36.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 >> [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] conn(0x7fef340ae000 msgr2=0x7fef340b0440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] conn(0x7fef340ae000 0x7fef340b0440 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:36.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef340a32d0 msgr2=0x7fef340a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef340a32d0 0x7fef340a5790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0058c0 tx=0x7fef3c005850 comp rx=0 tx=0).stop 2024-09-15T23:14:36.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef48102d90 msgr2=0x7fef481985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:36.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 --2- 172.21.15.62:0/3836730389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef48102d90 0x7fef481985f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fef44009230 tx=0x7fef44004270 comp rx=0 tx=0).stop 2024-09-15T23:14:36.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4e3e4640 1 -- 172.21.15.62:0/3836730389 reap_dead start 2024-09-15T23:14:36.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 shutdown_connections 2024-09-15T23:14:36.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 >> 172.21.15.62:0/3836730389 conn(0x7fef480fc7d0 msgr2=0x7fef48105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:36.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 shutdown_connections 2024-09-15T23:14:36.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:36.895+0000 7fef4fe47640 1 -- 172.21.15.62:0/3836730389 wait complete. 2024-09-15T23:14:37.927 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:37 smithi062 ceph-mon[30949]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-09-15T23:14:38.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.010+0000 7f91d3b6f640 1 -- 172.21.15.62:0/658244698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91cc102ef0 msgr2=0x7f91cc10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.010+0000 7f91d3b6f640 1 --2- 172.21.15.62:0/658244698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91cc102ef0 0x7f91cc10da40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f91c4009900 tx=0x7f91c402a480 comp rx=0 tx=0).stop 2024-09-15T23:14:38.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.010+0000 7f91d3b6f640 1 -- 172.21.15.62:0/658244698 shutdown_connections 2024-09-15T23:14:38.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.010+0000 7f91d3b6f640 1 --2- 172.21.15.62:0/658244698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91cc102ef0 0x7f91cc10da40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.010+0000 7f91d3b6f640 1 --2- 172.21.15.62:0/658244698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91cc102570 0x7f91cc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.010+0000 7f91d3b6f640 1 --2- 172.21.15.62:0/658244698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91cc100e20 0x7f91cc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.010+0000 7f91d3b6f640 1 -- 172.21.15.62:0/658244698 >> 172.21.15.62:0/658244698 conn(0x7f91cc0fc7d0 msgr2=0x7f91cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.010+0000 7f91d3b6f640 1 -- 172.21.15.62:0/658244698 shutdown_connections 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.010+0000 7f91d3b6f640 1 -- 172.21.15.62:0/658244698 wait complete. 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d3b6f640 1 Processor -- start 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d3b6f640 1 -- start start 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d3b6f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91cc100e20 0x7f91cc199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d3b6f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91cc102570 0x7f91cc1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d3b6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91cc102ef0 0x7f91cc1a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d3b6f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc110f80 con 0x7f91cc100e20 2024-09-15T23:14:38.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d3b6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc110e00 con 0x7f91cc102ef0 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d3b6f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91cc111100 con 0x7f91cc102570 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d190b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91cc100e20 0x7f91cc199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d190b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91cc100e20 0x7f91cc199290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51678/0 (socket says 172.21.15.62:51678) 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d190b640 1 -- 172.21.15.62:0/1201321956 learned_addr learned my addr 172.21.15.62:0/1201321956 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d110a640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91cc102570 0x7f91cc1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d210c640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91cc102ef0 0x7f91cc1a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d190b640 1 -- 172.21.15.62:0/1201321956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91cc102570 msgr2=0x7f91cc1997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d190b640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91cc102570 0x7f91cc1997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d190b640 1 -- 172.21.15.62:0/1201321956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91cc102ef0 msgr2=0x7f91cc1a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d190b640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91cc102ef0 0x7f91cc1a4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.011+0000 7f91d190b640 1 -- 172.21.15.62:0/1201321956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91cc1a4850 con 0x7f91cc100e20 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.012+0000 7f91d190b640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91cc100e20 0x7f91cc199290 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f91c000d950 tx=0x7f91c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:38.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.012+0000 7f91beffd640 1 -- 172.21.15.62:0/1201321956 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c0017070 con 0x7f91cc100e20 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.012+0000 7f91beffd640 1 -- 172.21.15.62:0/1201321956 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91c0009c30 con 0x7f91cc100e20 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.012+0000 7f91beffd640 1 -- 172.21.15.62:0/1201321956 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91c000f6b0 con 0x7f91cc100e20 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.012+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91cc1a4a20 con 0x7f91cc100e20 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.012+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91cc1a4f10 con 0x7f91cc100e20 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.013+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f91cc107280 con 0x7f91cc100e20 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.014+0000 7f91beffd640 1 -- 172.21.15.62:0/1201321956 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f91c00109d0 con 0x7f91cc100e20 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.015+0000 7f91beffd640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91b80a32d0 0x7f91b80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.015+0000 7f91d110a640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91b80a32d0 0x7f91b80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.015+0000 7f91beffd640 1 -- 172.21.15.62:0/1201321956 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f91c00cfca0 con 0x7f91cc100e20 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.015+0000 7f91beffd640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] conn(0x7f91b80ae000 0x7f91b80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.015+0000 7f91beffd640 1 -- 172.21.15.62:0/1201321956 --> [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91b80b0b10 con 0x7f91b80ae000 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.015+0000 7f91beffd640 1 -- 172.21.15.62:0/1201321956 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f91c00d00f0 con 0x7f91cc100e20 2024-09-15T23:14:38.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.015+0000 7f91d210c640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] conn(0x7f91b80ae000 0x7f91b80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.015+0000 7f91d110a640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91b80a32d0 0x7f91b80a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f91cc06ab00 tx=0x7f91c8009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:38.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.015+0000 7f91d210c640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] conn(0x7f91b80ae000 0x7f91b80b0440 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:38.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.016+0000 7f91beffd640 1 -- 172.21.15.62:0/1201321956 <== osd.9 v2:172.21.15.62:6826/3552755977 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91b80b0b10 con 0x7f91b80ae000 2024-09-15T23:14:38.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:37 smithi067 ceph-mon[35665]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-09-15T23:14:38.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.030+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 --> [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f91cc1021b0 con 0x7f91b80ae000 2024-09-15T23:14:38.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.030+0000 7f91beffd640 1 -- 172.21.15.62:0/1201321956 <== osd.9 v2:172.21.15.62:6826/3552755977 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f91cc1021b0 con 0x7f91b80ae000 2024-09-15T23:14:38.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] conn(0x7f91b80ae000 msgr2=0x7f91b80b0440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] conn(0x7f91b80ae000 0x7f91b80b0440 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91b80a32d0 msgr2=0x7f91b80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91b80a32d0 0x7f91b80a5790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f91cc06ab00 tx=0x7f91c8009680 comp rx=0 tx=0).stop 2024-09-15T23:14:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91cc100e20 msgr2=0x7f91cc199290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 --2- 172.21.15.62:0/1201321956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91cc100e20 0x7f91cc199290 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f91c000d950 tx=0x7f91c000de10 comp rx=0 tx=0).stop 2024-09-15T23:14:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d210c640 1 -- 172.21.15.62:0/1201321956 reap_dead start 2024-09-15T23:14:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 shutdown_connections 2024-09-15T23:14:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 >> 172.21.15.62:0/1201321956 conn(0x7f91cc0fc7d0 msgr2=0x7f91cc106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 shutdown_connections 2024-09-15T23:14:38.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.031+0000 7f91d3b6f640 1 -- 172.21.15.62:0/1201321956 wait complete. 2024-09-15T23:14:38.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:37 smithi148 ceph-mon[35610]: pgmap v316: 129 pgs: 129 active+clean; 640 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.714+0000 7f42dffa2640 1 -- 172.21.15.62:0/1037937446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42d81036e0 msgr2=0x7f42d8107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.714+0000 7f42dffa2640 1 --2- 172.21.15.62:0/1037937446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42d81036e0 0x7f42d8107b90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009930 tx=0x7f42cc02a420 comp rx=0 tx=0).stop 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 -- 172.21.15.62:0/1037937446 shutdown_connections 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 --2- 172.21.15.62:0/1037937446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42d81082a0 0x7f42d810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 --2- 172.21.15.62:0/1037937446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42d81036e0 0x7f42d8107b90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 --2- 172.21.15.62:0/1037937446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42d8102d30 0x7f42d8103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 -- 172.21.15.62:0/1037937446 >> 172.21.15.62:0/1037937446 conn(0x7f42d80fc7d0 msgr2=0x7f42d80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 -- 172.21.15.62:0/1037937446 shutdown_connections 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 -- 172.21.15.62:0/1037937446 wait complete. 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 Processor -- start 2024-09-15T23:14:38.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 -- start start 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.715+0000 7f42dffa2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42d8102d30 0x7f42d81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42dffa2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42d81036e0 0x7f42d8198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42dffa2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42d81082a0 0x7f42d819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42dffa2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f42d81120a0 con 0x7f42d8102d30 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42dffa2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f42d8111f20 con 0x7f42d81082a0 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42dffa2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f42d8112220 con 0x7f42d81036e0 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42ddd3e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42d8102d30 0x7f42d81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42ddd3e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42d8102d30 0x7f42d81985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51708/0 (socket says 172.21.15.62:51708) 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42ddd3e640 1 -- 172.21.15.62:0/1969580605 learned_addr learned my addr 172.21.15.62:0/1969580605 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42dd53d640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42d81036e0 0x7f42d8198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42de53f640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42d81082a0 0x7f42d819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42ddd3e640 1 -- 172.21.15.62:0/1969580605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42d81036e0 msgr2=0x7f42d8198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42ddd3e640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42d81036e0 0x7f42d8198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42ddd3e640 1 -- 172.21.15.62:0/1969580605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42d81082a0 msgr2=0x7f42d819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42ddd3e640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42d81082a0 0x7f42d819ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42ddd3e640 1 -- 172.21.15.62:0/1969580605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d819d570 con 0x7f42d8102d30 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42de53f640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42d81082a0 0x7f42d819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42ddd3e640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42d8102d30 0x7f42d81985c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f42d000d950 tx=0x7f42d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42d6ffd640 1 -- 172.21.15.62:0/1969580605 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d0017070 con 0x7f42d8102d30 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42d6ffd640 1 -- 172.21.15.62:0/1969580605 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42d0009c30 con 0x7f42d8102d30 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d819d860 con 0x7f42d8102d30 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.716+0000 7f42d6ffd640 1 -- 172.21.15.62:0/1969580605 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d000f6b0 con 0x7f42d8102d30 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.717+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d81a5140 con 0x7f42d8102d30 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.717+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f42d8062980 con 0x7f42d8102d30 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.719+0000 7f42d6ffd640 1 -- 172.21.15.62:0/1969580605 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f42d00109d0 con 0x7f42d8102d30 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.719+0000 7f42d6ffd640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42c00a3350 0x7f42c00a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.720+0000 7f42dd53d640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42c00a3350 0x7f42c00a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.720+0000 7f42d6ffd640 1 -- 172.21.15.62:0/1969580605 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f42d00cfcb0 con 0x7f42d8102d30 2024-09-15T23:14:38.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.720+0000 7f42d6ffd640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] conn(0x7f42c00ae080 0x7f42c00b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:38.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.720+0000 7f42d6ffd640 1 -- 172.21.15.62:0/1969580605 --> [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42d001de50 con 0x7f42c00ae080 2024-09-15T23:14:38.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.720+0000 7f42d6ffd640 1 -- 172.21.15.62:0/1969580605 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f42d00d0100 con 0x7f42d8102d30 2024-09-15T23:14:38.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.720+0000 7f42de53f640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] conn(0x7f42c00ae080 0x7f42c00b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:38.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.720+0000 7f42dd53d640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42c00a3350 0x7f42c00a5810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f42cc005940 tx=0x7f42cc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:38.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.720+0000 7f42de53f640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] conn(0x7f42c00ae080 0x7f42c00b04c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:38.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.721+0000 7f42d6ffd640 1 -- 172.21.15.62:0/1969580605 <== osd.0 v2:172.21.15.62:6802/2006762506 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42d001de50 con 0x7f42c00ae080 2024-09-15T23:14:38.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.735+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 --> [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f42d8062e00 con 0x7f42c00ae080 2024-09-15T23:14:38.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.735+0000 7f42d6ffd640 1 -- 172.21.15.62:0/1969580605 <== osd.0 v2:172.21.15.62:6802/2006762506 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f42d8062e00 con 0x7f42c00ae080 2024-09-15T23:14:38.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] conn(0x7f42c00ae080 msgr2=0x7f42c00b04c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] conn(0x7f42c00ae080 0x7f42c00b04c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42c00a3350 msgr2=0x7f42c00a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42c00a3350 0x7f42c00a5810 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f42cc005940 tx=0x7f42cc0058d0 comp rx=0 tx=0).stop 2024-09-15T23:14:38.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42d8102d30 msgr2=0x7f42d81985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:38.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 --2- 172.21.15.62:0/1969580605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42d8102d30 0x7f42d81985c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f42d000d950 tx=0x7f42d000de10 comp rx=0 tx=0).stop 2024-09-15T23:14:38.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42de53f640 1 -- 172.21.15.62:0/1969580605 reap_dead start 2024-09-15T23:14:38.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 shutdown_connections 2024-09-15T23:14:38.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 >> 172.21.15.62:0/1969580605 conn(0x7f42d80fc7d0 msgr2=0x7f42d8105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:38.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 shutdown_connections 2024-09-15T23:14:38.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.736+0000 7f42dffa2640 1 -- 172.21.15.62:0/1969580605 wait complete. 2024-09-15T23:14:39.428 INFO:teuthology.orchestra.run.smithi062.stdout:34359738451 2024-09-15T23:14:39.428 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-15T23:14:39.667 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:39 smithi062 ceph-mon[30949]: pgmap v317: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-15T23:14:40.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:39 smithi067 ceph-mon[35665]: pgmap v317: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-15T23:14:40.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:39 smithi148 ceph-mon[35610]: pgmap v317: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-09-15T23:14:40.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.851+0000 7f2526899640 1 -- 172.21.15.62:0/1933917243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2520102da0 msgr2=0x7f2520103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.851+0000 7f2526899640 1 --2- 172.21.15.62:0/1933917243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2520102da0 0x7f2520103180 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f2510009900 tx=0x7f251002a480 comp rx=0 tx=0).stop 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 -- 172.21.15.62:0/1933917243 shutdown_connections 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 --2- 172.21.15.62:0/1933917243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f252010c180 0x7f252010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 --2- 172.21.15.62:0/1933917243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25201036c0 0x7f252010bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 --2- 172.21.15.62:0/1933917243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2520102da0 0x7f2520103180 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 -- 172.21.15.62:0/1933917243 >> 172.21.15.62:0/1933917243 conn(0x7f25200fc7a0 msgr2=0x7f25200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 -- 172.21.15.62:0/1933917243 shutdown_connections 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 -- 172.21.15.62:0/1933917243 wait complete. 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 Processor -- start 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 -- start start 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2520102da0 0x7f2520198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.852+0000 7f2526899640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25201036c0 0x7f2520198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2526899640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f252010c180 0x7f252019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2526899640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2520111fa0 con 0x7f252010c180 2024-09-15T23:14:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2526899640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2520111e20 con 0x7f25201036c0 2024-09-15T23:14:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2526899640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2520112120 con 0x7f2520102da0 2024-09-15T23:14:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2524e36640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f252010c180 0x7f252019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2524e36640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f252010c180 0x7f252019cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51724/0 (socket says 172.21.15.62:51724) 2024-09-15T23:14:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2524e36640 1 -- 172.21.15.62:0/394893959 learned_addr learned my addr 172.21.15.62:0/394893959 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f251ffff640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2520102da0 0x7f2520198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2517fff640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25201036c0 0x7f2520198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2524e36640 1 -- 172.21.15.62:0/394893959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2520102da0 msgr2=0x7f2520198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2524e36640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2520102da0 0x7f2520198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2524e36640 1 -- 172.21.15.62:0/394893959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25201036c0 msgr2=0x7f2520198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2524e36640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25201036c0 0x7f2520198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2524e36640 1 -- 172.21.15.62:0/394893959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252019d4d0 con 0x7f252010c180 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2517fff640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25201036c0 0x7f2520198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2524e36640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f252010c180 0x7f252019cdf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f251800eaa0 tx=0x7f251800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f251dffb640 1 -- 172.21.15.62:0/394893959 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251800cdb0 con 0x7f252010c180 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f251dffb640 1 -- 172.21.15.62:0/394893959 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2518005020 con 0x7f252010c180 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252019d7c0 con 0x7f252010c180 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f251dffb640 1 -- 172.21.15.62:0/394893959 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25180116b0 con 0x7f252010c180 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.853+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25201a50a0 con 0x7f252010c180 2024-09-15T23:14:40.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.854+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2520062900 con 0x7f252010c180 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.855+0000 7f251dffb640 1 -- 172.21.15.62:0/394893959 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2518005560 con 0x7f252010c180 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.856+0000 7f251dffb640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a3350 0x7f250c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.856+0000 7f251dffb640 1 -- 172.21.15.62:0/394893959 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f25180cff00 con 0x7f252010c180 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.856+0000 7f251ffff640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a3350 0x7f250c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.856+0000 7f251dffb640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] conn(0x7f250c0ae080 0x7f250c0b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.856+0000 7f251dffb640 1 -- 172.21.15.62:0/394893959 --> [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2518018ec0 con 0x7f250c0ae080 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.856+0000 7f251dffb640 1 -- 172.21.15.62:0/394893959 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f25180d25c0 con 0x7f252010c180 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.857+0000 7f251ffff640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a3350 0x7f250c0a5810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2510009230 tx=0x7f25100058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.857+0000 7f2517fff640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] conn(0x7f250c0ae080 0x7f250c0b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.857+0000 7f2517fff640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] conn(0x7f250c0ae080 0x7f250c0b04c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.858+0000 7f251dffb640 1 -- 172.21.15.62:0/394893959 <== osd.8 v2:172.21.15.148:6816/381356822 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2518018ec0 con 0x7f250c0ae080 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.872+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 --> [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2520062d80 con 0x7f250c0ae080 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.872+0000 7f251dffb640 1 -- 172.21.15.62:0/394893959 <== osd.8 v2:172.21.15.148:6816/381356822 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2520062d80 con 0x7f250c0ae080 2024-09-15T23:14:40.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 >> [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] conn(0x7f250c0ae080 msgr2=0x7f250c0b04c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] conn(0x7f250c0ae080 0x7f250c0b04c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a3350 msgr2=0x7f250c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a3350 0x7f250c0a5810 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2510009230 tx=0x7f25100058d0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f252010c180 msgr2=0x7f252019cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 --2- 172.21.15.62:0/394893959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f252010c180 0x7f252019cdf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f251800eaa0 tx=0x7f251800ef60 comp rx=0 tx=0).stop 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2524e36640 1 -- 172.21.15.62:0/394893959 reap_dead start 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 shutdown_connections 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 >> 172.21.15.62:0/394893959 conn(0x7f25200fc7a0 msgr2=0x7f25200fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 shutdown_connections 2024-09-15T23:14:40.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:39.873+0000 7f2526899640 1 -- 172.21.15.62:0/394893959 wait complete. 2024-09-15T23:14:40.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.244+0000 7f834bf53640 1 -- 172.21.15.62:0/1474025025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8344102d40 msgr2=0x7f8344103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.244+0000 7f834bf53640 1 --2- 172.21.15.62:0/1474025025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8344102d40 0x7f8344103120 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8340009900 tx=0x7f834002a480 comp rx=0 tx=0).stop 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.244+0000 7f834bf53640 1 -- 172.21.15.62:0/1474025025 shutdown_connections 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.244+0000 7f834bf53640 1 --2- 172.21.15.62:0/1474025025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f834410c170 0x7f834410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.244+0000 7f834bf53640 1 --2- 172.21.15.62:0/1474025025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8344103660 0x7f834410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.244+0000 7f834bf53640 1 --2- 172.21.15.62:0/1474025025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8344102d40 0x7f8344103120 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.244+0000 7f834bf53640 1 -- 172.21.15.62:0/1474025025 >> 172.21.15.62:0/1474025025 conn(0x7f83440fc7a0 msgr2=0x7f83440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 -- 172.21.15.62:0/1474025025 shutdown_connections 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 -- 172.21.15.62:0/1474025025 wait complete. 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 Processor -- start 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 -- start start 2024-09-15T23:14:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8344102d40 0x7f8344198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8344103660 0x7f8344198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834410c170 0x7f834419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8344111f70 con 0x7f834410c170 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8344111df0 con 0x7f8344103660 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834bf53640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f83441120f0 con 0x7f8344102d40 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.245+0000 7f834a4f0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834410c170 0x7f834419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834a4f0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834410c170 0x7f834419cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51700/0 (socket says 172.21.15.62:51700) 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834a4f0640 1 -- 172.21.15.62:0/1298340866 learned_addr learned my addr 172.21.15.62:0/1298340866 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f83494ee640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8344103660 0x7f8344198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f8349cef640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8344102d40 0x7f8344198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834a4f0640 1 -- 172.21.15.62:0/1298340866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8344102d40 msgr2=0x7f8344198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834a4f0640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8344102d40 0x7f8344198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834a4f0640 1 -- 172.21.15.62:0/1298340866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8344103660 msgr2=0x7f8344198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834a4f0640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8344103660 0x7f8344198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834a4f0640 1 -- 172.21.15.62:0/1298340866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834419d3f0 con 0x7f834410c170 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f8349cef640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8344102d40 0x7f8344198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834a4f0640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834410c170 0x7f834419cd10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f833c00eaa0 tx=0x7f833c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f8336ffd640 1 -- 172.21.15.62:0/1298340866 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c00cdb0 con 0x7f834410c170 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f8336ffd640 1 -- 172.21.15.62:0/1298340866 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c005020 con 0x7f834410c170 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f834419d6e0 con 0x7f834410c170 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f8336ffd640 1 -- 172.21.15.62:0/1298340866 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c0116b0 con 0x7f834410c170 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.246+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83441a4fc0 con 0x7f834410c170 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.247+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f83440628c0 con 0x7f834410c170 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.249+0000 7f8336ffd640 1 -- 172.21.15.62:0/1298340866 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f833c005560 con 0x7f834410c170 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.249+0000 7f8336ffd640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83300a3350 0x7f83300a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.249+0000 7f8349cef640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83300a3350 0x7f83300a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.249+0000 7f8336ffd640 1 -- 172.21.15.62:0/1298340866 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8802+0+0 (secure 0 0 0) 0x7f833c0d0210 con 0x7f834410c170 2024-09-15T23:14:40.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.249+0000 7f8336ffd640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] conn(0x7f83300ae080 0x7f83300b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:40.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.249+0000 7f8336ffd640 1 -- 172.21.15.62:0/1298340866 --> [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f833c014ec0 con 0x7f83300ae080 2024-09-15T23:14:40.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.249+0000 7f8336ffd640 1 -- 172.21.15.62:0/1298340866 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) ==== 24+0+0 (secure 0 0 0) 0x7f833c0d0660 con 0x7f834410c170 2024-09-15T23:14:40.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.250+0000 7f83494ee640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] conn(0x7f83300ae080 0x7f83300b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:40.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.250+0000 7f8349cef640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83300a3350 0x7f83300a5810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8340005ab0 tx=0x7f8340005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.250+0000 7f83494ee640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] conn(0x7f83300ae080 0x7f83300b04c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.251+0000 7f8336ffd640 1 -- 172.21.15.62:0/1298340866 <== osd.3 v2:172.21.15.62:6810/2959993284 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f833c014ec0 con 0x7f83300ae080 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.264+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 --> [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8344062d40 con 0x7f83300ae080 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.264+0000 7f8336ffd640 1 -- 172.21.15.62:0/1298340866 <== osd.3 v2:172.21.15.62:6810/2959993284 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8344062d40 con 0x7f83300ae080 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] conn(0x7f83300ae080 msgr2=0x7f83300b04c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] conn(0x7f83300ae080 0x7f83300b04c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83300a3350 msgr2=0x7f83300a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83300a3350 0x7f83300a5810 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8340005ab0 tx=0x7f8340005a20 comp rx=0 tx=0).stop 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834410c170 msgr2=0x7f834419cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 --2- 172.21.15.62:0/1298340866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f834410c170 0x7f834419cd10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f833c00eaa0 tx=0x7f833c00ef60 comp rx=0 tx=0).stop 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834a4f0640 1 -- 172.21.15.62:0/1298340866 reap_dead start 2024-09-15T23:14:40.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 shutdown_connections 2024-09-15T23:14:40.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 >> 172.21.15.62:0/1298340866 conn(0x7f83440fc7a0 msgr2=0x7f83440fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:40.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 shutdown_connections 2024-09-15T23:14:40.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:38.265+0000 7f834bf53640 1 -- 172.21.15.62:0/1298340866 wait complete. 2024-09-15T23:14:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-09-15T23:14:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-09-15T23:14:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-09-15T23:14:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-09-15T23:14:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-15T23:14:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-09-15T23:14:40.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2024-09-15T23:14:40.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-09-15T23:14:40.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-09-15T23:14:41.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-09-15T23:14:41.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-09-15T23:14:41.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-09-15T23:14:41.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-09-15T23:14:41.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-15T23:14:41.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-09-15T23:14:41.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2024-09-15T23:14:41.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-09-15T23:14:41.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-09-15T23:14:41.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-09-15T23:14:41.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-09-15T23:14:41.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-09-15T23:14:41.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2024-09-15T23:14:41.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-09-15T23:14:41.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-09-15T23:14:41.041 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]: dispatch 2024-09-15T23:14:41.041 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-09-15T23:14:41.041 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-09-15T23:14:41.620 INFO:teuthology.orchestra.run.smithi062.stdout:193273528372 2024-09-15T23:14:41.620 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-15T23:14:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: pgmap v318: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-09-15T23:14:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-09-15T23:14:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-09-15T23:14:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-09-15T23:14:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-09-15T23:14:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-15T23:14:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-09-15T23:14:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2024-09-15T23:14:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-09-15T23:14:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-15T23:14:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:41 smithi062 ceph-mon[30949]: osdmap e89: 12 total, 12 up, 12 in 2024-09-15T23:14:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: pgmap v318: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-09-15T23:14:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-09-15T23:14:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-09-15T23:14:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-09-15T23:14:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-09-15T23:14:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-15T23:14:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-09-15T23:14:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2024-09-15T23:14:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-09-15T23:14:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-15T23:14:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:41 smithi067 ceph-mon[35665]: osdmap e89: 12 total, 12 up, 12 in 2024-09-15T23:14:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: pgmap v318: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-09-15T23:14:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-09-15T23:14:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-09-15T23:14:42.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-09-15T23:14:42.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2024-09-15T23:14:42.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-09-15T23:14:42.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-09-15T23:14:42.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 10]}]': finished 2024-09-15T23:14:42.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-09-15T23:14:42.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-09-15T23:14:42.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:41 smithi148 ceph-mon[35610]: osdmap e89: 12 total, 12 up, 12 in 2024-09-15T23:14:42.183 INFO:teuthology.orchestra.run.smithi062.stdout:107374182467 2024-09-15T23:14:42.183 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-15T23:14:42.599 INFO:teuthology.orchestra.run.smithi062.stdout:77309411400 2024-09-15T23:14:42.599 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-15T23:14:42.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.362+0000 7fc31000b640 1 -- 172.21.15.62:0/2214814484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc308100e20 msgr2=0x7fc308101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:42.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.362+0000 7fc31000b640 1 --2- 172.21.15.62:0/2214814484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc308100e20 0x7fc308101200 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc304009900 tx=0x7fc30402a480 comp rx=0 tx=0).stop 2024-09-15T23:14:42.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.362+0000 7fc31000b640 1 -- 172.21.15.62:0/2214814484 shutdown_connections 2024-09-15T23:14:42.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.362+0000 7fc31000b640 1 --2- 172.21.15.62:0/2214814484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc308102ef0 0x7fc30810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.362+0000 7fc31000b640 1 --2- 172.21.15.62:0/2214814484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc308102570 0x7fc3081029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.362+0000 7fc31000b640 1 --2- 172.21.15.62:0/2214814484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc308100e20 0x7fc308101200 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.362+0000 7fc31000b640 1 -- 172.21.15.62:0/2214814484 >> 172.21.15.62:0/2214814484 conn(0x7fc3080fc7d0 msgr2=0x7fc3080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:42.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.362+0000 7fc31000b640 1 -- 172.21.15.62:0/2214814484 shutdown_connections 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.362+0000 7fc31000b640 1 -- 172.21.15.62:0/2214814484 wait complete. 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc31000b640 1 Processor -- start 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc31000b640 1 -- start start 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc31000b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc308100e20 0x7fc308198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc31000b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc308102570 0x7fc308198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc31000b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc308102ef0 0x7fc30819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc31000b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc308110f80 con 0x7fc308102570 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc31000b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc308110e00 con 0x7fc308102ef0 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc31000b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc308111100 con 0x7fc308100e20 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc30d5a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc308102570 0x7fc308198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:42.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc30d5a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc308102570 0x7fc308198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51742/0 (socket says 172.21.15.62:51742) 2024-09-15T23:14:42.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.363+0000 7fc30d5a6640 1 -- 172.21.15.62:0/3260374747 learned_addr learned my addr 172.21.15.62:0/3260374747 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:42.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc30dda7640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc308100e20 0x7fc308198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:42.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc30e5a8640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc308102ef0 0x7fc30819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:42.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc30d5a6640 1 -- 172.21.15.62:0/3260374747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc308100e20 msgr2=0x7fc308198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:42.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc30d5a6640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc308100e20 0x7fc308198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc30d5a6640 1 -- 172.21.15.62:0/3260374747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc308102ef0 msgr2=0x7fc30819ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:42.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc30d5a6640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc308102ef0 0x7fc30819ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc30d5a6640 1 -- 172.21.15.62:0/3260374747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30819d4f0 con 0x7fc308102570 2024-09-15T23:14:42.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc30d5a6640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc308102570 0x7fc308198a80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800d770 tx=0x7fc2f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:42.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc2feffd640 1 -- 172.21.15.62:0/3260374747 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8017070 con 0x7fc308102570 2024-09-15T23:14:42.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc2feffd640 1 -- 172.21.15.62:0/3260374747 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2f8005500 con 0x7fc308102570 2024-09-15T23:14:42.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc2feffd640 1 -- 172.21.15.62:0/3260374747 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8004e00 con 0x7fc308102570 2024-09-15T23:14:42.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30819d7e0 con 0x7fc308102570 2024-09-15T23:14:42.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.364+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3081a50c0 con 0x7fc308102570 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.365+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3080fe290 con 0x7fc308102570 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.366+0000 7fc2feffd640 1 -- 172.21.15.62:0/3260374747 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc2f8004fa0 con 0x7fc308102570 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.367+0000 7fc2feffd640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2f40a32d0 0x7fc2f40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.367+0000 7fc30dda7640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2f40a32d0 0x7fc2f40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.367+0000 7fc2feffd640 1 -- 172.21.15.62:0/3260374747 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9071+0+0 (secure 0 0 0) 0x7fc2f80cfd00 con 0x7fc308102570 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.367+0000 7fc2feffd640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] conn(0x7fc2f40ae410 0x7fc2f40b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.367+0000 7fc2feffd640 1 -- 172.21.15.62:0/3260374747 --> [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2f8001d30 con 0x7fc2f40ae410 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.367+0000 7fc2feffd640 1 -- 172.21.15.62:0/3260374747 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fc2f80d0150 con 0x7fc308102570 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.368+0000 7fc30dda7640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2f40a32d0 0x7fc2f40a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc304005ab0 tx=0x7fc304005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.368+0000 7fc30e5a8640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] conn(0x7fc2f40ae410 0x7fc2f40b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.368+0000 7fc30e5a8640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] conn(0x7fc2f40ae410 0x7fc2f40b08b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:42.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.369+0000 7fc2feffd640 1 -- 172.21.15.62:0/3260374747 <== osd.1 v2:172.21.15.67:6800/1021530759 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc2f8001d30 con 0x7fc2f40ae410 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.383+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 --> [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc3081021b0 con 0x7fc2f40ae410 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.383+0000 7fc2feffd640 1 -- 172.21.15.62:0/3260374747 <== osd.1 v2:172.21.15.67:6800/1021530759 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc3081021b0 con 0x7fc2f40ae410 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.383+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 >> [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] conn(0x7fc2f40ae410 msgr2=0x7fc2f40b08b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.383+0000 7fc31000b640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] conn(0x7fc2f40ae410 0x7fc2f40b08b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.383+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2f40a32d0 msgr2=0x7fc2f40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.383+0000 7fc31000b640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2f40a32d0 0x7fc2f40a5790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc304005ab0 tx=0x7fc304005a20 comp rx=0 tx=0).stop 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.383+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc308102570 msgr2=0x7fc308198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.383+0000 7fc31000b640 1 --2- 172.21.15.62:0/3260374747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc308102570 0x7fc308198a80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800d770 tx=0x7fc2f800dc30 comp rx=0 tx=0).stop 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.384+0000 7fc30e5a8640 1 -- 172.21.15.62:0/3260374747 reap_dead start 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.384+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 shutdown_connections 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.384+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 >> 172.21.15.62:0/3260374747 conn(0x7fc3080fc7d0 msgr2=0x7fc308105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.384+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 shutdown_connections 2024-09-15T23:14:42.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:41.384+0000 7fc31000b640 1 -- 172.21.15.62:0/3260374747 wait complete. 2024-09-15T23:14:42.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:42 smithi062 ceph-mon[30949]: osdmap e90: 12 total, 12 up, 12 in 2024-09-15T23:14:42.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:42 smithi062 ceph-mon[30949]: pgmap v321: 129 pgs: 3 peering, 126 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s 2024-09-15T23:14:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:42 smithi067 ceph-mon[35665]: osdmap e90: 12 total, 12 up, 12 in 2024-09-15T23:14:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:42 smithi067 ceph-mon[35665]: pgmap v321: 129 pgs: 3 peering, 126 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s 2024-09-15T23:14:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:42 smithi148 ceph-mon[35610]: osdmap e90: 12 total, 12 up, 12 in 2024-09-15T23:14:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:42 smithi148 ceph-mon[35610]: pgmap v321: 129 pgs: 3 peering, 126 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s 2024-09-15T23:14:43.542 INFO:teuthology.orchestra.run.smithi062.stdout:274877906981 2024-09-15T23:14:43.542 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-15T23:14:43.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.724+0000 7ff993521640 1 -- 172.21.15.62:0/2252553230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff98c10c180 msgr2=0x7ff98c10e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.724+0000 7ff993521640 1 --2- 172.21.15.62:0/2252553230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff98c10c180 0x7ff98c10e620 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff984009ca0 tx=0x7ff98402a950 comp rx=0 tx=0).stop 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.724+0000 7ff993521640 1 -- 172.21.15.62:0/2252553230 shutdown_connections 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.724+0000 7ff993521640 1 --2- 172.21.15.62:0/2252553230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff98c10c180 0x7ff98c10e620 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.724+0000 7ff993521640 1 --2- 172.21.15.62:0/2252553230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c103670 0x7ff98c10bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.724+0000 7ff993521640 1 --2- 172.21.15.62:0/2252553230 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff98c102d50 0x7ff98c103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.724+0000 7ff993521640 1 -- 172.21.15.62:0/2252553230 >> 172.21.15.62:0/2252553230 conn(0x7ff98c0fc7d0 msgr2=0x7ff98c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.724+0000 7ff993521640 1 -- 172.21.15.62:0/2252553230 shutdown_connections 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.724+0000 7ff993521640 1 -- 172.21.15.62:0/2252553230 wait complete. 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff993521640 1 Processor -- start 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff993521640 1 -- start start 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff993521640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff98c102d50 0x7ff98c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff993521640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c103670 0x7ff98c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:43.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff993521640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff98c10c180 0x7ff98c19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff993521640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c111f80 con 0x7ff98c102d50 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff993521640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c111e00 con 0x7ff98c103670 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff993521640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c112100 con 0x7ff98c10c180 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff9912bd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff98c102d50 0x7ff98c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff9912bd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff98c102d50 0x7ff98c198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37998/0 (socket says 172.21.15.62:37998) 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff9912bd640 1 -- 172.21.15.62:0/1297943729 learned_addr learned my addr 172.21.15.62:0/1297943729 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.725+0000 7ff990abc640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c103670 0x7ff98c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff991abe640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff98c10c180 0x7ff98c19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff9912bd640 1 -- 172.21.15.62:0/1297943729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff98c10c180 msgr2=0x7ff98c19ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff9912bd640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff98c10c180 0x7ff98c19ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff9912bd640 1 -- 172.21.15.62:0/1297943729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c103670 msgr2=0x7ff98c198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:43.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff9912bd640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c103670 0x7ff98c198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff9912bd640 1 -- 172.21.15.62:0/1297943729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98c19d3d0 con 0x7ff98c102d50 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff991abe640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff98c10c180 0x7ff98c19ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff990abc640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff98c103670 0x7ff98c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff9912bd640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff98c102d50 0x7ff98c198420 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff98800e9f0 tx=0x7ff98800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff9827fc640 1 -- 172.21.15.62:0/1297943729 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98800cd80 con 0x7ff98c102d50 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98c19d6c0 con 0x7ff98c102d50 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff9827fc640 1 -- 172.21.15.62:0/1297943729 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff988005500 con 0x7ff98c102d50 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff9827fc640 1 -- 172.21.15.62:0/1297943729 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff988010690 con 0x7ff98c102d50 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.726+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff98c1a4fa0 con 0x7ff98c102d50 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.727+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff98c062980 con 0x7ff98c102d50 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.729+0000 7ff9827fc640 1 -- 172.21.15.62:0/1297943729 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff9880046c0 con 0x7ff98c102d50 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.729+0000 7ff9827fc640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9780a32d0 0x7ff9780a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:43.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.729+0000 7ff9827fc640 1 -- 172.21.15.62:0/1297943729 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7ff9880d06d0 con 0x7ff98c102d50 2024-09-15T23:14:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.729+0000 7ff990abc640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9780a32d0 0x7ff9780a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.730+0000 7ff9827fc640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] conn(0x7ff9780ae420 0x7ff9780b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.730+0000 7ff9827fc640 1 -- 172.21.15.62:0/1297943729 --> [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff988005350 con 0x7ff9780ae420 2024-09-15T23:14:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.730+0000 7ff9827fc640 1 -- 172.21.15.62:0/1297943729 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff9880d0b20 con 0x7ff98c102d50 2024-09-15T23:14:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.730+0000 7ff991abe640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] conn(0x7ff9780ae420 0x7ff9780b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.730+0000 7ff990abc640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9780a32d0 0x7ff9780a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0fed60 tx=0x7ff97c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.730+0000 7ff991abe640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] conn(0x7ff9780ae420 0x7ff9780b08c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:43.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.731+0000 7ff9827fc640 1 -- 172.21.15.62:0/1297943729 <== osd.7 v2:172.21.15.67:6816/2000628718 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff988005350 con 0x7ff9780ae420 2024-09-15T23:14:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.746+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 --> [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff98c062e00 con 0x7ff9780ae420 2024-09-15T23:14:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.747+0000 7ff9827fc640 1 -- 172.21.15.62:0/1297943729 <== osd.7 v2:172.21.15.67:6816/2000628718 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff98c062e00 con 0x7ff9780ae420 2024-09-15T23:14:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.747+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 >> [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] conn(0x7ff9780ae420 msgr2=0x7ff9780b08c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.747+0000 7ff993521640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] conn(0x7ff9780ae420 0x7ff9780b08c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.747+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9780a32d0 msgr2=0x7ff9780a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:43.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.747+0000 7ff993521640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9780a32d0 0x7ff9780a5790 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff98c0fed60 tx=0x7ff97c005f90 comp rx=0 tx=0).stop 2024-09-15T23:14:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.747+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff98c102d50 msgr2=0x7ff98c198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.747+0000 7ff993521640 1 --2- 172.21.15.62:0/1297943729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff98c102d50 0x7ff98c198420 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff98800e9f0 tx=0x7ff98800edf0 comp rx=0 tx=0).stop 2024-09-15T23:14:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.747+0000 7ff991abe640 1 -- 172.21.15.62:0/1297943729 reap_dead start 2024-09-15T23:14:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.748+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 shutdown_connections 2024-09-15T23:14:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.748+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 >> 172.21.15.62:0/1297943729 conn(0x7ff98c0fc7d0 msgr2=0x7ff98c0fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.748+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 shutdown_connections 2024-09-15T23:14:43.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:43.748+0000 7ff993521640 1 -- 172.21.15.62:0/1297943729 wait complete. 2024-09-15T23:14:44.142 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:43 smithi062 ceph-mon[30949]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-15T23:14:44.170 INFO:teuthology.orchestra.run.smithi062.stdout:158913790008 2024-09-15T23:14:44.171 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-15T23:14:44.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:43 smithi067 ceph-mon[35665]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-15T23:14:44.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:43 smithi148 ceph-mon[35610]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-15T23:14:44.458 INFO:teuthology.orchestra.run.smithi062.stdout:55834574926 2024-09-15T23:14:44.458 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.628+0000 7f4d54f68640 1 -- 172.21.15.62:0/1454235013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d50102570 msgr2=0x7f4d501029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.628+0000 7f4d54f68640 1 --2- 172.21.15.62:0/1454235013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d50102570 0x7f4d501029b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009900 tx=0x7f4d3802a480 comp rx=0 tx=0).stop 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.629+0000 7f4d54f68640 1 -- 172.21.15.62:0/1454235013 shutdown_connections 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.629+0000 7f4d54f68640 1 --2- 172.21.15.62:0/1454235013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d50102ef0 0x7f4d5010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.629+0000 7f4d54f68640 1 --2- 172.21.15.62:0/1454235013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d50102570 0x7f4d501029b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.629+0000 7f4d54f68640 1 --2- 172.21.15.62:0/1454235013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d50100e20 0x7f4d50101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.629+0000 7f4d54f68640 1 -- 172.21.15.62:0/1454235013 >> 172.21.15.62:0/1454235013 conn(0x7f4d500fc7d0 msgr2=0x7f4d500febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.629+0000 7f4d54f68640 1 -- 172.21.15.62:0/1454235013 shutdown_connections 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.629+0000 7f4d54f68640 1 -- 172.21.15.62:0/1454235013 wait complete. 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.629+0000 7f4d54f68640 1 Processor -- start 2024-09-15T23:14:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d54f68640 1 -- start start 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d54f68640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d50100e20 0x7f4d501973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d54f68640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d50102570 0x7f4d50197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d54f68640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d50102ef0 0x7f4d5019bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d54f68640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4d50110f80 con 0x7f4d50100e20 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d54f68640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d50110e00 con 0x7f4d50102570 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d54f68640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4d50111100 con 0x7f4d50102ef0 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d50100e20 0x7f4d501973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d50100e20 0x7f4d501973f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38020/0 (socket says 172.21.15.62:38020) 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4e59c640 1 -- 172.21.15.62:0/2124097804 learned_addr learned my addr 172.21.15.62:0/2124097804 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4dd9b640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d50102570 0x7f4d50197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4e59c640 1 -- 172.21.15.62:0/2124097804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d50102ef0 msgr2=0x7f4d5019bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4e59c640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d50102ef0 0x7f4d5019bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:44.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4e59c640 1 -- 172.21.15.62:0/2124097804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d50102570 msgr2=0x7f4d50197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4e59c640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d50102570 0x7f4d50197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4e59c640 1 -- 172.21.15.62:0/2124097804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5019c3a0 con 0x7f4d50100e20 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4dd9b640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d50102570 0x7f4d50197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.630+0000 7f4d4e59c640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d50100e20 0x7f4d501973f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000cce0 tx=0x7f4d4000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.631+0000 7f4d477fe640 1 -- 172.21.15.62:0/2124097804 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40004b20 con 0x7f4d50100e20 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.631+0000 7f4d477fe640 1 -- 172.21.15.62:0/2124097804 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d40004cc0 con 0x7f4d50100e20 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.631+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5019c690 con 0x7f4d50100e20 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.631+0000 7f4d477fe640 1 -- 172.21.15.62:0/2124097804 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4000f7d0 con 0x7f4d50100e20 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.631+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d501a3f70 con 0x7f4d50100e20 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.631+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d50062980 con 0x7f4d50100e20 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.633+0000 7f4d477fe640 1 -- 172.21.15.62:0/2124097804 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4d40004e70 con 0x7f4d50100e20 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.634+0000 7f4d477fe640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4d3c0a3350 0x7f4d3c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:44.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.634+0000 7f4d4dd9b640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4d3c0a3350 0x7f4d3c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:44.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.634+0000 7f4d477fe640 1 -- 172.21.15.62:0/2124097804 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f4d400d0280 con 0x7f4d50100e20 2024-09-15T23:14:44.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.634+0000 7f4d477fe640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] conn(0x7f4d3c0ae4a0 0x7f4d3c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:44.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.634+0000 7f4d477fe640 1 -- 172.21.15.62:0/2124097804 --> [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d3c0b1010 con 0x7f4d3c0ae4a0 2024-09-15T23:14:44.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.634+0000 7f4d477fe640 1 -- 172.21.15.62:0/2124097804 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f4d400d06d0 con 0x7f4d50100e20 2024-09-15T23:14:44.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.634+0000 7f4d4dd9b640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4d3c0a3350 0x7f4d3c0a5810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4d500ffe00 tx=0x7f4d38005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:44.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.634+0000 7f4d4ed9d640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] conn(0x7f4d3c0ae4a0 0x7f4d3c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:44.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.635+0000 7f4d4ed9d640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] conn(0x7f4d3c0ae4a0 0x7f4d3c0b0940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:44.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.636+0000 7f4d477fe640 1 -- 172.21.15.62:0/2124097804 <== osd.11 v2:172.21.15.148:6824/759280989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d3c0b1010 con 0x7f4d3c0ae4a0 2024-09-15T23:14:44.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.650+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 --> [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d50062d80 con 0x7f4d3c0ae4a0 2024-09-15T23:14:44.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.651+0000 7f4d477fe640 1 -- 172.21.15.62:0/2124097804 <== osd.11 v2:172.21.15.148:6824/759280989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4d50062d80 con 0x7f4d3c0ae4a0 2024-09-15T23:14:44.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.651+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 >> [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] conn(0x7f4d3c0ae4a0 msgr2=0x7f4d3c0b0940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:44.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.651+0000 7f4d54f68640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] conn(0x7f4d3c0ae4a0 0x7f4d3c0b0940 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:44.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.651+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4d3c0a3350 msgr2=0x7f4d3c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:44.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.651+0000 7f4d54f68640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4d3c0a3350 0x7f4d3c0a5810 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4d500ffe00 tx=0x7f4d38005a20 comp rx=0 tx=0).stop 2024-09-15T23:14:44.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.651+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d50100e20 msgr2=0x7f4d501973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:44.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.651+0000 7f4d54f68640 1 --2- 172.21.15.62:0/2124097804 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d50100e20 0x7f4d501973f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000cce0 tx=0x7f4d4000a620 comp rx=0 tx=0).stop 2024-09-15T23:14:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.651+0000 7f4d4ed9d640 1 -- 172.21.15.62:0/2124097804 reap_dead start 2024-09-15T23:14:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.652+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 shutdown_connections 2024-09-15T23:14:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.652+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 >> 172.21.15.62:0/2124097804 conn(0x7f4d500fc7d0 msgr2=0x7f4d500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.652+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 shutdown_connections 2024-09-15T23:14:44.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:44.652+0000 7f4d54f68640 1 -- 172.21.15.62:0/2124097804 wait complete. 2024-09-15T23:14:45.066 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:44 smithi062 ceph-mon[30949]: pgmap v322: 129 pgs: 3 peering, 126 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 383 B/s wr, 5 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:45.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:44 smithi067 ceph-mon[35665]: pgmap v322: 129 pgs: 3 peering, 126 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 383 B/s wr, 5 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:45.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:44 smithi148 ceph-mon[35610]: pgmap v322: 129 pgs: 3 peering, 126 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 383 B/s wr, 5 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:45.407 INFO:teuthology.orchestra.run.smithi062.stdout:137438953533 2024-09-15T23:14:45.407 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-15T23:14:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 -- 172.21.15.62:0/3125067405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0102d90 msgr2=0x7f61b0103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 --2- 172.21.15.62:0/3125067405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0102d90 0x7f61b0103170 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009930 tx=0x7f61a002a420 comp rx=0 tx=0).stop 2024-09-15T23:14:45.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 -- 172.21.15.62:0/3125067405 shutdown_connections 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 --2- 172.21.15.62:0/3125067405 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61b0108290 0x7f61b010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 --2- 172.21.15.62:0/3125067405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61b0103740 0x7f61b0107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 --2- 172.21.15.62:0/3125067405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0102d90 0x7f61b0103170 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 -- 172.21.15.62:0/3125067405 >> 172.21.15.62:0/3125067405 conn(0x7f61b00fc7d0 msgr2=0x7f61b00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 -- 172.21.15.62:0/3125067405 shutdown_connections 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 -- 172.21.15.62:0/3125067405 wait complete. 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.567+0000 7f61b6214640 1 Processor -- start 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61b6214640 1 -- start start 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61b6214640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61b0102d90 0x7f61b0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61b6214640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0103740 0x7f61b0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:45.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61b6214640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61b0108290 0x7f61b019ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61b6214640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0112090 con 0x7f61b0103740 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61b6214640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0111f10 con 0x7f61b0102d90 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61b6214640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f61b0112210 con 0x7f61b0108290 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61aeffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0103740 0x7f61b0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61aeffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0103740 0x7f61b0198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38036/0 (socket says 172.21.15.62:38036) 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61aeffd640 1 -- 172.21.15.62:0/219347335 learned_addr learned my addr 172.21.15.62:0/219347335 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61af7fe640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61b0102d90 0x7f61b0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61affff640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61b0108290 0x7f61b019ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61aeffd640 1 -- 172.21.15.62:0/219347335 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61b0108290 msgr2=0x7f61b019ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:45.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61aeffd640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61b0108290 0x7f61b019ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61aeffd640 1 -- 172.21.15.62:0/219347335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61b0102d90 msgr2=0x7f61b0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61aeffd640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61b0102d90 0x7f61b0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.568+0000 7f61aeffd640 1 -- 172.21.15.62:0/219347335 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b019d500 con 0x7f61b0103740 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.569+0000 7f61aeffd640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0103740 0x7f61b0198a90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f619800d770 tx=0x7f619800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.569+0000 7f61acff9640 1 -- 172.21.15.62:0/219347335 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198017070 con 0x7f61b0103740 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.569+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b019d7f0 con 0x7f61b0103740 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.569+0000 7f61acff9640 1 -- 172.21.15.62:0/219347335 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6198009c30 con 0x7f61b0103740 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.569+0000 7f61acff9640 1 -- 172.21.15.62:0/219347335 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619800f620 con 0x7f61b0103740 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.569+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b01a50d0 con 0x7f61b0103740 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.570+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f61b0062980 con 0x7f61b0103740 2024-09-15T23:14:45.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.571+0000 7f61acff9640 1 -- 172.21.15.62:0/219347335 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f619801a070 con 0x7f61b0103740 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.572+0000 7f61acff9640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f619c0a32d0 0x7f619c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.572+0000 7f61af7fe640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f619c0a32d0 0x7f619c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.572+0000 7f61acff9640 1 -- 172.21.15.62:0/219347335 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f61980cfc60 con 0x7f61b0103740 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.573+0000 7f61acff9640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] conn(0x7f619c0ae420 0x7f619c0b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.573+0000 7f61acff9640 1 -- 172.21.15.62:0/219347335 --> [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f619c0b0f90 con 0x7f619c0ae420 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.573+0000 7f61acff9640 1 -- 172.21.15.62:0/219347335 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f61980d00b0 con 0x7f61b0103740 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.573+0000 7f61affff640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] conn(0x7f619c0ae420 0x7f619c0b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.573+0000 7f61af7fe640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f619c0a32d0 0x7f619c0a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009900 tx=0x7f61a00058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.573+0000 7f61affff640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] conn(0x7f619c0ae420 0x7f619c0b08c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:45.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.574+0000 7f61acff9640 1 -- 172.21.15.62:0/219347335 <== osd.10 v2:172.21.15.67:6824/3693654334 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f619c0b0f90 con 0x7f619c0ae420 2024-09-15T23:14:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.588+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 --> [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f61b0062e00 con 0x7f619c0ae420 2024-09-15T23:14:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61acff9640 1 -- 172.21.15.62:0/219347335 <== osd.10 v2:172.21.15.67:6824/3693654334 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f61b0062e00 con 0x7f619c0ae420 2024-09-15T23:14:45.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 >> [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] conn(0x7f619c0ae420 msgr2=0x7f619c0b08c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61b6214640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] conn(0x7f619c0ae420 0x7f619c0b08c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f619c0a32d0 msgr2=0x7f619c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61b6214640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f619c0a32d0 0x7f619c0a5790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009900 tx=0x7f61a00058d0 comp rx=0 tx=0).stop 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0103740 msgr2=0x7f61b0198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61b6214640 1 --2- 172.21.15.62:0/219347335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61b0103740 0x7f61b0198a90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f619800d770 tx=0x7f619800dc30 comp rx=0 tx=0).stop 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61affff640 1 -- 172.21.15.62:0/219347335 reap_dead start 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 shutdown_connections 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.589+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 >> 172.21.15.62:0/219347335 conn(0x7f61b00fc7d0 msgr2=0x7f61b0105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.590+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 shutdown_connections 2024-09-15T23:14:45.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:45.590+0000 7f61b6214640 1 -- 172.21.15.62:0/219347335 wait complete. 2024-09-15T23:14:46.291 INFO:teuthology.orchestra.run.smithi062.stdout:244813135914 2024-09-15T23:14:46.292 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-15T23:14:46.928 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:46 smithi062 ceph-mon[30949]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:46.958 INFO:teuthology.orchestra.run.smithi062.stdout:296352743456 2024-09-15T23:14:46.958 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-15T23:14:47.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:46 smithi067 ceph-mon[35665]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:47.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:46 smithi148 ceph-mon[35610]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:47.517 INFO:teuthology.orchestra.run.smithi062.stdout:214748364847 2024-09-15T23:14:47.518 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-15T23:14:47.551 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:47.552 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:47.969 INFO:teuthology.orchestra.run.smithi062.stdout:326417514524 2024-09-15T23:14:47.969 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-15T23:14:48.090 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:48.092 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:48.097 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:48.225 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:47 smithi062 ceph-mon[30949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-15T23:14:48.225 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:47 smithi062 ceph-mon[30949]: Cluster is now healthy 2024-09-15T23:14:48.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:47 smithi067 ceph-mon[35665]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-15T23:14:48.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:47 smithi067 ceph-mon[35665]: Cluster is now healthy 2024-09-15T23:14:48.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:47 smithi148 ceph-mon[35610]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-15T23:14:48.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:47 smithi148 ceph-mon[35610]: Cluster is now healthy 2024-09-15T23:14:49.053 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:48 smithi062 ceph-mon[30949]: pgmap v324: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:49.175 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:48 smithi067 ceph-mon[35665]: pgmap v324: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:49.190 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:48 smithi148 ceph-mon[35610]: pgmap v324: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:49.369 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.584+0000 7ff62580d640 1 -- 172.21.15.62:0/2525147006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620100e20 msgr2=0x7ff620101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.584+0000 7ff62580d640 1 --2- 172.21.15.62:0/2525147006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620100e20 0x7ff620101200 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff610009ca0 tx=0x7ff61002a950 comp rx=0 tx=0).stop 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.584+0000 7ff62580d640 1 -- 172.21.15.62:0/2525147006 shutdown_connections 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.584+0000 7ff62580d640 1 --2- 172.21.15.62:0/2525147006 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff620102ef0 0x7ff62010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.584+0000 7ff62580d640 1 --2- 172.21.15.62:0/2525147006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620102570 0x7ff6201029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.584+0000 7ff62580d640 1 --2- 172.21.15.62:0/2525147006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620100e20 0x7ff620101200 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.584+0000 7ff62580d640 1 -- 172.21.15.62:0/2525147006 >> 172.21.15.62:0/2525147006 conn(0x7ff6200fc7d0 msgr2=0x7ff6200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.584+0000 7ff62580d640 1 -- 172.21.15.62:0/2525147006 shutdown_connections 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.584+0000 7ff62580d640 1 -- 172.21.15.62:0/2525147006 wait complete. 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff62580d640 1 Processor -- start 2024-09-15T23:14:49.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff62580d640 1 -- start start 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff62580d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620100e20 0x7ff6201973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff62580d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620102570 0x7ff620197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff62580d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff620102ef0 0x7ff62019bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff62580d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff620110f80 con 0x7ff620102570 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff62580d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff620110e00 con 0x7ff620100e20 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff62580d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff620111100 con 0x7ff620102ef0 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff61e7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620102570 0x7ff620197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff61e7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620102570 0x7ff620197930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38082/0 (socket says 172.21.15.62:38082) 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff61e7fc640 1 -- 172.21.15.62:0/1555580622 learned_addr learned my addr 172.21.15.62:0/1555580622 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff61f7fe640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff620102ef0 0x7ff62019bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.585+0000 7ff61effd640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620100e20 0x7ff6201973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff61e7fc640 1 -- 172.21.15.62:0/1555580622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff620102ef0 msgr2=0x7ff62019bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff61e7fc640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff620102ef0 0x7ff62019bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff61e7fc640 1 -- 172.21.15.62:0/1555580622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620100e20 msgr2=0x7ff6201973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff61e7fc640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620100e20 0x7ff6201973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff61e7fc640 1 -- 172.21.15.62:0/1555580622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62019c3a0 con 0x7ff620102570 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff61e7fc640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620102570 0x7ff620197930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff608009900 tx=0x7ff608009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff62480b640 1 -- 172.21.15.62:0/1555580622 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60800d030 con 0x7ff620102570 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff62480b640 1 -- 172.21.15.62:0/1555580622 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff60800d1d0 con 0x7ff620102570 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62019c690 con 0x7ff620102570 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff62480b640 1 -- 172.21.15.62:0/1555580622 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff608005260 con 0x7ff620102570 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.586+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff620069f70 con 0x7ff620102570 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.587+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6201021b0 con 0x7ff620102570 2024-09-15T23:14:49.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.589+0000 7ff62480b640 1 -- 172.21.15.62:0/1555580622 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff6080078f0 con 0x7ff620102570 2024-09-15T23:14:49.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.589+0000 7ff62480b640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff60c0a32d0 0x7ff60c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.590+0000 7ff62480b640 1 -- 172.21.15.62:0/1555580622 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7ff6080d14e0 con 0x7ff620102570 2024-09-15T23:14:49.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.590+0000 7ff61effd640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff60c0a32d0 0x7ff60c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.590+0000 7ff61effd640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff60c0a32d0 0x7ff60c0a5790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff610004ad0 tx=0x7ff610004a60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:49.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.594+0000 7ff62480b640 1 -- 172.21.15.62:0/1555580622 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff60809d710 con 0x7ff620102570 2024-09-15T23:14:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.453+0000 7fa4d4832640 1 -- 172.21.15.62:0/3649406963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0108300 msgr2=0x7fa4d010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.453+0000 7fa4d4832640 1 --2- 172.21.15.62:0/3649406963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0108300 0x7fa4d010ec70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009cd0 tx=0x7fa4c802a8d0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.453+0000 7fa4d4832640 1 -- 172.21.15.62:0/3649406963 shutdown_connections 2024-09-15T23:14:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.453+0000 7fa4d4832640 1 --2- 172.21.15.62:0/3649406963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0108300 0x7fa4d010ec70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.453+0000 7fa4d4832640 1 --2- 172.21.15.62:0/3649406963 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4d0103780 0x7fa4d0107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.453+0000 7fa4d4832640 1 --2- 172.21.15.62:0/3649406963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d0102dd0 0x7fa4d01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.453+0000 7fa4d4832640 1 -- 172.21.15.62:0/3649406963 >> 172.21.15.62:0/3649406963 conn(0x7fa4d00fc7d0 msgr2=0x7fa4d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.453+0000 7fa4d4832640 1 -- 172.21.15.62:0/3649406963 shutdown_connections 2024-09-15T23:14:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.453+0000 7fa4d4832640 1 -- 172.21.15.62:0/3649406963 wait complete. 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.454+0000 7fa4d4832640 1 Processor -- start 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.454+0000 7fa4d4832640 1 -- start start 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4d4832640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4d0102dd0 0x7fa4d0198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4d4832640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0103780 0x7fa4d0198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4d4832640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d0108300 0x7fa4d019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4d4832640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d01120d0 con 0x7fa4d0103780 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4d4832640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0111f50 con 0x7fa4d0108300 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4d4832640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d0112250 con 0x7fa4d0102dd0 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4cdd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0103780 0x7fa4d0198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4cdd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0103780 0x7fa4d0198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38062/0 (socket says 172.21.15.62:38062) 2024-09-15T23:14:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4cdd9b640 1 -- 172.21.15.62:0/1261006830 learned_addr learned my addr 172.21.15.62:0/1261006830 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:49.701 INFO:teuthology.orchestra.run.smithi062.stdout:107374182468 2024-09-15T23:14:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4ced9d640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d0108300 0x7fa4d019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4ce59c640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4d0102dd0 0x7fa4d0198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4cdd9b640 1 -- 172.21.15.62:0/1261006830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4d0102dd0 msgr2=0x7fa4d0198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4cdd9b640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4d0102dd0 0x7fa4d0198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4cdd9b640 1 -- 172.21.15.62:0/1261006830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d0108300 msgr2=0x7fa4d019ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4cdd9b640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d0108300 0x7fa4d019ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4cdd9b640 1 -- 172.21.15.62:0/1261006830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d019d530 con 0x7fa4d0103780 2024-09-15T23:14:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4ced9d640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d0108300 0x7fa4d019ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.455+0000 7fa4cdd9b640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0103780 0x7fa4d0198ac0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400d950 tx=0x7fa4c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.456+0000 7fa4bf7fe640 1 -- 172.21.15.62:0/1261006830 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4017070 con 0x7fa4d0103780 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.456+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d019d820 con 0x7fa4d0103780 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.456+0000 7fa4bf7fe640 1 -- 172.21.15.62:0/1261006830 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4c4009c30 con 0x7fa4d0103780 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.456+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d01a5100 con 0x7fa4d0103780 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.456+0000 7fa4bf7fe640 1 -- 172.21.15.62:0/1261006830 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c40047a0 con 0x7fa4d0103780 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.457+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d01031b0 con 0x7fa4d0103780 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.458+0000 7fa4bf7fe640 1 -- 172.21.15.62:0/1261006830 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa4c4004020 con 0x7fa4d0103780 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.459+0000 7fa4bf7fe640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4b80a32d0 0x7fa4b80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.459+0000 7fa4ce59c640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4b80a32d0 0x7fa4b80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.459+0000 7fa4bf7fe640 1 -- 172.21.15.62:0/1261006830 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fa4c40cfdd0 con 0x7fa4d0103780 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.459+0000 7fa4ce59c640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4b80a32d0 0x7fa4b80a5790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0009b60 tx=0x7fa4c0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.462+0000 7fa4bf7fe640 1 -- 172.21.15.62:0/1261006830 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa4c4020020 con 0x7fa4d0103780 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.647+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa4d010a290 con 0x7fa4d0103780 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.647+0000 7fa4bf7fe640 1 -- 172.21.15.62:0/1261006830 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa4c409c650 con 0x7fa4d0103780 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4b80a32d0 msgr2=0x7fa4b80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4b80a32d0 0x7fa4b80a5790 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0009b60 tx=0x7fa4c0005f90 comp rx=0 tx=0).stop 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0103780 msgr2=0x7fa4d0198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0103780 0x7fa4d0198ac0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400d950 tx=0x7fa4c400de10 comp rx=0 tx=0).stop 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 shutdown_connections 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa4b80a32d0 0x7fa4b80a5790 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4d0108300 0x7fa4d019ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4d0103780 0x7fa4d0198ac0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 --2- 172.21.15.62:0/1261006830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa4d0102dd0 0x7fa4d0198580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 >> 172.21.15.62:0/1261006830 conn(0x7fa4d00fc7d0 msgr2=0x7fa4d0105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 shutdown_connections 2024-09-15T23:14:49.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.650+0000 7fa4d4832640 1 -- 172.21.15.62:0/1261006830 wait complete. 2024-09-15T23:14:49.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.775+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff62006a9f0 con 0x7ff620102570 2024-09-15T23:14:49.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.776+0000 7ff62480b640 1 -- 172.21.15.62:0/1555580622 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff6080a1a10 con 0x7ff620102570 2024-09-15T23:14:49.778 INFO:teuthology.orchestra.run.smithi062.stdout:55834574927 2024-09-15T23:14:49.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff60c0a32d0 msgr2=0x7ff60c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff60c0a32d0 0x7ff60c0a5790 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff610004ad0 tx=0x7ff610004a60 comp rx=0 tx=0).stop 2024-09-15T23:14:49.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620102570 msgr2=0x7ff620197930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620102570 0x7ff620197930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff608009900 tx=0x7ff608009dc0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 shutdown_connections 2024-09-15T23:14:49.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff60c0a32d0 0x7ff60c0a5790 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff620102ef0 0x7ff62019bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff620102570 0x7ff620197930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 --2- 172.21.15.62:0/1555580622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620100e20 0x7ff6201973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 >> 172.21.15.62:0/1555580622 conn(0x7ff6200fc7d0 msgr2=0x7ff6200fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:49.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 shutdown_connections 2024-09-15T23:14:49.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.778+0000 7ff62580d640 1 -- 172.21.15.62:0/1555580622 wait complete. 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 -- 172.21.15.62:0/3043777311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab41055b0 msgr2=0x7ffab410d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 --2- 172.21.15.62:0/3043777311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab41055b0 0x7ffab410d130 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009ca0 tx=0x7ffaac02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 -- 172.21.15.62:0/3043777311 shutdown_connections 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 --2- 172.21.15.62:0/3043777311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab41055b0 0x7ffab410d130 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 --2- 172.21.15.62:0/3043777311 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab4100e30 0x7ffab4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 --2- 172.21.15.62:0/3043777311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffab4100480 0x7ffab4100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 -- 172.21.15.62:0/3043777311 >> 172.21.15.62:0/3043777311 conn(0x7ffab4077210 msgr2=0x7ffab40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 -- 172.21.15.62:0/3043777311 shutdown_connections 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 -- 172.21.15.62:0/3043777311 wait complete. 2024-09-15T23:14:49.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 Processor -- start 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.338+0000 7ffabac1f640 1 -- start start 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffabac1f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffab4100480 0x7ffab4073ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffabac1f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab4100e30 0x7ffab406ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffabac1f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab41055b0 0x7ffab406f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffabac1f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffab410fd90 con 0x7ffab4100e30 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffabac1f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffab410fc10 con 0x7ffab4100480 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffabac1f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffab410ff10 con 0x7ffab41055b0 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab4100e30 0x7ffab406ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab4100e30 0x7ffab406ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38050/0 (socket says 172.21.15.62:38050) 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab3fff640 1 -- 172.21.15.62:0/980172337 learned_addr learned my addr 172.21.15.62:0/980172337 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab89bb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffab4100480 0x7ffab4073ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab91bc640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab41055b0 0x7ffab406f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab3fff640 1 -- 172.21.15.62:0/980172337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab41055b0 msgr2=0x7ffab406f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab3fff640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab41055b0 0x7ffab406f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab3fff640 1 -- 172.21.15.62:0/980172337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffab4100480 msgr2=0x7ffab4073ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab3fff640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffab4100480 0x7ffab4073ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab3fff640 1 -- 172.21.15.62:0/980172337 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab406f8a0 con 0x7ffab4100e30 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab91bc640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab41055b0 0x7ffab406f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab89bb640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffab4100480 0x7ffab4073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab3fff640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab4100e30 0x7ffab406ec30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d950 tx=0x7ffaa800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.339+0000 7ffab1ffb640 1 -- 172.21.15.62:0/980172337 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8017030 con 0x7ffab4100e30 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.340+0000 7ffab1ffb640 1 -- 172.21.15.62:0/980172337 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffaa8004020 con 0x7ffab4100e30 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.340+0000 7ffab1ffb640 1 -- 172.21.15.62:0/980172337 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa80047a0 con 0x7ffab4100e30 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.340+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab41ad250 con 0x7ffab4100e30 2024-09-15T23:14:49.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.340+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab41ad6b0 con 0x7ffab4100e30 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.341+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab4101960 con 0x7ffab4100e30 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.342+0000 7ffab1ffb640 1 -- 172.21.15.62:0/980172337 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ffaa80041d0 con 0x7ffab4100e30 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.343+0000 7ffab1ffb640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffa9c0a32d0 0x7ffa9c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.343+0000 7ffab89bb640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffa9c0a32d0 0x7ffa9c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.343+0000 7ffab1ffb640 1 -- 172.21.15.62:0/980172337 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7ffaa80cf780 con 0x7ffab4100e30 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.343+0000 7ffab89bb640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffa9c0a32d0 0x7ffa9c0a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400e4a0 tx=0x7ffaa4010040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.347+0000 7ffab1ffb640 1 -- 172.21.15.62:0/980172337 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffaa809c000 con 0x7ffab4100e30 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.537+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ffab4108260 con 0x7ffab4100e30 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.537+0000 7ffab1ffb640 1 -- 172.21.15.62:0/980172337 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ffaa809b900 con 0x7ffab4100e30 2024-09-15T23:14:49.996 INFO:teuthology.orchestra.run.smithi062.stdout:34359738452 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffa9c0a32d0 msgr2=0x7ffa9c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffa9c0a32d0 0x7ffa9c0a5790 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400e4a0 tx=0x7ffaa4010040 comp rx=0 tx=0).stop 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab4100e30 msgr2=0x7ffab406ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab4100e30 0x7ffab406ec30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffaa800d950 tx=0x7ffaa800de10 comp rx=0 tx=0).stop 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 shutdown_connections 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffa9c0a32d0 0x7ffa9c0a5790 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffab41055b0 0x7ffab406f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffab4100e30 0x7ffab406ec30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 --2- 172.21.15.62:0/980172337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffab4100480 0x7ffab4073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 >> 172.21.15.62:0/980172337 conn(0x7ffab4077210 msgr2=0x7ffab4103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:49.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 shutdown_connections 2024-09-15T23:14:49.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:49.539+0000 7ffabac1f640 1 -- 172.21.15.62:0/980172337 wait complete. 2024-09-15T23:14:50.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:49 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/980172337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-15T23:14:50.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:49 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1261006830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-15T23:14:50.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:49 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1555580622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-15T23:14:50.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:49 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/980172337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-15T23:14:50.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:49 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1261006830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-15T23:14:50.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:49 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1555580622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-15T23:14:50.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:49 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/980172337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-15T23:14:50.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:49 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1261006830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-15T23:14:50.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:49 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1555580622' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-15T23:14:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.596+0000 7f7c5696f640 1 -- 172.21.15.62:0/1112908321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c50100a80 msgr2=0x7f7c50100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.596+0000 7f7c5696f640 1 --2- 172.21.15.62:0/1112908321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c50100a80 0x7f7c50100ee0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009930 tx=0x7f7c4402a420 comp rx=0 tx=0).stop 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.596+0000 7f7c5696f640 1 -- 172.21.15.62:0/1112908321 shutdown_connections 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.596+0000 7f7c5696f640 1 --2- 172.21.15.62:0/1112908321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50101420 0x7f7c5010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.596+0000 7f7c5696f640 1 --2- 172.21.15.62:0/1112908321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c50100a80 0x7f7c50100ee0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.596+0000 7f7c5696f640 1 --2- 172.21.15.62:0/1112908321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c50106a80 0x7f7c50106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.596+0000 7f7c5696f640 1 -- 172.21.15.62:0/1112908321 >> 172.21.15.62:0/1112908321 conn(0x7f7c500fc7a0 msgr2=0x7f7c500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.597+0000 7f7c5696f640 1 -- 172.21.15.62:0/1112908321 shutdown_connections 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.597+0000 7f7c5696f640 1 -- 172.21.15.62:0/1112908321 wait complete. 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.597+0000 7f7c5696f640 1 Processor -- start 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.597+0000 7f7c5696f640 1 -- start start 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c5696f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50100a80 0x7f7c50199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c5696f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c50101420 0x7f7c50199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c5696f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c50106a80 0x7f7c501a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c5696f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7c50110e80 con 0x7f7c50106a80 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c5696f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7c50110d00 con 0x7f7c50100a80 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c5696f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7c50111000 con 0x7f7c50101420 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c4ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50100a80 0x7f7c50199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c4f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c50101420 0x7f7c50199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c4ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50100a80 0x7f7c50199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:37174/0 (socket says 172.21.15.62:37174) 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c4f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c50101420 0x7f7c50199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38406/0 (socket says 172.21.15.62:38406) 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.598+0000 7f7c4ffff640 1 -- 172.21.15.62:0/2887320251 learned_addr learned my addr 172.21.15.62:0/2887320251 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c54f0c640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c50106a80 0x7f7c501a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c4ffff640 1 -- 172.21.15.62:0/2887320251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c50101420 msgr2=0x7f7c50199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c4ffff640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c50101420 0x7f7c50199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c4ffff640 1 -- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c50106a80 msgr2=0x7f7c501a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c4ffff640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c50106a80 0x7f7c501a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c4ffff640 1 -- 172.21.15.62:0/2887320251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c501a48f0 con 0x7f7c50100a80 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c54f0c640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c50106a80 0x7f7c501a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c4ffff640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50100a80 0x7f7c50199330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000d770 tx=0x7f7c4000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c4d7fa640 1 -- 172.21.15.62:0/2887320251 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40017070 con 0x7f7c50100a80 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c501a4ac0 con 0x7f7c50100a80 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c4d7fa640 1 -- 172.21.15.62:0/2887320251 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c40005500 con 0x7f7c50100a80 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c4d7fa640 1 -- 172.21.15.62:0/2887320251 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40004e00 con 0x7f7c50100a80 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.599+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c501a4f80 con 0x7f7c50100a80 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.600+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c50100ee0 con 0x7f7c50100a80 2024-09-15T23:14:50.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.602+0000 7f7c4d7fa640 1 -- 172.21.15.62:0/2887320251 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7c40004fa0 con 0x7f7c50100a80 2024-09-15T23:14:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.602+0000 7f7c4d7fa640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7c380a32d0 0x7f7c380a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.602+0000 7f7c4f7fe640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7c380a32d0 0x7f7c380a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.602+0000 7f7c4d7fa640 1 -- 172.21.15.62:0/2887320251 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f7c400cf8a0 con 0x7f7c50100a80 2024-09-15T23:14:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.603+0000 7f7c4f7fe640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7c380a32d0 0x7f7c380a5790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009900 tx=0x7f7c440058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:50.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.605+0000 7f7c4d7fa640 1 -- 172.21.15.62:0/2887320251 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7c4009c0c0 con 0x7f7c50100a80 2024-09-15T23:14:50.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.781+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f7c50108ff0 con 0x7f7c50100a80 2024-09-15T23:14:50.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.782+0000 7f7c4d7fa640 1 -- 172.21.15.62:0/2887320251 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7c4009b9c0 con 0x7f7c50100a80 2024-09-15T23:14:50.783 INFO:teuthology.orchestra.run.smithi062.stdout:274877906983 2024-09-15T23:14:50.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7c380a32d0 msgr2=0x7f7c380a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:50.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7c380a32d0 0x7f7c380a5790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009900 tx=0x7f7c440058d0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50100a80 msgr2=0x7f7c50199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:50.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50100a80 0x7f7c50199330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000d770 tx=0x7f7c4000dc30 comp rx=0 tx=0).stop 2024-09-15T23:14:50.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 shutdown_connections 2024-09-15T23:14:50.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7c380a32d0 0x7f7c380a5790 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c50106a80 0x7f7c501a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7c50101420 0x7f7c50199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 --2- 172.21.15.62:0/2887320251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50100a80 0x7f7c50199330 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:50.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 >> 172.21.15.62:0/2887320251 conn(0x7f7c500fc7a0 msgr2=0x7f7c500fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:50.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 shutdown_connections 2024-09-15T23:14:50.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:50.784+0000 7f7c5696f640 1 -- 172.21.15.62:0/2887320251 wait complete. 2024-09-15T23:14:51.218 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:50 smithi062 ceph-mon[30949]: pgmap v325: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:51.218 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:50 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2887320251' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-15T23:14:51.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:50 smithi067 ceph-mon[35665]: pgmap v325: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:51.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:50 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2887320251' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-15T23:14:51.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:50 smithi148 ceph-mon[35610]: pgmap v325: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2024-09-15T23:14:51.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:50 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2887320251' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-09-15T23:14:51.992 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:52.003 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738452 for osd.0 2024-09-15T23:14:52.003 DEBUG:teuthology.parallel:result is None 2024-09-15T23:14:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.191+0000 7f1ad6213640 1 -- 172.21.15.62:0/824430068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 msgr2=0x7f1ad010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.191+0000 7f1ad6213640 1 --2- 172.21.15.62:0/824430068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 0x7f1ad010ec40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009cd0 tx=0x7f1ac802a910 comp rx=0 tx=0).stop 2024-09-15T23:14:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.191+0000 7f1ad6213640 1 -- 172.21.15.62:0/824430068 shutdown_connections 2024-09-15T23:14:52.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.191+0000 7f1ad6213640 1 --2- 172.21.15.62:0/824430068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 0x7f1ad010ec40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.191+0000 7f1ad6213640 1 --2- 172.21.15.62:0/824430068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ad01036e0 0x7f1ad0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.191+0000 7f1ad6213640 1 --2- 172.21.15.62:0/824430068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ad0102d30 0x7f1ad0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.191+0000 7f1ad6213640 1 -- 172.21.15.62:0/824430068 >> 172.21.15.62:0/824430068 conn(0x7f1ad00fc7d0 msgr2=0x7f1ad00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.191+0000 7f1ad6213640 1 -- 172.21.15.62:0/824430068 shutdown_connections 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.191+0000 7f1ad6213640 1 -- 172.21.15.62:0/824430068 wait complete. 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1ad6213640 1 Processor -- start 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1ad6213640 1 -- start start 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1ad6213640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ad0102d30 0x7f1ad0100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1ad6213640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ad01036e0 0x7f1ad0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1ad6213640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 0x7f1ad0101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1ad6213640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad01120e0 con 0x7f1ad01082a0 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1ad6213640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad0111f60 con 0x7f1ad0102d30 2024-09-15T23:14:52.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1ad6213640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad0112260 con 0x7f1ad01036e0 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1acffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 0x7f1ad0101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1acffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 0x7f1ad0101410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38112/0 (socket says 172.21.15.62:38112) 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1acffff640 1 -- 172.21.15.62:0/3489191106 learned_addr learned my addr 172.21.15.62:0/3489191106 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.192+0000 7f1aceffd640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ad01036e0 0x7f1ad0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1acf7fe640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ad0102d30 0x7f1ad0100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1acffff640 1 -- 172.21.15.62:0/3489191106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ad01036e0 msgr2=0x7f1ad0100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1acffff640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ad01036e0 0x7f1ad0100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1acffff640 1 -- 172.21.15.62:0/3489191106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ad0102d30 msgr2=0x7f1ad0100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1acffff640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ad0102d30 0x7f1ad0100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1acffff640 1 -- 172.21.15.62:0/3489191106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad01a6a90 con 0x7f1ad01082a0 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1acffff640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 0x7f1ad0101410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8005cc0 tx=0x7f1ac8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1accff9640 1 -- 172.21.15.62:0/3489191106 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8031660 con 0x7f1ad01082a0 2024-09-15T23:14:52.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad01a6d20 con 0x7f1ad01082a0 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1accff9640 1 -- 172.21.15.62:0/3489191106 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ac8031800 con 0x7f1ad01082a0 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1accff9640 1 -- 172.21.15.62:0/3489191106 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac802fae0 con 0x7f1ad01082a0 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.193+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad01a7260 con 0x7f1ad01082a0 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.194+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad0103110 con 0x7f1ad01082a0 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.195+0000 7f1accff9640 1 -- 172.21.15.62:0/3489191106 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f1ac802fc80 con 0x7f1ad01082a0 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.196+0000 7f1accff9640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ab80a32d0 0x7f1ab80a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.196+0000 7f1accff9640 1 -- 172.21.15.62:0/3489191106 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f1ac80f12e0 con 0x7f1ad01082a0 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.196+0000 7f1acf7fe640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ab80a32d0 0x7f1ab80a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.196+0000 7f1acf7fe640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ab80a32d0 0x7f1ab80a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0005fd0 tx=0x7f1ac0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:52.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.199+0000 7f1accff9640 1 -- 172.21.15.62:0/3489191106 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1ac80bdc10 con 0x7f1ad01082a0 2024-09-15T23:14:52.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.382+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1ad010a2b0 con 0x7f1ad01082a0 2024-09-15T23:14:52.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.382+0000 7f1accff9640 1 -- 172.21.15.62:0/3489191106 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1ac80bd510 con 0x7f1ad01082a0 2024-09-15T23:14:52.384 INFO:teuthology.orchestra.run.smithi062.stdout:77309411403 2024-09-15T23:14:52.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ab80a32d0 msgr2=0x7f1ab80a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:52.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ab80a32d0 0x7f1ab80a5790 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0005fd0 tx=0x7f1ac0005850 comp rx=0 tx=0).stop 2024-09-15T23:14:52.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 msgr2=0x7f1ad0101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:52.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 0x7f1ad0101410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8005cc0 tx=0x7f1ac8003f90 comp rx=0 tx=0).stop 2024-09-15T23:14:52.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 shutdown_connections 2024-09-15T23:14:52.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1ab80a32d0 0x7f1ab80a5790 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:52.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ad01082a0 0x7f1ad0101410 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:52.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ad01036e0 0x7f1ad0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:52.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 --2- 172.21.15.62:0/3489191106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ad0102d30 0x7f1ad0100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:52.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 >> 172.21.15.62:0/3489191106 conn(0x7f1ad00fc7d0 msgr2=0x7f1ad0106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:52.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 shutdown_connections 2024-09-15T23:14:52.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:52.385+0000 7f1ad6213640 1 -- 172.21.15.62:0/3489191106 wait complete. 2024-09-15T23:14:52.832 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182467 got 107374182468 for osd.3 2024-09-15T23:14:52.832 DEBUG:teuthology.parallel:result is None 2024-09-15T23:14:53.134 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:52 smithi062 ceph-mon[30949]: pgmap v326: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 11 B/s, 0 keys/s, 1 objects/s recovering 2024-09-15T23:14:53.134 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:52 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3489191106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-15T23:14:53.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:52 smithi067 ceph-mon[35665]: pgmap v326: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 11 B/s, 0 keys/s, 1 objects/s recovering 2024-09-15T23:14:53.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:52 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3489191106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-15T23:14:53.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:52 smithi148 ceph-mon[35610]: pgmap v326: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 11 B/s, 0 keys/s, 1 objects/s recovering 2024-09-15T23:14:53.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:52 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3489191106' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-15T23:14:53.474 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411400 got 77309411403 for osd.2 2024-09-15T23:14:53.474 DEBUG:teuthology.parallel:result is None 2024-09-15T23:14:53.801 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574926 got 55834574927 for osd.1 2024-09-15T23:14:53.801 DEBUG:teuthology.parallel:result is None 2024-09-15T23:14:54.165 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:54.203 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906981 got 274877906983 for osd.9 2024-09-15T23:14:54.203 DEBUG:teuthology.parallel:result is None 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 -- 172.21.15.62:0/1361697858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc1036e0 msgr2=0x7ff2bc107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/1361697858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc1036e0 0x7ff2bc107b90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009930 tx=0x7ff2a402a420 comp rx=0 tx=0).stop 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 -- 172.21.15.62:0/1361697858 shutdown_connections 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/1361697858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc1082a0 0x7ff2bc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/1361697858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc1036e0 0x7ff2bc107b90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/1361697858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff2bc102d30 0x7ff2bc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 -- 172.21.15.62:0/1361697858 >> 172.21.15.62:0/1361697858 conn(0x7ff2bc0fc7d0 msgr2=0x7ff2bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 -- 172.21.15.62:0/1361697858 shutdown_connections 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 -- 172.21.15.62:0/1361697858 wait complete. 2024-09-15T23:14:54.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.346+0000 7ff2c22c6640 1 Processor -- start 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2c22c6640 1 -- start start 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2c22c6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102d30 0x7ff2bc1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2c22c6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff2bc1036e0 0x7ff2bc198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2c22c6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc1082a0 0x7ff2bc19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2c22c6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc1120c0 con 0x7ff2bc102d30 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2c22c6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc111f40 con 0x7ff2bc1082a0 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2c22c6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff2bc112240 con 0x7ff2bc1036e0 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2bbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102d30 0x7ff2bc1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2bbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102d30 0x7ff2bc1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50042/0 (socket says 172.21.15.62:50042) 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2bbfff640 1 -- 172.21.15.62:0/462065385 learned_addr learned my addr 172.21.15.62:0/462065385 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2c0863640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc1082a0 0x7ff2bc19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2b3fff640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff2bc1036e0 0x7ff2bc198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2bbfff640 1 -- 172.21.15.62:0/462065385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff2bc1036e0 msgr2=0x7ff2bc198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2bbfff640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff2bc1036e0 0x7ff2bc198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2bbfff640 1 -- 172.21.15.62:0/462065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc1082a0 msgr2=0x7ff2bc19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2bbfff640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc1082a0 0x7ff2bc19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2bbfff640 1 -- 172.21.15.62:0/462065385 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2bc19d570 con 0x7ff2bc102d30 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.347+0000 7ff2c0863640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc1082a0 0x7ff2bc19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.348+0000 7ff2bbfff640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102d30 0x7ff2bc1985c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac00b4a0 tx=0x7ff2ac00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.348+0000 7ff2b9ffb640 1 -- 172.21.15.62:0/462065385 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac014070 con 0x7ff2bc102d30 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.348+0000 7ff2b9ffb640 1 -- 172.21.15.62:0/462065385 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2ac010af0 con 0x7ff2bc102d30 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.348+0000 7ff2b9ffb640 1 -- 172.21.15.62:0/462065385 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac017d10 con 0x7ff2bc102d30 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.349+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2bc19d860 con 0x7ff2bc102d30 2024-09-15T23:14:54.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.349+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2bc1a5140 con 0x7ff2bc102d30 2024-09-15T23:14:54.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.350+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2bc06a850 con 0x7ff2bc102d30 2024-09-15T23:14:54.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.351+0000 7ff2b9ffb640 1 -- 172.21.15.62:0/462065385 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff2ac017070 con 0x7ff2bc102d30 2024-09-15T23:14:54.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.352+0000 7ff2b9ffb640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2a80a3350 0x7ff2a80a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.352+0000 7ff2b3fff640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2a80a3350 0x7ff2a80a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.352+0000 7ff2b9ffb640 1 -- 172.21.15.62:0/462065385 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7ff2ac0cecc0 con 0x7ff2bc102d30 2024-09-15T23:14:54.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.352+0000 7ff2b3fff640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2a80a3350 0x7ff2a80a5810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009900 tx=0x7ff2a4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:54.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.356+0000 7ff2b9ffb640 1 -- 172.21.15.62:0/462065385 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff2ac01e070 con 0x7ff2bc102d30 2024-09-15T23:14:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.534+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7ff2bc10a2b0 con 0x7ff2bc102d30 2024-09-15T23:14:54.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.534+0000 7ff2b9ffb640 1 -- 172.21.15.62:0/462065385 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff2ac09b650 con 0x7ff2bc102d30 2024-09-15T23:14:54.535 INFO:teuthology.orchestra.run.smithi062.stdout:158913790012 2024-09-15T23:14:54.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2a80a3350 msgr2=0x7ff2a80a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2a80a3350 0x7ff2a80a5810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009900 tx=0x7ff2a4000bc0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102d30 msgr2=0x7ff2bc1985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102d30 0x7ff2bc1985c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac00b4a0 tx=0x7ff2ac00b960 comp rx=0 tx=0).stop 2024-09-15T23:14:54.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 shutdown_connections 2024-09-15T23:14:54.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2a80a3350 0x7ff2a80a5810 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2bc1082a0 0x7ff2bc19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff2bc1036e0 0x7ff2bc198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 --2- 172.21.15.62:0/462065385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2bc102d30 0x7ff2bc1985c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 >> 172.21.15.62:0/462065385 conn(0x7ff2bc0fc7d0 msgr2=0x7ff2bc105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:54.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 shutdown_connections 2024-09-15T23:14:54.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.536+0000 7ff2c22c6640 1 -- 172.21.15.62:0/462065385 wait complete. 2024-09-15T23:14:54.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.558+0000 7f262a3dd640 1 -- 172.21.15.62:0/3167471005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624106b80 msgr2=0x7f2624106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.558+0000 7f262a3dd640 1 --2- 172.21.15.62:0/3167471005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624106b80 0x7f2624106f60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2614009900 tx=0x7f261402a410 comp rx=0 tx=0).stop 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.558+0000 7f262a3dd640 1 -- 172.21.15.62:0/3167471005 shutdown_connections 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.558+0000 7f262a3dd640 1 --2- 172.21.15.62:0/3167471005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2624100aa0 0x7f262410ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.558+0000 7f262a3dd640 1 --2- 172.21.15.62:0/3167471005 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2624100100 0x7f2624100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.558+0000 7f262a3dd640 1 --2- 172.21.15.62:0/3167471005 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624106b80 0x7f2624106f60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.558+0000 7f262a3dd640 1 -- 172.21.15.62:0/3167471005 >> 172.21.15.62:0/3167471005 conn(0x7f2624077230 msgr2=0x7f26240fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.558+0000 7f262a3dd640 1 -- 172.21.15.62:0/3167471005 shutdown_connections 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.558+0000 7f262a3dd640 1 -- 172.21.15.62:0/3167471005 wait complete. 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f262a3dd640 1 Processor -- start 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f262a3dd640 1 -- start start 2024-09-15T23:14:54.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f262a3dd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2624100100 0x7f26241012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f262a3dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624100aa0 0x7f2624101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f262a3dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2624106b80 0x7f2624101d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f262a3dd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f26241120a0 con 0x7f2624100aa0 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f262a3dd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2624111f20 con 0x7f2624106b80 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f262a3dd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2624112220 con 0x7f2624100100 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f26237fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624100aa0 0x7f2624101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f26237fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624100aa0 0x7f2624101820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50056/0 (socket says 172.21.15.62:50056) 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f26237fe640 1 -- 172.21.15.62:0/545524013 learned_addr learned my addr 172.21.15.62:0/545524013 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.559+0000 7f262897a640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2624106b80 0x7f2624101d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f2623fff640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2624100100 0x7f26241012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f26237fe640 1 -- 172.21.15.62:0/545524013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2624100100 msgr2=0x7f26241012e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f26237fe640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2624100100 0x7f26241012e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f26237fe640 1 -- 172.21.15.62:0/545524013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2624106b80 msgr2=0x7f2624101d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f26237fe640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2624106b80 0x7f2624101d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f26237fe640 1 -- 172.21.15.62:0/545524013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26241a6a00 con 0x7f2624100aa0 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f26237fe640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624100aa0 0x7f2624101820 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f261800d770 tx=0x7f261800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f26217fa640 1 -- 172.21.15.62:0/545524013 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618017070 con 0x7f2624100aa0 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26241a6cf0 con 0x7f2624100aa0 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f26217fa640 1 -- 172.21.15.62:0/545524013 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2618005500 con 0x7f2624100aa0 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f26217fa640 1 -- 172.21.15.62:0/545524013 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618004e00 con 0x7f2624100aa0 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.560+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26241a71d0 con 0x7f2624100aa0 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.561+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2624100560 con 0x7f2624100aa0 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.563+0000 7f26217fa640 1 -- 172.21.15.62:0/545524013 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2618004fa0 con 0x7f2624100aa0 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.563+0000 7f26217fa640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f260c0a32d0 0x7f260c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.564+0000 7f26217fa640 1 -- 172.21.15.62:0/545524013 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f26180cf760 con 0x7f2624100aa0 2024-09-15T23:14:54.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.564+0000 7f2623fff640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f260c0a32d0 0x7f260c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:54.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.564+0000 7f2623fff640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f260c0a32d0 0x7f260c0a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2614000c00 tx=0x7f2614005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:54.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.571+0000 7f26217fa640 1 -- 172.21.15.62:0/545524013 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f261809b990 con 0x7f2624100aa0 2024-09-15T23:14:54.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.751+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f262410a0c0 con 0x7f2624100aa0 2024-09-15T23:14:54.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.752+0000 7f26217fa640 1 -- 172.21.15.62:0/545524013 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f261809fc90 con 0x7f2624100aa0 2024-09-15T23:14:54.753 INFO:teuthology.orchestra.run.smithi062.stdout:193273528375 2024-09-15T23:14:54.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f260c0a32d0 msgr2=0x7f260c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f260c0a32d0 0x7f260c0a5790 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2614000c00 tx=0x7f2614005b50 comp rx=0 tx=0).stop 2024-09-15T23:14:54.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624100aa0 msgr2=0x7f2624101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:54.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624100aa0 0x7f2624101820 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f261800d770 tx=0x7f261800dc30 comp rx=0 tx=0).stop 2024-09-15T23:14:54.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 shutdown_connections 2024-09-15T23:14:54.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f260c0a32d0 0x7f260c0a5790 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2624106b80 0x7f2624101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624100aa0 0x7f2624101820 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 --2- 172.21.15.62:0/545524013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2624100100 0x7f26241012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:54.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 >> 172.21.15.62:0/545524013 conn(0x7f2624077230 msgr2=0x7f2624104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:54.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 shutdown_connections 2024-09-15T23:14:54.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:54.754+0000 7f262a3dd640 1 -- 172.21.15.62:0/545524013 wait complete. 2024-09-15T23:14:55.131 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:54 smithi062 ceph-mon[30949]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-09-15T23:14:55.131 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:54 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/462065385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-15T23:14:55.131 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:54 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/545524013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-15T23:14:55.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:54 smithi067 ceph-mon[35665]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-09-15T23:14:55.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:54 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/462065385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-15T23:14:55.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:54 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/545524013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-15T23:14:55.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:54 smithi148 ceph-mon[35610]: pgmap v327: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 9 B/s, 0 keys/s, 0 objects/s recovering 2024-09-15T23:14:55.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:54 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/462065385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-15T23:14:55.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:54 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/545524013' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-15T23:14:55.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.664+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1054669364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 msgr2=0x7ffbdc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:55.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.664+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1054669364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 0x7ffbdc1021c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009900 tx=0x7ffbd802a410 comp rx=0 tx=0).stop 2024-09-15T23:14:55.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.664+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1054669364 shutdown_connections 2024-09-15T23:14:55.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.664+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1054669364 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbdc106f10 0x7ffbdc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.664+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1054669364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbdc102790 0x7ffbdc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.664+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1054669364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 0x7ffbdc1021c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.664+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1054669364 >> 172.21.15.62:0/1054669364 conn(0x7ffbdc0fd8c0 msgr2=0x7ffbdc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:55.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.664+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1054669364 shutdown_connections 2024-09-15T23:14:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.664+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1054669364 wait complete. 2024-09-15T23:14:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe3dd7640 1 Processor -- start 2024-09-15T23:14:55.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe3dd7640 1 -- start start 2024-09-15T23:14:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe3dd7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 0x7ffbdc198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe3dd7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbdc102790 0x7ffbdc198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe3dd7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbdc106f10 0x7ffbdc19cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe3dd7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffbdc111fc0 con 0x7ffbdc101de0 2024-09-15T23:14:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe3dd7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffbdc111e40 con 0x7ffbdc106f10 2024-09-15T23:14:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe3dd7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffbdc112140 con 0x7ffbdc102790 2024-09-15T23:14:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe1b73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 0x7ffbdc198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe1b73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 0x7ffbdc198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50066/0 (socket says 172.21.15.62:50066) 2024-09-15T23:14:55.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.665+0000 7ffbe1b73640 1 -- 172.21.15.62:0/1753257557 learned_addr learned my addr 172.21.15.62:0/1753257557 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe2374640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbdc106f10 0x7ffbdc19cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe1372640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbdc102790 0x7ffbdc198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe1b73640 1 -- 172.21.15.62:0/1753257557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbdc102790 msgr2=0x7ffbdc198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe1b73640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbdc102790 0x7ffbdc198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe1b73640 1 -- 172.21.15.62:0/1753257557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbdc106f10 msgr2=0x7ffbdc19cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe1b73640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbdc106f10 0x7ffbdc19cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe1b73640 1 -- 172.21.15.62:0/1753257557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbdc19d2c0 con 0x7ffbdc101de0 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe2374640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbdc106f10 0x7ffbdc19cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe1b73640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 0x7ffbdc198310 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8006010 tx=0x7ffbd8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:55.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbd2ffd640 1 -- 172.21.15.62:0/1753257557 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd803f070 con 0x7ffbdc101de0 2024-09-15T23:14:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbd2ffd640 1 -- 172.21.15.62:0/1753257557 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd8031c90 con 0x7ffbdc101de0 2024-09-15T23:14:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbd2ffd640 1 -- 172.21.15.62:0/1753257557 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8035030 con 0x7ffbdc101de0 2024-09-15T23:14:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbdc19d550 con 0x7ffbdc101de0 2024-09-15T23:14:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.666+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbdc1a4e90 con 0x7ffbdc101de0 2024-09-15T23:14:55.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.667+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbdc1021c0 con 0x7ffbdc101de0 2024-09-15T23:14:55.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.668+0000 7ffbd2ffd640 1 -- 172.21.15.62:0/1753257557 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ffbd8042070 con 0x7ffbdc101de0 2024-09-15T23:14:55.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.669+0000 7ffbd2ffd640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffbc80a3350 0x7ffbc80a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:55.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.669+0000 7ffbd2ffd640 1 -- 172.21.15.62:0/1753257557 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7ffbd80f0c10 con 0x7ffbdc101de0 2024-09-15T23:14:55.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.669+0000 7ffbe1372640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffbc80a3350 0x7ffbc80a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.670+0000 7ffbe1372640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffbc80a3350 0x7ffbc80a5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc005e10 tx=0x7ffbcc005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:55.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.672+0000 7ffbd2ffd640 1 -- 172.21.15.62:0/1753257557 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffbd80bd490 con 0x7ffbdc101de0 2024-09-15T23:14:55.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.849+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ffbdc1a5060 con 0x7ffbdc101de0 2024-09-15T23:14:55.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.849+0000 7ffbd2ffd640 1 -- 172.21.15.62:0/1753257557 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ffbd80bcd90 con 0x7ffbdc101de0 2024-09-15T23:14:55.850 INFO:teuthology.orchestra.run.smithi062.stdout:296352743458 2024-09-15T23:14:55.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffbc80a3350 msgr2=0x7ffbc80a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:55.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffbc80a3350 0x7ffbc80a5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc005e10 tx=0x7ffbcc005da0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 msgr2=0x7ffbdc198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:55.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 0x7ffbdc198310 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8006010 tx=0x7ffbd8003ba0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 shutdown_connections 2024-09-15T23:14:55.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffbc80a3350 0x7ffbc80a5810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbdc106f10 0x7ffbdc19cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffbdc102790 0x7ffbdc198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 --2- 172.21.15.62:0/1753257557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc101de0 0x7ffbdc198310 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 >> 172.21.15.62:0/1753257557 conn(0x7ffbdc0fd8c0 msgr2=0x7ffbdc0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:55.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 shutdown_connections 2024-09-15T23:14:55.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.851+0000 7ffbe3dd7640 1 -- 172.21.15.62:0/1753257557 wait complete. 2024-09-15T23:14:55.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.867+0000 7f744675a640 1 -- 172.21.15.62:0/4033373843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440100e20 msgr2=0x7f7440101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.867+0000 7f744675a640 1 --2- 172.21.15.62:0/4033373843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440100e20 0x7f7440101200 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7430009900 tx=0x7f743002a480 comp rx=0 tx=0).stop 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.867+0000 7f744675a640 1 -- 172.21.15.62:0/4033373843 shutdown_connections 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.867+0000 7f744675a640 1 --2- 172.21.15.62:0/4033373843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7440102ef0 0x7f744010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.867+0000 7f744675a640 1 --2- 172.21.15.62:0/4033373843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7440102570 0x7f74401029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.867+0000 7f744675a640 1 --2- 172.21.15.62:0/4033373843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440100e20 0x7f7440101200 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.867+0000 7f744675a640 1 -- 172.21.15.62:0/4033373843 >> 172.21.15.62:0/4033373843 conn(0x7f74400fc7d0 msgr2=0x7f74400febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.867+0000 7f744675a640 1 -- 172.21.15.62:0/4033373843 shutdown_connections 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.868+0000 7f744675a640 1 -- 172.21.15.62:0/4033373843 wait complete. 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.868+0000 7f744675a640 1 Processor -- start 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.868+0000 7f744675a640 1 -- start start 2024-09-15T23:14:55.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f744675a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7440100e20 0x7f74401a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f744675a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440102570 0x7f74401a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f744675a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7440102ef0 0x7f744019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f744675a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7440110fa0 con 0x7f7440102570 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f744675a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7440110e20 con 0x7f7440100e20 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f744675a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7440111120 con 0x7f7440102ef0 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440102570 0x7f74401a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440102570 0x7f74401a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50090/0 (socket says 172.21.15.62:50090) 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743f7fe640 1 -- 172.21.15.62:0/3997274143 learned_addr learned my addr 172.21.15.62:0/3997274143 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743ffff640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7440100e20 0x7f74401a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f7444cf7640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7440102ef0 0x7f744019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743f7fe640 1 -- 172.21.15.62:0/3997274143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7440102ef0 msgr2=0x7f744019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743f7fe640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7440102ef0 0x7f744019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743f7fe640 1 -- 172.21.15.62:0/3997274143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7440100e20 msgr2=0x7f74401a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743f7fe640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7440100e20 0x7f74401a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743f7fe640 1 -- 172.21.15.62:0/3997274143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744019acf0 con 0x7f7440102570 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.869+0000 7f743f7fe640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440102570 0x7f74401a0940 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f743400d950 tx=0x7f743400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.870+0000 7f743d7fa640 1 -- 172.21.15.62:0/3997274143 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434017030 con 0x7f7440102570 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.870+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744019afe0 con 0x7f7440102570 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.870+0000 7f743d7fa640 1 -- 172.21.15.62:0/3997274143 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7434004020 con 0x7f7440102570 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.870+0000 7f743d7fa640 1 -- 172.21.15.62:0/3997274143 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340047a0 con 0x7f7440102570 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.870+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74401a72d0 con 0x7f7440102570 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.871+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7440101200 con 0x7f7440102570 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.872+0000 7f743d7fa640 1 -- 172.21.15.62:0/3997274143 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f74340041d0 con 0x7f7440102570 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.873+0000 7f743d7fa640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74280a32d0 0x7f74280a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.873+0000 7f743d7fa640 1 -- 172.21.15.62:0/3997274143 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f74340cfe60 con 0x7f7440102570 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.873+0000 7f743ffff640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74280a32d0 0x7f74280a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:55.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.874+0000 7f743ffff640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74280a32d0 0x7f74280a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7430000c00 tx=0x7f7430005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:55.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:55.877+0000 7f743d7fa640 1 -- 172.21.15.62:0/3997274143 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f743409c6e0 con 0x7f7440102570 2024-09-15T23:14:55.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:55 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1753257557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-15T23:14:56.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.056+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f74401a7560 con 0x7f7440102570 2024-09-15T23:14:56.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.056+0000 7f743d7fa640 1 -- 172.21.15.62:0/3997274143 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f743409bfe0 con 0x7f7440102570 2024-09-15T23:14:56.058 INFO:teuthology.orchestra.run.smithi062.stdout:214748364850 2024-09-15T23:14:56.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74280a32d0 msgr2=0x7f74280a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:56.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74280a32d0 0x7f74280a5790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7430000c00 tx=0x7f7430005b50 comp rx=0 tx=0).stop 2024-09-15T23:14:56.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440102570 msgr2=0x7f74401a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:56.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440102570 0x7f74401a0940 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f743400d950 tx=0x7f743400de10 comp rx=0 tx=0).stop 2024-09-15T23:14:56.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 shutdown_connections 2024-09-15T23:14:56.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74280a32d0 0x7f74280a5790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:56.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7440102ef0 0x7f744019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:56.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7440102570 0x7f74401a0940 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:56.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 --2- 172.21.15.62:0/3997274143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7440100e20 0x7f74401a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:56.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 >> 172.21.15.62:0/3997274143 conn(0x7f74400fc7d0 msgr2=0x7f744006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:56.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 shutdown_connections 2024-09-15T23:14:56.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:56.059+0000 7f744675a640 1 -- 172.21.15.62:0/3997274143 wait complete. 2024-09-15T23:14:56.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:55 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1753257557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-15T23:14:56.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:55 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1753257557' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-09-15T23:14:56.754 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743456 got 296352743458 for osd.10 2024-09-15T23:14:56.754 DEBUG:teuthology.parallel:result is None 2024-09-15T23:14:56.783 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:56 smithi062 ceph-mon[30949]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T23:14:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:56 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3997274143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-15T23:14:57.201 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:57.233 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790008 got 158913790012 for osd.5 2024-09-15T23:14:57.233 DEBUG:teuthology.parallel:result is None 2024-09-15T23:14:57.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:56 smithi067 ceph-mon[35665]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T23:14:57.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:56 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3997274143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-15T23:14:57.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:56 smithi148 ceph-mon[35610]: pgmap v328: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T23:14:57.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:56 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3997274143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-15T23:14:57.629 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528372 got 193273528375 for osd.6 2024-09-15T23:14:57.629 DEBUG:teuthology.parallel:result is None 2024-09-15T23:14:57.637 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:14:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 -- 172.21.15.62:0/3163471502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102ef0 msgr2=0x7fb04810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 --2- 172.21.15.62:0/3163471502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102ef0 0x7fb04810da40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb040009ca0 tx=0x7fb04002a950 comp rx=0 tx=0).stop 2024-09-15T23:14:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 -- 172.21.15.62:0/3163471502 shutdown_connections 2024-09-15T23:14:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 --2- 172.21.15.62:0/3163471502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102ef0 0x7fb04810da40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 --2- 172.21.15.62:0/3163471502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb048102570 0x7fb0481029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 --2- 172.21.15.62:0/3163471502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb048100e20 0x7fb048101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 -- 172.21.15.62:0/3163471502 >> 172.21.15.62:0/3163471502 conn(0x7fb0480fc7d0 msgr2=0x7fb0480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 -- 172.21.15.62:0/3163471502 shutdown_connections 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 -- 172.21.15.62:0/3163471502 wait complete. 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 Processor -- start 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.195+0000 7fb04f233640 1 -- start start 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb04f233640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb048100e20 0x7fb048198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb04f233640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102570 0x7fb048198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb04f233640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb048102ef0 0x7fb04819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb04f233640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb0481120a0 con 0x7fb048102570 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb04f233640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb048111f20 con 0x7fb048102ef0 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb04f233640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb048112220 con 0x7fb048100e20 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb047fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102570 0x7fb048198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb047fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102570 0x7fb048198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50110/0 (socket says 172.21.15.62:50110) 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb047fff640 1 -- 172.21.15.62:0/3079127650 learned_addr learned my addr 172.21.15.62:0/3079127650 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:58.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb04cfcf640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb048100e20 0x7fb048198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb04d7d0640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb048102ef0 0x7fb04819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb047fff640 1 -- 172.21.15.62:0/3079127650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb048100e20 msgr2=0x7fb048198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb047fff640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb048100e20 0x7fb048198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb047fff640 1 -- 172.21.15.62:0/3079127650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb048102ef0 msgr2=0x7fb04819cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb047fff640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb048102ef0 0x7fb04819cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb047fff640 1 -- 172.21.15.62:0/3079127650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04819d4b0 con 0x7fb048102570 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb047fff640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102570 0x7fb048198a40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb03400d770 tx=0x7fb03400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb045ffb640 1 -- 172.21.15.62:0/3079127650 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034017070 con 0x7fb048102570 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04819d7a0 con 0x7fb048102570 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb045ffb640 1 -- 172.21.15.62:0/3079127650 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb034005500 con 0x7fb048102570 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.196+0000 7fb045ffb640 1 -- 172.21.15.62:0/3079127650 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034004e00 con 0x7fb048102570 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.197+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0481a5080 con 0x7fb048102570 2024-09-15T23:14:58.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.198+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb048101200 con 0x7fb048102570 2024-09-15T23:14:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.199+0000 7fb045ffb640 1 -- 172.21.15.62:0/3079127650 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb034004fa0 con 0x7fb048102570 2024-09-15T23:14:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.199+0000 7fb045ffb640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0380a32d0 0x7fb0380a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.200+0000 7fb04cfcf640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0380a32d0 0x7fb0380a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.200+0000 7fb045ffb640 1 -- 172.21.15.62:0/3079127650 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fb0340cf750 con 0x7fb048102570 2024-09-15T23:14:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.200+0000 7fb04cfcf640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0380a32d0 0x7fb0380a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00bbe0 tx=0x7fb03c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:58.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.204+0000 7fb045ffb640 1 -- 172.21.15.62:0/3079127650 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb03409b980 con 0x7fb048102570 2024-09-15T23:14:58.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.385+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fb048109090 con 0x7fb048102570 2024-09-15T23:14:58.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.386+0000 7fb045ffb640 1 -- 172.21.15.62:0/3079127650 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb03409fc80 con 0x7fb048102570 2024-09-15T23:14:58.387 INFO:teuthology.orchestra.run.smithi062.stdout:244813135918 2024-09-15T23:14:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0380a32d0 msgr2=0x7fb0380a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0380a32d0 0x7fb0380a5790 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb03c00bbe0 tx=0x7fb03c00b390 comp rx=0 tx=0).stop 2024-09-15T23:14:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102570 msgr2=0x7fb048198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102570 0x7fb048198a40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb03400d770 tx=0x7fb03400dc30 comp rx=0 tx=0).stop 2024-09-15T23:14:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 shutdown_connections 2024-09-15T23:14:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0380a32d0 0x7fb0380a5790 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb048102ef0 0x7fb04819cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb048102570 0x7fb048198a40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 --2- 172.21.15.62:0/3079127650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb048100e20 0x7fb048198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 >> 172.21.15.62:0/3079127650 conn(0x7fb0480fc7d0 msgr2=0x7fb0481068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 shutdown_connections 2024-09-15T23:14:58.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.388+0000 7fb04f233640 1 -- 172.21.15.62:0/3079127650 wait complete. 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 -- 172.21.15.62:0/2495896750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 msgr2=0x7f3370100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 --2- 172.21.15.62:0/2495896750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 0x7f3370100ed0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3364009930 tx=0x7f336402a420 comp rx=0 tx=0).stop 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 -- 172.21.15.62:0/2495896750 shutdown_connections 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 --2- 172.21.15.62:0/2495896750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3370101410 0x7f337010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 --2- 172.21.15.62:0/2495896750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 0x7f3370100ed0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 --2- 172.21.15.62:0/2495896750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370106a70 0x7f3370106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 -- 172.21.15.62:0/2495896750 >> 172.21.15.62:0/2495896750 conn(0x7f33700fc790 msgr2=0x7f33700febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 -- 172.21.15.62:0/2495896750 shutdown_connections 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 -- 172.21.15.62:0/2495896750 wait complete. 2024-09-15T23:14:58.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 Processor -- start 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.827+0000 7f337655c640 1 -- start start 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f337655c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 0x7f33701992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f337655c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3370101410 0x7f3370199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f337655c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370106a70 0x7f33701a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f337655c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3370110e70 con 0x7f3370100a70 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f337655c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3370110cf0 con 0x7f3370106a70 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f337655c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3370110ff0 con 0x7f3370101410 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 0x7f33701992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 0x7f33701992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50136/0 (socket says 172.21.15.62:50136) 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336ffff640 1 -- 172.21.15.62:0/3613077530 learned_addr learned my addr 172.21.15.62:0/3613077530 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f3374af9640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370106a70 0x7f33701a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336f7fe640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3370101410 0x7f3370199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:58.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336ffff640 1 -- 172.21.15.62:0/3613077530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3370101410 msgr2=0x7f3370199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336ffff640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3370101410 0x7f3370199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336ffff640 1 -- 172.21.15.62:0/3613077530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370106a70 msgr2=0x7f33701a4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336ffff640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370106a70 0x7f33701a4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336ffff640 1 -- 172.21.15.62:0/3613077530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33701a4880 con 0x7f3370100a70 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.828+0000 7f336f7fe640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3370101410 0x7f3370199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.829+0000 7f336ffff640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 0x7f33701992c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f336000e9f0 tx=0x7f336000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.829+0000 7f336d7fa640 1 -- 172.21.15.62:0/3613077530 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336000cd80 con 0x7f3370100a70 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.829+0000 7f336d7fa640 1 -- 172.21.15.62:0/3613077530 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3360010910 con 0x7f3370100a70 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.829+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33701a4a50 con 0x7f3370100a70 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.829+0000 7f336d7fa640 1 -- 172.21.15.62:0/3613077530 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360018990 con 0x7f3370100a70 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.829+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33701a4f60 con 0x7f3370100a70 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.830+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3370100ed0 con 0x7f3370100a70 2024-09-15T23:14:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.831+0000 7f336d7fa640 1 -- 172.21.15.62:0/3613077530 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3360010ac0 con 0x7f3370100a70 2024-09-15T23:14:58.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.832+0000 7f336d7fa640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33580a3350 0x7f33580a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:58.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.832+0000 7f336f7fe640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33580a3350 0x7f33580a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:58.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.832+0000 7f336d7fa640 1 -- 172.21.15.62:0/3613077530 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f33600d04d0 con 0x7f3370100a70 2024-09-15T23:14:58.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.832+0000 7f336f7fe640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33580a3350 0x7f33580a5810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3364000c00 tx=0x7f3364005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:58.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:58.835+0000 7f336d7fa640 1 -- 172.21.15.62:0/3613077530 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f336009cd50 con 0x7f3370100a70 2024-09-15T23:14:59.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.012+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f337006ad80 con 0x7f3370100a70 2024-09-15T23:14:59.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:58 smithi062 ceph-mon[30949]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T23:14:59.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:58 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3079127650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-15T23:14:59.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.012+0000 7f336d7fa640 1 -- 172.21.15.62:0/3613077530 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f337006ad80 con 0x7f3370100a70 2024-09-15T23:14:59.014 INFO:teuthology.orchestra.run.smithi062.stdout:137438953538 2024-09-15T23:14:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33580a3350 msgr2=0x7f33580a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33580a3350 0x7f33580a5810 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3364000c00 tx=0x7f3364005a20 comp rx=0 tx=0).stop 2024-09-15T23:14:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 msgr2=0x7f33701992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:59.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 0x7f33701992c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f336000e9f0 tx=0x7f336000eeb0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 shutdown_connections 2024-09-15T23:14:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33580a3350 0x7f33580a5810 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3370106a70 0x7f33701a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3370101410 0x7f3370199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 --2- 172.21.15.62:0/3613077530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3370100a70 0x7f33701992c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 >> 172.21.15.62:0/3613077530 conn(0x7f33700fc790 msgr2=0x7f33700fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 shutdown_connections 2024-09-15T23:14:59.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.015+0000 7f337655c640 1 -- 172.21.15.62:0/3613077530 wait complete. 2024-09-15T23:14:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.166+0000 7f3e38622640 1 -- 172.21.15.62:0/2135292911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30100ca0 msgr2=0x7f3e30106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.166+0000 7f3e38622640 1 --2- 172.21.15.62:0/2135292911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30100ca0 0x7f3e30106840 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009930 tx=0x7f3e2c02a420 comp rx=0 tx=0).stop 2024-09-15T23:14:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.166+0000 7f3e38622640 1 -- 172.21.15.62:0/2135292911 shutdown_connections 2024-09-15T23:14:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.166+0000 7f3e38622640 1 --2- 172.21.15.62:0/2135292911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e30106f50 0x7f3e3010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.166+0000 7f3e38622640 1 --2- 172.21.15.62:0/2135292911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30100ca0 0x7f3e30106840 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.166+0000 7f3e38622640 1 --2- 172.21.15.62:0/2135292911 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e301002f0 0x7f3e301006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.166+0000 7f3e38622640 1 -- 172.21.15.62:0/2135292911 >> 172.21.15.62:0/2135292911 conn(0x7f3e30077210 msgr2=0x7f3e300fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.166+0000 7f3e38622640 1 -- 172.21.15.62:0/2135292911 shutdown_connections 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.166+0000 7f3e38622640 1 -- 172.21.15.62:0/2135292911 wait complete. 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.167+0000 7f3e38622640 1 Processor -- start 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.167+0000 7f3e38622640 1 -- start start 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.167+0000 7f3e38622640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e301002f0 0x7f3e30198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.167+0000 7f3e38622640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e30100ca0 0x7f3e301989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.167+0000 7f3e38622640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30106f50 0x7f3e3019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.167+0000 7f3e38622640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30111fa0 con 0x7f3e30106f50 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.167+0000 7f3e38622640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30111e20 con 0x7f3e301002f0 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.167+0000 7f3e38622640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3e30112120 con 0x7f3e30100ca0 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.167+0000 7f3e36bbf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30106f50 0x7f3e3019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e36bbf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30106f50 0x7f3e3019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50154/0 (socket says 172.21.15.62:50154) 2024-09-15T23:14:59.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e36bbf640 1 -- 172.21.15.62:0/856457953 learned_addr learned my addr 172.21.15.62:0/856457953 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e35bbd640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e30100ca0 0x7f3e301989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e363be640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e301002f0 0x7f3e30198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e36bbf640 1 -- 172.21.15.62:0/856457953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e30100ca0 msgr2=0x7f3e301989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e36bbf640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e30100ca0 0x7f3e301989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e36bbf640 1 -- 172.21.15.62:0/856457953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e301002f0 msgr2=0x7f3e30198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e36bbf640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e301002f0 0x7f3e30198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e36bbf640 1 -- 172.21.15.62:0/856457953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3019d440 con 0x7f3e30106f50 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e35bbd640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e30100ca0 0x7f3e301989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e36bbf640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30106f50 0x7f3e3019cd60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800eaa0 tx=0x7f3e2800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e277fe640 1 -- 172.21.15.62:0/856457953 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800cdb0 con 0x7f3e30106f50 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e277fe640 1 -- 172.21.15.62:0/856457953 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e28005020 con 0x7f3e30106f50 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3019d730 con 0x7f3e30106f50 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e277fe640 1 -- 172.21.15.62:0/856457953 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e280116b0 con 0x7f3e30106f50 2024-09-15T23:14:59.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.168+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e301a5010 con 0x7f3e30106f50 2024-09-15T23:14:59.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.169+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e301006d0 con 0x7f3e30106f50 2024-09-15T23:14:59.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.170+0000 7f3e277fe640 1 -- 172.21.15.62:0/856457953 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3e28005560 con 0x7f3e30106f50 2024-09-15T23:14:59.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.171+0000 7f3e277fe640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e1c0a32d0 0x7f3e1c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:14:59.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.171+0000 7f3e363be640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e1c0a32d0 0x7f3e1c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:14:59.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.171+0000 7f3e277fe640 1 -- 172.21.15.62:0/856457953 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f3e280d00e0 con 0x7f3e30106f50 2024-09-15T23:14:59.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.172+0000 7f3e363be640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e1c0a32d0 0x7f3e1c0a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3e20005fd0 tx=0x7f3e20005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:14:59.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.175+0000 7f3e277fe640 1 -- 172.21.15.62:0/856457953 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3e28016030 con 0x7f3e30106f50 2024-09-15T23:14:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:58 smithi067 ceph-mon[35665]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T23:14:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:58 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3079127650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-15T23:14:59.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:58 smithi148 ceph-mon[35610]: pgmap v329: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-15T23:14:59.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:58 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3079127650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-09-15T23:14:59.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.356+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f3e30109b90 con 0x7f3e30106f50 2024-09-15T23:14:59.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.356+0000 7f3e277fe640 1 -- 172.21.15.62:0/856457953 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f3e2809c900 con 0x7f3e30106f50 2024-09-15T23:14:59.358 INFO:teuthology.orchestra.run.smithi062.stdout:326417514526 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.358+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e1c0a32d0 msgr2=0x7f3e1c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e1c0a32d0 0x7f3e1c0a5790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3e20005fd0 tx=0x7f3e20005850 comp rx=0 tx=0).stop 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30106f50 msgr2=0x7f3e3019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30106f50 0x7f3e3019cd60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800eaa0 tx=0x7f3e2800ef60 comp rx=0 tx=0).stop 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 shutdown_connections 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e1c0a32d0 0x7f3e1c0a5790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e30106f50 0x7f3e3019cd60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e30100ca0 0x7f3e301989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 --2- 172.21.15.62:0/856457953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e301002f0 0x7f3e30198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:14:59.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 >> 172.21.15.62:0/856457953 conn(0x7f3e30077210 msgr2=0x7f3e300fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:14:59.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 shutdown_connections 2024-09-15T23:14:59.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:14:59.359+0000 7f3e38622640 1 -- 172.21.15.62:0/856457953 wait complete. 2024-09-15T23:14:59.634 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364847 got 214748364850 for osd.7 2024-09-15T23:14:59.634 DEBUG:teuthology.parallel:result is None 2024-09-15T23:15:00.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:59 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3613077530' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-15T23:15:00.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:14:59 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/856457953' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-15T23:15:00.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:59 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3613077530' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-15T23:15:00.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:14:59 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/856457953' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-15T23:15:00.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:59 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3613077530' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-15T23:15:00.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:14:59 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/856457953' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-09-15T23:15:00.296 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953533 got 137438953538 for osd.4 2024-09-15T23:15:00.296 DEBUG:teuthology.parallel:result is None 2024-09-15T23:15:00.954 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514524 got 326417514526 for osd.11 2024-09-15T23:15:00.954 DEBUG:teuthology.parallel:result is None 2024-09-15T23:15:01.163 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:00 smithi067 ceph-mon[35665]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:01.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:00 smithi062 ceph-mon[30949]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:00 smithi148 ceph-mon[35610]: pgmap v330: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:01.441 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135914 got 244813135918 for osd.8 2024-09-15T23:15:01.442 DEBUG:teuthology.parallel:result is None 2024-09-15T23:15:01.442 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-15T23:15:01.442 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:15:01.705 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:02.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.175+0000 7f387043b640 1 -- 172.21.15.62:0/3558515052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868101420 msgr2=0x7f386810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:02.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.175+0000 7f387043b640 1 --2- 172.21.15.62:0/3558515052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868101420 0x7f386810d950 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3860009930 tx=0x7f386002a420 comp rx=0 tx=0).stop 2024-09-15T23:15:02.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.176+0000 7f387043b640 1 -- 172.21.15.62:0/3558515052 shutdown_connections 2024-09-15T23:15:02.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.176+0000 7f387043b640 1 --2- 172.21.15.62:0/3558515052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868101420 0x7f386810d950 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.176+0000 7f387043b640 1 --2- 172.21.15.62:0/3558515052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3868100a80 0x7f3868100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.176+0000 7f387043b640 1 --2- 172.21.15.62:0/3558515052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3868106a80 0x7f3868106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.176+0000 7f387043b640 1 -- 172.21.15.62:0/3558515052 >> 172.21.15.62:0/3558515052 conn(0x7f38680fc7a0 msgr2=0x7f38680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.176+0000 7f387043b640 1 -- 172.21.15.62:0/3558515052 shutdown_connections 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.176+0000 7f387043b640 1 -- 172.21.15.62:0/3558515052 wait complete. 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.176+0000 7f387043b640 1 Processor -- start 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.176+0000 7f387043b640 1 -- start start 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f387043b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868100a80 0x7f38680fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f387043b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3868101420 0x7f38680fd320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f387043b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3868106a80 0x7f38680fd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f387043b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3868110e80 con 0x7f3868100a80 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f387043b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3868110d00 con 0x7f3868101420 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f387043b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3868111000 con 0x7f3868106a80 2024-09-15T23:15:02.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e1d7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868100a80 0x7f38680fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e1d7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868100a80 0x7f38680fcde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50164/0 (socket says 172.21.15.62:50164) 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e1d7640 1 -- 172.21.15.62:0/2342210763 learned_addr learned my addr 172.21.15.62:0/2342210763 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386d9d6640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3868101420 0x7f38680fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e9d8640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3868106a80 0x7f38680fd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e1d7640 1 -- 172.21.15.62:0/2342210763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3868106a80 msgr2=0x7f38680fd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e1d7640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3868106a80 0x7f38680fd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e1d7640 1 -- 172.21.15.62:0/2342210763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3868101420 msgr2=0x7f38680fd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e1d7640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3868101420 0x7f38680fd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e1d7640 1 -- 172.21.15.62:0/2342210763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38681a6950 con 0x7f3868100a80 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386d9d6640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3868101420 0x7f38680fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e9d8640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3868106a80 0x7f38680fd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f386e1d7640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868100a80 0x7f38680fcde0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f385800d770 tx=0x7f385800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:02.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f38677fe640 1 -- 172.21.15.62:0/2342210763 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858017070 con 0x7f3868100a80 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38681a6c40 con 0x7f3868100a80 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f38677fe640 1 -- 172.21.15.62:0/2342210763 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3858005500 con 0x7f3868100a80 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.177+0000 7f38677fe640 1 -- 172.21.15.62:0/2342210763 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3858004e00 con 0x7f3868100a80 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.178+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38681a7130 con 0x7f3868100a80 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.178+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3868100ee0 con 0x7f3868100a80 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.180+0000 7f38677fe640 1 -- 172.21.15.62:0/2342210763 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3858004fa0 con 0x7f3868100a80 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.180+0000 7f38677fe640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f38500a32d0 0x7f38500a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.180+0000 7f386d9d6640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f38500a32d0 0x7f38500a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.180+0000 7f38677fe640 1 -- 172.21.15.62:0/2342210763 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f38580cf860 con 0x7f3868100a80 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.181+0000 7f386d9d6640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f38500a32d0 0x7f38500a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f38680fe870 tx=0x7f385c009520 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:02.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.184+0000 7f38677fe640 1 -- 172.21.15.62:0/2342210763 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f385809c0e0 con 0x7f3868100a80 2024-09-15T23:15:02.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.356+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3868062630 con 0x7f38500a32d0 2024-09-15T23:15:02.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.365+0000 7f38677fe640 1 -- 172.21.15.62:0/2342210763 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340611 (secure 0 0 0) 0x7f3868062630 con 0x7f38500a32d0 2024-09-15T23:15:02.383 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:02.385 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:15:02.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.368+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f38500a32d0 msgr2=0x7f38500a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.368+0000 7f387043b640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f38500a32d0 0x7f38500a5790 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f38680fe870 tx=0x7f385c009520 comp rx=0 tx=0).stop 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.368+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868100a80 msgr2=0x7f38680fcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.368+0000 7f387043b640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868100a80 0x7f38680fcde0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f385800d770 tx=0x7f385800dc30 comp rx=0 tx=0).stop 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.369+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 shutdown_connections 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.369+0000 7f387043b640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f38500a32d0 0x7f38500a5790 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.369+0000 7f387043b640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3868106a80 0x7f38680fd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.369+0000 7f387043b640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3868101420 0x7f38680fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.369+0000 7f387043b640 1 --2- 172.21.15.62:0/2342210763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3868100a80 0x7f38680fcde0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.369+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 >> 172.21.15.62:0/2342210763 conn(0x7f38680fc7a0 msgr2=0x7f386806aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.369+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 shutdown_connections 2024-09-15T23:15:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:02.369+0000 7f387043b640 1 -- 172.21.15.62:0/2342210763 wait complete. 2024-09-15T23:15:03.117 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:02 smithi062 ceph-mon[30949]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:03.117 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:02 smithi062 ceph-mon[30949]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:03.146 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-15T23:15:01.713156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657536,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":472,"num_read_kb":353,"num_write":351,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282,"ondisk_log_size":282,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365548,"kb_used_data":12068,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576276,"statfs":{"total":1151940427776,"available":1151566106624,"internally_reserved":0,"allocated":12357632,"data_stored":7989266,"data_compressed":389796,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003419"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008791+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:14:42.008791+0000","last_peered":"2024-09-15T23:14:42.008791+0000","last_clean":"2024-09-15T23:14:42.008791+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:14:42.008791+0000","last_undegraded":"2024-09-15T23:14:42.008791+0000","last_fullsized":"2024-09-15T23:14:42.008791+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007527+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:14:42.007527+0000","last_peered":"2024-09-15T23:14:42.007527+0000","last_clean":"2024-09-15T23:14:42.007527+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:14:42.007527+0000","last_undegraded":"2024-09-15T23:14:42.007527+0000","last_fullsized":"2024-09-15T23:14:42.007527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638628+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:14:41.638628+0000","last_peered":"2024-09-15T23:14:41.638628+0000","last_clean":"2024-09-15T23:14:41.638628+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:14:41.638628+0000","last_undegraded":"2024-09-15T23:14:41.638628+0000","last_fullsized":"2024-09-15T23:14:41.638628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007826+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:14:42.007826+0000","last_peered":"2024-09-15T23:14:42.007826+0000","last_clean":"2024-09-15T23:14:42.007826+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:14:42.007826+0000","last_undegraded":"2024-09-15T23:14:42.007826+0000","last_fullsized":"2024-09-15T23:14:42.007826+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634945+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:14:41.634945+0000","last_peered":"2024-09-15T23:14:41.634945+0000","last_clean":"2024-09-15T23:14:41.634945+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:14:41.634945+0000","last_undegraded":"2024-09-15T23:14:41.634945+0000","last_fullsized":"2024-09-15T23:14:41.634945+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635661+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:14:41.635661+0000","last_peered":"2024-09-15T23:14:41.635661+0000","last_clean":"2024-09-15T23:14:41.635661+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:14:41.635661+0000","last_undegraded":"2024-09-15T23:14:41.635661+0000","last_fullsized":"2024-09-15T23:14:41.635661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.660261+0000","last_change":"2024-09-15T23:14:41.659816+0000","last_active":"2024-09-15T23:14:41.660261+0000","last_peered":"2024-09-15T23:14:41.660261+0000","last_clean":"2024-09-15T23:14:41.660261+0000","last_became_active":"2024-09-15T23:14:41.641611+0000","last_became_peered":"2024-09-15T23:14:41.641611+0000","last_unstale":"2024-09-15T23:14:41.660261+0000","last_undegraded":"2024-09-15T23:14:41.660261+0000","last_fullsized":"2024-09-15T23:14:41.660261+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:23.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633717+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:14:41.633717+0000","last_peered":"2024-09-15T23:14:41.633717+0000","last_clean":"2024-09-15T23:14:41.633717+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:14:41.633717+0000","last_undegraded":"2024-09-15T23:14:41.633717+0000","last_fullsized":"2024-09-15T23:14:41.633717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636932+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:14:41.636932+0000","last_peered":"2024-09-15T23:14:41.636932+0000","last_clean":"2024-09-15T23:14:41.636932+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:14:41.636932+0000","last_undegraded":"2024-09-15T23:14:41.636932+0000","last_fullsized":"2024-09-15T23:14:41.636932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636429+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:14:41.636429+0000","last_peered":"2024-09-15T23:14:41.636429+0000","last_clean":"2024-09-15T23:14:41.636429+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:14:41.636429+0000","last_undegraded":"2024-09-15T23:14:41.636429+0000","last_fullsized":"2024-09-15T23:14:41.636429+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632933+0000","last_change":"2024-09-15T23:13:32.511232+0000","last_active":"2024-09-15T23:14:41.632933+0000","last_peered":"2024-09-15T23:14:41.632933+0000","last_clean":"2024-09-15T23:14:41.632933+0000","last_became_active":"2024-09-15T23:13:32.510875+0000","last_became_peered":"2024-09-15T23:13:32.510875+0000","last_unstale":"2024-09-15T23:14:41.632933+0000","last_undegraded":"2024-09-15T23:14:41.632933+0000","last_fullsized":"2024-09-15T23:14:41.632933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:54:29.918573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635945+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:14:41.635945+0000","last_peered":"2024-09-15T23:14:41.635945+0000","last_clean":"2024-09-15T23:14:41.635945+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:14:41.635945+0000","last_undegraded":"2024-09-15T23:14:41.635945+0000","last_fullsized":"2024-09-15T23:14:41.635945+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638773+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:14:41.638773+0000","last_peered":"2024-09-15T23:14:41.638773+0000","last_clean":"2024-09-15T23:14:41.638773+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:14:41.638773+0000","last_undegraded":"2024-09-15T23:14:41.638773+0000","last_fullsized":"2024-09-15T23:14:41.638773+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.646320+0000","last_change":"2024-09-15T23:14:41.646320+0000","last_active":"2024-09-15T23:14:41.646320+0000","last_peered":"2024-09-15T23:14:41.646320+0000","last_clean":"2024-09-15T23:14:41.646320+0000","last_became_active":"2024-09-15T23:14:41.641933+0000","last_became_peered":"2024-09-15T23:14:41.641933+0000","last_unstale":"2024-09-15T23:14:41.646320+0000","last_undegraded":"2024-09-15T23:14:41.646320+0000","last_fullsized":"2024-09-15T23:14:41.646320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:49.417407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635920+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:14:41.635920+0000","last_peered":"2024-09-15T23:14:41.635920+0000","last_clean":"2024-09-15T23:14:41.635920+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:14:41.635920+0000","last_undegraded":"2024-09-15T23:14:41.635920+0000","last_fullsized":"2024-09-15T23:14:41.635920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007641+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:14:42.007641+0000","last_peered":"2024-09-15T23:14:42.007641+0000","last_clean":"2024-09-15T23:14:42.007641+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:14:42.007641+0000","last_undegraded":"2024-09-15T23:14:42.007641+0000","last_fullsized":"2024-09-15T23:14:42.007641+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639214+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:14:41.639214+0000","last_peered":"2024-09-15T23:14:41.639214+0000","last_clean":"2024-09-15T23:14:41.639214+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:14:41.639214+0000","last_undegraded":"2024-09-15T23:14:41.639214+0000","last_fullsized":"2024-09-15T23:14:41.639214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636523+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:14:41.636523+0000","last_peered":"2024-09-15T23:14:41.636523+0000","last_clean":"2024-09-15T23:14:41.636523+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:14:41.636523+0000","last_undegraded":"2024-09-15T23:14:41.636523+0000","last_fullsized":"2024-09-15T23:14:41.636523+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635294+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:14:41.635294+0000","last_peered":"2024-09-15T23:14:41.635294+0000","last_clean":"2024-09-15T23:14:41.635294+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:14:41.635294+0000","last_undegraded":"2024-09-15T23:14:41.635294+0000","last_fullsized":"2024-09-15T23:14:41.635294+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008225+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:14:42.008225+0000","last_peered":"2024-09-15T23:14:42.008225+0000","last_clean":"2024-09-15T23:14:42.008225+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:14:42.008225+0000","last_undegraded":"2024-09-15T23:14:42.008225+0000","last_fullsized":"2024-09-15T23:14:42.008225+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008480+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:14:42.008480+0000","last_peered":"2024-09-15T23:14:42.008480+0000","last_clean":"2024-09-15T23:14:42.008480+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:14:42.008480+0000","last_undegraded":"2024-09-15T23:14:42.008480+0000","last_fullsized":"2024-09-15T23:14:42.008480+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635579+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:14:41.635579+0000","last_peered":"2024-09-15T23:14:41.635579+0000","last_clean":"2024-09-15T23:14:41.635579+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:14:41.635579+0000","last_undegraded":"2024-09-15T23:14:41.635579+0000","last_fullsized":"2024-09-15T23:14:41.635579+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633263+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:14:41.633263+0000","last_peered":"2024-09-15T23:14:41.633263+0000","last_clean":"2024-09-15T23:14:41.633263+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:14:41.633263+0000","last_undegraded":"2024-09-15T23:14:41.633263+0000","last_fullsized":"2024-09-15T23:14:41.633263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636123+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:14:41.636123+0000","last_peered":"2024-09-15T23:14:41.636123+0000","last_clean":"2024-09-15T23:14:41.636123+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:14:41.636123+0000","last_undegraded":"2024-09-15T23:14:41.636123+0000","last_fullsized":"2024-09-15T23:14:41.636123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.009605+0000","last_change":"2024-09-15T23:14:42.009605+0000","last_active":"2024-09-15T23:14:42.009605+0000","last_peered":"2024-09-15T23:14:42.009605+0000","last_clean":"2024-09-15T23:14:42.009605+0000","last_became_active":"2024-09-15T23:14:42.009237+0000","last_became_peered":"2024-09-15T23:14:42.009237+0000","last_unstale":"2024-09-15T23:14:42.009605+0000","last_undegraded":"2024-09-15T23:14:42.009605+0000","last_fullsized":"2024-09-15T23:14:42.009605+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:11:08.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.637301+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:14:41.637301+0000","last_peered":"2024-09-15T23:14:41.637301+0000","last_clean":"2024-09-15T23:14:41.637301+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:14:41.637301+0000","last_undegraded":"2024-09-15T23:14:41.637301+0000","last_fullsized":"2024-09-15T23:14:41.637301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639750+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:14:41.639750+0000","last_peered":"2024-09-15T23:14:41.639750+0000","last_clean":"2024-09-15T23:14:41.639750+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:14:41.639750+0000","last_undegraded":"2024-09-15T23:14:41.639750+0000","last_fullsized":"2024-09-15T23:14:41.639750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008151+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:14:42.008151+0000","last_peered":"2024-09-15T23:14:42.008151+0000","last_clean":"2024-09-15T23:14:42.008151+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:14:42.008151+0000","last_undegraded":"2024-09-15T23:14:42.008151+0000","last_fullsized":"2024-09-15T23:14:42.008151+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'12","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008816+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:14:42.008816+0000","last_peered":"2024-09-15T23:14:42.008816+0000","last_clean":"2024-09-15T23:14:42.008816+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:14:42.008816+0000","last_undegraded":"2024-09-15T23:14:42.008816+0000","last_fullsized":"2024-09-15T23:14:42.008816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007629+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:14:42.007629+0000","last_peered":"2024-09-15T23:14:42.007629+0000","last_clean":"2024-09-15T23:14:42.007629+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:14:42.007629+0000","last_undegraded":"2024-09-15T23:14:42.007629+0000","last_fullsized":"2024-09-15T23:14:42.007629+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631552+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:14:41.631552+0000","last_peered":"2024-09-15T23:14:41.631552+0000","last_clean":"2024-09-15T23:14:41.631552+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:14:41.631552+0000","last_undegraded":"2024-09-15T23:14:41.631552+0000","last_fullsized":"2024-09-15T23:14:41.631552+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636929+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:14:41.636929+0000","last_peered":"2024-09-15T23:14:41.636929+0000","last_clean":"2024-09-15T23:14:41.636929+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:14:41.636929+0000","last_undegraded":"2024-09-15T23:14:41.636929+0000","last_fullsized":"2024-09-15T23:14:41.636929+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635760+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:14:41.635760+0000","last_peered":"2024-09-15T23:14:41.635760+0000","last_clean":"2024-09-15T23:14:41.635760+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:14:41.635760+0000","last_undegraded":"2024-09-15T23:14:41.635760+0000","last_fullsized":"2024-09-15T23:14:41.635760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993522+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:14:41.993522+0000","last_peered":"2024-09-15T23:14:41.993522+0000","last_clean":"2024-09-15T23:14:41.993522+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:14:41.993522+0000","last_undegraded":"2024-09-15T23:14:41.993522+0000","last_fullsized":"2024-09-15T23:14:41.993522+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.009055+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:14:42.009055+0000","last_peered":"2024-09-15T23:14:42.009055+0000","last_clean":"2024-09-15T23:14:42.009055+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:14:42.009055+0000","last_undegraded":"2024-09-15T23:14:42.009055+0000","last_fullsized":"2024-09-15T23:14:42.009055+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.995466+0000","last_change":"2024-09-15T23:14:41.995466+0000","last_active":"2024-09-15T23:14:41.995466+0000","last_peered":"2024-09-15T23:14:41.995466+0000","last_clean":"2024-09-15T23:14:41.995466+0000","last_became_active":"2024-09-15T23:14:41.994915+0000","last_became_peered":"2024-09-15T23:14:41.994915+0000","last_unstale":"2024-09-15T23:14:41.995466+0000","last_undegraded":"2024-09-15T23:14:41.995466+0000","last_fullsized":"2024-09-15T23:14:41.995466+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:04.121596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993461+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:14:41.993461+0000","last_peered":"2024-09-15T23:14:41.993461+0000","last_clean":"2024-09-15T23:14:41.993461+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:14:41.993461+0000","last_undegraded":"2024-09-15T23:14:41.993461+0000","last_fullsized":"2024-09-15T23:14:41.993461+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639308+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:14:41.639308+0000","last_peered":"2024-09-15T23:14:41.639308+0000","last_clean":"2024-09-15T23:14:41.639308+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:14:41.639308+0000","last_undegraded":"2024-09-15T23:14:41.639308+0000","last_fullsized":"2024-09-15T23:14:41.639308+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631244+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:14:41.631244+0000","last_peered":"2024-09-15T23:14:41.631244+0000","last_clean":"2024-09-15T23:14:41.631244+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:14:41.631244+0000","last_undegraded":"2024-09-15T23:14:41.631244+0000","last_fullsized":"2024-09-15T23:14:41.631244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632290+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:14:41.632290+0000","last_peered":"2024-09-15T23:14:41.632290+0000","last_clean":"2024-09-15T23:14:41.632290+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:14:41.632290+0000","last_undegraded":"2024-09-15T23:14:41.632290+0000","last_fullsized":"2024-09-15T23:14:41.632290+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:58.557277+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:14:58.557277+0000","last_peered":"2024-09-15T23:14:58.557277+0000","last_clean":"2024-09-15T23:14:58.557277+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:14:58.557277+0000","last_undegraded":"2024-09-15T23:14:58.557277+0000","last_fullsized":"2024-09-15T23:14:58.557277+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631849+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:14:41.631849+0000","last_peered":"2024-09-15T23:14:41.631849+0000","last_clean":"2024-09-15T23:14:41.631849+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:14:41.631849+0000","last_undegraded":"2024-09-15T23:14:41.631849+0000","last_fullsized":"2024-09-15T23:14:41.631849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.630899+0000","last_change":"2024-09-15T23:13:32.513091+0000","last_active":"2024-09-15T23:14:41.630899+0000","last_peered":"2024-09-15T23:14:41.630899+0000","last_clean":"2024-09-15T23:14:41.630899+0000","last_became_active":"2024-09-15T23:13:32.512752+0000","last_became_peered":"2024-09-15T23:13:32.512752+0000","last_unstale":"2024-09-15T23:14:41.630899+0000","last_undegraded":"2024-09-15T23:14:41.630899+0000","last_fullsized":"2024-09-15T23:14:41.630899+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:48:58.131659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638988+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:14:41.638988+0000","last_peered":"2024-09-15T23:14:41.638988+0000","last_clean":"2024-09-15T23:14:41.638988+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:14:41.638988+0000","last_undegraded":"2024-09-15T23:14:41.638988+0000","last_fullsized":"2024-09-15T23:14:41.638988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632857+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:14:41.632857+0000","last_peered":"2024-09-15T23:14:41.632857+0000","last_clean":"2024-09-15T23:14:41.632857+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:14:41.632857+0000","last_undegraded":"2024-09-15T23:14:41.632857+0000","last_fullsized":"2024-09-15T23:14:41.632857+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008444+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:14:42.008444+0000","last_peered":"2024-09-15T23:14:42.008444+0000","last_clean":"2024-09-15T23:14:42.008444+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:14:42.008444+0000","last_undegraded":"2024-09-15T23:14:42.008444+0000","last_fullsized":"2024-09-15T23:14:42.008444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636821+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:14:41.636821+0000","last_peered":"2024-09-15T23:14:41.636821+0000","last_clean":"2024-09-15T23:14:41.636821+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:14:41.636821+0000","last_undegraded":"2024-09-15T23:14:41.636821+0000","last_fullsized":"2024-09-15T23:14:41.636821+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640419+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:14:41.640419+0000","last_peered":"2024-09-15T23:14:41.640419+0000","last_clean":"2024-09-15T23:14:41.640419+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:14:41.640419+0000","last_undegraded":"2024-09-15T23:14:41.640419+0000","last_fullsized":"2024-09-15T23:14:41.640419+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640417+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:14:41.640417+0000","last_peered":"2024-09-15T23:14:41.640417+0000","last_clean":"2024-09-15T23:14:41.640417+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:14:41.640417+0000","last_undegraded":"2024-09-15T23:14:41.640417+0000","last_fullsized":"2024-09-15T23:14:41.640417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640638+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:14:41.640638+0000","last_peered":"2024-09-15T23:14:41.640638+0000","last_clean":"2024-09-15T23:14:41.640638+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:14:41.640638+0000","last_undegraded":"2024-09-15T23:14:41.640638+0000","last_fullsized":"2024-09-15T23:14:41.640638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639748+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:14:41.639748+0000","last_peered":"2024-09-15T23:14:41.639748+0000","last_clean":"2024-09-15T23:14:41.639748+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:14:41.639748+0000","last_undegraded":"2024-09-15T23:14:41.639748+0000","last_fullsized":"2024-09-15T23:14:41.639748+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.994950+0000","last_change":"2024-09-15T23:14:41.994950+0000","last_active":"2024-09-15T23:14:41.994950+0000","last_peered":"2024-09-15T23:14:41.994950+0000","last_clean":"2024-09-15T23:14:41.994950+0000","last_became_active":"2024-09-15T23:14:41.994429+0000","last_became_peered":"2024-09-15T23:14:41.994429+0000","last_unstale":"2024-09-15T23:14:41.994950+0000","last_undegraded":"2024-09-15T23:14:41.994950+0000","last_fullsized":"2024-09-15T23:14:41.994950+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:24.056756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639610+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:14:41.639610+0000","last_peered":"2024-09-15T23:14:41.639610+0000","last_clean":"2024-09-15T23:14:41.639610+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:14:41.639610+0000","last_undegraded":"2024-09-15T23:14:41.639610+0000","last_fullsized":"2024-09-15T23:14:41.639610+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633147+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:14:41.633147+0000","last_peered":"2024-09-15T23:14:41.633147+0000","last_clean":"2024-09-15T23:14:41.633147+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:14:41.633147+0000","last_undegraded":"2024-09-15T23:14:41.633147+0000","last_fullsized":"2024-09-15T23:14:41.633147+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640471+0000","last_change":"2024-09-15T23:14:41.640471+0000","last_active":"2024-09-15T23:14:41.640471+0000","last_peered":"2024-09-15T23:14:41.640471+0000","last_clean":"2024-09-15T23:14:41.640471+0000","last_became_active":"2024-09-15T23:14:41.640017+0000","last_became_peered":"2024-09-15T23:14:41.640017+0000","last_unstale":"2024-09-15T23:14:41.640471+0000","last_undegraded":"2024-09-15T23:14:41.640471+0000","last_fullsized":"2024-09-15T23:14:41.640471+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:58.842636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639778+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:14:41.639778+0000","last_peered":"2024-09-15T23:14:41.639778+0000","last_clean":"2024-09-15T23:14:41.639778+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:14:41.639778+0000","last_undegraded":"2024-09-15T23:14:41.639778+0000","last_fullsized":"2024-09-15T23:14:41.639778+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631856+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:14:41.631856+0000","last_peered":"2024-09-15T23:14:41.631856+0000","last_clean":"2024-09-15T23:14:41.631856+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:14:41.631856+0000","last_undegraded":"2024-09-15T23:14:41.631856+0000","last_fullsized":"2024-09-15T23:14:41.631856+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632120+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:14:41.632120+0000","last_peered":"2024-09-15T23:14:41.632120+0000","last_clean":"2024-09-15T23:14:41.632120+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:14:41.632120+0000","last_undegraded":"2024-09-15T23:14:41.632120+0000","last_fullsized":"2024-09-15T23:14:41.632120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635550+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:14:41.635550+0000","last_peered":"2024-09-15T23:14:41.635550+0000","last_clean":"2024-09-15T23:14:41.635550+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:14:41.635550+0000","last_undegraded":"2024-09-15T23:14:41.635550+0000","last_fullsized":"2024-09-15T23:14:41.635550+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636227+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:14:41.636227+0000","last_peered":"2024-09-15T23:14:41.636227+0000","last_clean":"2024-09-15T23:14:41.636227+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:14:41.636227+0000","last_undegraded":"2024-09-15T23:14:41.636227+0000","last_fullsized":"2024-09-15T23:14:41.636227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632125+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:14:41.632125+0000","last_peered":"2024-09-15T23:14:41.632125+0000","last_clean":"2024-09-15T23:14:41.632125+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:14:41.632125+0000","last_undegraded":"2024-09-15T23:14:41.632125+0000","last_fullsized":"2024-09-15T23:14:41.632125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632302+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:14:41.632302+0000","last_peered":"2024-09-15T23:14:41.632302+0000","last_clean":"2024-09-15T23:14:41.632302+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:14:41.632302+0000","last_undegraded":"2024-09-15T23:14:41.632302+0000","last_fullsized":"2024-09-15T23:14:41.632302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.654253+0000","last_change":"2024-09-15T23:14:41.653786+0000","last_active":"2024-09-15T23:14:41.654253+0000","last_peered":"2024-09-15T23:14:41.654253+0000","last_clean":"2024-09-15T23:14:41.654253+0000","last_became_active":"2024-09-15T23:14:41.641472+0000","last_became_peered":"2024-09-15T23:14:41.641472+0000","last_unstale":"2024-09-15T23:14:41.654253+0000","last_undegraded":"2024-09-15T23:14:41.654253+0000","last_fullsized":"2024-09-15T23:14:41.654253+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:16.010142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632061+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:14:41.632061+0000","last_peered":"2024-09-15T23:14:41.632061+0000","last_clean":"2024-09-15T23:14:41.632061+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:14:41.632061+0000","last_undegraded":"2024-09-15T23:14:41.632061+0000","last_fullsized":"2024-09-15T23:14:41.632061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638792+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:14:41.638792+0000","last_peered":"2024-09-15T23:14:41.638792+0000","last_clean":"2024-09-15T23:14:41.638792+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:14:41.638792+0000","last_undegraded":"2024-09-15T23:14:41.638792+0000","last_fullsized":"2024-09-15T23:14:41.638792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:58.557738+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:14:58.557738+0000","last_peered":"2024-09-15T23:14:58.557738+0000","last_clean":"2024-09-15T23:14:58.557738+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:14:58.557738+0000","last_undegraded":"2024-09-15T23:14:58.557738+0000","last_fullsized":"2024-09-15T23:14:58.557738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008576+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:14:42.008576+0000","last_peered":"2024-09-15T23:14:42.008576+0000","last_clean":"2024-09-15T23:14:42.008576+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:14:42.008576+0000","last_undegraded":"2024-09-15T23:14:42.008576+0000","last_fullsized":"2024-09-15T23:14:42.008576+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639361+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:14:41.639361+0000","last_peered":"2024-09-15T23:14:41.639361+0000","last_clean":"2024-09-15T23:14:41.639361+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:14:41.639361+0000","last_undegraded":"2024-09-15T23:14:41.639361+0000","last_fullsized":"2024-09-15T23:14:41.639361+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632532+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:14:41.632532+0000","last_peered":"2024-09-15T23:14:41.632532+0000","last_clean":"2024-09-15T23:14:41.632532+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:14:41.632532+0000","last_undegraded":"2024-09-15T23:14:41.632532+0000","last_fullsized":"2024-09-15T23:14:41.632532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635193+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:14:41.635193+0000","last_peered":"2024-09-15T23:14:41.635193+0000","last_clean":"2024-09-15T23:14:41.635193+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:14:41.635193+0000","last_undegraded":"2024-09-15T23:14:41.635193+0000","last_fullsized":"2024-09-15T23:14:41.635193+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631322+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:14:41.631322+0000","last_peered":"2024-09-15T23:14:41.631322+0000","last_clean":"2024-09-15T23:14:41.631322+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:14:41.631322+0000","last_undegraded":"2024-09-15T23:14:41.631322+0000","last_fullsized":"2024-09-15T23:14:41.631322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640350+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:14:41.640350+0000","last_peered":"2024-09-15T23:14:41.640350+0000","last_clean":"2024-09-15T23:14:41.640350+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:14:41.640350+0000","last_undegraded":"2024-09-15T23:14:41.640350+0000","last_fullsized":"2024-09-15T23:14:41.640350+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632904+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:14:41.632904+0000","last_peered":"2024-09-15T23:14:41.632904+0000","last_clean":"2024-09-15T23:14:41.632904+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:14:41.632904+0000","last_undegraded":"2024-09-15T23:14:41.632904+0000","last_fullsized":"2024-09-15T23:14:41.632904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636830+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:14:41.636830+0000","last_peered":"2024-09-15T23:14:41.636830+0000","last_clean":"2024-09-15T23:14:41.636830+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:14:41.636830+0000","last_undegraded":"2024-09-15T23:14:41.636830+0000","last_fullsized":"2024-09-15T23:14:41.636830+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'32","reported_seq":183,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:15:01.560580+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:15:01.560580+0000","last_peered":"2024-09-15T23:15:01.560580+0000","last_clean":"2024-09-15T23:15:01.560580+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:15:01.560580+0000","last_undegraded":"2024-09-15T23:15:01.560580+0000","last_fullsized":"2024-09-15T23:15:01.560580+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62931,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":120,"num_write":33,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008184+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:14:42.008184+0000","last_peered":"2024-09-15T23:14:42.008184+0000","last_clean":"2024-09-15T23:14:42.008184+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:14:42.008184+0000","last_undegraded":"2024-09-15T23:14:42.008184+0000","last_fullsized":"2024-09-15T23:14:42.008184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633683+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:14:41.633683+0000","last_peered":"2024-09-15T23:14:41.633683+0000","last_clean":"2024-09-15T23:14:41.633683+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:14:41.633683+0000","last_undegraded":"2024-09-15T23:14:41.633683+0000","last_fullsized":"2024-09-15T23:14:41.633683+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633092+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:14:41.633092+0000","last_peered":"2024-09-15T23:14:41.633092+0000","last_clean":"2024-09-15T23:14:41.633092+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:14:41.633092+0000","last_undegraded":"2024-09-15T23:14:41.633092+0000","last_fullsized":"2024-09-15T23:14:41.633092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.641820+0000","last_change":"2024-09-15T23:14:41.641820+0000","last_active":"2024-09-15T23:14:41.641820+0000","last_peered":"2024-09-15T23:14:41.641820+0000","last_clean":"2024-09-15T23:14:41.641820+0000","last_became_active":"2024-09-15T23:14:41.641282+0000","last_became_peered":"2024-09-15T23:14:41.641282+0000","last_unstale":"2024-09-15T23:14:41.641820+0000","last_undegraded":"2024-09-15T23:14:41.641820+0000","last_fullsized":"2024-09-15T23:14:41.641820+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:43.472601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635042+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:14:41.635042+0000","last_peered":"2024-09-15T23:14:41.635042+0000","last_clean":"2024-09-15T23:14:41.635042+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:14:41.635042+0000","last_undegraded":"2024-09-15T23:14:41.635042+0000","last_fullsized":"2024-09-15T23:14:41.635042+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640088+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:14:41.640088+0000","last_peered":"2024-09-15T23:14:41.640088+0000","last_clean":"2024-09-15T23:14:41.640088+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:14:41.640088+0000","last_undegraded":"2024-09-15T23:14:41.640088+0000","last_fullsized":"2024-09-15T23:14:41.640088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007194+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:14:42.007194+0000","last_peered":"2024-09-15T23:14:42.007194+0000","last_clean":"2024-09-15T23:14:42.007194+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:14:42.007194+0000","last_undegraded":"2024-09-15T23:14:42.007194+0000","last_fullsized":"2024-09-15T23:14:42.007194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634019+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:14:41.634019+0000","last_peered":"2024-09-15T23:14:41.634019+0000","last_clean":"2024-09-15T23:14:41.634019+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:14:41.634019+0000","last_undegraded":"2024-09-15T23:14:41.634019+0000","last_fullsized":"2024-09-15T23:14:41.634019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:15:00.559293+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:15:00.559293+0000","last_peered":"2024-09-15T23:15:00.559293+0000","last_clean":"2024-09-15T23:15:00.559293+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:15:00.559293+0000","last_undegraded":"2024-09-15T23:15:00.559293+0000","last_fullsized":"2024-09-15T23:15:00.559293+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":54,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007527+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:14:42.007527+0000","last_peered":"2024-09-15T23:14:42.007527+0000","last_clean":"2024-09-15T23:14:42.007527+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:14:42.007527+0000","last_undegraded":"2024-09-15T23:14:42.007527+0000","last_fullsized":"2024-09-15T23:14:42.007527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635001+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:14:41.635001+0000","last_peered":"2024-09-15T23:14:41.635001+0000","last_clean":"2024-09-15T23:14:41.635001+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:14:41.635001+0000","last_undegraded":"2024-09-15T23:14:41.635001+0000","last_fullsized":"2024-09-15T23:14:41.635001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007590+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:14:42.007590+0000","last_peered":"2024-09-15T23:14:42.007590+0000","last_clean":"2024-09-15T23:14:42.007590+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:14:42.007590+0000","last_undegraded":"2024-09-15T23:14:42.007590+0000","last_fullsized":"2024-09-15T23:14:42.007590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632461+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:14:41.632461+0000","last_peered":"2024-09-15T23:14:41.632461+0000","last_clean":"2024-09-15T23:14:41.632461+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:14:41.632461+0000","last_undegraded":"2024-09-15T23:14:41.632461+0000","last_fullsized":"2024-09-15T23:14:41.632461+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635438+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:14:41.635438+0000","last_peered":"2024-09-15T23:14:41.635438+0000","last_clean":"2024-09-15T23:14:41.635438+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:14:41.635438+0000","last_undegraded":"2024-09-15T23:14:41.635438+0000","last_fullsized":"2024-09-15T23:14:41.635438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007884+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:14:42.007884+0000","last_peered":"2024-09-15T23:14:42.007884+0000","last_clean":"2024-09-15T23:14:42.007884+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:14:42.007884+0000","last_undegraded":"2024-09-15T23:14:42.007884+0000","last_fullsized":"2024-09-15T23:14:42.007884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.630922+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:14:41.630922+0000","last_peered":"2024-09-15T23:14:41.630922+0000","last_clean":"2024-09-15T23:14:41.630922+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:14:41.630922+0000","last_undegraded":"2024-09-15T23:14:41.630922+0000","last_fullsized":"2024-09-15T23:14:41.630922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993682+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:14:41.993682+0000","last_peered":"2024-09-15T23:14:41.993682+0000","last_clean":"2024-09-15T23:14:41.993682+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:14:41.993682+0000","last_undegraded":"2024-09-15T23:14:41.993682+0000","last_fullsized":"2024-09-15T23:14:41.993682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632654+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:14:41.632654+0000","last_peered":"2024-09-15T23:14:41.632654+0000","last_clean":"2024-09-15T23:14:41.632654+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:14:41.632654+0000","last_undegraded":"2024-09-15T23:14:41.632654+0000","last_fullsized":"2024-09-15T23:14:41.632654+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639342+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:14:41.639342+0000","last_peered":"2024-09-15T23:14:41.639342+0000","last_clean":"2024-09-15T23:14:41.639342+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:14:41.639342+0000","last_undegraded":"2024-09-15T23:14:41.639342+0000","last_fullsized":"2024-09-15T23:14:41.639342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":96,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:59.558641+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:14:59.558641+0000","last_peered":"2024-09-15T23:14:59.558641+0000","last_clean":"2024-09-15T23:14:59.558641+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:14:59.558641+0000","last_undegraded":"2024-09-15T23:14:59.558641+0000","last_fullsized":"2024-09-15T23:14:59.558641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636592+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:14:41.636592+0000","last_peered":"2024-09-15T23:14:41.636592+0000","last_clean":"2024-09-15T23:14:41.636592+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:14:41.636592+0000","last_undegraded":"2024-09-15T23:14:41.636592+0000","last_fullsized":"2024-09-15T23:14:41.636592+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639974+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:14:41.639974+0000","last_peered":"2024-09-15T23:14:41.639974+0000","last_clean":"2024-09-15T23:14:41.639974+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:14:41.639974+0000","last_undegraded":"2024-09-15T23:14:41.639974+0000","last_fullsized":"2024-09-15T23:14:41.639974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635174+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:14:41.635174+0000","last_peered":"2024-09-15T23:14:41.635174+0000","last_clean":"2024-09-15T23:14:41.635174+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:14:41.635174+0000","last_undegraded":"2024-09-15T23:14:41.635174+0000","last_fullsized":"2024-09-15T23:14:41.635174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638994+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:14:41.638994+0000","last_peered":"2024-09-15T23:14:41.638994+0000","last_clean":"2024-09-15T23:14:41.638994+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:14:41.638994+0000","last_undegraded":"2024-09-15T23:14:41.638994+0000","last_fullsized":"2024-09-15T23:14:41.638994+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639105+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:14:41.639105+0000","last_peered":"2024-09-15T23:14:41.639105+0000","last_clean":"2024-09-15T23:14:41.639105+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:14:41.639105+0000","last_undegraded":"2024-09-15T23:14:41.639105+0000","last_fullsized":"2024-09-15T23:14:41.639105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634855+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:14:41.634855+0000","last_peered":"2024-09-15T23:14:41.634855+0000","last_clean":"2024-09-15T23:14:41.634855+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:14:41.634855+0000","last_undegraded":"2024-09-15T23:14:41.634855+0000","last_fullsized":"2024-09-15T23:14:41.634855+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.010010+0000","last_change":"2024-09-15T23:14:42.010010+0000","last_active":"2024-09-15T23:14:42.010010+0000","last_peered":"2024-09-15T23:14:42.010010+0000","last_clean":"2024-09-15T23:14:42.010010+0000","last_became_active":"2024-09-15T23:14:42.009603+0000","last_became_peered":"2024-09-15T23:14:42.009603+0000","last_unstale":"2024-09-15T23:14:42.010010+0000","last_undegraded":"2024-09-15T23:14:42.010010+0000","last_fullsized":"2024-09-15T23:14:42.010010+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:34:51.849548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635951+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:14:41.635951+0000","last_peered":"2024-09-15T23:14:41.635951+0000","last_clean":"2024-09-15T23:14:41.635951+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:14:41.635951+0000","last_undegraded":"2024-09-15T23:14:41.635951+0000","last_fullsized":"2024-09-15T23:14:41.635951+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639616+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:14:41.639616+0000","last_peered":"2024-09-15T23:14:41.639616+0000","last_clean":"2024-09-15T23:14:41.639616+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:14:41.639616+0000","last_undegraded":"2024-09-15T23:14:41.639616+0000","last_fullsized":"2024-09-15T23:14:41.639616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008057+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:14:42.008057+0000","last_peered":"2024-09-15T23:14:42.008057+0000","last_clean":"2024-09-15T23:14:42.008057+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:14:42.008057+0000","last_undegraded":"2024-09-15T23:14:42.008057+0000","last_fullsized":"2024-09-15T23:14:42.008057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632233+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:14:41.632233+0000","last_peered":"2024-09-15T23:14:41.632233+0000","last_clean":"2024-09-15T23:14:41.632233+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:14:41.632233+0000","last_undegraded":"2024-09-15T23:14:41.632233+0000","last_fullsized":"2024-09-15T23:14:41.632233+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008066+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:14:42.008066+0000","last_peered":"2024-09-15T23:14:42.008066+0000","last_clean":"2024-09-15T23:14:42.008066+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:14:42.008066+0000","last_undegraded":"2024-09-15T23:14:42.008066+0000","last_fullsized":"2024-09-15T23:14:42.008066+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634158+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:14:41.634158+0000","last_peered":"2024-09-15T23:14:41.634158+0000","last_clean":"2024-09-15T23:14:41.634158+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:14:41.634158+0000","last_undegraded":"2024-09-15T23:14:41.634158+0000","last_fullsized":"2024-09-15T23:14:41.634158+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631480+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:14:41.631480+0000","last_peered":"2024-09-15T23:14:41.631480+0000","last_clean":"2024-09-15T23:14:41.631480+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:14:41.631480+0000","last_undegraded":"2024-09-15T23:14:41.631480+0000","last_fullsized":"2024-09-15T23:14:41.631480+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635318+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:14:41.635318+0000","last_peered":"2024-09-15T23:14:41.635318+0000","last_clean":"2024-09-15T23:14:41.635318+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:14:41.635318+0000","last_undegraded":"2024-09-15T23:14:41.635318+0000","last_fullsized":"2024-09-15T23:14:41.635318+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640168+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:14:41.640168+0000","last_peered":"2024-09-15T23:14:41.640168+0000","last_clean":"2024-09-15T23:14:41.640168+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:14:41.640168+0000","last_undegraded":"2024-09-15T23:14:41.640168+0000","last_fullsized":"2024-09-15T23:14:41.640168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008717+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:14:42.008717+0000","last_peered":"2024-09-15T23:14:42.008717+0000","last_clean":"2024-09-15T23:14:42.008717+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:14:42.008717+0000","last_undegraded":"2024-09-15T23:14:42.008717+0000","last_fullsized":"2024-09-15T23:14:42.008717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635838+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:14:41.635838+0000","last_peered":"2024-09-15T23:14:41.635838+0000","last_clean":"2024-09-15T23:14:41.635838+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:14:41.635838+0000","last_undegraded":"2024-09-15T23:14:41.635838+0000","last_fullsized":"2024-09-15T23:14:41.635838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633133+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:14:41.633133+0000","last_peered":"2024-09-15T23:14:41.633133+0000","last_clean":"2024-09-15T23:14:41.633133+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:14:41.633133+0000","last_undegraded":"2024-09-15T23:14:41.633133+0000","last_fullsized":"2024-09-15T23:14:41.633133+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635063+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:14:41.635063+0000","last_peered":"2024-09-15T23:14:41.635063+0000","last_clean":"2024-09-15T23:14:41.635063+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:14:41.635063+0000","last_undegraded":"2024-09-15T23:14:41.635063+0000","last_fullsized":"2024-09-15T23:14:41.635063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634405+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:14:41.634405+0000","last_peered":"2024-09-15T23:14:41.634405+0000","last_clean":"2024-09-15T23:14:41.634405+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:14:41.634405+0000","last_undegraded":"2024-09-15T23:14:41.634405+0000","last_fullsized":"2024-09-15T23:14:41.634405+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631378+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:14:41.631378+0000","last_peered":"2024-09-15T23:14:41.631378+0000","last_clean":"2024-09-15T23:14:41.631378+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:14:41.631378+0000","last_undegraded":"2024-09-15T23:14:41.631378+0000","last_fullsized":"2024-09-15T23:14:41.631378+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636882+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:14:41.636882+0000","last_peered":"2024-09-15T23:14:41.636882+0000","last_clean":"2024-09-15T23:14:41.636882+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:14:41.636882+0000","last_undegraded":"2024-09-15T23:14:41.636882+0000","last_fullsized":"2024-09-15T23:14:41.636882+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640638+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:14:41.640638+0000","last_peered":"2024-09-15T23:14:41.640638+0000","last_clean":"2024-09-15T23:14:41.640638+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:14:41.640638+0000","last_undegraded":"2024-09-15T23:14:41.640638+0000","last_fullsized":"2024-09-15T23:14:41.640638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993959+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:14:41.993959+0000","last_peered":"2024-09-15T23:14:41.993959+0000","last_clean":"2024-09-15T23:14:41.993959+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:14:41.993959+0000","last_undegraded":"2024-09-15T23:14:41.993959+0000","last_fullsized":"2024-09-15T23:14:41.993959+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634160+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:14:41.634160+0000","last_peered":"2024-09-15T23:14:41.634160+0000","last_clean":"2024-09-15T23:14:41.634160+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:14:41.634160+0000","last_undegraded":"2024-09-15T23:14:41.634160+0000","last_fullsized":"2024-09-15T23:14:41.634160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007787+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:14:42.007787+0000","last_peered":"2024-09-15T23:14:42.007787+0000","last_clean":"2024-09-15T23:14:42.007787+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:14:42.007787+0000","last_undegraded":"2024-09-15T23:14:42.007787+0000","last_fullsized":"2024-09-15T23:14:42.007787+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639002+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:14:41.639002+0000","last_peered":"2024-09-15T23:14:41.639002+0000","last_clean":"2024-09-15T23:14:41.639002+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:14:41.639002+0000","last_undegraded":"2024-09-15T23:14:41.639002+0000","last_fullsized":"2024-09-15T23:14:41.639002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993633+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:14:41.993633+0000","last_peered":"2024-09-15T23:14:41.993633+0000","last_clean":"2024-09-15T23:14:41.993633+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:14:41.993633+0000","last_undegraded":"2024-09-15T23:14:41.993633+0000","last_fullsized":"2024-09-15T23:14:41.993633+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.637111+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:14:41.637111+0000","last_peered":"2024-09-15T23:14:41.637111+0000","last_clean":"2024-09-15T23:14:41.637111+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:14:41.637111+0000","last_undegraded":"2024-09-15T23:14:41.637111+0000","last_fullsized":"2024-09-15T23:14:41.637111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636813+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:14:41.636813+0000","last_peered":"2024-09-15T23:14:41.636813+0000","last_clean":"2024-09-15T23:14:41.636813+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:14:41.636813+0000","last_undegraded":"2024-09-15T23:14:41.636813+0000","last_fullsized":"2024-09-15T23:14:41.636813+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634425+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:14:41.634425+0000","last_peered":"2024-09-15T23:14:41.634425+0000","last_clean":"2024-09-15T23:14:41.634425+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:14:41.634425+0000","last_undegraded":"2024-09-15T23:14:41.634425+0000","last_fullsized":"2024-09-15T23:14:41.634425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635105+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:14:41.635105+0000","last_peered":"2024-09-15T23:14:41.635105+0000","last_clean":"2024-09-15T23:14:41.635105+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:14:41.635105+0000","last_undegraded":"2024-09-15T23:14:41.635105+0000","last_fullsized":"2024-09-15T23:14:41.635105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635722+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:14:41.635722+0000","last_peered":"2024-09-15T23:14:41.635722+0000","last_clean":"2024-09-15T23:14:41.635722+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:14:41.635722+0000","last_undegraded":"2024-09-15T23:14:41.635722+0000","last_fullsized":"2024-09-15T23:14:41.635722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66934,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":346,"num_read_kb":244,"num_write":168,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514527,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":995328,"data_stored":503506,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743460,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1134592,"data_stored":1153593,"data_compressed":59306,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906985,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":507726,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135919,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":503438,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364851,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1052672,"data_stored":1099461,"data_compressed":59306,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528377,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":983040,"data_stored":498806,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790014,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1060864,"data_stored":1095049,"data_compressed":59306,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953539,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":995328,"data_stored":503456,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":499000,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":563390,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1052672,"data_stored":561647,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":500194,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:15:03.152 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:15:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:02 smithi067 ceph-mon[35665]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:02 smithi067 ceph-mon[35665]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:02 smithi148 ceph-mon[35610]: pgmap v331: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:02 smithi148 ceph-mon[35610]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:03.412 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.893+0000 7f6536d0e640 1 -- 172.21.15.62:0/3790838536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6530102d40 msgr2=0x7f6530103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.893+0000 7f6536d0e640 1 --2- 172.21.15.62:0/3790838536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6530102d40 0x7f6530103120 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f6524009900 tx=0x7f652402a480 comp rx=0 tx=0).stop 2024-09-15T23:15:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.893+0000 7f6536d0e640 1 -- 172.21.15.62:0/3790838536 shutdown_connections 2024-09-15T23:15:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.893+0000 7f6536d0e640 1 --2- 172.21.15.62:0/3790838536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f653010c170 0x7f653010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.893+0000 7f6536d0e640 1 --2- 172.21.15.62:0/3790838536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6530103660 0x7f653010bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.893+0000 7f6536d0e640 1 --2- 172.21.15.62:0/3790838536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6530102d40 0x7f6530103120 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.893+0000 7f6536d0e640 1 -- 172.21.15.62:0/3790838536 >> 172.21.15.62:0/3790838536 conn(0x7f65300fc7a0 msgr2=0x7f65300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.893+0000 7f6536d0e640 1 -- 172.21.15.62:0/3790838536 shutdown_connections 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.893+0000 7f6536d0e640 1 -- 172.21.15.62:0/3790838536 wait complete. 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f6536d0e640 1 Processor -- start 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f6536d0e640 1 -- start start 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f6536d0e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6530102d40 0x7f65301983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f6536d0e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6530103660 0x7f6530198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f6536d0e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f653010c170 0x7f653019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f6536d0e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6530111f70 con 0x7f653010c170 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f6536d0e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6530111df0 con 0x7f6530102d40 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f6536d0e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f65301120f0 con 0x7f6530103660 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f65352ab640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f653010c170 0x7f653019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f65352ab640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f653010c170 0x7f653019ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49662/0 (socket says 172.21.15.62:49662) 2024-09-15T23:15:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f65352ab640 1 -- 172.21.15.62:0/106404026 learned_addr learned my addr 172.21.15.62:0/106404026 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f652ffff640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6530103660 0x7f6530198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.894+0000 7f6534aaa640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6530102d40 0x7f65301983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f65352ab640 1 -- 172.21.15.62:0/106404026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6530103660 msgr2=0x7f6530198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f65352ab640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6530103660 0x7f6530198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f65352ab640 1 -- 172.21.15.62:0/106404026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6530102d40 msgr2=0x7f65301983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f65352ab640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6530102d40 0x7f65301983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f65352ab640 1 -- 172.21.15.62:0/106404026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653019d3a0 con 0x7f653010c170 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f65352ab640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f653010c170 0x7f653019ccc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f652800eaa0 tx=0x7f652800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f652dffb640 1 -- 172.21.15.62:0/106404026 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652800cdb0 con 0x7f653010c170 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f652dffb640 1 -- 172.21.15.62:0/106404026 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6528005020 con 0x7f653010c170 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653019d690 con 0x7f653010c170 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f652dffb640 1 -- 172.21.15.62:0/106404026 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65280116b0 con 0x7f653010c170 2024-09-15T23:15:03.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.895+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65301a4f70 con 0x7f653010c170 2024-09-15T23:15:03.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.896+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6530103120 con 0x7f653010c170 2024-09-15T23:15:03.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.897+0000 7f652dffb640 1 -- 172.21.15.62:0/106404026 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f6528005560 con 0x7f653010c170 2024-09-15T23:15:03.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.898+0000 7f652dffb640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65200a33d0 0x7f65200a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:03.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.898+0000 7f652dffb640 1 -- 172.21.15.62:0/106404026 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f65280cff80 con 0x7f653010c170 2024-09-15T23:15:03.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.898+0000 7f6534aaa640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65200a33d0 0x7f65200a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:03.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.899+0000 7f6534aaa640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65200a33d0 0x7f65200a5890 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6524005ab0 tx=0x7f6524005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:03.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:03.902+0000 7f652dffb640 1 -- 172.21.15.62:0/106404026 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f6528016030 con 0x7f653010c170 2024-09-15T23:15:04.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.074+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6530062630 con 0x7f65200a33d0 2024-09-15T23:15:04.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.082+0000 7f652dffb640 1 -- 172.21.15.62:0/106404026 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340611 (secure 0 0 0) 0x7f6530062630 con 0x7f65200a33d0 2024-09-15T23:15:04.086 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:04.088 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:15:04.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65200a33d0 msgr2=0x7f65200a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65200a33d0 0x7f65200a5890 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6524005ab0 tx=0x7f6524005a20 comp rx=0 tx=0).stop 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f653010c170 msgr2=0x7f653019ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f653010c170 0x7f653019ccc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f652800eaa0 tx=0x7f652800ef60 comp rx=0 tx=0).stop 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 shutdown_connections 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65200a33d0 0x7f65200a5890 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f653010c170 0x7f653019ccc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6530103660 0x7f6530198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 --2- 172.21.15.62:0/106404026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6530102d40 0x7f65301983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 >> 172.21.15.62:0/106404026 conn(0x7f65300fc7a0 msgr2=0x7f65300fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 shutdown_connections 2024-09-15T23:15:04.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:04.086+0000 7f6536d0e640 1 -- 172.21.15.62:0/106404026 wait complete. 2024-09-15T23:15:04.804 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-15T23:15:03.713636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657536,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":472,"num_read_kb":353,"num_write":351,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":282,"ondisk_log_size":282,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365548,"kb_used_data":12068,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576276,"statfs":{"total":1151940427776,"available":1151566106624,"internally_reserved":0,"allocated":12357632,"data_stored":7989266,"data_compressed":389796,"data_compressed_allocated":528384,"data_compressed_original":2555904,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003351"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008791+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:14:42.008791+0000","last_peered":"2024-09-15T23:14:42.008791+0000","last_clean":"2024-09-15T23:14:42.008791+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:14:42.008791+0000","last_undegraded":"2024-09-15T23:14:42.008791+0000","last_fullsized":"2024-09-15T23:14:42.008791+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007527+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:14:42.007527+0000","last_peered":"2024-09-15T23:14:42.007527+0000","last_clean":"2024-09-15T23:14:42.007527+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:14:42.007527+0000","last_undegraded":"2024-09-15T23:14:42.007527+0000","last_fullsized":"2024-09-15T23:14:42.007527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638628+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:14:41.638628+0000","last_peered":"2024-09-15T23:14:41.638628+0000","last_clean":"2024-09-15T23:14:41.638628+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:14:41.638628+0000","last_undegraded":"2024-09-15T23:14:41.638628+0000","last_fullsized":"2024-09-15T23:14:41.638628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007826+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:14:42.007826+0000","last_peered":"2024-09-15T23:14:42.007826+0000","last_clean":"2024-09-15T23:14:42.007826+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:14:42.007826+0000","last_undegraded":"2024-09-15T23:14:42.007826+0000","last_fullsized":"2024-09-15T23:14:42.007826+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634945+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:14:41.634945+0000","last_peered":"2024-09-15T23:14:41.634945+0000","last_clean":"2024-09-15T23:14:41.634945+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:14:41.634945+0000","last_undegraded":"2024-09-15T23:14:41.634945+0000","last_fullsized":"2024-09-15T23:14:41.634945+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635661+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:14:41.635661+0000","last_peered":"2024-09-15T23:14:41.635661+0000","last_clean":"2024-09-15T23:14:41.635661+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:14:41.635661+0000","last_undegraded":"2024-09-15T23:14:41.635661+0000","last_fullsized":"2024-09-15T23:14:41.635661+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":19,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.660261+0000","last_change":"2024-09-15T23:14:41.659816+0000","last_active":"2024-09-15T23:14:41.660261+0000","last_peered":"2024-09-15T23:14:41.660261+0000","last_clean":"2024-09-15T23:14:41.660261+0000","last_became_active":"2024-09-15T23:14:41.641611+0000","last_became_peered":"2024-09-15T23:14:41.641611+0000","last_unstale":"2024-09-15T23:14:41.660261+0000","last_undegraded":"2024-09-15T23:14:41.660261+0000","last_fullsized":"2024-09-15T23:14:41.660261+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:23.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633717+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:14:41.633717+0000","last_peered":"2024-09-15T23:14:41.633717+0000","last_clean":"2024-09-15T23:14:41.633717+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:14:41.633717+0000","last_undegraded":"2024-09-15T23:14:41.633717+0000","last_fullsized":"2024-09-15T23:14:41.633717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636932+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:14:41.636932+0000","last_peered":"2024-09-15T23:14:41.636932+0000","last_clean":"2024-09-15T23:14:41.636932+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:14:41.636932+0000","last_undegraded":"2024-09-15T23:14:41.636932+0000","last_fullsized":"2024-09-15T23:14:41.636932+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636429+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:14:41.636429+0000","last_peered":"2024-09-15T23:14:41.636429+0000","last_clean":"2024-09-15T23:14:41.636429+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:14:41.636429+0000","last_undegraded":"2024-09-15T23:14:41.636429+0000","last_fullsized":"2024-09-15T23:14:41.636429+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632933+0000","last_change":"2024-09-15T23:13:32.511232+0000","last_active":"2024-09-15T23:14:41.632933+0000","last_peered":"2024-09-15T23:14:41.632933+0000","last_clean":"2024-09-15T23:14:41.632933+0000","last_became_active":"2024-09-15T23:13:32.510875+0000","last_became_peered":"2024-09-15T23:13:32.510875+0000","last_unstale":"2024-09-15T23:14:41.632933+0000","last_undegraded":"2024-09-15T23:14:41.632933+0000","last_fullsized":"2024-09-15T23:14:41.632933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:54:29.918573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635945+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:14:41.635945+0000","last_peered":"2024-09-15T23:14:41.635945+0000","last_clean":"2024-09-15T23:14:41.635945+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:14:41.635945+0000","last_undegraded":"2024-09-15T23:14:41.635945+0000","last_fullsized":"2024-09-15T23:14:41.635945+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638773+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:14:41.638773+0000","last_peered":"2024-09-15T23:14:41.638773+0000","last_clean":"2024-09-15T23:14:41.638773+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:14:41.638773+0000","last_undegraded":"2024-09-15T23:14:41.638773+0000","last_fullsized":"2024-09-15T23:14:41.638773+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.646320+0000","last_change":"2024-09-15T23:14:41.646320+0000","last_active":"2024-09-15T23:14:41.646320+0000","last_peered":"2024-09-15T23:14:41.646320+0000","last_clean":"2024-09-15T23:14:41.646320+0000","last_became_active":"2024-09-15T23:14:41.641933+0000","last_became_peered":"2024-09-15T23:14:41.641933+0000","last_unstale":"2024-09-15T23:14:41.646320+0000","last_undegraded":"2024-09-15T23:14:41.646320+0000","last_fullsized":"2024-09-15T23:14:41.646320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:49.417407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635920+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:14:41.635920+0000","last_peered":"2024-09-15T23:14:41.635920+0000","last_clean":"2024-09-15T23:14:41.635920+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:14:41.635920+0000","last_undegraded":"2024-09-15T23:14:41.635920+0000","last_fullsized":"2024-09-15T23:14:41.635920+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007641+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:14:42.007641+0000","last_peered":"2024-09-15T23:14:42.007641+0000","last_clean":"2024-09-15T23:14:42.007641+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:14:42.007641+0000","last_undegraded":"2024-09-15T23:14:42.007641+0000","last_fullsized":"2024-09-15T23:14:42.007641+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639214+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:14:41.639214+0000","last_peered":"2024-09-15T23:14:41.639214+0000","last_clean":"2024-09-15T23:14:41.639214+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:14:41.639214+0000","last_undegraded":"2024-09-15T23:14:41.639214+0000","last_fullsized":"2024-09-15T23:14:41.639214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636523+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:14:41.636523+0000","last_peered":"2024-09-15T23:14:41.636523+0000","last_clean":"2024-09-15T23:14:41.636523+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:14:41.636523+0000","last_undegraded":"2024-09-15T23:14:41.636523+0000","last_fullsized":"2024-09-15T23:14:41.636523+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635294+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:14:41.635294+0000","last_peered":"2024-09-15T23:14:41.635294+0000","last_clean":"2024-09-15T23:14:41.635294+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:14:41.635294+0000","last_undegraded":"2024-09-15T23:14:41.635294+0000","last_fullsized":"2024-09-15T23:14:41.635294+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008225+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:14:42.008225+0000","last_peered":"2024-09-15T23:14:42.008225+0000","last_clean":"2024-09-15T23:14:42.008225+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:14:42.008225+0000","last_undegraded":"2024-09-15T23:14:42.008225+0000","last_fullsized":"2024-09-15T23:14:42.008225+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008480+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:14:42.008480+0000","last_peered":"2024-09-15T23:14:42.008480+0000","last_clean":"2024-09-15T23:14:42.008480+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:14:42.008480+0000","last_undegraded":"2024-09-15T23:14:42.008480+0000","last_fullsized":"2024-09-15T23:14:42.008480+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635579+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:14:41.635579+0000","last_peered":"2024-09-15T23:14:41.635579+0000","last_clean":"2024-09-15T23:14:41.635579+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:14:41.635579+0000","last_undegraded":"2024-09-15T23:14:41.635579+0000","last_fullsized":"2024-09-15T23:14:41.635579+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633263+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:14:41.633263+0000","last_peered":"2024-09-15T23:14:41.633263+0000","last_clean":"2024-09-15T23:14:41.633263+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:14:41.633263+0000","last_undegraded":"2024-09-15T23:14:41.633263+0000","last_fullsized":"2024-09-15T23:14:41.633263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636123+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:14:41.636123+0000","last_peered":"2024-09-15T23:14:41.636123+0000","last_clean":"2024-09-15T23:14:41.636123+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:14:41.636123+0000","last_undegraded":"2024-09-15T23:14:41.636123+0000","last_fullsized":"2024-09-15T23:14:41.636123+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.009605+0000","last_change":"2024-09-15T23:14:42.009605+0000","last_active":"2024-09-15T23:14:42.009605+0000","last_peered":"2024-09-15T23:14:42.009605+0000","last_clean":"2024-09-15T23:14:42.009605+0000","last_became_active":"2024-09-15T23:14:42.009237+0000","last_became_peered":"2024-09-15T23:14:42.009237+0000","last_unstale":"2024-09-15T23:14:42.009605+0000","last_undegraded":"2024-09-15T23:14:42.009605+0000","last_fullsized":"2024-09-15T23:14:42.009605+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:11:08.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.637301+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:14:41.637301+0000","last_peered":"2024-09-15T23:14:41.637301+0000","last_clean":"2024-09-15T23:14:41.637301+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:14:41.637301+0000","last_undegraded":"2024-09-15T23:14:41.637301+0000","last_fullsized":"2024-09-15T23:14:41.637301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639750+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:14:41.639750+0000","last_peered":"2024-09-15T23:14:41.639750+0000","last_clean":"2024-09-15T23:14:41.639750+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:14:41.639750+0000","last_undegraded":"2024-09-15T23:14:41.639750+0000","last_fullsized":"2024-09-15T23:14:41.639750+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008151+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:14:42.008151+0000","last_peered":"2024-09-15T23:14:42.008151+0000","last_clean":"2024-09-15T23:14:42.008151+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:14:42.008151+0000","last_undegraded":"2024-09-15T23:14:42.008151+0000","last_fullsized":"2024-09-15T23:14:42.008151+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'12","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008816+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:14:42.008816+0000","last_peered":"2024-09-15T23:14:42.008816+0000","last_clean":"2024-09-15T23:14:42.008816+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:14:42.008816+0000","last_undegraded":"2024-09-15T23:14:42.008816+0000","last_fullsized":"2024-09-15T23:14:42.008816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007629+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:14:42.007629+0000","last_peered":"2024-09-15T23:14:42.007629+0000","last_clean":"2024-09-15T23:14:42.007629+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:14:42.007629+0000","last_undegraded":"2024-09-15T23:14:42.007629+0000","last_fullsized":"2024-09-15T23:14:42.007629+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631552+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:14:41.631552+0000","last_peered":"2024-09-15T23:14:41.631552+0000","last_clean":"2024-09-15T23:14:41.631552+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:14:41.631552+0000","last_undegraded":"2024-09-15T23:14:41.631552+0000","last_fullsized":"2024-09-15T23:14:41.631552+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636929+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:14:41.636929+0000","last_peered":"2024-09-15T23:14:41.636929+0000","last_clean":"2024-09-15T23:14:41.636929+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:14:41.636929+0000","last_undegraded":"2024-09-15T23:14:41.636929+0000","last_fullsized":"2024-09-15T23:14:41.636929+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635760+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:14:41.635760+0000","last_peered":"2024-09-15T23:14:41.635760+0000","last_clean":"2024-09-15T23:14:41.635760+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:14:41.635760+0000","last_undegraded":"2024-09-15T23:14:41.635760+0000","last_fullsized":"2024-09-15T23:14:41.635760+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993522+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:14:41.993522+0000","last_peered":"2024-09-15T23:14:41.993522+0000","last_clean":"2024-09-15T23:14:41.993522+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:14:41.993522+0000","last_undegraded":"2024-09-15T23:14:41.993522+0000","last_fullsized":"2024-09-15T23:14:41.993522+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.009055+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:14:42.009055+0000","last_peered":"2024-09-15T23:14:42.009055+0000","last_clean":"2024-09-15T23:14:42.009055+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:14:42.009055+0000","last_undegraded":"2024-09-15T23:14:42.009055+0000","last_fullsized":"2024-09-15T23:14:42.009055+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.995466+0000","last_change":"2024-09-15T23:14:41.995466+0000","last_active":"2024-09-15T23:14:41.995466+0000","last_peered":"2024-09-15T23:14:41.995466+0000","last_clean":"2024-09-15T23:14:41.995466+0000","last_became_active":"2024-09-15T23:14:41.994915+0000","last_became_peered":"2024-09-15T23:14:41.994915+0000","last_unstale":"2024-09-15T23:14:41.995466+0000","last_undegraded":"2024-09-15T23:14:41.995466+0000","last_fullsized":"2024-09-15T23:14:41.995466+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:04.121596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993461+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:14:41.993461+0000","last_peered":"2024-09-15T23:14:41.993461+0000","last_clean":"2024-09-15T23:14:41.993461+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:14:41.993461+0000","last_undegraded":"2024-09-15T23:14:41.993461+0000","last_fullsized":"2024-09-15T23:14:41.993461+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639308+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:14:41.639308+0000","last_peered":"2024-09-15T23:14:41.639308+0000","last_clean":"2024-09-15T23:14:41.639308+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:14:41.639308+0000","last_undegraded":"2024-09-15T23:14:41.639308+0000","last_fullsized":"2024-09-15T23:14:41.639308+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631244+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:14:41.631244+0000","last_peered":"2024-09-15T23:14:41.631244+0000","last_clean":"2024-09-15T23:14:41.631244+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:14:41.631244+0000","last_undegraded":"2024-09-15T23:14:41.631244+0000","last_fullsized":"2024-09-15T23:14:41.631244+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632290+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:14:41.632290+0000","last_peered":"2024-09-15T23:14:41.632290+0000","last_clean":"2024-09-15T23:14:41.632290+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:14:41.632290+0000","last_undegraded":"2024-09-15T23:14:41.632290+0000","last_fullsized":"2024-09-15T23:14:41.632290+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:58.557277+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:14:58.557277+0000","last_peered":"2024-09-15T23:14:58.557277+0000","last_clean":"2024-09-15T23:14:58.557277+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:14:58.557277+0000","last_undegraded":"2024-09-15T23:14:58.557277+0000","last_fullsized":"2024-09-15T23:14:58.557277+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631849+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:14:41.631849+0000","last_peered":"2024-09-15T23:14:41.631849+0000","last_clean":"2024-09-15T23:14:41.631849+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:14:41.631849+0000","last_undegraded":"2024-09-15T23:14:41.631849+0000","last_fullsized":"2024-09-15T23:14:41.631849+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.630899+0000","last_change":"2024-09-15T23:13:32.513091+0000","last_active":"2024-09-15T23:14:41.630899+0000","last_peered":"2024-09-15T23:14:41.630899+0000","last_clean":"2024-09-15T23:14:41.630899+0000","last_became_active":"2024-09-15T23:13:32.512752+0000","last_became_peered":"2024-09-15T23:13:32.512752+0000","last_unstale":"2024-09-15T23:14:41.630899+0000","last_undegraded":"2024-09-15T23:14:41.630899+0000","last_fullsized":"2024-09-15T23:14:41.630899+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:48:58.131659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638988+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:14:41.638988+0000","last_peered":"2024-09-15T23:14:41.638988+0000","last_clean":"2024-09-15T23:14:41.638988+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:14:41.638988+0000","last_undegraded":"2024-09-15T23:14:41.638988+0000","last_fullsized":"2024-09-15T23:14:41.638988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632857+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:14:41.632857+0000","last_peered":"2024-09-15T23:14:41.632857+0000","last_clean":"2024-09-15T23:14:41.632857+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:14:41.632857+0000","last_undegraded":"2024-09-15T23:14:41.632857+0000","last_fullsized":"2024-09-15T23:14:41.632857+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008444+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:14:42.008444+0000","last_peered":"2024-09-15T23:14:42.008444+0000","last_clean":"2024-09-15T23:14:42.008444+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:14:42.008444+0000","last_undegraded":"2024-09-15T23:14:42.008444+0000","last_fullsized":"2024-09-15T23:14:42.008444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636821+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:14:41.636821+0000","last_peered":"2024-09-15T23:14:41.636821+0000","last_clean":"2024-09-15T23:14:41.636821+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:14:41.636821+0000","last_undegraded":"2024-09-15T23:14:41.636821+0000","last_fullsized":"2024-09-15T23:14:41.636821+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640419+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:14:41.640419+0000","last_peered":"2024-09-15T23:14:41.640419+0000","last_clean":"2024-09-15T23:14:41.640419+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:14:41.640419+0000","last_undegraded":"2024-09-15T23:14:41.640419+0000","last_fullsized":"2024-09-15T23:14:41.640419+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640417+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:14:41.640417+0000","last_peered":"2024-09-15T23:14:41.640417+0000","last_clean":"2024-09-15T23:14:41.640417+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:14:41.640417+0000","last_undegraded":"2024-09-15T23:14:41.640417+0000","last_fullsized":"2024-09-15T23:14:41.640417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640638+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:14:41.640638+0000","last_peered":"2024-09-15T23:14:41.640638+0000","last_clean":"2024-09-15T23:14:41.640638+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:14:41.640638+0000","last_undegraded":"2024-09-15T23:14:41.640638+0000","last_fullsized":"2024-09-15T23:14:41.640638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639748+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:14:41.639748+0000","last_peered":"2024-09-15T23:14:41.639748+0000","last_clean":"2024-09-15T23:14:41.639748+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:14:41.639748+0000","last_undegraded":"2024-09-15T23:14:41.639748+0000","last_fullsized":"2024-09-15T23:14:41.639748+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.994950+0000","last_change":"2024-09-15T23:14:41.994950+0000","last_active":"2024-09-15T23:14:41.994950+0000","last_peered":"2024-09-15T23:14:41.994950+0000","last_clean":"2024-09-15T23:14:41.994950+0000","last_became_active":"2024-09-15T23:14:41.994429+0000","last_became_peered":"2024-09-15T23:14:41.994429+0000","last_unstale":"2024-09-15T23:14:41.994950+0000","last_undegraded":"2024-09-15T23:14:41.994950+0000","last_fullsized":"2024-09-15T23:14:41.994950+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:24.056756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639610+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:14:41.639610+0000","last_peered":"2024-09-15T23:14:41.639610+0000","last_clean":"2024-09-15T23:14:41.639610+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:14:41.639610+0000","last_undegraded":"2024-09-15T23:14:41.639610+0000","last_fullsized":"2024-09-15T23:14:41.639610+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633147+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:14:41.633147+0000","last_peered":"2024-09-15T23:14:41.633147+0000","last_clean":"2024-09-15T23:14:41.633147+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:14:41.633147+0000","last_undegraded":"2024-09-15T23:14:41.633147+0000","last_fullsized":"2024-09-15T23:14:41.633147+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640471+0000","last_change":"2024-09-15T23:14:41.640471+0000","last_active":"2024-09-15T23:14:41.640471+0000","last_peered":"2024-09-15T23:14:41.640471+0000","last_clean":"2024-09-15T23:14:41.640471+0000","last_became_active":"2024-09-15T23:14:41.640017+0000","last_became_peered":"2024-09-15T23:14:41.640017+0000","last_unstale":"2024-09-15T23:14:41.640471+0000","last_undegraded":"2024-09-15T23:14:41.640471+0000","last_fullsized":"2024-09-15T23:14:41.640471+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:58.842636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639778+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:14:41.639778+0000","last_peered":"2024-09-15T23:14:41.639778+0000","last_clean":"2024-09-15T23:14:41.639778+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:14:41.639778+0000","last_undegraded":"2024-09-15T23:14:41.639778+0000","last_fullsized":"2024-09-15T23:14:41.639778+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631856+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:14:41.631856+0000","last_peered":"2024-09-15T23:14:41.631856+0000","last_clean":"2024-09-15T23:14:41.631856+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:14:41.631856+0000","last_undegraded":"2024-09-15T23:14:41.631856+0000","last_fullsized":"2024-09-15T23:14:41.631856+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632120+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:14:41.632120+0000","last_peered":"2024-09-15T23:14:41.632120+0000","last_clean":"2024-09-15T23:14:41.632120+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:14:41.632120+0000","last_undegraded":"2024-09-15T23:14:41.632120+0000","last_fullsized":"2024-09-15T23:14:41.632120+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635550+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:14:41.635550+0000","last_peered":"2024-09-15T23:14:41.635550+0000","last_clean":"2024-09-15T23:14:41.635550+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:14:41.635550+0000","last_undegraded":"2024-09-15T23:14:41.635550+0000","last_fullsized":"2024-09-15T23:14:41.635550+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636227+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:14:41.636227+0000","last_peered":"2024-09-15T23:14:41.636227+0000","last_clean":"2024-09-15T23:14:41.636227+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:14:41.636227+0000","last_undegraded":"2024-09-15T23:14:41.636227+0000","last_fullsized":"2024-09-15T23:14:41.636227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632125+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:14:41.632125+0000","last_peered":"2024-09-15T23:14:41.632125+0000","last_clean":"2024-09-15T23:14:41.632125+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:14:41.632125+0000","last_undegraded":"2024-09-15T23:14:41.632125+0000","last_fullsized":"2024-09-15T23:14:41.632125+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632302+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:14:41.632302+0000","last_peered":"2024-09-15T23:14:41.632302+0000","last_clean":"2024-09-15T23:14:41.632302+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:14:41.632302+0000","last_undegraded":"2024-09-15T23:14:41.632302+0000","last_fullsized":"2024-09-15T23:14:41.632302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.654253+0000","last_change":"2024-09-15T23:14:41.653786+0000","last_active":"2024-09-15T23:14:41.654253+0000","last_peered":"2024-09-15T23:14:41.654253+0000","last_clean":"2024-09-15T23:14:41.654253+0000","last_became_active":"2024-09-15T23:14:41.641472+0000","last_became_peered":"2024-09-15T23:14:41.641472+0000","last_unstale":"2024-09-15T23:14:41.654253+0000","last_undegraded":"2024-09-15T23:14:41.654253+0000","last_fullsized":"2024-09-15T23:14:41.654253+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:16.010142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632061+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:14:41.632061+0000","last_peered":"2024-09-15T23:14:41.632061+0000","last_clean":"2024-09-15T23:14:41.632061+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:14:41.632061+0000","last_undegraded":"2024-09-15T23:14:41.632061+0000","last_fullsized":"2024-09-15T23:14:41.632061+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638792+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:14:41.638792+0000","last_peered":"2024-09-15T23:14:41.638792+0000","last_clean":"2024-09-15T23:14:41.638792+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:14:41.638792+0000","last_undegraded":"2024-09-15T23:14:41.638792+0000","last_fullsized":"2024-09-15T23:14:41.638792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:58.557738+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:14:58.557738+0000","last_peered":"2024-09-15T23:14:58.557738+0000","last_clean":"2024-09-15T23:14:58.557738+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:14:58.557738+0000","last_undegraded":"2024-09-15T23:14:58.557738+0000","last_fullsized":"2024-09-15T23:14:58.557738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008576+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:14:42.008576+0000","last_peered":"2024-09-15T23:14:42.008576+0000","last_clean":"2024-09-15T23:14:42.008576+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:14:42.008576+0000","last_undegraded":"2024-09-15T23:14:42.008576+0000","last_fullsized":"2024-09-15T23:14:42.008576+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639361+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:14:41.639361+0000","last_peered":"2024-09-15T23:14:41.639361+0000","last_clean":"2024-09-15T23:14:41.639361+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:14:41.639361+0000","last_undegraded":"2024-09-15T23:14:41.639361+0000","last_fullsized":"2024-09-15T23:14:41.639361+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632532+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:14:41.632532+0000","last_peered":"2024-09-15T23:14:41.632532+0000","last_clean":"2024-09-15T23:14:41.632532+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:14:41.632532+0000","last_undegraded":"2024-09-15T23:14:41.632532+0000","last_fullsized":"2024-09-15T23:14:41.632532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635193+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:14:41.635193+0000","last_peered":"2024-09-15T23:14:41.635193+0000","last_clean":"2024-09-15T23:14:41.635193+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:14:41.635193+0000","last_undegraded":"2024-09-15T23:14:41.635193+0000","last_fullsized":"2024-09-15T23:14:41.635193+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631322+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:14:41.631322+0000","last_peered":"2024-09-15T23:14:41.631322+0000","last_clean":"2024-09-15T23:14:41.631322+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:14:41.631322+0000","last_undegraded":"2024-09-15T23:14:41.631322+0000","last_fullsized":"2024-09-15T23:14:41.631322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640350+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:14:41.640350+0000","last_peered":"2024-09-15T23:14:41.640350+0000","last_clean":"2024-09-15T23:14:41.640350+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:14:41.640350+0000","last_undegraded":"2024-09-15T23:14:41.640350+0000","last_fullsized":"2024-09-15T23:14:41.640350+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632904+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:14:41.632904+0000","last_peered":"2024-09-15T23:14:41.632904+0000","last_clean":"2024-09-15T23:14:41.632904+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:14:41.632904+0000","last_undegraded":"2024-09-15T23:14:41.632904+0000","last_fullsized":"2024-09-15T23:14:41.632904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636830+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:14:41.636830+0000","last_peered":"2024-09-15T23:14:41.636830+0000","last_clean":"2024-09-15T23:14:41.636830+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:14:41.636830+0000","last_undegraded":"2024-09-15T23:14:41.636830+0000","last_fullsized":"2024-09-15T23:14:41.636830+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"88'32","reported_seq":183,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:15:01.560580+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:15:01.560580+0000","last_peered":"2024-09-15T23:15:01.560580+0000","last_clean":"2024-09-15T23:15:01.560580+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:15:01.560580+0000","last_undegraded":"2024-09-15T23:15:01.560580+0000","last_fullsized":"2024-09-15T23:15:01.560580+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62931,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":117,"num_read_kb":120,"num_write":33,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008184+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:14:42.008184+0000","last_peered":"2024-09-15T23:14:42.008184+0000","last_clean":"2024-09-15T23:14:42.008184+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:14:42.008184+0000","last_undegraded":"2024-09-15T23:14:42.008184+0000","last_fullsized":"2024-09-15T23:14:42.008184+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633683+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:14:41.633683+0000","last_peered":"2024-09-15T23:14:41.633683+0000","last_clean":"2024-09-15T23:14:41.633683+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:14:41.633683+0000","last_undegraded":"2024-09-15T23:14:41.633683+0000","last_fullsized":"2024-09-15T23:14:41.633683+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633092+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:14:41.633092+0000","last_peered":"2024-09-15T23:14:41.633092+0000","last_clean":"2024-09-15T23:14:41.633092+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:14:41.633092+0000","last_undegraded":"2024-09-15T23:14:41.633092+0000","last_fullsized":"2024-09-15T23:14:41.633092+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.641820+0000","last_change":"2024-09-15T23:14:41.641820+0000","last_active":"2024-09-15T23:14:41.641820+0000","last_peered":"2024-09-15T23:14:41.641820+0000","last_clean":"2024-09-15T23:14:41.641820+0000","last_became_active":"2024-09-15T23:14:41.641282+0000","last_became_peered":"2024-09-15T23:14:41.641282+0000","last_unstale":"2024-09-15T23:14:41.641820+0000","last_undegraded":"2024-09-15T23:14:41.641820+0000","last_fullsized":"2024-09-15T23:14:41.641820+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:43.472601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635042+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:14:41.635042+0000","last_peered":"2024-09-15T23:14:41.635042+0000","last_clean":"2024-09-15T23:14:41.635042+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:14:41.635042+0000","last_undegraded":"2024-09-15T23:14:41.635042+0000","last_fullsized":"2024-09-15T23:14:41.635042+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640088+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:14:41.640088+0000","last_peered":"2024-09-15T23:14:41.640088+0000","last_clean":"2024-09-15T23:14:41.640088+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:14:41.640088+0000","last_undegraded":"2024-09-15T23:14:41.640088+0000","last_fullsized":"2024-09-15T23:14:41.640088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007194+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:14:42.007194+0000","last_peered":"2024-09-15T23:14:42.007194+0000","last_clean":"2024-09-15T23:14:42.007194+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:14:42.007194+0000","last_undegraded":"2024-09-15T23:14:42.007194+0000","last_fullsized":"2024-09-15T23:14:42.007194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634019+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:14:41.634019+0000","last_peered":"2024-09-15T23:14:41.634019+0000","last_clean":"2024-09-15T23:14:41.634019+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:14:41.634019+0000","last_undegraded":"2024-09-15T23:14:41.634019+0000","last_fullsized":"2024-09-15T23:14:41.634019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"88'3","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:15:00.559293+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:15:00.559293+0000","last_peered":"2024-09-15T23:15:00.559293+0000","last_clean":"2024-09-15T23:15:00.559293+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:15:00.559293+0000","last_undegraded":"2024-09-15T23:15:00.559293+0000","last_fullsized":"2024-09-15T23:15:00.559293+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":54,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007527+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:14:42.007527+0000","last_peered":"2024-09-15T23:14:42.007527+0000","last_clean":"2024-09-15T23:14:42.007527+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:14:42.007527+0000","last_undegraded":"2024-09-15T23:14:42.007527+0000","last_fullsized":"2024-09-15T23:14:42.007527+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635001+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:14:41.635001+0000","last_peered":"2024-09-15T23:14:41.635001+0000","last_clean":"2024-09-15T23:14:41.635001+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:14:41.635001+0000","last_undegraded":"2024-09-15T23:14:41.635001+0000","last_fullsized":"2024-09-15T23:14:41.635001+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007590+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:14:42.007590+0000","last_peered":"2024-09-15T23:14:42.007590+0000","last_clean":"2024-09-15T23:14:42.007590+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:14:42.007590+0000","last_undegraded":"2024-09-15T23:14:42.007590+0000","last_fullsized":"2024-09-15T23:14:42.007590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632461+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:14:41.632461+0000","last_peered":"2024-09-15T23:14:41.632461+0000","last_clean":"2024-09-15T23:14:41.632461+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:14:41.632461+0000","last_undegraded":"2024-09-15T23:14:41.632461+0000","last_fullsized":"2024-09-15T23:14:41.632461+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635438+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:14:41.635438+0000","last_peered":"2024-09-15T23:14:41.635438+0000","last_clean":"2024-09-15T23:14:41.635438+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:14:41.635438+0000","last_undegraded":"2024-09-15T23:14:41.635438+0000","last_fullsized":"2024-09-15T23:14:41.635438+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007884+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:14:42.007884+0000","last_peered":"2024-09-15T23:14:42.007884+0000","last_clean":"2024-09-15T23:14:42.007884+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:14:42.007884+0000","last_undegraded":"2024-09-15T23:14:42.007884+0000","last_fullsized":"2024-09-15T23:14:42.007884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.630922+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:14:41.630922+0000","last_peered":"2024-09-15T23:14:41.630922+0000","last_clean":"2024-09-15T23:14:41.630922+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:14:41.630922+0000","last_undegraded":"2024-09-15T23:14:41.630922+0000","last_fullsized":"2024-09-15T23:14:41.630922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993682+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:14:41.993682+0000","last_peered":"2024-09-15T23:14:41.993682+0000","last_clean":"2024-09-15T23:14:41.993682+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:14:41.993682+0000","last_undegraded":"2024-09-15T23:14:41.993682+0000","last_fullsized":"2024-09-15T23:14:41.993682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632654+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:14:41.632654+0000","last_peered":"2024-09-15T23:14:41.632654+0000","last_clean":"2024-09-15T23:14:41.632654+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:14:41.632654+0000","last_undegraded":"2024-09-15T23:14:41.632654+0000","last_fullsized":"2024-09-15T23:14:41.632654+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639342+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:14:41.639342+0000","last_peered":"2024-09-15T23:14:41.639342+0000","last_clean":"2024-09-15T23:14:41.639342+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:14:41.639342+0000","last_undegraded":"2024-09-15T23:14:41.639342+0000","last_fullsized":"2024-09-15T23:14:41.639342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":96,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:59.558641+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:14:59.558641+0000","last_peered":"2024-09-15T23:14:59.558641+0000","last_clean":"2024-09-15T23:14:59.558641+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:14:59.558641+0000","last_undegraded":"2024-09-15T23:14:59.558641+0000","last_fullsized":"2024-09-15T23:14:59.558641+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636592+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:14:41.636592+0000","last_peered":"2024-09-15T23:14:41.636592+0000","last_clean":"2024-09-15T23:14:41.636592+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:14:41.636592+0000","last_undegraded":"2024-09-15T23:14:41.636592+0000","last_fullsized":"2024-09-15T23:14:41.636592+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639974+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:14:41.639974+0000","last_peered":"2024-09-15T23:14:41.639974+0000","last_clean":"2024-09-15T23:14:41.639974+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:14:41.639974+0000","last_undegraded":"2024-09-15T23:14:41.639974+0000","last_fullsized":"2024-09-15T23:14:41.639974+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635174+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:14:41.635174+0000","last_peered":"2024-09-15T23:14:41.635174+0000","last_clean":"2024-09-15T23:14:41.635174+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:14:41.635174+0000","last_undegraded":"2024-09-15T23:14:41.635174+0000","last_fullsized":"2024-09-15T23:14:41.635174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.638994+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:14:41.638994+0000","last_peered":"2024-09-15T23:14:41.638994+0000","last_clean":"2024-09-15T23:14:41.638994+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:14:41.638994+0000","last_undegraded":"2024-09-15T23:14:41.638994+0000","last_fullsized":"2024-09-15T23:14:41.638994+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639105+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:14:41.639105+0000","last_peered":"2024-09-15T23:14:41.639105+0000","last_clean":"2024-09-15T23:14:41.639105+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:14:41.639105+0000","last_undegraded":"2024-09-15T23:14:41.639105+0000","last_fullsized":"2024-09-15T23:14:41.639105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634855+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:14:41.634855+0000","last_peered":"2024-09-15T23:14:41.634855+0000","last_clean":"2024-09-15T23:14:41.634855+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:14:41.634855+0000","last_undegraded":"2024-09-15T23:14:41.634855+0000","last_fullsized":"2024-09-15T23:14:41.634855+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.010010+0000","last_change":"2024-09-15T23:14:42.010010+0000","last_active":"2024-09-15T23:14:42.010010+0000","last_peered":"2024-09-15T23:14:42.010010+0000","last_clean":"2024-09-15T23:14:42.010010+0000","last_became_active":"2024-09-15T23:14:42.009603+0000","last_became_peered":"2024-09-15T23:14:42.009603+0000","last_unstale":"2024-09-15T23:14:42.010010+0000","last_undegraded":"2024-09-15T23:14:42.010010+0000","last_fullsized":"2024-09-15T23:14:42.010010+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:34:51.849548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635951+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:14:41.635951+0000","last_peered":"2024-09-15T23:14:41.635951+0000","last_clean":"2024-09-15T23:14:41.635951+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:14:41.635951+0000","last_undegraded":"2024-09-15T23:14:41.635951+0000","last_fullsized":"2024-09-15T23:14:41.635951+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639616+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:14:41.639616+0000","last_peered":"2024-09-15T23:14:41.639616+0000","last_clean":"2024-09-15T23:14:41.639616+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:14:41.639616+0000","last_undegraded":"2024-09-15T23:14:41.639616+0000","last_fullsized":"2024-09-15T23:14:41.639616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008057+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:14:42.008057+0000","last_peered":"2024-09-15T23:14:42.008057+0000","last_clean":"2024-09-15T23:14:42.008057+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:14:42.008057+0000","last_undegraded":"2024-09-15T23:14:42.008057+0000","last_fullsized":"2024-09-15T23:14:42.008057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.632233+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:14:41.632233+0000","last_peered":"2024-09-15T23:14:41.632233+0000","last_clean":"2024-09-15T23:14:41.632233+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:14:41.632233+0000","last_undegraded":"2024-09-15T23:14:41.632233+0000","last_fullsized":"2024-09-15T23:14:41.632233+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008066+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:14:42.008066+0000","last_peered":"2024-09-15T23:14:42.008066+0000","last_clean":"2024-09-15T23:14:42.008066+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:14:42.008066+0000","last_undegraded":"2024-09-15T23:14:42.008066+0000","last_fullsized":"2024-09-15T23:14:42.008066+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634158+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:14:41.634158+0000","last_peered":"2024-09-15T23:14:41.634158+0000","last_clean":"2024-09-15T23:14:41.634158+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:14:41.634158+0000","last_undegraded":"2024-09-15T23:14:41.634158+0000","last_fullsized":"2024-09-15T23:14:41.634158+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631480+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:14:41.631480+0000","last_peered":"2024-09-15T23:14:41.631480+0000","last_clean":"2024-09-15T23:14:41.631480+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:14:41.631480+0000","last_undegraded":"2024-09-15T23:14:41.631480+0000","last_fullsized":"2024-09-15T23:14:41.631480+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635318+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:14:41.635318+0000","last_peered":"2024-09-15T23:14:41.635318+0000","last_clean":"2024-09-15T23:14:41.635318+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:14:41.635318+0000","last_undegraded":"2024-09-15T23:14:41.635318+0000","last_fullsized":"2024-09-15T23:14:41.635318+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640168+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:14:41.640168+0000","last_peered":"2024-09-15T23:14:41.640168+0000","last_clean":"2024-09-15T23:14:41.640168+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:14:41.640168+0000","last_undegraded":"2024-09-15T23:14:41.640168+0000","last_fullsized":"2024-09-15T23:14:41.640168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.008717+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:14:42.008717+0000","last_peered":"2024-09-15T23:14:42.008717+0000","last_clean":"2024-09-15T23:14:42.008717+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:14:42.008717+0000","last_undegraded":"2024-09-15T23:14:42.008717+0000","last_fullsized":"2024-09-15T23:14:42.008717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635838+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:14:41.635838+0000","last_peered":"2024-09-15T23:14:41.635838+0000","last_clean":"2024-09-15T23:14:41.635838+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:14:41.635838+0000","last_undegraded":"2024-09-15T23:14:41.635838+0000","last_fullsized":"2024-09-15T23:14:41.635838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.633133+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:14:41.633133+0000","last_peered":"2024-09-15T23:14:41.633133+0000","last_clean":"2024-09-15T23:14:41.633133+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:14:41.633133+0000","last_undegraded":"2024-09-15T23:14:41.633133+0000","last_fullsized":"2024-09-15T23:14:41.633133+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635063+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:14:41.635063+0000","last_peered":"2024-09-15T23:14:41.635063+0000","last_clean":"2024-09-15T23:14:41.635063+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:14:41.635063+0000","last_undegraded":"2024-09-15T23:14:41.635063+0000","last_fullsized":"2024-09-15T23:14:41.635063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634405+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:14:41.634405+0000","last_peered":"2024-09-15T23:14:41.634405+0000","last_clean":"2024-09-15T23:14:41.634405+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:14:41.634405+0000","last_undegraded":"2024-09-15T23:14:41.634405+0000","last_fullsized":"2024-09-15T23:14:41.634405+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.631378+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:14:41.631378+0000","last_peered":"2024-09-15T23:14:41.631378+0000","last_clean":"2024-09-15T23:14:41.631378+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:14:41.631378+0000","last_undegraded":"2024-09-15T23:14:41.631378+0000","last_fullsized":"2024-09-15T23:14:41.631378+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636882+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:14:41.636882+0000","last_peered":"2024-09-15T23:14:41.636882+0000","last_clean":"2024-09-15T23:14:41.636882+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:14:41.636882+0000","last_undegraded":"2024-09-15T23:14:41.636882+0000","last_fullsized":"2024-09-15T23:14:41.636882+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.640638+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:14:41.640638+0000","last_peered":"2024-09-15T23:14:41.640638+0000","last_clean":"2024-09-15T23:14:41.640638+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:14:41.640638+0000","last_undegraded":"2024-09-15T23:14:41.640638+0000","last_fullsized":"2024-09-15T23:14:41.640638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993959+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:14:41.993959+0000","last_peered":"2024-09-15T23:14:41.993959+0000","last_clean":"2024-09-15T23:14:41.993959+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:14:41.993959+0000","last_undegraded":"2024-09-15T23:14:41.993959+0000","last_fullsized":"2024-09-15T23:14:41.993959+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634160+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:14:41.634160+0000","last_peered":"2024-09-15T23:14:41.634160+0000","last_clean":"2024-09-15T23:14:41.634160+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:14:41.634160+0000","last_undegraded":"2024-09-15T23:14:41.634160+0000","last_fullsized":"2024-09-15T23:14:41.634160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:42.007787+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:14:42.007787+0000","last_peered":"2024-09-15T23:14:42.007787+0000","last_clean":"2024-09-15T23:14:42.007787+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:14:42.007787+0000","last_undegraded":"2024-09-15T23:14:42.007787+0000","last_fullsized":"2024-09-15T23:14:42.007787+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.639002+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:14:41.639002+0000","last_peered":"2024-09-15T23:14:41.639002+0000","last_clean":"2024-09-15T23:14:41.639002+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:14:41.639002+0000","last_undegraded":"2024-09-15T23:14:41.639002+0000","last_fullsized":"2024-09-15T23:14:41.639002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.993633+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:14:41.993633+0000","last_peered":"2024-09-15T23:14:41.993633+0000","last_clean":"2024-09-15T23:14:41.993633+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:14:41.993633+0000","last_undegraded":"2024-09-15T23:14:41.993633+0000","last_fullsized":"2024-09-15T23:14:41.993633+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.637111+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:14:41.637111+0000","last_peered":"2024-09-15T23:14:41.637111+0000","last_clean":"2024-09-15T23:14:41.637111+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:14:41.637111+0000","last_undegraded":"2024-09-15T23:14:41.637111+0000","last_fullsized":"2024-09-15T23:14:41.637111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.636813+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:14:41.636813+0000","last_peered":"2024-09-15T23:14:41.636813+0000","last_clean":"2024-09-15T23:14:41.636813+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:14:41.636813+0000","last_undegraded":"2024-09-15T23:14:41.636813+0000","last_fullsized":"2024-09-15T23:14:41.636813+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.634425+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:14:41.634425+0000","last_peered":"2024-09-15T23:14:41.634425+0000","last_clean":"2024-09-15T23:14:41.634425+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:14:41.634425+0000","last_undegraded":"2024-09-15T23:14:41.634425+0000","last_fullsized":"2024-09-15T23:14:41.634425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635105+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:14:41.635105+0000","last_peered":"2024-09-15T23:14:41.635105+0000","last_clean":"2024-09-15T23:14:41.635105+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:14:41.635105+0000","last_undegraded":"2024-09-15T23:14:41.635105+0000","last_fullsized":"2024-09-15T23:14:41.635105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-15T23:14:41.635722+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:14:41.635722+0000","last_peered":"2024-09-15T23:14:41.635722+0000","last_clean":"2024-09-15T23:14:41.635722+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:14:41.635722+0000","last_undegraded":"2024-09-15T23:14:41.635722+0000","last_fullsized":"2024-09-15T23:14:41.635722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66934,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":346,"num_read_kb":244,"num_write":168,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":200802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514528,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":995328,"data_stored":503506,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743460,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1134592,"data_stored":1153593,"data_compressed":59306,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906986,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":507726,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135919,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":503438,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364851,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1052672,"data_stored":1099461,"data_compressed":59306,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528377,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":983040,"data_stored":498806,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790014,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1060864,"data_stored":1095049,"data_compressed":59306,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953539,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":995328,"data_stored":503456,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":995328,"data_stored":499000,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":563390,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1052672,"data_stored":561647,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":500194,"data_compressed":23542,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":64674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:15:04.809 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-15T23:15:04.809 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T23:15:04.809 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-15T23:15:04.809 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-15T23:15:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:05.190 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:04 smithi062 ceph-mon[30949]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:05.190 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:04 smithi062 ceph-mon[30949]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:05.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:04 smithi067 ceph-mon[35665]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:05.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:04 smithi067 ceph-mon[35665]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:05.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:04 smithi148 ceph-mon[35610]: pgmap v332: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:05.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:04 smithi148 ceph-mon[35610]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:05.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.575+0000 7f994ba96640 1 -- 172.21.15.62:0/3946975443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f994410e480 msgr2=0x7f9944110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.575+0000 7f994ba96640 1 --2- 172.21.15.62:0/3946975443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f994410e480 0x7f9944110870 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f993c009ca0 tx=0x7f993c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.575+0000 7f994ba96640 1 -- 172.21.15.62:0/3946975443 shutdown_connections 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.575+0000 7f994ba96640 1 --2- 172.21.15.62:0/3946975443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f994410e480 0x7f9944110870 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.576+0000 7f994ba96640 1 --2- 172.21.15.62:0/3946975443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9944101280 0x7f994410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.576+0000 7f994ba96640 1 --2- 172.21.15.62:0/3946975443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99441008d0 0x7f9944100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.576+0000 7f994ba96640 1 -- 172.21.15.62:0/3946975443 >> 172.21.15.62:0/3946975443 conn(0x7f99440fc7a0 msgr2=0x7f99440febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.576+0000 7f994ba96640 1 -- 172.21.15.62:0/3946975443 shutdown_connections 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.576+0000 7f994ba96640 1 -- 172.21.15.62:0/3946975443 wait complete. 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.576+0000 7f994ba96640 1 Processor -- start 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.576+0000 7f994ba96640 1 -- start start 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.576+0000 7f994ba96640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99441008d0 0x7f994419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:05.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f994ba96640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9944101280 0x7f994419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f994ba96640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f994410e480 0x7f994419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f994ba96640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9944114160 con 0x7f99441008d0 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f994ba96640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9944113fe0 con 0x7f9944101280 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f994ba96640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f99441142e0 con 0x7f994410e480 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949832640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99441008d0 0x7f994419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949832640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99441008d0 0x7f994419a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49676/0 (socket says 172.21.15.62:49676) 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949832640 1 -- 172.21.15.62:0/2082670817 learned_addr learned my addr 172.21.15.62:0/2082670817 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949031640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9944101280 0x7f994419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f994a033640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f994410e480 0x7f994419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949832640 1 -- 172.21.15.62:0/2082670817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f994410e480 msgr2=0x7f994419eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949832640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f994410e480 0x7f994419eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949832640 1 -- 172.21.15.62:0/2082670817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9944101280 msgr2=0x7f994419ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:05.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949832640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9944101280 0x7f994419ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949832640 1 -- 172.21.15.62:0/2082670817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994419f5b0 con 0x7f99441008d0 2024-09-15T23:15:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f994a033640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f994410e480 0x7f994419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f9949832640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99441008d0 0x7f994419a600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f994000c920 tx=0x7f994000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f993affd640 1 -- 172.21.15.62:0/2082670817 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9940007c70 con 0x7f99441008d0 2024-09-15T23:15:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f993affd640 1 -- 172.21.15.62:0/2082670817 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9940007e10 con 0x7f99441008d0 2024-09-15T23:15:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f994419f8a0 con 0x7f99441008d0 2024-09-15T23:15:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f993affd640 1 -- 172.21.15.62:0/2082670817 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9940018890 con 0x7f99441008d0 2024-09-15T23:15:05.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.577+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99441a7180 con 0x7f99441008d0 2024-09-15T23:15:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.578+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9944100cb0 con 0x7f99441008d0 2024-09-15T23:15:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.580+0000 7f993affd640 1 -- 172.21.15.62:0/2082670817 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f9940018b30 con 0x7f99441008d0 2024-09-15T23:15:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.580+0000 7f993affd640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99300a32d0 0x7f99300a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.580+0000 7f9949031640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99300a32d0 0x7f99300a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.580+0000 7f993affd640 1 -- 172.21.15.62:0/2082670817 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f99400d0060 con 0x7f99441008d0 2024-09-15T23:15:05.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.581+0000 7f9949031640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99300a32d0 0x7f99300a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9934005fd0 tx=0x7f9934005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:05.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.584+0000 7f993affd640 1 -- 172.21.15.62:0/2082670817 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9940014030 con 0x7f99441008d0 2024-09-15T23:15:05.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.797+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9944109670 con 0x7f99441008d0 2024-09-15T23:15:05.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.797+0000 7f993affd640 1 -- 172.21.15.62:0/2082670817 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f994000de20 con 0x7f99441008d0 2024-09-15T23:15:05.798 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:05.799 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T23:15:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99300a32d0 msgr2=0x7f99300a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99300a32d0 0x7f99300a5790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9934005fd0 tx=0x7f9934005850 comp rx=0 tx=0).stop 2024-09-15T23:15:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99441008d0 msgr2=0x7f994419a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99441008d0 0x7f994419a600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f994000c920 tx=0x7f994000cde0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 shutdown_connections 2024-09-15T23:15:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99300a32d0 0x7f99300a5790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f994410e480 0x7f994419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9944101280 0x7f994419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 --2- 172.21.15.62:0/2082670817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f99441008d0 0x7f994419a600 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:05.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 >> 172.21.15.62:0/2082670817 conn(0x7f99440fc7a0 msgr2=0x7f99440feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:05.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 shutdown_connections 2024-09-15T23:15:05.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:05.800+0000 7f994ba96640 1 -- 172.21.15.62:0/2082670817 wait complete. 2024-09-15T23:15:06.115 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:05 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2082670817' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-15T23:15:06.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:05 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2082670817' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-15T23:15:06.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:05 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2082670817' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-15T23:15:06.547 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-15T23:15:06.548 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:15:06.808 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:07.195 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:06 smithi062 ceph-mon[30949]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:07.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:06 smithi067 ceph-mon[35665]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:07.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:06 smithi148 ceph-mon[35610]: pgmap v333: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:07.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.276+0000 7fe670e05640 1 -- 172.21.15.62:0/1614931952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c100a80 msgr2=0x7fe66c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:07.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.276+0000 7fe670e05640 1 --2- 172.21.15.62:0/1614931952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c100a80 0x7fe66c100ee0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe660009930 tx=0x7fe66002a420 comp rx=0 tx=0).stop 2024-09-15T23:15:07.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.276+0000 7fe670e05640 1 -- 172.21.15.62:0/1614931952 shutdown_connections 2024-09-15T23:15:07.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.276+0000 7fe670e05640 1 --2- 172.21.15.62:0/1614931952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe66c101420 0x7fe66c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.276+0000 7fe670e05640 1 --2- 172.21.15.62:0/1614931952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c100a80 0x7fe66c100ee0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.276+0000 7fe670e05640 1 --2- 172.21.15.62:0/1614931952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe66c106a80 0x7fe66c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.276+0000 7fe670e05640 1 -- 172.21.15.62:0/1614931952 >> 172.21.15.62:0/1614931952 conn(0x7fe66c0fc7a0 msgr2=0x7fe66c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.276+0000 7fe670e05640 1 -- 172.21.15.62:0/1614931952 shutdown_connections 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.276+0000 7fe670e05640 1 -- 172.21.15.62:0/1614931952 wait complete. 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe670e05640 1 Processor -- start 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe670e05640 1 -- start start 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe670e05640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe66c100a80 0x7fe66c1973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe670e05640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c101420 0x7fe66c1978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe670e05640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe66c106a80 0x7fe66c19bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe670e05640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe66c110e80 con 0x7fe66c101420 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe670e05640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe66c110d00 con 0x7fe66c100a80 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe670e05640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe66c111000 con 0x7fe66c106a80 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe669d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c101420 0x7fe66c1978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:07.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe669d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c101420 0x7fe66c1978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49698/0 (socket says 172.21.15.62:49698) 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.277+0000 7fe669d9b640 1 -- 172.21.15.62:0/816864136 learned_addr learned my addr 172.21.15.62:0/816864136 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe66a59c640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe66c100a80 0x7fe66c1973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe66ad9d640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe66c106a80 0x7fe66c19bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe669d9b640 1 -- 172.21.15.62:0/816864136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe66c106a80 msgr2=0x7fe66c19bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe669d9b640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe66c106a80 0x7fe66c19bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe669d9b640 1 -- 172.21.15.62:0/816864136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe66c100a80 msgr2=0x7fe66c1973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe669d9b640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe66c100a80 0x7fe66c1973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe669d9b640 1 -- 172.21.15.62:0/816864136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe66c19c350 con 0x7fe66c101420 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe669d9b640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c101420 0x7fe66c1978e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe660009230 tx=0x7fe660004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe65b7fe640 1 -- 172.21.15.62:0/816864136 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe66003f030 con 0x7fe66c101420 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe66c19c5e0 con 0x7fe66c101420 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe65b7fe640 1 -- 172.21.15.62:0/816864136 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe660003ad0 con 0x7fe66c101420 2024-09-15T23:15:07.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe65b7fe640 1 -- 172.21.15.62:0/816864136 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe660035070 con 0x7fe66c101420 2024-09-15T23:15:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.278+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe66c1a3f20 con 0x7fe66c101420 2024-09-15T23:15:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.279+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe66c1021c0 con 0x7fe66c101420 2024-09-15T23:15:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.280+0000 7fe65b7fe640 1 -- 172.21.15.62:0/816864136 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fe660042030 con 0x7fe66c101420 2024-09-15T23:15:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.281+0000 7fe65b7fe640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe6540a32d0 0x7fe6540a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.282+0000 7fe66a59c640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe6540a32d0 0x7fe6540a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.282+0000 7fe65b7fe640 1 -- 172.21.15.62:0/816864136 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fe6600f0360 con 0x7fe66c101420 2024-09-15T23:15:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.282+0000 7fe66a59c640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe6540a32d0 0x7fe6540a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe65c005fd0 tx=0x7fe65c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:07.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.286+0000 7fe65b7fe640 1 -- 172.21.15.62:0/816864136 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe6600bcbe0 con 0x7fe66c101420 2024-09-15T23:15:07.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.497+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe66c109020 con 0x7fe66c101420 2024-09-15T23:15:07.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.497+0000 7fe65b7fe640 1 -- 172.21.15.62:0/816864136 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11903 (secure 0 0 0) 0x7fe6600bc4e0 con 0x7fe66c101420 2024-09-15T23:15:07.498 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:07.498 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"btime":"2024-09-15T23:14:10:623582+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:14:10.188556+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:15:07.499 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 24 2024-09-15T23:15:07.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe6540a32d0 msgr2=0x7fe6540a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe6540a32d0 0x7fe6540a5790 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe65c005fd0 tx=0x7fe65c005850 comp rx=0 tx=0).stop 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c101420 msgr2=0x7fe66c1978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c101420 0x7fe66c1978e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe660009230 tx=0x7fe660004270 comp rx=0 tx=0).stop 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 shutdown_connections 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe6540a32d0 0x7fe6540a5790 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe66c106a80 0x7fe66c19bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe66c101420 0x7fe66c1978e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 --2- 172.21.15.62:0/816864136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe66c100a80 0x7fe66c1973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.499+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 >> 172.21.15.62:0/816864136 conn(0x7fe66c0fc7a0 msgr2=0x7fe66c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.500+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 shutdown_connections 2024-09-15T23:15:07.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:07.500+0000 7fe670e05640 1 -- 172.21.15.62:0/816864136 wait complete. 2024-09-15T23:15:08.210 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:07 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/816864136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:08.241 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:15:08.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:07 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/816864136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:08.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:07 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/816864136' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:08.499 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:08.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 -- 172.21.15.62:0/1579613494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec103740 msgr2=0x7f06ec107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:08.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1579613494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec103740 0x7f06ec107b80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f06d4009930 tx=0x7f06d402a420 comp rx=0 tx=0).stop 2024-09-15T23:15:08.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 -- 172.21.15.62:0/1579613494 shutdown_connections 2024-09-15T23:15:08.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1579613494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06ec108290 0x7f06ec10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1579613494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec103740 0x7f06ec107b80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1579613494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec102d90 0x7f06ec103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 -- 172.21.15.62:0/1579613494 >> 172.21.15.62:0/1579613494 conn(0x7f06ec0fc7d0 msgr2=0x7f06ec0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 -- 172.21.15.62:0/1579613494 shutdown_connections 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 -- 172.21.15.62:0/1579613494 wait complete. 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 Processor -- start 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.971+0000 7f06f16dc640 1 -- start start 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06f16dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec102d90 0x7f06ec1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06f16dc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06ec103740 0x7f06ec198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06f16dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec108290 0x7f06ec19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06f16dc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec112090 con 0x7f06ec102d90 2024-09-15T23:15:08.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06f16dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec111f10 con 0x7f06ec108290 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06f16dc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec112210 con 0x7f06ec103740 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec102d90 0x7f06ec1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec102d90 0x7f06ec1985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49720/0 (socket says 172.21.15.62:49720) 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eaffd640 1 -- 172.21.15.62:0/1119781653 learned_addr learned my addr 172.21.15.62:0/1119781653 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eb7fe640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec108290 0x7f06ec19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06ea7fc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06ec103740 0x7f06ec198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eaffd640 1 -- 172.21.15.62:0/1119781653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06ec103740 msgr2=0x7f06ec198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eaffd640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06ec103740 0x7f06ec198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eaffd640 1 -- 172.21.15.62:0/1119781653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec108290 msgr2=0x7f06ec19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eaffd640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec108290 0x7f06ec19ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eaffd640 1 -- 172.21.15.62:0/1119781653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ec19d560 con 0x7f06ec102d90 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06ea7fc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06ec103740 0x7f06ec198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:08.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.972+0000 7f06eaffd640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec102d90 0x7f06ec1985b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f06e400eab0 tx=0x7f06e400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.973+0000 7f06e3fff640 1 -- 172.21.15.62:0/1119781653 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e400ce40 con 0x7f06ec102d90 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.973+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06ec19d850 con 0x7f06ec102d90 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.973+0000 7f06e3fff640 1 -- 172.21.15.62:0/1119781653 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06e4011950 con 0x7f06ec102d90 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.973+0000 7f06e3fff640 1 -- 172.21.15.62:0/1119781653 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4010ae0 con 0x7f06ec102d90 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.973+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06ec1a5130 con 0x7f06ec102d90 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.974+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06ec103170 con 0x7f06ec102d90 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.975+0000 7f06e3fff640 1 -- 172.21.15.62:0/1119781653 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f06e4011b00 con 0x7f06ec102d90 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.976+0000 7f06e3fff640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f06d80a3350 0x7f06d80a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.976+0000 7f06ea7fc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f06d80a3350 0x7f06d80a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.976+0000 7f06e3fff640 1 -- 172.21.15.62:0/1119781653 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f06e40d0480 con 0x7f06ec102d90 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.977+0000 7f06ea7fc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f06d80a3350 0x7f06d80a5810 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f06d4000c00 tx=0x7f06d4032c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:08.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:08.980+0000 7f06e3fff640 1 -- 172.21.15.62:0/1119781653 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f06e4016030 con 0x7f06ec102d90 2024-09-15T23:15:09.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.189+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06ec199910 con 0x7f06ec102d90 2024-09-15T23:15:09.190 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:08 smithi062 ceph-mon[30949]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:09.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.190+0000 7f06e3fff640 1 -- 172.21.15.62:0/1119781653 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11903 (secure 0 0 0) 0x7f06e409cd20 con 0x7f06ec102d90 2024-09-15T23:15:09.191 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:09.191 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"btime":"2024-09-15T23:14:10:623582+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:14:10.188556+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:15:09.191 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 24 2024-09-15T23:15:09.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f06d80a3350 msgr2=0x7f06d80a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:09.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f06d80a3350 0x7f06d80a5810 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f06d4000c00 tx=0x7f06d4032c70 comp rx=0 tx=0).stop 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec102d90 msgr2=0x7f06ec1985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec102d90 0x7f06ec1985b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f06e400eab0 tx=0x7f06e400ef70 comp rx=0 tx=0).stop 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 shutdown_connections 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f06d80a3350 0x7f06d80a5810 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06ec108290 0x7f06ec19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f06ec103740 0x7f06ec198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 --2- 172.21.15.62:0/1119781653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06ec102d90 0x7f06ec1985b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 >> 172.21.15.62:0/1119781653 conn(0x7f06ec0fc7d0 msgr2=0x7f06ec106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 shutdown_connections 2024-09-15T23:15:09.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:09.192+0000 7f06f16dc640 1 -- 172.21.15.62:0/1119781653 wait complete. 2024-09-15T23:15:09.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:08 smithi067 ceph-mon[35665]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:09.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:08 smithi148 ceph-mon[35610]: pgmap v334: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:09.967 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:15:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:09 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1119781653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:10.235 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:10.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:09 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1119781653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:10.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:09 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1119781653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:10.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.696+0000 7f2b51bcf640 1 -- 172.21.15.62:0/1251759120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 msgr2=0x7f2b4c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:10.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.696+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/1251759120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 0x7f2b4c100ee0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009930 tx=0x7f2b4002a420 comp rx=0 tx=0).stop 2024-09-15T23:15:10.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.697+0000 7f2b51bcf640 1 -- 172.21.15.62:0/1251759120 shutdown_connections 2024-09-15T23:15:10.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.697+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/1251759120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4c101420 0x7f2b4c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.697+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/1251759120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 0x7f2b4c100ee0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.697+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/1251759120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b4c106a80 0x7f2b4c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.697+0000 7f2b51bcf640 1 -- 172.21.15.62:0/1251759120 >> 172.21.15.62:0/1251759120 conn(0x7f2b4c0fc7a0 msgr2=0x7f2b4c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.697+0000 7f2b51bcf640 1 -- 172.21.15.62:0/1251759120 shutdown_connections 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.697+0000 7f2b51bcf640 1 -- 172.21.15.62:0/1251759120 wait complete. 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.697+0000 7f2b51bcf640 1 Processor -- start 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.697+0000 7f2b51bcf640 1 -- start start 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b51bcf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 0x7f2b4c199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b51bcf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4c101420 0x7f2b4c199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b51bcf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b4c106a80 0x7f2b4c1a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b51bcf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c110ea0 con 0x7f2b4c100a80 2024-09-15T23:15:10.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b51bcf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c110d20 con 0x7f2b4c106a80 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b51bcf640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4c111020 con 0x7f2b4c101420 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 0x7f2b4c199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 0x7f2b4c199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49746/0 (socket says 172.21.15.62:49746) 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4b7fe640 1 -- 172.21.15.62:0/4104604242 learned_addr learned my addr 172.21.15.62:0/4104604242 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4affd640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4c101420 0x7f2b4c199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4bfff640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b4c106a80 0x7f2b4c1a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4b7fe640 1 -- 172.21.15.62:0/4104604242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4c101420 msgr2=0x7f2b4c199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4b7fe640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4c101420 0x7f2b4c199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4b7fe640 1 -- 172.21.15.62:0/4104604242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b4c106a80 msgr2=0x7f2b4c1a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4b7fe640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b4c106a80 0x7f2b4c1a42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4b7fe640 1 -- 172.21.15.62:0/4104604242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4c1a4810 con 0x7f2b4c100a80 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b4b7fe640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 0x7f2b4c199250 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00c920 tx=0x7f2b3c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:10.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b48ff9640 1 -- 172.21.15.62:0/4104604242 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c007c70 con 0x7f2b4c100a80 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4c1a49e0 con 0x7f2b4c100a80 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b48ff9640 1 -- 172.21.15.62:0/4104604242 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b3c007e10 con 0x7f2b4c100a80 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.698+0000 7f2b48ff9640 1 -- 172.21.15.62:0/4104604242 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c018890 con 0x7f2b4c100a80 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.699+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4c1a4ef0 con 0x7f2b4c100a80 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.700+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4c100ee0 con 0x7f2b4c100a80 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.701+0000 7f2b48ff9640 1 -- 172.21.15.62:0/4104604242 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f2b3c018a70 con 0x7f2b4c100a80 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.701+0000 7f2b48ff9640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b340a32d0 0x7f2b340a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.701+0000 7f2b4affd640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b340a32d0 0x7f2b340a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.701+0000 7f2b48ff9640 1 -- 172.21.15.62:0/4104604242 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f2b3c0cfe80 con 0x7f2b4c100a80 2024-09-15T23:15:10.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.702+0000 7f2b4affd640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b340a32d0 0x7f2b340a5790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009900 tx=0x7f2b400058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:10.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.706+0000 7f2b48ff9640 1 -- 172.21.15.62:0/4104604242 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2b3c014030 con 0x7f2b4c100a80 2024-09-15T23:15:10.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.883+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b4c1a5180 con 0x7f2b4c100a80 2024-09-15T23:15:10.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.884+0000 7f2b48ff9640 1 -- 172.21.15.62:0/4104604242 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24684 (secure 0 0 0) 0x7f2b3c09c700 con 0x7f2b4c100a80 2024-09-15T23:15:10.887 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:10.887 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":90,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:14:41.627161+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:15:10.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b340a32d0 msgr2=0x7f2b340a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:10.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b340a32d0 0x7f2b340a5790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2b40009900 tx=0x7f2b400058d0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 msgr2=0x7f2b4c199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:10.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 0x7f2b4c199250 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00c920 tx=0x7f2b3c00cde0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 shutdown_connections 2024-09-15T23:15:10.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b340a32d0 0x7f2b340a5790 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b4c106a80 0x7f2b4c1a42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4c101420 0x7f2b4c199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 --2- 172.21.15.62:0/4104604242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b4c100a80 0x7f2b4c199250 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:10.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 >> 172.21.15.62:0/4104604242 conn(0x7f2b4c0fc7a0 msgr2=0x7f2b4c0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:10.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 shutdown_connections 2024-09-15T23:15:10.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:10.887+0000 7f2b51bcf640 1 -- 172.21.15.62:0/4104604242 wait complete. 2024-09-15T23:15:11.210 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:10 smithi062 ceph-mon[30949]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:11.210 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:10 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4104604242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:11.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:10 smithi067 ceph-mon[35665]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:11.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:10 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4104604242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:11.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:10 smithi148 ceph-mon[35610]: pgmap v335: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:11.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:10 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4104604242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:11.633 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:15:11.895 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.344+0000 7fb62fb54640 1 -- 172.21.15.62:0/2653340415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628106a80 msgr2=0x7fb628106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.344+0000 7fb62fb54640 1 --2- 172.21.15.62:0/2653340415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628106a80 0x7fb628106e60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb624009930 tx=0x7fb62402a420 comp rx=0 tx=0).stop 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.344+0000 7fb62fb54640 1 -- 172.21.15.62:0/2653340415 shutdown_connections 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.344+0000 7fb62fb54640 1 --2- 172.21.15.62:0/2653340415 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb628101420 0x7fb62810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.344+0000 7fb62fb54640 1 --2- 172.21.15.62:0/2653340415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb628100a80 0x7fb628100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.344+0000 7fb62fb54640 1 --2- 172.21.15.62:0/2653340415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628106a80 0x7fb628106e60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.344+0000 7fb62fb54640 1 -- 172.21.15.62:0/2653340415 >> 172.21.15.62:0/2653340415 conn(0x7fb6280fc7a0 msgr2=0x7fb6280febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.344+0000 7fb62fb54640 1 -- 172.21.15.62:0/2653340415 shutdown_connections 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.344+0000 7fb62fb54640 1 -- 172.21.15.62:0/2653340415 wait complete. 2024-09-15T23:15:12.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62fb54640 1 Processor -- start 2024-09-15T23:15:12.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62fb54640 1 -- start start 2024-09-15T23:15:12.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62fb54640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628100a80 0x7fb628198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:12.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62fb54640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb628101420 0x7fb6281988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:12.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62fb54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb628106a80 0x7fb62819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:12.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62fb54640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb628111fa0 con 0x7fb628100a80 2024-09-15T23:15:12.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62fb54640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb628111e20 con 0x7fb628106a80 2024-09-15T23:15:12.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62fb54640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb628112120 con 0x7fb628101420 2024-09-15T23:15:12.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62d8f0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628100a80 0x7fb628198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62d8f0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628100a80 0x7fb628198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53996/0 (socket says 172.21.15.62:53996) 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62d8f0640 1 -- 172.21.15.62:0/3512308822 learned_addr learned my addr 172.21.15.62:0/3512308822 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62e0f1640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb628106a80 0x7fb62819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62d0ef640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb628101420 0x7fb6281988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62d8f0640 1 -- 172.21.15.62:0/3512308822 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb628101420 msgr2=0x7fb6281988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62d8f0640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb628101420 0x7fb6281988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62d8f0640 1 -- 172.21.15.62:0/3512308822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb628106a80 msgr2=0x7fb62819cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62d8f0640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb628106a80 0x7fb62819cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.345+0000 7fb62d8f0640 1 -- 172.21.15.62:0/3512308822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62819d330 con 0x7fb628100a80 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.346+0000 7fb62d8f0640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628100a80 0x7fb628198380 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb624009a50 tx=0x7fb624004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.346+0000 7fb61effd640 1 -- 172.21.15.62:0/3512308822 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb62403f030 con 0x7fb628100a80 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.346+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb62819d5c0 con 0x7fb628100a80 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.346+0000 7fb61effd640 1 -- 172.21.15.62:0/3512308822 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb624003ce0 con 0x7fb628100a80 2024-09-15T23:15:12.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.346+0000 7fb61effd640 1 -- 172.21.15.62:0/3512308822 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb624035070 con 0x7fb628100a80 2024-09-15T23:15:12.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.346+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6281a4f00 con 0x7fb628100a80 2024-09-15T23:15:12.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.347+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6281021c0 con 0x7fb628100a80 2024-09-15T23:15:12.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.348+0000 7fb61effd640 1 -- 172.21.15.62:0/3512308822 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fb624042030 con 0x7fb628100a80 2024-09-15T23:15:12.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.349+0000 7fb61effd640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb6140a32d0 0x7fb6140a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:12.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.349+0000 7fb62d0ef640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb6140a32d0 0x7fb6140a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:12.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.349+0000 7fb61effd640 1 -- 172.21.15.62:0/3512308822 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fb6240f0790 con 0x7fb628100a80 2024-09-15T23:15:12.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.349+0000 7fb62d0ef640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb6140a32d0 0x7fb6140a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb618005e10 tx=0x7fb618005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:12.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.352+0000 7fb61effd640 1 -- 172.21.15.62:0/3512308822 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb6240bd010 con 0x7fb628100a80 2024-09-15T23:15:12.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.568+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6281a50d0 con 0x7fb628100a80 2024-09-15T23:15:12.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.569+0000 7fb61effd640 1 -- 172.21.15.62:0/3512308822 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11903 (secure 0 0 0) 0x7fb6240bc910 con 0x7fb628100a80 2024-09-15T23:15:12.571 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:12.571 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"btime":"2024-09-15T23:14:10:623582+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:14:10.188556+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:15:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 24 2024-09-15T23:15:12.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb6140a32d0 msgr2=0x7fb6140a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:12.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb6140a32d0 0x7fb6140a5790 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb618005e10 tx=0x7fb618005da0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628100a80 msgr2=0x7fb628198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:12.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628100a80 0x7fb628198380 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb624009a50 tx=0x7fb624004270 comp rx=0 tx=0).stop 2024-09-15T23:15:12.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 shutdown_connections 2024-09-15T23:15:12.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb6140a32d0 0x7fb6140a5790 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb628106a80 0x7fb62819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb628101420 0x7fb6281988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 --2- 172.21.15.62:0/3512308822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb628100a80 0x7fb628198380 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:12.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 >> 172.21.15.62:0/3512308822 conn(0x7fb6280fc7a0 msgr2=0x7fb6280fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:12.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 shutdown_connections 2024-09-15T23:15:12.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:12.571+0000 7fb62fb54640 1 -- 172.21.15.62:0/3512308822 wait complete. 2024-09-15T23:15:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:12 smithi062 ceph-mon[30949]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:12 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3512308822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:13.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:12 smithi067 ceph-mon[35665]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:13.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:12 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3512308822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:13.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:12 smithi148 ceph-mon[35610]: pgmap v336: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:13.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:12 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3512308822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:13.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-15T23:13:33.200855+0000', 'modified': '2024-09-15T23:14:10.188556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24497, 'mds_1': 24470, 'mds_2': 24466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.148:6835/1239577031', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6834', 'nonce': 1239577031}, {'type': 'v1', 'addr': '172.21.15.148:6835', 'nonce': 1239577031}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.67:6835/3132447542', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 3132447542}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3132447542}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.148:6837/870913016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6836', 'nonce': 870913016}, {'type': 'v1', 'addr': '172.21.15.148:6837', 'nonce': 870913016}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.148:6839/2241485315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6838', 'nonce': 2241485315}, {'type': 'v1', 'addr': '172.21.15.148:6839', 'nonce': 2241485315}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.67:6837/747270351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6836', 'nonce': 747270351}, {'type': 'v1', 'addr': '172.21.15.67:6837', 'nonce': 747270351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24497': {'gid': 24497, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6839/818760439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6838', 'nonce': 818760439}, {'type': 'v1', 'addr': '172.21.15.67:6839', 'nonce': 818760439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24497, 'qdb_cluster': [24497, 24470, 24466]} 2024-09-15T23:15:13.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-15T23:15:13.371 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-15T23:15:13.633 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:14.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 -- 172.21.15.62:0/3284721401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240101420 msgr2=0x7f424010d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:14.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 --2- 172.21.15.62:0/3284721401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240101420 0x7f424010d950 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4238009cd0 tx=0x7f423802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 -- 172.21.15.62:0/3284721401 shutdown_connections 2024-09-15T23:15:14.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 --2- 172.21.15.62:0/3284721401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240101420 0x7f424010d950 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 --2- 172.21.15.62:0/3284721401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4240100a80 0x7f4240100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 --2- 172.21.15.62:0/3284721401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4240106a80 0x7f4240106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 -- 172.21.15.62:0/3284721401 >> 172.21.15.62:0/3284721401 conn(0x7f42400fc7a0 msgr2=0x7f42400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 -- 172.21.15.62:0/3284721401 shutdown_connections 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 -- 172.21.15.62:0/3284721401 wait complete. 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 Processor -- start 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.079+0000 7f4246730640 1 -- start start 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f4246730640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240100a80 0x7f4240198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f4246730640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4240101420 0x7f4240198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f4246730640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4240106a80 0x7f424019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f4246730640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4240110f70 con 0x7f4240100a80 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f4246730640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4240110df0 con 0x7f4240101420 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f4246730640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f42401110f0 con 0x7f4240106a80 2024-09-15T23:15:14.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f423ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240100a80 0x7f4240198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f423ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240100a80 0x7f4240198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54010/0 (socket says 172.21.15.62:54010) 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f423ffff640 1 -- 172.21.15.62:0/2029482495 learned_addr learned my addr 172.21.15.62:0/2029482495 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f423f7fe640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4240101420 0x7f4240198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f4244ccd640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4240106a80 0x7f424019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f423ffff640 1 -- 172.21.15.62:0/2029482495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4240106a80 msgr2=0x7f424019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f423ffff640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4240106a80 0x7f424019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f423ffff640 1 -- 172.21.15.62:0/2029482495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4240101420 msgr2=0x7f4240198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f423ffff640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4240101420 0x7f4240198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.080+0000 7f423ffff640 1 -- 172.21.15.62:0/2029482495 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424019d510 con 0x7f4240100a80 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.081+0000 7f423ffff640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240100a80 0x7f4240198560 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f423000b4a0 tx=0x7f423000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.081+0000 7f423d7fa640 1 -- 172.21.15.62:0/2029482495 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230015020 con 0x7f4240100a80 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.081+0000 7f423d7fa640 1 -- 172.21.15.62:0/2029482495 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4230011d10 con 0x7f4240100a80 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.081+0000 7f4246730640 1 -- 172.21.15.62:0/2029482495 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424019d800 con 0x7f4240100a80 2024-09-15T23:15:14.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.081+0000 7f423d7fa640 1 -- 172.21.15.62:0/2029482495 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423000f2e0 con 0x7f4240100a80 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.081+0000 7f4246730640 1 -- 172.21.15.62:0/2029482495 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42401a50e0 con 0x7f4240100a80 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.083+0000 7f423d7fa640 1 -- 172.21.15.62:0/2029482495 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f4230018020 con 0x7f4240100a80 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4248b09740 1 Processor -- start 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4248b09740 1 -- start start 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4248b09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 0x55d96898bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4248b09740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d96899aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4248b09740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 0x55d96899d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4248b09740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9688f2090 con 0x55d96898c1d0 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4248b09740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d968899a70 con 0x55d96899afc0 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4248b09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d9688ee050 con 0x55d96898b8b0 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f423ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d96899aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f423d7fa640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f422c0a3350 0x7f422c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f423ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d96899aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54022/0 (socket says 172.21.15.62:54022) 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f423ffff640 1 -- 172.21.15.62:0/405796879 learned_addr learned my addr 172.21.15.62:0/405796879 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4244ccd640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f422c0a3350 0x7f422c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f423f7fe640 1 --2- 172.21.15.62:0/405796879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 0x55d96899d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f4244ccd640 1 --2- 172.21.15.62:0/405796879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 0x55d96898bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.084+0000 7f423d7fa640 1 -- 172.21.15.62:0/2029482495 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f42300cf540 con 0x7f4240100a80 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f423ffff640 1 -- 172.21.15.62:0/405796879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 msgr2=0x55d96898bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f423ffff640 1 --2- 172.21.15.62:0/405796879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 0x55d96898bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f423ffff640 1 -- 172.21.15.62:0/405796879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 msgr2=0x55d96899d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f423ffff640 1 --2- 172.21.15.62:0/405796879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 0x55d96899d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f423ffff640 1 -- 172.21.15.62:0/405796879 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9688a6130 con 0x55d96898c1d0 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f423ffff640 1 --2- 172.21.15.62:0/405796879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d96899aa80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f42300a3f10 tx=0x7f4230010910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c56646e667a278 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:14.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f4244ccd640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f422c0a3350 0x7f422c0a5810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4238033040 tx=0x7f4238032fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f42367fc640 1 -- 172.21.15.62:0/405796879 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42300092b0 con 0x55d96898c1d0 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f42367fc640 1 -- 172.21.15.62:0/405796879 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f423009fb80 con 0x55d96898c1d0 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.085+0000 7f42367fc640 1 -- 172.21.15.62:0/405796879 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230097e40 con 0x55d96898c1d0 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 -- 172.21.15.62:0/405796879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 msgr2=0x55d96899aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 --2- 172.21.15.62:0/405796879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d96899aa80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f42300a3f10 tx=0x7f4230010910 comp rx=0 tx=0).stop 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 -- 172.21.15.62:0/405796879 shutdown_connections 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 --2- 172.21.15.62:0/405796879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 0x55d96899d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 --2- 172.21.15.62:0/405796879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d96899aa80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 --2- 172.21.15.62:0/405796879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 0x55d96898bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 -- 172.21.15.62:0/405796879 >> 172.21.15.62:0/405796879 conn(0x55d968987f60 msgr2=0x55d968988340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 -- 172.21.15.62:0/405796879 shutdown_connections 2024-09-15T23:15:14.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 -- 172.21.15.62:0/405796879 wait complete. 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 -- start start 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.086+0000 7f4248b09740 1 Processor -- start 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4248b09740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898b8b0 0x55d96899a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4248b09740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d968997c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4248b09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968998140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4248b09740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d96899db90 con 0x55d96898c1d0 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4248b09740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d9688acf20 con 0x55d96898b8b0 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4248b09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d96899dd10 con 0x55d96899afc0 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4248b09740 10 client.0 _ll_register_callbacks cb 0x55d968876030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d968997c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d968997c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54038/0 (socket says 172.21.15.62:54038) 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423f7fe640 1 -- 172.21.15.62:0/3527334396 learned_addr learned my addr 172.21.15.62:0/3527334396 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:14.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423ffff640 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898b8b0 0x55d96899a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4244ccd640 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968998140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423f7fe640 1 -- 172.21.15.62:0/3527334396 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 msgr2=0x55d968998140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423f7fe640 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968998140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423f7fe640 1 -- 172.21.15.62:0/3527334396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898b8b0 msgr2=0x55d96899a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423f7fe640 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898b8b0 0x55d96899a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423f7fe640 1 -- 172.21.15.62:0/3527334396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9686632b0 con 0x55d96898c1d0 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4244ccd640 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968998140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f423f7fe640 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d968997c00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4228011570 tx=0x7f4228011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5d28925eb31dc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f421f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f421f7fe640 1 -- 172.21.15.62:0/3527334396 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4228013670 con 0x55d96898c1d0 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f4248b09740 1 -- 172.21.15.62:0/3527334396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d9688ff600 con 0x55d96898c1d0 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.087+0000 7f421f7fe640 1 -- 172.21.15.62:0/3527334396 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f4228013810 con 0x55d96898c1d0 2024-09-15T23:15:14.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f421f7fe640 1 -- 172.21.15.62:0/3527334396 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4228013a50 con 0x55d96898c1d0 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f4248b09740 10 client.15069 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f4248b09740 1 -- 172.21.15.62:0/3527334396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d9689a95a0 con 0x55d96898c1d0 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f421f7fe640 1 -- 172.21.15.62:0/3527334396 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d9688ff600 con 0x55d96898c1d0 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f421f7fe640 1 -- 172.21.15.62:0/3527334396 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f422801d400 con 0x55d96898c1d0 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f4248b09740 10 client.15069 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f4248b09740 10 client.15069 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f4248b09740 20 client.15069 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f4248b09740 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55d9689abc10 0x55d9689ad080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f4248b09740 4 client.15069 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f4248b09740 1 -- 172.21.15.62:0/3527334396 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9688f9b10 con 0x55d9689abc10 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.088+0000 7f423ffff640 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55d9689abc10 0x55d9689ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.089+0000 7f421f7fe640 10 client.15069 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.089+0000 7f423ffff640 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55d9689abc10 0x55d9689ad080 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a9e637eceb2d2cf9 server_cookie=d33da15599659e9 in_seq=0 out_seq=0 2024-09-15T23:15:14.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f421f7fe640 1 -- 172.21.15.62:0/3527334396 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d9688f9b10 con 0x55d9689abc10 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f421f7fe640 10 client.15069 handle_command_reply: tid=0 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 client.15069 shutdown 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 -- 172.21.15.62:0/3527334396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 msgr2=0x55d968997c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d968997c00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4228011570 tx=0x7f4228011a30 comp rx=0 tx=0).stop 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 -- 172.21.15.62:0/3527334396 shutdown_connections 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55d9689abc10 0x55d9689ad080 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968998140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d968997c00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 --2- 172.21.15.62:0/3527334396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898b8b0 0x55d96899a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 -- 172.21.15.62:0/3527334396 >> 172.21.15.62:0/3527334396 conn(0x55d968987f60 msgr2=0x55d96898dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f421f7fe640 0 client.15069 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 -- 172.21.15.62:0/3527334396 shutdown_connections 2024-09-15T23:15:14.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 1 -- 172.21.15.62:0/3527334396 wait complete. 2024-09-15T23:15:14.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.090+0000 7f4248b09740 20 client.15069 trim_cache size 0 max 16384 2024-09-15T23:15:14.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f4248b09740 1 Processor -- start 2024-09-15T23:15:14.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f4248b09740 1 -- start start 2024-09-15T23:15:14.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f4248b09740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898b8b0 0x55d9689ab540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f4248b09740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898c1d0 0x55d968993130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f4248b09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968994e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f4248b09740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9689ab3c0 con 0x55d96898b8b0 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f4248b09740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d9689aba90 con 0x55d96898c1d0 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f4248b09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d9689ac160 con 0x55d96899afc0 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f423ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898b8b0 0x55d9689ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f423ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898b8b0 0x55d9689ab540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54056/0 (socket says 172.21.15.62:54056) 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.108+0000 7f423ffff640 1 -- 172.21.15.62:0/3628211785 learned_addr learned my addr 172.21.15.62:0/3628211785 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f423f7fe640 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898c1d0 0x55d968993130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f423ffff640 1 -- 172.21.15.62:0/3628211785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 msgr2=0x55d968994e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f4244ccd640 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968994e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f423ffff640 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968994e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f423ffff640 1 -- 172.21.15.62:0/3628211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898c1d0 msgr2=0x55d968993130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:14.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f423ffff640 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898c1d0 0x55d968993130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f423ffff640 1 -- 172.21.15.62:0/3628211785 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d96898c000 con 0x55d96898b8b0 2024-09-15T23:15:14.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f4244ccd640 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968994e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:14.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f423f7fe640 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898c1d0 0x55d968993130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:14.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f423ffff640 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898b8b0 0x55d9689ab540 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f423009b4d0 tx=0x7f42300047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85863ae44928a59 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:14.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f4235ffb640 1 -- 172.21.15.62:0/3628211785 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423009bed0 con 0x55d96898b8b0 2024-09-15T23:15:14.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f4235ffb640 1 -- 172.21.15.62:0/3628211785 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f423009b7d0 con 0x55d96898b8b0 2024-09-15T23:15:14.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.109+0000 7f4235ffb640 1 -- 172.21.15.62:0/3628211785 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423009ac20 con 0x55d96898b8b0 2024-09-15T23:15:14.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 -- 172.21.15.62:0/3628211785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898b8b0 msgr2=0x55d9689ab540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:14.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898b8b0 0x55d9689ab540 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f423009b4d0 tx=0x7f42300047d0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 -- 172.21.15.62:0/3628211785 shutdown_connections 2024-09-15T23:15:14.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96899afc0 0x55d968994e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96898c1d0 0x55d968993130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 --2- 172.21.15.62:0/3628211785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898b8b0 0x55d9689ab540 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 -- 172.21.15.62:0/3628211785 >> 172.21.15.62:0/3628211785 conn(0x55d968987f60 msgr2=0x55d968992a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:14.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 -- 172.21.15.62:0/3628211785 shutdown_connections 2024-09-15T23:15:14.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 -- 172.21.15.62:0/3628211785 wait complete. 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 -- start start 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.110+0000 7f4248b09740 1 Processor -- start 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f4248b09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 0x55d9689ae660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f4248b09740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d9689aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f4248b09740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 0x55d968a14310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f4248b09740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9689acf00 con 0x55d96898c1d0 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f4248b09740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d9689ac830 con 0x55d96899afc0 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f4248b09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d9689ad5d0 con 0x55d96898b8b0 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f4248b09740 10 client.0 _ll_register_callbacks cb 0x7f4228009510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:14.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d9689aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d9689aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54058/0 (socket says 172.21.15.62:54058) 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423f7fe640 1 -- 172.21.15.62:0/383513464 learned_addr learned my addr 172.21.15.62:0/383513464 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423ffff640 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 0x55d9689ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f4244ccd640 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 0x55d968a14310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423f7fe640 1 -- 172.21.15.62:0/383513464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 msgr2=0x55d9689ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423f7fe640 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 0x55d9689ae660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423f7fe640 1 -- 172.21.15.62:0/383513464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 msgr2=0x55d968a14310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423f7fe640 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 0x55d968a14310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423f7fe640 1 -- 172.21.15.62:0/383513464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d968a14850 con 0x55d96898c1d0 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f4244ccd640 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 0x55d968a14310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f423f7fe640 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d9689aeba0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f422801d8a0 tx=0x7f4228009270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1831aa766c6e7b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f421f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:14.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.111+0000 7f421f7fe640 1 -- 172.21.15.62:0/383513464 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422801de60 con 0x55d96898c1d0 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f421f7fe640 1 -- 172.21.15.62:0/383513464 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f4228013bf0 con 0x55d96898c1d0 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f4248b09740 1 -- 172.21.15.62:0/383513464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f423009b7d0 con 0x55d96898c1d0 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f421f7fe640 1 -- 172.21.15.62:0/383513464 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f4228038050 con 0x55d96898c1d0 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f4248b09740 10 client.15075 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f4248b09740 1 -- 172.21.15.62:0/383513464 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d968a15980 con 0x55d96898c1d0 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f421f7fe640 1 -- 172.21.15.62:0/383513464 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f423009b7d0 con 0x55d96898c1d0 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f421f7fe640 1 -- 172.21.15.62:0/383513464 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f422802abd0 con 0x55d96898c1d0 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f4248b09740 10 client.15075 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f4248b09740 10 client.15075 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f4248b09740 20 client.15075 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f4248b09740 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55d968a1a550 0x55d968a19260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f4248b09740 4 client.15075 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:15:14.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f4248b09740 1 -- 172.21.15.62:0/383513464 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "status"}) -- 0x55d9688760e0 con 0x55d968a1a550 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.112+0000 7f423ffff640 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55d968a1a550 0x55d968a19260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f421f7fe640 10 client.15075 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f423ffff640 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55d968a1a550 0x55d968a19260 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=beb25a6de3d73e98 server_cookie=b72e7a26bbf40d5d in_seq=0 out_seq=0 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f421f7fe640 1 -- 172.21.15.62:0/383513464 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d9688760e0 con 0x55d968a1a550 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f421f7fe640 10 client.15075 handle_command_reply: tid=0 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 client.15075 shutdown 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 -- 172.21.15.62:0/383513464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 msgr2=0x55d9689aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d9689aeba0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f422801d8a0 tx=0x7f4228009270 comp rx=0 tx=0).stop 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 -- 172.21.15.62:0/383513464 shutdown_connections 2024-09-15T23:15:14.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55d968a1a550 0x55d968a19260 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d96899afc0 0x55d968a14310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d96898c1d0 0x55d9689aeba0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f421f7fe640 0 client.15075 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 --2- 172.21.15.62:0/383513464 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d96898b8b0 0x55d9689ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 -- 172.21.15.62:0/383513464 >> 172.21.15.62:0/383513464 conn(0x55d968987f60 msgr2=0x55d968992a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 -- 172.21.15.62:0/383513464 shutdown_connections 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.113+0000 7f4248b09740 1 -- 172.21.15.62:0/383513464 wait complete. 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4248b09740 20 client.15075 trim_cache size 0 max 16384 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:15:14.122 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 2, 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24466, 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 66.489857185000005, 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 24, 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 90, 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 121.250168143 2024-09-15T23:15:14.123 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:15:14.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 -- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f422c0a3350 msgr2=0x7f422c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:14.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f422c0a3350 0x7f422c0a5810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4238033040 tx=0x7f4238032fd0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 -- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240100a80 msgr2=0x7f4240198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:14.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240100a80 0x7f4240198560 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f423000b4a0 tx=0x7f423000b8a0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 -- 172.21.15.62:0/2029482495 shutdown_connections 2024-09-15T23:15:14.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f422c0a3350 0x7f422c0a5810 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4240106a80 0x7f424019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4240101420 0x7f4240198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 --2- 172.21.15.62:0/2029482495 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4240100a80 0x7f4240198560 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:14.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 -- 172.21.15.62:0/2029482495 >> 172.21.15.62:0/2029482495 conn(0x7f42400fc7a0 msgr2=0x7f424006aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:14.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 -- 172.21.15.62:0/2029482495 shutdown_connections 2024-09-15T23:15:14.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:14.114+0000 7f4246730640 1 -- 172.21.15.62:0/2029482495 wait complete. 2024-09-15T23:15:14.819 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-15T23:15:15.077 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:15.200 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:14 smithi062 ceph-mon[30949]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:15.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:14 smithi067 ceph-mon[35665]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:14 smithi148 ceph-mon[35610]: pgmap v337: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:15.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.597+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1104481944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4102db0 msgr2=0x7f3da4103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:15.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.597+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1104481944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4102db0 0x7f3da4103190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009930 tx=0x7f3da002a420 comp rx=0 tx=0).stop 2024-09-15T23:15:15.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.597+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1104481944 shutdown_connections 2024-09-15T23:15:15.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.597+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1104481944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da41082e0 0x7f3da410ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.597+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1104481944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3da4103760 0x7f3da4107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.597+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1104481944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4102db0 0x7f3da4103190 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.597+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1104481944 >> 172.21.15.62:0/1104481944 conn(0x7f3da40fc7d0 msgr2=0x7f3da40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.597+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1104481944 shutdown_connections 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.597+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1104481944 wait complete. 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3dabfcd640 1 Processor -- start 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3dabfcd640 1 -- start start 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3dabfcd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3da4102db0 0x7f3da4100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3dabfcd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4103760 0x7f3da4100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3dabfcd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da41082e0 0x7f3da4101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3dabfcd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3da4112090 con 0x7f3da4103760 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3dabfcd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3da4111f10 con 0x7f3da41082e0 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3dabfcd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3da4112210 con 0x7f3da4102db0 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3da9568640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4103760 0x7f3da4100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3da9568640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4103760 0x7f3da4100f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54068/0 (socket says 172.21.15.62:54068) 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3da9568640 1 -- 172.21.15.62:0/1810453490 learned_addr learned my addr 172.21.15.62:0/1810453490 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3daa56a640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da41082e0 0x7f3da4101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.598+0000 7f3da9d69640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3da4102db0 0x7f3da4100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3da9568640 1 -- 172.21.15.62:0/1810453490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3da4102db0 msgr2=0x7f3da4100a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3da9568640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3da4102db0 0x7f3da4100a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3da9568640 1 -- 172.21.15.62:0/1810453490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da41082e0 msgr2=0x7f3da4101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3da9568640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da41082e0 0x7f3da4101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3da9568640 1 -- 172.21.15.62:0/1810453490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da41a6a80 con 0x7f3da4103760 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3da9568640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4103760 0x7f3da4100f50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400c970 tx=0x7f3d9400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3d9affd640 1 -- 172.21.15.62:0/1810453490 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d94007bb0 con 0x7f3da4103760 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1810453490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da41a6d70 con 0x7f3da4103760 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3d9affd640 1 -- 172.21.15.62:0/1810453490 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d94007d50 con 0x7f3da4103760 2024-09-15T23:15:15.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3d9affd640 1 -- 172.21.15.62:0/1810453490 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9400f450 con 0x7f3da4103760 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.599+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1810453490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da41a7250 con 0x7f3da4103760 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.601+0000 7f3d9affd640 1 -- 172.21.15.62:0/1810453490 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f3d94004f60 con 0x7f3da4103760 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3dae3a6740 1 Processor -- start 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3dae3a6740 1 -- start start 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db1e4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ef670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a4db14b090 con 0x55a4db1e52f0 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a4db0f2a70 con 0x55a4db1efbb0 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a4db147050 con 0x55a4db1e49d0 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3da9d69640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ef670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3da9d69640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ef670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54076/0 (socket says 172.21.15.62:54076) 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3da9d69640 1 -- 172.21.15.62:0/2378633914 learned_addr learned my addr 172.21.15.62:0/2378633914 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.602+0000 7f3d9affd640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d900a32d0 0x7f3d900a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3daa56a640 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db1e4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3da9568640 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3daa56a640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d900a32d0 0x7f3d900a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3da9d69640 1 -- 172.21.15.62:0/2378633914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 msgr2=0x55a4db1e4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3da9d69640 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db1e4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3da9d69640 1 -- 172.21.15.62:0/2378633914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 msgr2=0x55a4db1f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3d9affd640 1 -- 172.21.15.62:0/1810453490 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f3d940cfa20 con 0x7f3da4103760 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3da9d69640 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1f1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3da9d69640 1 -- 172.21.15.62:0/2378633914 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4db0ff130 con 0x55a4db1e52f0 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3da9568640 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3da9d69640 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ef670 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009930 tx=0x7f3da0004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae38c6a96a3704b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3daa56a640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d900a32d0 0x7f3d900a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00afd0 tx=0x7f3d9c00a4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3d8ffff640 1 -- 172.21.15.62:0/2378633914 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0035030 con 0x55a4db1e52f0 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3d8ffff640 1 -- 172.21.15.62:0/2378633914 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da0040070 con 0x55a4db1e52f0 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.603+0000 7f3d8ffff640 1 -- 172.21.15.62:0/2378633914 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0031c10 con 0x55a4db1e52f0 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.604+0000 7f3dae3a6740 1 -- 172.21.15.62:0/2378633914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 msgr2=0x55a4db1ef670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.604+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ef670 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009930 tx=0x7f3da0004200 comp rx=0 tx=0).stop 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.604+0000 7f3dae3a6740 1 -- 172.21.15.62:0/2378633914 shutdown_connections 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.604+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.604+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ef670 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.604+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/2378633914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db1e4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.604+0000 7f3dae3a6740 1 -- 172.21.15.62:0/2378633914 >> 172.21.15.62:0/2378633914 conn(0x55a4db1e0f60 msgr2=0x55a4db1e1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.604+0000 7f3dae3a6740 1 -- 172.21.15.62:0/2378633914 shutdown_connections 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.604+0000 7f3dae3a6740 1 -- 172.21.15.62:0/2378633914 wait complete. 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3dae3a6740 1 -- start start 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3dae3a6740 1 Processor -- start 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db1ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1efbb0 0x55a4db1ecc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a4db1f2780 con 0x55a4db1efbb0 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a4db105f20 con 0x55a4db1e52f0 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a4db1f2900 con 0x55a4db1e49d0 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3dae3a6740 10 client.0 _ll_register_callbacks cb 0x55a4db0cf030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3daa56a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1efbb0 0x55a4db1ecc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3daa56a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1efbb0 0x55a4db1ecc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54078/0 (socket says 172.21.15.62:54078) 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.605+0000 7f3daa56a640 1 -- 172.21.15.62:0/845095908 learned_addr learned my addr 172.21.15.62:0/845095908 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3da9568640 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3da9d69640 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db1ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3daa56a640 1 -- 172.21.15.62:0/845095908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 msgr2=0x55a4db1ef3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3daa56a640 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db1ef3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3daa56a640 1 -- 172.21.15.62:0/845095908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1e52f0 msgr2=0x55a4db1ec740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3daa56a640 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3daa56a640 1 -- 172.21.15.62:0/845095908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4daebc2b0 con 0x55a4db1efbb0 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3da9d69640 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db1ef3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3daa56a640 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1efbb0 0x55a4db1ecc80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c011f20 tx=0x7f3d9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea05d883237f610d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:15.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/845095908 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00fe60 con 0x55a4db1efbb0 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/845095908 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f3d9c0205f0 con 0x55a4db1efbb0 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3dae3a6740 1 -- 172.21.15.62:0/845095908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da0040070 con 0x55a4db1efbb0 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/845095908 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3d9c030050 con 0x55a4db1efbb0 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3dae3a6740 10 client.15087 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3dae3a6740 1 -- 172.21.15.62:0/845095908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a4db1fe070 con 0x55a4db1efbb0 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.606+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/845095908 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da0040070 con 0x55a4db1efbb0 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/845095908 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f3d9c021070 con 0x55a4db1efbb0 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3dae3a6740 10 client.15087 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3dae3a6740 10 client.15087 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3dae3a6740 20 client.15087 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4db200710 0x55a4db201b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3dae3a6740 4 client.15087 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3dae3a6740 1 -- 172.21.15.62:0/845095908 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4db158600 con 0x55a4db200710 2024-09-15T23:15:15.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3da9d69640 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4db200710 0x55a4db201b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3da9d69640 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4db200710 0x55a4db201b80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a8cfe76c2f87e391 server_cookie=5efb36408162182b in_seq=0 out_seq=0 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.607+0000 7f3d8d7fa640 10 client.15087 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.608+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/845095908 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a4db158600 con 0x55a4db200710 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.608+0000 7f3d8d7fa640 10 client.15087 handle_command_reply: tid=0 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.608+0000 7f3dae3a6740 1 client.15087 shutdown 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.608+0000 7f3dae3a6740 1 -- 172.21.15.62:0/845095908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1efbb0 msgr2=0x55a4db1ecc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.608+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1efbb0 0x55a4db1ecc80 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c011f20 tx=0x7f3d9c004a40 comp rx=0 tx=0).stop 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3dae3a6740 1 -- 172.21.15.62:0/845095908 shutdown_connections 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4db200710 0x55a4db201b80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1efbb0 0x55a4db1ecc80 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3d8d7fa640 0 client.15087 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/845095908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db1ef3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3dae3a6740 1 -- 172.21.15.62:0/845095908 >> 172.21.15.62:0/845095908 conn(0x55a4db1e0f60 msgr2=0x55a4db1e6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3dae3a6740 1 -- 172.21.15.62:0/845095908 shutdown_connections 2024-09-15T23:15:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3dae3a6740 1 -- 172.21.15.62:0/845095908 wait complete. 2024-09-15T23:15:15.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.609+0000 7f3dae3a6740 20 client.15087 trim_cache size 0 max 16384 2024-09-15T23:15:15.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3dae3a6740 1 Processor -- start 2024-09-15T23:15:15.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3dae3a6740 1 -- start start 2024-09-15T23:15:15.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db201b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1eb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a4db1ffec0 con 0x55a4db1e52f0 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a4db200590 con 0x55a4db1efbb0 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a4db200c60 con 0x55a4db1e49d0 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3da9568640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3da9568640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54080/0 (socket says 172.21.15.62:54080) 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3da9568640 1 -- 172.21.15.62:0/679490082 learned_addr learned my addr 172.21.15.62:0/679490082 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3da9d69640 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db201b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3daa56a640 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1eb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3da9568640 1 -- 172.21.15.62:0/679490082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 msgr2=0x55a4db201b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:15.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3da9568640 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db201b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3da9568640 1 -- 172.21.15.62:0/679490082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 msgr2=0x55a4db1eb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3da9568640 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1eb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.627+0000 7f3da9568640 1 -- 172.21.15.62:0/679490082 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4db1e5120 con 0x55a4db1e52f0 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.628+0000 7f3daa56a640 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.628+0000 7f3da9568640 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec000 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3d940cd0c0 tx=0x7f3d9401c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bae73f8e5c6a115 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.628+0000 7f3d8f7fe640 1 -- 172.21.15.62:0/679490082 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9409bc50 con 0x55a4db1e52f0 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.628+0000 7f3d8f7fe640 1 -- 172.21.15.62:0/679490082 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d9409ff50 con 0x55a4db1e52f0 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.628+0000 7f3d8f7fe640 1 -- 172.21.15.62:0/679490082 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9401c170 con 0x55a4db1e52f0 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.628+0000 7f3dae3a6740 1 -- 172.21.15.62:0/679490082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 msgr2=0x55a4db1ec000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.628+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec000 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3d940cd0c0 tx=0x7f3d9401c5d0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.629+0000 7f3dae3a6740 1 -- 172.21.15.62:0/679490082 shutdown_connections 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.629+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db1eb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.629+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db1ec000 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.629+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/679490082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db201b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.629+0000 7f3dae3a6740 1 -- 172.21.15.62:0/679490082 >> 172.21.15.62:0/679490082 conn(0x55a4db1e0f60 msgr2=0x55a4db1ec700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.629+0000 7f3dae3a6740 1 -- 172.21.15.62:0/679490082 shutdown_connections 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.629+0000 7f3dae3a6740 1 -- 172.21.15.62:0/679490082 wait complete. 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.629+0000 7f3dae3a6740 1 -- start start 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.629+0000 7f3dae3a6740 1 Processor -- start 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db203520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db203a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3dae3a6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db268e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a4db201a00 con 0x55a4db1e52f0 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a4db201330 con 0x55a4db1efbb0 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3dae3a6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a4db2020d0 con 0x55a4db1e49d0 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3dae3a6740 10 client.0 _ll_register_callbacks cb 0x7f3d9c011ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:15.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9568640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db203a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9568640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db203a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54086/0 (socket says 172.21.15.62:54086) 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9568640 1 -- 172.21.15.62:0/152736728 learned_addr learned my addr 172.21.15.62:0/152736728 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9d69640 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db203520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3daa56a640 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db268e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9568640 1 -- 172.21.15.62:0/152736728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 msgr2=0x55a4db203520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9568640 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db203520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9568640 1 -- 172.21.15.62:0/152736728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 msgr2=0x55a4db268e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9568640 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db268e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9568640 1 -- 172.21.15.62:0/152736728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4db269420 con 0x55a4db1e52f0 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3da9568640 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db203a60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3d94007590 tx=0x7f3d9409bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11743b7c0609f966 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3d8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/152736728 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9409b0a0 con 0x55a4db1e52f0 2024-09-15T23:15:15.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/152736728 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f3d9409b240 con 0x55a4db1e52f0 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.630+0000 7f3dae3a6740 1 -- 172.21.15.62:0/152736728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d9409ff50 con 0x55a4db1e52f0 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/152736728 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3d940ba270 con 0x55a4db1e52f0 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3dae3a6740 10 client.15093 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3dae3a6740 1 -- 172.21.15.62:0/152736728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a4db26a6e0 con 0x55a4db1e52f0 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/152736728 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d9409ff50 con 0x55a4db1e52f0 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/152736728 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f3d940a10e0 con 0x55a4db1e52f0 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3dae3a6740 10 client.15093 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3dae3a6740 10 client.15093 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3dae3a6740 20 client.15093 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4db26eca0 0x55a4db26d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3dae3a6740 4 client.15093 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3dae3a6740 1 -- 172.21.15.62:0/152736728 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "status"}) -- 0x55a4db152b10 con 0x55a4db26eca0 2024-09-15T23:15:15.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.631+0000 7f3da9d69640 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4db26eca0 0x55a4db26d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3da9d69640 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4db26eca0 0x55a4db26d9b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e73451ba9af5ceb4 server_cookie=8cb3e076b7dde19f in_seq=0 out_seq=0 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3d8d7fa640 10 client.15093 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3d8d7fa640 1 -- 172.21.15.62:0/152736728 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a4db152b10 con 0x55a4db26eca0 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3d8d7fa640 10 client.15093 handle_command_reply: tid=0 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 client.15093 shutdown 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 -- 172.21.15.62:0/152736728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 msgr2=0x55a4db203a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db203a60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3d94007590 tx=0x7f3d9409bd80 comp rx=0 tx=0).stop 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 -- 172.21.15.62:0/152736728 shutdown_connections 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4db26eca0 0x55a4db26d9b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4db1efbb0 0x55a4db268e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4db1e52f0 0x55a4db203a60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 --2- 172.21.15.62:0/152736728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4db1e49d0 0x55a4db203520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3d8d7fa640 0 client.15093 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:15:15.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 -- 172.21.15.62:0/152736728 >> 172.21.15.62:0/152736728 conn(0x55a4db1e0f60 msgr2=0x55a4db1e6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 -- 172.21.15.62:0/152736728 shutdown_connections 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.632+0000 7f3dae3a6740 1 -- 172.21.15.62:0/152736728 wait complete. 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dae3a6740 20 client.15093 trim_cache size 0 max 16384 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 1, 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24470, 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:15:15.643 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:15:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 69.017567057999997, 2024-09-15T23:15:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 24, 2024-09-15T23:15:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 90, 2024-09-15T23:15:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T23:15:15.644 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 125.35643789700001 2024-09-15T23:15:15.644 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d900a32d0 msgr2=0x7f3d900a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d900a32d0 0x7f3d900a5790 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00afd0 tx=0x7f3d9c00a4d0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4103760 msgr2=0x7f3da4100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4103760 0x7f3da4100f50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3d9400c970 tx=0x7f3d9400ce30 comp rx=0 tx=0).stop 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1810453490 shutdown_connections 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3d900a32d0 0x7f3d900a5790 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3da41082e0 0x7f3da4101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3da4103760 0x7f3da4100f50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 --2- 172.21.15.62:0/1810453490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3da4102db0 0x7f3da4100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1810453490 >> 172.21.15.62:0/1810453490 conn(0x7f3da40fc7d0 msgr2=0x7f3da4105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:15.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1810453490 shutdown_connections 2024-09-15T23:15:15.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:15.633+0000 7f3dabfcd640 1 -- 172.21.15.62:0/1810453490 wait complete. 2024-09-15T23:15:16.342 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T23:15:16.602 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:16.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:16 smithi062 ceph-mon[30949]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:17.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 -- 172.21.15.62:0/923262262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c101380 msgr2=0x7f0a8c10dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 --2- 172.21.15.62:0/923262262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c101380 0x7f0a8c10dfb0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0a74009900 tx=0x7f0a7402a410 comp rx=0 tx=0).stop 2024-09-15T23:15:17.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 -- 172.21.15.62:0/923262262 shutdown_connections 2024-09-15T23:15:17.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 --2- 172.21.15.62:0/923262262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c10e4f0 0x7f0a8c110900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 --2- 172.21.15.62:0/923262262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c101380 0x7f0a8c10dfb0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 --2- 172.21.15.62:0/923262262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a8c1009d0 0x7f0a8c100db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 -- 172.21.15.62:0/923262262 >> 172.21.15.62:0/923262262 conn(0x7f0a8c0fc7d0 msgr2=0x7f0a8c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 -- 172.21.15.62:0/923262262 shutdown_connections 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 -- 172.21.15.62:0/923262262 wait complete. 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.089+0000 7f0a9201e640 1 Processor -- start 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a9201e640 1 -- start start 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a9201e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c1009d0 0x7f0a8c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a9201e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a8c101380 0x7f0a8c19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a9201e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c10e4f0 0x7f0a8c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a9201e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c114260 con 0x7f0a8c10e4f0 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a9201e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1140e0 con 0x7f0a8c1009d0 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a9201e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8c1143e0 con 0x7f0a8c101380 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c10e4f0 0x7f0a8c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c10e4f0 0x7f0a8c19f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54102/0 (socket says 172.21.15.62:54102) 2024-09-15T23:15:17.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8bfff640 1 -- 172.21.15.62:0/795751511 learned_addr learned my addr 172.21.15.62:0/795751511 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c1009d0 0x7f0a8c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8affd640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a8c101380 0x7f0a8c19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8bfff640 1 -- 172.21.15.62:0/795751511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a8c101380 msgr2=0x7f0a8c19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8bfff640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a8c101380 0x7f0a8c19ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8bfff640 1 -- 172.21.15.62:0/795751511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c1009d0 msgr2=0x7f0a8c19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8bfff640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c1009d0 0x7f0a8c19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.090+0000 7f0a8bfff640 1 -- 172.21.15.62:0/795751511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8c19f6f0 con 0x7f0a8c10e4f0 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.091+0000 7f0a8bfff640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c10e4f0 0x7f0a8c19f010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400c970 tx=0x7f0a8400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.091+0000 7f0a88ff9640 1 -- 172.21.15.62:0/795751511 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84007bb0 con 0x7f0a8c10e4f0 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.091+0000 7f0a88ff9640 1 -- 172.21.15.62:0/795751511 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a84007d50 con 0x7f0a8c10e4f0 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.091+0000 7f0a9201e640 1 -- 172.21.15.62:0/795751511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8c19f9e0 con 0x7f0a8c10e4f0 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.091+0000 7f0a88ff9640 1 -- 172.21.15.62:0/795751511 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a84018800 con 0x7f0a8c10e4f0 2024-09-15T23:15:17.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.091+0000 7f0a9201e640 1 -- 172.21.15.62:0/795751511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8c1a72c0 con 0x7f0a8c10e4f0 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.094+0000 7f0a88ff9640 1 -- 172.21.15.62:0/795751511 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f0a8400f450 con 0x7f0a8c10e4f0 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.094+0000 7f0a943f7740 1 Processor -- start 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.094+0000 7f0a943f7740 1 -- start start 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.094+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e77db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.094+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e82670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.094+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 0x55a468e84fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.094+0000 7f0a943f7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a468dde090 con 0x55a468e782f0 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.094+0000 7f0a943f7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a468d85a70 con 0x55a468e779d0 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.094+0000 7f0a943f7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a468dda050 con 0x55a468e82bb0 2024-09-15T23:15:17.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e82670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e82670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54108/0 (socket says 172.21.15.62:54108) 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8b7fe640 1 -- 172.21.15.62:0/496527330 learned_addr learned my addr 172.21.15.62:0/496527330 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a88ff9640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a780a32d0 0x7f0a780a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8affd640 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 0x55a468e84fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8affd640 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e77db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8b7fe640 1 -- 172.21.15.62:0/496527330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 msgr2=0x55a468e84fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 0x55a468e84fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8b7fe640 1 -- 172.21.15.62:0/496527330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 msgr2=0x55a468e77db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8bfff640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a780a32d0 0x7f0a780a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e77db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8b7fe640 1 -- 172.21.15.62:0/496527330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a468d92130 con 0x55a468e782f0 2024-09-15T23:15:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8affd640 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e77db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a88ff9640 1 -- 172.21.15.62:0/795751511 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f0a840cfe20 con 0x7f0a8c10e4f0 2024-09-15T23:15:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e82670 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00af30 tx=0x7f0a7c00f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4396a1339e093f50 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a81ffb640 1 -- 172.21.15.62:0/496527330 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c010900 con 0x55a468e782f0 2024-09-15T23:15:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a8bfff640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a780a32d0 0x7f0a780a5790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0a8409bf30 tx=0x7f0a8406ace0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a81ffb640 1 -- 172.21.15.62:0/496527330 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a7c018430 con 0x55a468e782f0 2024-09-15T23:15:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.095+0000 7f0a81ffb640 1 -- 172.21.15.62:0/496527330 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c017640 con 0x55a468e782f0 2024-09-15T23:15:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.096+0000 7f0a943f7740 1 -- 172.21.15.62:0/496527330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 msgr2=0x55a468e82670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.096+0000 7f0a943f7740 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e82670 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00af30 tx=0x7f0a7c00f850 comp rx=0 tx=0).stop 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.096+0000 7f0a943f7740 1 -- 172.21.15.62:0/496527330 shutdown_connections 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.096+0000 7f0a943f7740 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 0x55a468e84fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.096+0000 7f0a943f7740 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e82670 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.096+0000 7f0a943f7740 1 --2- 172.21.15.62:0/496527330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e77db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.096+0000 7f0a943f7740 1 -- 172.21.15.62:0/496527330 >> 172.21.15.62:0/496527330 conn(0x55a468e73f60 msgr2=0x55a468e74340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.096+0000 7f0a943f7740 1 -- 172.21.15.62:0/496527330 shutdown_connections 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.096+0000 7f0a943f7740 1 -- 172.21.15.62:0/496527330 wait complete. 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a943f7740 1 -- start start 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a943f7740 1 Processor -- start 2024-09-15T23:15:17.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e82390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e782f0 0x55a468e7f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e7fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a943f7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a468e85780 con 0x55a468e82bb0 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a943f7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a468d98f20 con 0x55a468e779d0 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a943f7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a468e85900 con 0x55a468e782f0 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a943f7740 10 client.0 _ll_register_callbacks cb 0x55a468d62030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a8affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e7fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a8affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e7fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54110/0 (socket says 172.21.15.62:54110) 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a8affd640 1 -- 172.21.15.62:0/3403364468 learned_addr learned my addr 172.21.15.62:0/3403364468 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:17.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e782f0 0x55a468e7f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.097+0000 7f0a8affd640 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e82390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a8affd640 1 -- 172.21.15.62:0/3403364468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e782f0 msgr2=0x55a468e7f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:17.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a8affd640 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e782f0 0x55a468e7f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a8affd640 1 -- 172.21.15.62:0/3403364468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 msgr2=0x55a468e82390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:17.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a8affd640 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e82390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a8affd640 1 -- 172.21.15.62:0/3403364468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a468b4f2b0 con 0x55a468e82bb0 2024-09-15T23:15:17.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a8affd640 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e7fc70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0a74005ab0 tx=0x7f0a74005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3842de08809e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a6affd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a6affd640 1 -- 172.21.15.62:0/3403364468 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74035040 con 0x55a468e82bb0 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a6affd640 1 -- 172.21.15.62:0/3403364468 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f0a74004460 con 0x55a468e82bb0 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a943f7740 1 -- 172.21.15.62:0/3403364468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a7c018430 con 0x55a468e82bb0 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a6affd640 1 -- 172.21.15.62:0/3403364468 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0a74050050 con 0x55a468e82bb0 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a943f7740 10 client.15105 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a943f7740 1 -- 172.21.15.62:0/3403364468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a468e90f90 con 0x55a468e82bb0 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.098+0000 7f0a6affd640 1 -- 172.21.15.62:0/3403364468 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a7c018430 con 0x55a468e82bb0 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a6affd640 1 -- 172.21.15.62:0/3403364468 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f0a7403a040 con 0x55a468e82bb0 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a943f7740 10 client.15105 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:17.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a943f7740 10 client.15105 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a943f7740 20 client.15105 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a943f7740 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a468e93540 0x55a468e949b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a943f7740 4 client.15105 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a943f7740 1 -- 172.21.15.62:0/3403364468 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a468deb600 con 0x55a468e93540 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a468e93540 0x55a468e949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a6affd640 10 client.15105 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.099+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a468e93540 0x55a468e949b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12c5f83abb4046f7 server_cookie=572fb32d9c7b73ae in_seq=0 out_seq=0 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.100+0000 7f0a6affd640 1 -- 172.21.15.62:0/3403364468 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a468deb600 con 0x55a468e93540 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.100+0000 7f0a6affd640 10 client.15105 handle_command_reply: tid=0 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.100+0000 7f0a943f7740 1 client.15105 shutdown 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.100+0000 7f0a943f7740 1 -- 172.21.15.62:0/3403364468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 msgr2=0x55a468e7fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.100+0000 7f0a943f7740 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e7fc70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0a74005ab0 tx=0x7f0a74005a20 comp rx=0 tx=0).stop 2024-09-15T23:15:17.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.100+0000 7f0a943f7740 1 -- 172.21.15.62:0/3403364468 shutdown_connections 2024-09-15T23:15:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.101+0000 7f0a943f7740 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a468e93540 0x55a468e949b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.101+0000 7f0a943f7740 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e7fc70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.101+0000 7f0a943f7740 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e782f0 0x55a468e7f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.101+0000 7f0a943f7740 1 --2- 172.21.15.62:0/3403364468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e82390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.101+0000 7f0a943f7740 1 -- 172.21.15.62:0/3403364468 >> 172.21.15.62:0/3403364468 conn(0x55a468e73f60 msgr2=0x55a468e79c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.101+0000 7f0a6affd640 0 client.15105 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.101+0000 7f0a943f7740 1 -- 172.21.15.62:0/3403364468 shutdown_connections 2024-09-15T23:15:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.101+0000 7f0a943f7740 1 -- 172.21.15.62:0/3403364468 wait complete. 2024-09-15T23:15:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.101+0000 7f0a943f7740 20 client.15105 trim_cache size 0 max 16384 2024-09-15T23:15:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a943f7740 1 Processor -- start 2024-09-15T23:15:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a943f7740 1 -- start start 2024-09-15T23:15:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e779d0 0x55a468e93c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e782f0 0x55a468e942e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e92e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a943f7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a468e92cf0 con 0x55a468e82bb0 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a943f7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a468e933c0 con 0x55a468e782f0 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a943f7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a468e93a90 con 0x55a468e779d0 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e92e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e92e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54126/0 (socket says 172.21.15.62:54126) 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 -- 172.21.15.62:0/4092602256 learned_addr learned my addr 172.21.15.62:0/4092602256 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/4092602256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e782f0 0x55a468e942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 --2- 172.21.15.62:0/4092602256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e779d0 0x55a468e93c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 -- 172.21.15.62:0/4092602256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e779d0 msgr2=0x55a468e93c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 --2- 172.21.15.62:0/4092602256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e779d0 0x55a468e93c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 -- 172.21.15.62:0/4092602256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e782f0 msgr2=0x55a468e942e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 --2- 172.21.15.62:0/4092602256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e782f0 0x55a468e942e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 -- 172.21.15.62:0/4092602256 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a468e78120 con 0x55a468e82bb0 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.119+0000 7f0a8affd640 1 --2- 172.21.15.62:0/4092602256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e92e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0a7402a930 tx=0x7f0a7402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbcd9980befed43e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a817fa640 1 -- 172.21.15.62:0/4092602256 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74035070 con 0x55a468e82bb0 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a817fa640 1 -- 172.21.15.62:0/4092602256 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a74005500 con 0x55a468e82bb0 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a817fa640 1 -- 172.21.15.62:0/4092602256 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a740035f0 con 0x55a468e82bb0 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a943f7740 1 -- 172.21.15.62:0/4092602256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 msgr2=0x55a468e92e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a943f7740 1 --2- 172.21.15.62:0/4092602256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e92e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0a7402a930 tx=0x7f0a7402a410 comp rx=0 tx=0).stop 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a943f7740 1 -- 172.21.15.62:0/4092602256 shutdown_connections 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a943f7740 1 --2- 172.21.15.62:0/4092602256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e82bb0 0x55a468e92e70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a943f7740 1 --2- 172.21.15.62:0/4092602256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e782f0 0x55a468e942e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a943f7740 1 --2- 172.21.15.62:0/4092602256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e779d0 0x55a468e93c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a943f7740 1 -- 172.21.15.62:0/4092602256 >> 172.21.15.62:0/4092602256 conn(0x55a468e73f60 msgr2=0x55a468e7f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a943f7740 1 -- 172.21.15.62:0/4092602256 shutdown_connections 2024-09-15T23:15:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.120+0000 7f0a943f7740 1 -- 172.21.15.62:0/4092602256 wait complete. 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a943f7740 1 -- start start 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a943f7740 1 Processor -- start 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e9d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e97bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a943f7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 0x55a468e98300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a943f7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a468e94830 con 0x55a468e782f0 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a943f7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a468e94160 con 0x55a468e779d0 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a943f7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a468e94f00 con 0x55a468e82bb0 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a943f7740 10 client.0 _ll_register_callbacks cb 0x7f0a74009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a8b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e97bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a8b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e97bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54138/0 (socket says 172.21.15.62:54138) 2024-09-15T23:15:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.121+0000 7f0a8b7fe640 1 -- 172.21.15.62:0/2134993658 learned_addr learned my addr 172.21.15.62:0/2134993658 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a8affd640 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e9d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a8affd640 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 0x55a468e98300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a8b7fe640 1 -- 172.21.15.62:0/2134993658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 msgr2=0x55a468e98300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 0x55a468e98300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a8b7fe640 1 -- 172.21.15.62:0/2134993658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 msgr2=0x55a468e9d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e9d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a8b7fe640 1 -- 172.21.15.62:0/2134993658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a468efbc40 con 0x55a468e782f0 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a8affd640 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e9d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a8b7fe640 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e97bc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00af30 tx=0x7f0a7c00a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78de9206d54727d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a6affd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a6affd640 1 -- 172.21.15.62:0/2134993658 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c02e030 con 0x55a468e782f0 2024-09-15T23:15:17.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a6affd640 1 -- 172.21.15.62:0/2134993658 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f0a7c018640 con 0x55a468e782f0 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a943f7740 1 -- 172.21.15.62:0/2134993658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a74005500 con 0x55a468e782f0 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a6affd640 1 -- 172.21.15.62:0/2134993658 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0a7c00ab40 con 0x55a468e782f0 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a943f7740 10 client.15111 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a943f7740 1 -- 172.21.15.62:0/2134993658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a468efce40 con 0x55a468e782f0 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.122+0000 7f0a6affd640 1 -- 172.21.15.62:0/2134993658 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a74005500 con 0x55a468e782f0 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a6affd640 1 -- 172.21.15.62:0/2134993658 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f0a7c02a3e0 con 0x55a468e782f0 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a943f7740 10 client.15111 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a943f7740 10 client.15111 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a943f7740 20 client.15111 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a943f7740 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a468f01610 0x55a468f00320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a943f7740 4 client.15111 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:15:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a943f7740 1 -- 172.21.15.62:0/2134993658 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "status"}) -- 0x55a468de5b10 con 0x55a468f01610 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a8affd640 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a468f01610 0x55a468f00320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a8affd640 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a468f01610 0x55a468f00320 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=594cc67b99c3f465 server_cookie=dca82f8ed8f75f2a in_seq=0 out_seq=0 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.123+0000 7f0a6affd640 10 client.15111 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a6affd640 1 -- 172.21.15.62:0/2134993658 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a468de5b10 con 0x55a468f01610 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a6affd640 10 client.15111 handle_command_reply: tid=0 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 client.15111 shutdown 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 -- 172.21.15.62:0/2134993658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 msgr2=0x55a468e97bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e97bc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c00af30 tx=0x7f0a7c00a570 comp rx=0 tx=0).stop 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 -- 172.21.15.62:0/2134993658 shutdown_connections 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a468f01610 0x55a468f00320 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a468e82bb0 0x55a468e98300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a468e782f0 0x55a468e97bc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 --2- 172.21.15.62:0/2134993658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a468e779d0 0x55a468e9d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 -- 172.21.15.62:0/2134993658 >> 172.21.15.62:0/2134993658 conn(0x55a468e73f60 msgr2=0x55a468e7dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a6affd640 0 client.15111 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 -- 172.21.15.62:0/2134993658 shutdown_connections 2024-09-15T23:15:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 1 -- 172.21.15.62:0/2134993658 wait complete. 2024-09-15T23:15:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.124+0000 7f0a943f7740 20 client.15111 trim_cache size 0 max 16384 2024-09-15T23:15:17.141 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:15:17.141 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24497, 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 103.619027004, 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 24, 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 90, 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 114.76768287900001 2024-09-15T23:15:17.142 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:15:17.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 -- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a780a32d0 msgr2=0x7f0a780a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a780a32d0 0x7f0a780a5790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0a8409bf30 tx=0x7f0a8406ace0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 -- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c10e4f0 msgr2=0x7f0a8c19f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:17.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c10e4f0 0x7f0a8c19f010 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0a8400c970 tx=0x7f0a8400ce30 comp rx=0 tx=0).stop 2024-09-15T23:15:17.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 -- 172.21.15.62:0/795751511 shutdown_connections 2024-09-15T23:15:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a780a32d0 0x7f0a780a5790 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a8c10e4f0 0x7f0a8c19f010 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a8c101380 0x7f0a8c19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 --2- 172.21.15.62:0/795751511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c1009d0 0x7f0a8c19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 -- 172.21.15.62:0/795751511 >> 172.21.15.62:0/795751511 conn(0x7f0a8c0fc7d0 msgr2=0x7f0a8c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 -- 172.21.15.62:0/795751511 shutdown_connections 2024-09-15T23:15:17.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:17.125+0000 7f0a9201e640 1 -- 172.21.15.62:0/795751511 wait complete. 2024-09-15T23:15:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:16 smithi067 ceph-mon[35665]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:17.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:16 smithi148 ceph-mon[35610]: pgmap v338: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:17.913 INFO:tasks.cephadm:Setup complete, yielding 2024-09-15T23:15:17.914 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-15T23:15:17.923 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T23:15:17.923 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch status' 2024-09-15T23:15:18.183 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.665+0000 7f6374ced640 1 -- 172.21.15.62:0/3962413434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 msgr2=0x7f6370100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.665+0000 7f6374ced640 1 --2- 172.21.15.62:0/3962413434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 0x7f6370100680 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6360009900 tx=0x7f636002a410 comp rx=0 tx=0).stop 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.665+0000 7f6374ced640 1 -- 172.21.15.62:0/3962413434 shutdown_connections 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.665+0000 7f6374ced640 1 --2- 172.21.15.62:0/3962413434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6370101e40 0x7f63701086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.665+0000 7f6374ced640 1 --2- 172.21.15.62:0/3962413434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6370101480 0x7f6370101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.665+0000 7f6374ced640 1 --2- 172.21.15.62:0/3962413434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 0x7f6370100680 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.665+0000 7f6374ced640 1 -- 172.21.15.62:0/3962413434 >> 172.21.15.62:0/3962413434 conn(0x7f63700fba30 msgr2=0x7f63700fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.665+0000 7f6374ced640 1 -- 172.21.15.62:0/3962413434 shutdown_connections 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.665+0000 7f6374ced640 1 -- 172.21.15.62:0/3962413434 wait complete. 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f6374ced640 1 Processor -- start 2024-09-15T23:15:18.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f6374ced640 1 -- start start 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f6374ced640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 0x7f6370104210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f6374ced640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6370101480 0x7f6370102860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f6374ced640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6370101e40 0x7f6370102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f6374ced640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f637010b730 con 0x7f6370100280 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f6374ced640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f637010b5b0 con 0x7f6370101480 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f6374ced640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f637010b8b0 con 0x7f6370101e40 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f636e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 0x7f6370104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f636e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 0x7f6370104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54156/0 (socket says 172.21.15.62:54156) 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f636e59c640 1 -- 172.21.15.62:0/3103278676 learned_addr learned my addr 172.21.15.62:0/3103278676 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f6367fff640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6370101480 0x7f6370102860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.666+0000 7f636ed9d640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6370101e40 0x7f6370102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f636e59c640 1 -- 172.21.15.62:0/3103278676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6370101e40 msgr2=0x7f6370102da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:18.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f636e59c640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6370101e40 0x7f6370102da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f636e59c640 1 -- 172.21.15.62:0/3103278676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6370101480 msgr2=0x7f6370102860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f636e59c640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6370101480 0x7f6370102860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f636e59c640 1 -- 172.21.15.62:0/3103278676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6370103660 con 0x7f6370100280 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f636e59c640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 0x7f6370104210 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6360031440 tx=0x7f6360032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f63677fe640 1 -- 172.21.15.62:0/3103278676 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636003f030 con 0x7f6370100280 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f63677fe640 1 -- 172.21.15.62:0/3103278676 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6360003ce0 con 0x7f6370100280 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63701a2100 con 0x7f6370100280 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f63677fe640 1 -- 172.21.15.62:0/3103278676 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6360035070 con 0x7f6370100280 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.667+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63701a2630 con 0x7f6370100280 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.668+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63701a2920 con 0x7f6370100280 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.669+0000 7f63677fe640 1 -- 172.21.15.62:0/3103278676 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f6360042030 con 0x7f6370100280 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.670+0000 7f63677fe640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f635c0a32d0 0x7f635c0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.670+0000 7f6367fff640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f635c0a32d0 0x7f635c0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:18.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.670+0000 7f63677fe640 1 -- 172.21.15.62:0/3103278676 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f63600f06f0 con 0x7f6370100280 2024-09-15T23:15:18.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.670+0000 7f6367fff640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f635c0a32d0 0x7f635c0a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6370103f90 tx=0x7f6358009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:18.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.673+0000 7f63677fe640 1 -- 172.21.15.62:0/3103278676 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f63600bcf70 con 0x7f6370100280 2024-09-15T23:15:18.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.867+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f63701080d0 con 0x7f635c0a32d0 2024-09-15T23:15:18.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.869+0000 7f63677fe640 1 -- 172.21.15.62:0/3103278676 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f63701080d0 con 0x7f635c0a32d0 2024-09-15T23:15:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:Backend: cephadm 2024-09-15T23:15:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:Available: Yes 2024-09-15T23:15:18.871 INFO:teuthology.orchestra.run.smithi062.stdout:Paused: No 2024-09-15T23:15:18.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f635c0a32d0 msgr2=0x7f635c0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:18.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f635c0a32d0 0x7f635c0a5790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6370103f90 tx=0x7f6358009110 comp rx=0 tx=0).stop 2024-09-15T23:15:18.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 msgr2=0x7f6370104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:18.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 0x7f6370104210 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6360031440 tx=0x7f6360032e20 comp rx=0 tx=0).stop 2024-09-15T23:15:18.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 shutdown_connections 2024-09-15T23:15:18.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f635c0a32d0 0x7f635c0a5790 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6370101e40 0x7f6370102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6370101480 0x7f6370102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 --2- 172.21.15.62:0/3103278676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6370100280 0x7f6370104210 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 >> 172.21.15.62:0/3103278676 conn(0x7f63700fba30 msgr2=0x7f63701069b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 shutdown_connections 2024-09-15T23:15:18.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:18.872+0000 7f6374ced640 1 -- 172.21.15.62:0/3103278676 wait complete. 2024-09-15T23:15:19.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:18 smithi062 ceph-mon[30949]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:19.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:18 smithi067 ceph-mon[35665]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:19.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:18 smithi148 ceph-mon[35610]: pgmap v339: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:19.668 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-15T23:15:19.931 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:20.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:19 smithi062 ceph-mon[30949]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:20.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:19 smithi067 ceph-mon[35665]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:20.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:19 smithi148 ceph-mon[35610]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.351+0000 7ff062ffd640 1 -- 172.21.15.62:0/2214752624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c1026e0 msgr2=0x7ff05c104ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.351+0000 7ff062ffd640 1 --2- 172.21.15.62:0/2214752624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c1026e0 0x7ff05c104ad0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff054009900 tx=0x7ff05402a410 comp rx=0 tx=0).stop 2024-09-15T23:15:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.351+0000 7ff062ffd640 1 -- 172.21.15.62:0/2214752624 shutdown_connections 2024-09-15T23:15:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.351+0000 7ff062ffd640 1 --2- 172.21.15.62:0/2214752624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff05c107940 0x7ff05c109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.351+0000 7ff062ffd640 1 --2- 172.21.15.62:0/2214752624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff05c105010 0x7ff05c107400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.351+0000 7ff062ffd640 1 --2- 172.21.15.62:0/2214752624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c1026e0 0x7ff05c104ad0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.351+0000 7ff062ffd640 1 -- 172.21.15.62:0/2214752624 >> 172.21.15.62:0/2214752624 conn(0x7ff05c0fc280 msgr2=0x7ff05c0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.351+0000 7ff062ffd640 1 -- 172.21.15.62:0/2214752624 shutdown_connections 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.351+0000 7ff062ffd640 1 -- 172.21.15.62:0/2214752624 wait complete. 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff062ffd640 1 Processor -- start 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff062ffd640 1 -- start start 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff062ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff05c1026e0 0x7ff05c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff062ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c105010 0x7ff05c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff062ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff05c107940 0x7ff05c19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff062ffd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c10d880 con 0x7ff05c105010 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff062ffd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c10d700 con 0x7ff05c107940 2024-09-15T23:15:20.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff062ffd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c10da00 con 0x7ff05c1026e0 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff05bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c105010 0x7ff05c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff05bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c105010 0x7ff05c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54178/0 (socket says 172.21.15.62:54178) 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff05bfff640 1 -- 172.21.15.62:0/990721767 learned_addr learned my addr 172.21.15.62:0/990721767 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff06159a640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff05c107940 0x7ff05c19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff060d99640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff05c1026e0 0x7ff05c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff05bfff640 1 -- 172.21.15.62:0/990721767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff05c1026e0 msgr2=0x7ff05c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff05bfff640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff05c1026e0 0x7ff05c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff05bfff640 1 -- 172.21.15.62:0/990721767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff05c107940 msgr2=0x7ff05c19f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff05bfff640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff05c107940 0x7ff05c19f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.352+0000 7ff05bfff640 1 -- 172.21.15.62:0/990721767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05c19fb60 con 0x7ff05c105010 2024-09-15T23:15:20.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.353+0000 7ff05bfff640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c105010 0x7ff05c198470 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff044005fd0 tx=0x7ff04400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.353+0000 7ff059ffb640 1 -- 172.21.15.62:0/990721767 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044017070 con 0x7ff05c105010 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.353+0000 7ff059ffb640 1 -- 172.21.15.62:0/990721767 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff044009c30 con 0x7ff05c105010 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.353+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff05c19fe50 con 0x7ff05c105010 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.353+0000 7ff059ffb640 1 -- 172.21.15.62:0/990721767 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff044010460 con 0x7ff05c105010 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.353+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff05c1a0300 con 0x7ff05c105010 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.354+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff05c0fdd70 con 0x7ff05c105010 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.355+0000 7ff059ffb640 1 -- 172.21.15.62:0/990721767 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7ff044003e70 con 0x7ff05c105010 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.356+0000 7ff059ffb640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0480a3300 0x7ff0480a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.356+0000 7ff060d99640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0480a3300 0x7ff0480a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.356+0000 7ff059ffb640 1 -- 172.21.15.62:0/990721767 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7ff0440cfaa0 con 0x7ff05c105010 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.356+0000 7ff060d99640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0480a3300 0x7ff0480a57c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff054000c00 tx=0x7ff0540315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:20.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.360+0000 7ff059ffb640 1 -- 172.21.15.62:0/990721767 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff044020030 con 0x7ff05c105010 2024-09-15T23:15:20.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.543+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7ff05c062630 con 0x7ff0480a3300 2024-09-15T23:15:20.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.555+0000 7ff059ffb640 1 -- 172.21.15.62:0/990721767 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4154 (secure 0 0 0) 0x7ff05c062630 con 0x7ff0480a3300 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:mds.a smithi062 running (2m) 50s ago 2m 14.3M - 19.1.1-326-g8921c8ef f1455bd0219c 334aab8fbdd1 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:mds.b smithi067 running (118s) 51s ago 118s 26.5M - 19.1.1-326-g8921c8ef f1455bd0219c c2da18fa89c1 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:mds.c smithi148 running (2m) 51s ago 2m 16.9M - 19.1.1-326-g8921c8ef f1455bd0219c adb39dd76ca1 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:mds.d smithi062 running (2m) 50s ago 2m 16.5M - 19.1.1-326-g8921c8ef f1455bd0219c 46315ff11dd3 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:mds.e smithi067 running (2m) 51s ago 2m 15.9M - 19.1.1-326-g8921c8ef f1455bd0219c 0e3456702d7a 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:mds.f smithi148 running (2m) 51s ago 2m 17.3M - 19.1.1-326-g8921c8ef f1455bd0219c 7463f5940306 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:mds.g smithi062 running (2m) 50s ago 2m 13.4M - 19.1.1-326-g8921c8ef f1455bd0219c 7c00931df3a6 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:mds.h smithi067 running (2m) 51s ago 2m 19.0M - 19.1.1-326-g8921c8ef f1455bd0219c 4273c63666ae 2024-09-15T23:15:20.558 INFO:teuthology.orchestra.run.smithi062.stdout:mds.i smithi148 running (2m) 51s ago 2m 19.2M - 19.1.1-326-g8921c8ef f1455bd0219c bc6ff9a3ba66 2024-09-15T23:15:20.559 INFO:teuthology.orchestra.run.smithi062.stdout:mds.j smithi062 running (2m) 50s ago 2m 14.5M - 19.1.1-326-g8921c8ef f1455bd0219c 51f3dd19b664 2024-09-15T23:15:20.559 INFO:teuthology.orchestra.run.smithi062.stdout:mds.k smithi067 running (2m) 51s ago 2m 14.1M - 19.1.1-326-g8921c8ef f1455bd0219c b7512f6e00c8 2024-09-15T23:15:20.559 INFO:teuthology.orchestra.run.smithi062.stdout:mds.l smithi148 running (2m) 51s ago 2m 16.0M - 19.1.1-326-g8921c8ef f1455bd0219c c31cf6fdb387 2024-09-15T23:15:20.559 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi062 *:9283,8765 running (9m) 50s ago 9m 509M - 19.1.1-326-g8921c8ef f1455bd0219c 460390df780a 2024-09-15T23:15:20.559 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi067 *:8443,8765 running (7m) 51s ago 7m 444M - 19.1.1-326-g8921c8ef f1455bd0219c 73e4e6a40fa4 2024-09-15T23:15:20.559 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.z smithi148 *:8443,8765 running (7m) 51s ago 7m 442M - 19.1.1-326-g8921c8ef f1455bd0219c 39d719a09112 2024-09-15T23:15:20.559 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (9m) 50s ago 9m 49.5M 2048M 19.1.1-326-g8921c8ef f1455bd0219c 293eb3210083 2024-09-15T23:15:20.559 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi067 running (7m) 51s ago 7m 43.0M 2048M 19.1.1-326-g8921c8ef f1455bd0219c 58bf2b6c8495 2024-09-15T23:15:20.559 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi148 running (8m) 51s ago 8m 41.0M 2048M 19.1.1-326-g8921c8ef f1455bd0219c 2e8816ac8b2a 2024-09-15T23:15:20.560 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (7m) 50s ago 7m 57.0M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 169f9de9b005 2024-09-15T23:15:20.560 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi067 running (7m) 51s ago 7m 46.2M 4272M 19.1.1-326-g8921c8ef f1455bd0219c e78cdad9887a 2024-09-15T23:15:20.560 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi148 running (6m) 51s ago 6m 47.2M 4272M 19.1.1-326-g8921c8ef f1455bd0219c f9773566b554 2024-09-15T23:15:20.560 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (6m) 50s ago 6m 47.3M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 39cf981d3a4c 2024-09-15T23:15:20.560 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi067 running (5m) 51s ago 5m 53.5M 4272M 19.1.1-326-g8921c8ef f1455bd0219c ec3bf7036180 2024-09-15T23:15:20.560 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi148 running (5m) 51s ago 5m 56.7M 4272M 19.1.1-326-g8921c8ef f1455bd0219c b3aaac7e0212 2024-09-15T23:15:20.560 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi062 running (4m) 50s ago 4m 54.0M 4272M 19.1.1-326-g8921c8ef f1455bd0219c e002307370d8 2024-09-15T23:15:20.560 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi067 running (4m) 51s ago 4m 54.2M 4272M 19.1.1-326-g8921c8ef f1455bd0219c e150347fc53b 2024-09-15T23:15:20.561 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8 smithi148 running (4m) 51s ago 4m 45.0M 4272M 19.1.1-326-g8921c8ef f1455bd0219c c28d748a6a1d 2024-09-15T23:15:20.561 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9 smithi062 running (3m) 50s ago 3m 55.7M 4272M 19.1.1-326-g8921c8ef f1455bd0219c b628b8da1a7d 2024-09-15T23:15:20.561 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10 smithi067 running (3m) 51s ago 3m 54.2M 4272M 19.1.1-326-g8921c8ef f1455bd0219c de6e1146ab60 2024-09-15T23:15:20.561 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11 smithi148 running (2m) 51s ago 2m 43.4M 4272M 19.1.1-326-g8921c8ef f1455bd0219c 182856f661c0 2024-09-15T23:15:20.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0480a3300 msgr2=0x7ff0480a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:20.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0480a3300 0x7ff0480a57c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff054000c00 tx=0x7ff0540315d0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c105010 msgr2=0x7ff05c198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:20.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c105010 0x7ff05c198470 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff044005fd0 tx=0x7ff04400dde0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 shutdown_connections 2024-09-15T23:15:20.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0480a3300 0x7ff0480a57c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff05c107940 0x7ff05c19f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff05c105010 0x7ff05c198470 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 --2- 172.21.15.62:0/990721767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff05c1026e0 0x7ff05c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:20.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 >> 172.21.15.62:0/990721767 conn(0x7ff05c0fc280 msgr2=0x7ff05c103060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:20.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 shutdown_connections 2024-09-15T23:15:20.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:20.558+0000 7ff062ffd640 1 -- 172.21.15.62:0/990721767 wait complete. 2024-09-15T23:15:21.191 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:20 smithi062 ceph-mon[30949]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:21.191 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:20 smithi062 ceph-mon[30949]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:21.223 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-15T23:15:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:20 smithi067 ceph-mon[35665]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:20 smithi067 ceph-mon[35665]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:20 smithi148 ceph-mon[35610]: pgmap v340: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:20 smithi148 ceph-mon[35610]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:21.484 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:21.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.952+0000 7fa86e949640 1 -- 172.21.15.62:0/2892248136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 msgr2=0x7fa86810a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:21.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.952+0000 7fa86e949640 1 --2- 172.21.15.62:0/2892248136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 0x7fa86810a050 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa860009ca0 tx=0x7fa86002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:15:21.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.952+0000 7fa86e949640 1 -- 172.21.15.62:0/2892248136 shutdown_connections 2024-09-15T23:15:21.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.952+0000 7fa86e949640 1 --2- 172.21.15.62:0/2892248136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 0x7fa86810a050 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:21.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 --2- 172.21.15.62:0/2892248136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa868102e00 0x7fa868103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:21.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 --2- 172.21.15.62:0/2892248136 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa868101c00 0x7fa868102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:21.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 -- 172.21.15.62:0/2892248136 >> 172.21.15.62:0/2892248136 conn(0x7fa8680fd390 msgr2=0x7fa8680ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 -- 172.21.15.62:0/2892248136 shutdown_connections 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 -- 172.21.15.62:0/2892248136 wait complete. 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 Processor -- start 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 -- start start 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa868101c00 0x7fa868197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa868102e00 0x7fa868198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 0x7fa86819f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810d880 con 0x7fa8681037c0 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810d700 con 0x7fa868102e00 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.953+0000 7fa86e949640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa86810da00 con 0x7fa868101c00 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa86cee6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 0x7fa86819f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa86cee6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 0x7fa86819f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54202/0 (socket says 172.21.15.62:54202) 2024-09-15T23:15:21.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa86cee6640 1 -- 172.21.15.62:0/1246784181 learned_addr learned my addr 172.21.15.62:0/1246784181 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa867fff640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa868101c00 0x7fa868197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa8677fe640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa868102e00 0x7fa868198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa86cee6640 1 -- 172.21.15.62:0/1246784181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa868101c00 msgr2=0x7fa868197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa86cee6640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa868101c00 0x7fa868197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa86cee6640 1 -- 172.21.15.62:0/1246784181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa868102e00 msgr2=0x7fa868198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa86cee6640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa868102e00 0x7fa868198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa86cee6640 1 -- 172.21.15.62:0/1246784181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86819fa20 con 0x7fa8681037c0 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.954+0000 7fa86cee6640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 0x7fa86819f3b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa860005e30 tx=0x7fa860031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.955+0000 7fa8657fa640 1 -- 172.21.15.62:0/1246784181 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860003c30 con 0x7fa8681037c0 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.955+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86819fcb0 con 0x7fa8681037c0 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.955+0000 7fa8657fa640 1 -- 172.21.15.62:0/1246784181 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa860003dd0 con 0x7fa8681037c0 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.955+0000 7fa8657fa640 1 -- 172.21.15.62:0/1246784181 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86002fde0 con 0x7fa8681037c0 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.955+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8681a0190 con 0x7fa8681037c0 2024-09-15T23:15:21.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.956+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa868108f30 con 0x7fa8681037c0 2024-09-15T23:15:21.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.957+0000 7fa8657fa640 1 -- 172.21.15.62:0/1246784181 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa860004050 con 0x7fa8681037c0 2024-09-15T23:15:21.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.958+0000 7fa8657fa640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8500a3300 0x7fa8500a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:21.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.958+0000 7fa867fff640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8500a3300 0x7fa8500a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:21.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.958+0000 7fa8657fa640 1 -- 172.21.15.62:0/1246784181 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fa8600f1170 con 0x7fa8681037c0 2024-09-15T23:15:21.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.958+0000 7fa867fff640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8500a3300 0x7fa8500a57c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa858005fd0 tx=0x7fa85800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:21.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:21.963+0000 7fa8657fa640 1 -- 172.21.15.62:0/1246784181 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa8600bdb00 con 0x7fa8681037c0 2024-09-15T23:15:22.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.152+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fa868062630 con 0x7fa8500a3300 2024-09-15T23:15:22.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.160+0000 7fa8657fa640 1 -- 172.21.15.62:0/1246784181 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fa868062630 con 0x7fa8500a3300 2024-09-15T23:15:22.162 INFO:teuthology.orchestra.run.smithi062.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-15T23:15:22.162 INFO:teuthology.orchestra.run.smithi062.stdout:mds.all 12/12 53s ago 2m smithi062=a;smithi062=d;smithi062=g;smithi062=j;smithi067=b;smithi067=e;smithi067=h;smithi067=k;smithi148=c;smithi148=f;smithi148=i;smithi148=l;count:12 2024-09-15T23:15:22.162 INFO:teuthology.orchestra.run.smithi062.stdout:mgr 3/3 53s ago 7m smithi062=x;smithi067=y;smithi148=z;count:3 2024-09-15T23:15:22.162 INFO:teuthology.orchestra.run.smithi062.stdout:mon 3/3 53s ago 8m smithi062:172.21.15.62=a;smithi067:172.21.15.67=b;smithi148:172.21.15.148=c;count:3 2024-09-15T23:15:22.162 INFO:teuthology.orchestra.run.smithi062.stdout:osd 12 53s ago - 2024-09-15T23:15:22.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8500a3300 msgr2=0x7fa8500a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:22.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8500a3300 0x7fa8500a57c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa858005fd0 tx=0x7fa85800b390 comp rx=0 tx=0).stop 2024-09-15T23:15:22.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 msgr2=0x7fa86819f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:22.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 0x7fa86819f3b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa860005e30 tx=0x7fa860031ee0 comp rx=0 tx=0).stop 2024-09-15T23:15:22.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 shutdown_connections 2024-09-15T23:15:22.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa8500a3300 0x7fa8500a57c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:22.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8681037c0 0x7fa86819f3b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:22.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa868102e00 0x7fa868198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:22.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 --2- 172.21.15.62:0/1246784181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa868101c00 0x7fa868197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:22.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 >> 172.21.15.62:0/1246784181 conn(0x7fa8680fd390 msgr2=0x7fa868106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:22.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 shutdown_connections 2024-09-15T23:15:22.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:22.162+0000 7fa86e949640 1 -- 172.21.15.62:0/1246784181 wait complete. 2024-09-15T23:15:22.917 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-15T23:15:23.177 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:23.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:22 smithi062 ceph-mon[30949]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:23.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:22 smithi062 ceph-mon[30949]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:23.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:22 smithi067 ceph-mon[35665]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:23.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:22 smithi067 ceph-mon[35665]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:23.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:22 smithi148 ceph-mon[35610]: pgmap v341: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:23.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:22 smithi148 ceph-mon[35610]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:23.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 -- 172.21.15.62:0/2107203963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 msgr2=0x7fd2f8108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 --2- 172.21.15.62:0/2107203963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 0x7fd2f8108f20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009ca0 tx=0x7fd2f002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 -- 172.21.15.62:0/2107203963 shutdown_connections 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 --2- 172.21.15.62:0/2107203963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 0x7fd2f8108f20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 --2- 172.21.15.62:0/2107203963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f8101cd0 0x7fd2f8102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 --2- 172.21.15.62:0/2107203963 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f8100ad0 0x7fd2f8100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 -- 172.21.15.62:0/2107203963 >> 172.21.15.62:0/2107203963 conn(0x7fd2f80fc280 msgr2=0x7fd2f80fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 -- 172.21.15.62:0/2107203963 shutdown_connections 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 -- 172.21.15.62:0/2107203963 wait complete. 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 Processor -- start 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.737+0000 7fd300207640 1 -- start start 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd300207640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f8100ad0 0x7fd2f8197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd300207640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f8101cd0 0x7fd2f8198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:23.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd300207640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 0x7fd2f819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd300207640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f810d8e0 con 0x7fd2f8102690 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd300207640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f810d760 con 0x7fd2f8101cd0 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd300207640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f810da60 con 0x7fd2f8100ad0 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fe7a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 0x7fd2f819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fe7a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 0x7fd2f819f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56876/0 (socket says 172.21.15.62:56876) 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fe7a4640 1 -- 172.21.15.62:0/271549120 learned_addr learned my addr 172.21.15.62:0/271549120 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fdfa3640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f8100ad0 0x7fd2f8197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fd7a2640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f8101cd0 0x7fd2f8198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fe7a4640 1 -- 172.21.15.62:0/271549120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f8100ad0 msgr2=0x7fd2f8197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fe7a4640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f8100ad0 0x7fd2f8197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fe7a4640 1 -- 172.21.15.62:0/271549120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f8101cd0 msgr2=0x7fd2f8198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fe7a4640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f8101cd0 0x7fd2f8198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fe7a4640 1 -- 172.21.15.62:0/271549120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f819fb60 con 0x7fd2f8102690 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2fe7a4640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 0x7fd2f819f4f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0005e30 tx=0x7fd2f0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.738+0000 7fd2eeffd640 1 -- 172.21.15.62:0/271549120 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0003b70 con 0x7fd2f8102690 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.739+0000 7fd2eeffd640 1 -- 172.21.15.62:0/271549120 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2f0003d10 con 0x7fd2f8102690 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.739+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f819fdf0 con 0x7fd2f8102690 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.739+0000 7fd2eeffd640 1 -- 172.21.15.62:0/271549120 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f002fd30 con 0x7fd2f8102690 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.739+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f81a0320 con 0x7fd2f8102690 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.739+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2f8107ea0 con 0x7fd2f8102690 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.741+0000 7fd2eeffd640 1 -- 172.21.15.62:0/271549120 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd2f0004050 con 0x7fd2f8102690 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.741+0000 7fd2eeffd640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.741+0000 7fd2fdfa3640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.742+0000 7fd2eeffd640 1 -- 172.21.15.62:0/271549120 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fd2f00f11c0 con 0x7fd2f8102690 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.742+0000 7fd2fdfa3640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4005fd0 tx=0x7fd2f400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:23.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.745+0000 7fd2eeffd640 1 -- 172.21.15.62:0/271549120 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd2f00bdd00 con 0x7fd2f8102690 2024-09-15T23:15:23.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.926+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fd2f8062630 con 0x7fd2e40a3300 2024-09-15T23:15:23.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.927+0000 7fd2eeffd640 1 -- 172.21.15.62:0/271549120 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fd2f8062630 con 0x7fd2e40a3300 2024-09-15T23:15:23.928 INFO:teuthology.orchestra.run.smithi062.stdout:HOST ADDR LABELS STATUS 2024-09-15T23:15:23.929 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062 172.21.15.62 2024-09-15T23:15:23.929 INFO:teuthology.orchestra.run.smithi062.stdout:smithi067 172.21.15.67 2024-09-15T23:15:23.929 INFO:teuthology.orchestra.run.smithi062.stdout:smithi148 172.21.15.148 2024-09-15T23:15:23.929 INFO:teuthology.orchestra.run.smithi062.stdout:3 hosts in cluster 2024-09-15T23:15:23.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.929+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 msgr2=0x7fd2e40a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:23.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.929+0000 7fd300207640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4005fd0 tx=0x7fd2f400b390 comp rx=0 tx=0).stop 2024-09-15T23:15:23.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 msgr2=0x7fd2f819f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:23.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 0x7fd2f819f4f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0005e30 tx=0x7fd2f0031ee0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 shutdown_connections 2024-09-15T23:15:23.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2f8102690 0x7fd2f819f4f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2f8101cd0 0x7fd2f8198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 --2- 172.21.15.62:0/271549120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2f8100ad0 0x7fd2f8197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:23.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 >> 172.21.15.62:0/271549120 conn(0x7fd2f80fc280 msgr2=0x7fd2f8105070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:23.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 shutdown_connections 2024-09-15T23:15:23.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:23.930+0000 7fd300207640 1 -- 172.21.15.62:0/271549120 wait complete. 2024-09-15T23:15:24.615 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-15T23:15:24.873 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:25.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:24 smithi062 ceph-mon[30949]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:25.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:24 smithi062 ceph-mon[30949]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:25.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:24 smithi067 ceph-mon[35665]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:25.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:24 smithi067 ceph-mon[35665]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:25.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:24 smithi148 ceph-mon[35610]: pgmap v342: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:25.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:24 smithi148 ceph-mon[35610]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:25.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.359+0000 7f098d3c4640 1 -- 172.21.15.62:0/2959068182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988101d10 msgr2=0x7f0988102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.359+0000 7f098d3c4640 1 --2- 172.21.15.62:0/2959068182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988101d10 0x7f0988102190 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f097c009900 tx=0x7f097c02a410 comp rx=0 tx=0).stop 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.359+0000 7f098d3c4640 1 -- 172.21.15.62:0/2959068182 shutdown_connections 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.359+0000 7f098d3c4640 1 --2- 172.21.15.62:0/2959068182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09881026d0 0x7f0988108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.359+0000 7f098d3c4640 1 --2- 172.21.15.62:0/2959068182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988101d10 0x7f0988102190 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.359+0000 7f098d3c4640 1 --2- 172.21.15.62:0/2959068182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0988100b10 0x7f0988100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.359+0000 7f098d3c4640 1 -- 172.21.15.62:0/2959068182 >> 172.21.15.62:0/2959068182 conn(0x7f09880fc2a0 msgr2=0x7f09880fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.359+0000 7f098d3c4640 1 -- 172.21.15.62:0/2959068182 shutdown_connections 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.359+0000 7f098d3c4640 1 -- 172.21.15.62:0/2959068182 wait complete. 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.360+0000 7f098d3c4640 1 Processor -- start 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.360+0000 7f098d3c4640 1 -- start start 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.360+0000 7f098d3c4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988100b10 0x7f0988104aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.360+0000 7f098d3c4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988101d10 0x7f09881030f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:25.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.360+0000 7f098d3c4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09881026d0 0x7f0988103630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.360+0000 7f098d3c4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f098810c770 con 0x7f0988100b10 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.360+0000 7f098d3c4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f098810c5f0 con 0x7f0988101d10 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.360+0000 7f098d3c4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f098810c8f0 con 0x7f09881026d0 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.360+0000 7f0986ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988100b10 0x7f0988104aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f0986ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988100b10 0x7f0988104aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56890/0 (socket says 172.21.15.62:56890) 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f0986ffd640 1 -- 172.21.15.62:0/3700332060 learned_addr learned my addr 172.21.15.62:0/3700332060 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f09867fc640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988101d10 0x7f09881030f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f09877fe640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09881026d0 0x7f0988103630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f0986ffd640 1 -- 172.21.15.62:0/3700332060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09881026d0 msgr2=0x7f0988103630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f0986ffd640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09881026d0 0x7f0988103630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f0986ffd640 1 -- 172.21.15.62:0/3700332060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988101d10 msgr2=0x7f09881030f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:25.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f0986ffd640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988101d10 0x7f09881030f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f0986ffd640 1 -- 172.21.15.62:0/3700332060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0988103ef0 con 0x7f0988100b10 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f0986ffd640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988100b10 0x7f0988104aa0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f097800c970 tx=0x7f097800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f096ffff640 1 -- 172.21.15.62:0/3700332060 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0978007c10 con 0x7f0988100b10 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f096ffff640 1 -- 172.21.15.62:0/3700332060 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0978007db0 con 0x7f0988100b10 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f096ffff640 1 -- 172.21.15.62:0/3700332060 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097800f940 con 0x7f0988100b10 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09881a2080 con 0x7f0988100b10 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.361+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09881a25e0 con 0x7f0988100b10 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.362+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09881a2870 con 0x7f0988100b10 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.363+0000 7f096ffff640 1 -- 172.21.15.62:0/3700332060 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f097800fae0 con 0x7f0988100b10 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.364+0000 7f096ffff640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09700a3350 0x7f09700a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.364+0000 7f09867fc640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09700a3350 0x7f09700a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.364+0000 7f096ffff640 1 -- 172.21.15.62:0/3700332060 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f09780d01f0 con 0x7f0988100b10 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.365+0000 7f09867fc640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09700a3350 0x7f09700a5810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f097c000c00 tx=0x7f097c031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.368+0000 7f096ffff640 1 -- 172.21.15.62:0/3700332060 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f097809cb00 con 0x7f0988100b10 2024-09-15T23:15:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.553+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f098806aef0 con 0x7f09700a3350 2024-09-15T23:15:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.556+0000 7f096ffff640 1 -- 172.21.15.62:0/3700332060 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f098806aef0 con 0x7f09700a3350 2024-09-15T23:15:25.557 INFO:teuthology.orchestra.run.smithi062.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-15T23:15:25.557 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440094400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-15T23:15:25.557 INFO:teuthology.orchestra.run.smithi062.stdout:smithi067 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440042400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-09-15T23:15:25.557 INFO:teuthology.orchestra.run.smithi062.stdout:smithi148 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233005G400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-15T23:15:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09700a3350 msgr2=0x7f09700a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09700a3350 0x7f09700a5810 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f097c000c00 tx=0x7f097c031400 comp rx=0 tx=0).stop 2024-09-15T23:15:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988100b10 msgr2=0x7f0988104aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988100b10 0x7f0988104aa0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f097800c970 tx=0x7f097800ce30 comp rx=0 tx=0).stop 2024-09-15T23:15:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 shutdown_connections 2024-09-15T23:15:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09700a3350 0x7f09700a5810 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09881026d0 0x7f0988103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0988101d10 0x7f09881030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 --2- 172.21.15.62:0/3700332060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0988100b10 0x7f0988104aa0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:25.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 >> 172.21.15.62:0/3700332060 conn(0x7f09880fc2a0 msgr2=0x7f098806a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:25.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 shutdown_connections 2024-09-15T23:15:25.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:25.558+0000 7f098d3c4640 1 -- 172.21.15.62:0/3700332060 wait complete. 2024-09-15T23:15:26.196 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:25 smithi062 ceph-mon[30949]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:26.224 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-15T23:15:26.232 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T23:15:26.232 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- bash -c 'ceph fs dump' 2024-09-15T23:15:26.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:25 smithi067 ceph-mon[35665]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:26.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:25 smithi148 ceph-mon[35610]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-15T23:15:26.489 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:26.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.973+0000 7f3993dca640 1 -- 172.21.15.62:0/1411246264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c1026a0 msgr2=0x7f398c104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:26.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.973+0000 7f3993dca640 1 --2- 172.21.15.62:0/1411246264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c1026a0 0x7f398c104a90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3988009900 tx=0x7f398802a410 comp rx=0 tx=0).stop 2024-09-15T23:15:26.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.973+0000 7f3993dca640 1 -- 172.21.15.62:0/1411246264 shutdown_connections 2024-09-15T23:15:26.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.973+0000 7f3993dca640 1 --2- 172.21.15.62:0/1411246264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398c107900 0x7f398c109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.973+0000 7f3993dca640 1 --2- 172.21.15.62:0/1411246264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f398c104fd0 0x7f398c1073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.973+0000 7f3993dca640 1 --2- 172.21.15.62:0/1411246264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c1026a0 0x7f398c104a90 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.973+0000 7f3993dca640 1 -- 172.21.15.62:0/1411246264 >> 172.21.15.62:0/1411246264 conn(0x7f398c0fc2a0 msgr2=0x7f398c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.973+0000 7f3993dca640 1 -- 172.21.15.62:0/1411246264 shutdown_connections 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.973+0000 7f3993dca640 1 -- 172.21.15.62:0/1411246264 wait complete. 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3993dca640 1 Processor -- start 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3993dca640 1 -- start start 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3993dca640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f398c1026a0 0x7f398c102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3993dca640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c104fd0 0x7f398c100650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3993dca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398c107900 0x7f398c100b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3993dca640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f398c10d8b0 con 0x7f398c104fd0 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3993dca640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f398c10d730 con 0x7f398c107900 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3993dca640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f398c10da30 con 0x7f398c1026a0 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3991365640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c104fd0 0x7f398c100650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3991365640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c104fd0 0x7f398c100650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56914/0 (socket says 172.21.15.62:56914) 2024-09-15T23:15:26.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3991365640 1 -- 172.21.15.62:0/1587937372 learned_addr learned my addr 172.21.15.62:0/1587937372 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3992367640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398c107900 0x7f398c100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.974+0000 7f3991b66640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f398c1026a0 0x7f398c102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3991365640 1 -- 172.21.15.62:0/1587937372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f398c1026a0 msgr2=0x7f398c102000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3991365640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f398c1026a0 0x7f398c102000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3991365640 1 -- 172.21.15.62:0/1587937372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398c107900 msgr2=0x7f398c100b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3991365640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398c107900 0x7f398c100b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3991365640 1 -- 172.21.15.62:0/1587937372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c101450 con 0x7f398c104fd0 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3992367640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398c107900 0x7f398c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3991b66640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f398c1026a0 0x7f398c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3991365640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c104fd0 0x7f398c100650 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f397c00d770 tx=0x7f397c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3982ffd640 1 -- 172.21.15.62:0/1587937372 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c017070 con 0x7f398c104fd0 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f398c1a20c0 con 0x7f398c104fd0 2024-09-15T23:15:26.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3982ffd640 1 -- 172.21.15.62:0/1587937372 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f397c009c30 con 0x7f398c104fd0 2024-09-15T23:15:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3982ffd640 1 -- 172.21.15.62:0/1587937372 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c00f620 con 0x7f398c104fd0 2024-09-15T23:15:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.975+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f398c1a25d0 con 0x7f398c104fd0 2024-09-15T23:15:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.976+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f398c0fdd90 con 0x7f398c104fd0 2024-09-15T23:15:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.977+0000 7f3982ffd640 1 -- 172.21.15.62:0/1587937372 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f397c01a070 con 0x7f398c104fd0 2024-09-15T23:15:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.978+0000 7f3982ffd640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39780a32d0 0x7f39780a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.978+0000 7f3991b66640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39780a32d0 0x7f39780a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.978+0000 7f3982ffd640 1 -- 172.21.15.62:0/1587937372 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f397c0cf990 con 0x7f398c104fd0 2024-09-15T23:15:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.979+0000 7f3991b66640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39780a32d0 0x7f39780a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3988000c00 tx=0x7f39880095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:26.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:26.982+0000 7f3982ffd640 1 -- 172.21.15.62:0/1587937372 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f397c009de0 con 0x7f398c104fd0 2024-09-15T23:15:27.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.192+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f398c02d970 con 0x7f398c104fd0 2024-09-15T23:15:27.194 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:26 smithi062 ceph-mon[30949]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:27.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.193+0000 7f3982ffd640 1 -- 172.21.15.62:0/1587937372 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3082 (secure 0 0 0) 0x7f397c09c210 con 0x7f398c104fd0 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:e24 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:btime 2024-09-15T23:14:10:623582+0000 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:legacy client fscid: 1 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem 'cephfs' (1) 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:fs_name cephfs 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 24 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-09-15T23:15:27.195 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-09-15T23:13:33.200855+0000 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-09-15T23:14:10.188556+0000 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:tableserver 0 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:root 0 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:session_timeout 300 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:session_autoclose 300 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:max_file_size 1099511627776 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:max_xattr_size 65536 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:required_client_features {} 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure 0 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:last_failure_osd_epoch 0 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:max_mds 3 2024-09-15T23:15:27.196 INFO:teuthology.orchestra.run.smithi062.stdout:in 0,1,2 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:up {0=24497,1=24470,2=24466} 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:failed 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:damaged 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:stopped 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:data_pools [3,4] 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:metadata_pool 2 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:inline_data disabled 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:balancer 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:bal_rank_mask -1 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:standby_count_wanted 0 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:qdb_cluster leader: 24497 members: 24497,24470,24466 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b{0:24497} state up:active seq 4 addr [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.c{0:24454} state up:standby-replay seq 1 addr [v2:172.21.15.148:6836/870913016,v1:172.21.15.148:6837/870913016] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.197 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.h{1:24470} state up:active seq 16 addr [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.e{1:24449} state up:standby-replay seq 1 addr [v2:172.21.15.67:6834/3132447542,v1:172.21.15.67:6835/3132447542] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.i{2:24466} state up:active seq 15 addr [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.f{2:24442} state up:standby-replay seq 1 addr [v2:172.21.15.148:6834/1239577031,v1:172.21.15.148:6835/1239577031] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:Standby daemons: 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.j{-1:14538} state up:standby seq 1 addr [v2:172.21.15.62:6834/3727539953,v1:172.21.15.62:6835/3727539953] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.d{-1:14550} state up:standby seq 1 addr [v2:172.21.15.62:6836/862835307,v1:172.21.15.62:6837/862835307] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.g{-1:14604} state up:standby seq 1 addr [v2:172.21.15.62:6838/2389718073,v1:172.21.15.62:6839/2389718073] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a{-1:14616} state up:standby seq 1 addr [v2:172.21.15.62:6840/3484718619,v1:172.21.15.62:6841/3484718619] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.l{-1:24418} state up:standby seq 1 addr [v2:172.21.15.148:6832/2970459675,v1:172.21.15.148:6833/2970459675] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.198 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.k{-1:24425} state up:standby seq 1 addr [v2:172.21.15.67:6832/190864103,v1:172.21.15.67:6833/190864103] compat {c=[1],r=[1],i=[1fff]}] 2024-09-15T23:15:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 24 2024-09-15T23:15:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39780a32d0 msgr2=0x7f39780a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39780a32d0 0x7f39780a5790 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3988000c00 tx=0x7f39880095f0 comp rx=0 tx=0).stop 2024-09-15T23:15:27.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c104fd0 msgr2=0x7f398c100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c104fd0 0x7f398c100650 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f397c00d770 tx=0x7f397c00dc30 comp rx=0 tx=0).stop 2024-09-15T23:15:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 shutdown_connections 2024-09-15T23:15:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39780a32d0 0x7f39780a5790 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f398c107900 0x7f398c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f398c104fd0 0x7f398c100650 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 --2- 172.21.15.62:0/1587937372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f398c1026a0 0x7f398c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 >> 172.21.15.62:0/1587937372 conn(0x7f398c0fc2a0 msgr2=0x7f398c103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 shutdown_connections 2024-09-15T23:15:27.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:27.196+0000 7f3993dca640 1 -- 172.21.15.62:0/1587937372 wait complete. 2024-09-15T23:15:27.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:26 smithi067 ceph-mon[35665]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:27.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:26 smithi148 ceph-mon[35610]: pgmap v343: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:27.950 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- bash -c 'ceph osd dump' 2024-09-15T23:15:28.209 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:28.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:27 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1587937372' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-15T23:15:28.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:27 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1587937372' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-15T23:15:28.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:27 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1587937372' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.673+0000 7fea24b17640 1 -- 172.21.15.62:0/2450112960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea20102e00 msgr2=0x7fea20103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.673+0000 7fea24b17640 1 --2- 172.21.15.62:0/2450112960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea20102e00 0x7fea20103280 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fea14009900 tx=0x7fea1402a410 comp rx=0 tx=0).stop 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.673+0000 7fea24b17640 1 -- 172.21.15.62:0/2450112960 shutdown_connections 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.673+0000 7fea24b17640 1 --2- 172.21.15.62:0/2450112960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea201037c0 0x7fea2010a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.673+0000 7fea24b17640 1 --2- 172.21.15.62:0/2450112960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea20102e00 0x7fea20103280 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.673+0000 7fea24b17640 1 --2- 172.21.15.62:0/2450112960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea20101c00 0x7fea20102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.673+0000 7fea24b17640 1 -- 172.21.15.62:0/2450112960 >> 172.21.15.62:0/2450112960 conn(0x7fea200fd390 msgr2=0x7fea200ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.673+0000 7fea24b17640 1 -- 172.21.15.62:0/2450112960 shutdown_connections 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.673+0000 7fea24b17640 1 -- 172.21.15.62:0/2450112960 wait complete. 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea24b17640 1 Processor -- start 2024-09-15T23:15:28.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea24b17640 1 -- start start 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea24b17640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea20101c00 0x7fea20197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea24b17640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea20102e00 0x7fea20198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea24b17640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea201037c0 0x7fea2019f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea24b17640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fea2010d880 con 0x7fea201037c0 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea24b17640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fea2010d700 con 0x7fea20102e00 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea24b17640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fea2010da00 con 0x7fea20101c00 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea1ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea201037c0 0x7fea2019f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea1ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea201037c0 0x7fea2019f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56932/0 (socket says 172.21.15.62:56932) 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea1ed9d640 1 -- 172.21.15.62:0/2718261833 learned_addr learned my addr 172.21.15.62:0/2718261833 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea1dd9b640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea20102e00 0x7fea20198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.674+0000 7fea1e59c640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea20101c00 0x7fea20197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea1ed9d640 1 -- 172.21.15.62:0/2718261833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea20101c00 msgr2=0x7fea20197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:28.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea1ed9d640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea20101c00 0x7fea20197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea1ed9d640 1 -- 172.21.15.62:0/2718261833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea20102e00 msgr2=0x7fea20198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:28.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea1ed9d640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea20102e00 0x7fea20198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea1ed9d640 1 -- 172.21.15.62:0/2718261833 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea2019faf0 con 0x7fea201037c0 2024-09-15T23:15:28.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea1ed9d640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea201037c0 0x7fea2019f480 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fea1800eaa0 tx=0x7fea1800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:28.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea0f7fe640 1 -- 172.21.15.62:0/2718261833 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea1800cdb0 con 0x7fea201037c0 2024-09-15T23:15:28.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea0f7fe640 1 -- 172.21.15.62:0/2718261833 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea18005020 con 0x7fea201037c0 2024-09-15T23:15:28.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea2019fde0 con 0x7fea201037c0 2024-09-15T23:15:28.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea0f7fe640 1 -- 172.21.15.62:0/2718261833 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea180116b0 con 0x7fea201037c0 2024-09-15T23:15:28.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.675+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea201a0240 con 0x7fea201037c0 2024-09-15T23:15:28.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.676+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea20108fa0 con 0x7fea201037c0 2024-09-15T23:15:28.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.677+0000 7fea0f7fe640 1 -- 172.21.15.62:0/2718261833 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fea18005560 con 0x7fea201037c0 2024-09-15T23:15:28.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.678+0000 7fea0f7fe640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fea080a3300 0x7fea080a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:28.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.678+0000 7fea1e59c640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fea080a3300 0x7fea080a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:28.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.679+0000 7fea0f7fe640 1 -- 172.21.15.62:0/2718261833 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fea180cff60 con 0x7fea201037c0 2024-09-15T23:15:28.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.679+0000 7fea1e59c640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fea080a3300 0x7fea080a57c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea10005fd0 tx=0x7fea1000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:28.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.683+0000 7fea0f7fe640 1 -- 172.21.15.62:0/2718261833 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fea18016030 con 0x7fea201037c0 2024-09-15T23:15:28.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.860+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fea2002d970 con 0x7fea201037c0 2024-09-15T23:15:28.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.861+0000 7fea0f7fe640 1 -- 172.21.15.62:0/2718261833 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5566 (secure 0 0 0) 0x7fea1809c190 con 0x7fea201037c0 2024-09-15T23:15:28.862 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 90 2024-09-15T23:15:28.862 INFO:teuthology.orchestra.run.smithi062.stdout:fsid ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:15:28.862 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-09-15T23:05:50.747034+0000 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:modified 2024-09-15T23:14:41.627161+0000 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:crush_version 27 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:full_ratio 0.95 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:backfillfull_ratio 0.9 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:nearfull_ratio 0.85 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:require_min_compat_client luminous 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:min_compat_client luminous 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:require_osd_release squid 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:stretch_mode_enabled false 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-09-15T23:15:28.863 INFO:teuthology.orchestra.run.smithi062.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 88 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:max_osd 12 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6802/2006762506,v1:172.21.15.62:6803/2006762506] [v2:172.21.15.62:6804/2006762506,v1:172.21.15.62:6805/2006762506] exists,up f2a0a912-0888-41cb-a20e-745c2c97e6df 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 up in weight 1 up_from 13 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6800/1021530759,v1:172.21.15.67:6801/1021530759] [v2:172.21.15.67:6802/1021530759,v1:172.21.15.67:6803/1021530759] exists,up ebfe65d3-c394-48c5-8063-078760acf36c 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6800/1833942254,v1:172.21.15.148:6801/1833942254] [v2:172.21.15.148:6802/1833942254,v1:172.21.15.148:6803/1833942254] exists,up 51482605-4595-4d0b-b982-8613d1e73969 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6810/2959993284,v1:172.21.15.62:6811/2959993284] [v2:172.21.15.62:6812/2959993284,v1:172.21.15.62:6813/2959993284] exists,up 7cbd02ce-8ea0-479b-812d-5540bfd4fbd0 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 up in weight 1 up_from 32 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6808/3506283519,v1:172.21.15.67:6809/3506283519] [v2:172.21.15.67:6810/3506283519,v1:172.21.15.67:6811/3506283519] exists,up f6441578-9a31-46f2-a84e-0bfa41cd1c49 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 up in weight 1 up_from 37 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6808/1859303200,v1:172.21.15.148:6809/1859303200] [v2:172.21.15.148:6810/1859303200,v1:172.21.15.148:6811/1859303200] exists,up 5d4f2bbc-07fb-40de-88f0-ecf68cdb2579 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 up in weight 1 up_from 45 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6818/2615321035,v1:172.21.15.62:6819/2615321035] [v2:172.21.15.62:6820/2615321035,v1:172.21.15.62:6821/2615321035] exists,up 75629f41-6236-4d44-82fd-0f1d5e0df841 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 up in weight 1 up_from 50 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6816/2000628718,v1:172.21.15.67:6817/2000628718] [v2:172.21.15.67:6818/2000628718,v1:172.21.15.67:6819/2000628718] exists,up f605b7f7-4cee-43c8-aae4-f197c5664f49 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.8 up in weight 1 up_from 57 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6816/381356822,v1:172.21.15.148:6817/381356822] [v2:172.21.15.148:6818/381356822,v1:172.21.15.148:6819/381356822] exists,up 8a81c2c5-9444-460b-b7a4-403598e61ca3 2024-09-15T23:15:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:osd.9 up in weight 1 up_from 64 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6826/3552755977,v1:172.21.15.62:6827/3552755977] [v2:172.21.15.62:6828/3552755977,v1:172.21.15.62:6829/3552755977] exists,up 2220ce09-96e7-4fcf-af2a-2c7fa4081679 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:osd.10 up in weight 1 up_from 69 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6824/3693654334,v1:172.21.15.67:6825/3693654334] [v2:172.21.15.67:6826/3693654334,v1:172.21.15.67:6827/3693654334] exists,up 0bb9d76c-f64e-4792-b7c4-9e873a5837c5 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:osd.11 up in weight 1 up_from 76 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.148:6824/759280989,v1:172.21.15.148:6825/759280989] [v2:172.21.15.148:6826/759280989,v1:172.21.15.148:6827/759280989] exists,up c08c325c-1728-416e-893e-35954d5d8cd7 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:pg_upmap_items 2.b [2,1] 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:pg_upmap_items 2.10 [8,11] 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:pg_upmap_items 2.1d [2,4,8,11] 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:pg_upmap_items 3.6 [4,6] 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:pg_upmap_items 3.24 [4,9] 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:pg_upmap_items 4.2 [9,1] 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:pg_upmap_items 4.a [5,10] 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:pg_upmap_items 4.b [9,8] 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:pg_upmap_items 4.13 [9,8] 2024-09-15T23:15:28.865 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:0/1303697831 expires 2024-09-16T23:06:39.607848+0000 2024-09-15T23:15:28.866 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:6800/2614061737 expires 2024-09-16T23:06:39.607848+0000 2024-09-15T23:15:28.866 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:0/574363632 expires 2024-09-16T23:06:15.815154+0000 2024-09-15T23:15:28.866 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:6801/1277095060 expires 2024-09-16T23:06:15.815154+0000 2024-09-15T23:15:28.866 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:6801/2614061737 expires 2024-09-16T23:06:39.607848+0000 2024-09-15T23:15:28.866 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:6800/1277095060 expires 2024-09-16T23:06:15.815154+0000 2024-09-15T23:15:28.866 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:0/50968825 expires 2024-09-16T23:06:39.607848+0000 2024-09-15T23:15:28.866 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:0/2637169275 expires 2024-09-16T23:06:39.607848+0000 2024-09-15T23:15:28.866 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:0/3526091028 expires 2024-09-16T23:06:15.815154+0000 2024-09-15T23:15:28.866 INFO:teuthology.orchestra.run.smithi062.stdout:blocklist 172.21.15.62:0/3293859590 expires 2024-09-16T23:06:15.815154+0000 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fea080a3300 msgr2=0x7fea080a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fea080a3300 0x7fea080a57c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fea10005fd0 tx=0x7fea1000b390 comp rx=0 tx=0).stop 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea201037c0 msgr2=0x7fea2019f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea201037c0 0x7fea2019f480 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fea1800eaa0 tx=0x7fea1800ef60 comp rx=0 tx=0).stop 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 shutdown_connections 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fea080a3300 0x7fea080a57c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea201037c0 0x7fea2019f480 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea20102e00 0x7fea20198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 --2- 172.21.15.62:0/2718261833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fea20101c00 0x7fea20197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 >> 172.21.15.62:0/2718261833 conn(0x7fea200fd390 msgr2=0x7fea20106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:28.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 shutdown_connections 2024-09-15T23:15:28.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:28.864+0000 7fea24b17640 1 -- 172.21.15.62:0/2718261833 wait complete. 2024-09-15T23:15:29.179 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:28 smithi062 ceph-mon[30949]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:29.180 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:28 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2718261833' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-15T23:15:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:28 smithi067 ceph-mon[35665]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:28 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2718261833' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-15T23:15:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:28 smithi148 ceph-mon[35610]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:28 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2718261833' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-15T23:15:29.739 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-15T23:15:29.750 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:15:30.011 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:30.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.481+0000 7f168c8e6640 1 -- 172.21.15.62:0/3842667366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688106f10 msgr2=0x7f168810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.481+0000 7f168c8e6640 1 --2- 172.21.15.62:0/3842667366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688106f10 0x7f168810ea90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1680009ca0 tx=0x7f168002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.481+0000 7f168c8e6640 1 -- 172.21.15.62:0/3842667366 shutdown_connections 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.481+0000 7f168c8e6640 1 --2- 172.21.15.62:0/3842667366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688106f10 0x7f168810ea90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.481+0000 7f168c8e6640 1 --2- 172.21.15.62:0/3842667366 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1688102790 0x7f16881067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.481+0000 7f168c8e6640 1 --2- 172.21.15.62:0/3842667366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1688101de0 0x7f16881021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.481+0000 7f168c8e6640 1 -- 172.21.15.62:0/3842667366 >> 172.21.15.62:0/3842667366 conn(0x7f16880fd8c0 msgr2=0x7f16880ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.481+0000 7f168c8e6640 1 -- 172.21.15.62:0/3842667366 shutdown_connections 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.481+0000 7f168c8e6640 1 -- 172.21.15.62:0/3842667366 wait complete. 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168c8e6640 1 Processor -- start 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168c8e6640 1 -- start start 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168c8e6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688101de0 0x7f16881983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:30.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168c8e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1688102790 0x7f16881988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168c8e6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1688106f10 0x7f168819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168c8e6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1688111f70 con 0x7f1688101de0 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168c8e6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1688111df0 con 0x7f1688102790 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168c8e6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f16881120f0 con 0x7f1688106f10 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688101de0 0x7f16881983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688101de0 0x7f16881983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56952/0 (socket says 172.21.15.62:56952) 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f168659c640 1 -- 172.21.15.62:0/2378350616 learned_addr learned my addr 172.21.15.62:0/2378350616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f1685d9b640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1688102790 0x7f16881988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.482+0000 7f1686d9d640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1688106f10 0x7f168819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f168659c640 1 -- 172.21.15.62:0/2378350616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1688106f10 msgr2=0x7f168819cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f168659c640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1688106f10 0x7f168819cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f168659c640 1 -- 172.21.15.62:0/2378350616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1688102790 msgr2=0x7f16881988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f168659c640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1688102790 0x7f16881988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f168659c640 1 -- 172.21.15.62:0/2378350616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f168819d360 con 0x7f1688101de0 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f1685d9b640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1688102790 0x7f16881988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f168659c640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688101de0 0x7f16881983b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f167800d950 tx=0x7f167800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f167f7fe640 1 -- 172.21.15.62:0/2378350616 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1678017030 con 0x7f1688101de0 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f167f7fe640 1 -- 172.21.15.62:0/2378350616 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1678004020 con 0x7f1688101de0 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f168819d650 con 0x7f1688101de0 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f167f7fe640 1 -- 172.21.15.62:0/2378350616 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16780047a0 con 0x7f1688101de0 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.483+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16881a4f30 con 0x7f1688101de0 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.484+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16881021c0 con 0x7f1688101de0 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.485+0000 7f167f7fe640 1 -- 172.21.15.62:0/2378350616 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f16780041d0 con 0x7f1688101de0 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.486+0000 7f167f7fe640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16740a32d0 0x7f16740a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:30.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.486+0000 7f1685d9b640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16740a32d0 0x7f16740a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:30.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.486+0000 7f167f7fe640 1 -- 172.21.15.62:0/2378350616 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f16780cf730 con 0x7f1688101de0 2024-09-15T23:15:30.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.487+0000 7f1685d9b640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16740a32d0 0x7f16740a5790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f16881088e0 tx=0x7f167000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:30.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.490+0000 7f167f7fe640 1 -- 172.21.15.62:0/2378350616 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f167809bfb0 con 0x7f1688101de0 2024-09-15T23:15:30.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.716+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f168802d7c0 con 0x7f1688101de0 2024-09-15T23:15:30.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.716+0000 7f167f7fe640 1 -- 172.21.15.62:0/2378350616 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11903 (secure 0 0 0) 0x7f167809b8b0 con 0x7f1688101de0 2024-09-15T23:15:30.718 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:30.718 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"btime":"2024-09-15T23:14:10:623582+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:14:10.188556+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:15:30.718 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 24 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.718+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16740a32d0 msgr2=0x7f16740a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.718+0000 7f168c8e6640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16740a32d0 0x7f16740a5790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f16881088e0 tx=0x7f167000b390 comp rx=0 tx=0).stop 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688101de0 msgr2=0x7f16881983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688101de0 0x7f16881983b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f167800d950 tx=0x7f167800de10 comp rx=0 tx=0).stop 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 shutdown_connections 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16740a32d0 0x7f16740a5790 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1688106f10 0x7f168819cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1688102790 0x7f16881988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 --2- 172.21.15.62:0/2378350616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1688101de0 0x7f16881983b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 >> 172.21.15.62:0/2378350616 conn(0x7f16880fd8c0 msgr2=0x7f16880fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 shutdown_connections 2024-09-15T23:15:30.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:30.719+0000 7f168c8e6640 1 -- 172.21.15.62:0/2378350616 wait complete. 2024-09-15T23:15:30.978 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:30 smithi148 ceph-mon[35610]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:30.978 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:30 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:30.978 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:30 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2378350616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:31.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:30 smithi062 ceph-mon[30949]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:31.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:30 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:31.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:30 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2378350616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:31.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:30 smithi067 ceph-mon[35665]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:31.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:30 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:15:31.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:30 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2378350616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:31.796 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:15:32.166 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:32.582 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:32 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:32.582 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:32 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:32.583 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:32 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:32.583 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:32 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:32.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.665+0000 7fa2d3c66640 1 -- 172.21.15.62:0/1298499956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 msgr2=0x7fa2cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:32.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.665+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/1298499956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 0x7fa2cc1067e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0009900 tx=0x7fa2c002a410 comp rx=0 tx=0).stop 2024-09-15T23:15:32.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.665+0000 7fa2d3c66640 1 -- 172.21.15.62:0/1298499956 shutdown_connections 2024-09-15T23:15:32.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.665+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/1298499956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2cc106f10 0x7fa2cc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.665+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/1298499956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 0x7fa2cc1067e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.665+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/1298499956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc101de0 0x7fa2cc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.665+0000 7fa2d3c66640 1 -- 172.21.15.62:0/1298499956 >> 172.21.15.62:0/1298499956 conn(0x7fa2cc0fd8c0 msgr2=0x7fa2cc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:32.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.665+0000 7fa2d3c66640 1 -- 172.21.15.62:0/1298499956 shutdown_connections 2024-09-15T23:15:32.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.665+0000 7fa2d3c66640 1 -- 172.21.15.62:0/1298499956 wait complete. 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d3c66640 1 Processor -- start 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d3c66640 1 -- start start 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d3c66640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2cc101de0 0x7fa2cc198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d3c66640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 0x7fa2cc198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d3c66640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc106f10 0x7fa2cc19ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d3c66640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc111fc0 con 0x7fa2cc102790 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d3c66640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc111e40 con 0x7fa2cc106f10 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d3c66640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa2cc112140 con 0x7fa2cc101de0 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1201640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 0x7fa2cc198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1201640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 0x7fa2cc198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45802/0 (socket says 172.21.15.62:45802) 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1201640 1 -- 172.21.15.62:0/2340723972 learned_addr learned my addr 172.21.15.62:0/2340723972 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:32.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1a02640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2cc101de0 0x7fa2cc198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d2203640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc106f10 0x7fa2cc19ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1201640 1 -- 172.21.15.62:0/2340723972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2cc101de0 msgr2=0x7fa2cc198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1201640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2cc101de0 0x7fa2cc198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1201640 1 -- 172.21.15.62:0/2340723972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc106f10 msgr2=0x7fa2cc19ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1201640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc106f10 0x7fa2cc19ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1201640 1 -- 172.21.15.62:0/2340723972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2cc19d3b0 con 0x7fa2cc102790 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.666+0000 7fa2d1a02640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2cc101de0 0x7fa2cc198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.667+0000 7fa2d2203640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc106f10 0x7fa2cc19ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.667+0000 7fa2d1201640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 0x7fa2cc198940 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0004770 tx=0x7fa2c0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.667+0000 7fa2beffd640 1 -- 172.21.15.62:0/2340723972 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c003f070 con 0x7fa2cc102790 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.667+0000 7fa2beffd640 1 -- 172.21.15.62:0/2340723972 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2c0031ca0 con 0x7fa2cc102790 2024-09-15T23:15:32.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.667+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2cc19d640 con 0x7fa2cc102790 2024-09-15T23:15:32.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.667+0000 7fa2beffd640 1 -- 172.21.15.62:0/2340723972 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c0035030 con 0x7fa2cc102790 2024-09-15T23:15:32.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.667+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2cc1a4f80 con 0x7fa2cc102790 2024-09-15T23:15:32.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.668+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2cc1021c0 con 0x7fa2cc102790 2024-09-15T23:15:32.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.669+0000 7fa2beffd640 1 -- 172.21.15.62:0/2340723972 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fa2c0042070 con 0x7fa2cc102790 2024-09-15T23:15:32.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.670+0000 7fa2beffd640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa2b80a3350 0x7fa2b80a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:32.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.670+0000 7fa2d1a02640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa2b80a3350 0x7fa2b80a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:32.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.670+0000 7fa2beffd640 1 -- 172.21.15.62:0/2340723972 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fa2c00f0510 con 0x7fa2cc102790 2024-09-15T23:15:32.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.671+0000 7fa2d1a02640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa2b80a3350 0x7fa2b80a5810 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8005fd0 tx=0x7fa2c800b5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:32.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.675+0000 7fa2beffd640 1 -- 172.21.15.62:0/2340723972 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fa2c00bcd90 con 0x7fa2cc102790 2024-09-15T23:15:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:32 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:32 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:32.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:32 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:32.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:32 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:32.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.931+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2cc199760 con 0x7fa2cc102790 2024-09-15T23:15:32.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.932+0000 7fa2beffd640 1 -- 172.21.15.62:0/2340723972 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11903 (secure 0 0 0) 0x7fa2cc199760 con 0x7fa2cc102790 2024-09-15T23:15:32.933 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:32.933 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":24,"btime":"2024-09-15T23:14:10:623582+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:14:10.188556+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:15:32.934 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 24 2024-09-15T23:15:32.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa2b80a3350 msgr2=0x7fa2b80a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:32.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa2b80a3350 0x7fa2b80a5810 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8005fd0 tx=0x7fa2c800b5f0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 msgr2=0x7fa2cc198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 0x7fa2cc198940 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0004770 tx=0x7fa2c0003c10 comp rx=0 tx=0).stop 2024-09-15T23:15:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 shutdown_connections 2024-09-15T23:15:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa2b80a3350 0x7fa2b80a5810 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa2cc106f10 0x7fa2cc19ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2cc102790 0x7fa2cc198940 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 --2- 172.21.15.62:0/2340723972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa2cc101de0 0x7fa2cc198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.934+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 >> 172.21.15.62:0/2340723972 conn(0x7fa2cc0fd8c0 msgr2=0x7fa2cc0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.935+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 shutdown_connections 2024-09-15T23:15:32.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:32.935+0000 7fa2d3c66640 1 -- 172.21.15.62:0/2340723972 wait complete. 2024-09-15T23:15:33.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:32 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:33.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:32 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:33.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:32 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:33.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:32 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:33.706 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:15:33.846 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:33 smithi062 ceph-mon[30949]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:33.846 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:33 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2340723972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:33.847 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:33 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:33.847 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:33 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:33 smithi067 ceph-mon[35665]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:33 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2340723972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:33 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:33 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:33 smithi148 ceph-mon[35610]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-09-15T23:15:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:33 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2340723972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:33 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:33 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:34.088 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.614+0000 7f7b6461d640 1 -- 172.21.15.62:0/3781627952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c103660 msgr2=0x7f7b5c10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.614+0000 7f7b6461d640 1 --2- 172.21.15.62:0/3781627952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c103660 0x7f7b5c10bb50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c009900 tx=0x7f7b4c02a480 comp rx=0 tx=0).stop 2024-09-15T23:15:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.615+0000 7f7b6461d640 1 -- 172.21.15.62:0/3781627952 shutdown_connections 2024-09-15T23:15:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.615+0000 7f7b6461d640 1 --2- 172.21.15.62:0/3781627952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b5c10c170 0x7f7b5c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.615+0000 7f7b6461d640 1 --2- 172.21.15.62:0/3781627952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c103660 0x7f7b5c10bb50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.615+0000 7f7b6461d640 1 --2- 172.21.15.62:0/3781627952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5c102d40 0x7f7b5c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.615+0000 7f7b6461d640 1 -- 172.21.15.62:0/3781627952 >> 172.21.15.62:0/3781627952 conn(0x7f7b5c0fc7a0 msgr2=0x7f7b5c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.615+0000 7f7b6461d640 1 -- 172.21.15.62:0/3781627952 shutdown_connections 2024-09-15T23:15:34.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.615+0000 7f7b6461d640 1 -- 172.21.15.62:0/3781627952 wait complete. 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.615+0000 7f7b6461d640 1 Processor -- start 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.615+0000 7f7b6461d640 1 -- start start 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b6461d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b5c102d40 0x7f7b5c1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b6461d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5c103660 0x7f7b5c198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b6461d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c10c170 0x7f7b5c19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b6461d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c111fc0 con 0x7f7b5c10c170 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b6461d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c111e40 con 0x7f7b5c103660 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b6461d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c112140 con 0x7f7b5c102d40 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b62bba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c10c170 0x7f7b5c19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b62bba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c10c170 0x7f7b5c19cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45824/0 (socket says 172.21.15.62:45824) 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b62bba640 1 -- 172.21.15.62:0/2970836333 learned_addr learned my addr 172.21.15.62:0/2970836333 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:34.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b61bb8640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5c103660 0x7f7b5c198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b623b9640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b5c102d40 0x7f7b5c1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b62bba640 1 -- 172.21.15.62:0/2970836333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b5c102d40 msgr2=0x7f7b5c1983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b62bba640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b5c102d40 0x7f7b5c1983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b62bba640 1 -- 172.21.15.62:0/2970836333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5c103660 msgr2=0x7f7b5c198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b62bba640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5c103660 0x7f7b5c198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b62bba640 1 -- 172.21.15.62:0/2970836333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5c19d370 con 0x7f7b5c10c170 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b61bb8640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5c103660 0x7f7b5c198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.616+0000 7f7b62bba640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c10c170 0x7f7b5c19cc90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400eaa0 tx=0x7f7b5400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.617+0000 7f7b537fe640 1 -- 172.21.15.62:0/2970836333 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5400cdb0 con 0x7f7b5c10c170 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.617+0000 7f7b537fe640 1 -- 172.21.15.62:0/2970836333 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b54005020 con 0x7f7b5c10c170 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.617+0000 7f7b537fe640 1 -- 172.21.15.62:0/2970836333 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b540116b0 con 0x7f7b5c10c170 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.617+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b5c19d660 con 0x7f7b5c10c170 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.617+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b5c1a4f40 con 0x7f7b5c10c170 2024-09-15T23:15:34.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.618+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b5c103120 con 0x7f7b5c10c170 2024-09-15T23:15:34.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.619+0000 7f7b537fe640 1 -- 172.21.15.62:0/2970836333 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f7b54005560 con 0x7f7b5c10c170 2024-09-15T23:15:34.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.619+0000 7f7b537fe640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b480a3350 0x7f7b480a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:34.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.620+0000 7f7b623b9640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b480a3350 0x7f7b480a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:34.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.620+0000 7f7b537fe640 1 -- 172.21.15.62:0/2970836333 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f7b540d0400 con 0x7f7b5c10c170 2024-09-15T23:15:34.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.620+0000 7f7b623b9640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b480a3350 0x7f7b480a5810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b58005e10 tx=0x7f7b58005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:34.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.623+0000 7f7b537fe640 1 -- 172.21.15.62:0/2970836333 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7b54016030 con 0x7f7b5c10c170 2024-09-15T23:15:34.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.799+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b5c199720 con 0x7f7b5c10c170 2024-09-15T23:15:34.800 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:34.800 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:34.800 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:34.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.801+0000 7f7b537fe640 1 -- 172.21.15.62:0/2970836333 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24684 (secure 0 0 0) 0x7f7b5c199720 con 0x7f7b5c10c170 2024-09-15T23:15:34.802 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:34.802 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":90,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:14:41.627161+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:15:34.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b480a3350 msgr2=0x7f7b480a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:34.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b480a3350 0x7f7b480a5810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b58005e10 tx=0x7f7b58005da0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c10c170 msgr2=0x7f7b5c19cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:34.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c10c170 0x7f7b5c19cc90 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400eaa0 tx=0x7f7b5400ef60 comp rx=0 tx=0).stop 2024-09-15T23:15:34.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 shutdown_connections 2024-09-15T23:15:34.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b480a3350 0x7f7b480a5810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b5c10c170 0x7f7b5c19cc90 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b5c103660 0x7f7b5c198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 --2- 172.21.15.62:0/2970836333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b5c102d40 0x7f7b5c1983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:34.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 >> 172.21.15.62:0/2970836333 conn(0x7f7b5c0fc7a0 msgr2=0x7f7b5c0fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:34.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 shutdown_connections 2024-09-15T23:15:34.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:34.804+0000 7f7b6461d640 1 -- 172.21.15.62:0/2970836333 wait complete. 2024-09-15T23:15:35.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:35.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:35.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:35.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:15:35.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:15:35.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:15:35.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-15T23:13:33.200855+0000', 'modified': '2024-09-15T23:14:10.188556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24497, 'mds_1': 24470, 'mds_2': 24466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.148:6835/1239577031', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6834', 'nonce': 1239577031}, {'type': 'v1', 'addr': '172.21.15.148:6835', 'nonce': 1239577031}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.67:6835/3132447542', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 3132447542}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3132447542}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.148:6837/870913016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6836', 'nonce': 870913016}, {'type': 'v1', 'addr': '172.21.15.148:6837', 'nonce': 870913016}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.148:6839/2241485315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6838', 'nonce': 2241485315}, {'type': 'v1', 'addr': '172.21.15.148:6839', 'nonce': 2241485315}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.67:6837/747270351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6836', 'nonce': 747270351}, {'type': 'v1', 'addr': '172.21.15.67:6837', 'nonce': 747270351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24497': {'gid': 24497, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6839/818760439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6838', 'nonce': 818760439}, {'type': 'v1', 'addr': '172.21.15.67:6839', 'nonce': 818760439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24497, 'qdb_cluster': [24497, 24470, 24466]} 2024-09-15T23:15:35.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-15T23:15:35.472 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-15T23:15:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:35 smithi062 ceph-mon[30949]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:35 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2970836333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:35.772 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:36.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:35 smithi067 ceph-mon[35665]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:36.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:35 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2970836333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:36.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:35 smithi148 ceph-mon[35610]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:36.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:35 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2970836333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:36.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.261+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2538298574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 msgr2=0x7fd3e4101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:36.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.261+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2538298574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 0x7fd3e4101e20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0009930 tx=0x7fd3d002a420 comp rx=0 tx=0).stop 2024-09-15T23:15:36.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.261+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2538298574 shutdown_connections 2024-09-15T23:15:36.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.261+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2538298574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e4102360 0x7fd3e410e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.261+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2538298574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 0x7fd3e4101e20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.261+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2538298574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e41079c0 0x7fd3e4107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.261+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2538298574 >> 172.21.15.62:0/2538298574 conn(0x7fd3e40fd6e0 msgr2=0x7fd3e40ffb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.261+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2538298574 shutdown_connections 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.261+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2538298574 wait complete. 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e9fa0640 1 Processor -- start 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e9fa0640 1 -- start start 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e9fa0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 0x7fd3e41982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e9fa0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e4102360 0x7fd3e4198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e9fa0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e41079c0 0x7fd3e419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e9fa0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4111de0 con 0x7fd3e41019c0 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e9fa0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4111c60 con 0x7fd3e41079c0 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e9fa0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd3e4111f60 con 0x7fd3e4102360 2024-09-15T23:15:36.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e8f9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 0x7fd3e41982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e8f9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 0x7fd3e41982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45846/0 (socket says 172.21.15.62:45846) 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.262+0000 7fd3e8f9e640 1 -- 172.21.15.62:0/2697215591 learned_addr learned my addr 172.21.15.62:0/2697215591 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e979f640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e41079c0 0x7fd3e419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e4102360 0x7fd3e4198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e8f9e640 1 -- 172.21.15.62:0/2697215591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e4102360 msgr2=0x7fd3e4198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e4102360 0x7fd3e4198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e8f9e640 1 -- 172.21.15.62:0/2697215591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e41079c0 msgr2=0x7fd3e419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e41079c0 0x7fd3e419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e8f9e640 1 -- 172.21.15.62:0/2697215591 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3e419d280 con 0x7fd3e41019c0 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e979f640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e41079c0 0x7fd3e419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 0x7fd3e41982d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800c890 tx=0x7fd3d800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e1ffb640 1 -- 172.21.15.62:0/2697215591 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8004220 con 0x7fd3e41019c0 2024-09-15T23:15:36.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2697215591 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3e419d570 con 0x7fd3e41019c0 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e1ffb640 1 -- 172.21.15.62:0/2697215591 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d80043c0 con 0x7fd3e41019c0 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e1ffb640 1 -- 172.21.15.62:0/2697215591 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80049e0 con 0x7fd3e41019c0 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.263+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2697215591 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3e41a4e50 con 0x7fd3e41019c0 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3e1ffb640 1 -- 172.21.15.62:0/2697215591 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fd3d80079b0 con 0x7fd3e41019c0 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3ed5db740 1 Processor -- start 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3ed5db740 1 -- start start 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f6740bd0 0x55e9f6740fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f674b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f674bdb0 0x55e9f674e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e9f66a7090 con 0x55e9f67414f0 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e9f664ea70 con 0x55e9f6740bd0 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e9f66a3050 con 0x55e9f674bdb0 2024-09-15T23:15:36.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3e8f9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f674b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3e8f9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f674b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45860/0 (socket says 172.21.15.62:45860) 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3e1ffb640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3d40a32d0 0x7fd3d40a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3e8f9e640 1 -- 172.21.15.62:0/1138417637 learned_addr learned my addr 172.21.15.62:0/1138417637 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.266+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/1138417637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f674bdb0 0x55e9f674e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e979f640 1 --2- 172.21.15.62:0/1138417637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f6740bd0 0x55e9f6740fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e979f640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3d40a32d0 0x7fd3d40a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e1ffb640 1 -- 172.21.15.62:0/2697215591 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fd3d80cfef0 con 0x7fd3e41019c0 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e8f9e640 1 -- 172.21.15.62:0/1138417637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f674bdb0 msgr2=0x55e9f674e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/1138417637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f674bdb0 0x55e9f674e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e8f9e640 1 -- 172.21.15.62:0/1138417637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f6740bd0 msgr2=0x55e9f6740fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/1138417637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f6740bd0 0x55e9f6740fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e8f9e640 1 -- 172.21.15.62:0/1138417637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f665b130 con 0x55e9f67414f0 2024-09-15T23:15:36.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/1138417637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f674b870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd3d809c000 tx=0x7fd3d8005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fbb2a80531d0a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:36.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3e979f640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3d40a32d0 0x7fd3d40a5790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009fd0 tx=0x7fd3dc008220 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:36.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3ceffd640 1 -- 172.21.15.62:0/1138417637 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d801d050 con 0x55e9f67414f0 2024-09-15T23:15:36.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3ceffd640 1 -- 172.21.15.62:0/1138417637 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d809c770 con 0x55e9f67414f0 2024-09-15T23:15:36.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.267+0000 7fd3ceffd640 1 -- 172.21.15.62:0/1138417637 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d80986e0 con 0x55e9f67414f0 2024-09-15T23:15:36.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1138417637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 msgr2=0x55e9f674b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:36.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1138417637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f674b870 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd3d809c000 tx=0x7fd3d8005300 comp rx=0 tx=0).stop 2024-09-15T23:15:36.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1138417637 shutdown_connections 2024-09-15T23:15:36.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1138417637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f674bdb0 0x55e9f674e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1138417637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f674b870 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1138417637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f6740bd0 0x55e9f6740fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1138417637 >> 172.21.15.62:0/1138417637 conn(0x55e9f673d160 msgr2=0x55e9f673d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:36.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1138417637 shutdown_connections 2024-09-15T23:15:36.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1138417637 wait complete. 2024-09-15T23:15:36.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.268+0000 7fd3ed5db740 1 -- start start 2024-09-15T23:15:36.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3ed5db740 1 Processor -- start 2024-09-15T23:15:36.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f674b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f6748ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e9f674e920 con 0x55e9f67414f0 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e9f6661f20 con 0x55e9f674bdb0 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e9f674eaa0 con 0x55e9f6740bd0 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3ed5db740 10 client.0 _ll_register_callbacks cb 0x55e9f662b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45870/0 (socket says 172.21.15.62:45870) 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e3fff640 1 -- 172.21.15.62:0/1855987325 learned_addr learned my addr 172.21.15.62:0/1855987325 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e979f640 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f6748ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f674b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e3fff640 1 -- 172.21.15.62:0/1855987325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 msgr2=0x55e9f674b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f674b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e3fff640 1 -- 172.21.15.62:0/1855987325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 msgr2=0x55e9f6748ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f6748ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e3fff640 1 -- 172.21.15.62:0/1855987325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f64182b0 con 0x55e9f67414f0 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.269+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f674b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0031550 tx=0x7fd3d0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7898ab5cfbfb201 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3c3fff640 1 -- 172.21.15.62:0/1855987325 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0035040 con 0x55e9f67414f0 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1855987325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3d809c770 con 0x55e9f67414f0 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3c3fff640 1 -- 172.21.15.62:0/1855987325 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fd3d0005020 con 0x55e9f67414f0 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3c3fff640 1 -- 172.21.15.62:0/1855987325 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd3d004e050 con 0x55e9f67414f0 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3ed5db740 10 client.15183 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:36.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1855987325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e9f675a210 con 0x55e9f67414f0 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3c3fff640 1 -- 172.21.15.62:0/1855987325 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d809c770 con 0x55e9f67414f0 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3c3fff640 1 -- 172.21.15.62:0/1855987325 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fd3d00035f0 con 0x55e9f67414f0 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3ed5db740 10 client.15183 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3ed5db740 10 client.15183 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3ed5db740 20 client.15183 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55e9f675c870 0x55e9f675dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3ed5db740 4 client.15183 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.270+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1855987325 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9f66b4600 con 0x55e9f675c870 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.271+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55e9f675c870 0x55e9f675dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.271+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55e9f675c870 0x55e9f675dce0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1819565deebc8e0 server_cookie=bfa129de5d05c792 in_seq=0 out_seq=0 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.271+0000 7fd3c3fff640 10 client.15183 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3c3fff640 1 -- 172.21.15.62:0/1855987325 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e9f66b4600 con 0x55e9f675c870 2024-09-15T23:15:36.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3c3fff640 10 client.15183 handle_command_reply: tid=0 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 client.15183 shutdown 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1855987325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 msgr2=0x55e9f6748960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748960 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0031550 tx=0x7fd3d0004270 comp rx=0 tx=0).stop 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1855987325 shutdown_connections 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55e9f675c870 0x55e9f675dce0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f6748ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748960 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3c3fff640 0 client.15183 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/1855987325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f674b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1855987325 >> 172.21.15.62:0/1855987325 conn(0x55e9f673d160 msgr2=0x55e9f6742e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1855987325 shutdown_connections 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 1 -- 172.21.15.62:0/1855987325 wait complete. 2024-09-15T23:15:36.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.272+0000 7fd3ed5db740 20 client.15183 trim_cache size 0 max 16384 2024-09-15T23:15:36.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3ed5db740 1 Processor -- start 2024-09-15T23:15:36.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3ed5db740 1 -- start start 2024-09-15T23:15:36.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f675d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f6748e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e9f675c020 con 0x55e9f67414f0 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e9f675c6f0 con 0x55e9f674bdb0 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e9f675cdc0 con 0x55e9f6740bd0 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45882/0 (socket says 172.21.15.62:45882) 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.290+0000 7fd3e3fff640 1 -- 172.21.15.62:0/397954928 learned_addr learned my addr 172.21.15.62:0/397954928 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/397954928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f675d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3e979f640 1 --2- 172.21.15.62:0/397954928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f6748e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3e3fff640 1 -- 172.21.15.62:0/397954928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 msgr2=0x55e9f675d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/397954928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f675d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3e3fff640 1 -- 172.21.15.62:0/397954928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 msgr2=0x55e9f6748e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/397954928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f6748e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3e3fff640 1 -- 172.21.15.62:0/397954928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f673dd10 con 0x55e9f67414f0 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/397954928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd3d002a940 tx=0x7fd3d002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729e67103088b870 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3ce7fc640 1 -- 172.21.15.62:0/397954928 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0035070 con 0x55e9f67414f0 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3ce7fc640 1 -- 172.21.15.62:0/397954928 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d0057070 con 0x55e9f67414f0 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.291+0000 7fd3ce7fc640 1 -- 172.21.15.62:0/397954928 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0003bb0 con 0x55e9f67414f0 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 -- 172.21.15.62:0/397954928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 msgr2=0x55e9f6748930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/397954928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd3d002a940 tx=0x7fd3d002a420 comp rx=0 tx=0).stop 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 -- 172.21.15.62:0/397954928 shutdown_connections 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/397954928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f6748e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/397954928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6748930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/397954928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f675d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 -- 172.21.15.62:0/397954928 >> 172.21.15.62:0/397954928 conn(0x55e9f673d160 msgr2=0x55e9f6748220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:36.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 -- 172.21.15.62:0/397954928 shutdown_connections 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 -- 172.21.15.62:0/397954928 wait complete. 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 -- start start 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.292+0000 7fd3ed5db740 1 Processor -- start 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f6766320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6766860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3ed5db740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f67618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e9f675db60 con 0x55e9f67414f0 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e9f675d490 con 0x55e9f674bdb0 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3ed5db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e9f675e230 con 0x55e9f6740bd0 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3ed5db740 10 client.0 _ll_register_callbacks cb 0x7fd3d0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6766860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6766860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45894/0 (socket says 172.21.15.62:45894) 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e3fff640 1 -- 172.21.15.62:0/2317844836 learned_addr learned my addr 172.21.15.62:0/2317844836 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e979f640 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f67618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f6766320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e3fff640 1 -- 172.21.15.62:0/2317844836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 msgr2=0x55e9f6766320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f6766320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e3fff640 1 -- 172.21.15.62:0/2317844836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 msgr2=0x55e9f67618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f67618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.293+0000 7fd3e3fff640 1 -- 172.21.15.62:0/2317844836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f67620d0 con 0x55e9f67414f0 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3e3fff640 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6766860 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00039c0 tx=0x7fd3d00045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790e115230effdc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3c3fff640 1 -- 172.21.15.62:0/2317844836 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0035030 con 0x55e9f67414f0 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3c3fff640 1 -- 172.21.15.62:0/2317844836 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fd3d00047a0 con 0x55e9f67414f0 2024-09-15T23:15:36.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3ed5db740 1 -- 172.21.15.62:0/2317844836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3d0057070 con 0x55e9f67414f0 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3c3fff640 1 -- 172.21.15.62:0/2317844836 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd3d0003db0 con 0x55e9f67414f0 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3ed5db740 10 client.15189 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3ed5db740 1 -- 172.21.15.62:0/2317844836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e9f67c6be0 con 0x55e9f67414f0 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3c3fff640 1 -- 172.21.15.62:0/2317844836 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3d0057070 con 0x55e9f67414f0 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.294+0000 7fd3c3fff640 1 -- 172.21.15.62:0/2317844836 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fd3d0045480 con 0x55e9f67414f0 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.295+0000 7fd3ed5db740 10 client.15189 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.295+0000 7fd3ed5db740 10 client.15189 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.295+0000 7fd3ed5db740 20 client.15189 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.295+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55e9f67cb480 0x55e9f67ca190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.295+0000 7fd3ed5db740 4 client.15189 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:15:36.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.295+0000 7fd3ed5db740 1 -- 172.21.15.62:0/2317844836 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "status"}) -- 0x55e9f66aeb10 con 0x55e9f67cb480 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.295+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55e9f67cb480 0x55e9f67ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.295+0000 7fd3c3fff640 10 client.15189 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.295+0000 7fd3e8f9e640 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55e9f67cb480 0x55e9f67ca190 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3832d6c2429858ab server_cookie=e97609916307fa2b in_seq=0 out_seq=0 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3c3fff640 1 -- 172.21.15.62:0/2317844836 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55e9f66aeb10 con 0x55e9f67cb480 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3c3fff640 10 client.15189 handle_command_reply: tid=0 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 client.15189 shutdown 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 -- 172.21.15.62:0/2317844836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 msgr2=0x55e9f6766860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6766860 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00039c0 tx=0x7fd3d00045d0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 -- 172.21.15.62:0/2317844836 shutdown_connections 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55e9f67cb480 0x55e9f67ca190 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3c3fff640 0 client.15189 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e9f674bdb0 0x55e9f67618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e9f67414f0 0x55e9f6766860 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 --2- 172.21.15.62:0/2317844836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e9f6740bd0 0x55e9f6766320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 -- 172.21.15.62:0/2317844836 >> 172.21.15.62:0/2317844836 conn(0x55e9f673d160 msgr2=0x55e9f67480d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:36.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 -- 172.21.15.62:0/2317844836 shutdown_connections 2024-09-15T23:15:36.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 1 -- 172.21.15.62:0/2317844836 wait complete. 2024-09-15T23:15:36.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.296+0000 7fd3ed5db740 20 client.15189 trim_cache size 0 max 16384 2024-09-15T23:15:36.304 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:15:36.304 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:15:36.304 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 2, 2024-09-15T23:15:36.304 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24466, 2024-09-15T23:15:36.305 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:15:36.305 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:15:36.305 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:15:36.305 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 88.671373183, 2024-09-15T23:15:36.305 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 24, 2024-09-15T23:15:36.305 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 90, 2024-09-15T23:15:36.305 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T23:15:36.305 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 143.43168363999999 2024-09-15T23:15:36.305 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3d40a32d0 msgr2=0x7fd3d40a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3d40a32d0 0x7fd3d40a5790 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009fd0 tx=0x7fd3dc008220 comp rx=0 tx=0).stop 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 msgr2=0x7fd3e41982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 0x7fd3e41982d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800c890 tx=0x7fd3d800cd50 comp rx=0 tx=0).stop 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2697215591 shutdown_connections 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3d40a32d0 0x7fd3d40a5790 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3e41079c0 0x7fd3e419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3e4102360 0x7fd3e4198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 --2- 172.21.15.62:0/2697215591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3e41019c0 0x7fd3e41982d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:36.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2697215591 >> 172.21.15.62:0/2697215591 conn(0x7fd3e40fd6e0 msgr2=0x7fd3e40fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2697215591 shutdown_connections 2024-09-15T23:15:36.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:36.297+0000 7fd3e9fa0640 1 -- 172.21.15.62:0/2697215591 wait complete. 2024-09-15T23:15:36.985 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-15T23:15:37.243 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:37.637 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:37 smithi062 ceph-mon[30949]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:37.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.720+0000 7f740e2f0640 1 -- 172.21.15.62:0/640202303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408101420 msgr2=0x7f740810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.720+0000 7f740e2f0640 1 --2- 172.21.15.62:0/640202303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408101420 0x7f740810d950 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7400009cd0 tx=0x7f740002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.720+0000 7f740e2f0640 1 -- 172.21.15.62:0/640202303 shutdown_connections 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.720+0000 7f740e2f0640 1 --2- 172.21.15.62:0/640202303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408101420 0x7f740810d950 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.720+0000 7f740e2f0640 1 --2- 172.21.15.62:0/640202303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7408100a80 0x7f7408100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.720+0000 7f740e2f0640 1 --2- 172.21.15.62:0/640202303 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7408106a80 0x7f7408106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.720+0000 7f740e2f0640 1 -- 172.21.15.62:0/640202303 >> 172.21.15.62:0/640202303 conn(0x7f74080fc7a0 msgr2=0x7f74080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.720+0000 7f740e2f0640 1 -- 172.21.15.62:0/640202303 shutdown_connections 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.720+0000 7f740e2f0640 1 -- 172.21.15.62:0/640202303 wait complete. 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.721+0000 7f740e2f0640 1 Processor -- start 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.721+0000 7f740e2f0640 1 -- start start 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.721+0000 7f740e2f0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7408100a80 0x7f7408198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.721+0000 7f740e2f0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7408101420 0x7f74081989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.721+0000 7f740e2f0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408106a80 0x7f740819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.721+0000 7f740e2f0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7408110e80 con 0x7f7408106a80 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.721+0000 7f740e2f0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7408110d00 con 0x7f7408100a80 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.721+0000 7f740e2f0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7408111000 con 0x7f7408101420 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f740c88d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408106a80 0x7f740819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f740c88d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408106a80 0x7f740819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45898/0 (socket says 172.21.15.62:45898) 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f740c88d640 1 -- 172.21.15.62:0/3543972251 learned_addr learned my addr 172.21.15.62:0/3543972251 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f74077fe640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7408101420 0x7f74081989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f7407fff640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7408100a80 0x7f7408198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f740c88d640 1 -- 172.21.15.62:0/3543972251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7408101420 msgr2=0x7f74081989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f740c88d640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7408101420 0x7f74081989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f740c88d640 1 -- 172.21.15.62:0/3543972251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7408100a80 msgr2=0x7f7408198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f740c88d640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7408100a80 0x7f7408198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f740c88d640 1 -- 172.21.15.62:0/3543972251 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740819d420 con 0x7f7408106a80 2024-09-15T23:15:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.722+0000 7f740c88d640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408106a80 0x7f740819cd40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7400005cc0 tx=0x7f7400032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.723+0000 7f74057fa640 1 -- 172.21.15.62:0/3543972251 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7400004040 con 0x7f7408106a80 2024-09-15T23:15:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.723+0000 7f74057fa640 1 -- 172.21.15.62:0/3543972251 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74000046c0 con 0x7f7408106a80 2024-09-15T23:15:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.723+0000 7f740e2f0640 1 -- 172.21.15.62:0/3543972251 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f740819d6b0 con 0x7f7408106a80 2024-09-15T23:15:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.723+0000 7f74057fa640 1 -- 172.21.15.62:0/3543972251 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7400030e00 con 0x7f7408106a80 2024-09-15T23:15:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.723+0000 7f740e2f0640 1 -- 172.21.15.62:0/3543972251 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74081a4ff0 con 0x7f7408106a80 2024-09-15T23:15:37.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.725+0000 7f74057fa640 1 -- 172.21.15.62:0/3543972251 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7f74000305a0 con 0x7f7408106a80 2024-09-15T23:15:37.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.725+0000 7f74106c9740 1 Processor -- start 2024-09-15T23:15:37.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.725+0000 7f74106c9740 1 -- start start 2024-09-15T23:15:37.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74106c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 0x55c4bbd10db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74106c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd112f0 0x55c4bbd1b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74106c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd1dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74106c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbc77090 con 0x55c4bbd1bbb0 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74106c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbc1ea70 con 0x55c4bbd109d0 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74106c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbc73050 con 0x55c4bbd112f0 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74057fa640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73f40a3350 0x7f73f40a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74077fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd1dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74077fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd1dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45908/0 (socket says 172.21.15.62:45908) 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74077fe640 1 -- 172.21.15.62:0/1861132482 learned_addr learned my addr 172.21.15.62:0/1861132482 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74077fe640 1 --2- 172.21.15.62:0/1861132482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 0x55c4bbd10db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f740c88d640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73f40a3350 0x7f73f40a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f7407fff640 1 --2- 172.21.15.62:0/1861132482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd112f0 0x55c4bbd1b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74057fa640 1 -- 172.21.15.62:0/3543972251 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f740002e030 con 0x7f7408106a80 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74077fe640 1 -- 172.21.15.62:0/1861132482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd112f0 msgr2=0x55c4bbd1b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74077fe640 1 --2- 172.21.15.62:0/1861132482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd112f0 0x55c4bbd1b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74077fe640 1 -- 172.21.15.62:0/1861132482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 msgr2=0x55c4bbd10db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74077fe640 1 --2- 172.21.15.62:0/1861132482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 0x55c4bbd10db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.726+0000 7f74077fe640 1 -- 172.21.15.62:0/1861132482 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4bbc2b130 con 0x55c4bbd1bbb0 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f74077fe640 1 --2- 172.21.15.62:0/1861132482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd1dfa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f73f00127c0 tx=0x7f73f000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed7b5392c17e4f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f740c88d640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73f40a3350 0x7f73f40a5810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7400009ca0 tx=0x7f74000b8040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f73fe7fc640 1 -- 172.21.15.62:0/1861132482 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0012e00 con 0x55c4bbd1bbb0 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f73fe7fc640 1 -- 172.21.15.62:0/1861132482 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f0010c50 con 0x55c4bbd1bbb0 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f73fe7fc640 1 -- 172.21.15.62:0/1861132482 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f001ebe0 con 0x55c4bbd1bbb0 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f74106c9740 1 -- 172.21.15.62:0/1861132482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd1bbb0 msgr2=0x55c4bbd1dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f74106c9740 1 --2- 172.21.15.62:0/1861132482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd1dfa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f73f00127c0 tx=0x7f73f000b7c0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f74106c9740 1 -- 172.21.15.62:0/1861132482 shutdown_connections 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f74106c9740 1 --2- 172.21.15.62:0/1861132482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd1dfa0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f74106c9740 1 --2- 172.21.15.62:0/1861132482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd112f0 0x55c4bbd1b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f74106c9740 1 --2- 172.21.15.62:0/1861132482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 0x55c4bbd10db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.727+0000 7f74106c9740 1 -- 172.21.15.62:0/1861132482 >> 172.21.15.62:0/1861132482 conn(0x55c4bbd0cf60 msgr2=0x55c4bbd0d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 -- 172.21.15.62:0/1861132482 shutdown_connections 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 -- 172.21.15.62:0/1861132482 wait complete. 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 -- start start 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 Processor -- start 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd109d0 0x55c4bbd1b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd18740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd18c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbd1e780 con 0x55c4bbd112f0 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbc31f20 con 0x55c4bbd1bbb0 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbd1e900 con 0x55c4bbd109d0 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f74106c9740 10 client.0 _ll_register_callbacks cb 0x55c4bbbfb030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.728+0000 7f7407fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd18740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f7407fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd18740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45924/0 (socket says 172.21.15.62:45924) 2024-09-15T23:15:37.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f7407fff640 1 -- 172.21.15.62:0/1777384784 learned_addr learned my addr 172.21.15.62:0/1777384784 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f74077fe640 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd18c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f74077fe640 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd109d0 0x55c4bbd1b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f7407fff640 1 -- 172.21.15.62:0/1777384784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd109d0 msgr2=0x55c4bbd1b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f7407fff640 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd109d0 0x55c4bbd1b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f7407fff640 1 -- 172.21.15.62:0/1777384784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 msgr2=0x55c4bbd18c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f7407fff640 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd18c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f7407fff640 1 -- 172.21.15.62:0/1777384784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4bb9e82b0 con 0x55c4bbd112f0 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f74077fe640 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd18c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f7407fff640 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd18740 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f73f8012700 tx=0x7f73f8012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8911fe77d496984 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f73e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f73e77fe640 1 -- 172.21.15.62:0/1777384784 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f8011bb0 con 0x55c4bbd112f0 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f73e77fe640 1 -- 172.21.15.62:0/1777384784 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f73f8011d50 con 0x55c4bbd112f0 2024-09-15T23:15:37.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f74106c9740 1 -- 172.21.15.62:0/1777384784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73f0010c50 con 0x55c4bbd112f0 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f73e77fe640 1 -- 172.21.15.62:0/1777384784 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f73f80214d0 con 0x55c4bbd112f0 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f74106c9740 10 client.15201 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f74106c9740 1 -- 172.21.15.62:0/1777384784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4bbd2a120 con 0x55c4bbd112f0 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.729+0000 7f73e77fe640 1 -- 172.21.15.62:0/1777384784 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f0010c50 con 0x55c4bbd112f0 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f73e77fe640 1 -- 172.21.15.62:0/1777384784 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f73f800f4f0 con 0x55c4bbd112f0 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f74106c9740 10 client.15201 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f74106c9740 10 client.15201 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f74106c9740 20 client.15201 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f74106c9740 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55c4bbd2c7a0 0x55c4bbd2dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f74106c9740 4 client.15201 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f74106c9740 1 -- 172.21.15.62:0/1777384784 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4bbc84600 con 0x55c4bbd2c7a0 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f74077fe640 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55c4bbd2c7a0 0x55c4bbd2dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f73e77fe640 10 client.15201 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:15:37.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.730+0000 7f74077fe640 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55c4bbd2c7a0 0x55c4bbd2dc10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b5c2dc93414cd78 server_cookie=40d20eef3a5de72c in_seq=0 out_seq=0 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f73e77fe640 1 -- 172.21.15.62:0/1777384784 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c4bbc84600 con 0x55c4bbd2c7a0 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f73e77fe640 10 client.15201 handle_command_reply: tid=0 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f74106c9740 1 client.15201 shutdown 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f74106c9740 1 -- 172.21.15.62:0/1777384784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 msgr2=0x55c4bbd18740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f74106c9740 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd18740 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f73f8012700 tx=0x7f73f8012bc0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f74106c9740 1 -- 172.21.15.62:0/1777384784 shutdown_connections 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f74106c9740 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55c4bbd2c7a0 0x55c4bbd2dc10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f74106c9740 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd18c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f74106c9740 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd18740 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f74106c9740 1 --2- 172.21.15.62:0/1777384784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd109d0 0x55c4bbd1b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.731+0000 7f73e77fe640 0 client.15201 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.732+0000 7f74106c9740 1 -- 172.21.15.62:0/1777384784 >> 172.21.15.62:0/1777384784 conn(0x55c4bbd0cf60 msgr2=0x55c4bbd12c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.732+0000 7f74106c9740 1 -- 172.21.15.62:0/1777384784 shutdown_connections 2024-09-15T23:15:37.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.732+0000 7f74106c9740 1 -- 172.21.15.62:0/1777384784 wait complete. 2024-09-15T23:15:37.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.732+0000 7f74106c9740 20 client.15201 trim_cache size 0 max 16384 2024-09-15T23:15:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.749+0000 7f74106c9740 1 Processor -- start 2024-09-15T23:15:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.749+0000 7f74106c9740 1 -- start start 2024-09-15T23:15:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74106c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd109d0 0x55c4bbd2c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74106c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd112f0 0x55c4bbd2ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74106c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd362e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74106c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbd2bf50 con 0x55c4bbd109d0 2024-09-15T23:15:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74106c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbd2c620 con 0x55c4bbd1bbb0 2024-09-15T23:15:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74106c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbd2ccf0 con 0x55c4bbd112f0 2024-09-15T23:15:37.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd109d0 0x55c4bbd2c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd109d0 0x55c4bbd2c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45936/0 (socket says 172.21.15.62:45936) 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 -- 172.21.15.62:0/3649508891 learned_addr learned my addr 172.21.15.62:0/3649508891 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 --2- 172.21.15.62:0/3649508891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd362e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 -- 172.21.15.62:0/3649508891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd112f0 msgr2=0x55c4bbd2ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 --2- 172.21.15.62:0/3649508891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd112f0 0x55c4bbd2ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 -- 172.21.15.62:0/3649508891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 msgr2=0x55c4bbd362e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 --2- 172.21.15.62:0/3649508891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd362e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 -- 172.21.15.62:0/3649508891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4bbd1b760 con 0x55c4bbd109d0 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f74077fe640 1 --2- 172.21.15.62:0/3649508891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd109d0 0x55c4bbd2c0d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f73f00127c0 tx=0x7f73f000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ffd2174c74cfa79 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f73fdffb640 1 -- 172.21.15.62:0/3649508891 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0024040 con 0x55c4bbd109d0 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f73fdffb640 1 -- 172.21.15.62:0/3649508891 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f001ed70 con 0x55c4bbd109d0 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.750+0000 7f73fdffb640 1 -- 172.21.15.62:0/3649508891 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0010740 con 0x55c4bbd109d0 2024-09-15T23:15:37.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 -- 172.21.15.62:0/3649508891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd109d0 msgr2=0x55c4bbd2c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 --2- 172.21.15.62:0/3649508891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd109d0 0x55c4bbd2c0d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f73f00127c0 tx=0x7f73f000b7c0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 -- 172.21.15.62:0/3649508891 shutdown_connections 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 --2- 172.21.15.62:0/3649508891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd362e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 --2- 172.21.15.62:0/3649508891 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd112f0 0x55c4bbd2ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 --2- 172.21.15.62:0/3649508891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd109d0 0x55c4bbd2c0d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 -- 172.21.15.62:0/3649508891 >> 172.21.15.62:0/3649508891 conn(0x55c4bbd0cf60 msgr2=0x55c4bbd0d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 -- 172.21.15.62:0/3649508891 shutdown_connections 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 -- 172.21.15.62:0/3649508891 wait complete. 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.751+0000 7f74106c9740 1 -- start start 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74106c9740 1 Processor -- start 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74106c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 0x55c4bbd36600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74106c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd339a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74106c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd33ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74106c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbd2da90 con 0x55c4bbd112f0 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74106c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbd2d3c0 con 0x55c4bbd109d0 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74106c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c4bbd2e160 con 0x55c4bbd1bbb0 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74106c9740 10 client.0 _ll_register_callbacks cb 0x7f73f8009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f7407fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd339a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f7407fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd339a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45938/0 (socket says 172.21.15.62:45938) 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f7407fff640 1 -- 172.21.15.62:0/3039273724 learned_addr learned my addr 172.21.15.62:0/3039273724 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74077fe640 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 0x55c4bbd36600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f74077fe640 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd33ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f7407fff640 1 -- 172.21.15.62:0/3039273724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd1bbb0 msgr2=0x55c4bbd33ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f7407fff640 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd33ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f7407fff640 1 -- 172.21.15.62:0/3039273724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 msgr2=0x55c4bbd36600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f7407fff640 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 0x55c4bbd36600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.752+0000 7f7407fff640 1 -- 172.21.15.62:0/3039273724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4bbd34710 con 0x55c4bbd112f0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f7407fff640 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd339a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f73f800fb50 tx=0x7f73f8021430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfd40de99695a73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f73e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f73e77fe640 1 -- 172.21.15.62:0/3039273724 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f802f060 con 0x55c4bbd112f0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f73e77fe640 1 -- 172.21.15.62:0/3039273724 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7f73f8011dc0 con 0x55c4bbd112f0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f74106c9740 1 -- 172.21.15.62:0/3039273724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73f001ed70 con 0x55c4bbd112f0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f73e77fe640 1 -- 172.21.15.62:0/3039273724 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f73f8036050 con 0x55c4bbd112f0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f74106c9740 10 client.15207 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f74106c9740 1 -- 172.21.15.62:0/3039273724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4bbd9aed0 con 0x55c4bbd112f0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f73e77fe640 1 -- 172.21.15.62:0/3039273724 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f001ed70 con 0x55c4bbd112f0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f73e77fe640 1 -- 172.21.15.62:0/3039273724 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f73f8021770 con 0x55c4bbd112f0 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.753+0000 7f74106c9740 10 client.15207 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:37.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f74106c9740 10 client.15207 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f74106c9740 20 client.15207 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f74106c9740 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55c4bbd9faa0 0x55c4bbd9e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f74106c9740 4 client.15207 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f74106c9740 1 -- 172.21.15.62:0/3039273724 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "status"}) -- 0x55c4bbc7eb10 con 0x55c4bbd9faa0 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f74077fe640 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55c4bbd9faa0 0x55c4bbd9e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f74077fe640 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55c4bbd9faa0 0x55c4bbd9e7b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4615422ea3043b6d server_cookie=d35a2550b92aa5b3 in_seq=0 out_seq=0 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f73e77fe640 10 client.15207 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f73e77fe640 1 -- 172.21.15.62:0/3039273724 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c4bbc7eb10 con 0x55c4bbd9faa0 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f73e77fe640 10 client.15207 handle_command_reply: tid=0 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.754+0000 7f74106c9740 1 client.15207 shutdown 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 -- 172.21.15.62:0/3039273724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 msgr2=0x55c4bbd339a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd339a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f73f800fb50 tx=0x7f73f8021430 comp rx=0 tx=0).stop 2024-09-15T23:15:37.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 -- 172.21.15.62:0/3039273724 shutdown_connections 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55c4bbd9faa0 0x55c4bbd9e7b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c4bbd1bbb0 0x55c4bbd33ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c4bbd112f0 0x55c4bbd339a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 --2- 172.21.15.62:0/3039273724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c4bbd109d0 0x55c4bbd36600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 -- 172.21.15.62:0/3039273724 >> 172.21.15.62:0/3039273724 conn(0x55c4bbd0cf60 msgr2=0x55c4bbd13560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f73e77fe640 0 client.15207 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 -- 172.21.15.62:0/3039273724 shutdown_connections 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 1 -- 172.21.15.62:0/3039273724 wait complete. 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f74106c9740 20 client.15207 trim_cache size 0 max 16384 2024-09-15T23:15:37.769 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 1, 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24470, 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 91.138898089999998, 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 24, 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 90, 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 147.47776809600001 2024-09-15T23:15:37.770 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:15:37.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f740e2f0640 1 -- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73f40a3350 msgr2=0x7f73f40a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:37.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f740e2f0640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73f40a3350 0x7f73f40a5810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7400009ca0 tx=0x7f74000b8040 comp rx=0 tx=0).stop 2024-09-15T23:15:37.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f740e2f0640 1 -- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408106a80 msgr2=0x7f740819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:37.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f740e2f0640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408106a80 0x7f740819cd40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7400005cc0 tx=0x7f7400032f40 comp rx=0 tx=0).stop 2024-09-15T23:15:37.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f740e2f0640 1 -- 172.21.15.62:0/3543972251 shutdown_connections 2024-09-15T23:15:37.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f740e2f0640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f73f40a3350 0x7f73f40a5810 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f740e2f0640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7408106a80 0x7f740819cd40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f740e2f0640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7408101420 0x7f74081989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.755+0000 7f740e2f0640 1 --2- 172.21.15.62:0/3543972251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7408100a80 0x7f7408198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.756+0000 7f740e2f0640 1 -- 172.21.15.62:0/3543972251 >> 172.21.15.62:0/3543972251 conn(0x7f74080fc7a0 msgr2=0x7f740806aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.756+0000 7f740e2f0640 1 -- 172.21.15.62:0/3543972251 shutdown_connections 2024-09-15T23:15:37.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:37.756+0000 7f740e2f0640 1 -- 172.21.15.62:0/3543972251 wait complete. 2024-09-15T23:15:38.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:37 smithi067 ceph-mon[35665]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:38.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:37 smithi148 ceph-mon[35610]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-09-15T23:15:38.449 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T23:15:38.711 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:39.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.152+0000 7fc5d7e90640 1 -- 172.21.15.62:0/2585717776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 msgr2=0x7fc5d01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.152+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/2585717776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 0x7fc5d01067e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4009900 tx=0x7fc5c402a410 comp rx=0 tx=0).stop 2024-09-15T23:15:39.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.152+0000 7fc5d7e90640 1 -- 172.21.15.62:0/2585717776 shutdown_connections 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.152+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/2585717776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d0106f10 0x7fc5d010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.152+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/2585717776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 0x7fc5d01067e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.152+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/2585717776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc5d0101de0 0x7fc5d01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.152+0000 7fc5d7e90640 1 -- 172.21.15.62:0/2585717776 >> 172.21.15.62:0/2585717776 conn(0x7fc5d00fd8c0 msgr2=0x7fc5d00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.152+0000 7fc5d7e90640 1 -- 172.21.15.62:0/2585717776 shutdown_connections 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.152+0000 7fc5d7e90640 1 -- 172.21.15.62:0/2585717776 wait complete. 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d7e90640 1 Processor -- start 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d7e90640 1 -- start start 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d7e90640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d0101de0 0x7fc5d0198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d7e90640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 0x7fc5d01989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d7e90640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc5d0106f10 0x7fc5d019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d7e90640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d0111f70 con 0x7fc5d0102790 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d7e90640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d0111df0 con 0x7fc5d0101de0 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d7e90640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d01120f0 con 0x7fc5d0106f10 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d542b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 0x7fc5d01989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d542b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 0x7fc5d01989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45952/0 (socket says 172.21.15.62:45952) 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d542b640 1 -- 172.21.15.62:0/3712342339 learned_addr learned my addr 172.21.15.62:0/3712342339 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d642d640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc5d0106f10 0x7fc5d019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.153+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d0101de0 0x7fc5d0198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5d542b640 1 -- 172.21.15.62:0/3712342339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc5d0106f10 msgr2=0x7fc5d019cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5d542b640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc5d0106f10 0x7fc5d019cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5d542b640 1 -- 172.21.15.62:0/3712342339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d0101de0 msgr2=0x7fc5d0198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5d542b640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d0101de0 0x7fc5d0198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5d542b640 1 -- 172.21.15.62:0/3712342339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d019d440 con 0x7fc5d0102790 2024-09-15T23:15:39.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5d542b640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 0x7fc5d01989d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4006010 tx=0x7fc5c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5c2ffd640 1 -- 172.21.15.62:0/3712342339 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c403f030 con 0x7fc5d0102790 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5d7e90640 1 -- 172.21.15.62:0/3712342339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d019d6d0 con 0x7fc5d0102790 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5c2ffd640 1 -- 172.21.15.62:0/3712342339 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5c4003ce0 con 0x7fc5d0102790 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5c2ffd640 1 -- 172.21.15.62:0/3712342339 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4035070 con 0x7fc5d0102790 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.154+0000 7fc5d7e90640 1 -- 172.21.15.62:0/3712342339 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d01a5010 con 0x7fc5d0102790 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5c2ffd640 1 -- 172.21.15.62:0/3712342339 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) ==== 138406+0+0 (secure 0 0 0) 0x7fc5c4042030 con 0x7fc5d0102790 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5da269740 1 Processor -- start 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5da269740 1 -- start start 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5da269740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af07940cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5da269740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5da269740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 0x55af079523b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5da269740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55af078a7090 con 0x55af079411f0 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5da269740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55af0784ea70 con 0x55af079408d0 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5da269740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55af078a3050 con 0x55af0794ffc0 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5d5c2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5d5c2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45956/0 (socket says 172.21.15.62:45956) 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.157+0000 7fc5d5c2c640 1 -- 172.21.15.62:0/858189701 learned_addr learned my addr 172.21.15.62:0/858189701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d542b640 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 0x55af079523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d642d640 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af07940cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5c2ffd640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc5bc0a32d0 0x7fc5bc0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d5c2c640 1 -- 172.21.15.62:0/858189701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 msgr2=0x55af079523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 0x55af079523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d5c2c640 1 -- 172.21.15.62:0/858189701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 msgr2=0x55af07940cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:39.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af07940cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d5c2c640 1 -- 172.21.15.62:0/858189701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af0785b130 con 0x55af079411f0 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d642d640 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af07940cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d642d640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc5bc0a32d0 0x7fc5bc0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5c2ffd640 1 -- 172.21.15.62:0/3712342339 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fc5c40f03f0 con 0x7fc5d0102790 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794fa80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00b760 tx=0x7fc5cc00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d3a0f9868d9134 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5bbfff640 1 -- 172.21.15.62:0/858189701 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc003dd0 con 0x55af079411f0 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5bbfff640 1 -- 172.21.15.62:0/858189701 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5cc015070 con 0x55af079411f0 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.158+0000 7fc5bbfff640 1 -- 172.21.15.62:0/858189701 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc011440 con 0x55af079411f0 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5da269740 1 -- 172.21.15.62:0/858189701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 msgr2=0x55af0794fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5da269740 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794fa80 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00b760 tx=0x7fc5cc00bc20 comp rx=0 tx=0).stop 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5d642d640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc5bc0a32d0 0x7fc5bc0a5790 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8005fd0 tx=0x7fc5c8004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:39.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5da269740 1 -- 172.21.15.62:0/858189701 shutdown_connections 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5da269740 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 0x55af079523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5da269740 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794fa80 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5da269740 1 --2- 172.21.15.62:0/858189701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af07940cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5da269740 1 -- 172.21.15.62:0/858189701 >> 172.21.15.62:0/858189701 conn(0x55af0793cf60 msgr2=0x55af0793d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5da269740 1 -- 172.21.15.62:0/858189701 shutdown_connections 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.159+0000 7fc5da269740 1 -- 172.21.15.62:0/858189701 wait complete. 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5da269740 1 -- start start 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5da269740 1 Processor -- start 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5da269740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af0794f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5da269740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5da269740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 0x55af0794d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5da269740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55af07952bf0 con 0x55af079411f0 2024-09-15T23:15:39.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5da269740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55af07861f20 con 0x55af079408d0 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5da269740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55af07952d70 con 0x55af0794ffc0 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5da269740 10 client.0 _ll_register_callbacks cb 0x55af0782b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.160+0000 7fc5d542b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d542b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45964/0 (socket says 172.21.15.62:45964) 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d542b640 1 -- 172.21.15.62:0/4187785791 learned_addr learned my addr 172.21.15.62:0/4187785791 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d642d640 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 0x55af0794d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af0794f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d542b640 1 -- 172.21.15.62:0/4187785791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 msgr2=0x55af0794d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d542b640 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 0x55af0794d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d542b640 1 -- 172.21.15.62:0/4187785791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 msgr2=0x55af0794f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:39.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d542b640 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af0794f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d542b640 1 -- 172.21.15.62:0/4187785791 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af076182b0 con 0x55af079411f0 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d642d640 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 0x55af0794d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af0794f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5d542b640 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794cc40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40b7180 tx=0x7fc5c40b82e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb491c6c8cd11b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5b97fa640 1 -- 172.21.15.62:0/4187785791 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4049040 con 0x55af079411f0 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5b97fa640 1 -- 172.21.15.62:0/4187785791 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fc5c40edcb0 con 0x55af079411f0 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5da269740 1 -- 172.21.15.62:0/4187785791 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5cc015070 con 0x55af079411f0 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5b97fa640 1 -- 172.21.15.62:0/4187785791 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc5c40bc4b0 con 0x55af079411f0 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5da269740 10 client.15219 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.161+0000 7fc5da269740 1 -- 172.21.15.62:0/4187785791 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55af0795e5e0 con 0x55af079411f0 2024-09-15T23:15:39.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.162+0000 7fc5b97fa640 1 -- 172.21.15.62:0/4187785791 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5cc015070 con 0x55af079411f0 2024-09-15T23:15:39.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.162+0000 7fc5b97fa640 1 -- 172.21.15.62:0/4187785791 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fc5c40b8d50 con 0x55af079411f0 2024-09-15T23:15:39.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.162+0000 7fc5da269740 10 client.15219 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:39.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.162+0000 7fc5da269740 10 client.15219 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:15:39.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.162+0000 7fc5da269740 20 client.15219 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:39.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.162+0000 7fc5da269740 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55af07960c40 0x55af079620b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.162+0000 7fc5da269740 4 client.15219 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:15:39.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.162+0000 7fc5da269740 1 -- 172.21.15.62:0/4187785791 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af078b4600 con 0x55af07960c40 2024-09-15T23:15:39.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.162+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55af07960c40 0x55af079620b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.163+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55af07960c40 0x55af079620b0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1abf1a838608c2e server_cookie=e84e0323122b918a in_seq=0 out_seq=0 2024-09-15T23:15:39.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.163+0000 7fc5b97fa640 10 client.15219 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:39.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5b97fa640 1 -- 172.21.15.62:0/4187785791 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55af078b4600 con 0x55af07960c40 2024-09-15T23:15:39.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5b97fa640 10 client.15219 handle_command_reply: tid=0 2024-09-15T23:15:39.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 client.15219 shutdown 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 -- 172.21.15.62:0/4187785791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 msgr2=0x55af0794cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794cc40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc5c40b7180 tx=0x7fc5c40b82e0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 -- 172.21.15.62:0/4187785791 shutdown_connections 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55af07960c40 0x55af079620b0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af0794ffc0 0x55af0794d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079411f0 0x55af0794cc40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 --2- 172.21.15.62:0/4187785791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af0794f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5b97fa640 0 client.15219 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 -- 172.21.15.62:0/4187785791 >> 172.21.15.62:0/4187785791 conn(0x55af0793cf60 msgr2=0x55af07942c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:39.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 -- 172.21.15.62:0/4187785791 shutdown_connections 2024-09-15T23:15:39.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 1 -- 172.21.15.62:0/4187785791 wait complete. 2024-09-15T23:15:39.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.164+0000 7fc5da269740 20 client.15219 trim_cache size 0 max 16384 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5da269740 1 Processor -- start 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5da269740 1 -- start start 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5da269740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079408d0 0x55af07960570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5da269740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 0x55af079620b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5da269740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af0794ffc0 0x55af0794c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5da269740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55af079603f0 con 0x55af079408d0 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5da269740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55af07960ac0 con 0x55af0794ffc0 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5da269740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55af07961190 con 0x55af079411f0 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d5c2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079408d0 0x55af07960570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d5c2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079408d0 0x55af07960570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45968/0 (socket says 172.21.15.62:45968) 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d5c2c640 1 -- 172.21.15.62:0/3394481313 learned_addr learned my addr 172.21.15.62:0/3394481313 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d642d640 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af0794ffc0 0x55af0794c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d5c2c640 1 -- 172.21.15.62:0/3394481313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 msgr2=0x55af079620b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d542b640 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 0x55af079620b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 0x55af079620b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d5c2c640 1 -- 172.21.15.62:0/3394481313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af0794ffc0 msgr2=0x55af0794c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af0794ffc0 0x55af0794c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.182+0000 7fc5d5c2c640 1 -- 172.21.15.62:0/3394481313 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af0794fc60 con 0x55af079408d0 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5d542b640 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 0x55af079620b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079408d0 0x55af07960570 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc004a90 tx=0x7fc5cc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9d41307bd201c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:39.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5bb7fe640 1 -- 172.21.15.62:0/3394481313 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc003bd0 con 0x55af079408d0 2024-09-15T23:15:39.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5bb7fe640 1 -- 172.21.15.62:0/3394481313 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5cc015030 con 0x55af079408d0 2024-09-15T23:15:39.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5bb7fe640 1 -- 172.21.15.62:0/3394481313 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc019680 con 0x55af079408d0 2024-09-15T23:15:39.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5da269740 1 -- 172.21.15.62:0/3394481313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079408d0 msgr2=0x55af07960570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5da269740 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079408d0 0x55af07960570 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc004a90 tx=0x7fc5cc0039c0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5da269740 1 -- 172.21.15.62:0/3394481313 shutdown_connections 2024-09-15T23:15:39.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5da269740 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af0794ffc0 0x55af0794c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5da269740 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 0x55af079620b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5da269740 1 --2- 172.21.15.62:0/3394481313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af079408d0 0x55af07960570 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.183+0000 7fc5da269740 1 -- 172.21.15.62:0/3394481313 >> 172.21.15.62:0/3394481313 conn(0x55af0793cf60 msgr2=0x55af0793d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:39.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 -- 172.21.15.62:0/3394481313 shutdown_connections 2024-09-15T23:15:39.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 -- 172.21.15.62:0/3394481313 wait complete. 2024-09-15T23:15:39.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 -- start start 2024-09-15T23:15:39.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 Processor -- start 2024-09-15T23:15:39.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af07963660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 0x55af07963ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af0794ffc0 0x55af079c9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55af07961f30 con 0x55af0794ffc0 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55af07961860 con 0x55af079408d0 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55af07962600 con 0x55af079411f0 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.184+0000 7fc5da269740 10 client.0 _ll_register_callbacks cb 0x7fc5c4005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d642d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af0794ffc0 0x55af079c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d642d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af0794ffc0 0x55af079c9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45972/0 (socket says 172.21.15.62:45972) 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d5c2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af07963660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d642d640 1 -- 172.21.15.62:0/2765996828 learned_addr learned my addr 172.21.15.62:0/2765996828 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d542b640 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 0x55af07963ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d642d640 1 -- 172.21.15.62:0/2765996828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 msgr2=0x55af07963ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d642d640 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 0x55af07963ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d642d640 1 -- 172.21.15.62:0/2765996828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 msgr2=0x55af07963660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d642d640 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af07963660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d642d640 1 -- 172.21.15.62:0/2765996828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af079c9880 con 0x55af0794ffc0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5d642d640 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af0794ffc0 0x55af079c9340 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc5c801e890 tx=0x7fc5c8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22b08f36607a2c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5b97fa640 1 -- 172.21.15.62:0/2765996828 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c802d060 con 0x55af0794ffc0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5da269740 1 -- 172.21.15.62:0/2765996828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5cc015030 con 0x55af0794ffc0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5b97fa640 1 -- 172.21.15.62:0/2765996828 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7fc5c800ee00 con 0x55af0794ffc0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.185+0000 7fc5b97fa640 1 -- 172.21.15.62:0/2765996828 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc5c8004b20 con 0x55af0794ffc0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5da269740 10 client.15225 fetch_fsmap learned FSMap version 24 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5da269740 1 -- 172.21.15.62:0/2765996828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55af079caa30 con 0x55af0794ffc0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5b97fa640 1 -- 172.21.15.62:0/2765996828 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5cc015030 con 0x55af0794ffc0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5b97fa640 1 -- 172.21.15.62:0/2765996828 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fc5c8027070 con 0x55af0794ffc0 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5da269740 10 client.15225 fetch_fsmap finished waiting for FSMap version 24 2024-09-15T23:15:39.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5da269740 10 client.15225 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5da269740 20 client.15225 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5da269740 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55af079cf690 0x55af079ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5da269740 4 client.15225 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5da269740 1 -- 172.21.15.62:0/2765996828 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "status"}) -- 0x55af078aeb10 con 0x55af079cf690 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.186+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55af079cf690 0x55af079ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5d5c2c640 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55af079cf690 0x55af079ce3a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fafd9a22d3324ab5 server_cookie=f5bb3ee7d954b542 in_seq=0 out_seq=0 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5b97fa640 10 client.15225 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5b97fa640 1 -- 172.21.15.62:0/2765996828 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55af078aeb10 con 0x55af079cf690 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5b97fa640 10 client.15225 handle_command_reply: tid=0 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 client.15225 shutdown 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 -- 172.21.15.62:0/2765996828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af0794ffc0 msgr2=0x55af079c9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af0794ffc0 0x55af079c9340 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc5c801e890 tx=0x7fc5c8004ab0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 -- 172.21.15.62:0/2765996828 shutdown_connections 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55af079cf690 0x55af079ce3a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55af0794ffc0 0x55af079c9340 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5b97fa640 0 client.15225 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55af079411f0 0x55af07963ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 --2- 172.21.15.62:0/2765996828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55af079408d0 0x55af07963660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 -- 172.21.15.62:0/2765996828 >> 172.21.15.62:0/2765996828 conn(0x55af0793cf60 msgr2=0x55af0794b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 -- 172.21.15.62:0/2765996828 shutdown_connections 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 1 -- 172.21.15.62:0/2765996828 wait complete. 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.187+0000 7fc5da269740 20 client.15225 trim_cache size 0 max 16384 2024-09-15T23:15:39.196 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24497, 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 125.68131422499999, 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 24, 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 90, 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 88, 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 136.82997014899999 2024-09-15T23:15:39.197 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:15:39.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 -- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc5bc0a32d0 msgr2=0x7fc5bc0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc5bc0a32d0 0x7fc5bc0a5790 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8005fd0 tx=0x7fc5c8004a40 comp rx=0 tx=0).stop 2024-09-15T23:15:39.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 -- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 msgr2=0x7fc5d01989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:39.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 0x7fc5d01989d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4006010 tx=0x7fc5c40042f0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 -- 172.21.15.62:0/3712342339 shutdown_connections 2024-09-15T23:15:39.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc5bc0a32d0 0x7fc5bc0a5790 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc5d0106f10 0x7fc5d019cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc5d0102790 0x7fc5d01989d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 --2- 172.21.15.62:0/3712342339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5d0101de0 0x7fc5d0198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:39.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 -- 172.21.15.62:0/3712342339 >> 172.21.15.62:0/3712342339 conn(0x7fc5d00fd8c0 msgr2=0x7fc5d00fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:39.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 -- 172.21.15.62:0/3712342339 shutdown_connections 2024-09-15T23:15:39.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:39.188+0000 7fc5d7e90640 1 -- 172.21.15.62:0/3712342339 wait complete. 2024-09-15T23:15:39.950 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:39 smithi062 ceph-mon[30949]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:39.980 INFO:teuthology.run_tasks:Running task sequential... 2024-09-15T23:15:39.990 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-15T23:15:39.990 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-15T23:15:39.991 INFO:teuthology.task.print:Enabling mgr modules 2024-09-15T23:15:39.991 INFO:teuthology.run_tasks:Running task kclient... 2024-09-15T23:15:40.047 INFO:tasks.kclient:Mounting kernel clients... 2024-09-15T23:15:40.047 INFO:tasks.kclient:config is {'client.0': {}} 2024-09-15T23:15:40.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:40.048 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-09-15T23:15:40.049 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:39 smithi148 ceph-mon[35610]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:40.049 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:39 smithi067 ceph-mon[35665]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-09-15T23:15:40.066 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-15T23:15:40.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:40.066 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-15T23:15:40.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:40.134 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link delete ceph-brx 2024-09-15T23:15:40.251 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v1', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-09-15T23:15:40.251 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-15T23:15:40.251 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-09-15T23:15:40.251 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-09-15T23:15:40.251 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-15T23:15:40.518 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:41.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 -- 172.21.15.62:0/95543015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102ef0 msgr2=0x7eff5010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 --2- 172.21.15.62:0/95543015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102ef0 0x7eff5010da40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7eff40009900 tx=0x7eff4002a480 comp rx=0 tx=0).stop 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 -- 172.21.15.62:0/95543015 shutdown_connections 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 --2- 172.21.15.62:0/95543015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102ef0 0x7eff5010da40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 --2- 172.21.15.62:0/95543015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff50102570 0x7eff501029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 --2- 172.21.15.62:0/95543015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff50100e20 0x7eff50101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 -- 172.21.15.62:0/95543015 >> 172.21.15.62:0/95543015 conn(0x7eff500fc7d0 msgr2=0x7eff500febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 -- 172.21.15.62:0/95543015 shutdown_connections 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 -- 172.21.15.62:0/95543015 wait complete. 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.012+0000 7eff56cb2640 1 Processor -- start 2024-09-15T23:15:41.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff56cb2640 1 -- start start 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff56cb2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff50100e20 0x7eff501992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff56cb2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102570 0x7eff501997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff56cb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff50102ef0 0x7eff501a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff56cb2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7eff50110f80 con 0x7eff50102570 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff56cb2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7eff50110e00 con 0x7eff50102ef0 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff56cb2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7eff50111100 con 0x7eff50100e20 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff4ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102570 0x7eff501997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff4ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102570 0x7eff501997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45990/0 (socket says 172.21.15.62:45990) 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff4ffff640 1 -- 172.21.15.62:0/1883245614 learned_addr learned my addr 172.21.15.62:0/1883245614 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff5524f640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff50102ef0 0x7eff501a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff54a4e640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff50100e20 0x7eff501992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:41.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff4ffff640 1 -- 172.21.15.62:0/1883245614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff50100e20 msgr2=0x7eff501992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff4ffff640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff50100e20 0x7eff501992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff4ffff640 1 -- 172.21.15.62:0/1883245614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff50102ef0 msgr2=0x7eff501a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff4ffff640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff50102ef0 0x7eff501a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.013+0000 7eff4ffff640 1 -- 172.21.15.62:0/1883245614 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff501a4870 con 0x7eff50102570 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.014+0000 7eff4ffff640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102570 0x7eff501997f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7eff4400c890 tx=0x7eff4400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.014+0000 7eff4dffb640 1 -- 172.21.15.62:0/1883245614 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff44004220 con 0x7eff50102570 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.014+0000 7eff4dffb640 1 -- 172.21.15.62:0/1883245614 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff440043c0 con 0x7eff50102570 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.014+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff501a4a40 con 0x7eff50102570 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.014+0000 7eff4dffb640 1 -- 172.21.15.62:0/1883245614 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff440049e0 con 0x7eff50102570 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.014+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff501a4f50 con 0x7eff50102570 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.015+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff50101200 con 0x7eff50102570 2024-09-15T23:15:41.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.016+0000 7eff4dffb640 1 -- 172.21.15.62:0/1883245614 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7eff440079b0 con 0x7eff50102570 2024-09-15T23:15:41.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.017+0000 7eff4dffb640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7eff380a3240 0x7eff380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:41.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.017+0000 7eff4dffb640 1 -- 172.21.15.62:0/1883245614 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9071+0+0 (secure 0 0 0) 0x7eff440cfd20 con 0x7eff50102570 2024-09-15T23:15:41.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.017+0000 7eff54a4e640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7eff380a3240 0x7eff380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:41.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.018+0000 7eff54a4e640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7eff380a3240 0x7eff380a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7eff48005fd0 tx=0x7eff48009870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:41.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.020+0000 7eff4dffb640 1 -- 172.21.15.62:0/1883245614 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7eff44014030 con 0x7eff50102570 2024-09-15T23:15:41.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.236+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7eff5006a560 con 0x7eff50102570 2024-09-15T23:15:41.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.236+0000 7eff4dffb640 1 -- 172.21.15.62:0/1883245614 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7eff5006a560 con 0x7eff50102570 2024-09-15T23:15:41.241 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-15T23:15:41.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7eff380a3240 msgr2=0x7eff380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:41.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7eff380a3240 0x7eff380a5700 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7eff48005fd0 tx=0x7eff48009870 comp rx=0 tx=0).stop 2024-09-15T23:15:41.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102570 msgr2=0x7eff501997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:41.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102570 0x7eff501997f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7eff4400c890 tx=0x7eff4400cd50 comp rx=0 tx=0).stop 2024-09-15T23:15:41.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 shutdown_connections 2024-09-15T23:15:41.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7eff380a3240 0x7eff380a5700 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:41.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff50102ef0 0x7eff501a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:41.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff50102570 0x7eff501997f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:41.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 --2- 172.21.15.62:0/1883245614 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7eff50100e20 0x7eff501992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:41.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 >> 172.21.15.62:0/1883245614 conn(0x7eff500fc7d0 msgr2=0x7eff50106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:41.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 shutdown_connections 2024-09-15T23:15:41.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:41.239+0000 7eff56cb2640 1 -- 172.21.15.62:0/1883245614 wait complete. 2024-09-15T23:15:41.933 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:41 smithi062 ceph-mon[30949]: pgmap v350: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:41.933 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:41 smithi062 ceph-mon[30949]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-15T23:15:41.933 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-09-15T23:15:41.934 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-09-15T23:15:41.934 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:41 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1883245614' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:15:41.966 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-15T23:15:41.966 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-15T23:15:41.966 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-09-15T23:15:41.966 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-15T23:15:41.966 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-09-15T23:15:41.967 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-15T23:15:41.967 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-15T23:15:41.967 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-15T23:15:41.967 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-09-15T23:15:41.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:41.967 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-09-15T23:15:41.986 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-15T23:15:41.987 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:15:41.988 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-15T23:15:41.988 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-09-15T23:15:41.988 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-15T23:15:41.988 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42191sec preferred_lft 42191sec 2024-09-15T23:15:41.988 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link noprefixroute 2024-09-15T23:15:41.988 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-09-15T23:15:41.988 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-15T23:15:41.988 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:15:41.989 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:15:41.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add name ceph-brx type bridge 2024-09-15T23:15:41.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr flush dev ceph-brx 2024-09-15T23:15:41.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set ceph-brx up 2024-09-15T23:15:41.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-15T23:15:41.989 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:15:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:41 smithi067 ceph-mon[35665]: pgmap v350: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:41 smithi067 ceph-mon[35665]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-15T23:15:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-09-15T23:15:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-09-15T23:15:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:41 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1883245614' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:15:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:41 smithi148 ceph-mon[35610]: pgmap v350: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-09-15T23:15:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:41 smithi148 ceph-mon[35610]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-09-15T23:15:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2024-09-15T23:15:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-09-15T23:15:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:41 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1883245614' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-15T23:15:42.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:15:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:15:42.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:42.136 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-15T23:15:42.161 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-09-15T23:15:42.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:42.163 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-09-15T23:15:42.221 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-09-15T23:15:42.221 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-15T23:15:42.221 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-15T23:15:42.222 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-15T23:15:42.222 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:15:42.222 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:15:42.222 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-15T23:15:42.223 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-15T23:15:42.223 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-15T23:15:42.223 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:15:42.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:15:42.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:15:42.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:42.351 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-09-15T23:15:42.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:42.368 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list-id 2024-09-15T23:15:42.424 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:15:42.424 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:15:42.424 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns add ceph-ns--tmp-cephfs 2024-09-15T23:15:42.424 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-09-15T23:15:42.424 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:15:42.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:15:42.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:15:42.543 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-09-15T23:15:42.544 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:15:42.544 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:15:42.544 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-09-15T23:15:42.544 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-15T23:15:42.544 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-09-15T23:15:42.544 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-09-15T23:15:42.544 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-09-15T23:15:42.544 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:15:42.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:15:42.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:15:42.665 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-15T23:15:42.665 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-09-15T23:15:42.665 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-09-15T23:15:42.665 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set dev brx.0 master ceph-brx 2024-09-15T23:15:42.665 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-09-15T23:15:42.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:15:42.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:15:42.750 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T23:15:42.750 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /tmp/cephfs 2024-09-15T23:15:42.766 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/tmp/cephfs' 2024-09-15T23:15:42.766 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-15T23:15:42.766 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /tmp/cephfs 2024-09-15T23:15:42.822 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-09-15T23:15:42.823 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-09-15T23:15:42.823 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e 2024-09-15T23:15:42.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:42.823 DEBUG:teuthology.orchestra.run.smithi062:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,context=system_u:object_r:postgresql_db_t:s0 2024-09-15T23:15:42.885 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-09-15T23:15:42.885 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-09-15T23:15:42.885 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:42 smithi062 ceph-mon[30949]: osdmap e91: 12 total, 12 up, 12 in 2024-09-15T23:15:42.885 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:42 smithi062 ceph-mon[30949]: osdmap e92: 12 total, 12 up, 12 in 2024-09-15T23:15:42.907 INFO:teuthology.orchestra.run.smithi062.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0" 2024-09-15T23:15:42.908 INFO:teuthology.orchestra.run.smithi062.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-09-15T23:15:42.908 INFO:teuthology.orchestra.run.smithi062.stdout:invalid new device string format 2024-09-15T23:15:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-09-15T23:15:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-09-15T23:15:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:42 smithi067 ceph-mon[35665]: osdmap e91: 12 total, 12 up, 12 in 2024-09-15T23:15:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:42 smithi067 ceph-mon[35665]: osdmap e92: 12 total, 12 up, 12 in 2024-09-15T23:15:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2024-09-15T23:15:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-09-15T23:15:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:42 smithi148 ceph-mon[35610]: osdmap e91: 12 total, 12 up, 12 in 2024-09-15T23:15:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:42 smithi148 ceph-mon[35610]: osdmap e92: 12 total, 12 up, 12 in 2024-09-15T23:15:43.051 INFO:teuthology.orchestra.run.smithi062.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0" 2024-09-15T23:15:43.051 INFO:teuthology.orchestra.run.smithi062.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-09-15T23:15:43.051 INFO:teuthology.orchestra.run.smithi062.stdout:invalid new device string format 2024-09-15T23:15:43.051 INFO:teuthology.orchestra.run.smithi062.stdout:mount.ceph: resolved to: "172.21.15.62:3300,172.21.15.67:3300,172.21.15.148:3300" 2024-09-15T23:15:43.051 INFO:teuthology.orchestra.run.smithi062.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.62:3300,172.21.15.67:3300,172.21.15.148:3300:/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e 2024-09-15T23:15:43.051 INFO:teuthology.orchestra.run.smithi062.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,context="system_u:object_r:postgresql_db_t:s0",key=0,fsid=ee7f888e-73b6-11ef-bceb-c7b262605968" will pass to kernel 2024-09-15T23:15:43.054 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-15T23:15:43.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:15:43.054 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /tmp/cephfs 2024-09-15T23:15:43.121 DEBUG:teuthology.orchestra.run.smithi062:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-15T23:15:43.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T23:15:43.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:15:43.218 INFO:teuthology.orchestra.run.smithi062.stdout:client24941 2024-09-15T23:15:43.219 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:15:43.219 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:15:43.237 DEBUG:teuthology.orchestra.run.smithi062:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-15T23:15:43.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T23:15:43.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:15:43.334 INFO:teuthology.orchestra.run.smithi062.stdout:client24941 2024-09-15T23:15:43.335 DEBUG:teuthology.orchestra.run.smithi062:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-15T23:15:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-15T23:15:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:15:43.392 INFO:teuthology.orchestra.run.smithi062.stdout:client24941 2024-09-15T23:15:43.393 DEBUG:teuthology.orchestra.run.smithi062:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968.client24941/status') 2024-09-15T23:15:43.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-15T23:15:43.452 INFO:teuthology.orchestra.run.smithi062.stdout:instance: client.24941 (3)172.21.15.62:0/1950023177 2024-09-15T23:15:43.452 INFO:teuthology.orchestra.run.smithi062.stdout:blocklisted: false 2024-09-15T23:15:43.452 INFO:teuthology.orchestra.run.smithi062.stderr:0+1 records in 2024-09-15T23:15:43.453 INFO:teuthology.orchestra.run.smithi062.stderr:0+1 records out 2024-09-15T23:15:43.453 INFO:teuthology.orchestra.run.smithi062.stderr:71 bytes copied, 2.5585e-05 s, 2.8 MB/s 2024-09-15T23:15:43.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-15T23:15:43.460 INFO:teuthology.run_tasks:Running task exec... 2024-09-15T23:15:43.470 INFO:teuthology.task.exec:Executing custom commands... 2024-09-15T23:15:43.470 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T23:15:43.470 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-09-15T23:15:43.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.574+0000 7ff73cd4b640 1 -- 172.21.15.62:0/138331759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7381025d0 msgr2=0x7ff7381049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:43.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.574+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/138331759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7381025d0 0x7ff7381049c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff728009900 tx=0x7ff72802a410 comp rx=0 tx=0).stop 2024-09-15T23:15:43.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.574+0000 7ff73cd4b640 1 -- 172.21.15.62:0/138331759 shutdown_connections 2024-09-15T23:15:43.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.574+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/138331759 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff738107830 0x7ff738109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:43.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.574+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/138331759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff738104f00 0x7ff7381072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:43.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.574+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/138331759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7381025d0 0x7ff7381049c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:43.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.574+0000 7ff73cd4b640 1 -- 172.21.15.62:0/138331759 >> 172.21.15.62:0/138331759 conn(0x7ff7380fc1b0 msgr2=0x7ff7380fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:43.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.574+0000 7ff73cd4b640 1 -- 172.21.15.62:0/138331759 shutdown_connections 2024-09-15T23:15:43.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.574+0000 7ff73cd4b640 1 -- 172.21.15.62:0/138331759 wait complete. 2024-09-15T23:15:43.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff73cd4b640 1 Processor -- start 2024-09-15T23:15:43.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff73cd4b640 1 -- start start 2024-09-15T23:15:43.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff73cd4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7381025d0 0x7ff738197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:43.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff73cd4b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff738104f00 0x7ff738198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:43.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff73cd4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff738107830 0x7ff73819f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff73cd4b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff73810d840 con 0x7ff738104f00 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff73cd4b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff73810d6c0 con 0x7ff738107830 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff73cd4b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff73810d9c0 con 0x7ff7381025d0 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff735d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff738104f00 0x7ff738198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff735d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff738104f00 0x7ff738198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44654/0 (socket says 172.21.15.62:44654) 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff735d9b640 1 -- 172.21.15.62:0/3759226342 learned_addr learned my addr 172.21.15.62:0/3759226342 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.575+0000 7ff73659c640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7381025d0 0x7ff738197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff736d9d640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff738107830 0x7ff73819f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff735d9b640 1 -- 172.21.15.62:0/3759226342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7381025d0 msgr2=0x7ff738197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff735d9b640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7381025d0 0x7ff738197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff735d9b640 1 -- 172.21.15.62:0/3759226342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff738107830 msgr2=0x7ff73819f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff735d9b640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff738107830 0x7ff73819f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:43.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff735d9b640 1 -- 172.21.15.62:0/3759226342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73819fa60 con 0x7ff738104f00 2024-09-15T23:15:43.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff735d9b640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff738104f00 0x7ff738198380 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00c970 tx=0x7ff72c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:43.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff7277fe640 1 -- 172.21.15.62:0/3759226342 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c007bb0 con 0x7ff738104f00 2024-09-15T23:15:43.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff73819fd50 con 0x7ff738104f00 2024-09-15T23:15:43.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff7277fe640 1 -- 172.21.15.62:0/3759226342 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff72c007d50 con 0x7ff738104f00 2024-09-15T23:15:43.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff7277fe640 1 -- 172.21.15.62:0/3759226342 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c00f450 con 0x7ff738104f00 2024-09-15T23:15:43.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.576+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7381a0260 con 0x7ff738104f00 2024-09-15T23:15:43.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.577+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7380fdca0 con 0x7ff738104f00 2024-09-15T23:15:43.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.578+0000 7ff7277fe640 1 -- 172.21.15.62:0/3759226342 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff72c004f90 con 0x7ff738104f00 2024-09-15T23:15:43.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.579+0000 7ff7277fe640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7200a32f0 0x7ff7200a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:43.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.579+0000 7ff73659c640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7200a32f0 0x7ff7200a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:43.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.579+0000 7ff7277fe640 1 -- 172.21.15.62:0/3759226342 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff72c0cf9e0 con 0x7ff738104f00 2024-09-15T23:15:43.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.580+0000 7ff73659c640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7200a32f0 0x7ff7200a57b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff7280314e0 tx=0x7ff728000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:43.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.583+0000 7ff7277fe640 1 -- 172.21.15.62:0/3759226342 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff72c014030 con 0x7ff738104f00 2024-09-15T23:15:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:43 smithi062 ceph-mon[30949]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-15T23:15:43.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:43.797+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7ff738062e90 con 0x7ff738104f00 2024-09-15T23:15:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:43 smithi067 ceph-mon[35665]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-15T23:15:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:43 smithi148 ceph-mon[35610]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-15T23:15:44.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.626+0000 7ff7277fe640 1 -- 172.21.15.62:0/3759226342 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7ff72c09c380 con 0x7ff738104f00 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7200a32f0 msgr2=0x7ff7200a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7200a32f0 0x7ff7200a57b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff7280314e0 tx=0x7ff728000bc0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff738104f00 msgr2=0x7ff738198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff738104f00 0x7ff738198380 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00c970 tx=0x7ff72c00ce30 comp rx=0 tx=0).stop 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 shutdown_connections 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7200a32f0 0x7ff7200a57b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff738107830 0x7ff73819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff738104f00 0x7ff738198380 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 --2- 172.21.15.62:0/3759226342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7381025d0 0x7ff738197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 >> 172.21.15.62:0/3759226342 conn(0x7ff7380fc1b0 msgr2=0x7ff738102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:44.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 shutdown_connections 2024-09-15T23:15:44.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.629+0000 7ff73cd4b640 1 -- 172.21.15.62:0/3759226342 wait complete. 2024-09-15T23:15:44.642 INFO:teuthology.run_tasks:Running task check-counter... 2024-09-15T23:15:44.653 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-09-15T23:15:44.680 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T23:15:44.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.787+0000 7f1430435640 1 -- 172.21.15.62:0/582987501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428100a30 msgr2=0x7f1428100e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:44.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.787+0000 7f1430435640 1 --2- 172.21.15.62:0/582987501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428100a30 0x7f1428100e30 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1424009900 tx=0x7f142402a410 comp rx=0 tx=0).stop 2024-09-15T23:15:44.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 -- 172.21.15.62:0/582987501 shutdown_connections 2024-09-15T23:15:44.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 --2- 172.21.15.62:0/582987501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281025e0 0x7f1428108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 --2- 172.21.15.62:0/582987501 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1428101c20 0x7f14281020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 --2- 172.21.15.62:0/582987501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428100a30 0x7f1428100e30 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 -- 172.21.15.62:0/582987501 >> 172.21.15.62:0/582987501 conn(0x7f14280fc1b0 msgr2=0x7f14280fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:44.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 -- 172.21.15.62:0/582987501 shutdown_connections 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 -- 172.21.15.62:0/582987501 wait complete. 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 Processor -- start 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 -- start start 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1428100a30 0x7f142806af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428101c20 0x7f142819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281025e0 0x7f142819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f142810c700 con 0x7f1428101c20 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f142810c580 con 0x7f14281025e0 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f1430435640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f142810c880 con 0x7f1428100a30 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.788+0000 7f142d9d0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428101c20 0x7f142819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142d9d0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428101c20 0x7f142819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44672/0 (socket says 172.21.15.62:44672) 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142d9d0640 1 -- 172.21.15.62:0/3807868476 learned_addr learned my addr 172.21.15.62:0/3807868476 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:44.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142e9d2640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281025e0 0x7f142819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142e1d1640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1428100a30 0x7f142806af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142d9d0640 1 -- 172.21.15.62:0/3807868476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1428100a30 msgr2=0x7f142806af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142d9d0640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1428100a30 0x7f142806af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142d9d0640 1 -- 172.21.15.62:0/3807868476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281025e0 msgr2=0x7f142819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142d9d0640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281025e0 0x7f142819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142d9d0640 1 -- 172.21.15.62:0/3807868476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142819faa0 con 0x7f1428101c20 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142e9d2640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281025e0 0x7f142819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142e1d1640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1428100a30 0x7f142806af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f142d9d0640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428101c20 0x7f142819cb70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f141800e9f0 tx=0x7f141800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f141f7fe640 1 -- 172.21.15.62:0/3807868476 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141800cd80 con 0x7f1428101c20 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f141f7fe640 1 -- 172.21.15.62:0/3807868476 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1418005500 con 0x7f1428101c20 2024-09-15T23:15:44.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142819fd30 con 0x7f1428101c20 2024-09-15T23:15:44.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f141f7fe640 1 -- 172.21.15.62:0/3807868476 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418010690 con 0x7f1428101c20 2024-09-15T23:15:44.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.789+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14281a01d0 con 0x7f1428101c20 2024-09-15T23:15:44.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.790+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1428107dd0 con 0x7f1428101c20 2024-09-15T23:15:44.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.791+0000 7f141f7fe640 1 -- 172.21.15.62:0/3807868476 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f14180046c0 con 0x7f1428101c20 2024-09-15T23:15:44.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.792+0000 7f141f7fe640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14140a3250 0x7f14140a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:44.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.793+0000 7f142e1d1640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14140a3250 0x7f14140a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:44.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.793+0000 7f141f7fe640 1 -- 172.21.15.62:0/3807868476 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f14180d03f0 con 0x7f1428101c20 2024-09-15T23:15:44.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.793+0000 7f142e1d1640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14140a3250 0x7f14140a5710 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f142402a930 tx=0x7f1424000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:44.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.796+0000 7f141f7fe640 1 -- 172.21.15.62:0/3807868476 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f14180541b0 con 0x7f1428101c20 2024-09-15T23:15:44.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.972+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f142802da10 con 0x7f1428101c20 2024-09-15T23:15:44.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.974+0000 7f141f7fe640 1 -- 172.21.15.62:0/3807868476 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24777 (secure 0 0 0) 0x7f141809cd90 con 0x7f1428101c20 2024-09-15T23:15:44.975 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:44.975 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":92,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:15:42.605470+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:15:44.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14140a3250 msgr2=0x7f14140a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:44.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14140a3250 0x7f14140a5710 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f142402a930 tx=0x7f1424000bc0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428101c20 msgr2=0x7f142819cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:44.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428101c20 0x7f142819cb70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f141800e9f0 tx=0x7f141800edf0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 shutdown_connections 2024-09-15T23:15:44.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14140a3250 0x7f14140a5710 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14281025e0 0x7f142819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1428101c20 0x7f142819cb70 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 --2- 172.21.15.62:0/3807868476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1428100a30 0x7f142806af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:44.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 >> 172.21.15.62:0/3807868476 conn(0x7f14280fc1b0 msgr2=0x7f1428104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:44.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 shutdown_connections 2024-09-15T23:15:44.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:44.976+0000 7f1430435640 1 -- 172.21.15.62:0/3807868476 wait complete. 2024-09-15T23:15:44.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:44 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3759226342' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-09-15T23:15:44.988 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T23:08:54.129326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-15T23:13:29.172210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-15T23:13:31.408446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-15T23:13:37.961318+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-15T23:15:44.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T23:15:45.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:44 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3759226342' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-09-15T23:15:45.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:44 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3759226342' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-09-15T23:15:45.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.094+0000 7ffaec518640 1 -- 172.21.15.62:0/3498333041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4104ee0 msgr2=0x7ffae41072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.094+0000 7ffaec518640 1 --2- 172.21.15.62:0/3498333041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4104ee0 0x7ffae41072d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ffad8009900 tx=0x7ffad802a410 comp rx=0 tx=0).stop 2024-09-15T23:15:45.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.094+0000 7ffaec518640 1 -- 172.21.15.62:0/3498333041 shutdown_connections 2024-09-15T23:15:45.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.094+0000 7ffaec518640 1 --2- 172.21.15.62:0/3498333041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffae4107810 0x7ffae4109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.094+0000 7ffaec518640 1 --2- 172.21.15.62:0/3498333041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4104ee0 0x7ffae41072d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.094+0000 7ffaec518640 1 --2- 172.21.15.62:0/3498333041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffae41025b0 0x7ffae41049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.094+0000 7ffaec518640 1 -- 172.21.15.62:0/3498333041 >> 172.21.15.62:0/3498333041 conn(0x7ffae40fc1b0 msgr2=0x7ffae40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:45.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.094+0000 7ffaec518640 1 -- 172.21.15.62:0/3498333041 shutdown_connections 2024-09-15T23:15:45.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.094+0000 7ffaec518640 1 -- 172.21.15.62:0/3498333041 wait complete. 2024-09-15T23:15:45.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaec518640 1 Processor -- start 2024-09-15T23:15:45.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaec518640 1 -- start start 2024-09-15T23:15:45.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaec518640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffae41025b0 0x7ffae4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaec518640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffae4104ee0 0x7ffae41984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaec518640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4107810 0x7ffae419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaec518640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffae410d7f0 con 0x7ffae4107810 2024-09-15T23:15:45.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaec518640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffae410d670 con 0x7ffae4104ee0 2024-09-15T23:15:45.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaec518640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffae410d970 con 0x7ffae41025b0 2024-09-15T23:15:45.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaeaab5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4107810 0x7ffae419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaeaab5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4107810 0x7ffae419f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44684/0 (socket says 172.21.15.62:44684) 2024-09-15T23:15:45.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaeaab5640 1 -- 172.21.15.62:0/2742351913 learned_addr learned my addr 172.21.15.62:0/2742351913 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:45.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaea2b4640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffae41025b0 0x7ffae4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffae9ab3640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffae4104ee0 0x7ffae41984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaeaab5640 1 -- 172.21.15.62:0/2742351913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffae41025b0 msgr2=0x7ffae4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaeaab5640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffae41025b0 0x7ffae4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaeaab5640 1 -- 172.21.15.62:0/2742351913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffae4104ee0 msgr2=0x7ffae41984d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaeaab5640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffae4104ee0 0x7ffae41984d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffaeaab5640 1 -- 172.21.15.62:0/2742351913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae419fbb0 con 0x7ffae4107810 2024-09-15T23:15:45.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.095+0000 7ffae9ab3640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffae4104ee0 0x7ffae41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:45.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.096+0000 7ffaeaab5640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4107810 0x7ffae419f540 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ffad400cab0 tx=0x7ffad400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:45.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.096+0000 7ffae37fe640 1 -- 172.21.15.62:0/2742351913 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad400ae20 con 0x7ffae4107810 2024-09-15T23:15:45.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.096+0000 7ffae37fe640 1 -- 172.21.15.62:0/2742351913 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffad400f930 con 0x7ffae4107810 2024-09-15T23:15:45.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.096+0000 7ffae37fe640 1 -- 172.21.15.62:0/2742351913 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad4018b80 con 0x7ffae4107810 2024-09-15T23:15:45.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.096+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae419fea0 con 0x7ffae4107810 2024-09-15T23:15:45.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.096+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae41a0310 con 0x7ffae4107810 2024-09-15T23:15:45.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.097+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae40fdca0 con 0x7ffae4107810 2024-09-15T23:15:45.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.098+0000 7ffae37fe640 1 -- 172.21.15.62:0/2742351913 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ffad400fae0 con 0x7ffae4107810 2024-09-15T23:15:45.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.099+0000 7ffae37fe640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffacc0a3250 0x7ffacc0a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.099+0000 7ffae37fe640 1 -- 172.21.15.62:0/2742351913 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ffad40d04e0 con 0x7ffae4107810 2024-09-15T23:15:45.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.099+0000 7ffaea2b4640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffacc0a3250 0x7ffacc0a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.099+0000 7ffaea2b4640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffacc0a3250 0x7ffacc0a5710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00bbe0 tx=0x7ffadc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:45.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.102+0000 7ffae37fe640 1 -- 172.21.15.62:0/2742351913 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ffad4014030 con 0x7ffae4107810 2024-09-15T23:15:45.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.278+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffae402da10 con 0x7ffae4107810 2024-09-15T23:15:45.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.278+0000 7ffae37fe640 1 -- 172.21.15.62:0/2742351913 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7ffad409ce80 con 0x7ffae4107810 2024-09-15T23:15:45.279 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffacc0a3250 msgr2=0x7ffacc0a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffacc0a3250 0x7ffacc0a5710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00bbe0 tx=0x7ffadc00b390 comp rx=0 tx=0).stop 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4107810 msgr2=0x7ffae419f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4107810 0x7ffae419f540 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ffad400cab0 tx=0x7ffad400cf70 comp rx=0 tx=0).stop 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 shutdown_connections 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffacc0a3250 0x7ffacc0a5710 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffae4107810 0x7ffae419f540 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffae4104ee0 0x7ffae41984d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 --2- 172.21.15.62:0/2742351913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffae41025b0 0x7ffae4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.280+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 >> 172.21.15.62:0/2742351913 conn(0x7ffae40fc1b0 msgr2=0x7ffae4103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.281+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 shutdown_connections 2024-09-15T23:15:45.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.281+0000 7ffaec518640 1 -- 172.21.15.62:0/2742351913 wait complete. 2024-09-15T23:15:45.292 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 -- 172.21.15.62:0/1651114795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec101bf0 msgr2=0x7f14ec102070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 --2- 172.21.15.62:0/1651114795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec101bf0 0x7f14ec102070 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f14d4009900 tx=0x7f14d402a410 comp rx=0 tx=0).stop 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 -- 172.21.15.62:0/1651114795 shutdown_connections 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 --2- 172.21.15.62:0/1651114795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ec1025b0 0x7f14ec108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 --2- 172.21.15.62:0/1651114795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec101bf0 0x7f14ec102070 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 --2- 172.21.15.62:0/1651114795 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14ec100a00 0x7f14ec100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 -- 172.21.15.62:0/1651114795 >> 172.21.15.62:0/1651114795 conn(0x7f14ec0fc140 msgr2=0x7f14ec0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 -- 172.21.15.62:0/1651114795 shutdown_connections 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 -- 172.21.15.62:0/1651114795 wait complete. 2024-09-15T23:15:45.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 Processor -- start 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 -- start start 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec100a00 0x7f14ec196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ec101bf0 0x7f14ec1971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.398+0000 7f14f2eae640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14ec1025b0 0x7f14ec19e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f2eae640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec10c660 con 0x7f14ec100a00 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f2eae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec10c4e0 con 0x7f14ec101bf0 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f2eae640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec10c7e0 con 0x7f14ec1025b0 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f0c4a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec100a00 0x7f14ec196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f0c4a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec100a00 0x7f14ec196cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44702/0 (socket says 172.21.15.62:44702) 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f0c4a640 1 -- 172.21.15.62:0/2889015547 learned_addr learned my addr 172.21.15.62:0/2889015547 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f144b640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14ec1025b0 0x7f14ec19e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14ebfff640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ec101bf0 0x7f14ec1971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f0c4a640 1 -- 172.21.15.62:0/2889015547 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14ec1025b0 msgr2=0x7f14ec19e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f0c4a640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14ec1025b0 0x7f14ec19e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f0c4a640 1 -- 172.21.15.62:0/2889015547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ec101bf0 msgr2=0x7f14ec1971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f0c4a640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ec101bf0 0x7f14ec1971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f0c4a640 1 -- 172.21.15.62:0/2889015547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ec19e8f0 con 0x7f14ec100a00 2024-09-15T23:15:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f0c4a640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec100a00 0x7f14ec196cb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f14e400c970 tx=0x7f14e400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14e9ffb640 1 -- 172.21.15.62:0/2889015547 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e4007c10 con 0x7f14ec100a00 2024-09-15T23:15:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14ec19ebe0 con 0x7f14ec100a00 2024-09-15T23:15:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14e9ffb640 1 -- 172.21.15.62:0/2889015547 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14e4007db0 con 0x7f14ec100a00 2024-09-15T23:15:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.399+0000 7f14e9ffb640 1 -- 172.21.15.62:0/2889015547 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e400f8a0 con 0x7f14ec100a00 2024-09-15T23:15:45.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.400+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14ec19f100 con 0x7f14ec100a00 2024-09-15T23:15:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.401+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14ec107e40 con 0x7f14ec100a00 2024-09-15T23:15:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.402+0000 7f14e9ffb640 1 -- 172.21.15.62:0/2889015547 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f14e400fa60 con 0x7f14ec100a00 2024-09-15T23:15:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.403+0000 7f14e9ffb640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14d80a3250 0x7f14d80a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.403+0000 7f14ebfff640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14d80a3250 0x7f14d80a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.403+0000 7f14e9ffb640 1 -- 172.21.15.62:0/2889015547 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f14e40cffe0 con 0x7f14ec100a00 2024-09-15T23:15:45.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.403+0000 7f14ebfff640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14d80a3250 0x7f14d80a5710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f14d4000c00 tx=0x7f14d40315d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:45.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.407+0000 7f14e9ffb640 1 -- 172.21.15.62:0/2889015547 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f14e409ca10 con 0x7f14ec100a00 2024-09-15T23:15:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.585+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f14ec062e90 con 0x7f14ec100a00 2024-09-15T23:15:45.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.585+0000 7f14e9ffb640 1 -- 172.21.15.62:0/2889015547 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f14e409c310 con 0x7f14ec100a00 2024-09-15T23:15:45.586 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-09-15T23:15:45.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14d80a3250 msgr2=0x7f14d80a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14d80a3250 0x7f14d80a5710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f14d4000c00 tx=0x7f14d40315d0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec100a00 msgr2=0x7f14ec196cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec100a00 0x7f14ec196cb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f14e400c970 tx=0x7f14e400ce30 comp rx=0 tx=0).stop 2024-09-15T23:15:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 shutdown_connections 2024-09-15T23:15:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14d80a3250 0x7f14d80a5710 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f14ec1025b0 0x7f14ec19e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ec101bf0 0x7f14ec1971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 --2- 172.21.15.62:0/2889015547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14ec100a00 0x7f14ec196cb0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 >> 172.21.15.62:0/2889015547 conn(0x7f14ec0fc140 msgr2=0x7f14ec105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 shutdown_connections 2024-09-15T23:15:45.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.587+0000 7f14f2eae640 1 -- 172.21.15.62:0/2889015547 wait complete. 2024-09-15T23:15:45.599 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-15T23:15:45.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 -- 172.21.15.62:0/1194215805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4100a00 msgr2=0x7f89c4100e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 --2- 172.21.15.62:0/1194215805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4100a00 0x7f89c4100e00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009900 tx=0x7f89b402a410 comp rx=0 tx=0).stop 2024-09-15T23:15:45.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 -- 172.21.15.62:0/1194215805 shutdown_connections 2024-09-15T23:15:45.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 --2- 172.21.15.62:0/1194215805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89c41025b0 0x7f89c4108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 --2- 172.21.15.62:0/1194215805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c4101bf0 0x7f89c4102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 --2- 172.21.15.62:0/1194215805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4100a00 0x7f89c4100e00 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 -- 172.21.15.62:0/1194215805 >> 172.21.15.62:0/1194215805 conn(0x7f89c40fc140 msgr2=0x7f89c40fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 -- 172.21.15.62:0/1194215805 shutdown_connections 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 -- 172.21.15.62:0/1194215805 wait complete. 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 Processor -- start 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.706+0000 7f89caf11640 1 -- start start 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89caf11640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c4100a00 0x7f89c4069ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89caf11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4101bf0 0x7f89c419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89caf11640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89c41025b0 0x7f89c419f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89caf11640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f89c410c680 con 0x7f89c4101bf0 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89caf11640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89c410c500 con 0x7f89c4100a00 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89caf11640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f89c410c800 con 0x7f89c41025b0 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4101bf0 0x7f89c419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4101bf0 0x7f89c419cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44718/0 (socket says 172.21.15.62:44718) 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c3fff640 1 -- 172.21.15.62:0/3427846329 learned_addr learned my addr 172.21.15.62:0/3427846329 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c94ae640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89c41025b0 0x7f89c419f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c8cad640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c4100a00 0x7f89c4069ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c3fff640 1 -- 172.21.15.62:0/3427846329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89c41025b0 msgr2=0x7f89c419f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c3fff640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89c41025b0 0x7f89c419f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c3fff640 1 -- 172.21.15.62:0/3427846329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c4100a00 msgr2=0x7f89c4069ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c3fff640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c4100a00 0x7f89c4069ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c3fff640 1 -- 172.21.15.62:0/3427846329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c419fb40 con 0x7f89c4101bf0 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c3fff640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4101bf0 0x7f89c419cc10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f89b800d950 tx=0x7f89b800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c1ffb640 1 -- 172.21.15.62:0/3427846329 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8017070 con 0x7f89c4101bf0 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c1ffb640 1 -- 172.21.15.62:0/3427846329 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89b8009c30 con 0x7f89c4101bf0 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c419fdd0 con 0x7f89c4101bf0 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.707+0000 7f89c1ffb640 1 -- 172.21.15.62:0/3427846329 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b800f6b0 con 0x7f89c4101bf0 2024-09-15T23:15:45.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.708+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c41a02e0 con 0x7f89c4101bf0 2024-09-15T23:15:45.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.709+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c4107e80 con 0x7f89c4101bf0 2024-09-15T23:15:45.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.710+0000 7f89c1ffb640 1 -- 172.21.15.62:0/3427846329 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f89b80109d0 con 0x7f89c4101bf0 2024-09-15T23:15:45.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.710+0000 7f89c1ffb640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f89ac0a32d0 0x7f89ac0a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:45.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.711+0000 7f89c8cad640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f89ac0a32d0 0x7f89ac0a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:45.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.711+0000 7f89c1ffb640 1 -- 172.21.15.62:0/3427846329 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f89b80cf750 con 0x7f89c4101bf0 2024-09-15T23:15:45.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.711+0000 7f89c8cad640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f89ac0a32d0 0x7f89ac0a5790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f89b4005ab0 tx=0x7f89b4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:45.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.715+0000 7f89c1ffb640 1 -- 172.21.15.62:0/3427846329 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f89b8009de0 con 0x7f89c4101bf0 2024-09-15T23:15:45.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.889+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f89c4062e90 con 0x7f89c4101bf0 2024-09-15T23:15:45.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.889+0000 7f89c1ffb640 1 -- 172.21.15.62:0/3427846329 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f89b809baa0 con 0x7f89c4101bf0 2024-09-15T23:15:45.890 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-09-15T23:15:45.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f89ac0a32d0 msgr2=0x7f89ac0a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f89ac0a32d0 0x7f89ac0a5790 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f89b4005ab0 tx=0x7f89b4000bc0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4101bf0 msgr2=0x7f89c419cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4101bf0 0x7f89c419cc10 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f89b800d950 tx=0x7f89b800de10 comp rx=0 tx=0).stop 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 shutdown_connections 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f89ac0a32d0 0x7f89ac0a5790 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89c41025b0 0x7f89c419f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f89c4101bf0 0x7f89c419cc10 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 --2- 172.21.15.62:0/3427846329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89c4100a00 0x7f89c4069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 >> 172.21.15.62:0/3427846329 conn(0x7f89c40fc140 msgr2=0x7f89c4105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 shutdown_connections 2024-09-15T23:15:45.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:45.891+0000 7f89caf11640 1 -- 172.21.15.62:0/3427846329 wait complete. 2024-09-15T23:15:45.902 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:45 smithi062 ceph-mon[30949]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 7 op/s 2024-09-15T23:15:45.902 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:45 smithi062 ceph-mon[30949]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-15T23:15:45.902 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:45 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3759226342' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-15T23:15:45.902 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:45 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:15:45.902 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:45 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3807868476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:45.903 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:45 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2742351913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:15:45.903 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:45 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2889015547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:15:45.904 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-15T23:15:46.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.010+0000 7f726b718640 1 -- 172.21.15.62:0/3490709143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 msgr2=0x7f7264109f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:46.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.010+0000 7f726b718640 1 --2- 172.21.15.62:0/3490709143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 0x7f7264109f60 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f725c009ca0 tx=0x7f725c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.010+0000 7f726b718640 1 -- 172.21.15.62:0/3490709143 shutdown_connections 2024-09-15T23:15:46.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.010+0000 7f726b718640 1 --2- 172.21.15.62:0/3490709143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 0x7f7264109f60 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.010+0000 7f726b718640 1 --2- 172.21.15.62:0/3490709143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7264102d10 0x7f7264103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.010+0000 7f726b718640 1 --2- 172.21.15.62:0/3490709143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7264101b20 0x7f7264101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.010+0000 7f726b718640 1 -- 172.21.15.62:0/3490709143 >> 172.21.15.62:0/3490709143 conn(0x7f72640fd2a0 msgr2=0x7f72640ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:46.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.010+0000 7f726b718640 1 -- 172.21.15.62:0/3490709143 shutdown_connections 2024-09-15T23:15:46.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.010+0000 7f726b718640 1 -- 172.21.15.62:0/3490709143 wait complete. 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f726b718640 1 Processor -- start 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f726b718640 1 -- start start 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f726b718640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7264101b20 0x7f7264197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f726b718640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7264102d10 0x7f72641982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f726b718640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 0x7f726419f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f726b718640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f726410d7a0 con 0x7f72641036d0 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f726b718640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f726410d620 con 0x7f7264101b20 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f726b718640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f726410d920 con 0x7f7264102d10 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f7269cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 0x7f726419f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f7269cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 0x7f726419f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44736/0 (socket says 172.21.15.62:44736) 2024-09-15T23:15:46.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f7269cb5640 1 -- 172.21.15.62:0/2998802135 learned_addr learned my addr 172.21.15.62:0/2998802135 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f7268cb3640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7264102d10 0x7f72641982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f72694b4640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7264101b20 0x7f7264197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f7269cb5640 1 -- 172.21.15.62:0/2998802135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7264102d10 msgr2=0x7f72641982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f7269cb5640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7264102d10 0x7f72641982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f7269cb5640 1 -- 172.21.15.62:0/2998802135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7264101b20 msgr2=0x7f7264197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f7269cb5640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7264101b20 0x7f7264197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.011+0000 7f7269cb5640 1 -- 172.21.15.62:0/2998802135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726419f980 con 0x7f72641036d0 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.012+0000 7f7269cb5640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 0x7f726419f310 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f725c005e30 tx=0x7f725c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.012+0000 7f725a7fc640 1 -- 172.21.15.62:0/2998802135 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c003c30 con 0x7f72641036d0 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.012+0000 7f725a7fc640 1 -- 172.21.15.62:0/2998802135 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f725c003dd0 con 0x7f72641036d0 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.012+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f726419fc10 con 0x7f72641036d0 2024-09-15T23:15:46.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.012+0000 7f725a7fc640 1 -- 172.21.15.62:0/2998802135 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c02fde0 con 0x7f72641036d0 2024-09-15T23:15:46.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.012+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72641a01b0 con 0x7f72641036d0 2024-09-15T23:15:46.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.013+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7264108e90 con 0x7f72641036d0 2024-09-15T23:15:46.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.014+0000 7f725a7fc640 1 -- 172.21.15.62:0/2998802135 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f725c004050 con 0x7f72641036d0 2024-09-15T23:15:46.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.015+0000 7f725a7fc640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72500a32d0 0x7f72500a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:46.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.015+0000 7f72694b4640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72500a32d0 0x7f72500a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:46.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.015+0000 7f725a7fc640 1 -- 172.21.15.62:0/2998802135 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f725c0f1230 con 0x7f72641036d0 2024-09-15T23:15:46.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.016+0000 7f72694b4640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72500a32d0 0x7f72500a5790 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f72600058c0 tx=0x7f7260005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:46.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.018+0000 7f725a7fc640 1 -- 172.21.15.62:0/2998802135 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f725c0bdb70 con 0x7f72641036d0 2024-09-15T23:15:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:45 smithi067 ceph-mon[35665]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 7 op/s 2024-09-15T23:15:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:45 smithi067 ceph-mon[35665]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-15T23:15:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:45 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3759226342' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-15T23:15:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:45 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:15:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:45 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3807868476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:45 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2742351913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:15:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:45 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2889015547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:15:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:45 smithi148 ceph-mon[35610]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 639 B/s wr, 7 op/s 2024-09-15T23:15:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:45 smithi148 ceph-mon[35610]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-15T23:15:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:45 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3759226342' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-09-15T23:15:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:45 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:15:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:45 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3807868476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:45 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2742351913' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:15:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:45 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2889015547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:15:46.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.195+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f726402da10 con 0x7f72641036d0 2024-09-15T23:15:46.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.195+0000 7f725a7fc640 1 -- 172.21.15.62:0/2998802135 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f725c0bd470 con 0x7f72641036d0 2024-09-15T23:15:46.197 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-09-15T23:15:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72500a32d0 msgr2=0x7f72500a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72500a32d0 0x7f72500a5790 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f72600058c0 tx=0x7f7260005850 comp rx=0 tx=0).stop 2024-09-15T23:15:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 msgr2=0x7f726419f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 0x7f726419f310 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f725c005e30 tx=0x7f725c031ee0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 shutdown_connections 2024-09-15T23:15:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72500a32d0 0x7f72500a5790 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72641036d0 0x7f726419f310 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7264102d10 0x7f72641982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 --2- 172.21.15.62:0/2998802135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7264101b20 0x7f7264197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 >> 172.21.15.62:0/2998802135 conn(0x7f72640fd2a0 msgr2=0x7f7264106060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:46.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 shutdown_connections 2024-09-15T23:15:46.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.198+0000 7f726b718640 1 -- 172.21.15.62:0/2998802135 wait complete. 2024-09-15T23:15:46.209 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-09-15T23:15:46.210 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:15:46.466 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:46.854 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:46 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3427846329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:15:46.855 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:46 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2998802135' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1663454849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc102ef0 msgr2=0x7fefcc10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1663454849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc102ef0 0x7fefcc10da40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fefc4009ca0 tx=0x7fefc402a950 comp rx=0 tx=0).stop 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1663454849 shutdown_connections 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1663454849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc102ef0 0x7fefcc10da40 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1663454849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc102570 0x7fefcc1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1663454849 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefcc100e20 0x7fefcc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1663454849 >> 172.21.15.62:0/1663454849 conn(0x7fefcc0fc7d0 msgr2=0x7fefcc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1663454849 shutdown_connections 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1663454849 wait complete. 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 Processor -- start 2024-09-15T23:15:46.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.937+0000 7fefd3f6b640 1 -- start start 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd3f6b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc100e20 0x7fefcc1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd3f6b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc102570 0x7fefcc198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd3f6b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefcc102ef0 0x7fefcc19cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd3f6b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc110f80 con 0x7fefcc100e20 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd3f6b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc110e00 con 0x7fefcc102570 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd3f6b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc111100 con 0x7fefcc102ef0 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1d07640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc100e20 0x7fefcc1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1d07640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc100e20 0x7fefcc1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44756/0 (socket says 172.21.15.62:44756) 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1d07640 1 -- 172.21.15.62:0/1244428398 learned_addr learned my addr 172.21.15.62:0/1244428398 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1506640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc102570 0x7fefcc198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd2508640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefcc102ef0 0x7fefcc19cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1d07640 1 -- 172.21.15.62:0/1244428398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefcc102ef0 msgr2=0x7fefcc19cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:46.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1d07640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefcc102ef0 0x7fefcc19cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1d07640 1 -- 172.21.15.62:0/1244428398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc102570 msgr2=0x7fefcc198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1d07640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc102570 0x7fefcc198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1d07640 1 -- 172.21.15.62:0/1244428398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc19d380 con 0x7fefcc100e20 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd2508640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefcc102ef0 0x7fefcc19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefd1d07640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc100e20 0x7fefcc1983d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fefc800c920 tx=0x7fefc800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.938+0000 7fefc2ffd640 1 -- 172.21.15.62:0/1244428398 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8007c70 con 0x7fefcc100e20 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.939+0000 7fefc2ffd640 1 -- 172.21.15.62:0/1244428398 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc8007e10 con 0x7fefcc100e20 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.939+0000 7fefc2ffd640 1 -- 172.21.15.62:0/1244428398 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8018890 con 0x7fefcc100e20 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.939+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefcc19d670 con 0x7fefcc100e20 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.939+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefcc1a4f50 con 0x7fefcc100e20 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.940+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefcc101200 con 0x7fefcc100e20 2024-09-15T23:15:46.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.941+0000 7fefc2ffd640 1 -- 172.21.15.62:0/1244428398 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fefc8018a70 con 0x7fefcc100e20 2024-09-15T23:15:46.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.941+0000 7fefc2ffd640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefb80a3240 0x7fefb80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:46.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.941+0000 7fefd1506640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefb80a3240 0x7fefb80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:46.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.941+0000 7fefc2ffd640 1 -- 172.21.15.62:0/1244428398 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fefc80cfe90 con 0x7fefcc100e20 2024-09-15T23:15:46.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.942+0000 7fefd1506640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefb80a3240 0x7fefb80a5700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fefcc105150 tx=0x7fefbc00b590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:46.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:46.945+0000 7fefc2ffd640 1 -- 172.21.15.62:0/1244428398 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fefc8014030 con 0x7fefcc100e20 2024-09-15T23:15:47.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:46 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3427846329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:15:47.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:46 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2998802135' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:15:47.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:46 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3427846329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:15:47.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:46 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2998802135' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:15:47.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.156+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefcc199730 con 0x7fefcc100e20 2024-09-15T23:15:47.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.157+0000 7fefc2ffd640 1 -- 172.21.15.62:0/1244428398 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7fefc800de20 con 0x7fefcc100e20 2024-09-15T23:15:47.158 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:47.158 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"btime":"2024-09-15T23:15:44:620975+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:15:43.799068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:15:47.158 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefb80a3240 msgr2=0x7fefb80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefb80a3240 0x7fefb80a5700 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fefcc105150 tx=0x7fefbc00b590 comp rx=0 tx=0).stop 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc100e20 msgr2=0x7fefcc1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc100e20 0x7fefcc1983d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fefc800c920 tx=0x7fefc800cde0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 shutdown_connections 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fefb80a3240 0x7fefb80a5700 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fefcc102ef0 0x7fefcc19cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefcc102570 0x7fefcc198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 --2- 172.21.15.62:0/1244428398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fefcc100e20 0x7fefcc1983d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 >> 172.21.15.62:0/1244428398 conn(0x7fefcc0fc7d0 msgr2=0x7fefcc06a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:47.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 shutdown_connections 2024-09-15T23:15:47.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.159+0000 7fefd3f6b640 1 -- 172.21.15.62:0/1244428398 wait complete. 2024-09-15T23:15:47.786 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-09-15T23:15:47.787 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-09-15T23:15:47.787 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T23:15:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.892+0000 7f3084d91640 1 -- 172.21.15.62:0/2022175656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080101c20 msgr2=0x7f30801020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.892+0000 7f3084d91640 1 --2- 172.21.15.62:0/2022175656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080101c20 0x7f30801020a0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3074009900 tx=0x7f307402a410 comp rx=0 tx=0).stop 2024-09-15T23:15:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.892+0000 7f3084d91640 1 -- 172.21.15.62:0/2022175656 shutdown_connections 2024-09-15T23:15:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.892+0000 7f3084d91640 1 --2- 172.21.15.62:0/2022175656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f30801025e0 0x7f3080108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.892+0000 7f3084d91640 1 --2- 172.21.15.62:0/2022175656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080101c20 0x7f30801020a0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.893+0000 7f3084d91640 1 --2- 172.21.15.62:0/2022175656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3080100a30 0x7f3080100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.893+0000 7f3084d91640 1 -- 172.21.15.62:0/2022175656 >> 172.21.15.62:0/2022175656 conn(0x7f30800fc1b0 msgr2=0x7f30800fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.893+0000 7f3084d91640 1 -- 172.21.15.62:0/2022175656 shutdown_connections 2024-09-15T23:15:47.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.893+0000 7f3084d91640 1 -- 172.21.15.62:0/2022175656 wait complete. 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.893+0000 7f3084d91640 1 Processor -- start 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.893+0000 7f3084d91640 1 -- start start 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f3084d91640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3080100a30 0x7f3080197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f3084d91640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080101c20 0x7f30801983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f3084d91640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30801025e0 0x7f308019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f3084d91640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f308010d800 con 0x7f3080101c20 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f3084d91640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f308010d680 con 0x7f30801025e0 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f3084d91640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f308010d980 con 0x7f3080100a30 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307dd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080101c20 0x7f30801983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:47.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307e59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3080100a30 0x7f3080197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307e59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3080100a30 0x7f3080197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48102/0 (socket says 172.21.15.62:48102) 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307e59c640 1 -- 172.21.15.62:0/3764008083 learned_addr learned my addr 172.21.15.62:0/3764008083 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307ed9d640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30801025e0 0x7f308019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307dd9b640 1 -- 172.21.15.62:0/3764008083 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3080100a30 msgr2=0x7f3080197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307dd9b640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3080100a30 0x7f3080197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307dd9b640 1 -- 172.21.15.62:0/3764008083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30801025e0 msgr2=0x7f308019f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307dd9b640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30801025e0 0x7f308019f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307dd9b640 1 -- 172.21.15.62:0/3764008083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308019faa0 con 0x7f3080101c20 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307ed9d640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30801025e0 0x7f308019f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307e59c640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3080100a30 0x7f3080197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f307dd9b640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080101c20 0x7f30801983c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3074031440 tx=0x7f30740043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.894+0000 7f306f7fe640 1 -- 172.21.15.62:0/3764008083 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307403f070 con 0x7f3080101c20 2024-09-15T23:15:47.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.895+0000 7f306f7fe640 1 -- 172.21.15.62:0/3764008083 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f307403b460 con 0x7f3080101c20 2024-09-15T23:15:47.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.895+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f308019fd30 con 0x7f3080101c20 2024-09-15T23:15:47.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.895+0000 7f306f7fe640 1 -- 172.21.15.62:0/3764008083 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3074035070 con 0x7f3080101c20 2024-09-15T23:15:47.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.895+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30801a0210 con 0x7f3080101c20 2024-09-15T23:15:47.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.896+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3080107df0 con 0x7f3080101c20 2024-09-15T23:15:47.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.897+0000 7f306f7fe640 1 -- 172.21.15.62:0/3764008083 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f307403b980 con 0x7f3080101c20 2024-09-15T23:15:47.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.897+0000 7f306f7fe640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30680a32d0 0x7f30680a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:47.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.897+0000 7f307e59c640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30680a32d0 0x7f30680a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:47.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.897+0000 7f306f7fe640 1 -- 172.21.15.62:0/3764008083 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f30740f0210 con 0x7f3080101c20 2024-09-15T23:15:47.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.898+0000 7f307e59c640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30680a32d0 0x7f30680a5790 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3070005fd0 tx=0x7f307000b4a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:47.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:47.902+0000 7f306f7fe640 1 -- 172.21.15.62:0/3764008083 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f30740bc560 con 0x7f3080101c20 2024-09-15T23:15:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:47 smithi062 ceph-mon[30949]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s 2024-09-15T23:15:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:47 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1244428398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:48.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:47 smithi067 ceph-mon[35665]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s 2024-09-15T23:15:48.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:47 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1244428398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:48.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:47 smithi148 ceph-mon[35610]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 639 B/s wr, 6 op/s 2024-09-15T23:15:48.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:47 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1244428398' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:48.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.078+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3080062630 con 0x7f30680a32d0 2024-09-15T23:15:48.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.086+0000 7f306f7fe640 1 -- 172.21.15.62:0/3764008083 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340816 (secure 0 0 0) 0x7f3080062630 con 0x7f30680a32d0 2024-09-15T23:15:48.091 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:48.093 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:15:48.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30680a32d0 msgr2=0x7f30680a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30680a32d0 0x7f30680a5790 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3070005fd0 tx=0x7f307000b4a0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080101c20 msgr2=0x7f30801983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080101c20 0x7f30801983c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3074031440 tx=0x7f30740043b0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 shutdown_connections 2024-09-15T23:15:48.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f30680a32d0 0x7f30680a5790 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30801025e0 0x7f308019f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080101c20 0x7f30801983c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 --2- 172.21.15.62:0/3764008083 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3080100a30 0x7f3080197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 >> 172.21.15.62:0/3764008083 conn(0x7f30800fc1b0 msgr2=0x7f3080104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:48.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 shutdown_connections 2024-09-15T23:15:48.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.090+0000 7f3084d91640 1 -- 172.21.15.62:0/3764008083 wait complete. 2024-09-15T23:15:48.102 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-15T23:15:47.725593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658234,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":742,"num_read_kb":489,"num_write":361,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369972,"kb_used_data":12332,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571852,"statfs":{"total":1151940427776,"available":1151561576448,"internally_reserved":0,"allocated":12627968,"data_stored":8197454,"data_compressed":455690,"data_compressed_allocated":618496,"data_compressed_original":2736128,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":698,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":31,"num_write":10,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002136"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609573+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:15:42.609573+0000","last_peered":"2024-09-15T23:15:42.609573+0000","last_clean":"2024-09-15T23:15:42.609573+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:15:42.609573+0000","last_undegraded":"2024-09-15T23:15:42.609573+0000","last_fullsized":"2024-09-15T23:15:42.609573+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221889+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:15:42.221889+0000","last_peered":"2024-09-15T23:15:42.221889+0000","last_clean":"2024-09-15T23:15:42.221889+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:15:42.221889+0000","last_undegraded":"2024-09-15T23:15:42.221889+0000","last_fullsized":"2024-09-15T23:15:42.221889+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613665+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:15:42.613665+0000","last_peered":"2024-09-15T23:15:42.613665+0000","last_clean":"2024-09-15T23:15:42.613665+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:15:42.613665+0000","last_undegraded":"2024-09-15T23:15:42.613665+0000","last_fullsized":"2024-09-15T23:15:42.613665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.222006+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:15:42.222006+0000","last_peered":"2024-09-15T23:15:42.222006+0000","last_clean":"2024-09-15T23:15:42.222006+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:15:42.222006+0000","last_undegraded":"2024-09-15T23:15:42.222006+0000","last_fullsized":"2024-09-15T23:15:42.222006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096731+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:15:42.096731+0000","last_peered":"2024-09-15T23:15:42.096731+0000","last_clean":"2024-09-15T23:15:42.096731+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:15:42.096731+0000","last_undegraded":"2024-09-15T23:15:42.096731+0000","last_fullsized":"2024-09-15T23:15:42.096731+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611378+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:15:42.611378+0000","last_peered":"2024-09-15T23:15:42.611378+0000","last_clean":"2024-09-15T23:15:42.611378+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:15:42.611378+0000","last_undegraded":"2024-09-15T23:15:42.611378+0000","last_fullsized":"2024-09-15T23:15:42.611378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909808+0000","last_change":"2024-09-15T23:14:41.659816+0000","last_active":"2024-09-15T23:15:42.909808+0000","last_peered":"2024-09-15T23:15:42.909808+0000","last_clean":"2024-09-15T23:15:42.909808+0000","last_became_active":"2024-09-15T23:14:41.641611+0000","last_became_peered":"2024-09-15T23:14:41.641611+0000","last_unstale":"2024-09-15T23:15:42.909808+0000","last_undegraded":"2024-09-15T23:15:42.909808+0000","last_fullsized":"2024-09-15T23:15:42.909808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:23.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612963+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:15:42.612963+0000","last_peered":"2024-09-15T23:15:42.612963+0000","last_clean":"2024-09-15T23:15:42.612963+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:15:42.612963+0000","last_undegraded":"2024-09-15T23:15:42.612963+0000","last_fullsized":"2024-09-15T23:15:42.612963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610025+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:15:42.610025+0000","last_peered":"2024-09-15T23:15:42.610025+0000","last_clean":"2024-09-15T23:15:42.610025+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:15:42.610025+0000","last_undegraded":"2024-09-15T23:15:42.610025+0000","last_fullsized":"2024-09-15T23:15:42.610025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614867+0000","last_change":"2024-09-15T23:15:42.614867+0000","last_active":"2024-09-15T23:15:42.614867+0000","last_peered":"2024-09-15T23:15:42.614867+0000","last_clean":"2024-09-15T23:15:42.614867+0000","last_became_active":"2024-09-15T23:15:42.614459+0000","last_became_peered":"2024-09-15T23:15:42.614459+0000","last_unstale":"2024-09-15T23:15:42.614867+0000","last_undegraded":"2024-09-15T23:15:42.614867+0000","last_fullsized":"2024-09-15T23:15:42.614867+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:15.254442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611583+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:15:42.611583+0000","last_peered":"2024-09-15T23:15:42.611583+0000","last_clean":"2024-09-15T23:15:42.611583+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:15:42.611583+0000","last_undegraded":"2024-09-15T23:15:42.611583+0000","last_fullsized":"2024-09-15T23:15:42.611583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317095+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:15:43.317095+0000","last_peered":"2024-09-15T23:15:43.317095+0000","last_clean":"2024-09-15T23:15:43.317095+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:15:43.317095+0000","last_undegraded":"2024-09-15T23:15:43.317095+0000","last_fullsized":"2024-09-15T23:15:43.317095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":56,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609750+0000","last_change":"2024-09-15T23:14:41.646320+0000","last_active":"2024-09-15T23:15:42.609750+0000","last_peered":"2024-09-15T23:15:42.609750+0000","last_clean":"2024-09-15T23:15:42.609750+0000","last_became_active":"2024-09-15T23:14:41.641933+0000","last_became_peered":"2024-09-15T23:14:41.641933+0000","last_unstale":"2024-09-15T23:15:42.609750+0000","last_undegraded":"2024-09-15T23:15:42.609750+0000","last_fullsized":"2024-09-15T23:15:42.609750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:49.417407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611897+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:15:42.611897+0000","last_peered":"2024-09-15T23:15:42.611897+0000","last_clean":"2024-09-15T23:15:42.611897+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:15:42.611897+0000","last_undegraded":"2024-09-15T23:15:42.611897+0000","last_fullsized":"2024-09-15T23:15:42.611897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221618+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:15:42.221618+0000","last_peered":"2024-09-15T23:15:42.221618+0000","last_clean":"2024-09-15T23:15:42.221618+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:15:42.221618+0000","last_undegraded":"2024-09-15T23:15:42.221618+0000","last_fullsized":"2024-09-15T23:15:42.221618+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613807+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:15:42.613807+0000","last_peered":"2024-09-15T23:15:42.613807+0000","last_clean":"2024-09-15T23:15:42.613807+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:15:42.613807+0000","last_undegraded":"2024-09-15T23:15:42.613807+0000","last_fullsized":"2024-09-15T23:15:42.613807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612402+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:15:42.612402+0000","last_peered":"2024-09-15T23:15:42.612402+0000","last_clean":"2024-09-15T23:15:42.612402+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:15:42.612402+0000","last_undegraded":"2024-09-15T23:15:42.612402+0000","last_fullsized":"2024-09-15T23:15:42.612402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909877+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:15:42.909877+0000","last_peered":"2024-09-15T23:15:42.909877+0000","last_clean":"2024-09-15T23:15:42.909877+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:15:42.909877+0000","last_undegraded":"2024-09-15T23:15:42.909877+0000","last_fullsized":"2024-09-15T23:15:42.909877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221790+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:15:42.221790+0000","last_peered":"2024-09-15T23:15:42.221790+0000","last_clean":"2024-09-15T23:15:42.221790+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:15:42.221790+0000","last_undegraded":"2024-09-15T23:15:42.221790+0000","last_fullsized":"2024-09-15T23:15:42.221790+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609295+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:15:42.609295+0000","last_peered":"2024-09-15T23:15:42.609295+0000","last_clean":"2024-09-15T23:15:42.609295+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:15:42.609295+0000","last_undegraded":"2024-09-15T23:15:42.609295+0000","last_fullsized":"2024-09-15T23:15:42.609295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609453+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:15:42.609453+0000","last_peered":"2024-09-15T23:15:42.609453+0000","last_clean":"2024-09-15T23:15:42.609453+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:15:42.609453+0000","last_undegraded":"2024-09-15T23:15:42.609453+0000","last_fullsized":"2024-09-15T23:15:42.609453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613324+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:15:42.613324+0000","last_peered":"2024-09-15T23:15:42.613324+0000","last_clean":"2024-09-15T23:15:42.613324+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:15:42.613324+0000","last_undegraded":"2024-09-15T23:15:42.613324+0000","last_fullsized":"2024-09-15T23:15:42.613324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612027+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:15:42.612027+0000","last_peered":"2024-09-15T23:15:42.612027+0000","last_clean":"2024-09-15T23:15:42.612027+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:15:42.612027+0000","last_undegraded":"2024-09-15T23:15:42.612027+0000","last_fullsized":"2024-09-15T23:15:42.612027+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096594+0000","last_change":"2024-09-15T23:14:42.009605+0000","last_active":"2024-09-15T23:15:42.096594+0000","last_peered":"2024-09-15T23:15:42.096594+0000","last_clean":"2024-09-15T23:15:42.096594+0000","last_became_active":"2024-09-15T23:14:42.009237+0000","last_became_peered":"2024-09-15T23:14:42.009237+0000","last_unstale":"2024-09-15T23:15:42.096594+0000","last_undegraded":"2024-09-15T23:15:42.096594+0000","last_fullsized":"2024-09-15T23:15:42.096594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:11:08.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612739+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:15:42.612739+0000","last_peered":"2024-09-15T23:15:42.612739+0000","last_clean":"2024-09-15T23:15:42.612739+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:15:42.612739+0000","last_undegraded":"2024-09-15T23:15:42.612739+0000","last_fullsized":"2024-09-15T23:15:42.612739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614124+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:15:42.614124+0000","last_peered":"2024-09-15T23:15:42.614124+0000","last_clean":"2024-09-15T23:15:42.614124+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:15:42.614124+0000","last_undegraded":"2024-09-15T23:15:42.614124+0000","last_fullsized":"2024-09-15T23:15:42.614124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.222041+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:15:42.222041+0000","last_peered":"2024-09-15T23:15:42.222041+0000","last_clean":"2024-09-15T23:15:42.222041+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:15:42.222041+0000","last_undegraded":"2024-09-15T23:15:42.222041+0000","last_fullsized":"2024-09-15T23:15:42.222041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'14","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610065+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:15:42.610065+0000","last_peered":"2024-09-15T23:15:42.610065+0000","last_clean":"2024-09-15T23:15:42.610065+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:15:42.610065+0000","last_undegraded":"2024-09-15T23:15:42.610065+0000","last_fullsized":"2024-09-15T23:15:42.610065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221420+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:15:42.221420+0000","last_peered":"2024-09-15T23:15:42.221420+0000","last_clean":"2024-09-15T23:15:42.221420+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:15:42.221420+0000","last_undegraded":"2024-09-15T23:15:42.221420+0000","last_fullsized":"2024-09-15T23:15:42.221420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612814+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:15:42.612814+0000","last_peered":"2024-09-15T23:15:42.612814+0000","last_clean":"2024-09-15T23:15:42.612814+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:15:42.612814+0000","last_undegraded":"2024-09-15T23:15:42.612814+0000","last_fullsized":"2024-09-15T23:15:42.612814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610084+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:15:42.610084+0000","last_peered":"2024-09-15T23:15:42.610084+0000","last_clean":"2024-09-15T23:15:42.610084+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:15:42.610084+0000","last_undegraded":"2024-09-15T23:15:42.610084+0000","last_fullsized":"2024-09-15T23:15:42.610084+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909751+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:15:42.909751+0000","last_peered":"2024-09-15T23:15:42.909751+0000","last_clean":"2024-09-15T23:15:42.909751+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:15:42.909751+0000","last_undegraded":"2024-09-15T23:15:42.909751+0000","last_fullsized":"2024-09-15T23:15:42.909751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610218+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:15:42.610218+0000","last_peered":"2024-09-15T23:15:42.610218+0000","last_clean":"2024-09-15T23:15:42.610218+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:15:42.610218+0000","last_undegraded":"2024-09-15T23:15:42.610218+0000","last_fullsized":"2024-09-15T23:15:42.610218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609669+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:15:42.609669+0000","last_peered":"2024-09-15T23:15:42.609669+0000","last_clean":"2024-09-15T23:15:42.609669+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:15:42.609669+0000","last_undegraded":"2024-09-15T23:15:42.609669+0000","last_fullsized":"2024-09-15T23:15:42.609669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609473+0000","last_change":"2024-09-15T23:14:41.995466+0000","last_active":"2024-09-15T23:15:42.609473+0000","last_peered":"2024-09-15T23:15:42.609473+0000","last_clean":"2024-09-15T23:15:42.609473+0000","last_became_active":"2024-09-15T23:14:41.994915+0000","last_became_peered":"2024-09-15T23:14:41.994915+0000","last_unstale":"2024-09-15T23:15:42.609473+0000","last_undegraded":"2024-09-15T23:15:42.609473+0000","last_fullsized":"2024-09-15T23:15:42.609473+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:04.121596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610393+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:15:42.610393+0000","last_peered":"2024-09-15T23:15:42.610393+0000","last_clean":"2024-09-15T23:15:42.610393+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:15:42.610393+0000","last_undegraded":"2024-09-15T23:15:42.610393+0000","last_fullsized":"2024-09-15T23:15:42.610393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317131+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:15:43.317131+0000","last_peered":"2024-09-15T23:15:43.317131+0000","last_clean":"2024-09-15T23:15:43.317131+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:15:43.317131+0000","last_undegraded":"2024-09-15T23:15:43.317131+0000","last_fullsized":"2024-09-15T23:15:43.317131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626237+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:15:42.626237+0000","last_peered":"2024-09-15T23:15:42.626237+0000","last_clean":"2024-09-15T23:15:42.626237+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:15:42.626237+0000","last_undegraded":"2024-09-15T23:15:42.626237+0000","last_fullsized":"2024-09-15T23:15:42.626237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.618846+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:15:43.618846+0000","last_peered":"2024-09-15T23:15:43.618846+0000","last_clean":"2024-09-15T23:15:43.618846+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:15:43.618846+0000","last_undegraded":"2024-09-15T23:15:43.618846+0000","last_fullsized":"2024-09-15T23:15:43.618846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":97,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612986+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:15:42.612986+0000","last_peered":"2024-09-15T23:15:42.612986+0000","last_clean":"2024-09-15T23:15:42.612986+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:15:42.612986+0000","last_undegraded":"2024-09-15T23:15:42.612986+0000","last_fullsized":"2024-09-15T23:15:42.612986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.617748+0000","last_change":"2024-09-15T23:15:42.617748+0000","last_active":"2024-09-15T23:15:42.617748+0000","last_peered":"2024-09-15T23:15:42.617748+0000","last_clean":"2024-09-15T23:15:42.617748+0000","last_became_active":"2024-09-15T23:15:42.617320+0000","last_became_peered":"2024-09-15T23:15:42.617320+0000","last_unstale":"2024-09-15T23:15:42.617748+0000","last_undegraded":"2024-09-15T23:15:42.617748+0000","last_fullsized":"2024-09-15T23:15:42.617748+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:47.900049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316911+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:15:43.316911+0000","last_peered":"2024-09-15T23:15:43.316911+0000","last_clean":"2024-09-15T23:15:43.316911+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:15:43.316911+0000","last_undegraded":"2024-09-15T23:15:43.316911+0000","last_fullsized":"2024-09-15T23:15:43.316911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626536+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:15:42.626536+0000","last_peered":"2024-09-15T23:15:42.626536+0000","last_clean":"2024-09-15T23:15:42.626536+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:15:42.626536+0000","last_undegraded":"2024-09-15T23:15:42.626536+0000","last_fullsized":"2024-09-15T23:15:42.626536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221750+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:42.221750+0000","last_peered":"2024-09-15T23:15:42.221750+0000","last_clean":"2024-09-15T23:15:42.221750+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:15:42.221750+0000","last_undegraded":"2024-09-15T23:15:42.221750+0000","last_fullsized":"2024-09-15T23:15:42.221750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612474+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:15:42.612474+0000","last_peered":"2024-09-15T23:15:42.612474+0000","last_clean":"2024-09-15T23:15:42.612474+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:15:42.612474+0000","last_undegraded":"2024-09-15T23:15:42.612474+0000","last_fullsized":"2024-09-15T23:15:42.612474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317208+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:15:43.317208+0000","last_peered":"2024-09-15T23:15:43.317208+0000","last_clean":"2024-09-15T23:15:43.317208+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:15:43.317208+0000","last_undegraded":"2024-09-15T23:15:43.317208+0000","last_fullsized":"2024-09-15T23:15:43.317208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317366+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:15:43.317366+0000","last_peered":"2024-09-15T23:15:43.317366+0000","last_clean":"2024-09-15T23:15:43.317366+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:15:43.317366+0000","last_undegraded":"2024-09-15T23:15:43.317366+0000","last_fullsized":"2024-09-15T23:15:43.317366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317438+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:15:43.317438+0000","last_peered":"2024-09-15T23:15:43.317438+0000","last_clean":"2024-09-15T23:15:43.317438+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:15:43.317438+0000","last_undegraded":"2024-09-15T23:15:43.317438+0000","last_fullsized":"2024-09-15T23:15:43.317438+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614172+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:15:42.614172+0000","last_peered":"2024-09-15T23:15:42.614172+0000","last_clean":"2024-09-15T23:15:42.614172+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:15:42.614172+0000","last_undegraded":"2024-09-15T23:15:42.614172+0000","last_fullsized":"2024-09-15T23:15:42.614172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612428+0000","last_change":"2024-09-15T23:14:41.994950+0000","last_active":"2024-09-15T23:15:42.612428+0000","last_peered":"2024-09-15T23:15:42.612428+0000","last_clean":"2024-09-15T23:15:42.612428+0000","last_became_active":"2024-09-15T23:14:41.994429+0000","last_became_peered":"2024-09-15T23:14:41.994429+0000","last_unstale":"2024-09-15T23:15:42.612428+0000","last_undegraded":"2024-09-15T23:15:42.612428+0000","last_fullsized":"2024-09-15T23:15:42.612428+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:24.056756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317319+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:15:43.317319+0000","last_peered":"2024-09-15T23:15:43.317319+0000","last_clean":"2024-09-15T23:15:43.317319+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:15:43.317319+0000","last_undegraded":"2024-09-15T23:15:43.317319+0000","last_fullsized":"2024-09-15T23:15:43.317319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.627013+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:15:42.627013+0000","last_peered":"2024-09-15T23:15:42.627013+0000","last_clean":"2024-09-15T23:15:42.627013+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:15:42.627013+0000","last_undegraded":"2024-09-15T23:15:42.627013+0000","last_fullsized":"2024-09-15T23:15:42.627013+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096349+0000","last_change":"2024-09-15T23:14:41.640471+0000","last_active":"2024-09-15T23:15:42.096349+0000","last_peered":"2024-09-15T23:15:42.096349+0000","last_clean":"2024-09-15T23:15:42.096349+0000","last_became_active":"2024-09-15T23:14:41.640017+0000","last_became_peered":"2024-09-15T23:14:41.640017+0000","last_unstale":"2024-09-15T23:15:42.096349+0000","last_undegraded":"2024-09-15T23:15:42.096349+0000","last_fullsized":"2024-09-15T23:15:42.096349+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:58.842636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317343+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:15:43.317343+0000","last_peered":"2024-09-15T23:15:43.317343+0000","last_clean":"2024-09-15T23:15:43.317343+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:15:43.317343+0000","last_undegraded":"2024-09-15T23:15:43.317343+0000","last_fullsized":"2024-09-15T23:15:43.317343+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613353+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:15:42.613353+0000","last_peered":"2024-09-15T23:15:42.613353+0000","last_clean":"2024-09-15T23:15:42.613353+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:15:42.613353+0000","last_undegraded":"2024-09-15T23:15:42.613353+0000","last_fullsized":"2024-09-15T23:15:42.613353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.625937+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:15:42.625937+0000","last_peered":"2024-09-15T23:15:42.625937+0000","last_clean":"2024-09-15T23:15:42.625937+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:15:42.625937+0000","last_undegraded":"2024-09-15T23:15:42.625937+0000","last_fullsized":"2024-09-15T23:15:42.625937+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909962+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:15:42.909962+0000","last_peered":"2024-09-15T23:15:42.909962+0000","last_clean":"2024-09-15T23:15:42.909962+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:15:42.909962+0000","last_undegraded":"2024-09-15T23:15:42.909962+0000","last_fullsized":"2024-09-15T23:15:42.909962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610024+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:15:42.610024+0000","last_peered":"2024-09-15T23:15:42.610024+0000","last_clean":"2024-09-15T23:15:42.610024+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:15:42.610024+0000","last_undegraded":"2024-09-15T23:15:42.610024+0000","last_fullsized":"2024-09-15T23:15:42.610024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.625935+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:15:42.625935+0000","last_peered":"2024-09-15T23:15:42.625935+0000","last_clean":"2024-09-15T23:15:42.625935+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:15:42.625935+0000","last_undegraded":"2024-09-15T23:15:42.625935+0000","last_fullsized":"2024-09-15T23:15:42.625935+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626111+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:15:42.626111+0000","last_peered":"2024-09-15T23:15:42.626111+0000","last_clean":"2024-09-15T23:15:42.626111+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:15:42.626111+0000","last_undegraded":"2024-09-15T23:15:42.626111+0000","last_fullsized":"2024-09-15T23:15:42.626111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909569+0000","last_change":"2024-09-15T23:14:41.653786+0000","last_active":"2024-09-15T23:15:42.909569+0000","last_peered":"2024-09-15T23:15:42.909569+0000","last_clean":"2024-09-15T23:15:42.909569+0000","last_became_active":"2024-09-15T23:14:41.641472+0000","last_became_peered":"2024-09-15T23:14:41.641472+0000","last_unstale":"2024-09-15T23:15:42.909569+0000","last_undegraded":"2024-09-15T23:15:42.909569+0000","last_fullsized":"2024-09-15T23:15:42.909569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:16.010142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613267+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:15:42.613267+0000","last_peered":"2024-09-15T23:15:42.613267+0000","last_clean":"2024-09-15T23:15:42.613267+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:15:42.613267+0000","last_undegraded":"2024-09-15T23:15:42.613267+0000","last_fullsized":"2024-09-15T23:15:42.613267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316776+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:15:43.316776+0000","last_peered":"2024-09-15T23:15:43.316776+0000","last_clean":"2024-09-15T23:15:43.316776+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:15:43.316776+0000","last_undegraded":"2024-09-15T23:15:43.316776+0000","last_fullsized":"2024-09-15T23:15:43.316776+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.619522+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:15:43.619522+0000","last_peered":"2024-09-15T23:15:43.619522+0000","last_clean":"2024-09-15T23:15:43.619522+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:15:43.619522+0000","last_undegraded":"2024-09-15T23:15:43.619522+0000","last_fullsized":"2024-09-15T23:15:43.619522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609376+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:15:42.609376+0000","last_peered":"2024-09-15T23:15:42.609376+0000","last_clean":"2024-09-15T23:15:42.609376+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:15:42.609376+0000","last_undegraded":"2024-09-15T23:15:42.609376+0000","last_fullsized":"2024-09-15T23:15:42.609376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":48,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317071+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:15:43.317071+0000","last_peered":"2024-09-15T23:15:43.317071+0000","last_clean":"2024-09-15T23:15:43.317071+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:15:43.317071+0000","last_undegraded":"2024-09-15T23:15:43.317071+0000","last_fullsized":"2024-09-15T23:15:43.317071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626254+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:15:42.626254+0000","last_peered":"2024-09-15T23:15:42.626254+0000","last_clean":"2024-09-15T23:15:42.626254+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:15:42.626254+0000","last_undegraded":"2024-09-15T23:15:42.626254+0000","last_fullsized":"2024-09-15T23:15:42.626254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909759+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:15:42.909759+0000","last_peered":"2024-09-15T23:15:42.909759+0000","last_clean":"2024-09-15T23:15:42.909759+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:15:42.909759+0000","last_undegraded":"2024-09-15T23:15:42.909759+0000","last_fullsized":"2024-09-15T23:15:42.909759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612005+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:15:42.612005+0000","last_peered":"2024-09-15T23:15:42.612005+0000","last_clean":"2024-09-15T23:15:42.612005+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:15:42.612005+0000","last_undegraded":"2024-09-15T23:15:42.612005+0000","last_fullsized":"2024-09-15T23:15:42.612005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613889+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:15:42.613889+0000","last_peered":"2024-09-15T23:15:42.613889+0000","last_clean":"2024-09-15T23:15:42.613889+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:15:42.613889+0000","last_undegraded":"2024-09-15T23:15:42.613889+0000","last_fullsized":"2024-09-15T23:15:42.613889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626638+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:15:42.626638+0000","last_peered":"2024-09-15T23:15:42.626638+0000","last_clean":"2024-09-15T23:15:42.626638+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:15:42.626638+0000","last_undegraded":"2024-09-15T23:15:42.626638+0000","last_fullsized":"2024-09-15T23:15:42.626638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609857+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:15:42.609857+0000","last_peered":"2024-09-15T23:15:42.609857+0000","last_clean":"2024-09-15T23:15:42.609857+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:15:42.609857+0000","last_undegraded":"2024-09-15T23:15:42.609857+0000","last_fullsized":"2024-09-15T23:15:42.609857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":281,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:45.631571+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:15:45.631571+0000","last_peered":"2024-09-15T23:15:45.631571+0000","last_clean":"2024-09-15T23:15:45.631571+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:15:45.631571+0000","last_undegraded":"2024-09-15T23:15:45.631571+0000","last_fullsized":"2024-09-15T23:15:45.631571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63488,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":207,"num_read_kb":166,"num_write":37,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609007+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:15:42.609007+0000","last_peered":"2024-09-15T23:15:42.609007+0000","last_clean":"2024-09-15T23:15:42.609007+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:15:42.609007+0000","last_undegraded":"2024-09-15T23:15:42.609007+0000","last_fullsized":"2024-09-15T23:15:42.609007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612977+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:15:42.612977+0000","last_peered":"2024-09-15T23:15:42.612977+0000","last_clean":"2024-09-15T23:15:42.612977+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:15:42.612977+0000","last_undegraded":"2024-09-15T23:15:42.612977+0000","last_fullsized":"2024-09-15T23:15:42.612977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626922+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:15:42.626922+0000","last_peered":"2024-09-15T23:15:42.626922+0000","last_clean":"2024-09-15T23:15:42.626922+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:15:42.626922+0000","last_undegraded":"2024-09-15T23:15:42.626922+0000","last_fullsized":"2024-09-15T23:15:42.626922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612007+0000","last_change":"2024-09-15T23:14:41.641820+0000","last_active":"2024-09-15T23:15:42.612007+0000","last_peered":"2024-09-15T23:15:42.612007+0000","last_clean":"2024-09-15T23:15:42.612007+0000","last_became_active":"2024-09-15T23:14:41.641282+0000","last_became_peered":"2024-09-15T23:14:41.641282+0000","last_unstale":"2024-09-15T23:15:42.612007+0000","last_undegraded":"2024-09-15T23:15:42.612007+0000","last_fullsized":"2024-09-15T23:15:42.612007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:43.472601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096735+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:15:42.096735+0000","last_peered":"2024-09-15T23:15:42.096735+0000","last_clean":"2024-09-15T23:15:42.096735+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:15:42.096735+0000","last_undegraded":"2024-09-15T23:15:42.096735+0000","last_fullsized":"2024-09-15T23:15:42.096735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317368+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:15:43.317368+0000","last_peered":"2024-09-15T23:15:43.317368+0000","last_clean":"2024-09-15T23:15:43.317368+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:15:43.317368+0000","last_undegraded":"2024-09-15T23:15:43.317368+0000","last_fullsized":"2024-09-15T23:15:43.317368+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608807+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:15:42.608807+0000","last_peered":"2024-09-15T23:15:42.608807+0000","last_clean":"2024-09-15T23:15:42.608807+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:15:42.608807+0000","last_undegraded":"2024-09-15T23:15:42.608807+0000","last_fullsized":"2024-09-15T23:15:42.608807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612577+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:15:42.612577+0000","last_peered":"2024-09-15T23:15:42.612577+0000","last_clean":"2024-09-15T23:15:42.612577+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:15:42.612577+0000","last_undegraded":"2024-09-15T23:15:42.612577+0000","last_fullsized":"2024-09-15T23:15:42.612577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:44.628850+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:15:44.628850+0000","last_peered":"2024-09-15T23:15:44.628850+0000","last_clean":"2024-09-15T23:15:44.628850+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:15:44.628850+0000","last_undegraded":"2024-09-15T23:15:44.628850+0000","last_fullsized":"2024-09-15T23:15:44.628850+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221487+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:15:42.221487+0000","last_peered":"2024-09-15T23:15:42.221487+0000","last_clean":"2024-09-15T23:15:42.221487+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:15:42.221487+0000","last_undegraded":"2024-09-15T23:15:42.221487+0000","last_fullsized":"2024-09-15T23:15:42.221487+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096328+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:15:42.096328+0000","last_peered":"2024-09-15T23:15:42.096328+0000","last_clean":"2024-09-15T23:15:42.096328+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:15:42.096328+0000","last_undegraded":"2024-09-15T23:15:42.096328+0000","last_fullsized":"2024-09-15T23:15:42.096328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608984+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:15:42.608984+0000","last_peered":"2024-09-15T23:15:42.608984+0000","last_clean":"2024-09-15T23:15:42.608984+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:15:42.608984+0000","last_undegraded":"2024-09-15T23:15:42.608984+0000","last_fullsized":"2024-09-15T23:15:42.608984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626347+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:15:42.626347+0000","last_peered":"2024-09-15T23:15:42.626347+0000","last_clean":"2024-09-15T23:15:42.626347+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:15:42.626347+0000","last_undegraded":"2024-09-15T23:15:42.626347+0000","last_fullsized":"2024-09-15T23:15:42.626347+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909956+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:15:42.909956+0000","last_peered":"2024-09-15T23:15:42.909956+0000","last_clean":"2024-09-15T23:15:42.909956+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:15:42.909956+0000","last_undegraded":"2024-09-15T23:15:42.909956+0000","last_fullsized":"2024-09-15T23:15:42.909956+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221496+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:15:42.221496+0000","last_peered":"2024-09-15T23:15:42.221496+0000","last_clean":"2024-09-15T23:15:42.221496+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:15:42.221496+0000","last_undegraded":"2024-09-15T23:15:42.221496+0000","last_fullsized":"2024-09-15T23:15:42.221496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611692+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:15:42.611692+0000","last_peered":"2024-09-15T23:15:42.611692+0000","last_clean":"2024-09-15T23:15:42.611692+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:15:42.611692+0000","last_undegraded":"2024-09-15T23:15:42.611692+0000","last_fullsized":"2024-09-15T23:15:42.611692+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610033+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:15:42.610033+0000","last_peered":"2024-09-15T23:15:42.610033+0000","last_clean":"2024-09-15T23:15:42.610033+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:15:42.610033+0000","last_undegraded":"2024-09-15T23:15:42.610033+0000","last_fullsized":"2024-09-15T23:15:42.610033+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626302+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:15:42.626302+0000","last_peered":"2024-09-15T23:15:42.626302+0000","last_clean":"2024-09-15T23:15:42.626302+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:15:42.626302+0000","last_undegraded":"2024-09-15T23:15:42.626302+0000","last_fullsized":"2024-09-15T23:15:42.626302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613815+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:15:42.613815+0000","last_peered":"2024-09-15T23:15:42.613815+0000","last_clean":"2024-09-15T23:15:42.613815+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:15:42.613815+0000","last_undegraded":"2024-09-15T23:15:42.613815+0000","last_fullsized":"2024-09-15T23:15:42.613815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:44.629643+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:15:44.629643+0000","last_peered":"2024-09-15T23:15:44.629643+0000","last_clean":"2024-09-15T23:15:44.629643+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:15:44.629643+0000","last_undegraded":"2024-09-15T23:15:44.629643+0000","last_fullsized":"2024-09-15T23:15:44.629643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1156,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609500+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:15:42.609500+0000","last_peered":"2024-09-15T23:15:42.609500+0000","last_clean":"2024-09-15T23:15:42.609500+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:15:42.609500+0000","last_undegraded":"2024-09-15T23:15:42.609500+0000","last_fullsized":"2024-09-15T23:15:42.609500+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613805+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:15:42.613805+0000","last_peered":"2024-09-15T23:15:42.613805+0000","last_clean":"2024-09-15T23:15:42.613805+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:15:42.613805+0000","last_undegraded":"2024-09-15T23:15:42.613805+0000","last_fullsized":"2024-09-15T23:15:42.613805+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909660+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:15:42.909660+0000","last_peered":"2024-09-15T23:15:42.909660+0000","last_clean":"2024-09-15T23:15:42.909660+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:15:42.909660+0000","last_undegraded":"2024-09-15T23:15:42.909660+0000","last_fullsized":"2024-09-15T23:15:42.909660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614459+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:42.614459+0000","last_peered":"2024-09-15T23:15:42.614459+0000","last_clean":"2024-09-15T23:15:42.614459+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:15:42.614459+0000","last_undegraded":"2024-09-15T23:15:42.614459+0000","last_fullsized":"2024-09-15T23:15:42.614459+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613647+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:15:42.613647+0000","last_peered":"2024-09-15T23:15:42.613647+0000","last_clean":"2024-09-15T23:15:42.613647+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:15:42.613647+0000","last_undegraded":"2024-09-15T23:15:42.613647+0000","last_fullsized":"2024-09-15T23:15:42.613647+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096457+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:15:42.096457+0000","last_peered":"2024-09-15T23:15:42.096457+0000","last_clean":"2024-09-15T23:15:42.096457+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:15:42.096457+0000","last_undegraded":"2024-09-15T23:15:42.096457+0000","last_fullsized":"2024-09-15T23:15:42.096457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612897+0000","last_change":"2024-09-15T23:14:42.010010+0000","last_active":"2024-09-15T23:15:42.612897+0000","last_peered":"2024-09-15T23:15:42.612897+0000","last_clean":"2024-09-15T23:15:42.612897+0000","last_became_active":"2024-09-15T23:14:42.009603+0000","last_became_peered":"2024-09-15T23:14:42.009603+0000","last_unstale":"2024-09-15T23:15:42.612897+0000","last_undegraded":"2024-09-15T23:15:42.612897+0000","last_fullsized":"2024-09-15T23:15:42.612897+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:34:51.849548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910096+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:15:42.910096+0000","last_peered":"2024-09-15T23:15:42.910096+0000","last_clean":"2024-09-15T23:15:42.910096+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:15:42.910096+0000","last_undegraded":"2024-09-15T23:15:42.910096+0000","last_fullsized":"2024-09-15T23:15:42.910096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614200+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:15:42.614200+0000","last_peered":"2024-09-15T23:15:42.614200+0000","last_clean":"2024-09-15T23:15:42.614200+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:15:42.614200+0000","last_undegraded":"2024-09-15T23:15:42.614200+0000","last_fullsized":"2024-09-15T23:15:42.614200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221977+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:15:42.221977+0000","last_peered":"2024-09-15T23:15:42.221977+0000","last_clean":"2024-09-15T23:15:42.221977+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:15:42.221977+0000","last_undegraded":"2024-09-15T23:15:42.221977+0000","last_fullsized":"2024-09-15T23:15:42.221977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626113+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:15:42.626113+0000","last_peered":"2024-09-15T23:15:42.626113+0000","last_clean":"2024-09-15T23:15:42.626113+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:15:42.626113+0000","last_undegraded":"2024-09-15T23:15:42.626113+0000","last_fullsized":"2024-09-15T23:15:42.626113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221577+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:15:42.221577+0000","last_peered":"2024-09-15T23:15:42.221577+0000","last_clean":"2024-09-15T23:15:42.221577+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:15:42.221577+0000","last_undegraded":"2024-09-15T23:15:42.221577+0000","last_fullsized":"2024-09-15T23:15:42.221577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612683+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:15:42.612683+0000","last_peered":"2024-09-15T23:15:42.612683+0000","last_clean":"2024-09-15T23:15:42.612683+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:15:42.612683+0000","last_undegraded":"2024-09-15T23:15:42.612683+0000","last_fullsized":"2024-09-15T23:15:42.612683+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612593+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:15:42.612593+0000","last_peered":"2024-09-15T23:15:42.612593+0000","last_clean":"2024-09-15T23:15:42.612593+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:15:42.612593+0000","last_undegraded":"2024-09-15T23:15:42.612593+0000","last_fullsized":"2024-09-15T23:15:42.612593+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909865+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:15:42.909865+0000","last_peered":"2024-09-15T23:15:42.909865+0000","last_clean":"2024-09-15T23:15:42.909865+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:15:42.909865+0000","last_undegraded":"2024-09-15T23:15:42.909865+0000","last_fullsized":"2024-09-15T23:15:42.909865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614031+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:15:42.614031+0000","last_peered":"2024-09-15T23:15:42.614031+0000","last_clean":"2024-09-15T23:15:42.614031+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:15:42.614031+0000","last_undegraded":"2024-09-15T23:15:42.614031+0000","last_fullsized":"2024-09-15T23:15:42.614031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610009+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:15:42.610009+0000","last_peered":"2024-09-15T23:15:42.610009+0000","last_clean":"2024-09-15T23:15:42.610009+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:15:42.610009+0000","last_undegraded":"2024-09-15T23:15:42.610009+0000","last_fullsized":"2024-09-15T23:15:42.610009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910174+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:15:42.910174+0000","last_peered":"2024-09-15T23:15:42.910174+0000","last_clean":"2024-09-15T23:15:42.910174+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:15:42.910174+0000","last_undegraded":"2024-09-15T23:15:42.910174+0000","last_fullsized":"2024-09-15T23:15:42.910174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612880+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:15:42.612880+0000","last_peered":"2024-09-15T23:15:42.612880+0000","last_clean":"2024-09-15T23:15:42.612880+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:15:42.612880+0000","last_undegraded":"2024-09-15T23:15:42.612880+0000","last_fullsized":"2024-09-15T23:15:42.612880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096207+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:15:42.096207+0000","last_peered":"2024-09-15T23:15:42.096207+0000","last_clean":"2024-09-15T23:15:42.096207+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:15:42.096207+0000","last_undegraded":"2024-09-15T23:15:42.096207+0000","last_fullsized":"2024-09-15T23:15:42.096207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096366+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:15:42.096366+0000","last_peered":"2024-09-15T23:15:42.096366+0000","last_clean":"2024-09-15T23:15:42.096366+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:15:42.096366+0000","last_undegraded":"2024-09-15T23:15:42.096366+0000","last_fullsized":"2024-09-15T23:15:42.096366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612501+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:15:42.612501+0000","last_peered":"2024-09-15T23:15:42.612501+0000","last_clean":"2024-09-15T23:15:42.612501+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:15:42.612501+0000","last_undegraded":"2024-09-15T23:15:42.612501+0000","last_fullsized":"2024-09-15T23:15:42.612501+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612646+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:15:42.612646+0000","last_peered":"2024-09-15T23:15:42.612646+0000","last_clean":"2024-09-15T23:15:42.612646+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:15:42.612646+0000","last_undegraded":"2024-09-15T23:15:42.612646+0000","last_fullsized":"2024-09-15T23:15:42.612646+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317377+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:15:43.317377+0000","last_peered":"2024-09-15T23:15:43.317377+0000","last_clean":"2024-09-15T23:15:43.317377+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:15:43.317377+0000","last_undegraded":"2024-09-15T23:15:43.317377+0000","last_fullsized":"2024-09-15T23:15:43.317377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610726+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:15:42.610726+0000","last_peered":"2024-09-15T23:15:42.610726+0000","last_clean":"2024-09-15T23:15:42.610726+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:15:42.610726+0000","last_undegraded":"2024-09-15T23:15:42.610726+0000","last_fullsized":"2024-09-15T23:15:42.610726+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613153+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:15:42.613153+0000","last_peered":"2024-09-15T23:15:42.613153+0000","last_clean":"2024-09-15T23:15:42.613153+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:15:42.613153+0000","last_undegraded":"2024-09-15T23:15:42.613153+0000","last_fullsized":"2024-09-15T23:15:42.613153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221488+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:15:42.221488+0000","last_peered":"2024-09-15T23:15:42.221488+0000","last_clean":"2024-09-15T23:15:42.221488+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:15:42.221488+0000","last_undegraded":"2024-09-15T23:15:42.221488+0000","last_fullsized":"2024-09-15T23:15:42.221488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316851+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:15:43.316851+0000","last_peered":"2024-09-15T23:15:43.316851+0000","last_clean":"2024-09-15T23:15:43.316851+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:15:43.316851+0000","last_undegraded":"2024-09-15T23:15:43.316851+0000","last_fullsized":"2024-09-15T23:15:43.316851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610204+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:15:42.610204+0000","last_peered":"2024-09-15T23:15:42.610204+0000","last_clean":"2024-09-15T23:15:42.610204+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:15:42.610204+0000","last_undegraded":"2024-09-15T23:15:42.610204+0000","last_fullsized":"2024-09-15T23:15:42.610204+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609588+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:15:42.609588+0000","last_peered":"2024-09-15T23:15:42.609588+0000","last_clean":"2024-09-15T23:15:42.609588+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:15:42.609588+0000","last_undegraded":"2024-09-15T23:15:42.609588+0000","last_fullsized":"2024-09-15T23:15:42.609588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612535+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:15:42.612535+0000","last_peered":"2024-09-15T23:15:42.612535+0000","last_clean":"2024-09-15T23:15:42.612535+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:15:42.612535+0000","last_undegraded":"2024-09-15T23:15:42.612535+0000","last_fullsized":"2024-09-15T23:15:42.612535+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096427+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:15:42.096427+0000","last_peered":"2024-09-15T23:15:42.096427+0000","last_clean":"2024-09-15T23:15:42.096427+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:15:42.096427+0000","last_undegraded":"2024-09-15T23:15:42.096427+0000","last_fullsized":"2024-09-15T23:15:42.096427+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096366+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:15:42.096366+0000","last_peered":"2024-09-15T23:15:42.096366+0000","last_clean":"2024-09-15T23:15:42.096366+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:15:42.096366+0000","last_undegraded":"2024-09-15T23:15:42.096366+0000","last_fullsized":"2024-09-15T23:15:42.096366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910172+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:15:42.910172+0000","last_peered":"2024-09-15T23:15:42.910172+0000","last_clean":"2024-09-15T23:15:42.910172+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:15:42.910172+0000","last_undegraded":"2024-09-15T23:15:42.910172+0000","last_fullsized":"2024-09-15T23:15:42.910172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67632,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":616,"num_read_kb":380,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":202896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514536,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522238,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743469,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1159168,"data_stored":1172882,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906995,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1032192,"data_stored":526458,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135928,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522170,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364860,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1077248,"data_stored":1118334,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528386,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":517538,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790023,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1085440,"data_stored":1113922,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1019904,"data_stored":522188,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1007616,"data_stored":508387,"data_compressed":26534,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":582820,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":580936,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1019904,"data_stored":509581,"data_compressed":26534,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":65372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:15:48.108 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T23:15:48.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 -- 172.21.15.62:0/3722900717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8100a10 msgr2=0x7fe1b8100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 --2- 172.21.15.62:0/3722900717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8100a10 0x7fe1b8100e10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8009900 tx=0x7fe1a802a410 comp rx=0 tx=0).stop 2024-09-15T23:15:48.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 -- 172.21.15.62:0/3722900717 shutdown_connections 2024-09-15T23:15:48.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 --2- 172.21.15.62:0/3722900717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b81025c0 0x7fe1b8108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 --2- 172.21.15.62:0/3722900717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1b8101c00 0x7fe1b8102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 --2- 172.21.15.62:0/3722900717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8100a10 0x7fe1b8100e10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 -- 172.21.15.62:0/3722900717 >> 172.21.15.62:0/3722900717 conn(0x7fe1b80fc170 msgr2=0x7fe1b80fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:48.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 -- 172.21.15.62:0/3722900717 shutdown_connections 2024-09-15T23:15:48.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 -- 172.21.15.62:0/3722900717 wait complete. 2024-09-15T23:15:48.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 Processor -- start 2024-09-15T23:15:48.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.245+0000 7fe1be31a640 1 -- start start 2024-09-15T23:15:48.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1be31a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1b8100a10 0x7fe1b81049a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1be31a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8101c00 0x7fe1b8102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1be31a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b81025c0 0x7fe1b8103520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1be31a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b810c690 con 0x7fe1b8101c00 2024-09-15T23:15:48.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1be31a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b810c510 con 0x7fe1b81025c0 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1be31a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b810c810 con 0x7fe1b8100a10 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8101c00 0x7fe1b8102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8101c00 0x7fe1b8102fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44792/0 (socket says 172.21.15.62:44792) 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b77fe640 1 -- 172.21.15.62:0/505245532 learned_addr learned my addr 172.21.15.62:0/505245532 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b7fff640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1b8100a10 0x7fe1b81049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1bc8b7640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b81025c0 0x7fe1b8103520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b77fe640 1 -- 172.21.15.62:0/505245532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1b8100a10 msgr2=0x7fe1b81049a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b77fe640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1b8100a10 0x7fe1b81049a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b77fe640 1 -- 172.21.15.62:0/505245532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b81025c0 msgr2=0x7fe1b8103520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b77fe640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b81025c0 0x7fe1b8103520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b77fe640 1 -- 172.21.15.62:0/505245532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b8103db0 con 0x7fe1b8101c00 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1bc8b7640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b81025c0 0x7fe1b8103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b77fe640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8101c00 0x7fe1b8102fe0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00c970 tx=0x7fe1ac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:48.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.246+0000 7fe1b57fa640 1 -- 172.21.15.62:0/505245532 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac007bb0 con 0x7fe1b8101c00 2024-09-15T23:15:48.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.247+0000 7fe1b57fa640 1 -- 172.21.15.62:0/505245532 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1ac007d50 con 0x7fe1b8101c00 2024-09-15T23:15:48.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.247+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b81a1f30 con 0x7fe1b8101c00 2024-09-15T23:15:48.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.247+0000 7fe1b57fa640 1 -- 172.21.15.62:0/505245532 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac00f450 con 0x7fe1b8101c00 2024-09-15T23:15:48.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.247+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b81a2340 con 0x7fe1b8101c00 2024-09-15T23:15:48.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.248+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b81a2630 con 0x7fe1b8101c00 2024-09-15T23:15:48.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.249+0000 7fe1b57fa640 1 -- 172.21.15.62:0/505245532 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe1ac004fe0 con 0x7fe1b8101c00 2024-09-15T23:15:48.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.250+0000 7fe1b57fa640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1a00a3250 0x7fe1a00a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.250+0000 7fe1b7fff640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1a00a3250 0x7fe1a00a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.250+0000 7fe1b57fa640 1 -- 172.21.15.62:0/505245532 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe1ac0cfda0 con 0x7fe1b8101c00 2024-09-15T23:15:48.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.250+0000 7fe1b7fff640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1a00a3250 0x7fe1a00a5710 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8005ab0 tx=0x7fe1a8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:48.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.253+0000 7fe1b57fa640 1 -- 172.21.15.62:0/505245532 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe1ac014030 con 0x7fe1b8101c00 2024-09-15T23:15:48.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.426+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1b8069ac0 con 0x7fe1a00a3250 2024-09-15T23:15:48.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.434+0000 7fe1b57fa640 1 -- 172.21.15.62:0/505245532 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340816 (secure 0 0 0) 0x7fe1b8069ac0 con 0x7fe1a00a3250 2024-09-15T23:15:48.438 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:48.440 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1a00a3250 msgr2=0x7fe1a00a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1a00a3250 0x7fe1a00a5710 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8005ab0 tx=0x7fe1a8000bc0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8101c00 msgr2=0x7fe1b8102fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8101c00 0x7fe1b8102fe0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00c970 tx=0x7fe1ac00ce30 comp rx=0 tx=0).stop 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 shutdown_connections 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1a00a3250 0x7fe1a00a5710 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b81025c0 0x7fe1b8103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1b8101c00 0x7fe1b8102fe0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 --2- 172.21.15.62:0/505245532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1b8100a10 0x7fe1b81049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 >> 172.21.15.62:0/505245532 conn(0x7fe1b80fc170 msgr2=0x7fe1b80693b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:48.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 shutdown_connections 2024-09-15T23:15:48.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.437+0000 7fe1be31a640 1 -- 172.21.15.62:0/505245532 wait complete. 2024-09-15T23:15:48.449 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-15T23:15:47.725593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658234,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":742,"num_read_kb":489,"num_write":361,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369972,"kb_used_data":12332,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571852,"statfs":{"total":1151940427776,"available":1151561576448,"internally_reserved":0,"allocated":12627968,"data_stored":8197454,"data_compressed":455690,"data_compressed_allocated":618496,"data_compressed_original":2736128,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":698,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":31,"num_write":10,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002136"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609573+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:15:42.609573+0000","last_peered":"2024-09-15T23:15:42.609573+0000","last_clean":"2024-09-15T23:15:42.609573+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:15:42.609573+0000","last_undegraded":"2024-09-15T23:15:42.609573+0000","last_fullsized":"2024-09-15T23:15:42.609573+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221889+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:15:42.221889+0000","last_peered":"2024-09-15T23:15:42.221889+0000","last_clean":"2024-09-15T23:15:42.221889+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:15:42.221889+0000","last_undegraded":"2024-09-15T23:15:42.221889+0000","last_fullsized":"2024-09-15T23:15:42.221889+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613665+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:15:42.613665+0000","last_peered":"2024-09-15T23:15:42.613665+0000","last_clean":"2024-09-15T23:15:42.613665+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:15:42.613665+0000","last_undegraded":"2024-09-15T23:15:42.613665+0000","last_fullsized":"2024-09-15T23:15:42.613665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.222006+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:15:42.222006+0000","last_peered":"2024-09-15T23:15:42.222006+0000","last_clean":"2024-09-15T23:15:42.222006+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:15:42.222006+0000","last_undegraded":"2024-09-15T23:15:42.222006+0000","last_fullsized":"2024-09-15T23:15:42.222006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096731+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:15:42.096731+0000","last_peered":"2024-09-15T23:15:42.096731+0000","last_clean":"2024-09-15T23:15:42.096731+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:15:42.096731+0000","last_undegraded":"2024-09-15T23:15:42.096731+0000","last_fullsized":"2024-09-15T23:15:42.096731+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611378+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:15:42.611378+0000","last_peered":"2024-09-15T23:15:42.611378+0000","last_clean":"2024-09-15T23:15:42.611378+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:15:42.611378+0000","last_undegraded":"2024-09-15T23:15:42.611378+0000","last_fullsized":"2024-09-15T23:15:42.611378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909808+0000","last_change":"2024-09-15T23:14:41.659816+0000","last_active":"2024-09-15T23:15:42.909808+0000","last_peered":"2024-09-15T23:15:42.909808+0000","last_clean":"2024-09-15T23:15:42.909808+0000","last_became_active":"2024-09-15T23:14:41.641611+0000","last_became_peered":"2024-09-15T23:14:41.641611+0000","last_unstale":"2024-09-15T23:15:42.909808+0000","last_undegraded":"2024-09-15T23:15:42.909808+0000","last_fullsized":"2024-09-15T23:15:42.909808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:23.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612963+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:15:42.612963+0000","last_peered":"2024-09-15T23:15:42.612963+0000","last_clean":"2024-09-15T23:15:42.612963+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:15:42.612963+0000","last_undegraded":"2024-09-15T23:15:42.612963+0000","last_fullsized":"2024-09-15T23:15:42.612963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610025+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:15:42.610025+0000","last_peered":"2024-09-15T23:15:42.610025+0000","last_clean":"2024-09-15T23:15:42.610025+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:15:42.610025+0000","last_undegraded":"2024-09-15T23:15:42.610025+0000","last_fullsized":"2024-09-15T23:15:42.610025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614867+0000","last_change":"2024-09-15T23:15:42.614867+0000","last_active":"2024-09-15T23:15:42.614867+0000","last_peered":"2024-09-15T23:15:42.614867+0000","last_clean":"2024-09-15T23:15:42.614867+0000","last_became_active":"2024-09-15T23:15:42.614459+0000","last_became_peered":"2024-09-15T23:15:42.614459+0000","last_unstale":"2024-09-15T23:15:42.614867+0000","last_undegraded":"2024-09-15T23:15:42.614867+0000","last_fullsized":"2024-09-15T23:15:42.614867+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:15.254442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611583+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:15:42.611583+0000","last_peered":"2024-09-15T23:15:42.611583+0000","last_clean":"2024-09-15T23:15:42.611583+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:15:42.611583+0000","last_undegraded":"2024-09-15T23:15:42.611583+0000","last_fullsized":"2024-09-15T23:15:42.611583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317095+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:15:43.317095+0000","last_peered":"2024-09-15T23:15:43.317095+0000","last_clean":"2024-09-15T23:15:43.317095+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:15:43.317095+0000","last_undegraded":"2024-09-15T23:15:43.317095+0000","last_fullsized":"2024-09-15T23:15:43.317095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":56,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609750+0000","last_change":"2024-09-15T23:14:41.646320+0000","last_active":"2024-09-15T23:15:42.609750+0000","last_peered":"2024-09-15T23:15:42.609750+0000","last_clean":"2024-09-15T23:15:42.609750+0000","last_became_active":"2024-09-15T23:14:41.641933+0000","last_became_peered":"2024-09-15T23:14:41.641933+0000","last_unstale":"2024-09-15T23:15:42.609750+0000","last_undegraded":"2024-09-15T23:15:42.609750+0000","last_fullsized":"2024-09-15T23:15:42.609750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:49.417407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611897+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:15:42.611897+0000","last_peered":"2024-09-15T23:15:42.611897+0000","last_clean":"2024-09-15T23:15:42.611897+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:15:42.611897+0000","last_undegraded":"2024-09-15T23:15:42.611897+0000","last_fullsized":"2024-09-15T23:15:42.611897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221618+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:15:42.221618+0000","last_peered":"2024-09-15T23:15:42.221618+0000","last_clean":"2024-09-15T23:15:42.221618+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:15:42.221618+0000","last_undegraded":"2024-09-15T23:15:42.221618+0000","last_fullsized":"2024-09-15T23:15:42.221618+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613807+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:15:42.613807+0000","last_peered":"2024-09-15T23:15:42.613807+0000","last_clean":"2024-09-15T23:15:42.613807+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:15:42.613807+0000","last_undegraded":"2024-09-15T23:15:42.613807+0000","last_fullsized":"2024-09-15T23:15:42.613807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612402+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:15:42.612402+0000","last_peered":"2024-09-15T23:15:42.612402+0000","last_clean":"2024-09-15T23:15:42.612402+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:15:42.612402+0000","last_undegraded":"2024-09-15T23:15:42.612402+0000","last_fullsized":"2024-09-15T23:15:42.612402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909877+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:15:42.909877+0000","last_peered":"2024-09-15T23:15:42.909877+0000","last_clean":"2024-09-15T23:15:42.909877+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:15:42.909877+0000","last_undegraded":"2024-09-15T23:15:42.909877+0000","last_fullsized":"2024-09-15T23:15:42.909877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221790+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:15:42.221790+0000","last_peered":"2024-09-15T23:15:42.221790+0000","last_clean":"2024-09-15T23:15:42.221790+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:15:42.221790+0000","last_undegraded":"2024-09-15T23:15:42.221790+0000","last_fullsized":"2024-09-15T23:15:42.221790+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609295+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:15:42.609295+0000","last_peered":"2024-09-15T23:15:42.609295+0000","last_clean":"2024-09-15T23:15:42.609295+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:15:42.609295+0000","last_undegraded":"2024-09-15T23:15:42.609295+0000","last_fullsized":"2024-09-15T23:15:42.609295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609453+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:15:42.609453+0000","last_peered":"2024-09-15T23:15:42.609453+0000","last_clean":"2024-09-15T23:15:42.609453+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:15:42.609453+0000","last_undegraded":"2024-09-15T23:15:42.609453+0000","last_fullsized":"2024-09-15T23:15:42.609453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613324+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:15:42.613324+0000","last_peered":"2024-09-15T23:15:42.613324+0000","last_clean":"2024-09-15T23:15:42.613324+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:15:42.613324+0000","last_undegraded":"2024-09-15T23:15:42.613324+0000","last_fullsized":"2024-09-15T23:15:42.613324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612027+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:15:42.612027+0000","last_peered":"2024-09-15T23:15:42.612027+0000","last_clean":"2024-09-15T23:15:42.612027+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:15:42.612027+0000","last_undegraded":"2024-09-15T23:15:42.612027+0000","last_fullsized":"2024-09-15T23:15:42.612027+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096594+0000","last_change":"2024-09-15T23:14:42.009605+0000","last_active":"2024-09-15T23:15:42.096594+0000","last_peered":"2024-09-15T23:15:42.096594+0000","last_clean":"2024-09-15T23:15:42.096594+0000","last_became_active":"2024-09-15T23:14:42.009237+0000","last_became_peered":"2024-09-15T23:14:42.009237+0000","last_unstale":"2024-09-15T23:15:42.096594+0000","last_undegraded":"2024-09-15T23:15:42.096594+0000","last_fullsized":"2024-09-15T23:15:42.096594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:11:08.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612739+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:15:42.612739+0000","last_peered":"2024-09-15T23:15:42.612739+0000","last_clean":"2024-09-15T23:15:42.612739+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:15:42.612739+0000","last_undegraded":"2024-09-15T23:15:42.612739+0000","last_fullsized":"2024-09-15T23:15:42.612739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614124+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:15:42.614124+0000","last_peered":"2024-09-15T23:15:42.614124+0000","last_clean":"2024-09-15T23:15:42.614124+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:15:42.614124+0000","last_undegraded":"2024-09-15T23:15:42.614124+0000","last_fullsized":"2024-09-15T23:15:42.614124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.222041+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:15:42.222041+0000","last_peered":"2024-09-15T23:15:42.222041+0000","last_clean":"2024-09-15T23:15:42.222041+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:15:42.222041+0000","last_undegraded":"2024-09-15T23:15:42.222041+0000","last_fullsized":"2024-09-15T23:15:42.222041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'14","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610065+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:15:42.610065+0000","last_peered":"2024-09-15T23:15:42.610065+0000","last_clean":"2024-09-15T23:15:42.610065+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:15:42.610065+0000","last_undegraded":"2024-09-15T23:15:42.610065+0000","last_fullsized":"2024-09-15T23:15:42.610065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221420+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:15:42.221420+0000","last_peered":"2024-09-15T23:15:42.221420+0000","last_clean":"2024-09-15T23:15:42.221420+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:15:42.221420+0000","last_undegraded":"2024-09-15T23:15:42.221420+0000","last_fullsized":"2024-09-15T23:15:42.221420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612814+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:15:42.612814+0000","last_peered":"2024-09-15T23:15:42.612814+0000","last_clean":"2024-09-15T23:15:42.612814+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:15:42.612814+0000","last_undegraded":"2024-09-15T23:15:42.612814+0000","last_fullsized":"2024-09-15T23:15:42.612814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610084+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:15:42.610084+0000","last_peered":"2024-09-15T23:15:42.610084+0000","last_clean":"2024-09-15T23:15:42.610084+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:15:42.610084+0000","last_undegraded":"2024-09-15T23:15:42.610084+0000","last_fullsized":"2024-09-15T23:15:42.610084+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909751+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:15:42.909751+0000","last_peered":"2024-09-15T23:15:42.909751+0000","last_clean":"2024-09-15T23:15:42.909751+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:15:42.909751+0000","last_undegraded":"2024-09-15T23:15:42.909751+0000","last_fullsized":"2024-09-15T23:15:42.909751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610218+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:15:42.610218+0000","last_peered":"2024-09-15T23:15:42.610218+0000","last_clean":"2024-09-15T23:15:42.610218+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:15:42.610218+0000","last_undegraded":"2024-09-15T23:15:42.610218+0000","last_fullsized":"2024-09-15T23:15:42.610218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609669+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:15:42.609669+0000","last_peered":"2024-09-15T23:15:42.609669+0000","last_clean":"2024-09-15T23:15:42.609669+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:15:42.609669+0000","last_undegraded":"2024-09-15T23:15:42.609669+0000","last_fullsized":"2024-09-15T23:15:42.609669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609473+0000","last_change":"2024-09-15T23:14:41.995466+0000","last_active":"2024-09-15T23:15:42.609473+0000","last_peered":"2024-09-15T23:15:42.609473+0000","last_clean":"2024-09-15T23:15:42.609473+0000","last_became_active":"2024-09-15T23:14:41.994915+0000","last_became_peered":"2024-09-15T23:14:41.994915+0000","last_unstale":"2024-09-15T23:15:42.609473+0000","last_undegraded":"2024-09-15T23:15:42.609473+0000","last_fullsized":"2024-09-15T23:15:42.609473+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:04.121596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610393+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:15:42.610393+0000","last_peered":"2024-09-15T23:15:42.610393+0000","last_clean":"2024-09-15T23:15:42.610393+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:15:42.610393+0000","last_undegraded":"2024-09-15T23:15:42.610393+0000","last_fullsized":"2024-09-15T23:15:42.610393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317131+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:15:43.317131+0000","last_peered":"2024-09-15T23:15:43.317131+0000","last_clean":"2024-09-15T23:15:43.317131+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:15:43.317131+0000","last_undegraded":"2024-09-15T23:15:43.317131+0000","last_fullsized":"2024-09-15T23:15:43.317131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626237+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:15:42.626237+0000","last_peered":"2024-09-15T23:15:42.626237+0000","last_clean":"2024-09-15T23:15:42.626237+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:15:42.626237+0000","last_undegraded":"2024-09-15T23:15:42.626237+0000","last_fullsized":"2024-09-15T23:15:42.626237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.618846+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:15:43.618846+0000","last_peered":"2024-09-15T23:15:43.618846+0000","last_clean":"2024-09-15T23:15:43.618846+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:15:43.618846+0000","last_undegraded":"2024-09-15T23:15:43.618846+0000","last_fullsized":"2024-09-15T23:15:43.618846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":97,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612986+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:15:42.612986+0000","last_peered":"2024-09-15T23:15:42.612986+0000","last_clean":"2024-09-15T23:15:42.612986+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:15:42.612986+0000","last_undegraded":"2024-09-15T23:15:42.612986+0000","last_fullsized":"2024-09-15T23:15:42.612986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.617748+0000","last_change":"2024-09-15T23:15:42.617748+0000","last_active":"2024-09-15T23:15:42.617748+0000","last_peered":"2024-09-15T23:15:42.617748+0000","last_clean":"2024-09-15T23:15:42.617748+0000","last_became_active":"2024-09-15T23:15:42.617320+0000","last_became_peered":"2024-09-15T23:15:42.617320+0000","last_unstale":"2024-09-15T23:15:42.617748+0000","last_undegraded":"2024-09-15T23:15:42.617748+0000","last_fullsized":"2024-09-15T23:15:42.617748+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:47.900049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316911+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:15:43.316911+0000","last_peered":"2024-09-15T23:15:43.316911+0000","last_clean":"2024-09-15T23:15:43.316911+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:15:43.316911+0000","last_undegraded":"2024-09-15T23:15:43.316911+0000","last_fullsized":"2024-09-15T23:15:43.316911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626536+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:15:42.626536+0000","last_peered":"2024-09-15T23:15:42.626536+0000","last_clean":"2024-09-15T23:15:42.626536+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:15:42.626536+0000","last_undegraded":"2024-09-15T23:15:42.626536+0000","last_fullsized":"2024-09-15T23:15:42.626536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221750+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:42.221750+0000","last_peered":"2024-09-15T23:15:42.221750+0000","last_clean":"2024-09-15T23:15:42.221750+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:15:42.221750+0000","last_undegraded":"2024-09-15T23:15:42.221750+0000","last_fullsized":"2024-09-15T23:15:42.221750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612474+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:15:42.612474+0000","last_peered":"2024-09-15T23:15:42.612474+0000","last_clean":"2024-09-15T23:15:42.612474+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:15:42.612474+0000","last_undegraded":"2024-09-15T23:15:42.612474+0000","last_fullsized":"2024-09-15T23:15:42.612474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317208+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:15:43.317208+0000","last_peered":"2024-09-15T23:15:43.317208+0000","last_clean":"2024-09-15T23:15:43.317208+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:15:43.317208+0000","last_undegraded":"2024-09-15T23:15:43.317208+0000","last_fullsized":"2024-09-15T23:15:43.317208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317366+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:15:43.317366+0000","last_peered":"2024-09-15T23:15:43.317366+0000","last_clean":"2024-09-15T23:15:43.317366+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:15:43.317366+0000","last_undegraded":"2024-09-15T23:15:43.317366+0000","last_fullsized":"2024-09-15T23:15:43.317366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317438+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:15:43.317438+0000","last_peered":"2024-09-15T23:15:43.317438+0000","last_clean":"2024-09-15T23:15:43.317438+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:15:43.317438+0000","last_undegraded":"2024-09-15T23:15:43.317438+0000","last_fullsized":"2024-09-15T23:15:43.317438+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614172+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:15:42.614172+0000","last_peered":"2024-09-15T23:15:42.614172+0000","last_clean":"2024-09-15T23:15:42.614172+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:15:42.614172+0000","last_undegraded":"2024-09-15T23:15:42.614172+0000","last_fullsized":"2024-09-15T23:15:42.614172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612428+0000","last_change":"2024-09-15T23:14:41.994950+0000","last_active":"2024-09-15T23:15:42.612428+0000","last_peered":"2024-09-15T23:15:42.612428+0000","last_clean":"2024-09-15T23:15:42.612428+0000","last_became_active":"2024-09-15T23:14:41.994429+0000","last_became_peered":"2024-09-15T23:14:41.994429+0000","last_unstale":"2024-09-15T23:15:42.612428+0000","last_undegraded":"2024-09-15T23:15:42.612428+0000","last_fullsized":"2024-09-15T23:15:42.612428+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:24.056756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317319+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:15:43.317319+0000","last_peered":"2024-09-15T23:15:43.317319+0000","last_clean":"2024-09-15T23:15:43.317319+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:15:43.317319+0000","last_undegraded":"2024-09-15T23:15:43.317319+0000","last_fullsized":"2024-09-15T23:15:43.317319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.627013+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:15:42.627013+0000","last_peered":"2024-09-15T23:15:42.627013+0000","last_clean":"2024-09-15T23:15:42.627013+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:15:42.627013+0000","last_undegraded":"2024-09-15T23:15:42.627013+0000","last_fullsized":"2024-09-15T23:15:42.627013+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096349+0000","last_change":"2024-09-15T23:14:41.640471+0000","last_active":"2024-09-15T23:15:42.096349+0000","last_peered":"2024-09-15T23:15:42.096349+0000","last_clean":"2024-09-15T23:15:42.096349+0000","last_became_active":"2024-09-15T23:14:41.640017+0000","last_became_peered":"2024-09-15T23:14:41.640017+0000","last_unstale":"2024-09-15T23:15:42.096349+0000","last_undegraded":"2024-09-15T23:15:42.096349+0000","last_fullsized":"2024-09-15T23:15:42.096349+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:58.842636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317343+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:15:43.317343+0000","last_peered":"2024-09-15T23:15:43.317343+0000","last_clean":"2024-09-15T23:15:43.317343+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:15:43.317343+0000","last_undegraded":"2024-09-15T23:15:43.317343+0000","last_fullsized":"2024-09-15T23:15:43.317343+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613353+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:15:42.613353+0000","last_peered":"2024-09-15T23:15:42.613353+0000","last_clean":"2024-09-15T23:15:42.613353+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:15:42.613353+0000","last_undegraded":"2024-09-15T23:15:42.613353+0000","last_fullsized":"2024-09-15T23:15:42.613353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.625937+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:15:42.625937+0000","last_peered":"2024-09-15T23:15:42.625937+0000","last_clean":"2024-09-15T23:15:42.625937+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:15:42.625937+0000","last_undegraded":"2024-09-15T23:15:42.625937+0000","last_fullsized":"2024-09-15T23:15:42.625937+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909962+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:15:42.909962+0000","last_peered":"2024-09-15T23:15:42.909962+0000","last_clean":"2024-09-15T23:15:42.909962+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:15:42.909962+0000","last_undegraded":"2024-09-15T23:15:42.909962+0000","last_fullsized":"2024-09-15T23:15:42.909962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610024+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:15:42.610024+0000","last_peered":"2024-09-15T23:15:42.610024+0000","last_clean":"2024-09-15T23:15:42.610024+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:15:42.610024+0000","last_undegraded":"2024-09-15T23:15:42.610024+0000","last_fullsized":"2024-09-15T23:15:42.610024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.625935+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:15:42.625935+0000","last_peered":"2024-09-15T23:15:42.625935+0000","last_clean":"2024-09-15T23:15:42.625935+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:15:42.625935+0000","last_undegraded":"2024-09-15T23:15:42.625935+0000","last_fullsized":"2024-09-15T23:15:42.625935+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626111+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:15:42.626111+0000","last_peered":"2024-09-15T23:15:42.626111+0000","last_clean":"2024-09-15T23:15:42.626111+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:15:42.626111+0000","last_undegraded":"2024-09-15T23:15:42.626111+0000","last_fullsized":"2024-09-15T23:15:42.626111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909569+0000","last_change":"2024-09-15T23:14:41.653786+0000","last_active":"2024-09-15T23:15:42.909569+0000","last_peered":"2024-09-15T23:15:42.909569+0000","last_clean":"2024-09-15T23:15:42.909569+0000","last_became_active":"2024-09-15T23:14:41.641472+0000","last_became_peered":"2024-09-15T23:14:41.641472+0000","last_unstale":"2024-09-15T23:15:42.909569+0000","last_undegraded":"2024-09-15T23:15:42.909569+0000","last_fullsized":"2024-09-15T23:15:42.909569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:16.010142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613267+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:15:42.613267+0000","last_peered":"2024-09-15T23:15:42.613267+0000","last_clean":"2024-09-15T23:15:42.613267+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:15:42.613267+0000","last_undegraded":"2024-09-15T23:15:42.613267+0000","last_fullsized":"2024-09-15T23:15:42.613267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316776+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:15:43.316776+0000","last_peered":"2024-09-15T23:15:43.316776+0000","last_clean":"2024-09-15T23:15:43.316776+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:15:43.316776+0000","last_undegraded":"2024-09-15T23:15:43.316776+0000","last_fullsized":"2024-09-15T23:15:43.316776+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.619522+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:15:43.619522+0000","last_peered":"2024-09-15T23:15:43.619522+0000","last_clean":"2024-09-15T23:15:43.619522+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:15:43.619522+0000","last_undegraded":"2024-09-15T23:15:43.619522+0000","last_fullsized":"2024-09-15T23:15:43.619522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609376+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:15:42.609376+0000","last_peered":"2024-09-15T23:15:42.609376+0000","last_clean":"2024-09-15T23:15:42.609376+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:15:42.609376+0000","last_undegraded":"2024-09-15T23:15:42.609376+0000","last_fullsized":"2024-09-15T23:15:42.609376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":48,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317071+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:15:43.317071+0000","last_peered":"2024-09-15T23:15:43.317071+0000","last_clean":"2024-09-15T23:15:43.317071+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:15:43.317071+0000","last_undegraded":"2024-09-15T23:15:43.317071+0000","last_fullsized":"2024-09-15T23:15:43.317071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626254+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:15:42.626254+0000","last_peered":"2024-09-15T23:15:42.626254+0000","last_clean":"2024-09-15T23:15:42.626254+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:15:42.626254+0000","last_undegraded":"2024-09-15T23:15:42.626254+0000","last_fullsized":"2024-09-15T23:15:42.626254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909759+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:15:42.909759+0000","last_peered":"2024-09-15T23:15:42.909759+0000","last_clean":"2024-09-15T23:15:42.909759+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:15:42.909759+0000","last_undegraded":"2024-09-15T23:15:42.909759+0000","last_fullsized":"2024-09-15T23:15:42.909759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612005+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:15:42.612005+0000","last_peered":"2024-09-15T23:15:42.612005+0000","last_clean":"2024-09-15T23:15:42.612005+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:15:42.612005+0000","last_undegraded":"2024-09-15T23:15:42.612005+0000","last_fullsized":"2024-09-15T23:15:42.612005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613889+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:15:42.613889+0000","last_peered":"2024-09-15T23:15:42.613889+0000","last_clean":"2024-09-15T23:15:42.613889+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:15:42.613889+0000","last_undegraded":"2024-09-15T23:15:42.613889+0000","last_fullsized":"2024-09-15T23:15:42.613889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626638+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:15:42.626638+0000","last_peered":"2024-09-15T23:15:42.626638+0000","last_clean":"2024-09-15T23:15:42.626638+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:15:42.626638+0000","last_undegraded":"2024-09-15T23:15:42.626638+0000","last_fullsized":"2024-09-15T23:15:42.626638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609857+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:15:42.609857+0000","last_peered":"2024-09-15T23:15:42.609857+0000","last_clean":"2024-09-15T23:15:42.609857+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:15:42.609857+0000","last_undegraded":"2024-09-15T23:15:42.609857+0000","last_fullsized":"2024-09-15T23:15:42.609857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":281,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:45.631571+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:15:45.631571+0000","last_peered":"2024-09-15T23:15:45.631571+0000","last_clean":"2024-09-15T23:15:45.631571+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:15:45.631571+0000","last_undegraded":"2024-09-15T23:15:45.631571+0000","last_fullsized":"2024-09-15T23:15:45.631571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63488,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":207,"num_read_kb":166,"num_write":37,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609007+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:15:42.609007+0000","last_peered":"2024-09-15T23:15:42.609007+0000","last_clean":"2024-09-15T23:15:42.609007+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:15:42.609007+0000","last_undegraded":"2024-09-15T23:15:42.609007+0000","last_fullsized":"2024-09-15T23:15:42.609007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612977+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:15:42.612977+0000","last_peered":"2024-09-15T23:15:42.612977+0000","last_clean":"2024-09-15T23:15:42.612977+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:15:42.612977+0000","last_undegraded":"2024-09-15T23:15:42.612977+0000","last_fullsized":"2024-09-15T23:15:42.612977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626922+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:15:42.626922+0000","last_peered":"2024-09-15T23:15:42.626922+0000","last_clean":"2024-09-15T23:15:42.626922+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:15:42.626922+0000","last_undegraded":"2024-09-15T23:15:42.626922+0000","last_fullsized":"2024-09-15T23:15:42.626922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612007+0000","last_change":"2024-09-15T23:14:41.641820+0000","last_active":"2024-09-15T23:15:42.612007+0000","last_peered":"2024-09-15T23:15:42.612007+0000","last_clean":"2024-09-15T23:15:42.612007+0000","last_became_active":"2024-09-15T23:14:41.641282+0000","last_became_peered":"2024-09-15T23:14:41.641282+0000","last_unstale":"2024-09-15T23:15:42.612007+0000","last_undegraded":"2024-09-15T23:15:42.612007+0000","last_fullsized":"2024-09-15T23:15:42.612007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:43.472601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096735+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:15:42.096735+0000","last_peered":"2024-09-15T23:15:42.096735+0000","last_clean":"2024-09-15T23:15:42.096735+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:15:42.096735+0000","last_undegraded":"2024-09-15T23:15:42.096735+0000","last_fullsized":"2024-09-15T23:15:42.096735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317368+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:15:43.317368+0000","last_peered":"2024-09-15T23:15:43.317368+0000","last_clean":"2024-09-15T23:15:43.317368+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:15:43.317368+0000","last_undegraded":"2024-09-15T23:15:43.317368+0000","last_fullsized":"2024-09-15T23:15:43.317368+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608807+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:15:42.608807+0000","last_peered":"2024-09-15T23:15:42.608807+0000","last_clean":"2024-09-15T23:15:42.608807+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:15:42.608807+0000","last_undegraded":"2024-09-15T23:15:42.608807+0000","last_fullsized":"2024-09-15T23:15:42.608807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612577+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:15:42.612577+0000","last_peered":"2024-09-15T23:15:42.612577+0000","last_clean":"2024-09-15T23:15:42.612577+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:15:42.612577+0000","last_undegraded":"2024-09-15T23:15:42.612577+0000","last_fullsized":"2024-09-15T23:15:42.612577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:44.628850+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:15:44.628850+0000","last_peered":"2024-09-15T23:15:44.628850+0000","last_clean":"2024-09-15T23:15:44.628850+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:15:44.628850+0000","last_undegraded":"2024-09-15T23:15:44.628850+0000","last_fullsized":"2024-09-15T23:15:44.628850+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221487+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:15:42.221487+0000","last_peered":"2024-09-15T23:15:42.221487+0000","last_clean":"2024-09-15T23:15:42.221487+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:15:42.221487+0000","last_undegraded":"2024-09-15T23:15:42.221487+0000","last_fullsized":"2024-09-15T23:15:42.221487+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096328+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:15:42.096328+0000","last_peered":"2024-09-15T23:15:42.096328+0000","last_clean":"2024-09-15T23:15:42.096328+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:15:42.096328+0000","last_undegraded":"2024-09-15T23:15:42.096328+0000","last_fullsized":"2024-09-15T23:15:42.096328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608984+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:15:42.608984+0000","last_peered":"2024-09-15T23:15:42.608984+0000","last_clean":"2024-09-15T23:15:42.608984+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:15:42.608984+0000","last_undegraded":"2024-09-15T23:15:42.608984+0000","last_fullsized":"2024-09-15T23:15:42.608984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626347+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:15:42.626347+0000","last_peered":"2024-09-15T23:15:42.626347+0000","last_clean":"2024-09-15T23:15:42.626347+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:15:42.626347+0000","last_undegraded":"2024-09-15T23:15:42.626347+0000","last_fullsized":"2024-09-15T23:15:42.626347+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909956+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:15:42.909956+0000","last_peered":"2024-09-15T23:15:42.909956+0000","last_clean":"2024-09-15T23:15:42.909956+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:15:42.909956+0000","last_undegraded":"2024-09-15T23:15:42.909956+0000","last_fullsized":"2024-09-15T23:15:42.909956+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221496+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:15:42.221496+0000","last_peered":"2024-09-15T23:15:42.221496+0000","last_clean":"2024-09-15T23:15:42.221496+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:15:42.221496+0000","last_undegraded":"2024-09-15T23:15:42.221496+0000","last_fullsized":"2024-09-15T23:15:42.221496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611692+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:15:42.611692+0000","last_peered":"2024-09-15T23:15:42.611692+0000","last_clean":"2024-09-15T23:15:42.611692+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:15:42.611692+0000","last_undegraded":"2024-09-15T23:15:42.611692+0000","last_fullsized":"2024-09-15T23:15:42.611692+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610033+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:15:42.610033+0000","last_peered":"2024-09-15T23:15:42.610033+0000","last_clean":"2024-09-15T23:15:42.610033+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:15:42.610033+0000","last_undegraded":"2024-09-15T23:15:42.610033+0000","last_fullsized":"2024-09-15T23:15:42.610033+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626302+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:15:42.626302+0000","last_peered":"2024-09-15T23:15:42.626302+0000","last_clean":"2024-09-15T23:15:42.626302+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:15:42.626302+0000","last_undegraded":"2024-09-15T23:15:42.626302+0000","last_fullsized":"2024-09-15T23:15:42.626302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613815+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:15:42.613815+0000","last_peered":"2024-09-15T23:15:42.613815+0000","last_clean":"2024-09-15T23:15:42.613815+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:15:42.613815+0000","last_undegraded":"2024-09-15T23:15:42.613815+0000","last_fullsized":"2024-09-15T23:15:42.613815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:44.629643+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:15:44.629643+0000","last_peered":"2024-09-15T23:15:44.629643+0000","last_clean":"2024-09-15T23:15:44.629643+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:15:44.629643+0000","last_undegraded":"2024-09-15T23:15:44.629643+0000","last_fullsized":"2024-09-15T23:15:44.629643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1156,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609500+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:15:42.609500+0000","last_peered":"2024-09-15T23:15:42.609500+0000","last_clean":"2024-09-15T23:15:42.609500+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:15:42.609500+0000","last_undegraded":"2024-09-15T23:15:42.609500+0000","last_fullsized":"2024-09-15T23:15:42.609500+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613805+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:15:42.613805+0000","last_peered":"2024-09-15T23:15:42.613805+0000","last_clean":"2024-09-15T23:15:42.613805+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:15:42.613805+0000","last_undegraded":"2024-09-15T23:15:42.613805+0000","last_fullsized":"2024-09-15T23:15:42.613805+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909660+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:15:42.909660+0000","last_peered":"2024-09-15T23:15:42.909660+0000","last_clean":"2024-09-15T23:15:42.909660+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:15:42.909660+0000","last_undegraded":"2024-09-15T23:15:42.909660+0000","last_fullsized":"2024-09-15T23:15:42.909660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614459+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:42.614459+0000","last_peered":"2024-09-15T23:15:42.614459+0000","last_clean":"2024-09-15T23:15:42.614459+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:15:42.614459+0000","last_undegraded":"2024-09-15T23:15:42.614459+0000","last_fullsized":"2024-09-15T23:15:42.614459+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613647+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:15:42.613647+0000","last_peered":"2024-09-15T23:15:42.613647+0000","last_clean":"2024-09-15T23:15:42.613647+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:15:42.613647+0000","last_undegraded":"2024-09-15T23:15:42.613647+0000","last_fullsized":"2024-09-15T23:15:42.613647+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096457+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:15:42.096457+0000","last_peered":"2024-09-15T23:15:42.096457+0000","last_clean":"2024-09-15T23:15:42.096457+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:15:42.096457+0000","last_undegraded":"2024-09-15T23:15:42.096457+0000","last_fullsized":"2024-09-15T23:15:42.096457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612897+0000","last_change":"2024-09-15T23:14:42.010010+0000","last_active":"2024-09-15T23:15:42.612897+0000","last_peered":"2024-09-15T23:15:42.612897+0000","last_clean":"2024-09-15T23:15:42.612897+0000","last_became_active":"2024-09-15T23:14:42.009603+0000","last_became_peered":"2024-09-15T23:14:42.009603+0000","last_unstale":"2024-09-15T23:15:42.612897+0000","last_undegraded":"2024-09-15T23:15:42.612897+0000","last_fullsized":"2024-09-15T23:15:42.612897+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:34:51.849548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910096+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:15:42.910096+0000","last_peered":"2024-09-15T23:15:42.910096+0000","last_clean":"2024-09-15T23:15:42.910096+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:15:42.910096+0000","last_undegraded":"2024-09-15T23:15:42.910096+0000","last_fullsized":"2024-09-15T23:15:42.910096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614200+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:15:42.614200+0000","last_peered":"2024-09-15T23:15:42.614200+0000","last_clean":"2024-09-15T23:15:42.614200+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:15:42.614200+0000","last_undegraded":"2024-09-15T23:15:42.614200+0000","last_fullsized":"2024-09-15T23:15:42.614200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221977+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:15:42.221977+0000","last_peered":"2024-09-15T23:15:42.221977+0000","last_clean":"2024-09-15T23:15:42.221977+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:15:42.221977+0000","last_undegraded":"2024-09-15T23:15:42.221977+0000","last_fullsized":"2024-09-15T23:15:42.221977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626113+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:15:42.626113+0000","last_peered":"2024-09-15T23:15:42.626113+0000","last_clean":"2024-09-15T23:15:42.626113+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:15:42.626113+0000","last_undegraded":"2024-09-15T23:15:42.626113+0000","last_fullsized":"2024-09-15T23:15:42.626113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221577+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:15:42.221577+0000","last_peered":"2024-09-15T23:15:42.221577+0000","last_clean":"2024-09-15T23:15:42.221577+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:15:42.221577+0000","last_undegraded":"2024-09-15T23:15:42.221577+0000","last_fullsized":"2024-09-15T23:15:42.221577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612683+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:15:42.612683+0000","last_peered":"2024-09-15T23:15:42.612683+0000","last_clean":"2024-09-15T23:15:42.612683+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:15:42.612683+0000","last_undegraded":"2024-09-15T23:15:42.612683+0000","last_fullsized":"2024-09-15T23:15:42.612683+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612593+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:15:42.612593+0000","last_peered":"2024-09-15T23:15:42.612593+0000","last_clean":"2024-09-15T23:15:42.612593+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:15:42.612593+0000","last_undegraded":"2024-09-15T23:15:42.612593+0000","last_fullsized":"2024-09-15T23:15:42.612593+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909865+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:15:42.909865+0000","last_peered":"2024-09-15T23:15:42.909865+0000","last_clean":"2024-09-15T23:15:42.909865+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:15:42.909865+0000","last_undegraded":"2024-09-15T23:15:42.909865+0000","last_fullsized":"2024-09-15T23:15:42.909865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614031+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:15:42.614031+0000","last_peered":"2024-09-15T23:15:42.614031+0000","last_clean":"2024-09-15T23:15:42.614031+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:15:42.614031+0000","last_undegraded":"2024-09-15T23:15:42.614031+0000","last_fullsized":"2024-09-15T23:15:42.614031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610009+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:15:42.610009+0000","last_peered":"2024-09-15T23:15:42.610009+0000","last_clean":"2024-09-15T23:15:42.610009+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:15:42.610009+0000","last_undegraded":"2024-09-15T23:15:42.610009+0000","last_fullsized":"2024-09-15T23:15:42.610009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910174+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:15:42.910174+0000","last_peered":"2024-09-15T23:15:42.910174+0000","last_clean":"2024-09-15T23:15:42.910174+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:15:42.910174+0000","last_undegraded":"2024-09-15T23:15:42.910174+0000","last_fullsized":"2024-09-15T23:15:42.910174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612880+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:15:42.612880+0000","last_peered":"2024-09-15T23:15:42.612880+0000","last_clean":"2024-09-15T23:15:42.612880+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:15:42.612880+0000","last_undegraded":"2024-09-15T23:15:42.612880+0000","last_fullsized":"2024-09-15T23:15:42.612880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096207+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:15:42.096207+0000","last_peered":"2024-09-15T23:15:42.096207+0000","last_clean":"2024-09-15T23:15:42.096207+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:15:42.096207+0000","last_undegraded":"2024-09-15T23:15:42.096207+0000","last_fullsized":"2024-09-15T23:15:42.096207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096366+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:15:42.096366+0000","last_peered":"2024-09-15T23:15:42.096366+0000","last_clean":"2024-09-15T23:15:42.096366+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:15:42.096366+0000","last_undegraded":"2024-09-15T23:15:42.096366+0000","last_fullsized":"2024-09-15T23:15:42.096366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612501+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:15:42.612501+0000","last_peered":"2024-09-15T23:15:42.612501+0000","last_clean":"2024-09-15T23:15:42.612501+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:15:42.612501+0000","last_undegraded":"2024-09-15T23:15:42.612501+0000","last_fullsized":"2024-09-15T23:15:42.612501+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612646+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:15:42.612646+0000","last_peered":"2024-09-15T23:15:42.612646+0000","last_clean":"2024-09-15T23:15:42.612646+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:15:42.612646+0000","last_undegraded":"2024-09-15T23:15:42.612646+0000","last_fullsized":"2024-09-15T23:15:42.612646+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317377+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:15:43.317377+0000","last_peered":"2024-09-15T23:15:43.317377+0000","last_clean":"2024-09-15T23:15:43.317377+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:15:43.317377+0000","last_undegraded":"2024-09-15T23:15:43.317377+0000","last_fullsized":"2024-09-15T23:15:43.317377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610726+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:15:42.610726+0000","last_peered":"2024-09-15T23:15:42.610726+0000","last_clean":"2024-09-15T23:15:42.610726+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:15:42.610726+0000","last_undegraded":"2024-09-15T23:15:42.610726+0000","last_fullsized":"2024-09-15T23:15:42.610726+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613153+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:15:42.613153+0000","last_peered":"2024-09-15T23:15:42.613153+0000","last_clean":"2024-09-15T23:15:42.613153+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:15:42.613153+0000","last_undegraded":"2024-09-15T23:15:42.613153+0000","last_fullsized":"2024-09-15T23:15:42.613153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221488+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:15:42.221488+0000","last_peered":"2024-09-15T23:15:42.221488+0000","last_clean":"2024-09-15T23:15:42.221488+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:15:42.221488+0000","last_undegraded":"2024-09-15T23:15:42.221488+0000","last_fullsized":"2024-09-15T23:15:42.221488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316851+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:15:43.316851+0000","last_peered":"2024-09-15T23:15:43.316851+0000","last_clean":"2024-09-15T23:15:43.316851+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:15:43.316851+0000","last_undegraded":"2024-09-15T23:15:43.316851+0000","last_fullsized":"2024-09-15T23:15:43.316851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610204+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:15:42.610204+0000","last_peered":"2024-09-15T23:15:42.610204+0000","last_clean":"2024-09-15T23:15:42.610204+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:15:42.610204+0000","last_undegraded":"2024-09-15T23:15:42.610204+0000","last_fullsized":"2024-09-15T23:15:42.610204+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609588+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:15:42.609588+0000","last_peered":"2024-09-15T23:15:42.609588+0000","last_clean":"2024-09-15T23:15:42.609588+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:15:42.609588+0000","last_undegraded":"2024-09-15T23:15:42.609588+0000","last_fullsized":"2024-09-15T23:15:42.609588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612535+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:15:42.612535+0000","last_peered":"2024-09-15T23:15:42.612535+0000","last_clean":"2024-09-15T23:15:42.612535+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:15:42.612535+0000","last_undegraded":"2024-09-15T23:15:42.612535+0000","last_fullsized":"2024-09-15T23:15:42.612535+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096427+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:15:42.096427+0000","last_peered":"2024-09-15T23:15:42.096427+0000","last_clean":"2024-09-15T23:15:42.096427+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:15:42.096427+0000","last_undegraded":"2024-09-15T23:15:42.096427+0000","last_fullsized":"2024-09-15T23:15:42.096427+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096366+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:15:42.096366+0000","last_peered":"2024-09-15T23:15:42.096366+0000","last_clean":"2024-09-15T23:15:42.096366+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:15:42.096366+0000","last_undegraded":"2024-09-15T23:15:42.096366+0000","last_fullsized":"2024-09-15T23:15:42.096366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910172+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:15:42.910172+0000","last_peered":"2024-09-15T23:15:42.910172+0000","last_clean":"2024-09-15T23:15:42.910172+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:15:42.910172+0000","last_undegraded":"2024-09-15T23:15:42.910172+0000","last_fullsized":"2024-09-15T23:15:42.910172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67632,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":616,"num_read_kb":380,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":202896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514536,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522238,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743469,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1159168,"data_stored":1172882,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906995,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1032192,"data_stored":526458,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135928,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522170,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364860,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1077248,"data_stored":1118334,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528386,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":517538,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790023,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1085440,"data_stored":1113922,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1019904,"data_stored":522188,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1007616,"data_stored":508387,"data_compressed":26534,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":582820,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":580936,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1019904,"data_stored":509581,"data_compressed":26534,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":65372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:15:48.454 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-09-15T23:15:48.454 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.594+0000 7f7d22b42640 1 -- 172.21.15.62:0/1593356497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c102630 msgr2=0x7f7d1c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.594+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1593356497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c102630 0x7f7d1c104a20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009900 tx=0x7f7d1802a3f0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.594+0000 7f7d22b42640 1 -- 172.21.15.62:0/1593356497 shutdown_connections 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.594+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1593356497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d1c107890 0x7f7d1c109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.594+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1593356497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d1c104f60 0x7f7d1c107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.594+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1593356497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c102630 0x7f7d1c104a20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.594+0000 7f7d22b42640 1 -- 172.21.15.62:0/1593356497 >> 172.21.15.62:0/1593356497 conn(0x7f7d1c0fc1b0 msgr2=0x7f7d1c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.594+0000 7f7d22b42640 1 -- 172.21.15.62:0/1593356497 shutdown_connections 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.594+0000 7f7d22b42640 1 -- 172.21.15.62:0/1593356497 wait complete. 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d22b42640 1 Processor -- start 2024-09-15T23:15:48.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d22b42640 1 -- start start 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d22b42640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d1c102630 0x7f7d1c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d22b42640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d1c104f60 0x7f7d1c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d22b42640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c107890 0x7f7d1c19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d22b42640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c10d7f0 con 0x7f7d1c107890 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d22b42640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c10d670 con 0x7f7d1c104f60 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d22b42640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c10d970 con 0x7f7d1c102630 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d210df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c107890 0x7f7d1c19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d210df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c107890 0x7f7d1c19f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44808/0 (socket says 172.21.15.62:44808) 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d210df640 1 -- 172.21.15.62:0/1522050488 learned_addr learned my addr 172.21.15.62:0/1522050488 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d208de640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d1c102630 0x7f7d1c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.595+0000 7f7d13fff640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d1c104f60 0x7f7d1c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d210df640 1 -- 172.21.15.62:0/1522050488 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d1c102630 msgr2=0x7f7d1c197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d210df640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d1c102630 0x7f7d1c197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d210df640 1 -- 172.21.15.62:0/1522050488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d1c104f60 msgr2=0x7f7d1c198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d210df640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d1c104f60 0x7f7d1c198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d210df640 1 -- 172.21.15.62:0/1522050488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1c19fa70 con 0x7f7d1c107890 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d13fff640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d1c104f60 0x7f7d1c198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d210df640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c107890 0x7f7d1c19f400 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400e9d0 tx=0x7f7d1400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d11ffb640 1 -- 172.21.15.62:0/1522050488 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1400cd80 con 0x7f7d1c107890 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d11ffb640 1 -- 172.21.15.62:0/1522050488 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d14005020 con 0x7f7d1c107890 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1c19fd60 con 0x7f7d1c107890 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d11ffb640 1 -- 172.21.15.62:0/1522050488 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d140116d0 con 0x7f7d1c107890 2024-09-15T23:15:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.596+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d1c1a0200 con 0x7f7d1c107890 2024-09-15T23:15:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.597+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d1c0fdca0 con 0x7f7d1c107890 2024-09-15T23:15:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.598+0000 7f7d11ffb640 1 -- 172.21.15.62:0/1522050488 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7d140051d0 con 0x7f7d1c107890 2024-09-15T23:15:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.599+0000 7f7d11ffb640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7d040a3250 0x7f7d040a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.599+0000 7f7d208de640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7d040a3250 0x7f7d040a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.599+0000 7f7d11ffb640 1 -- 172.21.15.62:0/1522050488 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7d140d0280 con 0x7f7d1c107890 2024-09-15T23:15:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.600+0000 7f7d208de640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7d040a3250 0x7f7d040a5710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7d180314e0 tx=0x7f7d18000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:48.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.604+0000 7f7d11ffb640 1 -- 172.21.15.62:0/1522050488 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7d14016030 con 0x7f7d1c107890 2024-09-15T23:15:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:48 smithi062 ceph-mon[30949]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 895 B/s wr, 8 op/s 2024-09-15T23:15:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:48 smithi062 ceph-mon[30949]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:48 smithi062 ceph-mon[30949]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:48 smithi062 ceph-mon[30949]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-15T23:15:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:48 smithi062 ceph-mon[30949]: Cluster is now healthy 2024-09-15T23:15:48.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.778+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d1c062630 con 0x7f7d040a3250 2024-09-15T23:15:48.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.786+0000 7f7d11ffb640 1 -- 172.21.15.62:0/1522050488 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340816 (secure 0 0 0) 0x7f7d1c062630 con 0x7f7d040a3250 2024-09-15T23:15:48.791 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:48.793 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:15:48.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7d040a3250 msgr2=0x7f7d040a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7d040a3250 0x7f7d040a5710 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7d180314e0 tx=0x7f7d18000bc0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c107890 msgr2=0x7f7d1c19f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c107890 0x7f7d1c19f400 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7d1400e9d0 tx=0x7f7d1400ee90 comp rx=0 tx=0).stop 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 shutdown_connections 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7d040a3250 0x7f7d040a5710 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d1c107890 0x7f7d1c19f400 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d1c104f60 0x7f7d1c198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 --2- 172.21.15.62:0/1522050488 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7d1c102630 0x7f7d1c197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 >> 172.21.15.62:0/1522050488 conn(0x7f7d1c0fc1b0 msgr2=0x7f7d1c102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 shutdown_connections 2024-09-15T23:15:48.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:48.790+0000 7f7d22b42640 1 -- 172.21.15.62:0/1522050488 wait complete. 2024-09-15T23:15:48.802 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-15T23:15:47.725593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658234,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":742,"num_read_kb":489,"num_write":361,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":790,"ondisk_log_size":790,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369972,"kb_used_data":12332,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571852,"statfs":{"total":1151940427776,"available":1151561576448,"internally_reserved":0,"allocated":12627968,"data_stored":8197454,"data_compressed":455690,"data_compressed_allocated":618496,"data_compressed_original":2736128,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":698,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":31,"num_write":10,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002136"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609573+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:15:42.609573+0000","last_peered":"2024-09-15T23:15:42.609573+0000","last_clean":"2024-09-15T23:15:42.609573+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:15:42.609573+0000","last_undegraded":"2024-09-15T23:15:42.609573+0000","last_fullsized":"2024-09-15T23:15:42.609573+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221889+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:15:42.221889+0000","last_peered":"2024-09-15T23:15:42.221889+0000","last_clean":"2024-09-15T23:15:42.221889+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:15:42.221889+0000","last_undegraded":"2024-09-15T23:15:42.221889+0000","last_fullsized":"2024-09-15T23:15:42.221889+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613665+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:15:42.613665+0000","last_peered":"2024-09-15T23:15:42.613665+0000","last_clean":"2024-09-15T23:15:42.613665+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:15:42.613665+0000","last_undegraded":"2024-09-15T23:15:42.613665+0000","last_fullsized":"2024-09-15T23:15:42.613665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.222006+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:15:42.222006+0000","last_peered":"2024-09-15T23:15:42.222006+0000","last_clean":"2024-09-15T23:15:42.222006+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:15:42.222006+0000","last_undegraded":"2024-09-15T23:15:42.222006+0000","last_fullsized":"2024-09-15T23:15:42.222006+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096731+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:15:42.096731+0000","last_peered":"2024-09-15T23:15:42.096731+0000","last_clean":"2024-09-15T23:15:42.096731+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:15:42.096731+0000","last_undegraded":"2024-09-15T23:15:42.096731+0000","last_fullsized":"2024-09-15T23:15:42.096731+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611378+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:15:42.611378+0000","last_peered":"2024-09-15T23:15:42.611378+0000","last_clean":"2024-09-15T23:15:42.611378+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:15:42.611378+0000","last_undegraded":"2024-09-15T23:15:42.611378+0000","last_fullsized":"2024-09-15T23:15:42.611378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909808+0000","last_change":"2024-09-15T23:14:41.659816+0000","last_active":"2024-09-15T23:15:42.909808+0000","last_peered":"2024-09-15T23:15:42.909808+0000","last_clean":"2024-09-15T23:15:42.909808+0000","last_became_active":"2024-09-15T23:14:41.641611+0000","last_became_peered":"2024-09-15T23:14:41.641611+0000","last_unstale":"2024-09-15T23:15:42.909808+0000","last_undegraded":"2024-09-15T23:15:42.909808+0000","last_fullsized":"2024-09-15T23:15:42.909808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:23.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612963+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:15:42.612963+0000","last_peered":"2024-09-15T23:15:42.612963+0000","last_clean":"2024-09-15T23:15:42.612963+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:15:42.612963+0000","last_undegraded":"2024-09-15T23:15:42.612963+0000","last_fullsized":"2024-09-15T23:15:42.612963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610025+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:15:42.610025+0000","last_peered":"2024-09-15T23:15:42.610025+0000","last_clean":"2024-09-15T23:15:42.610025+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:15:42.610025+0000","last_undegraded":"2024-09-15T23:15:42.610025+0000","last_fullsized":"2024-09-15T23:15:42.610025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614867+0000","last_change":"2024-09-15T23:15:42.614867+0000","last_active":"2024-09-15T23:15:42.614867+0000","last_peered":"2024-09-15T23:15:42.614867+0000","last_clean":"2024-09-15T23:15:42.614867+0000","last_became_active":"2024-09-15T23:15:42.614459+0000","last_became_peered":"2024-09-15T23:15:42.614459+0000","last_unstale":"2024-09-15T23:15:42.614867+0000","last_undegraded":"2024-09-15T23:15:42.614867+0000","last_fullsized":"2024-09-15T23:15:42.614867+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:15.254442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611583+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:15:42.611583+0000","last_peered":"2024-09-15T23:15:42.611583+0000","last_clean":"2024-09-15T23:15:42.611583+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:15:42.611583+0000","last_undegraded":"2024-09-15T23:15:42.611583+0000","last_fullsized":"2024-09-15T23:15:42.611583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317095+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:15:43.317095+0000","last_peered":"2024-09-15T23:15:43.317095+0000","last_clean":"2024-09-15T23:15:43.317095+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:15:43.317095+0000","last_undegraded":"2024-09-15T23:15:43.317095+0000","last_fullsized":"2024-09-15T23:15:43.317095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":56,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609750+0000","last_change":"2024-09-15T23:14:41.646320+0000","last_active":"2024-09-15T23:15:42.609750+0000","last_peered":"2024-09-15T23:15:42.609750+0000","last_clean":"2024-09-15T23:15:42.609750+0000","last_became_active":"2024-09-15T23:14:41.641933+0000","last_became_peered":"2024-09-15T23:14:41.641933+0000","last_unstale":"2024-09-15T23:15:42.609750+0000","last_undegraded":"2024-09-15T23:15:42.609750+0000","last_fullsized":"2024-09-15T23:15:42.609750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:49.417407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611897+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:15:42.611897+0000","last_peered":"2024-09-15T23:15:42.611897+0000","last_clean":"2024-09-15T23:15:42.611897+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:15:42.611897+0000","last_undegraded":"2024-09-15T23:15:42.611897+0000","last_fullsized":"2024-09-15T23:15:42.611897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221618+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:15:42.221618+0000","last_peered":"2024-09-15T23:15:42.221618+0000","last_clean":"2024-09-15T23:15:42.221618+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:15:42.221618+0000","last_undegraded":"2024-09-15T23:15:42.221618+0000","last_fullsized":"2024-09-15T23:15:42.221618+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613807+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:15:42.613807+0000","last_peered":"2024-09-15T23:15:42.613807+0000","last_clean":"2024-09-15T23:15:42.613807+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:15:42.613807+0000","last_undegraded":"2024-09-15T23:15:42.613807+0000","last_fullsized":"2024-09-15T23:15:42.613807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612402+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:15:42.612402+0000","last_peered":"2024-09-15T23:15:42.612402+0000","last_clean":"2024-09-15T23:15:42.612402+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:15:42.612402+0000","last_undegraded":"2024-09-15T23:15:42.612402+0000","last_fullsized":"2024-09-15T23:15:42.612402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909877+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:15:42.909877+0000","last_peered":"2024-09-15T23:15:42.909877+0000","last_clean":"2024-09-15T23:15:42.909877+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:15:42.909877+0000","last_undegraded":"2024-09-15T23:15:42.909877+0000","last_fullsized":"2024-09-15T23:15:42.909877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221790+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:15:42.221790+0000","last_peered":"2024-09-15T23:15:42.221790+0000","last_clean":"2024-09-15T23:15:42.221790+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:15:42.221790+0000","last_undegraded":"2024-09-15T23:15:42.221790+0000","last_fullsized":"2024-09-15T23:15:42.221790+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609295+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:15:42.609295+0000","last_peered":"2024-09-15T23:15:42.609295+0000","last_clean":"2024-09-15T23:15:42.609295+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:15:42.609295+0000","last_undegraded":"2024-09-15T23:15:42.609295+0000","last_fullsized":"2024-09-15T23:15:42.609295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609453+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:15:42.609453+0000","last_peered":"2024-09-15T23:15:42.609453+0000","last_clean":"2024-09-15T23:15:42.609453+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:15:42.609453+0000","last_undegraded":"2024-09-15T23:15:42.609453+0000","last_fullsized":"2024-09-15T23:15:42.609453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613324+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:15:42.613324+0000","last_peered":"2024-09-15T23:15:42.613324+0000","last_clean":"2024-09-15T23:15:42.613324+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:15:42.613324+0000","last_undegraded":"2024-09-15T23:15:42.613324+0000","last_fullsized":"2024-09-15T23:15:42.613324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612027+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:15:42.612027+0000","last_peered":"2024-09-15T23:15:42.612027+0000","last_clean":"2024-09-15T23:15:42.612027+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:15:42.612027+0000","last_undegraded":"2024-09-15T23:15:42.612027+0000","last_fullsized":"2024-09-15T23:15:42.612027+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096594+0000","last_change":"2024-09-15T23:14:42.009605+0000","last_active":"2024-09-15T23:15:42.096594+0000","last_peered":"2024-09-15T23:15:42.096594+0000","last_clean":"2024-09-15T23:15:42.096594+0000","last_became_active":"2024-09-15T23:14:42.009237+0000","last_became_peered":"2024-09-15T23:14:42.009237+0000","last_unstale":"2024-09-15T23:15:42.096594+0000","last_undegraded":"2024-09-15T23:15:42.096594+0000","last_fullsized":"2024-09-15T23:15:42.096594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:11:08.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612739+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:15:42.612739+0000","last_peered":"2024-09-15T23:15:42.612739+0000","last_clean":"2024-09-15T23:15:42.612739+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:15:42.612739+0000","last_undegraded":"2024-09-15T23:15:42.612739+0000","last_fullsized":"2024-09-15T23:15:42.612739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614124+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:15:42.614124+0000","last_peered":"2024-09-15T23:15:42.614124+0000","last_clean":"2024-09-15T23:15:42.614124+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:15:42.614124+0000","last_undegraded":"2024-09-15T23:15:42.614124+0000","last_fullsized":"2024-09-15T23:15:42.614124+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.222041+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:15:42.222041+0000","last_peered":"2024-09-15T23:15:42.222041+0000","last_clean":"2024-09-15T23:15:42.222041+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:15:42.222041+0000","last_undegraded":"2024-09-15T23:15:42.222041+0000","last_fullsized":"2024-09-15T23:15:42.222041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'14","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610065+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:15:42.610065+0000","last_peered":"2024-09-15T23:15:42.610065+0000","last_clean":"2024-09-15T23:15:42.610065+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:15:42.610065+0000","last_undegraded":"2024-09-15T23:15:42.610065+0000","last_fullsized":"2024-09-15T23:15:42.610065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221420+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:15:42.221420+0000","last_peered":"2024-09-15T23:15:42.221420+0000","last_clean":"2024-09-15T23:15:42.221420+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:15:42.221420+0000","last_undegraded":"2024-09-15T23:15:42.221420+0000","last_fullsized":"2024-09-15T23:15:42.221420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612814+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:15:42.612814+0000","last_peered":"2024-09-15T23:15:42.612814+0000","last_clean":"2024-09-15T23:15:42.612814+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:15:42.612814+0000","last_undegraded":"2024-09-15T23:15:42.612814+0000","last_fullsized":"2024-09-15T23:15:42.612814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610084+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:15:42.610084+0000","last_peered":"2024-09-15T23:15:42.610084+0000","last_clean":"2024-09-15T23:15:42.610084+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:15:42.610084+0000","last_undegraded":"2024-09-15T23:15:42.610084+0000","last_fullsized":"2024-09-15T23:15:42.610084+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909751+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:15:42.909751+0000","last_peered":"2024-09-15T23:15:42.909751+0000","last_clean":"2024-09-15T23:15:42.909751+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:15:42.909751+0000","last_undegraded":"2024-09-15T23:15:42.909751+0000","last_fullsized":"2024-09-15T23:15:42.909751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610218+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:15:42.610218+0000","last_peered":"2024-09-15T23:15:42.610218+0000","last_clean":"2024-09-15T23:15:42.610218+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:15:42.610218+0000","last_undegraded":"2024-09-15T23:15:42.610218+0000","last_fullsized":"2024-09-15T23:15:42.610218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609669+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:15:42.609669+0000","last_peered":"2024-09-15T23:15:42.609669+0000","last_clean":"2024-09-15T23:15:42.609669+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:15:42.609669+0000","last_undegraded":"2024-09-15T23:15:42.609669+0000","last_fullsized":"2024-09-15T23:15:42.609669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609473+0000","last_change":"2024-09-15T23:14:41.995466+0000","last_active":"2024-09-15T23:15:42.609473+0000","last_peered":"2024-09-15T23:15:42.609473+0000","last_clean":"2024-09-15T23:15:42.609473+0000","last_became_active":"2024-09-15T23:14:41.994915+0000","last_became_peered":"2024-09-15T23:14:41.994915+0000","last_unstale":"2024-09-15T23:15:42.609473+0000","last_undegraded":"2024-09-15T23:15:42.609473+0000","last_fullsized":"2024-09-15T23:15:42.609473+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:04.121596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610393+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:15:42.610393+0000","last_peered":"2024-09-15T23:15:42.610393+0000","last_clean":"2024-09-15T23:15:42.610393+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:15:42.610393+0000","last_undegraded":"2024-09-15T23:15:42.610393+0000","last_fullsized":"2024-09-15T23:15:42.610393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317131+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:15:43.317131+0000","last_peered":"2024-09-15T23:15:43.317131+0000","last_clean":"2024-09-15T23:15:43.317131+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:15:43.317131+0000","last_undegraded":"2024-09-15T23:15:43.317131+0000","last_fullsized":"2024-09-15T23:15:43.317131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626237+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:15:42.626237+0000","last_peered":"2024-09-15T23:15:42.626237+0000","last_clean":"2024-09-15T23:15:42.626237+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:15:42.626237+0000","last_undegraded":"2024-09-15T23:15:42.626237+0000","last_fullsized":"2024-09-15T23:15:42.626237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.618846+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:15:43.618846+0000","last_peered":"2024-09-15T23:15:43.618846+0000","last_clean":"2024-09-15T23:15:43.618846+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:15:43.618846+0000","last_undegraded":"2024-09-15T23:15:43.618846+0000","last_fullsized":"2024-09-15T23:15:43.618846+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":97,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612986+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:15:42.612986+0000","last_peered":"2024-09-15T23:15:42.612986+0000","last_clean":"2024-09-15T23:15:42.612986+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:15:42.612986+0000","last_undegraded":"2024-09-15T23:15:42.612986+0000","last_fullsized":"2024-09-15T23:15:42.612986+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.617748+0000","last_change":"2024-09-15T23:15:42.617748+0000","last_active":"2024-09-15T23:15:42.617748+0000","last_peered":"2024-09-15T23:15:42.617748+0000","last_clean":"2024-09-15T23:15:42.617748+0000","last_became_active":"2024-09-15T23:15:42.617320+0000","last_became_peered":"2024-09-15T23:15:42.617320+0000","last_unstale":"2024-09-15T23:15:42.617748+0000","last_undegraded":"2024-09-15T23:15:42.617748+0000","last_fullsized":"2024-09-15T23:15:42.617748+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:47.900049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316911+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:15:43.316911+0000","last_peered":"2024-09-15T23:15:43.316911+0000","last_clean":"2024-09-15T23:15:43.316911+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:15:43.316911+0000","last_undegraded":"2024-09-15T23:15:43.316911+0000","last_fullsized":"2024-09-15T23:15:43.316911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626536+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:15:42.626536+0000","last_peered":"2024-09-15T23:15:42.626536+0000","last_clean":"2024-09-15T23:15:42.626536+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:15:42.626536+0000","last_undegraded":"2024-09-15T23:15:42.626536+0000","last_fullsized":"2024-09-15T23:15:42.626536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221750+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:42.221750+0000","last_peered":"2024-09-15T23:15:42.221750+0000","last_clean":"2024-09-15T23:15:42.221750+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:15:42.221750+0000","last_undegraded":"2024-09-15T23:15:42.221750+0000","last_fullsized":"2024-09-15T23:15:42.221750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612474+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:15:42.612474+0000","last_peered":"2024-09-15T23:15:42.612474+0000","last_clean":"2024-09-15T23:15:42.612474+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:15:42.612474+0000","last_undegraded":"2024-09-15T23:15:42.612474+0000","last_fullsized":"2024-09-15T23:15:42.612474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317208+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:15:43.317208+0000","last_peered":"2024-09-15T23:15:43.317208+0000","last_clean":"2024-09-15T23:15:43.317208+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:15:43.317208+0000","last_undegraded":"2024-09-15T23:15:43.317208+0000","last_fullsized":"2024-09-15T23:15:43.317208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317366+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:15:43.317366+0000","last_peered":"2024-09-15T23:15:43.317366+0000","last_clean":"2024-09-15T23:15:43.317366+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:15:43.317366+0000","last_undegraded":"2024-09-15T23:15:43.317366+0000","last_fullsized":"2024-09-15T23:15:43.317366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317438+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:15:43.317438+0000","last_peered":"2024-09-15T23:15:43.317438+0000","last_clean":"2024-09-15T23:15:43.317438+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:15:43.317438+0000","last_undegraded":"2024-09-15T23:15:43.317438+0000","last_fullsized":"2024-09-15T23:15:43.317438+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614172+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:15:42.614172+0000","last_peered":"2024-09-15T23:15:42.614172+0000","last_clean":"2024-09-15T23:15:42.614172+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:15:42.614172+0000","last_undegraded":"2024-09-15T23:15:42.614172+0000","last_fullsized":"2024-09-15T23:15:42.614172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612428+0000","last_change":"2024-09-15T23:14:41.994950+0000","last_active":"2024-09-15T23:15:42.612428+0000","last_peered":"2024-09-15T23:15:42.612428+0000","last_clean":"2024-09-15T23:15:42.612428+0000","last_became_active":"2024-09-15T23:14:41.994429+0000","last_became_peered":"2024-09-15T23:14:41.994429+0000","last_unstale":"2024-09-15T23:15:42.612428+0000","last_undegraded":"2024-09-15T23:15:42.612428+0000","last_fullsized":"2024-09-15T23:15:42.612428+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:24.056756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317319+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:15:43.317319+0000","last_peered":"2024-09-15T23:15:43.317319+0000","last_clean":"2024-09-15T23:15:43.317319+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:15:43.317319+0000","last_undegraded":"2024-09-15T23:15:43.317319+0000","last_fullsized":"2024-09-15T23:15:43.317319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.627013+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:15:42.627013+0000","last_peered":"2024-09-15T23:15:42.627013+0000","last_clean":"2024-09-15T23:15:42.627013+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:15:42.627013+0000","last_undegraded":"2024-09-15T23:15:42.627013+0000","last_fullsized":"2024-09-15T23:15:42.627013+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096349+0000","last_change":"2024-09-15T23:14:41.640471+0000","last_active":"2024-09-15T23:15:42.096349+0000","last_peered":"2024-09-15T23:15:42.096349+0000","last_clean":"2024-09-15T23:15:42.096349+0000","last_became_active":"2024-09-15T23:14:41.640017+0000","last_became_peered":"2024-09-15T23:14:41.640017+0000","last_unstale":"2024-09-15T23:15:42.096349+0000","last_undegraded":"2024-09-15T23:15:42.096349+0000","last_fullsized":"2024-09-15T23:15:42.096349+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:58.842636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317343+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:15:43.317343+0000","last_peered":"2024-09-15T23:15:43.317343+0000","last_clean":"2024-09-15T23:15:43.317343+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:15:43.317343+0000","last_undegraded":"2024-09-15T23:15:43.317343+0000","last_fullsized":"2024-09-15T23:15:43.317343+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613353+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:15:42.613353+0000","last_peered":"2024-09-15T23:15:42.613353+0000","last_clean":"2024-09-15T23:15:42.613353+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:15:42.613353+0000","last_undegraded":"2024-09-15T23:15:42.613353+0000","last_fullsized":"2024-09-15T23:15:42.613353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.625937+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:15:42.625937+0000","last_peered":"2024-09-15T23:15:42.625937+0000","last_clean":"2024-09-15T23:15:42.625937+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:15:42.625937+0000","last_undegraded":"2024-09-15T23:15:42.625937+0000","last_fullsized":"2024-09-15T23:15:42.625937+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909962+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:15:42.909962+0000","last_peered":"2024-09-15T23:15:42.909962+0000","last_clean":"2024-09-15T23:15:42.909962+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:15:42.909962+0000","last_undegraded":"2024-09-15T23:15:42.909962+0000","last_fullsized":"2024-09-15T23:15:42.909962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610024+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:15:42.610024+0000","last_peered":"2024-09-15T23:15:42.610024+0000","last_clean":"2024-09-15T23:15:42.610024+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:15:42.610024+0000","last_undegraded":"2024-09-15T23:15:42.610024+0000","last_fullsized":"2024-09-15T23:15:42.610024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.625935+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:15:42.625935+0000","last_peered":"2024-09-15T23:15:42.625935+0000","last_clean":"2024-09-15T23:15:42.625935+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:15:42.625935+0000","last_undegraded":"2024-09-15T23:15:42.625935+0000","last_fullsized":"2024-09-15T23:15:42.625935+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626111+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:15:42.626111+0000","last_peered":"2024-09-15T23:15:42.626111+0000","last_clean":"2024-09-15T23:15:42.626111+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:15:42.626111+0000","last_undegraded":"2024-09-15T23:15:42.626111+0000","last_fullsized":"2024-09-15T23:15:42.626111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909569+0000","last_change":"2024-09-15T23:14:41.653786+0000","last_active":"2024-09-15T23:15:42.909569+0000","last_peered":"2024-09-15T23:15:42.909569+0000","last_clean":"2024-09-15T23:15:42.909569+0000","last_became_active":"2024-09-15T23:14:41.641472+0000","last_became_peered":"2024-09-15T23:14:41.641472+0000","last_unstale":"2024-09-15T23:15:42.909569+0000","last_undegraded":"2024-09-15T23:15:42.909569+0000","last_fullsized":"2024-09-15T23:15:42.909569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:16.010142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613267+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:15:42.613267+0000","last_peered":"2024-09-15T23:15:42.613267+0000","last_clean":"2024-09-15T23:15:42.613267+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:15:42.613267+0000","last_undegraded":"2024-09-15T23:15:42.613267+0000","last_fullsized":"2024-09-15T23:15:42.613267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316776+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:15:43.316776+0000","last_peered":"2024-09-15T23:15:43.316776+0000","last_clean":"2024-09-15T23:15:43.316776+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:15:43.316776+0000","last_undegraded":"2024-09-15T23:15:43.316776+0000","last_fullsized":"2024-09-15T23:15:43.316776+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.619522+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:15:43.619522+0000","last_peered":"2024-09-15T23:15:43.619522+0000","last_clean":"2024-09-15T23:15:43.619522+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:15:43.619522+0000","last_undegraded":"2024-09-15T23:15:43.619522+0000","last_fullsized":"2024-09-15T23:15:43.619522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609376+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:15:42.609376+0000","last_peered":"2024-09-15T23:15:42.609376+0000","last_clean":"2024-09-15T23:15:42.609376+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:15:42.609376+0000","last_undegraded":"2024-09-15T23:15:42.609376+0000","last_fullsized":"2024-09-15T23:15:42.609376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":48,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317071+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:15:43.317071+0000","last_peered":"2024-09-15T23:15:43.317071+0000","last_clean":"2024-09-15T23:15:43.317071+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:15:43.317071+0000","last_undegraded":"2024-09-15T23:15:43.317071+0000","last_fullsized":"2024-09-15T23:15:43.317071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626254+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:15:42.626254+0000","last_peered":"2024-09-15T23:15:42.626254+0000","last_clean":"2024-09-15T23:15:42.626254+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:15:42.626254+0000","last_undegraded":"2024-09-15T23:15:42.626254+0000","last_fullsized":"2024-09-15T23:15:42.626254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909759+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:15:42.909759+0000","last_peered":"2024-09-15T23:15:42.909759+0000","last_clean":"2024-09-15T23:15:42.909759+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:15:42.909759+0000","last_undegraded":"2024-09-15T23:15:42.909759+0000","last_fullsized":"2024-09-15T23:15:42.909759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612005+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:15:42.612005+0000","last_peered":"2024-09-15T23:15:42.612005+0000","last_clean":"2024-09-15T23:15:42.612005+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:15:42.612005+0000","last_undegraded":"2024-09-15T23:15:42.612005+0000","last_fullsized":"2024-09-15T23:15:42.612005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613889+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:15:42.613889+0000","last_peered":"2024-09-15T23:15:42.613889+0000","last_clean":"2024-09-15T23:15:42.613889+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:15:42.613889+0000","last_undegraded":"2024-09-15T23:15:42.613889+0000","last_fullsized":"2024-09-15T23:15:42.613889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626638+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:15:42.626638+0000","last_peered":"2024-09-15T23:15:42.626638+0000","last_clean":"2024-09-15T23:15:42.626638+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:15:42.626638+0000","last_undegraded":"2024-09-15T23:15:42.626638+0000","last_fullsized":"2024-09-15T23:15:42.626638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609857+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:15:42.609857+0000","last_peered":"2024-09-15T23:15:42.609857+0000","last_clean":"2024-09-15T23:15:42.609857+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:15:42.609857+0000","last_undegraded":"2024-09-15T23:15:42.609857+0000","last_fullsized":"2024-09-15T23:15:42.609857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":281,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:45.631571+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:15:45.631571+0000","last_peered":"2024-09-15T23:15:45.631571+0000","last_clean":"2024-09-15T23:15:45.631571+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:15:45.631571+0000","last_undegraded":"2024-09-15T23:15:45.631571+0000","last_fullsized":"2024-09-15T23:15:45.631571+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63488,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":207,"num_read_kb":166,"num_write":37,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609007+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:15:42.609007+0000","last_peered":"2024-09-15T23:15:42.609007+0000","last_clean":"2024-09-15T23:15:42.609007+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:15:42.609007+0000","last_undegraded":"2024-09-15T23:15:42.609007+0000","last_fullsized":"2024-09-15T23:15:42.609007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612977+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:15:42.612977+0000","last_peered":"2024-09-15T23:15:42.612977+0000","last_clean":"2024-09-15T23:15:42.612977+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:15:42.612977+0000","last_undegraded":"2024-09-15T23:15:42.612977+0000","last_fullsized":"2024-09-15T23:15:42.612977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626922+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:15:42.626922+0000","last_peered":"2024-09-15T23:15:42.626922+0000","last_clean":"2024-09-15T23:15:42.626922+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:15:42.626922+0000","last_undegraded":"2024-09-15T23:15:42.626922+0000","last_fullsized":"2024-09-15T23:15:42.626922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612007+0000","last_change":"2024-09-15T23:14:41.641820+0000","last_active":"2024-09-15T23:15:42.612007+0000","last_peered":"2024-09-15T23:15:42.612007+0000","last_clean":"2024-09-15T23:15:42.612007+0000","last_became_active":"2024-09-15T23:14:41.641282+0000","last_became_peered":"2024-09-15T23:14:41.641282+0000","last_unstale":"2024-09-15T23:15:42.612007+0000","last_undegraded":"2024-09-15T23:15:42.612007+0000","last_fullsized":"2024-09-15T23:15:42.612007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:43.472601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096735+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:15:42.096735+0000","last_peered":"2024-09-15T23:15:42.096735+0000","last_clean":"2024-09-15T23:15:42.096735+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:15:42.096735+0000","last_undegraded":"2024-09-15T23:15:42.096735+0000","last_fullsized":"2024-09-15T23:15:42.096735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317368+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:15:43.317368+0000","last_peered":"2024-09-15T23:15:43.317368+0000","last_clean":"2024-09-15T23:15:43.317368+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:15:43.317368+0000","last_undegraded":"2024-09-15T23:15:43.317368+0000","last_fullsized":"2024-09-15T23:15:43.317368+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608807+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:15:42.608807+0000","last_peered":"2024-09-15T23:15:42.608807+0000","last_clean":"2024-09-15T23:15:42.608807+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:15:42.608807+0000","last_undegraded":"2024-09-15T23:15:42.608807+0000","last_fullsized":"2024-09-15T23:15:42.608807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612577+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:15:42.612577+0000","last_peered":"2024-09-15T23:15:42.612577+0000","last_clean":"2024-09-15T23:15:42.612577+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:15:42.612577+0000","last_undegraded":"2024-09-15T23:15:42.612577+0000","last_fullsized":"2024-09-15T23:15:42.612577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":140,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:44.628850+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:15:44.628850+0000","last_peered":"2024-09-15T23:15:44.628850+0000","last_clean":"2024-09-15T23:15:44.628850+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:15:44.628850+0000","last_undegraded":"2024-09-15T23:15:44.628850+0000","last_fullsized":"2024-09-15T23:15:44.628850+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221487+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:15:42.221487+0000","last_peered":"2024-09-15T23:15:42.221487+0000","last_clean":"2024-09-15T23:15:42.221487+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:15:42.221487+0000","last_undegraded":"2024-09-15T23:15:42.221487+0000","last_fullsized":"2024-09-15T23:15:42.221487+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096328+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:15:42.096328+0000","last_peered":"2024-09-15T23:15:42.096328+0000","last_clean":"2024-09-15T23:15:42.096328+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:15:42.096328+0000","last_undegraded":"2024-09-15T23:15:42.096328+0000","last_fullsized":"2024-09-15T23:15:42.096328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608984+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:15:42.608984+0000","last_peered":"2024-09-15T23:15:42.608984+0000","last_clean":"2024-09-15T23:15:42.608984+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:15:42.608984+0000","last_undegraded":"2024-09-15T23:15:42.608984+0000","last_fullsized":"2024-09-15T23:15:42.608984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626347+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:15:42.626347+0000","last_peered":"2024-09-15T23:15:42.626347+0000","last_clean":"2024-09-15T23:15:42.626347+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:15:42.626347+0000","last_undegraded":"2024-09-15T23:15:42.626347+0000","last_fullsized":"2024-09-15T23:15:42.626347+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909956+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:15:42.909956+0000","last_peered":"2024-09-15T23:15:42.909956+0000","last_clean":"2024-09-15T23:15:42.909956+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:15:42.909956+0000","last_undegraded":"2024-09-15T23:15:42.909956+0000","last_fullsized":"2024-09-15T23:15:42.909956+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221496+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:15:42.221496+0000","last_peered":"2024-09-15T23:15:42.221496+0000","last_clean":"2024-09-15T23:15:42.221496+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:15:42.221496+0000","last_undegraded":"2024-09-15T23:15:42.221496+0000","last_fullsized":"2024-09-15T23:15:42.221496+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611692+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:15:42.611692+0000","last_peered":"2024-09-15T23:15:42.611692+0000","last_clean":"2024-09-15T23:15:42.611692+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:15:42.611692+0000","last_undegraded":"2024-09-15T23:15:42.611692+0000","last_fullsized":"2024-09-15T23:15:42.611692+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610033+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:15:42.610033+0000","last_peered":"2024-09-15T23:15:42.610033+0000","last_clean":"2024-09-15T23:15:42.610033+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:15:42.610033+0000","last_undegraded":"2024-09-15T23:15:42.610033+0000","last_fullsized":"2024-09-15T23:15:42.610033+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626302+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:15:42.626302+0000","last_peered":"2024-09-15T23:15:42.626302+0000","last_clean":"2024-09-15T23:15:42.626302+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:15:42.626302+0000","last_undegraded":"2024-09-15T23:15:42.626302+0000","last_fullsized":"2024-09-15T23:15:42.626302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613815+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:15:42.613815+0000","last_peered":"2024-09-15T23:15:42.613815+0000","last_clean":"2024-09-15T23:15:42.613815+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:15:42.613815+0000","last_undegraded":"2024-09-15T23:15:42.613815+0000","last_fullsized":"2024-09-15T23:15:42.613815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":147,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:44.629643+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:15:44.629643+0000","last_peered":"2024-09-15T23:15:44.629643+0000","last_clean":"2024-09-15T23:15:44.629643+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:15:44.629643+0000","last_undegraded":"2024-09-15T23:15:44.629643+0000","last_fullsized":"2024-09-15T23:15:44.629643+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1156,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609500+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:15:42.609500+0000","last_peered":"2024-09-15T23:15:42.609500+0000","last_clean":"2024-09-15T23:15:42.609500+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:15:42.609500+0000","last_undegraded":"2024-09-15T23:15:42.609500+0000","last_fullsized":"2024-09-15T23:15:42.609500+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613805+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:15:42.613805+0000","last_peered":"2024-09-15T23:15:42.613805+0000","last_clean":"2024-09-15T23:15:42.613805+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:15:42.613805+0000","last_undegraded":"2024-09-15T23:15:42.613805+0000","last_fullsized":"2024-09-15T23:15:42.613805+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909660+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:15:42.909660+0000","last_peered":"2024-09-15T23:15:42.909660+0000","last_clean":"2024-09-15T23:15:42.909660+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:15:42.909660+0000","last_undegraded":"2024-09-15T23:15:42.909660+0000","last_fullsized":"2024-09-15T23:15:42.909660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614459+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:42.614459+0000","last_peered":"2024-09-15T23:15:42.614459+0000","last_clean":"2024-09-15T23:15:42.614459+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:15:42.614459+0000","last_undegraded":"2024-09-15T23:15:42.614459+0000","last_fullsized":"2024-09-15T23:15:42.614459+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613647+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:15:42.613647+0000","last_peered":"2024-09-15T23:15:42.613647+0000","last_clean":"2024-09-15T23:15:42.613647+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:15:42.613647+0000","last_undegraded":"2024-09-15T23:15:42.613647+0000","last_fullsized":"2024-09-15T23:15:42.613647+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096457+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:15:42.096457+0000","last_peered":"2024-09-15T23:15:42.096457+0000","last_clean":"2024-09-15T23:15:42.096457+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:15:42.096457+0000","last_undegraded":"2024-09-15T23:15:42.096457+0000","last_fullsized":"2024-09-15T23:15:42.096457+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612897+0000","last_change":"2024-09-15T23:14:42.010010+0000","last_active":"2024-09-15T23:15:42.612897+0000","last_peered":"2024-09-15T23:15:42.612897+0000","last_clean":"2024-09-15T23:15:42.612897+0000","last_became_active":"2024-09-15T23:14:42.009603+0000","last_became_peered":"2024-09-15T23:14:42.009603+0000","last_unstale":"2024-09-15T23:15:42.612897+0000","last_undegraded":"2024-09-15T23:15:42.612897+0000","last_fullsized":"2024-09-15T23:15:42.612897+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:34:51.849548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910096+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:15:42.910096+0000","last_peered":"2024-09-15T23:15:42.910096+0000","last_clean":"2024-09-15T23:15:42.910096+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:15:42.910096+0000","last_undegraded":"2024-09-15T23:15:42.910096+0000","last_fullsized":"2024-09-15T23:15:42.910096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614200+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:15:42.614200+0000","last_peered":"2024-09-15T23:15:42.614200+0000","last_clean":"2024-09-15T23:15:42.614200+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:15:42.614200+0000","last_undegraded":"2024-09-15T23:15:42.614200+0000","last_fullsized":"2024-09-15T23:15:42.614200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221977+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:15:42.221977+0000","last_peered":"2024-09-15T23:15:42.221977+0000","last_clean":"2024-09-15T23:15:42.221977+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:15:42.221977+0000","last_undegraded":"2024-09-15T23:15:42.221977+0000","last_fullsized":"2024-09-15T23:15:42.221977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626113+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:15:42.626113+0000","last_peered":"2024-09-15T23:15:42.626113+0000","last_clean":"2024-09-15T23:15:42.626113+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:15:42.626113+0000","last_undegraded":"2024-09-15T23:15:42.626113+0000","last_fullsized":"2024-09-15T23:15:42.626113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221577+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:15:42.221577+0000","last_peered":"2024-09-15T23:15:42.221577+0000","last_clean":"2024-09-15T23:15:42.221577+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:15:42.221577+0000","last_undegraded":"2024-09-15T23:15:42.221577+0000","last_fullsized":"2024-09-15T23:15:42.221577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612683+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:15:42.612683+0000","last_peered":"2024-09-15T23:15:42.612683+0000","last_clean":"2024-09-15T23:15:42.612683+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:15:42.612683+0000","last_undegraded":"2024-09-15T23:15:42.612683+0000","last_fullsized":"2024-09-15T23:15:42.612683+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612593+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:15:42.612593+0000","last_peered":"2024-09-15T23:15:42.612593+0000","last_clean":"2024-09-15T23:15:42.612593+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:15:42.612593+0000","last_undegraded":"2024-09-15T23:15:42.612593+0000","last_fullsized":"2024-09-15T23:15:42.612593+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909865+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:15:42.909865+0000","last_peered":"2024-09-15T23:15:42.909865+0000","last_clean":"2024-09-15T23:15:42.909865+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:15:42.909865+0000","last_undegraded":"2024-09-15T23:15:42.909865+0000","last_fullsized":"2024-09-15T23:15:42.909865+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614031+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:15:42.614031+0000","last_peered":"2024-09-15T23:15:42.614031+0000","last_clean":"2024-09-15T23:15:42.614031+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:15:42.614031+0000","last_undegraded":"2024-09-15T23:15:42.614031+0000","last_fullsized":"2024-09-15T23:15:42.614031+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610009+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:15:42.610009+0000","last_peered":"2024-09-15T23:15:42.610009+0000","last_clean":"2024-09-15T23:15:42.610009+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:15:42.610009+0000","last_undegraded":"2024-09-15T23:15:42.610009+0000","last_fullsized":"2024-09-15T23:15:42.610009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910174+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:15:42.910174+0000","last_peered":"2024-09-15T23:15:42.910174+0000","last_clean":"2024-09-15T23:15:42.910174+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:15:42.910174+0000","last_undegraded":"2024-09-15T23:15:42.910174+0000","last_fullsized":"2024-09-15T23:15:42.910174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612880+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:15:42.612880+0000","last_peered":"2024-09-15T23:15:42.612880+0000","last_clean":"2024-09-15T23:15:42.612880+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:15:42.612880+0000","last_undegraded":"2024-09-15T23:15:42.612880+0000","last_fullsized":"2024-09-15T23:15:42.612880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096207+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:15:42.096207+0000","last_peered":"2024-09-15T23:15:42.096207+0000","last_clean":"2024-09-15T23:15:42.096207+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:15:42.096207+0000","last_undegraded":"2024-09-15T23:15:42.096207+0000","last_fullsized":"2024-09-15T23:15:42.096207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096366+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:15:42.096366+0000","last_peered":"2024-09-15T23:15:42.096366+0000","last_clean":"2024-09-15T23:15:42.096366+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:15:42.096366+0000","last_undegraded":"2024-09-15T23:15:42.096366+0000","last_fullsized":"2024-09-15T23:15:42.096366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612501+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:15:42.612501+0000","last_peered":"2024-09-15T23:15:42.612501+0000","last_clean":"2024-09-15T23:15:42.612501+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:15:42.612501+0000","last_undegraded":"2024-09-15T23:15:42.612501+0000","last_fullsized":"2024-09-15T23:15:42.612501+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612646+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:15:42.612646+0000","last_peered":"2024-09-15T23:15:42.612646+0000","last_clean":"2024-09-15T23:15:42.612646+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:15:42.612646+0000","last_undegraded":"2024-09-15T23:15:42.612646+0000","last_fullsized":"2024-09-15T23:15:42.612646+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317377+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:15:43.317377+0000","last_peered":"2024-09-15T23:15:43.317377+0000","last_clean":"2024-09-15T23:15:43.317377+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:15:43.317377+0000","last_undegraded":"2024-09-15T23:15:43.317377+0000","last_fullsized":"2024-09-15T23:15:43.317377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610726+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:15:42.610726+0000","last_peered":"2024-09-15T23:15:42.610726+0000","last_clean":"2024-09-15T23:15:42.610726+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:15:42.610726+0000","last_undegraded":"2024-09-15T23:15:42.610726+0000","last_fullsized":"2024-09-15T23:15:42.610726+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613153+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:15:42.613153+0000","last_peered":"2024-09-15T23:15:42.613153+0000","last_clean":"2024-09-15T23:15:42.613153+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:15:42.613153+0000","last_undegraded":"2024-09-15T23:15:42.613153+0000","last_fullsized":"2024-09-15T23:15:42.613153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.221488+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:15:42.221488+0000","last_peered":"2024-09-15T23:15:42.221488+0000","last_clean":"2024-09-15T23:15:42.221488+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:15:42.221488+0000","last_undegraded":"2024-09-15T23:15:42.221488+0000","last_fullsized":"2024-09-15T23:15:42.221488+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316851+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:15:43.316851+0000","last_peered":"2024-09-15T23:15:43.316851+0000","last_clean":"2024-09-15T23:15:43.316851+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:15:43.316851+0000","last_undegraded":"2024-09-15T23:15:43.316851+0000","last_fullsized":"2024-09-15T23:15:43.316851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610204+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:15:42.610204+0000","last_peered":"2024-09-15T23:15:42.610204+0000","last_clean":"2024-09-15T23:15:42.610204+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:15:42.610204+0000","last_undegraded":"2024-09-15T23:15:42.610204+0000","last_fullsized":"2024-09-15T23:15:42.610204+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609588+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:15:42.609588+0000","last_peered":"2024-09-15T23:15:42.609588+0000","last_clean":"2024-09-15T23:15:42.609588+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:15:42.609588+0000","last_undegraded":"2024-09-15T23:15:42.609588+0000","last_fullsized":"2024-09-15T23:15:42.609588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612535+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:15:42.612535+0000","last_peered":"2024-09-15T23:15:42.612535+0000","last_clean":"2024-09-15T23:15:42.612535+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:15:42.612535+0000","last_undegraded":"2024-09-15T23:15:42.612535+0000","last_fullsized":"2024-09-15T23:15:42.612535+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096427+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:15:42.096427+0000","last_peered":"2024-09-15T23:15:42.096427+0000","last_clean":"2024-09-15T23:15:42.096427+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:15:42.096427+0000","last_undegraded":"2024-09-15T23:15:42.096427+0000","last_fullsized":"2024-09-15T23:15:42.096427+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.096366+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:15:42.096366+0000","last_peered":"2024-09-15T23:15:42.096366+0000","last_clean":"2024-09-15T23:15:42.096366+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:15:42.096366+0000","last_undegraded":"2024-09-15T23:15:42.096366+0000","last_fullsized":"2024-09-15T23:15:42.096366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910172+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:15:42.910172+0000","last_peered":"2024-09-15T23:15:42.910172+0000","last_clean":"2024-09-15T23:15:42.910172+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:15:42.910172+0000","last_undegraded":"2024-09-15T23:15:42.910172+0000","last_fullsized":"2024-09-15T23:15:42.910172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67632,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":616,"num_read_kb":380,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":202896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151,"ondisk_log_size":151,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514536,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522238,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743469,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1159168,"data_stored":1172882,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906995,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1032192,"data_stored":526458,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135928,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522170,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364860,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1077248,"data_stored":1118334,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528386,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":517538,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790023,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1085440,"data_stored":1113922,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1019904,"data_stored":522188,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1007616,"data_stored":508387,"data_compressed":26534,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1101824,"data_stored":582820,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1077248,"data_stored":580936,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1019904,"data_stored":509581,"data_compressed":26534,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":65372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":65066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:15:48.807 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:15:49.026 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:48 smithi067 ceph-mon[35665]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 895 B/s wr, 8 op/s 2024-09-15T23:15:49.026 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:48 smithi067 ceph-mon[35665]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:49.026 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:48 smithi067 ceph-mon[35665]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:49.026 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:48 smithi067 ceph-mon[35665]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-15T23:15:49.026 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:48 smithi067 ceph-mon[35665]: Cluster is now healthy 2024-09-15T23:15:49.045 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:48 smithi148 ceph-mon[35610]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 895 B/s wr, 8 op/s 2024-09-15T23:15:49.045 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:48 smithi148 ceph-mon[35610]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:49.046 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:48 smithi148 ceph-mon[35610]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:49.046 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:48 smithi148 ceph-mon[35610]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-15T23:15:49.046 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:48 smithi148 ceph-mon[35610]: Cluster is now healthy 2024-09-15T23:15:49.099 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:49.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 -- 172.21.15.62:0/2630769324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 msgr2=0x7f5230110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:49.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 --2- 172.21.15.62:0/2630769324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 0x7f5230110870 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5228009ca0 tx=0x7f522802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 -- 172.21.15.62:0/2630769324 shutdown_connections 2024-09-15T23:15:49.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 --2- 172.21.15.62:0/2630769324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 0x7f5230110870 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 --2- 172.21.15.62:0/2630769324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5230101280 0x7f523010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 --2- 172.21.15.62:0/2630769324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52301008d0 0x7f5230100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 -- 172.21.15.62:0/2630769324 >> 172.21.15.62:0/2630769324 conn(0x7f52300fc7a0 msgr2=0x7f52300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 -- 172.21.15.62:0/2630769324 shutdown_connections 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 -- 172.21.15.62:0/2630769324 wait complete. 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 Processor -- start 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.568+0000 7f5237bb0640 1 -- start start 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f5237bb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52301008d0 0x7f523019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f5237bb0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5230101280 0x7f523019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f5237bb0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 0x7f523019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f5237bb0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f52301141c0 con 0x7f523010e480 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f5237bb0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5230114040 con 0x7f52301008d0 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f5237bb0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5230114340 con 0x7f5230101280 2024-09-15T23:15:49.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523614d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 0x7f523019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523594c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52301008d0 0x7f523019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523614d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 0x7f523019ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44834/0 (socket says 172.21.15.62:44834) 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523594c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52301008d0 0x7f523019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:33840/0 (socket says 172.21.15.62:33840) 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523614d640 1 -- 172.21.15.62:0/3419104516 learned_addr learned my addr 172.21.15.62:0/3419104516 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523514b640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5230101280 0x7f523019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523614d640 1 -- 172.21.15.62:0/3419104516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5230101280 msgr2=0x7f523019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523614d640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5230101280 0x7f523019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523614d640 1 -- 172.21.15.62:0/3419104516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52301008d0 msgr2=0x7f523019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523614d640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52301008d0 0x7f523019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523614d640 1 -- 172.21.15.62:0/3419104516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523019f610 con 0x7f523010e480 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f523614d640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 0x7f523019ef30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f522802ade0 tx=0x7f5228031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f5226ffd640 1 -- 172.21.15.62:0/3419104516 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228003b70 con 0x7f523010e480 2024-09-15T23:15:49.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.569+0000 7f5226ffd640 1 -- 172.21.15.62:0/3419104516 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5228003d10 con 0x7f523010e480 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.570+0000 7f5226ffd640 1 -- 172.21.15.62:0/3419104516 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228004730 con 0x7f523010e480 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.570+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523019f8a0 con 0x7f523010e480 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.570+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52301a71e0 con 0x7f523010e480 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.571+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5230100cb0 con 0x7f523010e480 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.572+0000 7f5226ffd640 1 -- 172.21.15.62:0/3419104516 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f522802f710 con 0x7f523010e480 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.572+0000 7f5226ffd640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f521c0a3240 0x7f521c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.572+0000 7f523594c640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f521c0a3240 0x7f521c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.572+0000 7f5226ffd640 1 -- 172.21.15.62:0/3419104516 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f52280f0f50 con 0x7f523010e480 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.573+0000 7f523594c640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f521c0a3240 0x7f521c0a5700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f522c00abe0 tx=0x7f522c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:49.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.577+0000 7f5226ffd640 1 -- 172.21.15.62:0/3419104516 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f52280bd8f0 con 0x7f523010e480 2024-09-15T23:15:49.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.788+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5230109610 con 0x7f523010e480 2024-09-15T23:15:49.790 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:49 smithi062 ceph-mon[30949]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:49.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.789+0000 7f5226ffd640 1 -- 172.21.15.62:0/3419104516 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7f52280bd1f0 con 0x7f523010e480 2024-09-15T23:15:49.791 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:49.791 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"btime":"2024-09-15T23:15:44:620975+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:15:43.799068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:15:49.791 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f521c0a3240 msgr2=0x7f521c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f521c0a3240 0x7f521c0a5700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f522c00abe0 tx=0x7f522c00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 msgr2=0x7f523019ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 0x7f523019ef30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f522802ade0 tx=0x7f5228031a60 comp rx=0 tx=0).stop 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 shutdown_connections 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f521c0a3240 0x7f521c0a5700 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f523010e480 0x7f523019ef30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5230101280 0x7f523019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 --2- 172.21.15.62:0/3419104516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52301008d0 0x7f523019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 >> 172.21.15.62:0/3419104516 conn(0x7f52300fc7a0 msgr2=0x7f52300feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:49.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 shutdown_connections 2024-09-15T23:15:49.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:49.792+0000 7f5237bb0640 1 -- 172.21.15.62:0/3419104516 wait complete. 2024-09-15T23:15:49.939 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:49 smithi067 ceph-mon[35665]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:49.940 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:49 smithi148 ceph-mon[35610]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:50.429 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:15:50.692 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:50.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:50 smithi062 ceph-mon[30949]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 756 B/s wr, 6 op/s 2024-09-15T23:15:50.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:50 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3419104516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:51.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:50 smithi067 ceph-mon[35665]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 756 B/s wr, 6 op/s 2024-09-15T23:15:51.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:50 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3419104516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:51.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:50 smithi148 ceph-mon[35610]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 756 B/s wr, 6 op/s 2024-09-15T23:15:51.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:50 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3419104516' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.174+0000 7f2e01ef5640 1 -- 172.21.15.62:0/24545180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 msgr2=0x7f2dfc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.174+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/24545180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 0x7f2dfc1021c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f2dec009900 tx=0x7f2dec02a410 comp rx=0 tx=0).stop 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.174+0000 7f2e01ef5640 1 -- 172.21.15.62:0/24545180 shutdown_connections 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.174+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/24545180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dfc106f10 0x7f2dfc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.174+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/24545180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc102790 0x7f2dfc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.174+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/24545180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 0x7f2dfc1021c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.174+0000 7f2e01ef5640 1 -- 172.21.15.62:0/24545180 >> 172.21.15.62:0/24545180 conn(0x7f2dfc0fd8c0 msgr2=0x7f2dfc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.174+0000 7f2e01ef5640 1 -- 172.21.15.62:0/24545180 shutdown_connections 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.174+0000 7f2e01ef5640 1 -- 172.21.15.62:0/24545180 wait complete. 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2e01ef5640 1 Processor -- start 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2e01ef5640 1 -- start start 2024-09-15T23:15:51.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2e01ef5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 0x7f2dfc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2e01ef5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dfc102790 0x7f2dfc198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2e01ef5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc106f10 0x7f2dfc19cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2e01ef5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc111f70 con 0x7f2dfc101de0 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2e01ef5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc111df0 con 0x7f2dfc106f10 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2e01ef5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2dfc1120f0 con 0x7f2dfc102790 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 0x7f2dfc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 0x7f2dfc198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44854/0 (socket says 172.21.15.62:44854) 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfb7fe640 1 -- 172.21.15.62:0/3033776049 learned_addr learned my addr 172.21.15.62:0/3033776049 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfbfff640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc106f10 0x7f2dfc19cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfaffd640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dfc102790 0x7f2dfc198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfb7fe640 1 -- 172.21.15.62:0/3033776049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dfc102790 msgr2=0x7f2dfc198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfb7fe640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dfc102790 0x7f2dfc198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfb7fe640 1 -- 172.21.15.62:0/3033776049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc106f10 msgr2=0x7f2dfc19cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:51.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfb7fe640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc106f10 0x7f2dfc19cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfb7fe640 1 -- 172.21.15.62:0/3033776049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dfc19d3c0 con 0x7f2dfc101de0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.175+0000 7f2dfaffd640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dfc102790 0x7f2dfc198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.176+0000 7f2dfb7fe640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 0x7f2dfc198410 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2dec006010 tx=0x7f2dec0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.176+0000 7f2df8ff9640 1 -- 172.21.15.62:0/3033776049 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec03f030 con 0x7f2dfc101de0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.176+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dfc19d650 con 0x7f2dfc101de0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.176+0000 7f2df8ff9640 1 -- 172.21.15.62:0/3033776049 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dec003ce0 con 0x7f2dfc101de0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.176+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dfc1a4f90 con 0x7f2dfc101de0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.176+0000 7f2df8ff9640 1 -- 172.21.15.62:0/3033776049 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dec049060 con 0x7f2dfc101de0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.177+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dfc1032c0 con 0x7f2dfc101de0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.178+0000 7f2df8ff9640 1 -- 172.21.15.62:0/3033776049 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2dec042030 con 0x7f2dfc101de0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.179+0000 7f2df8ff9640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2de40a3240 0x7f2de40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.179+0000 7f2dfaffd640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2de40a3240 0x7f2de40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.179+0000 7f2df8ff9640 1 -- 172.21.15.62:0/3033776049 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2dec0f04a0 con 0x7f2dfc101de0 2024-09-15T23:15:51.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.180+0000 7f2dfaffd640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2de40a3240 0x7f2de40a5700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc1088e0 tx=0x7f2df0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:51.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.183+0000 7f2df8ff9640 1 -- 172.21.15.62:0/3033776049 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2dec0bcf00 con 0x7f2dfc101de0 2024-09-15T23:15:51.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.373+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2dfc10a0f0 con 0x7f2dfc101de0 2024-09-15T23:15:51.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.374+0000 7f2df8ff9640 1 -- 172.21.15.62:0/3033776049 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24777 (secure 0 0 0) 0x7f2dec0bc800 con 0x7f2dfc101de0 2024-09-15T23:15:51.376 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:51.376 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":92,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:15:42.605470+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:15:51.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2de40a3240 msgr2=0x7f2de40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:51.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2de40a3240 0x7f2de40a5700 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc1088e0 tx=0x7f2df0005f90 comp rx=0 tx=0).stop 2024-09-15T23:15:51.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 msgr2=0x7f2dfc198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:51.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 0x7f2dfc198410 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2dec006010 tx=0x7f2dec0042f0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 shutdown_connections 2024-09-15T23:15:51.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2de40a3240 0x7f2de40a5700 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc106f10 0x7f2dfc19cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dfc102790 0x7f2dfc198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 --2- 172.21.15.62:0/3033776049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dfc101de0 0x7f2dfc198410 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:51.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 >> 172.21.15.62:0/3033776049 conn(0x7f2dfc0fd8c0 msgr2=0x7f2dfc0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:51.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 shutdown_connections 2024-09-15T23:15:51.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:51.377+0000 7f2e01ef5640 1 -- 172.21.15.62:0/3033776049 wait complete. 2024-09-15T23:15:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:51 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3033776049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:52.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:51 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3033776049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:52.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:51 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3033776049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:15:52.177 DEBUG:tasks.fwd_scrub:Yielding 2024-09-15T23:15:52.177 INFO:teuthology.run_tasks:Running task background_exec... 2024-09-15T23:15:52.189 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T23:15:52.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-09-15T23:15:52.190 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-09-15T23:15:52.190 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:15:52.190 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:15:52.192 INFO:teuthology.run_tasks:Running task quiescer... 2024-09-15T23:15:52.243 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-15T23:15:52.243 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:15:52.462 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:52.741 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:52.843 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:52 smithi062 ceph-mon[30949]: pgmap v358: 129 pgs: 129 active+clean; 648 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 818 B/s wr, 6 op/s 2024-09-15T23:15:52.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.929+0000 7f1c0fcba640 1 -- 172.21.15.62:0/557550054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 msgr2=0x7f1c0810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.929+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/557550054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 0x7f1c0810d950 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009cd0 tx=0x7f1c0002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.929+0000 7f1c0fcba640 1 -- 172.21.15.62:0/557550054 shutdown_connections 2024-09-15T23:15:52.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.929+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/557550054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 0x7f1c0810d950 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.929+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/557550054 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c08100a80 0x7f1c08100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.929+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/557550054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c08106a80 0x7f1c08106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.929+0000 7f1c0fcba640 1 -- 172.21.15.62:0/557550054 >> 172.21.15.62:0/557550054 conn(0x7f1c080fc7a0 msgr2=0x7f1c080febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.929+0000 7f1c0fcba640 1 -- 172.21.15.62:0/557550054 shutdown_connections 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.929+0000 7f1c0fcba640 1 -- 172.21.15.62:0/557550054 wait complete. 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0fcba640 1 Processor -- start 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0fcba640 1 -- start start 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0fcba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c08100a80 0x7f1c080fcda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0fcba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 0x7f1c080fd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0fcba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c08106a80 0x7f1c080fd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0fcba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1c08110ea0 con 0x7f1c08101420 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0fcba640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c08110d20 con 0x7f1c08100a80 2024-09-15T23:15:52.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0fcba640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c08111020 con 0x7f1c08106a80 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0d255640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 0x7f1c080fd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0d255640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 0x7f1c080fd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42568/0 (socket says 172.21.15.62:42568) 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0d255640 1 -- 172.21.15.62:0/1700769148 learned_addr learned my addr 172.21.15.62:0/1700769148 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0da56640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c08100a80 0x7f1c080fcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0e257640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c08106a80 0x7f1c080fd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0d255640 1 -- 172.21.15.62:0/1700769148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c08106a80 msgr2=0x7f1c080fd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0d255640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c08106a80 0x7f1c080fd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0d255640 1 -- 172.21.15.62:0/1700769148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c08100a80 msgr2=0x7f1c080fcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0d255640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c08100a80 0x7f1c080fcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.930+0000 7f1c0d255640 1 -- 172.21.15.62:0/1700769148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c081a6930 con 0x7f1c08101420 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.931+0000 7f1c0d255640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 0x7f1c080fd2e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400b4a0 tx=0x7f1c0400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.931+0000 7f1bfaffd640 1 -- 172.21.15.62:0/1700769148 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04014070 con 0x7f1c08101420 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.931+0000 7f1bfaffd640 1 -- 172.21.15.62:0/1700769148 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c04010af0 con 0x7f1c08101420 2024-09-15T23:15:52.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.931+0000 7f1c0fcba640 1 -- 172.21.15.62:0/1700769148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c081a6c20 con 0x7f1c08101420 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.931+0000 7f1bfaffd640 1 -- 172.21.15.62:0/1700769148 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04017d10 con 0x7f1c08101420 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.931+0000 7f1c0fcba640 1 -- 172.21.15.62:0/1700769148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c081a7160 con 0x7f1c08101420 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.933+0000 7f1bfaffd640 1 -- 172.21.15.62:0/1700769148 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1c04017070 con 0x7f1c08101420 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.933+0000 7f1c12093740 1 Processor -- start 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c12093740 1 -- start start 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c12093740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105ab60 0x55a4b105af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c12093740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b1065800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c12093740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1068130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c12093740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a4b0fc0280 con 0x55a4b1065d40 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c12093740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a4b0f67b10 con 0x55a4b105b480 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c12093740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a4b0fbc240 con 0x55a4b105ab60 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1bfaffd640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1bf40a32c0 0x7f1bf40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c0d255640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1068130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c0da56640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b1065800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c0d255640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1068130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42584/0 (socket says 172.21.15.62:42584) 2024-09-15T23:15:52.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c0d255640 1 -- 172.21.15.62:0/1545418897 learned_addr learned my addr 172.21.15.62:0/1545418897 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:52.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c0e257640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1bf40a32c0 0x7f1bf40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.934+0000 7f1c0e257640 1 --2- 172.21.15.62:0/1545418897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105ab60 0x55a4b105af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1bfaffd640 1 -- 172.21.15.62:0/1700769148 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1c040cf0a0 con 0x7f1c08101420 2024-09-15T23:15:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1c0da56640 1 -- 172.21.15.62:0/1545418897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105ab60 msgr2=0x55a4b105af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1c0da56640 1 --2- 172.21.15.62:0/1545418897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105ab60 0x55a4b105af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1c0da56640 1 -- 172.21.15.62:0/1545418897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 msgr2=0x55a4b1068130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1c0da56640 1 --2- 172.21.15.62:0/1545418897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1068130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1c0da56640 1 -- 172.21.15.62:0/1545418897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4b0f741f0 con 0x55a4b105b480 2024-09-15T23:15:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1c0da56640 1 --2- 172.21.15.62:0/1545418897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b1065800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009b70 tx=0x7f1bfc0117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78ea969efee4ab3c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1c0e257640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1bf40a32c0 0x7f1bf40a5780 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009a60 tx=0x7f1c000095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1bf3fff640 1 -- 172.21.15.62:0/1545418897 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc009e60 con 0x55a4b105b480 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1bf3fff640 1 -- 172.21.15.62:0/1545418897 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bfc012930 con 0x55a4b105b480 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.935+0000 7f1bf3fff640 1 -- 172.21.15.62:0/1545418897 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc00fb90 con 0x55a4b105b480 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 -- 172.21.15.62:0/1545418897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 msgr2=0x55a4b1065800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 --2- 172.21.15.62:0/1545418897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b1065800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009b70 tx=0x7f1bfc0117b0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 -- 172.21.15.62:0/1545418897 shutdown_connections 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 --2- 172.21.15.62:0/1545418897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1068130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 --2- 172.21.15.62:0/1545418897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b1065800 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 --2- 172.21.15.62:0/1545418897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105ab60 0x55a4b105af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 -- 172.21.15.62:0/1545418897 >> 172.21.15.62:0/1545418897 conn(0x55a4b10570f0 msgr2=0x55a4b10574d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 -- 172.21.15.62:0/1545418897 shutdown_connections 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 -- 172.21.15.62:0/1545418897 wait complete. 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 -- start start 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.936+0000 7f1c12093740 1 Processor -- start 2024-09-15T23:15:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c12093740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105ab60 0x55a4b1062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c12093740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 0x55a4b1063100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c12093740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1063640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c12093740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a4b1068910 con 0x55a4b1065d40 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c12093740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a4b0f7af10 con 0x55a4b105ab60 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c12093740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a4b1068a90 con 0x55a4b105b480 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c12093740 10 client.0 _ll_register_callbacks cb 0x55a4b0f43a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0e257640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0e257640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1063640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42588/0 (socket says 172.21.15.62:42588) 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0da56640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105ab60 0x55a4b1062bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0e257640 1 -- 172.21.15.62:0/875621710 learned_addr learned my addr 172.21.15.62:0/875621710 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:52.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0d255640 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 0x55a4b1063100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0e257640 1 -- 172.21.15.62:0/875621710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 msgr2=0x55a4b1063100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0e257640 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 0x55a4b1063100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0e257640 1 -- 172.21.15.62:0/875621710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105ab60 msgr2=0x55a4b1062bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0e257640 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105ab60 0x55a4b1062bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0e257640 1 -- 172.21.15.62:0/875621710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4b0d316a0 con 0x55a4b1065d40 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0da56640 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105ab60 0x55a4b1062bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.937+0000 7f1c0d255640 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 0x55a4b1063100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1c0e257640 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1063640 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1c00035000 tx=0x7f1c00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e025c48e1e96f21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1bf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1bf17fa640 1 -- 172.21.15.62:0/875621710 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0004f070 con 0x55a4b1065d40 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1bf17fa640 1 -- 172.21.15.62:0/875621710 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1c00004510 con 0x55a4b1065d40 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1c12093740 1 -- 172.21.15.62:0/875621710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bfc012930 con 0x55a4b1065d40 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1bf17fa640 1 -- 172.21.15.62:0/875621710 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1c00004710 con 0x55a4b1065d40 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1c12093740 10 client.15315 fetch_fsmap learned FSMap version 25 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1c12093740 1 -- 172.21.15.62:0/875621710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a4b10744f0 con 0x55a4b1065d40 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1bf17fa640 1 -- 172.21.15.62:0/875621710 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bfc012930 con 0x55a4b1065d40 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1bf17fa640 1 -- 172.21.15.62:0/875621710 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f1c0004a070 con 0x55a4b1065d40 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1c12093740 10 client.15315 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.938+0000 7f1c12093740 10 client.15315 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.939+0000 7f1c12093740 20 client.15315 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.939+0000 7f1c12093740 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a4b1075950 0x55a4b1077e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.939+0000 7f1c12093740 4 client.15315 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.939+0000 7f1c12093740 1 -- 172.21.15.62:0/875621710 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4b0fcd7f0 con 0x55a4b1075950 2024-09-15T23:15:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.939+0000 7f1c0da56640 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a4b1075950 0x55a4b1077e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.939+0000 7f1c0da56640 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a4b1075950 0x55a4b1077e10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47cdffc8ab70d9b5 server_cookie=7de9876fe5d13fc6 in_seq=0 out_seq=0 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.939+0000 7f1bf17fa640 10 client.15315 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1bf17fa640 1 -- 172.21.15.62:0/875621710 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a4b0fcd7f0 con 0x55a4b1075950 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1bf17fa640 10 client.15315 handle_command_reply: tid=0 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 client.15315 shutdown 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 -- 172.21.15.62:0/875621710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 msgr2=0x55a4b1063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1063640 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1c00035000 tx=0x7f1c00004a40 comp rx=0 tx=0).stop 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 -- 172.21.15.62:0/875621710 shutdown_connections 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a4b1075950 0x55a4b1077e10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b1065d40 0x55a4b1063640 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 0x55a4b1063100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1bf17fa640 0 client.15315 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 --2- 172.21.15.62:0/875621710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105ab60 0x55a4b1062bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 -- 172.21.15.62:0/875621710 >> 172.21.15.62:0/875621710 conn(0x55a4b10570f0 msgr2=0x55a4b105cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 -- 172.21.15.62:0/875621710 shutdown_connections 2024-09-15T23:15:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.940+0000 7f1c12093740 1 -- 172.21.15.62:0/875621710 wait complete. 2024-09-15T23:15:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.941+0000 7f1c12093740 20 client.15315 trim_cache size 0 max 16384 2024-09-15T23:15:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c12093740 1 Processor -- start 2024-09-15T23:15:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c12093740 1 -- start start 2024-09-15T23:15:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c12093740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b105ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c12093740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 0x55a4b107d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c12093740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b1065d40 0x55a4b107dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c12093740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1c081b3760 con 0x55a4b105ab60 2024-09-15T23:15:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c12093740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c081b1030 con 0x55a4b1065d40 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c12093740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c081aee80 con 0x55a4b105b480 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0da56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b105ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0da56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b105ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42602/0 (socket says 172.21.15.62:42602) 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0da56640 1 -- 172.21.15.62:0/554514997 learned_addr learned my addr 172.21.15.62:0/554514997 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0d255640 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 0x55a4b107d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0e257640 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b1065d40 0x55a4b107dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0da56640 1 -- 172.21.15.62:0/554514997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 msgr2=0x55a4b107d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0da56640 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 0x55a4b107d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0da56640 1 -- 172.21.15.62:0/554514997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b1065d40 msgr2=0x55a4b107dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0da56640 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b1065d40 0x55a4b107dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.964+0000 7f1c0da56640 1 -- 172.21.15.62:0/554514997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4b1056f20 con 0x55a4b105ab60 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.965+0000 7f1c0e257640 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b1065d40 0x55a4b107dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.965+0000 7f1c0da56640 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b105ccc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc00f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93a2f85d12619ec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.965+0000 7f1bf37fe640 1 -- 172.21.15.62:0/554514997 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc033070 con 0x55a4b105ab60 2024-09-15T23:15:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.965+0000 7f1bf37fe640 1 -- 172.21.15.62:0/554514997 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bfc00f440 con 0x55a4b105ab60 2024-09-15T23:15:52.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.965+0000 7f1bf37fe640 1 -- 172.21.15.62:0/554514997 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc00fe00 con 0x55a4b105ab60 2024-09-15T23:15:52.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 -- 172.21.15.62:0/554514997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 msgr2=0x55a4b105ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b105ccc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc000c00 tx=0x7f1bfc00f6f0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 -- 172.21.15.62:0/554514997 shutdown_connections 2024-09-15T23:15:52.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b1065d40 0x55a4b107dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b105b480 0x55a4b107d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 --2- 172.21.15.62:0/554514997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b105ccc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 -- 172.21.15.62:0/554514997 >> 172.21.15.62:0/554514997 conn(0x55a4b10570f0 msgr2=0x55a4b105c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 -- 172.21.15.62:0/554514997 shutdown_connections 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 -- 172.21.15.62:0/554514997 wait complete. 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 -- start start 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.966+0000 7f1c12093740 1 Processor -- start 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c12093740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b106dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c12093740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b106e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c12093740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b1065d40 0x55a4b106ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0da56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b106dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c12093740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1c081af4f0 con 0x55a4b105ab60 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c12093740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c081b0960 con 0x55a4b105b480 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c12093740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c081afbc0 con 0x55a4b1065d40 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0da56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b106dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42614/0 (socket says 172.21.15.62:42614) 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0da56640 1 -- 172.21.15.62:0/2511943813 learned_addr learned my addr 172.21.15.62:0/2511943813 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c12093740 10 client.0 _ll_register_callbacks cb 0x7f1bfc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0d255640 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b106e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0e257640 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b1065d40 0x55a4b106ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0da56640 1 -- 172.21.15.62:0/2511943813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b1065d40 msgr2=0x55a4b106ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0da56640 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b1065d40 0x55a4b106ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0da56640 1 -- 172.21.15.62:0/2511943813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 msgr2=0x55a4b106e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:52.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0da56640 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b106e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0da56640 1 -- 172.21.15.62:0/2511943813 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4b106b220 con 0x55a4b105ab60 2024-09-15T23:15:52.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0e257640 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b1065d40 0x55a4b106ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:52.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0d255640 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b106e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:52.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c0da56640 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b106dd10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009a90 tx=0x7f1bfc011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fbb4d8804d1a529 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:52.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1bf17fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:52.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1bf17fa640 1 -- 172.21.15.62:0/2511943813 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc033030 con 0x55a4b105ab60 2024-09-15T23:15:52.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1bf17fa640 1 -- 172.21.15.62:0/2511943813 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1bfc02b050 con 0x55a4b105ab60 2024-09-15T23:15:52.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.967+0000 7f1c12093740 1 -- 172.21.15.62:0/2511943813 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bfc00f440 con 0x55a4b105ab60 2024-09-15T23:15:52.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1bf17fa640 1 -- 172.21.15.62:0/2511943813 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1bfc00fd50 con 0x55a4b105ab60 2024-09-15T23:15:52.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1c12093740 10 client.15321 fetch_fsmap learned FSMap version 25 2024-09-15T23:15:52.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1c12093740 1 -- 172.21.15.62:0/2511943813 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a4b10e0d40 con 0x55a4b105ab60 2024-09-15T23:15:52.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1bf17fa640 1 -- 172.21.15.62:0/2511943813 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bfc00f440 con 0x55a4b105ab60 2024-09-15T23:15:52.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1bf17fa640 1 -- 172.21.15.62:0/2511943813 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f1bfc0129c0 con 0x55a4b105ab60 2024-09-15T23:15:52.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1c12093740 10 client.15321 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:15:52.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1c12093740 10 client.15321 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:15:52.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1c12093740 20 client.15321 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:52.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1c12093740 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a4b10e1200 0x55a4b10e35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:52.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1c12093740 4 client.15321 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:15:52.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.968+0000 7f1c12093740 1 -- 172.21.15.62:0/2511943813 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a4b0fc7d00 con 0x55a4b10e1200 2024-09-15T23:15:52.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.969+0000 7f1c0d255640 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a4b10e1200 0x55a4b10e35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:52.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.969+0000 7f1bf17fa640 10 client.15321 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:52.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.969+0000 7f1c0d255640 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a4b10e1200 0x55a4b10e35f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4488ad5bc0393bd8 server_cookie=83f2118782f3ba8e in_seq=0 out_seq=0 2024-09-15T23:15:52.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1bf17fa640 1 -- 172.21.15.62:0/2511943813 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a4b0fc7d00 con 0x55a4b10e1200 2024-09-15T23:15:52.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1bf17fa640 10 client.15321 handle_command_reply: tid=0 2024-09-15T23:15:52.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 client.15321 shutdown 2024-09-15T23:15:52.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 -- 172.21.15.62:0/2511943813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 msgr2=0x55a4b106dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b106dd10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009a90 tx=0x7f1bfc011cb0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 -- 172.21.15.62:0/2511943813 shutdown_connections 2024-09-15T23:15:52.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a4b10e1200 0x55a4b10e35f0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4b1065d40 0x55a4b106ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4b105b480 0x55a4b106e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1bf17fa640 0 client.15321 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:52.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 --2- 172.21.15.62:0/2511943813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4b105ab60 0x55a4b106dd10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 -- 172.21.15.62:0/2511943813 >> 172.21.15.62:0/2511943813 conn(0x55a4b10570f0 msgr2=0x55a4b10633a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:52.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.970+0000 7f1c12093740 1 -- 172.21.15.62:0/2511943813 shutdown_connections 2024-09-15T23:15:52.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c12093740 1 -- 172.21.15.62:0/2511943813 wait complete. 2024-09-15T23:15:52.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c12093740 20 client.15321 trim_cache size 0 max 16384 2024-09-15T23:15:52.985 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:15:52.986 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:15:52.986 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "283e8d57-36b1-485d-9f31-7d1835989b67", 2024-09-15T23:15:52.986 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:15:52.986 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:15:52.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 -- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1bf40a32c0 msgr2=0x7f1bf40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1bf40a32c0 0x7f1bf40a5780 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009a60 tx=0x7f1c000095c0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 -- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 msgr2=0x7f1c080fd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:52.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 0x7f1c080fd2e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400b4a0 tx=0x7f1c0400b960 comp rx=0 tx=0).stop 2024-09-15T23:15:52.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 -- 172.21.15.62:0/1700769148 shutdown_connections 2024-09-15T23:15:52.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1bf40a32c0 0x7f1bf40a5780 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c08106a80 0x7f1c080fd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c08101420 0x7f1c080fd2e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 --2- 172.21.15.62:0/1700769148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c08100a80 0x7f1c080fcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:52.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 -- 172.21.15.62:0/1700769148 >> 172.21.15.62:0/1700769148 conn(0x7f1c080fc7a0 msgr2=0x7f1c0806aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:52.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 -- 172.21.15.62:0/1700769148 shutdown_connections 2024-09-15T23:15:52.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:52.971+0000 7f1c0fcba640 1 -- 172.21.15.62:0/1700769148 wait complete. 2024-09-15T23:15:53.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:52 smithi067 ceph-mon[35665]: pgmap v358: 129 pgs: 129 active+clean; 648 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 818 B/s wr, 6 op/s 2024-09-15T23:15:53.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:52 smithi148 ceph-mon[35610]: pgmap v358: 129 pgs: 129 active+clean; 648 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 818 B/s wr, 6 op/s 2024-09-15T23:15:53.842 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:53 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:15:53.842 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:53 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:15:53.842 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:53 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:15:53.842 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:53 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:15:53.871 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:15:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:53 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:15:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:53 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:15:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:53 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:15:54.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:53 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:15:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:53 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:15:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:53 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:15:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:53 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:15:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:53 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:15:54.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.100+0000 7fe2032cd640 1 -- 172.21.15.62:0/3858378134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc1036e0 msgr2=0x7fe1fc107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.100+0000 7fe2032cd640 1 --2- 172.21.15.62:0/3858378134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc1036e0 0x7fe1fc107b90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009930 tx=0x7fe1ec02a420 comp rx=0 tx=0).stop 2024-09-15T23:15:54.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.100+0000 7fe2032cd640 1 -- 172.21.15.62:0/3858378134 shutdown_connections 2024-09-15T23:15:54.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.100+0000 7fe2032cd640 1 --2- 172.21.15.62:0/3858378134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1fc1082a0 0x7fe1fc10ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.100+0000 7fe2032cd640 1 --2- 172.21.15.62:0/3858378134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc1036e0 0x7fe1fc107b90 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.100+0000 7fe2032cd640 1 --2- 172.21.15.62:0/3858378134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1fc102d30 0x7fe1fc103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.100+0000 7fe2032cd640 1 -- 172.21.15.62:0/3858378134 >> 172.21.15.62:0/3858378134 conn(0x7fe1fc0fc7d0 msgr2=0x7fe1fc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:54.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.100+0000 7fe2032cd640 1 -- 172.21.15.62:0/3858378134 shutdown_connections 2024-09-15T23:15:54.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.100+0000 7fe2032cd640 1 -- 172.21.15.62:0/3858378134 wait complete. 2024-09-15T23:15:54.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.101+0000 7fe2032cd640 1 Processor -- start 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.101+0000 7fe2032cd640 1 -- start start 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.101+0000 7fe2032cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc102d30 0x7fe1fc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.101+0000 7fe2032cd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1fc1036e0 0x7fe1fc198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.101+0000 7fe2032cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1fc1082a0 0x7fe1fc19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.101+0000 7fe2032cd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc112040 con 0x7fe1fc102d30 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.101+0000 7fe2032cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc111ec0 con 0x7fe1fc1082a0 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.101+0000 7fe2032cd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc1121c0 con 0x7fe1fc1036e0 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.101+0000 7fe201069640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc102d30 0x7fe1fc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe201069640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc102d30 0x7fe1fc198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42638/0 (socket says 172.21.15.62:42638) 2024-09-15T23:15:54.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe201069640 1 -- 172.21.15.62:0/1598584162 learned_addr learned my addr 172.21.15.62:0/1598584162 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:54.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe200868640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1fc1036e0 0x7fe1fc198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe20186a640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1fc1082a0 0x7fe1fc19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe201069640 1 -- 172.21.15.62:0/1598584162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1fc1036e0 msgr2=0x7fe1fc198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe201069640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1fc1036e0 0x7fe1fc198b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe201069640 1 -- 172.21.15.62:0/1598584162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1fc1082a0 msgr2=0x7fe1fc19ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe201069640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1fc1082a0 0x7fe1fc19ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe201069640 1 -- 172.21.15.62:0/1598584162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1fc19d5b0 con 0x7fe1fc102d30 2024-09-15T23:15:54.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe20186a640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1fc1082a0 0x7fe1fc19ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:15:54.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe201069640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc102d30 0x7fe1fc198600 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800d950 tx=0x7fe1f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:54.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe1f27fc640 1 -- 172.21.15.62:0/1598584162 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8017070 con 0x7fe1fc102d30 2024-09-15T23:15:54.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1fc19d8a0 con 0x7fe1fc102d30 2024-09-15T23:15:54.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe1f27fc640 1 -- 172.21.15.62:0/1598584162 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f8009c30 con 0x7fe1fc102d30 2024-09-15T23:15:54.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe1f27fc640 1 -- 172.21.15.62:0/1598584162 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800f6b0 con 0x7fe1fc102d30 2024-09-15T23:15:54.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.102+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1fc1a5180 con 0x7fe1fc102d30 2024-09-15T23:15:54.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.103+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1fc103110 con 0x7fe1fc102d30 2024-09-15T23:15:54.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.105+0000 7fe1f27fc640 1 -- 172.21.15.62:0/1598584162 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe1f80109d0 con 0x7fe1fc102d30 2024-09-15T23:15:54.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.105+0000 7fe1f27fc640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1e80a3240 0x7fe1e80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.105+0000 7fe200868640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1e80a3240 0x7fe1e80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.105+0000 7fe1f27fc640 1 -- 172.21.15.62:0/1598584162 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe1f80cf9b0 con 0x7fe1fc102d30 2024-09-15T23:15:54.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.106+0000 7fe200868640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1e80a3240 0x7fe1e80a5700 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009900 tx=0x7fe1ec032d10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:54.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.110+0000 7fe1f27fc640 1 -- 172.21.15.62:0/1598584162 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe1f8009de0 con 0x7fe1fc102d30 2024-09-15T23:15:54.191 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:54.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.302+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1fc062630 con 0x7fe1e80a3240 2024-09-15T23:15:54.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.310+0000 7fe1f27fc640 1 -- 172.21.15.62:0/1598584162 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340763 (secure 0 0 0) 0x7fe1fc062630 con 0x7fe1e80a3240 2024-09-15T23:15:54.314 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:15:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1e80a3240 msgr2=0x7fe1e80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1e80a3240 0x7fe1e80a5700 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009900 tx=0x7fe1ec032d10 comp rx=0 tx=0).stop 2024-09-15T23:15:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc102d30 msgr2=0x7fe1fc198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc102d30 0x7fe1fc198600 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe1f800d950 tx=0x7fe1f800de10 comp rx=0 tx=0).stop 2024-09-15T23:15:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 shutdown_connections 2024-09-15T23:15:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe1e80a3240 0x7fe1e80a5700 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1fc1082a0 0x7fe1fc19ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe1fc1036e0 0x7fe1fc198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 --2- 172.21.15.62:0/1598584162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1fc102d30 0x7fe1fc198600 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.313+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 >> 172.21.15.62:0/1598584162 conn(0x7fe1fc0fc7d0 msgr2=0x7fe1fc105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.314+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 shutdown_connections 2024-09-15T23:15:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.314+0000 7fe2032cd640 1 -- 172.21.15.62:0/1598584162 wait complete. 2024-09-15T23:15:54.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.864+0000 7fd26343f640 1 -- 172.21.15.62:0/2077714774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 msgr2=0x7fd25c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.864+0000 7fd26343f640 1 --2- 172.21.15.62:0/2077714774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 0x7fd25c1029b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd250009900 tx=0x7fd25002a480 comp rx=0 tx=0).stop 2024-09-15T23:15:54.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.864+0000 7fd26343f640 1 -- 172.21.15.62:0/2077714774 shutdown_connections 2024-09-15T23:15:54.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.864+0000 7fd26343f640 1 --2- 172.21.15.62:0/2077714774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd25c102ef0 0x7fd25c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.864+0000 7fd26343f640 1 --2- 172.21.15.62:0/2077714774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 0x7fd25c1029b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.864+0000 7fd26343f640 1 --2- 172.21.15.62:0/2077714774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd25c100e20 0x7fd25c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.864+0000 7fd26343f640 1 -- 172.21.15.62:0/2077714774 >> 172.21.15.62:0/2077714774 conn(0x7fd25c0fc7d0 msgr2=0x7fd25c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:54.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.864+0000 7fd26343f640 1 -- 172.21.15.62:0/2077714774 shutdown_connections 2024-09-15T23:15:54.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.864+0000 7fd26343f640 1 -- 172.21.15.62:0/2077714774 wait complete. 2024-09-15T23:15:54.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.865+0000 7fd26343f640 1 Processor -- start 2024-09-15T23:15:54.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.865+0000 7fd26343f640 1 -- start start 2024-09-15T23:15:54.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.865+0000 7fd26343f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd25c100e20 0x7fd25c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.865+0000 7fd26343f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 0x7fd25c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.865+0000 7fd26343f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd25c102ef0 0x7fd25c19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.865+0000 7fd26343f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c06a300 con 0x7fd25c102570 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.865+0000 7fd26343f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c06a180 con 0x7fd25c102ef0 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.865+0000 7fd26343f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c06a480 con 0x7fd25c100e20 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2609da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 0x7fd25c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2609da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 0x7fd25c198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42656/0 (socket says 172.21.15.62:42656) 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2609da640 1 -- 172.21.15.62:0/4087887198 learned_addr learned my addr 172.21.15.62:0/4087887198 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2619dc640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd25c102ef0 0x7fd25c19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2611db640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd25c100e20 0x7fd25c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2609da640 1 -- 172.21.15.62:0/4087887198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd25c100e20 msgr2=0x7fd25c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2609da640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd25c100e20 0x7fd25c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2609da640 1 -- 172.21.15.62:0/4087887198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd25c102ef0 msgr2=0x7fd25c19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2609da640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd25c102ef0 0x7fd25c19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2609da640 1 -- 172.21.15.62:0/4087887198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c19d550 con 0x7fd25c102570 2024-09-15T23:15:54.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd2609da640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 0x7fd25c198ae0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd250006010 tx=0x7fd250003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:54.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd24e7fc640 1 -- 172.21.15.62:0/4087887198 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25003f070 con 0x7fd25c102570 2024-09-15T23:15:54.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd24e7fc640 1 -- 172.21.15.62:0/4087887198 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd250031c30 con 0x7fd25c102570 2024-09-15T23:15:54.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd26343f640 1 -- 172.21.15.62:0/4087887198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd25c19d7e0 con 0x7fd25c102570 2024-09-15T23:15:54.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd24e7fc640 1 -- 172.21.15.62:0/4087887198 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250035030 con 0x7fd25c102570 2024-09-15T23:15:54.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.866+0000 7fd26343f640 1 -- 172.21.15.62:0/4087887198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd25c1a5120 con 0x7fd25c102570 2024-09-15T23:15:54.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.869+0000 7fd24e7fc640 1 -- 172.21.15.62:0/4087887198 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd250042070 con 0x7fd25c102570 2024-09-15T23:15:54.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd265818740 1 Processor -- start 2024-09-15T23:15:54.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd265818740 1 -- start start 2024-09-15T23:15:54.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd24e7fc640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2480a3240 0x7fd2480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd265818740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5f0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd265818740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5ff550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd265818740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 0x561daf601e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd265818740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561daf5562e0 con 0x561daf5f0c60 2024-09-15T23:15:54.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd265818740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561daf4fdce0 con 0x561daf5ffa90 2024-09-15T23:15:54.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd265818740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561daf5522a0 con 0x561daf5f0340 2024-09-15T23:15:54.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.870+0000 7fd2619dc640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2480a3240 0x7fd2480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd24e7fc640 1 -- 172.21.15.62:0/4087887198 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd2500f0560 con 0x7fd25c102570 2024-09-15T23:15:54.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2611db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2611db640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5f0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:60270/0 (socket says 172.21.15.62:60270) 2024-09-15T23:15:54.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2611db640 1 -- 172.21.15.62:0/2784891439 learned_addr learned my addr 172.21.15.62:0/2784891439 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:54.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2609da640 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5ff550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2619dc640 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 0x561daf601e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2611db640 1 -- 172.21.15.62:0/2784891439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 msgr2=0x561daf601e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:54.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2611db640 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 0x561daf601e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2611db640 1 -- 172.21.15.62:0/2784891439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 msgr2=0x561daf5ff550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2619dc640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2480a3240 0x7fd2480a5700 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd254005fd0 tx=0x7fd2540096a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2611db640 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5ff550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2611db640 1 -- 172.21.15.62:0/2784891439 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561daf4e2440 con 0x561daf5f0340 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2609da640 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.871+0000 7fd2611db640 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5f0720 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd25800bcf0 tx=0x7fd25800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a43753a9c7f7dd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.872+0000 7fd2477fe640 1 -- 172.21.15.62:0/2784891439 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd258015070 con 0x561daf5f0340 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.872+0000 7fd2477fe640 1 -- 172.21.15.62:0/2784891439 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd258011920 con 0x561daf5f0340 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.872+0000 7fd2477fe640 1 -- 172.21.15.62:0/2784891439 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd258010930 con 0x561daf5f0340 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 -- 172.21.15.62:0/2784891439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 msgr2=0x561daf5f0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5f0720 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd25800bcf0 tx=0x7fd25800f730 comp rx=0 tx=0).stop 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 -- 172.21.15.62:0/2784891439 shutdown_connections 2024-09-15T23:15:54.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 0x561daf601e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 --2- 172.21.15.62:0/2784891439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5f0720 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 -- 172.21.15.62:0/2784891439 >> 172.21.15.62:0/2784891439 conn(0x561daf5eca30 msgr2=0x561daf5ece10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 -- 172.21.15.62:0/2784891439 shutdown_connections 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 -- 172.21.15.62:0/2784891439 wait complete. 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 -- start start 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.873+0000 7fd265818740 1 Processor -- start 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd265818740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5ff3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd265818740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0c60 0x561daf5fc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd265818740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5ffa90 0x561daf5fcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd265818740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561daf602660 con 0x561daf5ffa90 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd265818740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561daf511050 con 0x561daf5f0c60 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd265818740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561daf6027e0 con 0x561daf5f0340 2024-09-15T23:15:54.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd265818740 10 client.0 _ll_register_callbacks cb 0x561daf551160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2619dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5ffa90 0x561daf5fcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2619dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5ffa90 0x561daf5fcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42666/0 (socket says 172.21.15.62:42666) 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2619dc640 1 -- 172.21.15.62:0/3411043627 learned_addr learned my addr 172.21.15.62:0/3411043627 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2609da640 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0c60 0x561daf5fc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2611db640 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5ff3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2619dc640 1 -- 172.21.15.62:0/3411043627 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 msgr2=0x561daf5ff3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2619dc640 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5ff3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2619dc640 1 -- 172.21.15.62:0/3411043627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0c60 msgr2=0x561daf5fc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2619dc640 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0c60 0x561daf5fc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2619dc640 1 -- 172.21.15.62:0/3411043627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561daf2c73c0 con 0x561daf5ffa90 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2609da640 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0c60 0x561daf5fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd2619dc640 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5ffa90 0x561daf5fcbf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd254011950 tx=0x7fd254004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba181df2227544a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:54.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.874+0000 7fd244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd244ff9640 1 -- 172.21.15.62:0/3411043627 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd25401dd60 con 0x561daf5ffa90 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd265818740 1 -- 172.21.15.62:0/3411043627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd258011920 con 0x561daf5ffa90 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd244ff9640 1 -- 172.21.15.62:0/3411043627 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd25401e440 con 0x561daf5ffa90 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd244ff9640 1 -- 172.21.15.62:0/3411043627 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd25401e640 con 0x561daf5ffa90 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd265818740 10 client.15339 fetch_fsmap learned FSMap version 25 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd265818740 1 -- 172.21.15.62:0/3411043627 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561daf60e120 con 0x561daf5ffa90 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd244ff9640 1 -- 172.21.15.62:0/3411043627 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd258011920 con 0x561daf5ffa90 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd244ff9640 1 -- 172.21.15.62:0/3411043627 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fd25401b960 con 0x561daf5ffa90 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd265818740 10 client.15339 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd265818740 10 client.15339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd265818740 20 client.15339 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd265818740 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561daf60f530 0x561daf611980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd265818740 4 client.15339 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:15:54.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.875+0000 7fd265818740 1 -- 172.21.15.62:0/3411043627 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561daf563850 con 0x561daf60f530 2024-09-15T23:15:54.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.876+0000 7fd2611db640 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561daf60f530 0x561daf611980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.876+0000 7fd244ff9640 10 client.15339 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:54.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.876+0000 7fd2611db640 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561daf60f530 0x561daf611980 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edc8d64801b88edb server_cookie=995e1eb87eefb3b1 in_seq=0 out_seq=0 2024-09-15T23:15:54.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd244ff9640 1 -- 172.21.15.62:0/3411043627 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561daf563850 con 0x561daf60f530 2024-09-15T23:15:54.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd244ff9640 10 client.15339 handle_command_reply: tid=0 2024-09-15T23:15:54.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 client.15339 shutdown 2024-09-15T23:15:54.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 -- 172.21.15.62:0/3411043627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5ffa90 msgr2=0x561daf5fcbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5ffa90 0x561daf5fcbf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd254011950 tx=0x7fd254004ab0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 -- 172.21.15.62:0/3411043627 shutdown_connections 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561daf60f530 0x561daf611980 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5ffa90 0x561daf5fcbf0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0c60 0x561daf5fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 --2- 172.21.15.62:0/3411043627 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf5ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 -- 172.21.15.62:0/3411043627 >> 172.21.15.62:0/3411043627 conn(0x561daf5eca30 msgr2=0x561daf5f2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd244ff9640 0 client.15339 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 -- 172.21.15.62:0/3411043627 shutdown_connections 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 1 -- 172.21.15.62:0/3411043627 wait complete. 2024-09-15T23:15:54.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.877+0000 7fd265818740 20 client.15339 trim_cache size 0 max 16384 2024-09-15T23:15:54.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.894+0000 7fd265818740 1 Processor -- start 2024-09-15T23:15:54.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd265818740 1 -- start start 2024-09-15T23:15:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd265818740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0340 0x561daf60b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd265818740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5fbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd265818740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5ffa90 0x561daf5f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd265818740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1af290 con 0x561daf5f0c60 2024-09-15T23:15:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd265818740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1acb60 con 0x561daf5f0340 2024-09-15T23:15:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd265818740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1aa9b0 con 0x561daf5ffa90 2024-09-15T23:15:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2609da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5fbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2609da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5fbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42678/0 (socket says 172.21.15.62:42678) 2024-09-15T23:15:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2609da640 1 -- 172.21.15.62:0/1671232889 learned_addr learned my addr 172.21.15.62:0/1671232889 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2619dc640 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5ffa90 0x561daf5f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2611db640 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0340 0x561daf60b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2609da640 1 -- 172.21.15.62:0/1671232889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5ffa90 msgr2=0x561daf5f7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2609da640 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5ffa90 0x561daf5f7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2609da640 1 -- 172.21.15.62:0/1671232889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0340 msgr2=0x561daf60b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2609da640 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0340 0x561daf60b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2609da640 1 -- 172.21.15.62:0/1671232889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561daf5fb640 con 0x561daf5f0c60 2024-09-15T23:15:54.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2619dc640 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5ffa90 0x561daf5f7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.895+0000 7fd2609da640 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5fbe00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd250005940 tx=0x7fd2500bc2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc160abe25eba6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd246ffd640 1 -- 172.21.15.62:0/1671232889 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250049040 con 0x561daf5f0c60 2024-09-15T23:15:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd246ffd640 1 -- 172.21.15.62:0/1671232889 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25002e440 con 0x561daf5f0c60 2024-09-15T23:15:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd246ffd640 1 -- 172.21.15.62:0/1671232889 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2500c5410 con 0x561daf5f0c60 2024-09-15T23:15:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd265818740 1 -- 172.21.15.62:0/1671232889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 msgr2=0x561daf5fbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd265818740 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5fbe00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd250005940 tx=0x7fd2500bc2e0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd265818740 1 -- 172.21.15.62:0/1671232889 shutdown_connections 2024-09-15T23:15:54.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd265818740 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5ffa90 0x561daf5f7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd265818740 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf5fbe00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd265818740 1 --2- 172.21.15.62:0/1671232889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5f0340 0x561daf60b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd265818740 1 -- 172.21.15.62:0/1671232889 >> 172.21.15.62:0/1671232889 conn(0x561daf5eca30 msgr2=0x561daf5f74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd265818740 1 -- 172.21.15.62:0/1671232889 shutdown_connections 2024-09-15T23:15:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.896+0000 7fd265818740 1 -- 172.21.15.62:0/1671232889 wait complete. 2024-09-15T23:15:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd265818740 1 -- start start 2024-09-15T23:15:54.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd265818740 1 Processor -- start 2024-09-15T23:15:54.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd265818740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf608110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd265818740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf603440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd265818740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 0x561daf603980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd265818740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1ab020 con 0x561daf5f0c60 2024-09-15T23:15:54.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd265818740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1ac490 con 0x561daf5ffa90 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd265818740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd25c1ab6f0 con 0x561daf5f0340 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd265818740 10 client.0 _ll_register_callbacks cb 0x7fd254009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd2609da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf603440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd2609da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf603440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42688/0 (socket says 172.21.15.62:42688) 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.897+0000 7fd2609da640 1 -- 172.21.15.62:0/2623583714 learned_addr learned my addr 172.21.15.62:0/2623583714 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd2611db640 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf608110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd2619dc640 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 0x561daf603980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd2609da640 1 -- 172.21.15.62:0/2623583714 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 msgr2=0x561daf608110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd2609da640 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf608110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd2609da640 1 -- 172.21.15.62:0/2623583714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 msgr2=0x561daf603980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd2609da640 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 0x561daf603980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd2609da640 1 -- 172.21.15.62:0/2623583714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561daf5f0a90 con 0x561daf5f0c60 2024-09-15T23:15:54.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd2619dc640 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 0x561daf603980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd2609da640 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf603440 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd250005ab0 tx=0x7fd2500bacb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124d028b3290319a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd244ff9640 1 -- 172.21.15.62:0/2623583714 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd250049070 con 0x561daf5f0c60 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd244ff9640 1 -- 172.21.15.62:0/2623583714 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd25003e4a0 con 0x561daf5f0c60 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd265818740 1 -- 172.21.15.62:0/2623583714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561daf55dd60 con 0x561daf5f0c60 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd244ff9640 1 -- 172.21.15.62:0/2623583714 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd25003e6e0 con 0x561daf5f0c60 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd265818740 10 client.15345 fetch_fsmap learned FSMap version 25 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd265818740 1 -- 172.21.15.62:0/2623583714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561daf679f80 con 0x561daf5f0c60 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.898+0000 7fd244ff9640 1 -- 172.21.15.62:0/2623583714 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561daf55dd60 con 0x561daf5f0c60 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd244ff9640 1 -- 172.21.15.62:0/2623583714 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fd2500c4f80 con 0x561daf5f0c60 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd265818740 10 client.15345 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd265818740 10 client.15345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd265818740 20 client.15345 populate_metadata read hostname 'smithi062' 2024-09-15T23:15:54.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd265818740 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561daf67a680 0x561daf67caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd265818740 4 client.15345 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd265818740 1 -- 172.21.15.62:0/2623583714 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561daf4da210 con 0x561daf67a680 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd2611db640 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561daf67a680 0x561daf67caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd2611db640 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561daf67a680 0x561daf67caa0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eab2c776d3c30ea1 server_cookie=cd7591ae0d8b8989 in_seq=0 out_seq=0 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd244ff9640 10 client.15345 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd244ff9640 1 -- 172.21.15.62:0/2623583714 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x561daf4da210 con 0x561daf67a680 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd244ff9640 10 client.15345 handle_command_reply: tid=0 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.899+0000 7fd265818740 1 client.15345 shutdown 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 -- 172.21.15.62:0/2623583714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 msgr2=0x561daf603440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf603440 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd250005ab0 tx=0x7fd2500bacb0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 -- 172.21.15.62:0/2623583714 shutdown_connections 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561daf67a680 0x561daf67caa0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561daf5ffa90 0x561daf603980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561daf5f0c60 0x561daf603440 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd244ff9640 0 client.15345 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 --2- 172.21.15.62:0/2623583714 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561daf5f0340 0x561daf608110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 -- 172.21.15.62:0/2623583714 >> 172.21.15.62:0/2623583714 conn(0x561daf5eca30 msgr2=0x561daf5f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 -- 172.21.15.62:0/2623583714 shutdown_connections 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 1 -- 172.21.15.62:0/2623583714 wait complete. 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd265818740 20 client.15345 trim_cache size 0 max 16384 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stdout: "283e8d57-36b1-485d-9f31-7d1835989b67": { 2024-09-15T23:15:54.908 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:15:54.909 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "283e8d57-36b1-485d-9f31-7d1835989b67", 2024-09-15T23:15:54.909 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:15:54.909 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:15:54.909 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:15:54.909 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:15:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd26343f640 1 -- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2480a3240 msgr2=0x7fd2480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd26343f640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2480a3240 0x7fd2480a5700 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd254005fd0 tx=0x7fd2540096a0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd26343f640 1 -- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 msgr2=0x7fd25c198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd26343f640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 0x7fd25c198ae0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd250006010 tx=0x7fd250003c10 comp rx=0 tx=0).stop 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd26343f640 1 -- 172.21.15.62:0/4087887198 shutdown_connections 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd26343f640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2480a3240 0x7fd2480a5700 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.900+0000 7fd26343f640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd25c102ef0 0x7fd25c19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.901+0000 7fd26343f640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd25c102570 0x7fd25c198ae0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.901+0000 7fd26343f640 1 --2- 172.21.15.62:0/4087887198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd25c100e20 0x7fd25c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.901+0000 7fd26343f640 1 -- 172.21.15.62:0/4087887198 >> 172.21.15.62:0/4087887198 conn(0x7fd25c0fc7d0 msgr2=0x7fd25c104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.901+0000 7fd26343f640 1 -- 172.21.15.62:0/4087887198 shutdown_connections 2024-09-15T23:15:54.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:54.901+0000 7fd26343f640 1 -- 172.21.15.62:0/4087887198 wait complete. 2024-09-15T23:15:54.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:54 smithi062 ceph-mon[30949]: pgmap v359: 129 pgs: 129 active+clean; 648 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 736 B/s wr, 5 op/s 2024-09-15T23:15:54.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:54 smithi062 ceph-mon[30949]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:55.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:54 smithi067 ceph-mon[35665]: pgmap v359: 129 pgs: 129 active+clean; 648 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 736 B/s wr, 5 op/s 2024-09-15T23:15:55.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:54 smithi067 ceph-mon[35665]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:55.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:54 smithi148 ceph-mon[35610]: pgmap v359: 129 pgs: 129 active+clean; 648 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 736 B/s wr, 5 op/s 2024-09-15T23:15:55.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:54 smithi148 ceph-mon[35610]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:55.363 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-15T23:15:53.727230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663607,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":773,"num_read_kb":510,"num_write":362,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369996,"kb_used_data":12368,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571828,"statfs":{"total":1151940427776,"available":1151561551872,"internally_reserved":0,"allocated":12664832,"data_stored":8233401,"data_compressed":461688,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5681,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":35,"num_write":4,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.119763"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609573+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:15:42.609573+0000","last_peered":"2024-09-15T23:15:42.609573+0000","last_clean":"2024-09-15T23:15:42.609573+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:15:42.609573+0000","last_undegraded":"2024-09-15T23:15:42.609573+0000","last_fullsized":"2024-09-15T23:15:42.609573+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317224+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:15:43.317224+0000","last_peered":"2024-09-15T23:15:43.317224+0000","last_clean":"2024-09-15T23:15:43.317224+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:15:43.317224+0000","last_undegraded":"2024-09-15T23:15:43.317224+0000","last_fullsized":"2024-09-15T23:15:43.317224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613665+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:15:42.613665+0000","last_peered":"2024-09-15T23:15:42.613665+0000","last_clean":"2024-09-15T23:15:42.613665+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:15:42.613665+0000","last_undegraded":"2024-09-15T23:15:42.613665+0000","last_fullsized":"2024-09-15T23:15:42.613665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317397+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:15:43.317397+0000","last_peered":"2024-09-15T23:15:43.317397+0000","last_clean":"2024-09-15T23:15:43.317397+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:15:43.317397+0000","last_undegraded":"2024-09-15T23:15:43.317397+0000","last_fullsized":"2024-09-15T23:15:43.317397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.628111+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:15:43.628111+0000","last_peered":"2024-09-15T23:15:43.628111+0000","last_clean":"2024-09-15T23:15:43.628111+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:15:43.628111+0000","last_undegraded":"2024-09-15T23:15:43.628111+0000","last_fullsized":"2024-09-15T23:15:43.628111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611378+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:15:42.611378+0000","last_peered":"2024-09-15T23:15:42.611378+0000","last_clean":"2024-09-15T23:15:42.611378+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:15:42.611378+0000","last_undegraded":"2024-09-15T23:15:42.611378+0000","last_fullsized":"2024-09-15T23:15:42.611378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909808+0000","last_change":"2024-09-15T23:14:41.659816+0000","last_active":"2024-09-15T23:15:42.909808+0000","last_peered":"2024-09-15T23:15:42.909808+0000","last_clean":"2024-09-15T23:15:42.909808+0000","last_became_active":"2024-09-15T23:14:41.641611+0000","last_became_peered":"2024-09-15T23:14:41.641611+0000","last_unstale":"2024-09-15T23:15:42.909808+0000","last_undegraded":"2024-09-15T23:15:42.909808+0000","last_fullsized":"2024-09-15T23:15:42.909808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:23.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612963+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:15:42.612963+0000","last_peered":"2024-09-15T23:15:42.612963+0000","last_clean":"2024-09-15T23:15:42.612963+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:15:42.612963+0000","last_undegraded":"2024-09-15T23:15:42.612963+0000","last_fullsized":"2024-09-15T23:15:42.612963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610025+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:15:42.610025+0000","last_peered":"2024-09-15T23:15:42.610025+0000","last_clean":"2024-09-15T23:15:42.610025+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:15:42.610025+0000","last_undegraded":"2024-09-15T23:15:42.610025+0000","last_fullsized":"2024-09-15T23:15:42.610025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614867+0000","last_change":"2024-09-15T23:15:42.614867+0000","last_active":"2024-09-15T23:15:42.614867+0000","last_peered":"2024-09-15T23:15:42.614867+0000","last_clean":"2024-09-15T23:15:42.614867+0000","last_became_active":"2024-09-15T23:15:42.614459+0000","last_became_peered":"2024-09-15T23:15:42.614459+0000","last_unstale":"2024-09-15T23:15:42.614867+0000","last_undegraded":"2024-09-15T23:15:42.614867+0000","last_fullsized":"2024-09-15T23:15:42.614867+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:15.254442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611583+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:15:42.611583+0000","last_peered":"2024-09-15T23:15:42.611583+0000","last_clean":"2024-09-15T23:15:42.611583+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:15:42.611583+0000","last_undegraded":"2024-09-15T23:15:42.611583+0000","last_fullsized":"2024-09-15T23:15:42.611583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317095+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:15:43.317095+0000","last_peered":"2024-09-15T23:15:43.317095+0000","last_clean":"2024-09-15T23:15:43.317095+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:15:43.317095+0000","last_undegraded":"2024-09-15T23:15:43.317095+0000","last_fullsized":"2024-09-15T23:15:43.317095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":56,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609750+0000","last_change":"2024-09-15T23:14:41.646320+0000","last_active":"2024-09-15T23:15:42.609750+0000","last_peered":"2024-09-15T23:15:42.609750+0000","last_clean":"2024-09-15T23:15:42.609750+0000","last_became_active":"2024-09-15T23:14:41.641933+0000","last_became_peered":"2024-09-15T23:14:41.641933+0000","last_unstale":"2024-09-15T23:15:42.609750+0000","last_undegraded":"2024-09-15T23:15:42.609750+0000","last_fullsized":"2024-09-15T23:15:42.609750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:49.417407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611897+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:15:42.611897+0000","last_peered":"2024-09-15T23:15:42.611897+0000","last_clean":"2024-09-15T23:15:42.611897+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:15:42.611897+0000","last_undegraded":"2024-09-15T23:15:42.611897+0000","last_fullsized":"2024-09-15T23:15:42.611897+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316939+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:15:43.316939+0000","last_peered":"2024-09-15T23:15:43.316939+0000","last_clean":"2024-09-15T23:15:43.316939+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:15:43.316939+0000","last_undegraded":"2024-09-15T23:15:43.316939+0000","last_fullsized":"2024-09-15T23:15:43.316939+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613807+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:15:42.613807+0000","last_peered":"2024-09-15T23:15:42.613807+0000","last_clean":"2024-09-15T23:15:42.613807+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:15:42.613807+0000","last_undegraded":"2024-09-15T23:15:42.613807+0000","last_fullsized":"2024-09-15T23:15:42.613807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612402+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:15:42.612402+0000","last_peered":"2024-09-15T23:15:42.612402+0000","last_clean":"2024-09-15T23:15:42.612402+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:15:42.612402+0000","last_undegraded":"2024-09-15T23:15:42.612402+0000","last_fullsized":"2024-09-15T23:15:42.612402+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909877+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:15:42.909877+0000","last_peered":"2024-09-15T23:15:42.909877+0000","last_clean":"2024-09-15T23:15:42.909877+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:15:42.909877+0000","last_undegraded":"2024-09-15T23:15:42.909877+0000","last_fullsized":"2024-09-15T23:15:42.909877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316960+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:15:43.316960+0000","last_peered":"2024-09-15T23:15:43.316960+0000","last_clean":"2024-09-15T23:15:43.316960+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:15:43.316960+0000","last_undegraded":"2024-09-15T23:15:43.316960+0000","last_fullsized":"2024-09-15T23:15:43.316960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609295+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:15:42.609295+0000","last_peered":"2024-09-15T23:15:42.609295+0000","last_clean":"2024-09-15T23:15:42.609295+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:15:42.609295+0000","last_undegraded":"2024-09-15T23:15:42.609295+0000","last_fullsized":"2024-09-15T23:15:42.609295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609453+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:15:42.609453+0000","last_peered":"2024-09-15T23:15:42.609453+0000","last_clean":"2024-09-15T23:15:42.609453+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:15:42.609453+0000","last_undegraded":"2024-09-15T23:15:42.609453+0000","last_fullsized":"2024-09-15T23:15:42.609453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613324+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:15:42.613324+0000","last_peered":"2024-09-15T23:15:42.613324+0000","last_clean":"2024-09-15T23:15:42.613324+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:15:42.613324+0000","last_undegraded":"2024-09-15T23:15:42.613324+0000","last_fullsized":"2024-09-15T23:15:42.613324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612027+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:15:42.612027+0000","last_peered":"2024-09-15T23:15:42.612027+0000","last_clean":"2024-09-15T23:15:42.612027+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:15:42.612027+0000","last_undegraded":"2024-09-15T23:15:42.612027+0000","last_fullsized":"2024-09-15T23:15:42.612027+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":47,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.628020+0000","last_change":"2024-09-15T23:14:42.009605+0000","last_active":"2024-09-15T23:15:43.628020+0000","last_peered":"2024-09-15T23:15:43.628020+0000","last_clean":"2024-09-15T23:15:43.628020+0000","last_became_active":"2024-09-15T23:14:42.009237+0000","last_became_peered":"2024-09-15T23:14:42.009237+0000","last_unstale":"2024-09-15T23:15:43.628020+0000","last_undegraded":"2024-09-15T23:15:43.628020+0000","last_fullsized":"2024-09-15T23:15:43.628020+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:11:08.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612739+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:15:42.612739+0000","last_peered":"2024-09-15T23:15:42.612739+0000","last_clean":"2024-09-15T23:15:42.612739+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:15:42.612739+0000","last_undegraded":"2024-09-15T23:15:42.612739+0000","last_fullsized":"2024-09-15T23:15:42.612739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614193+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:15:42.614193+0000","last_peered":"2024-09-15T23:15:42.614193+0000","last_clean":"2024-09-15T23:15:42.614193+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:15:42.614193+0000","last_undegraded":"2024-09-15T23:15:42.614193+0000","last_fullsized":"2024-09-15T23:15:42.614193+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317410+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:15:43.317410+0000","last_peered":"2024-09-15T23:15:43.317410+0000","last_clean":"2024-09-15T23:15:43.317410+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:15:43.317410+0000","last_undegraded":"2024-09-15T23:15:43.317410+0000","last_fullsized":"2024-09-15T23:15:43.317410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'14","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610065+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:15:42.610065+0000","last_peered":"2024-09-15T23:15:42.610065+0000","last_clean":"2024-09-15T23:15:42.610065+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:15:42.610065+0000","last_undegraded":"2024-09-15T23:15:42.610065+0000","last_fullsized":"2024-09-15T23:15:42.610065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316638+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:15:43.316638+0000","last_peered":"2024-09-15T23:15:43.316638+0000","last_clean":"2024-09-15T23:15:43.316638+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:15:43.316638+0000","last_undegraded":"2024-09-15T23:15:43.316638+0000","last_fullsized":"2024-09-15T23:15:43.316638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612814+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:15:42.612814+0000","last_peered":"2024-09-15T23:15:42.612814+0000","last_clean":"2024-09-15T23:15:42.612814+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:15:42.612814+0000","last_undegraded":"2024-09-15T23:15:42.612814+0000","last_fullsized":"2024-09-15T23:15:42.612814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610084+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:15:42.610084+0000","last_peered":"2024-09-15T23:15:42.610084+0000","last_clean":"2024-09-15T23:15:42.610084+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:15:42.610084+0000","last_undegraded":"2024-09-15T23:15:42.610084+0000","last_fullsized":"2024-09-15T23:15:42.610084+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909814+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:15:42.909814+0000","last_peered":"2024-09-15T23:15:42.909814+0000","last_clean":"2024-09-15T23:15:42.909814+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:15:42.909814+0000","last_undegraded":"2024-09-15T23:15:42.909814+0000","last_fullsized":"2024-09-15T23:15:42.909814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610218+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:15:42.610218+0000","last_peered":"2024-09-15T23:15:42.610218+0000","last_clean":"2024-09-15T23:15:42.610218+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:15:42.610218+0000","last_undegraded":"2024-09-15T23:15:42.610218+0000","last_fullsized":"2024-09-15T23:15:42.610218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609669+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:15:42.609669+0000","last_peered":"2024-09-15T23:15:42.609669+0000","last_clean":"2024-09-15T23:15:42.609669+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:15:42.609669+0000","last_undegraded":"2024-09-15T23:15:42.609669+0000","last_fullsized":"2024-09-15T23:15:42.609669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609473+0000","last_change":"2024-09-15T23:14:41.995466+0000","last_active":"2024-09-15T23:15:42.609473+0000","last_peered":"2024-09-15T23:15:42.609473+0000","last_clean":"2024-09-15T23:15:42.609473+0000","last_became_active":"2024-09-15T23:14:41.994915+0000","last_became_peered":"2024-09-15T23:14:41.994915+0000","last_unstale":"2024-09-15T23:15:42.609473+0000","last_undegraded":"2024-09-15T23:15:42.609473+0000","last_fullsized":"2024-09-15T23:15:42.609473+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:04.121596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610393+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:15:42.610393+0000","last_peered":"2024-09-15T23:15:42.610393+0000","last_clean":"2024-09-15T23:15:42.610393+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:15:42.610393+0000","last_undegraded":"2024-09-15T23:15:42.610393+0000","last_fullsized":"2024-09-15T23:15:42.610393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317131+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:15:43.317131+0000","last_peered":"2024-09-15T23:15:43.317131+0000","last_clean":"2024-09-15T23:15:43.317131+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:15:43.317131+0000","last_undegraded":"2024-09-15T23:15:43.317131+0000","last_fullsized":"2024-09-15T23:15:43.317131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626237+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:15:42.626237+0000","last_peered":"2024-09-15T23:15:42.626237+0000","last_clean":"2024-09-15T23:15:42.626237+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:15:42.626237+0000","last_undegraded":"2024-09-15T23:15:42.626237+0000","last_fullsized":"2024-09-15T23:15:42.626237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":145,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:48.626995+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:15:48.626995+0000","last_peered":"2024-09-15T23:15:48.626995+0000","last_clean":"2024-09-15T23:15:48.626995+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:15:48.626995+0000","last_undegraded":"2024-09-15T23:15:48.626995+0000","last_fullsized":"2024-09-15T23:15:48.626995+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613035+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:15:42.613035+0000","last_peered":"2024-09-15T23:15:42.613035+0000","last_clean":"2024-09-15T23:15:42.613035+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:15:42.613035+0000","last_undegraded":"2024-09-15T23:15:42.613035+0000","last_fullsized":"2024-09-15T23:15:42.613035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.617748+0000","last_change":"2024-09-15T23:15:42.617748+0000","last_active":"2024-09-15T23:15:42.617748+0000","last_peered":"2024-09-15T23:15:42.617748+0000","last_clean":"2024-09-15T23:15:42.617748+0000","last_became_active":"2024-09-15T23:15:42.617320+0000","last_became_peered":"2024-09-15T23:15:42.617320+0000","last_unstale":"2024-09-15T23:15:42.617748+0000","last_undegraded":"2024-09-15T23:15:42.617748+0000","last_fullsized":"2024-09-15T23:15:42.617748+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:47.900049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316911+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:15:43.316911+0000","last_peered":"2024-09-15T23:15:43.316911+0000","last_clean":"2024-09-15T23:15:43.316911+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:15:43.316911+0000","last_undegraded":"2024-09-15T23:15:43.316911+0000","last_fullsized":"2024-09-15T23:15:43.316911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626588+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:15:42.626588+0000","last_peered":"2024-09-15T23:15:42.626588+0000","last_clean":"2024-09-15T23:15:42.626588+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:15:42.626588+0000","last_undegraded":"2024-09-15T23:15:42.626588+0000","last_fullsized":"2024-09-15T23:15:42.626588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316962+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:43.316962+0000","last_peered":"2024-09-15T23:15:43.316962+0000","last_clean":"2024-09-15T23:15:43.316962+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:15:43.316962+0000","last_undegraded":"2024-09-15T23:15:43.316962+0000","last_fullsized":"2024-09-15T23:15:43.316962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612474+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:15:42.612474+0000","last_peered":"2024-09-15T23:15:42.612474+0000","last_clean":"2024-09-15T23:15:42.612474+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:15:42.612474+0000","last_undegraded":"2024-09-15T23:15:42.612474+0000","last_fullsized":"2024-09-15T23:15:42.612474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317208+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:15:43.317208+0000","last_peered":"2024-09-15T23:15:43.317208+0000","last_clean":"2024-09-15T23:15:43.317208+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:15:43.317208+0000","last_undegraded":"2024-09-15T23:15:43.317208+0000","last_fullsized":"2024-09-15T23:15:43.317208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317366+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:15:43.317366+0000","last_peered":"2024-09-15T23:15:43.317366+0000","last_clean":"2024-09-15T23:15:43.317366+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:15:43.317366+0000","last_undegraded":"2024-09-15T23:15:43.317366+0000","last_fullsized":"2024-09-15T23:15:43.317366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317493+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:15:43.317493+0000","last_peered":"2024-09-15T23:15:43.317493+0000","last_clean":"2024-09-15T23:15:43.317493+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:15:43.317493+0000","last_undegraded":"2024-09-15T23:15:43.317493+0000","last_fullsized":"2024-09-15T23:15:43.317493+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614232+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:15:42.614232+0000","last_peered":"2024-09-15T23:15:42.614232+0000","last_clean":"2024-09-15T23:15:42.614232+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:15:42.614232+0000","last_undegraded":"2024-09-15T23:15:42.614232+0000","last_fullsized":"2024-09-15T23:15:42.614232+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612481+0000","last_change":"2024-09-15T23:14:41.994950+0000","last_active":"2024-09-15T23:15:42.612481+0000","last_peered":"2024-09-15T23:15:42.612481+0000","last_clean":"2024-09-15T23:15:42.612481+0000","last_became_active":"2024-09-15T23:14:41.994429+0000","last_became_peered":"2024-09-15T23:14:41.994429+0000","last_unstale":"2024-09-15T23:15:42.612481+0000","last_undegraded":"2024-09-15T23:15:42.612481+0000","last_fullsized":"2024-09-15T23:15:42.612481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:24.056756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317374+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:15:43.317374+0000","last_peered":"2024-09-15T23:15:43.317374+0000","last_clean":"2024-09-15T23:15:43.317374+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:15:43.317374+0000","last_undegraded":"2024-09-15T23:15:43.317374+0000","last_fullsized":"2024-09-15T23:15:43.317374+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.627013+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:15:42.627013+0000","last_peered":"2024-09-15T23:15:42.627013+0000","last_clean":"2024-09-15T23:15:42.627013+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:15:42.627013+0000","last_undegraded":"2024-09-15T23:15:42.627013+0000","last_fullsized":"2024-09-15T23:15:42.627013+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627691+0000","last_change":"2024-09-15T23:14:41.640471+0000","last_active":"2024-09-15T23:15:43.627691+0000","last_peered":"2024-09-15T23:15:43.627691+0000","last_clean":"2024-09-15T23:15:43.627691+0000","last_became_active":"2024-09-15T23:14:41.640017+0000","last_became_peered":"2024-09-15T23:14:41.640017+0000","last_unstale":"2024-09-15T23:15:43.627691+0000","last_undegraded":"2024-09-15T23:15:43.627691+0000","last_fullsized":"2024-09-15T23:15:43.627691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:58.842636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317463+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:15:43.317463+0000","last_peered":"2024-09-15T23:15:43.317463+0000","last_clean":"2024-09-15T23:15:43.317463+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:15:43.317463+0000","last_undegraded":"2024-09-15T23:15:43.317463+0000","last_fullsized":"2024-09-15T23:15:43.317463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613353+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:15:42.613353+0000","last_peered":"2024-09-15T23:15:42.613353+0000","last_clean":"2024-09-15T23:15:42.613353+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:15:42.613353+0000","last_undegraded":"2024-09-15T23:15:42.613353+0000","last_fullsized":"2024-09-15T23:15:42.613353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626135+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:15:42.626135+0000","last_peered":"2024-09-15T23:15:42.626135+0000","last_clean":"2024-09-15T23:15:42.626135+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:15:42.626135+0000","last_undegraded":"2024-09-15T23:15:42.626135+0000","last_fullsized":"2024-09-15T23:15:42.626135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910014+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:15:42.910014+0000","last_peered":"2024-09-15T23:15:42.910014+0000","last_clean":"2024-09-15T23:15:42.910014+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:15:42.910014+0000","last_undegraded":"2024-09-15T23:15:42.910014+0000","last_fullsized":"2024-09-15T23:15:42.910014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610024+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:15:42.610024+0000","last_peered":"2024-09-15T23:15:42.610024+0000","last_clean":"2024-09-15T23:15:42.610024+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:15:42.610024+0000","last_undegraded":"2024-09-15T23:15:42.610024+0000","last_fullsized":"2024-09-15T23:15:42.610024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626004+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:15:42.626004+0000","last_peered":"2024-09-15T23:15:42.626004+0000","last_clean":"2024-09-15T23:15:42.626004+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:15:42.626004+0000","last_undegraded":"2024-09-15T23:15:42.626004+0000","last_fullsized":"2024-09-15T23:15:42.626004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626111+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:15:42.626111+0000","last_peered":"2024-09-15T23:15:42.626111+0000","last_clean":"2024-09-15T23:15:42.626111+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:15:42.626111+0000","last_undegraded":"2024-09-15T23:15:42.626111+0000","last_fullsized":"2024-09-15T23:15:42.626111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909569+0000","last_change":"2024-09-15T23:14:41.653786+0000","last_active":"2024-09-15T23:15:42.909569+0000","last_peered":"2024-09-15T23:15:42.909569+0000","last_clean":"2024-09-15T23:15:42.909569+0000","last_became_active":"2024-09-15T23:14:41.641472+0000","last_became_peered":"2024-09-15T23:14:41.641472+0000","last_unstale":"2024-09-15T23:15:42.909569+0000","last_undegraded":"2024-09-15T23:15:42.909569+0000","last_fullsized":"2024-09-15T23:15:42.909569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:16.010142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613267+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:15:42.613267+0000","last_peered":"2024-09-15T23:15:42.613267+0000","last_clean":"2024-09-15T23:15:42.613267+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:15:42.613267+0000","last_undegraded":"2024-09-15T23:15:42.613267+0000","last_fullsized":"2024-09-15T23:15:42.613267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316776+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:15:43.316776+0000","last_peered":"2024-09-15T23:15:43.316776+0000","last_clean":"2024-09-15T23:15:43.316776+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:15:43.316776+0000","last_undegraded":"2024-09-15T23:15:43.316776+0000","last_fullsized":"2024-09-15T23:15:43.316776+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":146,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:49.629154+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:15:49.629154+0000","last_peered":"2024-09-15T23:15:49.629154+0000","last_clean":"2024-09-15T23:15:49.629154+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:15:49.629154+0000","last_undegraded":"2024-09-15T23:15:49.629154+0000","last_fullsized":"2024-09-15T23:15:49.629154+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":105,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609376+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:15:42.609376+0000","last_peered":"2024-09-15T23:15:42.609376+0000","last_clean":"2024-09-15T23:15:42.609376+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:15:42.609376+0000","last_undegraded":"2024-09-15T23:15:42.609376+0000","last_fullsized":"2024-09-15T23:15:42.609376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":48,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317071+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:15:43.317071+0000","last_peered":"2024-09-15T23:15:43.317071+0000","last_clean":"2024-09-15T23:15:43.317071+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:15:43.317071+0000","last_undegraded":"2024-09-15T23:15:43.317071+0000","last_fullsized":"2024-09-15T23:15:43.317071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626312+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:15:42.626312+0000","last_peered":"2024-09-15T23:15:42.626312+0000","last_clean":"2024-09-15T23:15:42.626312+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:15:42.626312+0000","last_undegraded":"2024-09-15T23:15:42.626312+0000","last_fullsized":"2024-09-15T23:15:42.626312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909759+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:15:42.909759+0000","last_peered":"2024-09-15T23:15:42.909759+0000","last_clean":"2024-09-15T23:15:42.909759+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:15:42.909759+0000","last_undegraded":"2024-09-15T23:15:42.909759+0000","last_fullsized":"2024-09-15T23:15:42.909759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612005+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:15:42.612005+0000","last_peered":"2024-09-15T23:15:42.612005+0000","last_clean":"2024-09-15T23:15:42.612005+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:15:42.612005+0000","last_undegraded":"2024-09-15T23:15:42.612005+0000","last_fullsized":"2024-09-15T23:15:42.612005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613972+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:15:42.613972+0000","last_peered":"2024-09-15T23:15:42.613972+0000","last_clean":"2024-09-15T23:15:42.613972+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:15:42.613972+0000","last_undegraded":"2024-09-15T23:15:42.613972+0000","last_fullsized":"2024-09-15T23:15:42.613972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626638+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:15:42.626638+0000","last_peered":"2024-09-15T23:15:42.626638+0000","last_clean":"2024-09-15T23:15:42.626638+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:15:42.626638+0000","last_undegraded":"2024-09-15T23:15:42.626638+0000","last_fullsized":"2024-09-15T23:15:42.626638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609857+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:15:42.609857+0000","last_peered":"2024-09-15T23:15:42.609857+0000","last_clean":"2024-09-15T23:15:42.609857+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:15:42.609857+0000","last_undegraded":"2024-09-15T23:15:42.609857+0000","last_fullsized":"2024-09-15T23:15:42.609857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"92'37","reported_seq":294,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:50.638149+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:15:50.638149+0000","last_peered":"2024-09-15T23:15:50.638149+0000","last_clean":"2024-09-15T23:15:50.638149+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:15:50.638149+0000","last_undegraded":"2024-09-15T23:15:50.638149+0000","last_fullsized":"2024-09-15T23:15:50.638149+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68861,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":218,"num_read_kb":177,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609007+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:15:42.609007+0000","last_peered":"2024-09-15T23:15:42.609007+0000","last_clean":"2024-09-15T23:15:42.609007+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:15:42.609007+0000","last_undegraded":"2024-09-15T23:15:42.609007+0000","last_fullsized":"2024-09-15T23:15:42.609007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612977+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:15:42.612977+0000","last_peered":"2024-09-15T23:15:42.612977+0000","last_clean":"2024-09-15T23:15:42.612977+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:15:42.612977+0000","last_undegraded":"2024-09-15T23:15:42.612977+0000","last_fullsized":"2024-09-15T23:15:42.612977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626922+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:15:42.626922+0000","last_peered":"2024-09-15T23:15:42.626922+0000","last_clean":"2024-09-15T23:15:42.626922+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:15:42.626922+0000","last_undegraded":"2024-09-15T23:15:42.626922+0000","last_fullsized":"2024-09-15T23:15:42.626922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612007+0000","last_change":"2024-09-15T23:14:41.641820+0000","last_active":"2024-09-15T23:15:42.612007+0000","last_peered":"2024-09-15T23:15:42.612007+0000","last_clean":"2024-09-15T23:15:42.612007+0000","last_became_active":"2024-09-15T23:14:41.641282+0000","last_became_peered":"2024-09-15T23:14:41.641282+0000","last_unstale":"2024-09-15T23:15:42.612007+0000","last_undegraded":"2024-09-15T23:15:42.612007+0000","last_fullsized":"2024-09-15T23:15:42.612007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:43.472601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.628116+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:15:43.628116+0000","last_peered":"2024-09-15T23:15:43.628116+0000","last_clean":"2024-09-15T23:15:43.628116+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:15:43.628116+0000","last_undegraded":"2024-09-15T23:15:43.628116+0000","last_fullsized":"2024-09-15T23:15:43.628116+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317368+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:15:43.317368+0000","last_peered":"2024-09-15T23:15:43.317368+0000","last_clean":"2024-09-15T23:15:43.317368+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:15:43.317368+0000","last_undegraded":"2024-09-15T23:15:43.317368+0000","last_fullsized":"2024-09-15T23:15:43.317368+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608807+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:15:42.608807+0000","last_peered":"2024-09-15T23:15:42.608807+0000","last_clean":"2024-09-15T23:15:42.608807+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:15:42.608807+0000","last_undegraded":"2024-09-15T23:15:42.608807+0000","last_fullsized":"2024-09-15T23:15:42.608807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612577+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:15:42.612577+0000","last_peered":"2024-09-15T23:15:42.612577+0000","last_clean":"2024-09-15T23:15:42.612577+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:15:42.612577+0000","last_undegraded":"2024-09-15T23:15:42.612577+0000","last_fullsized":"2024-09-15T23:15:42.612577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":145,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:49.635646+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:15:49.635646+0000","last_peered":"2024-09-15T23:15:49.635646+0000","last_clean":"2024-09-15T23:15:49.635646+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:15:49.635646+0000","last_undegraded":"2024-09-15T23:15:49.635646+0000","last_fullsized":"2024-09-15T23:15:49.635646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316758+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:15:43.316758+0000","last_peered":"2024-09-15T23:15:43.316758+0000","last_clean":"2024-09-15T23:15:43.316758+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:15:43.316758+0000","last_undegraded":"2024-09-15T23:15:43.316758+0000","last_fullsized":"2024-09-15T23:15:43.316758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627914+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:15:43.627914+0000","last_peered":"2024-09-15T23:15:43.627914+0000","last_clean":"2024-09-15T23:15:43.627914+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:15:43.627914+0000","last_undegraded":"2024-09-15T23:15:43.627914+0000","last_fullsized":"2024-09-15T23:15:43.627914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608984+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:15:42.608984+0000","last_peered":"2024-09-15T23:15:42.608984+0000","last_clean":"2024-09-15T23:15:42.608984+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:15:42.608984+0000","last_undegraded":"2024-09-15T23:15:42.608984+0000","last_fullsized":"2024-09-15T23:15:42.608984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626347+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:15:42.626347+0000","last_peered":"2024-09-15T23:15:42.626347+0000","last_clean":"2024-09-15T23:15:42.626347+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:15:42.626347+0000","last_undegraded":"2024-09-15T23:15:42.626347+0000","last_fullsized":"2024-09-15T23:15:42.626347+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910018+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:15:42.910018+0000","last_peered":"2024-09-15T23:15:42.910018+0000","last_clean":"2024-09-15T23:15:42.910018+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:15:42.910018+0000","last_undegraded":"2024-09-15T23:15:42.910018+0000","last_fullsized":"2024-09-15T23:15:42.910018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316784+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:15:43.316784+0000","last_peered":"2024-09-15T23:15:43.316784+0000","last_clean":"2024-09-15T23:15:43.316784+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:15:43.316784+0000","last_undegraded":"2024-09-15T23:15:43.316784+0000","last_fullsized":"2024-09-15T23:15:43.316784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611692+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:15:42.611692+0000","last_peered":"2024-09-15T23:15:42.611692+0000","last_clean":"2024-09-15T23:15:42.611692+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:15:42.611692+0000","last_undegraded":"2024-09-15T23:15:42.611692+0000","last_fullsized":"2024-09-15T23:15:42.611692+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610033+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:15:42.610033+0000","last_peered":"2024-09-15T23:15:42.610033+0000","last_clean":"2024-09-15T23:15:42.610033+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:15:42.610033+0000","last_undegraded":"2024-09-15T23:15:42.610033+0000","last_fullsized":"2024-09-15T23:15:42.610033+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626302+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:15:42.626302+0000","last_peered":"2024-09-15T23:15:42.626302+0000","last_clean":"2024-09-15T23:15:42.626302+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:15:42.626302+0000","last_undegraded":"2024-09-15T23:15:42.626302+0000","last_fullsized":"2024-09-15T23:15:42.626302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613815+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:15:42.613815+0000","last_peered":"2024-09-15T23:15:42.613815+0000","last_clean":"2024-09-15T23:15:42.613815+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:15:42.613815+0000","last_undegraded":"2024-09-15T23:15:42.613815+0000","last_fullsized":"2024-09-15T23:15:42.613815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":152,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:49.636304+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:15:49.636304+0000","last_peered":"2024-09-15T23:15:49.636304+0000","last_clean":"2024-09-15T23:15:49.636304+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:15:49.636304+0000","last_undegraded":"2024-09-15T23:15:49.636304+0000","last_fullsized":"2024-09-15T23:15:49.636304+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1156,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609500+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:15:42.609500+0000","last_peered":"2024-09-15T23:15:42.609500+0000","last_clean":"2024-09-15T23:15:42.609500+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:15:42.609500+0000","last_undegraded":"2024-09-15T23:15:42.609500+0000","last_fullsized":"2024-09-15T23:15:42.609500+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613805+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:15:42.613805+0000","last_peered":"2024-09-15T23:15:42.613805+0000","last_clean":"2024-09-15T23:15:42.613805+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:15:42.613805+0000","last_undegraded":"2024-09-15T23:15:42.613805+0000","last_fullsized":"2024-09-15T23:15:42.613805+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909660+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:15:42.909660+0000","last_peered":"2024-09-15T23:15:42.909660+0000","last_clean":"2024-09-15T23:15:42.909660+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:15:42.909660+0000","last_undegraded":"2024-09-15T23:15:42.909660+0000","last_fullsized":"2024-09-15T23:15:42.909660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614459+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:42.614459+0000","last_peered":"2024-09-15T23:15:42.614459+0000","last_clean":"2024-09-15T23:15:42.614459+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:15:42.614459+0000","last_undegraded":"2024-09-15T23:15:42.614459+0000","last_fullsized":"2024-09-15T23:15:42.614459+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613647+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:15:42.613647+0000","last_peered":"2024-09-15T23:15:42.613647+0000","last_clean":"2024-09-15T23:15:42.613647+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:15:42.613647+0000","last_undegraded":"2024-09-15T23:15:42.613647+0000","last_fullsized":"2024-09-15T23:15:42.613647+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627750+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:15:43.627750+0000","last_peered":"2024-09-15T23:15:43.627750+0000","last_clean":"2024-09-15T23:15:43.627750+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:15:43.627750+0000","last_undegraded":"2024-09-15T23:15:43.627750+0000","last_fullsized":"2024-09-15T23:15:43.627750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612897+0000","last_change":"2024-09-15T23:14:42.010010+0000","last_active":"2024-09-15T23:15:42.612897+0000","last_peered":"2024-09-15T23:15:42.612897+0000","last_clean":"2024-09-15T23:15:42.612897+0000","last_became_active":"2024-09-15T23:14:42.009603+0000","last_became_peered":"2024-09-15T23:14:42.009603+0000","last_unstale":"2024-09-15T23:15:42.612897+0000","last_undegraded":"2024-09-15T23:15:42.612897+0000","last_fullsized":"2024-09-15T23:15:42.612897+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:34:51.849548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910096+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:15:42.910096+0000","last_peered":"2024-09-15T23:15:42.910096+0000","last_clean":"2024-09-15T23:15:42.910096+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:15:42.910096+0000","last_undegraded":"2024-09-15T23:15:42.910096+0000","last_fullsized":"2024-09-15T23:15:42.910096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614200+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:15:42.614200+0000","last_peered":"2024-09-15T23:15:42.614200+0000","last_clean":"2024-09-15T23:15:42.614200+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:15:42.614200+0000","last_undegraded":"2024-09-15T23:15:42.614200+0000","last_fullsized":"2024-09-15T23:15:42.614200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317395+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:15:43.317395+0000","last_peered":"2024-09-15T23:15:43.317395+0000","last_clean":"2024-09-15T23:15:43.317395+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:15:43.317395+0000","last_undegraded":"2024-09-15T23:15:43.317395+0000","last_fullsized":"2024-09-15T23:15:43.317395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626113+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:15:42.626113+0000","last_peered":"2024-09-15T23:15:42.626113+0000","last_clean":"2024-09-15T23:15:42.626113+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:15:42.626113+0000","last_undegraded":"2024-09-15T23:15:42.626113+0000","last_fullsized":"2024-09-15T23:15:42.626113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316925+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:15:43.316925+0000","last_peered":"2024-09-15T23:15:43.316925+0000","last_clean":"2024-09-15T23:15:43.316925+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:15:43.316925+0000","last_undegraded":"2024-09-15T23:15:43.316925+0000","last_fullsized":"2024-09-15T23:15:43.316925+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612732+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:15:42.612732+0000","last_peered":"2024-09-15T23:15:42.612732+0000","last_clean":"2024-09-15T23:15:42.612732+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:15:42.612732+0000","last_undegraded":"2024-09-15T23:15:42.612732+0000","last_fullsized":"2024-09-15T23:15:42.612732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612638+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:15:42.612638+0000","last_peered":"2024-09-15T23:15:42.612638+0000","last_clean":"2024-09-15T23:15:42.612638+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:15:42.612638+0000","last_undegraded":"2024-09-15T23:15:42.612638+0000","last_fullsized":"2024-09-15T23:15:42.612638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909941+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:15:42.909941+0000","last_peered":"2024-09-15T23:15:42.909941+0000","last_clean":"2024-09-15T23:15:42.909941+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:15:42.909941+0000","last_undegraded":"2024-09-15T23:15:42.909941+0000","last_fullsized":"2024-09-15T23:15:42.909941+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614098+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:15:42.614098+0000","last_peered":"2024-09-15T23:15:42.614098+0000","last_clean":"2024-09-15T23:15:42.614098+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:15:42.614098+0000","last_undegraded":"2024-09-15T23:15:42.614098+0000","last_fullsized":"2024-09-15T23:15:42.614098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610060+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:15:42.610060+0000","last_peered":"2024-09-15T23:15:42.610060+0000","last_clean":"2024-09-15T23:15:42.610060+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:15:42.610060+0000","last_undegraded":"2024-09-15T23:15:42.610060+0000","last_fullsized":"2024-09-15T23:15:42.610060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910207+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:15:42.910207+0000","last_peered":"2024-09-15T23:15:42.910207+0000","last_clean":"2024-09-15T23:15:42.910207+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:15:42.910207+0000","last_undegraded":"2024-09-15T23:15:42.910207+0000","last_fullsized":"2024-09-15T23:15:42.910207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612880+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:15:42.612880+0000","last_peered":"2024-09-15T23:15:42.612880+0000","last_clean":"2024-09-15T23:15:42.612880+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:15:42.612880+0000","last_undegraded":"2024-09-15T23:15:42.612880+0000","last_fullsized":"2024-09-15T23:15:42.612880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627665+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:15:43.627665+0000","last_peered":"2024-09-15T23:15:43.627665+0000","last_clean":"2024-09-15T23:15:43.627665+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:15:43.627665+0000","last_undegraded":"2024-09-15T23:15:43.627665+0000","last_fullsized":"2024-09-15T23:15:43.627665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627685+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:15:43.627685+0000","last_peered":"2024-09-15T23:15:43.627685+0000","last_clean":"2024-09-15T23:15:43.627685+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:15:43.627685+0000","last_undegraded":"2024-09-15T23:15:43.627685+0000","last_fullsized":"2024-09-15T23:15:43.627685+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612501+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:15:42.612501+0000","last_peered":"2024-09-15T23:15:42.612501+0000","last_clean":"2024-09-15T23:15:42.612501+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:15:42.612501+0000","last_undegraded":"2024-09-15T23:15:42.612501+0000","last_fullsized":"2024-09-15T23:15:42.612501+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612646+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:15:42.612646+0000","last_peered":"2024-09-15T23:15:42.612646+0000","last_clean":"2024-09-15T23:15:42.612646+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:15:42.612646+0000","last_undegraded":"2024-09-15T23:15:42.612646+0000","last_fullsized":"2024-09-15T23:15:42.612646+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317377+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:15:43.317377+0000","last_peered":"2024-09-15T23:15:43.317377+0000","last_clean":"2024-09-15T23:15:43.317377+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:15:43.317377+0000","last_undegraded":"2024-09-15T23:15:43.317377+0000","last_fullsized":"2024-09-15T23:15:43.317377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610726+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:15:42.610726+0000","last_peered":"2024-09-15T23:15:42.610726+0000","last_clean":"2024-09-15T23:15:42.610726+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:15:42.610726+0000","last_undegraded":"2024-09-15T23:15:42.610726+0000","last_fullsized":"2024-09-15T23:15:42.610726+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613153+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:15:42.613153+0000","last_peered":"2024-09-15T23:15:42.613153+0000","last_clean":"2024-09-15T23:15:42.613153+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:15:42.613153+0000","last_undegraded":"2024-09-15T23:15:42.613153+0000","last_fullsized":"2024-09-15T23:15:42.613153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316763+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:15:43.316763+0000","last_peered":"2024-09-15T23:15:43.316763+0000","last_clean":"2024-09-15T23:15:43.316763+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:15:43.316763+0000","last_undegraded":"2024-09-15T23:15:43.316763+0000","last_fullsized":"2024-09-15T23:15:43.316763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316851+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:15:43.316851+0000","last_peered":"2024-09-15T23:15:43.316851+0000","last_clean":"2024-09-15T23:15:43.316851+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:15:43.316851+0000","last_undegraded":"2024-09-15T23:15:43.316851+0000","last_fullsized":"2024-09-15T23:15:43.316851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610204+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:15:42.610204+0000","last_peered":"2024-09-15T23:15:42.610204+0000","last_clean":"2024-09-15T23:15:42.610204+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:15:42.610204+0000","last_undegraded":"2024-09-15T23:15:42.610204+0000","last_fullsized":"2024-09-15T23:15:42.610204+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609588+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:15:42.609588+0000","last_peered":"2024-09-15T23:15:42.609588+0000","last_clean":"2024-09-15T23:15:42.609588+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:15:42.609588+0000","last_undegraded":"2024-09-15T23:15:42.609588+0000","last_fullsized":"2024-09-15T23:15:42.609588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612535+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:15:42.612535+0000","last_peered":"2024-09-15T23:15:42.612535+0000","last_clean":"2024-09-15T23:15:42.612535+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:15:42.612535+0000","last_undegraded":"2024-09-15T23:15:42.612535+0000","last_fullsized":"2024-09-15T23:15:42.612535+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627888+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:15:43.627888+0000","last_peered":"2024-09-15T23:15:43.627888+0000","last_clean":"2024-09-15T23:15:43.627888+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:15:43.627888+0000","last_undegraded":"2024-09-15T23:15:43.627888+0000","last_fullsized":"2024-09-15T23:15:43.627888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627940+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:15:43.627940+0000","last_peered":"2024-09-15T23:15:43.627940+0000","last_clean":"2024-09-15T23:15:43.627940+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:15:43.627940+0000","last_undegraded":"2024-09-15T23:15:43.627940+0000","last_fullsized":"2024-09-15T23:15:43.627940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910172+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:15:42.910172+0000","last_peered":"2024-09-15T23:15:42.910172+0000","last_clean":"2024-09-15T23:15:42.910172+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:15:42.910172+0000","last_undegraded":"2024-09-15T23:15:42.910172+0000","last_fullsized":"2024-09-15T23:15:42.910172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73005,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":647,"num_read_kb":401,"num_write":179,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":220153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514538,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522238,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743470,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712832,"statfs":{"total":95995035648,"available":95961939968,"internally_reserved":0,"allocated":1163264,"data_stored":1178255,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906996,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1032192,"data_stored":526458,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135929,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522170,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364861,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1077248,"data_stored":1118334,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528387,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":517538,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790024,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1085440,"data_stored":1113922,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953549,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1019904,"data_stored":522188,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":517732,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":589331,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":586309,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1032192,"data_stored":518926,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":68861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:15:55.369 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:15:56.001 INFO:tasks.cephfs.filesystem:scrub status for tag:283e8d57-36b1-485d-9f31-7d1835989b67 - {'path': '/', 'tag': '283e8d57-36b1-485d-9f31-7d1835989b67', 'options': 'recursive,force'} 2024-09-15T23:15:56.037 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:56.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.506+0000 7fd5a486d640 1 -- 172.21.15.62:0/4223865415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0103660 msgr2=0x7fd5a010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:56.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.506+0000 7fd5a486d640 1 --2- 172.21.15.62:0/4223865415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0103660 0x7fd5a010bb50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd588009900 tx=0x7fd58802a480 comp rx=0 tx=0).stop 2024-09-15T23:15:56.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.506+0000 7fd5a486d640 1 -- 172.21.15.62:0/4223865415 shutdown_connections 2024-09-15T23:15:56.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.506+0000 7fd5a486d640 1 --2- 172.21.15.62:0/4223865415 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5a010c170 0x7fd5a010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.506+0000 7fd5a486d640 1 --2- 172.21.15.62:0/4223865415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0103660 0x7fd5a010bb50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.506+0000 7fd5a486d640 1 --2- 172.21.15.62:0/4223865415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5a0102d40 0x7fd5a0103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.506+0000 7fd5a486d640 1 -- 172.21.15.62:0/4223865415 >> 172.21.15.62:0/4223865415 conn(0x7fd5a00fc7a0 msgr2=0x7fd5a00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:56.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.506+0000 7fd5a486d640 1 -- 172.21.15.62:0/4223865415 shutdown_connections 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.506+0000 7fd5a486d640 1 -- 172.21.15.62:0/4223865415 wait complete. 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd5a486d640 1 Processor -- start 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd5a486d640 1 -- start start 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd5a486d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0102d40 0x7fd5a019a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd5a486d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5a0103660 0x7fd5a019ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd5a486d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5a010c170 0x7fd5a019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd5a486d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a0111f70 con 0x7fd5a0102d40 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd5a486d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a0111df0 con 0x7fd5a0103660 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd5a486d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a01120f0 con 0x7fd5a010c170 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd59e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0102d40 0x7fd5a019a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd59e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0102d40 0x7fd5a019a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42706/0 (socket says 172.21.15.62:42706) 2024-09-15T23:15:56.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd59e59c640 1 -- 172.21.15.62:0/3216660333 learned_addr learned my addr 172.21.15.62:0/3216660333 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd597fff640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5a0103660 0x7fd5a019ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.507+0000 7fd59ed9d640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5a010c170 0x7fd5a019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd59e59c640 1 -- 172.21.15.62:0/3216660333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5a010c170 msgr2=0x7fd5a019ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd59e59c640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5a010c170 0x7fd5a019ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd59e59c640 1 -- 172.21.15.62:0/3216660333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5a0103660 msgr2=0x7fd5a019ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd59e59c640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5a0103660 0x7fd5a019ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd59e59c640 1 -- 172.21.15.62:0/3216660333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a019f600 con 0x7fd5a0102d40 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd59e59c640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0102d40 0x7fd5a019a650 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd59000d950 tx=0x7fd59000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd5977fe640 1 -- 172.21.15.62:0/3216660333 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590017070 con 0x7fd5a0102d40 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd5977fe640 1 -- 172.21.15.62:0/3216660333 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd590009c30 con 0x7fd5a0102d40 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a019f8f0 con 0x7fd5a0102d40 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd5977fe640 1 -- 172.21.15.62:0/3216660333 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59000f6b0 con 0x7fd5a0102d40 2024-09-15T23:15:56.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.508+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a01a71d0 con 0x7fd5a0102d40 2024-09-15T23:15:56.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.509+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a0103120 con 0x7fd5a0102d40 2024-09-15T23:15:56.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.510+0000 7fd5977fe640 1 -- 172.21.15.62:0/3216660333 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd5900109d0 con 0x7fd5a0102d40 2024-09-15T23:15:56.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.511+0000 7fd5977fe640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd58c0a3240 0x7fd58c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:56.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.512+0000 7fd5977fe640 1 -- 172.21.15.62:0/3216660333 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd5900cf830 con 0x7fd5a0102d40 2024-09-15T23:15:56.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.512+0000 7fd597fff640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd58c0a3240 0x7fd58c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:56.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.512+0000 7fd597fff640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd58c0a3240 0x7fd58c0a5700 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd588009230 tx=0x7fd5880058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:56.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.516+0000 7fd5977fe640 1 -- 172.21.15.62:0/3216660333 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd590009de0 con 0x7fd5a0102d40 2024-09-15T23:15:56.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.689+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd5a00fe2a0 con 0x7fd58c0a3240 2024-09-15T23:15:56.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.697+0000 7fd5977fe640 1 -- 172.21.15.62:0/3216660333 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340739 (secure 0 0 0) 0x7fd5a00fe2a0 con 0x7fd58c0a3240 2024-09-15T23:15:56.701 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:56.703 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:15:56.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.700+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd58c0a3240 msgr2=0x7fd58c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:56.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.700+0000 7fd5a486d640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd58c0a3240 0x7fd58c0a5700 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd588009230 tx=0x7fd5880058d0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.700+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0102d40 msgr2=0x7fd5a019a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:56.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.700+0000 7fd5a486d640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0102d40 0x7fd5a019a650 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd59000d950 tx=0x7fd59000de10 comp rx=0 tx=0).stop 2024-09-15T23:15:56.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.701+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 shutdown_connections 2024-09-15T23:15:56.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.701+0000 7fd5a486d640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd58c0a3240 0x7fd58c0a5700 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.701+0000 7fd5a486d640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5a010c170 0x7fd5a019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.701+0000 7fd5a486d640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5a0103660 0x7fd5a019ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.701+0000 7fd5a486d640 1 --2- 172.21.15.62:0/3216660333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5a0102d40 0x7fd5a019a650 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:56.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.701+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 >> 172.21.15.62:0/3216660333 conn(0x7fd5a00fc7a0 msgr2=0x7fd5a00fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:56.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.701+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 shutdown_connections 2024-09-15T23:15:56.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:56.701+0000 7fd5a486d640 1 -- 172.21.15.62:0/3216660333 wait complete. 2024-09-15T23:15:56.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:56 smithi062 ceph-mon[30949]: pgmap v360: 129 pgs: 129 active+clean; 648 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 682 B/s wr, 6 op/s 2024-09-15T23:15:57.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:56 smithi148 ceph-mon[35610]: pgmap v360: 129 pgs: 129 active+clean; 648 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 682 B/s wr, 6 op/s 2024-09-15T23:15:57.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:56 smithi067 ceph-mon[35665]: pgmap v360: 129 pgs: 129 active+clean; 648 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 682 B/s wr, 6 op/s 2024-09-15T23:15:57.468 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-15T23:15:55.727756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663607,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":783,"num_read_kb":515,"num_write":362,"num_write_kb":3069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369996,"kb_used_data":12368,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571828,"statfs":{"total":1151940427776,"available":1151561551872,"internally_reserved":0,"allocated":12664832,"data_stored":8234539,"data_compressed":461688,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5681,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":69,"num_read_kb":40,"num_write":4,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003145"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609573+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:15:42.609573+0000","last_peered":"2024-09-15T23:15:42.609573+0000","last_clean":"2024-09-15T23:15:42.609573+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:15:42.609573+0000","last_undegraded":"2024-09-15T23:15:42.609573+0000","last_fullsized":"2024-09-15T23:15:42.609573+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317224+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:15:43.317224+0000","last_peered":"2024-09-15T23:15:43.317224+0000","last_clean":"2024-09-15T23:15:43.317224+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:15:43.317224+0000","last_undegraded":"2024-09-15T23:15:43.317224+0000","last_fullsized":"2024-09-15T23:15:43.317224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613665+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:15:42.613665+0000","last_peered":"2024-09-15T23:15:42.613665+0000","last_clean":"2024-09-15T23:15:42.613665+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:15:42.613665+0000","last_undegraded":"2024-09-15T23:15:42.613665+0000","last_fullsized":"2024-09-15T23:15:42.613665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317397+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:15:43.317397+0000","last_peered":"2024-09-15T23:15:43.317397+0000","last_clean":"2024-09-15T23:15:43.317397+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:15:43.317397+0000","last_undegraded":"2024-09-15T23:15:43.317397+0000","last_fullsized":"2024-09-15T23:15:43.317397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.628111+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:15:43.628111+0000","last_peered":"2024-09-15T23:15:43.628111+0000","last_clean":"2024-09-15T23:15:43.628111+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:15:43.628111+0000","last_undegraded":"2024-09-15T23:15:43.628111+0000","last_fullsized":"2024-09-15T23:15:43.628111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611378+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:15:42.611378+0000","last_peered":"2024-09-15T23:15:42.611378+0000","last_clean":"2024-09-15T23:15:42.611378+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:15:42.611378+0000","last_undegraded":"2024-09-15T23:15:42.611378+0000","last_fullsized":"2024-09-15T23:15:42.611378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909808+0000","last_change":"2024-09-15T23:14:41.659816+0000","last_active":"2024-09-15T23:15:42.909808+0000","last_peered":"2024-09-15T23:15:42.909808+0000","last_clean":"2024-09-15T23:15:42.909808+0000","last_became_active":"2024-09-15T23:14:41.641611+0000","last_became_peered":"2024-09-15T23:14:41.641611+0000","last_unstale":"2024-09-15T23:15:42.909808+0000","last_undegraded":"2024-09-15T23:15:42.909808+0000","last_fullsized":"2024-09-15T23:15:42.909808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:23.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612963+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:15:42.612963+0000","last_peered":"2024-09-15T23:15:42.612963+0000","last_clean":"2024-09-15T23:15:42.612963+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:15:42.612963+0000","last_undegraded":"2024-09-15T23:15:42.612963+0000","last_fullsized":"2024-09-15T23:15:42.612963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610025+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:15:42.610025+0000","last_peered":"2024-09-15T23:15:42.610025+0000","last_clean":"2024-09-15T23:15:42.610025+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:15:42.610025+0000","last_undegraded":"2024-09-15T23:15:42.610025+0000","last_fullsized":"2024-09-15T23:15:42.610025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614867+0000","last_change":"2024-09-15T23:15:42.614867+0000","last_active":"2024-09-15T23:15:42.614867+0000","last_peered":"2024-09-15T23:15:42.614867+0000","last_clean":"2024-09-15T23:15:42.614867+0000","last_became_active":"2024-09-15T23:15:42.614459+0000","last_became_peered":"2024-09-15T23:15:42.614459+0000","last_unstale":"2024-09-15T23:15:42.614867+0000","last_undegraded":"2024-09-15T23:15:42.614867+0000","last_fullsized":"2024-09-15T23:15:42.614867+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:15.254442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611583+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:15:42.611583+0000","last_peered":"2024-09-15T23:15:42.611583+0000","last_clean":"2024-09-15T23:15:42.611583+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:15:42.611583+0000","last_undegraded":"2024-09-15T23:15:42.611583+0000","last_fullsized":"2024-09-15T23:15:42.611583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317095+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:15:43.317095+0000","last_peered":"2024-09-15T23:15:43.317095+0000","last_clean":"2024-09-15T23:15:43.317095+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:15:43.317095+0000","last_undegraded":"2024-09-15T23:15:43.317095+0000","last_fullsized":"2024-09-15T23:15:43.317095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":56,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609750+0000","last_change":"2024-09-15T23:14:41.646320+0000","last_active":"2024-09-15T23:15:42.609750+0000","last_peered":"2024-09-15T23:15:42.609750+0000","last_clean":"2024-09-15T23:15:42.609750+0000","last_became_active":"2024-09-15T23:14:41.641933+0000","last_became_peered":"2024-09-15T23:14:41.641933+0000","last_unstale":"2024-09-15T23:15:42.609750+0000","last_undegraded":"2024-09-15T23:15:42.609750+0000","last_fullsized":"2024-09-15T23:15:42.609750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:49.417407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611971+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:15:42.611971+0000","last_peered":"2024-09-15T23:15:42.611971+0000","last_clean":"2024-09-15T23:15:42.611971+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:15:42.611971+0000","last_undegraded":"2024-09-15T23:15:42.611971+0000","last_fullsized":"2024-09-15T23:15:42.611971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316939+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:15:43.316939+0000","last_peered":"2024-09-15T23:15:43.316939+0000","last_clean":"2024-09-15T23:15:43.316939+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:15:43.316939+0000","last_undegraded":"2024-09-15T23:15:43.316939+0000","last_fullsized":"2024-09-15T23:15:43.316939+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613807+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:15:42.613807+0000","last_peered":"2024-09-15T23:15:42.613807+0000","last_clean":"2024-09-15T23:15:42.613807+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:15:42.613807+0000","last_undegraded":"2024-09-15T23:15:42.613807+0000","last_fullsized":"2024-09-15T23:15:42.613807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612468+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:15:42.612468+0000","last_peered":"2024-09-15T23:15:42.612468+0000","last_clean":"2024-09-15T23:15:42.612468+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:15:42.612468+0000","last_undegraded":"2024-09-15T23:15:42.612468+0000","last_fullsized":"2024-09-15T23:15:42.612468+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909877+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:15:42.909877+0000","last_peered":"2024-09-15T23:15:42.909877+0000","last_clean":"2024-09-15T23:15:42.909877+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:15:42.909877+0000","last_undegraded":"2024-09-15T23:15:42.909877+0000","last_fullsized":"2024-09-15T23:15:42.909877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316960+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:15:43.316960+0000","last_peered":"2024-09-15T23:15:43.316960+0000","last_clean":"2024-09-15T23:15:43.316960+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:15:43.316960+0000","last_undegraded":"2024-09-15T23:15:43.316960+0000","last_fullsized":"2024-09-15T23:15:43.316960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609295+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:15:42.609295+0000","last_peered":"2024-09-15T23:15:42.609295+0000","last_clean":"2024-09-15T23:15:42.609295+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:15:42.609295+0000","last_undegraded":"2024-09-15T23:15:42.609295+0000","last_fullsized":"2024-09-15T23:15:42.609295+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609453+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:15:42.609453+0000","last_peered":"2024-09-15T23:15:42.609453+0000","last_clean":"2024-09-15T23:15:42.609453+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:15:42.609453+0000","last_undegraded":"2024-09-15T23:15:42.609453+0000","last_fullsized":"2024-09-15T23:15:42.609453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613324+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:15:42.613324+0000","last_peered":"2024-09-15T23:15:42.613324+0000","last_clean":"2024-09-15T23:15:42.613324+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:15:42.613324+0000","last_undegraded":"2024-09-15T23:15:42.613324+0000","last_fullsized":"2024-09-15T23:15:42.613324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612082+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:15:42.612082+0000","last_peered":"2024-09-15T23:15:42.612082+0000","last_clean":"2024-09-15T23:15:42.612082+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:15:42.612082+0000","last_undegraded":"2024-09-15T23:15:42.612082+0000","last_fullsized":"2024-09-15T23:15:42.612082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":47,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.628020+0000","last_change":"2024-09-15T23:14:42.009605+0000","last_active":"2024-09-15T23:15:43.628020+0000","last_peered":"2024-09-15T23:15:43.628020+0000","last_clean":"2024-09-15T23:15:43.628020+0000","last_became_active":"2024-09-15T23:14:42.009237+0000","last_became_peered":"2024-09-15T23:14:42.009237+0000","last_unstale":"2024-09-15T23:15:43.628020+0000","last_undegraded":"2024-09-15T23:15:43.628020+0000","last_fullsized":"2024-09-15T23:15:43.628020+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:11:08.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612739+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:15:42.612739+0000","last_peered":"2024-09-15T23:15:42.612739+0000","last_clean":"2024-09-15T23:15:42.612739+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:15:42.612739+0000","last_undegraded":"2024-09-15T23:15:42.612739+0000","last_fullsized":"2024-09-15T23:15:42.612739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614193+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:15:42.614193+0000","last_peered":"2024-09-15T23:15:42.614193+0000","last_clean":"2024-09-15T23:15:42.614193+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:15:42.614193+0000","last_undegraded":"2024-09-15T23:15:42.614193+0000","last_fullsized":"2024-09-15T23:15:42.614193+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317410+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:15:43.317410+0000","last_peered":"2024-09-15T23:15:43.317410+0000","last_clean":"2024-09-15T23:15:43.317410+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:15:43.317410+0000","last_undegraded":"2024-09-15T23:15:43.317410+0000","last_fullsized":"2024-09-15T23:15:43.317410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'14","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610065+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:15:42.610065+0000","last_peered":"2024-09-15T23:15:42.610065+0000","last_clean":"2024-09-15T23:15:42.610065+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:15:42.610065+0000","last_undegraded":"2024-09-15T23:15:42.610065+0000","last_fullsized":"2024-09-15T23:15:42.610065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316638+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:15:43.316638+0000","last_peered":"2024-09-15T23:15:43.316638+0000","last_clean":"2024-09-15T23:15:43.316638+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:15:43.316638+0000","last_undegraded":"2024-09-15T23:15:43.316638+0000","last_fullsized":"2024-09-15T23:15:43.316638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612814+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:15:42.612814+0000","last_peered":"2024-09-15T23:15:42.612814+0000","last_clean":"2024-09-15T23:15:42.612814+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:15:42.612814+0000","last_undegraded":"2024-09-15T23:15:42.612814+0000","last_fullsized":"2024-09-15T23:15:42.612814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610084+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:15:42.610084+0000","last_peered":"2024-09-15T23:15:42.610084+0000","last_clean":"2024-09-15T23:15:42.610084+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:15:42.610084+0000","last_undegraded":"2024-09-15T23:15:42.610084+0000","last_fullsized":"2024-09-15T23:15:42.610084+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909814+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:15:42.909814+0000","last_peered":"2024-09-15T23:15:42.909814+0000","last_clean":"2024-09-15T23:15:42.909814+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:15:42.909814+0000","last_undegraded":"2024-09-15T23:15:42.909814+0000","last_fullsized":"2024-09-15T23:15:42.909814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610218+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:15:42.610218+0000","last_peered":"2024-09-15T23:15:42.610218+0000","last_clean":"2024-09-15T23:15:42.610218+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:15:42.610218+0000","last_undegraded":"2024-09-15T23:15:42.610218+0000","last_fullsized":"2024-09-15T23:15:42.610218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609669+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:15:42.609669+0000","last_peered":"2024-09-15T23:15:42.609669+0000","last_clean":"2024-09-15T23:15:42.609669+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:15:42.609669+0000","last_undegraded":"2024-09-15T23:15:42.609669+0000","last_fullsized":"2024-09-15T23:15:42.609669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609473+0000","last_change":"2024-09-15T23:14:41.995466+0000","last_active":"2024-09-15T23:15:42.609473+0000","last_peered":"2024-09-15T23:15:42.609473+0000","last_clean":"2024-09-15T23:15:42.609473+0000","last_became_active":"2024-09-15T23:14:41.994915+0000","last_became_peered":"2024-09-15T23:14:41.994915+0000","last_unstale":"2024-09-15T23:15:42.609473+0000","last_undegraded":"2024-09-15T23:15:42.609473+0000","last_fullsized":"2024-09-15T23:15:42.609473+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:04.121596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610393+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:15:42.610393+0000","last_peered":"2024-09-15T23:15:42.610393+0000","last_clean":"2024-09-15T23:15:42.610393+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:15:42.610393+0000","last_undegraded":"2024-09-15T23:15:42.610393+0000","last_fullsized":"2024-09-15T23:15:42.610393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317131+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:15:43.317131+0000","last_peered":"2024-09-15T23:15:43.317131+0000","last_clean":"2024-09-15T23:15:43.317131+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:15:43.317131+0000","last_undegraded":"2024-09-15T23:15:43.317131+0000","last_fullsized":"2024-09-15T23:15:43.317131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626237+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:15:42.626237+0000","last_peered":"2024-09-15T23:15:42.626237+0000","last_clean":"2024-09-15T23:15:42.626237+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:15:42.626237+0000","last_undegraded":"2024-09-15T23:15:42.626237+0000","last_fullsized":"2024-09-15T23:15:42.626237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":150,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:53.634948+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:15:53.634948+0000","last_peered":"2024-09-15T23:15:53.634948+0000","last_clean":"2024-09-15T23:15:53.634948+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:15:53.634948+0000","last_undegraded":"2024-09-15T23:15:53.634948+0000","last_fullsized":"2024-09-15T23:15:53.634948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613035+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:15:42.613035+0000","last_peered":"2024-09-15T23:15:42.613035+0000","last_clean":"2024-09-15T23:15:42.613035+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:15:42.613035+0000","last_undegraded":"2024-09-15T23:15:42.613035+0000","last_fullsized":"2024-09-15T23:15:42.613035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.617748+0000","last_change":"2024-09-15T23:15:42.617748+0000","last_active":"2024-09-15T23:15:42.617748+0000","last_peered":"2024-09-15T23:15:42.617748+0000","last_clean":"2024-09-15T23:15:42.617748+0000","last_became_active":"2024-09-15T23:15:42.617320+0000","last_became_peered":"2024-09-15T23:15:42.617320+0000","last_unstale":"2024-09-15T23:15:42.617748+0000","last_undegraded":"2024-09-15T23:15:42.617748+0000","last_fullsized":"2024-09-15T23:15:42.617748+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:47.900049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316911+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:15:43.316911+0000","last_peered":"2024-09-15T23:15:43.316911+0000","last_clean":"2024-09-15T23:15:43.316911+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:15:43.316911+0000","last_undegraded":"2024-09-15T23:15:43.316911+0000","last_fullsized":"2024-09-15T23:15:43.316911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626588+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:15:42.626588+0000","last_peered":"2024-09-15T23:15:42.626588+0000","last_clean":"2024-09-15T23:15:42.626588+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:15:42.626588+0000","last_undegraded":"2024-09-15T23:15:42.626588+0000","last_fullsized":"2024-09-15T23:15:42.626588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316962+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:43.316962+0000","last_peered":"2024-09-15T23:15:43.316962+0000","last_clean":"2024-09-15T23:15:43.316962+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:15:43.316962+0000","last_undegraded":"2024-09-15T23:15:43.316962+0000","last_fullsized":"2024-09-15T23:15:43.316962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612474+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:15:42.612474+0000","last_peered":"2024-09-15T23:15:42.612474+0000","last_clean":"2024-09-15T23:15:42.612474+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:15:42.612474+0000","last_undegraded":"2024-09-15T23:15:42.612474+0000","last_fullsized":"2024-09-15T23:15:42.612474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317208+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:15:43.317208+0000","last_peered":"2024-09-15T23:15:43.317208+0000","last_clean":"2024-09-15T23:15:43.317208+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:15:43.317208+0000","last_undegraded":"2024-09-15T23:15:43.317208+0000","last_fullsized":"2024-09-15T23:15:43.317208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317366+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:15:43.317366+0000","last_peered":"2024-09-15T23:15:43.317366+0000","last_clean":"2024-09-15T23:15:43.317366+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:15:43.317366+0000","last_undegraded":"2024-09-15T23:15:43.317366+0000","last_fullsized":"2024-09-15T23:15:43.317366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317493+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:15:43.317493+0000","last_peered":"2024-09-15T23:15:43.317493+0000","last_clean":"2024-09-15T23:15:43.317493+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:15:43.317493+0000","last_undegraded":"2024-09-15T23:15:43.317493+0000","last_fullsized":"2024-09-15T23:15:43.317493+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614232+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:15:42.614232+0000","last_peered":"2024-09-15T23:15:42.614232+0000","last_clean":"2024-09-15T23:15:42.614232+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:15:42.614232+0000","last_undegraded":"2024-09-15T23:15:42.614232+0000","last_fullsized":"2024-09-15T23:15:42.614232+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612481+0000","last_change":"2024-09-15T23:14:41.994950+0000","last_active":"2024-09-15T23:15:42.612481+0000","last_peered":"2024-09-15T23:15:42.612481+0000","last_clean":"2024-09-15T23:15:42.612481+0000","last_became_active":"2024-09-15T23:14:41.994429+0000","last_became_peered":"2024-09-15T23:14:41.994429+0000","last_unstale":"2024-09-15T23:15:42.612481+0000","last_undegraded":"2024-09-15T23:15:42.612481+0000","last_fullsized":"2024-09-15T23:15:42.612481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:24.056756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317374+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:15:43.317374+0000","last_peered":"2024-09-15T23:15:43.317374+0000","last_clean":"2024-09-15T23:15:43.317374+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:15:43.317374+0000","last_undegraded":"2024-09-15T23:15:43.317374+0000","last_fullsized":"2024-09-15T23:15:43.317374+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.627013+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:15:42.627013+0000","last_peered":"2024-09-15T23:15:42.627013+0000","last_clean":"2024-09-15T23:15:42.627013+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:15:42.627013+0000","last_undegraded":"2024-09-15T23:15:42.627013+0000","last_fullsized":"2024-09-15T23:15:42.627013+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627691+0000","last_change":"2024-09-15T23:14:41.640471+0000","last_active":"2024-09-15T23:15:43.627691+0000","last_peered":"2024-09-15T23:15:43.627691+0000","last_clean":"2024-09-15T23:15:43.627691+0000","last_became_active":"2024-09-15T23:14:41.640017+0000","last_became_peered":"2024-09-15T23:14:41.640017+0000","last_unstale":"2024-09-15T23:15:43.627691+0000","last_undegraded":"2024-09-15T23:15:43.627691+0000","last_fullsized":"2024-09-15T23:15:43.627691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:58.842636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317463+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:15:43.317463+0000","last_peered":"2024-09-15T23:15:43.317463+0000","last_clean":"2024-09-15T23:15:43.317463+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:15:43.317463+0000","last_undegraded":"2024-09-15T23:15:43.317463+0000","last_fullsized":"2024-09-15T23:15:43.317463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613353+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:15:42.613353+0000","last_peered":"2024-09-15T23:15:42.613353+0000","last_clean":"2024-09-15T23:15:42.613353+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:15:42.613353+0000","last_undegraded":"2024-09-15T23:15:42.613353+0000","last_fullsized":"2024-09-15T23:15:42.613353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626135+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:15:42.626135+0000","last_peered":"2024-09-15T23:15:42.626135+0000","last_clean":"2024-09-15T23:15:42.626135+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:15:42.626135+0000","last_undegraded":"2024-09-15T23:15:42.626135+0000","last_fullsized":"2024-09-15T23:15:42.626135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910014+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:15:42.910014+0000","last_peered":"2024-09-15T23:15:42.910014+0000","last_clean":"2024-09-15T23:15:42.910014+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:15:42.910014+0000","last_undegraded":"2024-09-15T23:15:42.910014+0000","last_fullsized":"2024-09-15T23:15:42.910014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610024+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:15:42.610024+0000","last_peered":"2024-09-15T23:15:42.610024+0000","last_clean":"2024-09-15T23:15:42.610024+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:15:42.610024+0000","last_undegraded":"2024-09-15T23:15:42.610024+0000","last_fullsized":"2024-09-15T23:15:42.610024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626004+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:15:42.626004+0000","last_peered":"2024-09-15T23:15:42.626004+0000","last_clean":"2024-09-15T23:15:42.626004+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:15:42.626004+0000","last_undegraded":"2024-09-15T23:15:42.626004+0000","last_fullsized":"2024-09-15T23:15:42.626004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626111+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:15:42.626111+0000","last_peered":"2024-09-15T23:15:42.626111+0000","last_clean":"2024-09-15T23:15:42.626111+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:15:42.626111+0000","last_undegraded":"2024-09-15T23:15:42.626111+0000","last_fullsized":"2024-09-15T23:15:42.626111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909569+0000","last_change":"2024-09-15T23:14:41.653786+0000","last_active":"2024-09-15T23:15:42.909569+0000","last_peered":"2024-09-15T23:15:42.909569+0000","last_clean":"2024-09-15T23:15:42.909569+0000","last_became_active":"2024-09-15T23:14:41.641472+0000","last_became_peered":"2024-09-15T23:14:41.641472+0000","last_unstale":"2024-09-15T23:15:42.909569+0000","last_undegraded":"2024-09-15T23:15:42.909569+0000","last_fullsized":"2024-09-15T23:15:42.909569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:16.010142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613267+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:15:42.613267+0000","last_peered":"2024-09-15T23:15:42.613267+0000","last_clean":"2024-09-15T23:15:42.613267+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:15:42.613267+0000","last_undegraded":"2024-09-15T23:15:42.613267+0000","last_fullsized":"2024-09-15T23:15:42.613267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316843+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:15:43.316843+0000","last_peered":"2024-09-15T23:15:43.316843+0000","last_clean":"2024-09-15T23:15:43.316843+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:15:43.316843+0000","last_undegraded":"2024-09-15T23:15:43.316843+0000","last_fullsized":"2024-09-15T23:15:43.316843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":151,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:53.635521+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:15:53.635521+0000","last_peered":"2024-09-15T23:15:53.635521+0000","last_clean":"2024-09-15T23:15:53.635521+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:15:53.635521+0000","last_undegraded":"2024-09-15T23:15:53.635521+0000","last_fullsized":"2024-09-15T23:15:53.635521+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609376+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:15:42.609376+0000","last_peered":"2024-09-15T23:15:42.609376+0000","last_clean":"2024-09-15T23:15:42.609376+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:15:42.609376+0000","last_undegraded":"2024-09-15T23:15:42.609376+0000","last_fullsized":"2024-09-15T23:15:42.609376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317127+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:15:43.317127+0000","last_peered":"2024-09-15T23:15:43.317127+0000","last_clean":"2024-09-15T23:15:43.317127+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:15:43.317127+0000","last_undegraded":"2024-09-15T23:15:43.317127+0000","last_fullsized":"2024-09-15T23:15:43.317127+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626312+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:15:42.626312+0000","last_peered":"2024-09-15T23:15:42.626312+0000","last_clean":"2024-09-15T23:15:42.626312+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:15:42.626312+0000","last_undegraded":"2024-09-15T23:15:42.626312+0000","last_fullsized":"2024-09-15T23:15:42.626312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909759+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:15:42.909759+0000","last_peered":"2024-09-15T23:15:42.909759+0000","last_clean":"2024-09-15T23:15:42.909759+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:15:42.909759+0000","last_undegraded":"2024-09-15T23:15:42.909759+0000","last_fullsized":"2024-09-15T23:15:42.909759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612055+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:15:42.612055+0000","last_peered":"2024-09-15T23:15:42.612055+0000","last_clean":"2024-09-15T23:15:42.612055+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:15:42.612055+0000","last_undegraded":"2024-09-15T23:15:42.612055+0000","last_fullsized":"2024-09-15T23:15:42.612055+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613972+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:15:42.613972+0000","last_peered":"2024-09-15T23:15:42.613972+0000","last_clean":"2024-09-15T23:15:42.613972+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:15:42.613972+0000","last_undegraded":"2024-09-15T23:15:42.613972+0000","last_fullsized":"2024-09-15T23:15:42.613972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626638+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:15:42.626638+0000","last_peered":"2024-09-15T23:15:42.626638+0000","last_clean":"2024-09-15T23:15:42.626638+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:15:42.626638+0000","last_undegraded":"2024-09-15T23:15:42.626638+0000","last_fullsized":"2024-09-15T23:15:42.626638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609857+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:15:42.609857+0000","last_peered":"2024-09-15T23:15:42.609857+0000","last_clean":"2024-09-15T23:15:42.609857+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:15:42.609857+0000","last_undegraded":"2024-09-15T23:15:42.609857+0000","last_fullsized":"2024-09-15T23:15:42.609857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"92'37","reported_seq":294,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:50.638149+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:15:50.638149+0000","last_peered":"2024-09-15T23:15:50.638149+0000","last_clean":"2024-09-15T23:15:50.638149+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:15:50.638149+0000","last_undegraded":"2024-09-15T23:15:50.638149+0000","last_fullsized":"2024-09-15T23:15:50.638149+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68861,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":218,"num_read_kb":177,"num_write":38,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609007+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:15:42.609007+0000","last_peered":"2024-09-15T23:15:42.609007+0000","last_clean":"2024-09-15T23:15:42.609007+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:15:42.609007+0000","last_undegraded":"2024-09-15T23:15:42.609007+0000","last_fullsized":"2024-09-15T23:15:42.609007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613046+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:15:42.613046+0000","last_peered":"2024-09-15T23:15:42.613046+0000","last_clean":"2024-09-15T23:15:42.613046+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:15:42.613046+0000","last_undegraded":"2024-09-15T23:15:42.613046+0000","last_fullsized":"2024-09-15T23:15:42.613046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626922+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:15:42.626922+0000","last_peered":"2024-09-15T23:15:42.626922+0000","last_clean":"2024-09-15T23:15:42.626922+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:15:42.626922+0000","last_undegraded":"2024-09-15T23:15:42.626922+0000","last_fullsized":"2024-09-15T23:15:42.626922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612007+0000","last_change":"2024-09-15T23:14:41.641820+0000","last_active":"2024-09-15T23:15:42.612007+0000","last_peered":"2024-09-15T23:15:42.612007+0000","last_clean":"2024-09-15T23:15:42.612007+0000","last_became_active":"2024-09-15T23:14:41.641282+0000","last_became_peered":"2024-09-15T23:14:41.641282+0000","last_unstale":"2024-09-15T23:15:42.612007+0000","last_undegraded":"2024-09-15T23:15:42.612007+0000","last_fullsized":"2024-09-15T23:15:42.612007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:43.472601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.628116+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:15:43.628116+0000","last_peered":"2024-09-15T23:15:43.628116+0000","last_clean":"2024-09-15T23:15:43.628116+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:15:43.628116+0000","last_undegraded":"2024-09-15T23:15:43.628116+0000","last_fullsized":"2024-09-15T23:15:43.628116+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317463+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:15:43.317463+0000","last_peered":"2024-09-15T23:15:43.317463+0000","last_clean":"2024-09-15T23:15:43.317463+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:15:43.317463+0000","last_undegraded":"2024-09-15T23:15:43.317463+0000","last_fullsized":"2024-09-15T23:15:43.317463+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608807+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:15:42.608807+0000","last_peered":"2024-09-15T23:15:42.608807+0000","last_clean":"2024-09-15T23:15:42.608807+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:15:42.608807+0000","last_undegraded":"2024-09-15T23:15:42.608807+0000","last_fullsized":"2024-09-15T23:15:42.608807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612577+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:15:42.612577+0000","last_peered":"2024-09-15T23:15:42.612577+0000","last_clean":"2024-09-15T23:15:42.612577+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:15:42.612577+0000","last_undegraded":"2024-09-15T23:15:42.612577+0000","last_fullsized":"2024-09-15T23:15:42.612577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":145,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:49.635646+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:15:49.635646+0000","last_peered":"2024-09-15T23:15:49.635646+0000","last_clean":"2024-09-15T23:15:49.635646+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:15:49.635646+0000","last_undegraded":"2024-09-15T23:15:49.635646+0000","last_fullsized":"2024-09-15T23:15:49.635646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316758+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:15:43.316758+0000","last_peered":"2024-09-15T23:15:43.316758+0000","last_clean":"2024-09-15T23:15:43.316758+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:15:43.316758+0000","last_undegraded":"2024-09-15T23:15:43.316758+0000","last_fullsized":"2024-09-15T23:15:43.316758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627914+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:15:43.627914+0000","last_peered":"2024-09-15T23:15:43.627914+0000","last_clean":"2024-09-15T23:15:43.627914+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:15:43.627914+0000","last_undegraded":"2024-09-15T23:15:43.627914+0000","last_fullsized":"2024-09-15T23:15:43.627914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608984+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:15:42.608984+0000","last_peered":"2024-09-15T23:15:42.608984+0000","last_clean":"2024-09-15T23:15:42.608984+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:15:42.608984+0000","last_undegraded":"2024-09-15T23:15:42.608984+0000","last_fullsized":"2024-09-15T23:15:42.608984+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626347+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:15:42.626347+0000","last_peered":"2024-09-15T23:15:42.626347+0000","last_clean":"2024-09-15T23:15:42.626347+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:15:42.626347+0000","last_undegraded":"2024-09-15T23:15:42.626347+0000","last_fullsized":"2024-09-15T23:15:42.626347+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910018+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:15:42.910018+0000","last_peered":"2024-09-15T23:15:42.910018+0000","last_clean":"2024-09-15T23:15:42.910018+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:15:42.910018+0000","last_undegraded":"2024-09-15T23:15:42.910018+0000","last_fullsized":"2024-09-15T23:15:42.910018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316784+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:15:43.316784+0000","last_peered":"2024-09-15T23:15:43.316784+0000","last_clean":"2024-09-15T23:15:43.316784+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:15:43.316784+0000","last_undegraded":"2024-09-15T23:15:43.316784+0000","last_fullsized":"2024-09-15T23:15:43.316784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611743+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:15:42.611743+0000","last_peered":"2024-09-15T23:15:42.611743+0000","last_clean":"2024-09-15T23:15:42.611743+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:15:42.611743+0000","last_undegraded":"2024-09-15T23:15:42.611743+0000","last_fullsized":"2024-09-15T23:15:42.611743+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610033+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:15:42.610033+0000","last_peered":"2024-09-15T23:15:42.610033+0000","last_clean":"2024-09-15T23:15:42.610033+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:15:42.610033+0000","last_undegraded":"2024-09-15T23:15:42.610033+0000","last_fullsized":"2024-09-15T23:15:42.610033+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626302+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:15:42.626302+0000","last_peered":"2024-09-15T23:15:42.626302+0000","last_clean":"2024-09-15T23:15:42.626302+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:15:42.626302+0000","last_undegraded":"2024-09-15T23:15:42.626302+0000","last_fullsized":"2024-09-15T23:15:42.626302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613815+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:15:42.613815+0000","last_peered":"2024-09-15T23:15:42.613815+0000","last_clean":"2024-09-15T23:15:42.613815+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:15:42.613815+0000","last_undegraded":"2024-09-15T23:15:42.613815+0000","last_fullsized":"2024-09-15T23:15:42.613815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":152,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:49.636304+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:15:49.636304+0000","last_peered":"2024-09-15T23:15:49.636304+0000","last_clean":"2024-09-15T23:15:49.636304+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:15:49.636304+0000","last_undegraded":"2024-09-15T23:15:49.636304+0000","last_fullsized":"2024-09-15T23:15:49.636304+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1156,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609500+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:15:42.609500+0000","last_peered":"2024-09-15T23:15:42.609500+0000","last_clean":"2024-09-15T23:15:42.609500+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:15:42.609500+0000","last_undegraded":"2024-09-15T23:15:42.609500+0000","last_fullsized":"2024-09-15T23:15:42.609500+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613805+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:15:42.613805+0000","last_peered":"2024-09-15T23:15:42.613805+0000","last_clean":"2024-09-15T23:15:42.613805+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:15:42.613805+0000","last_undegraded":"2024-09-15T23:15:42.613805+0000","last_fullsized":"2024-09-15T23:15:42.613805+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909660+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:15:42.909660+0000","last_peered":"2024-09-15T23:15:42.909660+0000","last_clean":"2024-09-15T23:15:42.909660+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:15:42.909660+0000","last_undegraded":"2024-09-15T23:15:42.909660+0000","last_fullsized":"2024-09-15T23:15:42.909660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614459+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:42.614459+0000","last_peered":"2024-09-15T23:15:42.614459+0000","last_clean":"2024-09-15T23:15:42.614459+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:15:42.614459+0000","last_undegraded":"2024-09-15T23:15:42.614459+0000","last_fullsized":"2024-09-15T23:15:42.614459+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613647+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:15:42.613647+0000","last_peered":"2024-09-15T23:15:42.613647+0000","last_clean":"2024-09-15T23:15:42.613647+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:15:42.613647+0000","last_undegraded":"2024-09-15T23:15:42.613647+0000","last_fullsized":"2024-09-15T23:15:42.613647+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627750+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:15:43.627750+0000","last_peered":"2024-09-15T23:15:43.627750+0000","last_clean":"2024-09-15T23:15:43.627750+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:15:43.627750+0000","last_undegraded":"2024-09-15T23:15:43.627750+0000","last_fullsized":"2024-09-15T23:15:43.627750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612897+0000","last_change":"2024-09-15T23:14:42.010010+0000","last_active":"2024-09-15T23:15:42.612897+0000","last_peered":"2024-09-15T23:15:42.612897+0000","last_clean":"2024-09-15T23:15:42.612897+0000","last_became_active":"2024-09-15T23:14:42.009603+0000","last_became_peered":"2024-09-15T23:14:42.009603+0000","last_unstale":"2024-09-15T23:15:42.612897+0000","last_undegraded":"2024-09-15T23:15:42.612897+0000","last_fullsized":"2024-09-15T23:15:42.612897+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:34:51.849548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910096+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:15:42.910096+0000","last_peered":"2024-09-15T23:15:42.910096+0000","last_clean":"2024-09-15T23:15:42.910096+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:15:42.910096+0000","last_undegraded":"2024-09-15T23:15:42.910096+0000","last_fullsized":"2024-09-15T23:15:42.910096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614200+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:15:42.614200+0000","last_peered":"2024-09-15T23:15:42.614200+0000","last_clean":"2024-09-15T23:15:42.614200+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:15:42.614200+0000","last_undegraded":"2024-09-15T23:15:42.614200+0000","last_fullsized":"2024-09-15T23:15:42.614200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317395+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:15:43.317395+0000","last_peered":"2024-09-15T23:15:43.317395+0000","last_clean":"2024-09-15T23:15:43.317395+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:15:43.317395+0000","last_undegraded":"2024-09-15T23:15:43.317395+0000","last_fullsized":"2024-09-15T23:15:43.317395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626113+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:15:42.626113+0000","last_peered":"2024-09-15T23:15:42.626113+0000","last_clean":"2024-09-15T23:15:42.626113+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:15:42.626113+0000","last_undegraded":"2024-09-15T23:15:42.626113+0000","last_fullsized":"2024-09-15T23:15:42.626113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316925+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:15:43.316925+0000","last_peered":"2024-09-15T23:15:43.316925+0000","last_clean":"2024-09-15T23:15:43.316925+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:15:43.316925+0000","last_undegraded":"2024-09-15T23:15:43.316925+0000","last_fullsized":"2024-09-15T23:15:43.316925+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612732+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:15:42.612732+0000","last_peered":"2024-09-15T23:15:42.612732+0000","last_clean":"2024-09-15T23:15:42.612732+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:15:42.612732+0000","last_undegraded":"2024-09-15T23:15:42.612732+0000","last_fullsized":"2024-09-15T23:15:42.612732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612638+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:15:42.612638+0000","last_peered":"2024-09-15T23:15:42.612638+0000","last_clean":"2024-09-15T23:15:42.612638+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:15:42.612638+0000","last_undegraded":"2024-09-15T23:15:42.612638+0000","last_fullsized":"2024-09-15T23:15:42.612638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909941+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:15:42.909941+0000","last_peered":"2024-09-15T23:15:42.909941+0000","last_clean":"2024-09-15T23:15:42.909941+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:15:42.909941+0000","last_undegraded":"2024-09-15T23:15:42.909941+0000","last_fullsized":"2024-09-15T23:15:42.909941+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614098+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:15:42.614098+0000","last_peered":"2024-09-15T23:15:42.614098+0000","last_clean":"2024-09-15T23:15:42.614098+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:15:42.614098+0000","last_undegraded":"2024-09-15T23:15:42.614098+0000","last_fullsized":"2024-09-15T23:15:42.614098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610060+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:15:42.610060+0000","last_peered":"2024-09-15T23:15:42.610060+0000","last_clean":"2024-09-15T23:15:42.610060+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:15:42.610060+0000","last_undegraded":"2024-09-15T23:15:42.610060+0000","last_fullsized":"2024-09-15T23:15:42.610060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910207+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:15:42.910207+0000","last_peered":"2024-09-15T23:15:42.910207+0000","last_clean":"2024-09-15T23:15:42.910207+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:15:42.910207+0000","last_undegraded":"2024-09-15T23:15:42.910207+0000","last_fullsized":"2024-09-15T23:15:42.910207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612963+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:15:42.612963+0000","last_peered":"2024-09-15T23:15:42.612963+0000","last_clean":"2024-09-15T23:15:42.612963+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:15:42.612963+0000","last_undegraded":"2024-09-15T23:15:42.612963+0000","last_fullsized":"2024-09-15T23:15:42.612963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627665+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:15:43.627665+0000","last_peered":"2024-09-15T23:15:43.627665+0000","last_clean":"2024-09-15T23:15:43.627665+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:15:43.627665+0000","last_undegraded":"2024-09-15T23:15:43.627665+0000","last_fullsized":"2024-09-15T23:15:43.627665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627685+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:15:43.627685+0000","last_peered":"2024-09-15T23:15:43.627685+0000","last_clean":"2024-09-15T23:15:43.627685+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:15:43.627685+0000","last_undegraded":"2024-09-15T23:15:43.627685+0000","last_fullsized":"2024-09-15T23:15:43.627685+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612501+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:15:42.612501+0000","last_peered":"2024-09-15T23:15:42.612501+0000","last_clean":"2024-09-15T23:15:42.612501+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:15:42.612501+0000","last_undegraded":"2024-09-15T23:15:42.612501+0000","last_fullsized":"2024-09-15T23:15:42.612501+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612646+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:15:42.612646+0000","last_peered":"2024-09-15T23:15:42.612646+0000","last_clean":"2024-09-15T23:15:42.612646+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:15:42.612646+0000","last_undegraded":"2024-09-15T23:15:42.612646+0000","last_fullsized":"2024-09-15T23:15:42.612646+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317377+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:15:43.317377+0000","last_peered":"2024-09-15T23:15:43.317377+0000","last_clean":"2024-09-15T23:15:43.317377+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:15:43.317377+0000","last_undegraded":"2024-09-15T23:15:43.317377+0000","last_fullsized":"2024-09-15T23:15:43.317377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610726+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:15:42.610726+0000","last_peered":"2024-09-15T23:15:42.610726+0000","last_clean":"2024-09-15T23:15:42.610726+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:15:42.610726+0000","last_undegraded":"2024-09-15T23:15:42.610726+0000","last_fullsized":"2024-09-15T23:15:42.610726+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613153+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:15:42.613153+0000","last_peered":"2024-09-15T23:15:42.613153+0000","last_clean":"2024-09-15T23:15:42.613153+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:15:42.613153+0000","last_undegraded":"2024-09-15T23:15:42.613153+0000","last_fullsized":"2024-09-15T23:15:42.613153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316763+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:15:43.316763+0000","last_peered":"2024-09-15T23:15:43.316763+0000","last_clean":"2024-09-15T23:15:43.316763+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:15:43.316763+0000","last_undegraded":"2024-09-15T23:15:43.316763+0000","last_fullsized":"2024-09-15T23:15:43.316763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316851+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:15:43.316851+0000","last_peered":"2024-09-15T23:15:43.316851+0000","last_clean":"2024-09-15T23:15:43.316851+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:15:43.316851+0000","last_undegraded":"2024-09-15T23:15:43.316851+0000","last_fullsized":"2024-09-15T23:15:43.316851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610204+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:15:42.610204+0000","last_peered":"2024-09-15T23:15:42.610204+0000","last_clean":"2024-09-15T23:15:42.610204+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:15:42.610204+0000","last_undegraded":"2024-09-15T23:15:42.610204+0000","last_fullsized":"2024-09-15T23:15:42.610204+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609588+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:15:42.609588+0000","last_peered":"2024-09-15T23:15:42.609588+0000","last_clean":"2024-09-15T23:15:42.609588+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:15:42.609588+0000","last_undegraded":"2024-09-15T23:15:42.609588+0000","last_fullsized":"2024-09-15T23:15:42.609588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612535+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:15:42.612535+0000","last_peered":"2024-09-15T23:15:42.612535+0000","last_clean":"2024-09-15T23:15:42.612535+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:15:42.612535+0000","last_undegraded":"2024-09-15T23:15:42.612535+0000","last_fullsized":"2024-09-15T23:15:42.612535+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627888+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:15:43.627888+0000","last_peered":"2024-09-15T23:15:43.627888+0000","last_clean":"2024-09-15T23:15:43.627888+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:15:43.627888+0000","last_undegraded":"2024-09-15T23:15:43.627888+0000","last_fullsized":"2024-09-15T23:15:43.627888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627940+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:15:43.627940+0000","last_peered":"2024-09-15T23:15:43.627940+0000","last_clean":"2024-09-15T23:15:43.627940+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:15:43.627940+0000","last_undegraded":"2024-09-15T23:15:43.627940+0000","last_fullsized":"2024-09-15T23:15:43.627940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910172+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:15:42.910172+0000","last_peered":"2024-09-15T23:15:42.910172+0000","last_clean":"2024-09-15T23:15:42.910172+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:15:42.910172+0000","last_undegraded":"2024-09-15T23:15:42.910172+0000","last_fullsized":"2024-09-15T23:15:42.910172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73005,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":657,"num_read_kb":406,"num_write":179,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":221291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514538,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522238,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743470,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712832,"statfs":{"total":95995035648,"available":95961939968,"internally_reserved":0,"allocated":1163264,"data_stored":1178255,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906996,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1032192,"data_stored":526458,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135929,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522170,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364862,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1077248,"data_stored":1118334,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528388,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":517538,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790024,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1085440,"data_stored":1113922,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1019904,"data_stored":522188,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":517732,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":589331,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":587447,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1032192,"data_stored":518926,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:15:57.473 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-15T23:15:57.473 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-15T23:15:57.731 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:57 smithi148 ceph-mon[35610]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:58.040 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:57 smithi062 ceph-mon[30949]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:58.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:57 smithi067 ceph-mon[35665]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:58.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.273+0000 7f0e3248d640 1 -- 172.21.15.62:0/969825813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 msgr2=0x7f0e2c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:58.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.273+0000 7f0e3248d640 1 --2- 172.21.15.62:0/969825813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 0x7f0e2c10ea90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009ca0 tx=0x7f0e2402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.273+0000 7f0e3248d640 1 -- 172.21.15.62:0/969825813 shutdown_connections 2024-09-15T23:15:58.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.273+0000 7f0e3248d640 1 --2- 172.21.15.62:0/969825813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 0x7f0e2c10ea90 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.273+0000 7f0e3248d640 1 --2- 172.21.15.62:0/969825813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e2c102790 0x7f0e2c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.273+0000 7f0e3248d640 1 --2- 172.21.15.62:0/969825813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e2c101de0 0x7f0e2c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.273+0000 7f0e3248d640 1 -- 172.21.15.62:0/969825813 >> 172.21.15.62:0/969825813 conn(0x7f0e2c0fd8c0 msgr2=0x7f0e2c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:58.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.273+0000 7f0e3248d640 1 -- 172.21.15.62:0/969825813 shutdown_connections 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.273+0000 7f0e3248d640 1 -- 172.21.15.62:0/969825813 wait complete. 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e3248d640 1 Processor -- start 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e3248d640 1 -- start start 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e3248d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e2c101de0 0x7f0e2c19a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e3248d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e2c102790 0x7f0e2c19ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e3248d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 0x7f0e2c19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e3248d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c111f70 con 0x7f0e2c106f10 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e3248d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c111df0 con 0x7f0e2c101de0 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e3248d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c1120f0 con 0x7f0e2c102790 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e30a2a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 0x7f0e2c19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e30a2a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 0x7f0e2c19eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42726/0 (socket says 172.21.15.62:42726) 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e30a2a640 1 -- 172.21.15.62:0/701047192 learned_addr learned my addr 172.21.15.62:0/701047192 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:58.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e2b7fe640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e2c102790 0x7f0e2c19ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.274+0000 7f0e2bfff640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e2c101de0 0x7f0e2c19a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e30a2a640 1 -- 172.21.15.62:0/701047192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e2c102790 msgr2=0x7f0e2c19ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e30a2a640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e2c102790 0x7f0e2c19ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e30a2a640 1 -- 172.21.15.62:0/701047192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e2c101de0 msgr2=0x7f0e2c19a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e30a2a640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e2c101de0 0x7f0e2c19a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e30a2a640 1 -- 172.21.15.62:0/701047192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c19f580 con 0x7f0e2c106f10 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e30a2a640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 0x7f0e2c19eea0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0e24031a60 tx=0x7f0e24031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e297fa640 1 -- 172.21.15.62:0/701047192 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e24003c30 con 0x7f0e2c106f10 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e297fa640 1 -- 172.21.15.62:0/701047192 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e24003dd0 con 0x7f0e2c106f10 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e2c19f810 con 0x7f0e2c106f10 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e297fa640 1 -- 172.21.15.62:0/701047192 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e240047e0 con 0x7f0e2c106f10 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.275+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e2c1a7150 con 0x7f0e2c106f10 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.276+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e2c1021c0 con 0x7f0e2c106f10 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.277+0000 7f0e297fa640 1 -- 172.21.15.62:0/701047192 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0e2402f790 con 0x7f0e2c106f10 2024-09-15T23:15:58.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.278+0000 7f0e297fa640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0e140a3240 0x7f0e140a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:58.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.278+0000 7f0e297fa640 1 -- 172.21.15.62:0/701047192 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0e240f10a0 con 0x7f0e2c106f10 2024-09-15T23:15:58.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.278+0000 7f0e2bfff640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0e140a3240 0x7f0e140a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:58.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.278+0000 7f0e2bfff640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0e140a3240 0x7f0e140a5700 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0058c0 tx=0x7f0e1c00a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:58.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.282+0000 7f0e297fa640 1 -- 172.21.15.62:0/701047192 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0e240bda40 con 0x7f0e2c106f10 2024-09-15T23:15:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.455+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 --> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e2c0feb80 con 0x7f0e140a3240 2024-09-15T23:15:58.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.463+0000 7f0e297fa640 1 -- 172.21.15.62:0/701047192 <== mgr.14150 v2:172.21.15.62:6800/4035090192 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340738 (secure 0 0 0) 0x7f0e2c0feb80 con 0x7f0e140a3240 2024-09-15T23:15:58.467 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:15:58.469 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-09-15T23:15:58.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0e140a3240 msgr2=0x7f0e140a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:58.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0e140a3240 0x7f0e140a5700 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0058c0 tx=0x7f0e1c00a4c0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 msgr2=0x7f0e2c19eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 0x7f0e2c19eea0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0e24031a60 tx=0x7f0e24031a90 comp rx=0 tx=0).stop 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 shutdown_connections 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0e140a3240 0x7f0e140a5700 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c106f10 0x7f0e2c19eea0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e2c102790 0x7f0e2c19ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 --2- 172.21.15.62:0/701047192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e2c101de0 0x7f0e2c19a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 >> 172.21.15.62:0/701047192 conn(0x7f0e2c0fd8c0 msgr2=0x7f0e2c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 shutdown_connections 2024-09-15T23:15:58.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:58.467+0000 7f0e3248d640 1 -- 172.21.15.62:0/701047192 wait complete. 2024-09-15T23:15:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:58 smithi148 ceph-mon[35610]: pgmap v361: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 7 op/s 2024-09-15T23:15:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:15:58 smithi148 ceph-mon[35610]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:59.070 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:58 smithi062 ceph-mon[30949]: pgmap v361: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 7 op/s 2024-09-15T23:15:59.070 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:15:58 smithi062 ceph-mon[30949]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:59.099 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-15T23:15:57.728261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664745,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":806,"num_read_kb":529,"num_write":363,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1180848,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":792,"ondisk_log_size":792,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369996,"kb_used_data":12368,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571828,"statfs":{"total":1151940427776,"available":1151561551872,"internally_reserved":0,"allocated":12664832,"data_stored":8235677,"data_compressed":461688,"data_compressed_allocated":626688,"data_compressed_original":2752512,"omap_allocated":19143,"internal_metadata":365933881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6819,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":49,"num_write":5,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003143"},"pg_stats":[{"pgid":"2.1f","version":"88'9","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609619+0000","last_change":"2024-09-15T23:13:30.489144+0000","last_active":"2024-09-15T23:15:42.609619+0000","last_peered":"2024-09-15T23:15:42.609619+0000","last_clean":"2024-09-15T23:15:42.609619+0000","last_became_active":"2024-09-15T23:13:30.488834+0000","last_became_peered":"2024-09-15T23:13:30.488834+0000","last_unstale":"2024-09-15T23:15:42.609619+0000","last_undegraded":"2024-09-15T23:15:42.609619+0000","last_fullsized":"2024-09-15T23:15:42.609619+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:56.106482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317224+0000","last_change":"2024-09-15T23:13:32.514252+0000","last_active":"2024-09-15T23:15:43.317224+0000","last_peered":"2024-09-15T23:15:43.317224+0000","last_clean":"2024-09-15T23:15:43.317224+0000","last_became_active":"2024-09-15T23:13:32.513952+0000","last_became_peered":"2024-09-15T23:13:32.513952+0000","last_unstale":"2024-09-15T23:15:43.317224+0000","last_undegraded":"2024-09-15T23:15:43.317224+0000","last_fullsized":"2024-09-15T23:15:43.317224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:32:33.854417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613665+0000","last_change":"2024-09-15T23:13:39.553448+0000","last_active":"2024-09-15T23:15:42.613665+0000","last_peered":"2024-09-15T23:15:42.613665+0000","last_clean":"2024-09-15T23:15:42.613665+0000","last_became_active":"2024-09-15T23:13:39.552735+0000","last_became_peered":"2024-09-15T23:13:39.552735+0000","last_unstale":"2024-09-15T23:15:42.613665+0000","last_undegraded":"2024-09-15T23:15:42.613665+0000","last_fullsized":"2024-09-15T23:15:42.613665+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:07.400427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317397+0000","last_change":"2024-09-15T23:13:30.597495+0000","last_active":"2024-09-15T23:15:43.317397+0000","last_peered":"2024-09-15T23:15:43.317397+0000","last_clean":"2024-09-15T23:15:43.317397+0000","last_became_active":"2024-09-15T23:13:30.596656+0000","last_became_peered":"2024-09-15T23:13:30.596656+0000","last_unstale":"2024-09-15T23:15:43.317397+0000","last_undegraded":"2024-09-15T23:15:43.317397+0000","last_fullsized":"2024-09-15T23:15:43.317397+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:10.082849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.628111+0000","last_change":"2024-09-15T23:13:32.508688+0000","last_active":"2024-09-15T23:15:43.628111+0000","last_peered":"2024-09-15T23:15:43.628111+0000","last_clean":"2024-09-15T23:15:43.628111+0000","last_became_active":"2024-09-15T23:13:32.508376+0000","last_became_peered":"2024-09-15T23:13:32.508376+0000","last_unstale":"2024-09-15T23:15:43.628111+0000","last_undegraded":"2024-09-15T23:15:43.628111+0000","last_fullsized":"2024-09-15T23:15:43.628111+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:40:06.860929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611378+0000","last_change":"2024-09-15T23:13:39.548061+0000","last_active":"2024-09-15T23:15:42.611378+0000","last_peered":"2024-09-15T23:15:42.611378+0000","last_clean":"2024-09-15T23:15:42.611378+0000","last_became_active":"2024-09-15T23:13:39.547100+0000","last_became_peered":"2024-09-15T23:13:39.547100+0000","last_unstale":"2024-09-15T23:15:42.611378+0000","last_undegraded":"2024-09-15T23:15:42.611378+0000","last_fullsized":"2024-09-15T23:15:42.611378+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:47:44.909736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'4","reported_seq":23,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909808+0000","last_change":"2024-09-15T23:14:41.659816+0000","last_active":"2024-09-15T23:15:42.909808+0000","last_peered":"2024-09-15T23:15:42.909808+0000","last_clean":"2024-09-15T23:15:42.909808+0000","last_became_active":"2024-09-15T23:14:41.641611+0000","last_became_peered":"2024-09-15T23:14:41.641611+0000","last_unstale":"2024-09-15T23:15:42.909808+0000","last_undegraded":"2024-09-15T23:15:42.909808+0000","last_fullsized":"2024-09-15T23:15:42.909808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:23.215938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612963+0000","last_change":"2024-09-15T23:13:32.508590+0000","last_active":"2024-09-15T23:15:42.612963+0000","last_peered":"2024-09-15T23:15:42.612963+0000","last_clean":"2024-09-15T23:15:42.612963+0000","last_became_active":"2024-09-15T23:13:32.508245+0000","last_became_peered":"2024-09-15T23:13:32.508245+0000","last_unstale":"2024-09-15T23:15:42.612963+0000","last_undegraded":"2024-09-15T23:15:42.612963+0000","last_fullsized":"2024-09-15T23:15:42.612963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:34:30.840473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.544104+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.543595+0000","last_became_peered":"2024-09-15T23:13:39.543595+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:44:35.770681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610025+0000","last_change":"2024-09-15T23:13:30.484769+0000","last_active":"2024-09-15T23:15:42.610025+0000","last_peered":"2024-09-15T23:15:42.610025+0000","last_clean":"2024-09-15T23:15:42.610025+0000","last_became_active":"2024-09-15T23:13:30.484382+0000","last_became_peered":"2024-09-15T23:13:30.484382+0000","last_unstale":"2024-09-15T23:15:42.610025+0000","last_undegraded":"2024-09-15T23:15:42.610025+0000","last_fullsized":"2024-09-15T23:15:42.610025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:01.910247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614867+0000","last_change":"2024-09-15T23:15:42.614867+0000","last_active":"2024-09-15T23:15:42.614867+0000","last_peered":"2024-09-15T23:15:42.614867+0000","last_clean":"2024-09-15T23:15:42.614867+0000","last_became_active":"2024-09-15T23:15:42.614459+0000","last_became_peered":"2024-09-15T23:15:42.614459+0000","last_unstale":"2024-09-15T23:15:42.614867+0000","last_undegraded":"2024-09-15T23:15:42.614867+0000","last_fullsized":"2024-09-15T23:15:42.614867+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:58:15.254442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611583+0000","last_change":"2024-09-15T23:13:39.549195+0000","last_active":"2024-09-15T23:15:42.611583+0000","last_peered":"2024-09-15T23:15:42.611583+0000","last_clean":"2024-09-15T23:15:42.611583+0000","last_became_active":"2024-09-15T23:13:39.548465+0000","last_became_peered":"2024-09-15T23:13:39.548465+0000","last_unstale":"2024-09-15T23:15:42.611583+0000","last_undegraded":"2024-09-15T23:15:42.611583+0000","last_fullsized":"2024-09-15T23:15:42.611583+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:06:51.432234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317095+0000","last_change":"2024-09-15T23:13:32.507654+0000","last_active":"2024-09-15T23:15:43.317095+0000","last_peered":"2024-09-15T23:15:43.317095+0000","last_clean":"2024-09-15T23:15:43.317095+0000","last_became_active":"2024-09-15T23:13:32.507339+0000","last_became_peered":"2024-09-15T23:13:32.507339+0000","last_unstale":"2024-09-15T23:15:43.317095+0000","last_undegraded":"2024-09-15T23:15:43.317095+0000","last_fullsized":"2024-09-15T23:15:43.317095+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:15:44.973326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"88'4","reported_seq":56,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609750+0000","last_change":"2024-09-15T23:14:41.646320+0000","last_active":"2024-09-15T23:15:42.609750+0000","last_peered":"2024-09-15T23:15:42.609750+0000","last_clean":"2024-09-15T23:15:42.609750+0000","last_became_active":"2024-09-15T23:14:41.641933+0000","last_became_peered":"2024-09-15T23:14:41.641933+0000","last_unstale":"2024-09-15T23:15:42.609750+0000","last_undegraded":"2024-09-15T23:15:42.609750+0000","last_fullsized":"2024-09-15T23:15:42.609750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:49.417407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611971+0000","last_change":"2024-09-15T23:13:39.747425+0000","last_active":"2024-09-15T23:15:42.611971+0000","last_peered":"2024-09-15T23:15:42.611971+0000","last_clean":"2024-09-15T23:15:42.611971+0000","last_became_active":"2024-09-15T23:13:39.746999+0000","last_became_peered":"2024-09-15T23:13:39.746999+0000","last_unstale":"2024-09-15T23:15:42.611971+0000","last_undegraded":"2024-09-15T23:15:42.611971+0000","last_fullsized":"2024-09-15T23:15:42.611971+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:58:16.458198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316939+0000","last_change":"2024-09-15T23:13:32.508968+0000","last_active":"2024-09-15T23:15:43.316939+0000","last_peered":"2024-09-15T23:15:43.316939+0000","last_clean":"2024-09-15T23:15:43.316939+0000","last_became_active":"2024-09-15T23:13:32.508748+0000","last_became_peered":"2024-09-15T23:13:32.508748+0000","last_unstale":"2024-09-15T23:15:43.316939+0000","last_undegraded":"2024-09-15T23:15:43.316939+0000","last_fullsized":"2024-09-15T23:15:43.316939+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:20.358354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613807+0000","last_change":"2024-09-15T23:13:30.487074+0000","last_active":"2024-09-15T23:15:42.613807+0000","last_peered":"2024-09-15T23:15:42.613807+0000","last_clean":"2024-09-15T23:15:42.613807+0000","last_became_active":"2024-09-15T23:13:30.486744+0000","last_became_peered":"2024-09-15T23:13:30.486744+0000","last_unstale":"2024-09-15T23:15:42.613807+0000","last_undegraded":"2024-09-15T23:15:42.613807+0000","last_fullsized":"2024-09-15T23:15:42.613807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:04.417034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"88'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612468+0000","last_change":"2024-09-15T23:13:39.547813+0000","last_active":"2024-09-15T23:15:42.612468+0000","last_peered":"2024-09-15T23:15:42.612468+0000","last_clean":"2024-09-15T23:15:42.612468+0000","last_became_active":"2024-09-15T23:13:39.547172+0000","last_became_peered":"2024-09-15T23:13:39.547172+0000","last_unstale":"2024-09-15T23:15:42.612468+0000","last_undegraded":"2024-09-15T23:15:42.612468+0000","last_fullsized":"2024-09-15T23:15:42.612468+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:37:20.590881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909877+0000","last_change":"2024-09-15T23:13:32.506153+0000","last_active":"2024-09-15T23:15:42.909877+0000","last_peered":"2024-09-15T23:15:42.909877+0000","last_clean":"2024-09-15T23:15:42.909877+0000","last_became_active":"2024-09-15T23:13:32.505205+0000","last_became_peered":"2024-09-15T23:13:32.505205+0000","last_unstale":"2024-09-15T23:15:42.909877+0000","last_undegraded":"2024-09-15T23:15:42.909877+0000","last_fullsized":"2024-09-15T23:15:42.909877+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:06:27.096179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316960+0000","last_change":"2024-09-15T23:13:30.488282+0000","last_active":"2024-09-15T23:15:43.316960+0000","last_peered":"2024-09-15T23:15:43.316960+0000","last_clean":"2024-09-15T23:15:43.316960+0000","last_became_active":"2024-09-15T23:13:30.487907+0000","last_became_peered":"2024-09-15T23:13:30.487907+0000","last_unstale":"2024-09-15T23:15:43.316960+0000","last_undegraded":"2024-09-15T23:15:43.316960+0000","last_fullsized":"2024-09-15T23:15:43.316960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:11:00.260780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609332+0000","last_change":"2024-09-15T23:13:39.555673+0000","last_active":"2024-09-15T23:15:42.609332+0000","last_peered":"2024-09-15T23:15:42.609332+0000","last_clean":"2024-09-15T23:15:42.609332+0000","last_became_active":"2024-09-15T23:13:39.555208+0000","last_became_peered":"2024-09-15T23:13:39.555208+0000","last_unstale":"2024-09-15T23:15:42.609332+0000","last_undegraded":"2024-09-15T23:15:42.609332+0000","last_fullsized":"2024-09-15T23:15:42.609332+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:22:25.307994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609453+0000","last_change":"2024-09-15T23:13:32.508227+0000","last_active":"2024-09-15T23:15:42.609453+0000","last_peered":"2024-09-15T23:15:42.609453+0000","last_clean":"2024-09-15T23:15:42.609453+0000","last_became_active":"2024-09-15T23:13:32.507878+0000","last_became_peered":"2024-09-15T23:13:32.507878+0000","last_unstale":"2024-09-15T23:15:42.609453+0000","last_undegraded":"2024-09-15T23:15:42.609453+0000","last_fullsized":"2024-09-15T23:15:42.609453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:03:39.964553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613324+0000","last_change":"2024-09-15T23:13:30.597185+0000","last_active":"2024-09-15T23:15:42.613324+0000","last_peered":"2024-09-15T23:15:42.613324+0000","last_clean":"2024-09-15T23:15:42.613324+0000","last_became_active":"2024-09-15T23:13:30.596876+0000","last_became_peered":"2024-09-15T23:13:30.596876+0000","last_unstale":"2024-09-15T23:15:42.613324+0000","last_undegraded":"2024-09-15T23:15:42.613324+0000","last_fullsized":"2024-09-15T23:15:42.613324+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:35:38.175899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612082+0000","last_change":"2024-09-15T23:13:39.749010+0000","last_active":"2024-09-15T23:15:42.612082+0000","last_peered":"2024-09-15T23:15:42.612082+0000","last_clean":"2024-09-15T23:15:42.612082+0000","last_became_active":"2024-09-15T23:13:39.748509+0000","last_became_peered":"2024-09-15T23:13:39.748509+0000","last_unstale":"2024-09-15T23:15:42.612082+0000","last_undegraded":"2024-09-15T23:15:42.612082+0000","last_fullsized":"2024-09-15T23:15:42.612082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:34:11.767234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":47,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.628020+0000","last_change":"2024-09-15T23:14:42.009605+0000","last_active":"2024-09-15T23:15:43.628020+0000","last_peered":"2024-09-15T23:15:43.628020+0000","last_clean":"2024-09-15T23:15:43.628020+0000","last_became_active":"2024-09-15T23:14:42.009237+0000","last_became_peered":"2024-09-15T23:14:42.009237+0000","last_unstale":"2024-09-15T23:15:43.628020+0000","last_undegraded":"2024-09-15T23:15:43.628020+0000","last_fullsized":"2024-09-15T23:15:43.628020+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:11:08.835758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"88'4","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612739+0000","last_change":"2024-09-15T23:13:30.553590+0000","last_active":"2024-09-15T23:15:42.612739+0000","last_peered":"2024-09-15T23:15:42.612739+0000","last_clean":"2024-09-15T23:15:42.612739+0000","last_became_active":"2024-09-15T23:13:30.553292+0000","last_became_peered":"2024-09-15T23:13:30.553292+0000","last_unstale":"2024-09-15T23:15:42.612739+0000","last_undegraded":"2024-09-15T23:15:42.612739+0000","last_fullsized":"2024-09-15T23:15:42.612739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:17:52.561964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614193+0000","last_change":"2024-09-15T23:13:39.552213+0000","last_active":"2024-09-15T23:15:42.614193+0000","last_peered":"2024-09-15T23:15:42.614193+0000","last_clean":"2024-09-15T23:15:42.614193+0000","last_became_active":"2024-09-15T23:13:39.551769+0000","last_became_peered":"2024-09-15T23:13:39.551769+0000","last_unstale":"2024-09-15T23:15:42.614193+0000","last_undegraded":"2024-09-15T23:15:42.614193+0000","last_fullsized":"2024-09-15T23:15:42.614193+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:40:05.305749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317410+0000","last_change":"2024-09-15T23:13:32.516356+0000","last_active":"2024-09-15T23:15:43.317410+0000","last_peered":"2024-09-15T23:15:43.317410+0000","last_clean":"2024-09-15T23:15:43.317410+0000","last_became_active":"2024-09-15T23:13:32.515954+0000","last_became_peered":"2024-09-15T23:13:32.515954+0000","last_unstale":"2024-09-15T23:15:43.317410+0000","last_undegraded":"2024-09-15T23:15:43.317410+0000","last_fullsized":"2024-09-15T23:15:43.317410+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:18:39.259559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'14","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610065+0000","last_change":"2024-09-15T23:13:30.487540+0000","last_active":"2024-09-15T23:15:42.610065+0000","last_peered":"2024-09-15T23:15:42.610065+0000","last_clean":"2024-09-15T23:15:42.610065+0000","last_became_active":"2024-09-15T23:13:30.487189+0000","last_became_peered":"2024-09-15T23:13:30.487189+0000","last_unstale":"2024-09-15T23:15:42.610065+0000","last_undegraded":"2024-09-15T23:15:42.610065+0000","last_fullsized":"2024-09-15T23:15:42.610065+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:47:45.049052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316638+0000","last_change":"2024-09-15T23:13:39.747994+0000","last_active":"2024-09-15T23:15:43.316638+0000","last_peered":"2024-09-15T23:15:43.316638+0000","last_clean":"2024-09-15T23:15:43.316638+0000","last_became_active":"2024-09-15T23:13:39.747592+0000","last_became_peered":"2024-09-15T23:13:39.747592+0000","last_unstale":"2024-09-15T23:15:43.316638+0000","last_undegraded":"2024-09-15T23:15:43.316638+0000","last_fullsized":"2024-09-15T23:15:43.316638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:59:27.147500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612814+0000","last_change":"2024-09-15T23:13:32.502589+0000","last_active":"2024-09-15T23:15:42.612814+0000","last_peered":"2024-09-15T23:15:42.612814+0000","last_clean":"2024-09-15T23:15:42.612814+0000","last_became_active":"2024-09-15T23:13:32.502264+0000","last_became_peered":"2024-09-15T23:13:32.502264+0000","last_unstale":"2024-09-15T23:15:42.612814+0000","last_undegraded":"2024-09-15T23:15:42.612814+0000","last_fullsized":"2024-09-15T23:15:42.612814+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:31:34.374170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"88'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610084+0000","last_change":"2024-09-15T23:13:30.599000+0000","last_active":"2024-09-15T23:15:42.610084+0000","last_peered":"2024-09-15T23:15:42.610084+0000","last_clean":"2024-09-15T23:15:42.610084+0000","last_became_active":"2024-09-15T23:13:30.598670+0000","last_became_peered":"2024-09-15T23:13:30.598670+0000","last_unstale":"2024-09-15T23:15:42.610084+0000","last_undegraded":"2024-09-15T23:15:42.610084+0000","last_fullsized":"2024-09-15T23:15:42.610084+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:30.044908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909814+0000","last_change":"2024-09-15T23:13:39.550192+0000","last_active":"2024-09-15T23:15:42.909814+0000","last_peered":"2024-09-15T23:15:42.909814+0000","last_clean":"2024-09-15T23:15:42.909814+0000","last_became_active":"2024-09-15T23:13:39.549756+0000","last_became_peered":"2024-09-15T23:13:39.549756+0000","last_unstale":"2024-09-15T23:15:42.909814+0000","last_undegraded":"2024-09-15T23:15:42.909814+0000","last_fullsized":"2024-09-15T23:15:42.909814+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:35:22.906887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610218+0000","last_change":"2024-09-15T23:13:32.503921+0000","last_active":"2024-09-15T23:15:42.610218+0000","last_peered":"2024-09-15T23:15:42.610218+0000","last_clean":"2024-09-15T23:15:42.610218+0000","last_became_active":"2024-09-15T23:13:32.503519+0000","last_became_peered":"2024-09-15T23:13:32.503519+0000","last_unstale":"2024-09-15T23:15:42.610218+0000","last_undegraded":"2024-09-15T23:15:42.610218+0000","last_fullsized":"2024-09-15T23:15:42.610218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:56:31.268719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609669+0000","last_change":"2024-09-15T23:13:30.597364+0000","last_active":"2024-09-15T23:15:42.609669+0000","last_peered":"2024-09-15T23:15:42.609669+0000","last_clean":"2024-09-15T23:15:42.609669+0000","last_became_active":"2024-09-15T23:13:30.597087+0000","last_became_peered":"2024-09-15T23:13:30.597087+0000","last_unstale":"2024-09-15T23:15:42.609669+0000","last_undegraded":"2024-09-15T23:15:42.609669+0000","last_fullsized":"2024-09-15T23:15:42.609669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:24:50.336456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609473+0000","last_change":"2024-09-15T23:14:41.995466+0000","last_active":"2024-09-15T23:15:42.609473+0000","last_peered":"2024-09-15T23:15:42.609473+0000","last_clean":"2024-09-15T23:15:42.609473+0000","last_became_active":"2024-09-15T23:14:41.994915+0000","last_became_peered":"2024-09-15T23:14:41.994915+0000","last_unstale":"2024-09-15T23:15:42.609473+0000","last_undegraded":"2024-09-15T23:15:42.609473+0000","last_fullsized":"2024-09-15T23:15:42.609473+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:36:04.121596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610393+0000","last_change":"2024-09-15T23:13:32.510117+0000","last_active":"2024-09-15T23:15:42.610393+0000","last_peered":"2024-09-15T23:15:42.610393+0000","last_clean":"2024-09-15T23:15:42.610393+0000","last_became_active":"2024-09-15T23:13:32.509683+0000","last_became_peered":"2024-09-15T23:13:32.509683+0000","last_unstale":"2024-09-15T23:15:42.610393+0000","last_undegraded":"2024-09-15T23:15:42.610393+0000","last_fullsized":"2024-09-15T23:15:42.610393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:36:21.781852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317131+0000","last_change":"2024-09-15T23:13:30.487039+0000","last_active":"2024-09-15T23:15:43.317131+0000","last_peered":"2024-09-15T23:15:43.317131+0000","last_clean":"2024-09-15T23:15:43.317131+0000","last_became_active":"2024-09-15T23:13:30.485966+0000","last_became_peered":"2024-09-15T23:13:30.485966+0000","last_unstale":"2024-09-15T23:15:43.317131+0000","last_undegraded":"2024-09-15T23:15:43.317131+0000","last_fullsized":"2024-09-15T23:15:43.317131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:38:20.267781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612342+0000","last_change":"2024-09-15T23:13:39.548485+0000","last_active":"2024-09-15T23:15:42.612342+0000","last_peered":"2024-09-15T23:15:42.612342+0000","last_clean":"2024-09-15T23:15:42.612342+0000","last_became_active":"2024-09-15T23:13:39.547923+0000","last_became_peered":"2024-09-15T23:13:39.547923+0000","last_unstale":"2024-09-15T23:15:42.612342+0000","last_undegraded":"2024-09-15T23:15:42.612342+0000","last_fullsized":"2024-09-15T23:15:42.612342+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:26:36.617773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626237+0000","last_change":"2024-09-15T23:13:32.512191+0000","last_active":"2024-09-15T23:15:42.626237+0000","last_peered":"2024-09-15T23:15:42.626237+0000","last_clean":"2024-09-15T23:15:42.626237+0000","last_became_active":"2024-09-15T23:13:32.511720+0000","last_became_peered":"2024-09-15T23:13:32.511720+0000","last_unstale":"2024-09-15T23:15:42.626237+0000","last_undegraded":"2024-09-15T23:15:42.626237+0000","last_fullsized":"2024-09-15T23:15:42.626237+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:55:30.964887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'4","reported_seq":150,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:53.634948+0000","last_change":"2024-09-15T23:13:30.597016+0000","last_active":"2024-09-15T23:15:53.634948+0000","last_peered":"2024-09-15T23:15:53.634948+0000","last_clean":"2024-09-15T23:15:53.634948+0000","last_became_active":"2024-09-15T23:13:30.596608+0000","last_became_peered":"2024-09-15T23:13:30.596608+0000","last_unstale":"2024-09-15T23:15:53.634948+0000","last_undegraded":"2024-09-15T23:15:53.634948+0000","last_fullsized":"2024-09-15T23:15:53.634948+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:09.930346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":107,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613035+0000","last_change":"2024-09-15T23:13:39.748147+0000","last_active":"2024-09-15T23:15:42.613035+0000","last_peered":"2024-09-15T23:15:42.613035+0000","last_clean":"2024-09-15T23:15:42.613035+0000","last_became_active":"2024-09-15T23:13:39.747700+0000","last_became_peered":"2024-09-15T23:13:39.747700+0000","last_unstale":"2024-09-15T23:15:42.613035+0000","last_undegraded":"2024-09-15T23:15:42.613035+0000","last_fullsized":"2024-09-15T23:15:42.613035+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:48.834807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.617748+0000","last_change":"2024-09-15T23:15:42.617748+0000","last_active":"2024-09-15T23:15:42.617748+0000","last_peered":"2024-09-15T23:15:42.617748+0000","last_clean":"2024-09-15T23:15:42.617748+0000","last_became_active":"2024-09-15T23:15:42.617320+0000","last_became_peered":"2024-09-15T23:15:42.617320+0000","last_unstale":"2024-09-15T23:15:42.617748+0000","last_undegraded":"2024-09-15T23:15:42.617748+0000","last_fullsized":"2024-09-15T23:15:42.617748+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:30:47.900049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,2],"acting":[6,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316911+0000","last_change":"2024-09-15T23:13:30.486969+0000","last_active":"2024-09-15T23:15:43.316911+0000","last_peered":"2024-09-15T23:15:43.316911+0000","last_clean":"2024-09-15T23:15:43.316911+0000","last_became_active":"2024-09-15T23:13:30.485899+0000","last_became_peered":"2024-09-15T23:13:30.485899+0000","last_unstale":"2024-09-15T23:15:43.316911+0000","last_undegraded":"2024-09-15T23:15:43.316911+0000","last_fullsized":"2024-09-15T23:15:43.316911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:50:16.329633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626588+0000","last_change":"2024-09-15T23:13:39.748545+0000","last_active":"2024-09-15T23:15:42.626588+0000","last_peered":"2024-09-15T23:15:42.626588+0000","last_clean":"2024-09-15T23:15:42.626588+0000","last_became_active":"2024-09-15T23:13:39.748099+0000","last_became_peered":"2024-09-15T23:13:39.748099+0000","last_unstale":"2024-09-15T23:15:42.626588+0000","last_undegraded":"2024-09-15T23:15:42.626588+0000","last_fullsized":"2024-09-15T23:15:42.626588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:50:29.478819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316962+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:43.316962+0000","last_peered":"2024-09-15T23:15:43.316962+0000","last_clean":"2024-09-15T23:15:43.316962+0000","last_became_active":"2024-09-15T23:13:32.514520+0000","last_became_peered":"2024-09-15T23:13:32.514520+0000","last_unstale":"2024-09-15T23:15:43.316962+0000","last_undegraded":"2024-09-15T23:15:43.316962+0000","last_fullsized":"2024-09-15T23:15:43.316962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:40.611599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"88'3","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612474+0000","last_change":"2024-09-15T23:13:30.486711+0000","last_active":"2024-09-15T23:15:42.612474+0000","last_peered":"2024-09-15T23:15:42.612474+0000","last_clean":"2024-09-15T23:15:42.612474+0000","last_became_active":"2024-09-15T23:13:30.486309+0000","last_became_peered":"2024-09-15T23:13:30.486309+0000","last_unstale":"2024-09-15T23:15:42.612474+0000","last_undegraded":"2024-09-15T23:15:42.612474+0000","last_fullsized":"2024-09-15T23:15:42.612474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:00:50.025630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317208+0000","last_change":"2024-09-15T23:12:20.293506+0000","last_active":"2024-09-15T23:15:43.317208+0000","last_peered":"2024-09-15T23:15:43.317208+0000","last_clean":"2024-09-15T23:15:43.317208+0000","last_became_active":"2024-09-15T23:12:20.293037+0000","last_became_peered":"2024-09-15T23:12:20.293037+0000","last_unstale":"2024-09-15T23:15:43.317208+0000","last_undegraded":"2024-09-15T23:15:43.317208+0000","last_fullsized":"2024-09-15T23:15:43.317208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:08:54.431171+0000","last_clean_scrub_stamp":"2024-09-15T23:08:54.431171+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:23:54.323469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"88'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317366+0000","last_change":"2024-09-15T23:13:39.548770+0000","last_active":"2024-09-15T23:15:43.317366+0000","last_peered":"2024-09-15T23:15:43.317366+0000","last_clean":"2024-09-15T23:15:43.317366+0000","last_became_active":"2024-09-15T23:13:39.547782+0000","last_became_peered":"2024-09-15T23:13:39.547782+0000","last_unstale":"2024-09-15T23:15:43.317366+0000","last_undegraded":"2024-09-15T23:15:43.317366+0000","last_fullsized":"2024-09-15T23:15:43.317366+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:59.834377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317493+0000","last_change":"2024-09-15T23:13:32.506663+0000","last_active":"2024-09-15T23:15:43.317493+0000","last_peered":"2024-09-15T23:15:43.317493+0000","last_clean":"2024-09-15T23:15:43.317493+0000","last_became_active":"2024-09-15T23:13:32.506311+0000","last_became_peered":"2024-09-15T23:13:32.506311+0000","last_unstale":"2024-09-15T23:15:43.317493+0000","last_undegraded":"2024-09-15T23:15:43.317493+0000","last_fullsized":"2024-09-15T23:15:43.317493+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:29:59.497769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'3","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614232+0000","last_change":"2024-09-15T23:13:30.598050+0000","last_active":"2024-09-15T23:15:42.614232+0000","last_peered":"2024-09-15T23:15:42.614232+0000","last_clean":"2024-09-15T23:15:42.614232+0000","last_became_active":"2024-09-15T23:13:30.597391+0000","last_became_peered":"2024-09-15T23:13:30.597391+0000","last_unstale":"2024-09-15T23:15:42.614232+0000","last_undegraded":"2024-09-15T23:15:42.614232+0000","last_fullsized":"2024-09-15T23:15:42.614232+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:42:27.724341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612481+0000","last_change":"2024-09-15T23:14:41.994950+0000","last_active":"2024-09-15T23:15:42.612481+0000","last_peered":"2024-09-15T23:15:42.612481+0000","last_clean":"2024-09-15T23:15:42.612481+0000","last_became_active":"2024-09-15T23:14:41.994429+0000","last_became_peered":"2024-09-15T23:14:41.994429+0000","last_unstale":"2024-09-15T23:15:42.612481+0000","last_undegraded":"2024-09-15T23:15:42.612481+0000","last_fullsized":"2024-09-15T23:15:42.612481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:28:24.056756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6,7],"acting":[1,10,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317374+0000","last_change":"2024-09-15T23:13:32.512821+0000","last_active":"2024-09-15T23:15:43.317374+0000","last_peered":"2024-09-15T23:15:43.317374+0000","last_clean":"2024-09-15T23:15:43.317374+0000","last_became_active":"2024-09-15T23:13:32.512242+0000","last_became_peered":"2024-09-15T23:13:32.512242+0000","last_unstale":"2024-09-15T23:15:43.317374+0000","last_undegraded":"2024-09-15T23:15:43.317374+0000","last_fullsized":"2024-09-15T23:15:43.317374+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:25.683110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.627013+0000","last_change":"2024-09-15T23:13:30.490302+0000","last_active":"2024-09-15T23:15:42.627013+0000","last_peered":"2024-09-15T23:15:42.627013+0000","last_clean":"2024-09-15T23:15:42.627013+0000","last_became_active":"2024-09-15T23:13:30.488844+0000","last_became_peered":"2024-09-15T23:13:30.488844+0000","last_unstale":"2024-09-15T23:15:42.627013+0000","last_undegraded":"2024-09-15T23:15:42.627013+0000","last_fullsized":"2024-09-15T23:15:42.627013+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:22:26.492373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627691+0000","last_change":"2024-09-15T23:14:41.640471+0000","last_active":"2024-09-15T23:15:43.627691+0000","last_peered":"2024-09-15T23:15:43.627691+0000","last_clean":"2024-09-15T23:15:43.627691+0000","last_became_active":"2024-09-15T23:14:41.640017+0000","last_became_peered":"2024-09-15T23:14:41.640017+0000","last_unstale":"2024-09-15T23:15:43.627691+0000","last_undegraded":"2024-09-15T23:15:43.627691+0000","last_fullsized":"2024-09-15T23:15:43.627691+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:29:58.842636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317463+0000","last_change":"2024-09-15T23:13:32.510914+0000","last_active":"2024-09-15T23:15:43.317463+0000","last_peered":"2024-09-15T23:15:43.317463+0000","last_clean":"2024-09-15T23:15:43.317463+0000","last_became_active":"2024-09-15T23:13:32.510555+0000","last_became_peered":"2024-09-15T23:13:32.510555+0000","last_unstale":"2024-09-15T23:15:43.317463+0000","last_undegraded":"2024-09-15T23:15:43.317463+0000","last_fullsized":"2024-09-15T23:15:43.317463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:43:45.120204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613353+0000","last_change":"2024-09-15T23:13:30.489360+0000","last_active":"2024-09-15T23:15:42.613353+0000","last_peered":"2024-09-15T23:15:42.613353+0000","last_clean":"2024-09-15T23:15:42.613353+0000","last_became_active":"2024-09-15T23:13:30.489025+0000","last_became_peered":"2024-09-15T23:13:30.489025+0000","last_unstale":"2024-09-15T23:15:42.613353+0000","last_undegraded":"2024-09-15T23:15:42.613353+0000","last_fullsized":"2024-09-15T23:15:42.613353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:12.231607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626135+0000","last_change":"2024-09-15T23:13:39.555773+0000","last_active":"2024-09-15T23:15:42.626135+0000","last_peered":"2024-09-15T23:15:42.626135+0000","last_clean":"2024-09-15T23:15:42.626135+0000","last_became_active":"2024-09-15T23:13:39.555330+0000","last_became_peered":"2024-09-15T23:13:39.555330+0000","last_unstale":"2024-09-15T23:15:42.626135+0000","last_undegraded":"2024-09-15T23:15:42.626135+0000","last_fullsized":"2024-09-15T23:15:42.626135+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:40:38.320071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910014+0000","last_change":"2024-09-15T23:13:32.509127+0000","last_active":"2024-09-15T23:15:42.910014+0000","last_peered":"2024-09-15T23:15:42.910014+0000","last_clean":"2024-09-15T23:15:42.910014+0000","last_became_active":"2024-09-15T23:13:32.508816+0000","last_became_peered":"2024-09-15T23:13:32.508816+0000","last_unstale":"2024-09-15T23:15:42.910014+0000","last_undegraded":"2024-09-15T23:15:42.910014+0000","last_fullsized":"2024-09-15T23:15:42.910014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:25:04.810091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"88'2","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610024+0000","last_change":"2024-09-15T23:13:30.486325+0000","last_active":"2024-09-15T23:15:42.610024+0000","last_peered":"2024-09-15T23:15:42.610024+0000","last_clean":"2024-09-15T23:15:42.610024+0000","last_became_active":"2024-09-15T23:13:30.485996+0000","last_became_peered":"2024-09-15T23:13:30.485996+0000","last_unstale":"2024-09-15T23:15:42.610024+0000","last_undegraded":"2024-09-15T23:15:42.610024+0000","last_fullsized":"2024-09-15T23:15:42.610024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:16:02.125949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626004+0000","last_change":"2024-09-15T23:13:39.548076+0000","last_active":"2024-09-15T23:15:42.626004+0000","last_peered":"2024-09-15T23:15:42.626004+0000","last_clean":"2024-09-15T23:15:42.626004+0000","last_became_active":"2024-09-15T23:13:39.547444+0000","last_became_peered":"2024-09-15T23:13:39.547444+0000","last_unstale":"2024-09-15T23:15:42.626004+0000","last_undegraded":"2024-09-15T23:15:42.626004+0000","last_fullsized":"2024-09-15T23:15:42.626004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:49:45.849102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626190+0000","last_change":"2024-09-15T23:13:32.512149+0000","last_active":"2024-09-15T23:15:42.626190+0000","last_peered":"2024-09-15T23:15:42.626190+0000","last_clean":"2024-09-15T23:15:42.626190+0000","last_became_active":"2024-09-15T23:13:32.511772+0000","last_became_peered":"2024-09-15T23:13:32.511772+0000","last_unstale":"2024-09-15T23:15:42.626190+0000","last_undegraded":"2024-09-15T23:15:42.626190+0000","last_fullsized":"2024-09-15T23:15:42.626190+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:47:16.689475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"88'15","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909569+0000","last_change":"2024-09-15T23:14:41.653786+0000","last_active":"2024-09-15T23:15:42.909569+0000","last_peered":"2024-09-15T23:15:42.909569+0000","last_clean":"2024-09-15T23:15:42.909569+0000","last_became_active":"2024-09-15T23:14:41.641472+0000","last_became_peered":"2024-09-15T23:14:41.641472+0000","last_unstale":"2024-09-15T23:15:42.909569+0000","last_undegraded":"2024-09-15T23:15:42.909569+0000","last_fullsized":"2024-09-15T23:15:42.909569+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:57:16.010142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613267+0000","last_change":"2024-09-15T23:13:39.549072+0000","last_active":"2024-09-15T23:15:42.613267+0000","last_peered":"2024-09-15T23:15:42.613267+0000","last_clean":"2024-09-15T23:15:42.613267+0000","last_became_active":"2024-09-15T23:13:39.548700+0000","last_became_peered":"2024-09-15T23:13:39.548700+0000","last_unstale":"2024-09-15T23:15:42.613267+0000","last_undegraded":"2024-09-15T23:15:42.613267+0000","last_fullsized":"2024-09-15T23:15:42.613267+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:56.902335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316843+0000","last_change":"2024-09-15T23:13:32.508311+0000","last_active":"2024-09-15T23:15:43.316843+0000","last_peered":"2024-09-15T23:15:43.316843+0000","last_clean":"2024-09-15T23:15:43.316843+0000","last_became_active":"2024-09-15T23:13:32.507651+0000","last_became_peered":"2024-09-15T23:13:32.507651+0000","last_unstale":"2024-09-15T23:15:43.316843+0000","last_undegraded":"2024-09-15T23:15:43.316843+0000","last_fullsized":"2024-09-15T23:15:43.316843+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:14.810293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'3","reported_seq":151,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:53.635521+0000","last_change":"2024-09-15T23:13:30.488852+0000","last_active":"2024-09-15T23:15:53.635521+0000","last_peered":"2024-09-15T23:15:53.635521+0000","last_clean":"2024-09-15T23:15:53.635521+0000","last_became_active":"2024-09-15T23:13:30.488547+0000","last_became_peered":"2024-09-15T23:13:30.488547+0000","last_unstale":"2024-09-15T23:15:53.635521+0000","last_undegraded":"2024-09-15T23:15:53.635521+0000","last_fullsized":"2024-09-15T23:15:53.635521+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:33:42.574621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609376+0000","last_change":"2024-09-15T23:13:39.546349+0000","last_active":"2024-09-15T23:15:42.609376+0000","last_peered":"2024-09-15T23:15:42.609376+0000","last_clean":"2024-09-15T23:15:42.609376+0000","last_became_active":"2024-09-15T23:13:39.545967+0000","last_became_peered":"2024-09-15T23:13:39.545967+0000","last_unstale":"2024-09-15T23:15:42.609376+0000","last_undegraded":"2024-09-15T23:15:42.609376+0000","last_fullsized":"2024-09-15T23:15:42.609376+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:28:31.081448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317127+0000","last_change":"2024-09-15T23:13:32.515726+0000","last_active":"2024-09-15T23:15:43.317127+0000","last_peered":"2024-09-15T23:15:43.317127+0000","last_clean":"2024-09-15T23:15:43.317127+0000","last_became_active":"2024-09-15T23:13:32.515426+0000","last_became_peered":"2024-09-15T23:13:32.515426+0000","last_unstale":"2024-09-15T23:15:43.317127+0000","last_undegraded":"2024-09-15T23:15:43.317127+0000","last_fullsized":"2024-09-15T23:15:43.317127+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:56:44.800997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"88'3","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626312+0000","last_change":"2024-09-15T23:13:30.490634+0000","last_active":"2024-09-15T23:15:42.626312+0000","last_peered":"2024-09-15T23:15:42.626312+0000","last_clean":"2024-09-15T23:15:42.626312+0000","last_became_active":"2024-09-15T23:13:30.488894+0000","last_became_peered":"2024-09-15T23:13:30.488894+0000","last_unstale":"2024-09-15T23:15:42.626312+0000","last_undegraded":"2024-09-15T23:15:42.626312+0000","last_fullsized":"2024-09-15T23:15:42.626312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:01:09.547911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909759+0000","last_change":"2024-09-15T23:13:39.552407+0000","last_active":"2024-09-15T23:15:42.909759+0000","last_peered":"2024-09-15T23:15:42.909759+0000","last_clean":"2024-09-15T23:15:42.909759+0000","last_became_active":"2024-09-15T23:13:39.552014+0000","last_became_peered":"2024-09-15T23:13:39.552014+0000","last_unstale":"2024-09-15T23:15:42.909759+0000","last_undegraded":"2024-09-15T23:15:42.909759+0000","last_fullsized":"2024-09-15T23:15:42.909759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:56:04.105370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612055+0000","last_change":"2024-09-15T23:13:32.509188+0000","last_active":"2024-09-15T23:15:42.612055+0000","last_peered":"2024-09-15T23:15:42.612055+0000","last_clean":"2024-09-15T23:15:42.612055+0000","last_became_active":"2024-09-15T23:13:32.508873+0000","last_became_peered":"2024-09-15T23:13:32.508873+0000","last_unstale":"2024-09-15T23:15:42.612055+0000","last_undegraded":"2024-09-15T23:15:42.612055+0000","last_fullsized":"2024-09-15T23:15:42.612055+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:31.844253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"88'6","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613972+0000","last_change":"2024-09-15T23:13:30.491911+0000","last_active":"2024-09-15T23:15:42.613972+0000","last_peered":"2024-09-15T23:15:42.613972+0000","last_clean":"2024-09-15T23:15:42.613972+0000","last_became_active":"2024-09-15T23:13:30.491535+0000","last_became_peered":"2024-09-15T23:13:30.491535+0000","last_unstale":"2024-09-15T23:15:42.613972+0000","last_undegraded":"2024-09-15T23:15:42.613972+0000","last_fullsized":"2024-09-15T23:15:42.613972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:10:35.004309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626638+0000","last_change":"2024-09-15T23:13:39.547675+0000","last_active":"2024-09-15T23:15:42.626638+0000","last_peered":"2024-09-15T23:15:42.626638+0000","last_clean":"2024-09-15T23:15:42.626638+0000","last_became_active":"2024-09-15T23:13:39.547252+0000","last_became_peered":"2024-09-15T23:13:39.547252+0000","last_unstale":"2024-09-15T23:15:42.626638+0000","last_undegraded":"2024-09-15T23:15:42.626638+0000","last_fullsized":"2024-09-15T23:15:42.626638+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:42:49.452151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609857+0000","last_change":"2024-09-15T23:13:32.516648+0000","last_active":"2024-09-15T23:15:42.609857+0000","last_peered":"2024-09-15T23:15:42.609857+0000","last_clean":"2024-09-15T23:15:42.609857+0000","last_became_active":"2024-09-15T23:13:32.516133+0000","last_became_peered":"2024-09-15T23:13:32.516133+0000","last_unstale":"2024-09-15T23:15:42.609857+0000","last_undegraded":"2024-09-15T23:15:42.609857+0000","last_fullsized":"2024-09-15T23:15:42.609857+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:11:07.156651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"92'38","reported_seq":307,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:55.644423+0000","last_change":"2024-09-15T23:13:30.492071+0000","last_active":"2024-09-15T23:15:55.644423+0000","last_peered":"2024-09-15T23:15:55.644423+0000","last_clean":"2024-09-15T23:15:55.644423+0000","last_became_active":"2024-09-15T23:13:30.491645+0000","last_became_peered":"2024-09-15T23:13:30.491645+0000","last_unstale":"2024-09-15T23:15:55.644423+0000","last_undegraded":"2024-09-15T23:15:55.644423+0000","last_fullsized":"2024-09-15T23:15:55.644423+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:40:09.613299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69999,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":230,"num_read_kb":185,"num_write":39,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609007+0000","last_change":"2024-09-15T23:13:39.551183+0000","last_active":"2024-09-15T23:15:42.609007+0000","last_peered":"2024-09-15T23:15:42.609007+0000","last_clean":"2024-09-15T23:15:42.609007+0000","last_became_active":"2024-09-15T23:13:39.550791+0000","last_became_peered":"2024-09-15T23:13:39.550791+0000","last_unstale":"2024-09-15T23:15:42.609007+0000","last_undegraded":"2024-09-15T23:15:42.609007+0000","last_fullsized":"2024-09-15T23:15:42.609007+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:49:44.142047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613046+0000","last_change":"2024-09-15T23:13:32.510721+0000","last_active":"2024-09-15T23:15:42.613046+0000","last_peered":"2024-09-15T23:15:42.613046+0000","last_clean":"2024-09-15T23:15:42.613046+0000","last_became_active":"2024-09-15T23:13:32.510426+0000","last_became_peered":"2024-09-15T23:13:32.510426+0000","last_unstale":"2024-09-15T23:15:42.613046+0000","last_undegraded":"2024-09-15T23:15:42.613046+0000","last_fullsized":"2024-09-15T23:15:42.613046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:45:01.768038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"88'4","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626979+0000","last_change":"2024-09-15T23:13:30.596739+0000","last_active":"2024-09-15T23:15:42.626979+0000","last_peered":"2024-09-15T23:15:42.626979+0000","last_clean":"2024-09-15T23:15:42.626979+0000","last_became_active":"2024-09-15T23:13:30.596422+0000","last_became_peered":"2024-09-15T23:13:30.596422+0000","last_unstale":"2024-09-15T23:15:42.626979+0000","last_undegraded":"2024-09-15T23:15:42.626979+0000","last_fullsized":"2024-09-15T23:15:42.626979+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:35:30.946301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612007+0000","last_change":"2024-09-15T23:14:41.641820+0000","last_active":"2024-09-15T23:15:42.612007+0000","last_peered":"2024-09-15T23:15:42.612007+0000","last_clean":"2024-09-15T23:15:42.612007+0000","last_became_active":"2024-09-15T23:14:41.641282+0000","last_became_peered":"2024-09-15T23:14:41.641282+0000","last_unstale":"2024-09-15T23:15:42.612007+0000","last_undegraded":"2024-09-15T23:15:42.612007+0000","last_fullsized":"2024-09-15T23:15:42.612007+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:49:43.472601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.628116+0000","last_change":"2024-09-15T23:13:32.511047+0000","last_active":"2024-09-15T23:15:43.628116+0000","last_peered":"2024-09-15T23:15:43.628116+0000","last_clean":"2024-09-15T23:15:43.628116+0000","last_became_active":"2024-09-15T23:13:32.510746+0000","last_became_peered":"2024-09-15T23:13:32.510746+0000","last_unstale":"2024-09-15T23:15:43.628116+0000","last_undegraded":"2024-09-15T23:15:43.628116+0000","last_fullsized":"2024-09-15T23:15:43.628116+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:35:06.925546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317463+0000","last_change":"2024-09-15T23:13:30.490794+0000","last_active":"2024-09-15T23:15:43.317463+0000","last_peered":"2024-09-15T23:15:43.317463+0000","last_clean":"2024-09-15T23:15:43.317463+0000","last_became_active":"2024-09-15T23:13:30.490441+0000","last_became_peered":"2024-09-15T23:13:30.490441+0000","last_unstale":"2024-09-15T23:15:43.317463+0000","last_undegraded":"2024-09-15T23:15:43.317463+0000","last_fullsized":"2024-09-15T23:15:43.317463+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:18.464726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.608807+0000","last_change":"2024-09-15T23:13:39.552475+0000","last_active":"2024-09-15T23:15:42.608807+0000","last_peered":"2024-09-15T23:15:42.608807+0000","last_clean":"2024-09-15T23:15:42.608807+0000","last_became_active":"2024-09-15T23:13:39.552106+0000","last_became_peered":"2024-09-15T23:13:39.552106+0000","last_unstale":"2024-09-15T23:15:42.608807+0000","last_undegraded":"2024-09-15T23:15:42.608807+0000","last_fullsized":"2024-09-15T23:15:42.608807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:25:37.674052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612577+0000","last_change":"2024-09-15T23:13:32.509021+0000","last_active":"2024-09-15T23:15:42.612577+0000","last_peered":"2024-09-15T23:15:42.612577+0000","last_clean":"2024-09-15T23:15:42.612577+0000","last_became_active":"2024-09-15T23:13:32.508684+0000","last_became_peered":"2024-09-15T23:13:32.508684+0000","last_unstale":"2024-09-15T23:15:42.612577+0000","last_undegraded":"2024-09-15T23:15:42.612577+0000","last_fullsized":"2024-09-15T23:15:42.612577+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:11:15.805067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":151,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:55.644146+0000","last_change":"2024-09-15T23:13:30.489844+0000","last_active":"2024-09-15T23:15:55.644146+0000","last_peered":"2024-09-15T23:15:55.644146+0000","last_clean":"2024-09-15T23:15:55.644146+0000","last_became_active":"2024-09-15T23:13:30.489473+0000","last_became_peered":"2024-09-15T23:13:30.489473+0000","last_unstale":"2024-09-15T23:15:55.644146+0000","last_undegraded":"2024-09-15T23:15:55.644146+0000","last_fullsized":"2024-09-15T23:15:55.644146+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:13:37.546103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316758+0000","last_change":"2024-09-15T23:13:39.547563+0000","last_active":"2024-09-15T23:15:43.316758+0000","last_peered":"2024-09-15T23:15:43.316758+0000","last_clean":"2024-09-15T23:15:43.316758+0000","last_became_active":"2024-09-15T23:13:39.547221+0000","last_became_peered":"2024-09-15T23:13:39.547221+0000","last_unstale":"2024-09-15T23:15:43.316758+0000","last_undegraded":"2024-09-15T23:15:43.316758+0000","last_fullsized":"2024-09-15T23:15:43.316758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:35:08.042961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627914+0000","last_change":"2024-09-15T23:13:39.552869+0000","last_active":"2024-09-15T23:15:43.627914+0000","last_peered":"2024-09-15T23:15:43.627914+0000","last_clean":"2024-09-15T23:15:43.627914+0000","last_became_active":"2024-09-15T23:13:39.552378+0000","last_became_peered":"2024-09-15T23:13:39.552378+0000","last_unstale":"2024-09-15T23:15:43.627914+0000","last_undegraded":"2024-09-15T23:15:43.627914+0000","last_fullsized":"2024-09-15T23:15:43.627914+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:10.467369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609033+0000","last_change":"2024-09-15T23:13:32.501233+0000","last_active":"2024-09-15T23:15:42.609033+0000","last_peered":"2024-09-15T23:15:42.609033+0000","last_clean":"2024-09-15T23:15:42.609033+0000","last_became_active":"2024-09-15T23:13:32.500888+0000","last_became_peered":"2024-09-15T23:13:32.500888+0000","last_unstale":"2024-09-15T23:15:42.609033+0000","last_undegraded":"2024-09-15T23:15:42.609033+0000","last_fullsized":"2024-09-15T23:15:42.609033+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:56:48.764187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"88'1","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626421+0000","last_change":"2024-09-15T23:13:30.490733+0000","last_active":"2024-09-15T23:15:42.626421+0000","last_peered":"2024-09-15T23:15:42.626421+0000","last_clean":"2024-09-15T23:15:42.626421+0000","last_became_active":"2024-09-15T23:13:30.490373+0000","last_became_peered":"2024-09-15T23:13:30.490373+0000","last_unstale":"2024-09-15T23:15:42.626421+0000","last_undegraded":"2024-09-15T23:15:42.626421+0000","last_fullsized":"2024-09-15T23:15:42.626421+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:16:34.191840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":26,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910018+0000","last_change":"2024-09-15T23:13:39.553643+0000","last_active":"2024-09-15T23:15:42.910018+0000","last_peered":"2024-09-15T23:15:42.910018+0000","last_clean":"2024-09-15T23:15:42.910018+0000","last_became_active":"2024-09-15T23:13:39.553259+0000","last_became_peered":"2024-09-15T23:13:39.553259+0000","last_unstale":"2024-09-15T23:15:42.910018+0000","last_undegraded":"2024-09-15T23:15:42.910018+0000","last_fullsized":"2024-09-15T23:15:42.910018+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:24.616605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316784+0000","last_change":"2024-09-15T23:13:32.506304+0000","last_active":"2024-09-15T23:15:43.316784+0000","last_peered":"2024-09-15T23:15:43.316784+0000","last_clean":"2024-09-15T23:15:43.316784+0000","last_became_active":"2024-09-15T23:13:32.505116+0000","last_became_peered":"2024-09-15T23:13:32.505116+0000","last_unstale":"2024-09-15T23:15:43.316784+0000","last_undegraded":"2024-09-15T23:15:43.316784+0000","last_fullsized":"2024-09-15T23:15:43.316784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:14:14.290284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"88'2","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.611743+0000","last_change":"2024-09-15T23:13:30.484913+0000","last_active":"2024-09-15T23:15:42.611743+0000","last_peered":"2024-09-15T23:15:42.611743+0000","last_clean":"2024-09-15T23:15:42.611743+0000","last_became_active":"2024-09-15T23:13:30.484573+0000","last_became_peered":"2024-09-15T23:13:30.484573+0000","last_unstale":"2024-09-15T23:15:42.611743+0000","last_undegraded":"2024-09-15T23:15:42.611743+0000","last_fullsized":"2024-09-15T23:15:42.611743+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:58:56.815793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"88'3","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610033+0000","last_change":"2024-09-15T23:13:30.489731+0000","last_active":"2024-09-15T23:15:42.610033+0000","last_peered":"2024-09-15T23:15:42.610033+0000","last_clean":"2024-09-15T23:15:42.610033+0000","last_became_active":"2024-09-15T23:13:30.489387+0000","last_became_peered":"2024-09-15T23:13:30.489387+0000","last_unstale":"2024-09-15T23:15:42.610033+0000","last_undegraded":"2024-09-15T23:15:42.610033+0000","last_fullsized":"2024-09-15T23:15:42.610033+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626302+0000","last_change":"2024-09-15T23:13:32.510220+0000","last_active":"2024-09-15T23:15:42.626302+0000","last_peered":"2024-09-15T23:15:42.626302+0000","last_clean":"2024-09-15T23:15:42.626302+0000","last_became_active":"2024-09-15T23:13:32.509726+0000","last_became_peered":"2024-09-15T23:13:32.509726+0000","last_unstale":"2024-09-15T23:15:42.626302+0000","last_undegraded":"2024-09-15T23:15:42.626302+0000","last_fullsized":"2024-09-15T23:15:42.626302+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:43:55.192392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613815+0000","last_change":"2024-09-15T23:13:39.555646+0000","last_active":"2024-09-15T23:15:42.613815+0000","last_peered":"2024-09-15T23:15:42.613815+0000","last_clean":"2024-09-15T23:15:42.613815+0000","last_became_active":"2024-09-15T23:13:39.555270+0000","last_became_peered":"2024-09-15T23:13:39.555270+0000","last_unstale":"2024-09-15T23:15:42.613815+0000","last_undegraded":"2024-09-15T23:15:42.613815+0000","last_fullsized":"2024-09-15T23:15:42.613815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:55:12.132017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'7","reported_seq":157,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:54.643249+0000","last_change":"2024-09-15T23:13:30.490535+0000","last_active":"2024-09-15T23:15:54.643249+0000","last_peered":"2024-09-15T23:15:54.643249+0000","last_clean":"2024-09-15T23:15:54.643249+0000","last_became_active":"2024-09-15T23:13:30.490172+0000","last_became_peered":"2024-09-15T23:13:30.490172+0000","last_unstale":"2024-09-15T23:15:54.643249+0000","last_undegraded":"2024-09-15T23:15:54.643249+0000","last_fullsized":"2024-09-15T23:15:54.643249+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:29.470198+0000","last_clean_scrub_stamp":"2024-09-15T23:13:29.470198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:53:37.715888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1156,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609500+0000","last_change":"2024-09-15T23:13:32.503903+0000","last_active":"2024-09-15T23:15:42.609500+0000","last_peered":"2024-09-15T23:15:42.609500+0000","last_clean":"2024-09-15T23:15:42.609500+0000","last_became_active":"2024-09-15T23:13:32.503501+0000","last_became_peered":"2024-09-15T23:13:32.503501+0000","last_unstale":"2024-09-15T23:15:42.609500+0000","last_undegraded":"2024-09-15T23:15:42.609500+0000","last_fullsized":"2024-09-15T23:15:42.609500+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:32:55.653633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613805+0000","last_change":"2024-09-15T23:13:39.554050+0000","last_active":"2024-09-15T23:15:42.613805+0000","last_peered":"2024-09-15T23:15:42.613805+0000","last_clean":"2024-09-15T23:15:42.613805+0000","last_became_active":"2024-09-15T23:13:39.553087+0000","last_became_peered":"2024-09-15T23:13:39.553087+0000","last_unstale":"2024-09-15T23:15:42.613805+0000","last_undegraded":"2024-09-15T23:15:42.613805+0000","last_fullsized":"2024-09-15T23:15:42.613805+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:38.525850+0000","last_clean_scrub_stamp":"2024-09-15T23:13:38.525850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:17:19.831032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909660+0000","last_change":"2024-09-15T23:13:32.505518+0000","last_active":"2024-09-15T23:15:42.909660+0000","last_peered":"2024-09-15T23:15:42.909660+0000","last_clean":"2024-09-15T23:15:42.909660+0000","last_became_active":"2024-09-15T23:13:32.504763+0000","last_became_peered":"2024-09-15T23:13:32.504763+0000","last_unstale":"2024-09-15T23:15:42.909660+0000","last_undegraded":"2024-09-15T23:15:42.909660+0000","last_fullsized":"2024-09-15T23:15:42.909660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:45:44.191272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614459+0000","last_change":"2024-09-15T23:13:32.514878+0000","last_active":"2024-09-15T23:15:42.614459+0000","last_peered":"2024-09-15T23:15:42.614459+0000","last_clean":"2024-09-15T23:15:42.614459+0000","last_became_active":"2024-09-15T23:13:32.512787+0000","last_became_peered":"2024-09-15T23:13:32.512787+0000","last_unstale":"2024-09-15T23:15:42.614459+0000","last_undegraded":"2024-09-15T23:15:42.614459+0000","last_fullsized":"2024-09-15T23:15:42.614459+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:42.772599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613647+0000","last_change":"2024-09-15T23:13:32.514522+0000","last_active":"2024-09-15T23:15:42.613647+0000","last_peered":"2024-09-15T23:15:42.613647+0000","last_clean":"2024-09-15T23:15:42.613647+0000","last_became_active":"2024-09-15T23:13:32.512854+0000","last_became_peered":"2024-09-15T23:13:32.512854+0000","last_unstale":"2024-09-15T23:15:42.613647+0000","last_undegraded":"2024-09-15T23:15:42.613647+0000","last_fullsized":"2024-09-15T23:15:42.613647+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:42:54.678090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627750+0000","last_change":"2024-09-15T23:13:32.510127+0000","last_active":"2024-09-15T23:15:43.627750+0000","last_peered":"2024-09-15T23:15:43.627750+0000","last_clean":"2024-09-15T23:15:43.627750+0000","last_became_active":"2024-09-15T23:13:32.509804+0000","last_became_peered":"2024-09-15T23:13:32.509804+0000","last_unstale":"2024-09-15T23:15:43.627750+0000","last_undegraded":"2024-09-15T23:15:43.627750+0000","last_fullsized":"2024-09-15T23:15:43.627750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:51:55.842078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612897+0000","last_change":"2024-09-15T23:14:42.010010+0000","last_active":"2024-09-15T23:15:42.612897+0000","last_peered":"2024-09-15T23:15:42.612897+0000","last_clean":"2024-09-15T23:15:42.612897+0000","last_became_active":"2024-09-15T23:14:42.009603+0000","last_became_peered":"2024-09-15T23:14:42.009603+0000","last_unstale":"2024-09-15T23:15:42.612897+0000","last_undegraded":"2024-09-15T23:15:42.612897+0000","last_fullsized":"2024-09-15T23:15:42.612897+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:34:51.849548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910096+0000","last_change":"2024-09-15T23:13:32.503129+0000","last_active":"2024-09-15T23:15:42.910096+0000","last_peered":"2024-09-15T23:15:42.910096+0000","last_clean":"2024-09-15T23:15:42.910096+0000","last_became_active":"2024-09-15T23:13:32.502794+0000","last_became_peered":"2024-09-15T23:13:32.502794+0000","last_unstale":"2024-09-15T23:15:42.910096+0000","last_undegraded":"2024-09-15T23:15:42.910096+0000","last_fullsized":"2024-09-15T23:15:42.910096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:12:15.519842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614200+0000","last_change":"2024-09-15T23:13:32.511261+0000","last_active":"2024-09-15T23:15:42.614200+0000","last_peered":"2024-09-15T23:15:42.614200+0000","last_clean":"2024-09-15T23:15:42.614200+0000","last_became_active":"2024-09-15T23:13:32.510748+0000","last_became_peered":"2024-09-15T23:13:32.510748+0000","last_unstale":"2024-09-15T23:15:42.614200+0000","last_undegraded":"2024-09-15T23:15:42.614200+0000","last_fullsized":"2024-09-15T23:15:42.614200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:14:45.405347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317395+0000","last_change":"2024-09-15T23:13:32.512226+0000","last_active":"2024-09-15T23:15:43.317395+0000","last_peered":"2024-09-15T23:15:43.317395+0000","last_clean":"2024-09-15T23:15:43.317395+0000","last_became_active":"2024-09-15T23:13:32.511897+0000","last_became_peered":"2024-09-15T23:13:32.511897+0000","last_unstale":"2024-09-15T23:15:43.317395+0000","last_undegraded":"2024-09-15T23:15:43.317395+0000","last_fullsized":"2024-09-15T23:15:43.317395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:55.592980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.626113+0000","last_change":"2024-09-15T23:13:32.513528+0000","last_active":"2024-09-15T23:15:42.626113+0000","last_peered":"2024-09-15T23:15:42.626113+0000","last_clean":"2024-09-15T23:15:42.626113+0000","last_became_active":"2024-09-15T23:13:32.513182+0000","last_became_peered":"2024-09-15T23:13:32.513182+0000","last_unstale":"2024-09-15T23:15:42.626113+0000","last_undegraded":"2024-09-15T23:15:42.626113+0000","last_fullsized":"2024-09-15T23:15:42.626113+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:27:59.405944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316925+0000","last_change":"2024-09-15T23:13:32.508241+0000","last_active":"2024-09-15T23:15:43.316925+0000","last_peered":"2024-09-15T23:15:43.316925+0000","last_clean":"2024-09-15T23:15:43.316925+0000","last_became_active":"2024-09-15T23:13:32.508002+0000","last_became_peered":"2024-09-15T23:13:32.508002+0000","last_unstale":"2024-09-15T23:15:43.316925+0000","last_undegraded":"2024-09-15T23:15:43.316925+0000","last_fullsized":"2024-09-15T23:15:43.316925+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:11:45.372282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612732+0000","last_change":"2024-09-15T23:13:32.514834+0000","last_active":"2024-09-15T23:15:42.612732+0000","last_peered":"2024-09-15T23:15:42.612732+0000","last_clean":"2024-09-15T23:15:42.612732+0000","last_became_active":"2024-09-15T23:13:32.514476+0000","last_became_peered":"2024-09-15T23:13:32.514476+0000","last_unstale":"2024-09-15T23:15:42.612732+0000","last_undegraded":"2024-09-15T23:15:42.612732+0000","last_fullsized":"2024-09-15T23:15:42.612732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:34:12.356436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612638+0000","last_change":"2024-09-15T23:13:32.510448+0000","last_active":"2024-09-15T23:15:42.612638+0000","last_peered":"2024-09-15T23:15:42.612638+0000","last_clean":"2024-09-15T23:15:42.612638+0000","last_became_active":"2024-09-15T23:13:32.510143+0000","last_became_peered":"2024-09-15T23:13:32.510143+0000","last_unstale":"2024-09-15T23:15:42.612638+0000","last_undegraded":"2024-09-15T23:15:42.612638+0000","last_fullsized":"2024-09-15T23:15:42.612638+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:07:34.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.909941+0000","last_change":"2024-09-15T23:13:32.511208+0000","last_active":"2024-09-15T23:15:42.909941+0000","last_peered":"2024-09-15T23:15:42.909941+0000","last_clean":"2024-09-15T23:15:42.909941+0000","last_became_active":"2024-09-15T23:13:32.510908+0000","last_became_peered":"2024-09-15T23:13:32.510908+0000","last_unstale":"2024-09-15T23:15:42.909941+0000","last_undegraded":"2024-09-15T23:15:42.909941+0000","last_fullsized":"2024-09-15T23:15:42.909941+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:35:13.664420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.614098+0000","last_change":"2024-09-15T23:13:32.513052+0000","last_active":"2024-09-15T23:15:42.614098+0000","last_peered":"2024-09-15T23:15:42.614098+0000","last_clean":"2024-09-15T23:15:42.614098+0000","last_became_active":"2024-09-15T23:13:32.512681+0000","last_became_peered":"2024-09-15T23:13:32.512681+0000","last_unstale":"2024-09-15T23:15:42.614098+0000","last_undegraded":"2024-09-15T23:15:42.614098+0000","last_fullsized":"2024-09-15T23:15:42.614098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:20:28.480979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610060+0000","last_change":"2024-09-15T23:13:32.501958+0000","last_active":"2024-09-15T23:15:42.610060+0000","last_peered":"2024-09-15T23:15:42.610060+0000","last_clean":"2024-09-15T23:15:42.610060+0000","last_became_active":"2024-09-15T23:13:32.501322+0000","last_became_peered":"2024-09-15T23:13:32.501322+0000","last_unstale":"2024-09-15T23:15:42.610060+0000","last_undegraded":"2024-09-15T23:15:42.610060+0000","last_fullsized":"2024-09-15T23:15:42.610060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:32:28.315317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910207+0000","last_change":"2024-09-15T23:13:32.504179+0000","last_active":"2024-09-15T23:15:42.910207+0000","last_peered":"2024-09-15T23:15:42.910207+0000","last_clean":"2024-09-15T23:15:42.910207+0000","last_became_active":"2024-09-15T23:13:32.503860+0000","last_became_peered":"2024-09-15T23:13:32.503860+0000","last_unstale":"2024-09-15T23:15:42.910207+0000","last_undegraded":"2024-09-15T23:15:42.910207+0000","last_fullsized":"2024-09-15T23:15:42.910207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:50:20.060412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612963+0000","last_change":"2024-09-15T23:13:32.516061+0000","last_active":"2024-09-15T23:15:42.612963+0000","last_peered":"2024-09-15T23:15:42.612963+0000","last_clean":"2024-09-15T23:15:42.612963+0000","last_became_active":"2024-09-15T23:13:32.514006+0000","last_became_peered":"2024-09-15T23:13:32.514006+0000","last_unstale":"2024-09-15T23:15:42.612963+0000","last_undegraded":"2024-09-15T23:15:42.612963+0000","last_fullsized":"2024-09-15T23:15:42.612963+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:29:46.945042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627665+0000","last_change":"2024-09-15T23:13:32.511159+0000","last_active":"2024-09-15T23:15:43.627665+0000","last_peered":"2024-09-15T23:15:43.627665+0000","last_clean":"2024-09-15T23:15:43.627665+0000","last_became_active":"2024-09-15T23:13:32.510794+0000","last_became_peered":"2024-09-15T23:13:32.510794+0000","last_unstale":"2024-09-15T23:15:43.627665+0000","last_undegraded":"2024-09-15T23:15:43.627665+0000","last_fullsized":"2024-09-15T23:15:43.627665+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:18:21.376854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627685+0000","last_change":"2024-09-15T23:13:32.503299+0000","last_active":"2024-09-15T23:15:43.627685+0000","last_peered":"2024-09-15T23:15:43.627685+0000","last_clean":"2024-09-15T23:15:43.627685+0000","last_became_active":"2024-09-15T23:13:32.502790+0000","last_became_peered":"2024-09-15T23:13:32.502790+0000","last_unstale":"2024-09-15T23:15:43.627685+0000","last_undegraded":"2024-09-15T23:15:43.627685+0000","last_fullsized":"2024-09-15T23:15:43.627685+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:08.908008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612501+0000","last_change":"2024-09-15T23:13:32.501725+0000","last_active":"2024-09-15T23:15:42.612501+0000","last_peered":"2024-09-15T23:15:42.612501+0000","last_clean":"2024-09-15T23:15:42.612501+0000","last_became_active":"2024-09-15T23:13:32.501400+0000","last_became_peered":"2024-09-15T23:13:32.501400+0000","last_unstale":"2024-09-15T23:15:42.612501+0000","last_undegraded":"2024-09-15T23:15:42.612501+0000","last_fullsized":"2024-09-15T23:15:42.612501+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:45:45.717360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612646+0000","last_change":"2024-09-15T23:13:32.511759+0000","last_active":"2024-09-15T23:15:42.612646+0000","last_peered":"2024-09-15T23:15:42.612646+0000","last_clean":"2024-09-15T23:15:42.612646+0000","last_became_active":"2024-09-15T23:13:32.511123+0000","last_became_peered":"2024-09-15T23:13:32.511123+0000","last_unstale":"2024-09-15T23:15:42.612646+0000","last_undegraded":"2024-09-15T23:15:42.612646+0000","last_fullsized":"2024-09-15T23:15:42.612646+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:13.559609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.317377+0000","last_change":"2024-09-15T23:13:32.513953+0000","last_active":"2024-09-15T23:15:43.317377+0000","last_peered":"2024-09-15T23:15:43.317377+0000","last_clean":"2024-09-15T23:15:43.317377+0000","last_became_active":"2024-09-15T23:13:32.513608+0000","last_became_peered":"2024-09-15T23:13:32.513608+0000","last_unstale":"2024-09-15T23:15:43.317377+0000","last_undegraded":"2024-09-15T23:15:43.317377+0000","last_fullsized":"2024-09-15T23:15:43.317377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:06:28.129609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610788+0000","last_change":"2024-09-15T23:13:32.510056+0000","last_active":"2024-09-15T23:15:42.610788+0000","last_peered":"2024-09-15T23:15:42.610788+0000","last_clean":"2024-09-15T23:15:42.610788+0000","last_became_active":"2024-09-15T23:13:32.509661+0000","last_became_peered":"2024-09-15T23:13:32.509661+0000","last_unstale":"2024-09-15T23:15:42.610788+0000","last_undegraded":"2024-09-15T23:15:42.610788+0000","last_fullsized":"2024-09-15T23:15:42.610788+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:27.946358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":44,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.613153+0000","last_change":"2024-09-15T23:13:32.506323+0000","last_active":"2024-09-15T23:15:42.613153+0000","last_peered":"2024-09-15T23:15:42.613153+0000","last_clean":"2024-09-15T23:15:42.613153+0000","last_became_active":"2024-09-15T23:13:32.505989+0000","last_became_peered":"2024-09-15T23:13:32.505989+0000","last_unstale":"2024-09-15T23:15:42.613153+0000","last_undegraded":"2024-09-15T23:15:42.613153+0000","last_fullsized":"2024-09-15T23:15:42.613153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:53:00.149575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316763+0000","last_change":"2024-09-15T23:13:32.510005+0000","last_active":"2024-09-15T23:15:43.316763+0000","last_peered":"2024-09-15T23:15:43.316763+0000","last_clean":"2024-09-15T23:15:43.316763+0000","last_became_active":"2024-09-15T23:13:32.509685+0000","last_became_peered":"2024-09-15T23:13:32.509685+0000","last_unstale":"2024-09-15T23:15:43.316763+0000","last_undegraded":"2024-09-15T23:15:43.316763+0000","last_fullsized":"2024-09-15T23:15:43.316763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:28:59.297990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.316851+0000","last_change":"2024-09-15T23:13:32.509225+0000","last_active":"2024-09-15T23:15:43.316851+0000","last_peered":"2024-09-15T23:15:43.316851+0000","last_clean":"2024-09-15T23:15:43.316851+0000","last_became_active":"2024-09-15T23:13:32.508814+0000","last_became_peered":"2024-09-15T23:13:32.508814+0000","last_unstale":"2024-09-15T23:15:43.316851+0000","last_undegraded":"2024-09-15T23:15:43.316851+0000","last_fullsized":"2024-09-15T23:15:43.316851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:03:47.410625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.610204+0000","last_change":"2024-09-15T23:13:32.501576+0000","last_active":"2024-09-15T23:15:42.610204+0000","last_peered":"2024-09-15T23:15:42.610204+0000","last_clean":"2024-09-15T23:15:42.610204+0000","last_became_active":"2024-09-15T23:13:32.500980+0000","last_became_peered":"2024-09-15T23:13:32.500980+0000","last_unstale":"2024-09-15T23:15:42.610204+0000","last_undegraded":"2024-09-15T23:15:42.610204+0000","last_fullsized":"2024-09-15T23:15:42.610204+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:40:17.034683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.609588+0000","last_change":"2024-09-15T23:13:32.508658+0000","last_active":"2024-09-15T23:15:42.609588+0000","last_peered":"2024-09-15T23:15:42.609588+0000","last_clean":"2024-09-15T23:15:42.609588+0000","last_became_active":"2024-09-15T23:13:32.507566+0000","last_became_peered":"2024-09-15T23:13:32.507566+0000","last_unstale":"2024-09-15T23:15:42.609588+0000","last_undegraded":"2024-09-15T23:15:42.609588+0000","last_fullsized":"2024-09-15T23:15:42.609588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:26.341164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.612535+0000","last_change":"2024-09-15T23:13:32.512078+0000","last_active":"2024-09-15T23:15:42.612535+0000","last_peered":"2024-09-15T23:15:42.612535+0000","last_clean":"2024-09-15T23:15:42.612535+0000","last_became_active":"2024-09-15T23:13:32.511497+0000","last_became_peered":"2024-09-15T23:13:32.511497+0000","last_unstale":"2024-09-15T23:15:42.612535+0000","last_undegraded":"2024-09-15T23:15:42.612535+0000","last_fullsized":"2024-09-15T23:15:42.612535+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:48:42.998705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627888+0000","last_change":"2024-09-15T23:13:32.505347+0000","last_active":"2024-09-15T23:15:43.627888+0000","last_peered":"2024-09-15T23:15:43.627888+0000","last_clean":"2024-09-15T23:15:43.627888+0000","last_became_active":"2024-09-15T23:13:32.505047+0000","last_became_peered":"2024-09-15T23:13:32.505047+0000","last_unstale":"2024-09-15T23:15:43.627888+0000","last_undegraded":"2024-09-15T23:15:43.627888+0000","last_fullsized":"2024-09-15T23:15:43.627888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:30:43.434698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:43.627940+0000","last_change":"2024-09-15T23:13:32.504567+0000","last_active":"2024-09-15T23:15:43.627940+0000","last_peered":"2024-09-15T23:15:43.627940+0000","last_clean":"2024-09-15T23:15:43.627940+0000","last_became_active":"2024-09-15T23:13:32.504217+0000","last_became_peered":"2024-09-15T23:13:32.504217+0000","last_unstale":"2024-09-15T23:15:43.627940+0000","last_undegraded":"2024-09-15T23:15:43.627940+0000","last_fullsized":"2024-09-15T23:15:43.627940+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:43:47.216427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-15T23:15:42.910172+0000","last_change":"2024-09-15T23:13:32.504210+0000","last_active":"2024-09-15T23:15:42.910172+0000","last_peered":"2024-09-15T23:15:42.910172+0000","last_clean":"2024-09-15T23:15:42.910172+0000","last_became_active":"2024-09-15T23:13:32.503886+0000","last_became_peered":"2024-09-15T23:13:32.503886+0000","last_unstale":"2024-09-15T23:15:42.910172+0000","last_undegraded":"2024-09-15T23:15:42.910172+0000","last_fullsized":"2024-09-15T23:15:42.910172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T23:13:31.484092+0000","last_clean_scrub_stamp":"2024-09-15T23:13:31.484092+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:45:28.703344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74143,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":680,"num_read_kb":420,"num_write":180,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":112,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":222429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153,"ondisk_log_size":153,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245431,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514538,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522238,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":69,"seq":296352743471,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32320,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712832,"statfs":{"total":95995035648,"available":95961939968,"internally_reserved":0,"allocated":1163264,"data_stored":1179393,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":64,"seq":274877906997,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1032192,"data_stored":526458,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":57,"seq":244813135930,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":522170,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364862,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1077248,"data_stored":1118334,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528388,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1007616,"data_stored":517538,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32252,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712900,"statfs":{"total":95995035648,"available":95962009600,"internally_reserved":0,"allocated":1085440,"data_stored":1113922,"data_compressed":65297,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1019904,"data_stored":522188,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182482,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1019904,"data_stored":517732,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":589331,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":587447,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1032192,"data_stored":518926,"data_compressed":29533,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34125,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35764,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":69999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":71883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":71577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T23:15:59.105 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:15:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:58 smithi067 ceph-mon[35665]: pgmap v361: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 7 op/s 2024-09-15T23:15:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:15:58 smithi067 ceph-mon[35665]: from='client.15357 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-15T23:15:59.362 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.830+0000 7f0044e8f640 1 -- 172.21.15.62:0/1534622281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 msgr2=0x7f004010e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.830+0000 7f0044e8f640 1 --2- 172.21.15.62:0/1534622281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 0x7f004010e610 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0038009ca0 tx=0x7f003802a950 comp rx=0 tx=0).stop 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.830+0000 7f0044e8f640 1 -- 172.21.15.62:0/1534622281 shutdown_connections 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.830+0000 7f0044e8f640 1 --2- 172.21.15.62:0/1534622281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 0x7f004010e610 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.830+0000 7f0044e8f640 1 --2- 172.21.15.62:0/1534622281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0040103700 0x7f004010bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.830+0000 7f0044e8f640 1 --2- 172.21.15.62:0/1534622281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0040102de0 0x7f00401031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.830+0000 7f0044e8f640 1 -- 172.21.15.62:0/1534622281 >> 172.21.15.62:0/1534622281 conn(0x7f00400fc7a0 msgr2=0x7f00400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.830+0000 7f0044e8f640 1 -- 172.21.15.62:0/1534622281 shutdown_connections 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.830+0000 7f0044e8f640 1 -- 172.21.15.62:0/1534622281 wait complete. 2024-09-15T23:15:59.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f0044e8f640 1 Processor -- start 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f0044e8f640 1 -- start start 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f0044e8f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0040102de0 0x7f0040198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f0044e8f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0040103700 0x7f0040198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f0044e8f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 0x7f004019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f0044e8f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0040111fb0 con 0x7f004010c180 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f0044e8f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0040111e30 con 0x7f0040102de0 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f0044e8f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0040112130 con 0x7f0040103700 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 0x7f004019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f003ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 0x7f004019cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42748/0 (socket says 172.21.15.62:42748) 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f003ed9d640 1 -- 172.21.15.62:0/3374708650 learned_addr learned my addr 172.21.15.62:0/3374708650 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f003e59c640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0040102de0 0x7f0040198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.831+0000 7f003ed9d640 1 -- 172.21.15.62:0/3374708650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0040103700 msgr2=0x7f0040198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:59.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f003ed9d640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0040103700 0x7f0040198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f003ed9d640 1 -- 172.21.15.62:0/3374708650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0040102de0 msgr2=0x7f0040198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f003ed9d640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0040102de0 0x7f0040198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f003ed9d640 1 -- 172.21.15.62:0/3374708650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004019d400 con 0x7f004010c180 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f003ed9d640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 0x7f004019cd20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f003802ae70 tx=0x7f003802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f002f7fe640 1 -- 172.21.15.62:0/3374708650 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0038034040 con 0x7f004010c180 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f002f7fe640 1 -- 172.21.15.62:0/3374708650 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0038030920 con 0x7f004010c180 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f004019d690 con 0x7f004010c180 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f002f7fe640 1 -- 172.21.15.62:0/3374708650 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00380047b0 con 0x7f004010c180 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.832+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00401a4fd0 con 0x7f004010c180 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.833+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f004006afd0 con 0x7f004010c180 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.834+0000 7f002f7fe640 1 -- 172.21.15.62:0/3374708650 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0038037020 con 0x7f004010c180 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.835+0000 7f002f7fe640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00280a3240 0x7f00280a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:15:59.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.835+0000 7f003e59c640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00280a3240 0x7f00280a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:15:59.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.835+0000 7f002f7fe640 1 -- 172.21.15.62:0/3374708650 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f00380f0b10 con 0x7f004010c180 2024-09-15T23:15:59.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.836+0000 7f003e59c640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00280a3240 0x7f00280a5700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0030009990 tx=0x7f0030009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:15:59.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:15:59.840+0000 7f002f7fe640 1 -- 172.21.15.62:0/3374708650 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f00380bd4b0 con 0x7f004010c180 2024-09-15T23:16:00.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.054+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0040107270 con 0x7f004010c180 2024-09-15T23:16:00.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.054+0000 7f002f7fe640 1 -- 172.21.15.62:0/3374708650 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7f00380bcdb0 con 0x7f004010c180 2024-09-15T23:16:00.056 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:00.056 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"btime":"2024-09-15T23:15:44:620975+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:15:43.799068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:16:00.056 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-09-15T23:16:00.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.056+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00280a3240 msgr2=0x7f00280a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:00.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.056+0000 7f0044e8f640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00280a3240 0x7f00280a5700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0030009990 tx=0x7f0030009110 comp rx=0 tx=0).stop 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.056+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 msgr2=0x7f004019cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.056+0000 7f0044e8f640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 0x7f004019cd20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f003802ae70 tx=0x7f003802f5b0 comp rx=0 tx=0).stop 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.057+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 shutdown_connections 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.057+0000 7f0044e8f640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00280a3240 0x7f00280a5700 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.057+0000 7f0044e8f640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f004010c180 0x7f004019cd20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.057+0000 7f0044e8f640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0040103700 0x7f0040198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.057+0000 7f0044e8f640 1 --2- 172.21.15.62:0/3374708650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0040102de0 0x7f0040198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.057+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 >> 172.21.15.62:0/3374708650 conn(0x7f00400fc7a0 msgr2=0x7f00400fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.057+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 shutdown_connections 2024-09-15T23:16:00.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:00.057+0000 7f0044e8f640 1 -- 172.21.15.62:0/3374708650 wait complete. 2024-09-15T23:16:00.691 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:16:00.950 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:01.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:00 smithi067 ceph-mon[35665]: pgmap v362: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 682 B/s wr, 5 op/s 2024-09-15T23:16:01.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:00 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3374708650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:01.072 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:00 smithi062 ceph-mon[30949]: pgmap v362: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 682 B/s wr, 5 op/s 2024-09-15T23:16:01.072 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:00 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3374708650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:00 smithi148 ceph-mon[35610]: pgmap v362: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 682 B/s wr, 5 op/s 2024-09-15T23:16:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:00 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3374708650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:01.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.452+0000 7f28712b1640 1 -- 172.21.15.62:0/2630354837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c106a80 msgr2=0x7f286c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:01.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.452+0000 7f28712b1640 1 --2- 172.21.15.62:0/2630354837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c106a80 0x7f286c106e60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f285c009930 tx=0x7f285c02a420 comp rx=0 tx=0).stop 2024-09-15T23:16:01.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.452+0000 7f28712b1640 1 -- 172.21.15.62:0/2630354837 shutdown_connections 2024-09-15T23:16:01.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.452+0000 7f28712b1640 1 --2- 172.21.15.62:0/2630354837 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f286c101420 0x7f286c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.452+0000 7f28712b1640 1 --2- 172.21.15.62:0/2630354837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f286c100a80 0x7f286c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.452+0000 7f28712b1640 1 --2- 172.21.15.62:0/2630354837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c106a80 0x7f286c106e60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.452+0000 7f28712b1640 1 -- 172.21.15.62:0/2630354837 >> 172.21.15.62:0/2630354837 conn(0x7f286c0fc7a0 msgr2=0x7f286c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:01.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.452+0000 7f28712b1640 1 -- 172.21.15.62:0/2630354837 shutdown_connections 2024-09-15T23:16:01.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.452+0000 7f28712b1640 1 -- 172.21.15.62:0/2630354837 wait complete. 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f28712b1640 1 Processor -- start 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f28712b1640 1 -- start start 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f28712b1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f286c100a80 0x7f286c198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f28712b1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c101420 0x7f286c198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f28712b1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f286c106a80 0x7f286c19cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f28712b1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f286c110ea0 con 0x7f286c101420 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f28712b1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f286c110d20 con 0x7f286c100a80 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f28712b1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f286c111020 con 0x7f286c106a80 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f286a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c101420 0x7f286c198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f286a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c101420 0x7f286c198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42778/0 (socket says 172.21.15.62:42778) 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f286a7fc640 1 -- 172.21.15.62:0/1165215803 learned_addr learned my addr 172.21.15.62:0/1165215803 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:01.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f286affd640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f286c100a80 0x7f286c198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.453+0000 7f286b7fe640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f286c106a80 0x7f286c19cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f286a7fc640 1 -- 172.21.15.62:0/1165215803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f286c106a80 msgr2=0x7f286c19cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f286a7fc640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f286c106a80 0x7f286c19cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f286a7fc640 1 -- 172.21.15.62:0/1165215803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f286c100a80 msgr2=0x7f286c198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f286a7fc640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f286c100a80 0x7f286c198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f286a7fc640 1 -- 172.21.15.62:0/1165215803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c19d4d0 con 0x7f286c101420 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f286a7fc640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c101420 0x7f286c198a60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f285400c890 tx=0x7f285400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f2863fff640 1 -- 172.21.15.62:0/1165215803 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854004220 con 0x7f286c101420 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f2863fff640 1 -- 172.21.15.62:0/1165215803 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28540043c0 con 0x7f286c101420 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f286c19d7c0 con 0x7f286c101420 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f2863fff640 1 -- 172.21.15.62:0/1165215803 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28540049e0 con 0x7f286c101420 2024-09-15T23:16:01.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.454+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f286c1a50a0 con 0x7f286c101420 2024-09-15T23:16:01.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.455+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286c100ee0 con 0x7f286c101420 2024-09-15T23:16:01.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.456+0000 7f2863fff640 1 -- 172.21.15.62:0/1165215803 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f28540079b0 con 0x7f286c101420 2024-09-15T23:16:01.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.457+0000 7f2863fff640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28580a3240 0x7f28580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:01.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.457+0000 7f2863fff640 1 -- 172.21.15.62:0/1165215803 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f28540cfcf0 con 0x7f286c101420 2024-09-15T23:16:01.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.457+0000 7f286affd640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28580a3240 0x7f28580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:01.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.458+0000 7f286affd640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28580a3240 0x7f28580a5700 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f285c009900 tx=0x7f285c0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:01.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.461+0000 7f2863fff640 1 -- 172.21.15.62:0/1165215803 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f28540d4070 con 0x7f286c101420 2024-09-15T23:16:01.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.681+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f286c1a5300 con 0x7f286c101420 2024-09-15T23:16:01.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.681+0000 7f2863fff640 1 -- 172.21.15.62:0/1165215803 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7f2854014030 con 0x7f286c101420 2024-09-15T23:16:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:01.683 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"btime":"2024-09-15T23:15:44:620975+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:15:43.799068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:16:01.683 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-09-15T23:16:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28580a3240 msgr2=0x7f28580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28580a3240 0x7f28580a5700 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f285c009900 tx=0x7f285c0058d0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c101420 msgr2=0x7f286c198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c101420 0x7f286c198a60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f285400c890 tx=0x7f285400cd50 comp rx=0 tx=0).stop 2024-09-15T23:16:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 shutdown_connections 2024-09-15T23:16:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28580a3240 0x7f28580a5700 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f286c106a80 0x7f286c19cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f286c101420 0x7f286c198a60 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 --2- 172.21.15.62:0/1165215803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f286c100a80 0x7f286c198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:01.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 >> 172.21.15.62:0/1165215803 conn(0x7f286c0fc7a0 msgr2=0x7f286c06a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:01.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 shutdown_connections 2024-09-15T23:16:01.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:01.684+0000 7f28712b1640 1 -- 172.21.15.62:0/1165215803 wait complete. 2024-09-15T23:16:01.976 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:01 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1165215803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:02.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:01 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1165215803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:02.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:01 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1165215803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:02.290 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:16:02.552 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:02.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:02 smithi062 ceph-mon[30949]: pgmap v363: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 682 B/s wr, 7 op/s 2024-09-15T23:16:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.060+0000 7f431c393640 1 -- 172.21.15.62:0/3616357698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314101420 msgr2=0x7f431410d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.060+0000 7f431c393640 1 --2- 172.21.15.62:0/3616357698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314101420 0x7f431410d950 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f430c009930 tx=0x7f430c02a420 comp rx=0 tx=0).stop 2024-09-15T23:16:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.060+0000 7f431c393640 1 -- 172.21.15.62:0/3616357698 shutdown_connections 2024-09-15T23:16:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.060+0000 7f431c393640 1 --2- 172.21.15.62:0/3616357698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314101420 0x7f431410d950 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.060+0000 7f431c393640 1 --2- 172.21.15.62:0/3616357698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4314100a80 0x7f4314100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.060+0000 7f431c393640 1 --2- 172.21.15.62:0/3616357698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4314106a80 0x7f4314106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.060+0000 7f431c393640 1 -- 172.21.15.62:0/3616357698 >> 172.21.15.62:0/3616357698 conn(0x7f43140fc7a0 msgr2=0x7f43140febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.060+0000 7f431c393640 1 -- 172.21.15.62:0/3616357698 shutdown_connections 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.060+0000 7f431c393640 1 -- 172.21.15.62:0/3616357698 wait complete. 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431c393640 1 Processor -- start 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431c393640 1 -- start start 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431c393640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4314100a80 0x7f4314199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431c393640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4314101420 0x7f4314199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431c393640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314106a80 0x7f43141a4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431c393640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4314110e80 con 0x7f4314106a80 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431c393640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4314110d00 con 0x7f4314101420 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431c393640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4314111000 con 0x7f4314100a80 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431a930640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314106a80 0x7f43141a4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431a930640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314106a80 0x7f43141a4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42080/0 (socket says 172.21.15.62:42080) 2024-09-15T23:16:03.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431a930640 1 -- 172.21.15.62:0/4285419297 learned_addr learned my addr 172.21.15.62:0/4285419297 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431992e640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4314101420 0x7f4314199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431a12f640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4314100a80 0x7f4314199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431a930640 1 -- 172.21.15.62:0/4285419297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4314100a80 msgr2=0x7f4314199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431a930640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4314100a80 0x7f4314199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431a930640 1 -- 172.21.15.62:0/4285419297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4314101420 msgr2=0x7f4314199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431a930640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4314101420 0x7f4314199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431a930640 1 -- 172.21.15.62:0/4285419297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43141a47c0 con 0x7f4314106a80 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.061+0000 7f431992e640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4314101420 0x7f4314199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.062+0000 7f431a930640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314106a80 0x7f43141a4280 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f430c005ab0 tx=0x7f430c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.062+0000 7f43077fe640 1 -- 172.21.15.62:0/4285419297 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c03f030 con 0x7f4314106a80 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.062+0000 7f43077fe640 1 -- 172.21.15.62:0/4285419297 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f430c003ce0 con 0x7f4314106a80 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.062+0000 7f43077fe640 1 -- 172.21.15.62:0/4285419297 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c035070 con 0x7f4314106a80 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.062+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43141a4990 con 0x7f4314106a80 2024-09-15T23:16:03.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.062+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43141a4de0 con 0x7f4314106a80 2024-09-15T23:16:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.063+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4314100ee0 con 0x7f4314106a80 2024-09-15T23:16:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.064+0000 7f43077fe640 1 -- 172.21.15.62:0/4285419297 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f430c042030 con 0x7f4314106a80 2024-09-15T23:16:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.065+0000 7f43077fe640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43000a3240 0x7f43000a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.065+0000 7f431a12f640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43000a3240 0x7f43000a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.065+0000 7f43077fe640 1 -- 172.21.15.62:0/4285419297 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f430c0f05d0 con 0x7f4314106a80 2024-09-15T23:16:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.066+0000 7f431a12f640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43000a3240 0x7f43000a5700 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4310005fd0 tx=0x7f4310009200 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:03.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.069+0000 7f43077fe640 1 -- 172.21.15.62:0/4285419297 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f430c0bcf70 con 0x7f4314106a80 2024-09-15T23:16:03.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.245+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43141a5040 con 0x7f4314106a80 2024-09-15T23:16:03.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.246+0000 7f43077fe640 1 -- 172.21.15.62:0/4285419297 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24777 (secure 0 0 0) 0x7f430c0bc870 con 0x7f4314106a80 2024-09-15T23:16:03.248 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:03.248 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":92,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:15:42.605470+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43000a3240 msgr2=0x7f43000a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43000a3240 0x7f43000a5700 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4310005fd0 tx=0x7f4310009200 comp rx=0 tx=0).stop 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314106a80 msgr2=0x7f43141a4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314106a80 0x7f43141a4280 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f430c005ab0 tx=0x7f430c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 shutdown_connections 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43000a3240 0x7f43000a5700 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4314106a80 0x7f43141a4280 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4314101420 0x7f4314199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 --2- 172.21.15.62:0/4285419297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4314100a80 0x7f4314199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 >> 172.21.15.62:0/4285419297 conn(0x7f43140fc7a0 msgr2=0x7f43140fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:03.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 shutdown_connections 2024-09-15T23:16:03.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:03.249+0000 7f431c393640 1 -- 172.21.15.62:0/4285419297 wait complete. 2024-09-15T23:16:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:02 smithi067 ceph-mon[35665]: pgmap v363: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 682 B/s wr, 7 op/s 2024-09-15T23:16:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:02 smithi148 ceph-mon[35610]: pgmap v363: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 682 B/s wr, 7 op/s 2024-09-15T23:16:03.880 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mon.a --format=json config get run_dir 2024-09-15T23:16:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:03 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4285419297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:16:04.136 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:03 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4285419297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:16:04.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:03 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4285419297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:16:05.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:04 smithi062 ceph-mon[30949]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-09-15T23:16:05.272 INFO:teuthology.orchestra.run.smithi062.stdout:{"run_dir":"/var/run/ceph"} 2024-09-15T23:16:05.273 DEBUG:tasks.cephfs.filesystem:_json_asok output { "run_dir": "/var/run/ceph" } 2024-09-15T23:16:05.273 INFO:tasks.quiescer.fs.[cephfs]:Initializing Quiescer with seed 3393 2024-09-15T23:16:05.273 DEBUG:tasks.quiescer:Yielding 2024-09-15T23:16:05.273 INFO:teuthology.run_tasks:Running task exec... 2024-09-15T23:16:05.282 INFO:teuthology.task.exec:Executing custom commands... 2024-09-15T23:16:05.283 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T23:16:05.283 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-09-15T23:16:05.283 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:16:05.284 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:04 smithi067 ceph-mon[35665]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-09-15T23:16:05.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:04 smithi148 ceph-mon[35610]: pgmap v364: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 170 B/s wr, 5 op/s 2024-09-15T23:16:05.321 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:05.337 INFO:teuthology.orchestra.run.smithi062.stdout:Filesystem Size Used Avail Use% Mounted on 2024-09-15T23:16:05.337 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:3300,172.21.15.67:3300,172.21.15.148:3300:/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e 1.1T 364M 1.1T 1% /tmp/cephfs 2024-09-15T23:16:05.341 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-09-15T23:16:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.340+0000 7f985372c640 1 -- 172.21.15.62:0/4162044124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 msgr2=0x7f984c109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.340+0000 7f985372c640 1 --2- 172.21.15.62:0/4162044124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 0x7f984c109fa0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9844009ca0 tx=0x7f984402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.340+0000 7f985372c640 1 -- 172.21.15.62:0/4162044124 shutdown_connections 2024-09-15T23:16:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.340+0000 7f985372c640 1 --2- 172.21.15.62:0/4162044124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 0x7f984c109fa0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.340+0000 7f985372c640 1 --2- 172.21.15.62:0/4162044124 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f984c102d50 0x7f984c1031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.340+0000 7f985372c640 1 --2- 172.21.15.62:0/4162044124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984c101b60 0x7f984c101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.340+0000 7f985372c640 1 -- 172.21.15.62:0/4162044124 >> 172.21.15.62:0/4162044124 conn(0x7f984c0fd2e0 msgr2=0x7f984c0ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.340+0000 7f985372c640 1 -- 172.21.15.62:0/4162044124 shutdown_connections 2024-09-15T23:16:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.340+0000 7f985372c640 1 -- 172.21.15.62:0/4162044124 wait complete. 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f985372c640 1 Processor -- start 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f985372c640 1 -- start start 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f985372c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984c101b60 0x7f984c197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f985372c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f984c102d50 0x7f984c1982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f985372c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 0x7f984c19f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f985372c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f984c10d7c0 con 0x7f984c103710 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f985372c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f984c10d640 con 0x7f984c101b60 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f985372c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f984c10d940 con 0x7f984c102d50 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f9851cc9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 0x7f984c19f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f9851cc9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 0x7f984c19f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42094/0 (socket says 172.21.15.62:42094) 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f9851cc9640 1 -- 172.21.15.62:0/606757637 learned_addr learned my addr 172.21.15.62:0/606757637 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f98514c8640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984c101b60 0x7f984c197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f9850cc7640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f984c102d50 0x7f984c1982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f9851cc9640 1 -- 172.21.15.62:0/606757637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f984c102d50 msgr2=0x7f984c1982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f9851cc9640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f984c102d50 0x7f984c1982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f9851cc9640 1 -- 172.21.15.62:0/606757637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984c101b60 msgr2=0x7f984c197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f9851cc9640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984c101b60 0x7f984c197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.341+0000 7f9851cc9640 1 -- 172.21.15.62:0/606757637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f984c19f9b0 con 0x7f984c103710 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.342+0000 7f9851cc9640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 0x7f984c19f310 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9844031ae0 tx=0x7f9844031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.342+0000 7f98427fc640 1 -- 172.21.15.62:0/606757637 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9844003b70 con 0x7f984c103710 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.342+0000 7f985372c640 1 -- 172.21.15.62:0/606757637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f984c19fc40 con 0x7f984c103710 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.342+0000 7f98427fc640 1 -- 172.21.15.62:0/606757637 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9844003d10 con 0x7f984c103710 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.342+0000 7f98427fc640 1 -- 172.21.15.62:0/606757637 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984402fc40 con 0x7f984c103710 2024-09-15T23:16:05.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.342+0000 7f985372c640 1 -- 172.21.15.62:0/606757637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f984c1a0190 con 0x7f984c103710 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.345+0000 7f98427fc640 1 -- 172.21.15.62:0/606757637 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9844004050 con 0x7f984c103710 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.345+0000 7f98564bd740 1 Processor -- start 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.345+0000 7f98564bd740 1 -- start start 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.345+0000 7f98564bd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 0x565098ed4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98564bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ed9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98564bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 0x565098edc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98564bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565098e382f0 con 0x565098ed52e0 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98564bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565098de0270 con 0x565098eda370 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98564bd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x565098e342b0 con 0x565098ed49a0 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98514c8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ed9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98514c8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ed9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42110/0 (socket says 172.21.15.62:42110) 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98514c8640 1 -- 172.21.15.62:0/869737494 learned_addr learned my addr 172.21.15.62:0/869737494 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f9850cc7640 1 --2- 172.21.15.62:0/869737494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 0x565098ed4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f9850cc7640 1 --2- 172.21.15.62:0/869737494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 0x565098edc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98427fc640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f98380a32f0 0x7f98380a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98514c8640 1 -- 172.21.15.62:0/869737494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 msgr2=0x565098ed4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98514c8640 1 --2- 172.21.15.62:0/869737494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 0x565098ed4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98514c8640 1 -- 172.21.15.62:0/869737494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 msgr2=0x565098edc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98514c8640 1 --2- 172.21.15.62:0/869737494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 0x565098edc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98514c8640 1 -- 172.21.15.62:0/869737494 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565098ddf340 con 0x565098ed52e0 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f9851cc9640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f98380a32f0 0x7f98380a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98514c8640 1 --2- 172.21.15.62:0/869737494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ed9da0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9848009bf0 tx=0x7f98480117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59227af868e6175c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.346+0000 7f98427fc640 1 -- 172.21.15.62:0/606757637 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f98440f0f10 con 0x7f984c103710 2024-09-15T23:16:05.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.347+0000 7f98377fe640 1 -- 172.21.15.62:0/869737494 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848016860 con 0x565098ed52e0 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.347+0000 7f98377fe640 1 -- 172.21.15.62:0/869737494 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f984800f440 con 0x565098ed52e0 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.347+0000 7f98377fe640 1 -- 172.21.15.62:0/869737494 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984801e650 con 0x565098ed52e0 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 -- 172.21.15.62:0/869737494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 msgr2=0x565098ed9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 --2- 172.21.15.62:0/869737494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ed9da0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9848009bf0 tx=0x7f98480117b0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f9851cc9640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f98380a32f0 0x7f98380a57b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f98440ee6e0 tx=0x7f98440c1040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 -- 172.21.15.62:0/869737494 shutdown_connections 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 --2- 172.21.15.62:0/869737494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 0x565098edc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 --2- 172.21.15.62:0/869737494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ed9da0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 --2- 172.21.15.62:0/869737494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 0x565098ed4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 -- 172.21.15.62:0/869737494 >> 172.21.15.62:0/869737494 conn(0x565098ed0010 msgr2=0x565098ed0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 -- 172.21.15.62:0/869737494 shutdown_connections 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 -- 172.21.15.62:0/869737494 wait complete. 2024-09-15T23:16:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 -- start start 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 Processor -- start 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed49a0 0x565098edeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed52e0 0x565098edf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098eda370 0x565098ee60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565098edd340 con 0x565098ed49a0 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565098dc1290 con 0x565098ed52e0 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.348+0000 7f98564bd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x565098edd4c0 con 0x565098eda370 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f98564bd740 10 client.0 _ll_register_callbacks cb 0x565098df2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed49a0 0x565098edeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed49a0 0x565098edeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42114/0 (socket says 172.21.15.62:42114) 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 -- 172.21.15.62:0/3866536302 learned_addr learned my addr 172.21.15.62:0/3866536302 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f98514c8640 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed52e0 0x565098edf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098eda370 0x565098ee60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 -- 172.21.15.62:0/3866536302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098eda370 msgr2=0x565098ee60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098eda370 0x565098ee60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 -- 172.21.15.62:0/3866536302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed52e0 msgr2=0x565098edf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed52e0 0x565098edf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 -- 172.21.15.62:0/3866536302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565098b4b210 con 0x565098ed49a0 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9850cc7640 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed49a0 0x565098edeb60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f983c012f20 tx=0x7f983c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880c86f7bcb20123 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9834ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.349+0000 7f9834ff9640 1 -- 172.21.15.62:0/3866536302 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c004020 con 0x565098ed49a0 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f9834ff9640 1 -- 172.21.15.62:0/3866536302 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f983c0041c0 con 0x565098ed49a0 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f98564bd740 1 -- 172.21.15.62:0/3866536302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f984800f440 con 0x565098ed49a0 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f9834ff9640 1 -- 172.21.15.62:0/3866536302 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f983c02b050 con 0x565098ed49a0 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f98564bd740 10 client.15387 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:05.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f98564bd740 1 -- 172.21.15.62:0/3866536302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x565098c17620 con 0x565098ed49a0 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f9834ff9640 1 -- 172.21.15.62:0/3866536302 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f984800f440 con 0x565098ed49a0 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f9834ff9640 1 -- 172.21.15.62:0/3866536302 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f983c00f630 con 0x565098ed49a0 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f98564bd740 10 client.15387 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f98564bd740 10 client.15387 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f98564bd740 20 client.15387 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f98564bd740 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565098eea820 0x565098eecc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f98564bd740 4 client.15387 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.350+0000 7f98564bd740 1 -- 172.21.15.62:0/3866536302 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565098e45a20 con 0x565098eea820 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.351+0000 7f98514c8640 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565098eea820 0x565098eecc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.351+0000 7f9834ff9640 10 client.15387 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:05.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.351+0000 7f98514c8640 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565098eea820 0x565098eecc90 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c1ffc4c5f5c6ee4 server_cookie=88c0c586accf4432 in_seq=0 out_seq=0 2024-09-15T23:16:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f9834ff9640 1 -- 172.21.15.62:0/3866536302 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565098e45a20 con 0x565098eea820 2024-09-15T23:16:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f9834ff9640 10 client.15387 handle_command_reply: tid=0 2024-09-15T23:16:05.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 client.15387 shutdown 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 -- 172.21.15.62:0/3866536302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed49a0 msgr2=0x565098edeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed49a0 0x565098edeb60 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f983c012f20 tx=0x7f983c0046c0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 -- 172.21.15.62:0/3866536302 shutdown_connections 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565098eea820 0x565098eecc90 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f9834ff9640 0 client.15387 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098eda370 0x565098ee60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed52e0 0x565098edf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 --2- 172.21.15.62:0/3866536302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed49a0 0x565098edeb60 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 -- 172.21.15.62:0/3866536302 >> 172.21.15.62:0/3866536302 conn(0x565098ed0010 msgr2=0x565098ed1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 -- 172.21.15.62:0/3866536302 shutdown_connections 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 1 -- 172.21.15.62:0/3866536302 wait complete. 2024-09-15T23:16:05.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.352+0000 7f98564bd740 20 client.15387 trim_cache size 0 max 16384 2024-09-15T23:16:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.367+0000 7f98564bd740 1 Processor -- start 2024-09-15T23:16:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.367+0000 7f98564bd740 1 -- start start 2024-09-15T23:16:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.367+0000 7f98564bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed49a0 0x565098ecf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.367+0000 7f98564bd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed52e0 0x565098ee6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.367+0000 7f98564bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098eda370 0x565098ef1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.367+0000 7f98564bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1aac20 con 0x565098eda370 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f98564bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1a84f0 con 0x565098ed49a0 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f98564bd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1a6340 con 0x565098ed52e0 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098eda370 0x565098ef1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098eda370 0x565098ef1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42130/0 (socket says 172.21.15.62:42130) 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 -- 172.21.15.62:0/236148513 learned_addr learned my addr 172.21.15.62:0/236148513 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f98514c8640 1 --2- 172.21.15.62:0/236148513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed52e0 0x565098ee6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 --2- 172.21.15.62:0/236148513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed49a0 0x565098ecf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 -- 172.21.15.62:0/236148513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed52e0 msgr2=0x565098ee6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 --2- 172.21.15.62:0/236148513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed52e0 0x565098ee6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 -- 172.21.15.62:0/236148513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed49a0 msgr2=0x565098ecf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 --2- 172.21.15.62:0/236148513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed49a0 0x565098ecf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 -- 172.21.15.62:0/236148513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565098ede6a0 con 0x565098eda370 2024-09-15T23:16:05.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.368+0000 7f9850cc7640 1 --2- 172.21.15.62:0/236148513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098eda370 0x565098ef1590 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f983c0096f0 tx=0x7f983c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bb4d60aabfc9ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f9836ffd640 1 -- 172.21.15.62:0/236148513 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c00fc40 con 0x565098eda370 2024-09-15T23:16:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f9836ffd640 1 -- 172.21.15.62:0/236148513 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f983c00fde0 con 0x565098eda370 2024-09-15T23:16:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f9836ffd640 1 -- 172.21.15.62:0/236148513 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f983c02f400 con 0x565098eda370 2024-09-15T23:16:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f98564bd740 1 -- 172.21.15.62:0/236148513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098eda370 msgr2=0x565098ef1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f98564bd740 1 --2- 172.21.15.62:0/236148513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098eda370 0x565098ef1590 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f983c0096f0 tx=0x7f983c005f90 comp rx=0 tx=0).stop 2024-09-15T23:16:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f98564bd740 1 -- 172.21.15.62:0/236148513 shutdown_connections 2024-09-15T23:16:05.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f98564bd740 1 --2- 172.21.15.62:0/236148513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098eda370 0x565098ef1590 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f98564bd740 1 --2- 172.21.15.62:0/236148513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed52e0 0x565098ee6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f98564bd740 1 --2- 172.21.15.62:0/236148513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098ed49a0 0x565098ecf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.369+0000 7f98564bd740 1 -- 172.21.15.62:0/236148513 >> 172.21.15.62:0/236148513 conn(0x565098ed0010 msgr2=0x565098ece9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 -- 172.21.15.62:0/236148513 shutdown_connections 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 -- 172.21.15.62:0/236148513 wait complete. 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 -- start start 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 Processor -- start 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 0x565098ee3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ee4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 0x565098ee5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1a69b0 con 0x565098ed52e0 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1a7e20 con 0x565098eda370 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f984c1a7080 con 0x565098ed49a0 2024-09-15T23:16:05.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.370+0000 7f98564bd740 10 client.0 _ll_register_callbacks cb 0x7f983c0124d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98514c8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ee4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98514c8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ee4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42144/0 (socket says 172.21.15.62:42144) 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98514c8640 1 -- 172.21.15.62:0/204218393 learned_addr learned my addr 172.21.15.62:0/204218393 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f9850cc7640 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 0x565098ee3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f9850cc7640 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 0x565098ee5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98514c8640 1 -- 172.21.15.62:0/204218393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 msgr2=0x565098ee3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98514c8640 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 0x565098ee3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98514c8640 1 -- 172.21.15.62:0/204218393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 msgr2=0x565098ee5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98514c8640 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 0x565098ee5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98514c8640 1 -- 172.21.15.62:0/204218393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565098ed0ca0 con 0x565098ed52e0 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98514c8640 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ee4140 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9848009ec0 tx=0x7f9848011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad16ee84c4d20ff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:05.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f9834ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f9834ff9640 1 -- 172.21.15.62:0/204218393 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848016c70 con 0x565098ed52e0 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f9834ff9640 1 -- 172.21.15.62:0/204218393 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f984800f440 con 0x565098ed52e0 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.371+0000 7f98564bd740 1 -- 172.21.15.62:0/204218393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f983c00fde0 con 0x565098ed52e0 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f9834ff9640 1 -- 172.21.15.62:0/204218393 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9848016e10 con 0x565098ed52e0 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f98564bd740 10 client.15393 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f98564bd740 1 -- 172.21.15.62:0/204218393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x565098f55960 con 0x565098ed52e0 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f9834ff9640 1 -- 172.21.15.62:0/204218393 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f983c00fde0 con 0x565098ed52e0 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f9834ff9640 1 -- 172.21.15.62:0/204218393 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f98480035f0 con 0x565098ed52e0 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f98564bd740 10 client.15393 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f98564bd740 10 client.15393 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:05.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f98564bd740 20 client.15393 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f98564bd740 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565098eea820 0x565098f581b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f98564bd740 4 client.15393 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:16:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.372+0000 7f98564bd740 1 -- 172.21.15.62:0/204218393 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565098e3fd70 con 0x565098eea820 2024-09-15T23:16:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.373+0000 7f9850cc7640 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565098eea820 0x565098f581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.373+0000 7f9850cc7640 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565098eea820 0x565098f581b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48a6a77f47e3b8d4 server_cookie=bf5f7cda2e80b827 in_seq=0 out_seq=0 2024-09-15T23:16:05.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.373+0000 7f9834ff9640 10 client.15393 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.385+0000 7f9834ff9640 1 -- 172.21.15.62:0/204218393 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565098e3fd70 con 0x565098eea820 2024-09-15T23:16:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.385+0000 7f9834ff9640 10 client.15393 handle_command_reply: tid=0 2024-09-15T23:16:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.385+0000 7f98564bd740 1 client.15393 shutdown 2024-09-15T23:16:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.385+0000 7f98564bd740 1 -- 172.21.15.62:0/204218393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 msgr2=0x565098ee4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ee4140 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9848009ec0 tx=0x7f9848011f80 comp rx=0 tx=0).stop 2024-09-15T23:16:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 1 -- 172.21.15.62:0/204218393 shutdown_connections 2024-09-15T23:16:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565098eea820 0x565098f581b0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565098eda370 0x565098ee5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f9834ff9640 0 client.15393 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565098ed52e0 0x565098ee4140 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 1 --2- 172.21.15.62:0/204218393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565098ed49a0 0x565098ee3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 1 -- 172.21.15.62:0/204218393 >> 172.21.15.62:0/204218393 conn(0x565098ed0010 msgr2=0x565098ece9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 1 -- 172.21.15.62:0/204218393 shutdown_connections 2024-09-15T23:16:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 1 -- 172.21.15.62:0/204218393 wait complete. 2024-09-15T23:16:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f98564bd740 20 client.15393 trim_cache size 0 max 16384 2024-09-15T23:16:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:16:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:16:05.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:16:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:16:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 -- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f98380a32f0 msgr2=0x7f98380a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f98380a32f0 0x7f98380a57b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f98440ee6e0 tx=0x7f98440c1040 comp rx=0 tx=0).stop 2024-09-15T23:16:05.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 -- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 msgr2=0x7f984c19f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 0x7f984c19f310 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9844031ae0 tx=0x7f9844031ee0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 -- 172.21.15.62:0/606757637 shutdown_connections 2024-09-15T23:16:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f98380a32f0 0x7f98380a57b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f984c103710 0x7f984c19f310 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f984c102d50 0x7f984c1982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 --2- 172.21.15.62:0/606757637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f984c101b60 0x7f984c197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.386+0000 7f985372c640 1 -- 172.21.15.62:0/606757637 >> 172.21.15.62:0/606757637 conn(0x7f984c0fd2e0 msgr2=0x7f984c106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.387+0000 7f985372c640 1 -- 172.21.15.62:0/606757637 shutdown_connections 2024-09-15T23:16:05.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:05.387+0000 7f985372c640 1 -- 172.21.15.62:0/606757637 wait complete. 2024-09-15T23:16:05.402 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-09-15T23:16:05.484 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-09-15T23:16:05.558 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.070+0000 7f8c48a51640 1 -- 172.21.15.62:0/215186137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c441082a0 msgr2=0x7f8c4410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.070+0000 7f8c48a51640 1 --2- 172.21.15.62:0/215186137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c441082a0 0x7f8c4410ec40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c009cd0 tx=0x7f8c3c02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.070+0000 7f8c48a51640 1 -- 172.21.15.62:0/215186137 shutdown_connections 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.070+0000 7f8c48a51640 1 --2- 172.21.15.62:0/215186137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c441082a0 0x7f8c4410ec40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.070+0000 7f8c48a51640 1 --2- 172.21.15.62:0/215186137 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c441036e0 0x7f8c44107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.070+0000 7f8c48a51640 1 --2- 172.21.15.62:0/215186137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c44102d30 0x7f8c44103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.070+0000 7f8c48a51640 1 -- 172.21.15.62:0/215186137 >> 172.21.15.62:0/215186137 conn(0x7f8c440fc7d0 msgr2=0x7f8c440febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.070+0000 7f8c48a51640 1 -- 172.21.15.62:0/215186137 shutdown_connections 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.070+0000 7f8c48a51640 1 -- 172.21.15.62:0/215186137 wait complete. 2024-09-15T23:16:06.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c48a51640 1 Processor -- start 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c48a51640 1 -- start start 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c48a51640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c44102d30 0x7f8c44100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c48a51640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c441036e0 0x7f8c44100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c48a51640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c441082a0 0x7f8c44101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c48a51640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8c441120a0 con 0x7f8c44102d30 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c48a51640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c44111f20 con 0x7f8c441036e0 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c48a51640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8c44112220 con 0x7f8c441082a0 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c4259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c44102d30 0x7f8c44100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c4259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c44102d30 0x7f8c44100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42164/0 (socket says 172.21.15.62:42164) 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c4259c640 1 -- 172.21.15.62:0/1817109679 learned_addr learned my addr 172.21.15.62:0/1817109679 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:06.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c3bfff640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c441036e0 0x7f8c44100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c4259c640 1 -- 172.21.15.62:0/1817109679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c441082a0 msgr2=0x7f8c44101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c4259c640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c441082a0 0x7f8c44101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c4259c640 1 -- 172.21.15.62:0/1817109679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c441036e0 msgr2=0x7f8c44100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c4259c640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c441036e0 0x7f8c44100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.071+0000 7f8c4259c640 1 -- 172.21.15.62:0/1817109679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c441a6a00 con 0x7f8c44102d30 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.072+0000 7f8c4259c640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c44102d30 0x7f8c44100990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400e9f0 tx=0x7f8c3400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.072+0000 7f8c3b7fe640 1 -- 172.21.15.62:0/1817109679 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3400cd80 con 0x7f8c44102d30 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.072+0000 7f8c3b7fe640 1 -- 172.21.15.62:0/1817109679 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c34010910 con 0x7f8c44102d30 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.072+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c441a6cf0 con 0x7f8c44102d30 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.072+0000 7f8c3b7fe640 1 -- 172.21.15.62:0/1817109679 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c34018990 con 0x7f8c44102d30 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.072+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c441a7220 con 0x7f8c44102d30 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.073+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c44103110 con 0x7f8c44102d30 2024-09-15T23:16:06.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.074+0000 7f8c3b7fe640 1 -- 172.21.15.62:0/1817109679 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8c34010ac0 con 0x7f8c44102d30 2024-09-15T23:16:06.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.075+0000 7f8c3b7fe640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c300a32c0 0x7f8c300a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:06.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.075+0000 7f8c3b7fe640 1 -- 172.21.15.62:0/1817109679 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8c340d03b0 con 0x7f8c44102d30 2024-09-15T23:16:06.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.075+0000 7f8c3bfff640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c300a32c0 0x7f8c300a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:06.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.076+0000 7f8c3bfff640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c300a32c0 0x7f8c300a5780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8c44102420 tx=0x7f8c2c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:06.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.079+0000 7f8c3b7fe640 1 -- 172.21.15.62:0/1817109679 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8c3409cd50 con 0x7f8c44102d30 2024-09-15T23:16:06.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.291+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c44101ec0 con 0x7f8c44102d30 2024-09-15T23:16:06.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.292+0000 7f8c3b7fe640 1 -- 172.21.15.62:0/1817109679 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7f8c3409c650 con 0x7f8c44102d30 2024-09-15T23:16:06.293 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:06.293 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"btime":"2024-09-15T23:15:44:620975+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:15:43.799068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:16:06.294 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-09-15T23:16:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c300a32c0 msgr2=0x7f8c300a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c300a32c0 0x7f8c300a5780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8c44102420 tx=0x7f8c2c00b410 comp rx=0 tx=0).stop 2024-09-15T23:16:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c44102d30 msgr2=0x7f8c44100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c44102d30 0x7f8c44100990 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400e9f0 tx=0x7f8c3400eeb0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 shutdown_connections 2024-09-15T23:16:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c300a32c0 0x7f8c300a5780 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c441082a0 0x7f8c44101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c441036e0 0x7f8c44100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 --2- 172.21.15.62:0/1817109679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c44102d30 0x7f8c44100990 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:06.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 >> 172.21.15.62:0/1817109679 conn(0x7f8c440fc7d0 msgr2=0x7f8c44105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:06.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 shutdown_connections 2024-09-15T23:16:06.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:06.294+0000 7f8c48a51640 1 -- 172.21.15.62:0/1817109679 wait complete. 2024-09-15T23:16:07.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:06 smithi062 ceph-mon[30949]: pgmap v365: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-15T23:16:07.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:06 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1817109679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:07.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:06 smithi067 ceph-mon[35665]: pgmap v365: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-15T23:16:07.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:06 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1817109679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:07.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:06 smithi148 ceph-mon[35610]: pgmap v365: 129 pgs: 129 active+clean; 649 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-09-15T23:16:07.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:06 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1817109679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:07.567 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-09-15T23:16:07.646 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-09-15T23:16:07.648 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-09-15T23:16:07.693 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 306, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': True}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-15T23:13:33.200855+0000', 'modified': '2024-09-15T23:15:43.799068+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24497, 'mds_1': 24470, 'mds_2': 24466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.148:6835/1239577031', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6834', 'nonce': 1239577031}, {'type': 'v1', 'addr': '172.21.15.148:6835', 'nonce': 1239577031}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.67:6835/3132447542', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 3132447542}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3132447542}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.148:6837/870913016', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6836', 'nonce': 870913016}, {'type': 'v1', 'addr': '172.21.15.148:6837', 'nonce': 870913016}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.148:6839/2241485315', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.148:6838', 'nonce': 2241485315}, {'type': 'v1', 'addr': '172.21.15.148:6839', 'nonce': 2241485315}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.67:6837/747270351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6836', 'nonce': 747270351}, {'type': 'v1', 'addr': '172.21.15.67:6837', 'nonce': 747270351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24497': {'gid': 24497, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6839/818760439', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6838', 'nonce': 818760439}, {'type': 'v1', 'addr': '172.21.15.67:6839', 'nonce': 818760439}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24497, 'qdb_cluster': [24466, 24470, 24497]} 2024-09-15T23:16:07.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-09-15T23:16:07.694 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.i status 2024-09-15T23:16:07.727 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-09-15T23:16:07.804 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/: 2024-09-15T23:16:07.804 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:07.804 INFO:teuthology.orchestra.run.smithi062.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:16:07.804 INFO:teuthology.orchestra.run.smithi062.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 .. 2024-09-15T23:16:07.804 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 data 2024-09-15T23:16:07.805 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:07.805 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data: 2024-09-15T23:16:07.805 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:07.805 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:07.805 INFO:teuthology.orchestra.run.smithi062.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 .. 2024-09-15T23:16:07.807 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-09-15T23:16:07.869 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:07 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:16:07.893 INFO:teuthology.orchestra.run.smithi062.stdout:[Service] 2024-09-15T23:16:07.893 INFO:teuthology.orchestra.run.smithi062.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-09-15T23:16:07.893 INFO:teuthology.orchestra.run.smithi062.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-09-15T23:16:07.895 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-09-15T23:16:08.035 INFO:teuthology.orchestra.run.smithi062.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-09-15T23:16:08.037 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:08.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:07 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:16:08.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:07 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:16:08.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.605+0000 7f6201f08640 1 -- 172.21.15.62:0/1016678823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc100a80 msgr2=0x7f61fc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:08.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.605+0000 7f6201f08640 1 --2- 172.21.15.62:0/1016678823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc100a80 0x7f61fc100ee0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f61ec009930 tx=0x7f61ec02a420 comp rx=0 tx=0).stop 2024-09-15T23:16:08.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.605+0000 7f6201f08640 1 -- 172.21.15.62:0/1016678823 shutdown_connections 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.605+0000 7f6201f08640 1 --2- 172.21.15.62:0/1016678823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc101420 0x7f61fc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.605+0000 7f6201f08640 1 --2- 172.21.15.62:0/1016678823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc100a80 0x7f61fc100ee0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.605+0000 7f6201f08640 1 --2- 172.21.15.62:0/1016678823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61fc106a80 0x7f61fc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.605+0000 7f6201f08640 1 -- 172.21.15.62:0/1016678823 >> 172.21.15.62:0/1016678823 conn(0x7f61fc0fc7a0 msgr2=0x7f61fc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.605+0000 7f6201f08640 1 -- 172.21.15.62:0/1016678823 shutdown_connections 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.605+0000 7f6201f08640 1 -- 172.21.15.62:0/1016678823 wait complete. 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f6201f08640 1 Processor -- start 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f6201f08640 1 -- start start 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f6201f08640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc100a80 0x7f61fc1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f6201f08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc101420 0x7f61fc198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f6201f08640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61fc106a80 0x7f61fc19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f6201f08640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc110e80 con 0x7f61fc101420 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f6201f08640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc110d00 con 0x7f61fc100a80 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f6201f08640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc111000 con 0x7f61fc106a80 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f61faffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc101420 0x7f61fc198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc100a80 0x7f61fc1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc100a80 0x7f61fc1984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:44578/0 (socket says 172.21.15.62:44578) 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f61faffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc101420 0x7f61fc198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42182/0 (socket says 172.21.15.62:42182) 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.606+0000 7f61fb7fe640 1 -- 172.21.15.62:0/2813469044 learned_addr learned my addr 172.21.15.62:0/2813469044 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61fbfff640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61fc106a80 0x7f61fc19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61faffd640 1 -- 172.21.15.62:0/2813469044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61fc106a80 msgr2=0x7f61fc19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61faffd640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61fc106a80 0x7f61fc19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61faffd640 1 -- 172.21.15.62:0/2813469044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc100a80 msgr2=0x7f61fc1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:08.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61faffd640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc100a80 0x7f61fc1984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61faffd640 1 -- 172.21.15.62:0/2813469044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc19d490 con 0x7f61fc101420 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61fbfff640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61fc106a80 0x7f61fc19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc100a80 0x7f61fc1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61faffd640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc101420 0x7f61fc198a20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f61ec005940 tx=0x7f61ec0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61f8ff9640 1 -- 172.21.15.62:0/2813469044 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec03f070 con 0x7f61fc101420 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61f8ff9640 1 -- 172.21.15.62:0/2813469044 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ec0044c0 con 0x7f61fc101420 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f6201f08640 1 -- 172.21.15.62:0/2813469044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61fc19d720 con 0x7f61fc101420 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f61f8ff9640 1 -- 172.21.15.62:0/2813469044 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec035070 con 0x7f61fc101420 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.607+0000 7f6201f08640 1 -- 172.21.15.62:0/2813469044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61fc1a5060 con 0x7f61fc101420 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.610+0000 7f61f8ff9640 1 -- 172.21.15.62:0/2813469044 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f61ec03b460 con 0x7f61fc101420 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f62042e1740 1 Processor -- start 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f62042e1740 1 -- start start 2024-09-15T23:16:08.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f62042e1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 0x5588245a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f62042e1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245b0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f62042e1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 0x5588245b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f62042e1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55882450c090 con 0x5588245b0bb0 2024-09-15T23:16:08.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f62042e1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5588244b3a70 con 0x5588245a62f0 2024-09-15T23:16:08.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f62042e1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558824508050 con 0x5588245a59d0 2024-09-15T23:16:08.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61f8ff9640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f61e40a3240 0x7f61e40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245b0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245b0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:44588/0 (socket says 172.21.15.62:44588) 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61fb7fe640 1 -- 172.21.15.62:0/445201732 learned_addr learned my addr 172.21.15.62:0/445201732 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61faffd640 1 --2- 172.21.15.62:0/445201732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 0x5588245b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61fbfff640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f61e40a3240 0x7f61e40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61f8ff9640 1 -- 172.21.15.62:0/2813469044 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f61ec0f01b0 con 0x7f61fc101420 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61fbfff640 1 --2- 172.21.15.62:0/445201732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 0x5588245a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61fb7fe640 1 -- 172.21.15.62:0/445201732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 msgr2=0x5588245a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/445201732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 0x5588245a5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61fb7fe640 1 -- 172.21.15.62:0/445201732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 msgr2=0x5588245b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.611+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/445201732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 0x5588245b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.612+0000 7f61fb7fe640 1 -- 172.21.15.62:0/445201732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588244c0130 con 0x5588245a62f0 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.612+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/445201732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245b0670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f61f000bfc0 tx=0x7f61f0010a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3352e0b1bef0c166 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.612+0000 7f61fbfff640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f61e40a3240 0x7f61e40a5700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009270 tx=0x7f61f40091e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:08.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.612+0000 7f61e1ffb640 1 -- 172.21.15.62:0/445201732 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0015b70 con 0x5588245a62f0 2024-09-15T23:16:08.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.612+0000 7f61e1ffb640 1 -- 172.21.15.62:0/445201732 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f0015d10 con 0x5588245a62f0 2024-09-15T23:16:08.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.612+0000 7f61e1ffb640 1 -- 172.21.15.62:0/445201732 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f001d810 con 0x5588245a62f0 2024-09-15T23:16:08.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 -- 172.21.15.62:0/445201732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 msgr2=0x5588245b0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:08.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 --2- 172.21.15.62:0/445201732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245b0670 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f61f000bfc0 tx=0x7f61f0010a10 comp rx=0 tx=0).stop 2024-09-15T23:16:08.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 -- 172.21.15.62:0/445201732 shutdown_connections 2024-09-15T23:16:08.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 --2- 172.21.15.62:0/445201732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 0x5588245b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 --2- 172.21.15.62:0/445201732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245b0670 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 --2- 172.21.15.62:0/445201732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 0x5588245a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 -- 172.21.15.62:0/445201732 >> 172.21.15.62:0/445201732 conn(0x5588245a1f60 msgr2=0x5588245a2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 -- 172.21.15.62:0/445201732 shutdown_connections 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 -- 172.21.15.62:0/445201732 wait complete. 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.613+0000 7f62042e1740 1 -- start start 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f62042e1740 1 Processor -- start 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f62042e1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a59d0 0x5588245b0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f62042e1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f62042e1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245b0bb0 0x5588245adc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f62042e1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5588245b3780 con 0x5588245a59d0 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f62042e1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5588244c6f20 con 0x5588245a62f0 2024-09-15T23:16:08.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f62042e1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5588245b3900 con 0x5588245b0bb0 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f62042e1740 10 client.0 _ll_register_callbacks cb 0x558824490030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a59d0 0x5588245b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a59d0 0x5588245b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42190/0 (socket says 172.21.15.62:42190) 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fb7fe640 1 -- 172.21.15.62:0/1371031292 learned_addr learned my addr 172.21.15.62:0/1371031292 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61faffd640 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fbfff640 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245b0bb0 0x5588245adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fb7fe640 1 -- 172.21.15.62:0/1371031292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245b0bb0 msgr2=0x5588245adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245b0bb0 0x5588245adc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fb7fe640 1 -- 172.21.15.62:0/1371031292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 msgr2=0x5588245ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245ad6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fb7fe640 1 -- 172.21.15.62:0/1371031292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55882427d2b0 con 0x5588245a59d0 2024-09-15T23:16:08.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61faffd640 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.614+0000 7f61fbfff640 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245b0bb0 0x5588245adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a59d0 0x5588245b0370 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f61f0010550 tx=0x7f61f000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b17c1d57d10b2d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f61daffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f61daffd640 1 -- 172.21.15.62:0/1371031292 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f002b030 con 0x5588245a59d0 2024-09-15T23:16:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f61daffd640 1 -- 172.21.15.62:0/1371031292 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f61f0023070 con 0x5588245a59d0 2024-09-15T23:16:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f62042e1740 1 -- 172.21.15.62:0/1371031292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61f0015d10 con 0x5588245a59d0 2024-09-15T23:16:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f61daffd640 1 -- 172.21.15.62:0/1371031292 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f61f0035050 con 0x5588245a59d0 2024-09-15T23:16:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f62042e1740 10 client.15411 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:08.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f62042e1740 1 -- 172.21.15.62:0/1371031292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5588245bf010 con 0x5588245a59d0 2024-09-15T23:16:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f61daffd640 1 -- 172.21.15.62:0/1371031292 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f0015d10 con 0x5588245a59d0 2024-09-15T23:16:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.615+0000 7f61daffd640 1 -- 172.21.15.62:0/1371031292 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f61f000e550 con 0x5588245a59d0 2024-09-15T23:16:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.616+0000 7f62042e1740 10 client.15411 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.616+0000 7f62042e1740 10 client.15411 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:16:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.616+0000 7f62042e1740 20 client.15411 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.616+0000 7f62042e1740 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5588245c1320 0x5588245c2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.616+0000 7f62042e1740 4 client.15411 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:08.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.616+0000 7f62042e1740 1 -- 172.21.15.62:0/1371031292 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558824519600 con 0x5588245c1320 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.616+0000 7f61faffd640 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5588245c1320 0x5588245c2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.616+0000 7f61faffd640 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5588245c1320 0x5588245c2790 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=83140eaea751dbf0 server_cookie=990a669f0f876f7c in_seq=0 out_seq=0 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.616+0000 7f61daffd640 10 client.15411 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.617+0000 7f61daffd640 1 -- 172.21.15.62:0/1371031292 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558824519600 con 0x5588245c1320 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.617+0000 7f61daffd640 10 client.15411 handle_command_reply: tid=0 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.617+0000 7f62042e1740 1 client.15411 shutdown 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.617+0000 7f62042e1740 1 -- 172.21.15.62:0/1371031292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a59d0 msgr2=0x5588245b0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.617+0000 7f62042e1740 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a59d0 0x5588245b0370 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f61f0010550 tx=0x7f61f000ee30 comp rx=0 tx=0).stop 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f62042e1740 1 -- 172.21.15.62:0/1371031292 shutdown_connections 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f62042e1740 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5588245c1320 0x5588245c2790 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f62042e1740 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245b0bb0 0x5588245adc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f62042e1740 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a62f0 0x5588245ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f62042e1740 1 --2- 172.21.15.62:0/1371031292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a59d0 0x5588245b0370 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f61daffd640 0 client.15411 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:16:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f62042e1740 1 -- 172.21.15.62:0/1371031292 >> 172.21.15.62:0/1371031292 conn(0x5588245a1f60 msgr2=0x5588245a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f62042e1740 1 -- 172.21.15.62:0/1371031292 shutdown_connections 2024-09-15T23:16:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f62042e1740 1 -- 172.21.15.62:0/1371031292 wait complete. 2024-09-15T23:16:08.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.618+0000 7f62042e1740 20 client.15411 trim_cache size 0 max 16384 2024-09-15T23:16:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f62042e1740 1 Processor -- start 2024-09-15T23:16:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f62042e1740 1 -- start start 2024-09-15T23:16:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f62042e1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 0x5588245c2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f62042e1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a62f0 0x5588245adc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f62042e1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245b0bb0 0x5588245af520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f62042e1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5588245c0ad0 con 0x5588245a62f0 2024-09-15T23:16:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f62042e1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5588245c11a0 con 0x5588245b0bb0 2024-09-15T23:16:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f62042e1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5588245c1870 con 0x5588245a59d0 2024-09-15T23:16:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61faffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a62f0 0x5588245adc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61faffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a62f0 0x5588245adc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42194/0 (socket says 172.21.15.62:42194) 2024-09-15T23:16:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61faffd640 1 -- 172.21.15.62:0/1186824390 learned_addr learned my addr 172.21.15.62:0/1186824390 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61fbfff640 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245b0bb0 0x5588245af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 0x5588245c2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61faffd640 1 -- 172.21.15.62:0/1186824390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 msgr2=0x5588245c2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61faffd640 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 0x5588245c2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61faffd640 1 -- 172.21.15.62:0/1186824390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245b0bb0 msgr2=0x5588245af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61faffd640 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245b0bb0 0x5588245af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61faffd640 1 -- 172.21.15.62:0/1186824390 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588245b3890 con 0x5588245a62f0 2024-09-15T23:16:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 0x5588245c2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61fbfff640 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245b0bb0 0x5588245af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.636+0000 7f61faffd640 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a62f0 0x5588245adc10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0ed980 tx=0x7f61ec08e220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290e7741c535aad8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f61e17fa640 1 -- 172.21.15.62:0/1186824390 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec048040 con 0x5588245a62f0 2024-09-15T23:16:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f61e17fa640 1 -- 172.21.15.62:0/1186824390 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ec0c0800 con 0x5588245a62f0 2024-09-15T23:16:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f61e17fa640 1 -- 172.21.15.62:0/1186824390 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec0d1f30 con 0x5588245a62f0 2024-09-15T23:16:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f62042e1740 1 -- 172.21.15.62:0/1186824390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a62f0 msgr2=0x5588245adc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f62042e1740 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a62f0 0x5588245adc10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f61ec0ed980 tx=0x7f61ec08e220 comp rx=0 tx=0).stop 2024-09-15T23:16:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f62042e1740 1 -- 172.21.15.62:0/1186824390 shutdown_connections 2024-09-15T23:16:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f62042e1740 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245b0bb0 0x5588245af520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f62042e1740 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245a62f0 0x5588245adc10 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f62042e1740 1 --2- 172.21.15.62:0/1186824390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a59d0 0x5588245c2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f62042e1740 1 -- 172.21.15.62:0/1186824390 >> 172.21.15.62:0/1186824390 conn(0x5588245a1f60 msgr2=0x5588245aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f62042e1740 1 -- 172.21.15.62:0/1186824390 shutdown_connections 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.637+0000 7f62042e1740 1 -- 172.21.15.62:0/1186824390 wait complete. 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f62042e1740 1 -- start start 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f62042e1740 1 Processor -- start 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f62042e1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a59d0 0x5588245c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f62042e1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a62f0 0x5588245c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f62042e1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 0x5588245c6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f62042e1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5588245c2610 con 0x5588245b0bb0 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f62042e1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5588245c1f40 con 0x5588245a59d0 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f62042e1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5588245c2ce0 con 0x5588245a62f0 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f62042e1740 10 client.0 _ll_register_callbacks cb 0x7f61ec0b8040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f61fbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 0x5588245c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a59d0 0x5588245c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f61fb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a59d0 0x5588245c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:44610/0 (socket says 172.21.15.62:44610) 2024-09-15T23:16:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.638+0000 7f61fb7fe640 1 -- 172.21.15.62:0/2914306026 learned_addr learned my addr 172.21.15.62:0/2914306026 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61faffd640 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a62f0 0x5588245c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61fbfff640 1 -- 172.21.15.62:0/2914306026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a62f0 msgr2=0x5588245c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61fbfff640 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a62f0 0x5588245c5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61fbfff640 1 -- 172.21.15.62:0/2914306026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a59d0 msgr2=0x5588245c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61fbfff640 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a59d0 0x5588245c5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61fbfff640 1 -- 172.21.15.62:0/2914306026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558824629bf0 con 0x5588245b0bb0 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61faffd640 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a62f0 0x5588245c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a59d0 0x5588245c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61fbfff640 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 0x5588245c6490 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009b60 tx=0x7f61f4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f66527f6fcc7fcf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61daffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:08.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61daffd640 1 -- 172.21.15.62:0/2914306026 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f40275d0 con 0x5588245b0bb0 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61daffd640 1 -- 172.21.15.62:0/2914306026 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f61f4027770 con 0x5588245b0bb0 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f62042e1740 1 -- 172.21.15.62:0/2914306026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61ec0c0800 con 0x5588245b0bb0 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f61daffd640 1 -- 172.21.15.62:0/2914306026 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f61f4037050 con 0x5588245b0bb0 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f62042e1740 10 client.15417 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.639+0000 7f62042e1740 1 -- 172.21.15.62:0/2914306026 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55882462af30 con 0x5588245b0bb0 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.640+0000 7f61daffd640 1 -- 172.21.15.62:0/2914306026 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61ec0c0800 con 0x5588245b0bb0 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.640+0000 7f61daffd640 1 -- 172.21.15.62:0/2914306026 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f61f4029a30 con 0x5588245b0bb0 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.640+0000 7f62042e1740 10 client.15417 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.640+0000 7f62042e1740 10 client.15417 resolve_mds: resolved name 'i' to daemon mds.i 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.640+0000 7f62042e1740 20 client.15417 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.640+0000 7f62042e1740 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55882462fbe0 0x55882462e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.640+0000 7f62042e1740 4 client.15417 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:16:08.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.640+0000 7f62042e1740 1 -- 172.21.15.62:0/2914306026 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "status"}) -- 0x558824513b10 con 0x55882462fbe0 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.640+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55882462fbe0 0x55882462e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f61fb7fe640 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55882462fbe0 0x55882462e8f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4eadbc21257e9bd7 server_cookie=749bfef58891136 in_seq=0 out_seq=0 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f61daffd640 10 client.15417 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f61daffd640 1 -- 172.21.15.62:0/2914306026 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558824513b10 con 0x55882462fbe0 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f61daffd640 10 client.15417 handle_command_reply: tid=0 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 client.15417 shutdown 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 -- 172.21.15.62:0/2914306026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 msgr2=0x5588245c6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 0x5588245c6490 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009b60 tx=0x7f61f4004ab0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 -- 172.21.15.62:0/2914306026 shutdown_connections 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55882462fbe0 0x55882462e8f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588245b0bb0 0x5588245c6490 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588245a62f0 0x5588245c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f61daffd640 0 client.15417 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 --2- 172.21.15.62:0/2914306026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588245a59d0 0x5588245c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 -- 172.21.15.62:0/2914306026 >> 172.21.15.62:0/2914306026 conn(0x5588245a1f60 msgr2=0x5588245a0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 -- 172.21.15.62:0/2914306026 shutdown_connections 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 1 -- 172.21.15.62:0/2914306026 wait complete. 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.641+0000 7f62042e1740 20 client.15417 trim_cache size 0 max 16384 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 2, 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24466, 2024-09-15T23:16:08.650 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:16:08.651 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:16:08.651 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:16:08.651 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 121.016792907, 2024-09-15T23:16:08.651 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 25, 2024-09-15T23:16:08.651 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 92, 2024-09-15T23:16:08.651 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 92, 2024-09-15T23:16:08.651 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 175.77710321199999 2024-09-15T23:16:08.651 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 -- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f61e40a3240 msgr2=0x7f61e40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f61e40a3240 0x7f61e40a5700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009270 tx=0x7f61f40091e0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 -- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc101420 msgr2=0x7f61fc198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc101420 0x7f61fc198a20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f61ec005940 tx=0x7f61ec0040c0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 -- 172.21.15.62:0/2813469044 shutdown_connections 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f61e40a3240 0x7f61e40a5700 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f61fc106a80 0x7f61fc19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61fc101420 0x7f61fc198a20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 --2- 172.21.15.62:0/2813469044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61fc100a80 0x7f61fc1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:08.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 -- 172.21.15.62:0/2813469044 >> 172.21.15.62:0/2813469044 conn(0x7f61fc0fc7a0 msgr2=0x7f61fc06aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:08.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 -- 172.21.15.62:0/2813469044 shutdown_connections 2024-09-15T23:16:08.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:08.642+0000 7f6201f08640 1 -- 172.21.15.62:0/2813469044 wait complete. 2024-09-15T23:16:09.229 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:08 smithi062 ceph-mon[30949]: pgmap v366: 129 pgs: 129 active+clean; 658 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-09-15T23:16:09.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:08 smithi067 ceph-mon[35665]: pgmap v366: 129 pgs: 129 active+clean; 658 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-09-15T23:16:09.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:08 smithi148 ceph-mon[35610]: pgmap v366: 129 pgs: 129 active+clean; 658 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.4 KiB/s wr, 11 op/s 2024-09-15T23:16:09.472 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.h status 2024-09-15T23:16:09.738 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:10.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 -- 172.21.15.62:0/2082078201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43841082e0 msgr2=0x7f438410ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:10.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 --2- 172.21.15.62:0/2082078201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43841082e0 0x7f438410ec50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f437c009cd0 tx=0x7f437c02a910 comp rx=0 tx=0).stop 2024-09-15T23:16:10.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 -- 172.21.15.62:0/2082078201 shutdown_connections 2024-09-15T23:16:10.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 --2- 172.21.15.62:0/2082078201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43841082e0 0x7f438410ec50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 --2- 172.21.15.62:0/2082078201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4384103760 0x7f4384107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 --2- 172.21.15.62:0/2082078201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4384102db0 0x7f4384103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 -- 172.21.15.62:0/2082078201 >> 172.21.15.62:0/2082078201 conn(0x7f43840fc7d0 msgr2=0x7f43840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:10.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 -- 172.21.15.62:0/2082078201 shutdown_connections 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 -- 172.21.15.62:0/2082078201 wait complete. 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.222+0000 7f438a164640 1 Processor -- start 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f438a164640 1 -- start start 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f438a164640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4384102db0 0x7f4384100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f438a164640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4384103760 0x7f4384100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f438a164640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43841082e0 0x7f4384101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f438a164640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f43841120f0 con 0x7f4384102db0 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f438a164640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4384111f70 con 0x7f43841082e0 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f438a164640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4384112270 con 0x7f4384103760 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f43837fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4384102db0 0x7f4384100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f43837fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4384102db0 0x7f4384100a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42218/0 (socket says 172.21.15.62:42218) 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f43837fe640 1 -- 172.21.15.62:0/3511230655 learned_addr learned my addr 172.21.15.62:0/3511230655 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:10.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f4383fff640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43841082e0 0x7f4384101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f4382ffd640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4384103760 0x7f4384100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f43837fe640 1 -- 172.21.15.62:0/3511230655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4384103760 msgr2=0x7f4384100f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f43837fe640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4384103760 0x7f4384100f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f43837fe640 1 -- 172.21.15.62:0/3511230655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43841082e0 msgr2=0x7f4384101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f43837fe640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43841082e0 0x7f4384101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f43837fe640 1 -- 172.21.15.62:0/3511230655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43841a6a30 con 0x7f4384102db0 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f4382ffd640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4384103760 0x7f4384100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.223+0000 7f43837fe640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4384102db0 0x7f4384100a10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f437400c920 tx=0x7f437400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.224+0000 7f4380ff9640 1 -- 172.21.15.62:0/3511230655 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374007c70 con 0x7f4384102db0 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.224+0000 7f438a164640 1 -- 172.21.15.62:0/3511230655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43841a6d20 con 0x7f4384102db0 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.224+0000 7f4380ff9640 1 -- 172.21.15.62:0/3511230655 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4374007e10 con 0x7f4384102db0 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.224+0000 7f4380ff9640 1 -- 172.21.15.62:0/3511230655 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374018890 con 0x7f4384102db0 2024-09-15T23:16:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.224+0000 7f438a164640 1 -- 172.21.15.62:0/3511230655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43841a7200 con 0x7f4384102db0 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.226+0000 7f4380ff9640 1 -- 172.21.15.62:0/3511230655 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4374018b10 con 0x7f4384102db0 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.226+0000 7f438c53d740 1 Processor -- start 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f438c53d740 1 -- start start 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f438c53d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 0x55d728aa2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f438c53d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 0x55d728aad670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f438c53d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728aaffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f438c53d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d728a09090 con 0x55d728aadbb0 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f438c53d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d7289b0a70 con 0x55d728aa32f0 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f438c53d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d728a05050 con 0x55d728aa29d0 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4380ff9640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f436c0a3240 0x7f436c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4382ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728aaffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4382ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728aaffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42222/0 (socket says 172.21.15.62:42222) 2024-09-15T23:16:10.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4382ffd640 1 -- 172.21.15.62:0/1992842115 learned_addr learned my addr 172.21.15.62:0/1992842115 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f43837fe640 1 --2- 172.21.15.62:0/1992842115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 0x55d728aad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4383fff640 1 --2- 172.21.15.62:0/1992842115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 0x55d728aa2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4383fff640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f436c0a3240 0x7f436c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4380ff9640 1 -- 172.21.15.62:0/3511230655 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f43740cfde0 con 0x7f4384102db0 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4382ffd640 1 -- 172.21.15.62:0/1992842115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 msgr2=0x55d728aa2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4382ffd640 1 --2- 172.21.15.62:0/1992842115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 0x55d728aa2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4382ffd640 1 -- 172.21.15.62:0/1992842115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 msgr2=0x55d728aad670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4382ffd640 1 --2- 172.21.15.62:0/1992842115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 0x55d728aad670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.227+0000 7f4382ffd640 1 -- 172.21.15.62:0/1992842115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7289bd130 con 0x55d728aadbb0 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.228+0000 7f4383fff640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f436c0a3240 0x7f436c0a5700 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f437c009430 tx=0x7f437c0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.228+0000 7f4382ffd640 1 --2- 172.21.15.62:0/1992842115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728aaffa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f437800a970 tx=0x7f437800ae30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29e1a4c092c48f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:10.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.228+0000 7f4369ffb640 1 -- 172.21.15.62:0/1992842115 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378015070 con 0x55d728aadbb0 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.228+0000 7f4369ffb640 1 -- 172.21.15.62:0/1992842115 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f437800cc00 con 0x55d728aadbb0 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.228+0000 7f4369ffb640 1 -- 172.21.15.62:0/1992842115 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43780185c0 con 0x55d728aadbb0 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 -- 172.21.15.62:0/1992842115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 msgr2=0x55d728aaffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 --2- 172.21.15.62:0/1992842115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728aaffa0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f437800a970 tx=0x7f437800ae30 comp rx=0 tx=0).stop 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 -- 172.21.15.62:0/1992842115 shutdown_connections 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 --2- 172.21.15.62:0/1992842115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728aaffa0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 --2- 172.21.15.62:0/1992842115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 0x55d728aad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 --2- 172.21.15.62:0/1992842115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 0x55d728aa2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 -- 172.21.15.62:0/1992842115 >> 172.21.15.62:0/1992842115 conn(0x55d728a9ef60 msgr2=0x55d728a9f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 -- 172.21.15.62:0/1992842115 shutdown_connections 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 -- 172.21.15.62:0/1992842115 wait complete. 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 -- start start 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.229+0000 7f438c53d740 1 Processor -- start 2024-09-15T23:16:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f438c53d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa29d0 0x55d728aad380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f438c53d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 0x55d728aaa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f438c53d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aadbb0 0x55d728aaac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f438c53d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d728ab0780 con 0x55d728aa29d0 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f438c53d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d7289c3f20 con 0x55d728aa32f0 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f438c53d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d728ab0900 con 0x55d728aadbb0 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f438c53d740 10 client.0 _ll_register_callbacks cb 0x55d72898d030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f43837fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa29d0 0x55d728aad380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f43837fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa29d0 0x55d728aad380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42224/0 (socket says 172.21.15.62:42224) 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f4382ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 0x55d728aaa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f43837fe640 1 -- 172.21.15.62:0/164814898 learned_addr learned my addr 172.21.15.62:0/164814898 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f4383fff640 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aadbb0 0x55d728aaac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f4382ffd640 1 -- 172.21.15.62:0/164814898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aadbb0 msgr2=0x55d728aaac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f4382ffd640 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aadbb0 0x55d728aaac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f4382ffd640 1 -- 172.21.15.62:0/164814898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa29d0 msgr2=0x55d728aad380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f4382ffd640 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa29d0 0x55d728aad380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f4382ffd640 1 -- 172.21.15.62:0/164814898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d72877a2b0 con 0x55d728aa32f0 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f43837fe640 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa29d0 0x55d728aad380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f4382ffd640 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 0x55d728aaa6d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4378000c00 tx=0x7f437800a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaf68a8094278dcb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.230+0000 7f4362ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f4362ffd640 1 -- 172.21.15.62:0/164814898 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378020070 con 0x55d728aa32f0 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f438c53d740 1 -- 172.21.15.62:0/164814898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f437800cc00 con 0x55d728aa32f0 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f4362ffd640 1 -- 172.21.15.62:0/164814898 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f437801a450 con 0x55d728aa32f0 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f4362ffd640 1 -- 172.21.15.62:0/164814898 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4378015040 con 0x55d728aa32f0 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f438c53d740 10 client.25097 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f438c53d740 1 -- 172.21.15.62:0/164814898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d728abc010 con 0x55d728aa32f0 2024-09-15T23:16:10.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f4362ffd640 1 -- 172.21.15.62:0/164814898 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f437800cc00 con 0x55d728aa32f0 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f4362ffd640 1 -- 172.21.15.62:0/164814898 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f4378024600 con 0x55d728aa32f0 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f438c53d740 10 client.25097 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f438c53d740 10 client.25097 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.231+0000 7f438c53d740 20 client.25097 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.232+0000 7f438c53d740 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55d728abe220 0x55d728abd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.232+0000 7f438c53d740 4 client.25097 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.232+0000 7f438c53d740 1 -- 172.21.15.62:0/164814898 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d728a16600 con 0x55d728abe220 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.232+0000 7f43837fe640 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55d728abe220 0x55d728abd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.232+0000 7f43837fe640 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55d728abe220 0x55d728abd480 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9f250c71f63ea09 server_cookie=f98252624fd80a0b in_seq=0 out_seq=0 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.232+0000 7f4362ffd640 10 client.25097 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f4362ffd640 1 -- 172.21.15.62:0/164814898 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d728a16600 con 0x55d728abe220 2024-09-15T23:16:10.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f4362ffd640 10 client.25097 handle_command_reply: tid=0 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 client.25097 shutdown 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 -- 172.21.15.62:0/164814898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 msgr2=0x55d728aaa6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 0x55d728aaa6d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4378000c00 tx=0x7f437800a230 comp rx=0 tx=0).stop 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 -- 172.21.15.62:0/164814898 shutdown_connections 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55d728abe220 0x55d728abd480 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aadbb0 0x55d728aaac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa32f0 0x55d728aaa6d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 --2- 172.21.15.62:0/164814898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa29d0 0x55d728aad380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f4362ffd640 0 client.25097 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 -- 172.21.15.62:0/164814898 >> 172.21.15.62:0/164814898 conn(0x55d728a9ef60 msgr2=0x55d728aa4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 -- 172.21.15.62:0/164814898 shutdown_connections 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 1 -- 172.21.15.62:0/164814898 wait complete. 2024-09-15T23:16:10.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.233+0000 7f438c53d740 20 client.25097 trim_cache size 0 max 16384 2024-09-15T23:16:10.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.251+0000 7f438c53d740 1 Processor -- start 2024-09-15T23:16:10.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.251+0000 7f438c53d740 1 -- start start 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f438c53d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 0x55d728abd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f438c53d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa32f0 0x55d728aa7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f438c53d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aadbb0 0x55d728aa7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f438c53d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d728abd9d0 con 0x55d728aa32f0 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f438c53d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d728abe0a0 con 0x55d728aadbb0 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f438c53d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d728abe770 con 0x55d728aa29d0 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4382ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa32f0 0x55d728aa7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4382ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa32f0 0x55d728aa7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42226/0 (socket says 172.21.15.62:42226) 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4382ffd640 1 -- 172.21.15.62:0/3376272951 learned_addr learned my addr 172.21.15.62:0/3376272951 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4383fff640 1 --2- 172.21.15.62:0/3376272951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aadbb0 0x55d728aa7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f43837fe640 1 --2- 172.21.15.62:0/3376272951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 0x55d728abd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4382ffd640 1 -- 172.21.15.62:0/3376272951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 msgr2=0x55d728abd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:10.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4382ffd640 1 --2- 172.21.15.62:0/3376272951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 0x55d728abd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4382ffd640 1 -- 172.21.15.62:0/3376272951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aadbb0 msgr2=0x55d728aa7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:10.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4382ffd640 1 --2- 172.21.15.62:0/3376272951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aadbb0 0x55d728aa7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4382ffd640 1 -- 172.21.15.62:0/3376272951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d728aa3120 con 0x55d728aa32f0 2024-09-15T23:16:10.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.252+0000 7f4382ffd640 1 --2- 172.21.15.62:0/3376272951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa32f0 0x55d728aa7970 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f437800a700 tx=0x7f4378006c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd28d820546a8e3d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:10.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f43697fa640 1 -- 172.21.15.62:0/3376272951 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437800a020 con 0x55d728aa32f0 2024-09-15T23:16:10.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f43697fa640 1 -- 172.21.15.62:0/3376272951 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4378011c90 con 0x55d728aa32f0 2024-09-15T23:16:10.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f43697fa640 1 -- 172.21.15.62:0/3376272951 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4378020040 con 0x55d728aa32f0 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f438c53d740 1 -- 172.21.15.62:0/3376272951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa32f0 msgr2=0x55d728aa7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f438c53d740 1 --2- 172.21.15.62:0/3376272951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa32f0 0x55d728aa7970 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f437800a700 tx=0x7f4378006c50 comp rx=0 tx=0).stop 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f438c53d740 1 -- 172.21.15.62:0/3376272951 shutdown_connections 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f438c53d740 1 --2- 172.21.15.62:0/3376272951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aadbb0 0x55d728aa7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f438c53d740 1 --2- 172.21.15.62:0/3376272951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aa32f0 0x55d728aa7970 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f438c53d740 1 --2- 172.21.15.62:0/3376272951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa29d0 0x55d728abd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.253+0000 7f438c53d740 1 -- 172.21.15.62:0/3376272951 >> 172.21.15.62:0/3376272951 conn(0x55d728a9ef60 msgr2=0x55d728a99ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 -- 172.21.15.62:0/3376272951 shutdown_connections 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 -- 172.21.15.62:0/3376272951 wait complete. 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 -- start start 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 Processor -- start 2024-09-15T23:16:10.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa29d0 0x55d728ac08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa32f0 0x55d728ac0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728ac1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d728abf510 con 0x55d728aadbb0 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d728abee40 con 0x55d728aa29d0 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d728abfbe0 con 0x55d728aa32f0 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.254+0000 7f438c53d740 10 client.0 _ll_register_callbacks cb 0x7f437800a7c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f43837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa29d0 0x55d728ac08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f4383fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728ac1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f43837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa29d0 0x55d728ac08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:44650/0 (socket says 172.21.15.62:44650) 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f43837fe640 1 -- 172.21.15.62:0/3537060377 learned_addr learned my addr 172.21.15.62:0/3537060377 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f4382ffd640 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa32f0 0x55d728ac0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f43837fe640 1 -- 172.21.15.62:0/3537060377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa32f0 msgr2=0x55d728ac0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f43837fe640 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa32f0 0x55d728ac0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f43837fe640 1 -- 172.21.15.62:0/3537060377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 msgr2=0x55d728ac1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f43837fe640 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728ac1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f43837fe640 1 -- 172.21.15.62:0/3537060377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d728b26bc0 con 0x55d728aa29d0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f4383fff640 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728ac1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f43837fe640 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa29d0 0x55d728ac08c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f43740cd5b0 tx=0x7f437406bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=957743b0bc1f8906 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f4362ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f4362ffd640 1 -- 172.21.15.62:0/3537060377 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43740a0430 con 0x55d728aa29d0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f4362ffd640 1 -- 172.21.15.62:0/3537060377 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f43740bc0b0 con 0x55d728aa29d0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.255+0000 7f438c53d740 1 -- 172.21.15.62:0/3537060377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4378011c90 con 0x55d728aa29d0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f4362ffd640 1 -- 172.21.15.62:0/3537060377 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f43740a4cf0 con 0x55d728aa29d0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f438c53d740 10 client.25103 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f438c53d740 1 -- 172.21.15.62:0/3537060377 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d728b27f00 con 0x55d728aa29d0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f4362ffd640 1 -- 172.21.15.62:0/3537060377 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4378011c90 con 0x55d728aa29d0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f4362ffd640 1 -- 172.21.15.62:0/3537060377 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f43740a4860 con 0x55d728aa29d0 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f438c53d740 10 client.25103 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f438c53d740 10 client.25103 resolve_mds: resolved name 'h' to daemon mds.h 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f438c53d740 20 client.25103 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f438c53d740 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55d728b2c9c0 0x55d728b2b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f438c53d740 4 client.25103 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f438c53d740 1 -- 172.21.15.62:0/3537060377 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "status"}) -- 0x55d728a10b10 con 0x55d728b2c9c0 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.256+0000 7f4382ffd640 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55d728b2c9c0 0x55d728b2b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f4362ffd640 10 client.25103 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f4382ffd640 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55d728b2c9c0 0x55d728b2b6d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2b4693809fe21455 server_cookie=f3b5812fd396b3ff in_seq=0 out_seq=0 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f4362ffd640 1 -- 172.21.15.62:0/3537060377 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d728a10b10 con 0x55d728b2c9c0 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f4362ffd640 10 client.25103 handle_command_reply: tid=0 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 client.25103 shutdown 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 -- 172.21.15.62:0/3537060377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa29d0 msgr2=0x55d728ac08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa29d0 0x55d728ac08c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f43740cd5b0 tx=0x7f437406bcf0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 -- 172.21.15.62:0/3537060377 shutdown_connections 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55d728b2c9c0 0x55d728b2b6d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d728aadbb0 0x55d728ac1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f4362ffd640 0 client.25103 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d728aa32f0 0x55d728ac0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 --2- 172.21.15.62:0/3537060377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d728aa29d0 0x55d728ac08c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 -- 172.21.15.62:0/3537060377 >> 172.21.15.62:0/3537060377 conn(0x55d728a9ef60 msgr2=0x55d728a9d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 -- 172.21.15.62:0/3537060377 shutdown_connections 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.257+0000 7f438c53d740 1 -- 172.21.15.62:0/3537060377 wait complete. 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438c53d740 20 client.25103 trim_cache size 0 max 16384 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 1, 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24470, 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:16:10.263 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 123.641572305, 2024-09-15T23:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 25, 2024-09-15T23:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 92, 2024-09-15T23:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 92, 2024-09-15T23:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 179.98044121999999 2024-09-15T23:16:10.264 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 -- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f436c0a3240 msgr2=0x7f436c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f436c0a3240 0x7f436c0a5700 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f437c009430 tx=0x7f437c0095c0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 -- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4384102db0 msgr2=0x7f4384100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4384102db0 0x7f4384100a10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f437400c920 tx=0x7f437400cde0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 -- 172.21.15.62:0/3511230655 shutdown_connections 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f436c0a3240 0x7f436c0a5700 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43841082e0 0x7f4384101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4384103760 0x7f4384100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 --2- 172.21.15.62:0/3511230655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4384102db0 0x7f4384100a10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 -- 172.21.15.62:0/3511230655 >> 172.21.15.62:0/3511230655 conn(0x7f43840fc7d0 msgr2=0x7f43840fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 -- 172.21.15.62:0/3511230655 shutdown_connections 2024-09-15T23:16:10.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:10.258+0000 7f438a164640 1 -- 172.21.15.62:0/3511230655 wait complete. 2024-09-15T23:16:10.963 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.b status 2024-09-15T23:16:11.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:10 smithi062 ceph-mon[30949]: pgmap v367: 129 pgs: 129 active+clean; 658 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-09-15T23:16:11.243 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:11.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:10 smithi067 ceph-mon[35665]: pgmap v367: 129 pgs: 129 active+clean; 658 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-09-15T23:16:11.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:10 smithi148 ceph-mon[35610]: pgmap v367: 129 pgs: 129 active+clean; 658 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-09-15T23:16:11.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.723+0000 7f2bc9faa640 1 -- 172.21.15.62:0/4231212703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4102d40 msgr2=0x7f2bc4103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:11.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.723+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/4231212703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4102d40 0x7f2bc4103120 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009900 tx=0x7f2bb402a480 comp rx=0 tx=0).stop 2024-09-15T23:16:11.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.723+0000 7f2bc9faa640 1 -- 172.21.15.62:0/4231212703 shutdown_connections 2024-09-15T23:16:11.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.723+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/4231212703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bc410c170 0x7f2bc410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.723+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/4231212703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc4103660 0x7f2bc410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.723+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/4231212703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4102d40 0x7f2bc4103120 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.723+0000 7f2bc9faa640 1 -- 172.21.15.62:0/4231212703 >> 172.21.15.62:0/4231212703 conn(0x7f2bc40fc7a0 msgr2=0x7f2bc40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.723+0000 7f2bc9faa640 1 -- 172.21.15.62:0/4231212703 shutdown_connections 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.723+0000 7f2bc9faa640 1 -- 172.21.15.62:0/4231212703 wait complete. 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc9faa640 1 Processor -- start 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc9faa640 1 -- start start 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc9faa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc4102d40 0x7f2bc41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc9faa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4103660 0x7f2bc4198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc9faa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bc410c170 0x7f2bc419cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc9faa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc4111f70 con 0x7f2bc4103660 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc9faa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc4111df0 con 0x7f2bc410c170 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc9faa640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc41120f0 con 0x7f2bc4102d40 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4103660 0x7f2bc4198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4103660 0x7f2bc4198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42254/0 (socket says 172.21.15.62:42254) 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc2ffd640 1 -- 172.21.15.62:0/2108355663 learned_addr learned my addr 172.21.15.62:0/2108355663 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc4102d40 0x7f2bc41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.724+0000 7f2bc3fff640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bc410c170 0x7f2bc419cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc2ffd640 1 -- 172.21.15.62:0/2108355663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc4102d40 msgr2=0x7f2bc41984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc4102d40 0x7f2bc41984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc2ffd640 1 -- 172.21.15.62:0/2108355663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bc410c170 msgr2=0x7f2bc419cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bc410c170 0x7f2bc419cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc2ffd640 1 -- 172.21.15.62:0/2108355663 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc419d4a0 con 0x7f2bc4103660 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc3fff640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bc410c170 0x7f2bc419cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4103660 0x7f2bc4198a30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800d950 tx=0x7f2bb800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc0ff9640 1 -- 172.21.15.62:0/2108355663 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8017030 con 0x7f2bc4103660 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc0ff9640 1 -- 172.21.15.62:0/2108355663 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bb8004020 con 0x7f2bc4103660 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc9faa640 1 -- 172.21.15.62:0/2108355663 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc419d790 con 0x7f2bc4103660 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc0ff9640 1 -- 172.21.15.62:0/2108355663 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80047a0 con 0x7f2bc4103660 2024-09-15T23:16:11.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.725+0000 7f2bc9faa640 1 -- 172.21.15.62:0/2108355663 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc41a5070 con 0x7f2bc4103660 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.727+0000 7f2bc0ff9640 1 -- 172.21.15.62:0/2108355663 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2bb80041d0 con 0x7f2bc4103660 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.727+0000 7f2bcc383740 1 Processor -- start 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.727+0000 7f2bcc383740 1 -- start start 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e0f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f32f0 0x56314e0fd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e0fffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bcc383740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56314e059090 con 0x56314e0f32f0 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bcc383740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56314e000a70 con 0x56314e0fdbb0 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bcc383740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56314e055050 con 0x56314e0f29d0 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f32f0 0x56314e0fd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e0f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f32f0 0x56314e0fd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42270/0 (socket says 172.21.15.62:42270) 2024-09-15T23:16:11.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc37fe640 1 -- 172.21.15.62:0/715093719 learned_addr learned my addr 172.21.15.62:0/715093719 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e0f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36066/0 (socket says 172.21.15.62:36066) 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/715093719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e0fffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc0ff9640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2bac0a3240 0x7f2bac0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc37fe640 1 -- 172.21.15.62:0/715093719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 msgr2=0x56314e0f2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/715093719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e0f2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc37fe640 1 -- 172.21.15.62:0/715093719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 msgr2=0x56314e0fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/715093719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e0fffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc37fe640 1 -- 172.21.15.62:0/715093719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56314e00d130 con 0x56314e0f32f0 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/715093719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e0fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.728+0000 7f2bc0ff9640 1 -- 172.21.15.62:0/2108355663 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2bb80cf610 con 0x7f2bc4103660 2024-09-15T23:16:11.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.729+0000 7f2bc3fff640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2bac0a3240 0x7f2bac0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.729+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/715093719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f32f0 0x56314e0fd670 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009900 tx=0x7f2bb4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d081d3deb664fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:11.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.729+0000 7f2ba9ffb640 1 -- 172.21.15.62:0/715093719 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4040070 con 0x56314e0f32f0 2024-09-15T23:16:11.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.729+0000 7f2ba9ffb640 1 -- 172.21.15.62:0/715093719 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bb40319f0 con 0x56314e0f32f0 2024-09-15T23:16:11.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.729+0000 7f2ba9ffb640 1 -- 172.21.15.62:0/715093719 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4035040 con 0x56314e0f32f0 2024-09-15T23:16:11.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bc3fff640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2bac0a3240 0x7f2bac0a5700 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005fd0 tx=0x7f2bbc004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:11.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 -- 172.21.15.62:0/715093719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f32f0 msgr2=0x56314e0fd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:11.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 --2- 172.21.15.62:0/715093719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f32f0 0x56314e0fd670 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009900 tx=0x7f2bb4005e80 comp rx=0 tx=0).stop 2024-09-15T23:16:11.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 -- 172.21.15.62:0/715093719 shutdown_connections 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 --2- 172.21.15.62:0/715093719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e0fffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 --2- 172.21.15.62:0/715093719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f32f0 0x56314e0fd670 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 --2- 172.21.15.62:0/715093719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e0f2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 -- 172.21.15.62:0/715093719 >> 172.21.15.62:0/715093719 conn(0x56314e0eef60 msgr2=0x56314e0ef340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 -- 172.21.15.62:0/715093719 shutdown_connections 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 -- 172.21.15.62:0/715093719 wait complete. 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.730+0000 7f2bcc383740 1 -- start start 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bcc383740 1 Processor -- start 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e0fd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0fdbb0 0x56314e0fabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bcc383740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56314e100780 con 0x56314e0f29d0 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bcc383740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56314e013f20 con 0x56314e0f32f0 2024-09-15T23:16:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bcc383740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56314e100900 con 0x56314e0fdbb0 2024-09-15T23:16:11.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e0fd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e0fd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42272/0 (socket says 172.21.15.62:42272) 2024-09-15T23:16:11.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc37fe640 1 -- 172.21.15.62:0/3030115799 learned_addr learned my addr 172.21.15.62:0/3030115799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:11.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bcc383740 10 client.0 _ll_register_callbacks cb 0x56314dfdd030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:11.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc3fff640 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0fdbb0 0x56314e0fabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc37fe640 1 -- 172.21.15.62:0/3030115799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0fdbb0 msgr2=0x56314e0fabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0fdbb0 0x56314e0fabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc37fe640 1 -- 172.21.15.62:0/3030115799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 msgr2=0x56314e0fa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.731+0000 7f2bc37fe640 1 -- 172.21.15.62:0/3030115799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56314ddca2b0 con 0x56314e0f29d0 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.732+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e0fd360 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2bb402a4c0 tx=0x7f2bb4005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ec092972eb9f28 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.732+0000 7f2ba2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.732+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/3030115799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4040030 con 0x56314e0f29d0 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.732+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/3030115799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2bb4004260 con 0x56314e0f29d0 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.732+0000 7f2bcc383740 1 -- 172.21.15.62:0/3030115799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bb40319f0 con 0x56314e0f29d0 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.732+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/3030115799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2bb4056050 con 0x56314e0f29d0 2024-09-15T23:16:11.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.732+0000 7f2bcc383740 10 client.15447 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.732+0000 7f2bcc383740 1 -- 172.21.15.62:0/3030115799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56314e10bf90 con 0x56314e0f29d0 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.732+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/3030115799 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bb40319f0 con 0x56314e0f29d0 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/3030115799 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f2bb4003790 con 0x56314e0f29d0 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2bcc383740 10 client.15447 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2bcc383740 10 client.15447 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2bcc383740 20 client.15447 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2bcc383740 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56314e10e110 0x56314e10f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2bcc383740 4 client.15447 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2bcc383740 1 -- 172.21.15.62:0/3030115799 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56314e066600 con 0x56314e10e110 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56314e10e110 0x56314e10f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56314e10e110 0x56314e10f580 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77545d48aa2114c0 server_cookie=e37e112cc53e77d9 in_seq=0 out_seq=0 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.733+0000 7f2ba2ffd640 10 client.15447 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:11.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/3030115799 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56314e066600 con 0x56314e10e110 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2ba2ffd640 10 client.15447 handle_command_reply: tid=0 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 client.15447 shutdown 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 -- 172.21.15.62:0/3030115799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 msgr2=0x56314e0fd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e0fd360 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2bb402a4c0 tx=0x7f2bb4005250 comp rx=0 tx=0).stop 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 -- 172.21.15.62:0/3030115799 shutdown_connections 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56314e10e110 0x56314e10f580 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0fdbb0 0x56314e0fabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 --2- 172.21.15.62:0/3030115799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e0fd360 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2ba2ffd640 0 client.15447 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:11.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 -- 172.21.15.62:0/3030115799 >> 172.21.15.62:0/3030115799 conn(0x56314e0eef60 msgr2=0x56314e0f4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:11.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 -- 172.21.15.62:0/3030115799 shutdown_connections 2024-09-15T23:16:11.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 1 -- 172.21.15.62:0/3030115799 wait complete. 2024-09-15T23:16:11.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.734+0000 7f2bcc383740 20 client.15447 trim_cache size 0 max 16384 2024-09-15T23:16:11.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bcc383740 1 Processor -- start 2024-09-15T23:16:11.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bcc383740 1 -- start start 2024-09-15T23:16:11.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e10f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0fdbb0 0x56314e0fc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bcc383740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56314e10d8c0 con 0x56314e0fdbb0 2024-09-15T23:16:11.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bcc383740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56314e10df90 con 0x56314e0f32f0 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bcc383740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56314e10e660 con 0x56314e0f29d0 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0fdbb0 0x56314e0fc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e10f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:44686/0 (socket says 172.21.15.62:44686) 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0fdbb0 0x56314e0fc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42276/0 (socket says 172.21.15.62:42276) 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc2ffd640 1 -- 172.21.15.62:0/1174271906 learned_addr learned my addr 172.21.15.62:0/1174271906 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e10f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36072/0 (socket says 172.21.15.62:36072) 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc2ffd640 1 -- 172.21.15.62:0/1174271906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 msgr2=0x56314e10f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/1174271906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e10f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc2ffd640 1 -- 172.21.15.62:0/1174271906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0fdbb0 msgr2=0x56314e0fc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:11.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/1174271906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0fdbb0 0x56314e0fc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc2ffd640 1 -- 172.21.15.62:0/1174271906 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56314e0eed90 con 0x56314e0f32f0 2024-09-15T23:16:11.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.753+0000 7f2bc3fff640 1 --2- 172.21.15.62:0/1174271906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0fdbb0 0x56314e0fc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:11.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.754+0000 7f2bc2ffd640 1 --2- 172.21.15.62:0/1174271906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fac00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009610 tx=0x7f2bb80ccc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f87ed546bbea2fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:11.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.754+0000 7f2ba97fa640 1 -- 172.21.15.62:0/1174271906 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80860b0 con 0x56314e0f32f0 2024-09-15T23:16:11.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.754+0000 7f2ba97fa640 1 -- 172.21.15.62:0/1174271906 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bb809bfb0 con 0x56314e0f32f0 2024-09-15T23:16:11.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.754+0000 7f2ba97fa640 1 -- 172.21.15.62:0/1174271906 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb8097f20 con 0x56314e0f32f0 2024-09-15T23:16:11.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.754+0000 7f2bcc383740 1 -- 172.21.15.62:0/1174271906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 msgr2=0x56314e0fac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.754+0000 7f2bcc383740 1 --2- 172.21.15.62:0/1174271906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fac00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009610 tx=0x7f2bb80ccc60 comp rx=0 tx=0).stop 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 -- 172.21.15.62:0/1174271906 shutdown_connections 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 --2- 172.21.15.62:0/1174271906 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0fdbb0 0x56314e0fc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 --2- 172.21.15.62:0/1174271906 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0f32f0 0x56314e0fac00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 --2- 172.21.15.62:0/1174271906 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f29d0 0x56314e10f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 -- 172.21.15.62:0/1174271906 >> 172.21.15.62:0/1174271906 conn(0x56314e0eef60 msgr2=0x56314e0e9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 -- 172.21.15.62:0/1174271906 shutdown_connections 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 -- 172.21.15.62:0/1174271906 wait complete. 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 -- start start 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 Processor -- start 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e117d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f32f0 0x56314e1130a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e1135e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56314e10f400 con 0x56314e0f29d0 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56314e10ed30 con 0x56314e0fdbb0 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.755+0000 7f2bcc383740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56314e10fad0 con 0x56314e0f32f0 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bcc383740 10 client.0 _ll_register_callbacks cb 0x7f2bbc009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e1135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e117d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e1135e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:44702/0 (socket says 172.21.15.62:44702) 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f32f0 0x56314e1130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc3fff640 1 -- 172.21.15.62:0/2313393673 learned_addr learned my addr 172.21.15.62:0/2313393673 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e117d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42282/0 (socket says 172.21.15.62:42282) 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc3fff640 1 -- 172.21.15.62:0/2313393673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f32f0 msgr2=0x56314e1130a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc3fff640 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f32f0 0x56314e1130a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc3fff640 1 -- 172.21.15.62:0/2313393673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 msgr2=0x56314e117d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc3fff640 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e117d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc3fff640 1 -- 172.21.15.62:0/2313393673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56314e113bd0 con 0x56314e0fdbb0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e117d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bc3fff640 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e1135e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc010a80 tx=0x7f2bbc004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd83cd56ba30b654 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2ba2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/2313393673 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc02d060 con 0x56314e0fdbb0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2bcc383740 1 -- 172.21.15.62:0/2313393673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bb809bfb0 con 0x56314e0fdbb0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.756+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/2313393673 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2bbc01d8a0 con 0x56314e0fdbb0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/2313393673 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2bbc034050 con 0x56314e0fdbb0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2bcc383740 10 client.25121 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2bcc383740 1 -- 172.21.15.62:0/2313393673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56314e178660 con 0x56314e0fdbb0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/2313393673 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bb809bfb0 con 0x56314e0fdbb0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/2313393673 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f2bbc00f040 con 0x56314e0fdbb0 2024-09-15T23:16:11.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2bcc383740 10 client.25121 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2bcc383740 10 client.25121 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2bcc383740 20 client.25121 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2bcc383740 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56314e17d2c0 0x56314e17bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2bcc383740 4 client.25121 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2bcc383740 1 -- 172.21.15.62:0/2313393673 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "status"}) -- 0x56314e060b10 con 0x56314e17d2c0 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.757+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56314e17d2c0 0x56314e17bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bc37fe640 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56314e17d2c0 0x56314e17bfd0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b2fe7f7df8e59d2 server_cookie=400d29b6402778e8 in_seq=0 out_seq=0 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2ba2ffd640 10 client.25121 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2ba2ffd640 1 -- 172.21.15.62:0/2313393673 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56314e060b10 con 0x56314e17d2c0 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2ba2ffd640 10 client.25121 handle_command_reply: tid=0 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 client.25121 shutdown 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 -- 172.21.15.62:0/2313393673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 msgr2=0x56314e1135e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:11.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e1135e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc010a80 tx=0x7f2bbc004ab0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 -- 172.21.15.62:0/2313393673 shutdown_connections 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56314e17d2c0 0x56314e17bfd0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56314e0fdbb0 0x56314e1135e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56314e0f32f0 0x56314e1130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2ba2ffd640 0 client.25121 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 --2- 172.21.15.62:0/2313393673 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56314e0f29d0 0x56314e117d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 -- 172.21.15.62:0/2313393673 >> 172.21.15.62:0/2313393673 conn(0x56314e0eef60 msgr2=0x56314e0ee000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 -- 172.21.15.62:0/2313393673 shutdown_connections 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 1 -- 172.21.15.62:0/2313393673 wait complete. 2024-09-15T23:16:11.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.758+0000 7f2bcc383740 20 client.25121 trim_cache size 0 max 16384 2024-09-15T23:16:11.766 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:16:11.766 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "ee7f888e-73b6-11ef-bceb-c7b262605968", 2024-09-15T23:16:11.766 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-09-15T23:16:11.766 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 24497, 2024-09-15T23:16:11.766 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-09-15T23:16:11.766 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-09-15T23:16:11.767 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-09-15T23:16:11.767 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 158.25220320899999, 2024-09-15T23:16:11.767 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 25, 2024-09-15T23:16:11.767 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 92, 2024-09-15T23:16:11.767 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 92, 2024-09-15T23:16:11.767 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 169.400857777 2024-09-15T23:16:11.768 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:16:11.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 -- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2bac0a3240 msgr2=0x7f2bac0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:11.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2bac0a3240 0x7f2bac0a5700 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc005fd0 tx=0x7f2bbc004a40 comp rx=0 tx=0).stop 2024-09-15T23:16:11.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 -- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4103660 msgr2=0x7f2bc4198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:11.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4103660 0x7f2bc4198a30 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800d950 tx=0x7f2bb800de10 comp rx=0 tx=0).stop 2024-09-15T23:16:11.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 -- 172.21.15.62:0/2108355663 shutdown_connections 2024-09-15T23:16:11.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2bac0a3240 0x7f2bac0a5700 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2bc410c170 0x7f2bc419cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bc4103660 0x7f2bc4198a30 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 --2- 172.21.15.62:0/2108355663 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2bc4102d40 0x7f2bc41984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:11.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 -- 172.21.15.62:0/2108355663 >> 172.21.15.62:0/2108355663 conn(0x7f2bc40fc7a0 msgr2=0x7f2bc40fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:11.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 -- 172.21.15.62:0/2108355663 shutdown_connections 2024-09-15T23:16:11.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:11.759+0000 7f2bc9faa640 1 -- 172.21.15.62:0/2108355663 wait complete. 2024-09-15T23:16:12.448 INFO:tasks.quiescer:Ready to start quiesce thrashing; initial delay: 15 sec 2024-09-15T23:16:12.988 INFO:teuthology.orchestra.run.smithi062.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-09-15T23:16:12.991 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-09-15T23:16:13.033 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/: 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 .. 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 data 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data: 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:total 35 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 . 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 .. 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 base 2024-09-15T23:16:13.034 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Sep 15 23:16 global 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 log 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_commit_ts 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_dynshmem 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Sep 15 23:16 pg_hba.conf 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Sep 15 23:16 pg_ident.conf 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 pg_logical 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 pg_multixact 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_notify 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_replslot 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_serial 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_snapshots 2024-09-15T23:16:13.035 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_stat 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_stat_tmp 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 pg_subtrans 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_tblspc 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_twophase 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 PG_VERSION 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 pg_wal 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 pg_xact 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Sep 15 23:16 postgresql.auto.conf 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28074 Sep 15 23:16 postgresql.conf 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/base: 2024-09-15T23:16:13.036 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 . 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 1 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 13493 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 13494 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/base/1: 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:total 7777 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 . 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 .. 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 112 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 113 2024-09-15T23:16:13.037 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 1247 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1247_fsm 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1247_vm 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 15 23:16 1249 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1249_fsm 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1249_vm 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 15 23:16 1255 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1255_fsm 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1255_vm 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 15 23:16 1259 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1259_fsm 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1259_vm 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 15 23:16 13344 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13344_fsm 2024-09-15T23:16:13.038 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13344_vm 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13346 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13348 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13349_fsm 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349_vm 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13351 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13353 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13354_fsm 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354_vm 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13356 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13358 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13359_fsm 2024-09-15T23:16:13.039 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359_vm 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13361 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13363 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1417 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1418 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 174 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 175 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2187 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2224 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2328 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2336 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2337 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2579 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2600 2024-09-15T23:16:13.040 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2600_fsm 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2600_vm 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2601_fsm 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601_vm 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2602 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2602_fsm 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2602_vm 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2603 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2603_fsm 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2603_vm 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2604 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2605 2024-09-15T23:16:13.041 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2605_fsm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2605_vm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2606_fsm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606_vm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2607 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2607_fsm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2607_vm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 15 23:16 2608 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2608_fsm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2608_vm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 15 23:16 2609 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2609_fsm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2609_vm 2024-09-15T23:16:13.042 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2610 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2610_fsm 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2610_vm 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2611 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2612_fsm 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612_vm 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2613 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2615_fsm 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615_vm 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2616 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2616_fsm 2024-09-15T23:16:13.043 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2616_vm 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 15 23:16 2617 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2617_fsm 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2617_vm 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 15 23:16 2618 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2618_fsm 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2618_vm 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 15 23:16 2619 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2619_fsm 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2619_vm 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2620 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2650 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2651 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2652 2024-09-15T23:16:13.044 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2653 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2654 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2655 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2656 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2657 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 15 23:16 2658 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 2659 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2660 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2661 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2662 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2663 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2664 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2665 2024-09-15T23:16:13.045 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2666 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2667 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2668 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2669 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2670 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 15 23:16 2673 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 15 23:16 2674 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 15 23:16 2675 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2678 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2679 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2680 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2681 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2682 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2683 2024-09-15T23:16:13.046 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2684 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2685 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2686 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2687 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2688 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2689 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 15 23:16 2690 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 15 23:16 2691 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2692 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2693 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2696 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2699 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2701 2024-09-15T23:16:13.047 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2702 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2703 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2704 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2753 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2753_fsm 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2753_vm 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2754 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2755 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2756 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2757 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2830 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2831 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2832 2024-09-15T23:16:13.048 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2833 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2834 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2835 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2836 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2837 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 15 23:16 2838 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2838_fsm 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2838_vm 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2839 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840_fsm 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2840_vm 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2841 2024-09-15T23:16:13.049 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2995 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2996 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3079_fsm 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079_vm 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3080 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3081 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3085 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3118 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3119 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 15 23:16 3164 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3256 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3257 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3258 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3350 2024-09-15T23:16:13.050 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3351 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3379 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3380 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3381 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3394 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3394_fsm 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3394_vm 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3395 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3429 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3430 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3431 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3433 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3439 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3440 2024-09-15T23:16:13.051 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3455 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 15 23:16 3456 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3456_fsm 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3456_vm 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3466 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3467 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3468 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3501 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3502 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3503 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3534 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3541_fsm 2024-09-15T23:16:13.052 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541_vm 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3542 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3574 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3575 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3576 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3596 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3597 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3598 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3599 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3600_fsm 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600_vm 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3601_fsm 2024-09-15T23:16:13.053 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601_vm 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3602_fsm 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602_vm 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603_fsm 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3603_vm 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3604 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3605 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3606 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3607 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3608 2024-09-15T23:16:13.054 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 3609 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3712 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3764_fsm 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764_vm 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3766 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3767 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3997 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4143 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4144 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4145 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4146 2024-09-15T23:16:13.055 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4147 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4148 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4149 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4150 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4151 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4152 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4153 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4154 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4155 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4156 2024-09-15T23:16:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4157 2024-09-15T23:16:13.057 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4158 2024-09-15T23:16:13.057 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4159 2024-09-15T23:16:13.057 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4160 2024-09-15T23:16:13.057 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4161 2024-09-15T23:16:13.057 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4162 2024-09-15T23:16:13.057 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4163 2024-09-15T23:16:13.057 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4164 2024-09-15T23:16:13.057 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4165 2024-09-15T23:16:13.060 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4166 2024-09-15T23:16:13.060 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4167 2024-09-15T23:16:13.060 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4168 2024-09-15T23:16:13.060 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4169 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4170 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4171 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4172 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4173 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4174 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 5002 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 548 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 549 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6102 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6104 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6106 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6110 2024-09-15T23:16:13.061 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6111 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6112 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6113 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6117 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 826 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 827 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 828 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 15 23:16 pg_filenode.map 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 PG_VERSION 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:total 7777 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 . 2024-09-15T23:16:13.062 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 .. 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 112 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 113 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 1247 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1247_fsm 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1247_vm 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 15 23:16 1249 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1249_fsm 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1249_vm 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 15 23:16 1255 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1255_fsm 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1255_vm 2024-09-15T23:16:13.063 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 15 23:16 1259 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1259_fsm 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1259_vm 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 15 23:16 13344 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13344_fsm 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13344_vm 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13346 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13348 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13349_fsm 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349_vm 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13351 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13353 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354 2024-09-15T23:16:13.064 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13354_fsm 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354_vm 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13356 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13358 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13359_fsm 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359_vm 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13361 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13363 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1417 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1418 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 174 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 175 2024-09-15T23:16:13.065 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2187 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2224 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2328 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2336 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2337 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2579 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2600 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2600_fsm 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2600_vm 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2601_fsm 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601_vm 2024-09-15T23:16:13.066 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2602 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2602_fsm 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2602_vm 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2603 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2603_fsm 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2603_vm 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2604 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2605 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2605_fsm 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2605_vm 2024-09-15T23:16:13.067 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2606_fsm 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606_vm 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2607 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2607_fsm 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2607_vm 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 15 23:16 2608 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2608_fsm 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2608_vm 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 15 23:16 2609 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2609_fsm 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2609_vm 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2610 2024-09-15T23:16:13.068 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2610_fsm 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2610_vm 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2611 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2612_fsm 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612_vm 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2613 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2615_fsm 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615_vm 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2616 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2616_fsm 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2616_vm 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 15 23:16 2617 2024-09-15T23:16:13.069 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2617_fsm 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2617_vm 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 15 23:16 2618 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2618_fsm 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2618_vm 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 15 23:16 2619 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2619_fsm 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2619_vm 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2620 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2650 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2651 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2652 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2653 2024-09-15T23:16:13.070 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2654 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2655 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2656 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2657 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 15 23:16 2658 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 2659 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2660 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2661 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2662 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2663 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2664 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2665 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2666 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2667 2024-09-15T23:16:13.071 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2668 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2669 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2670 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 15 23:16 2673 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 15 23:16 2674 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 15 23:16 2675 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2678 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2679 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2680 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2681 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2682 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2683 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2684 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2685 2024-09-15T23:16:13.072 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2686 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2687 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2688 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2689 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 15 23:16 2690 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 15 23:16 2691 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2692 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2693 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2696 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2699 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2701 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2702 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2703 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2704 2024-09-15T23:16:13.073 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2753 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2753_fsm 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2753_vm 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2754 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2755 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2756 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2757 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2830 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2831 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2832 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2833 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2834 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2835 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2836 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2837 2024-09-15T23:16:13.074 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 15 23:16 2838 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2838_fsm 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2838_vm 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2839 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840_fsm 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2840_vm 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2841 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2995 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2996 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3079_fsm 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079_vm 2024-09-15T23:16:13.075 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3080 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3081 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3085 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3118 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3119 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 15 23:16 3164 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3256 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3257 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3258 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3350 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3351 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3379 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3380 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3381 2024-09-15T23:16:13.076 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3394 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3394_fsm 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3394_vm 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3395 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3429 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3430 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3431 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3433 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3439 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3440 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3455 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 15 23:16 3456 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3456_fsm 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3456_vm 2024-09-15T23:16:13.077 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3466 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3467 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3468 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3501 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3502 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3503 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3534 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3541_fsm 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541_vm 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3542 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3574 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3575 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3576 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3596 2024-09-15T23:16:13.078 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3597 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3598 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3599 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3600_fsm 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600_vm 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3601_fsm 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601_vm 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3602_fsm 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602_vm 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603_fsm 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3603_vm 2024-09-15T23:16:13.079 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3604 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3605 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3606 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3607 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3608 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 3609 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3712 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3764_fsm 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764_vm 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3766 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3767 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3997 2024-09-15T23:16:13.080 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4143 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4144 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4145 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4146 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4147 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4148 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4149 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4150 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4151 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4152 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4153 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4154 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4155 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4156 2024-09-15T23:16:13.081 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4157 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4158 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4159 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4160 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4161 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4162 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4163 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4164 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4165 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4166 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4167 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4168 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4169 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4170 2024-09-15T23:16:13.082 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4171 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4172 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4173 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4174 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 5002 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 548 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 549 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6102 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6104 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6106 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6110 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6111 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6112 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6113 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6117 2024-09-15T23:16:13.083 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 826 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 827 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 828 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 15 23:16 pg_filenode.map 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 PG_VERSION 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:total 7777 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 . 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 .. 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 112 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 113 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 1247 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1247_fsm 2024-09-15T23:16:13.084 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1247_vm 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 15 23:16 1249 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1249_fsm 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1249_vm 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 15 23:16 1255 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1255_fsm 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1255_vm 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 15 23:16 1259 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1259_fsm 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1259_vm 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 15 23:16 13344 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13344_fsm 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13344_vm 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13346 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13348 2024-09-15T23:16:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13349_fsm 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349_vm 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13351 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13353 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13354_fsm 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354_vm 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13356 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13358 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13359_fsm 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359_vm 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13361 2024-09-15T23:16:13.086 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13363 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1417 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1418 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 174 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 175 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2187 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2224 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2328 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2336 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2337 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2579 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2600 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2600_fsm 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2600_vm 2024-09-15T23:16:13.087 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2601_fsm 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601_vm 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2602 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2602_fsm 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2602_vm 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2603 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2603_fsm 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2603_vm 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2604 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2605 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2605_fsm 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2605_vm 2024-09-15T23:16:13.088 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2606_fsm 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606_vm 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2607 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2607_fsm 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2607_vm 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 15 23:16 2608 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2608_fsm 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2608_vm 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 15 23:16 2609 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2609_fsm 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2609_vm 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2610 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2610_fsm 2024-09-15T23:16:13.089 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2610_vm 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2611 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2612_fsm 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612_vm 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2613 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2615_fsm 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615_vm 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2616 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2616_fsm 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2616_vm 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 15 23:16 2617 2024-09-15T23:16:13.090 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2617_fsm 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2617_vm 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 15 23:16 2618 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2618_fsm 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2618_vm 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 15 23:16 2619 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2619_fsm 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2619_vm 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2620 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2650 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2651 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2652 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2653 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2654 2024-09-15T23:16:13.091 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2655 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2656 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2657 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 15 23:16 2658 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 2659 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2660 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2661 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2662 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2663 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2664 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2665 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2666 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2667 2024-09-15T23:16:13.092 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2668 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2669 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2670 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 15 23:16 2673 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 15 23:16 2674 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 15 23:16 2675 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2678 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2679 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2680 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2681 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2682 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2683 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2684 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2685 2024-09-15T23:16:13.093 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2686 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2687 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2688 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2689 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 15 23:16 2690 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 15 23:16 2691 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2692 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2693 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2696 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2699 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2701 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2702 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2703 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2704 2024-09-15T23:16:13.094 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2753 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2753_fsm 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2753_vm 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2754 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2755 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2756 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2757 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2830 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2831 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2832 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2833 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2834 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2835 2024-09-15T23:16:13.095 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2836 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2837 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 15 23:16 2838 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2838_fsm 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2838_vm 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2839 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840_fsm 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2840_vm 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2841 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2995 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2996 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079 2024-09-15T23:16:13.096 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3079_fsm 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079_vm 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3080 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3081 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3085 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3118 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3119 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 15 23:16 3164 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3256 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3257 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3258 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3350 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3351 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3379 2024-09-15T23:16:13.097 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3380 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3381 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3394 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3394_fsm 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3394_vm 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3395 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3429 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3430 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3431 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3433 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3439 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3440 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3455 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 15 23:16 3456 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3456_fsm 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3456_vm 2024-09-15T23:16:13.098 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3466 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3467 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3468 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3501 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3502 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3503 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3534 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3541_fsm 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541_vm 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3542 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3574 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3575 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3576 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3596 2024-09-15T23:16:13.099 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3597 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3598 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3599 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3600_fsm 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600_vm 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3601_fsm 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601_vm 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3602_fsm 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602_vm 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603_fsm 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3603_vm 2024-09-15T23:16:13.100 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3604 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3605 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3606 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3607 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3608 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 3609 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3712 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3764_fsm 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764_vm 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3766 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3767 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3997 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4143 2024-09-15T23:16:13.101 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4144 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4145 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4146 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4147 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4148 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4149 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4150 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4151 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4152 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4153 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4154 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4155 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4156 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4157 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4158 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4159 2024-09-15T23:16:13.102 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4160 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4161 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4162 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4163 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4164 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4165 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4166 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4167 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4168 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4169 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4170 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4171 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4172 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4173 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4174 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 5002 2024-09-15T23:16:13.103 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 548 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 549 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6102 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6104 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6106 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6110 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6111 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6112 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6113 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6117 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 826 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 827 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 828 2024-09-15T23:16:13.104 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 15 23:16 pg_filenode.map 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 PG_VERSION 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/global: 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:total 537 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Sep 15 23:16 . 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1213 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1213_fsm 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1213_vm 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1214 2024-09-15T23:16:13.105 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1214_fsm 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1214_vm 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 1232 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 1233 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1260 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1260_fsm 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1260_vm 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1261 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1261_fsm 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1261_vm 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1262 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1262_fsm 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1262_vm 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2396 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2396_fsm 2024-09-15T23:16:13.106 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2396_vm 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2397 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2671 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2672 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2676 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2677 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2694 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2695 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2697 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2698 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2846 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2847 2024-09-15T23:16:13.107 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2964 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2965 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2966 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2967 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3592 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3593 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4060 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4061 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4175 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4176 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4177 2024-09-15T23:16:13.108 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4178 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4181 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4182 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4183 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4184 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4185 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4186 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6000 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6001 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6002 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6100 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6114 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6115 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 pg_control 2024-09-15T23:16:13.109 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 15 23:16 pg_filenode.map 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/log: 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.110 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:total 1 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 . 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 mappings 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Sep 15 23:16 replorigin_checkpoint 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 snapshots 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 .. 2024-09-15T23:16:13.111 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 .. 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 . 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 members 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 offsets 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-09-15T23:16:13.112 INFO:teuthology.orchestra.run.smithi062.stdout:total 8 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 .. 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 0000 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:total 8 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 .. 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 0000 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.113 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.114 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:total 8 2024-09-15T23:16:13.115 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 0000 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.116 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:total 16384 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 . 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:16 000000010000000000000001 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 archive_status 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 .. 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:total 8 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:16:13.117 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Sep 15 23:16 .. 2024-09-15T23:16:13.118 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 0000 2024-09-15T23:16:13.118 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-09-15T23:16:13.141 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:12 smithi062 ceph-mon[30949]: pgmap v368: 129 pgs: 129 active+clean; 4.6 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 342 KiB/s wr, 20 op/s 2024-09-15T23:16:13.243 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-09-15T23:16:13.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:12 smithi067 ceph-mon[35665]: pgmap v368: 129 pgs: 129 active+clean; 4.6 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 342 KiB/s wr, 20 op/s 2024-09-15T23:16:13.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:12 smithi148 ceph-mon[35610]: pgmap v368: 129 pgs: 129 active+clean; 4.6 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 342 KiB/s wr, 20 op/s 2024-09-15T23:16:13.330 INFO:teuthology.orchestra.run.smithi062.stderr:dropping old tables... 2024-09-15T23:16:13.330 INFO:teuthology.orchestra.run.smithi062.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-09-15T23:16:13.330 INFO:teuthology.orchestra.run.smithi062.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-09-15T23:16:13.330 INFO:teuthology.orchestra.run.smithi062.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-09-15T23:16:13.330 INFO:teuthology.orchestra.run.smithi062.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-09-15T23:16:13.330 INFO:teuthology.orchestra.run.smithi062.stderr:creating tables... 2024-09-15T23:16:13.345 INFO:teuthology.orchestra.run.smithi062.stderr:generating data (client-side)... 2024-09-15T23:16:13.422 INFO:teuthology.orchestra.run.smithi062.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.06 s, remaining 1.87 s) 2024-09-15T23:16:13.578 INFO:teuthology.orchestra.run.smithi062.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.22 s, remaining 3.24 s) 2024-09-15T23:16:13.691 INFO:teuthology.orchestra.run.smithi062.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.33 s, remaining 3.18 s) 2024-09-15T23:16:13.840 INFO:teuthology.orchestra.run.smithi062.stderr:400000 of 3200000 tuples (12%) done (elapsed 0.48 s, remaining 3.35 s) 2024-09-15T23:16:13.989 INFO:teuthology.orchestra.run.smithi062.stderr:500000 of 3200000 tuples (15%) done (elapsed 0.63 s, remaining 3.39 s) 2024-09-15T23:16:14.151 INFO:teuthology.orchestra.run.smithi062.stderr:600000 of 3200000 tuples (18%) done (elapsed 0.79 s, remaining 3.42 s) 2024-09-15T23:16:14.239 INFO:teuthology.orchestra.run.smithi062.stderr:700000 of 3200000 tuples (21%) done (elapsed 0.88 s, remaining 3.13 s) 2024-09-15T23:16:14.402 INFO:teuthology.orchestra.run.smithi062.stderr:800000 of 3200000 tuples (25%) done (elapsed 1.04 s, remaining 3.12 s) 2024-09-15T23:16:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:900000 of 3200000 tuples (28%) done (elapsed 1.20 s, remaining 3.06 s) 2024-09-15T23:16:14.631 INFO:teuthology.orchestra.run.smithi062.stderr:1000000 of 3200000 tuples (31%) done (elapsed 1.27 s, remaining 2.79 s) 2024-09-15T23:16:14.791 INFO:teuthology.orchestra.run.smithi062.stderr:1100000 of 3200000 tuples (34%) done (elapsed 1.43 s, remaining 2.73 s) 2024-09-15T23:16:14.879 INFO:teuthology.orchestra.run.smithi062.stderr:1200000 of 3200000 tuples (37%) done (elapsed 1.52 s, remaining 2.53 s) 2024-09-15T23:16:15.012 INFO:teuthology.orchestra.run.smithi062.stderr:1300000 of 3200000 tuples (40%) done (elapsed 1.65 s, remaining 2.41 s) 2024-09-15T23:16:15.169 INFO:teuthology.orchestra.run.smithi062.stderr:1400000 of 3200000 tuples (43%) done (elapsed 1.81 s, remaining 2.32 s) 2024-09-15T23:16:15.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:14 smithi062 ceph-mon[30949]: pgmap v369: 129 pgs: 129 active+clean; 18 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 1.7 MiB/s wr, 47 op/s 2024-09-15T23:16:15.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:14 smithi067 ceph-mon[35665]: pgmap v369: 129 pgs: 129 active+clean; 18 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 1.7 MiB/s wr, 47 op/s 2024-09-15T23:16:15.284 INFO:teuthology.orchestra.run.smithi062.stderr:1500000 of 3200000 tuples (46%) done (elapsed 1.92 s, remaining 2.18 s) 2024-09-15T23:16:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:14 smithi148 ceph-mon[35610]: pgmap v369: 129 pgs: 129 active+clean; 18 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 1.7 MiB/s wr, 47 op/s 2024-09-15T23:16:15.428 INFO:teuthology.orchestra.run.smithi062.stderr:1600000 of 3200000 tuples (50%) done (elapsed 2.07 s, remaining 2.07 s) 2024-09-15T23:16:15.514 INFO:teuthology.orchestra.run.smithi062.stderr:1700000 of 3200000 tuples (53%) done (elapsed 2.15 s, remaining 1.90 s) 2024-09-15T23:16:15.672 INFO:teuthology.orchestra.run.smithi062.stderr:1800000 of 3200000 tuples (56%) done (elapsed 2.31 s, remaining 1.80 s) 2024-09-15T23:16:15.788 INFO:teuthology.orchestra.run.smithi062.stderr:1900000 of 3200000 tuples (59%) done (elapsed 2.43 s, remaining 1.66 s) 2024-09-15T23:16:15.884 INFO:teuthology.orchestra.run.smithi062.stderr:2000000 of 3200000 tuples (62%) done (elapsed 2.52 s, remaining 1.51 s) 2024-09-15T23:16:16.028 INFO:teuthology.orchestra.run.smithi062.stderr:2100000 of 3200000 tuples (65%) done (elapsed 2.67 s, remaining 1.40 s) 2024-09-15T23:16:16.191 INFO:teuthology.orchestra.run.smithi062.stderr:2200000 of 3200000 tuples (68%) done (elapsed 2.83 s, remaining 1.29 s) 2024-09-15T23:16:16.310 INFO:teuthology.orchestra.run.smithi062.stderr:2300000 of 3200000 tuples (71%) done (elapsed 2.95 s, remaining 1.15 s) 2024-09-15T23:16:16.434 INFO:teuthology.orchestra.run.smithi062.stderr:2400000 of 3200000 tuples (75%) done (elapsed 3.07 s, remaining 1.02 s) 2024-09-15T23:16:16.614 INFO:teuthology.orchestra.run.smithi062.stderr:2500000 of 3200000 tuples (78%) done (elapsed 3.25 s, remaining 0.91 s) 2024-09-15T23:16:16.677 INFO:teuthology.orchestra.run.smithi062.stderr:2600000 of 3200000 tuples (81%) done (elapsed 3.31 s, remaining 0.76 s) 2024-09-15T23:16:16.837 INFO:teuthology.orchestra.run.smithi062.stderr:2700000 of 3200000 tuples (84%) done (elapsed 3.48 s, remaining 0.64 s) 2024-09-15T23:16:16.980 INFO:teuthology.orchestra.run.smithi062.stderr:2800000 of 3200000 tuples (87%) done (elapsed 3.62 s, remaining 0.52 s) 2024-09-15T23:16:17.065 INFO:teuthology.orchestra.run.smithi062.stderr:2900000 of 3200000 tuples (90%) done (elapsed 3.70 s, remaining 0.38 s) 2024-09-15T23:16:17.207 INFO:teuthology.orchestra.run.smithi062.stderr:3000000 of 3200000 tuples (93%) done (elapsed 3.85 s, remaining 0.26 s) 2024-09-15T23:16:17.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:16 smithi062 ceph-mon[30949]: pgmap v370: 129 pgs: 129 active+clean; 87 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 76 op/s 2024-09-15T23:16:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:16 smithi067 ceph-mon[35665]: pgmap v370: 129 pgs: 129 active+clean; 87 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 76 op/s 2024-09-15T23:16:17.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:16 smithi148 ceph-mon[35610]: pgmap v370: 129 pgs: 129 active+clean; 87 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 12 MiB/s wr, 76 op/s 2024-09-15T23:16:17.295 INFO:teuthology.orchestra.run.smithi062.stderr:3100000 of 3200000 tuples (96%) done (elapsed 3.93 s, remaining 0.13 s) 2024-09-15T23:16:17.462 INFO:teuthology.orchestra.run.smithi062.stderr:3200000 of 3200000 tuples (100%) done (elapsed 4.10 s, remaining 0.00 s) 2024-09-15T23:16:17.474 INFO:teuthology.orchestra.run.smithi062.stderr:vacuuming... 2024-09-15T23:16:18.077 INFO:teuthology.orchestra.run.smithi062.stderr:creating primary keys... 2024-09-15T23:16:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 -- 172.21.15.62:0/846641190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5958104f90 msgr2=0x7f5958107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 --2- 172.21.15.62:0/846641190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5958104f90 0x7f5958107380 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f594c009900 tx=0x7f594c02a410 comp rx=0 tx=0).stop 2024-09-15T23:16:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 -- 172.21.15.62:0/846641190 shutdown_connections 2024-09-15T23:16:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 --2- 172.21.15.62:0/846641190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f59581078c0 0x7f5958109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 --2- 172.21.15.62:0/846641190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5958104f90 0x7f5958107380 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 --2- 172.21.15.62:0/846641190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5958102660 0x7f5958104a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 -- 172.21.15.62:0/846641190 >> 172.21.15.62:0/846641190 conn(0x7f59580fc1e0 msgr2=0x7f59580fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 -- 172.21.15.62:0/846641190 shutdown_connections 2024-09-15T23:16:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 -- 172.21.15.62:0/846641190 wait complete. 2024-09-15T23:16:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 Processor -- start 2024-09-15T23:16:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.482+0000 7f595e754640 1 -- start start 2024-09-15T23:16:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595e754640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5958102660 0x7f5958101fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595e754640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5958104f90 0x7f5958100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595e754640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59581078c0 0x7f5958100b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595e754640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f595810d820 con 0x7f59581078c0 2024-09-15T23:16:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595e754640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f595810d6a0 con 0x7f5958104f90 2024-09-15T23:16:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595e754640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f595810d9a0 con 0x7f5958102660 2024-09-15T23:16:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595ccf1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59581078c0 0x7f5958100b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595ccf1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59581078c0 0x7f5958100b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41064/0 (socket says 172.21.15.62:41064) 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595ccf1640 1 -- 172.21.15.62:0/3526867592 learned_addr learned my addr 172.21.15.62:0/3526867592 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f59577fe640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5958104f90 0x7f5958100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f5957fff640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5958102660 0x7f5958101fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595ccf1640 1 -- 172.21.15.62:0/3526867592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5958102660 msgr2=0x7f5958101fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5958102660 0x7f5958101fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595ccf1640 1 -- 172.21.15.62:0/3526867592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5958104f90 msgr2=0x7f5958100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5958104f90 0x7f5958100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595ccf1640 1 -- 172.21.15.62:0/3526867592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59581013e0 con 0x7f59581078c0 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.483+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59581078c0 0x7f5958100b50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f595000eb40 tx=0x7f595000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.484+0000 7f59557fa640 1 -- 172.21.15.62:0/3526867592 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950004b20 con 0x7f59581078c0 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.484+0000 7f59557fa640 1 -- 172.21.15.62:0/3526867592 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5950004cc0 con 0x7f59581078c0 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.484+0000 7f595e754640 1 -- 172.21.15.62:0/3526867592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59581a1f40 con 0x7f59581078c0 2024-09-15T23:16:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.484+0000 7f59557fa640 1 -- 172.21.15.62:0/3526867592 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5950010850 con 0x7f59581078c0 2024-09-15T23:16:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.484+0000 7f595e754640 1 -- 172.21.15.62:0/3526867592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59581a23a0 con 0x7f59581078c0 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59615c8740 1 Processor -- start 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59557fa640 1 -- 172.21.15.62:0/3526867592 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f5950005020 con 0x7f59581078c0 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59615c8740 1 -- start start 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59615c8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x555741598da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59615c8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 0x55574159dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59615c8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 0x5557415a0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59615c8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5557414fc2f0 con 0x5557415989a0 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59615c8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5557414a4270 con 0x5557415992e0 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59615c8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5557414f82b0 con 0x55574159e370 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59577fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x555741598da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.487+0000 7f59577fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x555741598da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41066/0 (socket says 172.21.15.62:41066) 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59577fe640 1 -- 172.21.15.62:0/338852077 learned_addr learned my addr 172.21.15.62:0/338852077 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f5957fff640 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 0x55574159dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59577fe640 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 0x5557415a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59577fe640 1 -- 172.21.15.62:0/338852077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 msgr2=0x5557415a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59577fe640 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 0x5557415a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59577fe640 1 -- 172.21.15.62:0/338852077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 msgr2=0x55574159dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59577fe640 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 0x55574159dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59577fe640 1 -- 172.21.15.62:0/338852077 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557414a3340 con 0x5557415989a0 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f5957fff640 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 0x55574159dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59577fe640 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x555741598da0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f594c006010 tx=0x7f594c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2211e7ea82daaf88 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59557fa640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59400a32f0 0x7f59400a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f593e7fc640 1 -- 172.21.15.62:0/338852077 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c044020 con 0x5557415989a0 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f593e7fc640 1 -- 172.21.15.62:0/338852077 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f594c004770 con 0x5557415989a0 2024-09-15T23:16:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f593e7fc640 1 -- 172.21.15.62:0/338852077 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c035040 con 0x5557415989a0 2024-09-15T23:16:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f5957fff640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59400a32f0 0x7f59400a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.488+0000 7f59557fa640 1 -- 172.21.15.62:0/3526867592 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f59500d04c0 con 0x7f59581078c0 2024-09-15T23:16:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f59615c8740 1 -- 172.21.15.62:0/338852077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 msgr2=0x555741598da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f5957fff640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59400a32f0 0x7f59400a57b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x555741599d10 tx=0x7f5948005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f59615c8740 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x555741598da0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f594c006010 tx=0x7f594c005d50 comp rx=0 tx=0).stop 2024-09-15T23:16:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f59615c8740 1 -- 172.21.15.62:0/338852077 shutdown_connections 2024-09-15T23:16:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f59615c8740 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 0x5557415a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f59615c8740 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 0x55574159dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f59615c8740 1 --2- 172.21.15.62:0/338852077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x555741598da0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f59615c8740 1 -- 172.21.15.62:0/338852077 >> 172.21.15.62:0/338852077 conn(0x555741594010 msgr2=0x555741594410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f59615c8740 1 -- 172.21.15.62:0/338852077 shutdown_connections 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.489+0000 7f59615c8740 1 -- 172.21.15.62:0/338852077 wait complete. 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.490+0000 7f59615c8740 1 -- start start 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.490+0000 7f59615c8740 1 Processor -- start 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.490+0000 7f59615c8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x5557415a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.490+0000 7f59615c8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 0x5557415a30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f59615c8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 0x5557415aa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f59615c8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5557415a12e0 con 0x5557415989a0 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f5957fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x5557415a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f5957fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x5557415a2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41072/0 (socket says 172.21.15.62:41072) 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f5957fff640 1 -- 172.21.15.62:0/3234932140 learned_addr learned my addr 172.21.15.62:0/3234932140 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 0x5557415aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f59577fe640 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 0x5557415a30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555741485290 con 0x5557415992e0 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5557415a1460 con 0x55574159e370 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f59615c8740 10 client.0 _ll_register_callbacks cb 0x5557414b6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f595ccf1640 1 -- 172.21.15.62:0/3234932140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 msgr2=0x5557415a30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 0x5557415a30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f595ccf1640 1 -- 172.21.15.62:0/3234932140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 msgr2=0x5557415a2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x5557415a2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f595ccf1640 1 -- 172.21.15.62:0/3234932140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55574120f210 con 0x55574159e370 2024-09-15T23:16:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.491+0000 7f5957fff640 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x5557415a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 0x5557415aa110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f595000c7d0 tx=0x7f595009c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=962558b8833681a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59377fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59377fe640 1 -- 172.21.15.62:0/3234932140 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59500d0070 con 0x55574159e370 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59377fe640 1 -- 172.21.15.62:0/3234932140 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5950005310 con 0x55574159e370 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f594c004770 con 0x55574159e370 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59377fe640 1 -- 172.21.15.62:0/3234932140 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f595009c760 con 0x55574159e370 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59615c8740 10 client.25051 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5557412db620 con 0x55574159e370 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59377fe640 1 -- 172.21.15.62:0/3234932140 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f594c004770 con 0x55574159e370 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59377fe640 1 -- 172.21.15.62:0/3234932140 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f595009cac0 con 0x55574159e370 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59615c8740 10 client.25051 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59615c8740 10 client.25051 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59615c8740 20 client.25051 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59615c8740 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5557415ae870 0x5557415b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59615c8740 4 client.25051 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.492+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555741509a20 con 0x5557415ae870 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.493+0000 7f5957fff640 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5557415ae870 0x5557415b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.493+0000 7f5957fff640 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5557415ae870 0x5557415b0cc0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e16f777ab45ff249 server_cookie=e61dc58b40b4f95b in_seq=0 out_seq=0 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.493+0000 7f59377fe640 10 client.25051 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59377fe640 1 -- 172.21.15.62:0/3234932140 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555741509a20 con 0x5557415ae870 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59377fe640 10 client.25051 handle_command_reply: tid=0 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 client.25051 shutdown 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 msgr2=0x5557415aa110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 0x5557415aa110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f595000c7d0 tx=0x7f595009c360 comp rx=0 tx=0).stop 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 shutdown_connections 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5557415ae870 0x5557415b0cc0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55574159e370 0x5557415aa110 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415992e0 0x5557415a30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59377fe640 0 client.25051 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 --2- 172.21.15.62:0/3234932140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415989a0 0x5557415a2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 >> 172.21.15.62:0/3234932140 conn(0x555741594010 msgr2=0x555741595820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 shutdown_connections 2024-09-15T23:16:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 1 -- 172.21.15.62:0/3234932140 wait complete. 2024-09-15T23:16:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.494+0000 7f59615c8740 20 client.25051 trim_cache size 0 max 16384 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.522+0000 7f59615c8740 1 Processor -- start 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.522+0000 7f59615c8740 1 -- start start 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.522+0000 7f59615c8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555741594010 0x555741595400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f59615c8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 0x5557415aa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f59615c8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5557415992e0 0x555741595940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f59615c8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f59581aeff0 con 0x555741594010 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f59615c8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f59581ac8c0 con 0x5557415989a0 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f59615c8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f59581aa710 con 0x5557415992e0 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f5957fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555741594010 0x555741595400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f5957fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555741594010 0x555741595400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41086/0 (socket says 172.21.15.62:41086) 2024-09-15T23:16:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f5957fff640 1 -- 172.21.15.62:0/1410794478 learned_addr learned my addr 172.21.15.62:0/1410794478 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f59577fe640 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 0x5557415aa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f595ccf1640 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5557415992e0 0x555741595940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f5957fff640 1 -- 172.21.15.62:0/1410794478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5557415992e0 msgr2=0x555741595940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f5957fff640 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5557415992e0 0x555741595940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f5957fff640 1 -- 172.21.15.62:0/1410794478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 msgr2=0x5557415aa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f5957fff640 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 0x5557415aa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f5957fff640 1 -- 172.21.15.62:0/1410794478 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557415b1340 con 0x555741594010 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f595ccf1640 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5557415992e0 0x555741595940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f5957fff640 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555741594010 0x555741595400 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5948012cb0 tx=0x7f594801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6565849a4268e17 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.523+0000 7f593dffb640 1 -- 172.21.15.62:0/1410794478 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5948021da0 con 0x555741594010 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.524+0000 7f593dffb640 1 -- 172.21.15.62:0/1410794478 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5948004310 con 0x555741594010 2024-09-15T23:16:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.524+0000 7f593dffb640 1 -- 172.21.15.62:0/1410794478 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594802bc10 con 0x555741594010 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.524+0000 7f59615c8740 1 -- 172.21.15.62:0/1410794478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555741594010 msgr2=0x555741595400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.524+0000 7f59615c8740 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555741594010 0x555741595400 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5948012cb0 tx=0x7f594801a000 comp rx=0 tx=0).stop 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.525+0000 7f59615c8740 1 -- 172.21.15.62:0/1410794478 shutdown_connections 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.525+0000 7f59615c8740 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5557415992e0 0x555741595940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.525+0000 7f59615c8740 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 0x5557415aa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.525+0000 7f59615c8740 1 --2- 172.21.15.62:0/1410794478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555741594010 0x555741595400 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.525+0000 7f59615c8740 1 -- 172.21.15.62:0/1410794478 >> 172.21.15.62:0/1410794478 conn(0x55574159de90 msgr2=0x5557415931c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.525+0000 7f59615c8740 1 -- 172.21.15.62:0/1410794478 shutdown_connections 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.525+0000 7f59615c8740 1 -- 172.21.15.62:0/1410794478 wait complete. 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.525+0000 7f59615c8740 1 -- start start 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.525+0000 7f59615c8740 1 Processor -- start 2024-09-15T23:16:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59615c8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555741594010 0x5557415a77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59615c8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 0x5557415a7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59615c8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415992e0 0x5557415a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59615c8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f59581aad80 con 0x5557415992e0 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59615c8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f59581ac1f0 con 0x5557415989a0 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59615c8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f59581ab450 con 0x555741594010 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59615c8740 10 client.0 _ll_register_callbacks cb 0x7f5950004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f595ccf1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415992e0 0x5557415a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f595ccf1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415992e0 0x5557415a8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41088/0 (socket says 172.21.15.62:41088) 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f595ccf1640 1 -- 172.21.15.62:0/3315952050 learned_addr learned my addr 172.21.15.62:0/3315952050 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59577fe640 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 0x5557415a7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f5957fff640 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555741594010 0x5557415a77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f595ccf1640 1 -- 172.21.15.62:0/3315952050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555741594010 msgr2=0x5557415a77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555741594010 0x5557415a77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f595ccf1640 1 -- 172.21.15.62:0/3315952050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 msgr2=0x5557415a7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 0x5557415a7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f595ccf1640 1 -- 172.21.15.62:0/3315952050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557415a9290 con 0x5557415992e0 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59577fe640 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 0x5557415a7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f595ccf1640 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415992e0 0x5557415a8c80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f59500cdc90 tx=0x7f595009bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50210b7c12971fd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59377fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59377fe640 1 -- 172.21.15.62:0/3315952050 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59500d0040 con 0x5557415992e0 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59377fe640 1 -- 172.21.15.62:0/3315952050 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f595009c9c0 con 0x5557415992e0 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.526+0000 7f59615c8740 1 -- 172.21.15.62:0/3315952050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5948004310 con 0x5557415992e0 2024-09-15T23:16:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59377fe640 1 -- 172.21.15.62:0/3315952050 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5950083050 con 0x5557415992e0 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59615c8740 10 client.15471 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59615c8740 1 -- 172.21.15.62:0/3315952050 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5557416195a0 con 0x5557415992e0 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59377fe640 1 -- 172.21.15.62:0/3315952050 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5948004310 con 0x5557415992e0 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59377fe640 1 -- 172.21.15.62:0/3315952050 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f59500672a0 con 0x5557415992e0 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59615c8740 10 client.15471 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59615c8740 10 client.15471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59615c8740 20 client.15471 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59615c8740 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55574159e370 0x55574161bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59615c8740 4 client.15471 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f59615c8740 1 -- 172.21.15.62:0/3315952050 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555741503d70 con 0x55574159e370 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.527+0000 7f5957fff640 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55574159e370 0x55574161bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.528+0000 7f59377fe640 10 client.15471 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.528+0000 7f5957fff640 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55574159e370 0x55574161bd10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=116d340aeb32ffb5 server_cookie=e82596f1d71ef691 in_seq=0 out_seq=0 2024-09-15T23:16:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.832+0000 7f59377fe640 1 -- 172.21.15.62:0/3315952050 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555741503d70 con 0x55574159e370 2024-09-15T23:16:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.832+0000 7f59377fe640 10 client.15471 handle_command_reply: tid=0 2024-09-15T23:16:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.832+0000 7f59615c8740 1 client.15471 shutdown 2024-09-15T23:16:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.832+0000 7f59615c8740 1 -- 172.21.15.62:0/3315952050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415992e0 msgr2=0x5557415a8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.832+0000 7f59615c8740 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415992e0 0x5557415a8c80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f59500cdc90 tx=0x7f595009bbb0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59615c8740 1 -- 172.21.15.62:0/3315952050 shutdown_connections 2024-09-15T23:16:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59615c8740 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55574159e370 0x55574161bd10 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59615c8740 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557415992e0 0x5557415a8c80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59615c8740 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5557415989a0 0x5557415a7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59615c8740 1 --2- 172.21.15.62:0/3315952050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555741594010 0x5557415a77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59377fe640 0 client.15471 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59615c8740 1 -- 172.21.15.62:0/3315952050 >> 172.21.15.62:0/3315952050 conn(0x55574159de90 msgr2=0x5557415943f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59615c8740 1 -- 172.21.15.62:0/3315952050 shutdown_connections 2024-09-15T23:16:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59615c8740 1 -- 172.21.15.62:0/3315952050 wait complete. 2024-09-15T23:16:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f59615c8740 20 client.15471 trim_cache size 0 max 16384 2024-09-15T23:16:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:16:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:16:18.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:16:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:16:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 -- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59400a32f0 msgr2=0x7f59400a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59400a32f0 0x7f59400a57b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x555741599d10 tx=0x7f5948005f90 comp rx=0 tx=0).stop 2024-09-15T23:16:18.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 -- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59581078c0 msgr2=0x7f5958100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59581078c0 0x7f5958100b50 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f595000eb40 tx=0x7f595000c2d0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 -- 172.21.15.62:0/3526867592 shutdown_connections 2024-09-15T23:16:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59400a32f0 0x7f59400a57b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59581078c0 0x7f5958100b50 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5958104f90 0x7f5958100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 --2- 172.21.15.62:0/3526867592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5958102660 0x7f5958101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.833+0000 7f595e754640 1 -- 172.21.15.62:0/3526867592 >> 172.21.15.62:0/3526867592 conn(0x7f59580fc1e0 msgr2=0x7f5958103000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.834+0000 7f595e754640 1 -- 172.21.15.62:0/3526867592 shutdown_connections 2024-09-15T23:16:18.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:18.834+0000 7f595e754640 1 -- 172.21.15.62:0/3526867592 wait complete. 2024-09-15T23:16:19.091 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:18 smithi148 ceph-mon[35610]: pgmap v371: 129 pgs: 129 active+clean; 172 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 25 MiB/s wr, 133 op/s 2024-09-15T23:16:19.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:18 smithi062 ceph-mon[30949]: pgmap v371: 129 pgs: 129 active+clean; 172 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 25 MiB/s wr, 133 op/s 2024-09-15T23:16:19.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:18 smithi067 ceph-mon[35665]: pgmap v371: 129 pgs: 129 active+clean; 172 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 25 MiB/s wr, 133 op/s 2024-09-15T23:16:19.510 INFO:teuthology.orchestra.run.smithi062.stderr:done in 6.18 s (drop tables 0.00 s, create tables 0.02 s, client-side generate 4.13 s, vacuum 0.60 s, primary keys 1.43 s). 2024-09-15T23:16:19.513 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-09-15T23:16:19.569 INFO:teuthology.orchestra.run.smithi062.stderr:starting vacuum...end. 2024-09-15T23:16:21.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:20 smithi062 ceph-mon[30949]: pgmap v372: 129 pgs: 129 active+clean; 308 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 48 MiB/s wr, 148 op/s 2024-09-15T23:16:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:20 smithi067 ceph-mon[35665]: pgmap v372: 129 pgs: 129 active+clean; 308 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 48 MiB/s wr, 148 op/s 2024-09-15T23:16:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:20 smithi148 ceph-mon[35610]: pgmap v372: 129 pgs: 129 active+clean; 308 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 48 MiB/s wr, 148 op/s 2024-09-15T23:16:23.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:22 smithi062 ceph-mon[30949]: pgmap v373: 129 pgs: 129 active+clean; 457 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 70 MiB/s wr, 319 op/s 2024-09-15T23:16:23.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:22 smithi067 ceph-mon[35665]: pgmap v373: 129 pgs: 129 active+clean; 457 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 70 MiB/s wr, 319 op/s 2024-09-15T23:16:23.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:22 smithi148 ceph-mon[35610]: pgmap v373: 129 pgs: 129 active+clean; 457 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 70 MiB/s wr, 319 op/s 2024-09-15T23:16:24.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 5.0 s, 3286.8 tps, lat 29.823 ms stddev 34.952 2024-09-15T23:16:25.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:24 smithi062 ceph-mon[30949]: pgmap v374: 129 pgs: 129 active+clean; 617 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 76 MiB/s wr, 475 op/s 2024-09-15T23:16:25.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:24 smithi067 ceph-mon[35665]: pgmap v374: 129 pgs: 129 active+clean; 617 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 76 MiB/s wr, 475 op/s 2024-09-15T23:16:25.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:24 smithi148 ceph-mon[35610]: pgmap v374: 129 pgs: 129 active+clean; 617 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 76 MiB/s wr, 475 op/s 2024-09-15T23:16:26.002 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:16:26.404 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:26.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.955+0000 7f7989450640 1 -- 172.21.15.62:0/3659213592 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79800319b0 con 0x7f7984100e20 2024-09-15T23:16:26.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.956+0000 7f798c6b6640 1 -- 172.21.15.62:0/3659213592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7984100e20 msgr2=0x7f7984101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:26.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.956+0000 7f798c6b6640 1 --2- 172.21.15.62:0/3659213592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7984100e20 0x7f7984101200 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7980009930 tx=0x7f798002a4d0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.956+0000 7f798c6b6640 1 -- 172.21.15.62:0/3659213592 shutdown_connections 2024-09-15T23:16:26.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.956+0000 7f798c6b6640 1 --2- 172.21.15.62:0/3659213592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7984102ef0 0x7f798410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.956+0000 7f798c6b6640 1 --2- 172.21.15.62:0/3659213592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7984102570 0x7f79841029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.956+0000 7f798c6b6640 1 --2- 172.21.15.62:0/3659213592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7984100e20 0x7f7984101200 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.956+0000 7f798c6b6640 1 -- 172.21.15.62:0/3659213592 >> 172.21.15.62:0/3659213592 conn(0x7f79840fc7d0 msgr2=0x7f79840febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:26.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.957+0000 7f798c6b6640 1 -- 172.21.15.62:0/3659213592 shutdown_connections 2024-09-15T23:16:26.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.957+0000 7f798c6b6640 1 -- 172.21.15.62:0/3659213592 wait complete. 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.958+0000 7f798c6b6640 1 Processor -- start 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.958+0000 7f798c6b6640 1 -- start start 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.958+0000 7f798c6b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7984100e20 0x7f79841a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.958+0000 7f798c6b6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7984102570 0x7f79841a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.958+0000 7f798c6b6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7984102ef0 0x7f798419a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.958+0000 7f798c6b6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f798406ac00 con 0x7f7984102ef0 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.958+0000 7f798c6b6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f798406aa80 con 0x7f7984100e20 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.958+0000 7f798c6b6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f798406ad80 con 0x7f7984102570 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.959+0000 7f798a452640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7984100e20 0x7f79841a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.959+0000 7f798a452640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7984100e20 0x7f79841a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:33470/0 (socket says 172.21.15.62:33470) 2024-09-15T23:16:26.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.959+0000 7f798a452640 1 -- 172.21.15.62:0/2450673060 learned_addr learned my addr 172.21.15.62:0/2450673060 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:26.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.959+0000 7f7989c51640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7984102570 0x7f79841a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.960+0000 7f7989c51640 1 -- 172.21.15.62:0/2450673060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7984100e20 msgr2=0x7f79841a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.960+0000 7f7989c51640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7984100e20 0x7f79841a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.960+0000 7f7989c51640 1 -- 172.21.15.62:0/2450673060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7984102ef0 msgr2=0x7f798419a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.960+0000 7f7989c51640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7984102ef0 0x7f798419a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.960+0000 7f7989c51640 1 -- 172.21.15.62:0/2450673060 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798419acf0 con 0x7f7984102570 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.960+0000 7f7989c51640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7984102570 0x7f79841a0940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f797400e9f0 tx=0x7f797400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.960+0000 7f797b7fe640 1 -- 172.21.15.62:0/2450673060 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797400ccc0 con 0x7f7984102570 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.960+0000 7f797b7fe640 1 -- 172.21.15.62:0/2450673060 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7974005020 con 0x7f7984102570 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.960+0000 7f797b7fe640 1 -- 172.21.15.62:0/2450673060 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974004b20 con 0x7f7984102570 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.961+0000 7f798c6b6640 1 -- 172.21.15.62:0/2450673060 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f798419afe0 con 0x7f7984102570 2024-09-15T23:16:26.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.961+0000 7f798c6b6640 1 -- 172.21.15.62:0/2450673060 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7984110380 con 0x7f7984102570 2024-09-15T23:16:26.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.964+0000 7f797b7fe640 1 -- 172.21.15.62:0/2450673060 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7974004cc0 con 0x7f7984102570 2024-09-15T23:16:26.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.965+0000 7f797b7fe640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f79700a3390 0x7f79700a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.965+0000 7f797b7fe640 1 -- 172.21.15.62:0/2450673060 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f79740cfc60 con 0x7f7984102570 2024-09-15T23:16:26.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.966+0000 7f798a452640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f79700a3390 0x7f79700a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.967+0000 7f798a452640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f79700a3390 0x7f79700a5850 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7980005ab0 tx=0x7f7980005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:26.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.968+0000 7f798ea8f740 1 Processor -- start 2024-09-15T23:16:26.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ea8f740 1 -- start start 2024-09-15T23:16:26.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf0734a0 0x55accf073880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 0x55accf07e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf07e680 0x55accf080a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ea8f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55accefd92e0 con 0x55accf073dc0 2024-09-15T23:16:26.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ea8f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55accef80ce0 con 0x55accf07e680 2024-09-15T23:16:26.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ea8f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55accefd52a0 con 0x55accf0734a0 2024-09-15T23:16:26.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ac53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf07e680 0x55accf080a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f7989c51640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 0x55accf07e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ac53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf07e680 0x55accf080a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:33486/0 (socket says 172.21.15.62:33486) 2024-09-15T23:16:26.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ac53640 1 -- 172.21.15.62:0/699826988 learned_addr learned my addr 172.21.15.62:0/699826988 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:26.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798a452640 1 --2- 172.21.15.62:0/699826988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf0734a0 0x55accf073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ac53640 1 -- 172.21.15.62:0/699826988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf0734a0 msgr2=0x55accf073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:26.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ac53640 1 --2- 172.21.15.62:0/699826988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf0734a0 0x55accf073880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ac53640 1 -- 172.21.15.62:0/699826988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 msgr2=0x55accf07e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:26.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ac53640 1 --2- 172.21.15.62:0/699826988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 0x55accf07e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.969+0000 7f798ac53640 1 -- 172.21.15.62:0/699826988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55accef65440 con 0x55accf07e680 2024-09-15T23:16:26.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.970+0000 7f798ac53640 1 --2- 172.21.15.62:0/699826988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf07e680 0x55accf080a70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f797c009d00 tx=0x7f797c00b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e07b9ac4a3f102b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:26.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.970+0000 7f7978ff9640 1 -- 172.21.15.62:0/699826988 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c00bd40 con 0x55accf07e680 2024-09-15T23:16:26.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.970+0000 7f7978ff9640 1 -- 172.21.15.62:0/699826988 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f797c017070 con 0x55accf07e680 2024-09-15T23:16:26.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.970+0000 7f7978ff9640 1 -- 172.21.15.62:0/699826988 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c011720 con 0x55accf07e680 2024-09-15T23:16:26.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.972+0000 7f798ea8f740 1 -- 172.21.15.62:0/699826988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf07e680 msgr2=0x55accf080a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:26.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.972+0000 7f798ea8f740 1 --2- 172.21.15.62:0/699826988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf07e680 0x55accf080a70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f797c009d00 tx=0x7f797c00b420 comp rx=0 tx=0).stop 2024-09-15T23:16:26.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.972+0000 7f798ea8f740 1 -- 172.21.15.62:0/699826988 shutdown_connections 2024-09-15T23:16:26.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.972+0000 7f798ea8f740 1 --2- 172.21.15.62:0/699826988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf07e680 0x55accf080a70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.972+0000 7f798ea8f740 1 --2- 172.21.15.62:0/699826988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 0x55accf07e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.972+0000 7f798ea8f740 1 --2- 172.21.15.62:0/699826988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf0734a0 0x55accf073880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.972+0000 7f798ea8f740 1 -- 172.21.15.62:0/699826988 >> 172.21.15.62:0/699826988 conn(0x55accf06fa30 msgr2=0x55accf06fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:26.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.972+0000 7f798ea8f740 1 -- 172.21.15.62:0/699826988 shutdown_connections 2024-09-15T23:16:26.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.973+0000 7f798ea8f740 1 -- 172.21.15.62:0/699826988 wait complete. 2024-09-15T23:16:26.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.973+0000 7f798ea8f740 1 -- start start 2024-09-15T23:16:26.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.973+0000 7f798ea8f740 1 Processor -- start 2024-09-15T23:16:26.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.973+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf0734a0 0x55accf07df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf073dc0 0x55accf07b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798a452640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf0734a0 0x55accf07df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798a452640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf0734a0 0x55accf07df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44838/0 (socket says 172.21.15.62:44838) 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf07b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798a452640 1 -- 172.21.15.62:0/4279750936 learned_addr learned my addr 172.21.15.62:0/4279750936 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55accf081250 con 0x55accf0734a0 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55accef94050 con 0x55accf073dc0 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55accf0813d0 con 0x55accf07e680 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f7989c51640 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf073dc0 0x55accf07b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798ac53640 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf07b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798ea8f740 10 client.0 _ll_register_callbacks cb 0x55accefd4160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:26.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f7989c51640 1 -- 172.21.15.62:0/4279750936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 msgr2=0x55accf07b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f7989c51640 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf07b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f7989c51640 1 -- 172.21.15.62:0/4279750936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf0734a0 msgr2=0x55accf07df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f7989c51640 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf0734a0 0x55accf07df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f7989c51640 1 -- 172.21.15.62:0/4279750936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55acced4a3c0 con 0x55accf073dc0 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.974+0000 7f798a452640 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf0734a0 0x55accf07df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.975+0000 7f798ac53640 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf07b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.975+0000 7f7989c51640 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf073dc0 0x55accf07b240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f797400e9c0 tx=0x7f797409cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=939f93ca54c3bb01 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.975+0000 7f796dffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.975+0000 7f796dffb640 1 -- 172.21.15.62:0/4279750936 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79740b4070 con 0x55accf073dc0 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.975+0000 7f796dffb640 1 -- 172.21.15.62:0/4279750936 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7974098a60 con 0x55accf073dc0 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.975+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f797c017070 con 0x55accf073dc0 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.975+0000 7f796dffb640 1 -- 172.21.15.62:0/4279750936 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f79740ac050 con 0x55accf073dc0 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.975+0000 7f798ea8f740 10 client.25151 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:26.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.975+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55accf08cc10 con 0x55accf073dc0 2024-09-15T23:16:26.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.976+0000 7f796dffb640 1 -- 172.21.15.62:0/4279750936 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79740a5060 con 0x55accf073dc0 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.976+0000 7f796dffb640 1 -- 172.21.15.62:0/4279750936 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f79740bab40 con 0x55accf073dc0 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.977+0000 7f798ea8f740 10 client.25151 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.977+0000 7f798ea8f740 10 client.25151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.977+0000 7f798ea8f740 20 client.25151 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.977+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55accf08e120 0x55accf090580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.977+0000 7f798ea8f740 4 client.25151 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.977+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f797c017070 con 0x55accf08e120 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.977+0000 7f798a452640 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55accf08e120 0x55accf090580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.978+0000 7f798a452640 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55accf08e120 0x55accf090580 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c422a0886a2b58fc server_cookie=43453bd15fa08903 in_seq=0 out_seq=0 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.978+0000 7f796dffb640 10 client.25151 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.978+0000 7f796dffb640 1 -- 172.21.15.62:0/4279750936 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f797c017070 con 0x55accf08e120 2024-09-15T23:16:26.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.978+0000 7f796dffb640 10 client.25151 handle_command_reply: tid=0 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.978+0000 7f798ea8f740 1 client.25151 shutdown 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf073dc0 msgr2=0x55accf07b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf073dc0 0x55accf07b240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f797400e9c0 tx=0x7f797409cbb0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 shutdown_connections 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55accf08e120 0x55accf090580 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf07b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf073dc0 0x55accf07b240 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4279750936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf0734a0 0x55accf07df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 >> 172.21.15.62:0/4279750936 conn(0x55accf06fa30 msgr2=0x55accf075760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f796dffb640 0 client.25151 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 shutdown_connections 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 1 -- 172.21.15.62:0/4279750936 wait complete. 2024-09-15T23:16:26.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:26.979+0000 7f798ea8f740 20 client.25151 trim_cache size 0 max 16384 2024-09-15T23:16:27.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.004+0000 7f798ea8f740 1 Processor -- start 2024-09-15T23:16:27.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.004+0000 7f798ea8f740 1 -- start start 2024-09-15T23:16:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf08a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf073dc0 0x55accf095e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf07e680 0x55accf096390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f798ea8f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f79841b7ce0 con 0x55accf07e680 2024-09-15T23:16:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f798ea8f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79841b55b0 con 0x55accf0734a0 2024-09-15T23:16:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f798ea8f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f79841b3420 con 0x55accf073dc0 2024-09-15T23:16:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f7989c51640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf073dc0 0x55accf095e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f798a452640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf08a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f798a452640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf08a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:33508/0 (socket says 172.21.15.62:33508) 2024-09-15T23:16:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.005+0000 7f798a452640 1 -- 172.21.15.62:0/1382891557 learned_addr learned my addr 172.21.15.62:0/1382891557 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.006+0000 7f7989c51640 1 -- 172.21.15.62:0/1382891557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 msgr2=0x55accf08a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.006+0000 7f7989c51640 1 --2- 172.21.15.62:0/1382891557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf08a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.006+0000 7f7989c51640 1 -- 172.21.15.62:0/1382891557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf07e680 msgr2=0x55accf096390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.006+0000 7f7989c51640 1 --2- 172.21.15.62:0/1382891557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf07e680 0x55accf096390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.006+0000 7f7989c51640 1 -- 172.21.15.62:0/1382891557 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55accf070bc0 con 0x55accf073dc0 2024-09-15T23:16:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.006+0000 7f7989c51640 1 --2- 172.21.15.62:0/1382891557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf073dc0 0x55accf095e50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55accf07c880 tx=0x7f797409c2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97afb50dc272e72f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.007+0000 7f796ffff640 1 -- 172.21.15.62:0/1382891557 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79740b4070 con 0x55accf073dc0 2024-09-15T23:16:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.007+0000 7f796ffff640 1 -- 172.21.15.62:0/1382891557 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79740a5060 con 0x55accf073dc0 2024-09-15T23:16:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.007+0000 7f796ffff640 1 -- 172.21.15.62:0/1382891557 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797401c040 con 0x55accf073dc0 2024-09-15T23:16:27.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.007+0000 7f798ea8f740 1 -- 172.21.15.62:0/1382891557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf073dc0 msgr2=0x55accf095e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:27.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.007+0000 7f798ea8f740 1 --2- 172.21.15.62:0/1382891557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf073dc0 0x55accf095e50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55accf07c880 tx=0x7f797409c2f0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.007+0000 7f798ea8f740 1 -- 172.21.15.62:0/1382891557 shutdown_connections 2024-09-15T23:16:27.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.007+0000 7f798ea8f740 1 --2- 172.21.15.62:0/1382891557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf07e680 0x55accf096390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.007+0000 7f798ea8f740 1 --2- 172.21.15.62:0/1382891557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf073dc0 0x55accf095e50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 --2- 172.21.15.62:0/1382891557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf08a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 -- 172.21.15.62:0/1382891557 >> 172.21.15.62:0/1382891557 conn(0x55accf06fa30 msgr2=0x55accf076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:27.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 -- 172.21.15.62:0/1382891557 shutdown_connections 2024-09-15T23:16:27.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 -- 172.21.15.62:0/1382891557 wait complete. 2024-09-15T23:16:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 -- start start 2024-09-15T23:16:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 Processor -- start 2024-09-15T23:16:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf086f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 0x55accf078180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf078910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f79841b3a70 con 0x55accf073dc0 2024-09-15T23:16:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79841b4ee0 con 0x55accf0734a0 2024-09-15T23:16:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.008+0000 7f798ea8f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f79841b4140 con 0x55accf07e680 2024-09-15T23:16:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798ea8f740 10 client.0 _ll_register_callbacks cb 0x7f79740bc050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798a452640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf086f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798a452640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf086f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:33520/0 (socket says 172.21.15.62:33520) 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798a452640 1 -- 172.21.15.62:0/4244642132 learned_addr learned my addr 172.21.15.62:0/4244642132 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f7989c51640 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 0x55accf078180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798ac53640 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf078910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798a452640 1 -- 172.21.15.62:0/4244642132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 msgr2=0x55accf078910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798a452640 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf078910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798a452640 1 -- 172.21.15.62:0/4244642132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 msgr2=0x55accf078180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798a452640 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 0x55accf078180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.009+0000 7f798a452640 1 -- 172.21.15.62:0/4244642132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55accf0f7e20 con 0x55accf0734a0 2024-09-15T23:16:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.010+0000 7f7989c51640 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 0x55accf078180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.010+0000 7f798ac53640 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf078910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.010+0000 7f798a452640 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf086f40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7980053fd0 tx=0x7f7980009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=799ccdeb64b4cd90 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.010+0000 7f796dffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.010+0000 7f796dffb640 1 -- 172.21.15.62:0/4244642132 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7980035050 con 0x55accf0734a0 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.010+0000 7f796dffb640 1 -- 172.21.15.62:0/4244642132 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f79800048c0 con 0x55accf0734a0 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.010+0000 7f798ea8f740 1 -- 172.21.15.62:0/4244642132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79740a5060 con 0x55accf0734a0 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f796dffb640 1 -- 172.21.15.62:0/4244642132 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7980064050 con 0x55accf0734a0 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f798ea8f740 10 client.25157 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f798ea8f740 1 -- 172.21.15.62:0/4244642132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55accf0f9070 con 0x55accf0734a0 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f796dffb640 1 -- 172.21.15.62:0/4244642132 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79740a5060 con 0x55accf0734a0 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f796dffb640 1 -- 172.21.15.62:0/4244642132 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f7980032c00 con 0x55accf0734a0 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f798ea8f740 10 client.25157 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f798ea8f740 10 client.25157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f798ea8f740 20 client.25157 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55accf0f9780 0x55accf0fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f798ea8f740 4 client.25157 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.011+0000 7f798ea8f740 1 -- 172.21.15.62:0/4244642132 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55accefe6850 con 0x55accf0f9780 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f7989c51640 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55accf0f9780 0x55accf0fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f796dffb640 10 client.25157 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f7989c51640 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55accf0f9780 0x55accf0fbbc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93b82fa8f3a3a4eb server_cookie=a261edff0d1d8d0 in_seq=0 out_seq=0 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f796dffb640 1 -- 172.21.15.62:0/4244642132 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55accefe6850 con 0x55accf0f9780 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f796dffb640 10 client.25157 handle_command_reply: tid=0 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f798ea8f740 1 client.25157 shutdown 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f798ea8f740 1 -- 172.21.15.62:0/4244642132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 msgr2=0x55accf086f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf086f40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7980053fd0 tx=0x7f7980009260 comp rx=0 tx=0).stop 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f798ea8f740 1 -- 172.21.15.62:0/4244642132 shutdown_connections 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55accf0f9780 0x55accf0fbbc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55accf07e680 0x55accf078910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55accf073dc0 0x55accf078180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f796dffb640 0 client.25157 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.012+0000 7f798ea8f740 1 --2- 172.21.15.62:0/4244642132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55accf0734a0 0x55accf086f40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.013+0000 7f798ea8f740 1 -- 172.21.15.62:0/4244642132 >> 172.21.15.62:0/4244642132 conn(0x55accf06fa30 msgr2=0x55accf074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.013+0000 7f798ea8f740 1 -- 172.21.15.62:0/4244642132 shutdown_connections 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.013+0000 7f798ea8f740 1 -- 172.21.15.62:0/4244642132 wait complete. 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.013+0000 7f798ea8f740 20 client.25157 trim_cache size 0 max 16384 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:16:27.018 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 -- 172.21.15.62:0/2450673060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f79700a3390 msgr2=0x7f79700a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f79700a3390 0x7f79700a5850 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7980005ab0 tx=0x7f7980005a20 comp rx=0 tx=0).stop 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 -- 172.21.15.62:0/2450673060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7984102570 msgr2=0x7f79841a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7984102570 0x7f79841a0940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f797400e9f0 tx=0x7f797400eeb0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 -- 172.21.15.62:0/2450673060 shutdown_connections 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f79700a3390 0x7f79700a5850 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7984102ef0 0x7f798419a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7984102570 0x7f79841a0940 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 --2- 172.21.15.62:0/2450673060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7984100e20 0x7f79841a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 -- 172.21.15.62:0/2450673060 >> 172.21.15.62:0/2450673060 conn(0x7f79840fc7d0 msgr2=0x7f79840fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:27.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 -- 172.21.15.62:0/2450673060 shutdown_connections 2024-09-15T23:16:27.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:27.014+0000 7f798c6b6640 1 -- 172.21.15.62:0/2450673060 wait complete. 2024-09-15T23:16:27.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:26 smithi062 ceph-mon[30949]: pgmap v375: 129 pgs: 129 active+clean; 747 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 76 MiB/s wr, 551 op/s 2024-09-15T23:16:27.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:26 smithi067 ceph-mon[35665]: pgmap v375: 129 pgs: 129 active+clean; 747 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 76 MiB/s wr, 551 op/s 2024-09-15T23:16:27.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:26 smithi148 ceph-mon[35610]: pgmap v375: 129 pgs: 129 active+clean; 747 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 76 MiB/s wr, 551 op/s 2024-09-15T23:16:27.450 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 30.2 2024-09-15T23:16:27.450 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:16:28.453 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:16:28.454 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:16:28.619 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:29.073 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:28 smithi062 ceph-mon[30949]: pgmap v376: 129 pgs: 129 active+clean; 860 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 67 MiB/s wr, 578 op/s 2024-09-15T23:16:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:28 smithi067 ceph-mon[35665]: pgmap v376: 129 pgs: 129 active+clean; 860 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 67 MiB/s wr, 578 op/s 2024-09-15T23:16:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:28 smithi148 ceph-mon[35610]: pgmap v376: 129 pgs: 129 active+clean; 860 MiB data, 764 MiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 67 MiB/s wr, 578 op/s 2024-09-15T23:16:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.338+0000 7fe7b3c60640 1 -- 172.21.15.62:0/2203627078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac102790 msgr2=0x7fe7ac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.338+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/2203627078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac102790 0x7fe7ac1067e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe79c009900 tx=0x7fe79c02a410 comp rx=0 tx=0).stop 2024-09-15T23:16:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.338+0000 7fe7b3c60640 1 -- 172.21.15.62:0/2203627078 shutdown_connections 2024-09-15T23:16:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.338+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/2203627078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7ac106f10 0x7fe7ac10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:29.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.338+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/2203627078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac102790 0x7fe7ac1067e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.338+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/2203627078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac101de0 0x7fe7ac1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.339+0000 7fe7b3c60640 1 -- 172.21.15.62:0/2203627078 >> 172.21.15.62:0/2203627078 conn(0x7fe7ac0fd880 msgr2=0x7fe7ac0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.339+0000 7fe7b3c60640 1 -- 172.21.15.62:0/2203627078 shutdown_connections 2024-09-15T23:16:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.340+0000 7fe7b3c60640 1 -- 172.21.15.62:0/2203627078 wait complete. 2024-09-15T23:16:29.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.341+0000 7fe7b3c60640 1 Processor -- start 2024-09-15T23:16:29.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.341+0000 7fe7b3c60640 1 -- start start 2024-09-15T23:16:29.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b3c60640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac101de0 0x7fe7ac1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:29.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b3c60640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7ac106f10 0x7fe7ac1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b3c60640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac19a7d0 0x7fe7ac19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b3c60640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac112060 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b3c60640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac111ee0 con 0x7fe7ac101de0 2024-09-15T23:16:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b3c60640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac1121e0 con 0x7fe7ac106f10 2024-09-15T23:16:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b21fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac19a7d0 0x7fe7ac19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b21fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac19a7d0 0x7fe7ac19ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44856/0 (socket says 172.21.15.62:44856) 2024-09-15T23:16:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b21fd640 1 -- 172.21.15.62:0/1939908059 learned_addr learned my addr 172.21.15.62:0/1939908059 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:29.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.342+0000 7fe7b11fb640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7ac106f10 0x7fe7ac1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:29.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.343+0000 7fe7b19fc640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac101de0 0x7fe7ac1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:29.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.343+0000 7fe7b21fd640 1 -- 172.21.15.62:0/1939908059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7ac106f10 msgr2=0x7fe7ac1a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:29.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.343+0000 7fe7b21fd640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7ac106f10 0x7fe7ac1a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:29.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.343+0000 7fe7b21fd640 1 -- 172.21.15.62:0/1939908059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac101de0 msgr2=0x7fe7ac1a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:29.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.343+0000 7fe7b21fd640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac101de0 0x7fe7ac1a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:29.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.343+0000 7fe7b21fd640 1 -- 172.21.15.62:0/1939908059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ac19b3b0 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.343+0000 7fe7b21fd640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac19a7d0 0x7fe7ac19ac80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400b970 tx=0x7fe7a400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:29.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.344+0000 7fe7a2ffd640 1 -- 172.21.15.62:0/1939908059 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4019070 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.344+0000 7fe7a2ffd640 1 -- 172.21.15.62:0/1939908059 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7a400cb40 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.344+0000 7fe7a2ffd640 1 -- 172.21.15.62:0/1939908059 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4007650 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.344+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7ac19b640 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.344+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7ac1a77a0 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.345+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7ac1032c0 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.347+0000 7fe7a2ffd640 1 -- 172.21.15.62:0/1939908059 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe7a401c070 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.348+0000 7fe7a2ffd640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe7980a3240 0x7fe7980a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:29.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.348+0000 7fe7a2ffd640 1 -- 172.21.15.62:0/1939908059 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe7a40cf6e0 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.348+0000 7fe7b19fc640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe7980a3240 0x7fe7980a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:29.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.348+0000 7fe7b19fc640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe7980a3240 0x7fe7980a5700 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8009890 tx=0x7fe7a8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:29.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.352+0000 7fe7a2ffd640 1 -- 172.21.15.62:0/1939908059 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe7a409ba30 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.479 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:29.571 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 10.0 s, 2834.8 tps, lat 35.079 ms stddev 40.393 2024-09-15T23:16:29.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.737+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7ac102240 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.739+0000 7fe7a2ffd640 1 -- 172.21.15.62:0/1939908059 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7fe7ac102240 con 0x7fe7ac19a7d0 2024-09-15T23:16:29.740 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:29.741 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"btime":"2024-09-15T23:15:44:620975+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:15:43.799068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:16:29.741 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-09-15T23:16:29.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.743+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe7980a3240 msgr2=0x7fe7980a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:29.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.743+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe7980a3240 0x7fe7980a5700 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8009890 tx=0x7fe7a8008040 comp rx=0 tx=0).stop 2024-09-15T23:16:29.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.743+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac19a7d0 msgr2=0x7fe7ac19ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:29.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.743+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac19a7d0 0x7fe7ac19ac80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400b970 tx=0x7fe7a400be30 comp rx=0 tx=0).stop 2024-09-15T23:16:29.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.744+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 shutdown_connections 2024-09-15T23:16:29.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.744+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe7980a3240 0x7fe7980a5700 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:29.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.744+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe7ac19a7d0 0x7fe7ac19ac80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:29.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.744+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe7ac106f10 0x7fe7ac1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:29.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.744+0000 7fe7b3c60640 1 --2- 172.21.15.62:0/1939908059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7ac101de0 0x7fe7ac1a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:29.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.744+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 >> 172.21.15.62:0/1939908059 conn(0x7fe7ac0fd880 msgr2=0x7fe7ac0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:29.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.744+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 shutdown_connections 2024-09-15T23:16:29.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:29.745+0000 7fe7b3c60640 1 -- 172.21.15.62:0/1939908059 wait complete. 2024-09-15T23:16:29.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:29 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1939908059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:30.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.249+0000 7f8dad070640 1 -- 172.21.15.62:0/561953995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 msgr2=0x7f8da81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:30.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.249+0000 7f8dad070640 1 --2- 172.21.15.62:0/561953995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 0x7f8da81021c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009900 tx=0x7f8d9802a410 comp rx=0 tx=0).stop 2024-09-15T23:16:30.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.249+0000 7f8dad070640 1 -- 172.21.15.62:0/561953995 shutdown_connections 2024-09-15T23:16:30.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.249+0000 7f8dad070640 1 --2- 172.21.15.62:0/561953995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da8106f10 0x7f8da810ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:30.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.249+0000 7f8dad070640 1 --2- 172.21.15.62:0/561953995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8da8102790 0x7f8da81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:30.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.249+0000 7f8dad070640 1 --2- 172.21.15.62:0/561953995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 0x7f8da81021c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:30.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.249+0000 7f8dad070640 1 -- 172.21.15.62:0/561953995 >> 172.21.15.62:0/561953995 conn(0x7f8da80fd8e0 msgr2=0x7f8da80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:30.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.250+0000 7f8dad070640 1 -- 172.21.15.62:0/561953995 shutdown_connections 2024-09-15T23:16:30.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.250+0000 7f8dad070640 1 -- 172.21.15.62:0/561953995 wait complete. 2024-09-15T23:16:30.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.250+0000 7f8dad070640 1 Processor -- start 2024-09-15T23:16:30.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.251+0000 7f8dad070640 1 -- start start 2024-09-15T23:16:30.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8dad070640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 0x7f8da80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:30.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8dad070640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da8102790 0x7f8da8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:30.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8dad070640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8da8106f10 0x7f8da8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:30.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8da6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 0x7f8da80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:30.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8da6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 0x7f8da80744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44874/0 (socket says 172.21.15.62:44874) 2024-09-15T23:16:30.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8da6d9d640 1 -- 172.21.15.62:0/4237515881 learned_addr learned my addr 172.21.15.62:0/4237515881 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:30.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8da8111fc0 con 0x7f8da8101de0 2024-09-15T23:16:30.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8da8111e40 con 0x7f8da8102790 2024-09-15T23:16:30.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8da8112140 con 0x7f8da8106f10 2024-09-15T23:16:30.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.252+0000 7f8da759e640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8da8106f10 0x7f8da8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:30.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.253+0000 7f8da6d9d640 1 -- 172.21.15.62:0/4237515881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8da8106f10 msgr2=0x7f8da8074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:30.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.253+0000 7f8da6d9d640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8da8106f10 0x7f8da8074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:30.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.253+0000 7f8da6d9d640 1 -- 172.21.15.62:0/4237515881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da8102790 msgr2=0x7f8da8074a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:16:30.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.253+0000 7f8da6d9d640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da8102790 0x7f8da8074a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:30.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.253+0000 7f8da6d9d640 1 -- 172.21.15.62:0/4237515881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da81aadd0 con 0x7f8da8101de0 2024-09-15T23:16:30.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.253+0000 7f8da6d9d640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 0x7f8da80744f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004770 tx=0x7f8d98003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:30.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.254+0000 7f8d8ffff640 1 -- 172.21.15.62:0/4237515881 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9803f070 con 0x7f8da8101de0 2024-09-15T23:16:30.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.254+0000 7f8d8ffff640 1 -- 172.21.15.62:0/4237515881 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d98031ca0 con 0x7f8da8101de0 2024-09-15T23:16:30.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.254+0000 7f8d8ffff640 1 -- 172.21.15.62:0/4237515881 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98035030 con 0x7f8da8101de0 2024-09-15T23:16:30.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.254+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da81aafa0 con 0x7f8da8101de0 2024-09-15T23:16:30.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.254+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da81ab400 con 0x7f8da8101de0 2024-09-15T23:16:30.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.256+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da81032c0 con 0x7f8da8101de0 2024-09-15T23:16:30.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.257+0000 7f8d8ffff640 1 -- 172.21.15.62:0/4237515881 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8d98042070 con 0x7f8da8101de0 2024-09-15T23:16:30.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.258+0000 7f8d8ffff640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8d900a3390 0x7f8d900a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:30.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.258+0000 7f8d8ffff640 1 -- 172.21.15.62:0/4237515881 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8d980f0140 con 0x7f8da8101de0 2024-09-15T23:16:30.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.258+0000 7f8da659c640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8d900a3390 0x7f8d900a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:30.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.259+0000 7f8da659c640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8d900a3390 0x7f8d900a5850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8da8075cb0 tx=0x7f8d9c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:30.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.266+0000 7f8d8ffff640 1 -- 172.21.15.62:0/4237515881 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8d98049030 con 0x7f8da8101de0 2024-09-15T23:16:30.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:29 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1939908059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:29 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1939908059' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:30.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.640+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8da8109b60 con 0x7f8da8101de0 2024-09-15T23:16:30.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.641+0000 7f8d8ffff640 1 -- 172.21.15.62:0/4237515881 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11903 (secure 0 0 0) 0x7f8d980bcae0 con 0x7f8da8101de0 2024-09-15T23:16:30.642 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:30.642 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":25,"btime":"2024-09-15T23:15:44:620975+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:15:43.799068+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:16:30.642 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 25 2024-09-15T23:16:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8d900a3390 msgr2=0x7f8d900a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8d900a3390 0x7f8d900a5850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8da8075cb0 tx=0x7f8d9c005f90 comp rx=0 tx=0).stop 2024-09-15T23:16:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 msgr2=0x7f8da80744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 0x7f8da80744f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004770 tx=0x7f8d98003c10 comp rx=0 tx=0).stop 2024-09-15T23:16:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 shutdown_connections 2024-09-15T23:16:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8d900a3390 0x7f8d900a5850 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8da8106f10 0x7f8da8074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8da8102790 0x7f8da8074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:30.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 --2- 172.21.15.62:0/4237515881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8da8101de0 0x7f8da80744f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:30.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.645+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 >> 172.21.15.62:0/4237515881 conn(0x7f8da80fd8e0 msgr2=0x7f8da80fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:30.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.646+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 shutdown_connections 2024-09-15T23:16:30.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:30.646+0000 7f8dad070640 1 -- 172.21.15.62:0/4237515881 wait complete. 2024-09-15T23:16:30.949 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:30 smithi062 ceph-mon[30949]: pgmap v377: 129 pgs: 129 active+clean; 895 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 54 MiB/s wr, 607 op/s 2024-09-15T23:16:30.949 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:30 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4237515881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:30.951 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0' 2024-09-15T23:16:30.951 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 150.2 --await-for 45.0 2024-09-15T23:16:31.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:30 smithi067 ceph-mon[35665]: pgmap v377: 129 pgs: 129 active+clean; 895 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 54 MiB/s wr, 607 op/s 2024-09-15T23:16:31.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:30 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4237515881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:31.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:30 smithi148 ceph-mon[35610]: pgmap v377: 129 pgs: 129 active+clean; 895 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 54 MiB/s wr, 607 op/s 2024-09-15T23:16:31.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:30 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4237515881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:31.445 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:16:31.511 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.968+0000 7f7909f23640 1 -- 172.21.15.62:0/3713115861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7904072d20 msgr2=0x7f7904073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:31.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.968+0000 7f7909f23640 1 --2- 172.21.15.62:0/3713115861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7904072d20 0x7f7904073180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009900 tx=0x7f78f802a420 comp rx=0 tx=0).stop 2024-09-15T23:16:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.971+0000 7f7909f23640 1 -- 172.21.15.62:0/3713115861 shutdown_connections 2024-09-15T23:16:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.971+0000 7f7909f23640 1 --2- 172.21.15.62:0/3713115861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f79040736c0 0x7f7904119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.971+0000 7f7909f23640 1 --2- 172.21.15.62:0/3713115861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7904072d20 0x7f7904073180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.971+0000 7f7909f23640 1 --2- 172.21.15.62:0/3713115861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7904109700 0x7f7904109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.971+0000 7f7909f23640 1 -- 172.21.15.62:0/3713115861 >> 172.21.15.62:0/3713115861 conn(0x7f790406ece0 msgr2=0x7f7904071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:31.975 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.974+0000 7f7909f23640 1 -- 172.21.15.62:0/3713115861 shutdown_connections 2024-09-15T23:16:31.975 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.974+0000 7f7909f23640 1 -- 172.21.15.62:0/3713115861 wait complete. 2024-09-15T23:16:31.976 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.975+0000 7f7909f23640 1 Processor -- start 2024-09-15T23:16:31.976 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.975+0000 7f7909f23640 1 -- start start 2024-09-15T23:16:31.976 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.975+0000 7f7909f23640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7904072d20 0x7f79041a6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.976+0000 7f7909f23640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79040736c0 0x7f79041a70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.976+0000 7f7909f23640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7904109700 0x7f79041ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.976+0000 7f7909f23640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f790411c5e0 con 0x7f79040736c0 2024-09-15T23:16:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.976+0000 7f7909f23640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f790411c460 con 0x7f7904072d20 2024-09-15T23:16:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.976+0000 7f7909f23640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f790411c760 con 0x7f7904109700 2024-09-15T23:16:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.976+0000 7f7903fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7904109700 0x7f79041ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:31.977 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.976+0000 7f7903fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7904109700 0x7f79041ae150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33996/0 (socket says 172.21.15.62:33996) 2024-09-15T23:16:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.976+0000 7f7903fff640 1 -- 172.21.15.62:0/449159010 learned_addr learned my addr 172.21.15.62:0/449159010 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:31.978 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.977+0000 7f7902ffd640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79040736c0 0x7f79041a70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.977+0000 7f79037fe640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7904072d20 0x7f79041a6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.977+0000 7f7903fff640 1 -- 172.21.15.62:0/449159010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7904072d20 msgr2=0x7f79041a6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.977+0000 7f7903fff640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7904072d20 0x7f79041a6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.977+0000 7f7903fff640 1 -- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79040736c0 msgr2=0x7f79041a70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.977+0000 7f7903fff640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79040736c0 0x7f79041a70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.977+0000 7f7903fff640 1 -- 172.21.15.62:0/449159010 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79041ae7c0 con 0x7f7904109700 2024-09-15T23:16:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.977+0000 7f7902ffd640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79040736c0 0x7f79041a70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:31.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.977+0000 7f7903fff640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7904109700 0x7f79041ae150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00eb10 tx=0x7f78fc00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.978+0000 7f7900ff9640 1 -- 172.21.15.62:0/449159010 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc004b20 con 0x7f7904109700 2024-09-15T23:16:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.978+0000 7f7909f23640 1 -- 172.21.15.62:0/449159010 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79041aeab0 con 0x7f7904109700 2024-09-15T23:16:31.981 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.978+0000 7f7909f23640 1 -- 172.21.15.62:0/449159010 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79041aefd0 con 0x7f7904109700 2024-09-15T23:16:31.981 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.978+0000 7f7900ff9640 1 -- 172.21.15.62:0/449159010 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78fc004cc0 con 0x7f7904109700 2024-09-15T23:16:31.981 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.978+0000 7f7900ff9640 1 -- 172.21.15.62:0/449159010 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc010760 con 0x7f7904109700 2024-09-15T23:16:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.982+0000 7f790ccbb740 1 Processor -- start 2024-09-15T23:16:31.985 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.983+0000 7f7900ff9640 1 -- 172.21.15.62:0/449159010 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f78fc005020 con 0x7f7904109700 2024-09-15T23:16:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.984+0000 7f7900ff9640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78ec0a33c0 0x7f78ec0a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.984+0000 7f790ccbb740 1 -- start start 2024-09-15T23:16:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.984+0000 7f7900ff9640 1 -- 172.21.15.62:0/449159010 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78fc0d0270 con 0x7f7904109700 2024-09-15T23:16:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.985+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02eedda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.985+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eee2e0 0x55eb02ef2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.985+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02ef3370 0x55eb02ef5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.985+0000 7f79037fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02eedda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.985+0000 7f79037fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02eedda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34004/0 (socket says 172.21.15.62:34004) 2024-09-15T23:16:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.985+0000 7f79037fe640 1 -- 172.21.15.62:0/3227870967 learned_addr learned my addr 172.21.15.62:0/3227870967 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:31.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.986+0000 7f790ccbb740 1 -- 172.21.15.62:0/3227870967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55eb02e512f0 con 0x55eb02eee2e0 2024-09-15T23:16:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.986+0000 7f790ccbb740 1 -- 172.21.15.62:0/3227870967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55eb02df9270 con 0x55eb02ef3370 2024-09-15T23:16:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.986+0000 7f7903fff640 1 --2- 172.21.15.62:0/3227870967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02ef3370 0x55eb02ef5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.986+0000 7f790ccbb740 1 -- 172.21.15.62:0/3227870967 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55eb02e4d2b0 con 0x55eb02eed9a0 2024-09-15T23:16:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.988+0000 7f79037fe640 1 -- 172.21.15.62:0/3227870967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02ef3370 msgr2=0x55eb02ef5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.988+0000 7f79037fe640 1 --2- 172.21.15.62:0/3227870967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02ef3370 0x55eb02ef5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.988+0000 7f79037fe640 1 -- 172.21.15.62:0/3227870967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eee2e0 msgr2=0x55eb02ef2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.988+0000 7f79037fe640 1 --2- 172.21.15.62:0/3227870967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eee2e0 0x55eb02ef2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.988+0000 7f79037fe640 1 -- 172.21.15.62:0/3227870967 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb02df8340 con 0x55eb02eed9a0 2024-09-15T23:16:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.988+0000 7f79037fe640 1 --2- 172.21.15.62:0/3227870967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02eedda0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009c80 tx=0x7f78f40117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49b1f0715b7ae780 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.989+0000 7f7902ffd640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78ec0a33c0 0x7f78ec0a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.992+0000 7f78e9ffb640 1 -- 172.21.15.62:0/3227870967 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4012450 con 0x55eb02eed9a0 2024-09-15T23:16:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.992+0000 7f78e9ffb640 1 -- 172.21.15.62:0/3227870967 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78f4012ad0 con 0x55eb02eed9a0 2024-09-15T23:16:31.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.992+0000 7f78e9ffb640 1 -- 172.21.15.62:0/3227870967 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f400fcd0 con 0x55eb02eed9a0 2024-09-15T23:16:31.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.993+0000 7f7902ffd640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78ec0a33c0 0x7f78ec0a5880 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009850 tx=0x7f78f8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:31.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.995+0000 7f790ccbb740 1 -- 172.21.15.62:0/3227870967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 msgr2=0x55eb02eedda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:31.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.995+0000 7f790ccbb740 1 --2- 172.21.15.62:0/3227870967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02eedda0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009c80 tx=0x7f78f40117b0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.995+0000 7f790ccbb740 1 -- 172.21.15.62:0/3227870967 shutdown_connections 2024-09-15T23:16:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.995+0000 7f790ccbb740 1 --2- 172.21.15.62:0/3227870967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02ef3370 0x55eb02ef5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.995+0000 7f790ccbb740 1 --2- 172.21.15.62:0/3227870967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eee2e0 0x55eb02ef2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.995+0000 7f790ccbb740 1 --2- 172.21.15.62:0/3227870967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02eedda0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:31.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.996+0000 7f790ccbb740 1 -- 172.21.15.62:0/3227870967 >> 172.21.15.62:0/3227870967 conn(0x55eb02ee9010 msgr2=0x55eb02ee9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:31.999 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:31.999+0000 7f790ccbb740 1 -- 172.21.15.62:0/3227870967 shutdown_connections 2024-09-15T23:16:32.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.000+0000 7f790ccbb740 1 -- 172.21.15.62:0/3227870967 wait complete. 2024-09-15T23:16:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.001+0000 7f790ccbb740 1 -- start start 2024-09-15T23:16:32.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.001+0000 7f790ccbb740 1 Processor -- start 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.002+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 0x55eb02ef7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.002+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eee2e0 0x55eb02ef8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.002+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02ef3370 0x55eb02eff170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.002+0000 7f790ccbb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55eb02ef6340 con 0x55eb02eed9a0 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.002+0000 7f790ccbb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55eb02dda290 con 0x55eb02ef3370 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.002+0000 7f790ccbb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55eb02ef64c0 con 0x55eb02eee2e0 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.002+0000 7f790ccbb740 10 client.0 _ll_register_callbacks cb 0x55eb02e0b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.003+0000 7f7902ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eee2e0 0x55eb02ef8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.003+0000 7f7902ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eee2e0 0x55eb02ef8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34014/0 (socket says 172.21.15.62:34014) 2024-09-15T23:16:32.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.003+0000 7f7902ffd640 1 -- 172.21.15.62:0/457919818 learned_addr learned my addr 172.21.15.62:0/457919818 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.004+0000 7f7902ffd640 1 -- 172.21.15.62:0/457919818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02ef3370 msgr2=0x55eb02eff170 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:16:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.004+0000 7f7902ffd640 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02ef3370 0x55eb02eff170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.004+0000 7f7902ffd640 1 -- 172.21.15.62:0/457919818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 msgr2=0x55eb02ef7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.004+0000 7f7902ffd640 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 0x55eb02ef7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.004+0000 7f7902ffd640 1 -- 172.21.15.62:0/457919818 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb02b64210 con 0x55eb02eee2e0 2024-09-15T23:16:32.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.004+0000 7f7902ffd640 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eee2e0 0x55eb02ef8130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78f802aa10 tx=0x7f78f8005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29ca51879c3495d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.004+0000 7f78e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:16:32.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.005+0000 7f78e2ffd640 1 -- 172.21.15.62:0/457919818 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f8051670 con 0x55eb02eee2e0 2024-09-15T23:16:32.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.006+0000 7f78e2ffd640 1 -- 172.21.15.62:0/457919818 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78f8051810 con 0x55eb02eee2e0 2024-09-15T23:16:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.014+0000 7f790ccbb740 1 -- 172.21.15.62:0/457919818 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78f4012ad0 con 0x55eb02eee2e0 2024-09-15T23:16:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.014+0000 7f78e2ffd640 1 -- 172.21.15.62:0/457919818 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f78f8051a10 con 0x55eb02eee2e0 2024-09-15T23:16:32.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.014+0000 7f790ccbb740 10 client.25087 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.014+0000 7f790ccbb740 1 -- 172.21.15.62:0/457919818 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eb02c30620 con 0x55eb02eee2e0 2024-09-15T23:16:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.015+0000 7f78e2ffd640 1 -- 172.21.15.62:0/457919818 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78f4012ad0 con 0x55eb02eee2e0 2024-09-15T23:16:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.015+0000 7f78e2ffd640 1 -- 172.21.15.62:0/457919818 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f78f8044030 con 0x55eb02eee2e0 2024-09-15T23:16:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.015+0000 7f790ccbb740 10 client.25087 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:32.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.015+0000 7f790ccbb740 10 client.25087 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.015+0000 7f790ccbb740 20 client.25087 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.015+0000 7f790ccbb740 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55eb02f03d70 0x55eb02f06230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.015+0000 7f790ccbb740 4 client.25087 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:32.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.015+0000 7f790ccbb740 1 -- 172.21.15.62:0/457919818 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb02e5ea20 con 0x55eb02f03d70 2024-09-15T23:16:32.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.017+0000 7f79037fe640 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55eb02f03d70 0x55eb02f06230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.017+0000 7f79037fe640 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55eb02f03d70 0x55eb02f06230 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc3bfce43bc327e3 server_cookie=135a9a94f54c8be in_seq=0 out_seq=0 2024-09-15T23:16:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.017+0000 7f78e2ffd640 10 client.25087 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.018+0000 7f78e2ffd640 1 -- 172.21.15.62:0/457919818 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55eb02e5ea20 con 0x55eb02f03d70 2024-09-15T23:16:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.018+0000 7f78e2ffd640 10 client.25087 handle_command_reply: tid=0 2024-09-15T23:16:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.018+0000 7f790ccbb740 1 client.25087 shutdown 2024-09-15T23:16:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 -- 172.21.15.62:0/457919818 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eee2e0 msgr2=0x55eb02ef8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eee2e0 0x55eb02ef8130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78f802aa10 tx=0x7f78f8005fd0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 -- 172.21.15.62:0/457919818 shutdown_connections 2024-09-15T23:16:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55eb02f03d70 0x55eb02f06230 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02ef3370 0x55eb02eff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eee2e0 0x55eb02ef8130 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f78e2ffd640 0 client.25087 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 --2- 172.21.15.62:0/457919818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 0x55eb02ef7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 -- 172.21.15.62:0/457919818 >> 172.21.15.62:0/457919818 conn(0x55eb02ee9010 msgr2=0x55eb02eea820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 -- 172.21.15.62:0/457919818 shutdown_connections 2024-09-15T23:16:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 1 -- 172.21.15.62:0/457919818 wait complete. 2024-09-15T23:16:32.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.019+0000 7f790ccbb740 20 client.25087 trim_cache size 0 max 16384 2024-09-15T23:16:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.046+0000 7f790ccbb740 1 Processor -- start 2024-09-15T23:16:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.046+0000 7f790ccbb740 1 -- start start 2024-09-15T23:16:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.046+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 0x55eb02ee9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.046+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 0x55eb02eff130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.046+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02ef3370 0x55eb02eea400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.046+0000 7f790ccbb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78f0012450 con 0x55eb02eed9a0 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.046+0000 7f790ccbb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78f000fd20 con 0x55eb02eee2e0 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.046+0000 7f790ccbb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78f000db70 con 0x55eb02ef3370 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f79037fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 0x55eb02ee9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f79037fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 0x55eb02ee9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44912/0 (socket says 172.21.15.62:44912) 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f79037fe640 1 -- 172.21.15.62:0/2283478977 learned_addr learned my addr 172.21.15.62:0/2283478977 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f7902ffd640 1 --2- 172.21.15.62:0/2283478977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 0x55eb02eff130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f7903fff640 1 --2- 172.21.15.62:0/2283478977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02ef3370 0x55eb02eea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f79037fe640 1 -- 172.21.15.62:0/2283478977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02ef3370 msgr2=0x55eb02eea400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f79037fe640 1 --2- 172.21.15.62:0/2283478977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02ef3370 0x55eb02eea400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f79037fe640 1 -- 172.21.15.62:0/2283478977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 msgr2=0x55eb02eff130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f79037fe640 1 --2- 172.21.15.62:0/2283478977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 0x55eb02eff130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f79037fe640 1 -- 172.21.15.62:0/2283478977 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb02eee7c0 con 0x55eb02eed9a0 2024-09-15T23:16:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.047+0000 7f79037fe640 1 --2- 172.21.15.62:0/2283478977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 0x55eb02ee9ec0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x55eb02ee9e90 tx=0x7f78f400f630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6a0f3ca8766472 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.048+0000 7f78e97fa640 1 -- 172.21.15.62:0/2283478977 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f40050b0 con 0x55eb02eed9a0 2024-09-15T23:16:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.048+0000 7f78e97fa640 1 -- 172.21.15.62:0/2283478977 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78f400f820 con 0x55eb02eed9a0 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.048+0000 7f78e97fa640 1 -- 172.21.15.62:0/2283478977 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f4003850 con 0x55eb02eed9a0 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.049+0000 7f790ccbb740 1 -- 172.21.15.62:0/2283478977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 msgr2=0x55eb02ee9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.049+0000 7f790ccbb740 1 --2- 172.21.15.62:0/2283478977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 0x55eb02ee9ec0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x55eb02ee9e90 tx=0x7f78f400f630 comp rx=0 tx=0).stop 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.049+0000 7f790ccbb740 1 -- 172.21.15.62:0/2283478977 shutdown_connections 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.049+0000 7f790ccbb740 1 --2- 172.21.15.62:0/2283478977 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02ef3370 0x55eb02eea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.049+0000 7f790ccbb740 1 --2- 172.21.15.62:0/2283478977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 0x55eb02eff130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.050+0000 7f790ccbb740 1 --2- 172.21.15.62:0/2283478977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02eed9a0 0x55eb02ee9ec0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.050+0000 7f790ccbb740 1 -- 172.21.15.62:0/2283478977 >> 172.21.15.62:0/2283478977 conn(0x55eb02ee9010 msgr2=0x55eb02ef7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.050+0000 7f790ccbb740 1 -- 172.21.15.62:0/2283478977 shutdown_connections 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.050+0000 7f790ccbb740 1 -- 172.21.15.62:0/2283478977 wait complete. 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.050+0000 7f790ccbb740 1 -- start start 2024-09-15T23:16:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.051+0000 7f790ccbb740 1 Processor -- start 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02efc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f790ccbb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 0x55eb02efcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f79037fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02efc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f79037fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02efc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34032/0 (socket says 172.21.15.62:34032) 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f79037fe640 1 -- 172.21.15.62:0/1209452834 learned_addr learned my addr 172.21.15.62:0/1209452834 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f790ccbb740 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02ef3370 0x55eb02efddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f7902ffd640 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 0x55eb02efcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78f000e1e0 con 0x55eb02ef3370 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78f000f650 con 0x55eb02eee2e0 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f7903fff640 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02ef3370 0x55eb02efddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78f000e8b0 con 0x55eb02eed9a0 2024-09-15T23:16:32.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.052+0000 7f790ccbb740 10 client.0 _ll_register_callbacks cb 0x7f78f4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.053+0000 7f7903fff640 1 -- 172.21.15.62:0/1209452834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 msgr2=0x55eb02efc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.053+0000 7f7903fff640 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02efc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.053+0000 7f7903fff640 1 -- 172.21.15.62:0/1209452834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 msgr2=0x55eb02efcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.053+0000 7f7903fff640 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 0x55eb02efcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.053+0000 7f7903fff640 1 -- 172.21.15.62:0/1209452834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb02efe3c0 con 0x55eb02ef3370 2024-09-15T23:16:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.054+0000 7f7903fff640 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02ef3370 0x55eb02efddb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0cdd60 tx=0x7f78fc09c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7dc7a2ef85d49f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.054+0000 7f78e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:32.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.056+0000 7f78e2ffd640 1 -- 172.21.15.62:0/1209452834 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc0b4070 con 0x55eb02ef3370 2024-09-15T23:16:32.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.056+0000 7f78e2ffd640 1 -- 172.21.15.62:0/1209452834 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78fc0a4ef0 con 0x55eb02ef3370 2024-09-15T23:16:32.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.056+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78f400f820 con 0x55eb02ef3370 2024-09-15T23:16:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.057+0000 7f78e2ffd640 1 -- 172.21.15.62:0/1209452834 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f78fc09cc10 con 0x55eb02ef3370 2024-09-15T23:16:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.057+0000 7f790ccbb740 10 client.15501 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.057+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55eb02f6ebe0 con 0x55eb02ef3370 2024-09-15T23:16:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.058+0000 7f78e2ffd640 1 -- 172.21.15.62:0/1209452834 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78fc09c510 con 0x55eb02ef3370 2024-09-15T23:16:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.058+0000 7f78e2ffd640 1 -- 172.21.15.62:0/1209452834 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f78fc09b960 con 0x55eb02ef3370 2024-09-15T23:16:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.058+0000 7f790ccbb740 10 client.15501 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.058+0000 7f790ccbb740 10 client.15501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.058+0000 7f790ccbb740 20 client.15501 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.058+0000 7f790ccbb740 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55eb02f6f530 0x55eb02f71950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.058+0000 7f790ccbb740 4 client.15501 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:16:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.058+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f78f400f820 con 0x55eb02f6f530 2024-09-15T23:16:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.058+0000 7f79037fe640 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55eb02f6f530 0x55eb02f71950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.059+0000 7f78e2ffd640 10 client.15501 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:32.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.059+0000 7f79037fe640 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55eb02f6f530 0x55eb02f71950 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c22be25e958c77f server_cookie=799eccf86acc2e54 in_seq=0 out_seq=0 2024-09-15T23:16:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f78e2ffd640 1 -- 172.21.15.62:0/1209452834 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f78f400f820 con 0x55eb02f6f530 2024-09-15T23:16:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f78e2ffd640 10 client.15501 handle_command_reply: tid=0 2024-09-15T23:16:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f790ccbb740 1 client.15501 shutdown 2024-09-15T23:16:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02ef3370 msgr2=0x55eb02efddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f790ccbb740 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02ef3370 0x55eb02efddb0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0cdd60 tx=0x7f78fc09c3a0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 shutdown_connections 2024-09-15T23:16:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f790ccbb740 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55eb02f6f530 0x55eb02f71950 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f790ccbb740 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb02ef3370 0x55eb02efddb0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f790ccbb740 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eb02eee2e0 0x55eb02efcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f790ccbb740 1 --2- 172.21.15.62:0/1209452834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55eb02eed9a0 0x55eb02efc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 >> 172.21.15.62:0/1209452834 conn(0x55eb02ee9010 msgr2=0x55eb02ee79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.086+0000 7f78e2ffd640 0 client.15501 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.087+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 shutdown_connections 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.087+0000 7f790ccbb740 1 -- 172.21.15.62:0/1209452834 wait complete. 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.087+0000 7f790ccbb740 20 client.15501 trim_cache size 0 max 16384 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:16:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:16:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:16:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 -- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78ec0a33c0 msgr2=0x7f78ec0a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78ec0a33c0 0x7f78ec0a5880 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f78f8009850 tx=0x7f78f8000bc0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 -- 172.21.15.62:0/449159010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7904109700 msgr2=0x7f79041ae150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7904109700 0x7f79041ae150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00eb10 tx=0x7f78fc00c2d0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 -- 172.21.15.62:0/449159010 shutdown_connections 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78ec0a33c0 0x7f78ec0a5880 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7904109700 0x7f79041ae150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79040736c0 0x7f79041a70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 --2- 172.21.15.62:0/449159010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7904072d20 0x7f79041a6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.088+0000 7f7909f23640 1 -- 172.21.15.62:0/449159010 >> 172.21.15.62:0/449159010 conn(0x7f790406ece0 msgr2=0x7f79040716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.089+0000 7f7909f23640 1 -- 172.21.15.62:0/449159010 shutdown_connections 2024-09-15T23:16:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:32.090+0000 7f7909f23640 1 -- 172.21.15.62:0/449159010 wait complete. 2024-09-15T23:16:32.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2deb7fe640 1 -- 172.21.15.62:0/1586485153 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de4036070 con 0x7f2dec106f10 2024-09-15T23:16:32.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2df3247640 1 -- 172.21.15.62:0/1586485153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dec106f10 msgr2=0x7f2dec10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2df3247640 1 --2- 172.21.15.62:0/1586485153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dec106f10 0x7f2dec10ea90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009be0 tx=0x7f2de402a6f0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2df3247640 1 -- 172.21.15.62:0/1586485153 shutdown_connections 2024-09-15T23:16:32.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2df3247640 1 --2- 172.21.15.62:0/1586485153 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dec106f10 0x7f2dec10ea90 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2df3247640 1 --2- 172.21.15.62:0/1586485153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dec102790 0x7f2dec1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2df3247640 1 --2- 172.21.15.62:0/1586485153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec101de0 0x7f2dec1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2df3247640 1 -- 172.21.15.62:0/1586485153 >> 172.21.15.62:0/1586485153 conn(0x7f2dec0fd880 msgr2=0x7f2dec0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2df3247640 1 -- 172.21.15.62:0/1586485153 shutdown_connections 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.100+0000 7f2df3247640 1 -- 172.21.15.62:0/1586485153 wait complete. 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.101+0000 7f2df3247640 1 Processor -- start 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.102+0000 7f2df3247640 1 -- start start 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.102+0000 7f2df3247640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dec101de0 0x7f2dec073920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.102+0000 7f2df3247640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dec102790 0x7f2dec073e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.102+0000 7f2df3247640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec106f10 0x7f2dec06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.102+0000 7f2df3247640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2dec10fd50 con 0x7f2dec106f10 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.102+0000 7f2df3247640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2dec10fbd0 con 0x7f2dec101de0 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.102+0000 7f2df3247640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2dec10fed0 con 0x7f2dec102790 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.102+0000 7f2df17e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec106f10 0x7f2dec06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.103+0000 7f2df17e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec106f10 0x7f2dec06cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44944/0 (socket says 172.21.15.62:44944) 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.103+0000 7f2df17e4640 1 -- 172.21.15.62:0/3605605596 learned_addr learned my addr 172.21.15.62:0/3605605596 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.103+0000 7f2debfff640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dec102790 0x7f2dec073e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.103+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dec101de0 0x7f2dec073920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.103+0000 7f2df17e4640 1 -- 172.21.15.62:0/3605605596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dec102790 msgr2=0x7f2dec073e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.103+0000 7f2df17e4640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dec102790 0x7f2dec073e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.103+0000 7f2df17e4640 1 -- 172.21.15.62:0/3605605596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dec101de0 msgr2=0x7f2dec073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.103+0000 7f2df17e4640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dec101de0 0x7f2dec073920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.103+0000 7f2df17e4640 1 -- 172.21.15.62:0/3605605596 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dec06d450 con 0x7f2dec106f10 2024-09-15T23:16:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.104+0000 7f2df17e4640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec106f10 0x7f2dec06cb90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2de4004af0 tx=0x7f2de4032f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.104+0000 7f2de9ffb640 1 -- 172.21.15.62:0/3605605596 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de4041070 con 0x7f2dec106f10 2024-09-15T23:16:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.104+0000 7f2df3247640 1 -- 172.21.15.62:0/3605605596 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dec06d6e0 con 0x7f2dec106f10 2024-09-15T23:16:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.104+0000 7f2df3247640 1 -- 172.21.15.62:0/3605605596 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dec071620 con 0x7f2dec106f10 2024-09-15T23:16:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.104+0000 7f2de9ffb640 1 -- 172.21.15.62:0/3605605596 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2de4003e50 con 0x7f2dec106f10 2024-09-15T23:16:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.105+0000 7f2de9ffb640 1 -- 172.21.15.62:0/3605605596 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de404b060 con 0x7f2dec106f10 2024-09-15T23:16:32.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.108+0000 7f2de9ffb640 1 -- 172.21.15.62:0/3605605596 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2de4044070 con 0x7f2dec106f10 2024-09-15T23:16:32.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.109+0000 7f2de9ffb640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ddc0a3340 0x7f2ddc0a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.110+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ddc0a3340 0x7f2ddc0a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.110+0000 7f2de9ffb640 1 -- 172.21.15.62:0/3605605596 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2de40f03e0 con 0x7f2dec106f10 2024-09-15T23:16:32.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.110+0000 7f2df5620740 1 Processor -- start 2024-09-15T23:16:32.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.110+0000 7f2df5620740 1 -- start start 2024-09-15T23:16:32.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2df5620740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa3760 0x55d25afa3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2df5620740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afae400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2debfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2debfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:33650/0 (socket says 172.21.15.62:33650) 2024-09-15T23:16:32.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2df5620740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 0x55d25afb0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2debfff640 1 -- 172.21.15.62:0/3882069549 learned_addr learned my addr 172.21.15.62:0/3882069549 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2df5620740 1 -- 172.21.15.62:0/3882069549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d25af092a0 con 0x55d25afae940 2024-09-15T23:16:32.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2df5620740 1 -- 172.21.15.62:0/3882069549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d25aeb0b10 con 0x55d25afa4080 2024-09-15T23:16:32.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2df5620740 1 -- 172.21.15.62:0/3882069549 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d25af05260 con 0x55d25afa3760 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa3760 0x55d25afa3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2df17e4640 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 0x55d25afb0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2debfff640 1 -- 172.21.15.62:0/3882069549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa3760 msgr2=0x55d25afa3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2debfff640 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa3760 0x55d25afa3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.111+0000 7f2debfff640 1 -- 172.21.15.62:0/3882069549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 msgr2=0x55d25afb0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.112+0000 7f2debfff640 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 0x55d25afb0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.112+0000 7f2debfff640 1 -- 172.21.15.62:0/3882069549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d25ac7ab90 con 0x55d25afa4080 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.112+0000 7f2df17e4640 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 0x55d25afb0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.112+0000 7f2debfff640 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afae400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009b00 tx=0x7f2dd800f360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca3571994c978121 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.112+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ddc0a3340 0x7f2ddc0a5800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009f90 tx=0x7f2de00137d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.112+0000 7f2dd6ffd640 1 -- 172.21.15.62:0/3882069549 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd800fd90 con 0x55d25afa4080 2024-09-15T23:16:32.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.112+0000 7f2dd6ffd640 1 -- 172.21.15.62:0/3882069549 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd8011920 con 0x55d25afa4080 2024-09-15T23:16:32.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.112+0000 7f2dd6ffd640 1 -- 172.21.15.62:0/3882069549 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd801b9b0 con 0x55d25afa4080 2024-09-15T23:16:32.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.113+0000 7f2df5620740 1 -- 172.21.15.62:0/3882069549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 msgr2=0x55d25afae400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.113+0000 7f2df5620740 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afae400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009b00 tx=0x7f2dd800f360 comp rx=0 tx=0).stop 2024-09-15T23:16:32.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.115+0000 7f2df5620740 1 -- 172.21.15.62:0/3882069549 shutdown_connections 2024-09-15T23:16:32.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.115+0000 7f2df5620740 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 0x55d25afb0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.115+0000 7f2df5620740 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afae400 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.115+0000 7f2df5620740 1 --2- 172.21.15.62:0/3882069549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa3760 0x55d25afa3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.115+0000 7f2df5620740 1 -- 172.21.15.62:0/3882069549 >> 172.21.15.62:0/3882069549 conn(0x55d25af9fcf0 msgr2=0x55d25afa00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.115+0000 7f2df5620740 1 -- 172.21.15.62:0/3882069549 shutdown_connections 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.115+0000 7f2df5620740 1 -- 172.21.15.62:0/3882069549 wait complete. 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.116+0000 7f2df5620740 1 -- start start 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.116+0000 7f2df5620740 1 Processor -- start 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.116+0000 7f2df5620740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa3760 0x55d25afae400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.116+0000 7f2df5620740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2df5620740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 0x55d25afabe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2df5620740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d25afb1510 con 0x55d25afa3760 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2df5620740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d25ae916e0 con 0x55d25afa4080 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2df5620740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d25afb1690 con 0x55d25afae940 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2df5620740 10 client.0 _ll_register_callbacks cb 0x55d25aeb5a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2debfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2debfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afab6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:33660/0 (socket says 172.21.15.62:33660) 2024-09-15T23:16:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2debfff640 1 -- 172.21.15.62:0/3816503258 learned_addr learned my addr 172.21.15.62:0/3816503258 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa3760 0x55d25afae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2df17e4640 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 0x55d25afabe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2debfff640 1 -- 172.21.15.62:0/3816503258 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 msgr2=0x55d25afabe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2debfff640 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 0x55d25afabe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2debfff640 1 -- 172.21.15.62:0/3816503258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa3760 msgr2=0x55d25afae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2debfff640 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa3760 0x55d25afae400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.117+0000 7f2debfff640 1 -- 172.21.15.62:0/3816503258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d25afba9e0 con 0x55d25afa4080 2024-09-15T23:16:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2debfff640 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afab6b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800fea0 tx=0x7f2dd800fed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a7bd46a857e431a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2dcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:16:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2dcbfff640 1 -- 172.21.15.62:0/3816503258 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd802a060 con 0x55d25afa4080 2024-09-15T23:16:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2df5620740 1 -- 172.21.15.62:0/3816503258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dd8011920 con 0x55d25afa4080 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2dcbfff640 1 -- 172.21.15.62:0/3816503258 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2dd8011ad0 con 0x55d25afa4080 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2dcbfff640 1 -- 172.21.15.62:0/3816503258 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2dd8031050 con 0x55d25afa4080 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2df5620740 10 client.25190 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2df5620740 1 -- 172.21.15.62:0/3816503258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d25afbd270 con 0x55d25afa4080 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2dcbfff640 1 -- 172.21.15.62:0/3816503258 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd8011920 con 0x55d25afa4080 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2dcbfff640 1 -- 172.21.15.62:0/3816503258 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f2dd8004930 con 0x55d25afa4080 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2df5620740 10 client.25190 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2df5620740 10 client.25190 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2df5620740 20 client.25190 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2df5620740 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d25afbe680 0x55d25afc0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2df5620740 4 client.25190 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.118+0000 7f2df5620740 1 -- 172.21.15.62:0/3816503258 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d25af169d0 con 0x55d25afbe680 2024-09-15T23:16:32.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.119+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d25afbe680 0x55d25afc0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.120+0000 7f2dcbfff640 10 client.25190 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:32.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.120+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d25afbe680 0x55d25afc0a70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbd59185b536c2a9 server_cookie=4ca703c21f34b14d in_seq=0 out_seq=0 2024-09-15T23:16:32.122 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:32.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.120+0000 7f2dcbfff640 1 -- 172.21.15.62:0/3816503258 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d25af169d0 con 0x55d25afbe680 2024-09-15T23:16:32.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.120+0000 7f2dcbfff640 10 client.25190 handle_command_reply: tid=0 2024-09-15T23:16:32.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 client.25190 shutdown 2024-09-15T23:16:32.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 -- 172.21.15.62:0/3816503258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 msgr2=0x55d25afab6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afab6b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800fea0 tx=0x7f2dd800fed0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 -- 172.21.15.62:0/3816503258 shutdown_connections 2024-09-15T23:16:32.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d25afbe680 0x55d25afc0a70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2dcbfff640 0 client.25190 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:32.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 0x55d25afabe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa4080 0x55d25afab6b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 --2- 172.21.15.62:0/3816503258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa3760 0x55d25afae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 -- 172.21.15.62:0/3816503258 >> 172.21.15.62:0/3816503258 conn(0x55d25af9fcf0 msgr2=0x55d25afa59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 -- 172.21.15.62:0/3816503258 shutdown_connections 2024-09-15T23:16:32.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.121+0000 7f2df5620740 1 -- 172.21.15.62:0/3816503258 wait complete. 2024-09-15T23:16:32.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.122+0000 7f2df5620740 20 client.25190 trim_cache size 0 max 16384 2024-09-15T23:16:32.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df5620740 1 Processor -- start 2024-09-15T23:16:32.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df5620740 1 -- start start 2024-09-15T23:16:32.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df5620740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25af9ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df5620740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa4080 0x55d25afc60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df5620740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 0x55d25afc6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df5620740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd00123e0 con 0x55d25afae940 2024-09-15T23:16:32.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df5620740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd000fcb0 con 0x55d25afa3760 2024-09-15T23:16:32.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df5620740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd000db00 con 0x55d25afa4080 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df0fe3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25af9ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.154+0000 7f2df0fe3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25af9ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:33668/0 (socket says 172.21.15.62:33668) 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.155+0000 7f2df0fe3640 1 -- 172.21.15.62:0/1401593988 learned_addr learned my addr 172.21.15.62:0/1401593988 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.155+0000 7f2debfff640 1 --2- 172.21.15.62:0/1401593988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa4080 0x55d25afc60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.155+0000 7f2df0fe3640 1 -- 172.21.15.62:0/1401593988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa4080 msgr2=0x55d25afc60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.155+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/1401593988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa4080 0x55d25afc60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.155+0000 7f2df0fe3640 1 -- 172.21.15.62:0/1401593988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 msgr2=0x55d25afc6610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.155+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/1401593988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 0x55d25afc6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.155+0000 7f2df0fe3640 1 -- 172.21.15.62:0/1401593988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d25af9fb20 con 0x55d25afa3760 2024-09-15T23:16:32.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.155+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/1401593988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25af9ed60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2de000f680 tx=0x7f2de0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4248ffb283f38bb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.155+0000 7f2dd67fc640 1 -- 172.21.15.62:0/1401593988 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de0031070 con 0x55d25afa3760 2024-09-15T23:16:32.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.156+0000 7f2dd67fc640 1 -- 172.21.15.62:0/1401593988 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d25af169d0 con 0x55d25afa3760 2024-09-15T23:16:32.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.156+0000 7f2dd67fc640 1 -- 172.21.15.62:0/1401593988 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de002ccf0 con 0x55d25afa3760 2024-09-15T23:16:32.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.156+0000 7f2df5620740 1 -- 172.21.15.62:0/1401593988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 msgr2=0x55d25af9ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.156+0000 7f2df5620740 1 --2- 172.21.15.62:0/1401593988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25af9ed60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2de000f680 tx=0x7f2de0005f90 comp rx=0 tx=0).stop 2024-09-15T23:16:32.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.157+0000 7f2df5620740 1 -- 172.21.15.62:0/1401593988 shutdown_connections 2024-09-15T23:16:32.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.157+0000 7f2df5620740 1 --2- 172.21.15.62:0/1401593988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afae940 0x55d25afc6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.157+0000 7f2df5620740 1 --2- 172.21.15.62:0/1401593988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afa4080 0x55d25afc60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.157+0000 7f2df5620740 1 --2- 172.21.15.62:0/1401593988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25af9ed60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.157+0000 7f2df5620740 1 -- 172.21.15.62:0/1401593988 >> 172.21.15.62:0/1401593988 conn(0x55d25af9fcf0 msgr2=0x55d25af9e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.158+0000 7f2df5620740 1 -- 172.21.15.62:0/1401593988 shutdown_connections 2024-09-15T23:16:32.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.158+0000 7f2df5620740 1 -- 172.21.15.62:0/1401593988 wait complete. 2024-09-15T23:16:32.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.159+0000 7f2df5620740 1 -- start start 2024-09-15T23:16:32.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.159+0000 7f2df5620740 1 Processor -- start 2024-09-15T23:16:32.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.159+0000 7f2df5620740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25afb7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.159+0000 7f2df5620740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa4080 0x55d25afb1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.159+0000 7f2df5620740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 0x55d25afb1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.159+0000 7f2df5620740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd000e170 con 0x55d25afa4080 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.159+0000 7f2df5620740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd000f5e0 con 0x55d25afa3760 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.159+0000 7f2df5620740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd000e840 con 0x55d25afae940 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df5620740 10 client.0 _ll_register_callbacks cb 0x7f2dd8009980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df0fe3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25afb7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df0fe3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25afb7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:33680/0 (socket says 172.21.15.62:33680) 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df0fe3640 1 -- 172.21.15.62:0/715774154 learned_addr learned my addr 172.21.15.62:0/715774154 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2debfff640 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa4080 0x55d25afb1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df17e4640 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 0x55d25afb1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df0fe3640 1 -- 172.21.15.62:0/715774154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 msgr2=0x55d25afb1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 0x55d25afb1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df0fe3640 1 -- 172.21.15.62:0/715774154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa4080 msgr2=0x55d25afb1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa4080 0x55d25afb1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df0fe3640 1 -- 172.21.15.62:0/715774154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d25af9b260 con 0x55d25afa3760 2024-09-15T23:16:32.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df17e4640 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 0x55d25afb1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:32.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2df0fe3640 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25afb7060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2de002f040 tx=0x7f2de0009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd85fea5da0d5d94 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2dcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:16:32.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.160+0000 7f2dcbfff640 1 -- 172.21.15.62:0/715774154 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de0031070 con 0x55d25afa3760 2024-09-15T23:16:32.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.162+0000 7f2dcbfff640 1 -- 172.21.15.62:0/715774154 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2de0020980 con 0x55d25afa3760 2024-09-15T23:16:32.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.165+0000 7f2df5620740 1 -- 172.21.15.62:0/715774154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d25af169d0 con 0x55d25afa3760 2024-09-15T23:16:32.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.166+0000 7f2dcbfff640 1 -- 172.21.15.62:0/715774154 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2de0020b80 con 0x55d25afa3760 2024-09-15T23:16:32.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.166+0000 7f2df5620740 10 client.25196 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:32.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.166+0000 7f2df5620740 1 -- 172.21.15.62:0/715774154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d25b029e60 con 0x55d25afa3760 2024-09-15T23:16:32.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.166+0000 7f2dcbfff640 1 -- 172.21.15.62:0/715774154 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2de0020d20 con 0x55d25afa3760 2024-09-15T23:16:32.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.166+0000 7f2dcbfff640 1 -- 172.21.15.62:0/715774154 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f2de002b070 con 0x55d25afa3760 2024-09-15T23:16:32.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.166+0000 7f2df5620740 10 client.25196 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:32.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.167+0000 7f2df5620740 10 client.25196 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:16:32.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.167+0000 7f2df5620740 20 client.25196 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:32.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.167+0000 7f2df5620740 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d25afbe680 0x55d25b02c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.167+0000 7f2df5620740 4 client.25196 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}] 2024-09-15T23:16:32.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.167+0000 7f2df5620740 1 -- 172.21.15.62:0/715774154 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 150.2, "await_for": 45.0}) -- 0x55d25af169d0 con 0x55d25afbe680 2024-09-15T23:16:32.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.167+0000 7f2debfff640 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d25afbe680 0x55d25b02c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.168+0000 7f2debfff640 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d25afbe680 0x55d25b02c5b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97ef931d038ac4c6 server_cookie=db21223a4c1b45ac in_seq=0 out_seq=0 2024-09-15T23:16:32.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.168+0000 7f2dcbfff640 10 client.25196 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:32.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.652+0000 7f1280290640 1 -- 172.21.15.62:0/4112692131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278106a80 msgr2=0x7f1278106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.652+0000 7f1280290640 1 --2- 172.21.15.62:0/4112692131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278106a80 0x7f1278106e60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1274009930 tx=0x7f127402a420 comp rx=0 tx=0).stop 2024-09-15T23:16:32.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.652+0000 7f1280290640 1 -- 172.21.15.62:0/4112692131 shutdown_connections 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.652+0000 7f1280290640 1 --2- 172.21.15.62:0/4112692131 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1278101420 0x7f127810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.652+0000 7f1280290640 1 --2- 172.21.15.62:0/4112692131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1278100a80 0x7f1278100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.652+0000 7f1280290640 1 --2- 172.21.15.62:0/4112692131 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278106a80 0x7f1278106e60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.652+0000 7f1280290640 1 -- 172.21.15.62:0/4112692131 >> 172.21.15.62:0/4112692131 conn(0x7f12780fc7a0 msgr2=0x7f12780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 -- 172.21.15.62:0/4112692131 shutdown_connections 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 -- 172.21.15.62:0/4112692131 wait complete. 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 Processor -- start 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 -- start start 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278100a80 0x7f1278198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1278101420 0x7f1278198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1278106a80 0x7f127819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1278110ed0 con 0x7f1278100a80 2024-09-15T23:16:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1278110d50 con 0x7f1278101420 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.653+0000 7f1280290640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1278111050 con 0x7f1278106a80 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e02c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278100a80 0x7f1278198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e02c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278100a80 0x7f1278198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59974/0 (socket says 172.21.15.62:59974) 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e02c640 1 -- 172.21.15.62:0/3529491695 learned_addr learned my addr 172.21.15.62:0/3529491695 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127d82b640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1278101420 0x7f1278198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e82d640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1278106a80 0x7f127819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e02c640 1 -- 172.21.15.62:0/3529491695 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1278106a80 msgr2=0x7f127819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e02c640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1278106a80 0x7f127819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e02c640 1 -- 172.21.15.62:0/3529491695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1278101420 msgr2=0x7f1278198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e02c640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1278101420 0x7f1278198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e02c640 1 -- 172.21.15.62:0/3529491695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127819d400 con 0x7f1278100a80 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127d82b640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1278101420 0x7f1278198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f127e02c640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278100a80 0x7f1278198450 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f127402a940 tx=0x7f1274003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f126f7fe640 1 -- 172.21.15.62:0/3529491695 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127403f070 con 0x7f1278100a80 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f126f7fe640 1 -- 172.21.15.62:0/3529491695 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1274031c90 con 0x7f1278100a80 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f1280290640 1 -- 172.21.15.62:0/3529491695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f127819d690 con 0x7f1278100a80 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f126f7fe640 1 -- 172.21.15.62:0/3529491695 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1274035030 con 0x7f1278100a80 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.654+0000 7f1280290640 1 -- 172.21.15.62:0/3529491695 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12781a4fd0 con 0x7f1278100a80 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.657+0000 7f126f7fe640 1 -- 172.21.15.62:0/3529491695 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1274042070 con 0x7f1278100a80 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.657+0000 7f1282669740 1 Processor -- start 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.657+0000 7f1282669740 1 -- start start 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f1282669740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc293b40 0x55dccc292090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f1282669740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc292a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f1282669740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f1282669740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55dccc1f82f0 con 0x55dccc29d370 2024-09-15T23:16:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f1282669740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55dccc19fcf0 con 0x55dccc2925d0 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f1282669740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55dccc1f42b0 con 0x55dccc293b40 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e02c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc292a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc293b40 0x55dccc292090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127d82b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc293b40 0x55dccc292090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33640/0 (socket says 172.21.15.62:33640) 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127d82b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59988/0 (socket says 172.21.15.62:59988) 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 -- 172.21.15.62:0/1411289190 learned_addr learned my addr 172.21.15.62:0/1411289190 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f126f7fe640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12640a3240 0x7f12640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 -- 172.21.15.62:0/1411289190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 msgr2=0x55dccc292a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 --2- 172.21.15.62:0/1411289190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc292a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 -- 172.21.15.62:0/1411289190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 msgr2=0x55dccc29f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 --2- 172.21.15.62:0/1411289190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 -- 172.21.15.62:0/1411289190 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dccc1843a0 con 0x55dccc293b40 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127d82b640 1 --2- 172.21.15.62:0/1411289190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12640a3240 0x7f12640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f126f7fe640 1 -- 172.21.15.62:0/3529491695 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f12740f02f0 con 0x7f1278100a80 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.658+0000 7f127e82d640 1 --2- 172.21.15.62:0/1411289190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc293b40 0x55dccc292090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1270011590 tx=0x7f1270011a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a937f5980cdc80b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.659+0000 7f126cdea640 1 -- 172.21.15.62:0/1411289190 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1270017050 con 0x55dccc293b40 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.659+0000 7f127e82d640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12640a3240 0x7f12640a5700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1270022000 tx=0x7f1270023000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.659+0000 7f126cdea640 1 -- 172.21.15.62:0/1411289190 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f127001bb90 con 0x55dccc293b40 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.659+0000 7f126cdea640 1 -- 172.21.15.62:0/1411289190 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127000e3b0 con 0x55dccc293b40 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.659+0000 7f1282669740 1 -- 172.21.15.62:0/1411289190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc293b40 msgr2=0x55dccc292090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.659+0000 7f1282669740 1 --2- 172.21.15.62:0/1411289190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc293b40 0x55dccc292090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1270011590 tx=0x7f1270011a50 comp rx=0 tx=0).stop 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.659+0000 7f1282669740 1 -- 172.21.15.62:0/1411289190 shutdown_connections 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.659+0000 7f1282669740 1 --2- 172.21.15.62:0/1411289190 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.659+0000 7f1282669740 1 --2- 172.21.15.62:0/1411289190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc292a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 --2- 172.21.15.62:0/1411289190 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc293b40 0x55dccc292090 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 -- 172.21.15.62:0/1411289190 >> 172.21.15.62:0/1411289190 conn(0x55dccc25b310 msgr2=0x55dccc28ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 -- 172.21.15.62:0/1411289190 shutdown_connections 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 -- 172.21.15.62:0/1411289190 wait complete. 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 -- start start 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 Processor -- start 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 0x55dccc29cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc293b40 0x55dccc29a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55dccc2a0070 con 0x55dccc29d370 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55dccc1b3060 con 0x55dccc293b40 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55dccc2a01f0 con 0x55dccc2925d0 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f1282669740 10 client.0 _ll_register_callbacks cb 0x55dccbf068c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:32.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f127e82d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f127e02c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 0x55dccc29cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f127d82b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc293b40 0x55dccc29a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f127e82d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60000/0 (socket says 172.21.15.62:60000) 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.660+0000 7f127e82d640 1 -- 172.21.15.62:0/746586193 learned_addr learned my addr 172.21.15.62:0/746586193 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f127e82d640 1 -- 172.21.15.62:0/746586193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 msgr2=0x55dccc29cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f127e82d640 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 0x55dccc29cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f127e82d640 1 -- 172.21.15.62:0/746586193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc293b40 msgr2=0x55dccc29a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f127e82d640 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc293b40 0x55dccc29a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f127e82d640 1 -- 172.21.15.62:0/746586193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dccbf693c0 con 0x55dccc29d370 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f127e82d640 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29a800 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1270000f80 tx=0x7f127000f7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c94911d6935c2a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f1261ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:32.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f1261ffb640 1 -- 172.21.15.62:0/746586193 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1270033060 con 0x55dccc29d370 2024-09-15T23:16:32.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f1261ffb640 1 -- 172.21.15.62:0/746586193 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1270025630 con 0x55dccc29d370 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f1282669740 1 -- 172.21.15.62:0/746586193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f127001bb90 con 0x55dccc29d370 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f1261ffb640 1 -- 172.21.15.62:0/746586193 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f127003a050 con 0x55dccc29d370 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f1282669740 10 client.15528 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f1282669740 1 -- 172.21.15.62:0/746586193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dccc2abb80 con 0x55dccc29d370 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.661+0000 7f1261ffb640 1 -- 172.21.15.62:0/746586193 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f127001bb90 con 0x55dccc29d370 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f1261ffb640 1 -- 172.21.15.62:0/746586193 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f127002d380 con 0x55dccc29d370 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f1282669740 10 client.15528 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f1282669740 10 client.15528 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f1282669740 20 client.15528 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f1282669740 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55dccc2aced0 0x55dccc2af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f1282669740 4 client.15528 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f1282669740 1 -- 172.21.15.62:0/746586193 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dccc205860 con 0x55dccc2aced0 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f127e02c640 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55dccc2aced0 0x55dccc2af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f127e02c640 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55dccc2aced0 0x55dccc2af2c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1b1144f72948f376 server_cookie=8a612ec03b061cfc in_seq=0 out_seq=0 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.662+0000 7f1261ffb640 10 client.15528 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1261ffb640 1 -- 172.21.15.62:0/746586193 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dccc205860 con 0x55dccc2aced0 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1261ffb640 10 client.15528 handle_command_reply: tid=0 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 client.15528 shutdown 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 -- 172.21.15.62:0/746586193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 msgr2=0x55dccc29a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29a800 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1270000f80 tx=0x7f127000f7a0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 -- 172.21.15.62:0/746586193 shutdown_connections 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55dccc2aced0 0x55dccc2af2c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1261ffb640 0 client.15528 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc29d370 0x55dccc29a800 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc293b40 0x55dccc29a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 --2- 172.21.15.62:0/746586193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 0x55dccc29cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 -- 172.21.15.62:0/746586193 >> 172.21.15.62:0/746586193 conn(0x55dccc25b310 msgr2=0x55dccc294960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 -- 172.21.15.62:0/746586193 shutdown_connections 2024-09-15T23:16:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 1 -- 172.21.15.62:0/746586193 wait complete. 2024-09-15T23:16:32.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.663+0000 7f1282669740 20 client.15528 trim_cache size 0 max 16384 2024-09-15T23:16:32.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.676+0000 7f1282669740 1 Processor -- start 2024-09-15T23:16:32.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.676+0000 7f1282669740 1 -- start start 2024-09-15T23:16:32.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f1282669740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 0x55dccc28ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f1282669740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f1282669740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc29d370 0x55dccc295120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f1282669740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f12781b1570 con 0x55dccc293b40 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f1282669740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12781aee40 con 0x55dccc29d370 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f1282669740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f12781acc90 con 0x55dccc2925d0 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127d82b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127e02c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 0x55dccc28ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127e82d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc29d370 0x55dccc295120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127d82b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60012/0 (socket says 172.21.15.62:60012) 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127d82b640 1 -- 172.21.15.62:0/756075397 learned_addr learned my addr 172.21.15.62:0/756075397 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127e02c640 1 -- 172.21.15.62:0/756075397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc29d370 msgr2=0x55dccc295120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127e02c640 1 --2- 172.21.15.62:0/756075397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc29d370 0x55dccc295120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127e02c640 1 -- 172.21.15.62:0/756075397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 msgr2=0x55dccc2a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127e02c640 1 --2- 172.21.15.62:0/756075397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2a8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127e02c640 1 -- 172.21.15.62:0/756075397 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dccc29d130 con 0x55dccc2925d0 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127d82b640 1 --2- 172.21.15.62:0/756075397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127e82d640 1 --2- 172.21.15.62:0/756075397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc29d370 0x55dccc295120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.677+0000 7f127e02c640 1 --2- 172.21.15.62:0/756075397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 0x55dccc28ed70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12740bc2e0 tx=0x7f12740ed8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ac72f668ad67a6e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1263fff640 1 -- 172.21.15.62:0/756075397 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1274049040 con 0x55dccc2925d0 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1263fff640 1 -- 172.21.15.62:0/756075397 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1274031e40 con 0x55dccc2925d0 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1263fff640 1 -- 172.21.15.62:0/756075397 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12740dc040 con 0x55dccc2925d0 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1282669740 1 -- 172.21.15.62:0/756075397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 msgr2=0x55dccc28ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1282669740 1 --2- 172.21.15.62:0/756075397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 0x55dccc28ed70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12740bc2e0 tx=0x7f12740ed8e0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1282669740 1 -- 172.21.15.62:0/756075397 shutdown_connections 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1282669740 1 --2- 172.21.15.62:0/756075397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc29d370 0x55dccc295120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1282669740 1 --2- 172.21.15.62:0/756075397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1282669740 1 --2- 172.21.15.62:0/756075397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc2925d0 0x55dccc28ed70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1282669740 1 -- 172.21.15.62:0/756075397 >> 172.21.15.62:0/756075397 conn(0x55dccc25b310 msgr2=0x55dccc2a8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1282669740 1 -- 172.21.15.62:0/756075397 shutdown_connections 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.678+0000 7f1282669740 1 -- 172.21.15.62:0/756075397 wait complete. 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f1282669740 1 -- start start 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f1282669740 1 Processor -- start 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f1282669740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc2b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f1282669740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f1282669740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc29d370 0x55dccc2b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f1282669740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f12781ad300 con 0x55dccc293b40 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f1282669740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12781ae770 con 0x55dccc2925d0 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f1282669740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f12781ad9d0 con 0x55dccc29d370 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f1282669740 10 client.0 _ll_register_callbacks cb 0x7f1270011f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127d82b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127e02c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc2b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127e82d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc29d370 0x55dccc2b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127e02c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc2b2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39590/0 (socket says 172.21.15.62:39590) 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127e02c640 1 -- 172.21.15.62:0/329584541 learned_addr learned my addr 172.21.15.62:0/329584541 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127d82b640 1 -- 172.21.15.62:0/329584541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc29d370 msgr2=0x55dccc2b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127d82b640 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc29d370 0x55dccc2b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127d82b640 1 -- 172.21.15.62:0/329584541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 msgr2=0x55dccc2b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127d82b640 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc2b2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127d82b640 1 -- 172.21.15.62:0/329584541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dccc2b0d10 con 0x55dccc293b40 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.679+0000 7f127e02c640 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc2b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f127d82b640 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2afed0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x55dccc28d420 tx=0x7f126800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229a6d5d069adaad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1261ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1261ffb640 1 -- 172.21.15.62:0/329584541 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1268023060 con 0x55dccc293b40 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1261ffb640 1 -- 172.21.15.62:0/329584541 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1268004250 con 0x55dccc293b40 2024-09-15T23:16:32.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1282669740 1 -- 172.21.15.62:0/329584541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dccc1ffd70 con 0x55dccc293b40 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1261ffb640 1 -- 172.21.15.62:0/329584541 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f126802a050 con 0x55dccc293b40 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1282669740 10 client.15534 fetch_fsmap learned FSMap version 25 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1282669740 1 -- 172.21.15.62:0/329584541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dccc31bde0 con 0x55dccc293b40 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1261ffb640 1 -- 172.21.15.62:0/329584541 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dccc1ffd70 con 0x55dccc293b40 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1261ffb640 1 -- 172.21.15.62:0/329584541 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f1268010430 con 0x55dccc293b40 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1282669740 10 client.15534 fetch_fsmap finished waiting for FSMap version 25 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1282669740 10 client.15534 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1282669740 20 client.15534 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1282669740 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55dccc2aced0 0x55dccc31e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1282669740 4 client.15534 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.680+0000 7f1282669740 1 -- 172.21.15.62:0/329584541 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dccc17c170 con 0x55dccc2aced0 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f127e02c640 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55dccc2aced0 0x55dccc31e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1261ffb640 10 client.15534 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:16:32.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f127e02c640 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55dccc2aced0 0x55dccc31e5d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d2fde433692dba4d server_cookie=fe1bf29141386959 in_seq=0 out_seq=0 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1261ffb640 1 -- 172.21.15.62:0/329584541 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dccc17c170 con 0x55dccc2aced0 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1261ffb640 10 client.15534 handle_command_reply: tid=0 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1282669740 1 client.15534 shutdown 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1282669740 1 -- 172.21.15.62:0/329584541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 msgr2=0x55dccc2afed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1282669740 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2afed0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x55dccc28d420 tx=0x7f126800f790 comp rx=0 tx=0).stop 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1282669740 1 -- 172.21.15.62:0/329584541 shutdown_connections 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1282669740 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55dccc2aced0 0x55dccc31e5d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1282669740 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dccc29d370 0x55dccc2b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1282669740 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dccc293b40 0x55dccc2afed0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1261ffb640 0 client.15534 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1282669740 1 --2- 172.21.15.62:0/329584541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dccc2925d0 0x55dccc2b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.681+0000 7f1282669740 1 -- 172.21.15.62:0/329584541 >> 172.21.15.62:0/329584541 conn(0x55dccc25b310 msgr2=0x55dccc29b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1282669740 1 -- 172.21.15.62:0/329584541 shutdown_connections 2024-09-15T23:16:32.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1282669740 1 -- 172.21.15.62:0/329584541 wait complete. 2024-09-15T23:16:32.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1282669740 20 client.15534 trim_cache size 0 max 16384 2024-09-15T23:16:32.693 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 -- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12640a3240 msgr2=0x7f12640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12640a3240 0x7f12640a5700 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1270022000 tx=0x7f1270023000 comp rx=0 tx=0).stop 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 -- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278100a80 msgr2=0x7f1278198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278100a80 0x7f1278198450 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f127402a940 tx=0x7f1274003b20 comp rx=0 tx=0).stop 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 -- 172.21.15.62:0/3529491695 shutdown_connections 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12640a3240 0x7f12640a5700 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1278106a80 0x7f127819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1278101420 0x7f1278198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 --2- 172.21.15.62:0/3529491695 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1278100a80 0x7f1278198450 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.682+0000 7f1280290640 1 -- 172.21.15.62:0/3529491695 >> 172.21.15.62:0/3529491695 conn(0x7f12780fc7a0 msgr2=0x7f127806aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:32.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.683+0000 7f1280290640 1 -- 172.21.15.62:0/3529491695 shutdown_connections 2024-09-15T23:16:32.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:32.683+0000 7f1280290640 1 -- 172.21.15.62:0/3529491695 wait complete. 2024-09-15T23:16:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:32 smithi062 ceph-mon[30949]: pgmap v378: 129 pgs: 129 active+clean; 909 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 33 MiB/s wr, 746 op/s 2024-09-15T23:16:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:32 smithi067 ceph-mon[35665]: pgmap v378: 129 pgs: 129 active+clean; 909 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 33 MiB/s wr, 746 op/s 2024-09-15T23:16:33.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:32 smithi148 ceph-mon[35610]: pgmap v378: 129 pgs: 129 active+clean; 909 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 33 MiB/s wr, 746 op/s 2024-09-15T23:16:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2dcbfff640 1 -- 172.21.15.62:0/715774154 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x55d25af169d0 con 0x55d25afbe680 2024-09-15T23:16:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2dcbfff640 10 client.25196 handle_command_reply: tid=0 2024-09-15T23:16:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 client.25196 shutdown 2024-09-15T23:16:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 -- 172.21.15.62:0/715774154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 msgr2=0x55d25afb7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25afb7060 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2de002f040 tx=0x7f2de0009640 comp rx=0 tx=0).stop 2024-09-15T23:16:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 -- 172.21.15.62:0/715774154 shutdown_connections 2024-09-15T23:16:33.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d25afbe680 0x55d25b02c5b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:33.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d25afae940 0x55d25afb1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:33.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d25afa4080 0x55d25afb1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:33.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 --2- 172.21.15.62:0/715774154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d25afa3760 0x55d25afb7060 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:33.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2dcbfff640 0 client.25196 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:33.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 -- 172.21.15.62:0/715774154 >> 172.21.15.62:0/715774154 conn(0x55d25af9fcf0 msgr2=0x55d25afad730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:33.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 -- 172.21.15.62:0/715774154 shutdown_connections 2024-09-15T23:16:33.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.405+0000 7f2df5620740 1 -- 172.21.15.62:0/715774154 wait complete. 2024-09-15T23:16:33.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df5620740 20 client.25196 trim_cache size 0 max 16384 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 26, 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 2, 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "10fd8a97": { 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 2, 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:16:33.409 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 150.2, 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:16:33.410 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:16:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 -- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ddc0a3340 msgr2=0x7f2ddc0a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ddc0a3340 0x7f2ddc0a5800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009f90 tx=0x7f2de00137d0 comp rx=0 tx=0).stop 2024-09-15T23:16:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 -- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec106f10 msgr2=0x7f2dec06cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec106f10 0x7f2dec06cb90 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2de4004af0 tx=0x7f2de4032f10 comp rx=0 tx=0).stop 2024-09-15T23:16:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 -- 172.21.15.62:0/3605605596 shutdown_connections 2024-09-15T23:16:33.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ddc0a3340 0x7f2ddc0a5800 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dec106f10 0x7f2dec06cb90 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dec102790 0x7f2dec073e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 --2- 172.21.15.62:0/3605605596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dec101de0 0x7f2dec073920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 -- 172.21.15.62:0/3605605596 >> 172.21.15.62:0/3605605596 conn(0x7f2dec0fd880 msgr2=0x7f2dec0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 -- 172.21.15.62:0/3605605596 shutdown_connections 2024-09-15T23:16:33.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:33.406+0000 7f2df3247640 1 -- 172.21.15.62:0/3605605596 wait complete. 2024-09-15T23:16:34.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:33 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:16:34.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:33 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:16:34.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:33 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:16:34.572 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 15.0 s, 1358.3 tps, lat 38.739 ms stddev 40.435 2024-09-15T23:16:35.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:34 smithi062 ceph-mon[30949]: pgmap v379: 129 pgs: 129 active+clean; 926 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 14 MiB/s wr, 715 op/s 2024-09-15T23:16:35.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:34 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:35.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:34 smithi067 ceph-mon[35665]: pgmap v379: 129 pgs: 129 active+clean; 926 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 14 MiB/s wr, 715 op/s 2024-09-15T23:16:35.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:34 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:35.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:34 smithi148 ceph-mon[35610]: pgmap v379: 129 pgs: 129 active+clean; 926 MiB data, 930 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 14 MiB/s wr, 715 op/s 2024-09-15T23:16:35.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:34 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:16:35.857 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '10fd8a97', quiesce took 8.4 seconds. Will release after: 21.799999999999997 2024-09-15T23:16:36.045 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:16:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.809+0000 7f65731ad640 1 -- 172.21.15.62:0/2237582217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c100480 msgr2=0x7f656c100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.809+0000 7f65731ad640 1 --2- 172.21.15.62:0/2237582217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c100480 0x7f656c100860 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6554009900 tx=0x7f655402a410 comp rx=0 tx=0).stop 2024-09-15T23:16:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.809+0000 7f65731ad640 1 -- 172.21.15.62:0/2237582217 shutdown_connections 2024-09-15T23:16:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.809+0000 7f65731ad640 1 --2- 172.21.15.62:0/2237582217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c1055b0 0x7f656c10d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.809+0000 7f65731ad640 1 --2- 172.21.15.62:0/2237582217 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c100e30 0x7f656c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.809+0000 7f65731ad640 1 --2- 172.21.15.62:0/2237582217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c100480 0x7f656c100860 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.809+0000 7f65731ad640 1 -- 172.21.15.62:0/2237582217 >> 172.21.15.62:0/2237582217 conn(0x7f656c077210 msgr2=0x7f656c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 -- 172.21.15.62:0/2237582217 shutdown_connections 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 -- 172.21.15.62:0/2237582217 wait complete. 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 Processor -- start 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 -- start start 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c100480 0x7f656c199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c100e30 0x7f656c1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c1055b0 0x7f656c1a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f656c110eb0 con 0x7f656c1055b0 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f656c110d30 con 0x7f656c100480 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f65731ad640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f656c111030 con 0x7f656c100e30 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.810+0000 7f657174a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c1055b0 0x7f656c1a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f657174a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c1055b0 0x7f656c1a43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60038/0 (socket says 172.21.15.62:60038) 2024-09-15T23:16:36.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f657174a640 1 -- 172.21.15.62:0/1460464809 learned_addr learned my addr 172.21.15.62:0/1460464809 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f656bfff640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c100e30 0x7f656c1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f6570f49640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c100480 0x7f656c199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f657174a640 1 -- 172.21.15.62:0/1460464809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c100e30 msgr2=0x7f656c1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f657174a640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c100e30 0x7f656c1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f657174a640 1 -- 172.21.15.62:0/1460464809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c100480 msgr2=0x7f656c199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f657174a640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c100480 0x7f656c199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f657174a640 1 -- 172.21.15.62:0/1460464809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c1a4920 con 0x7f656c1055b0 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f657174a640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c1055b0 0x7f656c1a43e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f656400cce0 tx=0x7f656400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f6569ffb640 1 -- 172.21.15.62:0/1460464809 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656400f450 con 0x7f656c1055b0 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f6569ffb640 1 -- 172.21.15.62:0/1460464809 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656400fad0 con 0x7f656c1055b0 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f65731ad640 1 -- 172.21.15.62:0/1460464809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656c1a4af0 con 0x7f656c1055b0 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f6569ffb640 1 -- 172.21.15.62:0/1460464809 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564018c10 con 0x7f656c1055b0 2024-09-15T23:16:36.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.811+0000 7f65731ad640 1 -- 172.21.15.62:0/1460464809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656c1a4fb0 con 0x7f656c1055b0 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.813+0000 7f6569ffb640 1 -- 172.21.15.62:0/1460464809 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f656400f5f0 con 0x7f656c1055b0 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.814+0000 7f6569ffb640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3240 0x7f65580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.814+0000 7f6575586740 1 Processor -- start 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.814+0000 7f6570f49640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3240 0x7f65580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.814+0000 7f6575586740 1 -- start start 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.814+0000 7f6569ffb640 1 -- 172.21.15.62:0/1460464809 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f65640d0360 con 0x7f656c1055b0 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6575586740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x55608846a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6575586740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x55608846aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6575586740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 0x55608847bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6575586740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5560883d02f0 con 0x556088479780 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6575586740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556088377cf0 con 0x55608846bb40 2024-09-15T23:16:36.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6575586740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5560883cc2b0 con 0x55608846a5d0 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f657174a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 0x55608847bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6570f49640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3240 0x7f65580a5700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f65540096c0 tx=0x7f6554031490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f657174a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 0x55608847bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60052/0 (socket says 172.21.15.62:60052) 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f657174a640 1 -- 172.21.15.62:0/1961258669 learned_addr learned my addr 172.21.15.62:0/1961258669 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6570f49640 1 --2- 172.21.15.62:0/1961258669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x55608846a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f656bfff640 1 --2- 172.21.15.62:0/1961258669 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x55608846aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f657174a640 1 -- 172.21.15.62:0/1961258669 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 msgr2=0x55608846aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f657174a640 1 --2- 172.21.15.62:0/1961258669 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x55608846aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f657174a640 1 -- 172.21.15.62:0/1961258669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 msgr2=0x55608846a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f657174a640 1 --2- 172.21.15.62:0/1961258669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x55608846a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f657174a640 1 -- 172.21.15.62:0/1961258669 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55608835c3a0 con 0x556088479780 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f657174a640 1 --2- 172.21.15.62:0/1961258669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 0x55608847bb70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6564099070 tx=0x7f656409c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69afe3a38e562c6a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6562ffd640 1 -- 172.21.15.62:0/1961258669 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65640b5070 con 0x556088479780 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6562ffd640 1 -- 172.21.15.62:0/1961258669 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65640a0a50 con 0x556088479780 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.815+0000 7f6562ffd640 1 -- 172.21.15.62:0/1961258669 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656409aa30 con 0x556088479780 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.816+0000 7f6575586740 1 -- 172.21.15.62:0/1961258669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 msgr2=0x55608847bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.816+0000 7f6575586740 1 --2- 172.21.15.62:0/1961258669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 0x55608847bb70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6564099070 tx=0x7f656409c000 comp rx=0 tx=0).stop 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.816+0000 7f6575586740 1 -- 172.21.15.62:0/1961258669 shutdown_connections 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.816+0000 7f6575586740 1 --2- 172.21.15.62:0/1961258669 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 0x55608847bb70 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.816+0000 7f6575586740 1 --2- 172.21.15.62:0/1961258669 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x55608846aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.816+0000 7f6575586740 1 --2- 172.21.15.62:0/1961258669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x55608846a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.816+0000 7f6575586740 1 -- 172.21.15.62:0/1961258669 >> 172.21.15.62:0/1961258669 conn(0x556088433310 msgr2=0x556088466c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.816+0000 7f6575586740 1 -- 172.21.15.62:0/1961258669 shutdown_connections 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.816+0000 7f6575586740 1 -- 172.21.15.62:0/1961258669 wait complete. 2024-09-15T23:16:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f6575586740 1 -- start start 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f6575586740 1 Processor -- start 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f6575586740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x5560884791a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f6575586740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846bb40 0x556088476450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f6575586740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556088479780 0x556088476a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f6575586740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55608847c420 con 0x55608846bb40 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f6575586740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55608838b060 con 0x556088479780 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f6575586740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55608847c5a0 con 0x55608846a5d0 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f6575586740 10 client.0 _ll_register_callbacks cb 0x5560880de8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f656bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846bb40 0x556088476450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f656bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846bb40 0x556088476450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60066/0 (socket says 172.21.15.62:60066) 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.817+0000 7f656bfff640 1 -- 172.21.15.62:0/2021842557 learned_addr learned my addr 172.21.15.62:0/2021842557 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f6570f49640 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x5560884791a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f657174a640 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556088479780 0x556088476a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f656bfff640 1 -- 172.21.15.62:0/2021842557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 msgr2=0x5560884791a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f656bfff640 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x5560884791a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f656bfff640 1 -- 172.21.15.62:0/2021842557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556088479780 msgr2=0x556088476a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f656bfff640 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556088479780 0x556088476a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f656bfff640 1 -- 172.21.15.62:0/2021842557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560881413c0 con 0x55608846bb40 2024-09-15T23:16:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f656bfff640 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846bb40 0x556088476450 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f655c00bf90 tx=0x7f655c00f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd03d33f985a7fd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f654bfff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f654bfff640 1 -- 172.21.15.62:0/2021842557 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c018020 con 0x55608846bb40 2024-09-15T23:16:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f654bfff640 1 -- 172.21.15.62:0/2021842557 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f655c014870 con 0x55608846bb40 2024-09-15T23:16:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f6575586740 1 -- 172.21.15.62:0/2021842557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65640a0a50 con 0x55608846bb40 2024-09-15T23:16:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f654bfff640 1 -- 172.21.15.62:0/2021842557 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f655c023050 con 0x55608846bb40 2024-09-15T23:16:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f6575586740 10 client.15546 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.818+0000 7f6575586740 1 -- 172.21.15.62:0/2021842557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556088487f50 con 0x55608846bb40 2024-09-15T23:16:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.819+0000 7f654bfff640 1 -- 172.21.15.62:0/2021842557 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65640a0a50 con 0x55608846bb40 2024-09-15T23:16:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.819+0000 7f654bfff640 1 -- 172.21.15.62:0/2021842557 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f655c013b50 con 0x55608846bb40 2024-09-15T23:16:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.819+0000 7f6575586740 10 client.15546 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.819+0000 7f6575586740 10 client.15546 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.819+0000 7f6575586740 20 client.15546 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.819+0000 7f6575586740 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5560884894b0 0x55608848b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.819+0000 7f6575586740 4 client.15546 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.819+0000 7f6575586740 1 -- 172.21.15.62:0/2021842557 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560883dd860 con 0x5560884894b0 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.819+0000 7f6570f49640 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5560884894b0 0x55608848b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.820+0000 7f6570f49640 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5560884894b0 0x55608848b8d0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=585782595100db96 server_cookie=2d0ecfd99aa512e3 in_seq=0 out_seq=0 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.820+0000 7f654bfff640 10 client.15546 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.820+0000 7f654bfff640 1 -- 172.21.15.62:0/2021842557 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5560883dd860 con 0x5560884894b0 2024-09-15T23:16:36.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.820+0000 7f654bfff640 10 client.15546 handle_command_reply: tid=0 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.820+0000 7f6575586740 1 client.15546 shutdown 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 -- 172.21.15.62:0/2021842557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846bb40 msgr2=0x556088476450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846bb40 0x556088476450 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f655c00bf90 tx=0x7f655c00f670 comp rx=0 tx=0).stop 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 -- 172.21.15.62:0/2021842557 shutdown_connections 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5560884894b0 0x55608848b8d0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556088479780 0x556088476a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846bb40 0x556088476450 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 --2- 172.21.15.62:0/2021842557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x5560884791a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 -- 172.21.15.62:0/2021842557 >> 172.21.15.62:0/2021842557 conn(0x556088433310 msgr2=0x55608846c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f654bfff640 0 client.15546 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 -- 172.21.15.62:0/2021842557 shutdown_connections 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 1 -- 172.21.15.62:0/2021842557 wait complete. 2024-09-15T23:16:36.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.821+0000 7f6575586740 20 client.15546 trim_cache size 0 max 16384 2024-09-15T23:16:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.845+0000 7f6575586740 1 Processor -- start 2024-09-15T23:16:36.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.845+0000 7f6575586740 1 -- start start 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.845+0000 7f6575586740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x556088476420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.845+0000 7f6575586740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x5560884856c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.845+0000 7f6575586740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 0x556088491180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.846+0000 7f6575586740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1af2e0 con 0x556088479780 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.846+0000 7f6575586740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1acbb0 con 0x55608846bb40 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.846+0000 7f6575586740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1aaa00 con 0x55608846a5d0 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.846+0000 7f6570f49640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x556088476420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.846+0000 7f6570f49640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x556088476420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33704/0 (socket says 172.21.15.62:33704) 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.846+0000 7f6570f49640 1 -- 172.21.15.62:0/2477254698 learned_addr learned my addr 172.21.15.62:0/2477254698 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.847+0000 7f656bfff640 1 --2- 172.21.15.62:0/2477254698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x5560884856c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.847+0000 7f6570f49640 1 -- 172.21.15.62:0/2477254698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 msgr2=0x5560884856c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:36.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.847+0000 7f6570f49640 1 --2- 172.21.15.62:0/2477254698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x5560884856c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.847+0000 7f6570f49640 1 -- 172.21.15.62:0/2477254698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 msgr2=0x556088491180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.847+0000 7f6570f49640 1 --2- 172.21.15.62:0/2477254698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 0x556088491180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.847+0000 7f6570f49640 1 -- 172.21.15.62:0/2477254698 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556088466ed0 con 0x55608846a5d0 2024-09-15T23:16:36.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.849+0000 7f6570f49640 1 --2- 172.21.15.62:0/2477254698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x556088476420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f655403e040 tx=0x7f655403f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3aed530e7a6444 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:36.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.853+0000 7f65627fc640 1 -- 172.21.15.62:0/2477254698 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554035050 con 0x55608846a5d0 2024-09-15T23:16:36.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.853+0000 7f65627fc640 1 -- 172.21.15.62:0/2477254698 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6554003ad0 con 0x55608846a5d0 2024-09-15T23:16:36.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.853+0000 7f65627fc640 1 -- 172.21.15.62:0/2477254698 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554052a40 con 0x55608846a5d0 2024-09-15T23:16:36.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.854+0000 7f6575586740 1 -- 172.21.15.62:0/2477254698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 msgr2=0x556088476420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:36.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.854+0000 7f6575586740 1 --2- 172.21.15.62:0/2477254698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x556088476420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f655403e040 tx=0x7f655403f000 comp rx=0 tx=0).stop 2024-09-15T23:16:36.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.856+0000 7f6575586740 1 -- 172.21.15.62:0/2477254698 shutdown_connections 2024-09-15T23:16:36.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.856+0000 7f6575586740 1 --2- 172.21.15.62:0/2477254698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556088479780 0x556088491180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.856+0000 7f6575586740 1 --2- 172.21.15.62:0/2477254698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x5560884856c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.856+0000 7f6575586740 1 --2- 172.21.15.62:0/2477254698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608846a5d0 0x556088476420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.856+0000 7f6575586740 1 -- 172.21.15.62:0/2477254698 >> 172.21.15.62:0/2477254698 conn(0x556088433310 msgr2=0x556088475d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:36.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.862+0000 7f6575586740 1 -- 172.21.15.62:0/2477254698 shutdown_connections 2024-09-15T23:16:36.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.866+0000 7f6575586740 1 -- 172.21.15.62:0/2477254698 wait complete. 2024-09-15T23:16:36.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.866+0000 7f6575586740 1 -- start start 2024-09-15T23:16:36.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.866+0000 7f6575586740 1 Processor -- start 2024-09-15T23:16:36.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.867+0000 7f6575586740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846a5d0 0x55608846d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.867+0000 7f6575586740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x55608846dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.867+0000 7f6575586740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556088479780 0x55608846e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.867+0000 7f6575586740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1ab070 con 0x55608846a5d0 2024-09-15T23:16:36.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.867+0000 7f6575586740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1ac4e0 con 0x55608846bb40 2024-09-15T23:16:36.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.867+0000 7f6575586740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1ab740 con 0x556088479780 2024-09-15T23:16:36.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.867+0000 7f6575586740 10 client.0 _ll_register_callbacks cb 0x7f655c00b2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:36.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.869+0000 7f6570f49640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846a5d0 0x55608846d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.869+0000 7f6570f49640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846a5d0 0x55608846d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60084/0 (socket says 172.21.15.62:60084) 2024-09-15T23:16:36.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.869+0000 7f6570f49640 1 -- 172.21.15.62:0/4240493733 learned_addr learned my addr 172.21.15.62:0/4240493733 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:36.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.872+0000 7f6570f49640 1 -- 172.21.15.62:0/4240493733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556088479780 msgr2=0x55608846e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:36.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.872+0000 7f6570f49640 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556088479780 0x55608846e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.872+0000 7f6570f49640 1 -- 172.21.15.62:0/4240493733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 msgr2=0x55608846dd20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:16:36.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.872+0000 7f6570f49640 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x55608846dd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.872+0000 7f6570f49640 1 -- 172.21.15.62:0/4240493733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556088476dd0 con 0x55608846a5d0 2024-09-15T23:16:36.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.873+0000 7f6570f49640 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846a5d0 0x55608846d7e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6554031ba0 tx=0x7f6554052330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d7c0658fd3eefb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:36.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.873+0000 7f654bfff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:36.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.880+0000 7f654bfff640 1 -- 172.21.15.62:0/4240493733 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554035070 con 0x55608846a5d0 2024-09-15T23:16:36.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.880+0000 7f6575586740 1 -- 172.21.15.62:0/4240493733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6554003ad0 con 0x55608846a5d0 2024-09-15T23:16:36.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.880+0000 7f654bfff640 1 -- 172.21.15.62:0/4240493733 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f65540035f0 con 0x55608846a5d0 2024-09-15T23:16:36.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.881+0000 7f654bfff640 1 -- 172.21.15.62:0/4240493733 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6554069050 con 0x55608846a5d0 2024-09-15T23:16:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.886+0000 7f6575586740 10 client.15549 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:36.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.886+0000 7f6575586740 1 -- 172.21.15.62:0/4240493733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5560884819c0 con 0x55608846a5d0 2024-09-15T23:16:36.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.887+0000 7f654bfff640 1 -- 172.21.15.62:0/4240493733 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6554003ad0 con 0x55608846a5d0 2024-09-15T23:16:36.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.887+0000 7f654bfff640 1 -- 172.21.15.62:0/4240493733 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f655403b530 con 0x55608846a5d0 2024-09-15T23:16:36.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.887+0000 7f6575586740 10 client.15549 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:36.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.887+0000 7f6575586740 10 client.15549 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:16:36.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.887+0000 7f6575586740 20 client.15549 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:36.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.888+0000 7f6575586740 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x556088482120 0x55608850b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:36.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.888+0000 7f6575586740 4 client.15549 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:16:36.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.888+0000 7f6575586740 1 -- 172.21.15.62:0/4240493733 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5560883d7d70 con 0x556088482120 2024-09-15T23:16:36.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.896+0000 7f656bfff640 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x556088482120 0x55608850b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:36.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.910+0000 7f656bfff640 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x556088482120 0x55608850b960 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=28f67c78f8f2934b server_cookie=a2ee9d20b363c136 in_seq=0 out_seq=0 2024-09-15T23:16:36.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.910+0000 7f654bfff640 10 client.15549 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:16:36.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.910+0000 7f654bfff640 1 -- 172.21.15.62:0/4240493733 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5560883d7d70 con 0x556088482120 2024-09-15T23:16:36.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.910+0000 7f654bfff640 10 client.15549 handle_command_reply: tid=0 2024-09-15T23:16:36.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.910+0000 7f6575586740 1 client.15549 shutdown 2024-09-15T23:16:36.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.911+0000 7f6575586740 1 -- 172.21.15.62:0/4240493733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846a5d0 msgr2=0x55608846d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:36.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.911+0000 7f6575586740 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846a5d0 0x55608846d7e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6554031ba0 tx=0x7f6554052330 comp rx=0 tx=0).stop 2024-09-15T23:16:36.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.913+0000 7f6575586740 1 -- 172.21.15.62:0/4240493733 shutdown_connections 2024-09-15T23:16:36.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.913+0000 7f6575586740 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x556088482120 0x55608850b960 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.913+0000 7f6575586740 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556088479780 0x55608846e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.913+0000 7f6575586740 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608846bb40 0x55608846dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.913+0000 7f6575586740 1 --2- 172.21.15.62:0/4240493733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608846a5d0 0x55608846d7e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.913+0000 7f6575586740 1 -- 172.21.15.62:0/4240493733 >> 172.21.15.62:0/4240493733 conn(0x556088433310 msgr2=0x556088474d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:36.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.913+0000 7f654bfff640 0 client.15549 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:16:36.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.916+0000 7f6575586740 1 -- 172.21.15.62:0/4240493733 shutdown_connections 2024-09-15T23:16:36.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.919+0000 7f6575586740 1 -- 172.21.15.62:0/4240493733 wait complete. 2024-09-15T23:16:36.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.919+0000 7f6575586740 20 client.15549 trim_cache size 0 max 16384 2024-09-15T23:16:36.920 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:16:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 -- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3240 msgr2=0x7f65580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3240 0x7f65580a5700 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f65540096c0 tx=0x7f6554031490 comp rx=0 tx=0).stop 2024-09-15T23:16:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 -- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c1055b0 msgr2=0x7f656c1a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c1055b0 0x7f656c1a43e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f656400cce0 tx=0x7f656400a620 comp rx=0 tx=0).stop 2024-09-15T23:16:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 -- 172.21.15.62:0/1460464809 shutdown_connections 2024-09-15T23:16:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3240 0x7f65580a5700 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c1055b0 0x7f656c1a43e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c100e30 0x7f656c1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 --2- 172.21.15.62:0/1460464809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c100480 0x7f656c199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:36.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 -- 172.21.15.62:0/1460464809 >> 172.21.15.62:0/1460464809 conn(0x7f656c077210 msgr2=0x7f656c103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:36.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 -- 172.21.15.62:0/1460464809 shutdown_connections 2024-09-15T23:16:36.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:36.920+0000 7f65731ad640 1 -- 172.21.15.62:0/1460464809 wait complete. 2024-09-15T23:16:36.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:36 smithi062 ceph-mon[30949]: pgmap v380: 129 pgs: 129 active+clean; 926 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.8 MiB/s wr, 575 op/s 2024-09-15T23:16:36.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:16:36.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:16:36.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:36.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:36.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:36 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:16:37.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:36 smithi067 ceph-mon[35665]: pgmap v380: 129 pgs: 129 active+clean; 926 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.8 MiB/s wr, 575 op/s 2024-09-15T23:16:37.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:16:37.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:16:37.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:37.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:37.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:36 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:16:37.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:36 smithi148 ceph-mon[35610]: pgmap v380: 129 pgs: 129 active+clean; 926 MiB data, 957 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.8 MiB/s wr, 575 op/s 2024-09-15T23:16:37.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:16:37.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:16:37.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:16:37.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:16:37.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:36 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:16:37.811 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:16:38.070 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:38.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 -- 172.21.15.62:0/2107762430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 msgr2=0x7f16781029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:38.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 --2- 172.21.15.62:0/2107762430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 0x7f16781029b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1660009900 tx=0x7f166002a480 comp rx=0 tx=0).stop 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 -- 172.21.15.62:0/2107762430 shutdown_connections 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 --2- 172.21.15.62:0/2107762430 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1678102ef0 0x7f167810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 --2- 172.21.15.62:0/2107762430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 0x7f16781029b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 --2- 172.21.15.62:0/2107762430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678100e20 0x7f1678101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 -- 172.21.15.62:0/2107762430 >> 172.21.15.62:0/2107762430 conn(0x7f16780fc7d0 msgr2=0x7f16780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 -- 172.21.15.62:0/2107762430 shutdown_connections 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 -- 172.21.15.62:0/2107762430 wait complete. 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 Processor -- start 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.513+0000 7f167d81e640 1 -- start start 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f167d81e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1678100e20 0x7f16781992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f167d81e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 0x7f1678199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f167d81e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678102ef0 0x7f16781a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f167d81e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1678110fa0 con 0x7f1678102570 2024-09-15T23:16:38.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f167d81e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1678110e20 con 0x7f1678102ef0 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f167d81e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1678111120 con 0x7f1678100e20 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f166ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 0x7f1678199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f166ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 0x7f1678199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60106/0 (socket says 172.21.15.62:60106) 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f166ffff640 1 -- 172.21.15.62:0/1115672581 learned_addr learned my addr 172.21.15.62:0/1115672581 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f16777fe640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678102ef0 0x7f16781a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f1676ffd640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1678100e20 0x7f16781992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f166ffff640 1 -- 172.21.15.62:0/1115672581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1678100e20 msgr2=0x7f16781992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f166ffff640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1678100e20 0x7f16781992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f166ffff640 1 -- 172.21.15.62:0/1115672581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678102ef0 msgr2=0x7f16781a4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f166ffff640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678102ef0 0x7f16781a4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f166ffff640 1 -- 172.21.15.62:0/1115672581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16781a48b0 con 0x7f1678102570 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f16777fe640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678102ef0 0x7f16781a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.514+0000 7f166ffff640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 0x7f1678199830 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1660006010 tx=0x7f1660003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:38.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.515+0000 7f167c81c640 1 -- 172.21.15.62:0/1115672581 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166003f070 con 0x7f1678102570 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.515+0000 7f167c81c640 1 -- 172.21.15.62:0/1115672581 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1660031e30 con 0x7f1678102570 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.515+0000 7f167d81e640 1 -- 172.21.15.62:0/1115672581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16781a4a80 con 0x7f1678102570 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.515+0000 7f167c81c640 1 -- 172.21.15.62:0/1115672581 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660035030 con 0x7f1678102570 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.515+0000 7f167d81e640 1 -- 172.21.15.62:0/1115672581 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16781a4f60 con 0x7f1678102570 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.517+0000 7f167c81c640 1 -- 172.21.15.62:0/1115672581 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1660042070 con 0x7f1678102570 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.517+0000 7f167fbf7740 1 Processor -- start 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.517+0000 7f167fbf7740 1 -- start start 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c76c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 0x55586c76ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 0x55586c77db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f167fbf7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55586c6d22f0 con 0x55586c76db40 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f167fbf7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55586c679cf0 con 0x55586c76c5d0 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f167fbf7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55586c6ce2b0 con 0x55586c77b780 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c76c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c76c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60122/0 (socket says 172.21.15.62:60122) 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 -- 172.21.15.62:0/3814883886 learned_addr learned my addr 172.21.15.62:0/3814883886 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f167c81c640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16640a32c0 0x7f16640a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f1676ffd640 1 --2- 172.21.15.62:0/3814883886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 0x55586c76ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16640a32c0 0x7f16640a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f167c81c640 1 -- 172.21.15.62:0/1115672581 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f16600f0670 con 0x7f1678102570 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f166ffff640 1 --2- 172.21.15.62:0/3814883886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 0x55586c77db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 -- 172.21.15.62:0/3814883886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 msgr2=0x55586c77db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 --2- 172.21.15.62:0/3814883886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 0x55586c77db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 -- 172.21.15.62:0/3814883886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 msgr2=0x55586c76ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 --2- 172.21.15.62:0/3814883886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 0x55586c76ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 -- 172.21.15.62:0/3814883886 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55586c65e3a0 con 0x55586c76db40 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.518+0000 7f16777fe640 1 --2- 172.21.15.62:0/3814883886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c76c090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f167000cbf0 tx=0x7f1670007680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c62a21a4d76da57 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f166dffb640 1 -- 172.21.15.62:0/3814883886 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670015070 con 0x55586c76db40 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f166dffb640 1 -- 172.21.15.62:0/3814883886 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1670016040 con 0x55586c76db40 2024-09-15T23:16:38.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f166dffb640 1 -- 172.21.15.62:0/3814883886 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167001e020 con 0x55586c76db40 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f16777fe640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16640a32c0 0x7f16640a5780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f167001f000 tx=0x7f167001b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f167fbf7740 1 -- 172.21.15.62:0/3814883886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 msgr2=0x55586c76c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3814883886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c76c090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f167000cbf0 tx=0x7f1670007680 comp rx=0 tx=0).stop 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f167fbf7740 1 -- 172.21.15.62:0/3814883886 shutdown_connections 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3814883886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 0x55586c77db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3814883886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 0x55586c76ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3814883886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c76c090 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f167fbf7740 1 -- 172.21.15.62:0/3814883886 >> 172.21.15.62:0/3814883886 conn(0x55586c735310 msgr2=0x55586c768c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f167fbf7740 1 -- 172.21.15.62:0/3814883886 shutdown_connections 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.519+0000 7f167fbf7740 1 -- 172.21.15.62:0/3814883886 wait complete. 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f167fbf7740 1 -- start start 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f167fbf7740 1 Processor -- start 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76c5d0 0x55586c77b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76db40 0x55586c778430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 0x55586c778a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f167fbf7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55586c77e480 con 0x55586c76c5d0 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f167fbf7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55586c68d060 con 0x55586c77b780 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f167fbf7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55586c77e600 con 0x55586c76db40 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f167fbf7740 10 client.0 _ll_register_callbacks cb 0x55586c3e08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f1676ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76c5d0 0x55586c77b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f1676ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76c5d0 0x55586c77b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60138/0 (socket says 172.21.15.62:60138) 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.520+0000 7f1676ffd640 1 -- 172.21.15.62:0/3185986504 learned_addr learned my addr 172.21.15.62:0/3185986504 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f16777fe640 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 0x55586c778a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f166ffff640 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76db40 0x55586c778430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1676ffd640 1 -- 172.21.15.62:0/3185986504 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76db40 msgr2=0x55586c778430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1676ffd640 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76db40 0x55586c778430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1676ffd640 1 -- 172.21.15.62:0/3185986504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 msgr2=0x55586c778a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1676ffd640 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 0x55586c778a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1676ffd640 1 -- 172.21.15.62:0/3185986504 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55586c4433c0 con 0x55586c76c5d0 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1676ffd640 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76c5d0 0x55586c77b180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f166800ce00 tx=0x7f16680039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1b6909dcaee589 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1656ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1656ffd640 1 -- 172.21.15.62:0/3185986504 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1668011970 con 0x55586c76c5d0 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1656ffd640 1 -- 172.21.15.62:0/3185986504 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1668011b10 con 0x55586c76c5d0 2024-09-15T23:16:38.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f167fbf7740 1 -- 172.21.15.62:0/3185986504 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1670016040 con 0x55586c76c5d0 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f1656ffd640 1 -- 172.21.15.62:0/3185986504 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1668011d10 con 0x55586c76c5d0 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f167fbf7740 10 client.15561 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.521+0000 7f167fbf7740 1 -- 172.21.15.62:0/3185986504 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55586c789f80 con 0x55586c76c5d0 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f1656ffd640 1 -- 172.21.15.62:0/3185986504 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1670016040 con 0x55586c76c5d0 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f1656ffd640 1 -- 172.21.15.62:0/3185986504 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f1668018070 con 0x55586c76c5d0 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f167fbf7740 10 client.15561 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f167fbf7740 10 client.15561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f167fbf7740 20 client.15561 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55586c78b490 0x55586c78d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f167fbf7740 4 client.15561 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:38.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f167fbf7740 1 -- 172.21.15.62:0/3185986504 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55586c6df860 con 0x55586c78b490 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f166ffff640 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55586c78b490 0x55586c78d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f1656ffd640 10 client.15561 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.522+0000 7f166ffff640 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55586c78b490 0x55586c78d8e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1b70c5eaa414602 server_cookie=33107cca32a02811 in_seq=0 out_seq=0 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f1656ffd640 1 -- 172.21.15.62:0/3185986504 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55586c6df860 con 0x55586c78b490 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f1656ffd640 10 client.15561 handle_command_reply: tid=0 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f167fbf7740 1 client.15561 shutdown 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f167fbf7740 1 -- 172.21.15.62:0/3185986504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76c5d0 msgr2=0x55586c77b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76c5d0 0x55586c77b180 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f166800ce00 tx=0x7f16680039c0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f167fbf7740 1 -- 172.21.15.62:0/3185986504 shutdown_connections 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55586c78b490 0x55586c78d8e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 0x55586c778a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76db40 0x55586c778430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f167fbf7740 1 --2- 172.21.15.62:0/3185986504 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76c5d0 0x55586c77b180 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f1656ffd640 0 client.15561 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:38.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.523+0000 7f167fbf7740 1 -- 172.21.15.62:0/3185986504 >> 172.21.15.62:0/3185986504 conn(0x55586c735310 msgr2=0x55586c76e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:38.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.524+0000 7f167fbf7740 1 -- 172.21.15.62:0/3185986504 shutdown_connections 2024-09-15T23:16:38.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.524+0000 7f167fbf7740 1 -- 172.21.15.62:0/3185986504 wait complete. 2024-09-15T23:16:38.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.524+0000 7f167fbf7740 20 client.15561 trim_cache size 0 max 16384 2024-09-15T23:16:38.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f167fbf7740 1 Processor -- start 2024-09-15T23:16:38.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f167fbf7740 1 -- start start 2024-09-15T23:16:38.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 0x55586c787720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c7931b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 0x55586c7936f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f167fbf7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f16781af250 con 0x55586c76db40 2024-09-15T23:16:38.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f167fbf7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f16781acb20 con 0x55586c76c5d0 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f167fbf7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f16781aa970 con 0x55586c77b780 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f166ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c7931b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f166ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c7931b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60140/0 (socket says 172.21.15.62:60140) 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f166ffff640 1 -- 172.21.15.62:0/2087207215 learned_addr learned my addr 172.21.15.62:0/2087207215 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f1676ffd640 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 0x55586c787720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f16777fe640 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 0x55586c7936f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f166ffff640 1 -- 172.21.15.62:0/2087207215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 msgr2=0x55586c7936f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f166ffff640 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 0x55586c7936f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f166ffff640 1 -- 172.21.15.62:0/2087207215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 msgr2=0x55586c787720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f166ffff640 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 0x55586c787720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f166ffff640 1 -- 172.21.15.62:0/2087207215 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55586c77e5f0 con 0x55586c76db40 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f16777fe640 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 0x55586c7936f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.537+0000 7f1676ffd640 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 0x55586c787720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:38.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.538+0000 7f166ffff640 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c7931b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1660003fd0 tx=0x7f166002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4dd31a7588f944 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.538+0000 7f166d7fa640 1 -- 172.21.15.62:0/2087207215 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1660049040 con 0x55586c76db40 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.538+0000 7f166d7fa640 1 -- 172.21.15.62:0/2087207215 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16600bc8b0 con 0x55586c76db40 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.538+0000 7f166d7fa640 1 -- 172.21.15.62:0/2087207215 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16600b8f20 con 0x55586c76db40 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.538+0000 7f167fbf7740 1 -- 172.21.15.62:0/2087207215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 msgr2=0x55586c7931b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.538+0000 7f167fbf7740 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c7931b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1660003fd0 tx=0x7f166002a480 comp rx=0 tx=0).stop 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 -- 172.21.15.62:0/2087207215 shutdown_connections 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c77b780 0x55586c7936f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c7931b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 --2- 172.21.15.62:0/2087207215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c76c5d0 0x55586c787720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 -- 172.21.15.62:0/2087207215 >> 172.21.15.62:0/2087207215 conn(0x55586c735310 msgr2=0x55586c773720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 -- 172.21.15.62:0/2087207215 shutdown_connections 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 -- 172.21.15.62:0/2087207215 wait complete. 2024-09-15T23:16:38.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 -- start start 2024-09-15T23:16:38.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 Processor -- start 2024-09-15T23:16:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76c5d0 0x55586c783a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c783fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 0x55586c780860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f16781aafe0 con 0x55586c76db40 2024-09-15T23:16:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f16781ac450 con 0x55586c77b780 2024-09-15T23:16:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.539+0000 7f167fbf7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f16781ab6b0 con 0x55586c76c5d0 2024-09-15T23:16:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f167fbf7740 10 client.0 _ll_register_callbacks cb 0x7f1668004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:38.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f166ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c783fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f166ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c783fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60150/0 (socket says 172.21.15.62:60150) 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f166ffff640 1 -- 172.21.15.62:0/1956480059 learned_addr learned my addr 172.21.15.62:0/1956480059 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f16777fe640 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 0x55586c780860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f1676ffd640 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76c5d0 0x55586c783a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f16777fe640 1 -- 172.21.15.62:0/1956480059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76c5d0 msgr2=0x55586c783a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f16777fe640 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76c5d0 0x55586c783a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f16777fe640 1 -- 172.21.15.62:0/1956480059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 msgr2=0x55586c783fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f16777fe640 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c783fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f16777fe640 1 -- 172.21.15.62:0/1956480059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55586c787c60 con 0x55586c77b780 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f166ffff640 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c783fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f16777fe640 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 0x55586c780860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1670007b80 tx=0x7f167000e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec078113913c0e61 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f1656ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:16:38.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f1656ffd640 1 -- 172.21.15.62:0/1956480059 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670015050 con 0x55586c77b780 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.540+0000 7f167fbf7740 1 -- 172.21.15.62:0/1956480059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16600bc8b0 con 0x55586c77b780 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f1656ffd640 1 -- 172.21.15.62:0/1956480059 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f167000aa30 con 0x55586c77b780 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f1656ffd640 1 -- 172.21.15.62:0/1956480059 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f167001e040 con 0x55586c77b780 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f167fbf7740 10 client.25238 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f167fbf7740 1 -- 172.21.15.62:0/1956480059 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55586c769960 con 0x55586c77b780 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f1656ffd640 1 -- 172.21.15.62:0/1956480059 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16600bc8b0 con 0x55586c77b780 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f1656ffd640 1 -- 172.21.15.62:0/1956480059 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f1670035030 con 0x55586c77b780 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f167fbf7740 10 client.25238 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f167fbf7740 10 client.25238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f167fbf7740 20 client.25238 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f167fbf7740 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55586c7f6b80 0x55586c7f8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f167fbf7740 4 client.25238 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:16:38.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f167fbf7740 1 -- 172.21.15.62:0/1956480059 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55586c6d9d70 con 0x55586c7f6b80 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.541+0000 7f1676ffd640 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55586c7f6b80 0x55586c7f8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f1676ffd640 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55586c7f6b80 0x55586c7f8fa0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=822b61c1d392977f server_cookie=5884d0300cee1f45 in_seq=0 out_seq=0 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f1656ffd640 10 client.25238 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f1656ffd640 1 -- 172.21.15.62:0/1956480059 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55586c6d9d70 con 0x55586c7f6b80 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f1656ffd640 10 client.25238 handle_command_reply: tid=0 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 client.25238 shutdown 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 -- 172.21.15.62:0/1956480059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 msgr2=0x55586c780860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 0x55586c780860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1670007b80 tx=0x7f167000e070 comp rx=0 tx=0).stop 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 -- 172.21.15.62:0/1956480059 shutdown_connections 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55586c7f6b80 0x55586c7f8fa0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55586c77b780 0x55586c780860 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f1656ffd640 0 client.25238 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:38.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55586c76db40 0x55586c783fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 --2- 172.21.15.62:0/1956480059 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55586c76c5d0 0x55586c783a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 -- 172.21.15.62:0/1956480059 >> 172.21.15.62:0/1956480059 conn(0x55586c735310 msgr2=0x55586c767500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:38.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 -- 172.21.15.62:0/1956480059 shutdown_connections 2024-09-15T23:16:38.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 1 -- 172.21.15.62:0/1956480059 wait complete. 2024-09-15T23:16:38.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.542+0000 7f167fbf7740 20 client.25238 trim_cache size 0 max 16384 2024-09-15T23:16:38.550 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:16:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 -- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16640a32c0 msgr2=0x7f16640a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16640a32c0 0x7f16640a5780 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f167001f000 tx=0x7f167001b000 comp rx=0 tx=0).stop 2024-09-15T23:16:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 -- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 msgr2=0x7f1678199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 0x7f1678199830 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1660006010 tx=0x7f1660003c80 comp rx=0 tx=0).stop 2024-09-15T23:16:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 -- 172.21.15.62:0/1115672581 shutdown_connections 2024-09-15T23:16:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f16640a32c0 0x7f16640a5780 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1678102ef0 0x7f16781a4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1678102570 0x7f1678199830 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 --2- 172.21.15.62:0/1115672581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1678100e20 0x7f16781992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:38.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 -- 172.21.15.62:0/1115672581 >> 172.21.15.62:0/1115672581 conn(0x7f16780fc7d0 msgr2=0x7f16781068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:38.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 -- 172.21.15.62:0/1115672581 shutdown_connections 2024-09-15T23:16:38.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:38.543+0000 7f167d81e640 1 -- 172.21.15.62:0/1115672581 wait complete. 2024-09-15T23:16:39.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:38 smithi062 ceph-mon[30949]: pgmap v381: 129 pgs: 129 active+clean; 937 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 8.7 MiB/s wr, 611 op/s 2024-09-15T23:16:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:38 smithi067 ceph-mon[35665]: pgmap v381: 129 pgs: 129 active+clean; 937 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 8.7 MiB/s wr, 611 op/s 2024-09-15T23:16:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:38 smithi148 ceph-mon[35610]: pgmap v381: 129 pgs: 129 active+clean; 937 MiB data, 999 MiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 8.7 MiB/s wr, 611 op/s 2024-09-15T23:16:39.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 20.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:16:41.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:40 smithi062 ceph-mon[30949]: pgmap v382: 129 pgs: 129 active+clean; 938 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 13 MiB/s wr, 618 op/s 2024-09-15T23:16:41.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:40 smithi067 ceph-mon[35665]: pgmap v382: 129 pgs: 129 active+clean; 938 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 13 MiB/s wr, 618 op/s 2024-09-15T23:16:41.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:40 smithi148 ceph-mon[35610]: pgmap v382: 129 pgs: 129 active+clean; 938 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 13 MiB/s wr, 618 op/s 2024-09-15T23:16:42.455 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:16:42.455 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:16:42.731 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:43.138 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:42 smithi062 ceph-mon[30949]: pgmap v383: 129 pgs: 129 active+clean; 941 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 17 MiB/s wr, 651 op/s 2024-09-15T23:16:43.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.225+0000 7fd89bd68640 1 -- 172.21.15.62:0/1349166341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894102790 msgr2=0x7fd8941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.225+0000 7fd89bd68640 1 --2- 172.21.15.62:0/1349166341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894102790 0x7fd8941067e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd888009900 tx=0x7fd88802a410 comp rx=0 tx=0).stop 2024-09-15T23:16:43.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.225+0000 7fd89bd68640 1 -- 172.21.15.62:0/1349166341 shutdown_connections 2024-09-15T23:16:43.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.225+0000 7fd89bd68640 1 --2- 172.21.15.62:0/1349166341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd894106f10 0x7fd89410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.225+0000 7fd89bd68640 1 --2- 172.21.15.62:0/1349166341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894102790 0x7fd8941067e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.225+0000 7fd89bd68640 1 --2- 172.21.15.62:0/1349166341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd894101de0 0x7fd8941021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.225+0000 7fd89bd68640 1 -- 172.21.15.62:0/1349166341 >> 172.21.15.62:0/1349166341 conn(0x7fd8940fd8c0 msgr2=0x7fd8940ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:43.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.225+0000 7fd89bd68640 1 -- 172.21.15.62:0/1349166341 shutdown_connections 2024-09-15T23:16:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.225+0000 7fd89bd68640 1 -- 172.21.15.62:0/1349166341 wait complete. 2024-09-15T23:16:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd89bd68640 1 Processor -- start 2024-09-15T23:16:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd89bd68640 1 -- start start 2024-09-15T23:16:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd89bd68640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894101de0 0x7fd894198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd89bd68640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd894102790 0x7fd8941989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd89bd68640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd894106f10 0x7fd89419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd89bd68640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd894111f70 con 0x7fd894101de0 2024-09-15T23:16:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd89bd68640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd894111df0 con 0x7fd894106f10 2024-09-15T23:16:43.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd89bd68640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd8941120f0 con 0x7fd894102790 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd899b04640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894101de0 0x7fd894198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd899b04640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894101de0 0x7fd894198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48390/0 (socket says 172.21.15.62:48390) 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd899b04640 1 -- 172.21.15.62:0/3803810768 learned_addr learned my addr 172.21.15.62:0/3803810768 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd89a305640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd894106f10 0x7fd89419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.226+0000 7fd899303640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd894102790 0x7fd8941989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd899b04640 1 -- 172.21.15.62:0/3803810768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd894102790 msgr2=0x7fd8941989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd899b04640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd894102790 0x7fd8941989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd899b04640 1 -- 172.21.15.62:0/3803810768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd894106f10 msgr2=0x7fd89419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd899b04640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd894106f10 0x7fd89419cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd899b04640 1 -- 172.21.15.62:0/3803810768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89419d420 con 0x7fd894101de0 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd899b04640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894101de0 0x7fd894198470 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00d950 tx=0x7fd88c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd892ffd640 1 -- 172.21.15.62:0/3803810768 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c017070 con 0x7fd894101de0 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd892ffd640 1 -- 172.21.15.62:0/3803810768 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd88c009c30 con 0x7fd894101de0 2024-09-15T23:16:43.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd89bd68640 1 -- 172.21.15.62:0/3803810768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd89419d710 con 0x7fd894101de0 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd892ffd640 1 -- 172.21.15.62:0/3803810768 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88c00f6b0 con 0x7fd894101de0 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.227+0000 7fd89bd68640 1 -- 172.21.15.62:0/3803810768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8941a4ff0 con 0x7fd894101de0 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.229+0000 7fd892ffd640 1 -- 172.21.15.62:0/3803810768 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd88c0109d0 con 0x7fd894101de0 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd89e141740 1 Processor -- start 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd89e141740 1 -- start start 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd89e141740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 0x558af6506f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd89e141740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6507480 0x558af6511800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd89e141740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6514130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd89e141740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558af646c280 con 0x558af6511d40 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd89e141740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558af6413b10 con 0x558af6507480 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd89e141740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558af6468240 con 0x558af6506b60 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd899303640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6514130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd892ffd640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd87c0a32c0 0x7fd87c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd899303640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6514130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48404/0 (socket says 172.21.15.62:48404) 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd899303640 1 -- 172.21.15.62:0/2581117385 learned_addr learned my addr 172.21.15.62:0/2581117385 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.230+0000 7fd899b04640 1 --2- 172.21.15.62:0/2581117385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6507480 0x558af6511800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd89a305640 1 --2- 172.21.15.62:0/2581117385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 0x558af6506f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd89a305640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd87c0a32c0 0x7fd87c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd892ffd640 1 -- 172.21.15.62:0/3803810768 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd88c0cf770 con 0x7fd894101de0 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd899303640 1 -- 172.21.15.62:0/2581117385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 msgr2=0x558af6506f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd899303640 1 --2- 172.21.15.62:0/2581117385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 0x558af6506f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd899303640 1 -- 172.21.15.62:0/2581117385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6507480 msgr2=0x558af6511800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd899303640 1 --2- 172.21.15.62:0/2581117385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6507480 0x558af6511800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd899303640 1 -- 172.21.15.62:0/2581117385 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558af64201f0 con 0x558af6511d40 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd899303640 1 --2- 172.21.15.62:0/2581117385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6514130 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd888009900 tx=0x7fd888005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9783f4ba52a0200 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd89a305640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd87c0a32c0 0x7fd87c0a5780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd884009fd0 tx=0x7fd884009600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:43.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd87bfff640 1 -- 172.21.15.62:0/2581117385 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888040070 con 0x558af6511d40 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd87bfff640 1 -- 172.21.15.62:0/2581117385 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8880319f0 con 0x558af6511d40 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.231+0000 7fd87bfff640 1 -- 172.21.15.62:0/2581117385 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888035040 con 0x558af6511d40 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 -- 172.21.15.62:0/2581117385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 msgr2=0x558af6514130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 --2- 172.21.15.62:0/2581117385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6514130 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd888009900 tx=0x7fd888005e80 comp rx=0 tx=0).stop 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 -- 172.21.15.62:0/2581117385 shutdown_connections 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 --2- 172.21.15.62:0/2581117385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6514130 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 --2- 172.21.15.62:0/2581117385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6507480 0x558af6511800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 --2- 172.21.15.62:0/2581117385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 0x558af6506f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 -- 172.21.15.62:0/2581117385 >> 172.21.15.62:0/2581117385 conn(0x558af65030f0 msgr2=0x558af65034d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 -- 172.21.15.62:0/2581117385 shutdown_connections 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 -- 172.21.15.62:0/2581117385 wait complete. 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.232+0000 7fd89e141740 1 -- start start 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89e141740 1 Processor -- start 2024-09-15T23:16:43.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89e141740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af650eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89e141740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 0x558af650f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89e141740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af650f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89e141740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558af6514910 con 0x558af6511d40 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89e141740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558af6426f10 con 0x558af6506b60 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89e141740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558af6514a90 con 0x558af6507480 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89e141740 10 client.0 _ll_register_callbacks cb 0x558af63efa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89a305640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af650f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89a305640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af650f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48416/0 (socket says 172.21.15.62:48416) 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89a305640 1 -- 172.21.15.62:0/534428702 learned_addr learned my addr 172.21.15.62:0/534428702 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd899b04640 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af650eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd899303640 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 0x558af650f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89a305640 1 -- 172.21.15.62:0/534428702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 msgr2=0x558af650f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:43.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89a305640 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 0x558af650f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89a305640 1 -- 172.21.15.62:0/534428702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 msgr2=0x558af650eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.233+0000 7fd89a305640 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af650eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd89a305640 1 -- 172.21.15.62:0/534428702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558af61dd6a0 con 0x558af6511d40 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd899b04640 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af650eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd89a305640 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af650f620 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd88400ab90 tx=0x7fd884004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c25dc0382c969 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd8797fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd8797fa640 1 -- 172.21.15.62:0/534428702 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88401ebb0 con 0x558af6511d40 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd8797fa640 1 -- 172.21.15.62:0/534428702 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd88401ed50 con 0x558af6511d40 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd89e141740 1 -- 172.21.15.62:0/534428702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8880319f0 con 0x558af6511d40 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd8797fa640 1 -- 172.21.15.62:0/534428702 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd884033050 con 0x558af6511d40 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd89e141740 10 client.15579 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd89e141740 1 -- 172.21.15.62:0/534428702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558af6520520 con 0x558af6511d40 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd8797fa640 1 -- 172.21.15.62:0/534428702 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8880319f0 con 0x558af6511d40 2024-09-15T23:16:43.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd8797fa640 1 -- 172.21.15.62:0/534428702 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fd88400f040 con 0x558af6511d40 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd89e141740 10 client.15579 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.234+0000 7fd89e141740 10 client.15579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.235+0000 7fd89e141740 20 client.15579 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.235+0000 7fd89e141740 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558af6521a30 0x558af6523ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.235+0000 7fd89e141740 4 client.15579 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.235+0000 7fd89e141740 1 -- 172.21.15.62:0/534428702 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558af64797f0 con 0x558af6521a30 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.235+0000 7fd899b04640 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558af6521a30 0x558af6523ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.235+0000 7fd899b04640 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558af6521a30 0x558af6523ea0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18f520df26ae3a5c server_cookie=9b16b3ed72a4f5c7 in_seq=0 out_seq=0 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.235+0000 7fd8797fa640 10 client.15579 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.236+0000 7fd8797fa640 1 -- 172.21.15.62:0/534428702 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558af64797f0 con 0x558af6521a30 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.236+0000 7fd8797fa640 10 client.15579 handle_command_reply: tid=0 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.236+0000 7fd89e141740 1 client.15579 shutdown 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.236+0000 7fd89e141740 1 -- 172.21.15.62:0/534428702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 msgr2=0x558af650f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.236+0000 7fd89e141740 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af650f620 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd88400ab90 tx=0x7fd884004a40 comp rx=0 tx=0).stop 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd89e141740 1 -- 172.21.15.62:0/534428702 shutdown_connections 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd89e141740 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558af6521a30 0x558af6523ea0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd89e141740 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af650f620 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd89e141740 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 0x558af650f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd8797fa640 0 client.15579 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd89e141740 1 --2- 172.21.15.62:0/534428702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af650eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd89e141740 1 -- 172.21.15.62:0/534428702 >> 172.21.15.62:0/534428702 conn(0x558af65030f0 msgr2=0x558af6508dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd89e141740 1 -- 172.21.15.62:0/534428702 shutdown_connections 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd89e141740 1 -- 172.21.15.62:0/534428702 wait complete. 2024-09-15T23:16:43.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.237+0000 7fd89e141740 20 client.15579 trim_cache size 0 max 16384 2024-09-15T23:16:43.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.254+0000 7fd89e141740 1 Processor -- start 2024-09-15T23:16:43.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.254+0000 7fd89e141740 1 -- start start 2024-09-15T23:16:43.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd89e141740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 0x558af6508cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd89e141740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6507480 0x558af6529770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd89e141740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6511d40 0x558af6529cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd89e141740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd8941b1490 con 0x558af6507480 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd89e141740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd8941aed60 con 0x558af6511d40 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd89e141740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd8941acbb0 con 0x558af6506b60 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899303640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6507480 0x558af6529770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899303640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6507480 0x558af6529770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48420/0 (socket says 172.21.15.62:48420) 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899303640 1 -- 172.21.15.62:0/2432135536 learned_addr learned my addr 172.21.15.62:0/2432135536 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899b04640 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 0x558af6508cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd89a305640 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6511d40 0x558af6529cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899303640 1 -- 172.21.15.62:0/2432135536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 msgr2=0x558af6508cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899303640 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 0x558af6508cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899303640 1 -- 172.21.15.62:0/2432135536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6511d40 msgr2=0x558af6529cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899303640 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6511d40 0x558af6529cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899303640 1 -- 172.21.15.62:0/2432135536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558af65072b0 con 0x558af6507480 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd89a305640 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6511d40 0x558af6529cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd899303640 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6507480 0x558af6529770 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd888009900 tx=0x7fd888031500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74c8d758886bd43 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.255+0000 7fd87b7fe640 1 -- 172.21.15.62:0/2432135536 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888040040 con 0x558af6507480 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd87b7fe640 1 -- 172.21.15.62:0/2432135536 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8880035f0 con 0x558af6507480 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd87b7fe640 1 -- 172.21.15.62:0/2432135536 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd888035040 con 0x558af6507480 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd89e141740 1 -- 172.21.15.62:0/2432135536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6507480 msgr2=0x558af6529770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd89e141740 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6507480 0x558af6529770 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd888009900 tx=0x7fd888031500 comp rx=0 tx=0).stop 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd89e141740 1 -- 172.21.15.62:0/2432135536 shutdown_connections 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd89e141740 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6511d40 0x558af6529cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd89e141740 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6507480 0x558af6529770 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd89e141740 1 --2- 172.21.15.62:0/2432135536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6506b60 0x558af6508cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd89e141740 1 -- 172.21.15.62:0/2432135536 >> 172.21.15.62:0/2432135536 conn(0x558af65030f0 msgr2=0x558af65085b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd89e141740 1 -- 172.21.15.62:0/2432135536 shutdown_connections 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.256+0000 7fd89e141740 1 -- 172.21.15.62:0/2432135536 wait complete. 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89e141740 1 -- start start 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89e141740 1 Processor -- start 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89e141740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af6515a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89e141740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 0x558af6515f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89e141740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6516480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89e141740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd8941ad220 con 0x558af6511d40 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89e141740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd8941ae690 con 0x558af6506b60 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89e141740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd8941ad8f0 con 0x558af6507480 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89a305640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6516480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89e141740 10 client.0 _ll_register_callbacks cb 0x7fd884012760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:43.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89a305640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6516480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48434/0 (socket says 172.21.15.62:48434) 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd89a305640 1 -- 172.21.15.62:0/5174667 learned_addr learned my addr 172.21.15.62:0/5174667 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.257+0000 7fd899b04640 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af6515a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd899303640 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 0x558af6515f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd89a305640 1 -- 172.21.15.62:0/5174667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 msgr2=0x558af6515f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd89a305640 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 0x558af6515f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd89a305640 1 -- 172.21.15.62:0/5174667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 msgr2=0x558af6515a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd89a305640 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af6515a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd89a305640 1 -- 172.21.15.62:0/5174667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558af658b740 con 0x558af6511d40 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd899b04640 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af6515a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd89a305640 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6516480 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd884012810 tx=0x7fd884009c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fdc9e781adddaab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd8797fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd8797fa640 1 -- 172.21.15.62:0/5174667 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88403c060 con 0x558af6511d40 2024-09-15T23:16:43.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd8797fa640 1 -- 172.21.15.62:0/5174667 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd8840120b0 con 0x558af6511d40 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd89e141740 1 -- 172.21.15.62:0/5174667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8880035f0 con 0x558af6511d40 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd8797fa640 1 -- 172.21.15.62:0/5174667 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd884043050 con 0x558af6511d40 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd89e141740 10 client.15585 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd89e141740 1 -- 172.21.15.62:0/5174667 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558af658ca00 con 0x558af6511d40 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.258+0000 7fd8797fa640 1 -- 172.21.15.62:0/5174667 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8880035f0 con 0x558af6511d40 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd8797fa640 1 -- 172.21.15.62:0/5174667 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fd884011950 con 0x558af6511d40 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd89e141740 10 client.15585 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd89e141740 10 client.15585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd89e141740 20 client.15585 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd89e141740 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558af658d150 0x558af658f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd89e141740 4 client.15585 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd89e141740 1 -- 172.21.15.62:0/5174667 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558af6473d00 con 0x558af658d150 2024-09-15T23:16:43.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd899b04640 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558af658d150 0x558af658f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:43.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd899b04640 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558af658d150 0x558af658f590 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cdb06af8bbc6e8a server_cookie=83e9f3ff597e5790 in_seq=0 out_seq=0 2024-09-15T23:16:43.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.259+0000 7fd8797fa640 10 client.15585 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.260+0000 7fd8797fa640 1 -- 172.21.15.62:0/5174667 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558af6473d00 con 0x558af658d150 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.260+0000 7fd8797fa640 10 client.15585 handle_command_reply: tid=0 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.260+0000 7fd89e141740 1 client.15585 shutdown 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.260+0000 7fd89e141740 1 -- 172.21.15.62:0/5174667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 msgr2=0x558af6516480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.260+0000 7fd89e141740 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6516480 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd884012810 tx=0x7fd884009c40 comp rx=0 tx=0).stop 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd89e141740 1 -- 172.21.15.62:0/5174667 shutdown_connections 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd89e141740 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558af658d150 0x558af658f590 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd89e141740 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558af6511d40 0x558af6516480 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd89e141740 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558af6507480 0x558af6515f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd89e141740 1 --2- 172.21.15.62:0/5174667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558af6506b60 0x558af6515a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd8797fa640 0 client.15585 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd89e141740 1 -- 172.21.15.62:0/5174667 >> 172.21.15.62:0/5174667 conn(0x558af65030f0 msgr2=0x558af650f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:43.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd89e141740 1 -- 172.21.15.62:0/5174667 shutdown_connections 2024-09-15T23:16:43.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd89e141740 1 -- 172.21.15.62:0/5174667 wait complete. 2024-09-15T23:16:43.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.261+0000 7fd89e141740 20 client.15585 trim_cache size 0 max 16384 2024-09-15T23:16:43.268 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:16:43.268 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:16:43.268 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "37b08474-b25b-4afa-b111-83d888471b58", 2024-09-15T23:16:43.268 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:16:43.269 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:16:43.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 -- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd87c0a32c0 msgr2=0x7fd87c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd87c0a32c0 0x7fd87c0a5780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd884009fd0 tx=0x7fd884009600 comp rx=0 tx=0).stop 2024-09-15T23:16:43.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 -- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894101de0 msgr2=0x7fd894198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:43.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894101de0 0x7fd894198470 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00d950 tx=0x7fd88c00de10 comp rx=0 tx=0).stop 2024-09-15T23:16:43.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 -- 172.21.15.62:0/3803810768 shutdown_connections 2024-09-15T23:16:43.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd87c0a32c0 0x7fd87c0a5780 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd894106f10 0x7fd89419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd894102790 0x7fd8941989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 --2- 172.21.15.62:0/3803810768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd894101de0 0x7fd894198470 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:43.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 -- 172.21.15.62:0/3803810768 >> 172.21.15.62:0/3803810768 conn(0x7fd8940fd8c0 msgr2=0x7fd8940fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:43.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 -- 172.21.15.62:0/3803810768 shutdown_connections 2024-09-15T23:16:43.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:43.262+0000 7fd89bd68640 1 -- 172.21.15.62:0/3803810768 wait complete. 2024-09-15T23:16:43.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:42 smithi067 ceph-mon[35665]: pgmap v383: 129 pgs: 129 active+clean; 941 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 17 MiB/s wr, 651 op/s 2024-09-15T23:16:43.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:42 smithi148 ceph-mon[35610]: pgmap v383: 129 pgs: 129 active+clean; 941 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 17 MiB/s wr, 651 op/s 2024-09-15T23:16:44.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:16:44.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:43 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:16:44.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:43 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:16:44.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:43 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:16:44.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:43 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:16:44.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:43 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:16:44.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:43 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:16:44.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:43 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:16:44.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:43 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:16:44.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:43 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:16:44.295 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:44.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 25.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.795+0000 7f8c8cd02640 1 -- 172.21.15.62:0/3297051154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c881000d0 msgr2=0x7f8c88100530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.795+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/3297051154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c881000d0 0x7f8c88100530 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009900 tx=0x7f8c7002a410 comp rx=0 tx=0).stop 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.795+0000 7f8c8cd02640 1 -- 172.21.15.62:0/3297051154 shutdown_connections 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.795+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/3297051154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c88100a70 0x7f8c8810ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.795+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/3297051154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c881000d0 0x7f8c88100530 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.795+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/3297051154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c88106ba0 0x7f8c88106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.795+0000 7f8c8cd02640 1 -- 172.21.15.62:0/3297051154 >> 172.21.15.62:0/3297051154 conn(0x7f8c88077210 msgr2=0x7f8c880fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.795+0000 7f8c8cd02640 1 -- 172.21.15.62:0/3297051154 shutdown_connections 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.795+0000 7f8c8cd02640 1 -- 172.21.15.62:0/3297051154 wait complete. 2024-09-15T23:16:44.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.796+0000 7f8c8cd02640 1 Processor -- start 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.796+0000 7f8c8cd02640 1 -- start start 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c8cd02640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c881000d0 0x7f8c881984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c8cd02640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c88100a70 0x7f8c881989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c8cd02640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c88106ba0 0x7f8c8819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c8cd02640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8c88112050 con 0x7f8c88100a70 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c8cd02640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c88111ed0 con 0x7f8c88106ba0 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c8cd02640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8c881121d0 con 0x7f8c881000d0 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c85d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c88100a70 0x7f8c881989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c85d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c88100a70 0x7f8c881989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48440/0 (socket says 172.21.15.62:48440) 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c85d9b640 1 -- 172.21.15.62:0/4199668185 learned_addr learned my addr 172.21.15.62:0/4199668185 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c8659c640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c881000d0 0x7f8c881984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c86d9d640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c88106ba0 0x7f8c8819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c85d9b640 1 -- 172.21.15.62:0/4199668185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c881000d0 msgr2=0x7f8c881984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c881000d0 0x7f8c881984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c85d9b640 1 -- 172.21.15.62:0/4199668185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c88106ba0 msgr2=0x7f8c8819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c88106ba0 0x7f8c8819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c85d9b640 1 -- 172.21.15.62:0/4199668185 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8819d460 con 0x7f8c88100a70 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c88100a70 0x7f8c881989f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8c70006010 tx=0x7f8c70003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.797+0000 7f8c7f7fe640 1 -- 172.21.15.62:0/4199668185 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7003f070 con 0x7f8c88100a70 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.798+0000 7f8c8cd02640 1 -- 172.21.15.62:0/4199668185 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c8819d6f0 con 0x7f8c88100a70 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.798+0000 7f8c7f7fe640 1 -- 172.21.15.62:0/4199668185 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c70031c30 con 0x7f8c88100a70 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.798+0000 7f8c7f7fe640 1 -- 172.21.15.62:0/4199668185 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70035030 con 0x7f8c88100a70 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.798+0000 7f8c8cd02640 1 -- 172.21.15.62:0/4199668185 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c881a5030 con 0x7f8c88100a70 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c7f7fe640 1 -- 172.21.15.62:0/4199668185 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8c70042070 con 0x7f8c88100a70 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c8f0db740 1 Processor -- start 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c8f0db740 1 -- start start 2024-09-15T23:16:44.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700702760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e700711550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 0x55e700713e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e7006682e0 con 0x55e700702380 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e70060fce0 con 0x55e700702ca0 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e7006642a0 con 0x55e700711a90 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c86d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700702760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c86d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700702760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48450/0 (socket says 172.21.15.62:48450) 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c86d9d640 1 -- 172.21.15.62:0/2877869736 learned_addr learned my addr 172.21.15.62:0/2877869736 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c8659c640 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e700711550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.801+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 0x55e700713e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c86d9d640 1 -- 172.21.15.62:0/2877869736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 msgr2=0x55e700713e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c86d9d640 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 0x55e700713e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c86d9d640 1 -- 172.21.15.62:0/2877869736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 msgr2=0x55e700711550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c86d9d640 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e700711550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c86d9d640 1 -- 172.21.15.62:0/2877869736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7005f4440 con 0x55e700702380 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c8659c640 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e700711550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c7f7fe640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c740a3240 0x7f8c740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c86d9d640 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700702760 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009ee0 tx=0x7f8c80010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a3d3d5d4c4da82 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c8659c640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c740a3240 0x7f8c740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c7f7fe640 1 -- 172.21.15.62:0/4199668185 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8c700f05a0 con 0x7f8c88100a70 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c7cff9640 1 -- 172.21.15.62:0/2877869736 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c80016070 con 0x55e700702380 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.802+0000 7f8c7cff9640 1 -- 172.21.15.62:0/2877869736 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c80004d20 con 0x55e700702380 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c7cff9640 1 -- 172.21.15.62:0/2877869736 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c800127d0 con 0x55e700702380 2024-09-15T23:16:44.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 -- 172.21.15.62:0/2877869736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 msgr2=0x55e700702760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:44.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8659c640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c740a3240 0x7f8c740a5700 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8c780058c0 tx=0x7f8c78005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:44.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700702760 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8c80009ee0 tx=0x7f8c80010740 comp rx=0 tx=0).stop 2024-09-15T23:16:44.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 -- 172.21.15.62:0/2877869736 shutdown_connections 2024-09-15T23:16:44.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 0x55e700713e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e700711550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/2877869736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700702760 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 -- 172.21.15.62:0/2877869736 >> 172.21.15.62:0/2877869736 conn(0x55e7006fea30 msgr2=0x55e7006fee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:44.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 -- 172.21.15.62:0/2877869736 shutdown_connections 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 -- 172.21.15.62:0/2877869736 wait complete. 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 -- start start 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.803+0000 7f8c8f0db740 1 Processor -- start 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700711360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e70070e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 0x55e70070eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e7007146b0 con 0x55e700702380 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e700623050 con 0x55e700702ca0 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e700714830 con 0x55e700711a90 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8f0db740 10 client.0 _ll_register_callbacks cb 0x55e700663160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700711360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700711360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48460/0 (socket says 172.21.15.62:48460) 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8659c640 1 -- 172.21.15.62:0/3423957743 learned_addr learned my addr 172.21.15.62:0/3423957743 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e70070e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c86d9d640 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 0x55e70070eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8659c640 1 -- 172.21.15.62:0/3423957743 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 msgr2=0x55e70070eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8659c640 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 0x55e70070eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8659c640 1 -- 172.21.15.62:0/3423957743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 msgr2=0x55e70070e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8659c640 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e70070e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8659c640 1 -- 172.21.15.62:0/3423957743 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7003d93c0 con 0x55e700702380 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e70070e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c8659c640 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700711360 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8c78011e10 tx=0x7f8c780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586ec816c06afcf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.804+0000 7f8c6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3423957743 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7801e420 con 0x55e700702380 2024-09-15T23:16:44.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3423957743 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8c7801e5c0 con 0x55e700702380 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3423957743 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c80004d20 con 0x55e700702380 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3423957743 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8c7801e7c0 con 0x55e700702380 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c8f0db740 10 client.15597 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3423957743 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e7007200f0 con 0x55e700702380 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3423957743 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c80004d20 con 0x55e700702380 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3423957743 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f8c78026a20 con 0x55e700702380 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c8f0db740 10 client.15597 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c8f0db740 10 client.15597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c8f0db740 20 client.15597 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e7007215b0 0x55e700723a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c8f0db740 4 client.15597 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.805+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3423957743 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e700675850 con 0x55e7007215b0 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.806+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e7007215b0 0x55e700723a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.806+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e7007215b0 0x55e700723a00 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb4a2267730cabe7 server_cookie=82b7c5b9d34f65bf in_seq=0 out_seq=0 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.806+0000 7f8c6dffb640 10 client.15597 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3423957743 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e700675850 con 0x55e7007215b0 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c6dffb640 10 client.15597 handle_command_reply: tid=0 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 client.15597 shutdown 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3423957743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 msgr2=0x55e700711360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700711360 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8c78011e10 tx=0x7f8c780046c0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3423957743 shutdown_connections 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e7007215b0 0x55e700723a00 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700711a90 0x55e70070eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700702ca0 0x55e70070e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c6dffb640 0 client.15597 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3423957743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700711360 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3423957743 >> 172.21.15.62:0/3423957743 conn(0x55e7006fea30 msgr2=0x55e700704790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:44.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3423957743 shutdown_connections 2024-09-15T23:16:44.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3423957743 wait complete. 2024-09-15T23:16:44.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.807+0000 7f8c8f0db740 20 client.15597 trim_cache size 0 max 16384 2024-09-15T23:16:44.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.824+0000 7f8c8f0db740 1 Processor -- start 2024-09-15T23:16:44.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c8f0db740 1 -- start start 2024-09-15T23:16:44.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702380 0x55e70071d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702ca0 0x55e700709d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e7007045f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8c881af2a0 con 0x55e700702ca0 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c881acb70 con 0x55e700711a90 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8c881aa9c0 con 0x55e700702380 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c85d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702ca0 0x55e700709d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c85d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702ca0 0x55e700709d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48476/0 (socket says 172.21.15.62:48476) 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c85d9b640 1 -- 172.21.15.62:0/676194956 learned_addr learned my addr 172.21.15.62:0/676194956 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c86d9d640 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e7007045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c8659c640 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702380 0x55e70071d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c85d9b640 1 -- 172.21.15.62:0/676194956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702380 msgr2=0x55e70071d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:44.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702380 0x55e70071d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c85d9b640 1 -- 172.21.15.62:0/676194956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 msgr2=0x55e7007045f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e7007045f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c85d9b640 1 -- 172.21.15.62:0/676194956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7007141f0 con 0x55e700702ca0 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c86d9d640 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e7007045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.825+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702ca0 0x55e700709d10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8c700edd20 tx=0x7f8c700bc260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214e8852d07644cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c6ffff640 1 -- 172.21.15.62:0/676194956 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c70049040 con 0x55e700702ca0 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c6ffff640 1 -- 172.21.15.62:0/676194956 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c700bc840 con 0x55e700702ca0 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c6ffff640 1 -- 172.21.15.62:0/676194956 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c700babd0 con 0x55e700702ca0 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c8f0db740 1 -- 172.21.15.62:0/676194956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702ca0 msgr2=0x55e700709d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702ca0 0x55e700709d10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f8c700edd20 tx=0x7f8c700bc260 comp rx=0 tx=0).stop 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c8f0db740 1 -- 172.21.15.62:0/676194956 shutdown_connections 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e7007045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702ca0 0x55e700709d10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/676194956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702380 0x55e70071d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c8f0db740 1 -- 172.21.15.62:0/676194956 >> 172.21.15.62:0/676194956 conn(0x55e7006fea30 msgr2=0x55e700703ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:44.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c8f0db740 1 -- 172.21.15.62:0/676194956 shutdown_connections 2024-09-15T23:16:44.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.826+0000 7f8c8f0db740 1 -- 172.21.15.62:0/676194956 wait complete. 2024-09-15T23:16:44.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8f0db740 1 -- start start 2024-09-15T23:16:44.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8f0db740 1 Processor -- start 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700715980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702ca0 0x55e700715ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8f0db740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e700716400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8c881ab030 con 0x55e700702380 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8c881ac4a0 con 0x55e700711a90 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8f0db740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8c881ab700 con 0x55e700702ca0 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8f0db740 10 client.0 _ll_register_callbacks cb 0x7f8c7800c9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700715980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700715980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48486/0 (socket says 172.21.15.62:48486) 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8659c640 1 -- 172.21.15.62:0/3805699938 learned_addr learned my addr 172.21.15.62:0/3805699938 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:44.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c86d9d640 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e700716400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702ca0 0x55e700715ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8659c640 1 -- 172.21.15.62:0/3805699938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702ca0 msgr2=0x55e700715ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8659c640 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702ca0 0x55e700715ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8659c640 1 -- 172.21.15.62:0/3805699938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 msgr2=0x55e700716400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8659c640 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e700716400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c8659c640 1 -- 172.21.15.62:0/3805699938 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e70078b370 con 0x55e700702380 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.827+0000 7f8c86d9d640 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e700716400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c8659c640 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700715980 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8c78005940 tx=0x7f8c7801c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761ee27272cfce01 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3805699938 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c78038070 con 0x55e700702380 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3805699938 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c700bc840 con 0x55e700702380 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3805699938 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8c7801eb60 con 0x55e700702380 2024-09-15T23:16:44.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3805699938 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8c78046050 con 0x55e700702380 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c8f0db740 10 client.15603 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3805699938 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e70078c6c0 con 0x55e700702380 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3805699938 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c700bc840 con 0x55e700702380 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3805699938 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f8c7801c830 con 0x55e700702380 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c8f0db740 10 client.15603 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c8f0db740 10 client.15603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.828+0000 7f8c8f0db740 20 client.15603 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e70078ce40 0x55e70078f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c8f0db740 4 client.15603 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3805699938 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e70070eee0 con 0x55e70078ce40 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e70078ce40 0x55e70078f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c85d9b640 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e70078ce40 0x55e70078f260 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c04da4415198a4a server_cookie=a38200bfc9f8eac3 in_seq=0 out_seq=0 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c6dffb640 10 client.15603 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:44.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c6dffb640 1 -- 172.21.15.62:0/3805699938 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e70070eee0 con 0x55e70078ce40 2024-09-15T23:16:44.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c6dffb640 10 client.15603 handle_command_reply: tid=0 2024-09-15T23:16:44.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c8f0db740 1 client.15603 shutdown 2024-09-15T23:16:44.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3805699938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 msgr2=0x55e700715980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:44.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.829+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700715980 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8c78005940 tx=0x7f8c7801c730 comp rx=0 tx=0).stop 2024-09-15T23:16:44.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3805699938 shutdown_connections 2024-09-15T23:16:44.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e70078ce40 0x55e70078f260 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e700711a90 0x55e700716400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e700702ca0 0x55e700715ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8f0db740 1 --2- 172.21.15.62:0/3805699938 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e700702380 0x55e700715980 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3805699938 >> 172.21.15.62:0/3805699938 conn(0x55e7006fea30 msgr2=0x55e700709c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c6dffb640 0 client.15603 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3805699938 shutdown_connections 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8f0db740 1 -- 172.21.15.62:0/3805699938 wait complete. 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8f0db740 20 client.15603 trim_cache size 0 max 16384 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:16:44.837 INFO:teuthology.orchestra.run.smithi062.stdout: "37b08474-b25b-4afa-b111-83d888471b58": { 2024-09-15T23:16:44.838 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:16:44.838 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "37b08474-b25b-4afa-b111-83d888471b58", 2024-09-15T23:16:44.838 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:16:44.838 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:16:44.838 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:16:44.838 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 -- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c740a3240 msgr2=0x7f8c740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c740a3240 0x7f8c740a5700 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8c780058c0 tx=0x7f8c78005850 comp rx=0 tx=0).stop 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 -- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c88100a70 msgr2=0x7f8c881989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c88100a70 0x7f8c881989f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8c70006010 tx=0x7f8c70003c10 comp rx=0 tx=0).stop 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 -- 172.21.15.62:0/4199668185 shutdown_connections 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8c740a3240 0x7f8c740a5700 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c88106ba0 0x7f8c8819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c88100a70 0x7f8c881989f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 --2- 172.21.15.62:0/4199668185 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8c881000d0 0x7f8c881984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 -- 172.21.15.62:0/4199668185 >> 172.21.15.62:0/4199668185 conn(0x7f8c88077210 msgr2=0x7f8c88104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.830+0000 7f8c8cd02640 1 -- 172.21.15.62:0/4199668185 shutdown_connections 2024-09-15T23:16:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:44.831+0000 7f8c8cd02640 1 -- 172.21.15.62:0/4199668185 wait complete. 2024-09-15T23:16:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.189+0000 7febf564c640 1 -- 172.21.15.62:0/1174958118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 msgr2=0x7febf0109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.189+0000 7febf564c640 1 --2- 172.21.15.62:0/1174958118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 0x7febf0109c90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7febe8009ca0 tx=0x7febe802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.189+0000 7febf564c640 1 -- 172.21.15.62:0/1174958118 shutdown_connections 2024-09-15T23:16:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.189+0000 7febf564c640 1 --2- 172.21.15.62:0/1174958118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 0x7febf0109c90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.189+0000 7febf564c640 1 --2- 172.21.15.62:0/1174958118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf0104f70 0x7febf0107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.189+0000 7febf564c640 1 --2- 172.21.15.62:0/1174958118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febf0102640 0x7febf0104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.189+0000 7febf564c640 1 -- 172.21.15.62:0/1174958118 >> 172.21.15.62:0/1174958118 conn(0x7febf00fc1e0 msgr2=0x7febf00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.189+0000 7febf564c640 1 -- 172.21.15.62:0/1174958118 shutdown_connections 2024-09-15T23:16:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.189+0000 7febf564c640 1 -- 172.21.15.62:0/1174958118 wait complete. 2024-09-15T23:16:45.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febf564c640 1 Processor -- start 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febf564c640 1 -- start start 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febf564c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febf0102640 0x7febf0101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febf564c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf0104f70 0x7febf01005f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febf564c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 0x7febf0100b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febf564c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7febf010d7e0 con 0x7febf01078a0 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febf564c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7febf010d660 con 0x7febf0104f70 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febf564c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7febf010d960 con 0x7febf0102640 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 0x7febf0100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 0x7febf0100b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48500/0 (socket says 172.21.15.62:48500) 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.190+0000 7febef7fe640 1 -- 172.21.15.62:0/3733583638 learned_addr learned my addr 172.21.15.62:0/3733583638 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febee7fc640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf0104f70 0x7febf01005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febeeffd640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febf0102640 0x7febf0101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febef7fe640 1 -- 172.21.15.62:0/3733583638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febf0102640 msgr2=0x7febf0101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febef7fe640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febf0102640 0x7febf0101fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febef7fe640 1 -- 172.21.15.62:0/3733583638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf0104f70 msgr2=0x7febf01005f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febef7fe640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf0104f70 0x7febf01005f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febef7fe640 1 -- 172.21.15.62:0/3733583638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf01013c0 con 0x7febf01078a0 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febeeffd640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febf0102640 0x7febf0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febef7fe640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 0x7febf0100b30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7febe8031a40 tx=0x7febe8031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febe7fff640 1 -- 172.21.15.62:0/3733583638 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe8003c30 con 0x7febf01078a0 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febe7fff640 1 -- 172.21.15.62:0/3733583638 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe8003dd0 con 0x7febf01078a0 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febf564c640 1 -- 172.21.15.62:0/3733583638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf01a1e70 con 0x7febf01078a0 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febe7fff640 1 -- 172.21.15.62:0/3733583638 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe802fdb0 con 0x7febf01078a0 2024-09-15T23:16:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.191+0000 7febf564c640 1 -- 172.21.15.62:0/3733583638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febf01a22d0 con 0x7febf01078a0 2024-09-15T23:16:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.194+0000 7febe7fff640 1 -- 172.21.15.62:0/3733583638 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7febe8004050 con 0x7febf01078a0 2024-09-15T23:16:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.194+0000 7febf83e7740 1 Processor -- start 2024-09-15T23:16:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.194+0000 7febf83e7740 1 -- start start 2024-09-15T23:16:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febe7fff640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7febdc0a32f0 0x7febdc0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febf83e7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262a9a0 0x55c98262ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febf83e7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 0x55c98262fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febf83e7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982632830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febf83e7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c98258e2f0 con 0x55c982630370 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febf83e7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c982536270 con 0x55c98262b2e0 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febf83e7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c98258a2b0 con 0x55c98262a9a0 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febee7fc640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7febdc0a32f0 0x7febdc0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982632830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982632830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48512/0 (socket says 172.21.15.62:48512) 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febef7fe640 1 -- 172.21.15.62:0/1221551393 learned_addr learned my addr 172.21.15.62:0/1221551393 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febeeffd640 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262a9a0 0x55c98262ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febe7fff640 1 -- 172.21.15.62:0/3733583638 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7febe80f0e70 con 0x7febf01078a0 2024-09-15T23:16:45.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febee7fc640 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 0x55c98262fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febef7fe640 1 -- 172.21.15.62:0/1221551393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262a9a0 msgr2=0x55c98262ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febef7fe640 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262a9a0 0x55c98262ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febef7fe640 1 -- 172.21.15.62:0/1221551393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 msgr2=0x55c98262fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febef7fe640 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 0x55c98262fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febef7fe640 1 -- 172.21.15.62:0/1221551393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c982535340 con 0x55c982630370 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febee7fc640 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 0x55c98262fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febee7fc640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7febdc0a32f0 0x7febdc0a57b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7febd80058a0 tx=0x7febd80128d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.195+0000 7febef7fe640 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982632830 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7febe80b9570 tx=0x7febe802f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c177890817c1feb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febe57fa640 1 -- 172.21.15.62:0/1221551393 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe80bca70 con 0x55c982630370 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febe57fa640 1 -- 172.21.15.62:0/1221551393 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe8036040 con 0x55c982630370 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febe57fa640 1 -- 172.21.15.62:0/1221551393 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe80bb1b0 con 0x55c982630370 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febf83e7740 1 -- 172.21.15.62:0/1221551393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 msgr2=0x55c982632830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:45.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febf83e7740 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982632830 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7febe80b9570 tx=0x7febe802f870 comp rx=0 tx=0).stop 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febf83e7740 1 -- 172.21.15.62:0/1221551393 shutdown_connections 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febf83e7740 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982632830 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febf83e7740 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 0x55c98262fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febf83e7740 1 --2- 172.21.15.62:0/1221551393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262a9a0 0x55c98262ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.196+0000 7febf83e7740 1 -- 172.21.15.62:0/1221551393 >> 172.21.15.62:0/1221551393 conn(0x55c982626010 msgr2=0x55c982626410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 -- 172.21.15.62:0/1221551393 shutdown_connections 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 -- 172.21.15.62:0/1221551393 wait complete. 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 -- start start 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 Processor -- start 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c9826346b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c98262b2e0 0x55c982634bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c982630370 0x55c98263bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c982536270 con 0x55c98262b2e0 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c982517290 con 0x55c98262a9a0 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c98258e2f0 con 0x55c982630370 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febf83e7740 10 client.0 _ll_register_callbacks cb 0x55c982548140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febee7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c98262b2e0 0x55c982634bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febee7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c98262b2e0 0x55c982634bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48522/0 (socket says 172.21.15.62:48522) 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.197+0000 7febee7fc640 1 -- 172.21.15.62:0/2855086899 learned_addr learned my addr 172.21.15.62:0/2855086899 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febeeffd640 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c9826346b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febef7fe640 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c982630370 0x55c98263bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febee7fc640 1 -- 172.21.15.62:0/2855086899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c982630370 msgr2=0x55c98263bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febee7fc640 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c982630370 0x55c98263bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febee7fc640 1 -- 172.21.15.62:0/2855086899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 msgr2=0x55c9826346b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febee7fc640 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c9826346b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febee7fc640 1 -- 172.21.15.62:0/2855086899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9822a1210 con 0x55c98262b2e0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febeeffd640 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c9826346b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febef7fe640 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c982630370 0x55c98263bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febee7fc640 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c98262b2e0 0x55c982634bf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7febd800e950 tx=0x7febd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d33e2057da885b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febd67fc640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febd67fc640 1 -- 172.21.15.62:0/2855086899 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd801fd20 con 0x55c98262b2e0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febf83e7740 1 -- 172.21.15.62:0/2855086899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febe8036040 con 0x55c98262b2e0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febd67fc640 1 -- 172.21.15.62:0/2855086899 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7febd8020400 con 0x55c98262b2e0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febd67fc640 1 -- 172.21.15.62:0/2855086899 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7febd8033050 con 0x55c98262b2e0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febf83e7740 10 client.15615 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febf83e7740 1 -- 172.21.15.62:0/2855086899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c98236d620 con 0x55c98262b2e0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.198+0000 7febd67fc640 1 -- 172.21.15.62:0/2855086899 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe8036040 con 0x55c98262b2e0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febd67fc640 1 -- 172.21.15.62:0/2855086899 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7febd8028580 con 0x55c98262b2e0 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febf83e7740 10 client.15615 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:45.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febf83e7740 10 client.15615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febf83e7740 20 client.15615 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febf83e7740 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c9826403e0 0x55c982642870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febf83e7740 4 client.15615 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febf83e7740 1 -- 172.21.15.62:0/2855086899 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c98259ba20 con 0x55c9826403e0 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febeeffd640 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c9826403e0 0x55c982642870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febd67fc640 10 client.15615 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.199+0000 7febeeffd640 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c9826403e0 0x55c982642870 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b82c6525f4aefd0 server_cookie=4902a0f1731f6bcc in_seq=0 out_seq=0 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febd67fc640 1 -- 172.21.15.62:0/2855086899 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c98259ba20 con 0x55c9826403e0 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febd67fc640 10 client.15615 handle_command_reply: tid=0 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febf83e7740 1 client.15615 shutdown 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febf83e7740 1 -- 172.21.15.62:0/2855086899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c98262b2e0 msgr2=0x55c982634bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febf83e7740 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c98262b2e0 0x55c982634bf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7febd800e950 tx=0x7febd8004730 comp rx=0 tx=0).stop 2024-09-15T23:16:45.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febf83e7740 1 -- 172.21.15.62:0/2855086899 shutdown_connections 2024-09-15T23:16:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febf83e7740 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c9826403e0 0x55c982642870 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febf83e7740 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c982630370 0x55c98263bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febd67fc640 0 client.15615 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febf83e7740 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c98262b2e0 0x55c982634bf0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febf83e7740 1 --2- 172.21.15.62:0/2855086899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c9826346b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.200+0000 7febf83e7740 1 -- 172.21.15.62:0/2855086899 >> 172.21.15.62:0/2855086899 conn(0x55c982626010 msgr2=0x55c982627820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.201+0000 7febf83e7740 1 -- 172.21.15.62:0/2855086899 shutdown_connections 2024-09-15T23:16:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.201+0000 7febf83e7740 1 -- 172.21.15.62:0/2855086899 wait complete. 2024-09-15T23:16:45.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.201+0000 7febf83e7740 20 client.15615 trim_cache size 0 max 16384 2024-09-15T23:16:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febf83e7740 1 Processor -- start 2024-09-15T23:16:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febf83e7740 1 -- start start 2024-09-15T23:16:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febf83e7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262a9a0 0x55c982628ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febf83e7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 0x55c98263bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febf83e7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982629410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febf83e7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7febf01af0e0 con 0x55c982630370 2024-09-15T23:16:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febf83e7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7febf01ac9b0 con 0x55c98262b2e0 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febf83e7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7febf01aa800 con 0x55c98262a9a0 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982629410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982629410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48538/0 (socket says 172.21.15.62:48538) 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febef7fe640 1 -- 172.21.15.62:0/1888998470 learned_addr learned my addr 172.21.15.62:0/1888998470 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febef7fe640 1 -- 172.21.15.62:0/1888998470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262a9a0 msgr2=0x55c982628ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febef7fe640 1 --2- 172.21.15.62:0/1888998470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262a9a0 0x55c982628ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febef7fe640 1 -- 172.21.15.62:0/1888998470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 msgr2=0x55c98263bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febef7fe640 1 --2- 172.21.15.62:0/1888998470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 0x55c98263bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.215+0000 7febef7fe640 1 -- 172.21.15.62:0/1888998470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c982642ef0 con 0x55c982630370 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.216+0000 7febef7fe640 1 --2- 172.21.15.62:0/1888998470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982629410 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7febe8031ae0 tx=0x7febe8005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591b28f9a7d32844 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:45.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.216+0000 7febe4ff9640 1 -- 172.21.15.62:0/1888998470 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe80bd110 con 0x55c982630370 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.216+0000 7febe4ff9640 1 -- 172.21.15.62:0/1888998470 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe8036040 con 0x55c982630370 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.216+0000 7febe4ff9640 1 -- 172.21.15.62:0/1888998470 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe80dbde0 con 0x55c982630370 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.216+0000 7febf83e7740 1 -- 172.21.15.62:0/1888998470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 msgr2=0x55c982629410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.216+0000 7febf83e7740 1 --2- 172.21.15.62:0/1888998470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982629410 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7febe8031ae0 tx=0x7febe8005f30 comp rx=0 tx=0).stop 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.216+0000 7febf83e7740 1 -- 172.21.15.62:0/1888998470 shutdown_connections 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 --2- 172.21.15.62:0/1888998470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c982629410 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 --2- 172.21.15.62:0/1888998470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262b2e0 0x55c98263bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 --2- 172.21.15.62:0/1888998470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262a9a0 0x55c982628ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 -- 172.21.15.62:0/1888998470 >> 172.21.15.62:0/1888998470 conn(0x55c982626010 msgr2=0x55c9826346b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 -- 172.21.15.62:0/1888998470 shutdown_connections 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 -- 172.21.15.62:0/1888998470 wait complete. 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 -- start start 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 Processor -- start 2024-09-15T23:16:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c982639600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262b2e0 0x55c982639b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c98263ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7febf01aae70 con 0x55c982630370 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7febf01ac2e0 con 0x55c98262a9a0 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7febf01ab540 con 0x55c98262b2e0 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febf83e7740 10 client.0 _ll_register_callbacks cb 0x7febd800e620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c98263ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c98263ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48542/0 (socket says 172.21.15.62:48542) 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febef7fe640 1 -- 172.21.15.62:0/2183223954 learned_addr learned my addr 172.21.15.62:0/2183223954 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febee7fc640 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262b2e0 0x55c982639b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.217+0000 7febeeffd640 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c982639600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febef7fe640 1 -- 172.21.15.62:0/2183223954 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262b2e0 msgr2=0x55c982639b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:45.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febef7fe640 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262b2e0 0x55c982639b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febef7fe640 1 -- 172.21.15.62:0/2183223954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 msgr2=0x55c982639600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febef7fe640 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c982639600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febef7fe640 1 -- 172.21.15.62:0/2183223954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c982633680 con 0x55c982630370 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febeeffd640 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c982639600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febef7fe640 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c98263ab00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7febe80b9570 tx=0x7febe80c14c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978953661361b477 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febd67fc640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febd67fc640 1 -- 172.21.15.62:0/2183223954 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe80dbdc0 con 0x55c982630370 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febd67fc640 1 -- 172.21.15.62:0/2183223954 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7febe80dd0d0 con 0x55c982630370 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febf83e7740 1 -- 172.21.15.62:0/2183223954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febe8036040 con 0x55c982630370 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febd67fc640 1 -- 172.21.15.62:0/2183223954 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7febe80c5d80 con 0x55c982630370 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febf83e7740 10 client.15621 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febf83e7740 1 -- 172.21.15.62:0/2183223954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c9826ab270 con 0x55c982630370 2024-09-15T23:16:45.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febd67fc640 1 -- 172.21.15.62:0/2183223954 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe8036040 con 0x55c982630370 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febd67fc640 1 -- 172.21.15.62:0/2183223954 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7febe8004340 con 0x55c982630370 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febf83e7740 10 client.15621 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febf83e7740 10 client.15621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febf83e7740 20 client.15621 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febf83e7740 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c9826ab9d0 0x55c9826addc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febf83e7740 4 client.15621 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.218+0000 7febf83e7740 1 -- 172.21.15.62:0/2183223954 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c982595d70 con 0x55c9826ab9d0 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.219+0000 7febeeffd640 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c9826ab9d0 0x55c9826addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.219+0000 7febd67fc640 10 client.15621 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:45.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.219+0000 7febeeffd640 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c9826ab9d0 0x55c9826addc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaa7c8e5b0503f14 server_cookie=bc99457272da793e in_seq=0 out_seq=0 2024-09-15T23:16:45.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:44 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:16:45.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:44 smithi067 ceph-mon[35665]: pgmap v384: 129 pgs: 129 active+clean; 933 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 15 MiB/s wr, 509 op/s 2024-09-15T23:16:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.284+0000 7febd67fc640 1 -- 172.21.15.62:0/2183223954 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c982595d70 con 0x55c9826ab9d0 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.284+0000 7febd67fc640 10 client.15621 handle_command_reply: tid=0 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 client.15621 shutdown 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 -- 172.21.15.62:0/2183223954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 msgr2=0x55c98263ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c98263ab00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7febe80b9570 tx=0x7febe80c14c0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 -- 172.21.15.62:0/2183223954 shutdown_connections 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c9826ab9d0 0x55c9826addc0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c982630370 0x55c98263ab00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febd67fc640 0 client.15621 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c98262b2e0 0x55c982639b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 --2- 172.21.15.62:0/2183223954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c98262a9a0 0x55c982639600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 -- 172.21.15.62:0/2183223954 >> 172.21.15.62:0/2183223954 conn(0x55c982626010 msgr2=0x55c9826249a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 -- 172.21.15.62:0/2183223954 shutdown_connections 2024-09-15T23:16:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 1 -- 172.21.15.62:0/2183223954 wait complete. 2024-09-15T23:16:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf83e7740 20 client.15621 trim_cache size 0 max 16384 2024-09-15T23:16:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:16:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:16:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:16:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:16:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf564c640 1 -- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7febdc0a32f0 msgr2=0x7febdc0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf564c640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7febdc0a32f0 0x7febdc0a57b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7febd80058a0 tx=0x7febd80128d0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf564c640 1 -- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 msgr2=0x7febf0100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.285+0000 7febf564c640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 0x7febf0100b30 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7febe8031a40 tx=0x7febe8031ee0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.286+0000 7febf564c640 1 -- 172.21.15.62:0/3733583638 shutdown_connections 2024-09-15T23:16:45.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.286+0000 7febf564c640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7febdc0a32f0 0x7febdc0a57b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.286+0000 7febf564c640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7febf01078a0 0x7febf0100b30 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.286+0000 7febf564c640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febf0104f70 0x7febf01005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.286+0000 7febf564c640 1 --2- 172.21.15.62:0/3733583638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7febf0102640 0x7febf0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:45.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.286+0000 7febf564c640 1 -- 172.21.15.62:0/3733583638 >> 172.21.15.62:0/3733583638 conn(0x7febf00fc1e0 msgr2=0x7febf0102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:45.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.286+0000 7febf564c640 1 -- 172.21.15.62:0/3733583638 shutdown_connections 2024-09-15T23:16:45.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:45.286+0000 7febf564c640 1 -- 172.21.15.62:0/3733583638 wait complete. 2024-09-15T23:16:45.292 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:44 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:16:45.292 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:44 smithi148 ceph-mon[35610]: pgmap v384: 129 pgs: 129 active+clean; 933 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 15 MiB/s wr, 509 op/s 2024-09-15T23:16:45.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:44 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:16:45.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:44 smithi062 ceph-mon[30949]: pgmap v384: 129 pgs: 129 active+clean; 933 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 15 MiB/s wr, 509 op/s 2024-09-15T23:16:45.704 INFO:tasks.cephfs.filesystem:scrub status for tag:37b08474-b25b-4afa-b111-83d888471b58 - {'path': '/', 'tag': '37b08474-b25b-4afa-b111-83d888471b58', 'options': 'recursive,force'} 2024-09-15T23:16:47.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:46 smithi067 ceph-mon[35665]: pgmap v385: 129 pgs: 129 active+clean; 933 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 13 MiB/s wr, 405 op/s 2024-09-15T23:16:47.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:46 smithi148 ceph-mon[35610]: pgmap v385: 129 pgs: 129 active+clean; 933 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 13 MiB/s wr, 405 op/s 2024-09-15T23:16:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:46 smithi062 ceph-mon[30949]: pgmap v385: 129 pgs: 129 active+clean; 933 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 13 MiB/s wr, 405 op/s 2024-09-15T23:16:48.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:47 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:16:48.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:47 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:16:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:47 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:16:49.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:48 smithi067 ceph-mon[35665]: pgmap v386: 129 pgs: 129 active+clean; 934 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 15 MiB/s wr, 434 op/s 2024-09-15T23:16:49.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:48 smithi148 ceph-mon[35610]: pgmap v386: 129 pgs: 129 active+clean; 934 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 15 MiB/s wr, 434 op/s 2024-09-15T23:16:49.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:48 smithi062 ceph-mon[30949]: pgmap v386: 129 pgs: 129 active+clean; 934 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 15 MiB/s wr, 434 op/s 2024-09-15T23:16:49.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 30.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:16:51.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:51 smithi067 ceph-mon[35665]: pgmap v387: 129 pgs: 129 active+clean; 935 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 17 MiB/s wr, 350 op/s 2024-09-15T23:16:51.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:51 smithi148 ceph-mon[35610]: pgmap v387: 129 pgs: 129 active+clean; 935 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 17 MiB/s wr, 350 op/s 2024-09-15T23:16:51.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:51 smithi062 ceph-mon[30949]: pgmap v387: 129 pgs: 129 active+clean; 935 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 17 MiB/s wr, 350 op/s 2024-09-15T23:16:53.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:53 smithi067 ceph-mon[35665]: pgmap v388: 129 pgs: 129 active+clean; 937 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 20 MiB/s wr, 355 op/s 2024-09-15T23:16:53.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:53 smithi148 ceph-mon[35610]: pgmap v388: 129 pgs: 129 active+clean; 937 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 20 MiB/s wr, 355 op/s 2024-09-15T23:16:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:53 smithi062 ceph-mon[30949]: pgmap v388: 129 pgs: 129 active+clean; 937 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 20 MiB/s wr, 355 op/s 2024-09-15T23:16:54.572 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 35.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:16:55.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:55 smithi067 ceph-mon[35665]: pgmap v389: 129 pgs: 129 active+clean; 943 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 22 MiB/s wr, 291 op/s 2024-09-15T23:16:55.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:55 smithi148 ceph-mon[35610]: pgmap v389: 129 pgs: 129 active+clean; 943 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 22 MiB/s wr, 291 op/s 2024-09-15T23:16:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:55 smithi062 ceph-mon[30949]: pgmap v389: 129 pgs: 129 active+clean; 943 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 22 MiB/s wr, 291 op/s 2024-09-15T23:16:57.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:57 smithi067 ceph-mon[35665]: pgmap v390: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 26 MiB/s wr, 312 op/s 2024-09-15T23:16:57.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:57 smithi148 ceph-mon[35610]: pgmap v390: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 26 MiB/s wr, 312 op/s 2024-09-15T23:16:57.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:57 smithi062 ceph-mon[30949]: pgmap v390: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 26 MiB/s wr, 312 op/s 2024-09-15T23:16:57.659 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:16:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:16:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.376+0000 7fa40181e640 1 -- 172.21.15.62:0/1858126219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc102620 msgr2=0x7fa3fc104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.376+0000 7fa40181e640 1 --2- 172.21.15.62:0/1858126219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc102620 0x7fa3fc104a10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec009900 tx=0x7fa3ec02a410 comp rx=0 tx=0).stop 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.376+0000 7fa40181e640 1 -- 172.21.15.62:0/1858126219 shutdown_connections 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.376+0000 7fa40181e640 1 --2- 172.21.15.62:0/1858126219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc107880 0x7fa3fc109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.376+0000 7fa40181e640 1 --2- 172.21.15.62:0/1858126219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3fc104f50 0x7fa3fc107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.376+0000 7fa40181e640 1 --2- 172.21.15.62:0/1858126219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc102620 0x7fa3fc104a10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.376+0000 7fa40181e640 1 -- 172.21.15.62:0/1858126219 >> 172.21.15.62:0/1858126219 conn(0x7fa3fc0fc1e0 msgr2=0x7fa3fc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.376+0000 7fa40181e640 1 -- 172.21.15.62:0/1858126219 shutdown_connections 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.376+0000 7fa40181e640 1 -- 172.21.15.62:0/1858126219 wait complete. 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa40181e640 1 Processor -- start 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa40181e640 1 -- start start 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa40181e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3fc102620 0x7fa3fc197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa40181e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc104f50 0x7fa3fc1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa40181e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc107880 0x7fa3fc19f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa40181e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc10d830 con 0x7fa3fc104f50 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa40181e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc10d6b0 con 0x7fa3fc107880 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa40181e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc10d9b0 con 0x7fa3fc102620 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa3f3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc104f50 0x7fa3fc1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa3f3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc104f50 0x7fa3fc1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49612/0 (socket says 172.21.15.62:49612) 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa3f3fff640 1 -- 172.21.15.62:0/3153023070 learned_addr learned my addr 172.21.15.62:0/3153023070 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa3faffd640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3fc102620 0x7fa3fc197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.377+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc107880 0x7fa3fc19f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa3f3fff640 1 -- 172.21.15.62:0/3153023070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3fc102620 msgr2=0x7fa3fc197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3fc102620 0x7fa3fc197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa3f3fff640 1 -- 172.21.15.62:0/3153023070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc107880 msgr2=0x7fa3fc19f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc107880 0x7fa3fc19f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.382 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa3f3fff640 1 -- 172.21.15.62:0/3153023070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc19fab0 con 0x7fa3fc104f50 2024-09-15T23:16:58.382 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc104f50 0x7fa3fc1983d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400d740 tx=0x7fa3e400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:58.382 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa40081c640 1 -- 172.21.15.62:0/3153023070 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4017070 con 0x7fa3fc104f50 2024-09-15T23:16:58.382 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa40081c640 1 -- 172.21.15.62:0/3153023070 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3e400de50 con 0x7fa3fc104f50 2024-09-15T23:16:58.382 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa40181e640 1 -- 172.21.15.62:0/3153023070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3fc19fda0 con 0x7fa3fc104f50 2024-09-15T23:16:58.382 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa40081c640 1 -- 172.21.15.62:0/3153023070 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4004e00 con 0x7fa3fc104f50 2024-09-15T23:16:58.382 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.378+0000 7fa40181e640 1 -- 172.21.15.62:0/3153023070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3fc1a0210 con 0x7fa3fc104f50 2024-09-15T23:16:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa40081c640 1 -- 172.21.15.62:0/3153023070 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa3e4004fa0 con 0x7fa3fc104f50 2024-09-15T23:16:58.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa40468c740 1 Processor -- start 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa40468c740 1 -- start start 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa40468c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 0x564b3c5f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa40468c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5f91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa40468c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 0x564b3c5fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa40468c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564b3c5532f0 con 0x564b3c5f46f0 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa40468c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564b3c4fb270 con 0x564b3c5f3db0 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa40468c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564b3c54f2b0 con 0x564b3c5f9780 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa3faffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5f91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa3faffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5f91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49618/0 (socket says 172.21.15.62:49618) 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa3faffd640 1 -- 172.21.15.62:0/1892548200 learned_addr learned my addr 172.21.15.62:0/1892548200 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.381+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 0x564b3c5f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 0x564b3c5fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3faffd640 1 -- 172.21.15.62:0/1892548200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 msgr2=0x564b3c5fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3faffd640 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 0x564b3c5fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3faffd640 1 -- 172.21.15.62:0/1892548200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 msgr2=0x564b3c5f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3faffd640 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 0x564b3c5f41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3faffd640 1 -- 172.21.15.62:0/1892548200 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b3c4fa340 con 0x564b3c5f46f0 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 0x564b3c5fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3faffd640 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5f91b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec02a930 tx=0x7fa3ec005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26e130ee795fa25 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3f1ffb640 1 -- 172.21.15.62:0/1892548200 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec040070 con 0x564b3c5f46f0 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa40081c640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3e80a32f0 0x7fa3e80a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3f1ffb640 1 -- 172.21.15.62:0/1892548200 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3ec0318b0 con 0x564b3c5f46f0 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3f1ffb640 1 -- 172.21.15.62:0/1892548200 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec035040 con 0x564b3c5f46f0 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3e80a32f0 0x7fa3e80a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.382+0000 7fa40081c640 1 -- 172.21.15.62:0/3153023070 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa3e40cf660 con 0x7fa3fc104f50 2024-09-15T23:16:58.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 -- 172.21.15.62:0/1892548200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 msgr2=0x564b3c5f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3e80a32f0 0x7fa3e80a57b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc06a980 tx=0x7fa3f4009580 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5f91b0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec02a930 tx=0x7fa3ec005c00 comp rx=0 tx=0).stop 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 -- 172.21.15.62:0/1892548200 shutdown_connections 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 0x564b3c5fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5f91b0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 --2- 172.21.15.62:0/1892548200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 0x564b3c5f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 -- 172.21.15.62:0/1892548200 >> 172.21.15.62:0/1892548200 conn(0x564b3c5eb010 msgr2=0x564b3c5eb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 -- 172.21.15.62:0/1892548200 shutdown_connections 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 -- 172.21.15.62:0/1892548200 wait complete. 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.383+0000 7fa40468c740 1 -- start start 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa40468c740 1 Processor -- start 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa40468c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 0x564b3c5fdf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa40468c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5fe490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa40468c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 0x564b3c6054d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa40468c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564b3c5fc6f0 con 0x564b3c5f46f0 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa40468c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564b3c4dc290 con 0x564b3c5f3db0 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa40468c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564b3c5fc870 con 0x564b3c5f9780 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa40468c740 10 client.0 _ll_register_callbacks cb 0x564b3c50d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3f3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5fe490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3f3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5fe490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49622/0 (socket says 172.21.15.62:49622) 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3f3fff640 1 -- 172.21.15.62:0/3703619441 learned_addr learned my addr 172.21.15.62:0/3703619441 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3faffd640 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 0x564b3c5fdf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 0x564b3c6054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3f3fff640 1 -- 172.21.15.62:0/3703619441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 msgr2=0x564b3c6054d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 0x564b3c6054d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3f3fff640 1 -- 172.21.15.62:0/3703619441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 msgr2=0x564b3c5fdf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 0x564b3c5fdf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.384+0000 7fa3f3fff640 1 -- 172.21.15.62:0/3703619441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b3c266210 con 0x564b3c5f46f0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5fe490 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa3e409b600 tx=0x7fa3e4004780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376fa9edaa3ca4f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa3daffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa3daffd640 1 -- 172.21.15.62:0/3703619441 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4020070 con 0x564b3c5f46f0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa3daffd640 1 -- 172.21.15.62:0/3703619441 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa3e40cf3a0 con 0x564b3c5f46f0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa40468c740 1 -- 172.21.15.62:0/3703619441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3ec0318b0 con 0x564b3c5f46f0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa3daffd640 1 -- 172.21.15.62:0/3703619441 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa3e409fcb0 con 0x564b3c5f46f0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa40468c740 10 client.15633 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa40468c740 1 -- 172.21.15.62:0/3703619441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564b3c332620 con 0x564b3c5f46f0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa3daffd640 1 -- 172.21.15.62:0/3703619441 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3ec0318b0 con 0x564b3c5f46f0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa3daffd640 1 -- 172.21.15.62:0/3703619441 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fa3e409ae50 con 0x564b3c5f46f0 2024-09-15T23:16:58.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa40468c740 10 client.15633 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa40468c740 10 client.15633 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa40468c740 20 client.15633 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa40468c740 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564b3c609ae0 0x564b3c60bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa40468c740 4 client.15633 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.385+0000 7fa40468c740 1 -- 172.21.15.62:0/3703619441 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b3c560a20 con 0x564b3c609ae0 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.386+0000 7fa3faffd640 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564b3c609ae0 0x564b3c60bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.386+0000 7fa3faffd640 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564b3c609ae0 0x564b3c60bf50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87ce6cb3634d9d3c server_cookie=7ece9557de24fdc1 in_seq=0 out_seq=0 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.386+0000 7fa3daffd640 10 client.15633 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa3daffd640 1 -- 172.21.15.62:0/3703619441 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564b3c560a20 con 0x564b3c609ae0 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa3daffd640 10 client.15633 handle_command_reply: tid=0 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 client.15633 shutdown 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 -- 172.21.15.62:0/3703619441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 msgr2=0x564b3c5fe490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.394 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5fe490 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa3e409b600 tx=0x7fa3e4004780 comp rx=0 tx=0).stop 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 -- 172.21.15.62:0/3703619441 shutdown_connections 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564b3c609ae0 0x564b3c60bf50 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f9780 0x564b3c6054d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f46f0 0x564b3c5fe490 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 --2- 172.21.15.62:0/3703619441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f3db0 0x564b3c5fdf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa3daffd640 0 client.15633 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 -- 172.21.15.62:0/3703619441 >> 172.21.15.62:0/3703619441 conn(0x564b3c5eb010 msgr2=0x564b3c5ec820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 -- 172.21.15.62:0/3703619441 shutdown_connections 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 1 -- 172.21.15.62:0/3703619441 wait complete. 2024-09-15T23:16:58.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.387+0000 7fa40468c740 20 client.15633 trim_cache size 0 max 16384 2024-09-15T23:16:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa40468c740 1 Processor -- start 2024-09-15T23:16:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa40468c740 1 -- start start 2024-09-15T23:16:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa40468c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 0x564b3c5f8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa40468c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 0x564b3c605490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa40468c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c5ea0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa40468c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1aad30 con 0x564b3c5f9780 2024-09-15T23:16:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa40468c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1a8600 con 0x564b3c5f46f0 2024-09-15T23:16:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa40468c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1a6450 con 0x564b3c5f3db0 2024-09-15T23:16:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3fb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c5ea0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3fb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c5ea0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49632/0 (socket says 172.21.15.62:49632) 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3fb7fe640 1 -- 172.21.15.62:0/963233686 learned_addr learned my addr 172.21.15.62:0/963233686 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 0x564b3c605490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3faffd640 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 0x564b3c5f8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3fb7fe640 1 -- 172.21.15.62:0/963233686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 msgr2=0x564b3c5f8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 0x564b3c5f8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3fb7fe640 1 -- 172.21.15.62:0/963233686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 msgr2=0x564b3c605490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 0x564b3c605490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3fb7fe640 1 -- 172.21.15.62:0/963233686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b3c5f4bd0 con 0x564b3c5f9780 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.402+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 0x564b3c605490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.403+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c5ea0b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40129b0 tx=0x7fa3f4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f689a789b6d9fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.403+0000 7fa3f17fa640 1 -- 172.21.15.62:0/963233686 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4020bb0 con 0x564b3c5f9780 2024-09-15T23:16:58.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.403+0000 7fa3f17fa640 1 -- 172.21.15.62:0/963233686 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3f4020d50 con 0x564b3c5f9780 2024-09-15T23:16:58.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.403+0000 7fa3f17fa640 1 -- 172.21.15.62:0/963233686 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f401b6d0 con 0x564b3c5f9780 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 -- 172.21.15.62:0/963233686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 msgr2=0x564b3c5ea0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c5ea0b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40129b0 tx=0x7fa3f4014000 comp rx=0 tx=0).stop 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 -- 172.21.15.62:0/963233686 shutdown_connections 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c5ea0b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 0x564b3c605490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 --2- 172.21.15.62:0/963233686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 0x564b3c5f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 -- 172.21.15.62:0/963233686 >> 172.21.15.62:0/963233686 conn(0x564b3c5eb010 msgr2=0x564b3c5e99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 -- 172.21.15.62:0/963233686 shutdown_connections 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 -- 172.21.15.62:0/963233686 wait complete. 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 -- start start 2024-09-15T23:16:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 Processor -- start 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 0x564b3c603100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 0x564b3c603640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c604650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1a6ac0 con 0x564b3c5f9780 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1a7f30 con 0x564b3c5f46f0 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc1a7190 con 0x564b3c5f3db0 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.404+0000 7fa40468c740 10 client.0 _ll_register_callbacks cb 0x7fa3e4005cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3fb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c604650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3fb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c604650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49636/0 (socket says 172.21.15.62:49636) 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3fb7fe640 1 -- 172.21.15.62:0/173226678 learned_addr learned my addr 172.21.15.62:0/173226678 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3faffd640 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 0x564b3c603100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3f3fff640 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 0x564b3c603640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3fb7fe640 1 -- 172.21.15.62:0/173226678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 msgr2=0x564b3c603100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 0x564b3c603100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3fb7fe640 1 -- 172.21.15.62:0/173226678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 msgr2=0x564b3c603640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 0x564b3c603640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3fb7fe640 1 -- 172.21.15.62:0/173226678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b3c604c60 con 0x564b3c5f9780 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3fb7fe640 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c604650 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4020800 tx=0x7fa3f4020be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=860e7d0f2a312863 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3daffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3daffd640 1 -- 172.21.15.62:0/173226678 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4039060 con 0x564b3c5f9780 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa40468c740 1 -- 172.21.15.62:0/173226678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3f4020d50 con 0x564b3c5f9780 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3daffd640 1 -- 172.21.15.62:0/173226678 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa3f4004c50 con 0x564b3c5f9780 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.405+0000 7fa3daffd640 1 -- 172.21.15.62:0/173226678 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa3f4040050 con 0x564b3c5f9780 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa40468c740 10 client.15639 fetch_fsmap learned FSMap version 26 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa40468c740 1 -- 172.21.15.62:0/173226678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564b3c674e50 con 0x564b3c5f9780 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa3daffd640 1 -- 172.21.15.62:0/173226678 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3f4020d50 con 0x564b3c5f9780 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa3daffd640 1 -- 172.21.15.62:0/173226678 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fa3f400f040 con 0x564b3c5f9780 2024-09-15T23:16:58.411 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa40468c740 10 client.15639 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:16:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa40468c740 10 client.15639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:16:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa40468c740 20 client.15639 populate_metadata read hostname 'smithi062' 2024-09-15T23:16:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa40468c740 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564b3c675790 0x564b3c677bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa40468c740 4 client.15639 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:16:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa40468c740 1 -- 172.21.15.62:0/173226678 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564b3c55ad70 con 0x564b3c675790 2024-09-15T23:16:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.406+0000 7fa3faffd640 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564b3c675790 0x564b3c677bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.407+0000 7fa3faffd640 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564b3c675790 0x564b3c677bb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1686ff8597c785d9 server_cookie=1cf7ce502e353423 in_seq=0 out_seq=0 2024-09-15T23:16:58.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.407+0000 7fa3daffd640 10 client.15639 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.419+0000 7fa3daffd640 1 -- 172.21.15.62:0/173226678 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564b3c55ad70 con 0x564b3c675790 2024-09-15T23:16:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.419+0000 7fa3daffd640 10 client.15639 handle_command_reply: tid=0 2024-09-15T23:16:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.419+0000 7fa40468c740 1 client.15639 shutdown 2024-09-15T23:16:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 -- 172.21.15.62:0/173226678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 msgr2=0x564b3c604650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c604650 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4020800 tx=0x7fa3f4020be0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 -- 172.21.15.62:0/173226678 shutdown_connections 2024-09-15T23:16:58.421 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564b3c675790 0x564b3c677bb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564b3c5f9780 0x564b3c604650 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564b3c5f46f0 0x564b3c603640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa3daffd640 0 client.15639 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:16:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 --2- 172.21.15.62:0/173226678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564b3c5f3db0 0x564b3c603100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 -- 172.21.15.62:0/173226678 >> 172.21.15.62:0/173226678 conn(0x564b3c5eb010 msgr2=0x564b3c5e99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 -- 172.21.15.62:0/173226678 shutdown_connections 2024-09-15T23:16:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 1 -- 172.21.15.62:0/173226678 wait complete. 2024-09-15T23:16:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40468c740 20 client.15639 trim_cache size 0 max 16384 2024-09-15T23:16:58.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:16:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:16:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:16:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:16:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40181e640 1 -- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3e80a32f0 msgr2=0x7fa3e80a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40181e640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3e80a32f0 0x7fa3e80a57b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc06a980 tx=0x7fa3f4009580 comp rx=0 tx=0).stop 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40181e640 1 -- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc104f50 msgr2=0x7fa3fc1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.420+0000 7fa40181e640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc104f50 0x7fa3fc1983d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa3e400d740 tx=0x7fa3e400dc00 comp rx=0 tx=0).stop 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.421+0000 7fa40181e640 1 -- 172.21.15.62:0/3153023070 shutdown_connections 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.421+0000 7fa40181e640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3e80a32f0 0x7fa3e80a57b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.421+0000 7fa40181e640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3fc107880 0x7fa3fc19f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.421+0000 7fa40181e640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3fc104f50 0x7fa3fc1983d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.421+0000 7fa40181e640 1 --2- 172.21.15.62:0/3153023070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3fc102620 0x7fa3fc197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.421+0000 7fa40181e640 1 -- 172.21.15.62:0/3153023070 >> 172.21.15.62:0/3153023070 conn(0x7fa3fc0fc1e0 msgr2=0x7fa3fc103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.421+0000 7fa40181e640 1 -- 172.21.15.62:0/3153023070 shutdown_connections 2024-09-15T23:16:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:16:58.421+0000 7fa40181e640 1 -- 172.21.15.62:0/3153023070 wait complete. 2024-09-15T23:16:58.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 -- 172.21.15.62:0/654844958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00100e20 msgr2=0x7fca00101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 --2- 172.21.15.62:0/654844958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00100e20 0x7fca00101200 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009900 tx=0x7fc9fc02a480 comp rx=0 tx=0).stop 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 -- 172.21.15.62:0/654844958 shutdown_connections 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 --2- 172.21.15.62:0/654844958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00102ef0 0x7fca0010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 --2- 172.21.15.62:0/654844958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca00102570 0x7fca001029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 --2- 172.21.15.62:0/654844958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00100e20 0x7fca00101200 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 -- 172.21.15.62:0/654844958 >> 172.21.15.62:0/654844958 conn(0x7fca000fc7d0 msgr2=0x7fca000febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 -- 172.21.15.62:0/654844958 shutdown_connections 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 -- 172.21.15.62:0/654844958 wait complete. 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.436+0000 7fca086cd640 1 Processor -- start 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca086cd640 1 -- start start 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca086cd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00100e20 0x7fca001992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca086cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca00102570 0x7fca001997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca086cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00102ef0 0x7fca001a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca086cd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fca00110f80 con 0x7fca00102ef0 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca086cd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fca00110e00 con 0x7fca00102570 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca086cd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fca00111100 con 0x7fca00100e20 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca06c6a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00102ef0 0x7fca001a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca06c6a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00102ef0 0x7fca001a4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49662/0 (socket says 172.21.15.62:49662) 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca06c6a640 1 -- 172.21.15.62:0/1978540637 learned_addr learned my addr 172.21.15.62:0/1978540637 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca05c68640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca00102570 0x7fca001997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca06469640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00100e20 0x7fca001992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca06c6a640 1 -- 172.21.15.62:0/1978540637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00100e20 msgr2=0x7fca001992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca06c6a640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00100e20 0x7fca001992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca06c6a640 1 -- 172.21.15.62:0/1978540637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca00102570 msgr2=0x7fca001997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca06c6a640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca00102570 0x7fca001997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.437+0000 7fca06c6a640 1 -- 172.21.15.62:0/1978540637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca001a4870 con 0x7fca00102ef0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.438+0000 7fca06c6a640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00102ef0 0x7fca001a4330 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc9f800eaa0 tx=0x7fc9f800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.438+0000 7fc9f77fe640 1 -- 172.21.15.62:0/1978540637 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f800cdb0 con 0x7fca00102ef0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.438+0000 7fc9f77fe640 1 -- 172.21.15.62:0/1978540637 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9f8005020 con 0x7fca00102ef0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.438+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca001a4a40 con 0x7fca00102ef0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.438+0000 7fc9f77fe640 1 -- 172.21.15.62:0/1978540637 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f80116b0 con 0x7fca00102ef0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.438+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca001a4f50 con 0x7fca00102ef0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.439+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca00101200 con 0x7fca00102ef0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.440+0000 7fc9f77fe640 1 -- 172.21.15.62:0/1978540637 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc9f8005560 con 0x7fca00102ef0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.441+0000 7fc9f77fe640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9ec0a3240 0x7fc9ec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.441+0000 7fca06469640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9ec0a3240 0x7fc9ec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:16:58.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.441+0000 7fc9f77fe640 1 -- 172.21.15.62:0/1978540637 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc9f80d02f0 con 0x7fca00102ef0 2024-09-15T23:16:58.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.441+0000 7fca06469640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9ec0a3240 0x7fc9ec0a5700 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc005ab0 tx=0x7fc9fc005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:16:58.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.444+0000 7fc9f77fe640 1 -- 172.21.15.62:0/1978540637 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc9f8016030 con 0x7fca00102ef0 2024-09-15T23:16:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.658+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca001a5240 con 0x7fca00102ef0 2024-09-15T23:16:58.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.659+0000 7fc9f77fe640 1 -- 172.21.15.62:0/1978540637 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11904 (secure 0 0 0) 0x7fc9f809cc30 con 0x7fca00102ef0 2024-09-15T23:16:58.660 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:16:58.660 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":26,"btime":"2024-09-15T23:16:32:870187+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:16:32.375022+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:16:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 26 2024-09-15T23:16:58.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9ec0a3240 msgr2=0x7fc9ec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9ec0a3240 0x7fc9ec0a5700 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc005ab0 tx=0x7fc9fc005a20 comp rx=0 tx=0).stop 2024-09-15T23:16:58.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00102ef0 msgr2=0x7fca001a4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:16:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00102ef0 0x7fca001a4330 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc9f800eaa0 tx=0x7fc9f800ef60 comp rx=0 tx=0).stop 2024-09-15T23:16:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 shutdown_connections 2024-09-15T23:16:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9ec0a3240 0x7fc9ec0a5700 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fca00102ef0 0x7fca001a4330 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca00102570 0x7fca001997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 --2- 172.21.15.62:0/1978540637 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fca00100e20 0x7fca001992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:16:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.661+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 >> 172.21.15.62:0/1978540637 conn(0x7fca000fc7d0 msgr2=0x7fca00106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:16:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.662+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 shutdown_connections 2024-09-15T23:16:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:16:58.662+0000 7fca086cd640 1 -- 172.21.15.62:0/1978540637 wait complete. 2024-09-15T23:16:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:59 smithi067 ceph-mon[35665]: pgmap v391: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 27 MiB/s wr, 289 op/s 2024-09-15T23:16:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:16:59 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1978540637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:59.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:59 smithi148 ceph-mon[35610]: pgmap v391: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 27 MiB/s wr, 289 op/s 2024-09-15T23:16:59.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:16:59 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1978540637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:59.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:59 smithi062 ceph-mon[30949]: pgmap v391: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 27 MiB/s wr, 289 op/s 2024-09-15T23:16:59.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:16:59 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1978540637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:16:59.440 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 10fd8a97 --release --await' 2024-09-15T23:16:59.440 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 10fd8a97 --release --await 2024-09-15T23:16:59.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 40.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:16:59.743 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:00.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.175+0000 7fb67ad62640 1 -- 172.21.15.62:0/3836176953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102570 msgr2=0x7fb6741029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:00.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.175+0000 7fb67ad62640 1 --2- 172.21.15.62:0/3836176953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102570 0x7fb6741029b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb668009900 tx=0x7fb66802a480 comp rx=0 tx=0).stop 2024-09-15T23:17:00.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.175+0000 7fb67ad62640 1 -- 172.21.15.62:0/3836176953 shutdown_connections 2024-09-15T23:17:00.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.175+0000 7fb67ad62640 1 --2- 172.21.15.62:0/3836176953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb674102ef0 0x7fb67410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.175+0000 7fb67ad62640 1 --2- 172.21.15.62:0/3836176953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102570 0x7fb6741029b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.175+0000 7fb67ad62640 1 --2- 172.21.15.62:0/3836176953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb674100e20 0x7fb674101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.175+0000 7fb67ad62640 1 -- 172.21.15.62:0/3836176953 >> 172.21.15.62:0/3836176953 conn(0x7fb6740fc7d0 msgr2=0x7fb6740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:00.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.175+0000 7fb67ad62640 1 -- 172.21.15.62:0/3836176953 shutdown_connections 2024-09-15T23:17:00.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.175+0000 7fb67ad62640 1 -- 172.21.15.62:0/3836176953 wait complete. 2024-09-15T23:17:00.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb67ad62640 1 Processor -- start 2024-09-15T23:17:00.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb67ad62640 1 -- start start 2024-09-15T23:17:00.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb67ad62640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb674100e20 0x7fb6741973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb67ad62640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb674102570 0x7fb674197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb67ad62640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102ef0 0x7fb67419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb67ad62640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb674110f80 con 0x7fb674102ef0 2024-09-15T23:17:00.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb67ad62640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb674110e00 con 0x7fb674100e20 2024-09-15T23:17:00.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb67ad62640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb674111100 con 0x7fb674102570 2024-09-15T23:17:00.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb6792ff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102ef0 0x7fb67419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb6792ff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102ef0 0x7fb67419bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49682/0 (socket says 172.21.15.62:49682) 2024-09-15T23:17:00.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb6792ff640 1 -- 172.21.15.62:0/552047811 learned_addr learned my addr 172.21.15.62:0/552047811 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:00.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb673fff640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb674102570 0x7fb674197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.176+0000 7fb678afe640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb674100e20 0x7fb6741973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb6792ff640 1 -- 172.21.15.62:0/552047811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb674102570 msgr2=0x7fb674197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:00.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb6792ff640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb674102570 0x7fb674197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb6792ff640 1 -- 172.21.15.62:0/552047811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb674100e20 msgr2=0x7fb6741973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:00.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb6792ff640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb674100e20 0x7fb6741973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb6792ff640 1 -- 172.21.15.62:0/552047811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67419c390 con 0x7fb674102ef0 2024-09-15T23:17:00.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb6792ff640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102ef0 0x7fb67419bcb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00cad0 tx=0x7fb66c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:00.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb671ffb640 1 -- 172.21.15.62:0/552047811 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00ae20 con 0x7fb674102ef0 2024-09-15T23:17:00.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb671ffb640 1 -- 172.21.15.62:0/552047811 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb66c005020 con 0x7fb674102ef0 2024-09-15T23:17:00.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb67ad62640 1 -- 172.21.15.62:0/552047811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67419c680 con 0x7fb674102ef0 2024-09-15T23:17:00.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb671ffb640 1 -- 172.21.15.62:0/552047811 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00f6f0 con 0x7fb674102ef0 2024-09-15T23:17:00.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.177+0000 7fb67ad62640 1 -- 172.21.15.62:0/552047811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6741a3f60 con 0x7fb674102ef0 2024-09-15T23:17:00.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.180+0000 7fb67d13b740 1 Processor -- start 2024-09-15T23:17:00.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.180+0000 7fb671ffb640 1 -- 172.21.15.62:0/552047811 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb66c0051d0 con 0x7fb674102ef0 2024-09-15T23:17:00.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb67d13b740 1 -- start start 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468060 0x564953468440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953468980 0x564953472d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953473240 0x564953475630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb67d13b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5649533cd620 con 0x564953473240 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb67d13b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564953374e50 con 0x564953468980 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb67d13b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5649533c95e0 con 0x564953468060 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb673fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953473240 0x564953475630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb673fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953473240 0x564953475630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49692/0 (socket says 172.21.15.62:49692) 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb673fff640 1 -- 172.21.15.62:0/2828711424 learned_addr learned my addr 172.21.15.62:0/2828711424 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:00.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb678afe640 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953468980 0x564953472d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb671ffb640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb65c0a32c0 0x7fb65c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb673fff640 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468060 0x564953468440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb673fff640 1 -- 172.21.15.62:0/2828711424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468060 msgr2=0x564953468440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb673fff640 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468060 0x564953468440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb673fff640 1 -- 172.21.15.62:0/2828711424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953468980 msgr2=0x564953472d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb673fff640 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953468980 0x564953472d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb673fff640 1 -- 172.21.15.62:0/2828711424 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564953365d40 con 0x564953473240 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb678afe640 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953468980 0x564953472d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.181+0000 7fb6792ff640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb65c0a32c0 0x7fb65c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.182+0000 7fb671ffb640 1 -- 172.21.15.62:0/552047811 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb66c0cfeb0 con 0x7fb674102ef0 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.182+0000 7fb673fff640 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953473240 0x564953475630 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb6680314e0 tx=0x7fb668000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1202d75ba9066713 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.182+0000 7fb65affd640 1 -- 172.21.15.62:0/2828711424 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668035040 con 0x564953473240 2024-09-15T23:17:00.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.182+0000 7fb65affd640 1 -- 172.21.15.62:0/2828711424 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb668003cd0 con 0x564953473240 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.182+0000 7fb65affd640 1 -- 172.21.15.62:0/2828711424 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668031dc0 con 0x564953473240 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 -- 172.21.15.62:0/2828711424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953473240 msgr2=0x564953475630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953473240 0x564953475630 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb6680314e0 tx=0x7fb668000bc0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb6792ff640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb65c0a32c0 0x7fb65c0a5780 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb66c09c190 tx=0x7fb66c066c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 -- 172.21.15.62:0/2828711424 shutdown_connections 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953473240 0x564953475630 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953468980 0x564953472d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 --2- 172.21.15.62:0/2828711424 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468060 0x564953468440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 -- 172.21.15.62:0/2828711424 >> 172.21.15.62:0/2828711424 conn(0x5649534645f0 msgr2=0x5649534649d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 -- 172.21.15.62:0/2828711424 shutdown_connections 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 -- 172.21.15.62:0/2828711424 wait complete. 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 -- start start 2024-09-15T23:17:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 Processor -- start 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.183+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x564953470720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x564953470c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb67d13b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564953475e10 con 0x564953468060 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb67d13b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564953355f10 con 0x564953473240 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb67d13b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564953475f90 con 0x564953468980 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb67d13b740 10 client.0 _ll_register_callbacks cb 0x564953379db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49706/0 (socket says 172.21.15.62:49706) 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 -- 172.21.15.62:0/3471692293 learned_addr learned my addr 172.21.15.62:0/3471692293 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x564953470c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb678afe640 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x564953470720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 -- 172.21.15.62:0/3471692293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 msgr2=0x564953470720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x564953470720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 -- 172.21.15.62:0/3471692293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 msgr2=0x564953470c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x564953470c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 -- 172.21.15.62:0/3471692293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649530dc580 con 0x564953468060 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb673fff640 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534701e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb66802a9a0 tx=0x7fb668031aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db62de25026984a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb653fff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb653fff640 1 -- 172.21.15.62:0/3471692293 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668035070 con 0x564953468060 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb653fff640 1 -- 172.21.15.62:0/3471692293 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb6680035f0 con 0x564953468060 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.184+0000 7fb67d13b740 1 -- 172.21.15.62:0/3471692293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb668003cd0 con 0x564953468060 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb653fff640 1 -- 172.21.15.62:0/3471692293 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb668003e80 con 0x564953468060 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb67d13b740 10 client.15657 fetch_fsmap learned FSMap version 26 2024-09-15T23:17:00.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb67d13b740 1 -- 172.21.15.62:0/3471692293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5649531a6960 con 0x564953468060 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb653fff640 1 -- 172.21.15.62:0/3471692293 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb668003cd0 con 0x564953468060 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb653fff640 1 -- 172.21.15.62:0/3471692293 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fb668004880 con 0x564953468060 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb67d13b740 10 client.15657 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb67d13b740 10 client.15657 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb67d13b740 20 client.15657 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb67d13b740 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564953482ce0 0x5649534850d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb67d13b740 4 client.15657 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb67d13b740 1 -- 172.21.15.62:0/3471692293 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649533dab90 con 0x564953482ce0 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.185+0000 7fb678afe640 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564953482ce0 0x5649534850d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.186+0000 7fb678afe640 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564953482ce0 0x5649534850d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83719a6c20de3997 server_cookie=ab42921c92058e99 in_seq=0 out_seq=0 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.186+0000 7fb653fff640 10 client.15657 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.186+0000 7fb653fff640 1 -- 172.21.15.62:0/3471692293 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5649533dab90 con 0x564953482ce0 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.186+0000 7fb653fff640 10 client.15657 handle_command_reply: tid=0 2024-09-15T23:17:00.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.186+0000 7fb67d13b740 1 client.15657 shutdown 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 -- 172.21.15.62:0/3471692293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 msgr2=0x5649534701e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534701e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb66802a9a0 tx=0x7fb668031aa0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 -- 172.21.15.62:0/3471692293 shutdown_connections 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564953482ce0 0x5649534850d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x564953470c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x564953470720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb653fff640 0 client.15657 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 --2- 172.21.15.62:0/3471692293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534701e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 -- 172.21.15.62:0/3471692293 >> 172.21.15.62:0/3471692293 conn(0x5649534645f0 msgr2=0x56495346a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 -- 172.21.15.62:0/3471692293 shutdown_connections 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 1 -- 172.21.15.62:0/3471692293 wait complete. 2024-09-15T23:17:00.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.187+0000 7fb67d13b740 20 client.15657 trim_cache size 0 max 16384 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.204+0000 7fb67d13b740 1 Processor -- start 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.204+0000 7fb67d13b740 1 -- start start 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.204+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534882f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.204+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x564953488830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.204+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x56495346d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.204+0000 7fb67d13b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb6741b03c0 con 0x564953468060 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.204+0000 7fb67d13b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb6741adc90 con 0x564953473240 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.204+0000 7fb67d13b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb6741abae0 con 0x564953468980 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb673fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534882f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb673fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534882f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49708/0 (socket says 172.21.15.62:49708) 2024-09-15T23:17:00.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb673fff640 1 -- 172.21.15.62:0/746170544 learned_addr learned my addr 172.21.15.62:0/746170544 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb678afe640 1 --2- 172.21.15.62:0/746170544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x564953488830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb673fff640 1 -- 172.21.15.62:0/746170544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 msgr2=0x564953488830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb673fff640 1 --2- 172.21.15.62:0/746170544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x564953488830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb673fff640 1 -- 172.21.15.62:0/746170544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 msgr2=0x56495346d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb673fff640 1 --2- 172.21.15.62:0/746170544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x56495346d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb673fff640 1 -- 172.21.15.62:0/746170544 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56495346f1b0 con 0x564953468060 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb678afe640 1 --2- 172.21.15.62:0/746170544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x564953488830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb673fff640 1 --2- 172.21.15.62:0/746170544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534882f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb6680314e0 tx=0x7fb668000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=375603d402d35357 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb65a7fc640 1 -- 172.21.15.62:0/746170544 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668035070 con 0x564953468060 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb65a7fc640 1 -- 172.21.15.62:0/746170544 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6680035f0 con 0x564953468060 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.205+0000 7fb65a7fc640 1 -- 172.21.15.62:0/746170544 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668050070 con 0x564953468060 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 -- 172.21.15.62:0/746170544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 msgr2=0x5649534882f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 --2- 172.21.15.62:0/746170544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534882f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb6680314e0 tx=0x7fb668000bc0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 -- 172.21.15.62:0/746170544 shutdown_connections 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 --2- 172.21.15.62:0/746170544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x56495346d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 --2- 172.21.15.62:0/746170544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x564953488830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 --2- 172.21.15.62:0/746170544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534882f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 -- 172.21.15.62:0/746170544 >> 172.21.15.62:0/746170544 conn(0x5649534645f0 msgr2=0x564953471a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 -- 172.21.15.62:0/746170544 shutdown_connections 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 -- 172.21.15.62:0/746170544 wait complete. 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 -- start start 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.206+0000 7fb67d13b740 1 Processor -- start 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534e3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x5649534e4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb67d13b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x5649534e8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb67d13b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb6741ac150 con 0x564953468060 2024-09-15T23:17:00.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb67d13b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb6741ad5c0 con 0x564953473240 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb67d13b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb6741ac820 con 0x564953468980 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb67d13b740 10 client.0 _ll_register_callbacks cb 0x7fb668004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb673fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534e3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb673fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534e3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49716/0 (socket says 172.21.15.62:49716) 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb673fff640 1 -- 172.21.15.62:0/4294762592 learned_addr learned my addr 172.21.15.62:0/4294762592 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb678afe640 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x5649534e4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb673fff640 1 -- 172.21.15.62:0/4294762592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 msgr2=0x5649534e4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb673fff640 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x5649534e4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb673fff640 1 -- 172.21.15.62:0/4294762592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 msgr2=0x5649534e8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb673fff640 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x5649534e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.207+0000 7fb673fff640 1 -- 172.21.15.62:0/4294762592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649534e89a0 con 0x564953468060 2024-09-15T23:17:00.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb673fff640 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534e3d40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb66802a9a0 tx=0x7fb668031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299227dacea50299 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb653fff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:17:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb653fff640 1 -- 172.21.15.62:0/4294762592 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb668062060 con 0x564953468060 2024-09-15T23:17:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb653fff640 1 -- 172.21.15.62:0/4294762592 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb6680037a0 con 0x564953468060 2024-09-15T23:17:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb67d13b740 1 -- 172.21.15.62:0/4294762592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6680035f0 con 0x564953468060 2024-09-15T23:17:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb653fff640 1 -- 172.21.15.62:0/4294762592 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb668035070 con 0x564953468060 2024-09-15T23:17:00.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb67d13b740 10 client.15663 fetch_fsmap learned FSMap version 26 2024-09-15T23:17:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb67d13b740 1 -- 172.21.15.62:0/4294762592 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5649534f0ee0 con 0x564953468060 2024-09-15T23:17:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb653fff640 1 -- 172.21.15.62:0/4294762592 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6680035f0 con 0x564953468060 2024-09-15T23:17:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb653fff640 1 -- 172.21.15.62:0/4294762592 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fb66803d640 con 0x564953468060 2024-09-15T23:17:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb67d13b740 10 client.15663 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:17:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb67d13b740 10 client.15663 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:17:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb67d13b740 20 client.15663 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:00.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb67d13b740 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564953482ce0 0x5649534f3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:00.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb67d13b740 4 client.15663 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "10fd8a97", "release": true, "await": true}] 2024-09-15T23:17:00.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.208+0000 7fb67d13b740 1 -- 172.21.15.62:0/4294762592 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "10fd8a97", "release": true, "await": true}) -- 0x56495345f950 con 0x564953482ce0 2024-09-15T23:17:00.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.209+0000 7fb678afe640 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564953482ce0 0x5649534f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:00.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.209+0000 7fb653fff640 10 client.15663 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:00.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.209+0000 7fb678afe640 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564953482ce0 0x5649534f3370 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9815243d47aa59d2 server_cookie=4c3ee194379d4596 in_seq=0 out_seq=0 2024-09-15T23:17:00.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.255+0000 7fb653fff640 1 -- 172.21.15.62:0/4294762592 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x56495345f950 con 0x564953482ce0 2024-09-15T23:17:00.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.255+0000 7fb653fff640 10 client.15663 handle_command_reply: tid=0 2024-09-15T23:17:00.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.255+0000 7fb67d13b740 1 client.15663 shutdown 2024-09-15T23:17:00.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 -- 172.21.15.62:0/4294762592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 msgr2=0x5649534e3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:00.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534e3d40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb66802a9a0 tx=0x7fb668031b10 comp rx=0 tx=0).stop 2024-09-15T23:17:00.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 -- 172.21.15.62:0/4294762592 shutdown_connections 2024-09-15T23:17:00.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564953482ce0 0x5649534f3370 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564953473240 0x5649534e8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564953468980 0x5649534e4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb653fff640 0 client.15663 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:00.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 --2- 172.21.15.62:0/4294762592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564953468060 0x5649534e3d40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 -- 172.21.15.62:0/4294762592 >> 172.21.15.62:0/4294762592 conn(0x5649534645f0 msgr2=0x564953462f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:00.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 -- 172.21.15.62:0/4294762592 shutdown_connections 2024-09-15T23:17:00.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 1 -- 172.21.15.62:0/4294762592 wait complete. 2024-09-15T23:17:00.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.256+0000 7fb67d13b740 20 client.15663 trim_cache size 0 max 16384 2024-09-15T23:17:00.259 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:17:00.259 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 26, 2024-09-15T23:17:00.259 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:17:00.259 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 4, 2024-09-15T23:17:00.259 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "10fd8a97": { 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 4, 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 150.2, 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:17:00.260 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.257+0000 7fb67ad62640 1 -- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb65c0a32c0 msgr2=0x7fb65c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:00.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.257+0000 7fb67ad62640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb65c0a32c0 0x7fb65c0a5780 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb66c09c190 tx=0x7fb66c066c40 comp rx=0 tx=0).stop 2024-09-15T23:17:00.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.257+0000 7fb67ad62640 1 -- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102ef0 msgr2=0x7fb67419bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:00.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.257+0000 7fb67ad62640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102ef0 0x7fb67419bcb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00cad0 tx=0x7fb66c00cf90 comp rx=0 tx=0).stop 2024-09-15T23:17:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.260+0000 7fb67ad62640 1 -- 172.21.15.62:0/552047811 shutdown_connections 2024-09-15T23:17:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.260+0000 7fb67ad62640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb65c0a32c0 0x7fb65c0a5780 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.260+0000 7fb67ad62640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb674102ef0 0x7fb67419bcb0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.260+0000 7fb67ad62640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb674102570 0x7fb674197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.260+0000 7fb67ad62640 1 --2- 172.21.15.62:0/552047811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb674100e20 0x7fb6741973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.260+0000 7fb67ad62640 1 -- 172.21.15.62:0/552047811 >> 172.21.15.62:0/552047811 conn(0x7fb6740fc7d0 msgr2=0x7fb6741068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.262+0000 7fb67ad62640 1 -- 172.21.15.62:0/552047811 shutdown_connections 2024-09-15T23:17:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:00.262+0000 7fb67ad62640 1 -- 172.21.15.62:0/552047811 wait complete. 2024-09-15T23:17:00.952 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '10fd8a97', total seconds elapsed: 33.5 2024-09-15T23:17:01.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:01 smithi062 ceph-mon[30949]: pgmap v392: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 233 op/s 2024-09-15T23:17:01.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:01 smithi067 ceph-mon[35665]: pgmap v392: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 233 op/s 2024-09-15T23:17:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:01 smithi148 ceph-mon[35610]: pgmap v392: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 MiB/s wr, 233 op/s 2024-09-15T23:17:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:03 smithi067 ceph-mon[35665]: pgmap v393: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 20 MiB/s wr, 189 op/s 2024-09-15T23:17:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:03 smithi148 ceph-mon[35610]: pgmap v393: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 20 MiB/s wr, 189 op/s 2024-09-15T23:17:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:03 smithi062 ceph-mon[30949]: pgmap v393: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 20 MiB/s wr, 189 op/s 2024-09-15T23:17:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 45.0 s, 2876.3 tps, lat 225.443 ms stddev 2339.174 2024-09-15T23:17:05.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:05 smithi148 ceph-mon[35610]: pgmap v394: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 13 MiB/s wr, 198 op/s 2024-09-15T23:17:05.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:05 smithi062 ceph-mon[30949]: pgmap v394: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 13 MiB/s wr, 198 op/s 2024-09-15T23:17:05.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:05 smithi067 ceph-mon[35665]: pgmap v394: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 13 MiB/s wr, 198 op/s 2024-09-15T23:17:07.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:07 smithi148 ceph-mon[35610]: pgmap v395: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7.4 MiB/s wr, 255 op/s 2024-09-15T23:17:07.389 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:07 smithi062 ceph-mon[30949]: pgmap v395: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7.4 MiB/s wr, 255 op/s 2024-09-15T23:17:07.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:07 smithi067 ceph-mon[35665]: pgmap v395: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 7.4 MiB/s wr, 255 op/s 2024-09-15T23:17:09.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:09 smithi148 ceph-mon[35610]: pgmap v396: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.1 MiB/s wr, 327 op/s 2024-09-15T23:17:09.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:09 smithi062 ceph-mon[30949]: pgmap v396: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.1 MiB/s wr, 327 op/s 2024-09-15T23:17:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:09 smithi067 ceph-mon[35665]: pgmap v396: 129 pgs: 129 active+clean; 944 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 4.1 MiB/s wr, 327 op/s 2024-09-15T23:17:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 50.0 s, 3078.8 tps, lat 32.458 ms stddev 33.153 2024-09-15T23:17:11.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:11 smithi148 ceph-mon[35610]: pgmap v397: 129 pgs: 129 active+clean; 944 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 4.3 MiB/s wr, 424 op/s 2024-09-15T23:17:11.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:11 smithi062 ceph-mon[30949]: pgmap v397: 129 pgs: 129 active+clean; 944 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 4.3 MiB/s wr, 424 op/s 2024-09-15T23:17:11.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:11 smithi067 ceph-mon[35665]: pgmap v397: 129 pgs: 129 active+clean; 944 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 4.3 MiB/s wr, 424 op/s 2024-09-15T23:17:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.565+0000 7f2e48974640 1 -- 172.21.15.62:0/1188026165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e44101f20 msgr2=0x7f2e44109d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.565+0000 7f2e48974640 1 --2- 172.21.15.62:0/1188026165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e44101f20 0x7f2e44109d90 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009ca0 tx=0x7f2e3c02a8b0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.565+0000 7f2e48974640 1 -- 172.21.15.62:0/1188026165 shutdown_connections 2024-09-15T23:17:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.565+0000 7f2e48974640 1 --2- 172.21.15.62:0/1188026165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e44101f20 0x7f2e44109d90 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.565+0000 7f2e48974640 1 --2- 172.21.15.62:0/1188026165 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e44101560 0x7f2e441019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.565+0000 7f2e48974640 1 --2- 172.21.15.62:0/1188026165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e44103030 0x7f2e44103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.565+0000 7f2e48974640 1 -- 172.21.15.62:0/1188026165 >> 172.21.15.62:0/1188026165 conn(0x7f2e440fd0d0 msgr2=0x7f2e440ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.567+0000 7f2e48974640 1 -- 172.21.15.62:0/1188026165 shutdown_connections 2024-09-15T23:17:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.568+0000 7f2e48974640 1 -- 172.21.15.62:0/1188026165 wait complete. 2024-09-15T23:17:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.569+0000 7f2e48974640 1 Processor -- start 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.570+0000 7f2e48974640 1 -- start start 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.571+0000 7f2e48974640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e44101560 0x7f2e4419e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.571+0000 7f2e48974640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e44101f20 0x7f2e4419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.571+0000 7f2e48974640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e44103030 0x7f2e441a5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.571+0000 7f2e48974640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e440761f0 con 0x7f2e44103030 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.571+0000 7f2e48974640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e44076070 con 0x7f2e44101f20 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.571+0000 7f2e48974640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2e44076370 con 0x7f2e44101560 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.572+0000 7f2e42ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e44101f20 0x7f2e4419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.572+0000 7f2e42ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e44101f20 0x7f2e4419e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:55682/0 (socket says 172.21.15.62:55682) 2024-09-15T23:17:11.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.572+0000 7f2e42ffd640 1 -- 172.21.15.62:0/1818649949 learned_addr learned my addr 172.21.15.62:0/1818649949 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.572+0000 7f2e437fe640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e44101560 0x7f2e4419e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.573+0000 7f2e437fe640 1 -- 172.21.15.62:0/1818649949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e44101f20 msgr2=0x7f2e4419e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.573+0000 7f2e437fe640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e44101f20 0x7f2e4419e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.573+0000 7f2e437fe640 1 -- 172.21.15.62:0/1818649949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e44103030 msgr2=0x7f2e441a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.573+0000 7f2e437fe640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e44103030 0x7f2e441a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.573+0000 7f2e437fe640 1 -- 172.21.15.62:0/1818649949 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4410fe70 con 0x7f2e44101560 2024-09-15T23:17:11.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.576+0000 7f2e437fe640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e44101560 0x7f2e4419e2b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400e9c0 tx=0x7f2e3400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.577+0000 7f2e40ff9640 1 -- 172.21.15.62:0/1818649949 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3400ccc0 con 0x7f2e44101560 2024-09-15T23:17:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.577+0000 7f2e48974640 1 -- 172.21.15.62:0/1818649949 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e44110160 con 0x7f2e44101560 2024-09-15T23:17:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.577+0000 7f2e40ff9640 1 -- 172.21.15.62:0/1818649949 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e340035f0 con 0x7f2e44101560 2024-09-15T23:17:11.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.577+0000 7f2e40ff9640 1 -- 172.21.15.62:0/1818649949 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3400ce60 con 0x7f2e44101560 2024-09-15T23:17:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.577+0000 7f2e48974640 1 -- 172.21.15.62:0/1818649949 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e441106b0 con 0x7f2e44101560 2024-09-15T23:17:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.582+0000 7f2e4ca53740 1 Processor -- start 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.588+0000 7f2e40ff9640 1 -- 172.21.15.62:0/1818649949 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2e34003b10 con 0x7f2e44101560 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.589+0000 7f2e40ff9640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e2c0a3450 0x7f2e2c0a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.589+0000 7f2e40ff9640 1 -- 172.21.15.62:0/1818649949 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2e340cffe0 con 0x7f2e44101560 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.589+0000 7f2e4ca53740 1 -- start start 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.590+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a4b70 0x5616376a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.590+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a54b0 0x5616376a9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.590+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376aa540 0x5616376aca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.590+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5616376082f0 con 0x5616376a54b0 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.590+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5616375b0270 con 0x5616376a4b70 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.590+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5616376042b0 con 0x5616376aa540 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.590+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a4b70 0x5616376a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.590+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a4b70 0x5616376a4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:55694/0 (socket says 172.21.15.62:55694) 2024-09-15T23:17:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.590+0000 7f2e437fe640 1 -- 172.21.15.62:0/2882230159 learned_addr learned my addr 172.21.15.62:0/2882230159 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:11.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.591+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/2882230159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a54b0 0x5616376a9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.591+0000 7f2e42ffd640 1 -- 172.21.15.62:0/2882230159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376aa540 msgr2=0x5616376aca00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:17:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.591+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/2882230159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376aa540 0x5616376aca00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.591+0000 7f2e42ffd640 1 -- 172.21.15.62:0/2882230159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a4b70 msgr2=0x5616376a4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.591+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/2882230159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a4b70 0x5616376a4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.591+0000 7f2e42ffd640 1 -- 172.21.15.62:0/2882230159 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616375af340 con 0x5616376a54b0 2024-09-15T23:17:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.592+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/2882230159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a54b0 0x5616376a9f70 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2e4419f7c0 tx=0x7f2e38011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c1d69ecedd2a0c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.593+0000 7f2e43fff640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e2c0a3450 0x7f2e2c0a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.593+0000 7f2e29ffb640 1 -- 172.21.15.62:0/2882230159 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e38004230 con 0x5616376a54b0 2024-09-15T23:17:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.593+0000 7f2e29ffb640 1 -- 172.21.15.62:0/2882230159 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e380043d0 con 0x5616376a54b0 2024-09-15T23:17:11.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.594+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2882230159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a54b0 msgr2=0x5616376a9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.594+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/2882230159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a54b0 0x5616376a9f70 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2e4419f7c0 tx=0x7f2e38011810 comp rx=0 tx=0).stop 2024-09-15T23:17:11.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.596+0000 7f2e43fff640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e2c0a3450 0x7f2e2c0a5910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c004af0 tx=0x7f2e3c005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.598+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2882230159 shutdown_connections 2024-09-15T23:17:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.598+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/2882230159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376aa540 0x5616376aca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.598+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/2882230159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a54b0 0x5616376a9f70 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.598+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/2882230159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a4b70 0x5616376a4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.598+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2882230159 >> 172.21.15.62:0/2882230159 conn(0x5616376a01e0 msgr2=0x5616376a05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:11.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.601+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2882230159 shutdown_connections 2024-09-15T23:17:11.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.602+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2882230159 wait complete. 2024-09-15T23:17:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.602+0000 7f2e4ca53740 1 -- start start 2024-09-15T23:17:11.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.603+0000 7f2e4ca53740 1 Processor -- start 2024-09-15T23:17:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.604+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a4b70 0x5616376aed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.605+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a54b0 0x5616376af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.605+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376aa540 0x5616376b6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.605+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5616376ad560 con 0x5616376aa540 2024-09-15T23:17:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.605+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561637591290 con 0x5616376a54b0 2024-09-15T23:17:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.605+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5616376ad6e0 con 0x5616376a4b70 2024-09-15T23:17:11.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.605+0000 7f2e4ca53740 10 client.0 _ll_register_callbacks cb 0x5616375c2140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:11.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.605+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a4b70 0x5616376aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.606+0000 7f2e437fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a4b70 0x5616376aed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36044/0 (socket says 172.21.15.62:36044) 2024-09-15T23:17:11.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.606+0000 7f2e437fe640 1 -- 172.21.15.62:0/3001763206 learned_addr learned my addr 172.21.15.62:0/3001763206 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.607+0000 7f2e437fe640 1 -- 172.21.15.62:0/3001763206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a54b0 msgr2=0x5616376af2c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:17:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.607+0000 7f2e437fe640 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a54b0 0x5616376af2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.607+0000 7f2e437fe640 1 -- 172.21.15.62:0/3001763206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376aa540 msgr2=0x5616376b6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.607+0000 7f2e437fe640 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376aa540 0x5616376b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.608 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.607+0000 7f2e437fe640 1 -- 172.21.15.62:0/3001763206 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56163731b210 con 0x5616376a4b70 2024-09-15T23:17:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.607+0000 7f2e437fe640 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a4b70 0x5616376aed80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c6a0 tx=0x7f2e3409c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8ed7ed23c8fcf5a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.607+0000 7f2e22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:17:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.608+0000 7f2e22ffd640 1 -- 172.21.15.62:0/3001763206 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e340b4020 con 0x5616376a4b70 2024-09-15T23:17:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.608+0000 7f2e22ffd640 1 -- 172.21.15.62:0/3001763206 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2e340bc0d0 con 0x5616376a4b70 2024-09-15T23:17:11.609 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.608+0000 7f2e4ca53740 1 -- 172.21.15.62:0/3001763206 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e380043d0 con 0x5616376a4b70 2024-09-15T23:17:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.609+0000 7f2e22ffd640 1 -- 172.21.15.62:0/3001763206 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e340bae60 con 0x5616376a4b70 2024-09-15T23:17:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.609+0000 7f2e4ca53740 10 client.25216 fetch_fsmap learned FSMap version 26 2024-09-15T23:17:11.610 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.609+0000 7f2e4ca53740 1 -- 172.21.15.62:0/3001763206 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5616373e7620 con 0x5616376a4b70 2024-09-15T23:17:11.611 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.609+0000 7f2e22ffd640 1 -- 172.21.15.62:0/3001763206 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e380043d0 con 0x5616376a4b70 2024-09-15T23:17:11.611 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.609+0000 7f2e22ffd640 1 -- 172.21.15.62:0/3001763206 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f2e340cdbc0 con 0x5616376a4b70 2024-09-15T23:17:11.611 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.610+0000 7f2e4ca53740 10 client.25216 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:17:11.611 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.610+0000 7f2e4ca53740 10 client.25216 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:11.611 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.610+0000 7f2e4ca53740 20 client.25216 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:11.611 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.610+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616376baaf0 0x5616376bcf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.612 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.610+0000 7f2e4ca53740 4 client.25216 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:11.612 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.610+0000 7f2e4ca53740 1 -- 172.21.15.62:0/3001763206 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561637615a20 con 0x5616376baaf0 2024-09-15T23:17:11.615 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.614+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616376baaf0 0x5616376bcf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.615+0000 7f2e22ffd640 10 client.25216 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:11.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.615+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616376baaf0 0x5616376bcf70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b936e6d6a205217 server_cookie=1cb4923233027932 in_seq=0 out_seq=0 2024-09-15T23:17:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.616+0000 7f2e22ffd640 1 -- 172.21.15.62:0/3001763206 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561637615a20 con 0x5616376baaf0 2024-09-15T23:17:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.616+0000 7f2e22ffd640 10 client.25216 handle_command_reply: tid=0 2024-09-15T23:17:11.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.616+0000 7f2e4ca53740 1 client.25216 shutdown 2024-09-15T23:17:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 -- 172.21.15.62:0/3001763206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a4b70 msgr2=0x5616376aed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a4b70 0x5616376aed80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400c6a0 tx=0x7f2e3409c2e0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 -- 172.21.15.62:0/3001763206 shutdown_connections 2024-09-15T23:17:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616376baaf0 0x5616376bcf70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376aa540 0x5616376b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376a54b0 0x5616376af2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/3001763206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a4b70 0x5616376aed80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 -- 172.21.15.62:0/3001763206 >> 172.21.15.62:0/3001763206 conn(0x5616376a01e0 msgr2=0x5616376a19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e22ffd640 0 client.25216 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 -- 172.21.15.62:0/3001763206 shutdown_connections 2024-09-15T23:17:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 1 -- 172.21.15.62:0/3001763206 wait complete. 2024-09-15T23:17:11.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.618+0000 7f2e4ca53740 20 client.25216 trim_cache size 0 max 16384 2024-09-15T23:17:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.644+0000 7f2e4ca53740 1 Processor -- start 2024-09-15T23:17:11.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.644+0000 7f2e4ca53740 1 -- start start 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.644+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 0x5616376af350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.644+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.644+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.645+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e300124f0 con 0x5616376a4b70 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.645+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3000fdc0 con 0x5616376aa540 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.645+0000 7f2e42ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.645+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3000dc10 con 0x5616376a54b0 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.645+0000 7f2e42ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36054/0 (socket says 172.21.15.62:36054) 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.645+0000 7f2e42ffd640 1 -- 172.21.15.62:0/2386213722 learned_addr learned my addr 172.21.15.62:0/2386213722 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.645+0000 7f2e43fff640 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.645+0000 7f2e437fe640 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 0x5616376af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.646+0000 7f2e43fff640 1 -- 172.21.15.62:0/2386213722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 msgr2=0x5616376b6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.646+0000 7f2e43fff640 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.646+0000 7f2e43fff640 1 -- 172.21.15.62:0/2386213722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 msgr2=0x5616376af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.646+0000 7f2e43fff640 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 0x5616376af350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.646+0000 7f2e43fff640 1 -- 172.21.15.62:0/2386213722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616376ae8c0 con 0x5616376aa540 2024-09-15T23:17:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.646+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.646+0000 7f2e43fff640 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376a05c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c02aea0 tx=0x7f2e3c03d070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=527cfa66cc4471fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:11.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.646+0000 7f2e297fa640 1 -- 172.21.15.62:0/2386213722 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c030c30 con 0x5616376aa540 2024-09-15T23:17:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.647+0000 7f2e297fa640 1 -- 172.21.15.62:0/2386213722 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e3c036050 con 0x5616376aa540 2024-09-15T23:17:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.647+0000 7f2e297fa640 1 -- 172.21.15.62:0/2386213722 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c004260 con 0x5616376aa540 2024-09-15T23:17:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.648+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2386213722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 msgr2=0x5616376a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:11.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.648+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376a05c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c02aea0 tx=0x7f2e3c03d070 comp rx=0 tx=0).stop 2024-09-15T23:17:11.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.649+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2386213722 shutdown_connections 2024-09-15T23:17:11.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.649+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376a05c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.649+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.649+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/2386213722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 0x5616376af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.649+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2386213722 >> 172.21.15.62:0/2386213722 conn(0x5616376a01e0 msgr2=0x5616376a1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:11.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.649+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2386213722 shutdown_connections 2024-09-15T23:17:11.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.649+0000 7f2e4ca53740 1 -- 172.21.15.62:0/2386213722 wait complete. 2024-09-15T23:17:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.650+0000 7f2e4ca53740 1 -- start start 2024-09-15T23:17:11.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.650+0000 7f2e4ca53740 1 Processor -- start 2024-09-15T23:17:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.650+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 0x5616376b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.650+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.650+0000 7f2e4ca53740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.650+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3000e280 con 0x5616376a4b70 2024-09-15T23:17:11.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.650+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3000f6f0 con 0x5616376aa540 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.650+0000 7f2e4ca53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3000e950 con 0x5616376a54b0 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.651+0000 7f2e42ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.651+0000 7f2e42ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36062/0 (socket says 172.21.15.62:36062) 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.651+0000 7f2e42ffd640 1 -- 172.21.15.62:0/1560810019 learned_addr learned my addr 172.21.15.62:0/1560810019 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.651+0000 7f2e4ca53740 10 client.0 _ll_register_callbacks cb 0x7f2e3409bff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.651+0000 7f2e437fe640 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 0x5616376b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.651+0000 7f2e43fff640 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e42ffd640 1 -- 172.21.15.62:0/1560810019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 msgr2=0x5616376b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e42ffd640 1 -- 172.21.15.62:0/1560810019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 msgr2=0x5616376b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 0x5616376b40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e42ffd640 1 -- 172.21.15.62:0/1560810019 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616376bd5f0 con 0x5616376a54b0 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e437fe640 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 0x5616376b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:17:11.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e43fff640 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:17:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e42ffd640 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b4620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5616376a5ee0 tx=0x7f2e38009860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2018b091e9a7ef8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e22ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:17:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e22ffd640 1 -- 172.21.15.62:0/1560810019 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3801e040 con 0x5616376a54b0 2024-09-15T23:17:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.652+0000 7f2e22ffd640 1 -- 172.21.15.62:0/1560810019 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2e38005410 con 0x5616376a54b0 2024-09-15T23:17:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.653+0000 7f2e4ca53740 1 -- 172.21.15.62:0/1560810019 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e3c036050 con 0x5616376a54b0 2024-09-15T23:17:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.654+0000 7f2e22ffd640 1 -- 172.21.15.62:0/1560810019 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e380035f0 con 0x5616376a54b0 2024-09-15T23:17:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.654+0000 7f2e4ca53740 10 client.25222 fetch_fsmap learned FSMap version 26 2024-09-15T23:17:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.654+0000 7f2e4ca53740 1 -- 172.21.15.62:0/1560810019 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561637725db0 con 0x5616376a54b0 2024-09-15T23:17:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.654+0000 7f2e22ffd640 1 -- 172.21.15.62:0/1560810019 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561637615a20 con 0x5616376a54b0 2024-09-15T23:17:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.655+0000 7f2e22ffd640 1 -- 172.21.15.62:0/1560810019 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f2e3803a080 con 0x5616376a54b0 2024-09-15T23:17:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.655+0000 7f2e4ca53740 10 client.25222 fetch_fsmap finished waiting for FSMap version 26 2024-09-15T23:17:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.655+0000 7f2e4ca53740 10 client.25222 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.655+0000 7f2e4ca53740 20 client.25222 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.655+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561637726720 0x561637728b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.655+0000 7f2e4ca53740 4 client.25222 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:17:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.655+0000 7f2e4ca53740 1 -- 172.21.15.62:0/1560810019 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f2e3c036050 con 0x561637726720 2024-09-15T23:17:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.655+0000 7f2e437fe640 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561637726720 0x561637728b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.656+0000 7f2e437fe640 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561637726720 0x561637728b40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f21b2cfd63e48441 server_cookie=5f6fcfcacda85a2c in_seq=0 out_seq=0 2024-09-15T23:17:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.656+0000 7f2e22ffd640 10 client.25222 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.668+0000 7f2e22ffd640 1 -- 172.21.15.62:0/1560810019 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f2e3c036050 con 0x561637726720 2024-09-15T23:17:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.668+0000 7f2e22ffd640 10 client.25222 handle_command_reply: tid=0 2024-09-15T23:17:11.669 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 client.25222 shutdown 2024-09-15T23:17:11.670 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 -- 172.21.15.62:0/1560810019 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 msgr2=0x5616376b4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b4620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x5616376a5ee0 tx=0x7f2e38009860 comp rx=0 tx=0).stop 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 -- 172.21.15.62:0/1560810019 shutdown_connections 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561637726720 0x561637728b40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616376aa540 0x5616376b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616376a54b0 0x5616376b4620 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 --2- 172.21.15.62:0/1560810019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616376a4b70 0x5616376b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 -- 172.21.15.62:0/1560810019 >> 172.21.15.62:0/1560810019 conn(0x5616376a01e0 msgr2=0x5616376a1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e22ffd640 0 client.25222 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 -- 172.21.15.62:0/1560810019 shutdown_connections 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 1 -- 172.21.15.62:0/1560810019 wait complete. 2024-09-15T23:17:11.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.669+0000 7f2e4ca53740 20 client.25222 trim_cache size 0 max 16384 2024-09-15T23:17:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:17:11.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:17:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:17:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:17:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 -- 172.21.15.62:0/1818649949 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e2c0a3450 msgr2=0x7f2e2c0a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e2c0a3450 0x7f2e2c0a5910 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c004af0 tx=0x7f2e3c005ef0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 -- 172.21.15.62:0/1818649949 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e44101560 msgr2=0x7f2e4419e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e44101560 0x7f2e4419e2b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400e9c0 tx=0x7f2e3400ee80 comp rx=0 tx=0).stop 2024-09-15T23:17:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 -- 172.21.15.62:0/1818649949 shutdown_connections 2024-09-15T23:17:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e2c0a3450 0x7f2e2c0a5910 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e44103030 0x7f2e441a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e44101f20 0x7f2e4419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 --2- 172.21.15.62:0/1818649949 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e44101560 0x7f2e4419e2b0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400e9c0 tx=0x7f2e3400ee80 comp rx=0 tx=0).stop 2024-09-15T23:17:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.671+0000 7f2e48974640 1 -- 172.21.15.62:0/1818649949 >> 172.21.15.62:0/1818649949 conn(0x7f2e440fd0d0 msgr2=0x7f2e4410dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.672+0000 7f2e48974640 1 -- 172.21.15.62:0/1818649949 shutdown_connections 2024-09-15T23:17:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:11.672+0000 7f2e48974640 1 -- 172.21.15.62:0/1818649949 wait complete. 2024-09-15T23:17:13.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:13 smithi148 ceph-mon[35610]: pgmap v398: 129 pgs: 129 active+clean; 961 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.2 MiB/s wr, 582 op/s 2024-09-15T23:17:13.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:13 smithi062 ceph-mon[30949]: pgmap v398: 129 pgs: 129 active+clean; 961 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.2 MiB/s wr, 582 op/s 2024-09-15T23:17:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:13 smithi067 ceph-mon[35665]: pgmap v398: 129 pgs: 129 active+clean; 961 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 7.2 MiB/s wr, 582 op/s 2024-09-15T23:17:14.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:14 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:17:14.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:14 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:17:14.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:14 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:17:14.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 55.0 s, 2980.8 tps, lat 33.483 ms stddev 36.307 2024-09-15T23:17:15.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:15 smithi062 ceph-mon[30949]: pgmap v399: 129 pgs: 129 active+clean; 961 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 8.0 MiB/s wr, 674 op/s 2024-09-15T23:17:15.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:15 smithi067 ceph-mon[35665]: pgmap v399: 129 pgs: 129 active+clean; 961 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 8.0 MiB/s wr, 674 op/s 2024-09-15T23:17:15.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:15 smithi148 ceph-mon[35610]: pgmap v399: 129 pgs: 129 active+clean; 961 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 8.0 MiB/s wr, 674 op/s 2024-09-15T23:17:15.705 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:17:16.213 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:17.162 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:17 smithi062 ceph-mon[30949]: pgmap v400: 129 pgs: 129 active+clean; 961 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 8.4 MiB/s wr, 707 op/s 2024-09-15T23:17:17.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.298+0000 7f8703fff640 1 -- 172.21.15.62:0/1537243595 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc037070 con 0x7f870c109830 2024-09-15T23:17:17.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.298+0000 7f8713786640 1 -- 172.21.15.62:0/1537243595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f870c109830 msgr2=0x7f870c109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:17.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.298+0000 7f8713786640 1 --2- 172.21.15.62:0/1537243595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f870c109830 0x7f870c109c90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009860 tx=0x7f86fc02a380 comp rx=0 tx=0).stop 2024-09-15T23:17:17.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.298+0000 7f8713786640 1 -- 172.21.15.62:0/1537243595 shutdown_connections 2024-09-15T23:17:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.298+0000 7f8713786640 1 --2- 172.21.15.62:0/1537243595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f870c11b140 0x7f870c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.298+0000 7f8713786640 1 --2- 172.21.15.62:0/1537243595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f870c109830 0x7f870c109c90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.298+0000 7f8713786640 1 --2- 172.21.15.62:0/1537243595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f870c109450 0x7f870c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.298+0000 7f8713786640 1 -- 172.21.15.62:0/1537243595 >> 172.21.15.62:0/1537243595 conn(0x7f870c06e140 msgr2=0x7f870c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.299+0000 7f8713786640 1 -- 172.21.15.62:0/1537243595 shutdown_connections 2024-09-15T23:17:17.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.299+0000 7f8713786640 1 -- 172.21.15.62:0/1537243595 wait complete. 2024-09-15T23:17:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.300+0000 7f8713786640 1 Processor -- start 2024-09-15T23:17:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.300+0000 7f8713786640 1 -- start start 2024-09-15T23:17:17.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.300+0000 7f8713786640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f870c109450 0x7f870c1a7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.300+0000 7f8713786640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f870c109830 0x7f870c1a7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.300+0000 7f8713786640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f870c11b140 0x7f870c1abb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.300+0000 7f8713786640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f870c120e30 con 0x7f870c109830 2024-09-15T23:17:17.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.300+0000 7f8713786640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f870c120cb0 con 0x7f870c109450 2024-09-15T23:17:17.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.300+0000 7f8713786640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f870c120fb0 con 0x7f870c11b140 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.300+0000 7f8711522640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f870c109450 0x7f870c1a7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711d23640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f870c11b140 0x7f870c1abb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711522640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f870c109450 0x7f870c1a7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41738/0 (socket says 172.21.15.62:41738) 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711522640 1 -- 172.21.15.62:0/3346093788 learned_addr learned my addr 172.21.15.62:0/3346093788 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8710d21640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f870c109830 0x7f870c1a7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711522640 1 -- 172.21.15.62:0/3346093788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f870c11b140 msgr2=0x7f870c1abb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711522640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f870c11b140 0x7f870c1abb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711522640 1 -- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f870c109830 msgr2=0x7f870c1a7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711522640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f870c109830 0x7f870c1a7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711522640 1 -- 172.21.15.62:0/3346093788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870c1ac200 con 0x7f870c109450 2024-09-15T23:17:17.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711d23640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f870c11b140 0x7f870c1abb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:17.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f8711522640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f870c109450 0x7f870c1a7250 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f870800ebc0 tx=0x7f870800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:17.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.301+0000 7f87027fc640 1 -- 172.21.15.62:0/3346093788 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708010630 con 0x7f870c109450 2024-09-15T23:17:17.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.302+0000 7f87027fc640 1 -- 172.21.15.62:0/3346093788 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8708010cb0 con 0x7f870c109450 2024-09-15T23:17:17.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.303+0000 7f87027fc640 1 -- 172.21.15.62:0/3346093788 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708018bd0 con 0x7f870c109450 2024-09-15T23:17:17.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.303+0000 7f8713786640 1 -- 172.21.15.62:0/3346093788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f870c1ac4f0 con 0x7f870c109450 2024-09-15T23:17:17.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.303+0000 7f8713786640 1 -- 172.21.15.62:0/3346093788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f870c1b3dd0 con 0x7f870c109450 2024-09-15T23:17:17.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.306+0000 7f8715b5f740 1 Processor -- start 2024-09-15T23:17:17.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.307+0000 7f87027fc640 1 -- 172.21.15.62:0/3346093788 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f87080107d0 con 0x7f870c109450 2024-09-15T23:17:17.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.307+0000 7f8715b5f740 1 -- start start 2024-09-15T23:17:17.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.307+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fb8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.307+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fc3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.307+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.307+0000 7f8715b5f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556197f1e2e0 con 0x556197fb84a0 2024-09-15T23:17:17.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.307+0000 7f8715b5f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556197ec5ce0 con 0x556197fb8dc0 2024-09-15T23:17:17.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.307+0000 7f8715b5f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556197f1a2a0 con 0x556197fc3680 2024-09-15T23:17:17.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.307+0000 7f8711522640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fc3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.308+0000 7f8710d21640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.308+0000 7f8711522640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fc3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41746/0 (socket says 172.21.15.62:41746) 2024-09-15T23:17:17.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.308+0000 7f8711522640 1 -- 172.21.15.62:0/4102212049 learned_addr learned my addr 172.21.15.62:0/4102212049 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:17.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.308+0000 7f8711d23640 1 --2- 172.21.15.62:0/4102212049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fb8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.308+0000 7f87027fc640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f86f80a3190 0x7f86f80a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.309+0000 7f8710d21640 1 -- 172.21.15.62:0/4102212049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 msgr2=0x556197fc3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:17.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.309+0000 7f8710d21640 1 --2- 172.21.15.62:0/4102212049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fc3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.309+0000 7f8710d21640 1 -- 172.21.15.62:0/4102212049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 msgr2=0x556197fb8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:17.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.309+0000 7f8710d21640 1 --2- 172.21.15.62:0/4102212049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fb8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.309+0000 7f8710d21640 1 -- 172.21.15.62:0/4102212049 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556197eaa440 con 0x556197fc3680 2024-09-15T23:17:17.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.309+0000 7f8711522640 1 --2- 172.21.15.62:0/4102212049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fc3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:17.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.309+0000 7f87027fc640 1 -- 172.21.15.62:0/3346093788 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f87080d0190 con 0x7f870c109450 2024-09-15T23:17:17.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.309+0000 7f8711d23640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f86f80a3190 0x7f86f80a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.311+0000 7f8710d21640 1 --2- 172.21.15.62:0/4102212049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc5a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009830 tx=0x7f86fc009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a580b67cb4992c71 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:17.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.312+0000 7f8711d23640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f86f80a3190 0x7f86f80a5650 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8704005fd0 tx=0x7f87040046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:17.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.312+0000 7f86f77fe640 1 -- 172.21.15.62:0/4102212049 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc041070 con 0x556197fc3680 2024-09-15T23:17:17.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.312+0000 7f86f77fe640 1 -- 172.21.15.62:0/4102212049 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86fc005520 con 0x556197fc3680 2024-09-15T23:17:17.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.312+0000 7f86f77fe640 1 -- 172.21.15.62:0/4102212049 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc037040 con 0x556197fc3680 2024-09-15T23:17:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.313+0000 7f8715b5f740 1 -- 172.21.15.62:0/4102212049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 msgr2=0x556197fc5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.313+0000 7f8715b5f740 1 --2- 172.21.15.62:0/4102212049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc5a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009830 tx=0x7f86fc009540 comp rx=0 tx=0).stop 2024-09-15T23:17:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.313+0000 7f8715b5f740 1 -- 172.21.15.62:0/4102212049 shutdown_connections 2024-09-15T23:17:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.313+0000 7f8715b5f740 1 --2- 172.21.15.62:0/4102212049 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc5a70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.313+0000 7f8715b5f740 1 --2- 172.21.15.62:0/4102212049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fc3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.313+0000 7f8715b5f740 1 --2- 172.21.15.62:0/4102212049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fb8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.313+0000 7f8715b5f740 1 -- 172.21.15.62:0/4102212049 >> 172.21.15.62:0/4102212049 conn(0x556197fb4a30 msgr2=0x556197fb4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.313+0000 7f8715b5f740 1 -- 172.21.15.62:0/4102212049 shutdown_connections 2024-09-15T23:17:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.314+0000 7f8715b5f740 1 -- 172.21.15.62:0/4102212049 wait complete. 2024-09-15T23:17:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.314+0000 7f8715b5f740 1 -- start start 2024-09-15T23:17:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.314+0000 7f8715b5f740 1 Processor -- start 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb84a0 0x556197fc2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb8dc0 0x556197fc0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8715b5f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556197fc6250 con 0x556197fb8dc0 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8715b5f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556197ed9050 con 0x556197fb84a0 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8715b5f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556197fc63d0 con 0x556197fc3680 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8711d23640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8711d23640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34622/0 (socket says 172.21.15.62:34622) 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8711d23640 1 -- 172.21.15.62:0/2092875596 learned_addr learned my addr 172.21.15.62:0/2092875596 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8711522640 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb84a0 0x556197fc2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8715b5f740 10 client.0 _ll_register_callbacks cb 0x556197f19160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8711d23640 1 -- 172.21.15.62:0/2092875596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb84a0 msgr2=0x556197fc2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8711d23640 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb84a0 0x556197fc2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8711d23640 1 -- 172.21.15.62:0/2092875596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb8dc0 msgr2=0x556197fc0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:17.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8711d23640 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb8dc0 0x556197fc0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.315+0000 7f8711d23640 1 -- 172.21.15.62:0/2092875596 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556197c8f3c0 con 0x556197fc3680 2024-09-15T23:17:17.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f8711d23640 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc0770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8704009d10 tx=0x7f8704004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eec9f3e45e3c769 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:17.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f86f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:17:17.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f86f4ff9640 1 -- 172.21.15.62:0/2092875596 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870402c060 con 0x556197fc3680 2024-09-15T23:17:17.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f86f4ff9640 1 -- 172.21.15.62:0/2092875596 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f870401dd00 con 0x556197fc3680 2024-09-15T23:17:17.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f8715b5f740 1 -- 172.21.15.62:0/2092875596 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86fc005520 con 0x556197fc3680 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f86f4ff9640 1 -- 172.21.15.62:0/2092875596 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8704033050 con 0x556197fc3680 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f8715b5f740 10 client.25234 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f8715b5f740 1 -- 172.21.15.62:0/2092875596 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556197fd1c20 con 0x556197fc3680 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f86f4ff9640 1 -- 172.21.15.62:0/2092875596 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87040278d0 con 0x556197fc3680 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.316+0000 7f86f4ff9640 1 -- 172.21.15.62:0/2092875596 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f8704026070 con 0x556197fc3680 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.317+0000 7f8715b5f740 10 client.25234 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.317+0000 7f8715b5f740 10 client.25234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.317+0000 7f8715b5f740 20 client.25234 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.317+0000 7f8715b5f740 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556197fd3570 0x556197fd5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.317+0000 7f8715b5f740 4 client.25234 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.317+0000 7f8715b5f740 1 -- 172.21.15.62:0/2092875596 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f86fc005520 con 0x556197fd3570 2024-09-15T23:17:17.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.317+0000 7f8711522640 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556197fd3570 0x556197fd5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.317+0000 7f86f4ff9640 10 client.25234 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:17.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.317+0000 7f8711522640 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556197fd3570 0x556197fd5a30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf25ce6fb9941fb3 server_cookie=d16889190662d384 in_seq=0 out_seq=0 2024-09-15T23:17:17.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.320+0000 7f86f4ff9640 1 -- 172.21.15.62:0/2092875596 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f86fc005520 con 0x556197fd3570 2024-09-15T23:17:17.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.320+0000 7f86f4ff9640 10 client.25234 handle_command_reply: tid=0 2024-09-15T23:17:17.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.320+0000 7f8715b5f740 1 client.25234 shutdown 2024-09-15T23:17:17.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.320+0000 7f8715b5f740 1 -- 172.21.15.62:0/2092875596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 msgr2=0x556197fc0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:17.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.320+0000 7f8715b5f740 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc0770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8704009d10 tx=0x7f8704004730 comp rx=0 tx=0).stop 2024-09-15T23:17:17.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f8715b5f740 1 -- 172.21.15.62:0/2092875596 shutdown_connections 2024-09-15T23:17:17.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f8715b5f740 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556197fd3570 0x556197fd5a30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f8715b5f740 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fc3680 0x556197fc0770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f8715b5f740 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb8dc0 0x556197fc0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f8715b5f740 1 --2- 172.21.15.62:0/2092875596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb84a0 0x556197fc2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f8715b5f740 1 -- 172.21.15.62:0/2092875596 >> 172.21.15.62:0/2092875596 conn(0x556197fb4a30 msgr2=0x556197fba760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:17.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f86f4ff9640 0 client.25234 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:17.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f8715b5f740 1 -- 172.21.15.62:0/2092875596 shutdown_connections 2024-09-15T23:17:17.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f8715b5f740 1 -- 172.21.15.62:0/2092875596 wait complete. 2024-09-15T23:17:17.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.321+0000 7f8715b5f740 20 client.25234 trim_cache size 0 max 16384 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.349+0000 7f8715b5f740 1 Processor -- start 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.349+0000 7f8715b5f740 1 -- start start 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.349+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb4a30 0x556197fcf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.349+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fbeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.349+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fba650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8715b5f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f870c1c0060 con 0x556197fb84a0 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8715b5f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f870c1bd930 con 0x556197fb8dc0 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8715b5f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f870c1bb780 con 0x556197fb4a30 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8711d23640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fba650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8711d23640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fba650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41762/0 (socket says 172.21.15.62:41762) 2024-09-15T23:17:17.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8711d23640 1 -- 172.21.15.62:0/3260935781 learned_addr learned my addr 172.21.15.62:0/3260935781 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8711d23640 1 -- 172.21.15.62:0/3260935781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb4a30 msgr2=0x556197fcf4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:17:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8711d23640 1 --2- 172.21.15.62:0/3260935781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb4a30 0x556197fcf4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8711d23640 1 -- 172.21.15.62:0/3260935781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 msgr2=0x556197fbeb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:17:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8711d23640 1 --2- 172.21.15.62:0/3260935781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fbeb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.350+0000 7f8711d23640 1 -- 172.21.15.62:0/3260935781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556197fbf1e0 con 0x556197fb8dc0 2024-09-15T23:17:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.351+0000 7f8711d23640 1 --2- 172.21.15.62:0/3260935781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fba650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f870402a040 tx=0x7f870401d840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a7816d05a7def9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.351+0000 7f86f6ffd640 1 -- 172.21.15.62:0/3260935781 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870401b920 con 0x556197fb8dc0 2024-09-15T23:17:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.351+0000 7f86f6ffd640 1 -- 172.21.15.62:0/3260935781 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f870403c070 con 0x556197fb8dc0 2024-09-15T23:17:17.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.351+0000 7f86f6ffd640 1 -- 172.21.15.62:0/3260935781 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704037b60 con 0x556197fb8dc0 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.352+0000 7f8715b5f740 1 -- 172.21.15.62:0/3260935781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 msgr2=0x556197fba650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.352+0000 7f8715b5f740 1 --2- 172.21.15.62:0/3260935781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fba650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f870402a040 tx=0x7f870401d840 comp rx=0 tx=0).stop 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.352+0000 7f8715b5f740 1 -- 172.21.15.62:0/3260935781 shutdown_connections 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.352+0000 7f8715b5f740 1 --2- 172.21.15.62:0/3260935781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb8dc0 0x556197fba650 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.352+0000 7f8715b5f740 1 --2- 172.21.15.62:0/3260935781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fbeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.352+0000 7f8715b5f740 1 --2- 172.21.15.62:0/3260935781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb4a30 0x556197fcf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.352+0000 7f8715b5f740 1 -- 172.21.15.62:0/3260935781 >> 172.21.15.62:0/3260935781 conn(0x556197fc08d0 msgr2=0x556197fb9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.352+0000 7f8715b5f740 1 -- 172.21.15.62:0/3260935781 shutdown_connections 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.352+0000 7f8715b5f740 1 -- 172.21.15.62:0/3260935781 wait complete. 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8715b5f740 1 -- start start 2024-09-15T23:17:17.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8715b5f740 1 Processor -- start 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb4a30 0x556197fc7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fc7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8715b5f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb8dc0 0x55619803d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8715b5f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f870c1bbdf0 con 0x556197fb84a0 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8715b5f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f870c1bd260 con 0x556197fb4a30 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8715b5f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f870c1bc4c0 con 0x556197fb8dc0 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8715b5f740 10 client.0 _ll_register_callbacks cb 0x7f8704009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.353+0000 7f8710d21640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fc7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8710d21640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fc7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40770/0 (socket says 172.21.15.62:40770) 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8710d21640 1 -- 172.21.15.62:0/1205650687 learned_addr learned my addr 172.21.15.62:0/1205650687 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8711d23640 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb8dc0 0x55619803d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8711522640 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb4a30 0x556197fc7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8711d23640 1 -- 172.21.15.62:0/1205650687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb4a30 msgr2=0x556197fc7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8711d23640 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb4a30 0x556197fc7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8711d23640 1 -- 172.21.15.62:0/1205650687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 msgr2=0x556197fc7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8711d23640 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fc7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8711d23640 1 -- 172.21.15.62:0/1205650687 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55619803d850 con 0x556197fb8dc0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.354+0000 7f8711d23640 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb8dc0 0x55619803d310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8704005f70 tx=0x7f870401bb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c523b48741b69eba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.355+0000 7f86f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.355+0000 7f86f4ff9640 1 -- 172.21.15.62:0/1205650687 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870402c070 con 0x556197fb8dc0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.355+0000 7f86f4ff9640 1 -- 172.21.15.62:0/1205650687 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f87040377a0 con 0x556197fb8dc0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.355+0000 7f8715b5f740 1 -- 172.21.15.62:0/1205650687 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f870403c070 con 0x556197fb8dc0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.355+0000 7f86f4ff9640 1 -- 172.21.15.62:0/1205650687 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f87040379e0 con 0x556197fb8dc0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.355+0000 7f8715b5f740 10 client.25237 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.355+0000 7f8715b5f740 1 -- 172.21.15.62:0/1205650687 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55619803e9f0 con 0x556197fb8dc0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.356+0000 7f86f4ff9640 1 -- 172.21.15.62:0/1205650687 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8704037b80 con 0x556197fb8dc0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.356+0000 7f86f4ff9640 1 -- 172.21.15.62:0/1205650687 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f8704026040 con 0x556197fb8dc0 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.356+0000 7f8715b5f740 10 client.25237 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.356+0000 7f8715b5f740 10 client.25237 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.356+0000 7f8715b5f740 20 client.25237 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:17.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.356+0000 7f8715b5f740 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556197fc3680 0x556198041090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.356+0000 7f8715b5f740 4 client.25237 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.356+0000 7f8715b5f740 1 -- 172.21.15.62:0/1205650687 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f870403c070 con 0x556197fc3680 2024-09-15T23:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.356+0000 7f8711522640 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556197fc3680 0x556198041090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f86f4ff9640 10 client.25237 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8711522640 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556197fc3680 0x556198041090 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=896edcfef524b73a server_cookie=e069477a3e05cf97 in_seq=0 out_seq=0 2024-09-15T23:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f86f4ff9640 1 -- 172.21.15.62:0/1205650687 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f870403c070 con 0x556197fc3680 2024-09-15T23:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f86f4ff9640 10 client.25237 handle_command_reply: tid=0 2024-09-15T23:17:17.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8715b5f740 1 client.25237 shutdown 2024-09-15T23:17:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8715b5f740 1 -- 172.21.15.62:0/1205650687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb8dc0 msgr2=0x55619803d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8715b5f740 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb8dc0 0x55619803d310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8704005f70 tx=0x7f870401bb50 comp rx=0 tx=0).stop 2024-09-15T23:17:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8715b5f740 1 -- 172.21.15.62:0/1205650687 shutdown_connections 2024-09-15T23:17:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8715b5f740 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556197fc3680 0x556198041090 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8715b5f740 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556197fb8dc0 0x55619803d310 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8715b5f740 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556197fb84a0 0x556197fc7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8715b5f740 1 --2- 172.21.15.62:0/1205650687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556197fb4a30 0x556197fc7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f8715b5f740 1 -- 172.21.15.62:0/1205650687 >> 172.21.15.62:0/1205650687 conn(0x556197fc08d0 msgr2=0x556197fc22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:17.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.357+0000 7f86f4ff9640 0 client.25237 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:17.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.358+0000 7f8715b5f740 1 -- 172.21.15.62:0/1205650687 shutdown_connections 2024-09-15T23:17:17.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.358+0000 7f8715b5f740 1 -- 172.21.15.62:0/1205650687 wait complete. 2024-09-15T23:17:17.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.360+0000 7f8715b5f740 20 client.25237 trim_cache size 0 max 16384 2024-09-15T23:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:17:17.363 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:17:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 -- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f86f80a3190 msgr2=0x7f86f80a5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f86f80a3190 0x7f86f80a5650 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8704005fd0 tx=0x7f87040046c0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 -- 172.21.15.62:0/3346093788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f870c109450 msgr2=0x7f870c1a7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f870c109450 0x7f870c1a7250 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f870800ebc0 tx=0x7f870800c6a0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 -- 172.21.15.62:0/3346093788 shutdown_connections 2024-09-15T23:17:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f86f80a3190 0x7f86f80a5650 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f870c11b140 0x7f870c1abb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f870c109830 0x7f870c1a7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 --2- 172.21.15.62:0/3346093788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f870c109450 0x7f870c1a7250 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 -- 172.21.15.62:0/3346093788 >> 172.21.15.62:0/3346093788 conn(0x7f870c06e140 msgr2=0x7f870c11bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 -- 172.21.15.62:0/3346093788 shutdown_connections 2024-09-15T23:17:17.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:17.361+0000 7f8713786640 1 -- 172.21.15.62:0/3346093788 wait complete. 2024-09-15T23:17:17.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:17 smithi067 ceph-mon[35665]: pgmap v400: 129 pgs: 129 active+clean; 961 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 8.4 MiB/s wr, 707 op/s 2024-09-15T23:17:17.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:17 smithi148 ceph-mon[35610]: pgmap v400: 129 pgs: 129 active+clean; 961 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 8.4 MiB/s wr, 707 op/s 2024-09-15T23:17:18.939 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:17:18.939 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:17:19.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:19 smithi062 ceph-mon[30949]: pgmap v401: 129 pgs: 129 active+clean; 963 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 11 MiB/s wr, 819 op/s 2024-09-15T23:17:19.327 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:19 smithi067 ceph-mon[35665]: pgmap v401: 129 pgs: 129 active+clean; 963 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 11 MiB/s wr, 819 op/s 2024-09-15T23:17:19.340 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:19 smithi148 ceph-mon[35610]: pgmap v401: 129 pgs: 129 active+clean; 963 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 11 MiB/s wr, 819 op/s 2024-09-15T23:17:19.368 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:19.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 60.0 s, 2498.0 tps, lat 39.955 ms stddev 56.644 2024-09-15T23:17:19.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.907+0000 7f8bc8dff640 1 -- 172.21.15.62:0/3213626991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bc4102d30 msgr2=0x7f8bc4103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:19.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.907+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/3213626991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bc4102d30 0x7f8bc4103110 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8bb4009930 tx=0x7f8bb402a420 comp rx=0 tx=0).stop 2024-09-15T23:17:19.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.907+0000 7f8bc8dff640 1 -- 172.21.15.62:0/3213626991 shutdown_connections 2024-09-15T23:17:19.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.907+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/3213626991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc41082a0 0x7f8bc410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:19.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.907+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/3213626991 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bc41036e0 0x7f8bc4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:19.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.907+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/3213626991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bc4102d30 0x7f8bc4103110 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:19.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.907+0000 7f8bc8dff640 1 -- 172.21.15.62:0/3213626991 >> 172.21.15.62:0/3213626991 conn(0x7f8bc40fc7d0 msgr2=0x7f8bc40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:19.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.907+0000 7f8bc8dff640 1 -- 172.21.15.62:0/3213626991 shutdown_connections 2024-09-15T23:17:19.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.907+0000 7f8bc8dff640 1 -- 172.21.15.62:0/3213626991 wait complete. 2024-09-15T23:17:19.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.908+0000 7f8bc8dff640 1 Processor -- start 2024-09-15T23:17:19.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.908+0000 7f8bc8dff640 1 -- start start 2024-09-15T23:17:19.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.909+0000 7f8bc8dff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc4102d30 0x7f8bc41008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:19.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.909+0000 7f8bc8dff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bc41036e0 0x7f8bc4100e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:19.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.910+0000 7f8bc8dff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bc41082a0 0x7f8bc4101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:19.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.910+0000 7f8bc259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc4102d30 0x7f8bc41008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:19.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.910+0000 7f8bc8dff640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41120e0 con 0x7f8bc41036e0 2024-09-15T23:17:19.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.910+0000 7f8bc259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc4102d30 0x7f8bc41008c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41784/0 (socket says 172.21.15.62:41784) 2024-09-15T23:17:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.910+0000 7f8bc259c640 1 -- 172.21.15.62:0/1369131648 learned_addr learned my addr 172.21.15.62:0/1369131648 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.910+0000 7f8bc8dff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc4111f60 con 0x7f8bc4102d30 2024-09-15T23:17:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.910+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc4112260 con 0x7f8bc41082a0 2024-09-15T23:17:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.910+0000 7f8bc1d9b640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bc41036e0 0x7f8bc4100e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.910+0000 7f8bc259c640 1 -- 172.21.15.62:0/1369131648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bc41082a0 msgr2=0x7f8bc4101340 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:17:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bc259c640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bc41082a0 0x7f8bc4101340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:19.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bc259c640 1 -- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bc41036e0 msgr2=0x7f8bc4100e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bc259c640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bc41036e0 0x7f8bc4100e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bc259c640 1 -- 172.21.15.62:0/1369131648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc4072cb0 con 0x7f8bc4102d30 2024-09-15T23:17:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bc259c640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc4102d30 0x7f8bc41008c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8bb402a940 tx=0x7f8bb4003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bb37fe640 1 -- 172.21.15.62:0/1369131648 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb403f070 con 0x7f8bc4102d30 2024-09-15T23:17:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bb37fe640 1 -- 172.21.15.62:0/1369131648 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bb4031c90 con 0x7f8bc4102d30 2024-09-15T23:17:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc4072f40 con 0x7f8bc4102d30 2024-09-15T23:17:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bb37fe640 1 -- 172.21.15.62:0/1369131648 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb4035030 con 0x7f8bc4102d30 2024-09-15T23:17:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.911+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc4073470 con 0x7f8bc4102d30 2024-09-15T23:17:19.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.913+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc4103110 con 0x7f8bc4102d30 2024-09-15T23:17:19.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.915+0000 7f8bb37fe640 1 -- 172.21.15.62:0/1369131648 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8bb4042070 con 0x7f8bc4102d30 2024-09-15T23:17:19.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.915+0000 7f8bb37fe640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8bac0a32c0 0x7f8bac0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:19.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.916+0000 7f8bc1d9b640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8bac0a32c0 0x7f8bac0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:19.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.916+0000 7f8bb37fe640 1 -- 172.21.15.62:0/1369131648 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8bb40f02c0 con 0x7f8bc4102d30 2024-09-15T23:17:19.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.916+0000 7f8bc1d9b640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8bac0a32c0 0x7f8bac0a5780 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4102030 tx=0x7f8bb8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:19.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:19.920+0000 7f8bb37fe640 1 -- 172.21.15.62:0/1369131648 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8bb40bcc60 con 0x7f8bc4102d30 2024-09-15T23:17:20.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.294+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bc4101da0 con 0x7f8bc4102d30 2024-09-15T23:17:20.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.296+0000 7f8bb37fe640 1 -- 172.21.15.62:0/1369131648 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11906 (secure 0 0 0) 0x7f8bb40bc560 con 0x7f8bc4102d30 2024-09-15T23:17:20.299 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:17:20.299 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":27,"btime":"2024-09-15T23:17:13:035711+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:17:12.375630+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:17:20.299 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 27 2024-09-15T23:17:20.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8bac0a32c0 msgr2=0x7f8bac0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:20.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8bac0a32c0 0x7f8bac0a5780 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4102030 tx=0x7f8bb8005f90 comp rx=0 tx=0).stop 2024-09-15T23:17:20.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc4102d30 msgr2=0x7f8bc41008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:20.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc4102d30 0x7f8bc41008c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8bb402a940 tx=0x7f8bb4003b20 comp rx=0 tx=0).stop 2024-09-15T23:17:20.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 shutdown_connections 2024-09-15T23:17:20.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8bac0a32c0 0x7f8bac0a5780 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:20.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8bc41082a0 0x7f8bc4101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:20.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8bc41036e0 0x7f8bc4100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:20.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 --2- 172.21.15.62:0/1369131648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc4102d30 0x7f8bc41008c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:20.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 >> 172.21.15.62:0/1369131648 conn(0x7f8bc40fc7d0 msgr2=0x7f8bc4106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:20.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 shutdown_connections 2024-09-15T23:17:20.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:20.300+0000 7f8bc8dff640 1 -- 172.21.15.62:0/1369131648 wait complete. 2024-09-15T23:17:21.463 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:21 smithi062 ceph-mon[30949]: pgmap v402: 129 pgs: 129 active+clean; 965 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-09-15T23:17:21.463 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:21 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1369131648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:17:21.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:21 smithi067 ceph-mon[35665]: pgmap v402: 129 pgs: 129 active+clean; 965 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-09-15T23:17:21.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:21 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1369131648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:17:21.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:21 smithi148 ceph-mon[35610]: pgmap v402: 129 pgs: 129 active+clean; 965 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 12 MiB/s wr, 869 op/s 2024-09-15T23:17:21.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:21 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1369131648' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:17:21.929 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:17:22.351 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:22.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.897+0000 7f3804f88640 1 -- 172.21.15.62:0/253092602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3800101de0 msgr2=0x7f38001021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:22.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.897+0000 7f3804f88640 1 --2- 172.21.15.62:0/253092602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3800101de0 0x7f38001021c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009900 tx=0x7f37f002a410 comp rx=0 tx=0).stop 2024-09-15T23:17:22.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.898+0000 7f3804f88640 1 -- 172.21.15.62:0/253092602 shutdown_connections 2024-09-15T23:17:22.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.898+0000 7f3804f88640 1 --2- 172.21.15.62:0/253092602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800106f10 0x7f380010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.898+0000 7f3804f88640 1 --2- 172.21.15.62:0/253092602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3800102790 0x7f38001067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.898+0000 7f3804f88640 1 --2- 172.21.15.62:0/253092602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3800101de0 0x7f38001021c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.898+0000 7f3804f88640 1 -- 172.21.15.62:0/253092602 >> 172.21.15.62:0/253092602 conn(0x7f38000fd8e0 msgr2=0x7f38000ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:22.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.898+0000 7f3804f88640 1 -- 172.21.15.62:0/253092602 shutdown_connections 2024-09-15T23:17:22.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.898+0000 7f3804f88640 1 -- 172.21.15.62:0/253092602 wait complete. 2024-09-15T23:17:22.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.899+0000 7f3804f88640 1 Processor -- start 2024-09-15T23:17:22.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.899+0000 7f3804f88640 1 -- start start 2024-09-15T23:17:22.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f3804f88640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3800101de0 0x7f38001a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f3804f88640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3800102790 0x7f38001a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f3804f88640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800106f10 0x7f380019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f3804f88640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f380010fdd0 con 0x7f3800101de0 2024-09-15T23:17:22.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f3804f88640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f380010fc50 con 0x7f3800106f10 2024-09-15T23:17:22.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f3804f88640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f380010ff50 con 0x7f3800102790 2024-09-15T23:17:22.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fe59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3800101de0 0x7f38001a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fe59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3800101de0 0x7f38001a46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43172/0 (socket says 172.21.15.62:43172) 2024-09-15T23:17:22.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fe59c640 1 -- 172.21.15.62:0/2794655120 learned_addr learned my addr 172.21.15.62:0/2794655120 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:22.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fed9d640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800106f10 0x7f380019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3800102790 0x7f38001a4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fed9d640 1 -- 172.21.15.62:0/2794655120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3800102790 msgr2=0x7f38001a4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fed9d640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3800102790 0x7f38001a4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fed9d640 1 -- 172.21.15.62:0/2794655120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3800101de0 msgr2=0x7f38001a46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fed9d640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3800101de0 0x7f38001a46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.900+0000 7f37fed9d640 1 -- 172.21.15.62:0/2794655120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380019eff0 con 0x7f3800106f10 2024-09-15T23:17:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.901+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3800102790 0x7f38001a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:17:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.901+0000 7f37fed9d640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800106f10 0x7f380019e850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f37f800ec50 tx=0x7f37f800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.901+0000 7f37f77fe640 1 -- 172.21.15.62:0/2794655120 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8003b70 con 0x7f3800106f10 2024-09-15T23:17:22.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.901+0000 7f37f77fe640 1 -- 172.21.15.62:0/2794655120 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f8003d10 con 0x7f3800106f10 2024-09-15T23:17:22.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.901+0000 7f37f77fe640 1 -- 172.21.15.62:0/2794655120 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8010890 con 0x7f3800106f10 2024-09-15T23:17:22.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.901+0000 7f3804f88640 1 -- 172.21.15.62:0/2794655120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f380019f2e0 con 0x7f3800106f10 2024-09-15T23:17:22.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.901+0000 7f3804f88640 1 -- 172.21.15.62:0/2794655120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38001ab530 con 0x7f3800106f10 2024-09-15T23:17:22.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.906+0000 7f3807361740 1 Processor -- start 2024-09-15T23:17:22.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.906+0000 7f37f77fe640 1 -- 172.21.15.62:0/2794655120 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f37f8003fc0 con 0x7f3800106f10 2024-09-15T23:17:22.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f3807361740 1 -- start start 2024-09-15T23:17:22.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f3807361740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c466c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f3807361740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466c5d0 0x5556c466ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f3807361740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c4677370 0x5556c4679760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f3807361740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5556c45d22f0 con 0x5556c466c5d0 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f3807361740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5556c4579cf0 con 0x5556c466db40 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f3807361740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5556c45ce2b0 con 0x5556c4677370 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f37fe59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466c5d0 0x5556c466ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f37fdd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c466c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f37fdd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c466c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41812/0 (socket says 172.21.15.62:41812) 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.907+0000 7f37fdd9b640 1 -- 172.21.15.62:0/468983145 learned_addr learned my addr 172.21.15.62:0/468983145 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/468983145 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c4677370 0x5556c4679760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37fdd9b640 1 -- 172.21.15.62:0/468983145 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c4677370 msgr2=0x5556c4679760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/468983145 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c4677370 0x5556c4679760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37fdd9b640 1 -- 172.21.15.62:0/468983145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466c5d0 msgr2=0x5556c466ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/468983145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466c5d0 0x5556c466ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37fdd9b640 1 -- 172.21.15.62:0/468983145 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556c455e3a0 con 0x5556c466db40 2024-09-15T23:17:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37fe59c640 1 --2- 172.21.15.62:0/468983145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466c5d0 0x5556c466ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37f77fe640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f37ec0a3310 0x7f37ec0a57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/468983145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c466c090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f37e8013520 tx=0x7f37e8009790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1cdf060078c0372 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37fe59c640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f37ec0a3310 0x7f37ec0a57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37f77fe640 1 -- 172.21.15.62:0/2794655120 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f37f80d0230 con 0x7f3800106f10 2024-09-15T23:17:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37f4ff9640 1 -- 172.21.15.62:0/468983145 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8004020 con 0x5556c466db40 2024-09-15T23:17:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37f4ff9640 1 -- 172.21.15.62:0/468983145 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37e80041c0 con 0x5556c466db40 2024-09-15T23:17:22.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.908+0000 7f37f4ff9640 1 -- 172.21.15.62:0/468983145 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e800f700 con 0x5556c466db40 2024-09-15T23:17:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.909+0000 7f3807361740 1 -- 172.21.15.62:0/468983145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 msgr2=0x5556c466c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.909+0000 7f3807361740 1 --2- 172.21.15.62:0/468983145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c466c090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f37e8013520 tx=0x7f37e8009790 comp rx=0 tx=0).stop 2024-09-15T23:17:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.909+0000 7f3807361740 1 -- 172.21.15.62:0/468983145 shutdown_connections 2024-09-15T23:17:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.909+0000 7f3807361740 1 --2- 172.21.15.62:0/468983145 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c4677370 0x5556c4679760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.909+0000 7f3807361740 1 --2- 172.21.15.62:0/468983145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466c5d0 0x5556c466ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.909+0000 7f3807361740 1 --2- 172.21.15.62:0/468983145 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c466c090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.909+0000 7f3807361740 1 -- 172.21.15.62:0/468983145 >> 172.21.15.62:0/468983145 conn(0x5556c4635310 msgr2=0x5556c4668c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.910+0000 7f37fe59c640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f37ec0a3310 0x7f37ec0a57d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x5556c4669ac0 tx=0x7f37f002a910 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:22.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.910+0000 7f3807361740 1 -- 172.21.15.62:0/468983145 shutdown_connections 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.910+0000 7f3807361740 1 -- 172.21.15.62:0/468983145 wait complete. 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.911+0000 7f3807361740 1 -- start start 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.911+0000 7f3807361740 1 Processor -- start 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f3807361740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4676db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f3807361740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c4674060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f3807361740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c4677370 0x5556c46747f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f37fe59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4676db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f37fe59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4676db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56334/0 (socket says 172.21.15.62:56334) 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f37fdd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c4674060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f37fdd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c4674060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41828/0 (socket says 172.21.15.62:41828) 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f37fe59c640 1 -- 172.21.15.62:0/3920686161 learned_addr learned my addr 172.21.15.62:0/3920686161 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f37fed9d640 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c4677370 0x5556c46747f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.912+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5556c467a070 con 0x5556c4677370 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5556c458d060 con 0x5556c466db40 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5556c467a1f0 con 0x5556c466c5d0 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f3807361740 10 client.0 _ll_register_callbacks cb 0x5556c42e08c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f37fed9d640 1 -- 172.21.15.62:0/3920686161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 msgr2=0x5556c4676db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f37fed9d640 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4676db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f37fed9d640 1 -- 172.21.15.62:0/3920686161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 msgr2=0x5556c4674060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f37fed9d640 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c4674060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f37fed9d640 1 -- 172.21.15.62:0/3920686161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556c43433c0 con 0x5556c4677370 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c4674060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f37fed9d640 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c4677370 0x5556c46747f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f37f800cba0 tx=0x7f37f809c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478ebcc339dba95d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f37e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.913+0000 7f37e5ffb640 1 -- 172.21.15.62:0/3920686161 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f80b4070 con 0x5556c4677370 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f37e5ffb640 1 -- 172.21.15.62:0/3920686161 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f37f80b5460 con 0x5556c4677370 2024-09-15T23:17:22.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37e80041c0 con 0x5556c4677370 2024-09-15T23:17:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f37e5ffb640 1 -- 172.21.15.62:0/3920686161 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f37f809cbd0 con 0x5556c4677370 2024-09-15T23:17:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f3807361740 10 client.15705 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5556c4685b80 con 0x5556c4677370 2024-09-15T23:17:22.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f37e5ffb640 1 -- 172.21.15.62:0/3920686161 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37e80041c0 con 0x5556c4677370 2024-09-15T23:17:22.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f37e5ffb640 1 -- 172.21.15.62:0/3920686161 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f37f80a4cb0 con 0x5556c4677370 2024-09-15T23:17:22.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f3807361740 10 client.15705 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:22.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f3807361740 10 client.15705 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:17:22.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f3807361740 20 client.15705 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:22.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f3807361740 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5556c4687040 0x5556c46894b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.914+0000 7f3807361740 4 client.15705 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:22.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.915+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556c45df860 con 0x5556c4687040 2024-09-15T23:17:22.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.915+0000 7f37fe59c640 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5556c4687040 0x5556c46894b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.917+0000 7f37fe59c640 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5556c4687040 0x5556c46894b0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b57e396c0de1a493 server_cookie=f6a0af03840530a0 in_seq=0 out_seq=0 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.918+0000 7f37e5ffb640 10 client.15705 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.918+0000 7f37e5ffb640 1 -- 172.21.15.62:0/3920686161 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5556c45df860 con 0x5556c4687040 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.918+0000 7f37e5ffb640 10 client.15705 handle_command_reply: tid=0 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.918+0000 7f3807361740 1 client.15705 shutdown 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.918+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c4677370 msgr2=0x5556c46747f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.918+0000 7f3807361740 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c4677370 0x5556c46747f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f37f800cba0 tx=0x7f37f809c010 comp rx=0 tx=0).stop 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 shutdown_connections 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f3807361740 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5556c4687040 0x5556c46894b0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f37e5ffb640 0 client.15705 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f3807361740 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c4677370 0x5556c46747f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f3807361740 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c466db40 0x5556c4674060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f3807361740 1 --2- 172.21.15.62:0/3920686161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4676db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 >> 172.21.15.62:0/3920686161 conn(0x5556c4635310 msgr2=0x5556c466e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:22.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 shutdown_connections 2024-09-15T23:17:22.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f3807361740 1 -- 172.21.15.62:0/3920686161 wait complete. 2024-09-15T23:17:22.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.919+0000 7f3807361740 20 client.15705 trim_cache size 0 max 16384 2024-09-15T23:17:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f3807361740 1 Processor -- start 2024-09-15T23:17:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f3807361740 1 -- start start 2024-09-15T23:17:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f3807361740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4672930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f3807361740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c4672e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f3807361740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c468edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f3807361740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f38001b7ae0 con 0x5556c466db40 2024-09-15T23:17:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f3807361740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f38001b53b0 con 0x5556c4677370 2024-09-15T23:17:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f3807361740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f38001b3200 con 0x5556c466c5d0 2024-09-15T23:17:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f37fed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c468edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f37fed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c468edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41842/0 (socket says 172.21.15.62:41842) 2024-09-15T23:17:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f37fed9d640 1 -- 172.21.15.62:0/3524443622 learned_addr learned my addr 172.21.15.62:0/3524443622 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.939+0000 7f37fe59c640 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4672930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c4672e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37fe59c640 1 -- 172.21.15.62:0/3524443622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 msgr2=0x5556c468edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37fe59c640 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c468edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37fe59c640 1 -- 172.21.15.62:0/3524443622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 msgr2=0x5556c4672e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37fe59c640 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c4672e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37fe59c640 1 -- 172.21.15.62:0/3524443622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556c4668510 con 0x5556c466c5d0 2024-09-15T23:17:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37fed9d640 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c468edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37fe59c640 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4672930 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x5556c46751d0 tx=0x7f37f003f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e39431bc9e748d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37e7fff640 1 -- 172.21.15.62:0/3524443622 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0035040 con 0x5556c466c5d0 2024-09-15T23:17:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37e7fff640 1 -- 172.21.15.62:0/3524443622 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f005a070 con 0x5556c466c5d0 2024-09-15T23:17:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.940+0000 7f37e7fff640 1 -- 172.21.15.62:0/3524443622 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0032ce0 con 0x5556c466c5d0 2024-09-15T23:17:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.941+0000 7f3807361740 1 -- 172.21.15.62:0/3524443622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 msgr2=0x5556c4672930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.941+0000 7f3807361740 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4672930 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x5556c46751d0 tx=0x7f37f003f000 comp rx=0 tx=0).stop 2024-09-15T23:17:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.943+0000 7f3807361740 1 -- 172.21.15.62:0/3524443622 shutdown_connections 2024-09-15T23:17:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.943+0000 7f3807361740 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c468edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.943+0000 7f3807361740 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c4672e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.943+0000 7f3807361740 1 --2- 172.21.15.62:0/3524443622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c4672930 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.943+0000 7f3807361740 1 -- 172.21.15.62:0/3524443622 >> 172.21.15.62:0/3524443622 conn(0x5556c4635310 msgr2=0x5556c4663a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.943+0000 7f3807361740 1 -- 172.21.15.62:0/3524443622 shutdown_connections 2024-09-15T23:17:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.943+0000 7f3807361740 1 -- 172.21.15.62:0/3524443622 wait complete. 2024-09-15T23:17:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.944+0000 7f3807361740 1 -- start start 2024-09-15T23:17:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.944+0000 7f3807361740 1 Processor -- start 2024-09-15T23:17:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.944+0000 7f3807361740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c467fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.944+0000 7f3807361740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c467af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.944+0000 7f3807361740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c467b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.944+0000 7f3807361740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f38001b3870 con 0x5556c466db40 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.944+0000 7f3807361740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f38001b4ce0 con 0x5556c4677370 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.944+0000 7f3807361740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f38001b3f40 con 0x5556c466c5d0 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f3807361740 10 client.0 _ll_register_callbacks cb 0x7f37f8003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fdd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c467af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fdd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c467af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43200/0 (socket says 172.21.15.62:43200) 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fdd9b640 1 -- 172.21.15.62:0/2154971236 learned_addr learned my addr 172.21.15.62:0/2154971236 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fed9d640 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c467b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fe59c640 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c467fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fdd9b640 1 -- 172.21.15.62:0/2154971236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 msgr2=0x5556c467fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c467fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fdd9b640 1 -- 172.21.15.62:0/2154971236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 msgr2=0x5556c467b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c467b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.945+0000 7f37fdd9b640 1 -- 172.21.15.62:0/2154971236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556c467bcb0 con 0x5556c466db40 2024-09-15T23:17:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.946+0000 7f37fdd9b640 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c467af20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f37e8003cb0 tx=0x7f37e8005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59e4fdbc2a41534 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.946+0000 7f37e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:17:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.946+0000 7f37e5ffb640 1 -- 172.21.15.62:0/2154971236 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e80055e0 con 0x5556c466db40 2024-09-15T23:17:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.946+0000 7f37e5ffb640 1 -- 172.21.15.62:0/2154971236 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f37e8003db0 con 0x5556c466db40 2024-09-15T23:17:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.946+0000 7f3807361740 1 -- 172.21.15.62:0/2154971236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37f005a070 con 0x5556c466db40 2024-09-15T23:17:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.947+0000 7f37e5ffb640 1 -- 172.21.15.62:0/2154971236 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f37e8003fb0 con 0x5556c466db40 2024-09-15T23:17:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.947+0000 7f3807361740 10 client.15711 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.947+0000 7f3807361740 1 -- 172.21.15.62:0/2154971236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5556c46f1dd0 con 0x5556c466db40 2024-09-15T23:17:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.947+0000 7f37e5ffb640 1 -- 172.21.15.62:0/2154971236 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37e8004660 con 0x5556c466db40 2024-09-15T23:17:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.947+0000 7f37e5ffb640 1 -- 172.21.15.62:0/2154971236 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f37e801d020 con 0x5556c466db40 2024-09-15T23:17:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.948+0000 7f3807361740 10 client.15711 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.948+0000 7f3807361740 10 client.15711 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:17:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.948+0000 7f3807361740 20 client.15711 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.948+0000 7f3807361740 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5556c46f22e0 0x5556c46f4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.948+0000 7f3807361740 4 client.15711 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:17:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.948+0000 7f3807361740 1 -- 172.21.15.62:0/2154971236 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f37f005a070 con 0x5556c46f22e0 2024-09-15T23:17:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.950+0000 7f37fe59c640 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5556c46f22e0 0x5556c46f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.954+0000 7f37fe59c640 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5556c46f22e0 0x5556c46f4700 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=632c0aa73430373e server_cookie=7d97d4a9ffb84036 in_seq=0 out_seq=0 2024-09-15T23:17:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.954+0000 7f37e5ffb640 10 client.15711 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:17:22.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.958+0000 7f37e5ffb640 1 -- 172.21.15.62:0/2154971236 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f37f005a070 con 0x5556c46f22e0 2024-09-15T23:17:22.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.958+0000 7f37e5ffb640 10 client.15711 handle_command_reply: tid=0 2024-09-15T23:17:22.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.958+0000 7f3807361740 1 client.15711 shutdown 2024-09-15T23:17:22.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.959+0000 7f3807361740 1 -- 172.21.15.62:0/2154971236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 msgr2=0x5556c467af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:22.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.959+0000 7f3807361740 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c467af20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f37e8003cb0 tx=0x7f37e8005270 comp rx=0 tx=0).stop 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.959+0000 7f3807361740 1 -- 172.21.15.62:0/2154971236 shutdown_connections 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.960+0000 7f3807361740 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5556c46f22e0 0x5556c46f4700 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.960+0000 7f3807361740 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5556c4677370 0x5556c467b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.960+0000 7f37e5ffb640 0 client.15711 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.960+0000 7f3807361740 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5556c466db40 0x5556c467af20 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.960+0000 7f3807361740 1 --2- 172.21.15.62:0/2154971236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5556c466c5d0 0x5556c467fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.960+0000 7f3807361740 1 -- 172.21.15.62:0/2154971236 >> 172.21.15.62:0/2154971236 conn(0x5556c4635310 msgr2=0x5556c46728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.960+0000 7f3807361740 1 -- 172.21.15.62:0/2154971236 shutdown_connections 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.960+0000 7f3807361740 1 -- 172.21.15.62:0/2154971236 wait complete. 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.960+0000 7f3807361740 20 client.15711 trim_cache size 0 max 16384 2024-09-15T23:17:22.963 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:17:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.961+0000 7f3804f88640 1 -- 172.21.15.62:0/2794655120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f37ec0a3310 msgr2=0x7f37ec0a57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.961+0000 7f3804f88640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f37ec0a3310 0x7f37ec0a57d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x5556c4669ac0 tx=0x7f37f002a910 comp rx=0 tx=0).stop 2024-09-15T23:17:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.962+0000 7f3804f88640 1 -- 172.21.15.62:0/2794655120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800106f10 msgr2=0x7f380019e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:22.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.962+0000 7f3804f88640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800106f10 0x7f380019e850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f37f800ec50 tx=0x7f37f800c6a0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.962+0000 7f3804f88640 1 -- 172.21.15.62:0/2794655120 shutdown_connections 2024-09-15T23:17:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.962+0000 7f3804f88640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f37ec0a3310 0x7f37ec0a57d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.962+0000 7f3804f88640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3800106f10 0x7f380019e850 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.962+0000 7f3804f88640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3800102790 0x7f38001a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.962+0000 7f3804f88640 1 --2- 172.21.15.62:0/2794655120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3800101de0 0x7f38001a46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.962+0000 7f3804f88640 1 -- 172.21.15.62:0/2794655120 >> 172.21.15.62:0/2794655120 conn(0x7f38000fd8e0 msgr2=0x7f38000fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.963+0000 7f3804f88640 1 -- 172.21.15.62:0/2794655120 shutdown_connections 2024-09-15T23:17:22.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:22.963+0000 7f3804f88640 1 -- 172.21.15.62:0/2794655120 wait complete. 2024-09-15T23:17:23.351 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:23 smithi062 ceph-mon[30949]: pgmap v403: 129 pgs: 129 active+clean; 978 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 15 MiB/s wr, 1.00k op/s 2024-09-15T23:17:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:23 smithi067 ceph-mon[35665]: pgmap v403: 129 pgs: 129 active+clean; 978 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 15 MiB/s wr, 1.00k op/s 2024-09-15T23:17:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:23 smithi148 ceph-mon[35610]: pgmap v403: 129 pgs: 129 active+clean; 978 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 15 MiB/s wr, 1.00k op/s 2024-09-15T23:17:23.693 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:17:24.100 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:24.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 65.0 s, 2630.8 tps, lat 38.054 ms stddev 42.691 2024-09-15T23:17:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.632+0000 7f2b7eb71640 1 -- 172.21.15.62:0/2972478892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b78100a80 msgr2=0x7f2b78100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.632+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/2972478892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b78100a80 0x7f2b78100ee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009900 tx=0x7f2b6402a400 comp rx=0 tx=0).stop 2024-09-15T23:17:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.632+0000 7f2b7eb71640 1 -- 172.21.15.62:0/2972478892 shutdown_connections 2024-09-15T23:17:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.632+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/2972478892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b78101420 0x7f2b7810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.632+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/2972478892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b78100a80 0x7f2b78100ee0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.632+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/2972478892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b78106a80 0x7f2b78106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.632+0000 7f2b7eb71640 1 -- 172.21.15.62:0/2972478892 >> 172.21.15.62:0/2972478892 conn(0x7f2b780fc7a0 msgr2=0x7f2b780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.632+0000 7f2b7eb71640 1 -- 172.21.15.62:0/2972478892 shutdown_connections 2024-09-15T23:17:24.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.632+0000 7f2b7eb71640 1 -- 172.21.15.62:0/2972478892 wait complete. 2024-09-15T23:17:24.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.633+0000 7f2b7eb71640 1 Processor -- start 2024-09-15T23:17:24.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.633+0000 7f2b7eb71640 1 -- start start 2024-09-15T23:17:24.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.633+0000 7f2b7eb71640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b78100a80 0x7f2b7819a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.633+0000 7f2b7eb71640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b78101420 0x7f2b7819ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.634+0000 7f2b7c90d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b78100a80 0x7f2b7819a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.634+0000 7f2b7c90d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b78100a80 0x7f2b7819a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56372/0 (socket says 172.21.15.62:56372) 2024-09-15T23:17:24.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.634+0000 7f2b7eb71640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b78106a80 0x7f2b7819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.634+0000 7f2b7eb71640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2b78114180 con 0x7f2b78101420 2024-09-15T23:17:24.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.634+0000 7f2b7eb71640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b78114000 con 0x7f2b78106a80 2024-09-15T23:17:24.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.634+0000 7f2b7eb71640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2b78114300 con 0x7f2b78100a80 2024-09-15T23:17:24.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.634+0000 7f2b7c90d640 1 -- 172.21.15.62:0/3475397604 learned_addr learned my addr 172.21.15.62:0/3475397604 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.634+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b78106a80 0x7f2b7819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.634+0000 7f2b77fff640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b78101420 0x7f2b7819ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.635+0000 7f2b7d10e640 1 -- 172.21.15.62:0/3475397604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b78100a80 msgr2=0x7f2b7819a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.635+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b78100a80 0x7f2b7819a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.635+0000 7f2b7d10e640 1 -- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b78101420 msgr2=0x7f2b7819ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.635+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b78101420 0x7f2b7819ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.635+0000 7f2b7d10e640 1 -- 172.21.15.62:0/3475397604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7819f5f0 con 0x7f2b78106a80 2024-09-15T23:17:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.635+0000 7f2b7c90d640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b78100a80 0x7f2b7819a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.635+0000 7f2b77fff640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b78101420 0x7f2b7819ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.635+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b78106a80 0x7f2b7819ef10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000ec50 tx=0x7f2b7000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.636+0000 7f2b75ffb640 1 -- 172.21.15.62:0/3475397604 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70003b70 con 0x7f2b78106a80 2024-09-15T23:17:24.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.636+0000 7f2b75ffb640 1 -- 172.21.15.62:0/3475397604 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b70003d10 con 0x7f2b78106a80 2024-09-15T23:17:24.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.636+0000 7f2b7eb71640 1 -- 172.21.15.62:0/3475397604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7819f8e0 con 0x7f2b78106a80 2024-09-15T23:17:24.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.636+0000 7f2b75ffb640 1 -- 172.21.15.62:0/3475397604 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70010890 con 0x7f2b78106a80 2024-09-15T23:17:24.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.636+0000 7f2b7eb71640 1 -- 172.21.15.62:0/3475397604 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b781a71c0 con 0x7f2b78106a80 2024-09-15T23:17:24.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.638+0000 7f2b75ffb640 1 -- 172.21.15.62:0/3475397604 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2b70003fe0 con 0x7f2b78106a80 2024-09-15T23:17:24.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.638+0000 7f2b80f4a740 1 Processor -- start 2024-09-15T23:17:24.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b75ffb640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b680a32b0 0x7f2b680a5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b80f4a740 1 -- start start 2024-09-15T23:17:24.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b75ffb640 1 -- 172.21.15.62:0/3475397604 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2b700d0270 con 0x7f2b78106a80 2024-09-15T23:17:24.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b77fff640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b680a32b0 0x7f2b680a5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e44b40 0x565280e43090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e43a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e4e370 0x565280e50760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b80f4a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565280da92f0 con 0x565280e44b40 2024-09-15T23:17:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b80f4a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565280d50cf0 con 0x565280e435d0 2024-09-15T23:17:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b80f4a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x565280da52b0 con 0x565280e4e370 2024-09-15T23:17:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b7d10e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e4e370 0x565280e50760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b7d10e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e4e370 0x565280e50760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56384/0 (socket says 172.21.15.62:56384) 2024-09-15T23:17:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b7d10e640 1 -- 172.21.15.62:0/1522385130 learned_addr learned my addr 172.21.15.62:0/1522385130 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b77fff640 1 --2- 172.21.15.62:0/1522385130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e43a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.639+0000 7f2b7c90d640 1 --2- 172.21.15.62:0/1522385130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e44b40 0x565280e43090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.640+0000 7f2b7d10e640 1 -- 172.21.15.62:0/1522385130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 msgr2=0x565280e43a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.640+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/1522385130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e43a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.640+0000 7f2b7d10e640 1 -- 172.21.15.62:0/1522385130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e44b40 msgr2=0x565280e43090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.640+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/1522385130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e44b40 0x565280e43090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.640+0000 7f2b7d10e640 1 -- 172.21.15.62:0/1522385130 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565280d353a0 con 0x565280e4e370 2024-09-15T23:17:24.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.640+0000 7f2b77fff640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b680a32b0 0x7f2b680a5770 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b64004450 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.640+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/1522385130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e4e370 0x565280e50760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2b7009c440 tx=0x7f2b700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0021ede366bec06 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.641+0000 7f2b62ffd640 1 -- 172.21.15.62:0/1522385130 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70009820 con 0x565280e4e370 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.641+0000 7f2b62ffd640 1 -- 172.21.15.62:0/1522385130 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b700cde50 con 0x565280e4e370 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.641+0000 7f2b62ffd640 1 -- 172.21.15.62:0/1522385130 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b700042d0 con 0x565280e4e370 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.642+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1522385130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e4e370 msgr2=0x565280e50760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.642+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1522385130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e4e370 0x565280e50760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2b7009c440 tx=0x7f2b700046c0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.642+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1522385130 shutdown_connections 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.642+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1522385130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e4e370 0x565280e50760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.642+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1522385130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e43a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.642+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1522385130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e44b40 0x565280e43090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.642+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1522385130 >> 172.21.15.62:0/1522385130 conn(0x565280e0c310 msgr2=0x565280e3fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.642+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1522385130 shutdown_connections 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.642+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1522385130 wait complete. 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b80f4a740 1 -- start start 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b80f4a740 1 Processor -- start 2024-09-15T23:17:24.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e435d0 0x565280e4dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 0x565280e4b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e4e370 0x565280e4b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b7c90d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e435d0 0x565280e4dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b7c90d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e435d0 0x565280e4dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43242/0 (socket says 172.21.15.62:43242) 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b80f4a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565280e51070 con 0x565280e435d0 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b7c90d640 1 -- 172.21.15.62:0/696853540 learned_addr learned my addr 172.21.15.62:0/696853540 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565280d64060 con 0x565280e4e370 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x565280e511f0 con 0x565280e44b40 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.643+0000 7f2b80f4a740 10 client.0 _ll_register_callbacks cb 0x565280ab78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b77fff640 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 0x565280e4b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e4e370 0x565280e4b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b77fff640 1 -- 172.21.15.62:0/696853540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e4e370 msgr2=0x565280e4b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b77fff640 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e4e370 0x565280e4b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b77fff640 1 -- 172.21.15.62:0/696853540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e435d0 msgr2=0x565280e4dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b77fff640 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e435d0 0x565280e4dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b77fff640 1 -- 172.21.15.62:0/696853540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565280b1a3c0 con 0x565280e44b40 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e4e370 0x565280e4b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b77fff640 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 0x565280e4b090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b6402a9f0 tx=0x7f2b6402a400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10b80cfc03d907a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b57fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b57fff640 1 -- 172.21.15.62:0/696853540 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b64035040 con 0x565280e44b40 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b700cde50 con 0x565280e44b40 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.644+0000 7f2b57fff640 1 -- 172.21.15.62:0/696853540 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2b640037f0 con 0x565280e44b40 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b57fff640 1 -- 172.21.15.62:0/696853540 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2b6405c050 con 0x565280e44b40 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b80f4a740 10 client.25270 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565280e5cbd0 con 0x565280e44b40 2024-09-15T23:17:24.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b57fff640 1 -- 172.21.15.62:0/696853540 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b700cde50 con 0x565280e44b40 2024-09-15T23:17:24.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b57fff640 1 -- 172.21.15.62:0/696853540 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2b64032c70 con 0x565280e44b40 2024-09-15T23:17:24.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b80f4a740 10 client.25270 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:24.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b80f4a740 10 client.25270 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:17:24.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b80f4a740 20 client.25270 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:24.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x565280e5dfe0 0x565280e603f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b80f4a740 4 client.25270 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:24.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565280db6860 con 0x565280e5dfe0 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.645+0000 7f2b7c90d640 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x565280e5dfe0 0x565280e603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.646+0000 7f2b57fff640 10 client.25270 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.646+0000 7f2b7c90d640 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x565280e5dfe0 0x565280e603f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2e81298a099852e4 server_cookie=a7a0cb303c770502 in_seq=0 out_seq=0 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.647+0000 7f2b57fff640 1 -- 172.21.15.62:0/696853540 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565280db6860 con 0x565280e5dfe0 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.647+0000 7f2b57fff640 10 client.25270 handle_command_reply: tid=0 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.647+0000 7f2b80f4a740 1 client.25270 shutdown 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 msgr2=0x565280e4b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 0x565280e4b090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b6402a9f0 tx=0x7f2b6402a400 comp rx=0 tx=0).stop 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 shutdown_connections 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x565280e5dfe0 0x565280e603f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e4e370 0x565280e4b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 0x565280e4b090 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/696853540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e435d0 0x565280e4dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 >> 172.21.15.62:0/696853540 conn(0x565280e0c310 msgr2=0x565280e45930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b57fff640 0 client.25270 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:17:24.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 shutdown_connections 2024-09-15T23:17:24.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 1 -- 172.21.15.62:0/696853540 wait complete. 2024-09-15T23:17:24.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.648+0000 7f2b80f4a740 20 client.25270 trim_cache size 0 max 16384 2024-09-15T23:17:24.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.680+0000 7f2b80f4a740 1 Processor -- start 2024-09-15T23:17:24.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.681+0000 7f2b80f4a740 1 -- start start 2024-09-15T23:17:24.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.681+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e4a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.681+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 0x565280e5a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.681+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e4e370 0x565280e4a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.681+0000 7f2b80f4a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2b781af2a0 con 0x565280e4e370 2024-09-15T23:17:24.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.681+0000 7f2b80f4a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b781acb70 con 0x565280e435d0 2024-09-15T23:17:24.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.681+0000 7f2b80f4a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2b781aaa00 con 0x565280e44b40 2024-09-15T23:17:24.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.682+0000 7f2b7c90d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e4a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.682+0000 7f2b7c90d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e4a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41886/0 (socket says 172.21.15.62:41886) 2024-09-15T23:17:24.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.682+0000 7f2b7c90d640 1 -- 172.21.15.62:0/125405007 learned_addr learned my addr 172.21.15.62:0/125405007 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.682+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/125405007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e4e370 0x565280e4a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.682+0000 7f2b7c90d640 1 -- 172.21.15.62:0/125405007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 msgr2=0x565280e5a310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:17:24.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.682+0000 7f2b7c90d640 1 --2- 172.21.15.62:0/125405007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 0x565280e5a310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.682+0000 7f2b7c90d640 1 -- 172.21.15.62:0/125405007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e4e370 msgr2=0x565280e4a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.682+0000 7f2b7c90d640 1 --2- 172.21.15.62:0/125405007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e4e370 0x565280e4a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.682+0000 7f2b7c90d640 1 -- 172.21.15.62:0/125405007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565280e3fde0 con 0x565280e435d0 2024-09-15T23:17:24.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.683+0000 7f2b7c90d640 1 --2- 172.21.15.62:0/125405007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e4a950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005e00 tx=0x7f2b6c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a70356767c385a42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.683+0000 7f2b627fc640 1 -- 172.21.15.62:0/125405007 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c00f890 con 0x565280e435d0 2024-09-15T23:17:24.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.683+0000 7f2b627fc640 1 -- 172.21.15.62:0/125405007 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b6c005020 con 0x565280e435d0 2024-09-15T23:17:24.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.683+0000 7f2b627fc640 1 -- 172.21.15.62:0/125405007 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c004b20 con 0x565280e435d0 2024-09-15T23:17:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.684+0000 7f2b80f4a740 1 -- 172.21.15.62:0/125405007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 msgr2=0x565280e4a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.684+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/125405007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e4a950 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005e00 tx=0x7f2b6c005f90 comp rx=0 tx=0).stop 2024-09-15T23:17:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.684+0000 7f2b80f4a740 1 -- 172.21.15.62:0/125405007 shutdown_connections 2024-09-15T23:17:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.684+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/125405007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e4e370 0x565280e4a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.684+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/125405007 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e44b40 0x565280e5a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.684+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/125405007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e435d0 0x565280e4a950 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.685+0000 7f2b80f4a740 1 -- 172.21.15.62:0/125405007 >> 172.21.15.62:0/125405007 conn(0x565280e0c310 msgr2=0x565280e49960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.686+0000 7f2b80f4a740 1 -- 172.21.15.62:0/125405007 shutdown_connections 2024-09-15T23:17:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.686+0000 7f2b80f4a740 1 -- 172.21.15.62:0/125405007 wait complete. 2024-09-15T23:17:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.687+0000 7f2b80f4a740 1 -- start start 2024-09-15T23:17:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.687+0000 7f2b80f4a740 1 Processor -- start 2024-09-15T23:17:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.688+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e435d0 0x565280e569b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.688+0000 7f2b80f4a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e4e370 0x565280e51c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.688+0000 7f2b7c90d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e435d0 0x565280e569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.688+0000 7f2b7c90d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e435d0 0x565280e569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56412/0 (socket says 172.21.15.62:56412) 2024-09-15T23:17:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.689+0000 7f2b7c90d640 1 -- 172.21.15.62:0/1972801955 learned_addr learned my addr 172.21.15.62:0/1972801955 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.689+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e52180 0x565280e52600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.689+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2b781ab050 con 0x565280e4e370 2024-09-15T23:17:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.689+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e52180 0x565280e52600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.689+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b781ac4a0 con 0x565280e52180 2024-09-15T23:17:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.689+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2b781ab700 con 0x565280e435d0 2024-09-15T23:17:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.690+0000 7f2b80f4a740 10 client.0 _ll_register_callbacks cb 0x7f2b64009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.690+0000 7f2b7d10e640 1 -- 172.21.15.62:0/1972801955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e435d0 msgr2=0x565280e569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.690+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e435d0 0x565280e569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.690+0000 7f2b7d10e640 1 -- 172.21.15.62:0/1972801955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e4e370 msgr2=0x565280e51c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.690+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e4e370 0x565280e51c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.690+0000 7f2b7d10e640 1 -- 172.21.15.62:0/1972801955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565280ec7c90 con 0x565280e52180 2024-09-15T23:17:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.691+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b6c005020 con 0x565280e52180 2024-09-15T23:17:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.691+0000 7f2b7d10e640 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e52180 0x565280e52600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b7009c1b0 tx=0x7f2b70003ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4ff5ff97ff13178 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.692+0000 7f2b57fff640 10 client.25391 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:17:24.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.693+0000 7f2b57fff640 1 -- 172.21.15.62:0/1972801955 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b70009820 con 0x565280e52180 2024-09-15T23:17:24.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.693+0000 7f2b57fff640 1 -- 172.21.15.62:0/1972801955 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2b700b5260 con 0x565280e52180 2024-09-15T23:17:24.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.693+0000 7f2b57fff640 1 -- 172.21.15.62:0/1972801955 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2b700044d0 con 0x565280e52180 2024-09-15T23:17:24.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.694+0000 7f2b80f4a740 10 client.25391 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:24.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.694+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=27}) -- 0x565280ec8cd0 con 0x565280e52180 2024-09-15T23:17:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.695+0000 7f2b57fff640 1 -- 172.21.15.62:0/1972801955 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2b700a4c90 con 0x565280e52180 2024-09-15T23:17:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.695+0000 7f2b80f4a740 10 client.25391 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.695+0000 7f2b80f4a740 10 client.25391 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:17:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.695+0000 7f2b80f4a740 20 client.25391 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.695+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x565280ec8ea0 0x565280ecb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.695+0000 7f2b80f4a740 4 client.25391 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:17:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.695+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f2b6c005020 con 0x565280ec8ea0 2024-09-15T23:17:24.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.696+0000 7f2b7c90d640 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x565280ec8ea0 0x565280ecb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.697+0000 7f2b7c90d640 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x565280ec8ea0 0x565280ecb290 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a3b08d38a94f70ed server_cookie=e0a851b3927a30f7 in_seq=0 out_seq=0 2024-09-15T23:17:24.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.697+0000 7f2b57fff640 10 client.25391 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:17:24.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.697+0000 7f2b57fff640 1 -- 172.21.15.62:0/1972801955 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f2b6c005020 con 0x565280ec8ea0 2024-09-15T23:17:24.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.697+0000 7f2b57fff640 10 client.25391 handle_command_reply: tid=0 2024-09-15T23:17:24.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.698+0000 7f2b80f4a740 1 client.25391 shutdown 2024-09-15T23:17:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.700+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e52180 msgr2=0x565280e52600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.700+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e52180 0x565280e52600 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2b7009c1b0 tx=0x7f2b70003ec0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.700+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 shutdown_connections 2024-09-15T23:17:24.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.700+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x565280ec8ea0 0x565280ecb290 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.700+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565280e52180 0x565280e52600 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.700+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565280e4e370 0x565280e51c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.700+0000 7f2b80f4a740 1 --2- 172.21.15.62:0/1972801955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565280e435d0 0x565280e569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.700+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 >> 172.21.15.62:0/1972801955 conn(0x565280e0c310 msgr2=0x565280e4c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.700+0000 7f2b57fff640 0 client.25391 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:17:24.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.701+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 shutdown_connections 2024-09-15T23:17:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.701+0000 7f2b80f4a740 1 -- 172.21.15.62:0/1972801955 wait complete. 2024-09-15T23:17:24.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.702+0000 7f2b80f4a740 20 client.25391 trim_cache size 0 max 16384 2024-09-15T23:17:24.705 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:17:24.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 -- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b680a32b0 msgr2=0x7f2b680a5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b680a32b0 0x7f2b680a5770 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b64004450 comp rx=0 tx=0).stop 2024-09-15T23:17:24.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 -- 172.21.15.62:0/3475397604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b78106a80 msgr2=0x7f2b7819ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b78106a80 0x7f2b7819ef10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2b7000ec50 tx=0x7f2b7000c6a0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 -- 172.21.15.62:0/3475397604 shutdown_connections 2024-09-15T23:17:24.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b680a32b0 0x7f2b680a5770 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b78106a80 0x7f2b7819ef10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b78101420 0x7f2b7819ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 --2- 172.21.15.62:0/3475397604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b78100a80 0x7f2b7819a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.706+0000 7f2b7eb71640 1 -- 172.21.15.62:0/3475397604 >> 172.21.15.62:0/3475397604 conn(0x7f2b780fc7a0 msgr2=0x7f2b780fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.710+0000 7f2b7eb71640 1 -- 172.21.15.62:0/3475397604 shutdown_connections 2024-09-15T23:17:24.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:24.710+0000 7f2b7eb71640 1 -- 172.21.15.62:0/3475397604 wait complete. 2024-09-15T23:17:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.816+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1141367602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54107870 msgr2=0x7f0b54109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.816+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1141367602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54107870 0x7f0b54109ca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009be0 tx=0x7f0b4c02a6d0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.816+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1141367602 shutdown_connections 2024-09-15T23:17:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.816+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1141367602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54107870 0x7f0b54109ca0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.816+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1141367602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b54104f40 0x7f0b54107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.816+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1141367602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b54102610 0x7f0b54104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.816+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1141367602 >> 172.21.15.62:0/1141367602 conn(0x7f0b540fc1b0 msgr2=0x7f0b540fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.816+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1141367602 shutdown_connections 2024-09-15T23:17:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.816+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1141367602 wait complete. 2024-09-15T23:17:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.817+0000 7f0b59f6d640 1 Processor -- start 2024-09-15T23:17:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.818+0000 7f0b59f6d640 1 -- start start 2024-09-15T23:17:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.820+0000 7f0b59f6d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54102610 0x7f0b54199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.820+0000 7f0b59f6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b54104f40 0x7f0b5419a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.820+0000 7f0b59f6d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b54107870 0x7f0b541a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.820+0000 7f0b59f6d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541018b0 con 0x7f0b54104f40 2024-09-15T23:17:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.820+0000 7f0b59f6d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b54101730 con 0x7f0b54102610 2024-09-15T23:17:24.823 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.820+0000 7f0b59f6d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0b54101a30 con 0x7f0b54107870 2024-09-15T23:17:24.823 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.820+0000 7f0b52ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b54104f40 0x7f0b5419a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.823 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.820+0000 7f0b52ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b54104f40 0x7f0b5419a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43268/0 (socket says 172.21.15.62:43268) 2024-09-15T23:17:24.823 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.820+0000 7f0b52ffd640 1 -- 172.21.15.62:0/1829086023 learned_addr learned my addr 172.21.15.62:0/1829086023 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b52ffd640 1 -- 172.21.15.62:0/1829086023 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b54107870 msgr2=0x7f0b541a1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b54107870 0x7f0b541a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b537fe640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54102610 0x7f0b54199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b54107870 0x7f0b541a1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b52ffd640 1 -- 172.21.15.62:0/1829086023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54102610 msgr2=0x7f0b54199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54102610 0x7f0b54199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b52ffd640 1 -- 172.21.15.62:0/1829086023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b541a1ac0 con 0x7f0b54104f40 2024-09-15T23:17:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b537fe640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54102610 0x7f0b54199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:17:24.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b54107870 0x7f0b541a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:17:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.821+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b54104f40 0x7f0b5419a3e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800b450 tx=0x7f0b4800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.822+0000 7f0b50ff9640 1 -- 172.21.15.62:0/1829086023 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b48015020 con 0x7f0b54104f40 2024-09-15T23:17:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.822+0000 7f0b50ff9640 1 -- 172.21.15.62:0/1829086023 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b48011d70 con 0x7f0b54104f40 2024-09-15T23:17:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.822+0000 7f0b50ff9640 1 -- 172.21.15.62:0/1829086023 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4800f340 con 0x7f0b54104f40 2024-09-15T23:17:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.822+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1829086023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b541a1db0 con 0x7f0b54104f40 2024-09-15T23:17:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.823+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1829086023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b541a22a0 con 0x7f0b54104f40 2024-09-15T23:17:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b5cde6740 1 Processor -- start 2024-09-15T23:17:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b5cde6740 1 -- start start 2024-09-15T23:17:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b50ff9640 1 -- 172.21.15.62:0/1829086023 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0b48018020 con 0x7f0b54104f40 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bc7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc82e0 0x555c46bccda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bcd370 0x555c46bcf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555c46b2b2f0 con 0x555c46bc82e0 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555c46ad3270 con 0x555c46bcd370 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555c46b272b0 con 0x555c46bc79a0 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.828+0000 7f0b537fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc82e0 0x555c46bccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.829+0000 7f0b537fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc82e0 0x555c46bccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43284/0 (socket says 172.21.15.62:43284) 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.829+0000 7f0b537fe640 1 -- 172.21.15.62:0/1480229875 learned_addr learned my addr 172.21.15.62:0/1480229875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.829+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1480229875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bc7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.829+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/1480229875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bcd370 0x555c46bcf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.829+0000 7f0b53fff640 1 -- 172.21.15.62:0/1480229875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bcd370 msgr2=0x555c46bcf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.829+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1480229875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bcd370 0x555c46bcf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.829+0000 7f0b53fff640 1 -- 172.21.15.62:0/1480229875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc82e0 msgr2=0x555c46bccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.829+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1480229875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc82e0 0x555c46bccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.829+0000 7f0b53fff640 1 -- 172.21.15.62:0/1480229875 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c46ad2340 con 0x555c46bc79a0 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.830+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1480229875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bc7da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009b30 tx=0x7f0b4c00b140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=933c587a70bbac2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.830+0000 7f0b39ffb640 1 -- 172.21.15.62:0/1480229875 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c040020 con 0x555c46bc79a0 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.830+0000 7f0b50ff9640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0b3c0a34b0 0x7f0b3c0a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.830+0000 7f0b39ffb640 1 -- 172.21.15.62:0/1480229875 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b4c037040 con 0x555c46bc79a0 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.830+0000 7f0b39ffb640 1 -- 172.21.15.62:0/1480229875 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c036030 con 0x555c46bc79a0 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.830+0000 7f0b537fe640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0b3c0a34b0 0x7f0b3c0a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.830+0000 7f0b50ff9640 1 -- 172.21.15.62:0/1829086023 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0b480cf720 con 0x7f0b54104f40 2024-09-15T23:17:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b537fe640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0b3c0a34b0 0x7f0b3c0a5970 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x555c46bc8d10 tx=0x7f0b44009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1480229875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 msgr2=0x555c46bc7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1480229875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bc7da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009b30 tx=0x7f0b4c00b140 comp rx=0 tx=0).stop 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1480229875 shutdown_connections 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1480229875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bcd370 0x555c46bcf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1480229875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc82e0 0x555c46bccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1480229875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bc7da0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1480229875 >> 172.21.15.62:0/1480229875 conn(0x555c46bc3010 msgr2=0x555c46bc3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1480229875 shutdown_connections 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.831+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1480229875 wait complete. 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b5cde6740 1 -- start start 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b5cde6740 1 Processor -- start 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bda3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46bda8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46be1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555c46bd8af0 con 0x555c46bcd370 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555c46ab4290 con 0x555c46bc82e0 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555c46bd8c70 con 0x555c46bc79a0 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b537fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bda3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b52ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46bda8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b537fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bda3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56450/0 (socket says 172.21.15.62:56450) 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b537fe640 1 -- 172.21.15.62:0/1812609668 learned_addr learned my addr 172.21.15.62:0/1812609668 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46be1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.832+0000 7f0b5cde6740 10 client.0 _ll_register_callbacks cb 0x555c46ae5140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.833+0000 7f0b53fff640 1 -- 172.21.15.62:0/1812609668 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 msgr2=0x555c46bda3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.833+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bda3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.833+0000 7f0b53fff640 1 -- 172.21.15.62:0/1812609668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 msgr2=0x555c46bda8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.833+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46bda8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.833+0000 7f0b53fff640 1 -- 172.21.15.62:0/1812609668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c4683e210 con 0x555c46bcd370 2024-09-15T23:17:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.833+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46bda8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.833+0000 7f0b53fff640 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46be1920 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c034040 tx=0x7f0b4c009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89dbf398e73e0e2e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.833+0000 7f0b32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.835+0000 7f0b32ffd640 1 -- 172.21.15.62:0/1812609668 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c040070 con 0x555c46bcd370 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.835+0000 7f0b32ffd640 1 -- 172.21.15.62:0/1812609668 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0b4c0305a0 con 0x555c46bcd370 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.835+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1812609668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b4c037040 con 0x555c46bcd370 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.835+0000 7f0b32ffd640 1 -- 172.21.15.62:0/1812609668 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b4c036040 con 0x555c46bcd370 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.835+0000 7f0b5cde6740 10 client.15735 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.835+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1812609668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555c4690a620 con 0x555c46bcd370 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.835+0000 7f0b32ffd640 1 -- 172.21.15.62:0/1812609668 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b4c037040 con 0x555c46bcd370 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.836+0000 7f0b32ffd640 1 -- 172.21.15.62:0/1812609668 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0b4c031560 con 0x555c46bcd370 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.836+0000 7f0b5cde6740 10 client.15735 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.836+0000 7f0b5cde6740 10 client.15735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.836+0000 7f0b5cde6740 20 client.15735 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.836+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c46be5c00 0x555c46be8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.836+0000 7f0b5cde6740 4 client.15735 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.836+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1812609668 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c46b38a20 con 0x555c46be5c00 2024-09-15T23:17:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.836+0000 7f0b537fe640 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c46be5c00 0x555c46be8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.837+0000 7f0b537fe640 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c46be5c00 0x555c46be8030 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce7de02e9a1dfd3d server_cookie=7c890b3fa060c87d in_seq=0 out_seq=0 2024-09-15T23:17:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.837+0000 7f0b32ffd640 10 client.15735 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.837+0000 7f0b32ffd640 1 -- 172.21.15.62:0/1812609668 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555c46b38a20 con 0x555c46be5c00 2024-09-15T23:17:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.837+0000 7f0b32ffd640 10 client.15735 handle_command_reply: tid=0 2024-09-15T23:17:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.837+0000 7f0b5cde6740 1 client.15735 shutdown 2024-09-15T23:17:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1812609668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 msgr2=0x555c46be1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46be1920 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c034040 tx=0x7f0b4c009960 comp rx=0 tx=0).stop 2024-09-15T23:17:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1812609668 shutdown_connections 2024-09-15T23:17:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c46be5c00 0x555c46be8030 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46be1920 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46bda8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1812609668 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bda3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b32ffd640 0 client.15735 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1812609668 >> 172.21.15.62:0/1812609668 conn(0x555c46bc3010 msgr2=0x555c46bc4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1812609668 shutdown_connections 2024-09-15T23:17:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.838+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1812609668 wait complete. 2024-09-15T23:17:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.839+0000 7f0b5cde6740 20 client.15735 trim_cache size 0 max 16384 2024-09-15T23:17:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.861+0000 7f0b5cde6740 1 Processor -- start 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.861+0000 7f0b5cde6740 1 -- start start 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.861+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bc33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.861+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46be18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.861+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46bc3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541aad60 con 0x555c46bcd370 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46bc3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46bc3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43300/0 (socket says 172.21.15.62:43300) 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b53fff640 1 -- 172.21.15.62:0/1962230886 learned_addr learned my addr 172.21.15.62:0/1962230886 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1962230886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541a8630 con 0x555c46bc82e0 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/1962230886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46be18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1962230886 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541a6480 con 0x555c46bc79a0 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b52ffd640 1 -- 172.21.15.62:0/1962230886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 msgr2=0x555c46bc33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/1962230886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bc33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b52ffd640 1 -- 172.21.15.62:0/1962230886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 msgr2=0x555c46bc3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/1962230886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46bc3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.862+0000 7f0b52ffd640 1 -- 172.21.15.62:0/1962230886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c46bd9370 con 0x555c46bc82e0 2024-09-15T23:17:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.863+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/1962230886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46be18e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009fd0 tx=0x7f0b4809b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb63d982a32e6acf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.863+0000 7f0b397fa640 1 -- 172.21.15.62:0/1962230886 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4809b960 con 0x555c46bc82e0 2024-09-15T23:17:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.863+0000 7f0b397fa640 1 -- 172.21.15.62:0/1962230886 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b4809fd10 con 0x555c46bc82e0 2024-09-15T23:17:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.863+0000 7f0b397fa640 1 -- 172.21.15.62:0/1962230886 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b48010570 con 0x555c46bc82e0 2024-09-15T23:17:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.864+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1962230886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 msgr2=0x555c46be18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.864+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1962230886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46be18e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009fd0 tx=0x7f0b4809b7f0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.864+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1962230886 shutdown_connections 2024-09-15T23:17:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.864+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1962230886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bcd370 0x555c46bc3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.864+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1962230886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46be18e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.864+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/1962230886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bc79a0 0x555c46bc33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.864+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1962230886 >> 172.21.15.62:0/1962230886 conn(0x555c46bc3010 msgr2=0x555c46bda3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.864+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1962230886 shutdown_connections 2024-09-15T23:17:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.864+0000 7f0b5cde6740 1 -- 172.21.15.62:0/1962230886 wait complete. 2024-09-15T23:17:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.865+0000 7f0b5cde6740 1 -- start start 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.865+0000 7f0b5cde6740 1 Processor -- start 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.865+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc79a0 0x555c46bde770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.865+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46bdecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.865+0000 7f0b5cde6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bcd370 0x555c46c5f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.865+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541a6af0 con 0x555c46bc79a0 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.865+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541a7f60 con 0x555c46bc82e0 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.865+0000 7f0b5cde6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0b541a71c0 con 0x555c46bcd370 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b5cde6740 10 client.0 _ll_register_callbacks cb 0x7f0b44009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bcd370 0x555c46c5f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b537fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc79a0 0x555c46bde770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b53fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bcd370 0x555c46c5f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56458/0 (socket says 172.21.15.62:56458) 2024-09-15T23:17:24.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b53fff640 1 -- 172.21.15.62:0/660446996 learned_addr learned my addr 172.21.15.62:0/660446996 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b52ffd640 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46bdecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b537fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc79a0 0x555c46bde770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43304/0 (socket says 172.21.15.62:43304) 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b53fff640 1 -- 172.21.15.62:0/660446996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 msgr2=0x555c46bdecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b53fff640 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46bdecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b53fff640 1 -- 172.21.15.62:0/660446996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc79a0 msgr2=0x555c46bde770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b53fff640 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc79a0 0x555c46bde770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b53fff640 1 -- 172.21.15.62:0/660446996 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c46bd0100 con 0x555c46bcd370 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.866+0000 7f0b537fe640 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc79a0 0x555c46bde770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.867+0000 7f0b53fff640 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bcd370 0x555c46c5f920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c031960 tx=0x7f0b4c003970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f612bb8cb76b051 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.867+0000 7f0b32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.867+0000 7f0b32ffd640 1 -- 172.21.15.62:0/660446996 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c040050 con 0x555c46bcd370 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.867+0000 7f0b32ffd640 1 -- 172.21.15.62:0/660446996 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0b4c031440 con 0x555c46bcd370 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.867+0000 7f0b5cde6740 1 -- 172.21.15.62:0/660446996 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b4809fd10 con 0x555c46bcd370 2024-09-15T23:17:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.867+0000 7f0b32ffd640 1 -- 172.21.15.62:0/660446996 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0b4c064050 con 0x555c46bcd370 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.867+0000 7f0b5cde6740 10 client.25285 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.867+0000 7f0b5cde6740 1 -- 172.21.15.62:0/660446996 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555c46bd1480 con 0x555c46bcd370 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.867+0000 7f0b32ffd640 1 -- 172.21.15.62:0/660446996 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b4809fd10 con 0x555c46bcd370 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.868+0000 7f0b32ffd640 1 -- 172.21.15.62:0/660446996 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f0b4c046690 con 0x555c46bcd370 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.868+0000 7f0b5cde6740 10 client.25285 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.868+0000 7f0b5cde6740 10 client.25285 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.868+0000 7f0b5cde6740 20 client.25285 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.868+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c46bd1e30 0x555c46bd4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.868+0000 7f0b5cde6740 4 client.25285 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:17:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.868+0000 7f0b5cde6740 1 -- 172.21.15.62:0/660446996 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555c46b32d70 con 0x555c46bd1e30 2024-09-15T23:17:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.869+0000 7f0b537fe640 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c46bd1e30 0x555c46bd4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.870+0000 7f0b537fe640 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c46bd1e30 0x555c46bd4250 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc2cdc9679dc28ed server_cookie=fe5e35e2bcd293ac in_seq=0 out_seq=0 2024-09-15T23:17:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.870+0000 7f0b32ffd640 10 client.25285 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b32ffd640 1 -- 172.21.15.62:0/660446996 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555c46b32d70 con 0x555c46bd1e30 2024-09-15T23:17:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b32ffd640 10 client.25285 handle_command_reply: tid=0 2024-09-15T23:17:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 client.25285 shutdown 2024-09-15T23:17:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 -- 172.21.15.62:0/660446996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bcd370 msgr2=0x555c46c5f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bcd370 0x555c46c5f920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c031960 tx=0x7f0b4c003970 comp rx=0 tx=0).stop 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 -- 172.21.15.62:0/660446996 shutdown_connections 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c46bd1e30 0x555c46bd4250 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c46bcd370 0x555c46c5f920 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c46bc82e0 0x555c46bdecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 --2- 172.21.15.62:0/660446996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c46bc79a0 0x555c46bde770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b32ffd640 0 client.25285 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 -- 172.21.15.62:0/660446996 >> 172.21.15.62:0/660446996 conn(0x555c46bc3010 msgr2=0x555c46bc19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 -- 172.21.15.62:0/660446996 shutdown_connections 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 1 -- 172.21.15.62:0/660446996 wait complete. 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.907+0000 7f0b5cde6740 20 client.25285 trim_cache size 0 max 16384 2024-09-15T23:17:24.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:17:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:17:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:17:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0b3c0a34b0 msgr2=0x7f0b3c0a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0b3c0a34b0 0x7f0b3c0a5970 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x555c46bc8d10 tx=0x7f0b44009260 comp rx=0 tx=0).stop 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b54104f40 msgr2=0x7f0b5419a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b54104f40 0x7f0b5419a3e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800b450 tx=0x7f0b4800b910 comp rx=0 tx=0).stop 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1829086023 shutdown_connections 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0b3c0a34b0 0x7f0b3c0a5970 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0b54107870 0x7f0b541a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b54104f40 0x7f0b5419a3e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 --2- 172.21.15.62:0/1829086023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b54102610 0x7f0b54199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1829086023 >> 172.21.15.62:0/1829086023 conn(0x7f0b540fc1b0 msgr2=0x7f0b54107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1829086023 shutdown_connections 2024-09-15T23:17:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:24.908+0000 7f0b59f6d640 1 -- 172.21.15.62:0/1829086023 wait complete. 2024-09-15T23:17:25.401 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:25 smithi062 ceph-mon[30949]: pgmap v404: 129 pgs: 129 active+clean; 982 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 13 MiB/s wr, 944 op/s 2024-09-15T23:17:25.439 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:17:25.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:25 smithi067 ceph-mon[35665]: pgmap v404: 129 pgs: 129 active+clean; 982 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 13 MiB/s wr, 944 op/s 2024-09-15T23:17:25.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:25 smithi148 ceph-mon[35610]: pgmap v404: 129 pgs: 129 active+clean; 982 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 13 MiB/s wr, 944 op/s 2024-09-15T23:17:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:26.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.343+0000 7f2e6b527640 1 -- 172.21.15.62:0/249147703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e641049a0 msgr2=0x7f2e641089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:26.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.343+0000 7f2e6b527640 1 --2- 172.21.15.62:0/249147703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e641049a0 0x7f2e641089f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c009ca0 tx=0x7f2e5c02a8d0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.343+0000 7f2e6b527640 1 -- 172.21.15.62:0/249147703 shutdown_connections 2024-09-15T23:17:26.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.343+0000 7f2e6b527640 1 --2- 172.21.15.62:0/249147703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e64109120 0x7f2e64110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.343+0000 7f2e6b527640 1 --2- 172.21.15.62:0/249147703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e641049a0 0x7f2e641089f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.343+0000 7f2e6b527640 1 --2- 172.21.15.62:0/249147703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e64103ff0 0x7f2e641043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.343+0000 7f2e6b527640 1 -- 172.21.15.62:0/249147703 >> 172.21.15.62:0/249147703 conn(0x7f2e640ffab0 msgr2=0x7f2e64101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:26.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.343+0000 7f2e6b527640 1 -- 172.21.15.62:0/249147703 shutdown_connections 2024-09-15T23:17:26.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.344+0000 7f2e6b527640 1 -- 172.21.15.62:0/249147703 wait complete. 2024-09-15T23:17:26.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.345+0000 7f2e6b527640 1 Processor -- start 2024-09-15T23:17:26.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e6b527640 1 -- start start 2024-09-15T23:17:26.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e6b527640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e64103ff0 0x7f2e6419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e6b527640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e641049a0 0x7f2e6419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e6b527640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e64109120 0x7f2e6419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e6b527640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e64114170 con 0x7f2e641049a0 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e6b527640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e64113ff0 con 0x7f2e64103ff0 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e6b527640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2e641142f0 con 0x7f2e64109120 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e69ac4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e64109120 0x7f2e6419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e692c3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e64103ff0 0x7f2e6419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e68ac2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e641049a0 0x7f2e6419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e692c3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e64103ff0 0x7f2e6419a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41974/0 (socket says 172.21.15.62:41974) 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e69ac4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e64109120 0x7f2e6419ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56478/0 (socket says 172.21.15.62:56478) 2024-09-15T23:17:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.346+0000 7f2e692c3640 1 -- 172.21.15.62:0/2193249011 learned_addr learned my addr 172.21.15.62:0/2193249011 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:26.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.347+0000 7f2e69ac4640 1 -- 172.21.15.62:0/2193249011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e64103ff0 msgr2=0x7f2e6419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:26.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.347+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e64103ff0 0x7f2e6419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.347+0000 7f2e69ac4640 1 -- 172.21.15.62:0/2193249011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e641049a0 msgr2=0x7f2e6419ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:26.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.347+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e641049a0 0x7f2e6419ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.347+0000 7f2e69ac4640 1 -- 172.21.15.62:0/2193249011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e6419f5e0 con 0x7f2e64109120 2024-09-15T23:17:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.347+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e64109120 0x7f2e6419ef00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000e970 tx=0x7f2e6000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.348+0000 7f2e5a7fc640 1 -- 172.21.15.62:0/2193249011 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000ccc0 con 0x7f2e64109120 2024-09-15T23:17:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.348+0000 7f2e5a7fc640 1 -- 172.21.15.62:0/2193249011 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e60005020 con 0x7f2e64109120 2024-09-15T23:17:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.348+0000 7f2e5a7fc640 1 -- 172.21.15.62:0/2193249011 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e60004a00 con 0x7f2e64109120 2024-09-15T23:17:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.348+0000 7f2e6b527640 1 -- 172.21.15.62:0/2193249011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e6419f8d0 con 0x7f2e64109120 2024-09-15T23:17:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.348+0000 7f2e6b527640 1 -- 172.21.15.62:0/2193249011 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e641a71b0 con 0x7f2e64109120 2024-09-15T23:17:26.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.352+0000 7f2e5a7fc640 1 -- 172.21.15.62:0/2193249011 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2e600051d0 con 0x7f2e64109120 2024-09-15T23:17:26.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.354+0000 7f2e5a7fc640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e500a3380 0x7f2e500a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.355+0000 7f2e5a7fc640 1 -- 172.21.15.62:0/2193249011 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2e600cfb90 con 0x7f2e64109120 2024-09-15T23:17:26.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.355+0000 7f2e692c3640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e500a3380 0x7f2e500a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.355+0000 7f2e6d900740 1 Processor -- start 2024-09-15T23:17:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.356+0000 7f2e6d900740 1 -- start start 2024-09-15T23:17:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.356+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c1b40 0x55d9114c0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.356+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c05d0 0x55d9114c0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.356+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114cb370 0x55d9114cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.356+0000 7f2e6d900740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9114262f0 con 0x55d9114cb370 2024-09-15T23:17:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.356+0000 7f2e6d900740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d9113cdcf0 con 0x55d9114c1b40 2024-09-15T23:17:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.356+0000 7f2e6d900740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d9114222b0 con 0x55d9114c05d0 2024-09-15T23:17:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.356+0000 7f2e68ac2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c05d0 0x55d9114c0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.356+0000 7f2e69ac4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114cb370 0x55d9114cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.357+0000 7f2e69ac4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114cb370 0x55d9114cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43340/0 (socket says 172.21.15.62:43340) 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.357+0000 7f2e69ac4640 1 -- 172.21.15.62:0/918638616 learned_addr learned my addr 172.21.15.62:0/918638616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.357+0000 7f2e68ac2640 1 -- 172.21.15.62:0/918638616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c1b40 msgr2=0x55d9114c0090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.357+0000 7f2e68ac2640 1 --2- 172.21.15.62:0/918638616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c1b40 0x55d9114c0090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.357+0000 7f2e68ac2640 1 -- 172.21.15.62:0/918638616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114cb370 msgr2=0x55d9114cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.357+0000 7f2e68ac2640 1 --2- 172.21.15.62:0/918638616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114cb370 0x55d9114cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.357+0000 7f2e68ac2640 1 -- 172.21.15.62:0/918638616 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9113b23a0 con 0x55d9114c05d0 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.358+0000 7f2e692c3640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e500a3380 0x7f2e500a5840 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2e54005fd0 tx=0x7f2e5400b480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.358+0000 7f2e68ac2640 1 --2- 172.21.15.62:0/918638616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c05d0 0x55d9114c0a10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c005e30 tx=0x7f2e5c02f4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44fdeff0e330ba62 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:26.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.359+0000 7f2e4f7fe640 1 -- 172.21.15.62:0/918638616 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c031b10 con 0x55d9114c05d0 2024-09-15T23:17:26.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.359+0000 7f2e4f7fe640 1 -- 172.21.15.62:0/918638616 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e5c036030 con 0x55d9114c05d0 2024-09-15T23:17:26.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.359+0000 7f2e4f7fe640 1 -- 172.21.15.62:0/918638616 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c004040 con 0x55d9114c05d0 2024-09-15T23:17:26.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.360+0000 7f2e6d900740 1 -- 172.21.15.62:0/918638616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c05d0 msgr2=0x55d9114c0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:26.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.360+0000 7f2e6d900740 1 --2- 172.21.15.62:0/918638616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c05d0 0x55d9114c0a10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c005e30 tx=0x7f2e5c02f4d0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.361+0000 7f2e6d900740 1 -- 172.21.15.62:0/918638616 shutdown_connections 2024-09-15T23:17:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.361+0000 7f2e6d900740 1 --2- 172.21.15.62:0/918638616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114cb370 0x55d9114cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.361+0000 7f2e6d900740 1 --2- 172.21.15.62:0/918638616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c05d0 0x55d9114c0a10 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c005e30 tx=0x7f2e5c02f4d0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.361+0000 7f2e6d900740 1 --2- 172.21.15.62:0/918638616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c1b40 0x55d9114c0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.361+0000 7f2e6d900740 1 -- 172.21.15.62:0/918638616 >> 172.21.15.62:0/918638616 conn(0x55d911489310 msgr2=0x55d9114bcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:26.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.366+0000 7f2e6d900740 1 -- 172.21.15.62:0/918638616 shutdown_connections 2024-09-15T23:17:26.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.370+0000 7f2e6d900740 1 -- 172.21.15.62:0/918638616 wait complete. 2024-09-15T23:17:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.371+0000 7f2e6d900740 1 -- start start 2024-09-15T23:17:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.371+0000 7f2e6d900740 1 Processor -- start 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.372+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 0x55d9114c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.372+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114cb370 0x55d9114c8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.372+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c8d60 0x55d9114d74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.372+0000 7f2e6d900740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9114ce010 con 0x55d9114c1b40 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.372+0000 7f2e6d900740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d9113e1060 con 0x55d9114c8d60 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.372+0000 7f2e6d900740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d9114ce190 con 0x55d9114cb370 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.372+0000 7f2e6d900740 10 client.0 _ll_register_callbacks cb 0x55d9111348c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.373+0000 7f2e69ac4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c8d60 0x55d9114d74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.373+0000 7f2e69ac4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c8d60 0x55d9114d74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41996/0 (socket says 172.21.15.62:41996) 2024-09-15T23:17:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.373+0000 7f2e69ac4640 1 -- 172.21.15.62:0/1204575298 learned_addr learned my addr 172.21.15.62:0/1204575298 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.373+0000 7f2e69ac4640 1 -- 172.21.15.62:0/1204575298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114cb370 msgr2=0x55d9114c8820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:17:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.373+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114cb370 0x55d9114c8820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.373+0000 7f2e69ac4640 1 -- 172.21.15.62:0/1204575298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 msgr2=0x55d9114c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.373+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 0x55d9114c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.373+0000 7f2e69ac4640 1 -- 172.21.15.62:0/1204575298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9111973c0 con 0x55d9114c8d60 2024-09-15T23:17:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.373+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c8d60 0x55d9114d74c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000c6a0 tx=0x7f2e60005620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=365a6e957cbb32c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.374+0000 7f2e4cff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:17:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.374+0000 7f2e4cff9640 1 -- 172.21.15.62:0/1204575298 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e600b9060 con 0x55d9114c8d60 2024-09-15T23:17:26.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.374+0000 7f2e4cff9640 1 -- 172.21.15.62:0/1204575298 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2e600a47b0 con 0x55d9114c8d60 2024-09-15T23:17:26.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.374+0000 7f2e6d900740 1 -- 172.21.15.62:0/1204575298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e5c036030 con 0x55d9114c8d60 2024-09-15T23:17:26.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.375+0000 7f2e4cff9640 1 -- 172.21.15.62:0/1204575298 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2e6009f070 con 0x55d9114c8d60 2024-09-15T23:17:26.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.375+0000 7f2e6d900740 10 client.25415 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:26.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.375+0000 7f2e6d900740 1 -- 172.21.15.62:0/1204575298 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d9114da190 con 0x55d9114c8d60 2024-09-15T23:17:26.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.376+0000 7f2e4cff9640 1 -- 172.21.15.62:0/1204575298 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e600a0180 con 0x55d9114c8d60 2024-09-15T23:17:26.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.376+0000 7f2e4cff9640 1 -- 172.21.15.62:0/1204575298 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2e600b1070 con 0x55d9114c8d60 2024-09-15T23:17:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.378+0000 7f2e6d900740 10 client.25415 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.378+0000 7f2e6d900740 10 client.25415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.378+0000 7f2e6d900740 20 client.25415 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.378+0000 7f2e6d900740 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9114db4e0 0x55d9114dd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.378+0000 7f2e6d900740 4 client.25415 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.378+0000 7f2e6d900740 1 -- 172.21.15.62:0/1204575298 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2e5c036030 con 0x55d9114db4e0 2024-09-15T23:17:26.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.383+0000 7f2e692c3640 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9114db4e0 0x55d9114dd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.385+0000 7f2e4cff9640 10 client.25415 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:26.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.385+0000 7f2e692c3640 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9114db4e0 0x55d9114dd970 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e80b0259b5663a86 server_cookie=237b12b23c252525 in_seq=0 out_seq=0 2024-09-15T23:17:26.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.386+0000 7f2e4cff9640 1 -- 172.21.15.62:0/1204575298 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f2e5c036030 con 0x55d9114db4e0 2024-09-15T23:17:26.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.386+0000 7f2e4cff9640 10 client.25415 handle_command_reply: tid=0 2024-09-15T23:17:26.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.386+0000 7f2e6d900740 1 client.25415 shutdown 2024-09-15T23:17:26.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.387+0000 7f2e6d900740 1 -- 172.21.15.62:0/1204575298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c8d60 msgr2=0x55d9114d74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:26.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.387+0000 7f2e6d900740 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c8d60 0x55d9114d74c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000c6a0 tx=0x7f2e60005620 comp rx=0 tx=0).stop 2024-09-15T23:17:26.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.389+0000 7f2e6d900740 1 -- 172.21.15.62:0/1204575298 shutdown_connections 2024-09-15T23:17:26.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.389+0000 7f2e6d900740 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9114db4e0 0x55d9114dd970 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.389+0000 7f2e6d900740 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114c8d60 0x55d9114d74c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.389+0000 7f2e6d900740 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114cb370 0x55d9114c8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.389+0000 7f2e6d900740 1 --2- 172.21.15.62:0/1204575298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 0x55d9114c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.389+0000 7f2e6d900740 1 -- 172.21.15.62:0/1204575298 >> 172.21.15.62:0/1204575298 conn(0x55d911489310 msgr2=0x55d9114c2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:26.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.389+0000 7f2e4cff9640 0 client.25415 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:26.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.389+0000 7f2e6d900740 1 -- 172.21.15.62:0/1204575298 shutdown_connections 2024-09-15T23:17:26.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.389+0000 7f2e6d900740 1 -- 172.21.15.62:0/1204575298 wait complete. 2024-09-15T23:17:26.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.390+0000 7f2e6d900740 20 client.25415 trim_cache size 0 max 16384 2024-09-15T23:17:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.416+0000 7f2e6d900740 1 Processor -- start 2024-09-15T23:17:26.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.416+0000 7f2e6d900740 1 -- start start 2024-09-15T23:17:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 0x55d9114d72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c8d60 0x55d9114c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 0x55d9114e3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e6d900740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e641b3750 con 0x55d9114c1b40 2024-09-15T23:17:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e6d900740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e641b1020 con 0x55d9114cb370 2024-09-15T23:17:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e6d900740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2e641aee70 con 0x55d9114c8d60 2024-09-15T23:17:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e692c3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 0x55d9114d72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e692c3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 0x55d9114d72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43354/0 (socket says 172.21.15.62:43354) 2024-09-15T23:17:26.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e692c3640 1 -- 172.21.15.62:0/3430422472 learned_addr learned my addr 172.21.15.62:0/3430422472 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 0x55d9114e3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.417+0000 7f2e68ac2640 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c8d60 0x55d9114c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.418+0000 7f2e69ac4640 1 -- 172.21.15.62:0/3430422472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c8d60 msgr2=0x55d9114c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.418+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c8d60 0x55d9114c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.418+0000 7f2e69ac4640 1 -- 172.21.15.62:0/3430422472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 msgr2=0x55d9114d72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.418+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 0x55d9114d72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.418+0000 7f2e69ac4640 1 -- 172.21.15.62:0/3430422472 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9114c6830 con 0x55d9114cb370 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.418+0000 7f2e692c3640 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 0x55d9114d72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.418+0000 7f2e68ac2640 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c8d60 0x55d9114c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.418+0000 7f2e69ac4640 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 0x55d9114e3240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e6009b530 tx=0x7f2e6009bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12403fe8a746c714 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.419+0000 7f2e4effd640 1 -- 172.21.15.62:0/3430422472 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e600b9070 con 0x55d9114cb370 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.419+0000 7f2e4effd640 1 -- 172.21.15.62:0/3430422472 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e6009b220 con 0x55d9114cb370 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.419+0000 7f2e4effd640 1 -- 172.21.15.62:0/3430422472 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6001c040 con 0x55d9114cb370 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.420+0000 7f2e6d900740 1 -- 172.21.15.62:0/3430422472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 msgr2=0x55d9114e3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.420+0000 7f2e6d900740 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 0x55d9114e3240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2e6009b530 tx=0x7f2e6009bf50 comp rx=0 tx=0).stop 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.420+0000 7f2e6d900740 1 -- 172.21.15.62:0/3430422472 shutdown_connections 2024-09-15T23:17:26.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.420+0000 7f2e6d900740 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 0x55d9114e3240 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.420+0000 7f2e6d900740 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c8d60 0x55d9114c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.420+0000 7f2e6d900740 1 --2- 172.21.15.62:0/3430422472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c1b40 0x55d9114d72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.420+0000 7f2e6d900740 1 -- 172.21.15.62:0/3430422472 >> 172.21.15.62:0/3430422472 conn(0x55d911489310 msgr2=0x55d9114c6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:26.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.420+0000 7f2e6d900740 1 -- 172.21.15.62:0/3430422472 shutdown_connections 2024-09-15T23:17:26.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.420+0000 7f2e6d900740 1 -- 172.21.15.62:0/3430422472 wait complete. 2024-09-15T23:17:26.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.421+0000 7f2e6d900740 1 -- start start 2024-09-15T23:17:26.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.421+0000 7f2e6d900740 1 Processor -- start 2024-09-15T23:17:26.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.422+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c1b40 0x55d9114d3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.422+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c8d60 0x55d9114cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.423+0000 7f2e6d900740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 0x55d9114cc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.423+0000 7f2e6d900740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2e641af4e0 con 0x55d9114c8d60 2024-09-15T23:17:26.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.423+0000 7f2e6d900740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e641b0950 con 0x55d9114cb370 2024-09-15T23:17:26.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.423+0000 7f2e6d900740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2e641afbb0 con 0x55d9114c1b40 2024-09-15T23:17:26.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.423+0000 7f2e6d900740 10 client.0 _ll_register_callbacks cb 0x7f2e600cd1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.423+0000 7f2e68ac2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c8d60 0x55d9114cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.423+0000 7f2e68ac2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c8d60 0x55d9114cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43364/0 (socket says 172.21.15.62:43364) 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.423+0000 7f2e68ac2640 1 -- 172.21.15.62:0/4147660863 learned_addr learned my addr 172.21.15.62:0/4147660863 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.424+0000 7f2e68ac2640 1 -- 172.21.15.62:0/4147660863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c1b40 msgr2=0x55d9114d3a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.424+0000 7f2e68ac2640 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c1b40 0x55d9114d3a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.424+0000 7f2e68ac2640 1 -- 172.21.15.62:0/4147660863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 msgr2=0x55d9114cc520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.424+0000 7f2e68ac2640 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 0x55d9114cc520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.424+0000 7f2e68ac2640 1 -- 172.21.15.62:0/4147660863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9114ccbe0 con 0x55d9114c8d60 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.425+0000 7f2e68ac2640 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c8d60 0x55d9114cbfe0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c02aec0 tx=0x7f2e5c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69082b6f8acda7a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:26.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.425+0000 7f2e4cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:17:26.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.425+0000 7f2e4cff9640 1 -- 172.21.15.62:0/4147660863 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5c057060 con 0x55d9114c8d60 2024-09-15T23:17:26.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.425+0000 7f2e4cff9640 1 -- 172.21.15.62:0/4147660863 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2e5c031460 con 0x55d9114c8d60 2024-09-15T23:17:26.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.426+0000 7f2e6d900740 1 -- 172.21.15.62:0/4147660863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e6009b220 con 0x55d9114c8d60 2024-09-15T23:17:26.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.426+0000 7f2e4cff9640 1 -- 172.21.15.62:0/4147660863 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2e5c003e00 con 0x55d9114c8d60 2024-09-15T23:17:26.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.426+0000 7f2e6d900740 10 client.15756 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:26.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.426+0000 7f2e6d900740 1 -- 172.21.15.62:0/4147660863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d91154ea20 con 0x55d9114c8d60 2024-09-15T23:17:26.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.427+0000 7f2e4cff9640 1 -- 172.21.15.62:0/4147660863 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e5c031680 con 0x55d9114c8d60 2024-09-15T23:17:26.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.427+0000 7f2e4cff9640 1 -- 172.21.15.62:0/4147660863 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2e5c043070 con 0x55d9114c8d60 2024-09-15T23:17:26.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.427+0000 7f2e6d900740 10 client.15756 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:26.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.427+0000 7f2e6d900740 10 client.15756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:26.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.428+0000 7f2e6d900740 20 client.15756 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:26.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.428+0000 7f2e6d900740 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d91154ebf0 0x55d911550fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:26.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.428+0000 7f2e6d900740 4 client.15756 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:17:26.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.428+0000 7f2e6d900740 1 -- 172.21.15.62:0/4147660863 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f2e6009b220 con 0x55d91154ebf0 2024-09-15T23:17:26.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.428+0000 7f2e692c3640 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d91154ebf0 0x55d911550fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:26.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.429+0000 7f2e692c3640 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d91154ebf0 0x55d911550fe0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e18c4ae1fcee8f0 server_cookie=9ca7559ae92f7c09 in_seq=0 out_seq=0 2024-09-15T23:17:26.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.429+0000 7f2e4cff9640 10 client.15756 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:26.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.430+0000 7f2e4cff9640 1 -- 172.21.15.62:0/4147660863 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f2e6009b220 con 0x55d91154ebf0 2024-09-15T23:17:26.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.430+0000 7f2e4cff9640 10 client.15756 handle_command_reply: tid=0 2024-09-15T23:17:26.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.430+0000 7f2e6d900740 1 client.15756 shutdown 2024-09-15T23:17:26.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.430+0000 7f2e6d900740 1 -- 172.21.15.62:0/4147660863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c8d60 msgr2=0x55d9114cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:26.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.430+0000 7f2e6d900740 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c8d60 0x55d9114cbfe0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c02aec0 tx=0x7f2e5c005ef0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e6d900740 1 -- 172.21.15.62:0/4147660863 shutdown_connections 2024-09-15T23:17:26.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e6d900740 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d91154ebf0 0x55d911550fe0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e6d900740 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9114cb370 0x55d9114cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e6d900740 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9114c8d60 0x55d9114cbfe0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e6d900740 1 --2- 172.21.15.62:0/4147660863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9114c1b40 0x55d9114d3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e6d900740 1 -- 172.21.15.62:0/4147660863 >> 172.21.15.62:0/4147660863 conn(0x55d911489310 msgr2=0x55d9114c73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:26.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e4cff9640 0 client.15756 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:26.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e6d900740 1 -- 172.21.15.62:0/4147660863 shutdown_connections 2024-09-15T23:17:26.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e6d900740 1 -- 172.21.15.62:0/4147660863 wait complete. 2024-09-15T23:17:26.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.431+0000 7f2e6d900740 20 client.15756 trim_cache size 0 max 16384 2024-09-15T23:17:26.434 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:17:26.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 -- 172.21.15.62:0/2193249011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e500a3380 msgr2=0x7f2e500a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:26.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e500a3380 0x7f2e500a5840 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2e54005fd0 tx=0x7f2e5400b480 comp rx=0 tx=0).stop 2024-09-15T23:17:26.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 -- 172.21.15.62:0/2193249011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e64109120 msgr2=0x7f2e6419ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:26.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e64109120 0x7f2e6419ef00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000e970 tx=0x7f2e6000ee30 comp rx=0 tx=0).stop 2024-09-15T23:17:26.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 -- 172.21.15.62:0/2193249011 shutdown_connections 2024-09-15T23:17:26.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2e500a3380 0x7f2e500a5840 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2e64109120 0x7f2e6419ef00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e641049a0 0x7f2e6419ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 --2- 172.21.15.62:0/2193249011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e64103ff0 0x7f2e6419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:26.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.432+0000 7f2e6b527640 1 -- 172.21.15.62:0/2193249011 >> 172.21.15.62:0/2193249011 conn(0x7f2e640ffab0 msgr2=0x7f2e64100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:26.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.433+0000 7f2e6b527640 1 -- 172.21.15.62:0/2193249011 shutdown_connections 2024-09-15T23:17:26.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:26.433+0000 7f2e6b527640 1 -- 172.21.15.62:0/2193249011 wait complete. 2024-09-15T23:17:27.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:27 smithi062 ceph-mon[30949]: pgmap v405: 129 pgs: 129 active+clean; 982 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 14 MiB/s wr, 948 op/s 2024-09-15T23:17:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:27 smithi067 ceph-mon[35665]: pgmap v405: 129 pgs: 129 active+clean; 982 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 14 MiB/s wr, 948 op/s 2024-09-15T23:17:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:27 smithi148 ceph-mon[35610]: pgmap v405: 129 pgs: 129 active+clean; 982 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 14 MiB/s wr, 948 op/s 2024-09-15T23:17:28.320 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:17:28.320 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:17:28.709 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:29 smithi062 ceph-mon[30949]: pgmap v406: 129 pgs: 129 active+clean; 982 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 14 MiB/s wr, 957 op/s 2024-09-15T23:17:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:29 smithi067 ceph-mon[35665]: pgmap v406: 129 pgs: 129 active+clean; 982 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 14 MiB/s wr, 957 op/s 2024-09-15T23:17:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:29 smithi148 ceph-mon[35610]: pgmap v406: 129 pgs: 129 active+clean; 982 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 14 MiB/s wr, 957 op/s 2024-09-15T23:17:29.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 70.0 s, 3291.0 tps, lat 30.421 ms stddev 38.004 2024-09-15T23:17:29.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.783+0000 7f234a310640 1 -- 172.21.15.62:0/4068806565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234410ea80 msgr2=0x7f23441154f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:29.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.783+0000 7f234a310640 1 --2- 172.21.15.62:0/4068806565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234410ea80 0x7f23441154f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f233c009ca0 tx=0x7f233c02b9e0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.788+0000 7f234a310640 1 -- 172.21.15.62:0/4068806565 shutdown_connections 2024-09-15T23:17:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.788+0000 7f234a310640 1 --2- 172.21.15.62:0/4068806565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f234410ea80 0x7f23441154f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.788+0000 7f234a310640 1 --2- 172.21.15.62:0/4068806565 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2344109e00 0x7f234410e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.788+0000 7f234a310640 1 --2- 172.21.15.62:0/4068806565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2344109450 0x7f2344109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.788+0000 7f234a310640 1 -- 172.21.15.62:0/4068806565 >> 172.21.15.62:0/4068806565 conn(0x7f234406e2d0 msgr2=0x7f234406e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:29.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.790+0000 7f234a310640 1 -- 172.21.15.62:0/4068806565 shutdown_connections 2024-09-15T23:17:29.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.792+0000 7f234a310640 1 -- 172.21.15.62:0/4068806565 wait complete. 2024-09-15T23:17:29.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.793+0000 7f234a310640 1 Processor -- start 2024-09-15T23:17:29.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.793+0000 7f234a310640 1 -- start start 2024-09-15T23:17:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.794+0000 7f234a310640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2344109450 0x7f2344112ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.794+0000 7f234a310640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2344109e00 0x7f2344113410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.794+0000 7f234a310640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234410ea80 0x7f234411f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.794+0000 7f234a310640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2344118350 con 0x7f234410ea80 2024-09-15T23:17:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.794+0000 7f234a310640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f23441181d0 con 0x7f2344109450 2024-09-15T23:17:29.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.794+0000 7f234a310640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f23441184d0 con 0x7f2344109e00 2024-09-15T23:17:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.794+0000 7f23437fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2344109e00 0x7f2344113410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.795+0000 7f23437fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2344109e00 0x7f2344113410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56530/0 (socket says 172.21.15.62:56530) 2024-09-15T23:17:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.795+0000 7f23437fe640 1 -- 172.21.15.62:0/3116533127 learned_addr learned my addr 172.21.15.62:0/3116533127 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:29.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.795+0000 7f2343fff640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2344109450 0x7f2344112ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.795+0000 7f23488ad640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234410ea80 0x7f234411f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.796+0000 7f23437fe640 1 -- 172.21.15.62:0/3116533127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2344109450 msgr2=0x7f2344112ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:29.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.796+0000 7f23437fe640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2344109450 0x7f2344112ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.796+0000 7f23437fe640 1 -- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234410ea80 msgr2=0x7f234411f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.796+0000 7f23437fe640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234410ea80 0x7f234411f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.796+0000 7f23437fe640 1 -- 172.21.15.62:0/3116533127 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234411fe60 con 0x7f2344109e00 2024-09-15T23:17:29.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.797+0000 7f23437fe640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2344109e00 0x7f2344113410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f233400d770 tx=0x7f233400dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:29.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.797+0000 7f23417fa640 1 -- 172.21.15.62:0/3116533127 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2334017070 con 0x7f2344109e00 2024-09-15T23:17:29.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.797+0000 7f234a310640 1 -- 172.21.15.62:0/3116533127 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2344120150 con 0x7f2344109e00 2024-09-15T23:17:29.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.797+0000 7f234a310640 1 -- 172.21.15.62:0/3116533127 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2344124840 con 0x7f2344109e00 2024-09-15T23:17:29.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.798+0000 7f23417fa640 1 -- 172.21.15.62:0/3116533127 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2334009c30 con 0x7f2344109e00 2024-09-15T23:17:29.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.798+0000 7f23417fa640 1 -- 172.21.15.62:0/3116533127 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233400f6b0 con 0x7f2344109e00 2024-09-15T23:17:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.802+0000 7f234c6e9740 1 Processor -- start 2024-09-15T23:17:29.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.802+0000 7f234c6e9740 1 -- start start 2024-09-15T23:17:29.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e780f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e781480 0x556b4e78b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e78bd40 0x556b4e78e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f234c6e9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556b4e6e6280 con 0x556b4e780b60 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f234c6e9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556b4e68db10 con 0x556b4e78bd40 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f234c6e9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556b4e6e2240 con 0x556b4e781480 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f23488ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e780f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f2343fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e781480 0x556b4e78b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f23488ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e780f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43376/0 (socket says 172.21.15.62:43376) 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f23488ad640 1 -- 172.21.15.62:0/2958021818 learned_addr learned my addr 172.21.15.62:0/2958021818 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f23417fa640 1 -- 172.21.15.62:0/3116533127 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f233401a070 con 0x7f2344109e00 2024-09-15T23:17:29.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.803+0000 7f23437fe640 1 --2- 172.21.15.62:0/2958021818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e78bd40 0x556b4e78e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.804+0000 7f23417fa640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f232c0a33d0 0x7f232c0a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.805+0000 7f2343fff640 1 -- 172.21.15.62:0/2958021818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e78bd40 msgr2=0x556b4e78e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:29.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.805+0000 7f2343fff640 1 --2- 172.21.15.62:0/2958021818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e78bd40 0x556b4e78e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.805+0000 7f2343fff640 1 -- 172.21.15.62:0/2958021818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 msgr2=0x556b4e780f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:29.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.805+0000 7f2343fff640 1 --2- 172.21.15.62:0/2958021818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e780f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.805+0000 7f2343fff640 1 -- 172.21.15.62:0/2958021818 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b4e69a1f0 con 0x556b4e781480 2024-09-15T23:17:29.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.805+0000 7f23417fa640 1 -- 172.21.15.62:0/3116533127 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f23340cea40 con 0x7f2344109e00 2024-09-15T23:17:29.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.805+0000 7f23488ad640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f232c0a33d0 0x7f232c0a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.805+0000 7f23437fe640 1 --2- 172.21.15.62:0/2958021818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e78bd40 0x556b4e78e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:17:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.806+0000 7f2343fff640 1 --2- 172.21.15.62:0/2958021818 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e781480 0x556b4e78b800 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f233800bbf0 tx=0x7f23380107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51c2ffcc6b44f114 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.806+0000 7f232a7fc640 1 -- 172.21.15.62:0/2958021818 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338004030 con 0x556b4e781480 2024-09-15T23:17:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.806+0000 7f232a7fc640 1 -- 172.21.15.62:0/2958021818 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23380041d0 con 0x556b4e781480 2024-09-15T23:17:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.806+0000 7f232a7fc640 1 -- 172.21.15.62:0/2958021818 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23380116c0 con 0x556b4e781480 2024-09-15T23:17:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.807+0000 7f234c6e9740 1 -- 172.21.15.62:0/2958021818 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e781480 msgr2=0x556b4e78b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:29.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.807+0000 7f234c6e9740 1 --2- 172.21.15.62:0/2958021818 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e781480 0x556b4e78b800 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f233800bbf0 tx=0x7f23380107a0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.808+0000 7f234c6e9740 1 -- 172.21.15.62:0/2958021818 shutdown_connections 2024-09-15T23:17:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.808+0000 7f234c6e9740 1 --2- 172.21.15.62:0/2958021818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e78bd40 0x556b4e78e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.808+0000 7f234c6e9740 1 --2- 172.21.15.62:0/2958021818 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e781480 0x556b4e78b800 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.808+0000 7f234c6e9740 1 --2- 172.21.15.62:0/2958021818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e780f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.808+0000 7f234c6e9740 1 -- 172.21.15.62:0/2958021818 >> 172.21.15.62:0/2958021818 conn(0x556b4e77d0f0 msgr2=0x556b4e77d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.809+0000 7f23488ad640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f232c0a33d0 0x7f232c0a5890 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f233c005fa0 tx=0x7f233c02b9e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:29.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.809+0000 7f234c6e9740 1 -- 172.21.15.62:0/2958021818 shutdown_connections 2024-09-15T23:17:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.810+0000 7f234c6e9740 1 -- 172.21.15.62:0/2958021818 wait complete. 2024-09-15T23:17:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.810+0000 7f234c6e9740 1 -- start start 2024-09-15T23:17:29.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f234c6e9740 1 Processor -- start 2024-09-15T23:17:29.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e780b60 0x556b4e788b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e781480 0x556b4e7890d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e789610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f234c6e9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556b4e78e910 con 0x556b4e781480 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f234c6e9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556b4e6a0f10 con 0x556b4e780b60 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f234c6e9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556b4e78ea90 con 0x556b4e78bd40 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f234c6e9740 10 client.0 _ll_register_callbacks cb 0x556b4e669a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f2343fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e780b60 0x556b4e788b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f2343fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e780b60 0x556b4e788b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42046/0 (socket says 172.21.15.62:42046) 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.811+0000 7f2343fff640 1 -- 172.21.15.62:0/3303193622 learned_addr learned my addr 172.21.15.62:0/3303193622 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.812+0000 7f2343fff640 1 -- 172.21.15.62:0/3303193622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 msgr2=0x556b4e789610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.812+0000 7f2343fff640 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e789610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.812+0000 7f2343fff640 1 -- 172.21.15.62:0/3303193622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e781480 msgr2=0x556b4e7890d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.812+0000 7f2343fff640 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e781480 0x556b4e7890d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.812+0000 7f2343fff640 1 -- 172.21.15.62:0/3303193622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b4e4576a0 con 0x556b4e780b60 2024-09-15T23:17:29.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.812+0000 7f2343fff640 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e780b60 0x556b4e788b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2338011010 tx=0x7f233800bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94ebbb282cec3782 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.812+0000 7f23237fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.812+0000 7f23237fe640 1 -- 172.21.15.62:0/3303193622 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338005040 con 0x556b4e780b60 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f23237fe640 1 -- 172.21.15.62:0/3303193622 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f23380051e0 con 0x556b4e780b60 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f234c6e9740 1 -- 172.21.15.62:0/3303193622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23380041d0 con 0x556b4e780b60 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f23237fe640 1 -- 172.21.15.62:0/3303193622 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f23380053e0 con 0x556b4e780b60 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f234c6e9740 10 client.25427 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f234c6e9740 1 -- 172.21.15.62:0/3303193622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556b4e79a4a0 con 0x556b4e780b60 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f23237fe640 1 -- 172.21.15.62:0/3303193622 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23380041d0 con 0x556b4e780b60 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f23237fe640 1 -- 172.21.15.62:0/3303193622 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f2338004a50 con 0x556b4e780b60 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f234c6e9740 10 client.25427 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f234c6e9740 10 client.25427 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f234c6e9740 20 client.25427 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:29.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556b4e79be20 0x556b4e79e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f234c6e9740 4 client.25427 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:29.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.813+0000 7f234c6e9740 1 -- 172.21.15.62:0/3303193622 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b4e6f37f0 con 0x556b4e79be20 2024-09-15T23:17:29.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.814+0000 7f23437fe640 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556b4e79be20 0x556b4e79e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.814+0000 7f23237fe640 10 client.25427 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.814+0000 7f23437fe640 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556b4e79be20 0x556b4e79e2e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2b186137eef809f server_cookie=d209dc52429b4f49 in_seq=0 out_seq=0 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f23237fe640 1 -- 172.21.15.62:0/3303193622 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556b4e6f37f0 con 0x556b4e79be20 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f23237fe640 10 client.25427 handle_command_reply: tid=0 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 client.25427 shutdown 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 -- 172.21.15.62:0/3303193622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e780b60 msgr2=0x556b4e788b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e780b60 0x556b4e788b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2338011010 tx=0x7f233800bfd0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 -- 172.21.15.62:0/3303193622 shutdown_connections 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556b4e79be20 0x556b4e79e2e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e789610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e781480 0x556b4e7890d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3303193622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e780b60 0x556b4e788b90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 -- 172.21.15.62:0/3303193622 >> 172.21.15.62:0/3303193622 conn(0x556b4e77d0f0 msgr2=0x556b4e77c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:29.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f23237fe640 0 client.25427 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:29.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 -- 172.21.15.62:0/3303193622 shutdown_connections 2024-09-15T23:17:29.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 1 -- 172.21.15.62:0/3303193622 wait complete. 2024-09-15T23:17:29.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.815+0000 7f234c6e9740 20 client.25427 trim_cache size 0 max 16384 2024-09-15T23:17:29.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.843+0000 7f234c6e9740 1 Processor -- start 2024-09-15T23:17:29.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.843+0000 7f234c6e9740 1 -- start start 2024-09-15T23:17:29.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e78a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e7a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e7a4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f234c6e9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f23300123e0 con 0x556b4e780b60 2024-09-15T23:17:29.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f234c6e9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f233000fcb0 con 0x556b4e781480 2024-09-15T23:17:29.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f234c6e9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f233000db00 con 0x556b4e78bd40 2024-09-15T23:17:29.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f23437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e7a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f23437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e7a3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42060/0 (socket says 172.21.15.62:42060) 2024-09-15T23:17:29.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f23437fe640 1 -- 172.21.15.62:0/3151454 learned_addr learned my addr 172.21.15.62:0/3151454 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:29.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.844+0000 7f2343fff640 1 --2- 172.21.15.62:0/3151454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e78a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.845+0000 7f23488ad640 1 --2- 172.21.15.62:0/3151454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e7a4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.845+0000 7f23437fe640 1 -- 172.21.15.62:0/3151454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 msgr2=0x556b4e7a4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:29.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.845+0000 7f23437fe640 1 --2- 172.21.15.62:0/3151454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e7a4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.845+0000 7f23437fe640 1 -- 172.21.15.62:0/3151454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 msgr2=0x556b4e78a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:29.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.845+0000 7f23437fe640 1 --2- 172.21.15.62:0/3151454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e78a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.845+0000 7f23437fe640 1 -- 172.21.15.62:0/3151454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b4e7812b0 con 0x556b4e781480 2024-09-15T23:17:29.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.845+0000 7f23437fe640 1 --2- 172.21.15.62:0/3151454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e7a3bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2334009610 tx=0x7f23340ce040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e48e2c6a51d6d137 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:29.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.846+0000 7f2329ffb640 1 -- 172.21.15.62:0/3151454 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2334009de0 con 0x556b4e781480 2024-09-15T23:17:29.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.846+0000 7f2329ffb640 1 -- 172.21.15.62:0/3151454 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556b4e6f37f0 con 0x556b4e781480 2024-09-15T23:17:29.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.846+0000 7f2329ffb640 1 -- 172.21.15.62:0/3151454 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2334098e90 con 0x556b4e781480 2024-09-15T23:17:29.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.847+0000 7f234c6e9740 1 -- 172.21.15.62:0/3151454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 msgr2=0x556b4e7a3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:29.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.847+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3151454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e7a3bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2334009610 tx=0x7f23340ce040 comp rx=0 tx=0).stop 2024-09-15T23:17:29.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.847+0000 7f234c6e9740 1 -- 172.21.15.62:0/3151454 shutdown_connections 2024-09-15T23:17:29.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.847+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3151454 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e7a4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.847+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3151454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e7a3bf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.847+0000 7f234c6e9740 1 --2- 172.21.15.62:0/3151454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e78a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.847+0000 7f234c6e9740 1 -- 172.21.15.62:0/3151454 >> 172.21.15.62:0/3151454 conn(0x556b4e77d0f0 msgr2=0x556b4e78a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:29.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.848+0000 7f234c6e9740 1 -- 172.21.15.62:0/3151454 shutdown_connections 2024-09-15T23:17:29.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.848+0000 7f234c6e9740 1 -- 172.21.15.62:0/3151454 wait complete. 2024-09-15T23:17:29.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.848+0000 7f234c6e9740 1 -- start start 2024-09-15T23:17:29.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.849+0000 7f234c6e9740 1 Processor -- start 2024-09-15T23:17:29.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.849+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e7943d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.849+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e785780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.849+0000 7f234c6e9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e785cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f234c6e9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f233000e170 con 0x556b4e780b60 2024-09-15T23:17:29.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f234c6e9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f233000f5e0 con 0x556b4e781480 2024-09-15T23:17:29.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f234c6e9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f233000e840 con 0x556b4e78bd40 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.849+0000 7f23437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e785780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f234c6e9740 10 client.0 _ll_register_callbacks cb 0x7f233c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f23437fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e785780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42066/0 (socket says 172.21.15.62:42066) 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f2343fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e7943d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f23437fe640 1 -- 172.21.15.62:0/4040320605 learned_addr learned my addr 172.21.15.62:0/4040320605 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f23488ad640 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e785cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f2343fff640 1 -- 172.21.15.62:0/4040320605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 msgr2=0x556b4e785cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f2343fff640 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e785cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f2343fff640 1 -- 172.21.15.62:0/4040320605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 msgr2=0x556b4e785780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f2343fff640 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e785780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.850+0000 7f2343fff640 1 -- 172.21.15.62:0/4040320605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b4e786360 con 0x556b4e780b60 2024-09-15T23:17:29.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.851+0000 7f2343fff640 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e7943d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2338003ef0 tx=0x7f2338003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfdc9cf09a3f216 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:29.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.851+0000 7f23237fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:17:29.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.851+0000 7f23237fe640 1 -- 172.21.15.62:0/4040320605 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23380040c0 con 0x556b4e780b60 2024-09-15T23:17:29.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.851+0000 7f23237fe640 1 -- 172.21.15.62:0/4040320605 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2338011890 con 0x556b4e780b60 2024-09-15T23:17:29.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.851+0000 7f234c6e9740 1 -- 172.21.15.62:0/4040320605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556b4e6f37f0 con 0x556b4e780b60 2024-09-15T23:17:29.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f23237fe640 1 -- 172.21.15.62:0/4040320605 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2338011a90 con 0x556b4e780b60 2024-09-15T23:17:29.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f234c6e9740 10 client.15771 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:29.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f234c6e9740 1 -- 172.21.15.62:0/4040320605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556b4e806a90 con 0x556b4e780b60 2024-09-15T23:17:29.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f23237fe640 1 -- 172.21.15.62:0/4040320605 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556b4e6f37f0 con 0x556b4e780b60 2024-09-15T23:17:29.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f23237fe640 1 -- 172.21.15.62:0/4040320605 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f23380035f0 con 0x556b4e780b60 2024-09-15T23:17:29.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f234c6e9740 10 client.15771 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:29.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f234c6e9740 10 client.15771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:29.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f234c6e9740 20 client.15771 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:29.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f234c6e9740 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556b4e807100 0x556b4e809520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:29.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f234c6e9740 4 client.15771 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:17:29.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.854+0000 7f234c6e9740 1 -- 172.21.15.62:0/4040320605 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556b4e6edd00 con 0x556b4e807100 2024-09-15T23:17:29.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.855+0000 7f23437fe640 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556b4e807100 0x556b4e809520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:29.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.855+0000 7f23237fe640 10 client.15771 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:29.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.855+0000 7f23437fe640 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556b4e807100 0x556b4e809520 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f920125b6cad6f6 server_cookie=d02ac2dd8e17128b in_seq=0 out_seq=0 2024-09-15T23:17:29.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f23237fe640 1 -- 172.21.15.62:0/4040320605 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556b4e6edd00 con 0x556b4e807100 2024-09-15T23:17:29.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f23237fe640 10 client.15771 handle_command_reply: tid=0 2024-09-15T23:17:29.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 client.15771 shutdown 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 -- 172.21.15.62:0/4040320605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 msgr2=0x556b4e7943d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e7943d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2338003ef0 tx=0x7f2338003f90 comp rx=0 tx=0).stop 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 -- 172.21.15.62:0/4040320605 shutdown_connections 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556b4e807100 0x556b4e809520 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556b4e78bd40 0x556b4e785cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556b4e781480 0x556b4e785780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 --2- 172.21.15.62:0/4040320605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b4e780b60 0x556b4e7943d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 -- 172.21.15.62:0/4040320605 >> 172.21.15.62:0/4040320605 conn(0x556b4e77d0f0 msgr2=0x556b4e77c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f23237fe640 0 client.15771 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 -- 172.21.15.62:0/4040320605 shutdown_connections 2024-09-15T23:17:29.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.857+0000 7f234c6e9740 1 -- 172.21.15.62:0/4040320605 wait complete. 2024-09-15T23:17:29.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.858+0000 7f234c6e9740 20 client.15771 trim_cache size 0 max 16384 2024-09-15T23:17:29.860 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:17:29.860 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:17:29.860 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "e9ecaca8-5a24-4e19-8d7b-d27638b0cb24", 2024-09-15T23:17:29.860 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:17:29.860 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.858+0000 7f234a310640 1 -- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f232c0a33d0 msgr2=0x7f232c0a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.858+0000 7f234a310640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f232c0a33d0 0x7f232c0a5890 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f233c005fa0 tx=0x7f233c02b9e0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.858+0000 7f234a310640 1 -- 172.21.15.62:0/3116533127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2344109e00 msgr2=0x7f2344113410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.858+0000 7f234a310640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2344109e00 0x7f2344113410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f233400d770 tx=0x7f233400dc30 comp rx=0 tx=0).stop 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.859+0000 7f234a310640 1 -- 172.21.15.62:0/3116533127 shutdown_connections 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.859+0000 7f234a310640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f232c0a33d0 0x7f232c0a5890 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.859+0000 7f234a310640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f234410ea80 0x7f234411f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.859+0000 7f234a310640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2344109e00 0x7f2344113410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.859+0000 7f234a310640 1 --2- 172.21.15.62:0/3116533127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2344109450 0x7f2344112ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.859+0000 7f234a310640 1 -- 172.21.15.62:0/3116533127 >> 172.21.15.62:0/3116533127 conn(0x7f234406e2d0 msgr2=0x7f23440726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.859+0000 7f234a310640 1 -- 172.21.15.62:0/3116533127 shutdown_connections 2024-09-15T23:17:29.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:29.859+0000 7f234a310640 1 -- 172.21.15.62:0/3116533127 wait complete. 2024-09-15T23:17:30.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:30 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:17:30.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:30 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:30.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:30 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:17:30.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:30 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:17:30.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:30 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:30.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:30 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:17:30.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:30 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:17:30.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:30 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:30.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:30 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:17:30.580 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:17:30.998 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:31.405 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:31 smithi062 ceph-mon[30949]: pgmap v407: 129 pgs: 129 active+clean; 982 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 810 op/s 2024-09-15T23:17:31.405 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:31 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:17:31.405 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:31 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.405 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:31 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:17:31.405 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:31 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:17:31.405 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:31 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.406 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:31 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.406 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:31 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:31 smithi067 ceph-mon[35665]: pgmap v407: 129 pgs: 129 active+clean; 982 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 810 op/s 2024-09-15T23:17:31.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:31 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:17:31.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:31 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:31 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:17:31.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:31 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:17:31.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:31 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:31 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:31 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:31 smithi148 ceph-mon[35610]: pgmap v407: 129 pgs: 129 active+clean; 982 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 810 op/s 2024-09-15T23:17:31.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:31 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:17:31.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:31 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:31 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:17:31.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:31 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:17:31.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:31 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:31 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:31 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:17:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.538+0000 7fdc3eb72640 1 -- 172.21.15.62:0/3343185416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc3806ee10 msgr2=0x7fdc38114570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.538+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/3343185416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc3806ee10 0x7fdc38114570 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009c10 tx=0x7fdc3402a6e0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.539+0000 7fdc3eb72640 1 -- 172.21.15.62:0/3343185416 shutdown_connections 2024-09-15T23:17:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.539+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/3343185416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc38114ab0 0x7fdc38116ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.539+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/3343185416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc3806ee10 0x7fdc38114570 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.539+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/3343185416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc3806e4f0 0x7fdc3806e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.539+0000 7fdc3eb72640 1 -- 172.21.15.62:0/3343185416 >> 172.21.15.62:0/3343185416 conn(0x7fdc38073180 msgr2=0x7fdc38073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.540+0000 7fdc3eb72640 1 -- 172.21.15.62:0/3343185416 shutdown_connections 2024-09-15T23:17:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.540+0000 7fdc3eb72640 1 -- 172.21.15.62:0/3343185416 wait complete. 2024-09-15T23:17:31.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.541+0000 7fdc3eb72640 1 Processor -- start 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.541+0000 7fdc3eb72640 1 -- start start 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.542+0000 7fdc3eb72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3806e4f0 0x7fdc381abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.542+0000 7fdc3eb72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc38114ab0 0x7fdc381ac460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.542+0000 7fdc3eb72640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc381139e0 0x7fdc38113e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.542+0000 7fdc3eb72640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3811a550 con 0x7fdc381139e0 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.542+0000 7fdc3eb72640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3811a3d0 con 0x7fdc3806e4f0 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.542+0000 7fdc3eb72640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3811a6d0 con 0x7fdc38114ab0 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.542+0000 7fdc3e371640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc381139e0 0x7fdc38113e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.542+0000 7fdc3e371640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc381139e0 0x7fdc38113e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43410/0 (socket says 172.21.15.62:43410) 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.542+0000 7fdc3e371640 1 -- 172.21.15.62:0/172521025 learned_addr learned my addr 172.21.15.62:0/172521025 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.543+0000 7fdc3e371640 1 -- 172.21.15.62:0/172521025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc38114ab0 msgr2=0x7fdc381ac460 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:17:31.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.543+0000 7fdc3e371640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc38114ab0 0x7fdc381ac460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.543+0000 7fdc3e371640 1 -- 172.21.15.62:0/172521025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3806e4f0 msgr2=0x7fdc381abf00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:17:31.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.543+0000 7fdc3e371640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3806e4f0 0x7fdc381abf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.543+0000 7fdc3e371640 1 -- 172.21.15.62:0/172521025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc38114630 con 0x7fdc381139e0 2024-09-15T23:17:31.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.544+0000 7fdc3e371640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc381139e0 0x7fdc38113e90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000e940 tx=0x7fdc3000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:31.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.545+0000 7fdc2effd640 1 -- 172.21.15.62:0/172521025 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3000cd00 con 0x7fdc381139e0 2024-09-15T23:17:31.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.545+0000 7fdc2effd640 1 -- 172.21.15.62:0/172521025 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc300035f0 con 0x7fdc381139e0 2024-09-15T23:17:31.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.546+0000 7fdc2effd640 1 -- 172.21.15.62:0/172521025 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3000cea0 con 0x7fdc381139e0 2024-09-15T23:17:31.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.546+0000 7fdc3eb72640 1 -- 172.21.15.62:0/172521025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc381b3e10 con 0x7fdc381139e0 2024-09-15T23:17:31.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.546+0000 7fdc3eb72640 1 -- 172.21.15.62:0/172521025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc381b41b0 con 0x7fdc381139e0 2024-09-15T23:17:31.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.554+0000 7fdc2effd640 1 -- 172.21.15.62:0/172521025 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fdc30004000 con 0x7fdc381139e0 2024-09-15T23:17:31.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.555+0000 7fdc421ad740 1 Processor -- start 2024-09-15T23:17:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.556+0000 7fdc421ad740 1 -- start start 2024-09-15T23:17:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.556+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f28a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.556+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28f90 0x55b353f33310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.556+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f33850 0x55b353f35c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.556+0000 7fdc421ad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b353e8e2e0 con 0x55b353f28670 2024-09-15T23:17:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.556+0000 7fdc421ad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b353e35ce0 con 0x55b353f28f90 2024-09-15T23:17:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.556+0000 7fdc421ad740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b353e8a2a0 con 0x55b353f33850 2024-09-15T23:17:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.556+0000 7fdc2effd640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc240a32c0 0x7fdc240a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.557+0000 7fdc3e371640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc240a32c0 0x7fdc240a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.557+0000 7fdc2effd640 1 -- 172.21.15.62:0/172521025 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdc300cfee0 con 0x7fdc381139e0 2024-09-15T23:17:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.558+0000 7fdc3d36f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f28a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.558+0000 7fdc3d36f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f28a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43418/0 (socket says 172.21.15.62:43418) 2024-09-15T23:17:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.558+0000 7fdc3d36f640 1 -- 172.21.15.62:0/226629820 learned_addr learned my addr 172.21.15.62:0/226629820 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.558+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/226629820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f33850 0x55b353f35c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc3d36f640 1 -- 172.21.15.62:0/226629820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28f90 msgr2=0x55b353f33310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:17:31.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/226629820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28f90 0x55b353f33310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc3d36f640 1 -- 172.21.15.62:0/226629820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 msgr2=0x55b353f28a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:31.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/226629820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f28a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc3d36f640 1 -- 172.21.15.62:0/226629820 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b353e1a440 con 0x55b353f33850 2024-09-15T23:17:31.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/226629820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f33850 0x55b353f35c40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc34005ee0 tx=0x7fdc34005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c8d20ab9884ebaa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:31.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc23fff640 1 -- 172.21.15.62:0/226629820 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34046070 con 0x55b353f33850 2024-09-15T23:17:31.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc3e371640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc240a32c0 0x7fdc240a5780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdc300a4860 tx=0x7fdc30066b30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:31.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc23fff640 1 -- 172.21.15.62:0/226629820 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc34003970 con 0x55b353f33850 2024-09-15T23:17:31.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.559+0000 7fdc23fff640 1 -- 172.21.15.62:0/226629820 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34036070 con 0x55b353f33850 2024-09-15T23:17:31.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.561+0000 7fdc421ad740 1 -- 172.21.15.62:0/226629820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f33850 msgr2=0x55b353f35c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:31.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.561+0000 7fdc421ad740 1 --2- 172.21.15.62:0/226629820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f33850 0x55b353f35c40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc34005ee0 tx=0x7fdc34005fd0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.561+0000 7fdc421ad740 1 -- 172.21.15.62:0/226629820 shutdown_connections 2024-09-15T23:17:31.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.561+0000 7fdc421ad740 1 --2- 172.21.15.62:0/226629820 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f33850 0x55b353f35c40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.561+0000 7fdc421ad740 1 --2- 172.21.15.62:0/226629820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28f90 0x55b353f33310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.561+0000 7fdc421ad740 1 --2- 172.21.15.62:0/226629820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f28a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.561+0000 7fdc421ad740 1 -- 172.21.15.62:0/226629820 >> 172.21.15.62:0/226629820 conn(0x55b353f24c00 msgr2=0x55b353f24fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:31.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.565+0000 7fdc421ad740 1 -- 172.21.15.62:0/226629820 shutdown_connections 2024-09-15T23:17:31.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.565+0000 7fdc421ad740 1 -- 172.21.15.62:0/226629820 wait complete. 2024-09-15T23:17:31.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.565+0000 7fdc421ad740 1 -- start start 2024-09-15T23:17:31.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.566+0000 7fdc421ad740 1 Processor -- start 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28670 0x55b353f30610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f33850 0x55b353f30b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f3f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc421ad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b353f36420 con 0x55b353f33850 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc421ad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b353e49050 con 0x55b353f28670 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc421ad740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b353f365a0 con 0x55b353f31090 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc421ad740 10 client.0 _ll_register_callbacks cb 0x55b353e89160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc3d36f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28670 0x55b353f30610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc3d36f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28670 0x55b353f30610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42106/0 (socket says 172.21.15.62:42106) 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc3d36f640 1 -- 172.21.15.62:0/3444669362 learned_addr learned my addr 172.21.15.62:0/3444669362 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:31.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.567+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f3f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.568+0000 7fdc3db70640 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f33850 0x55b353f30b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.568+0000 7fdc3d36f640 1 -- 172.21.15.62:0/3444669362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 msgr2=0x55b353f3f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:31.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.568+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f3f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.568+0000 7fdc3d36f640 1 -- 172.21.15.62:0/3444669362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f33850 msgr2=0x55b353f30b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:31.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.568+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f33850 0x55b353f30b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.568+0000 7fdc3d36f640 1 -- 172.21.15.62:0/3444669362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b353bff3c0 con 0x55b353f28670 2024-09-15T23:17:31.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.568+0000 7fdc3db70640 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f33850 0x55b353f30b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:17:31.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.571+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28670 0x55b353f30610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdc34002230 tx=0x7fdc34030fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45c7e77a773de716 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:31.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.571+0000 7fdc217fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:17:31.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.572+0000 7fdc217fa640 1 -- 172.21.15.62:0/3444669362 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34046030 con 0x55b353f28670 2024-09-15T23:17:31.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.572+0000 7fdc217fa640 1 -- 172.21.15.62:0/3444669362 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdc34003d30 con 0x55b353f28670 2024-09-15T23:17:31.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.572+0000 7fdc421ad740 1 -- 172.21.15.62:0/3444669362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc34003970 con 0x55b353f28670 2024-09-15T23:17:31.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.572+0000 7fdc217fa640 1 -- 172.21.15.62:0/3444669362 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc34036070 con 0x55b353f28670 2024-09-15T23:17:31.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.572+0000 7fdc421ad740 10 client.25439 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:31.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.572+0000 7fdc421ad740 1 -- 172.21.15.62:0/3444669362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b353f42370 con 0x55b353f28670 2024-09-15T23:17:31.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.573+0000 7fdc217fa640 1 -- 172.21.15.62:0/3444669362 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc34003970 con 0x55b353f28670 2024-09-15T23:17:31.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.574+0000 7fdc217fa640 1 -- 172.21.15.62:0/3444669362 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fdc34004040 con 0x55b353f28670 2024-09-15T23:17:31.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.574+0000 7fdc421ad740 10 client.25439 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:31.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.574+0000 7fdc421ad740 10 client.25439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:31.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.574+0000 7fdc421ad740 20 client.25439 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:31.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.574+0000 7fdc421ad740 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b353f436c0 0x55b353f45ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.574+0000 7fdc421ad740 4 client.25439 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:31.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.574+0000 7fdc421ad740 1 -- 172.21.15.62:0/3444669362 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b353e9b850 con 0x55b353f436c0 2024-09-15T23:17:31.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.575+0000 7fdc3db70640 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b353f436c0 0x55b353f45ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.576+0000 7fdc3db70640 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b353f436c0 0x55b353f45ab0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb3bf51ecdc68535 server_cookie=4c63a86413fb2ddc in_seq=0 out_seq=0 2024-09-15T23:17:31.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.576+0000 7fdc217fa640 10 client.25439 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:31.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.578+0000 7fdc217fa640 1 -- 172.21.15.62:0/3444669362 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b353e9b850 con 0x55b353f436c0 2024-09-15T23:17:31.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.578+0000 7fdc217fa640 10 client.25439 handle_command_reply: tid=0 2024-09-15T23:17:31.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.578+0000 7fdc421ad740 1 client.25439 shutdown 2024-09-15T23:17:31.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.580+0000 7fdc421ad740 1 -- 172.21.15.62:0/3444669362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28670 msgr2=0x55b353f30610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:31.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.580+0000 7fdc421ad740 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28670 0x55b353f30610 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdc34002230 tx=0x7fdc34030fa0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.580+0000 7fdc421ad740 1 -- 172.21.15.62:0/3444669362 shutdown_connections 2024-09-15T23:17:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.580+0000 7fdc421ad740 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b353f436c0 0x55b353f45ab0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.580+0000 7fdc421ad740 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f3f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.580+0000 7fdc421ad740 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f33850 0x55b353f30b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.580+0000 7fdc421ad740 1 --2- 172.21.15.62:0/3444669362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f28670 0x55b353f30610 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.580+0000 7fdc421ad740 1 -- 172.21.15.62:0/3444669362 >> 172.21.15.62:0/3444669362 conn(0x55b353f24c00 msgr2=0x55b353f2a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.580+0000 7fdc217fa640 0 client.25439 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.581+0000 7fdc421ad740 1 -- 172.21.15.62:0/3444669362 shutdown_connections 2024-09-15T23:17:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.581+0000 7fdc421ad740 1 -- 172.21.15.62:0/3444669362 wait complete. 2024-09-15T23:17:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.581+0000 7fdc421ad740 20 client.25439 trim_cache size 0 max 16384 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc421ad740 1 Processor -- start 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc421ad740 1 -- start start 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f3f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f48ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 0x55b353f49010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc421ad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381c0580 con 0x55b353f28670 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc421ad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381bde50 con 0x55b353f33850 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc421ad740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381bbca0 con 0x55b353f31090 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc3d36f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f3f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc3db70640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f48ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc3db70640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f48ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56620/0 (socket says 172.21.15.62:56620) 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc3db70640 1 -- 172.21.15.62:0/1707871660 learned_addr learned my addr 172.21.15.62:0/1707871660 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.613+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/1707871660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 0x55b353f49010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc3d36f640 1 -- 172.21.15.62:0/1707871660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 msgr2=0x55b353f48ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/1707871660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f48ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc3d36f640 1 -- 172.21.15.62:0/1707871660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 msgr2=0x55b353f49010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/1707871660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 0x55b353f49010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc3d36f640 1 -- 172.21.15.62:0/1707871660 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b353f25890 con 0x55b353f28670 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc3db70640 1 --2- 172.21.15.62:0/1707871660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f48ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/1707871660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f3f800 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdc34030f70 tx=0x7fdc34005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7465d3df40dc5cbf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc237fe640 1 -- 172.21.15.62:0/1707871660 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34046070 con 0x55b353f28670 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc237fe640 1 -- 172.21.15.62:0/1707871660 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc34062070 con 0x55b353f28670 2024-09-15T23:17:31.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.614+0000 7fdc237fe640 1 -- 172.21.15.62:0/1707871660 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34004260 con 0x55b353f28670 2024-09-15T23:17:31.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 -- 172.21.15.62:0/1707871660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 msgr2=0x55b353f3f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:31.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 --2- 172.21.15.62:0/1707871660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f3f800 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdc34030f70 tx=0x7fdc34005fd0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 -- 172.21.15.62:0/1707871660 shutdown_connections 2024-09-15T23:17:31.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 --2- 172.21.15.62:0/1707871660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 0x55b353f49010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 --2- 172.21.15.62:0/1707871660 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f31090 0x55b353f48ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 --2- 172.21.15.62:0/1707871660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f28670 0x55b353f3f800 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 -- 172.21.15.62:0/1707871660 >> 172.21.15.62:0/1707871660 conn(0x55b353f24c00 msgr2=0x55b353f32690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 -- 172.21.15.62:0/1707871660 shutdown_connections 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 -- 172.21.15.62:0/1707871660 wait complete. 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 -- start start 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.615+0000 7fdc421ad740 1 Processor -- start 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.616+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f28670 0x55b353f49310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.616+0000 7fdc421ad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f31090 0x55b353f466b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.616+0000 7fdc3d36f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f28670 0x55b353f49310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.617+0000 7fdc3d36f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f28670 0x55b353f49310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56630/0 (socket says 172.21.15.62:56630) 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.617+0000 7fdc3d36f640 1 -- 172.21.15.62:0/2053278119 learned_addr learned my addr 172.21.15.62:0/2053278119 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.617+0000 7fdc421ad740 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 0x55b353f46ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.617+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381bc310 con 0x55b353f31090 2024-09-15T23:17:31.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.617+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381bd780 con 0x55b353f33850 2024-09-15T23:17:31.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.617+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdc381bc9e0 con 0x55b353f28670 2024-09-15T23:17:31.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.617+0000 7fdc421ad740 10 client.0 _ll_register_callbacks cb 0x7fdc34044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:31.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.617+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 0x55b353f46ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.617+0000 7fdc3db70640 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f31090 0x55b353f466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc3d36f640 1 -- 172.21.15.62:0/2053278119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 msgr2=0x55b353f46ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:31.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 0x55b353f46ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc3d36f640 1 -- 172.21.15.62:0/2053278119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f31090 msgr2=0x55b353f466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:31.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f31090 0x55b353f466b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc3d36f640 1 -- 172.21.15.62:0/2053278119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b353f474d0 con 0x55b353f28670 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc3d36f640 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f28670 0x55b353f49310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009560 tx=0x7fdc340045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95a9ddd6afa18a36 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc217fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc217fa640 1 -- 172.21.15.62:0/2053278119 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc34046030 con 0x55b353f28670 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc217fa640 1 -- 172.21.15.62:0/2053278119 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdc340305b0 con 0x55b353f28670 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.618+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc34062070 con 0x55b353f28670 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc217fa640 1 -- 172.21.15.62:0/2053278119 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc34003970 con 0x55b353f28670 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc421ad740 10 client.25324 fetch_fsmap learned FSMap version 27 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b353fae0d0 con 0x55b353f28670 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc217fa640 1 -- 172.21.15.62:0/2053278119 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc34062070 con 0x55b353f28670 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc217fa640 1 -- 172.21.15.62:0/2053278119 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fdc34066040 con 0x55b353f28670 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc421ad740 10 client.25324 fetch_fsmap finished waiting for FSMap version 27 2024-09-15T23:17:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc421ad740 10 client.25324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc421ad740 20 client.25324 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc421ad740 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b353fae840 0x55b353fb0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc421ad740 4 client.25324 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b353f30ee0 con 0x55b353fae840 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.619+0000 7fdc3db70640 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b353fae840 0x55b353fb0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.620+0000 7fdc217fa640 10 client.25324 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.620+0000 7fdc3db70640 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b353fae840 0x55b353fb0c60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=116875cd7c7a8969 server_cookie=d040a25e45c8b0ec in_seq=0 out_seq=0 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.620+0000 7fdc217fa640 1 -- 172.21.15.62:0/2053278119 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b353f30ee0 con 0x55b353fae840 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.620+0000 7fdc217fa640 10 client.25324 handle_command_reply: tid=0 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.620+0000 7fdc421ad740 1 client.25324 shutdown 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.621+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f28670 msgr2=0x55b353f49310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.621+0000 7fdc421ad740 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f28670 0x55b353f49310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009560 tx=0x7fdc340045e0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.621+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 shutdown_connections 2024-09-15T23:17:31.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.621+0000 7fdc421ad740 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b353fae840 0x55b353fb0c60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.621+0000 7fdc421ad740 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b353f33850 0x55b353f46ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.621+0000 7fdc217fa640 0 client.25324 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.621+0000 7fdc421ad740 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b353f31090 0x55b353f466b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.621+0000 7fdc421ad740 1 --2- 172.21.15.62:0/2053278119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b353f28670 0x55b353f49310 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.621+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 >> 172.21.15.62:0/2053278119 conn(0x55b353f24c00 msgr2=0x55b353f2b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.622+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 shutdown_connections 2024-09-15T23:17:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.622+0000 7fdc421ad740 1 -- 172.21.15.62:0/2053278119 wait complete. 2024-09-15T23:17:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.622+0000 7fdc421ad740 20 client.25324 trim_cache size 0 max 16384 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout: "e9ecaca8-5a24-4e19-8d7b-d27638b0cb24": { 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "e9ecaca8-5a24-4e19-8d7b-d27638b0cb24", 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:17:31.627 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:17:31.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 -- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc240a32c0 msgr2=0x7fdc240a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:31.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc240a32c0 0x7fdc240a5780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdc300a4860 tx=0x7fdc30066b30 comp rx=0 tx=0).stop 2024-09-15T23:17:31.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 -- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc381139e0 msgr2=0x7fdc38113e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:31.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc381139e0 0x7fdc38113e90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fdc3000e940 tx=0x7fdc3000ee00 comp rx=0 tx=0).stop 2024-09-15T23:17:31.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 -- 172.21.15.62:0/172521025 shutdown_connections 2024-09-15T23:17:31.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc240a32c0 0x7fdc240a5780 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc381139e0 0x7fdc38113e90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc38114ab0 0x7fdc381ac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 --2- 172.21.15.62:0/172521025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc3806e4f0 0x7fdc381abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:31.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.624+0000 7fdc3eb72640 1 -- 172.21.15.62:0/172521025 >> 172.21.15.62:0/172521025 conn(0x7fdc38073180 msgr2=0x7fdc3810a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:31.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.625+0000 7fdc3eb72640 1 -- 172.21.15.62:0/172521025 shutdown_connections 2024-09-15T23:17:31.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:31.625+0000 7fdc3eb72640 1 -- 172.21.15.62:0/172521025 wait complete. 2024-09-15T23:17:32.876 INFO:tasks.cephfs.filesystem:scrub status for tag:e9ecaca8-5a24-4e19-8d7b-d27638b0cb24 - {'path': '/', 'tag': 'e9ecaca8-5a24-4e19-8d7b-d27638b0cb24', 'options': 'recursive,force'} 2024-09-15T23:17:33.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:33 smithi062 ceph-mon[30949]: pgmap v408: 129 pgs: 129 active+clean; 982 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 11 MiB/s wr, 899 op/s 2024-09-15T23:17:33.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:33 smithi067 ceph-mon[35665]: pgmap v408: 129 pgs: 129 active+clean; 982 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 11 MiB/s wr, 899 op/s 2024-09-15T23:17:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:33 smithi148 ceph-mon[35610]: pgmap v408: 129 pgs: 129 active+clean; 982 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 11 MiB/s wr, 899 op/s 2024-09-15T23:17:34.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:34 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:17:34.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:34 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:17:34.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:34 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:17:34.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 75.0 s, 3428.1 tps, lat 29.296 ms stddev 33.863 2024-09-15T23:17:35.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:35 smithi062 ceph-mon[30949]: pgmap v409: 129 pgs: 129 active+clean; 982 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 7.9 MiB/s wr, 824 op/s 2024-09-15T23:17:35.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:35 smithi067 ceph-mon[35665]: pgmap v409: 129 pgs: 129 active+clean; 982 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 7.9 MiB/s wr, 824 op/s 2024-09-15T23:17:35.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:35 smithi148 ceph-mon[35610]: pgmap v409: 129 pgs: 129 active+clean; 982 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 7.9 MiB/s wr, 824 op/s 2024-09-15T23:17:37.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:37 smithi062 ceph-mon[30949]: pgmap v410: 129 pgs: 129 active+clean; 991 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 9.2 MiB/s wr, 862 op/s 2024-09-15T23:17:37.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:37 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:37.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:37 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:37 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:37 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:37 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:37 smithi067 ceph-mon[35665]: pgmap v410: 129 pgs: 129 active+clean; 991 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 9.2 MiB/s wr, 862 op/s 2024-09-15T23:17:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:37 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:37 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:37 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:37 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:37 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:37 smithi148 ceph-mon[35610]: pgmap v410: 129 pgs: 129 active+clean; 991 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 9.2 MiB/s wr, 862 op/s 2024-09-15T23:17:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:37 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:17:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:37 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:37 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:37 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:37 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.048+0000 7f1348ce6640 1 -- 172.21.15.62:0/631716140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1344102d30 msgr2=0x7f13441051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.048+0000 7f1348ce6640 1 --2- 172.21.15.62:0/631716140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1344102d30 0x7f13441051a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1330009900 tx=0x7f133002a400 comp rx=0 tx=0).stop 2024-09-15T23:17:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.048+0000 7f1348ce6640 1 -- 172.21.15.62:0/631716140 shutdown_connections 2024-09-15T23:17:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.048+0000 7f1348ce6640 1 --2- 172.21.15.62:0/631716140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13441056e0 0x7f1344105b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.048+0000 7f1348ce6640 1 --2- 172.21.15.62:0/631716140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1344102d30 0x7f13441051a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.048+0000 7f1348ce6640 1 --2- 172.21.15.62:0/631716140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13441003e0 0x7f13441027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.048+0000 7f1348ce6640 1 -- 172.21.15.62:0/631716140 >> 172.21.15.62:0/631716140 conn(0x7f13440fc1b0 msgr2=0x7f13440fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.048+0000 7f1348ce6640 1 -- 172.21.15.62:0/631716140 shutdown_connections 2024-09-15T23:17:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.048+0000 7f1348ce6640 1 -- 172.21.15.62:0/631716140 wait complete. 2024-09-15T23:17:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1348ce6640 1 Processor -- start 2024-09-15T23:17:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1348ce6640 1 -- start start 2024-09-15T23:17:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1348ce6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13441003e0 0x7f134419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1348ce6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1344102d30 0x7f134419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1348ce6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13441056e0 0x7f13441a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1348ce6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1344113e20 con 0x7f13441003e0 2024-09-15T23:17:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1348ce6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1344113ca0 con 0x7f13441056e0 2024-09-15T23:17:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1348ce6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1344113fa0 con 0x7f1344102d30 2024-09-15T23:17:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f134259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13441003e0 0x7f134419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13441056e0 0x7f13441a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13441056e0 0x7f13441a5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:59106/0 (socket says 172.21.15.62:59106) 2024-09-15T23:17:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f134259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13441003e0 0x7f134419e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34662/0 (socket says 172.21.15.62:34662) 2024-09-15T23:17:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1342d9d640 1 -- 172.21.15.62:0/4223822848 learned_addr learned my addr 172.21.15.62:0/4223822848 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.049+0000 7f1341d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1344102d30 0x7f134419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.050+0000 7f1342d9d640 1 -- 172.21.15.62:0/4223822848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1344102d30 msgr2=0x7f134419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.050+0000 7f1342d9d640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1344102d30 0x7f134419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.050+0000 7f1342d9d640 1 -- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13441003e0 msgr2=0x7f134419e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.050+0000 7f1342d9d640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13441003e0 0x7f134419e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.050+0000 7f1342d9d640 1 -- 172.21.15.62:0/4223822848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13441a5fd0 con 0x7f13441056e0 2024-09-15T23:17:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.050+0000 7f134259c640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13441003e0 0x7f134419e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:17:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.051+0000 7f1342d9d640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13441056e0 0x7f13441a5960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f133c00eb10 tx=0x7f133c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:38.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.051+0000 7f133b7fe640 1 -- 172.21.15.62:0/4223822848 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c004b20 con 0x7f13441056e0 2024-09-15T23:17:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.051+0000 7f133b7fe640 1 -- 172.21.15.62:0/4223822848 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c004cc0 con 0x7f13441056e0 2024-09-15T23:17:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.051+0000 7f133b7fe640 1 -- 172.21.15.62:0/4223822848 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c010760 con 0x7f13441056e0 2024-09-15T23:17:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.051+0000 7f1348ce6640 1 -- 172.21.15.62:0/4223822848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13441a62c0 con 0x7f13441056e0 2024-09-15T23:17:38.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.051+0000 7f1348ce6640 1 -- 172.21.15.62:0/4223822848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13441a6790 con 0x7f13441056e0 2024-09-15T23:17:38.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.055+0000 7f134ba81740 1 Processor -- start 2024-09-15T23:17:38.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.056+0000 7f133b7fe640 1 -- 172.21.15.62:0/4223822848 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f133c005020 con 0x7f13441056e0 2024-09-15T23:17:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f134ba81740 1 -- start start 2024-09-15T23:17:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f133b7fe640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13280a32f0 0x7f13280a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f134ba81740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed91da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f134ba81740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799ed96da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f134ba81740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799ed99830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f134ba81740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55799ecf52f0 con 0x55799ed97370 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f133b7fe640 1 -- 172.21.15.62:0/4223822848 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f133c0d02f0 con 0x7f13441056e0 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f134ba81740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55799ec9d270 con 0x55799ed919a0 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f134ba81740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55799ecf12b0 con 0x55799ed922e0 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f1341d9b640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13280a32f0 0x7f13280a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799ed99830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f1342d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799ed99830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34672/0 (socket says 172.21.15.62:34672) 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f1342d9d640 1 -- 172.21.15.62:0/592415990 learned_addr learned my addr 172.21.15.62:0/592415990 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.057+0000 7f1341d9b640 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799ed96da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f134259c640 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed91da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1342d9d640 1 -- 172.21.15.62:0/592415990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 msgr2=0x55799ed96da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:38.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1342d9d640 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799ed96da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1342d9d640 1 -- 172.21.15.62:0/592415990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 msgr2=0x55799ed91da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1342d9d640 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed91da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1342d9d640 1 -- 172.21.15.62:0/592415990 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55799ec9c340 con 0x55799ed97370 2024-09-15T23:17:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f134259c640 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed91da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:17:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1341d9b640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13280a32f0 0x7f13280a57b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f13300314e0 tx=0x7f1330032f20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1342d9d640 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799ed99830 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f133c0a4cd0 tx=0x7f133c098900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71636f4528b0357f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1338ff9640 1 -- 172.21.15.62:0/592415990 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c09bef0 con 0x55799ed97370 2024-09-15T23:17:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1338ff9640 1 -- 172.21.15.62:0/592415990 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c09cc90 con 0x55799ed97370 2024-09-15T23:17:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.058+0000 7f1338ff9640 1 -- 172.21.15.62:0/592415990 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0a4d70 con 0x55799ed97370 2024-09-15T23:17:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.059+0000 7f134ba81740 1 -- 172.21.15.62:0/592415990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 msgr2=0x55799ed99830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.059+0000 7f134ba81740 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799ed99830 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f133c0a4cd0 tx=0x7f133c098900 comp rx=0 tx=0).stop 2024-09-15T23:17:38.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.059+0000 7f134ba81740 1 -- 172.21.15.62:0/592415990 shutdown_connections 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.059+0000 7f134ba81740 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799ed99830 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.059+0000 7f134ba81740 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799ed96da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.059+0000 7f134ba81740 1 --2- 172.21.15.62:0/592415990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed91da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.059+0000 7f134ba81740 1 -- 172.21.15.62:0/592415990 >> 172.21.15.62:0/592415990 conn(0x55799ed8d010 msgr2=0x55799ed8d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.060+0000 7f134ba81740 1 -- 172.21.15.62:0/592415990 shutdown_connections 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.060+0000 7f134ba81740 1 -- 172.21.15.62:0/592415990 wait complete. 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.060+0000 7f134ba81740 1 -- start start 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134ba81740 1 Processor -- start 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134ba81740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed9bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134ba81740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799ed9c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134ba81740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799eda3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134ba81740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55799ed9a2e0 con 0x55799ed97370 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134ba81740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55799ec7e290 con 0x55799ed919a0 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134ba81740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55799ed9a460 con 0x55799ed922e0 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134ba81740 10 client.0 _ll_register_callbacks cb 0x55799ecaf140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed9bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed9bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:59126/0 (socket says 172.21.15.62:59126) 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f134259c640 1 -- 172.21.15.62:0/3047204297 learned_addr learned my addr 172.21.15.62:0/3047204297 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f1342d9d640 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799eda3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.061+0000 7f1341d9b640 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799ed9c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f134259c640 1 -- 172.21.15.62:0/3047204297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 msgr2=0x55799ed9c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f134259c640 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799ed9c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f134259c640 1 -- 172.21.15.62:0/3047204297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 msgr2=0x55799eda3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:38.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f134259c640 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799eda3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f134259c640 1 -- 172.21.15.62:0/3047204297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55799ea08210 con 0x55799ed919a0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f134259c640 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed9bbd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f133400bf80 tx=0x7f1334011870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6f8796869f750c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f1325ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f1325ffb640 1 -- 172.21.15.62:0/3047204297 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1334004230 con 0x55799ed919a0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f1325ffb640 1 -- 172.21.15.62:0/3047204297 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f13340043d0 con 0x55799ed919a0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.062+0000 7f134ba81740 1 -- 172.21.15.62:0/3047204297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f133c09cc90 con 0x55799ed919a0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f1325ffb640 1 -- 172.21.15.62:0/3047204297 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1334024050 con 0x55799ed919a0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f134ba81740 10 client.25457 fetch_fsmap learned FSMap version 28 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f134ba81740 1 -- 172.21.15.62:0/3047204297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55799ead4620 con 0x55799ed919a0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f1325ffb640 1 -- 172.21.15.62:0/3047204297 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c09cc90 con 0x55799ed919a0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f1325ffb640 1 -- 172.21.15.62:0/3047204297 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f1334012900 con 0x55799ed919a0 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f134ba81740 10 client.25457 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f134ba81740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f134ba81740 20 client.25457 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:38.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f134ba81740 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55799eda7970 0x55799eda9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f134ba81740 4 client.25457 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f134ba81740 1 -- 172.21.15.62:0/3047204297 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55799ed02a20 con 0x55799eda7970 2024-09-15T23:17:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.063+0000 7f1341d9b640 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55799eda7970 0x55799eda9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.064+0000 7f1325ffb640 10 client.25457 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.064+0000 7f1341d9b640 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55799eda7970 0x55799eda9d80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4936b9d0aae8b977 server_cookie=36604fc200a80f27 in_seq=0 out_seq=0 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.065+0000 7f1325ffb640 1 -- 172.21.15.62:0/3047204297 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55799ed02a20 con 0x55799eda7970 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.065+0000 7f1325ffb640 10 client.25457 handle_command_reply: tid=0 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.065+0000 7f134ba81740 1 client.25457 shutdown 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.066+0000 7f134ba81740 1 -- 172.21.15.62:0/3047204297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 msgr2=0x55799ed9bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.066+0000 7f134ba81740 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed9bbd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f133400bf80 tx=0x7f1334011870 comp rx=0 tx=0).stop 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.066+0000 7f134ba81740 1 -- 172.21.15.62:0/3047204297 shutdown_connections 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.066+0000 7f134ba81740 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55799eda7970 0x55799eda9d80 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.066+0000 7f134ba81740 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed97370 0x55799eda3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.066+0000 7f134ba81740 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799ed9c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.066+0000 7f134ba81740 1 --2- 172.21.15.62:0/3047204297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed919a0 0x55799ed9bbd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.066+0000 7f134ba81740 1 -- 172.21.15.62:0/3047204297 >> 172.21.15.62:0/3047204297 conn(0x55799ed8d010 msgr2=0x55799ed8e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.066+0000 7f1325ffb640 0 client.25457 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.067+0000 7f134ba81740 1 -- 172.21.15.62:0/3047204297 shutdown_connections 2024-09-15T23:17:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.067+0000 7f134ba81740 1 -- 172.21.15.62:0/3047204297 wait complete. 2024-09-15T23:17:38.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.067+0000 7f134ba81740 20 client.25457 trim_cache size 0 max 16384 2024-09-15T23:17:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.090+0000 7f134ba81740 1 Processor -- start 2024-09-15T23:17:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.090+0000 7f134ba81740 1 -- start start 2024-09-15T23:17:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.091+0000 7f134ba81740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed919a0 0x55799ed8d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.091+0000 7f134ba81740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799eda3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.091+0000 7f134ba81740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 0x55799ed8d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.091+0000 7f134259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed919a0 0x55799ed8d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.091+0000 7f134259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed919a0 0x55799ed8d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34680/0 (socket says 172.21.15.62:34680) 2024-09-15T23:17:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.091+0000 7f134259c640 1 -- 172.21.15.62:0/2303651890 learned_addr learned my addr 172.21.15.62:0/2303651890 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f134ba81740 1 -- 172.21.15.62:0/2303651890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f13441b3580 con 0x55799ed919a0 2024-09-15T23:17:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1342d9d640 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 0x55799ed8d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f134ba81740 1 -- 172.21.15.62:0/2303651890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13441b0e20 con 0x55799ed97370 2024-09-15T23:17:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f134ba81740 1 -- 172.21.15.62:0/2303651890 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f13441aec70 con 0x55799ed922e0 2024-09-15T23:17:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1341d9b640 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799eda3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1342d9d640 1 -- 172.21.15.62:0/2303651890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 msgr2=0x55799eda3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1342d9d640 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799eda3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1342d9d640 1 -- 172.21.15.62:0/2303651890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed919a0 msgr2=0x55799ed8d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1342d9d640 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed919a0 0x55799ed8d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1342d9d640 1 -- 172.21.15.62:0/2303651890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55799ed93d80 con 0x55799ed97370 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1341d9b640 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799eda3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1342d9d640 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 0x55799ed8d930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f133c0a4cd0 tx=0x7f133c009a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a158fb992c7ab7eb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.092+0000 7f1327fff640 1 -- 172.21.15.62:0/2303651890 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c09cc90 con 0x55799ed97370 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f1327fff640 1 -- 172.21.15.62:0/2303651890 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c09c590 con 0x55799ed97370 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f1327fff640 1 -- 172.21.15.62:0/2303651890 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0a4d70 con 0x55799ed97370 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f134ba81740 1 -- 172.21.15.62:0/2303651890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 msgr2=0x55799ed8d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f134ba81740 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 0x55799ed8d930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f133c0a4cd0 tx=0x7f133c009a70 comp rx=0 tx=0).stop 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f134ba81740 1 -- 172.21.15.62:0/2303651890 shutdown_connections 2024-09-15T23:17:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f134ba81740 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 0x55799ed8d930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f134ba81740 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed922e0 0x55799eda3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f134ba81740 1 --2- 172.21.15.62:0/2303651890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed919a0 0x55799ed8d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f134ba81740 1 -- 172.21.15.62:0/2303651890 >> 172.21.15.62:0/2303651890 conn(0x55799ed8d010 msgr2=0x55799ed9b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.093+0000 7f134ba81740 1 -- 172.21.15.62:0/2303651890 shutdown_connections 2024-09-15T23:17:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.094+0000 7f134ba81740 1 -- 172.21.15.62:0/2303651890 wait complete. 2024-09-15T23:17:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.094+0000 7f134ba81740 1 -- start start 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.094+0000 7f134ba81740 1 Processor -- start 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.094+0000 7f134ba81740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed919a0 0x55799eda0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.094+0000 7f134ba81740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed922e0 0x55799eda08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.094+0000 7f134ba81740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 0x55799eda18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.094+0000 7f134ba81740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f13441af2e0 con 0x55799ed922e0 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.094+0000 7f134259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed919a0 0x55799eda0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed919a0 0x55799eda0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:45592/0 (socket says 172.21.15.62:45592) 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134259c640 1 -- 172.21.15.62:0/2699362677 learned_addr learned my addr 172.21.15.62:0/2699362677 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f1342d9d640 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 0x55799eda18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13441b0750 con 0x55799ed97370 2024-09-15T23:17:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f13441af9b0 con 0x55799ed919a0 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134ba81740 10 client.0 _ll_register_callbacks cb 0x7f133400b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134259c640 1 -- 172.21.15.62:0/2699362677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 msgr2=0x55799eda18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134259c640 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 0x55799eda18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134259c640 1 -- 172.21.15.62:0/2699362677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed922e0 msgr2=0x55799eda08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134259c640 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed922e0 0x55799eda08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134259c640 1 -- 172.21.15.62:0/2699362677 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55799eda1ed0 con 0x55799ed919a0 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f134259c640 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed919a0 0x55799eda0370 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1334012d50 tx=0x7f133400b410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=714fdab51e0193cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.095+0000 7f1325ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.096+0000 7f1325ffb640 1 -- 172.21.15.62:0/2699362677 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1334021040 con 0x55799ed919a0 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.096+0000 7f1325ffb640 1 -- 172.21.15.62:0/2699362677 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1334004020 con 0x55799ed919a0 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.096+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f133c09c590 con 0x55799ed919a0 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.096+0000 7f1325ffb640 1 -- 172.21.15.62:0/2699362677 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f133405c050 con 0x55799ed919a0 2024-09-15T23:17:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.096+0000 7f134ba81740 10 client.25342 fetch_fsmap learned FSMap version 28 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.096+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55799eda3230 con 0x55799ed919a0 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.096+0000 7f1325ffb640 1 -- 172.21.15.62:0/2699362677 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133400b650 con 0x55799ed919a0 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f1325ffb640 1 -- 172.21.15.62:0/2699362677 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f1334004680 con 0x55799ed919a0 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f134ba81740 10 client.25342 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f134ba81740 10 client.25342 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f134ba81740 20 client.25342 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f134ba81740 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55799ee12c30 0x55799ee15050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f134ba81740 4 client.25342 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f133c09c590 con 0x55799ee12c30 2024-09-15T23:17:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f1341d9b640 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55799ee12c30 0x55799ee15050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f1325ffb640 10 client.25342 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.097+0000 7f1341d9b640 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55799ee12c30 0x55799ee15050 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdf551c6f1c64f28 server_cookie=dab5e3364bdc0216 in_seq=0 out_seq=0 2024-09-15T23:17:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.110+0000 7f1325ffb640 1 -- 172.21.15.62:0/2699362677 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f133c09c590 con 0x55799ee12c30 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.110+0000 7f1325ffb640 10 client.25342 handle_command_reply: tid=0 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.110+0000 7f134ba81740 1 client.25342 shutdown 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.110+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed919a0 msgr2=0x55799eda0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.110+0000 7f134ba81740 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed919a0 0x55799eda0370 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1334012d50 tx=0x7f133400b410 comp rx=0 tx=0).stop 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 shutdown_connections 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f134ba81740 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55799ee12c30 0x55799ee15050 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f134ba81740 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55799ed97370 0x55799eda18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f134ba81740 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55799ed922e0 0x55799eda08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f134ba81740 1 --2- 172.21.15.62:0/2699362677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55799ed919a0 0x55799eda0370 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 >> 172.21.15.62:0/2699362677 conn(0x55799ed8d010 msgr2=0x55799ed8b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f1325ffb640 0 client.25342 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 shutdown_connections 2024-09-15T23:17:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f134ba81740 1 -- 172.21.15.62:0/2699362677 wait complete. 2024-09-15T23:17:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.111+0000 7f134ba81740 20 client.25342 trim_cache size 0 max 16384 2024-09-15T23:17:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:17:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:17:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:17:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:17:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.112+0000 7f1348ce6640 1 -- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13280a32f0 msgr2=0x7f13280a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.112+0000 7f1348ce6640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13280a32f0 0x7f13280a57b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f13300314e0 tx=0x7f1330032f20 comp rx=0 tx=0).stop 2024-09-15T23:17:38.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.112+0000 7f1348ce6640 1 -- 172.21.15.62:0/4223822848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13441056e0 msgr2=0x7f13441a5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.112+0000 7f1348ce6640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13441056e0 0x7f13441a5960 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f133c00eb10 tx=0x7f133c00c2d0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.113+0000 7f1348ce6640 1 -- 172.21.15.62:0/4223822848 shutdown_connections 2024-09-15T23:17:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.113+0000 7f1348ce6640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13280a32f0 0x7f13280a57b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.113+0000 7f1348ce6640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13441056e0 0x7f13441a5960 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.113+0000 7f1348ce6640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1344102d30 0x7f134419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.113+0000 7f1348ce6640 1 --2- 172.21.15.62:0/4223822848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13441003e0 0x7f134419e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.113+0000 7f1348ce6640 1 -- 172.21.15.62:0/4223822848 >> 172.21.15.62:0/4223822848 conn(0x7f13440fc1b0 msgr2=0x7f1344100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.113+0000 7f1348ce6640 1 -- 172.21.15.62:0/4223822848 shutdown_connections 2024-09-15T23:17:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:38.113+0000 7f1348ce6640 1 -- 172.21.15.62:0/4223822848 wait complete. 2024-09-15T23:17:38.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:38.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:38.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:38.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:38 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:17:38.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:38 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:17:38.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:38.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:38.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:38.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:38 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:17:38.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:38 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:17:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:17:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:17:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:17:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:38 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:17:38.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:38 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:17:39.381 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:39 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:17:39.382 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:39 smithi062 ceph-mon[30949]: pgmap v411: 129 pgs: 129 active+clean; 999 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 10 MiB/s wr, 871 op/s 2024-09-15T23:17:39.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:39 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:17:39.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:39 smithi067 ceph-mon[35665]: pgmap v411: 129 pgs: 129 active+clean; 999 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 10 MiB/s wr, 871 op/s 2024-09-15T23:17:39.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:39 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:17:39.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:39 smithi148 ceph-mon[35610]: pgmap v411: 129 pgs: 129 active+clean; 999 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 10 MiB/s wr, 871 op/s 2024-09-15T23:17:39.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 80.0 s, 2837.8 tps, lat 35.125 ms stddev 39.096 2024-09-15T23:17:41.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:41 smithi062 ceph-mon[30949]: pgmap v412: 129 pgs: 129 active+clean; 1000 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 9.4 MiB/s wr, 830 op/s 2024-09-15T23:17:41.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:41 smithi067 ceph-mon[35665]: pgmap v412: 129 pgs: 129 active+clean; 1000 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 9.4 MiB/s wr, 830 op/s 2024-09-15T23:17:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:41 smithi148 ceph-mon[35610]: pgmap v412: 129 pgs: 129 active+clean; 1000 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 9.4 MiB/s wr, 830 op/s 2024-09-15T23:17:43.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:43 smithi062 ceph-mon[30949]: pgmap v413: 129 pgs: 129 active+clean; 1001 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 943 op/s 2024-09-15T23:17:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:43 smithi067 ceph-mon[35665]: pgmap v413: 129 pgs: 129 active+clean; 1001 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 943 op/s 2024-09-15T23:17:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:43 smithi148 ceph-mon[35610]: pgmap v413: 129 pgs: 129 active+clean; 1001 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 11 MiB/s wr, 943 op/s 2024-09-15T23:17:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 85.0 s, 2349.6 tps, lat 42.597 ms stddev 71.187 2024-09-15T23:17:45.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:45 smithi062 ceph-mon[30949]: pgmap v414: 129 pgs: 129 active+clean; 1006 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 15 MiB/s wr, 978 op/s 2024-09-15T23:17:45.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:45 smithi067 ceph-mon[35665]: pgmap v414: 129 pgs: 129 active+clean; 1006 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 15 MiB/s wr, 978 op/s 2024-09-15T23:17:45.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:45 smithi148 ceph-mon[35610]: pgmap v414: 129 pgs: 129 active+clean; 1006 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 15 MiB/s wr, 978 op/s 2024-09-15T23:17:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:47 smithi062 ceph-mon[30949]: pgmap v415: 129 pgs: 129 active+clean; 1011 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-09-15T23:17:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:47 smithi067 ceph-mon[35665]: pgmap v415: 129 pgs: 129 active+clean; 1011 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-09-15T23:17:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:47 smithi148 ceph-mon[35610]: pgmap v415: 129 pgs: 129 active+clean; 1011 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 19 MiB/s wr, 1.01k op/s 2024-09-15T23:17:49.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:49 smithi062 ceph-mon[30949]: pgmap v416: 129 pgs: 129 active+clean; 1011 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 1.05k op/s 2024-09-15T23:17:49.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:49 smithi067 ceph-mon[35665]: pgmap v416: 129 pgs: 129 active+clean; 1011 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 1.05k op/s 2024-09-15T23:17:49.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:49 smithi148 ceph-mon[35610]: pgmap v416: 129 pgs: 129 active+clean; 1011 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 MiB/s wr, 1.05k op/s 2024-09-15T23:17:49.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 90.0 s, 2322.8 tps, lat 42.863 ms stddev 45.412 2024-09-15T23:17:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 -- 172.21.15.62:0/2958444490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d8101750 msgr2=0x7f54d8101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 --2- 172.21.15.62:0/2958444490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d8101750 0x7f54d8101c00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009900 tx=0x7f54c802a410 comp rx=0 tx=0).stop 2024-09-15T23:17:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 -- 172.21.15.62:0/2958444490 shutdown_connections 2024-09-15T23:17:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 --2- 172.21.15.62:0/2958444490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d8101750 0x7f54d8101c00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 --2- 172.21.15.62:0/2958444490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d8100db0 0x7f54d8101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 --2- 172.21.15.62:0/2958444490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54d8100470 0x7f54d8100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 -- 172.21.15.62:0/2958444490 >> 172.21.15.62:0/2958444490 conn(0x7f54d80fc1e0 msgr2=0x7f54d80fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 -- 172.21.15.62:0/2958444490 shutdown_connections 2024-09-15T23:17:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 -- 172.21.15.62:0/2958444490 wait complete. 2024-09-15T23:17:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.254+0000 7f54dddd6640 1 Processor -- start 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54dddd6640 1 -- start start 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54dddd6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d8100470 0x7f54d8199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54dddd6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d8100db0 0x7f54d819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54dddd6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54d8101750 0x7f54d81a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54dddd6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f54d810f9c0 con 0x7f54d8100470 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54dddd6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54d810f840 con 0x7f54d8100db0 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54dddd6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f54d810fb40 con 0x7f54d8101750 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d8100470 0x7f54d8199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d8100470 0x7f54d8199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43636/0 (socket says 172.21.15.62:43636) 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54d77fe640 1 -- 172.21.15.62:0/3331520486 learned_addr learned my addr 172.21.15.62:0/3331520486 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54d7fff640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54d8101750 0x7f54d81a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.255+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d8100db0 0x7f54d819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.256+0000 7f54d6ffd640 1 -- 172.21.15.62:0/3331520486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54d8101750 msgr2=0x7f54d81a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.256+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54d8101750 0x7f54d81a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.256+0000 7f54d6ffd640 1 -- 172.21.15.62:0/3331520486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d8100470 msgr2=0x7f54d8199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.256+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d8100470 0x7f54d8199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.256+0000 7f54d6ffd640 1 -- 172.21.15.62:0/3331520486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d81a1bd0 con 0x7f54d8100db0 2024-09-15T23:17:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.256+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d8100db0 0x7f54d819a4f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00d6d0 tx=0x7f54cc00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.256+0000 7f54d4ff9640 1 -- 172.21.15.62:0/3331520486 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc017070 con 0x7f54d8100db0 2024-09-15T23:17:51.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.256+0000 7f54d4ff9640 1 -- 172.21.15.62:0/3331520486 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54cc009820 con 0x7f54d8100db0 2024-09-15T23:17:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.256+0000 7f54d4ff9640 1 -- 172.21.15.62:0/3331520486 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc004ce0 con 0x7f54d8100db0 2024-09-15T23:17:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.257+0000 7f54dddd6640 1 -- 172.21.15.62:0/3331520486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d81a1ec0 con 0x7f54d8100db0 2024-09-15T23:17:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.257+0000 7f54dddd6640 1 -- 172.21.15.62:0/3331520486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d81a2270 con 0x7f54d8100db0 2024-09-15T23:17:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.259+0000 7f54d4ff9640 1 -- 172.21.15.62:0/3331520486 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f54cc0099d0 con 0x7f54d8100db0 2024-09-15T23:17:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.260+0000 7f54d4ff9640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54c00a34e0 0x7f54c00a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.260+0000 7f54d4ff9640 1 -- 172.21.15.62:0/3331520486 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f54cc0cf3f0 con 0x7f54d8100db0 2024-09-15T23:17:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54e0c53740 1 Processor -- start 2024-09-15T23:17:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54e0c53740 1 -- start start 2024-09-15T23:17:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 0x558429c625b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c675b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c67b80 0x558429c6a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54e0c53740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558429bbd2f0 con 0x558429c621b0 2024-09-15T23:17:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54e0c53740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558429b65270 con 0x558429c62af0 2024-09-15T23:17:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54e0c53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558429bb92b0 con 0x558429c67b80 2024-09-15T23:17:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c67b80 0x558429c6a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 0x558429c625b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c67b80 0x558429c6a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43718/0 (socket says 172.21.15.62:43718) 2024-09-15T23:17:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54d7fff640 1 -- 172.21.15.62:0/379963860 learned_addr learned my addr 172.21.15.62:0/379963860 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.261+0000 7f54d77fe640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54c00a34e0 0x7f54c00a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/379963860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c675b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54d7fff640 1 -- 172.21.15.62:0/379963860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 msgr2=0x558429c675b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54d7fff640 1 --2- 172.21.15.62:0/379963860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c675b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54d7fff640 1 -- 172.21.15.62:0/379963860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 msgr2=0x558429c625b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54d7fff640 1 --2- 172.21.15.62:0/379963860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 0x558429c625b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54d7fff640 1 -- 172.21.15.62:0/379963860 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558429b64340 con 0x558429c67b80 2024-09-15T23:17:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/379963860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c675b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:17:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54d7fff640 1 --2- 172.21.15.62:0/379963860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c67b80 0x558429c6a040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009900 tx=0x7f54c8005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ee4c0325c7bc437 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54d77fe640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54c00a34e0 0x7f54c00a59a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f54d0013ed0 tx=0x7f54d0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54bdffb640 1 -- 172.21.15.62:0/379963860 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c8040070 con 0x558429c67b80 2024-09-15T23:17:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54bdffb640 1 -- 172.21.15.62:0/379963860 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54c80319f0 con 0x558429c67b80 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.262+0000 7f54bdffb640 1 -- 172.21.15.62:0/379963860 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c8035040 con 0x558429c67b80 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.263+0000 7f54e0c53740 1 -- 172.21.15.62:0/379963860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c67b80 msgr2=0x558429c6a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.263+0000 7f54e0c53740 1 --2- 172.21.15.62:0/379963860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c67b80 0x558429c6a040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009900 tx=0x7f54c8005c00 comp rx=0 tx=0).stop 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.263+0000 7f54e0c53740 1 -- 172.21.15.62:0/379963860 shutdown_connections 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.263+0000 7f54e0c53740 1 --2- 172.21.15.62:0/379963860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c67b80 0x558429c6a040 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.263+0000 7f54e0c53740 1 --2- 172.21.15.62:0/379963860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c675b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.263+0000 7f54e0c53740 1 --2- 172.21.15.62:0/379963860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 0x558429c625b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.263+0000 7f54e0c53740 1 -- 172.21.15.62:0/379963860 >> 172.21.15.62:0/379963860 conn(0x558429c5d820 msgr2=0x558429c5dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.265+0000 7f54e0c53740 1 -- 172.21.15.62:0/379963860 shutdown_connections 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.265+0000 7f54e0c53740 1 -- 172.21.15.62:0/379963860 wait complete. 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54e0c53740 1 -- start start 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54e0c53740 1 Processor -- start 2024-09-15T23:17:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c6c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c62af0 0x558429c6c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c67b80 0x558429c73820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54e0c53740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558429c6aaf0 con 0x558429c62af0 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54e0c53740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558429b46290 con 0x558429c67b80 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54e0c53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558429c6ac70 con 0x558429c621b0 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54e0c53740 10 client.0 _ll_register_callbacks cb 0x558429b77140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54d6ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c62af0 0x558429c6c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c6c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c6c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43730/0 (socket says 172.21.15.62:43730) 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54d6ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c62af0 0x558429c6c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43646/0 (socket says 172.21.15.62:43646) 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54d77fe640 1 -- 172.21.15.62:0/3328434106 learned_addr learned my addr 172.21.15.62:0/3328434106 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.266+0000 7f54d7fff640 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c67b80 0x558429c73820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54d77fe640 1 -- 172.21.15.62:0/3328434106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c67b80 msgr2=0x558429c73820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54d77fe640 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c67b80 0x558429c73820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54d77fe640 1 -- 172.21.15.62:0/3328434106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c62af0 msgr2=0x558429c6c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54d77fe640 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c62af0 0x558429c6c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54d77fe640 1 -- 172.21.15.62:0/3328434106 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584298d0210 con 0x558429c621b0 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54d77fe640 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c6c310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f54d00106a0 tx=0x7f54d0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3afe0e130ed8daa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54b6ffd640 1 -- 172.21.15.62:0/3328434106 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0026070 con 0x558429c621b0 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54e0c53740 1 -- 172.21.15.62:0/3328434106 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c80319f0 con 0x558429c621b0 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54b6ffd640 1 -- 172.21.15.62:0/3328434106 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f54d0009040 con 0x558429c621b0 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54b6ffd640 1 -- 172.21.15.62:0/3328434106 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54d0033050 con 0x558429c621b0 2024-09-15T23:17:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54e0c53740 10 client.25354 fetch_fsmap learned FSMap version 28 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.267+0000 7f54e0c53740 1 -- 172.21.15.62:0/3328434106 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55842999c620 con 0x558429c621b0 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.268+0000 7f54b6ffd640 1 -- 172.21.15.62:0/3328434106 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54c80319f0 con 0x558429c621b0 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.268+0000 7f54b6ffd640 1 -- 172.21.15.62:0/3328434106 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f54d001f040 con 0x558429c621b0 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.268+0000 7f54e0c53740 10 client.25354 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.268+0000 7f54e0c53740 10 client.25354 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.268+0000 7f54e0c53740 20 client.25354 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.268+0000 7f54e0c53740 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558429c77ee0 0x558429c7a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.268+0000 7f54e0c53740 4 client.25354 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.268+0000 7f54e0c53740 1 -- 172.21.15.62:0/3328434106 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558429bcaa20 con 0x558429c77ee0 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.269+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558429c77ee0 0x558429c7a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.269+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558429c77ee0 0x558429c7a3a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=983dfd64ba9855e7 server_cookie=bda45939eb2d487b in_seq=0 out_seq=0 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.269+0000 7f54b6ffd640 10 client.25354 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.270+0000 7f54b6ffd640 1 -- 172.21.15.62:0/3328434106 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558429bcaa20 con 0x558429c77ee0 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.270+0000 7f54b6ffd640 10 client.25354 handle_command_reply: tid=0 2024-09-15T23:17:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.270+0000 7f54e0c53740 1 client.25354 shutdown 2024-09-15T23:17:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 -- 172.21.15.62:0/3328434106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 msgr2=0x558429c6c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c6c310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f54d00106a0 tx=0x7f54d0004ab0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 -- 172.21.15.62:0/3328434106 shutdown_connections 2024-09-15T23:17:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558429c77ee0 0x558429c7a3a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c67b80 0x558429c73820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c62af0 0x558429c6c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 --2- 172.21.15.62:0/3328434106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c6c310 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54b6ffd640 0 client.25354 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 -- 172.21.15.62:0/3328434106 >> 172.21.15.62:0/3328434106 conn(0x558429c5d820 msgr2=0x558429c541c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 -- 172.21.15.62:0/3328434106 shutdown_connections 2024-09-15T23:17:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 1 -- 172.21.15.62:0/3328434106 wait complete. 2024-09-15T23:17:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.272+0000 7f54e0c53740 20 client.25354 trim_cache size 0 max 16384 2024-09-15T23:17:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.296+0000 7f54e0c53740 1 Processor -- start 2024-09-15T23:17:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.296+0000 7f54e0c53740 1 -- start start 2024-09-15T23:17:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c5d820 0x558429c6b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 0x558429c73740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c540b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54e0c53740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81aabf0 con 0x558429c621b0 2024-09-15T23:17:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54e0c53740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81a8490 con 0x558429c62af0 2024-09-15T23:17:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54e0c53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81a62e0 con 0x558429c5d820 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c5d820 0x558429c6b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d77fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c5d820 0x558429c6b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43746/0 (socket says 172.21.15.62:43746) 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d77fe640 1 -- 172.21.15.62:0/1578373742 learned_addr learned my addr 172.21.15.62:0/1578373742 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d7fff640 1 --2- 172.21.15.62:0/1578373742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/1578373742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 0x558429c73740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d7fff640 1 -- 172.21.15.62:0/1578373742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c5d820 msgr2=0x558429c6b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d7fff640 1 --2- 172.21.15.62:0/1578373742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c5d820 0x558429c6b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d7fff640 1 -- 172.21.15.62:0/1578373742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 msgr2=0x558429c73740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d7fff640 1 --2- 172.21.15.62:0/1578373742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 0x558429c73740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d7fff640 1 -- 172.21.15.62:0/1578373742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558429c5d640 con 0x558429c62af0 2024-09-15T23:17:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.297+0000 7f54d7fff640 1 --2- 172.21.15.62:0/1578373742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c540b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f54c8005ec0 tx=0x7f54c8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0f83f9890501de3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.298+0000 7f54bd7fa640 1 -- 172.21.15.62:0/1578373742 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c8040070 con 0x558429c62af0 2024-09-15T23:17:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.298+0000 7f54bd7fa640 1 -- 172.21.15.62:0/1578373742 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54c80035f0 con 0x558429c62af0 2024-09-15T23:17:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.298+0000 7f54bd7fa640 1 -- 172.21.15.62:0/1578373742 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c8035040 con 0x558429c62af0 2024-09-15T23:17:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.299+0000 7f54e0c53740 1 -- 172.21.15.62:0/1578373742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 msgr2=0x558429c540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.299+0000 7f54e0c53740 1 --2- 172.21.15.62:0/1578373742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c540b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f54c8005ec0 tx=0x7f54c8005fd0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.299+0000 7f54e0c53740 1 -- 172.21.15.62:0/1578373742 shutdown_connections 2024-09-15T23:17:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.299+0000 7f54e0c53740 1 --2- 172.21.15.62:0/1578373742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c540b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.299+0000 7f54e0c53740 1 --2- 172.21.15.62:0/1578373742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c621b0 0x558429c73740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.299+0000 7f54e0c53740 1 --2- 172.21.15.62:0/1578373742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c5d820 0x558429c6b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.299+0000 7f54e0c53740 1 -- 172.21.15.62:0/1578373742 >> 172.21.15.62:0/1578373742 conn(0x558429c635b0 msgr2=0x558429c539a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.299+0000 7f54e0c53740 1 -- 172.21.15.62:0/1578373742 shutdown_connections 2024-09-15T23:17:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.299+0000 7f54e0c53740 1 -- 172.21.15.62:0/1578373742 wait complete. 2024-09-15T23:17:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.300+0000 7f54e0c53740 1 -- start start 2024-09-15T23:17:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.300+0000 7f54e0c53740 1 Processor -- start 2024-09-15T23:17:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c5d820 0x558429c73580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c71be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54e0c53740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c72120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54e0c53740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81a6950 con 0x558429c5d820 2024-09-15T23:17:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54e0c53740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81a7dc0 con 0x558429c62af0 2024-09-15T23:17:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54e0c53740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81a7020 con 0x558429c621b0 2024-09-15T23:17:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54e0c53740 10 client.0 _ll_register_callbacks cb 0x7f54d00136b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:17:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c72120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54d7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c72120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:37320/0 (socket says 172.21.15.62:37320) 2024-09-15T23:17:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.301+0000 7f54d7fff640 1 -- 172.21.15.62:0/2817578075 learned_addr learned my addr 172.21.15.62:0/2817578075 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.302+0000 7f54d77fe640 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c5d820 0x558429c73580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.302+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c71be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.302+0000 7f54d77fe640 1 -- 172.21.15.62:0/2817578075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 msgr2=0x558429c71be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.302+0000 7f54d77fe640 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c71be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.302+0000 7f54d77fe640 1 -- 172.21.15.62:0/2817578075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 msgr2=0x558429c72120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:17:51.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.302+0000 7f54d77fe640 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c72120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.302+0000 7f54d77fe640 1 -- 172.21.15.62:0/2817578075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558429c728b0 con 0x558429c5d820 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.303+0000 7f54d77fe640 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c5d820 0x558429c73580 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f54d00106a0 tx=0x7f54d00132f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e8cb39334c480c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.303+0000 7f54b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.303+0000 7f54b6ffd640 1 -- 172.21.15.62:0/2817578075 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d0026040 con 0x558429c5d820 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.304+0000 7f54b6ffd640 1 -- 172.21.15.62:0/2817578075 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f54d0013bc0 con 0x558429c5d820 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.304+0000 7f54e0c53740 1 -- 172.21.15.62:0/2817578075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c80035f0 con 0x558429c5d820 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.304+0000 7f54b6ffd640 1 -- 172.21.15.62:0/2817578075 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f54d0003b80 con 0x558429c5d820 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.304+0000 7f54e0c53740 10 client.15819 fetch_fsmap learned FSMap version 28 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.304+0000 7f54e0c53740 1 -- 172.21.15.62:0/2817578075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558429c5a9b0 con 0x558429c5d820 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.304+0000 7f54b6ffd640 1 -- 172.21.15.62:0/2817578075 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54c80035f0 con 0x558429c5d820 2024-09-15T23:17:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.304+0000 7f54b6ffd640 1 -- 172.21.15.62:0/2817578075 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f54d0036070 con 0x558429c5d820 2024-09-15T23:17:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.305+0000 7f54e0c53740 10 client.15819 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:17:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.305+0000 7f54e0c53740 10 client.15819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:17:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.305+0000 7f54e0c53740 20 client.15819 populate_metadata read hostname 'smithi062' 2024-09-15T23:17:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.305+0000 7f54e0c53740 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558429c67b80 0x558429c5b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.305+0000 7f54e0c53740 4 client.15819 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:17:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.305+0000 7f54e0c53740 1 -- 172.21.15.62:0/2817578075 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558429bc4d70 con 0x558429c67b80 2024-09-15T23:17:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.306+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558429c67b80 0x558429c5b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:51.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.307+0000 7f54d6ffd640 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558429c67b80 0x558429c5b420 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70307717baf52e9d server_cookie=46e2b188fe61bbe9 in_seq=0 out_seq=0 2024-09-15T23:17:51.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.307+0000 7f54b6ffd640 10 client.15819 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.318+0000 7f54b6ffd640 1 -- 172.21.15.62:0/2817578075 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558429bc4d70 con 0x558429c67b80 2024-09-15T23:17:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.318+0000 7f54b6ffd640 10 client.15819 handle_command_reply: tid=0 2024-09-15T23:17:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.318+0000 7f54e0c53740 1 client.15819 shutdown 2024-09-15T23:17:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.319+0000 7f54e0c53740 1 -- 172.21.15.62:0/2817578075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c5d820 msgr2=0x558429c73580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.319+0000 7f54e0c53740 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c5d820 0x558429c73580 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f54d00106a0 tx=0x7f54d00132f0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.319+0000 7f54e0c53740 1 -- 172.21.15.62:0/2817578075 shutdown_connections 2024-09-15T23:17:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.319+0000 7f54e0c53740 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558429c67b80 0x558429c5b420 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.319+0000 7f54e0c53740 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558429c62af0 0x558429c72120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.320+0000 7f54e0c53740 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558429c621b0 0x558429c71be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.320+0000 7f54b6ffd640 0 client.15819 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:17:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.320+0000 7f54e0c53740 1 --2- 172.21.15.62:0/2817578075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558429c5d820 0x558429c73580 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.320+0000 7f54e0c53740 1 -- 172.21.15.62:0/2817578075 >> 172.21.15.62:0/2817578075 conn(0x558429c635b0 msgr2=0x558429c5eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.320+0000 7f54e0c53740 1 -- 172.21.15.62:0/2817578075 shutdown_connections 2024-09-15T23:17:51.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.320+0000 7f54e0c53740 1 -- 172.21.15.62:0/2817578075 wait complete. 2024-09-15T23:17:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.321+0000 7f54e0c53740 20 client.15819 trim_cache size 0 max 16384 2024-09-15T23:17:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:17:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:17:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:17:51.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:17:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 -- 172.21.15.62:0/3331520486 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54c00a34e0 msgr2=0x7f54c00a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54c00a34e0 0x7f54c00a59a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f54d0013ed0 tx=0x7f54d0009500 comp rx=0 tx=0).stop 2024-09-15T23:17:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 -- 172.21.15.62:0/3331520486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d8100db0 msgr2=0x7f54d819a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d8100db0 0x7f54d819a4f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00d6d0 tx=0x7f54cc00db90 comp rx=0 tx=0).stop 2024-09-15T23:17:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 -- 172.21.15.62:0/3331520486 shutdown_connections 2024-09-15T23:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f54c00a34e0 0x7f54c00a59a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f54d8101750 0x7f54d81a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f54d8100db0 0x7f54d819a4f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 --2- 172.21.15.62:0/3331520486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d8100470 0x7f54d8199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 -- 172.21.15.62:0/3331520486 >> 172.21.15.62:0/3331520486 conn(0x7f54d80fc1e0 msgr2=0x7f54d80fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 -- 172.21.15.62:0/3331520486 shutdown_connections 2024-09-15T23:17:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:17:51.322+0000 7f54dddd6640 1 -- 172.21.15.62:0/3331520486 wait complete. 2024-09-15T23:17:51.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:51 smithi062 ceph-mon[30949]: pgmap v417: 129 pgs: 129 active+clean; 1019 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 1.06k op/s 2024-09-15T23:17:51.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:51 smithi067 ceph-mon[35665]: pgmap v417: 129 pgs: 129 active+clean; 1019 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 1.06k op/s 2024-09-15T23:17:51.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:51 smithi148 ceph-mon[35610]: pgmap v417: 129 pgs: 129 active+clean; 1019 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 1.06k op/s 2024-09-15T23:17:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:53 smithi062 ceph-mon[30949]: pgmap v418: 129 pgs: 129 active+clean; 1019 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 21 MiB/s wr, 1.13k op/s 2024-09-15T23:17:53.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:53 smithi067 ceph-mon[35665]: pgmap v418: 129 pgs: 129 active+clean; 1019 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 21 MiB/s wr, 1.13k op/s 2024-09-15T23:17:53.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:53 smithi148 ceph-mon[35610]: pgmap v418: 129 pgs: 129 active+clean; 1019 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 21 MiB/s wr, 1.13k op/s 2024-09-15T23:17:54.571 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 95.0 s, 2842.6 tps, lat 35.223 ms stddev 40.848 2024-09-15T23:17:55.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:55 smithi148 ceph-mon[35610]: pgmap v419: 129 pgs: 129 active+clean; 1019 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 1.09k op/s 2024-09-15T23:17:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:55 smithi062 ceph-mon[30949]: pgmap v419: 129 pgs: 129 active+clean; 1019 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 1.09k op/s 2024-09-15T23:17:56.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:55 smithi067 ceph-mon[35665]: pgmap v419: 129 pgs: 129 active+clean; 1019 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 1.09k op/s 2024-09-15T23:17:57.039 INFO:tasks.quiescer.fs.[cephfs]:Total duration (39.1) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:17:57.039 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-15T23:17:57.040 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:17:57.431 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:57 smithi148 ceph-mon[35610]: pgmap v420: 129 pgs: 129 active+clean; 1019 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 883 op/s 2024-09-15T23:17:57.822 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:57 smithi062 ceph-mon[30949]: pgmap v420: 129 pgs: 129 active+clean; 1019 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 883 op/s 2024-09-15T23:17:57.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.955+0000 7f4bf5607640 1 -- 172.21.15.62:0/1206272904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf0102ef0 msgr2=0x7f4bf010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:57.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.955+0000 7f4bf5607640 1 --2- 172.21.15.62:0/1206272904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf0102ef0 0x7f4bf010da40 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4be8007290 tx=0x7f4be8032d90 comp rx=0 tx=0).stop 2024-09-15T23:17:57.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.956+0000 7f4bf5607640 1 -- 172.21.15.62:0/1206272904 shutdown_connections 2024-09-15T23:17:57.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.956+0000 7f4bf5607640 1 --2- 172.21.15.62:0/1206272904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf0102ef0 0x7f4bf010da40 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:57.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.956+0000 7f4bf5607640 1 --2- 172.21.15.62:0/1206272904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf0102570 0x7f4bf01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:57.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.956+0000 7f4bf5607640 1 --2- 172.21.15.62:0/1206272904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bf0100e20 0x7f4bf0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:57.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.956+0000 7f4bf5607640 1 -- 172.21.15.62:0/1206272904 >> 172.21.15.62:0/1206272904 conn(0x7f4bf00fc7d0 msgr2=0x7f4bf00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:57.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.957+0000 7f4bf5607640 1 -- 172.21.15.62:0/1206272904 shutdown_connections 2024-09-15T23:17:57.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.957+0000 7f4bf5607640 1 -- 172.21.15.62:0/1206272904 wait complete. 2024-09-15T23:17:57.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.957+0000 7f4bf5607640 1 Processor -- start 2024-09-15T23:17:57.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.961+0000 7f4bf5607640 1 -- start start 2024-09-15T23:17:57.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.961+0000 7f4bf5607640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bf0100e20 0x7f4bf0196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:57.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.961+0000 7f4bf5607640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf0102570 0x7f4bf01967b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:57.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.962+0000 7f4bf5607640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf0102ef0 0x7f4bf019ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:57.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.962+0000 7f4bf5607640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010fe50 con 0x7f4bf0102ef0 2024-09-15T23:17:57.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.962+0000 7f4bf5607640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010fcd0 con 0x7f4bf0102570 2024-09-15T23:17:57.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.962+0000 7f4bf5607640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010ffd0 con 0x7f4bf0100e20 2024-09-15T23:17:57.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.962+0000 7f4bee7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf0102570 0x7f4bf01967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:57.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.962+0000 7f4bee7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf0102570 0x7f4bf01967b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49716/0 (socket says 172.21.15.62:49716) 2024-09-15T23:17:57.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.962+0000 7f4bee7fc640 1 -- 172.21.15.62:0/929068833 learned_addr learned my addr 172.21.15.62:0/929068833 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:17:57.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.962+0000 7f4bef7fe640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf0102ef0 0x7f4bf019ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:57.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.963+0000 7f4beeffd640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bf0100e20 0x7f4bf0196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:57.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.963+0000 7f4beeffd640 1 -- 172.21.15.62:0/929068833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf0102570 msgr2=0x7f4bf01967b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:57.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.963+0000 7f4beeffd640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf0102570 0x7f4bf01967b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:57.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.963+0000 7f4beeffd640 1 -- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf0102ef0 msgr2=0x7f4bf019ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:57.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.963+0000 7f4beeffd640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf0102ef0 0x7f4bf019ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:57.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.963+0000 7f4beeffd640 1 -- 172.21.15.62:0/929068833 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf019b220 con 0x7f4bf0100e20 2024-09-15T23:17:57.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.964+0000 7f4beeffd640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bf0100e20 0x7f4bf0196270 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4be400e9f0 tx=0x7f4be400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.964+0000 7f4bd7fff640 1 -- 172.21.15.62:0/929068833 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be400ccc0 con 0x7f4bf0100e20 2024-09-15T23:17:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.965+0000 7f4bd7fff640 1 -- 172.21.15.62:0/929068833 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4be4010910 con 0x7f4bf0100e20 2024-09-15T23:17:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.965+0000 7f4bd7fff640 1 -- 172.21.15.62:0/929068833 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be4018980 con 0x7f4bf0100e20 2024-09-15T23:17:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.965+0000 7f4bef7fe640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf0102ef0 0x7f4bf019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:17:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.966+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf019b510 con 0x7f4bf0100e20 2024-09-15T23:17:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.966+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf01a2df0 con 0x7f4bf0100e20 2024-09-15T23:17:57.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.968+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf0101200 con 0x7f4bf0100e20 2024-09-15T23:17:57.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.973+0000 7f4bd7fff640 1 -- 172.21.15.62:0/929068833 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4be401f020 con 0x7f4bf0100e20 2024-09-15T23:17:57.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.974+0000 7f4bd7fff640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4bd80a3450 0x7f4bd80a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:17:57.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.974+0000 7f4bd7fff640 1 -- 172.21.15.62:0/929068833 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4be40d0950 con 0x7f4bf0100e20 2024-09-15T23:17:57.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.975+0000 7f4bee7fc640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4bd80a3450 0x7f4bd80a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:17:57.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.976+0000 7f4bee7fc640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4bd80a3450 0x7f4bd80a5910 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00fec10 tx=0x7f4be0009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:17:57.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:57.976+0000 7f4bd7fff640 1 -- 172.21.15.62:0/929068833 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4be4026030 con 0x7f4bf0100e20 2024-09-15T23:17:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:57 smithi067 ceph-mon[35665]: pgmap v420: 129 pgs: 129 active+clean; 1019 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 883 op/s 2024-09-15T23:17:58.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.368+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bf01975d0 con 0x7f4bf0100e20 2024-09-15T23:17:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.371+0000 7f4bd7fff640 1 -- 172.21.15.62:0/929068833 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11907 (secure 0 0 0) 0x7f4bf01975d0 con 0x7f4bf0100e20 2024-09-15T23:17:58.372 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:17:58.373 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":28,"btime":"2024-09-15T23:17:33:092139+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:17:32.873645+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:17:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 28 2024-09-15T23:17:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.375+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4bd80a3450 msgr2=0x7f4bd80a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.375+0000 7f4bf5607640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4bd80a3450 0x7f4bd80a5910 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4bf00fec10 tx=0x7f4be0009190 comp rx=0 tx=0).stop 2024-09-15T23:17:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.375+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bf0100e20 msgr2=0x7f4bf0196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:17:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.376+0000 7f4bf5607640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bf0100e20 0x7f4bf0196270 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4be400e9f0 tx=0x7f4be400eeb0 comp rx=0 tx=0).stop 2024-09-15T23:17:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.376+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 shutdown_connections 2024-09-15T23:17:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.376+0000 7f4bf5607640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4bd80a3450 0x7f4bd80a5910 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.376+0000 7f4bf5607640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4bf0102ef0 0x7f4bf019ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.376+0000 7f4bf5607640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bf0102570 0x7f4bf01967b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.376+0000 7f4bf5607640 1 --2- 172.21.15.62:0/929068833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4bf0100e20 0x7f4bf0196270 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:17:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.376+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 >> 172.21.15.62:0/929068833 conn(0x7f4bf00fc7d0 msgr2=0x7f4bf01068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:17:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.377+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 shutdown_connections 2024-09-15T23:17:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:17:58.377+0000 7f4bf5607640 1 -- 172.21.15.62:0/929068833 wait complete. 2024-09-15T23:17:58.637 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:58 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/929068833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:17:58.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:58 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/929068833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:17:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:58 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/929068833' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:17:59.198 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-15T23:17:59.198 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-15T23:17:59.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 100.0 s, 2156.6 tps, lat 46.013 ms stddev 49.693 2024-09-15T23:17:59.637 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:17:59.811 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:17:59 smithi062 ceph-mon[30949]: pgmap v421: 129 pgs: 129 active+clean; 1019 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 758 op/s 2024-09-15T23:18:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:17:59 smithi067 ceph-mon[35665]: pgmap v421: 129 pgs: 129 active+clean; 1019 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 758 op/s 2024-09-15T23:18:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:17:59 smithi148 ceph-mon[35610]: pgmap v421: 129 pgs: 129 active+clean; 1019 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.6 MiB/s wr, 758 op/s 2024-09-15T23:18:00.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.260+0000 7fede37fe640 1 -- 172.21.15.62:0/254073917 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4030ac0 con 0x7fede4109450 2024-09-15T23:18:00.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.260+0000 7fedeaf20640 1 -- 172.21.15.62:0/254073917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede4109450 msgr2=0x7fede4109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.260+0000 7fedeaf20640 1 --2- 172.21.15.62:0/254073917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede4109450 0x7fede4109830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fedd400aa10 tx=0x7fedd402b720 comp rx=0 tx=0).stop 2024-09-15T23:18:00.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.261+0000 7fedeaf20640 1 -- 172.21.15.62:0/254073917 shutdown_connections 2024-09-15T23:18:00.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.261+0000 7fedeaf20640 1 --2- 172.21.15.62:0/254073917 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fede410ddc0 0x7fede4115940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.261+0000 7fedeaf20640 1 --2- 172.21.15.62:0/254073917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fede4109e00 0x7fede410d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.261+0000 7fedeaf20640 1 --2- 172.21.15.62:0/254073917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede4109450 0x7fede4109830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.261+0000 7fedeaf20640 1 -- 172.21.15.62:0/254073917 >> 172.21.15.62:0/254073917 conn(0x7fede406e140 msgr2=0x7fede406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:00.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.261+0000 7fedeaf20640 1 -- 172.21.15.62:0/254073917 shutdown_connections 2024-09-15T23:18:00.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.261+0000 7fedeaf20640 1 -- 172.21.15.62:0/254073917 wait complete. 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.261+0000 7fedeaf20640 1 Processor -- start 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fedeaf20640 1 -- start start 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fedeaf20640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fede4109450 0x7fede4122b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fedeaf20640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fede4109e00 0x7fede41230a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fedeaf20640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede410ddc0 0x7fede4114b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fedeaf20640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fede4117f50 con 0x7fede4109450 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fedeaf20640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fede4117dd0 con 0x7fede410ddc0 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fedeaf20640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fede41180d0 con 0x7fede4109e00 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede94bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede410ddc0 0x7fede4114b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede8cbc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fede4109450 0x7fede4122b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede94bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede410ddc0 0x7fede4114b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49730/0 (socket says 172.21.15.62:49730) 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede94bd640 1 -- 172.21.15.62:0/773278600 learned_addr learned my addr 172.21.15.62:0/773278600 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede3fff640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fede4109e00 0x7fede41230a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede94bd640 1 -- 172.21.15.62:0/773278600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fede4109e00 msgr2=0x7fede41230a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede94bd640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fede4109e00 0x7fede41230a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede94bd640 1 -- 172.21.15.62:0/773278600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fede4109450 msgr2=0x7fede4122b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede94bd640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fede4109450 0x7fede4122b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.262+0000 7fede94bd640 1 -- 172.21.15.62:0/773278600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede41153e0 con 0x7fede410ddc0 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.263+0000 7fede94bd640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede410ddc0 0x7fede4114b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feddc00eaa0 tx=0x7feddc00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.263+0000 7fede1ffb640 1 -- 172.21.15.62:0/773278600 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc00cdb0 con 0x7fede410ddc0 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.263+0000 7fede1ffb640 1 -- 172.21.15.62:0/773278600 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feddc005020 con 0x7fede410ddc0 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.263+0000 7fede1ffb640 1 -- 172.21.15.62:0/773278600 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feddc0116d0 con 0x7fede410ddc0 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.263+0000 7fedeaf20640 1 -- 172.21.15.62:0/773278600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede41156d0 con 0x7fede410ddc0 2024-09-15T23:18:00.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.264+0000 7fedeaf20640 1 -- 172.21.15.62:0/773278600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede411e7e0 con 0x7fede410ddc0 2024-09-15T23:18:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.267+0000 7fede1ffb640 1 -- 172.21.15.62:0/773278600 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7feddc005540 con 0x7fede410ddc0 2024-09-15T23:18:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.267+0000 7feded2f9740 1 Processor -- start 2024-09-15T23:18:00.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.267+0000 7feded2f9740 1 -- start start 2024-09-15T23:18:00.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7feded2f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e43c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7feded2f9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7feded2f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e50e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7feded2f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5581d2da8160 con 0x5581d2e4ea30 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7feded2f9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5581d2d4f720 con 0x5581d2e44170 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7feded2f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5581d2da4120 con 0x5581d2e43850 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e50e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e50e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60036/0 (socket says 172.21.15.62:60036) 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede3fff640 1 -- 172.21.15.62:0/3948344332 learned_addr learned my addr 172.21.15.62:0/3948344332 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede8cbc640 1 --2- 172.21.15.62:0/3948344332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede3fff640 1 --2- 172.21.15.62:0/3948344332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e43c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede1ffb640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fedcc0a3410 0x7fedcc0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede1ffb640 1 -- 172.21.15.62:0/773278600 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7feddc0cfd60 con 0x7fede410ddc0 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede94bd640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fedcc0a3410 0x7fedcc0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede3fff640 1 -- 172.21.15.62:0/3948344332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 msgr2=0x5581d2e4e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:00.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede3fff640 1 --2- 172.21.15.62:0/3948344332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede3fff640 1 -- 172.21.15.62:0/3948344332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 msgr2=0x5581d2e50e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede3fff640 1 --2- 172.21.15.62:0/3948344332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e50e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.268+0000 7fede3fff640 1 -- 172.21.15.62:0/3948344332 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581d2b19ba0 con 0x5581d2e43850 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.269+0000 7fede3fff640 1 --2- 172.21.15.62:0/3948344332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e43c30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x5581d2e41160 tx=0x7fedd800b410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2168eb67d8caa7fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.269+0000 7fede94bd640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fedcc0a3410 0x7fedcc0a58d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feddc0058f0 tx=0x7feddc009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.269+0000 7fedcaffd640 1 -- 172.21.15.62:0/3948344332 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8017070 con 0x5581d2e43850 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.269+0000 7fedcaffd640 1 -- 172.21.15.62:0/3948344332 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd8009670 con 0x5581d2e43850 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.269+0000 7fedcaffd640 1 -- 172.21.15.62:0/3948344332 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8003850 con 0x5581d2e43850 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.270+0000 7feded2f9740 1 -- 172.21.15.62:0/3948344332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 msgr2=0x5581d2e43c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.270+0000 7feded2f9740 1 --2- 172.21.15.62:0/3948344332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e43c30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x5581d2e41160 tx=0x7fedd800b410 comp rx=0 tx=0).stop 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.270+0000 7feded2f9740 1 -- 172.21.15.62:0/3948344332 shutdown_connections 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.270+0000 7feded2f9740 1 --2- 172.21.15.62:0/3948344332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e50e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.270+0000 7feded2f9740 1 --2- 172.21.15.62:0/3948344332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.270+0000 7feded2f9740 1 --2- 172.21.15.62:0/3948344332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e43c30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.270+0000 7feded2f9740 1 -- 172.21.15.62:0/3948344332 >> 172.21.15.62:0/3948344332 conn(0x5581d2e3fde0 msgr2=0x5581d2e401c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:00.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.271+0000 7feded2f9740 1 -- 172.21.15.62:0/3948344332 shutdown_connections 2024-09-15T23:18:00.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.272+0000 7feded2f9740 1 -- 172.21.15.62:0/3948344332 wait complete. 2024-09-15T23:18:00.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.272+0000 7feded2f9740 1 -- start start 2024-09-15T23:18:00.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.272+0000 7feded2f9740 1 Processor -- start 2024-09-15T23:18:00.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.273+0000 7feded2f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e4b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.273+0000 7feded2f9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7feded2f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e4c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7feded2f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5581d2e51600 con 0x5581d2e4ea30 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7feded2f9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5581d2d30900 con 0x5581d2e44170 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7feded2f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5581d2e51780 con 0x5581d2e43850 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7feded2f9740 10 client.0 _ll_register_callbacks cb 0x5581d2d2e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e4b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e4b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35958/0 (socket says 172.21.15.62:35958) 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede3fff640 1 -- 172.21.15.62:0/2304144014 learned_addr learned my addr 172.21.15.62:0/2304144014 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede3fff640 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e4c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede8cbc640 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede3fff640 1 -- 172.21.15.62:0/2304144014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 msgr2=0x5581d2e4bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede3fff640 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede3fff640 1 -- 172.21.15.62:0/2304144014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 msgr2=0x5581d2e4c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede3fff640 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e4c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede3fff640 1 -- 172.21.15.62:0/2304144014 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581d2d70c50 con 0x5581d2e43850 2024-09-15T23:18:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.274+0000 7fede8cbc640 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.275+0000 7fedc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:18:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.275+0000 7fede3fff640 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e4b9d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fedd8005d00 tx=0x7fedd8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7914383c58c8795 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.275+0000 7fedc3fff640 1 -- 172.21.15.62:0/2304144014 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd8017050 con 0x5581d2e43850 2024-09-15T23:18:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.275+0000 7fedc3fff640 1 -- 172.21.15.62:0/2304144014 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fedd8005040 con 0x5581d2e43850 2024-09-15T23:18:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.275+0000 7feded2f9740 1 -- 172.21.15.62:0/2304144014 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd8009670 con 0x5581d2e43850 2024-09-15T23:18:00.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.276+0000 7fedc3fff640 1 -- 172.21.15.62:0/2304144014 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fedd8033050 con 0x5581d2e43850 2024-09-15T23:18:00.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.278+0000 7feded2f9740 10 client.25378 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:00.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.278+0000 7feded2f9740 1 -- 172.21.15.62:0/2304144014 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5581d2e5d350 con 0x5581d2e43850 2024-09-15T23:18:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.279+0000 7fedc3fff640 1 -- 172.21.15.62:0/2304144014 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd8009670 con 0x5581d2e43850 2024-09-15T23:18:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.279+0000 7fedc3fff640 1 -- 172.21.15.62:0/2304144014 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fedd8004c30 con 0x5581d2e43850 2024-09-15T23:18:00.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.280+0000 7feded2f9740 10 client.25378 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.280+0000 7feded2f9740 10 client.25378 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:18:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.280+0000 7feded2f9740 20 client.25378 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.280+0000 7feded2f9740 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5581d2e5e6a0 0x5581d2e60a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.280+0000 7feded2f9740 4 client.25378 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:00.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.280+0000 7feded2f9740 1 -- 172.21.15.62:0/2304144014 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581d2db5890 con 0x5581d2e5e6a0 2024-09-15T23:18:00.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.287+0000 7fede8cbc640 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5581d2e5e6a0 0x5581d2e60a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.289+0000 7fede8cbc640 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5581d2e5e6a0 0x5581d2e60a90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbc5162fc15619bc server_cookie=d9677c494c86f93d in_seq=0 out_seq=0 2024-09-15T23:18:00.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.289+0000 7fedc3fff640 10 client.25378 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.290+0000 7fedc3fff640 1 -- 172.21.15.62:0/2304144014 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5581d2db5890 con 0x5581d2e5e6a0 2024-09-15T23:18:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.290+0000 7fedc3fff640 10 client.25378 handle_command_reply: tid=0 2024-09-15T23:18:00.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.290+0000 7feded2f9740 1 client.25378 shutdown 2024-09-15T23:18:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.293+0000 7feded2f9740 1 -- 172.21.15.62:0/2304144014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 msgr2=0x5581d2e4b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.293+0000 7feded2f9740 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e4b9d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fedd8005d00 tx=0x7fedd8005850 comp rx=0 tx=0).stop 2024-09-15T23:18:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.300+0000 7feded2f9740 1 -- 172.21.15.62:0/2304144014 shutdown_connections 2024-09-15T23:18:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.300+0000 7feded2f9740 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5581d2e5e6a0 0x5581d2e60a90 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.300+0000 7fedc3fff640 0 client.25378 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.300+0000 7feded2f9740 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e4c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.300+0000 7feded2f9740 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.300+0000 7feded2f9740 1 --2- 172.21.15.62:0/2304144014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e43850 0x5581d2e4b9d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.300+0000 7feded2f9740 1 -- 172.21.15.62:0/2304144014 >> 172.21.15.62:0/2304144014 conn(0x5581d2e3fde0 msgr2=0x5581d2e45ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.303+0000 7feded2f9740 1 -- 172.21.15.62:0/2304144014 shutdown_connections 2024-09-15T23:18:00.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.304+0000 7feded2f9740 1 -- 172.21.15.62:0/2304144014 wait complete. 2024-09-15T23:18:00.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.304+0000 7feded2f9740 20 client.25378 trim_cache size 0 max 16384 2024-09-15T23:18:00.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.339+0000 7feded2f9740 1 Processor -- start 2024-09-15T23:18:00.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.340+0000 7feded2f9740 1 -- start start 2024-09-15T23:18:00.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.340+0000 7feded2f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e43850 0x5581d2e46ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.341+0000 7feded2f9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.341+0000 7feded2f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e4ea30 0x5581d2e66140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.341+0000 7feded2f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fede41c8b90 con 0x5581d2e43850 2024-09-15T23:18:00.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.341+0000 7feded2f9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fede41c6430 con 0x5581d2e44170 2024-09-15T23:18:00.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.341+0000 7feded2f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fede41c4280 con 0x5581d2e4ea30 2024-09-15T23:18:00.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.341+0000 7fede8cbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.341+0000 7fede8cbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49758/0 (socket says 172.21.15.62:49758) 2024-09-15T23:18:00.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.341+0000 7fede8cbc640 1 -- 172.21.15.62:0/3277433567 learned_addr learned my addr 172.21.15.62:0/3277433567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:00.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.342+0000 7fede8cbc640 1 -- 172.21.15.62:0/3277433567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e4ea30 msgr2=0x5581d2e66140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:18:00.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.342+0000 7fede8cbc640 1 --2- 172.21.15.62:0/3277433567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e4ea30 0x5581d2e66140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.342+0000 7fede8cbc640 1 -- 172.21.15.62:0/3277433567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e43850 msgr2=0x5581d2e46ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:18:00.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.342+0000 7fede8cbc640 1 --2- 172.21.15.62:0/3277433567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e43850 0x5581d2e46ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.342+0000 7fede8cbc640 1 -- 172.21.15.62:0/3277433567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581d2e40e90 con 0x5581d2e44170 2024-09-15T23:18:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.344+0000 7fede8cbc640 1 --2- 172.21.15.62:0/3277433567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4d940 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fedd402bc40 tx=0x7fedd40040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e38d5cc11d635bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.345+0000 7fedca7fc640 1 -- 172.21.15.62:0/3277433567 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4036040 con 0x5581d2e44170 2024-09-15T23:18:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.345+0000 7fedca7fc640 1 -- 172.21.15.62:0/3277433567 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd4004220 con 0x5581d2e44170 2024-09-15T23:18:00.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.345+0000 7fedca7fc640 1 -- 172.21.15.62:0/3277433567 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd40047a0 con 0x5581d2e44170 2024-09-15T23:18:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.346+0000 7feded2f9740 1 -- 172.21.15.62:0/3277433567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 msgr2=0x5581d2e4d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.346+0000 7feded2f9740 1 --2- 172.21.15.62:0/3277433567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4d940 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fedd402bc40 tx=0x7fedd40040f0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.346+0000 7feded2f9740 1 -- 172.21.15.62:0/3277433567 shutdown_connections 2024-09-15T23:18:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.346+0000 7feded2f9740 1 --2- 172.21.15.62:0/3277433567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e4ea30 0x5581d2e66140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.346+0000 7feded2f9740 1 --2- 172.21.15.62:0/3277433567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e44170 0x5581d2e4d940 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.346+0000 7feded2f9740 1 --2- 172.21.15.62:0/3277433567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e43850 0x5581d2e46ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.346+0000 7feded2f9740 1 -- 172.21.15.62:0/3277433567 >> 172.21.15.62:0/3277433567 conn(0x5581d2e3fde0 msgr2=0x5581d2e4d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.347+0000 7feded2f9740 1 -- 172.21.15.62:0/3277433567 shutdown_connections 2024-09-15T23:18:00.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.347+0000 7feded2f9740 1 -- 172.21.15.62:0/3277433567 wait complete. 2024-09-15T23:18:00.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.347+0000 7feded2f9740 1 -- start start 2024-09-15T23:18:00.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.348+0000 7feded2f9740 1 Processor -- start 2024-09-15T23:18:00.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.348+0000 7feded2f9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e43850 0x5581d2e57020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.348+0000 7feded2f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e44170 0x5581d2e52350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.348+0000 7feded2f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e52890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.348+0000 7feded2f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fede41c48f0 con 0x5581d2e4ea30 2024-09-15T23:18:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.348+0000 7feded2f9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fede41c5d60 con 0x5581d2e43850 2024-09-15T23:18:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.348+0000 7feded2f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fede41c4fc0 con 0x5581d2e44170 2024-09-15T23:18:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.348+0000 7feded2f9740 10 client.0 _ll_register_callbacks cb 0x7fedd800be40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.349+0000 7fede8cbc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e44170 0x5581d2e52350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.349+0000 7fede8cbc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e44170 0x5581d2e52350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35962/0 (socket says 172.21.15.62:35962) 2024-09-15T23:18:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.349+0000 7fede8cbc640 1 -- 172.21.15.62:0/3786988434 learned_addr learned my addr 172.21.15.62:0/3786988434 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.349+0000 7fede3fff640 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e43850 0x5581d2e57020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.350+0000 7fede8cbc640 1 -- 172.21.15.62:0/3786988434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e43850 msgr2=0x5581d2e57020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.350+0000 7fede8cbc640 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e43850 0x5581d2e57020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.350+0000 7fede8cbc640 1 -- 172.21.15.62:0/3786988434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 msgr2=0x5581d2e52890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.350+0000 7fede8cbc640 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e52890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.350+0000 7fede8cbc640 1 -- 172.21.15.62:0/3786988434 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581d2e530c0 con 0x5581d2e44170 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.350+0000 7fede8cbc640 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e44170 0x5581d2e52350 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fedd4049040 tx=0x7fedd4005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ce8cb505b5752da server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.350+0000 7fedc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.350+0000 7fedc3fff640 1 -- 172.21.15.62:0/3786988434 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4055060 con 0x5581d2e44170 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.351+0000 7feded2f9740 1 -- 172.21.15.62:0/3786988434 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd4004220 con 0x5581d2e44170 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.351+0000 7fedc3fff640 1 -- 172.21.15.62:0/3786988434 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fedd40308d0 con 0x5581d2e44170 2024-09-15T23:18:00.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.351+0000 7fedc3fff640 1 -- 172.21.15.62:0/3786988434 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fedd405c050 con 0x5581d2e44170 2024-09-15T23:18:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.352+0000 7feded2f9740 10 client.25381 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:00.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.352+0000 7feded2f9740 1 -- 172.21.15.62:0/3786988434 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5581d2ec9020 con 0x5581d2e44170 2024-09-15T23:18:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.352+0000 7fedc3fff640 1 -- 172.21.15.62:0/3786988434 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd4004220 con 0x5581d2e44170 2024-09-15T23:18:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.352+0000 7fedc3fff640 1 -- 172.21.15.62:0/3786988434 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fedd403b040 con 0x5581d2e44170 2024-09-15T23:18:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.353+0000 7feded2f9740 10 client.25381 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.353+0000 7feded2f9740 10 client.25381 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:18:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.353+0000 7feded2f9740 20 client.25381 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.353+0000 7feded2f9740 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5581d2ec9830 0x5581d2ecbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.353+0000 7feded2f9740 4 client.25381 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-15T23:18:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.353+0000 7feded2f9740 1 -- 172.21.15.62:0/3786988434 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x5581d2e46780 con 0x5581d2ec9830 2024-09-15T23:18:00.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.353+0000 7fede3fff640 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5581d2ec9830 0x5581d2ecbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:00.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.354+0000 7fedc3fff640 10 client.25381 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:00.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.355+0000 7fede3fff640 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5581d2ec9830 0x5581d2ecbc50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47ad324858ccb0ee server_cookie=a0314da7892b32 in_seq=0 out_seq=0 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7fedc3fff640 1 -- 172.21.15.62:0/3786988434 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x5581d2e46780 con 0x5581d2ec9830 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7fedc3fff640 10 client.25381 handle_command_reply: tid=0 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7feded2f9740 1 client.25381 shutdown 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7feded2f9740 1 -- 172.21.15.62:0/3786988434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e44170 msgr2=0x5581d2e52350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7feded2f9740 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e44170 0x5581d2e52350 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fedd4049040 tx=0x7fedd4005c00 comp rx=0 tx=0).stop 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7feded2f9740 1 -- 172.21.15.62:0/3786988434 shutdown_connections 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7feded2f9740 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5581d2ec9830 0x5581d2ecbc50 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7feded2f9740 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5581d2e4ea30 0x5581d2e52890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7feded2f9740 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5581d2e44170 0x5581d2e52350 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7fedc3fff640 0 client.25381 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7feded2f9740 1 --2- 172.21.15.62:0/3786988434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5581d2e43850 0x5581d2e57020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.575+0000 7feded2f9740 1 -- 172.21.15.62:0/3786988434 >> 172.21.15.62:0/3786988434 conn(0x5581d2e3fde0 msgr2=0x5581d2e4da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.576+0000 7feded2f9740 1 -- 172.21.15.62:0/3786988434 shutdown_connections 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.576+0000 7feded2f9740 1 -- 172.21.15.62:0/3786988434 wait complete. 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.576+0000 7feded2f9740 20 client.25381 trim_cache size 0 max 16384 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 28, 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 6, 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:18:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "3956db1e": { 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 6, 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 139.6, 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:18:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:00.580 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.576+0000 7fedeaf20640 1 -- 172.21.15.62:0/773278600 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fedcc0a3410 msgr2=0x7fedcc0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.576+0000 7fedeaf20640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fedcc0a3410 0x7fedcc0a58d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feddc0058f0 tx=0x7feddc009270 comp rx=0 tx=0).stop 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 -- 172.21.15.62:0/773278600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede410ddc0 msgr2=0x7fede4114b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede410ddc0 0x7fede4114b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feddc00eaa0 tx=0x7feddc00ef60 comp rx=0 tx=0).stop 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 -- 172.21.15.62:0/773278600 shutdown_connections 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fedcc0a3410 0x7fedcc0a58d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fede410ddc0 0x7fede4114b50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fede4109e00 0x7fede41230a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 --2- 172.21.15.62:0/773278600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fede4109450 0x7fede4122b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 -- 172.21.15.62:0/773278600 >> 172.21.15.62:0/773278600 conn(0x7fede406e140 msgr2=0x7fede4072870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 -- 172.21.15.62:0/773278600 shutdown_connections 2024-09-15T23:18:00.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:00.577+0000 7fedeaf20640 1 -- 172.21.15.62:0/773278600 wait complete. 2024-09-15T23:18:01.190 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '3956db1e', quiesce took 4.2 seconds. Will release after: 15.350000000000001 2024-09-15T23:18:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:01 smithi062 ceph-mon[30949]: pgmap v422: 129 pgs: 129 active+clean; 1.0 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 9.1 MiB/s wr, 763 op/s 2024-09-15T23:18:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:01 smithi067 ceph-mon[35665]: pgmap v422: 129 pgs: 129 active+clean; 1.0 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 9.1 MiB/s wr, 763 op/s 2024-09-15T23:18:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:01 smithi148 ceph-mon[35610]: pgmap v422: 129 pgs: 129 active+clean; 1.0 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 9.1 MiB/s wr, 763 op/s 2024-09-15T23:18:02.876 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:18:03.153 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.746+0000 7fdb832dc640 1 -- 172.21.15.62:0/3538581606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 msgr2=0x7fdb7c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.746+0000 7fdb832dc640 1 --2- 172.21.15.62:0/3538581606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 0x7fdb7c1067e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009900 tx=0x7fdb7002a410 comp rx=0 tx=0).stop 2024-09-15T23:18:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.746+0000 7fdb832dc640 1 -- 172.21.15.62:0/3538581606 shutdown_connections 2024-09-15T23:18:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.746+0000 7fdb832dc640 1 --2- 172.21.15.62:0/3538581606 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb7c106f10 0x7fdb7c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.746+0000 7fdb832dc640 1 --2- 172.21.15.62:0/3538581606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 0x7fdb7c1067e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.746+0000 7fdb832dc640 1 --2- 172.21.15.62:0/3538581606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c101de0 0x7fdb7c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.746+0000 7fdb832dc640 1 -- 172.21.15.62:0/3538581606 >> 172.21.15.62:0/3538581606 conn(0x7fdb7c0fd8c0 msgr2=0x7fdb7c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.746+0000 7fdb832dc640 1 -- 172.21.15.62:0/3538581606 shutdown_connections 2024-09-15T23:18:03.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.746+0000 7fdb832dc640 1 -- 172.21.15.62:0/3538581606 wait complete. 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb832dc640 1 Processor -- start 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb832dc640 1 -- start start 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb832dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c101de0 0x7fdb7c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb832dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 0x7fdb7c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb832dc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb7c106f10 0x7fdb7c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb832dc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c111f90 con 0x7fdb7c102790 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb832dc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c111e10 con 0x7fdb7c101de0 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb832dc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c112110 con 0x7fdb7c106f10 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb80877640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 0x7fdb7c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb80877640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 0x7fdb7c1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34984/0 (socket says 172.21.15.62:34984) 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb80877640 1 -- 172.21.15.62:0/2536110093 learned_addr learned my addr 172.21.15.62:0/2536110093 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:03.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb81078640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c101de0 0x7fdb7c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb80877640 1 -- 172.21.15.62:0/2536110093 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb7c106f10 msgr2=0x7fdb7c19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb81879640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb7c106f10 0x7fdb7c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb80877640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb7c106f10 0x7fdb7c19cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb80877640 1 -- 172.21.15.62:0/2536110093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c101de0 msgr2=0x7fdb7c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb80877640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c101de0 0x7fdb7c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb80877640 1 -- 172.21.15.62:0/2536110093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7c19d420 con 0x7fdb7c102790 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.747+0000 7fdb81078640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c101de0 0x7fdb7c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.748+0000 7fdb80877640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 0x7fdb7c1989b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fdb70005ab0 tx=0x7fdb700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.748+0000 7fdb6e7fc640 1 -- 172.21.15.62:0/2536110093 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7003f030 con 0x7fdb7c102790 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.748+0000 7fdb6e7fc640 1 -- 172.21.15.62:0/2536110093 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb70003ce0 con 0x7fdb7c102790 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.748+0000 7fdb832dc640 1 -- 172.21.15.62:0/2536110093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7c19d6b0 con 0x7fdb7c102790 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.748+0000 7fdb6e7fc640 1 -- 172.21.15.62:0/2536110093 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70035070 con 0x7fdb7c102790 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.748+0000 7fdb832dc640 1 -- 172.21.15.62:0/2536110093 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb7c1a4ff0 con 0x7fdb7c102790 2024-09-15T23:18:03.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.750+0000 7fdb6e7fc640 1 -- 172.21.15.62:0/2536110093 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fdb70042030 con 0x7fdb7c102790 2024-09-15T23:18:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb856b5740 1 Processor -- start 2024-09-15T23:18:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb856b5740 1 -- start start 2024-09-15T23:18:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 0x561ac0586880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac0591140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0593a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb856b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561ac04ec2e0 con 0x561ac0586dc0 2024-09-15T23:18:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb856b5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561ac0493ce0 con 0x561ac0591680 2024-09-15T23:18:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb856b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561ac04e82a0 con 0x561ac05864a0 2024-09-15T23:18:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb81078640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac0591140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb81078640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac0591140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34998/0 (socket says 172.21.15.62:34998) 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb81078640 1 -- 172.21.15.62:0/1804866958 learned_addr learned my addr 172.21.15.62:0/1804866958 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb6e7fc640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdb680a3240 0x7fdb680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb80877640 1 --2- 172.21.15.62:0/1804866958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0593a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb81879640 1 --2- 172.21.15.62:0/1804866958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 0x561ac0586880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.751+0000 7fdb81879640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdb680a3240 0x7fdb680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb6e7fc640 1 -- 172.21.15.62:0/2536110093 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdb700f0310 con 0x7fdb7c102790 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb81078640 1 -- 172.21.15.62:0/1804866958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 msgr2=0x561ac0586880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb81078640 1 --2- 172.21.15.62:0/1804866958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 0x561ac0586880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb81078640 1 -- 172.21.15.62:0/1804866958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 msgr2=0x561ac0593a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb81078640 1 --2- 172.21.15.62:0/1804866958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0593a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb81078640 1 -- 172.21.15.62:0/1804866958 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ac0478440 con 0x561ac0586dc0 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb81879640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdb680a3240 0x7fdb680a5700 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdb74014960 tx=0x7fdb74005df0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb81078640 1 --2- 172.21.15.62:0/1804866958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac0591140 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800cca0 tx=0x7fdb7800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8577fed4fd933d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb677fe640 1 -- 172.21.15.62:0/1804866958 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78005220 con 0x561ac0586dc0 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb677fe640 1 -- 172.21.15.62:0/1804866958 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb78004c80 con 0x561ac0586dc0 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.752+0000 7fdb677fe640 1 -- 172.21.15.62:0/1804866958 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78010890 con 0x561ac0586dc0 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 -- 172.21.15.62:0/1804866958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 msgr2=0x561ac0591140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 --2- 172.21.15.62:0/1804866958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac0591140 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800cca0 tx=0x7fdb7800e720 comp rx=0 tx=0).stop 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 -- 172.21.15.62:0/1804866958 shutdown_connections 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 --2- 172.21.15.62:0/1804866958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0593a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 --2- 172.21.15.62:0/1804866958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac0591140 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 --2- 172.21.15.62:0/1804866958 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 0x561ac0586880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 -- 172.21.15.62:0/1804866958 >> 172.21.15.62:0/1804866958 conn(0x561ac0582a30 msgr2=0x561ac0582e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 -- 172.21.15.62:0/1804866958 shutdown_connections 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 -- 172.21.15.62:0/1804866958 wait complete. 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 -- start start 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.753+0000 7fdb856b5740 1 Processor -- start 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac0590f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 0x561ac058e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac058e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb856b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561ac0594250 con 0x561ac05864a0 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb856b5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561ac04a7050 con 0x561ac0591680 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb856b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561ac05943d0 con 0x561ac0586dc0 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb856b5740 10 client.0 _ll_register_callbacks cb 0x561ac04e7160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81078640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac0590f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81078640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac0590f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35012/0 (socket says 172.21.15.62:35012) 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81078640 1 -- 172.21.15.62:0/2402297828 learned_addr learned my addr 172.21.15.62:0/2402297828 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb80877640 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 0x561ac058e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81879640 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac058e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81078640 1 -- 172.21.15.62:0/2402297828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 msgr2=0x561ac058e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81078640 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 0x561ac058e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81078640 1 -- 172.21.15.62:0/2402297828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 msgr2=0x561ac058e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81078640 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac058e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81078640 1 -- 172.21.15.62:0/2402297828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ac025d3c0 con 0x561ac05864a0 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.754+0000 7fdb81879640 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac058e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb81078640 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac0590f20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800ec20 tx=0x7fdb7800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a071eb1cac568cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb64ff9640 1 -- 172.21.15.62:0/2402297828 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7802a060 con 0x561ac05864a0 2024-09-15T23:18:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb856b5740 1 -- 172.21.15.62:0/2402297828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb78004c80 con 0x561ac05864a0 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb64ff9640 1 -- 172.21.15.62:0/2402297828 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdb780048d0 con 0x561ac05864a0 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb64ff9640 1 -- 172.21.15.62:0/2402297828 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdb78031050 con 0x561ac05864a0 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb856b5740 10 client.15849 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb856b5740 1 -- 172.21.15.62:0/2402297828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ac05a3f80 con 0x561ac05864a0 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb64ff9640 1 -- 172.21.15.62:0/2402297828 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb78004c80 con 0x561ac05864a0 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.755+0000 7fdb64ff9640 1 -- 172.21.15.62:0/2402297828 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fdb7801a410 con 0x561ac05864a0 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.756+0000 7fdb856b5740 10 client.15849 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.756+0000 7fdb856b5740 10 client.15849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.756+0000 7fdb856b5740 20 client.15849 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.756+0000 7fdb856b5740 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561ac05a5440 0x561ac05a7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.756+0000 7fdb856b5740 4 client.15849 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.756+0000 7fdb856b5740 1 -- 172.21.15.62:0/2402297828 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ac04f9850 con 0x561ac05a5440 2024-09-15T23:18:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.756+0000 7fdb80877640 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561ac05a5440 0x561ac05a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.756+0000 7fdb80877640 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561ac05a5440 0x561ac05a7830 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11dc0084bddc896b server_cookie=bcdb24003ac7788e in_seq=0 out_seq=0 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.756+0000 7fdb64ff9640 10 client.15849 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb64ff9640 1 -- 172.21.15.62:0/2402297828 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561ac04f9850 con 0x561ac05a5440 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb64ff9640 10 client.15849 handle_command_reply: tid=0 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb856b5740 1 client.15849 shutdown 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb856b5740 1 -- 172.21.15.62:0/2402297828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 msgr2=0x561ac0590f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb856b5740 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac0590f20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800ec20 tx=0x7fdb7800c9e0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb856b5740 1 -- 172.21.15.62:0/2402297828 shutdown_connections 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb856b5740 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561ac05a5440 0x561ac05a7830 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb856b5740 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac058e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb856b5740 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 0x561ac058e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb856b5740 1 --2- 172.21.15.62:0/2402297828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac0590f20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb856b5740 1 -- 172.21.15.62:0/2402297828 >> 172.21.15.62:0/2402297828 conn(0x561ac0582a30 msgr2=0x561ac0581b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:03.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.757+0000 7fdb64ff9640 0 client.15849 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:03.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.758+0000 7fdb856b5740 1 -- 172.21.15.62:0/2402297828 shutdown_connections 2024-09-15T23:18:03.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.758+0000 7fdb856b5740 1 -- 172.21.15.62:0/2402297828 wait complete. 2024-09-15T23:18:03.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.758+0000 7fdb856b5740 20 client.15849 trim_cache size 0 max 16384 2024-09-15T23:18:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.775+0000 7fdb856b5740 1 Processor -- start 2024-09-15T23:18:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.775+0000 7fdb856b5740 1 -- start start 2024-09-15T23:18:03.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.775+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac058dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.775+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 0x561ac0587ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.775+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0581a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.775+0000 7fdb856b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1af3a0 con 0x561ac05864a0 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.775+0000 7fdb856b5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1acc40 con 0x561ac0591680 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.775+0000 7fdb856b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1aaa90 con 0x561ac0586dc0 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81078640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac058dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81078640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac058dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35014/0 (socket says 172.21.15.62:35014) 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81078640 1 -- 172.21.15.62:0/525453517 learned_addr learned my addr 172.21.15.62:0/525453517 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb80877640 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 0x561ac0587ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81879640 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0581a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81078640 1 -- 172.21.15.62:0/525453517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 msgr2=0x561ac0587ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81078640 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 0x561ac0587ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81078640 1 -- 172.21.15.62:0/525453517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 msgr2=0x561ac0581a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81078640 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0581a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81078640 1 -- 172.21.15.62:0/525453517 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ac0594380 con 0x561ac05864a0 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81879640 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0581a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb80877640 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 0x561ac0587ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb81078640 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac058dab0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdb7801a7e0 tx=0x7fdb7800ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652480e3ea835c40 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb66ffd640 1 -- 172.21.15.62:0/525453517 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb780055c0 con 0x561ac05864a0 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb66ffd640 1 -- 172.21.15.62:0/525453517 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb78004db0 con 0x561ac05864a0 2024-09-15T23:18:03.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.776+0000 7fdb66ffd640 1 -- 172.21.15.62:0/525453517 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb78010590 con 0x561ac05864a0 2024-09-15T23:18:03.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 -- 172.21.15.62:0/525453517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 msgr2=0x561ac058dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:03.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac058dab0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdb7801a7e0 tx=0x7fdb7800ef50 comp rx=0 tx=0).stop 2024-09-15T23:18:03.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 -- 172.21.15.62:0/525453517 shutdown_connections 2024-09-15T23:18:03.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0581a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac0586dc0 0x561ac0587ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 --2- 172.21.15.62:0/525453517 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac05864a0 0x561ac058dab0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 -- 172.21.15.62:0/525453517 >> 172.21.15.62:0/525453517 conn(0x561ac0582a30 msgr2=0x561ac0581350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 -- 172.21.15.62:0/525453517 shutdown_connections 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 -- 172.21.15.62:0/525453517 wait complete. 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 -- start start 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.777+0000 7fdb856b5740 1 Processor -- start 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 0x561ac059e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac05995e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb856b5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0599b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb856b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1ab100 con 0x561ac0586dc0 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb856b5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1ac570 con 0x561ac0591680 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb856b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7c1ab7d0 con 0x561ac05864a0 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb856b5740 10 client.0 _ll_register_callbacks cb 0x7fdb78014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb80877640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac05995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb80877640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac05995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35030/0 (socket says 172.21.15.62:35030) 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb80877640 1 -- 172.21.15.62:0/500947288 learned_addr learned my addr 172.21.15.62:0/500947288 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb81078640 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 0x561ac059e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb81879640 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0599b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb80877640 1 -- 172.21.15.62:0/500947288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 msgr2=0x561ac059e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb80877640 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 0x561ac059e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb80877640 1 -- 172.21.15.62:0/500947288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 msgr2=0x561ac0599b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb80877640 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0599b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb80877640 1 -- 172.21.15.62:0/500947288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ac059a350 con 0x561ac0586dc0 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb81078640 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 0x561ac059e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.778+0000 7fdb80877640 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac05995e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdb70000c00 tx=0x7fdb700045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5c2c38c7497770 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb64ff9640 1 -- 172.21.15.62:0/500947288 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70049040 con 0x561ac0586dc0 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb64ff9640 1 -- 172.21.15.62:0/500947288 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdb700edb40 con 0x561ac0586dc0 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb856b5740 1 -- 172.21.15.62:0/500947288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb78004db0 con 0x561ac0586dc0 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb64ff9640 1 -- 172.21.15.62:0/500947288 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdb700bc550 con 0x561ac0586dc0 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb856b5740 10 client.15855 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:03.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb856b5740 1 -- 172.21.15.62:0/500947288 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561ac060fea0 con 0x561ac0586dc0 2024-09-15T23:18:03.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb64ff9640 1 -- 172.21.15.62:0/500947288 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb78004db0 con 0x561ac0586dc0 2024-09-15T23:18:03.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb64ff9640 1 -- 172.21.15.62:0/500947288 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fdb7003b460 con 0x561ac0586dc0 2024-09-15T23:18:03.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb856b5740 10 client.15855 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:03.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb856b5740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:03.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb856b5740 20 client.15855 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:03.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb856b5740 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561ac0610700 0x561ac0612af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb856b5740 4 client.15855 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.779+0000 7fdb856b5740 1 -- 172.21.15.62:0/500947288 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561ac058eaa0 con 0x561ac0610700 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb81078640 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561ac0610700 0x561ac0612af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb64ff9640 10 client.15855 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb81078640 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561ac0610700 0x561ac0612af0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3624e60a889d52e server_cookie=cfdf692269647130 in_seq=0 out_seq=0 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb64ff9640 1 -- 172.21.15.62:0/500947288 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561ac058eaa0 con 0x561ac0610700 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb64ff9640 10 client.15855 handle_command_reply: tid=0 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 client.15855 shutdown 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 -- 172.21.15.62:0/500947288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 msgr2=0x561ac05995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac05995e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdb70000c00 tx=0x7fdb700045e0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 -- 172.21.15.62:0/500947288 shutdown_connections 2024-09-15T23:18:03.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561ac0610700 0x561ac0612af0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac0591680 0x561ac0599b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561ac0586dc0 0x561ac05995e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 --2- 172.21.15.62:0/500947288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561ac05864a0 0x561ac059e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 -- 172.21.15.62:0/500947288 >> 172.21.15.62:0/500947288 conn(0x561ac0582a30 msgr2=0x561ac0581350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:03.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb64ff9640 0 client.15855 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:03.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 -- 172.21.15.62:0/500947288 shutdown_connections 2024-09-15T23:18:03.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.780+0000 7fdb856b5740 1 -- 172.21.15.62:0/500947288 wait complete. 2024-09-15T23:18:03.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb856b5740 20 client.15855 trim_cache size 0 max 16384 2024-09-15T23:18:03.791 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:18:03.791 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:18:03.791 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:18:03.791 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:18:03.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 -- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdb680a3240 msgr2=0x7fdb680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:03.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdb680a3240 0x7fdb680a5700 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdb74014960 tx=0x7fdb74005df0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 -- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 msgr2=0x7fdb7c1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:03.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 0x7fdb7c1989b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fdb70005ab0 tx=0x7fdb700042f0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 -- 172.21.15.62:0/2536110093 shutdown_connections 2024-09-15T23:18:03.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdb680a3240 0x7fdb680a5700 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdb7c106f10 0x7fdb7c19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb7c102790 0x7fdb7c1989b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 --2- 172.21.15.62:0/2536110093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb7c101de0 0x7fdb7c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 -- 172.21.15.62:0/2536110093 >> 172.21.15.62:0/2536110093 conn(0x7fdb7c0fd8c0 msgr2=0x7fdb7c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 -- 172.21.15.62:0/2536110093 shutdown_connections 2024-09-15T23:18:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:03.781+0000 7fdb832dc640 1 -- 172.21.15.62:0/2536110093 wait complete. 2024-09-15T23:18:03.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:03 smithi062 ceph-mon[30949]: pgmap v423: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.1 MiB/s wr, 758 op/s 2024-09-15T23:18:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:03 smithi067 ceph-mon[35665]: pgmap v423: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.1 MiB/s wr, 758 op/s 2024-09-15T23:18:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:03 smithi148 ceph-mon[35610]: pgmap v423: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.1 MiB/s wr, 758 op/s 2024-09-15T23:18:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 -- 172.21.15.62:0/2091634386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 msgr2=0x7f97d4100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 --2- 172.21.15.62:0/2091634386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 0x7f97d4100e60 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009900 tx=0x7f97d002a410 comp rx=0 tx=0).stop 2024-09-15T23:18:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 -- 172.21.15.62:0/2091634386 shutdown_connections 2024-09-15T23:18:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 --2- 172.21.15.62:0/2091634386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97d4102610 0x7f97d4108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 --2- 172.21.15.62:0/2091634386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97d4101c50 0x7f97d41020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 --2- 172.21.15.62:0/2091634386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 0x7f97d4100e60 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 -- 172.21.15.62:0/2091634386 >> 172.21.15.62:0/2091634386 conn(0x7f97d40fc1e0 msgr2=0x7f97d40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 -- 172.21.15.62:0/2091634386 shutdown_connections 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 -- 172.21.15.62:0/2091634386 wait complete. 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 Processor -- start 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.427+0000 7f97dc549640 1 -- start start 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97dc549640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 0x7f97d41049f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97dc549640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97d4101c50 0x7f97d4103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97dc549640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97d4102610 0x7f97d4103570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97dc549640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f97d410c6c0 con 0x7f97d4100a60 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97dc549640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97d410c540 con 0x7f97d4101c50 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97dc549640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f97d410c840 con 0x7f97d4102610 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97da2e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 0x7f97d41049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97da2e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 0x7f97d41049f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35038/0 (socket says 172.21.15.62:35038) 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97da2e5640 1 -- 172.21.15.62:0/4095717512 learned_addr learned my addr 172.21.15.62:0/4095717512 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97d9ae4640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97d4101c50 0x7f97d4103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97daae6640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97d4102610 0x7f97d4103570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97da2e5640 1 -- 172.21.15.62:0/4095717512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97d4102610 msgr2=0x7f97d4103570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97da2e5640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97d4102610 0x7f97d4103570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97da2e5640 1 -- 172.21.15.62:0/4095717512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97d4101c50 msgr2=0x7f97d4103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97da2e5640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97d4101c50 0x7f97d4103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97da2e5640 1 -- 172.21.15.62:0/4095717512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d4103e00 con 0x7f97d4100a60 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97da2e5640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 0x7f97d41049f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f97d0031610 tx=0x7f97d00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97cb7fe640 1 -- 172.21.15.62:0/4095717512 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d003f030 con 0x7f97d4100a60 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97cb7fe640 1 -- 172.21.15.62:0/4095717512 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d0003ce0 con 0x7f97d4100a60 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97dc549640 1 -- 172.21.15.62:0/4095717512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d41a1f00 con 0x7f97d4100a60 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97cb7fe640 1 -- 172.21.15.62:0/4095717512 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0035070 con 0x7f97d4100a60 2024-09-15T23:18:04.432 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.428+0000 7f97dc549640 1 -- 172.21.15.62:0/4095717512 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d41a2310 con 0x7f97d4100a60 2024-09-15T23:18:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.431+0000 7f97cb7fe640 1 -- 172.21.15.62:0/4095717512 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f97d0042030 con 0x7f97d4100a60 2024-09-15T23:18:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.431+0000 7f97df3bd740 1 Processor -- start 2024-09-15T23:18:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.431+0000 7f97df3bd740 1 -- start start 2024-09-15T23:18:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd39a0 0x561d1bbd3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbd8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd9370 0x561d1bbdb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97df3bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561d1bb372f0 con 0x561d1bbd42e0 2024-09-15T23:18:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97df3bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d1badf270 con 0x561d1bbd39a0 2024-09-15T23:18:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97df3bd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561d1bb332b0 con 0x561d1bbd9370 2024-09-15T23:18:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97da2e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbd8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97da2e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbd8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35046/0 (socket says 172.21.15.62:35046) 2024-09-15T23:18:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97da2e5640 1 -- 172.21.15.62:0/1320350310 learned_addr learned my addr 172.21.15.62:0/1320350310 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97d9ae4640 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd9370 0x561d1bbdb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97daae6640 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd39a0 0x561d1bbd3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97da2e5640 1 -- 172.21.15.62:0/1320350310 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd9370 msgr2=0x561d1bbdb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97da2e5640 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd9370 0x561d1bbdb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97cb7fe640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f97c00a32f0 0x7f97c00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97da2e5640 1 -- 172.21.15.62:0/1320350310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd39a0 msgr2=0x561d1bbd3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97da2e5640 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd39a0 0x561d1bbd3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97da2e5640 1 -- 172.21.15.62:0/1320350310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d1bade340 con 0x561d1bbd42e0 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97daae6640 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd39a0 0x561d1bbd3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97daae6640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f97c00a32f0 0x7f97c00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.432+0000 7f97cb7fe640 1 -- 172.21.15.62:0/4095717512 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f97d00efff0 con 0x7f97d4100a60 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.433+0000 7f97da2e5640 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbd8da0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f97d00c1330 tx=0x7f97d00b85f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab7a1b150d2ff47 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.433+0000 7f97c8bfe640 1 -- 172.21.15.62:0/1320350310 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0049070 con 0x561d1bbd42e0 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.433+0000 7f97c8bfe640 1 -- 172.21.15.62:0/1320350310 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d00bc990 con 0x561d1bbd42e0 2024-09-15T23:18:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.433+0000 7f97c8bfe640 1 -- 172.21.15.62:0/1320350310 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d00bc060 con 0x561d1bbd42e0 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.433+0000 7f97df3bd740 1 -- 172.21.15.62:0/1320350310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 msgr2=0x561d1bbd8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.433+0000 7f97daae6640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f97c00a32f0 0x7f97c00a57b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f97cc008e80 tx=0x7f97cc004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.433+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbd8da0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f97d00c1330 tx=0x7f97d00b85f0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.433+0000 7f97df3bd740 1 -- 172.21.15.62:0/1320350310 shutdown_connections 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd9370 0x561d1bbdb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbd8da0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1320350310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd39a0 0x561d1bbd3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 -- 172.21.15.62:0/1320350310 >> 172.21.15.62:0/1320350310 conn(0x561d1bbcf010 msgr2=0x561d1bbcf410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 -- 172.21.15.62:0/1320350310 shutdown_connections 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 -- 172.21.15.62:0/1320350310 wait complete. 2024-09-15T23:18:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 -- start start 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 Processor -- start 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd39a0 0x561d1bbdd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 0x561d1bbddd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd9370 0x561d1bbe4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561d1bb372f0 con 0x561d1bbd9370 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d1bac0290 con 0x561d1bbd42e0 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561d1bbdc100 con 0x561d1bbd39a0 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97df3bd740 10 client.0 _ll_register_callbacks cb 0x561d1baf1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97daae6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd9370 0x561d1bbe4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97daae6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd9370 0x561d1bbe4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35048/0 (socket says 172.21.15.62:35048) 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97daae6640 1 -- 172.21.15.62:0/1379235726 learned_addr learned my addr 172.21.15.62:0/1379235726 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.434+0000 7f97d9ae4640 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 0x561d1bbddd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97da2e5640 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd39a0 0x561d1bbdd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97daae6640 1 -- 172.21.15.62:0/1379235726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd39a0 msgr2=0x561d1bbdd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97daae6640 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd39a0 0x561d1bbdd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97daae6640 1 -- 172.21.15.62:0/1379235726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 msgr2=0x561d1bbddd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97daae6640 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 0x561d1bbddd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97daae6640 1 -- 172.21.15.62:0/1379235726 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d1b84a210 con 0x561d1bbd9370 2024-09-15T23:18:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97daae6640 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd9370 0x561d1bbe4d50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f97cc008cb0 tx=0x7f97cc0084e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817fc1904efcf400 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97bdffb640 1 -- 172.21.15.62:0/1379235726 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc01dd80 con 0x561d1bbd9370 2024-09-15T23:18:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97bdffb640 1 -- 172.21.15.62:0/1379235726 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f97cc01b450 con 0x561d1bbd9370 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97df3bd740 1 -- 172.21.15.62:0/1379235726 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97d00bc990 con 0x561d1bbd9370 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97bdffb640 1 -- 172.21.15.62:0/1379235726 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97cc01b650 con 0x561d1bbd9370 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97df3bd740 10 client.15867 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97df3bd740 1 -- 172.21.15.62:0/1379235726 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561d1b916620 con 0x561d1bbd9370 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.435+0000 7f97bdffb640 1 -- 172.21.15.62:0/1379235726 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97d00bc990 con 0x561d1bbd9370 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97bdffb640 1 -- 172.21.15.62:0/1379235726 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f97cc024070 con 0x561d1bbd9370 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97df3bd740 10 client.15867 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97df3bd740 10 client.15867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97df3bd740 20 client.15867 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d1bbe9530 0x561d1bbeb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97df3bd740 4 client.15867 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97df3bd740 1 -- 172.21.15.62:0/1379235726 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d1bb44a20 con 0x561d1bbe9530 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97da2e5640 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d1bbe9530 0x561d1bbeb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97da2e5640 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d1bbe9530 0x561d1bbeb970 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cd0ad5b7d0a4aef server_cookie=27550576605fa1c0 in_seq=0 out_seq=0 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.436+0000 7f97bdffb640 10 client.15867 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97bdffb640 1 -- 172.21.15.62:0/1379235726 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561d1bb44a20 con 0x561d1bbe9530 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97bdffb640 10 client.15867 handle_command_reply: tid=0 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 client.15867 shutdown 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 -- 172.21.15.62:0/1379235726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd9370 msgr2=0x561d1bbe4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd9370 0x561d1bbe4d50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f97cc008cb0 tx=0x7f97cc0084e0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 -- 172.21.15.62:0/1379235726 shutdown_connections 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d1bbe9530 0x561d1bbeb970 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd9370 0x561d1bbe4d50 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 0x561d1bbddd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97bdffb640 0 client.15867 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 --2- 172.21.15.62:0/1379235726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd39a0 0x561d1bbdd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 -- 172.21.15.62:0/1379235726 >> 172.21.15.62:0/1379235726 conn(0x561d1bbcf010 msgr2=0x561d1bbd0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 -- 172.21.15.62:0/1379235726 shutdown_connections 2024-09-15T23:18:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 1 -- 172.21.15.62:0/1379235726 wait complete. 2024-09-15T23:18:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.437+0000 7f97df3bd740 20 client.15867 trim_cache size 0 max 16384 2024-09-15T23:18:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97df3bd740 1 Processor -- start 2024-09-15T23:18:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97df3bd740 1 -- start start 2024-09-15T23:18:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd39a0 0x561d1bbce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.457 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbe4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd9370 0x561d1bbf0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97df3bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f97d41af270 con 0x561d1bbd42e0 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97df3bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97d41acb10 con 0x561d1bbd9370 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97df3bd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f97d41aa960 con 0x561d1bbd39a0 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97d9ae4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbe4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97d9ae4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbe4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35060/0 (socket says 172.21.15.62:35060) 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97d9ae4640 1 -- 172.21.15.62:0/701545687 learned_addr learned my addr 172.21.15.62:0/701545687 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97daae6640 1 --2- 172.21.15.62:0/701545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd9370 0x561d1bbf0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97d9ae4640 1 -- 172.21.15.62:0/701545687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd39a0 msgr2=0x561d1bbce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97d9ae4640 1 --2- 172.21.15.62:0/701545687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd39a0 0x561d1bbce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97d9ae4640 1 -- 172.21.15.62:0/701545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd9370 msgr2=0x561d1bbf0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97d9ae4640 1 --2- 172.21.15.62:0/701545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd9370 0x561d1bbf0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97d9ae4640 1 -- 172.21.15.62:0/701545687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d1bbcee30 con 0x561d1bbd42e0 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.452+0000 7f97daae6640 1 --2- 172.21.15.62:0/701545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd9370 0x561d1bbf0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97d9ae4640 1 --2- 172.21.15.62:0/701545687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbe4d10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f97c400cc40 tx=0x7f97c4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e8cb314b488c1a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97bffff640 1 -- 172.21.15.62:0/701545687 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c401e070 con 0x561d1bbd42e0 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97bffff640 1 -- 172.21.15.62:0/701545687 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c4005020 con 0x561d1bbd42e0 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97bffff640 1 -- 172.21.15.62:0/701545687 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4004b00 con 0x561d1bbd42e0 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97df3bd740 1 -- 172.21.15.62:0/701545687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 msgr2=0x561d1bbe4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97df3bd740 1 --2- 172.21.15.62:0/701545687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbe4d10 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f97c400cc40 tx=0x7f97c4011810 comp rx=0 tx=0).stop 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97df3bd740 1 -- 172.21.15.62:0/701545687 shutdown_connections 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97df3bd740 1 --2- 172.21.15.62:0/701545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd9370 0x561d1bbf0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97df3bd740 1 --2- 172.21.15.62:0/701545687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd42e0 0x561d1bbe4d10 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97df3bd740 1 --2- 172.21.15.62:0/701545687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd39a0 0x561d1bbce0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97df3bd740 1 -- 172.21.15.62:0/701545687 >> 172.21.15.62:0/701545687 conn(0x561d1bbcf010 msgr2=0x561d1bbcd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:04.459 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97df3bd740 1 -- 172.21.15.62:0/701545687 shutdown_connections 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.453+0000 7f97df3bd740 1 -- 172.21.15.62:0/701545687 wait complete. 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97df3bd740 1 -- start start 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97df3bd740 1 Processor -- start 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd39a0 0x561d1bbe2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 0x561d1bbe2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97df3bd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd9370 0x561d1bbe3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97df3bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f97d41aafd0 con 0x561d1bbd39a0 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97df3bd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97d41ac440 con 0x561d1bbd42e0 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97df3bd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f97d41ab6a0 con 0x561d1bbd9370 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97df3bd740 10 client.0 _ll_register_callbacks cb 0x7f97cc010e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97da2e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd39a0 0x561d1bbe2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97da2e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd39a0 0x561d1bbe2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35072/0 (socket says 172.21.15.62:35072) 2024-09-15T23:18:04.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97da2e5640 1 -- 172.21.15.62:0/2896870172 learned_addr learned my addr 172.21.15.62:0/2896870172 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97d9ae4640 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 0x561d1bbe2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97da2e5640 1 -- 172.21.15.62:0/2896870172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd9370 msgr2=0x561d1bbe3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97da2e5640 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd9370 0x561d1bbe3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97da2e5640 1 -- 172.21.15.62:0/2896870172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 msgr2=0x561d1bbe2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97da2e5640 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 0x561d1bbe2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97da2e5640 1 -- 172.21.15.62:0/2896870172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d1bbe41d0 con 0x561d1bbd39a0 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97da2e5640 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd39a0 0x561d1bbe2670 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f97d00edae0 tx=0x7f97d00f35e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b9889840899cc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.454+0000 7f97bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97bdffb640 1 -- 172.21.15.62:0/2896870172 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0049040 con 0x561d1bbd39a0 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97bdffb640 1 -- 172.21.15.62:0/2896870172 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f97d00ba590 con 0x561d1bbd39a0 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97df3bd740 1 -- 172.21.15.62:0/2896870172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97c4005020 con 0x561d1bbd39a0 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97bdffb640 1 -- 172.21.15.62:0/2896870172 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f97d00bbbf0 con 0x561d1bbd39a0 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97df3bd740 10 client.15873 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97df3bd740 1 -- 172.21.15.62:0/2896870172 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561d1bc54440 con 0x561d1bbd39a0 2024-09-15T23:18:04.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97bdffb640 1 -- 172.21.15.62:0/2896870172 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c4005020 con 0x561d1bbd39a0 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97bdffb640 1 -- 172.21.15.62:0/2896870172 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f97d00b8900 con 0x561d1bbd39a0 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97df3bd740 10 client.15873 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97df3bd740 10 client.15873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97df3bd740 20 client.15873 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97df3bd740 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d1bc54ce0 0x561d1bc57100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97df3bd740 4 client.15873 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97df3bd740 1 -- 172.21.15.62:0/2896870172 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d1bb3ed70 con 0x561d1bc54ce0 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.455+0000 7f97d9ae4640 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d1bc54ce0 0x561d1bc57100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.456+0000 7f97bdffb640 10 client.15873 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.456+0000 7f97d9ae4640 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d1bc54ce0 0x561d1bc57100 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e515a3224acad52 server_cookie=5c86caa038a81967 in_seq=0 out_seq=0 2024-09-15T23:18:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97bdffb640 1 -- 172.21.15.62:0/2896870172 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d1bb3ed70 con 0x561d1bc54ce0 2024-09-15T23:18:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97bdffb640 10 client.15873 handle_command_reply: tid=0 2024-09-15T23:18:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 client.15873 shutdown 2024-09-15T23:18:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 -- 172.21.15.62:0/2896870172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd39a0 msgr2=0x561d1bbe2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd39a0 0x561d1bbe2670 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f97d00edae0 tx=0x7f97d00f35e0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 -- 172.21.15.62:0/2896870172 shutdown_connections 2024-09-15T23:18:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d1bc54ce0 0x561d1bc57100 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d1bbd9370 0x561d1bbe3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d1bbd42e0 0x561d1bbe2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 --2- 172.21.15.62:0/2896870172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d1bbd39a0 0x561d1bbe2670 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97bdffb640 0 client.15873 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 -- 172.21.15.62:0/2896870172 >> 172.21.15.62:0/2896870172 conn(0x561d1bbcf010 msgr2=0x561d1bbce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 -- 172.21.15.62:0/2896870172 shutdown_connections 2024-09-15T23:18:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 1 -- 172.21.15.62:0/2896870172 wait complete. 2024-09-15T23:18:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.470+0000 7f97df3bd740 20 client.15873 trim_cache size 0 max 16384 2024-09-15T23:18:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:18:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:18:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:18:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 -- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f97c00a32f0 msgr2=0x7f97c00a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f97c00a32f0 0x7f97c00a57b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f97cc008e80 tx=0x7f97cc004a40 comp rx=0 tx=0).stop 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 -- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 msgr2=0x7f97d41049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 0x7f97d41049f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f97d0031610 tx=0x7f97d00043b0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 -- 172.21.15.62:0/4095717512 shutdown_connections 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f97c00a32f0 0x7f97c00a57b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f97d4102610 0x7f97d4103570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97d4101c50 0x7f97d4103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 --2- 172.21.15.62:0/4095717512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f97d4100a60 0x7f97d41049f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 -- 172.21.15.62:0/4095717512 >> 172.21.15.62:0/4095717512 conn(0x7f97d40fc1e0 msgr2=0x7f97d40fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 -- 172.21.15.62:0/4095717512 shutdown_connections 2024-09-15T23:18:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:04.471+0000 7f97dc549640 1 -- 172.21.15.62:0/4095717512 wait complete. 2024-09-15T23:18:04.573 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 105.0 s, 388.8 tps, lat 51.075 ms stddev 57.447 2024-09-15T23:18:04.636 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:18:04.637 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:18:04.935 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:05.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:05 smithi062 ceph-mon[30949]: pgmap v424: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 704 op/s 2024-09-15T23:18:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:05 smithi067 ceph-mon[35665]: pgmap v424: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 704 op/s 2024-09-15T23:18:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:05 smithi148 ceph-mon[35610]: pgmap v424: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.0 MiB/s wr, 704 op/s 2024-09-15T23:18:06.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 -- 172.21.15.62:0/2355295607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 msgr2=0x7f020c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:06.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 --2- 172.21.15.62:0/2355295607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 0x7f020c10da40 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0204009900 tx=0x7f020402a480 comp rx=0 tx=0).stop 2024-09-15T23:18:06.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 -- 172.21.15.62:0/2355295607 shutdown_connections 2024-09-15T23:18:06.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 --2- 172.21.15.62:0/2355295607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 0x7f020c10da40 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:06.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 --2- 172.21.15.62:0/2355295607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f020c102570 0x7f020c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 --2- 172.21.15.62:0/2355295607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f020c100e20 0x7f020c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 -- 172.21.15.62:0/2355295607 >> 172.21.15.62:0/2355295607 conn(0x7f020c0fc7d0 msgr2=0x7f020c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 -- 172.21.15.62:0/2355295607 shutdown_connections 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 -- 172.21.15.62:0/2355295607 wait complete. 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.832+0000 7f0212cd5640 1 Processor -- start 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0212cd5640 1 -- start start 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0212cd5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f020c100e20 0x7f020c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0212cd5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f020c102570 0x7f020c19ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0212cd5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 0x7f020c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0212cd5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f020c110f80 con 0x7f020c102ef0 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0212cd5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f020c110e00 con 0x7f020c102570 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0212cd5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f020c111100 con 0x7f020c100e20 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0211272640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 0x7f020c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:06.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0211272640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 0x7f020c19efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35092/0 (socket says 172.21.15.62:35092) 2024-09-15T23:18:06.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0211272640 1 -- 172.21.15.62:0/1412403273 learned_addr learned my addr 172.21.15.62:0/1412403273 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:06.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f020bfff640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f020c102570 0x7f020c19ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:06.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0210a71640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f020c100e20 0x7f020c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:06.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0211272640 1 -- 172.21.15.62:0/1412403273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f020c100e20 msgr2=0x7f020c19a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:06.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0211272640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f020c100e20 0x7f020c19a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:06.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0211272640 1 -- 172.21.15.62:0/1412403273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f020c102570 msgr2=0x7f020c19ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:06.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0211272640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f020c102570 0x7f020c19ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:06.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.833+0000 7f0211272640 1 -- 172.21.15.62:0/1412403273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020c19f6b0 con 0x7f020c102ef0 2024-09-15T23:18:06.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.834+0000 7f0211272640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 0x7f020c19efd0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0204004770 tx=0x7f0204003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:06.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.834+0000 7f0209ffb640 1 -- 172.21.15.62:0/1412403273 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020403f070 con 0x7f020c102ef0 2024-09-15T23:18:06.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.834+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020c19f940 con 0x7f020c102ef0 2024-09-15T23:18:06.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.834+0000 7f0209ffb640 1 -- 172.21.15.62:0/1412403273 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0204031c30 con 0x7f020c102ef0 2024-09-15T23:18:06.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.834+0000 7f0209ffb640 1 -- 172.21.15.62:0/1412403273 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0204035030 con 0x7f020c102ef0 2024-09-15T23:18:06.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.834+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f020c1a7280 con 0x7f020c102ef0 2024-09-15T23:18:06.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.835+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f020c1021b0 con 0x7f020c102ef0 2024-09-15T23:18:06.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.836+0000 7f0209ffb640 1 -- 172.21.15.62:0/1412403273 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0204042070 con 0x7f020c102ef0 2024-09-15T23:18:06.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.837+0000 7f0209ffb640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f01f80a3240 0x7f01f80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:06.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.837+0000 7f0210a71640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f01f80a3240 0x7f01f80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:06.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.837+0000 7f0209ffb640 1 -- 172.21.15.62:0/1412403273 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f02040f0cc0 con 0x7f020c102ef0 2024-09-15T23:18:06.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.838+0000 7f0210a71640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f01f80a3240 0x7f01f80a5700 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f01f4005fd0 tx=0x7f01f4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:06.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:06.840+0000 7f0209ffb640 1 -- 172.21.15.62:0/1412403273 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f02040bd600 con 0x7f020c102ef0 2024-09-15T23:18:07.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.169+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f020c101200 con 0x7f020c102ef0 2024-09-15T23:18:07.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.172+0000 7f0209ffb640 1 -- 172.21.15.62:0/1412403273 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11907 (secure 0 0 0) 0x7f020c101200 con 0x7f020c102ef0 2024-09-15T23:18:07.174 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:18:07.174 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":28,"btime":"2024-09-15T23:17:33:092139+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:17:32.873645+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:18:07.175 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 28 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f01f80a3240 msgr2=0x7f01f80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f01f80a3240 0x7f01f80a5700 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f01f4005fd0 tx=0x7f01f4005850 comp rx=0 tx=0).stop 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 msgr2=0x7f020c19efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 0x7f020c19efd0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0204004770 tx=0x7f0204003c10 comp rx=0 tx=0).stop 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 shutdown_connections 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f01f80a3240 0x7f01f80a5700 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f020c102ef0 0x7f020c19efd0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f020c102570 0x7f020c19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 --2- 172.21.15.62:0/1412403273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f020c100e20 0x7f020c19a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:07.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.176+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 >> 172.21.15.62:0/1412403273 conn(0x7f020c0fc7d0 msgr2=0x7f020c06a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:07.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.178+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 shutdown_connections 2024-09-15T23:18:07.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:07.178+0000 7f0212cd5640 1 -- 172.21.15.62:0/1412403273 wait complete. 2024-09-15T23:18:07.944 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:07 smithi062 ceph-mon[30949]: pgmap v425: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.7 MiB/s wr, 557 op/s 2024-09-15T23:18:07.944 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:07 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1412403273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:07.972 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:18:08.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:07 smithi067 ceph-mon[35665]: pgmap v425: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.7 MiB/s wr, 557 op/s 2024-09-15T23:18:08.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:07 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1412403273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:07 smithi148 ceph-mon[35610]: pgmap v425: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.7 MiB/s wr, 557 op/s 2024-09-15T23:18:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:07 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1412403273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:08.228 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:09.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.072+0000 7f1271f93640 1 -- 172.21.15.62:0/144024957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 msgr2=0x7f126c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.072+0000 7f1271f93640 1 --2- 172.21.15.62:0/144024957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 0x7f126c1021c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f125c009ca0 tx=0x7f125c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.072+0000 7f1271f93640 1 -- 172.21.15.62:0/144024957 shutdown_connections 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.072+0000 7f1271f93640 1 --2- 172.21.15.62:0/144024957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f126c106f10 0x7f126c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.072+0000 7f1271f93640 1 --2- 172.21.15.62:0/144024957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f126c102790 0x7f126c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.072+0000 7f1271f93640 1 --2- 172.21.15.62:0/144024957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 0x7f126c1021c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.072+0000 7f1271f93640 1 -- 172.21.15.62:0/144024957 >> 172.21.15.62:0/144024957 conn(0x7f126c0fd8c0 msgr2=0x7f126c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.072+0000 7f1271f93640 1 -- 172.21.15.62:0/144024957 shutdown_connections 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.072+0000 7f1271f93640 1 -- 172.21.15.62:0/144024957 wait complete. 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f1271f93640 1 Processor -- start 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f1271f93640 1 -- start start 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f1271f93640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 0x7f126c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f1271f93640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f126c102790 0x7f126c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f1271f93640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f126c106f10 0x7f126c19cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f1271f93640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f126c111f70 con 0x7f126c101de0 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f1271f93640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f126c111df0 con 0x7f126c106f10 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f1271f93640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1120f0 con 0x7f126c102790 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f126b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 0x7f126c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f126b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 0x7f126c198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35108/0 (socket says 172.21.15.62:35108) 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f126b7fe640 1 -- 172.21.15.62:0/2706010209 learned_addr learned my addr 172.21.15.62:0/2706010209 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f126affd640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f126c102790 0x7f126c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.073+0000 7f126bfff640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f126c106f10 0x7f126c19cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f126b7fe640 1 -- 172.21.15.62:0/2706010209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f126c102790 msgr2=0x7f126c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f126b7fe640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f126c102790 0x7f126c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f126b7fe640 1 -- 172.21.15.62:0/2706010209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f126c106f10 msgr2=0x7f126c19cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f126b7fe640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f126c106f10 0x7f126c19cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f126b7fe640 1 -- 172.21.15.62:0/2706010209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c19d400 con 0x7f126c101de0 2024-09-15T23:18:09.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f126b7fe640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 0x7f126c198450 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f125c02ade0 tx=0x7f125c0319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f1268ff9640 1 -- 172.21.15.62:0/2706010209 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c02fa20 con 0x7f126c101de0 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f1268ff9640 1 -- 172.21.15.62:0/2706010209 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125c02fbc0 con 0x7f126c101de0 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f1271f93640 1 -- 172.21.15.62:0/2706010209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f126c19d690 con 0x7f126c101de0 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f1268ff9640 1 -- 172.21.15.62:0/2706010209 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c0466e0 con 0x7f126c101de0 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.074+0000 7f1271f93640 1 -- 172.21.15.62:0/2706010209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f126c1a4fd0 con 0x7f126c101de0 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f1268ff9640 1 -- 172.21.15.62:0/2706010209 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f125c046940 con 0x7f126c101de0 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f127436c740 1 Processor -- start 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f127436c740 1 -- start start 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f127436c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 0x557331173090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f127436c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 0x557331173a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f127436c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x557331180760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f127436c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5573310d92f0 con 0x55733117e370 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f127436c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557331080cf0 con 0x5573311735d0 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f127436c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5573310d52b0 con 0x557331174b40 2024-09-15T23:18:09.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x557331180760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x557331180760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35118/0 (socket says 172.21.15.62:35118) 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126affd640 1 -- 172.21.15.62:0/970715923 learned_addr learned my addr 172.21.15.62:0/970715923 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126bfff640 1 --2- 172.21.15.62:0/970715923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 0x557331173090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126b7fe640 1 --2- 172.21.15.62:0/970715923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 0x557331173a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126affd640 1 -- 172.21.15.62:0/970715923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 msgr2=0x557331173090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126affd640 1 --2- 172.21.15.62:0/970715923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 0x557331173090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126affd640 1 -- 172.21.15.62:0/970715923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 msgr2=0x557331173a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126affd640 1 --2- 172.21.15.62:0/970715923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 0x557331173a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.077+0000 7f126affd640 1 -- 172.21.15.62:0/970715923 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573310653a0 con 0x55733117e370 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.078+0000 7f126affd640 1 --2- 172.21.15.62:0/970715923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x557331180760 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f125400bcb0 tx=0x7f12540107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff042544d6624fe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.078+0000 7f1268ff9640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12580a3240 0x7f12580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.078+0000 7f1261ffb640 1 -- 172.21.15.62:0/970715923 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254004030 con 0x55733117e370 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.078+0000 7f1261ffb640 1 -- 172.21.15.62:0/970715923 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12540041d0 con 0x55733117e370 2024-09-15T23:18:09.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.078+0000 7f1261ffb640 1 -- 172.21.15.62:0/970715923 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254011840 con 0x55733117e370 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.078+0000 7f126bfff640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12580a3240 0x7f12580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.078+0000 7f1268ff9640 1 -- 172.21.15.62:0/2706010209 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f125c0f1070 con 0x7f126c101de0 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 -- 172.21.15.62:0/970715923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 msgr2=0x557331180760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f126bfff640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12580a3240 0x7f12580a5700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1264009c70 tx=0x7f1264005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 --2- 172.21.15.62:0/970715923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x557331180760 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f125400bcb0 tx=0x7f12540107a0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 -- 172.21.15.62:0/970715923 shutdown_connections 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 --2- 172.21.15.62:0/970715923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x557331180760 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 --2- 172.21.15.62:0/970715923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 0x557331173a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 --2- 172.21.15.62:0/970715923 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 0x557331173090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 -- 172.21.15.62:0/970715923 >> 172.21.15.62:0/970715923 conn(0x55733113c310 msgr2=0x55733116fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 -- 172.21.15.62:0/970715923 shutdown_connections 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 -- 172.21.15.62:0/970715923 wait complete. 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 -- start start 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.079+0000 7f127436c740 1 Processor -- start 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f127436c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 0x55733117dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f127436c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 0x55733117afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f127436c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x55733117b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f127436c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557331181070 con 0x55733117e370 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f127436c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557331094060 con 0x5573311735d0 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f127436c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5573311811f0 con 0x557331174b40 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f127436c740 10 client.0 _ll_register_callbacks cb 0x557330de78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x55733117b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x55733117b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35134/0 (socket says 172.21.15.62:35134) 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 0x55733117dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126bfff640 1 -- 172.21.15.62:0/4252527557 learned_addr learned my addr 172.21.15.62:0/4252527557 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126affd640 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 0x55733117afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126bfff640 1 -- 172.21.15.62:0/4252527557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 msgr2=0x55733117afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126bfff640 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 0x55733117afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126bfff640 1 -- 172.21.15.62:0/4252527557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 msgr2=0x55733117dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126bfff640 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 0x55733117dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126bfff640 1 -- 172.21.15.62:0/4252527557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557330e4a3c0 con 0x55733117e370 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f126bfff640 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x55733117b5d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1264004310 tx=0x7f1264004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637d7aa9780bdfcb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.080+0000 7f124affd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f124affd640 1 -- 172.21.15.62:0/4252527557 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1264029440 con 0x55733117e370 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f124affd640 1 -- 172.21.15.62:0/4252527557 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f126401f430 con 0x55733117e370 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f127436c740 1 -- 172.21.15.62:0/4252527557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12540041d0 con 0x55733117e370 2024-09-15T23:18:09.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f124affd640 1 -- 172.21.15.62:0/4252527557 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f126401f630 con 0x55733117e370 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f127436c740 10 client.15891 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f127436c740 1 -- 172.21.15.62:0/4252527557 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55733118cb30 con 0x55733117e370 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f124affd640 1 -- 172.21.15.62:0/4252527557 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12540041d0 con 0x55733117e370 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f124affd640 1 -- 172.21.15.62:0/4252527557 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f1264027070 con 0x55733117e370 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f127436c740 10 client.15891 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f127436c740 10 client.15891 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f127436c740 20 client.15891 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f127436c740 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55733118e040 0x557331190490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f127436c740 4 client.15891 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.081+0000 7f127436c740 1 -- 172.21.15.62:0/4252527557 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573310e6860 con 0x55733118e040 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.082+0000 7f126b7fe640 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55733118e040 0x557331190490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.082+0000 7f126b7fe640 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55733118e040 0x557331190490 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=797ebd4df983676a server_cookie=263cd0752c40cb57 in_seq=0 out_seq=0 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.082+0000 7f124affd640 10 client.15891 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:18:09.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f124affd640 1 -- 172.21.15.62:0/4252527557 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5573310e6860 con 0x55733118e040 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f124affd640 10 client.15891 handle_command_reply: tid=0 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 client.15891 shutdown 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 -- 172.21.15.62:0/4252527557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 msgr2=0x55733117b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x55733117b5d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1264004310 tx=0x7f1264004730 comp rx=0 tx=0).stop 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 -- 172.21.15.62:0/4252527557 shutdown_connections 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55733118e040 0x557331190490 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f124affd640 0 client.15891 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55733117e370 0x55733117b5d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557331174b40 0x55733117afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 --2- 172.21.15.62:0/4252527557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5573311735d0 0x55733117dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 -- 172.21.15.62:0/4252527557 >> 172.21.15.62:0/4252527557 conn(0x55733113c310 msgr2=0x557331175960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 -- 172.21.15.62:0/4252527557 shutdown_connections 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 1 -- 172.21.15.62:0/4252527557 wait complete. 2024-09-15T23:18:09.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.083+0000 7f127436c740 20 client.15891 trim_cache size 0 max 16384 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.096+0000 7f127436c740 1 Processor -- start 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.096+0000 7f127436c740 1 -- start start 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f127436c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573311735d0 0x55733117cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f127436c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557331174b40 0x557331176120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f127436c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55733117e370 0x557331176660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f127436c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1b14e0 con 0x5573311735d0 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f127436c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1aed80 con 0x557331174b40 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f127436c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1acbd0 con 0x55733117e370 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573311735d0 0x55733117cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573311735d0 0x55733117cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35150/0 (socket says 172.21.15.62:35150) 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126b7fe640 1 -- 172.21.15.62:0/3935565124 learned_addr learned my addr 172.21.15.62:0/3935565124 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:09.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126affd640 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557331174b40 0x557331176120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126bfff640 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55733117e370 0x557331176660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126b7fe640 1 -- 172.21.15.62:0/3935565124 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55733117e370 msgr2=0x557331176660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:09.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126b7fe640 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55733117e370 0x557331176660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126b7fe640 1 -- 172.21.15.62:0/3935565124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557331174b40 msgr2=0x557331176120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:09.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126b7fe640 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557331174b40 0x557331176120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126b7fe640 1 -- 172.21.15.62:0/3935565124 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573311811e0 con 0x5573311735d0 2024-09-15T23:18:09.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126bfff640 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55733117e370 0x557331176660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:09.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126affd640 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557331174b40 0x557331176120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:09.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f126b7fe640 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573311735d0 0x55733117cf50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f125c000f80 tx=0x7f125c0ee6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b7aee113997917 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:09.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f12617fa640 1 -- 172.21.15.62:0/3935565124 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c0bd0e0 con 0x5573311735d0 2024-09-15T23:18:09.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f12617fa640 1 -- 172.21.15.62:0/3935565124 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125c036040 con 0x5573311735d0 2024-09-15T23:18:09.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.097+0000 7f12617fa640 1 -- 172.21.15.62:0/3935565124 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c0bcc70 con 0x5573311735d0 2024-09-15T23:18:09.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 -- 172.21.15.62:0/3935565124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573311735d0 msgr2=0x55733117cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573311735d0 0x55733117cf50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f125c000f80 tx=0x7f125c0ee6c0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 -- 172.21.15.62:0/3935565124 shutdown_connections 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55733117e370 0x557331176660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557331174b40 0x557331176120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 --2- 172.21.15.62:0/3935565124 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5573311735d0 0x55733117cf50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 -- 172.21.15.62:0/3935565124 >> 172.21.15.62:0/3935565124 conn(0x55733113c310 msgr2=0x55733117c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 -- 172.21.15.62:0/3935565124 shutdown_connections 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 -- 172.21.15.62:0/3935565124 wait complete. 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.098+0000 7f127436c740 1 -- start start 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f127436c740 1 Processor -- start 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f127436c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5573311735d0 0x557331186d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f127436c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557331174b40 0x557331182080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f127436c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55733117e370 0x557331182810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f127436c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1ad240 con 0x557331174b40 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f127436c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1ae6b0 con 0x55733117e370 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f127436c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f126c1ad910 con 0x5573311735d0 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f127436c740 10 client.0 _ll_register_callbacks cb 0x7f12640098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f126affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557331174b40 0x557331182080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f126affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557331174b40 0x557331182080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35152/0 (socket says 172.21.15.62:35152) 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f126affd640 1 -- 172.21.15.62:0/441256488 learned_addr learned my addr 172.21.15.62:0/441256488 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f126b7fe640 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5573311735d0 0x557331186d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.099+0000 7f126bfff640 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55733117e370 0x557331182810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f126affd640 1 -- 172.21.15.62:0/441256488 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5573311735d0 msgr2=0x557331186d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f126affd640 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5573311735d0 0x557331186d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f126affd640 1 -- 172.21.15.62:0/441256488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55733117e370 msgr2=0x557331182810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:09.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f126affd640 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55733117e370 0x557331182810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f126affd640 1 -- 172.21.15.62:0/441256488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573311fc140 con 0x557331174b40 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f126affd640 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557331174b40 0x557331182080 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x557331174070 tx=0x7f125400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ae6dc5709a83c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f124affd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f124affd640 1 -- 172.21.15.62:0/441256488 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1254005450 con 0x557331174b40 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f127436c740 1 -- 172.21.15.62:0/441256488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f125c036040 con 0x557331174b40 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f124affd640 1 -- 172.21.15.62:0/441256488 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f12540055f0 con 0x557331174b40 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f124affd640 1 -- 172.21.15.62:0/441256488 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1254034050 con 0x557331174b40 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f127436c740 10 client.15897 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f127436c740 1 -- 172.21.15.62:0/441256488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5573311fd300 con 0x557331174b40 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.100+0000 7f124affd640 1 -- 172.21.15.62:0/441256488 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f125c036040 con 0x557331174b40 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f124affd640 1 -- 172.21.15.62:0/441256488 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f12540119d0 con 0x557331174b40 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f127436c740 10 client.15897 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f127436c740 10 client.15897 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:18:09.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f127436c740 20 client.15897 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f127436c740 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55733118e040 0x5573311ff830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f127436c740 4 client.15897 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f127436c740 1 -- 172.21.15.62:0/441256488 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5573310e0d70 con 0x55733118e040 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f126b7fe640 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55733118e040 0x5573311ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f124affd640 10 client.15897 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.101+0000 7f126b7fe640 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55733118e040 0x5573311ff830 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9200481bd8fd8d66 server_cookie=e6da76ec0604e327 in_seq=0 out_seq=0 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f124affd640 1 -- 172.21.15.62:0/441256488 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5573310e0d70 con 0x55733118e040 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f124affd640 10 client.15897 handle_command_reply: tid=0 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 client.15897 shutdown 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 -- 172.21.15.62:0/441256488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557331174b40 msgr2=0x557331182080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557331174b40 0x557331182080 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x557331174070 tx=0x7f125400bfd0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 -- 172.21.15.62:0/441256488 shutdown_connections 2024-09-15T23:18:09.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55733118e040 0x5573311ff830 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55733117e370 0x557331182810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557331174b40 0x557331182080 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 --2- 172.21.15.62:0/441256488 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5573311735d0 0x557331186d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f124affd640 0 client.15897 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:18:09.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 -- 172.21.15.62:0/441256488 >> 172.21.15.62:0/441256488 conn(0x55733113c310 msgr2=0x55733117d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:09.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 -- 172.21.15.62:0/441256488 shutdown_connections 2024-09-15T23:18:09.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 1 -- 172.21.15.62:0/441256488 wait complete. 2024-09-15T23:18:09.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.102+0000 7f127436c740 20 client.15897 trim_cache size 0 max 16384 2024-09-15T23:18:09.109 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:18:09.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 -- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12580a3240 msgr2=0x7f12580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:09.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12580a3240 0x7f12580a5700 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1264009c70 tx=0x7f1264005f90 comp rx=0 tx=0).stop 2024-09-15T23:18:09.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 -- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 msgr2=0x7f126c198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:09.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 0x7f126c198450 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f125c02ade0 tx=0x7f125c0319f0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 -- 172.21.15.62:0/2706010209 shutdown_connections 2024-09-15T23:18:09.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12580a3240 0x7f12580a5700 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f126c106f10 0x7f126c19cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f126c102790 0x7f126c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 --2- 172.21.15.62:0/2706010209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f126c101de0 0x7f126c198450 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:09.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 -- 172.21.15.62:0/2706010209 >> 172.21.15.62:0/2706010209 conn(0x7f126c0fd8c0 msgr2=0x7f126c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:09.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 -- 172.21.15.62:0/2706010209 shutdown_connections 2024-09-15T23:18:09.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:09.103+0000 7f1271f93640 1 -- 172.21.15.62:0/2706010209 wait complete. 2024-09-15T23:18:09.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 110.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:18:09.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:09 smithi062 ceph-mon[30949]: pgmap v426: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.1 MiB/s wr, 526 op/s 2024-09-15T23:18:10.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:09 smithi067 ceph-mon[35665]: pgmap v426: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.1 MiB/s wr, 526 op/s 2024-09-15T23:18:10.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:09 smithi148 ceph-mon[35610]: pgmap v426: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 8.1 MiB/s wr, 526 op/s 2024-09-15T23:18:10.128 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:18:10.387 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:11.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.112+0000 7f7cf74b9640 1 -- 172.21.15.62:0/1004876925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 msgr2=0x7f7cf010e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:11.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.112+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/1004876925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 0x7f7cf010e5f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009ca0 tx=0x7f7ce802a950 comp rx=0 tx=0).stop 2024-09-15T23:18:11.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.112+0000 7f7cf74b9640 1 -- 172.21.15.62:0/1004876925 shutdown_connections 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.112+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/1004876925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 0x7f7cf010e5f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.112+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/1004876925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf01036b0 0x7f7cf010bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.112+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/1004876925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf0102d90 0x7f7cf0103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.112+0000 7f7cf74b9640 1 -- 172.21.15.62:0/1004876925 >> 172.21.15.62:0/1004876925 conn(0x7f7cf00fc790 msgr2=0x7f7cf00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.112+0000 7f7cf74b9640 1 -- 172.21.15.62:0/1004876925 shutdown_connections 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.112+0000 7f7cf74b9640 1 -- 172.21.15.62:0/1004876925 wait complete. 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf74b9640 1 Processor -- start 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf74b9640 1 -- start start 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf74b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf0102d90 0x7f7cf0198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf74b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf01036b0 0x7f7cf01988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf74b9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 0x7f7cf019cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf74b9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf0111f30 con 0x7f7cf010c170 2024-09-15T23:18:11.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf74b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf0111db0 con 0x7f7cf0102d90 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf74b9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf01120b0 con 0x7f7cf01036b0 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5a56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 0x7f7cf019cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5a56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 0x7f7cf019cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35168/0 (socket says 172.21.15.62:35168) 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5a56640 1 -- 172.21.15.62:0/220127243 learned_addr learned my addr 172.21.15.62:0/220127243 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5255640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf0102d90 0x7f7cf0198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf01036b0 0x7f7cf01988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5a56640 1 -- 172.21.15.62:0/220127243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf01036b0 msgr2=0x7f7cf01988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5a56640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf01036b0 0x7f7cf01988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5a56640 1 -- 172.21.15.62:0/220127243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf0102d90 msgr2=0x7f7cf0198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5a56640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf0102d90 0x7f7cf0198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5a56640 1 -- 172.21.15.62:0/220127243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf019d340 con 0x7f7cf010c170 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.113+0000 7f7cf5255640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf0102d90 0x7f7cf0198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:11.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.114+0000 7f7cf5a56640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 0x7f7cf019cc60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7ce802ae70 tx=0x7f7ce802f5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.114+0000 7f7ce67fc640 1 -- 172.21.15.62:0/220127243 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8034040 con 0x7f7cf010c170 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.114+0000 7f7ce67fc640 1 -- 172.21.15.62:0/220127243 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ce8030920 con 0x7f7cf010c170 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.114+0000 7f7cf74b9640 1 -- 172.21.15.62:0/220127243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf019d5d0 con 0x7f7cf010c170 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.114+0000 7f7ce67fc640 1 -- 172.21.15.62:0/220127243 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8045480 con 0x7f7cf010c170 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.114+0000 7f7cf74b9640 1 -- 172.21.15.62:0/220127243 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf01a4f10 con 0x7f7cf010c170 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.116+0000 7f7ce67fc640 1 -- 172.21.15.62:0/220127243 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7ce8037020 con 0x7f7cf010c170 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.116+0000 7f7cf9892740 1 Processor -- start 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf9892740 1 -- start start 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e840485b40 0x55e840484090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8404845d0 0x55e840484a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e84048f370 0x55e840491760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf9892740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e8403ea2f0 con 0x55e84048f370 2024-09-15T23:18:11.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf9892740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e840391cf0 con 0x55e8404845d0 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf9892740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e8403e62b0 con 0x55e840485b40 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf4a54640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e84048f370 0x55e840491760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7ce67fc640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7cdc0a3240 0x7f7cdc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf4a54640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e84048f370 0x55e840491760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35172/0 (socket says 172.21.15.62:35172) 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2933882447 learned_addr learned my addr 172.21.15.62:0/2933882447 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf5255640 1 --2- 172.21.15.62:0/2933882447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8404845d0 0x55e840484a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2933882447 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e840485b40 0x55e840484090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf5a56640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7cdc0a3240 0x7f7cdc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2933882447 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e840485b40 msgr2=0x55e840484090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2933882447 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e840485b40 0x55e840484090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2933882447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8404845d0 msgr2=0x55e840484a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2933882447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8404845d0 0x55e840484a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2933882447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8403763a0 con 0x55e84048f370 2024-09-15T23:18:11.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.117+0000 7f7ce67fc640 1 -- 172.21.15.62:0/220127243 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7ce80f0d10 con 0x7f7cf010c170 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.118+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2933882447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e84048f370 0x55e840491760 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000cf90 tx=0x7f7ce000c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610d931b8035443f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.118+0000 7f7cf5a56640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7cdc0a3240 0x7f7cdc0a5700 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80c5690 tx=0x7f7ce808fdd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.118+0000 7f7cdb7fe640 1 -- 172.21.15.62:0/2933882447 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0019030 con 0x55e84048f370 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.118+0000 7f7cdb7fe640 1 -- 172.21.15.62:0/2933882447 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ce0011a00 con 0x55e84048f370 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.118+0000 7f7cdb7fe640 1 -- 172.21.15.62:0/2933882447 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce000fa10 con 0x55e84048f370 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 -- 172.21.15.62:0/2933882447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e84048f370 msgr2=0x55e840491760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2933882447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e84048f370 0x55e840491760 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000cf90 tx=0x7f7ce000c630 comp rx=0 tx=0).stop 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 -- 172.21.15.62:0/2933882447 shutdown_connections 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2933882447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e84048f370 0x55e840491760 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2933882447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8404845d0 0x55e840484a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2933882447 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e840485b40 0x55e840484090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 -- 172.21.15.62:0/2933882447 >> 172.21.15.62:0/2933882447 conn(0x55e84044d310 msgr2=0x55e840480c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 -- 172.21.15.62:0/2933882447 shutdown_connections 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 -- 172.21.15.62:0/2933882447 wait complete. 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 -- start start 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 Processor -- start 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e84048ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 0x55e84048c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 0x55e84048c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e840492070 con 0x55e8404845d0 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e8403a5060 con 0x55e840485b40 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.119+0000 7f7cf9892740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e8404921f0 con 0x55e84048f370 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf9892740 10 client.0 _ll_register_callbacks cb 0x55e8400f88c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:11.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e84048ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e84048ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35180/0 (socket says 172.21.15.62:35180) 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2869315089 learned_addr learned my addr 172.21.15.62:0/2869315089 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf5255640 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 0x55e84048c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 0x55e84048c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2869315089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 msgr2=0x55e84048c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 0x55e84048c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2869315089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 msgr2=0x55e84048c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 0x55e84048c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2869315089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e84015b3c0 con 0x55e8404845d0 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e84048ed80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00058c0 tx=0x7f7ce0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d6a620bf10a58e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cd8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/2869315089 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0019040 con 0x55e8404845d0 2024-09-15T23:18:11.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/2869315089 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7ce000f790 con 0x55e8404845d0 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf9892740 1 -- 172.21.15.62:0/2869315089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ce0011a00 con 0x55e8404845d0 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/2869315089 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7ce0031050 con 0x55e8404845d0 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf9892740 10 client.15909 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.120+0000 7f7cf9892740 1 -- 172.21.15.62:0/2869315089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e84049db70 con 0x55e8404845d0 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.121+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/2869315089 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ce0011a00 con 0x55e8404845d0 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.121+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/2869315089 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f7ce002a060 con 0x55e8404845d0 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.121+0000 7f7cf9892740 10 client.15909 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.121+0000 7f7cf9892740 10 client.15909 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.121+0000 7f7cf9892740 20 client.15909 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.121+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e84049eec0 0x55e8404a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.121+0000 7f7cf9892740 4 client.15909 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.121+0000 7f7cf9892740 1 -- 172.21.15.62:0/2869315089 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8403f7860 con 0x55e84049eec0 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.121+0000 7f7cf5255640 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e84049eec0 0x55e8404a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cf5255640 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e84049eec0 0x55e8404a12b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ec1a871152ccc3e1 server_cookie=ba952f1b463d7d07 in_seq=0 out_seq=0 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cd8ff9640 10 client.15909 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/2869315089 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e8403f7860 con 0x55e84049eec0 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cd8ff9640 10 client.15909 handle_command_reply: tid=0 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cf9892740 1 client.15909 shutdown 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cf9892740 1 -- 172.21.15.62:0/2869315089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 msgr2=0x55e84048ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e84048ed80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00058c0 tx=0x7f7ce0005850 comp rx=0 tx=0).stop 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cf9892740 1 -- 172.21.15.62:0/2869315089 shutdown_connections 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e84049eec0 0x55e8404a12b0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 0x55e84048c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.122+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 0x55e84048c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.123+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2869315089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e84048ed80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.123+0000 7f7cf9892740 1 -- 172.21.15.62:0/2869315089 >> 172.21.15.62:0/2869315089 conn(0x55e84044d310 msgr2=0x55e840486960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:11.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.123+0000 7f7cd8ff9640 0 client.15909 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:18:11.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.123+0000 7f7cf9892740 1 -- 172.21.15.62:0/2869315089 shutdown_connections 2024-09-15T23:18:11.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.123+0000 7f7cf9892740 1 -- 172.21.15.62:0/2869315089 wait complete. 2024-09-15T23:18:11.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.123+0000 7f7cf9892740 20 client.15909 trim_cache size 0 max 16384 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.136+0000 7f7cf9892740 1 Processor -- start 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.136+0000 7f7cf9892740 1 -- start start 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.136+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e840480e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.136+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 0x55e84049b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.136+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 0x55e8404a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.136+0000 7f7cf9892740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf01b1450 con 0x55e8404845d0 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.136+0000 7f7cf9892740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf01aecf0 con 0x55e840485b40 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.136+0000 7f7cf9892740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf01acb40 con 0x55e84048f370 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.136+0000 7f7cf4a54640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e840480e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf4a54640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e840480e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35196/0 (socket says 172.21.15.62:35196) 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2080558624 learned_addr learned my addr 172.21.15.62:0/2080558624 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf5255640 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 0x55e84049b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 0x55e8404a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2080558624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 msgr2=0x55e8404a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 0x55e8404a6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2080558624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 msgr2=0x55e84049b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 0x55e84049b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf4a54640 1 -- 172.21.15.62:0/2080558624 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e84048cfd0 con 0x55e8404845d0 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf5255640 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 0x55e84049b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e840480e10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000c540 tx=0x7f7ce0014f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1cdf3f5c9622aaf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cdaffd640 1 -- 172.21.15.62:0/2080558624 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0019040 con 0x55e8404845d0 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cdaffd640 1 -- 172.21.15.62:0/2080558624 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ce0011730 con 0x55e8404845d0 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.137+0000 7f7cdaffd640 1 -- 172.21.15.62:0/2080558624 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00055f0 con 0x55e8404845d0 2024-09-15T23:18:11.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.138+0000 7f7cf9892740 1 -- 172.21.15.62:0/2080558624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 msgr2=0x55e840480e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.138+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e840480e10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000c540 tx=0x7f7ce0014f60 comp rx=0 tx=0).stop 2024-09-15T23:18:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.138+0000 7f7cf9892740 1 -- 172.21.15.62:0/2080558624 shutdown_connections 2024-09-15T23:18:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.138+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e84048f370 0x55e8404a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.138+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e840485b40 0x55e84049b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.138+0000 7f7cf9892740 1 --2- 172.21.15.62:0/2080558624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e8404845d0 0x55e840480e10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.138+0000 7f7cf9892740 1 -- 172.21.15.62:0/2080558624 >> 172.21.15.62:0/2080558624 conn(0x55e84044d310 msgr2=0x55e840480700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.138+0000 7f7cf9892740 1 -- 172.21.15.62:0/2080558624 shutdown_connections 2024-09-15T23:18:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.138+0000 7f7cf9892740 1 -- 172.21.15.62:0/2080558624 wait complete. 2024-09-15T23:18:11.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.139+0000 7f7cf9892740 1 -- start start 2024-09-15T23:18:11.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.139+0000 7f7cf9892740 1 Processor -- start 2024-09-15T23:18:11.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.139+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e8404845d0 0x55e8404886f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.139+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e840485b40 0x55e840488c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.139+0000 7f7cf9892740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e84048f370 0x55e840489170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.139+0000 7f7cf9892740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf01ad1b0 con 0x55e840485b40 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.139+0000 7f7cf9892740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf01ae620 con 0x55e84048f370 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.139+0000 7f7cf9892740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf01ad880 con 0x55e8404845d0 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.139+0000 7f7cf9892740 10 client.0 _ll_register_callbacks cb 0x7f7cec008390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e84048f370 0x55e840489170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e8404845d0 0x55e8404886f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e84048f370 0x55e840489170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:38560/0 (socket says 172.21.15.62:38560) 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 -- 172.21.15.62:0/1979903367 learned_addr learned my addr 172.21.15.62:0/1979903367 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf5255640 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e840485b40 0x55e840488c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 -- 172.21.15.62:0/1979903367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e8404845d0 msgr2=0x55e8404886f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e8404845d0 0x55e8404886f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 -- 172.21.15.62:0/1979903367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e840485b40 msgr2=0x55e840488c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e840485b40 0x55e840488c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 -- 172.21.15.62:0/1979903367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8404964a0 con 0x55e84048f370 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cf4a54640 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e84048f370 0x55e840489170 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7ce002cf00 tx=0x7f7ce0039040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d1872e54c0a181 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.140+0000 7f7cd8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/1979903367 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0019070 con 0x55e84048f370 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/1979903367 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7ce002c640 con 0x55e84048f370 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cf9892740 1 -- 172.21.15.62:0/1979903367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ce0011730 con 0x55e84048f370 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/1979903367 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7ce004f050 con 0x55e84048f370 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cf9892740 10 client.25562 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cf9892740 1 -- 172.21.15.62:0/1979903367 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e840497710 con 0x55e84048f370 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/1979903367 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ce0011730 con 0x55e84048f370 2024-09-15T23:18:11.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/1979903367 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f7ce0043070 con 0x55e84048f370 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cf9892740 10 client.25562 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cf9892740 10 client.25562 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cf9892740 20 client.25562 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.141+0000 7f7cf9892740 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e840497de0 0x55e840514640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cf9892740 4 client.25562 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cf9892740 1 -- 172.21.15.62:0/1979903367 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8403f1d70 con 0x55e840497de0 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cf5255640 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e840497de0 0x55e840514640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cf5255640 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e840497de0 0x55e840514640 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6dff924e93c02f44 server_cookie=f09e3c4306977d51 in_seq=0 out_seq=0 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cd8ff9640 10 client.25562 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cd8ff9640 1 -- 172.21.15.62:0/1979903367 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8403f1d70 con 0x55e840497de0 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cd8ff9640 10 client.25562 handle_command_reply: tid=0 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cf9892740 1 client.25562 shutdown 2024-09-15T23:18:11.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cf9892740 1 -- 172.21.15.62:0/1979903367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e84048f370 msgr2=0x55e840489170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.142+0000 7f7cf9892740 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e84048f370 0x55e840489170 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7ce002cf00 tx=0x7f7ce0039040 comp rx=0 tx=0).stop 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf9892740 1 -- 172.21.15.62:0/1979903367 shutdown_connections 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf9892740 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e840497de0 0x55e840514640 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf9892740 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e84048f370 0x55e840489170 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf9892740 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e840485b40 0x55e840488c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf9892740 1 --2- 172.21.15.62:0/1979903367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e8404845d0 0x55e8404886f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cd8ff9640 0 client.25562 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf9892740 1 -- 172.21.15.62:0/1979903367 >> 172.21.15.62:0/1979903367 conn(0x55e84044d310 msgr2=0x55e8404860e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf9892740 1 -- 172.21.15.62:0/1979903367 shutdown_connections 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf9892740 1 -- 172.21.15.62:0/1979903367 wait complete. 2024-09-15T23:18:11.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf9892740 20 client.25562 trim_cache size 0 max 16384 2024-09-15T23:18:11.150 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:18:11.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf74b9640 1 -- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7cdc0a3240 msgr2=0x7f7cdc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:11.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7cdc0a3240 0x7f7cdc0a5700 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80c5690 tx=0x7f7ce808fdd0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf74b9640 1 -- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 msgr2=0x7f7cf019cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:11.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.143+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 0x7f7cf019cc60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7ce802ae70 tx=0x7f7ce802f5b0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.144+0000 7f7cf74b9640 1 -- 172.21.15.62:0/220127243 shutdown_connections 2024-09-15T23:18:11.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.144+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7cdc0a3240 0x7f7cdc0a5700 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.144+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cf010c170 0x7f7cf019cc60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.144+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7cf01036b0 0x7f7cf01988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.144+0000 7f7cf74b9640 1 --2- 172.21.15.62:0/220127243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7cf0102d90 0x7f7cf0198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:11.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.144+0000 7f7cf74b9640 1 -- 172.21.15.62:0/220127243 >> 172.21.15.62:0/220127243 conn(0x7f7cf00fc790 msgr2=0x7f7cf00fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:11.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.144+0000 7f7cf74b9640 1 -- 172.21.15.62:0/220127243 shutdown_connections 2024-09-15T23:18:11.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:11.144+0000 7f7cf74b9640 1 -- 172.21.15.62:0/220127243 wait complete. 2024-09-15T23:18:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:11 smithi062 ceph-mon[30949]: pgmap v427: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 604 op/s 2024-09-15T23:18:11.802 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:18:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:11 smithi067 ceph-mon[35665]: pgmap v427: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 604 op/s 2024-09-15T23:18:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:11 smithi148 ceph-mon[35610]: pgmap v427: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 12 MiB/s wr, 604 op/s 2024-09-15T23:18:12.071 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 -- 172.21.15.62:0/475641511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 msgr2=0x7f80d41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 --2- 172.21.15.62:0/475641511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 0x7f80d41067d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f80c4009900 tx=0x7f80c402a410 comp rx=0 tx=0).stop 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 -- 172.21.15.62:0/475641511 shutdown_connections 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 --2- 172.21.15.62:0/475641511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d4106f00 0x7f80d410ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 --2- 172.21.15.62:0/475641511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 0x7f80d41067d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 --2- 172.21.15.62:0/475641511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d4101dd0 0x7f80d41021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 -- 172.21.15.62:0/475641511 >> 172.21.15.62:0/475641511 conn(0x7f80d40fd890 msgr2=0x7f80d40ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 -- 172.21.15.62:0/475641511 shutdown_connections 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 -- 172.21.15.62:0/475641511 wait complete. 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.580+0000 7f80dc384640 1 Processor -- start 2024-09-15T23:18:12.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80dc384640 1 -- start start 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80dc384640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d4101dd0 0x7f80d4198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80dc384640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 0x7f80d4198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80dc384640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d4106f00 0x7f80d419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80dc384640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f80d4111f60 con 0x7f80d4102780 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80dc384640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80d4111de0 con 0x7f80d4106f00 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80dc384640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f80d41120e0 con 0x7f80d4101dd0 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80d991f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 0x7f80d4198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80d991f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 0x7f80d4198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40632/0 (socket says 172.21.15.62:40632) 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80d991f640 1 -- 172.21.15.62:0/3510825479 learned_addr learned my addr 172.21.15.62:0/3510825479 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80da921640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d4106f00 0x7f80d419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80da120640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d4101dd0 0x7f80d4198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80d991f640 1 -- 172.21.15.62:0/3510825479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d4101dd0 msgr2=0x7f80d4198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:12.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80d991f640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d4101dd0 0x7f80d4198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80d991f640 1 -- 172.21.15.62:0/3510825479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d4106f00 msgr2=0x7f80d419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80d991f640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d4106f00 0x7f80d419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80d991f640 1 -- 172.21.15.62:0/3510825479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d419d3f0 con 0x7f80d4102780 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80da921640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d4106f00 0x7f80d419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80da120640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d4101dd0 0x7f80d4198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.581+0000 7f80d991f640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 0x7f80d4198980 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f80c4006010 tx=0x7f80c40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.582+0000 7f80cb7fe640 1 -- 172.21.15.62:0/3510825479 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c403f030 con 0x7f80d4102780 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.582+0000 7f80cb7fe640 1 -- 172.21.15.62:0/3510825479 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c4003ce0 con 0x7f80d4102780 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.582+0000 7f80dc384640 1 -- 172.21.15.62:0/3510825479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d419d680 con 0x7f80d4102780 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.582+0000 7f80cb7fe640 1 -- 172.21.15.62:0/3510825479 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4035070 con 0x7f80d4102780 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.582+0000 7f80dc384640 1 -- 172.21.15.62:0/3510825479 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d41a4fc0 con 0x7f80d4102780 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.584+0000 7f80cb7fe640 1 -- 172.21.15.62:0/3510825479 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f80c4042030 con 0x7f80d4102780 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.584+0000 7f80de75d740 1 Processor -- start 2024-09-15T23:18:12.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.584+0000 7f80de75d740 1 -- start start 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80de75d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 0x56173619c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80de75d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 0x56173619ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80de75d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361adb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80de75d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5617361022f0 con 0x5617361ab780 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80de75d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5617360a9cf0 con 0x56173619c5d0 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80de75d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5617360fe2b0 con 0x56173619db40 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80d991f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361adb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80d991f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361adb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40638/0 (socket says 172.21.15.62:40638) 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80d991f640 1 -- 172.21.15.62:0/2340824570 learned_addr learned my addr 172.21.15.62:0/2340824570 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80da120640 1 --2- 172.21.15.62:0/2340824570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 0x56173619ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80cb7fe640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80c00a32c0 0x7f80c00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80da921640 1 --2- 172.21.15.62:0/2340824570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 0x56173619c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80da921640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80c00a32c0 0x7f80c00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80cb7fe640 1 -- 172.21.15.62:0/3510825479 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f80c40eff10 con 0x7f80d4102780 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80d991f640 1 -- 172.21.15.62:0/2340824570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 msgr2=0x56173619c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80d991f640 1 --2- 172.21.15.62:0/2340824570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 0x56173619c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80d991f640 1 -- 172.21.15.62:0/2340824570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 msgr2=0x56173619ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80d991f640 1 --2- 172.21.15.62:0/2340824570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 0x56173619ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.585+0000 7f80d991f640 1 -- 172.21.15.62:0/2340824570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56173608e3a0 con 0x5617361ab780 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.586+0000 7f80d991f640 1 --2- 172.21.15.62:0/2340824570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361adb70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f80c40bbf10 tx=0x7f80c4086770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff2677ba7087b65 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.586+0000 7f80da921640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80c00a32c0 0x7f80c00a5780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f80cc007310 tx=0x7f80cc0072a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.586+0000 7f80c8dea640 1 -- 172.21.15.62:0/2340824570 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4049070 con 0x5617361ab780 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.586+0000 7f80c8dea640 1 -- 172.21.15.62:0/2340824570 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c403b580 con 0x5617361ab780 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.586+0000 7f80c8dea640 1 -- 172.21.15.62:0/2340824570 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c40c05c0 con 0x5617361ab780 2024-09-15T23:18:12.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 -- 172.21.15.62:0/2340824570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 msgr2=0x5617361adb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 --2- 172.21.15.62:0/2340824570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361adb70 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f80c40bbf10 tx=0x7f80c4086770 comp rx=0 tx=0).stop 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 -- 172.21.15.62:0/2340824570 shutdown_connections 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 --2- 172.21.15.62:0/2340824570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361adb70 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 --2- 172.21.15.62:0/2340824570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 0x56173619ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 --2- 172.21.15.62:0/2340824570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 0x56173619c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 -- 172.21.15.62:0/2340824570 >> 172.21.15.62:0/2340824570 conn(0x561736165310 msgr2=0x561736198c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 -- 172.21.15.62:0/2340824570 shutdown_connections 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 -- 172.21.15.62:0/2340824570 wait complete. 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 -- start start 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 Processor -- start 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56173619c5d0 0x5617361ab200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 0x5617361a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617361ab780 0x5617361a8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5617361ae240 con 0x56173619c5d0 2024-09-15T23:18:12.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5617360bd060 con 0x56173619db40 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.587+0000 7f80de75d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5617361ae3c0 con 0x5617361ab780 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80de75d740 10 client.0 _ll_register_callbacks cb 0x561735e108c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da120640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56173619c5d0 0x5617361ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da120640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56173619c5d0 0x5617361ab200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40650/0 (socket says 172.21.15.62:40650) 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da120640 1 -- 172.21.15.62:0/3923725825 learned_addr learned my addr 172.21.15.62:0/3923725825 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80d991f640 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 0x5617361a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da921640 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617361ab780 0x5617361a8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da120640 1 -- 172.21.15.62:0/3923725825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617361ab780 msgr2=0x5617361a8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da120640 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617361ab780 0x5617361a8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da120640 1 -- 172.21.15.62:0/3923725825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 msgr2=0x5617361a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da120640 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 0x5617361a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da120640 1 -- 172.21.15.62:0/3923725825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561735e733c0 con 0x56173619c5d0 2024-09-15T23:18:12.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80da120640 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56173619c5d0 0x5617361ab200 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f80d00113e0 tx=0x7f80d00118a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcce480ca2be8d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80bdffb640 1 -- 172.21.15.62:0/3923725825 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d0012980 con 0x56173619c5d0 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80bdffb640 1 -- 172.21.15.62:0/3923725825 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f80d0012b20 con 0x56173619c5d0 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80de75d740 1 -- 172.21.15.62:0/3923725825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56173610f860 con 0x56173619c5d0 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80bdffb640 1 -- 172.21.15.62:0/3923725825 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f80d0025050 con 0x56173619c5d0 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80de75d740 10 client.15927 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.588+0000 7f80de75d740 1 -- 172.21.15.62:0/3923725825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5617361b9cc0 con 0x56173619c5d0 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.589+0000 7f80bdffb640 1 -- 172.21.15.62:0/3923725825 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56173610f860 con 0x56173619c5d0 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.589+0000 7f80bdffb640 1 -- 172.21.15.62:0/3923725825 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f80d000f040 con 0x56173619c5d0 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.589+0000 7f80de75d740 10 client.15927 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.589+0000 7f80de75d740 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.589+0000 7f80de75d740 20 client.15927 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:12.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.589+0000 7f80de75d740 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617361bb220 0x5617361bd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.589+0000 7f80de75d740 4 client.15927 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.589+0000 7f80de75d740 1 -- 172.21.15.62:0/3923725825 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561736109d70 con 0x5617361bb220 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.589+0000 7f80d991f640 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617361bb220 0x5617361bd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80bdffb640 10 client.15927 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80d991f640 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617361bb220 0x5617361bd6a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f879976e8ba72273 server_cookie=dea7f21a4d73000a in_seq=0 out_seq=0 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80bdffb640 1 -- 172.21.15.62:0/3923725825 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561736109d70 con 0x5617361bb220 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80bdffb640 10 client.15927 handle_command_reply: tid=0 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80de75d740 1 client.15927 shutdown 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80de75d740 1 -- 172.21.15.62:0/3923725825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56173619c5d0 msgr2=0x5617361ab200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80de75d740 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56173619c5d0 0x5617361ab200 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f80d00113e0 tx=0x7f80d00118a0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80de75d740 1 -- 172.21.15.62:0/3923725825 shutdown_connections 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80de75d740 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617361bb220 0x5617361bd6a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80de75d740 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617361ab780 0x5617361a8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80de75d740 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 0x5617361a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80bdffb640 0 client.15927 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:12.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80de75d740 1 --2- 172.21.15.62:0/3923725825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56173619c5d0 0x5617361ab200 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.590+0000 7f80de75d740 1 -- 172.21.15.62:0/3923725825 >> 172.21.15.62:0/3923725825 conn(0x561736165310 msgr2=0x56173619e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:12.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.591+0000 7f80de75d740 1 -- 172.21.15.62:0/3923725825 shutdown_connections 2024-09-15T23:18:12.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.591+0000 7f80de75d740 1 -- 172.21.15.62:0/3923725825 wait complete. 2024-09-15T23:18:12.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.591+0000 7f80de75d740 20 client.15927 trim_cache size 0 max 16384 2024-09-15T23:18:12.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.603+0000 7f80de75d740 1 Processor -- start 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80de75d740 1 -- start start 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80de75d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 0x5617361a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80de75d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 0x5617361b7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80de75d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361a7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80de75d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f80d41af360 con 0x5617361ab780 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80de75d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80d41acc00 con 0x56173619c5d0 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80de75d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f80d41aaa50 con 0x56173619db40 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da921640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361a7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da921640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361a7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40662/0 (socket says 172.21.15.62:40662) 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da921640 1 -- 172.21.15.62:0/1964647554 learned_addr learned my addr 172.21.15.62:0/1964647554 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da120640 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 0x5617361a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80d991f640 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 0x5617361b7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da921640 1 -- 172.21.15.62:0/1964647554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 msgr2=0x5617361b7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:12.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da921640 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 0x5617361b7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da921640 1 -- 172.21.15.62:0/1964647554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 msgr2=0x5617361a7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:12.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da921640 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 0x5617361a7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da921640 1 -- 172.21.15.62:0/1964647554 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561736199960 con 0x5617361ab780 2024-09-15T23:18:12.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da120640 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 0x5617361a7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:12.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.604+0000 7f80da921640 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361a7490 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f80cc013ed0 tx=0x7f80cc0073c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4186e32c773e7887 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:12.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80bffff640 1 -- 172.21.15.62:0/1964647554 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc0119c0 con 0x5617361ab780 2024-09-15T23:18:12.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80bffff640 1 -- 172.21.15.62:0/1964647554 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80cc011b60 con 0x5617361ab780 2024-09-15T23:18:12.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80bffff640 1 -- 172.21.15.62:0/1964647554 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc01f440 con 0x5617361ab780 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80de75d740 1 -- 172.21.15.62:0/1964647554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 msgr2=0x5617361a7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80de75d740 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361a7490 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f80cc013ed0 tx=0x7f80cc0073c0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80de75d740 1 -- 172.21.15.62:0/1964647554 shutdown_connections 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80de75d740 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361a7490 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80de75d740 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619db40 0x5617361b7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80de75d740 1 --2- 172.21.15.62:0/1964647554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619c5d0 0x5617361a7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80de75d740 1 -- 172.21.15.62:0/1964647554 >> 172.21.15.62:0/1964647554 conn(0x561736165310 msgr2=0x5617361a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80de75d740 1 -- 172.21.15.62:0/1964647554 shutdown_connections 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.605+0000 7f80de75d740 1 -- 172.21.15.62:0/1964647554 wait complete. 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80de75d740 1 -- start start 2024-09-15T23:18:12.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80de75d740 1 Processor -- start 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80de75d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619c5d0 0x5617361b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80de75d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 0x5617361b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80de75d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361aea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80de75d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f80d41ab0c0 con 0x5617361ab780 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80de75d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80d41ac530 con 0x56173619db40 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80de75d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f80d41ab790 con 0x56173619c5d0 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80de75d740 10 client.0 _ll_register_callbacks cb 0x7f80d00097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80da921640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80da921640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361aea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40674/0 (socket says 172.21.15.62:40674) 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80da921640 1 -- 172.21.15.62:0/1038954134 learned_addr learned my addr 172.21.15.62:0/1038954134 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80d991f640 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 0x5617361b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80da120640 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619c5d0 0x5617361b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80da921640 1 -- 172.21.15.62:0/1038954134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619c5d0 msgr2=0x5617361b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80da921640 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619c5d0 0x5617361b35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80da921640 1 -- 172.21.15.62:0/1038954134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 msgr2=0x5617361b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80da921640 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 0x5617361b3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.606+0000 7f80da921640 1 -- 172.21.15.62:0/1038954134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617361af230 con 0x5617361ab780 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80da921640 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361aea00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f80cc005c90 tx=0x7f80cc028b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3843abf3648e0be6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80bdffb640 1 -- 172.21.15.62:0/1038954134 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc004e50 con 0x5617361ab780 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80bdffb640 1 -- 172.21.15.62:0/1038954134 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f80cc028480 con 0x5617361ab780 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80de75d740 1 -- 172.21.15.62:0/1038954134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80cc011b60 con 0x5617361ab780 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80bdffb640 1 -- 172.21.15.62:0/1038954134 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f80cc028680 con 0x5617361ab780 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80de75d740 10 client.15933 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80de75d740 1 -- 172.21.15.62:0/1038954134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561736225a00 con 0x5617361ab780 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80bdffb640 1 -- 172.21.15.62:0/1038954134 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80cc011b60 con 0x5617361ab780 2024-09-15T23:18:12.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80bdffb640 1 -- 172.21.15.62:0/1038954134 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f80cc02f070 con 0x5617361ab780 2024-09-15T23:18:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80de75d740 10 client.15933 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80de75d740 10 client.15933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80de75d740 20 client.15933 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80de75d740 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561736226040 0x561736228460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80de75d740 4 client.15933 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:18:12.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.607+0000 7f80de75d740 1 -- 172.21.15.62:0/1038954134 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561736086170 con 0x561736226040 2024-09-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80da120640 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561736226040 0x561736228460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80da120640 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561736226040 0x561736228460 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bd696e05b3ee9ca server_cookie=9fcb5d0ec16a02ca in_seq=0 out_seq=0 2024-09-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80bdffb640 10 client.15933 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80bdffb640 1 -- 172.21.15.62:0/1038954134 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561736086170 con 0x561736226040 2024-09-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80bdffb640 10 client.15933 handle_command_reply: tid=0 2024-09-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80de75d740 1 client.15933 shutdown 2024-09-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80de75d740 1 -- 172.21.15.62:0/1038954134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 msgr2=0x5617361aea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80de75d740 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361aea00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f80cc005c90 tx=0x7f80cc028b80 comp rx=0 tx=0).stop 2024-09-15T23:18:12.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80de75d740 1 -- 172.21.15.62:0/1038954134 shutdown_connections 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80de75d740 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561736226040 0x561736228460 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80de75d740 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617361ab780 0x5617361aea00 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80de75d740 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56173619db40 0x5617361b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80de75d740 1 --2- 172.21.15.62:0/1038954134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56173619c5d0 0x5617361b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80de75d740 1 -- 172.21.15.62:0/1038954134 >> 172.21.15.62:0/1038954134 conn(0x561736165310 msgr2=0x561736197d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.608+0000 7f80bdffb640 0 client.15933 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80de75d740 1 -- 172.21.15.62:0/1038954134 shutdown_connections 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80de75d740 1 -- 172.21.15.62:0/1038954134 wait complete. 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80de75d740 20 client.15933 trim_cache size 0 max 16384 2024-09-15T23:18:12.616 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:18:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 -- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80c00a32c0 msgr2=0x7f80c00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80c00a32c0 0x7f80c00a5780 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f80cc007310 tx=0x7f80cc0072a0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 -- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 msgr2=0x7f80d4198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 0x7f80d4198980 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f80c4006010 tx=0x7f80c40042f0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 -- 172.21.15.62:0/3510825479 shutdown_connections 2024-09-15T23:18:12.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80c00a32c0 0x7f80c00a5780 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80d4106f00 0x7f80d419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80d4102780 0x7f80d4198980 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 --2- 172.21.15.62:0/3510825479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80d4101dd0 0x7f80d4198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 -- 172.21.15.62:0/3510825479 >> 172.21.15.62:0/3510825479 conn(0x7f80d40fd890 msgr2=0x7f80d40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 -- 172.21.15.62:0/3510825479 shutdown_connections 2024-09-15T23:18:12.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:12.609+0000 7f80dc384640 1 -- 172.21.15.62:0/3510825479 wait complete. 2024-09-15T23:18:13.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:13 smithi062 ceph-mon[30949]: pgmap v428: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 641 op/s 2024-09-15T23:18:14.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:13 smithi067 ceph-mon[35665]: pgmap v428: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 641 op/s 2024-09-15T23:18:14.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:13 smithi148 ceph-mon[35610]: pgmap v428: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 641 op/s 2024-09-15T23:18:14.195 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:18:14.196 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:18:14.469 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:14.573 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 115.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:18:15.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.989+0000 7fdad30d1640 1 -- 172.21.15.62:0/78847592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc103700 msgr2=0x7fdacc10bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:15.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.989+0000 7fdad30d1640 1 --2- 172.21.15.62:0/78847592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc103700 0x7fdacc10bbf0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009900 tx=0x7fdac002a480 comp rx=0 tx=0).stop 2024-09-15T23:18:15.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.989+0000 7fdad30d1640 1 -- 172.21.15.62:0/78847592 shutdown_connections 2024-09-15T23:18:15.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.989+0000 7fdad30d1640 1 --2- 172.21.15.62:0/78847592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdacc10c180 0x7fdacc10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.989+0000 7fdad30d1640 1 --2- 172.21.15.62:0/78847592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc103700 0x7fdacc10bbf0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.989+0000 7fdad30d1640 1 --2- 172.21.15.62:0/78847592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdacc102de0 0x7fdacc1031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.989+0000 7fdad30d1640 1 -- 172.21.15.62:0/78847592 >> 172.21.15.62:0/78847592 conn(0x7fdacc0fc7a0 msgr2=0x7fdacc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 -- 172.21.15.62:0/78847592 shutdown_connections 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 -- 172.21.15.62:0/78847592 wait complete. 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 Processor -- start 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 -- start start 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc102de0 0x7fdacc1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdacc103700 0x7fdacc198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdacc10c180 0x7fdacc19ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc111f30 con 0x7fdacc102de0 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc111db0 con 0x7fdacc10c180 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad30d1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc1120b0 con 0x7fdacc103700 2024-09-15T23:18:15.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.990+0000 7fdad0e6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc102de0 0x7fdacc1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad0e6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc102de0 0x7fdacc1983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40694/0 (socket says 172.21.15.62:40694) 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad0e6d640 1 -- 172.21.15.62:0/3390850953 learned_addr learned my addr 172.21.15.62:0/3390850953 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdacbfff640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdacc103700 0x7fdacc198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad166e640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdacc10c180 0x7fdacc19ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad0e6d640 1 -- 172.21.15.62:0/3390850953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdacc103700 msgr2=0x7fdacc198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdacc103700 0x7fdacc198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad0e6d640 1 -- 172.21.15.62:0/3390850953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdacc10c180 msgr2=0x7fdacc19ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdacc10c180 0x7fdacc19ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad0e6d640 1 -- 172.21.15.62:0/3390850953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdacc19d3a0 con 0x7fdacc102de0 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad166e640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdacc10c180 0x7fdacc19ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:15.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc102de0 0x7fdacc1983f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdac400cce0 tx=0x7fdac400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdac9ffb640 1 -- 172.21.15.62:0/3390850953 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac400f650 con 0x7fdacc102de0 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdad30d1640 1 -- 172.21.15.62:0/3390850953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdacc19d690 con 0x7fdacc102de0 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdac9ffb640 1 -- 172.21.15.62:0/3390850953 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdac400fcd0 con 0x7fdacc102de0 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.991+0000 7fdac9ffb640 1 -- 172.21.15.62:0/3390850953 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac4018c70 con 0x7fdacc102de0 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.992+0000 7fdad30d1640 1 -- 172.21.15.62:0/3390850953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdacc100970 con 0x7fdacc102de0 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.994+0000 7fdac9ffb640 1 -- 172.21.15.62:0/3390850953 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fdac400f7f0 con 0x7fdacc102de0 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad54aa740 1 Processor -- start 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad54aa740 1 -- start start 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 0x5651f2ea1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eac800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 0x5651f2eaf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad54aa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5651f2e07280 con 0x5651f2ea2480 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad54aa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5651f2daeb10 con 0x5651f2eacd40 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad54aa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5651f2e03240 con 0x5651f2ea1b60 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdac9ffb640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdab40a3240 0x7fdab40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad0e6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eac800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad0e6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eac800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40708/0 (socket says 172.21.15.62:40708) 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad0e6d640 1 -- 172.21.15.62:0/1945575363 learned_addr learned my addr 172.21.15.62:0/1945575363 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad166e640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdab40a3240 0x7fdab40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdacbfff640 1 --2- 172.21.15.62:0/1945575363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 0x5651f2eaf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdad166e640 1 --2- 172.21.15.62:0/1945575363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 0x5651f2ea1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.995+0000 7fdac9ffb640 1 -- 172.21.15.62:0/3390850953 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdac40d0140 con 0x7fdacc102de0 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdad0e6d640 1 -- 172.21.15.62:0/1945575363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 msgr2=0x5651f2ea1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/1945575363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 0x5651f2ea1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdad0e6d640 1 -- 172.21.15.62:0/1945575363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 msgr2=0x5651f2eaf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:15.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/1945575363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 0x5651f2eaf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdad0e6d640 1 -- 172.21.15.62:0/1945575363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651f2dbb1f0 con 0x5651f2ea2480 2024-09-15T23:18:15.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/1945575363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eac800 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdac4098db0 tx=0x7fdac40987b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f83a56819b3c2b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:15.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdad166e640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdab40a3240 0x7fdab40a5700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdabc005d50 tx=0x7fdabc005e90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:15.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdab2ffd640 1 -- 172.21.15.62:0/1945575363 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac409bf20 con 0x5651f2ea2480 2024-09-15T23:18:15.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdab2ffd640 1 -- 172.21.15.62:0/1945575363 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdac40a0790 con 0x5651f2ea2480 2024-09-15T23:18:15.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.996+0000 7fdab2ffd640 1 -- 172.21.15.62:0/1945575363 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac409bd40 con 0x5651f2ea2480 2024-09-15T23:18:15.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 -- 172.21.15.62:0/1945575363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 msgr2=0x5651f2eac800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 --2- 172.21.15.62:0/1945575363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eac800 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdac4098db0 tx=0x7fdac40987b0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 -- 172.21.15.62:0/1945575363 shutdown_connections 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 --2- 172.21.15.62:0/1945575363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 0x5651f2eaf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 --2- 172.21.15.62:0/1945575363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eac800 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 --2- 172.21.15.62:0/1945575363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 0x5651f2ea1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 -- 172.21.15.62:0/1945575363 >> 172.21.15.62:0/1945575363 conn(0x5651f2e9e0f0 msgr2=0x5651f2e9e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 -- 172.21.15.62:0/1945575363 shutdown_connections 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 -- 172.21.15.62:0/1945575363 wait complete. 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 -- start start 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.997+0000 7fdad54aa740 1 Processor -- start 2024-09-15T23:18:15.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea1b60 0x5651f2ea9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eaa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2eacd40 0x5651f2eaa710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad54aa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5651f2eaf910 con 0x5651f2ea2480 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad54aa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5651f2dc1f10 con 0x5651f2ea1b60 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad54aa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5651f2eafa90 con 0x5651f2eacd40 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad54aa740 10 client.0 _ll_register_callbacks cb 0x5651f2d8aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdacbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eaa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdacbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eaa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40720/0 (socket says 172.21.15.62:40720) 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdacbfff640 1 -- 172.21.15.62:0/3179782901 learned_addr learned my addr 172.21.15.62:0/3179782901 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea1b60 0x5651f2ea9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad166e640 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2eacd40 0x5651f2eaa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdacbfff640 1 -- 172.21.15.62:0/3179782901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2eacd40 msgr2=0x5651f2eaa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:15.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdacbfff640 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2eacd40 0x5651f2eaa710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdacbfff640 1 -- 172.21.15.62:0/3179782901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea1b60 msgr2=0x5651f2ea9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:15.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdacbfff640 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea1b60 0x5651f2ea9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdacbfff640 1 -- 172.21.15.62:0/3179782901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651f2b786a0 con 0x5651f2ea2480 2024-09-15T23:18:15.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea1b60 0x5651f2ea9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:15.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdacbfff640 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eaa1d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fdac0033000 tx=0x7fdac0031a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e806369fbfc328bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:15.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdaabfff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.998+0000 7fdaabfff640 1 -- 172.21.15.62:0/3179782901 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac0035040 con 0x5651f2ea2480 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdaabfff640 1 -- 172.21.15.62:0/3179782901 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdac0004400 con 0x5651f2ea2480 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad54aa740 1 -- 172.21.15.62:0/3179782901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5651f2e147f0 con 0x5651f2ea2480 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdaabfff640 1 -- 172.21.15.62:0/3179782901 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdac004f050 con 0x5651f2ea2480 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad54aa740 10 client.15945 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad54aa740 1 -- 172.21.15.62:0/3179782901 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5651f2ebb590 con 0x5651f2ea2480 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdaabfff640 1 -- 172.21.15.62:0/3179782901 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5651f2e147f0 con 0x5651f2ea2480 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdaabfff640 1 -- 172.21.15.62:0/3179782901 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fdac0046880 con 0x5651f2ea2480 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad54aa740 10 client.15945 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad54aa740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad54aa740 20 client.15945 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad54aa740 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5651f2ebcaa0 0x5651f2ebef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad54aa740 4 client.15945 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad54aa740 1 -- 172.21.15.62:0/3179782901 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651f2e0ed00 con 0x5651f2ebcaa0 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:14.999+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5651f2ebcaa0 0x5651f2ebef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.000+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5651f2ebcaa0 0x5651f2ebef30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d1f06220b1bedd2 server_cookie=6e1a503f4ab07de in_seq=0 out_seq=0 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.000+0000 7fdaabfff640 10 client.15945 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdaabfff640 1 -- 172.21.15.62:0/3179782901 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5651f2e0ed00 con 0x5651f2ebcaa0 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdaabfff640 10 client.15945 handle_command_reply: tid=0 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 client.15945 shutdown 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 -- 172.21.15.62:0/3179782901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 msgr2=0x5651f2eaa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eaa1d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fdac0033000 tx=0x7fdac0031a60 comp rx=0 tx=0).stop 2024-09-15T23:18:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 -- 172.21.15.62:0/3179782901 shutdown_connections 2024-09-15T23:18:15.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5651f2ebcaa0 0x5651f2ebef30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2eacd40 0x5651f2eaa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eaa1d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdaabfff640 0 client.15945 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:15.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 --2- 172.21.15.62:0/3179782901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea1b60 0x5651f2ea9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 -- 172.21.15.62:0/3179782901 >> 172.21.15.62:0/3179782901 conn(0x5651f2e9e0f0 msgr2=0x5651f2ea3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:15.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 -- 172.21.15.62:0/3179782901 shutdown_connections 2024-09-15T23:18:15.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 1 -- 172.21.15.62:0/3179782901 wait complete. 2024-09-15T23:18:15.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.001+0000 7fdad54aa740 20 client.15945 trim_cache size 0 max 16384 2024-09-15T23:18:15.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.018+0000 7fdad54aa740 1 Processor -- start 2024-09-15T23:18:15.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.018+0000 7fdad54aa740 1 -- start start 2024-09-15T23:18:15.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.018+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea1b60 0x5651f2ec4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.018+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea2480 0x5651f2ec4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.018+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2eacd40 0x5651f2ea6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.018+0000 7fdad54aa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc1b3750 con 0x5651f2ea1b60 2024-09-15T23:18:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.018+0000 7fdad54aa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc1b0ff0 con 0x5651f2ea2480 2024-09-15T23:18:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.018+0000 7fdad54aa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc1aee40 con 0x5651f2eacd40 2024-09-15T23:18:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdad0e6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea1b60 0x5651f2ec4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdad0e6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea1b60 0x5651f2ec4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40736/0 (socket says 172.21.15.62:40736) 2024-09-15T23:18:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdad0e6d640 1 -- 172.21.15.62:0/233439855 learned_addr learned my addr 172.21.15.62:0/233439855 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdacbfff640 1 --2- 172.21.15.62:0/233439855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea2480 0x5651f2ec4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdad0e6d640 1 -- 172.21.15.62:0/233439855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2eacd40 msgr2=0x5651f2ea6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/233439855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2eacd40 0x5651f2ea6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdad0e6d640 1 -- 172.21.15.62:0/233439855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea2480 msgr2=0x5651f2ec4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/233439855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea2480 0x5651f2ec4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdad0e6d640 1 -- 172.21.15.62:0/233439855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651f2ea81e0 con 0x5651f2ea1b60 2024-09-15T23:18:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdacbfff640 1 --2- 172.21.15.62:0/233439855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea2480 0x5651f2ec4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/233439855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea1b60 0x5651f2ec4840 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdac40a1880 tx=0x7fdac4098680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71aaa8992f9cc410 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:15.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdab27fc640 1 -- 172.21.15.62:0/233439855 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac40a5050 con 0x5651f2ea1b60 2024-09-15T23:18:15.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdab27fc640 1 -- 172.21.15.62:0/233439855 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdac4018e10 con 0x5651f2ea1b60 2024-09-15T23:18:15.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.019+0000 7fdab27fc640 1 -- 172.21.15.62:0/233439855 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac40b53e0 con 0x5651f2ea1b60 2024-09-15T23:18:15.057 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:18:15.057 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:18:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "6b4b943c-4a01-4ae3-94b0-d5a898040e81", 2024-09-15T23:18:15.058 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:18:15.058 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:18:15.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 -- 172.21.15.62:0/233439855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea1b60 msgr2=0x5651f2ec4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:15.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 --2- 172.21.15.62:0/233439855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea1b60 0x5651f2ec4840 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fdac40a1880 tx=0x7fdac4098680 comp rx=0 tx=0).stop 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 -- 172.21.15.62:0/233439855 shutdown_connections 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 --2- 172.21.15.62:0/233439855 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2eacd40 0x5651f2ea6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 --2- 172.21.15.62:0/233439855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2ea2480 0x5651f2ec4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 --2- 172.21.15.62:0/233439855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea1b60 0x5651f2ec4840 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 -- 172.21.15.62:0/233439855 >> 172.21.15.62:0/233439855 conn(0x5651f2e9e0f0 msgr2=0x5651f2eabc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 -- 172.21.15.62:0/233439855 shutdown_connections 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 -- 172.21.15.62:0/233439855 wait complete. 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 -- start start 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.020+0000 7fdad54aa740 1 Processor -- start 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 0x5651f2eb4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eb5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad54aa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 0x5651f2eb11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad54aa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc1af4b0 con 0x5651f2ea2480 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad54aa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc1b0920 con 0x5651f2eacd40 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad54aa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdacc1afb80 con 0x5651f2ea1b60 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad54aa740 10 client.0 _ll_register_callbacks cb 0x7fdac0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdacbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eb5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdacbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eb5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40750/0 (socket says 172.21.15.62:40750) 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdacbfff640 1 -- 172.21.15.62:0/692724516 learned_addr learned my addr 172.21.15.62:0/692724516 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 0x5651f2eb4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad166e640 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 0x5651f2eb11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdacbfff640 1 -- 172.21.15.62:0/692724516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 msgr2=0x5651f2eb4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdacbfff640 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 0x5651f2eb4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdacbfff640 1 -- 172.21.15.62:0/692724516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 msgr2=0x5651f2eb11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdacbfff640 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 0x5651f2eb11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdacbfff640 1 -- 172.21.15.62:0/692724516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651f2eb19a0 con 0x5651f2ea2480 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 0x5651f2eb4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad166e640 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 0x5651f2eb11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdacbfff640 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eb5230 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009850 tx=0x7fdac0005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160efbbb8b260efa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdaabfff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdaabfff640 1 -- 172.21.15.62:0/692724516 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac0035040 con 0x5651f2ea2480 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdaabfff640 1 -- 172.21.15.62:0/692724516 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdac0031630 con 0x5651f2ea2480 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.021+0000 7fdad54aa740 1 -- 172.21.15.62:0/692724516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdac4018e10 con 0x5651f2ea2480 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdaabfff640 1 -- 172.21.15.62:0/692724516 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdac0061050 con 0x5651f2ea2480 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdad54aa740 10 client.15951 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdad54aa740 1 -- 172.21.15.62:0/692724516 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5651f2f28210 con 0x5651f2ea2480 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdaabfff640 1 -- 172.21.15.62:0/692724516 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdac4018e10 con 0x5651f2ea2480 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdaabfff640 1 -- 172.21.15.62:0/692724516 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7fdac003d9d0 con 0x5651f2ea2480 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdad54aa740 10 client.15951 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:15.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdad54aa740 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdad54aa740 20 client.15951 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdad54aa740 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5651f2f28a70 0x5651f2f2ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdad54aa740 4 client.15951 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdad54aa740 1 -- 172.21.15.62:0/692724516 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5651f2d8b110 con 0x5651f2f28a70 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.022+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5651f2f28a70 0x5651f2f2ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.023+0000 7fdad0e6d640 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5651f2f28a70 0x5651f2f2ae90 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=affcffba56011d1f server_cookie=5b8e8275e3f57c9f in_seq=0 out_seq=0 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.023+0000 7fdaabfff640 10 client.15951 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdaabfff640 1 -- 172.21.15.62:0/692724516 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5651f2d8b110 con 0x5651f2f28a70 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdaabfff640 10 client.15951 handle_command_reply: tid=0 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdad54aa740 1 client.15951 shutdown 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdad54aa740 1 -- 172.21.15.62:0/692724516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 msgr2=0x5651f2eb5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdad54aa740 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eb5230 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009850 tx=0x7fdac0005e00 comp rx=0 tx=0).stop 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdad54aa740 1 -- 172.21.15.62:0/692724516 shutdown_connections 2024-09-15T23:18:15.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdad54aa740 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5651f2f28a70 0x5651f2f2ae90 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdad54aa740 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5651f2eacd40 0x5651f2eb11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdad54aa740 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5651f2ea2480 0x5651f2eb5230 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdaabfff640 0 client.15951 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:15.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdad54aa740 1 --2- 172.21.15.62:0/692724516 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5651f2ea1b60 0x5651f2eb4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.024+0000 7fdad54aa740 1 -- 172.21.15.62:0/692724516 >> 172.21.15.62:0/692724516 conn(0x5651f2e9e0f0 msgr2=0x5651f2ea9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:15.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad54aa740 1 -- 172.21.15.62:0/692724516 shutdown_connections 2024-09-15T23:18:15.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad54aa740 1 -- 172.21.15.62:0/692724516 wait complete. 2024-09-15T23:18:15.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad54aa740 20 client.15951 trim_cache size 0 max 16384 2024-09-15T23:18:15.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 -- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdab40a3240 msgr2=0x7fdab40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:15.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdab40a3240 0x7fdab40a5700 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fdabc005d50 tx=0x7fdabc005e90 comp rx=0 tx=0).stop 2024-09-15T23:18:15.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 -- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc102de0 msgr2=0x7fdacc1983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:15.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc102de0 0x7fdacc1983f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fdac400cce0 tx=0x7fdac400a620 comp rx=0 tx=0).stop 2024-09-15T23:18:15.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 -- 172.21.15.62:0/3390850953 shutdown_connections 2024-09-15T23:18:15.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdab40a3240 0x7fdab40a5700 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdacc10c180 0x7fdacc19ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdacc103700 0x7fdacc198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 --2- 172.21.15.62:0/3390850953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdacc102de0 0x7fdacc1983f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:15.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 -- 172.21.15.62:0/3390850953 >> 172.21.15.62:0/3390850953 conn(0x7fdacc0fc7a0 msgr2=0x7fdacc0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:15.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 -- 172.21.15.62:0/3390850953 shutdown_connections 2024-09-15T23:18:15.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:15.025+0000 7fdad30d1640 1 -- 172.21.15.62:0/3390850953 wait complete. 2024-09-15T23:18:15.771 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:18:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:15 smithi062 ceph-mon[30949]: pgmap v429: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 14 MiB/s wr, 523 op/s 2024-09-15T23:18:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:15 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:18:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:15 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:15 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:18:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:15 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:18:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:15 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:15 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:18:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:15 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:15.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:15 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:15 smithi067 ceph-mon[35665]: pgmap v429: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 14 MiB/s wr, 523 op/s 2024-09-15T23:18:16.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:15 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:18:16.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:15 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:15 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:18:16.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:15 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:18:16.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:15 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:15 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:18:16.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:15 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:15 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.030 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:15 smithi148 ceph-mon[35610]: pgmap v429: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 14 MiB/s wr, 523 op/s 2024-09-15T23:18:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:15 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:18:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:15 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:15 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:18:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:15 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:18:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:15 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:15 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:18:16.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:15 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:15 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.541 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:18:16.828 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:16 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:18:16.828 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:16 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:16.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.837+0000 7f4392a11640 1 -- 172.21.15.62:0/3085283607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c103740 msgr2=0x7f438c107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:16.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.837+0000 7f4392a11640 1 --2- 172.21.15.62:0/3085283607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c103740 0x7f438c107b80 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4380009930 tx=0x7f438002a420 comp rx=0 tx=0).stop 2024-09-15T23:18:16.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.837+0000 7f4392a11640 1 -- 172.21.15.62:0/3085283607 shutdown_connections 2024-09-15T23:18:16.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.837+0000 7f4392a11640 1 --2- 172.21.15.62:0/3085283607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f438c108290 0x7f438c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.837+0000 7f4392a11640 1 --2- 172.21.15.62:0/3085283607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c103740 0x7f438c107b80 secure :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f4380009930 tx=0x7f438002a420 comp rx=0 tx=0).stop 2024-09-15T23:18:16.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.837+0000 7f4392a11640 1 --2- 172.21.15.62:0/3085283607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c102d90 0x7f438c103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.837+0000 7f4392a11640 1 -- 172.21.15.62:0/3085283607 >> 172.21.15.62:0/3085283607 conn(0x7f438c0fc7d0 msgr2=0x7f438c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:16.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.838+0000 7f4392a11640 1 -- 172.21.15.62:0/3085283607 shutdown_connections 2024-09-15T23:18:16.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.838+0000 7f4392a11640 1 -- 172.21.15.62:0/3085283607 wait complete. 2024-09-15T23:18:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.838+0000 7f4392a11640 1 Processor -- start 2024-09-15T23:18:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.838+0000 7f4392a11640 1 -- start start 2024-09-15T23:18:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4392a11640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f438c102d90 0x7f438c1009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4392a11640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c108290 0x7f438c100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4392a11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c101470 0x7f438c1a6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4392a11640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f438c112090 con 0x7f438c101470 2024-09-15T23:18:16.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4392a11640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f438c111f10 con 0x7f438c108290 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4392a11640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f438c112210 con 0x7f438c102d90 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4390fae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c101470 0x7f438c1a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4390fae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c101470 0x7f438c1a6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40772/0 (socket says 172.21.15.62:40772) 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4390fae640 1 -- 172.21.15.62:0/851815160 learned_addr learned my addr 172.21.15.62:0/851815160 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f438bfff640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f438c102d90 0x7f438c1009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f438b7fe640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c108290 0x7f438c100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4390fae640 1 -- 172.21.15.62:0/851815160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f438c102d90 msgr2=0x7f438c1009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4390fae640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f438c102d90 0x7f438c1009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4390fae640 1 -- 172.21.15.62:0/851815160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c108290 msgr2=0x7f438c100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4390fae640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c108290 0x7f438c100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.839+0000 7f4390fae640 1 -- 172.21.15.62:0/851815160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438c1a7030 con 0x7f438c101470 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.840+0000 7f4390fae640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c101470 0x7f438c1a6960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f438400cce0 tx=0x7f438400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.840+0000 7f43897fa640 1 -- 172.21.15.62:0/851815160 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438400f450 con 0x7f438c101470 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.840+0000 7f43897fa640 1 -- 172.21.15.62:0/851815160 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f438400fad0 con 0x7f438c101470 2024-09-15T23:18:16.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.840+0000 7f4392a11640 1 -- 172.21.15.62:0/851815160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f438c1a7320 con 0x7f438c101470 2024-09-15T23:18:16.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.840+0000 7f43897fa640 1 -- 172.21.15.62:0/851815160 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4384018c10 con 0x7f438c101470 2024-09-15T23:18:16.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.840+0000 7f4392a11640 1 -- 172.21.15.62:0/851815160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f438c1a7830 con 0x7f438c101470 2024-09-15T23:18:16.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.842+0000 7f43897fa640 1 -- 172.21.15.62:0/851815160 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f438400f5f0 con 0x7f438c101470 2024-09-15T23:18:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f4394dea740 1 Processor -- start 2024-09-15T23:18:16.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f43897fa640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43740a32c0 0x7f43740a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f4394dea740 1 -- start start 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f43897fa640 1 -- 172.21.15.62:0/851815160 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f43840d01b0 con 0x7f438c101470 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f438b7fe640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43740a32c0 0x7f43740a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f4394dea740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a25880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f4394dea740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a30140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f4394dea740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a30680 0x55f432a32a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f4394dea740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f43298b2e0 con 0x55f432a25dc0 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f4394dea740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f432932ce0 con 0x55f432a254a0 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f4394dea740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f4329872a0 con 0x55f432a30680 2024-09-15T23:18:16.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.843+0000 7f438b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a30140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a30140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40774/0 (socket says 172.21.15.62:40774) 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a25880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438b7fe640 1 -- 172.21.15.62:0/287787177 learned_addr learned my addr 172.21.15.62:0/287787177 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f4390fae640 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a30680 0x55f432a32a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438b7fe640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43740a32c0 0x7f43740a5780 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4380009230 tx=0x7f43800315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438bfff640 1 -- 172.21.15.62:0/287787177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a30680 msgr2=0x55f432a32a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438bfff640 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a30680 0x55f432a32a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438bfff640 1 -- 172.21.15.62:0/287787177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 msgr2=0x55f432a30140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438bfff640 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a30140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438bfff640 1 -- 172.21.15.62:0/287787177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f432917440 con 0x55f432a254a0 2024-09-15T23:18:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438b7fe640 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a30140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f4390fae640 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a30680 0x55f432a32a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f438bfff640 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a25880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f437c00bbf0 tx=0x7f437c0103d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d1d03d7836f68 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.844+0000 7f43727fc640 1 -- 172.21.15.62:0/287787177 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c004030 con 0x55f432a254a0 2024-09-15T23:18:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f43727fc640 1 -- 172.21.15.62:0/287787177 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f437c0041d0 con 0x55f432a254a0 2024-09-15T23:18:16.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f43727fc640 1 -- 172.21.15.62:0/287787177 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c0116a0 con 0x55f432a254a0 2024-09-15T23:18:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f4394dea740 1 -- 172.21.15.62:0/287787177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 msgr2=0x55f432a25880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f4394dea740 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a25880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f437c00bbf0 tx=0x7f437c0103d0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f4394dea740 1 -- 172.21.15.62:0/287787177 shutdown_connections 2024-09-15T23:18:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f4394dea740 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a30680 0x55f432a32a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f4394dea740 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a30140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f4394dea740 1 --2- 172.21.15.62:0/287787177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a25880 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f4394dea740 1 -- 172.21.15.62:0/287787177 >> 172.21.15.62:0/287787177 conn(0x55f432a21a30 msgr2=0x55f432a21e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f4394dea740 1 -- 172.21.15.62:0/287787177 shutdown_connections 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.845+0000 7f4394dea740 1 -- 172.21.15.62:0/287787177 wait complete. 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f4394dea740 1 -- start start 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f4394dea740 1 Processor -- start 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f4394dea740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a2ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f4394dea740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a25dc0 0x55f432a2d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f4394dea740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a30680 0x55f432a2d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f4394dea740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f432a33250 con 0x55f432a254a0 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f4394dea740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f432946050 con 0x55f432a30680 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f4394dea740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f432a333d0 con 0x55f432a25dc0 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f4394dea740 10 client.0 _ll_register_callbacks cb 0x55f432986160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:16.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f438bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a2ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.846+0000 7f438bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a2ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40776/0 (socket says 172.21.15.62:40776) 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f438bfff640 1 -- 172.21.15.62:0/192142698 learned_addr learned my addr 172.21.15.62:0/192142698 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f438b7fe640 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a25dc0 0x55f432a2d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f4390fae640 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a30680 0x55f432a2d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f438bfff640 1 -- 172.21.15.62:0/192142698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a25dc0 msgr2=0x55f432a2d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f438bfff640 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a25dc0 0x55f432a2d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f438bfff640 1 -- 172.21.15.62:0/192142698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a30680 msgr2=0x55f432a2d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f438bfff640 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a30680 0x55f432a2d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f438bfff640 1 -- 172.21.15.62:0/192142698 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4326fc3c0 con 0x55f432a254a0 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f4390fae640 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a30680 0x55f432a2d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f438b7fe640 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a25dc0 0x55f432a2d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f438bfff640 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a2ff60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f437c005890 tx=0x7f437c0045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89974b7a31953db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f436b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:16.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f436b7fe640 1 -- 172.21.15.62:0/192142698 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c02a060 con 0x55f432a254a0 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f4394dea740 1 -- 172.21.15.62:0/192142698 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f437c0041d0 con 0x55f432a254a0 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.847+0000 7f436b7fe640 1 -- 172.21.15.62:0/192142698 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f437c010cc0 con 0x55f432a254a0 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f436b7fe640 1 -- 172.21.15.62:0/192142698 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f437c031050 con 0x55f432a254a0 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f4394dea740 10 client.15963 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f4394dea740 1 -- 172.21.15.62:0/192142698 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f432a3ec20 con 0x55f432a254a0 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f436b7fe640 1 -- 172.21.15.62:0/192142698 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f437c0041d0 con 0x55f432a254a0 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f436b7fe640 1 -- 172.21.15.62:0/192142698 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f437c004a30 con 0x55f432a254a0 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f4394dea740 10 client.15963 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f4394dea740 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f4394dea740 20 client.15963 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f4394dea740 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f432a40030 0x55f432a42420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f4394dea740 4 client.15963 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f4394dea740 1 -- 172.21.15.62:0/192142698 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f432998850 con 0x55f432a40030 2024-09-15T23:18:16.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.848+0000 7f438b7fe640 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f432a40030 0x55f432a42420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.849+0000 7f438b7fe640 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f432a40030 0x55f432a42420 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdc862a5866a97e9 server_cookie=646b7af086fc4a45 in_seq=0 out_seq=0 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.849+0000 7f436b7fe640 10 client.15963 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f436b7fe640 1 -- 172.21.15.62:0/192142698 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f432998850 con 0x55f432a40030 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f436b7fe640 10 client.15963 handle_command_reply: tid=0 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 client.15963 shutdown 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 -- 172.21.15.62:0/192142698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 msgr2=0x55f432a2ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a2ff60 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f437c005890 tx=0x7f437c0045b0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 -- 172.21.15.62:0/192142698 shutdown_connections 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f432a40030 0x55f432a42420 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a30680 0x55f432a2d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a25dc0 0x55f432a2d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 --2- 172.21.15.62:0/192142698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a2ff60 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 -- 172.21.15.62:0/192142698 >> 172.21.15.62:0/192142698 conn(0x55f432a21a30 msgr2=0x55f432a27760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:16.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f436b7fe640 0 client.15963 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:16.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 -- 172.21.15.62:0/192142698 shutdown_connections 2024-09-15T23:18:16.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 1 -- 172.21.15.62:0/192142698 wait complete. 2024-09-15T23:18:16.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.850+0000 7f4394dea740 20 client.15963 trim_cache size 0 max 16384 2024-09-15T23:18:16.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4394dea740 1 Processor -- start 2024-09-15T23:18:16.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4394dea740 1 -- start start 2024-09-15T23:18:16.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4394dea740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a3c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4394dea740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a28900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4394dea740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a2c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4394dea740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f438c1b3d50 con 0x55f432a25dc0 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4394dea740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f438c1b15f0 con 0x55f432a254a0 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4394dea740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f438c1af440 con 0x55f432a21a30 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4390fae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a2c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4390fae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a2c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40786/0 (socket says 172.21.15.62:40786) 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f4390fae640 1 -- 172.21.15.62:0/3594156940 learned_addr learned my addr 172.21.15.62:0/3594156940 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.881+0000 7f438b7fe640 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a28900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f438bfff640 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a3c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f438b7fe640 1 -- 172.21.15.62:0/3594156940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 msgr2=0x55f432a3c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f438b7fe640 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a3c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f438b7fe640 1 -- 172.21.15.62:0/3594156940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 msgr2=0x55f432a2c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:16.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f438b7fe640 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a2c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f438b7fe640 1 -- 172.21.15.62:0/3594156940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f432a214d0 con 0x55f432a254a0 2024-09-15T23:18:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f438bfff640 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a3c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f4390fae640 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a2c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f438b7fe640 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a28900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4380005940 tx=0x7f4380047000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ba020e6cd20ca4c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f4371ffb640 1 -- 172.21.15.62:0/3594156940 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4380035040 con 0x55f432a254a0 2024-09-15T23:18:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f4371ffb640 1 -- 172.21.15.62:0/3594156940 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4380004490 con 0x55f432a254a0 2024-09-15T23:18:16.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.882+0000 7f4371ffb640 1 -- 172.21.15.62:0/3594156940 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43800035f0 con 0x55f432a254a0 2024-09-15T23:18:16.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.883+0000 7f4394dea740 1 -- 172.21.15.62:0/3594156940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 msgr2=0x55f432a28900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:16.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.883+0000 7f4394dea740 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a28900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4380005940 tx=0x7f4380047000 comp rx=0 tx=0).stop 2024-09-15T23:18:16.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.883+0000 7f4394dea740 1 -- 172.21.15.62:0/3594156940 shutdown_connections 2024-09-15T23:18:16.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.883+0000 7f4394dea740 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a25dc0 0x55f432a2c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.883+0000 7f4394dea740 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a254a0 0x55f432a28900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.883+0000 7f4394dea740 1 --2- 172.21.15.62:0/3594156940 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a3c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.883+0000 7f4394dea740 1 -- 172.21.15.62:0/3594156940 >> 172.21.15.62:0/3594156940 conn(0x55f432a2d8b0 msgr2=0x55f432a2b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.883+0000 7f4394dea740 1 -- 172.21.15.62:0/3594156940 shutdown_connections 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.883+0000 7f4394dea740 1 -- 172.21.15.62:0/3594156940 wait complete. 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f4394dea740 1 -- start start 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f4394dea740 1 Processor -- start 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f4394dea740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a33b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f4394dea740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a34080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f4394dea740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a25dc0 0x55f432aa8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f4394dea740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f438c1afab0 con 0x55f432a254a0 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f4394dea740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f438c1b0f20 con 0x55f432a25dc0 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f4394dea740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f438c1b0180 con 0x55f432a21a30 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f4394dea740 10 client.0 _ll_register_callbacks cb 0x7f43840077a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f438b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a34080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f438b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a34080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40802/0 (socket says 172.21.15.62:40802) 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f438b7fe640 1 -- 172.21.15.62:0/3486165108 learned_addr learned my addr 172.21.15.62:0/3486165108 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.884+0000 7f438bfff640 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a33b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f4390fae640 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a25dc0 0x55f432aa8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f438bfff640 1 -- 172.21.15.62:0/3486165108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a25dc0 msgr2=0x55f432aa8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f438bfff640 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a25dc0 0x55f432aa8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f438bfff640 1 -- 172.21.15.62:0/3486165108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 msgr2=0x55f432a34080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f438bfff640 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a34080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f438bfff640 1 -- 172.21.15.62:0/3486165108 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f432aa8f50 con 0x55f432a21a30 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f438b7fe640 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a34080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f438bfff640 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a33b40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f437c000c00 tx=0x7f437c003c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd2750934b9c9af2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f436b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f436b7fe640 1 -- 172.21.15.62:0/3486165108 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437c00ba40 con 0x55f432a21a30 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f4394dea740 1 -- 172.21.15.62:0/3486165108 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4380004490 con 0x55f432a21a30 2024-09-15T23:18:16.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f436b7fe640 1 -- 172.21.15.62:0/3486165108 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f437c0040a0 con 0x55f432a21a30 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.885+0000 7f436b7fe640 1 -- 172.21.15.62:0/3486165108 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f437c010cc0 con 0x55f432a21a30 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f4394dea740 10 client.25483 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f4394dea740 1 -- 172.21.15.62:0/3486165108 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f432aaa1e0 con 0x55f432a21a30 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f436b7fe640 1 -- 172.21.15.62:0/3486165108 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4380004490 con 0x55f432a21a30 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f436b7fe640 1 -- 172.21.15.62:0/3486165108 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f437c011930 con 0x55f432a21a30 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f4394dea740 10 client.25483 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f4394dea740 10 client.25483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f4394dea740 20 client.25483 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f4394dea740 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f432a30680 0x55f432aac940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f4394dea740 4 client.25483 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f4394dea740 1 -- 172.21.15.62:0/3486165108 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f432992d60 con 0x55f432a30680 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.886+0000 7f438b7fe640 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f432a30680 0x55f432aac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:16.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f438b7fe640 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f432a30680 0x55f432aac940 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=603038cefb03c219 server_cookie=e54c316f4555d46c in_seq=0 out_seq=0 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f436b7fe640 10 client.25483 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f436b7fe640 1 -- 172.21.15.62:0/3486165108 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f432992d60 con 0x55f432a30680 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f436b7fe640 10 client.25483 handle_command_reply: tid=0 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 client.25483 shutdown 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 -- 172.21.15.62:0/3486165108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 msgr2=0x55f432a33b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a33b40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f437c000c00 tx=0x7f437c003c90 comp rx=0 tx=0).stop 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 -- 172.21.15.62:0/3486165108 shutdown_connections 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f432a30680 0x55f432aac940 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f432a25dc0 0x55f432aa8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f432a254a0 0x55f432a34080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f436b7fe640 0 client.25483 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 --2- 172.21.15.62:0/3486165108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f432a21a30 0x55f432a33b40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 -- 172.21.15.62:0/3486165108 >> 172.21.15.62:0/3486165108 conn(0x55f432a2d8b0 msgr2=0x55f432a287f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:16.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 -- 172.21.15.62:0/3486165108 shutdown_connections 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 1 -- 172.21.15.62:0/3486165108 wait complete. 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.887+0000 7f4394dea740 20 client.25483 trim_cache size 0 max 16384 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout: "6b4b943c-4a01-4ae3-94b0-d5a898040e81": { 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "6b4b943c-4a01-4ae3-94b0-d5a898040e81", 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:16.896 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:18:16.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 -- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43740a32c0 msgr2=0x7f43740a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:16.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43740a32c0 0x7f43740a5780 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4380009230 tx=0x7f43800315b0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 -- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c101470 msgr2=0x7f438c1a6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:16.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c101470 0x7f438c1a6960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f438400cce0 tx=0x7f438400a620 comp rx=0 tx=0).stop 2024-09-15T23:18:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 -- 172.21.15.62:0/851815160 shutdown_connections 2024-09-15T23:18:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43740a32c0 0x7f43740a5780 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f438c101470 0x7f438c1a6960 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f438c108290 0x7f438c100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 --2- 172.21.15.62:0/851815160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f438c102d90 0x7f438c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:16.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 -- 172.21.15.62:0/851815160 >> 172.21.15.62:0/851815160 conn(0x7f438c0fc7d0 msgr2=0x7f438c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:16.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 -- 172.21.15.62:0/851815160 shutdown_connections 2024-09-15T23:18:16.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:16.888+0000 7f4392a11640 1 -- 172.21.15.62:0/851815160 wait complete. 2024-09-15T23:18:17.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:16 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:18:17.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:16 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:17.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:16 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:18:17.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:16 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:18:17.561 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.559+0000 7f84d0a53640 1 -- 172.21.15.62:0/3062171586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc102620 msgr2=0x7f84cc104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.559+0000 7f84d0a53640 1 --2- 172.21.15.62:0/3062171586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc102620 0x7f84cc104a10 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009900 tx=0x7f84c402a410 comp rx=0 tx=0).stop 2024-09-15T23:18:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.559+0000 7f84d0a53640 1 -- 172.21.15.62:0/3062171586 shutdown_connections 2024-09-15T23:18:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.559+0000 7f84d0a53640 1 --2- 172.21.15.62:0/3062171586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84cc107880 0x7f84cc109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.559+0000 7f84d0a53640 1 --2- 172.21.15.62:0/3062171586 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84cc104f50 0x7f84cc107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.559+0000 7f84d0a53640 1 --2- 172.21.15.62:0/3062171586 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc102620 0x7f84cc104a10 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.559+0000 7f84d0a53640 1 -- 172.21.15.62:0/3062171586 >> 172.21.15.62:0/3062171586 conn(0x7f84cc0fc1e0 msgr2=0x7f84cc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.559+0000 7f84d0a53640 1 -- 172.21.15.62:0/3062171586 shutdown_connections 2024-09-15T23:18:17.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.559+0000 7f84d0a53640 1 -- 172.21.15.62:0/3062171586 wait complete. 2024-09-15T23:18:17.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84d0a53640 1 Processor -- start 2024-09-15T23:18:17.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84d0a53640 1 -- start start 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84d0a53640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84cc102620 0x7f84cc197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84d0a53640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84cc104f50 0x7f84cc198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84d0a53640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc107880 0x7f84cc19f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84d0a53640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc10d880 con 0x7f84cc107880 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84d0a53640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc10d700 con 0x7f84cc102620 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84d0a53640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc10da00 con 0x7f84cc104f50 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84cad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc107880 0x7f84cc19f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84cad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc107880 0x7f84cc19f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40828/0 (socket says 172.21.15.62:40828) 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84cad9d640 1 -- 172.21.15.62:0/1645661272 learned_addr learned my addr 172.21.15.62:0/1645661272 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84ca59c640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84cc102620 0x7f84cc197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84cc104f50 0x7f84cc198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84cad9d640 1 -- 172.21.15.62:0/1645661272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84cc104f50 msgr2=0x7f84cc198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:17.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84cad9d640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84cc104f50 0x7f84cc198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84cad9d640 1 -- 172.21.15.62:0/1645661272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84cc102620 msgr2=0x7f84cc197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84cad9d640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84cc102620 0x7f84cc197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84cad9d640 1 -- 172.21.15.62:0/1645661272 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84cc19fb00 con 0x7f84cc107880 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84ca59c640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84cc102620 0x7f84cc197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84cc104f50 0x7f84cc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.560+0000 7f84cad9d640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc107880 0x7f84cc19f490 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00e9d0 tx=0x7f84bc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.561+0000 7f84bb7fe640 1 -- 172.21.15.62:0/1645661272 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc00cd80 con 0x7f84cc107880 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.561+0000 7f84bb7fe640 1 -- 172.21.15.62:0/1645661272 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84bc005020 con 0x7f84cc107880 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.561+0000 7f84d0a53640 1 -- 172.21.15.62:0/1645661272 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84cc19fdf0 con 0x7f84cc107880 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.561+0000 7f84bb7fe640 1 -- 172.21.15.62:0/1645661272 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0116d0 con 0x7f84cc107880 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.561+0000 7f84d0a53640 1 -- 172.21.15.62:0/1645661272 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84cc1a0330 con 0x7f84cc107880 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.564+0000 7f84d37ec740 1 Processor -- start 2024-09-15T23:18:17.856 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.564+0000 7f84bb7fe640 1 -- 172.21.15.62:0/1645661272 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f84bc0051d0 con 0x7f84cc107880 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.564+0000 7f84d37ec740 1 -- start start 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.564+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86066da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.564+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe8606bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.564+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe8606e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.564+0000 7f84d37ec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fe85fca2f0 con 0x55fe8606c370 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.564+0000 7f84d37ec740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe85f72270 con 0x55fe860669a0 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.564+0000 7f84d37ec740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55fe85fc62b0 con 0x55fe860672e0 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe8606e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe8606e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40834/0 (socket says 172.21.15.62:40834) 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 -- 172.21.15.62:0/3825718349 learned_addr learned my addr 172.21.15.62:0/3825718349 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86066da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84bb7fe640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f84b40a3370 0x7f84b40a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.857 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84ca59c640 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe8606bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 -- 172.21.15.62:0/3825718349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 msgr2=0x55fe8606bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe8606bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 -- 172.21.15.62:0/3825718349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 msgr2=0x55fe86066da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86066da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 -- 172.21.15.62:0/3825718349 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe85f71340 con 0x55fe8606c370 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84ca59c640 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe8606bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84cad9d640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f84b40a3370 0x7f84b40a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe8606e830 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f84c000ae40 tx=0x7f84c000a4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea17530d9a41aca8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84bb7fe640 1 -- 172.21.15.62:0/1645661272 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f84bc0d02a0 con 0x7f84cc107880 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84b8ff9640 1 -- 172.21.15.62:0/3825718349 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c001c070 con 0x55fe8606c370 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84b8ff9640 1 -- 172.21.15.62:0/3825718349 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c000fd70 con 0x55fe8606c370 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.565+0000 7f84b8ff9640 1 -- 172.21.15.62:0/3825718349 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c00147b0 con 0x55fe8606c370 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 -- 172.21.15.62:0/3825718349 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 msgr2=0x55fe8606e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:17.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe8606e830 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f84c000ae40 tx=0x7f84c000a4f0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84cad9d640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f84b40a3370 0x7f84b40a5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0989a0 tx=0x7f84bc06b280 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 -- 172.21.15.62:0/3825718349 shutdown_connections 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe8606e830 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe8606bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 --2- 172.21.15.62:0/3825718349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86066da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 -- 172.21.15.62:0/3825718349 >> 172.21.15.62:0/3825718349 conn(0x55fe86062010 msgr2=0x55fe86062410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 -- 172.21.15.62:0/3825718349 shutdown_connections 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 -- 172.21.15.62:0/3825718349 wait complete. 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.566+0000 7f84d37ec740 1 -- start start 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84d37ec740 1 Processor -- start 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86070c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe86071140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe86078180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84d37ec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fe8606f340 con 0x55fe8606c370 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84d37ec740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe85f53290 con 0x55fe860669a0 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84d37ec740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55fe8606f4c0 con 0x55fe860672e0 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84d37ec740 10 client.0 _ll_register_callbacks cb 0x55fe85f84140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe86078180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe86078180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40850/0 (socket says 172.21.15.62:40850) 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 -- 172.21.15.62:0/1409461623 learned_addr learned my addr 172.21.15.62:0/1409461623 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86070c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84ca59c640 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe86071140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 -- 172.21.15.62:0/1409461623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 msgr2=0x55fe86071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe86071140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 -- 172.21.15.62:0/1409461623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 msgr2=0x55fe86070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:17.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86070c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 -- 172.21.15.62:0/1409461623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe85cdd210 con 0x55fe8606c370 2024-09-15T23:18:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84ca59c640 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe86071140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe86078180 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f84c0017eb0 tx=0x7f84c0006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f9a3a9b8eabc57 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.567+0000 7f84b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84b1ffb640 1 -- 172.21.15.62:0/1409461623 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c002a060 con 0x55fe8606c370 2024-09-15T23:18:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84b1ffb640 1 -- 172.21.15.62:0/1409461623 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f84c0014de0 con 0x55fe8606c370 2024-09-15T23:18:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84d37ec740 1 -- 172.21.15.62:0/1409461623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84c000fd70 con 0x55fe8606c370 2024-09-15T23:18:17.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84b1ffb640 1 -- 172.21.15.62:0/1409461623 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f84c0031050 con 0x55fe8606c370 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84d37ec740 10 client.15981 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84d37ec740 1 -- 172.21.15.62:0/1409461623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fe85da9620 con 0x55fe8606c370 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84b1ffb640 1 -- 172.21.15.62:0/1409461623 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c000fd70 con 0x55fe8606c370 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84b1ffb640 1 -- 172.21.15.62:0/1409461623 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f84c0022070 con 0x55fe8606c370 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84d37ec740 10 client.15981 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84d37ec740 10 client.15981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84d37ec740 20 client.15981 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84d37ec740 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe8607c4b0 0x55fe8607e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84d37ec740 4 client.15981 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.568+0000 7f84d37ec740 1 -- 172.21.15.62:0/1409461623 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe85fd7a20 con 0x55fe8607c4b0 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.569+0000 7f84ca59c640 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe8607c4b0 0x55fe8607e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.569+0000 7f84ca59c640 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe8607c4b0 0x55fe8607e8a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ee60afaa61f057f server_cookie=8659df66d34e5c7b in_seq=0 out_seq=0 2024-09-15T23:18:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.569+0000 7f84b1ffb640 10 client.15981 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84b1ffb640 1 -- 172.21.15.62:0/1409461623 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe85fd7a20 con 0x55fe8607c4b0 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84b1ffb640 10 client.15981 handle_command_reply: tid=0 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 client.15981 shutdown 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 -- 172.21.15.62:0/1409461623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 msgr2=0x55fe86078180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe86078180 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f84c0017eb0 tx=0x7f84c0006f90 comp rx=0 tx=0).stop 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 -- 172.21.15.62:0/1409461623 shutdown_connections 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe8607c4b0 0x55fe8607e8a0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe8606c370 0x55fe86078180 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe860672e0 0x55fe86071140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84b1ffb640 0 client.15981 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 --2- 172.21.15.62:0/1409461623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86070c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 -- 172.21.15.62:0/1409461623 >> 172.21.15.62:0/1409461623 conn(0x55fe86062010 msgr2=0x55fe86063820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 -- 172.21.15.62:0/1409461623 shutdown_connections 2024-09-15T23:18:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 1 -- 172.21.15.62:0/1409461623 wait complete. 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.570+0000 7f84d37ec740 20 client.15981 trim_cache size 0 max 16384 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84d37ec740 1 Processor -- start 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84d37ec740 1 -- start start 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86062f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860672e0 0x55fe86063460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe8606c370 0x55fe860639a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84d37ec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1ad110 con 0x55fe860672e0 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84d37ec740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1aa9b0 con 0x55fe860669a0 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84d37ec740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1a8800 con 0x55fe8606c370 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84ca59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860672e0 0x55fe86063460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84ca59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860672e0 0x55fe86063460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40862/0 (socket says 172.21.15.62:40862) 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84ca59c640 1 -- 172.21.15.62:0/2555451749 learned_addr learned my addr 172.21.15.62:0/2555451749 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86062f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe8606c370 0x55fe860639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84ca59c640 1 -- 172.21.15.62:0/2555451749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe8606c370 msgr2=0x55fe860639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84ca59c640 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe8606c370 0x55fe860639a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84ca59c640 1 -- 172.21.15.62:0/2555451749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 msgr2=0x55fe86062f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84ca59c640 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86062f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84ca59c640 1 -- 172.21.15.62:0/2555451749 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe8606c110 con 0x55fe860672e0 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.585+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86062f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.586+0000 7f84ca59c640 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860672e0 0x55fe86063460 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f84c4032fd0 tx=0x7f84c4032990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc96f077c0d4999 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.586+0000 7f84b3fff640 1 -- 172.21.15.62:0/2555451749 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4049070 con 0x55fe860672e0 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.586+0000 7f84b3fff640 1 -- 172.21.15.62:0/2555451749 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c4005040 con 0x55fe860672e0 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.586+0000 7f84b3fff640 1 -- 172.21.15.62:0/2555451749 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c4035040 con 0x55fe860672e0 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.586+0000 7f84d37ec740 1 -- 172.21.15.62:0/2555451749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860672e0 msgr2=0x55fe86063460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.586+0000 7f84d37ec740 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860672e0 0x55fe86063460 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f84c4032fd0 tx=0x7f84c4032990 comp rx=0 tx=0).stop 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 -- 172.21.15.62:0/2555451749 shutdown_connections 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe8606c370 0x55fe860639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860672e0 0x55fe86063460 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 --2- 172.21.15.62:0/2555451749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860669a0 0x55fe86062f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 -- 172.21.15.62:0/2555451749 >> 172.21.15.62:0/2555451749 conn(0x55fe86062010 msgr2=0x55fe86070c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 -- 172.21.15.62:0/2555451749 shutdown_connections 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 -- 172.21.15.62:0/2555451749 wait complete. 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 -- start start 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 Processor -- start 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860669a0 0x55fe860e21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860672e0 0x55fe860e2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe8606c370 0x55fe860e9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1a8e70 con 0x55fe860669a0 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1aa2e0 con 0x55fe860672e0 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f84cc1a9540 con 0x55fe8606c370 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84d37ec740 10 client.0 _ll_register_callbacks cb 0x7f84c001a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84c9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860669a0 0x55fe860e21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84c9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860669a0 0x55fe860e21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40866/0 (socket says 172.21.15.62:40866) 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84c9d9b640 1 -- 172.21.15.62:0/280354572 learned_addr learned my addr 172.21.15.62:0/280354572 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.587+0000 7f84ca59c640 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860672e0 0x55fe860e2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84c9d9b640 1 -- 172.21.15.62:0/280354572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe8606c370 msgr2=0x55fe860e9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe8606c370 0x55fe860e9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84c9d9b640 1 -- 172.21.15.62:0/280354572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860672e0 msgr2=0x55fe860e2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860672e0 0x55fe860e2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84c9d9b640 1 -- 172.21.15.62:0/280354572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe860e9d50 con 0x55fe860669a0 2024-09-15T23:18:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84c9d9b640 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860669a0 0x55fe860e21c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f84c0006f60 tx=0x7f84c0023ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557d28c1c20a6c5e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84b1ffb640 1 -- 172.21.15.62:0/280354572 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c003b070 con 0x55fe860669a0 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84b1ffb640 1 -- 172.21.15.62:0/280354572 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f84c0023610 con 0x55fe860669a0 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84d37ec740 1 -- 172.21.15.62:0/280354572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84c4005040 con 0x55fe860669a0 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84b1ffb640 1 -- 172.21.15.62:0/280354572 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f84c0046050 con 0x55fe860669a0 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84d37ec740 10 client.15987 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84d37ec740 1 -- 172.21.15.62:0/280354572 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fe860eb0d0 con 0x55fe860669a0 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84b1ffb640 1 -- 172.21.15.62:0/280354572 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c4005040 con 0x55fe860669a0 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84b1ffb640 1 -- 172.21.15.62:0/280354572 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f84c0034040 con 0x55fe860669a0 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84d37ec740 10 client.15987 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84d37ec740 10 client.15987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84d37ec740 20 client.15987 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84d37ec740 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe860ebac0 0x55fe860edee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84d37ec740 4 client.15987 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.588+0000 7f84d37ec740 1 -- 172.21.15.62:0/280354572 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fe85fd1d70 con 0x55fe860ebac0 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.589+0000 7f84ca59c640 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe860ebac0 0x55fe860edee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.589+0000 7f84ca59c640 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe860ebac0 0x55fe860edee0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ed72e58c6ecbb14 server_cookie=1e93ca53774197a7 in_seq=0 out_seq=0 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.589+0000 7f84b1ffb640 10 client.15987 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.601+0000 7f84b1ffb640 1 -- 172.21.15.62:0/280354572 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fe85fd1d70 con 0x55fe860ebac0 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.601+0000 7f84b1ffb640 10 client.15987 handle_command_reply: tid=0 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.601+0000 7f84d37ec740 1 client.15987 shutdown 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 -- 172.21.15.62:0/280354572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860669a0 msgr2=0x55fe860e21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860669a0 0x55fe860e21c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f84c0006f60 tx=0x7f84c0023ee0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 -- 172.21.15.62:0/280354572 shutdown_connections 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe860ebac0 0x55fe860edee0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe8606c370 0x55fe860e9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84b1ffb640 0 client.15987 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe860672e0 0x55fe860e2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 --2- 172.21.15.62:0/280354572 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe860669a0 0x55fe860e21c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 -- 172.21.15.62:0/280354572 >> 172.21.15.62:0/280354572 conn(0x55fe86062010 msgr2=0x55fe860609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 -- 172.21.15.62:0/280354572 shutdown_connections 2024-09-15T23:18:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 1 -- 172.21.15.62:0/280354572 wait complete. 2024-09-15T23:18:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d37ec740 20 client.15987 trim_cache size 0 max 16384 2024-09-15T23:18:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:18:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:18:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:18:17.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:18:17.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 -- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f84b40a3370 msgr2=0x7f84b40a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:17.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f84b40a3370 0x7f84b40a5830 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0989a0 tx=0x7f84bc06b280 comp rx=0 tx=0).stop 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 -- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc107880 msgr2=0x7f84cc19f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc107880 0x7f84cc19f490 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00e9d0 tx=0x7f84bc00ee90 comp rx=0 tx=0).stop 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 -- 172.21.15.62:0/1645661272 shutdown_connections 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f84b40a3370 0x7f84b40a5830 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84cc107880 0x7f84cc19f490 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f84cc104f50 0x7f84cc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 --2- 172.21.15.62:0/1645661272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84cc102620 0x7f84cc197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 -- 172.21.15.62:0/1645661272 >> 172.21.15.62:0/1645661272 conn(0x7f84cc0fc1e0 msgr2=0x7f84cc103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.602+0000 7f84d0a53640 1 -- 172.21.15.62:0/1645661272 shutdown_connections 2024-09-15T23:18:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:17.603+0000 7f84d0a53640 1 -- 172.21.15.62:0/1645661272 wait complete. 2024-09-15T23:18:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:17 smithi062 ceph-mon[30949]: pgmap v430: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 464 op/s 2024-09-15T23:18:18.015 INFO:tasks.cephfs.filesystem:scrub status for tag:6b4b943c-4a01-4ae3-94b0-d5a898040e81 - {'path': '/', 'tag': '6b4b943c-4a01-4ae3-94b0-d5a898040e81', 'options': 'recursive,force'} 2024-09-15T23:18:18.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:17 smithi067 ceph-mon[35665]: pgmap v430: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 464 op/s 2024-09-15T23:18:18.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:17 smithi148 ceph-mon[35610]: pgmap v430: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 13 MiB/s wr, 464 op/s 2024-09-15T23:18:18.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.210+0000 7f09884fd640 1 -- 172.21.15.62:0/55574593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 msgr2=0x7f0980107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:18.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.210+0000 7f09884fd640 1 --2- 172.21.15.62:0/55574593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 0x7f0980107b90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0974009930 tx=0x7f097402a420 comp rx=0 tx=0).stop 2024-09-15T23:18:18.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 -- 172.21.15.62:0/55574593 shutdown_connections 2024-09-15T23:18:18.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 --2- 172.21.15.62:0/55574593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09801082a0 0x7f098010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 --2- 172.21.15.62:0/55574593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 0x7f0980107b90 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 --2- 172.21.15.62:0/55574593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0980102d30 0x7f0980103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 -- 172.21.15.62:0/55574593 >> 172.21.15.62:0/55574593 conn(0x7f09800fc7d0 msgr2=0x7f09800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 -- 172.21.15.62:0/55574593 shutdown_connections 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 -- 172.21.15.62:0/55574593 wait complete. 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 Processor -- start 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 -- start start 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0980102d30 0x7f0980100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.211+0000 7f09884fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 0x7f0980100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f09884fd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09801082a0 0x7f0980101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f09884fd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f09801120c0 con 0x7f09801036e0 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f09884fd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0980111f40 con 0x7f0980102d30 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f09884fd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0980112240 con 0x7f09801082a0 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0985a98640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 0x7f0980100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0985a98640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 0x7f0980100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40888/0 (socket says 172.21.15.62:40888) 2024-09-15T23:18:18.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0985a98640 1 -- 172.21.15.62:0/1537578767 learned_addr learned my addr 172.21.15.62:0/1537578767 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0986299640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0980102d30 0x7f0980100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0986a9a640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09801082a0 0x7f0980101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0985a98640 1 -- 172.21.15.62:0/1537578767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09801082a0 msgr2=0x7f0980101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0985a98640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09801082a0 0x7f0980101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0985a98640 1 -- 172.21.15.62:0/1537578767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0980102d30 msgr2=0x7f0980100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0985a98640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0980102d30 0x7f0980100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0985a98640 1 -- 172.21.15.62:0/1537578767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09801a6ae0 con 0x7f09801036e0 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0986299640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0980102d30 0x7f0980100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f0985a98640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 0x7f0980100ed0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0974005940 tx=0x7f09740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f097f7fe640 1 -- 172.21.15.62:0/1537578767 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097403f030 con 0x7f09801036e0 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09801a6d70 con 0x7f09801036e0 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.212+0000 7f097f7fe640 1 -- 172.21.15.62:0/1537578767 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0974003ce0 con 0x7f09801036e0 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.213+0000 7f097f7fe640 1 -- 172.21.15.62:0/1537578767 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974035070 con 0x7f09801036e0 2024-09-15T23:18:18.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.213+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09801a71f0 con 0x7f09801036e0 2024-09-15T23:18:18.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.213+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0980103110 con 0x7f09801036e0 2024-09-15T23:18:18.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.215+0000 7f097f7fe640 1 -- 172.21.15.62:0/1537578767 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0974042030 con 0x7f09801036e0 2024-09-15T23:18:18.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.215+0000 7f097f7fe640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09680a3240 0x7f09680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:18.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.216+0000 7f0986299640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09680a3240 0x7f09680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:18.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.216+0000 7f097f7fe640 1 -- 172.21.15.62:0/1537578767 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f09740f02d0 con 0x7f09801036e0 2024-09-15T23:18:18.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.216+0000 7f0986299640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09680a3240 0x7f09680a5700 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f097800bfd0 tx=0x7f097800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:18.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.219+0000 7f097f7fe640 1 -- 172.21.15.62:0/1537578767 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f09740bc620 con 0x7f09801036e0 2024-09-15T23:18:18.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.429+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f098010a2b0 con 0x7f09801036e0 2024-09-15T23:18:18.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.430+0000 7f097f7fe640 1 -- 172.21.15.62:0/1537578767 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11907 (secure 0 0 0) 0x7f09740dc020 con 0x7f09801036e0 2024-09-15T23:18:18.432 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:18:18.432 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":28,"btime":"2024-09-15T23:17:33:092139+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:17:32.873645+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:18:18.433 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 28 2024-09-15T23:18:18.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09680a3240 msgr2=0x7f09680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:18.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09680a3240 0x7f09680a5700 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f097800bfd0 tx=0x7f097800b390 comp rx=0 tx=0).stop 2024-09-15T23:18:18.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 msgr2=0x7f0980100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:18.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 0x7f0980100ed0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0974005940 tx=0x7f09740042f0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 shutdown_connections 2024-09-15T23:18:18.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09680a3240 0x7f09680a5700 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f09801082a0 0x7f0980101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09801036e0 0x7f0980100ed0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 --2- 172.21.15.62:0/1537578767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0980102d30 0x7f0980100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:18.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 >> 172.21.15.62:0/1537578767 conn(0x7f09800fc7d0 msgr2=0x7f0980105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:18.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 shutdown_connections 2024-09-15T23:18:18.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:18.432+0000 7f09884fd640 1 -- 172.21.15.62:0/1537578767 wait complete. 2024-09-15T23:18:18.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:18 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1537578767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:18 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1537578767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:18 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1537578767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:19.096 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 3956db1e --release --await' 2024-09-15T23:18:19.097 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 3956db1e --release --await 2024-09-15T23:18:19.353 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:19.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 120.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:18:19.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:19 smithi062 ceph-mon[30949]: pgmap v431: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 13 MiB/s wr, 486 op/s 2024-09-15T23:18:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:19 smithi067 ceph-mon[35665]: pgmap v431: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 13 MiB/s wr, 486 op/s 2024-09-15T23:18:19.940 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:19 smithi148 ceph-mon[35610]: pgmap v431: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 13 MiB/s wr, 486 op/s 2024-09-15T23:18:19.980 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:18:19.980 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 28, 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 8, 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "3956db1e": { 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 8, 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:18:19.981 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 139.6, 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:19.982 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:18:19.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.900+0000 7efdc3fd7640 1 -- 172.21.15.62:0/1785326292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 msgr2=0x7efdbc104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:19.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.900+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/1785326292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 0x7efdbc104e80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7efdb000aa10 tx=0x7efdb002b720 comp rx=0 tx=0).stop 2024-09-15T23:18:19.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.900+0000 7efdc3fd7640 1 -- 172.21.15.62:0/1785326292 shutdown_connections 2024-09-15T23:18:19.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.900+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/1785326292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdbc1055b0 0x7efdbc10d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.900+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/1785326292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 0x7efdbc104e80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.900+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/1785326292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdbc100480 0x7efdbc100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.900+0000 7efdc3fd7640 1 -- 172.21.15.62:0/1785326292 >> 172.21.15.62:0/1785326292 conn(0x7efdbc077210 msgr2=0x7efdbc0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.900+0000 7efdc3fd7640 1 -- 172.21.15.62:0/1785326292 shutdown_connections 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.900+0000 7efdc3fd7640 1 -- 172.21.15.62:0/1785326292 wait complete. 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc3fd7640 1 Processor -- start 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc3fd7640 1 -- start start 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc3fd7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdbc100480 0x7efdbc1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc3fd7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 0x7efdbc1968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc3fd7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdbc1055b0 0x7efdbc19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc3fd7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc10fdb0 con 0x7efdbc100e30 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc3fd7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc10fc30 con 0x7efdbc100480 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc3fd7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc10ff30 con 0x7efdbc1055b0 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc1572640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 0x7efdbc1968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc1572640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 0x7efdbc1968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40904/0 (socket says 172.21.15.62:40904) 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.901+0000 7efdc1572640 1 -- 172.21.15.62:0/3293962368 learned_addr learned my addr 172.21.15.62:0/3293962368 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc1d73640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdbc100480 0x7efdbc1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc2574640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdbc1055b0 0x7efdbc19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc1572640 1 -- 172.21.15.62:0/3293962368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdbc1055b0 msgr2=0x7efdbc19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc1572640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdbc1055b0 0x7efdbc19ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc1572640 1 -- 172.21.15.62:0/3293962368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdbc100480 msgr2=0x7efdbc1963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc1572640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdbc100480 0x7efdbc1963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc1572640 1 -- 172.21.15.62:0/3293962368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdbc19b360 con 0x7efdbc100e30 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc1572640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 0x7efdbc1968f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7efdb0006010 tx=0x7efdb00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdbaffd640 1 -- 172.21.15.62:0/3293962368 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0040070 con 0x7efdbc100e30 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdbaffd640 1 -- 172.21.15.62:0/3293962368 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdb003c460 con 0x7efdbc100e30 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc3fd7640 1 -- 172.21.15.62:0/3293962368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdbc19b5f0 con 0x7efdbc100e30 2024-09-15T23:18:19.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdbaffd640 1 -- 172.21.15.62:0/3293962368 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0036030 con 0x7efdbc100e30 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.902+0000 7efdc3fd7640 1 -- 172.21.15.62:0/3293962368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdbc1a2f30 con 0x7efdbc100e30 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.904+0000 7efdbaffd640 1 -- 172.21.15.62:0/3293962368 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7efdb0043070 con 0x7efdbc100e30 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc63b0740 1 Processor -- start 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc63b0740 1 -- start start 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c2f060 0x561d00c2f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f980 0x561d00c39d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c3a240 0x561d00c3c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc63b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561d00b94620 con 0x561d00c2f980 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc63b0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d00b3be50 con 0x561d00c3a240 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc63b0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561d00b905e0 con 0x561d00c2f060 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc1d73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f980 0x561d00c39d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc1d73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f980 0x561d00c39d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40918/0 (socket says 172.21.15.62:40918) 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc1d73640 1 -- 172.21.15.62:0/3220566648 learned_addr learned my addr 172.21.15.62:0/3220566648 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:19.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc1572640 1 --2- 172.21.15.62:0/3220566648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c3a240 0x561d00c3c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdbaffd640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a54d0 0x7efda40a7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.905+0000 7efdc2574640 1 --2- 172.21.15.62:0/3220566648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c2f060 0x561d00c2f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efdc1d73640 1 -- 172.21.15.62:0/3220566648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c2f060 msgr2=0x561d00c2f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efdc1d73640 1 --2- 172.21.15.62:0/3220566648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c2f060 0x561d00c2f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efdc1d73640 1 -- 172.21.15.62:0/3220566648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c3a240 msgr2=0x561d00c3c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efdc1d73640 1 --2- 172.21.15.62:0/3220566648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c3a240 0x561d00c3c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efdc1d73640 1 -- 172.21.15.62:0/3220566648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d00b2cd40 con 0x561d00c2f980 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efdc2574640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a54d0 0x7efda40a7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efdbaffd640 1 -- 172.21.15.62:0/3293962368 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7efdb00071c0 con 0x7efdbc100e30 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efdc1d73640 1 --2- 172.21.15.62:0/3220566648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f980 0x561d00c39d00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efdac0039f0 tx=0x7efdac003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04d1c1eebb0bffe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efda3fff640 1 -- 172.21.15.62:0/3220566648 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac004b20 con 0x561d00c2f980 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efda3fff640 1 -- 172.21.15.62:0/3220566648 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac004cc0 con 0x561d00c2f980 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.906+0000 7efda3fff640 1 -- 172.21.15.62:0/3220566648 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac012770 con 0x561d00c2f980 2024-09-15T23:18:19.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 -- 172.21.15.62:0/3220566648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f980 msgr2=0x561d00c39d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc2574640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a54d0 0x7efda40a7990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efdb4006fd0 tx=0x7efdb40047d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 --2- 172.21.15.62:0/3220566648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f980 0x561d00c39d00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efdac0039f0 tx=0x7efdac003eb0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 -- 172.21.15.62:0/3220566648 shutdown_connections 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 --2- 172.21.15.62:0/3220566648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c3a240 0x561d00c3c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 --2- 172.21.15.62:0/3220566648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f980 0x561d00c39d00 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 --2- 172.21.15.62:0/3220566648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c2f060 0x561d00c2f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 -- 172.21.15.62:0/3220566648 >> 172.21.15.62:0/3220566648 conn(0x561d00c2b5f0 msgr2=0x561d00c2b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 -- 172.21.15.62:0/3220566648 shutdown_connections 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 -- 172.21.15.62:0/3220566648 wait complete. 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 -- start start 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.907+0000 7efdc63b0740 1 Processor -- start 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c370e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c37620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c37b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc63b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561d00c3ce10 con 0x561d00c2f060 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc63b0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d00b1cf10 con 0x561d00c2f980 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc63b0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561d00c3cf90 con 0x561d00c3a240 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc63b0740 10 client.0 _ll_register_callbacks cb 0x561d00b40db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1d73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1d73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c370e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40932/0 (socket says 172.21.15.62:40932) 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1d73640 1 -- 172.21.15.62:0/432519353 learned_addr learned my addr 172.21.15.62:0/432519353 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1572640 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c37620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc2574640 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c37b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1d73640 1 -- 172.21.15.62:0/432519353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 msgr2=0x561d00c37b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1d73640 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c37b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1d73640 1 -- 172.21.15.62:0/432519353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 msgr2=0x561d00c37620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:19.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1d73640 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c37620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1d73640 1 -- 172.21.15.62:0/432519353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d008a3580 con 0x561d00c2f060 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efdc1d73640 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c370e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efdac009dd0 tx=0x7efdac0059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77d6c53d8b367c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.908+0000 7efda17fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efda17fa640 1 -- 172.21.15.62:0/432519353 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac012940 con 0x561d00c2f060 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efda17fa640 1 -- 172.21.15.62:0/432519353 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7efdac012ae0 con 0x561d00c2f060 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efdc63b0740 1 -- 172.21.15.62:0/432519353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdac004cc0 con 0x561d00c2f060 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efda17fa640 1 -- 172.21.15.62:0/432519353 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efdac012ce0 con 0x561d00c2f060 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efdc63b0740 10 client.16005 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efdc63b0740 1 -- 172.21.15.62:0/432519353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561d0096d960 con 0x561d00c2f060 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efda17fa640 1 -- 172.21.15.62:0/432519353 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac004cc0 con 0x561d00c2f060 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efda17fa640 1 -- 172.21.15.62:0/432519353 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7efdac00f040 con 0x561d00c2f060 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efdc63b0740 10 client.16005 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efdc63b0740 10 client.16005 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efdc63b0740 20 client.16005 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:19.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efdc63b0740 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d00c49d10 0x561d00c4c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efdc63b0740 4 client.16005 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.909+0000 7efdc63b0740 1 -- 172.21.15.62:0/432519353 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d00ba1b90 con 0x561d00c49d10 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.910+0000 7efdc1572640 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d00c49d10 0x561d00c4c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.910+0000 7efda17fa640 10 client.16005 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.910+0000 7efdc1572640 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d00c49d10 0x561d00c4c100 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d08d23e0022b6529 server_cookie=f05da98168d40ca0 in_seq=0 out_seq=0 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efda17fa640 1 -- 172.21.15.62:0/432519353 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561d00ba1b90 con 0x561d00c49d10 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efda17fa640 10 client.16005 handle_command_reply: tid=0 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 client.16005 shutdown 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 -- 172.21.15.62:0/432519353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 msgr2=0x561d00c370e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c370e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7efdac009dd0 tx=0x7efdac0059d0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 -- 172.21.15.62:0/432519353 shutdown_connections 2024-09-15T23:18:19.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d00c49d10 0x561d00c4c100 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c37b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efda17fa640 0 client.16005 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c37620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 --2- 172.21.15.62:0/432519353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c370e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 -- 172.21.15.62:0/432519353 >> 172.21.15.62:0/432519353 conn(0x561d00c2b5f0 msgr2=0x561d00c312d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 -- 172.21.15.62:0/432519353 shutdown_connections 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 1 -- 172.21.15.62:0/432519353 wait complete. 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.911+0000 7efdc63b0740 20 client.16005 trim_cache size 0 max 16384 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.928+0000 7efdc63b0740 1 Processor -- start 2024-09-15T23:18:19.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.928+0000 7efdc63b0740 1 -- start start 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.928+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c2a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.928+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c51930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.928+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c51e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.928+0000 7efdc63b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1af3f0 con 0x561d00c2f060 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.928+0000 7efdc63b0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1acc90 con 0x561d00c2f980 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.928+0000 7efdc63b0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1aaae0 con 0x561d00c3a240 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1d73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c2a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1d73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c2a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40948/0 (socket says 172.21.15.62:40948) 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1d73640 1 -- 172.21.15.62:0/694811879 learned_addr learned my addr 172.21.15.62:0/694811879 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc2574640 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c51e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1572640 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c51930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1d73640 1 -- 172.21.15.62:0/694811879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 msgr2=0x561d00c51e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1d73640 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c51e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1d73640 1 -- 172.21.15.62:0/694811879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 msgr2=0x561d00c51930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1d73640 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c51930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1d73640 1 -- 172.21.15.62:0/694811879 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d00c4c6e0 con 0x561d00c2f060 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1572640 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c51930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efdc1d73640 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c2a640 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efdac0265d0 tx=0x7efdac004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0120ad8c0e95ce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efda37fe640 1 -- 172.21.15.62:0/694811879 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac026650 con 0x561d00c2f060 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efda37fe640 1 -- 172.21.15.62:0/694811879 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac0267f0 con 0x561d00c2f060 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.929+0000 7efda37fe640 1 -- 172.21.15.62:0/694811879 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac011730 con 0x561d00c2f060 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 -- 172.21.15.62:0/694811879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 msgr2=0x561d00c2a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c2a640 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7efdac0265d0 tx=0x7efdac004410 comp rx=0 tx=0).stop 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 -- 172.21.15.62:0/694811879 shutdown_connections 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c51e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c51930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 --2- 172.21.15.62:0/694811879 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c2a640 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 -- 172.21.15.62:0/694811879 >> 172.21.15.62:0/694811879 conn(0x561d00c2b5f0 msgr2=0x561d00c29f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 -- 172.21.15.62:0/694811879 shutdown_connections 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 -- 172.21.15.62:0/694811879 wait complete. 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 -- start start 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.930+0000 7efdc63b0740 1 Processor -- start 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c3e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c3e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc63b0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c3ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc63b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1ab150 con 0x561d00c2f060 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc63b0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1ac5c0 con 0x561d00c2f980 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc63b0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1ab820 con 0x561d00c3a240 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc63b0740 10 client.0 _ll_register_callbacks cb 0x7efdac009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:19.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1d73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c3e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1d73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c3e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40962/0 (socket says 172.21.15.62:40962) 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1d73640 1 -- 172.21.15.62:0/826937751 learned_addr learned my addr 172.21.15.62:0/826937751 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1572640 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c3e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc2574640 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c3ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1d73640 1 -- 172.21.15.62:0/826937751 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 msgr2=0x561d00c3ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1d73640 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c3ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1d73640 1 -- 172.21.15.62:0/826937751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 msgr2=0x561d00c3e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1d73640 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c3e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1d73640 1 -- 172.21.15.62:0/826937751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d00cb3b00 con 0x561d00c2f060 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc2574640 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c3ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efdc1d73640 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c3e140 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7efdac026460 tx=0x7efdac009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a4ef85d23fdc26 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efda17fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:19.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.931+0000 7efda17fa640 1 -- 172.21.15.62:0/826937751 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac026650 con 0x561d00c2f060 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc63b0740 1 -- 172.21.15.62:0/826937751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdac0267f0 con 0x561d00c2f060 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efda17fa640 1 -- 172.21.15.62:0/826937751 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7efdac026db0 con 0x561d00c2f060 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efda17fa640 1 -- 172.21.15.62:0/826937751 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efdac004740 con 0x561d00c2f060 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc63b0740 10 client.16011 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc63b0740 1 -- 172.21.15.62:0/826937751 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561d00cb4d30 con 0x561d00c2f060 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efda17fa640 1 -- 172.21.15.62:0/826937751 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac0267f0 con 0x561d00c2f060 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efda17fa640 1 -- 172.21.15.62:0/826937751 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7efdac033030 con 0x561d00c2f060 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc63b0740 10 client.16011 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc63b0740 10 client.16011 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc63b0740 20 client.16011 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc63b0740 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d00cb5230 0x561d00cb7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc63b0740 4 client.16011 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "3956db1e", "release": true, "await": true}] 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc63b0740 1 -- 172.21.15.62:0/826937751 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "3956db1e", "release": true, "await": true}) -- 0x561d00b9c0a0 con 0x561d00cb5230 2024-09-15T23:18:19.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.932+0000 7efdc1572640 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d00cb5230 0x561d00cb7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.933+0000 7efda17fa640 10 client.16011 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.933+0000 7efdc1572640 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d00cb5230 0x561d00cb7650 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23a5d31828f4f4cd server_cookie=326b91121e54dc38 in_seq=0 out_seq=0 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.960+0000 7efda17fa640 1 -- 172.21.15.62:0/826937751 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+589 (crc 0 0 0) 0x561d00b9c0a0 con 0x561d00cb5230 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.960+0000 7efda17fa640 10 client.16011 handle_command_reply: tid=0 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.960+0000 7efdc63b0740 1 client.16011 shutdown 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.961+0000 7efdc63b0740 1 -- 172.21.15.62:0/826937751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 msgr2=0x561d00c3e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.961+0000 7efdc63b0740 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c3e140 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7efdac026460 tx=0x7efdac009fa0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.964+0000 7efdc63b0740 1 -- 172.21.15.62:0/826937751 shutdown_connections 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.964+0000 7efdc63b0740 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d00cb5230 0x561d00cb7650 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.964+0000 7efda17fa640 0 client.16011 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.964+0000 7efdc63b0740 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d00c3a240 0x561d00c3ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.964+0000 7efdc63b0740 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d00c2f980 0x561d00c3e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.964+0000 7efdc63b0740 1 --2- 172.21.15.62:0/826937751 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d00c2f060 0x561d00c3e140 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:19.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.964+0000 7efdc63b0740 1 -- 172.21.15.62:0/826937751 >> 172.21.15.62:0/826937751 conn(0x561d00c2b5f0 msgr2=0x561d00c394b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.965+0000 7efdc63b0740 1 -- 172.21.15.62:0/826937751 shutdown_connections 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.967+0000 7efdc63b0740 1 -- 172.21.15.62:0/826937751 wait complete. 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.968+0000 7efdc63b0740 20 client.16011 trim_cache size 0 max 16384 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 -- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a54d0 msgr2=0x7efda40a7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a54d0 0x7efda40a7990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efdb4006fd0 tx=0x7efdb40047d0 comp rx=0 tx=0).stop 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 -- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 msgr2=0x7efdbc1968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 0x7efdbc1968f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7efdb0006010 tx=0x7efdb00042f0 comp rx=0 tx=0).stop 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 -- 172.21.15.62:0/3293962368 shutdown_connections 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efda40a54d0 0x7efda40a7990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efdbc1055b0 0x7efdbc19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc100e30 0x7efdbc1968f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 --2- 172.21.15.62:0/3293962368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efdbc100480 0x7efdbc1963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 -- 172.21.15.62:0/3293962368 >> 172.21.15.62:0/3293962368 conn(0x7efdbc077210 msgr2=0x7efdbc0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:20.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 -- 172.21.15.62:0/3293962368 shutdown_connections 2024-09-15T23:18:20.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:19.971+0000 7efdc3fd7640 1 -- 172.21.15.62:0/3293962368 wait complete. 2024-09-15T23:18:20.770 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '3956db1e', total seconds elapsed: 23.7 2024-09-15T23:18:20.770 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 19.55 2024-09-15T23:18:20.770 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:18:21.203 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:21.640 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:21 smithi062 ceph-mon[30949]: pgmap v432: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 12 MiB/s wr, 474 op/s 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.787+0000 7f1cd21e1640 1 -- 172.21.15.62:0/1071759555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ccc106f10 msgr2=0x7f1ccc10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.787+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/1071759555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ccc106f10 0x7f1ccc10ea90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009ca0 tx=0x7f1cc402a8d0 comp rx=0 tx=0).stop 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.787+0000 7f1cd21e1640 1 -- 172.21.15.62:0/1071759555 shutdown_connections 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.787+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/1071759555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ccc106f10 0x7f1ccc10ea90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.787+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/1071759555 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ccc102790 0x7f1ccc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.787+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/1071759555 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ccc101de0 0x7f1ccc1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.787+0000 7f1cd21e1640 1 -- 172.21.15.62:0/1071759555 >> 172.21.15.62:0/1071759555 conn(0x7f1ccc0fd880 msgr2=0x7f1ccc0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.787+0000 7f1cd21e1640 1 -- 172.21.15.62:0/1071759555 shutdown_connections 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.787+0000 7f1cd21e1640 1 -- 172.21.15.62:0/1071759555 wait complete. 2024-09-15T23:18:21.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.788+0000 7f1cd21e1640 1 Processor -- start 2024-09-15T23:18:21.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.788+0000 7f1cd21e1640 1 -- start start 2024-09-15T23:18:21.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1cd21e1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ccc101de0 0x7f1ccc108ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:21.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1cd21e1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ccc102790 0x7f1ccc108ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:21.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1cd21e1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ccc106f10 0x7f1ccc109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:21.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1cd21e1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc111fb0 con 0x7f1ccc102790 2024-09-15T23:18:21.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1cd21e1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc111e30 con 0x7f1ccc106f10 2024-09-15T23:18:21.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1cd21e1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1ccc112130 con 0x7f1ccc101de0 2024-09-15T23:18:21.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ccc101de0 0x7f1ccc108ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:21.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ccc102790 0x7f1ccc108ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:21.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ccc102790 0x7f1ccc108ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40982/0 (socket says 172.21.15.62:40982) 2024-09-15T23:18:21.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccaffd640 1 -- 172.21.15.62:0/3397626613 learned_addr learned my addr 172.21.15.62:0/3397626613 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:21.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccb7fe640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ccc101de0 0x7f1ccc108ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56392/0 (socket says 172.21.15.62:56392) 2024-09-15T23:18:21.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccbfff640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ccc106f10 0x7f1ccc109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:21.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccbfff640 1 -- 172.21.15.62:0/3397626613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ccc101de0 msgr2=0x7f1ccc108ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:21.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccbfff640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ccc101de0 0x7f1ccc108ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:21.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccbfff640 1 -- 172.21.15.62:0/3397626613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ccc102790 msgr2=0x7f1ccc108ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:21.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.789+0000 7f1ccbfff640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ccc102790 0x7f1ccc108ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.790+0000 7f1ccbfff640 1 -- 172.21.15.62:0/3397626613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ccc1aad00 con 0x7f1ccc106f10 2024-09-15T23:18:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.790+0000 7f1ccbfff640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ccc106f10 0x7f1ccc109530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009430 tx=0x7f1cc4031690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.790+0000 7f1cc8ff9640 1 -- 172.21.15.62:0/3397626613 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4004040 con 0x7f1ccc106f10 2024-09-15T23:18:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.790+0000 7f1cc8ff9640 1 -- 172.21.15.62:0/3397626613 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cc40046c0 con 0x7f1ccc106f10 2024-09-15T23:18:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.790+0000 7f1cc8ff9640 1 -- 172.21.15.62:0/3397626613 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4030e10 con 0x7f1ccc106f10 2024-09-15T23:18:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.790+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ccc1aaf90 con 0x7f1ccc106f10 2024-09-15T23:18:21.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.791+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ccc1ab480 con 0x7f1ccc106f10 2024-09-15T23:18:21.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.792+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ccc1032c0 con 0x7f1ccc106f10 2024-09-15T23:18:21.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.794+0000 7f1cc8ff9640 1 -- 172.21.15.62:0/3397626613 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1cc40305d0 con 0x7f1ccc106f10 2024-09-15T23:18:21.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.795+0000 7f1cc8ff9640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1cb40a3410 0x7f1cb40a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:21.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.795+0000 7f1cc8ff9640 1 -- 172.21.15.62:0/3397626613 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1cc40f10b0 con 0x7f1ccc106f10 2024-09-15T23:18:21.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.795+0000 7f1ccb7fe640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1cb40a3410 0x7f1cb40a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:21.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.796+0000 7f1ccb7fe640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1cb40a3410 0x7f1cb40a58d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc009bd0 tx=0x7f1cbc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:21.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:21.800+0000 7f1cc8ff9640 1 -- 172.21.15.62:0/3397626613 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f1cc40bd9f0 con 0x7f1ccc106f10 2024-09-15T23:18:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:21 smithi067 ceph-mon[35665]: pgmap v432: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 12 MiB/s wr, 474 op/s 2024-09-15T23:18:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:21 smithi148 ceph-mon[35610]: pgmap v432: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 12 MiB/s wr, 474 op/s 2024-09-15T23:18:22.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.165+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ccc1ab840 con 0x7f1ccc106f10 2024-09-15T23:18:22.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.166+0000 7f1cc8ff9640 1 -- 172.21.15.62:0/3397626613 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11907 (secure 0 0 0) 0x7f1cc40bd2f0 con 0x7f1ccc106f10 2024-09-15T23:18:22.168 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:18:22.169 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":28,"btime":"2024-09-15T23:17:33:092139+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:17:32.873645+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:18:22.169 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 28 2024-09-15T23:18:22.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1cb40a3410 msgr2=0x7f1cb40a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1cb40a3410 0x7f1cb40a58d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc009bd0 tx=0x7f1cbc005f90 comp rx=0 tx=0).stop 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ccc106f10 msgr2=0x7f1ccc109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ccc106f10 0x7f1ccc109530 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009430 tx=0x7f1cc4031690 comp rx=0 tx=0).stop 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 shutdown_connections 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1cb40a3410 0x7f1cb40a58d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1ccc106f10 0x7f1ccc109530 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ccc102790 0x7f1ccc108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 --2- 172.21.15.62:0/3397626613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1ccc101de0 0x7f1ccc108ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.168+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 >> 172.21.15.62:0/3397626613 conn(0x7f1ccc0fd880 msgr2=0x7f1ccc0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.169+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 shutdown_connections 2024-09-15T23:18:22.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:22.169+0000 7f1cd21e1640 1 -- 172.21.15.62:0/3397626613 wait complete. 2024-09-15T23:18:22.919 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:22 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3397626613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:22.977 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0' 2024-09-15T23:18:22.977 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 139.55 --await-for 45.0 2024-09-15T23:18:23.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:22 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3397626613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:23.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:22 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3397626613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:23.363 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:23.824 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:23 smithi062 ceph-mon[30949]: pgmap v433: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 6.5 MiB/s wr, 336 op/s 2024-09-15T23:18:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.970+0000 7f4370f0e640 1 -- 172.21.15.62:0/569540995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 msgr2=0x7f436c106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.970+0000 7f4370f0e640 1 --2- 172.21.15.62:0/569540995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 0x7f436c106e50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f435c009900 tx=0x7f435c02a400 comp rx=0 tx=0).stop 2024-09-15T23:18:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.970+0000 7f4370f0e640 1 -- 172.21.15.62:0/569540995 shutdown_connections 2024-09-15T23:18:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.970+0000 7f4370f0e640 1 --2- 172.21.15.62:0/569540995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f436c101410 0x7f436c10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.970+0000 7f4370f0e640 1 --2- 172.21.15.62:0/569540995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436c100a70 0x7f436c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.970+0000 7f4370f0e640 1 --2- 172.21.15.62:0/569540995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 0x7f436c106e50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.970+0000 7f4370f0e640 1 -- 172.21.15.62:0/569540995 >> 172.21.15.62:0/569540995 conn(0x7f436c0fc790 msgr2=0x7f436c0febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.970+0000 7f4370f0e640 1 -- 172.21.15.62:0/569540995 shutdown_connections 2024-09-15T23:18:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.970+0000 7f4370f0e640 1 -- 172.21.15.62:0/569540995 wait complete. 2024-09-15T23:18:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.971+0000 7f4370f0e640 1 Processor -- start 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.971+0000 7f4370f0e640 1 -- start start 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f4370f0e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f436c100a70 0x7f436c1a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f4370f0e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436c101410 0x7f436c1a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f4370f0e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 0x7f436c19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f4370f0e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f436c110ee0 con 0x7f436c101410 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f4370f0e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f436c110d60 con 0x7f436c106a70 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f4370f0e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f436c111060 con 0x7f436c100a70 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f4363fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436c101410 0x7f436c1a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 0x7f436c19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f436a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f436c100a70 0x7f436c1a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f436a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f436c100a70 0x7f436c1a4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48156/0 (socket says 172.21.15.62:48156) 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 0x7f436c19ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54822/0 (socket says 172.21.15.62:54822) 2024-09-15T23:18:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f4363fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436c101410 0x7f436c1a4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60750/0 (socket says 172.21.15.62:60750) 2024-09-15T23:18:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.972+0000 7f436a59c640 1 -- 172.21.15.62:0/1018340167 learned_addr learned my addr 172.21.15.62:0/1018340167 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.973+0000 7f436ad9d640 1 -- 172.21.15.62:0/1018340167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f436c100a70 msgr2=0x7f436c1a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.973+0000 7f436ad9d640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f436c100a70 0x7f436c1a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.973+0000 7f436ad9d640 1 -- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436c101410 msgr2=0x7f436c1a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.973+0000 7f436ad9d640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436c101410 0x7f436c1a4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.973+0000 7f436ad9d640 1 -- 172.21.15.62:0/1018340167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436c19f340 con 0x7f436c106a70 2024-09-15T23:18:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.973+0000 7f436ad9d640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 0x7f436c19ebc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f436400be20 tx=0x7f436400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.974+0000 7f4363fff640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436c101410 0x7f436c1a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-15T23:18:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.974+0000 7f43637fe640 1 -- 172.21.15.62:0/1018340167 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364019070 con 0x7f436c106a70 2024-09-15T23:18:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.974+0000 7f43637fe640 1 -- 172.21.15.62:0/1018340167 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4364014920 con 0x7f436c106a70 2024-09-15T23:18:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.974+0000 7f4370f0e640 1 -- 172.21.15.62:0/1018340167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436c19f630 con 0x7f436c106a70 2024-09-15T23:18:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.974+0000 7f43637fe640 1 -- 172.21.15.62:0/1018340167 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364012900 con 0x7f436c106a70 2024-09-15T23:18:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.975+0000 7f4370f0e640 1 -- 172.21.15.62:0/1018340167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f436c1ab3f0 con 0x7f436c106a70 2024-09-15T23:18:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.978+0000 7f43732e7740 1 Processor -- start 2024-09-15T23:18:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.979+0000 7f43637fe640 1 -- 172.21.15.62:0/1018340167 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f436401c070 con 0x7f436c106a70 2024-09-15T23:18:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.979+0000 7f43732e7740 1 -- start start 2024-09-15T23:18:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.979+0000 7f43732e7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.979+0000 7f43732e7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 0x55b1ee5b14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.979+0000 7f43732e7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.979+0000 7f43732e7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b1ee50b160 con 0x55b1ee5a7170 2024-09-15T23:18:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.979+0000 7f43732e7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b1ee4b2720 con 0x55b1ee5a6850 2024-09-15T23:18:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.979+0000 7f43732e7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b1ee507120 con 0x55b1ee5b1a30 2024-09-15T23:18:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.980+0000 7f4363fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.980+0000 7f4363fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54834/0 (socket says 172.21.15.62:54834) 2024-09-15T23:18:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.980+0000 7f4363fff640 1 -- 172.21.15.62:0/1975392232 learned_addr learned my addr 172.21.15.62:0/1975392232 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.980+0000 7f4363fff640 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.980+0000 7f436a59c640 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 0x55b1ee5b14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.981+0000 7f4363fff640 1 -- 172.21.15.62:0/1975392232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 msgr2=0x55b1ee5b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.981+0000 7f4363fff640 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.981+0000 7f4363fff640 1 -- 172.21.15.62:0/1975392232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 msgr2=0x55b1ee5b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.981+0000 7f4363fff640 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 0x55b1ee5b14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.981+0000 7f4363fff640 1 -- 172.21.15.62:0/1975392232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1ee27cba0 con 0x55b1ee5a6850 2024-09-15T23:18:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.981+0000 7f436a59c640 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 0x55b1ee5b14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.981+0000 7f43637fe640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43580a3410 0x7f43580a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.981+0000 7f43637fe640 1 -- 172.21.15.62:0/1018340167 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f43640cf440 con 0x7f436c106a70 2024-09-15T23:18:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.982+0000 7f436a59c640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43580a3410 0x7f43580a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.982+0000 7f4363fff640 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a6c30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4354009fd0 tx=0x7f4354009670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e826234a9f19134c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.983+0000 7f4360ff9640 1 -- 172.21.15.62:0/1975392232 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354004030 con 0x55b1ee5a6850 2024-09-15T23:18:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.983+0000 7f4360ff9640 1 -- 172.21.15.62:0/1975392232 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435400fc10 con 0x55b1ee5a6850 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.983+0000 7f4360ff9640 1 -- 172.21.15.62:0/1975392232 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354003850 con 0x55b1ee5a6850 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 -- 172.21.15.62:0/1975392232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 msgr2=0x55b1ee5a6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a6c30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4354009fd0 tx=0x7f4354009670 comp rx=0 tx=0).stop 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f436a59c640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43580a3410 0x7f43580a58d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f435c0314e0 tx=0x7f435c00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 -- 172.21.15.62:0/1975392232 shutdown_connections 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 0x55b1ee5b14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 --2- 172.21.15.62:0/1975392232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a6c30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 -- 172.21.15.62:0/1975392232 >> 172.21.15.62:0/1975392232 conn(0x55b1ee5a2de0 msgr2=0x55b1ee5a31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 -- 172.21.15.62:0/1975392232 shutdown_connections 2024-09-15T23:18:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 -- 172.21.15.62:0/1975392232 wait complete. 2024-09-15T23:18:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 -- start start 2024-09-15T23:18:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.984+0000 7f43732e7740 1 Processor -- start 2024-09-15T23:18:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.985+0000 7f43732e7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5ae880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.985+0000 7f43732e7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a7170 0x55b1ee5aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.985+0000 7f43732e7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5af300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.985+0000 7f43732e7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b1ee5c5660 con 0x55b1ee5b1a30 2024-09-15T23:18:23.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.985+0000 7f43732e7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b1ee493900 con 0x55b1ee5a6850 2024-09-15T23:18:23.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.985+0000 7f43732e7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b1ee5c57e0 con 0x55b1ee5a7170 2024-09-15T23:18:23.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.985+0000 7f43732e7740 10 client.0 _ll_register_callbacks cb 0x55b1ee491780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f4363fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a7170 0x55b1ee5aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5af300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60772/0 (socket says 172.21.15.62:60772) 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f4363fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a7170 0x55b1ee5aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48172/0 (socket says 172.21.15.62:48172) 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f436ad9d640 1 -- 172.21.15.62:0/1358688360 learned_addr learned my addr 172.21.15.62:0/1358688360 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f436a59c640 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f4363fff640 1 -- 172.21.15.62:0/1358688360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 msgr2=0x55b1ee5ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f4363fff640 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5ae880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f4363fff640 1 -- 172.21.15.62:0/1358688360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 msgr2=0x55b1ee5af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:23.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f4363fff640 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.986+0000 7f4363fff640 1 -- 172.21.15.62:0/1358688360 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1ee4d3c50 con 0x55b1ee5a7170 2024-09-15T23:18:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.987+0000 7f4363fff640 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a7170 0x55b1ee5aedc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4354004320 tx=0x7f43540143b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e927cca4c36203c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.987+0000 7f4351ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:18:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.988+0000 7f4351ffb640 1 -- 172.21.15.62:0/1358688360 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435402c060 con 0x55b1ee5a7170 2024-09-15T23:18:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.988+0000 7f4351ffb640 1 -- 172.21.15.62:0/1358688360 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f435400fdc0 con 0x55b1ee5a7170 2024-09-15T23:18:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.988+0000 7f43732e7740 1 -- 172.21.15.62:0/1358688360 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f435400fc10 con 0x55b1ee5a7170 2024-09-15T23:18:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.988+0000 7f4351ffb640 1 -- 172.21.15.62:0/1358688360 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4354005b90 con 0x55b1ee5a7170 2024-09-15T23:18:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.988+0000 7f43732e7740 10 client.25513 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:23.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.988+0000 7f43732e7740 1 -- 172.21.15.62:0/1358688360 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b1ee5d1290 con 0x55b1ee5a7170 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f4351ffb640 1 -- 172.21.15.62:0/1358688360 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435400fc10 con 0x55b1ee5a7170 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f4351ffb640 1 -- 172.21.15.62:0/1358688360 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f435401f420 con 0x55b1ee5a7170 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f43732e7740 10 client.25513 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f43732e7740 10 client.25513 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f43732e7740 20 client.25513 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f43732e7740 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b1ee5d2630 0x55b1ee5d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f43732e7740 4 client.25513 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f43732e7740 1 -- 172.21.15.62:0/1358688360 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b1ee518890 con 0x55b1ee5d2630 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f436a59c640 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b1ee5d2630 0x55b1ee5d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:23.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f436a59c640 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b1ee5d2630 0x55b1ee5d4a20 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dccde90bb1c9f473 server_cookie=7d54c3c51ab07065 in_seq=0 out_seq=0 2024-09-15T23:18:23.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.989+0000 7f4351ffb640 10 client.25513 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:23.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.990+0000 7f4351ffb640 1 -- 172.21.15.62:0/1358688360 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b1ee518890 con 0x55b1ee5d2630 2024-09-15T23:18:23.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.990+0000 7f4351ffb640 10 client.25513 handle_command_reply: tid=0 2024-09-15T23:18:23.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.990+0000 7f43732e7740 1 client.25513 shutdown 2024-09-15T23:18:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 -- 172.21.15.62:0/1358688360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a7170 msgr2=0x55b1ee5aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a7170 0x55b1ee5aedc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4354004320 tx=0x7f43540143b0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 -- 172.21.15.62:0/1358688360 shutdown_connections 2024-09-15T23:18:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b1ee5d2630 0x55b1ee5d4a20 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a7170 0x55b1ee5aedc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 --2- 172.21.15.62:0/1358688360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 -- 172.21.15.62:0/1358688360 >> 172.21.15.62:0/1358688360 conn(0x55b1ee5a2de0 msgr2=0x55b1ee5a8b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:23.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f4351ffb640 0 client.25513 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:23.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 -- 172.21.15.62:0/1358688360 shutdown_connections 2024-09-15T23:18:23.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 1 -- 172.21.15.62:0/1358688360 wait complete. 2024-09-15T23:18:23.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:23.992+0000 7f43732e7740 20 client.25513 trim_cache size 0 max 16384 2024-09-15T23:18:24.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:23 smithi067 ceph-mon[35665]: pgmap v433: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 6.5 MiB/s wr, 336 op/s 2024-09-15T23:18:24.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.029+0000 7f43732e7740 1 Processor -- start 2024-09-15T23:18:24.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.033+0000 7f43732e7740 1 -- start start 2024-09-15T23:18:24.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.033+0000 7f43732e7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:24.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.033+0000 7f43732e7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 0x55b1ee5af150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:24.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.033+0000 7f43732e7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5da140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:24.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.033+0000 7f43732e7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1af3d0 con 0x55b1ee5a7170 2024-09-15T23:18:24.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f43732e7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1acc70 con 0x55b1ee5a6850 2024-09-15T23:18:24.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f436a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:24.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f43732e7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f436c19fc40 con 0x55b1ee5b1a30 2024-09-15T23:18:24.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f436a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54858/0 (socket says 172.21.15.62:54858) 2024-09-15T23:18:24.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f436a59c640 1 -- 172.21.15.62:0/1134952863 learned_addr learned my addr 172.21.15.62:0/1134952863 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:24.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f436ad9d640 1 --2- 172.21.15.62:0/1134952863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5da140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:24.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f436a59c640 1 -- 172.21.15.62:0/1134952863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 msgr2=0x55b1ee5da140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:24.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f436a59c640 1 --2- 172.21.15.62:0/1134952863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5da140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f436a59c640 1 -- 172.21.15.62:0/1134952863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 msgr2=0x55b1ee5af150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:24.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f436a59c640 1 --2- 172.21.15.62:0/1134952863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 0x55b1ee5af150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.034+0000 7f436a59c640 1 -- 172.21.15.62:0/1134952863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1ee5c5840 con 0x55b1ee5a6850 2024-09-15T23:18:24.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.035+0000 7f436a59c640 1 --2- 172.21.15.62:0/1134952863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a9300 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f435c009850 tx=0x7f435c03e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc156d4b9522ee36 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:24.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.036+0000 7f4353fff640 1 -- 172.21.15.62:0/1134952863 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c035050 con 0x55b1ee5a6850 2024-09-15T23:18:24.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.036+0000 7f4353fff640 1 -- 172.21.15.62:0/1134952863 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435c056070 con 0x55b1ee5a6850 2024-09-15T23:18:24.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.036+0000 7f4353fff640 1 -- 172.21.15.62:0/1134952863 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c03c9a0 con 0x55b1ee5a6850 2024-09-15T23:18:24.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:23 smithi148 ceph-mon[35610]: pgmap v433: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 6.5 MiB/s wr, 336 op/s 2024-09-15T23:18:24.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.038+0000 7f43732e7740 1 -- 172.21.15.62:0/1134952863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 msgr2=0x55b1ee5a9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:24.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.038+0000 7f43732e7740 1 --2- 172.21.15.62:0/1134952863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a9300 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f435c009850 tx=0x7f435c03e070 comp rx=0 tx=0).stop 2024-09-15T23:18:24.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.038+0000 7f43732e7740 1 -- 172.21.15.62:0/1134952863 shutdown_connections 2024-09-15T23:18:24.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.038+0000 7f43732e7740 1 --2- 172.21.15.62:0/1134952863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5da140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.038+0000 7f43732e7740 1 --2- 172.21.15.62:0/1134952863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5a7170 0x55b1ee5af150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.038+0000 7f43732e7740 1 --2- 172.21.15.62:0/1134952863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a6850 0x55b1ee5a9300 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.038+0000 7f43732e7740 1 -- 172.21.15.62:0/1134952863 >> 172.21.15.62:0/1134952863 conn(0x55b1ee5a2de0 msgr2=0x55b1ee5a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:24.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.039+0000 7f43732e7740 1 -- 172.21.15.62:0/1134952863 shutdown_connections 2024-09-15T23:18:24.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.040+0000 7f43732e7740 1 -- 172.21.15.62:0/1134952863 wait complete. 2024-09-15T23:18:24.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.041+0000 7f43732e7740 1 -- start start 2024-09-15T23:18:24.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.041+0000 7f43732e7740 1 Processor -- start 2024-09-15T23:18:24.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f43732e7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a6850 0x55b1ee5ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:24.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f43732e7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a7170 0x55b1ee5abe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f43732e7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5ac340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f43732e7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f436c073fd0 con 0x55b1ee5b1a30 2024-09-15T23:18:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f43732e7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1ac5a0 con 0x55b1ee5a7170 2024-09-15T23:18:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f43732e7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f436c1ab800 con 0x55b1ee5a6850 2024-09-15T23:18:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5ac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f436ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5ac340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60798/0 (socket says 172.21.15.62:60798) 2024-09-15T23:18:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f436ad9d640 1 -- 172.21.15.62:0/310989768 learned_addr learned my addr 172.21.15.62:0/310989768 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.042+0000 7f43732e7740 10 client.0 _ll_register_callbacks cb 0x7f4354009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.043+0000 7f436ad9d640 1 -- 172.21.15.62:0/310989768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a6850 msgr2=0x55b1ee5ab8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:18:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.043+0000 7f436ad9d640 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a6850 0x55b1ee5ab8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.043+0000 7f436ad9d640 1 -- 172.21.15.62:0/310989768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a7170 msgr2=0x55b1ee5abe00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:18:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.043+0000 7f436ad9d640 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a7170 0x55b1ee5abe00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.043+0000 7f436ad9d640 1 -- 172.21.15.62:0/310989768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1ee5a3e90 con 0x55b1ee5b1a30 2024-09-15T23:18:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.044+0000 7f436ad9d640 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5ac340 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4364097b90 tx=0x7f4364009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d69ebd4b44e9795 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.044+0000 7f4351ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.044+0000 7f4351ffb640 1 -- 172.21.15.62:0/310989768 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43640a3e60 con 0x55b1ee5b1a30 2024-09-15T23:18:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.044+0000 7f43732e7740 1 -- 172.21.15.62:0/310989768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f435c056070 con 0x55b1ee5b1a30 2024-09-15T23:18:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.044+0000 7f4351ffb640 1 -- 172.21.15.62:0/310989768 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f43640a4000 con 0x55b1ee5b1a30 2024-09-15T23:18:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f4351ffb640 1 -- 172.21.15.62:0/310989768 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f43640afba0 con 0x55b1ee5b1a30 2024-09-15T23:18:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f43732e7740 10 client.16026 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f43732e7740 1 -- 172.21.15.62:0/310989768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b1ee5a2c10 con 0x55b1ee5b1a30 2024-09-15T23:18:24.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f4351ffb640 1 -- 172.21.15.62:0/310989768 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435c056070 con 0x55b1ee5b1a30 2024-09-15T23:18:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f4351ffb640 1 -- 172.21.15.62:0/310989768 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f436409ad00 con 0x55b1ee5b1a30 2024-09-15T23:18:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f43732e7740 10 client.16026 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f43732e7740 10 client.16026 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:18:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f43732e7740 20 client.16026 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f43732e7740 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b1ee63d8c0 0x55b1ee63fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f43732e7740 4 client.16026 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}] 2024-09-15T23:18:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.045+0000 7f43732e7740 1 -- 172.21.15.62:0/310989768 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 139.55, "await_for": 45.0}) -- 0x55b1ee512be0 con 0x55b1ee63d8c0 2024-09-15T23:18:24.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.046+0000 7f436a59c640 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b1ee63d8c0 0x55b1ee63fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:24.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.048+0000 7f436a59c640 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b1ee63d8c0 0x55b1ee63fd40 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ecef77850779a2e server_cookie=c30fbfc6f1b35b73 in_seq=0 out_seq=0 2024-09-15T23:18:24.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.048+0000 7f4351ffb640 10 client.16026 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:24.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.287+0000 7f4351ffb640 1 -- 172.21.15.62:0/310989768 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b1ee512be0 con 0x55b1ee63d8c0 2024-09-15T23:18:24.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.287+0000 7f4351ffb640 10 client.16026 handle_command_reply: tid=0 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 client.16026 shutdown 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 -- 172.21.15.62:0/310989768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 msgr2=0x55b1ee5ac340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5ac340 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4364097b90 tx=0x7f4364009f90 comp rx=0 tx=0).stop 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 -- 172.21.15.62:0/310989768 shutdown_connections 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b1ee63d8c0 0x55b1ee63fd40 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b1ee5b1a30 0x55b1ee5ac340 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b1ee5a7170 0x55b1ee5abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f4351ffb640 0 client.16026 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 --2- 172.21.15.62:0/310989768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b1ee5a6850 0x55b1ee5ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 -- 172.21.15.62:0/310989768 >> 172.21.15.62:0/310989768 conn(0x55b1ee5a2de0 msgr2=0x55b1ee5cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 -- 172.21.15.62:0/310989768 shutdown_connections 2024-09-15T23:18:24.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 1 -- 172.21.15.62:0/310989768 wait complete. 2024-09-15T23:18:24.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.288+0000 7f43732e7740 20 client.16026 trim_cache size 0 max 16384 2024-09-15T23:18:24.292 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:18:24.292 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 28, 2024-09-15T23:18:24.292 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:18:24.292 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 10, 2024-09-15T23:18:24.292 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:18:24.292 INFO:teuthology.orchestra.run.smithi062.stdout: "5d3ecd09": { 2024-09-15T23:18:24.292 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 10, 2024-09-15T23:18:24.292 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 139.6, 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:24.293 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:24.294 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:24.294 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:24.294 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:24.294 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 -- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43580a3410 msgr2=0x7f43580a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43580a3410 0x7f43580a58d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f435c0314e0 tx=0x7f435c00b040 comp rx=0 tx=0).stop 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 -- 172.21.15.62:0/1018340167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 msgr2=0x7f436c19ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 0x7f436c19ebc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f436400be20 tx=0x7f436400bf10 comp rx=0 tx=0).stop 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 -- 172.21.15.62:0/1018340167 shutdown_connections 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43580a3410 0x7f43580a58d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f436c106a70 0x7f436c19ebc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f436c101410 0x7f436c1a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 --2- 172.21.15.62:0/1018340167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f436c100a70 0x7f436c1a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 -- 172.21.15.62:0/1018340167 >> 172.21.15.62:0/1018340167 conn(0x7f436c0fc790 msgr2=0x7f436c06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:24.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 -- 172.21.15.62:0/1018340167 shutdown_connections 2024-09-15T23:18:24.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:24.289+0000 7f4370f0e640 1 -- 172.21.15.62:0/1018340167 wait complete. 2024-09-15T23:18:24.572 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 125.0 s, 2624.1 tps, lat 181.013 ms stddev 1696.562 2024-09-15T23:18:25.086 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5d3ecd09', quiesce took 4.3 seconds. Will release after: 15.25 2024-09-15T23:18:25.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:25 smithi062 ceph-mon[30949]: pgmap v434: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-09-15T23:18:26.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:25 smithi067 ceph-mon[35665]: pgmap v434: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-09-15T23:18:26.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:25 smithi148 ceph-mon[35610]: pgmap v434: 129 pgs: 129 active+clean; 1.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.3 MiB/s wr, 211 op/s 2024-09-15T23:18:27.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:27 smithi062 ceph-mon[30949]: pgmap v435: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 2.2 MiB/s wr, 314 op/s 2024-09-15T23:18:27.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:27 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:18:28.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:27 smithi067 ceph-mon[35665]: pgmap v435: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 2.2 MiB/s wr, 314 op/s 2024-09-15T23:18:28.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:27 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:18:28.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:27 smithi148 ceph-mon[35610]: pgmap v435: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 2.2 MiB/s wr, 314 op/s 2024-09-15T23:18:28.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:27 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:18:28.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:28 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:18:28.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:28 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:18:29.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:28 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:18:29.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:28 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:18:29.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:28 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:18:29.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:28 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:18:29.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 130.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:18:29.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:29 smithi062 ceph-mon[30949]: pgmap v436: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 2.4 MiB/s wr, 312 op/s 2024-09-15T23:18:30.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:29 smithi067 ceph-mon[35665]: pgmap v436: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 2.4 MiB/s wr, 312 op/s 2024-09-15T23:18:30.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:29 smithi148 ceph-mon[35610]: pgmap v436: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 2.4 MiB/s wr, 312 op/s 2024-09-15T23:18:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.704+0000 7ff026342640 1 -- 172.21.15.62:0/2829940661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff020107840 msgr2=0x7ff020109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.704+0000 7ff026342640 1 --2- 172.21.15.62:0/2829940661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff020107840 0x7ff020109cb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff018009ca0 tx=0x7ff01802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.704+0000 7ff026342640 1 -- 172.21.15.62:0/2829940661 shutdown_connections 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.704+0000 7ff026342640 1 --2- 172.21.15.62:0/2829940661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff020107840 0x7ff020109cb0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.704+0000 7ff026342640 1 --2- 172.21.15.62:0/2829940661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020104f10 0x7ff020107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.704+0000 7ff026342640 1 --2- 172.21.15.62:0/2829940661 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff0201025e0 0x7ff0201049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.704+0000 7ff026342640 1 -- 172.21.15.62:0/2829940661 >> 172.21.15.62:0/2829940661 conn(0x7ff0200fc1e0 msgr2=0x7ff0200fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.704+0000 7ff026342640 1 -- 172.21.15.62:0/2829940661 shutdown_connections 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.704+0000 7ff026342640 1 -- 172.21.15.62:0/2829940661 wait complete. 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff026342640 1 Processor -- start 2024-09-15T23:18:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff026342640 1 -- start start 2024-09-15T23:18:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff026342640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0201025e0 0x7ff020101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff026342640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020104f10 0x7ff020100590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff026342640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff020107840 0x7ff020100ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff026342640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff02010d820 con 0x7ff0201025e0 2024-09-15T23:18:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff026342640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff02010d6a0 con 0x7ff020104f10 2024-09-15T23:18:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff026342640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff02010d9a0 con 0x7ff020107840 2024-09-15T23:18:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff01ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0201025e0 0x7ff020101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff01ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0201025e0 0x7ff020101f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60810/0 (socket says 172.21.15.62:60810) 2024-09-15T23:18:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.705+0000 7ff01ffff640 1 -- 172.21.15.62:0/1365953708 learned_addr learned my addr 172.21.15.62:0/1365953708 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01f7fe640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020104f10 0x7ff020100590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff0248df640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff020107840 0x7ff020100ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01ffff640 1 -- 172.21.15.62:0/1365953708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff020107840 msgr2=0x7ff020100ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01ffff640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff020107840 0x7ff020100ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01ffff640 1 -- 172.21.15.62:0/1365953708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020104f10 msgr2=0x7ff020100590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01ffff640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020104f10 0x7ff020100590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01ffff640 1 -- 172.21.15.62:0/1365953708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff020101360 con 0x7ff0201025e0 2024-09-15T23:18:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff0248df640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff020107840 0x7ff020100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01ffff640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0201025e0 0x7ff020101f50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff01000c970 tx=0x7ff01000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01d7fa640 1 -- 172.21.15.62:0/1365953708 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff010007bb0 con 0x7ff0201025e0 2024-09-15T23:18:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01d7fa640 1 -- 172.21.15.62:0/1365953708 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff010007d50 con 0x7ff0201025e0 2024-09-15T23:18:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff026342640 1 -- 172.21.15.62:0/1365953708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0201a1f30 con 0x7ff0201025e0 2024-09-15T23:18:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff01d7fa640 1 -- 172.21.15.62:0/1365953708 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01000f450 con 0x7ff0201025e0 2024-09-15T23:18:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.706+0000 7ff026342640 1 -- 172.21.15.62:0/1365953708 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0201a2330 con 0x7ff0201025e0 2024-09-15T23:18:30.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.709+0000 7ff01d7fa640 1 -- 172.21.15.62:0/1365953708 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff010005020 con 0x7ff0201025e0 2024-09-15T23:18:30.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.709+0000 7ff0290da740 1 Processor -- start 2024-09-15T23:18:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.709+0000 7ff0290da740 1 -- start start 2024-09-15T23:18:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.709+0000 7ff0290da740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b6399a0 0x56218b639da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.709+0000 7ff0290da740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b63a2e0 0x56218b63eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.709+0000 7ff0290da740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b641830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.709+0000 7ff0290da740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56218b59d2f0 con 0x56218b6399a0 2024-09-15T23:18:30.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.709+0000 7ff0290da740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56218b545270 con 0x56218b63f370 2024-09-15T23:18:30.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.709+0000 7ff0290da740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56218b5992b0 con 0x56218b63a2e0 2024-09-15T23:18:30.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b6399a0 0x56218b639da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b6399a0 0x56218b639da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60818/0 (socket says 172.21.15.62:60818) 2024-09-15T23:18:30.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 -- 172.21.15.62:0/2915249341 learned_addr learned my addr 172.21.15.62:0/2915249341 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:30.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff01ffff640 1 --2- 172.21.15.62:0/2915249341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b63a2e0 0x56218b63eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff01f7fe640 1 --2- 172.21.15.62:0/2915249341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b641830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 -- 172.21.15.62:0/2915249341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b63a2e0 msgr2=0x56218b63eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 --2- 172.21.15.62:0/2915249341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b63a2e0 0x56218b63eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff01d7fa640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0080a32f0 0x7ff0080a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 -- 172.21.15.62:0/2915249341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 msgr2=0x56218b641830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 --2- 172.21.15.62:0/2915249341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b641830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 -- 172.21.15.62:0/2915249341 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56218b544340 con 0x56218b6399a0 2024-09-15T23:18:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff01d7fa640 1 -- 172.21.15.62:0/1365953708 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff0100cfa70 con 0x7ff0201025e0 2024-09-15T23:18:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 --2- 172.21.15.62:0/2915249341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b6399a0 0x56218b639da0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff018035000 tx=0x7ff018009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5244040a95bbe584 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0248df640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0080a32f0 0x7ff0080a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.710+0000 7ff0067fc640 1 -- 172.21.15.62:0/2915249341 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff018031590 con 0x56218b6399a0 2024-09-15T23:18:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0067fc640 1 -- 172.21.15.62:0/2915249341 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff018036050 con 0x56218b6399a0 2024-09-15T23:18:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0067fc640 1 -- 172.21.15.62:0/2915249341 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01802fcf0 con 0x56218b6399a0 2024-09-15T23:18:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0290da740 1 -- 172.21.15.62:0/2915249341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b6399a0 msgr2=0x56218b639da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0290da740 1 --2- 172.21.15.62:0/2915249341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b6399a0 0x56218b639da0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ff018035000 tx=0x7ff018009800 comp rx=0 tx=0).stop 2024-09-15T23:18:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0248df640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0080a32f0 0x7ff0080a57b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff018009ca0 tx=0x7ff01804a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0290da740 1 -- 172.21.15.62:0/2915249341 shutdown_connections 2024-09-15T23:18:30.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0290da740 1 --2- 172.21.15.62:0/2915249341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b641830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0290da740 1 --2- 172.21.15.62:0/2915249341 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b63a2e0 0x56218b63eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0290da740 1 --2- 172.21.15.62:0/2915249341 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b6399a0 0x56218b639da0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0290da740 1 -- 172.21.15.62:0/2915249341 >> 172.21.15.62:0/2915249341 conn(0x56218b635010 msgr2=0x56218b635410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0290da740 1 -- 172.21.15.62:0/2915249341 shutdown_connections 2024-09-15T23:18:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.711+0000 7ff0290da740 1 -- 172.21.15.62:0/2915249341 wait complete. 2024-09-15T23:18:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0290da740 1 -- start start 2024-09-15T23:18:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0290da740 1 Processor -- start 2024-09-15T23:18:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0290da740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b6399a0 0x56218b643a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0290da740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b643fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0290da740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b64af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0290da740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56218b6421c0 con 0x56218b63a2e0 2024-09-15T23:18:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0290da740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56218b526290 con 0x56218b63f370 2024-09-15T23:18:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0290da740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56218b642340 con 0x56218b6399a0 2024-09-15T23:18:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0290da740 10 client.0 _ll_register_callbacks cb 0x56218b557140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff01f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b643fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff01f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b643fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60820/0 (socket says 172.21.15.62:60820) 2024-09-15T23:18:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff01f7fe640 1 -- 172.21.15.62:0/587346972 learned_addr learned my addr 172.21.15.62:0/587346972 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.712+0000 7ff0248df640 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b64af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff01ffff640 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b6399a0 0x56218b643a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff01f7fe640 1 -- 172.21.15.62:0/587346972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b6399a0 msgr2=0x56218b643a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff01f7fe640 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b6399a0 0x56218b643a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff01f7fe640 1 -- 172.21.15.62:0/587346972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 msgr2=0x56218b64af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff01f7fe640 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b64af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff01f7fe640 1 -- 172.21.15.62:0/587346972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56218b2b0210 con 0x56218b63a2e0 2024-09-15T23:18:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff01f7fe640 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b643fc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff014012700 tx=0x7ff014012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a4f0d376eda09d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7fefff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7fefff7fe640 1 -- 172.21.15.62:0/587346972 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014004050 con 0x56218b63a2e0 2024-09-15T23:18:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7fefff7fe640 1 -- 172.21.15.62:0/587346972 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff0140041f0 con 0x56218b63a2e0 2024-09-15T23:18:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff0290da740 1 -- 172.21.15.62:0/587346972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff018036050 con 0x56218b63a2e0 2024-09-15T23:18:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7fefff7fe640 1 -- 172.21.15.62:0/587346972 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff014025050 con 0x56218b63a2e0 2024-09-15T23:18:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff0290da740 10 client.16038 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:30.736 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7ff0290da740 1 -- 172.21.15.62:0/587346972 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56218b37c620 con 0x56218b63a2e0 2024-09-15T23:18:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.713+0000 7fefff7fe640 1 -- 172.21.15.62:0/587346972 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff018036050 con 0x56218b63a2e0 2024-09-15T23:18:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7fefff7fe640 1 -- 172.21.15.62:0/587346972 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7ff014003a40 con 0x56218b63a2e0 2024-09-15T23:18:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7ff0290da740 10 client.16038 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7ff0290da740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7ff0290da740 20 client.16038 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7ff0290da740 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56218b64f760 0x56218b651be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7ff0290da740 4 client.16038 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7ff0290da740 1 -- 172.21.15.62:0/587346972 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56218b5aaa20 con 0x56218b64f760 2024-09-15T23:18:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7ff01ffff640 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56218b64f760 0x56218b651be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7ff01ffff640 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56218b64f760 0x56218b651be0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=524b93ec768c22e0 server_cookie=6b128541ed8d0127 in_seq=0 out_seq=0 2024-09-15T23:18:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.714+0000 7fefff7fe640 10 client.16038 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.715+0000 7fefff7fe640 1 -- 172.21.15.62:0/587346972 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56218b5aaa20 con 0x56218b64f760 2024-09-15T23:18:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.715+0000 7fefff7fe640 10 client.16038 handle_command_reply: tid=0 2024-09-15T23:18:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.715+0000 7ff0290da740 1 client.16038 shutdown 2024-09-15T23:18:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 -- 172.21.15.62:0/587346972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 msgr2=0x56218b643fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b643fc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff014012700 tx=0x7ff014012bc0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 -- 172.21.15.62:0/587346972 shutdown_connections 2024-09-15T23:18:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56218b64f760 0x56218b651be0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b64af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b643fc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7fefff7fe640 0 client.16038 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 --2- 172.21.15.62:0/587346972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b6399a0 0x56218b643a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 -- 172.21.15.62:0/587346972 >> 172.21.15.62:0/587346972 conn(0x56218b635010 msgr2=0x56218b636820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 -- 172.21.15.62:0/587346972 shutdown_connections 2024-09-15T23:18:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 1 -- 172.21.15.62:0/587346972 wait complete. 2024-09-15T23:18:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.716+0000 7ff0290da740 20 client.16038 trim_cache size 0 max 16384 2024-09-15T23:18:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.730+0000 7ff0290da740 1 Processor -- start 2024-09-15T23:18:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.730+0000 7ff0290da740 1 -- start start 2024-09-15T23:18:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0290da740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b6399a0 0x56218b635e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0290da740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b63a2e0 0x56218b64af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0290da740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63f370 0x56218b6363d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0290da740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff0201af170 con 0x56218b63f370 2024-09-15T23:18:30.751 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0290da740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff0201aca10 con 0x56218b6399a0 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0290da740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff0201aa860 con 0x56218b63a2e0 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0248df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63f370 0x56218b6363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0248df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63f370 0x56218b6363d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60830/0 (socket says 172.21.15.62:60830) 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0248df640 1 -- 172.21.15.62:0/2663546632 learned_addr learned my addr 172.21.15.62:0/2663546632 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff01ffff640 1 --2- 172.21.15.62:0/2663546632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b6399a0 0x56218b635e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0248df640 1 -- 172.21.15.62:0/2663546632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b63a2e0 msgr2=0x56218b64af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0248df640 1 --2- 172.21.15.62:0/2663546632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b63a2e0 0x56218b64af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0248df640 1 -- 172.21.15.62:0/2663546632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b6399a0 msgr2=0x56218b635e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0248df640 1 --2- 172.21.15.62:0/2663546632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b6399a0 0x56218b635e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0248df640 1 -- 172.21.15.62:0/2663546632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56218b634e30 con 0x56218b63f370 2024-09-15T23:18:30.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff0248df640 1 --2- 172.21.15.62:0/2663546632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63f370 0x56218b6363d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff018009430 tx=0x7ff018050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474ad8fb68acbaed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff005ffb640 1 -- 172.21.15.62:0/2663546632 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0180044e0 con 0x56218b63f370 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff005ffb640 1 -- 172.21.15.62:0/2663546632 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff018058070 con 0x56218b63f370 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.731+0000 7ff005ffb640 1 -- 172.21.15.62:0/2663546632 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01802f870 con 0x56218b63f370 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 -- 172.21.15.62:0/2663546632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63f370 msgr2=0x56218b6363d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 --2- 172.21.15.62:0/2663546632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63f370 0x56218b6363d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff018009430 tx=0x7ff018050000 comp rx=0 tx=0).stop 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 -- 172.21.15.62:0/2663546632 shutdown_connections 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 --2- 172.21.15.62:0/2663546632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63f370 0x56218b6363d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 --2- 172.21.15.62:0/2663546632 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b63a2e0 0x56218b64af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 --2- 172.21.15.62:0/2663546632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b6399a0 0x56218b635e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 -- 172.21.15.62:0/2663546632 >> 172.21.15.62:0/2663546632 conn(0x56218b635010 msgr2=0x56218b63b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 -- 172.21.15.62:0/2663546632 shutdown_connections 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 -- 172.21.15.62:0/2663546632 wait complete. 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.732+0000 7ff0290da740 1 -- start start 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0290da740 1 Processor -- start 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0290da740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b6399a0 0x56218b6486b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0290da740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b648bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0290da740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b649b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0290da740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff0201aaed0 con 0x56218b63a2e0 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0290da740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff0201ac340 con 0x56218b63f370 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0290da740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff0201ab5a0 con 0x56218b6399a0 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0290da740 10 client.0 _ll_register_callbacks cb 0x7ff0140097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff01f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b648bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff01f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b648bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60846/0 (socket says 172.21.15.62:60846) 2024-09-15T23:18:30.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff01f7fe640 1 -- 172.21.15.62:0/2118689620 learned_addr learned my addr 172.21.15.62:0/2118689620 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0248df640 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b649b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff01f7fe640 1 -- 172.21.15.62:0/2118689620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b6399a0 msgr2=0x56218b6486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff01f7fe640 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b6399a0 0x56218b6486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff01f7fe640 1 -- 172.21.15.62:0/2118689620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 msgr2=0x56218b649b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff01f7fe640 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b649b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff01f7fe640 1 -- 172.21.15.62:0/2118689620 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56218b64a120 con 0x56218b63a2e0 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.733+0000 7ff0248df640 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b649b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff01f7fe640 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b648bf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff014009d30 tx=0x7ff014010760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bbbeb3f8db5d9bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7fefff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7fefff7fe640 1 -- 172.21.15.62:0/2118689620 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0140052f0 con 0x56218b63a2e0 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7fefff7fe640 1 -- 172.21.15.62:0/2118689620 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff014022070 con 0x56218b63a2e0 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff0290da740 1 -- 172.21.15.62:0/2118689620 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff018058070 con 0x56218b63a2e0 2024-09-15T23:18:30.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7fefff7fe640 1 -- 172.21.15.62:0/2118689620 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff014005490 con 0x56218b63a2e0 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff0290da740 10 client.16044 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff0290da740 1 -- 172.21.15.62:0/2118689620 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56218b6ba480 con 0x56218b63a2e0 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7fefff7fe640 1 -- 172.21.15.62:0/2118689620 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff018058070 con 0x56218b63a2e0 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7fefff7fe640 1 -- 172.21.15.62:0/2118689620 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7ff014010930 con 0x56218b63a2e0 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff0290da740 10 client.16044 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff0290da740 10 client.16044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff0290da740 20 client.16044 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff0290da740 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56218b6bae80 0x56218b6bd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff0290da740 4 client.16044 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.734+0000 7ff0290da740 1 -- 172.21.15.62:0/2118689620 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56218b5a4d70 con 0x56218b6bae80 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.735+0000 7ff01ffff640 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56218b6bae80 0x56218b6bd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.735+0000 7ff01ffff640 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56218b6bae80 0x56218b6bd2a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=679f3a1aeec44f57 server_cookie=89d391a99ab4306a in_seq=0 out_seq=0 2024-09-15T23:18:30.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.735+0000 7fefff7fe640 10 client.16044 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7fefff7fe640 1 -- 172.21.15.62:0/2118689620 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56218b5a4d70 con 0x56218b6bae80 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7fefff7fe640 10 client.16044 handle_command_reply: tid=0 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 client.16044 shutdown 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 -- 172.21.15.62:0/2118689620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 msgr2=0x56218b648bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b648bf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7ff014009d30 tx=0x7ff014010760 comp rx=0 tx=0).stop 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 -- 172.21.15.62:0/2118689620 shutdown_connections 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56218b6bae80 0x56218b6bd2a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56218b63f370 0x56218b649b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56218b63a2e0 0x56218b648bf0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7fefff7fe640 0 client.16044 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 --2- 172.21.15.62:0/2118689620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56218b6399a0 0x56218b6486b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 -- 172.21.15.62:0/2118689620 >> 172.21.15.62:0/2118689620 conn(0x56218b635010 msgr2=0x56218b6339a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 -- 172.21.15.62:0/2118689620 shutdown_connections 2024-09-15T23:18:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 1 -- 172.21.15.62:0/2118689620 wait complete. 2024-09-15T23:18:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.748+0000 7ff0290da740 20 client.16044 trim_cache size 0 max 16384 2024-09-15T23:18:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:18:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:18:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:18:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:18:30.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 -- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0080a32f0 msgr2=0x7ff0080a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0080a32f0 0x7ff0080a57b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff018009ca0 tx=0x7ff01804a000 comp rx=0 tx=0).stop 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 -- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0201025e0 msgr2=0x7ff020101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0201025e0 0x7ff020101f50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ff01000c970 tx=0x7ff01000ce30 comp rx=0 tx=0).stop 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 -- 172.21.15.62:0/1365953708 shutdown_connections 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff0080a32f0 0x7ff0080a57b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff020107840 0x7ff020100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020104f10 0x7ff020100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 --2- 172.21.15.62:0/1365953708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff0201025e0 0x7ff020101f50 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 -- 172.21.15.62:0/1365953708 >> 172.21.15.62:0/1365953708 conn(0x7ff0200fc1e0 msgr2=0x7ff020102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 -- 172.21.15.62:0/1365953708 shutdown_connections 2024-09-15T23:18:30.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:30.749+0000 7ff026342640 1 -- 172.21.15.62:0/1365953708 wait complete. 2024-09-15T23:18:31.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:31 smithi062 ceph-mon[30949]: pgmap v437: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 2.5 MiB/s wr, 293 op/s 2024-09-15T23:18:32.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:31 smithi067 ceph-mon[35665]: pgmap v437: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 2.5 MiB/s wr, 293 op/s 2024-09-15T23:18:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:31 smithi148 ceph-mon[35610]: pgmap v437: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 2.5 MiB/s wr, 293 op/s 2024-09-15T23:18:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:33 smithi062 ceph-mon[30949]: pgmap v438: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-09-15T23:18:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:33 smithi067 ceph-mon[35665]: pgmap v438: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-09-15T23:18:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:33 smithi148 ceph-mon[35610]: pgmap v438: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-09-15T23:18:34.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 135.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:18:35.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:35 smithi062 ceph-mon[30949]: pgmap v439: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 3.0 MiB/s wr, 264 op/s 2024-09-15T23:18:36.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:35 smithi067 ceph-mon[35665]: pgmap v439: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 3.0 MiB/s wr, 264 op/s 2024-09-15T23:18:36.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:35 smithi148 ceph-mon[35610]: pgmap v439: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 3.0 MiB/s wr, 264 op/s 2024-09-15T23:18:37.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:37 smithi062 ceph-mon[30949]: pgmap v440: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 2.2 MiB/s wr, 178 op/s 2024-09-15T23:18:37.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:37 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:38.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:37 smithi067 ceph-mon[35665]: pgmap v440: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 2.2 MiB/s wr, 178 op/s 2024-09-15T23:18:38.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:37 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:38.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:37 smithi148 ceph-mon[35610]: pgmap v440: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 2.2 MiB/s wr, 178 op/s 2024-09-15T23:18:38.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:37 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:18:38.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:38.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:38.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:18:39.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:39.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:39.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:18:39.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:18:39.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:18:39.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:18:39.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 140.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:18:39.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:39 smithi062 ceph-mon[30949]: pgmap v441: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 995 KiB/s wr, 68 op/s 2024-09-15T23:18:40.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:39 smithi067 ceph-mon[35665]: pgmap v441: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 995 KiB/s wr, 68 op/s 2024-09-15T23:18:40.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:39 smithi148 ceph-mon[35610]: pgmap v441: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 995 KiB/s wr, 68 op/s 2024-09-15T23:18:40.337 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:18:40.609 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.048+0000 7fcb0a368640 1 -- 172.21.15.62:0/2593740355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 msgr2=0x7fcb0410ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.048+0000 7fcb0a368640 1 --2- 172.21.15.62:0/2593740355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 0x7fcb0410ea90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009ca0 tx=0x7fcafc02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.048+0000 7fcb0a368640 1 -- 172.21.15.62:0/2593740355 shutdown_connections 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.048+0000 7fcb0a368640 1 --2- 172.21.15.62:0/2593740355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 0x7fcb0410ea90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.048+0000 7fcb0a368640 1 --2- 172.21.15.62:0/2593740355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb04102790 0x7fcb041067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.048+0000 7fcb0a368640 1 --2- 172.21.15.62:0/2593740355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb04101de0 0x7fcb041021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.048+0000 7fcb0a368640 1 -- 172.21.15.62:0/2593740355 >> 172.21.15.62:0/2593740355 conn(0x7fcb040fd8c0 msgr2=0x7fcb040ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.048+0000 7fcb0a368640 1 -- 172.21.15.62:0/2593740355 shutdown_connections 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.048+0000 7fcb0a368640 1 -- 172.21.15.62:0/2593740355 wait complete. 2024-09-15T23:18:41.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb0a368640 1 Processor -- start 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb0a368640 1 -- start start 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb0a368640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb04101de0 0x7fcb040744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb0a368640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb04102790 0x7fcb04074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb0a368640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 0x7fcb04074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb0a368640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcb04111f70 con 0x7fcb04106f10 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb0a368640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb04111df0 con 0x7fcb04102790 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb0a368640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcb041120f0 con 0x7fcb04101de0 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb08905640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 0x7fcb04074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb08905640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 0x7fcb04074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33630/0 (socket says 172.21.15.62:33630) 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb08905640 1 -- 172.21.15.62:0/3049139352 learned_addr learned my addr 172.21.15.62:0/3049139352 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb037fe640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb04102790 0x7fcb04074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:41.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.049+0000 7fcb03fff640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb04101de0 0x7fcb040744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb08905640 1 -- 172.21.15.62:0/3049139352 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb04101de0 msgr2=0x7fcb040744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb08905640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb04101de0 0x7fcb040744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb08905640 1 -- 172.21.15.62:0/3049139352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb04102790 msgr2=0x7fcb04074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb08905640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb04102790 0x7fcb04074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb08905640 1 -- 172.21.15.62:0/3049139352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb041a6890 con 0x7fcb04106f10 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb08905640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 0x7fcb04074f70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcafc031a20 tx=0x7fcafc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb017fa640 1 -- 172.21.15.62:0/3049139352 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc003c30 con 0x7fcb04106f10 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb017fa640 1 -- 172.21.15.62:0/3049139352 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcafc003dd0 con 0x7fcb04106f10 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb041a6b20 con 0x7fcb04106f10 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb017fa640 1 -- 172.21.15.62:0/3049139352 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc02fcf0 con 0x7fcb04106f10 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.050+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb041a7060 con 0x7fcb04106f10 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.051+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb041021c0 con 0x7fcb04106f10 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.052+0000 7fcb017fa640 1 -- 172.21.15.62:0/3049139352 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fcafc004050 con 0x7fcb04106f10 2024-09-15T23:18:41.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.053+0000 7fcb017fa640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcaec0a3240 0x7fcaec0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:41.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.053+0000 7fcb03fff640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcaec0a3240 0x7fcaec0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:41.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.053+0000 7fcb017fa640 1 -- 172.21.15.62:0/3049139352 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcafc0f0fe0 con 0x7fcb04106f10 2024-09-15T23:18:41.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.053+0000 7fcb03fff640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcaec0a3240 0x7fcaec0a5700 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4005fd0 tx=0x7fcaf4005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:41.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.057+0000 7fcb017fa640 1 -- 172.21.15.62:0/3049139352 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fcafc0bda10 con 0x7fcb04106f10 2024-09-15T23:18:41.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.268+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb0410a150 con 0x7fcb04106f10 2024-09-15T23:18:41.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.268+0000 7fcb017fa640 1 -- 172.21.15.62:0/3049139352 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11907 (secure 0 0 0) 0x7fcafc02fe90 con 0x7fcb04106f10 2024-09-15T23:18:41.269 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:18:41.270 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":28,"btime":"2024-09-15T23:17:33:092139+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:17:32.873645+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:18:41.270 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 28 2024-09-15T23:18:41.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.270+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcaec0a3240 msgr2=0x7fcaec0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.270+0000 7fcb0a368640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcaec0a3240 0x7fcaec0a5700 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4005fd0 tx=0x7fcaf4005e20 comp rx=0 tx=0).stop 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.270+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 msgr2=0x7fcb04074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.270+0000 7fcb0a368640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 0x7fcb04074f70 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcafc031a20 tx=0x7fcafc031a50 comp rx=0 tx=0).stop 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.271+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 shutdown_connections 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.271+0000 7fcb0a368640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcaec0a3240 0x7fcaec0a5700 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.271+0000 7fcb0a368640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb04106f10 0x7fcb04074f70 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.271+0000 7fcb0a368640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb04102790 0x7fcb04074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.271+0000 7fcb0a368640 1 --2- 172.21.15.62:0/3049139352 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb04101de0 0x7fcb040744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.271+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 >> 172.21.15.62:0/3049139352 conn(0x7fcb040fd8c0 msgr2=0x7fcb040feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.271+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 shutdown_connections 2024-09-15T23:18:41.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:41.271+0000 7fcb0a368640 1 -- 172.21.15.62:0/3049139352 wait complete. 2024-09-15T23:18:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:41 smithi062 ceph-mon[30949]: pgmap v442: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.0 MiB/s wr, 182 op/s 2024-09-15T23:18:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:41 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3049139352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:41 smithi067 ceph-mon[35665]: pgmap v442: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.0 MiB/s wr, 182 op/s 2024-09-15T23:18:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:41 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3049139352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:41 smithi148 ceph-mon[35610]: pgmap v442: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.0 MiB/s wr, 182 op/s 2024-09-15T23:18:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:41 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3049139352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:42.041 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 5d3ecd09 --release --await' 2024-09-15T23:18:42.042 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 5d3ecd09 --release --await 2024-09-15T23:18:42.303 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:43.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.118+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3002756875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4101400 msgr2=0x7f96d410d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.118+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3002756875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4101400 0x7f96d410d930 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009cd0 tx=0x7f96cc02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3002756875 shutdown_connections 2024-09-15T23:18:43.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3002756875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4101400 0x7f96d410d930 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3002756875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96d4100a60 0x7f96d4100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3002756875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d4106a60 0x7f96d4106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3002756875 >> 172.21.15.62:0/3002756875 conn(0x7f96d40fc780 msgr2=0x7f96d40feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3002756875 shutdown_connections 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3002756875 wait complete. 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 Processor -- start 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 -- start start 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4100a60 0x7f96d4198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96d4101400 0x7f96d41989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d4106a60 0x7f96d419cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f96d4111f70 con 0x7f96d4100a60 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96d4111df0 con 0x7f96d4106a60 2024-09-15T23:18:43.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.119+0000 7f96dbfc1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41120f0 con 0x7f96d4101400 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d9d5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4100a60 0x7f96d4198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d9d5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4100a60 0x7f96d4198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41984/0 (socket says 172.21.15.62:41984) 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d9d5d640 1 -- 172.21.15.62:0/3069294452 learned_addr learned my addr 172.21.15.62:0/3069294452 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96da55e640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d4106a60 0x7f96d419cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d955c640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96d4101400 0x7f96d41989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d9d5d640 1 -- 172.21.15.62:0/3069294452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96d4101400 msgr2=0x7f96d41989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96d4101400 0x7f96d41989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d9d5d640 1 -- 172.21.15.62:0/3069294452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d4106a60 msgr2=0x7f96d419cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d4106a60 0x7f96d419cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d9d5d640 1 -- 172.21.15.62:0/3069294452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d419d420 con 0x7f96d4100a60 2024-09-15T23:18:43.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d955c640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96d4101400 0x7f96d41989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4100a60 0x7f96d4198470 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f96d000d950 tx=0x7f96d000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96caffd640 1 -- 172.21.15.62:0/3069294452 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0017070 con 0x7f96d4100a60 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3069294452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d419d710 con 0x7f96d4100a60 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96caffd640 1 -- 172.21.15.62:0/3069294452 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96d0009c30 con 0x7f96d4100a60 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96caffd640 1 -- 172.21.15.62:0/3069294452 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d000f6b0 con 0x7f96d4100a60 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.120+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3069294452 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d41a4ff0 con 0x7f96d4100a60 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.123+0000 7f96caffd640 1 -- 172.21.15.62:0/3069294452 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f96d00109d0 con 0x7f96d4100a60 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.123+0000 7f96de39a740 1 Processor -- start 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.123+0000 7f96de39a740 1 -- start start 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96de39a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607db320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96de39a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607db860 0x55fe607ea110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96de39a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607ea650 0x55fe607eca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96de39a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fe60740620 con 0x55fe607daf40 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96de39a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe606e7e50 con 0x55fe607db860 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96de39a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55fe6073c5e0 con 0x55fe607ea650 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607db320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607db320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42000/0 (socket says 172.21.15.62:42000) 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 -- 172.21.15.62:0/3654839775 learned_addr learned my addr 172.21.15.62:0/3654839775 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607db860 0x55fe607ea110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96caffd640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f96c00a3240 0x7f96c00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96d955c640 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607ea650 0x55fe607eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 -- 172.21.15.62:0/3654839775 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607ea650 msgr2=0x55fe607eca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607ea650 0x55fe607eca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 -- 172.21.15.62:0/3654839775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607db860 msgr2=0x55fe607ea110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607db860 0x55fe607ea110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 -- 172.21.15.62:0/3654839775 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe606d8d40 con 0x55fe607daf40 2024-09-15T23:18:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607db860 0x55fe607ea110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f96c00a3240 0x7f96c00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96da55e640 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607db320 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009a60 tx=0x7f96cc031430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399a79ca32fa926c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.124+0000 7f96caffd640 1 -- 172.21.15.62:0/3069294452 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f96d00cf9a0 con 0x7f96d4100a60 2024-09-15T23:18:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.125+0000 7f96bffff640 1 -- 172.21.15.62:0/3654839775 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc04d070 con 0x55fe607daf40 2024-09-15T23:18:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.125+0000 7f96bffff640 1 -- 172.21.15.62:0/3654839775 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96cc043070 con 0x55fe607daf40 2024-09-15T23:18:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.125+0000 7f96bffff640 1 -- 172.21.15.62:0/3654839775 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc030c60 con 0x55fe607daf40 2024-09-15T23:18:43.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.125+0000 7f96da55e640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f96c00a3240 0x7f96c00a5700 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f96cc046000 tx=0x7f96cc04b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.125+0000 7f96de39a740 1 -- 172.21.15.62:0/3654839775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 msgr2=0x55fe607db320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607db320 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009a60 tx=0x7f96cc031430 comp rx=0 tx=0).stop 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 -- 172.21.15.62:0/3654839775 shutdown_connections 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607ea650 0x55fe607eca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607db860 0x55fe607ea110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 --2- 172.21.15.62:0/3654839775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607db320 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 -- 172.21.15.62:0/3654839775 >> 172.21.15.62:0/3654839775 conn(0x55fe607d75f0 msgr2=0x55fe607d79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 -- 172.21.15.62:0/3654839775 shutdown_connections 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 -- 172.21.15.62:0/3654839775 wait complete. 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 -- start start 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.126+0000 7f96de39a740 1 Processor -- start 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96de39a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607daf40 0x55fe607e75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96de39a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe607e7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96de39a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607ea650 0x55fe607e8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96de39a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fe607ed220 con 0x55fe607ea650 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96de39a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe606c8f10 con 0x55fe607daf40 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96de39a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55fe607ed3a0 con 0x55fe607db860 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96da55e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607ea650 0x55fe607e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96d9d5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607daf40 0x55fe607e75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96da55e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607ea650 0x55fe607e8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42016/0 (socket says 172.21.15.62:42016) 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96da55e640 1 -- 172.21.15.62:0/3206798214 learned_addr learned my addr 172.21.15.62:0/3206798214 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96de39a740 10 client.0 _ll_register_callbacks cb 0x55fe606ecdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96d955c640 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe607e7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96da55e640 1 -- 172.21.15.62:0/3206798214 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 msgr2=0x55fe607e7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96da55e640 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe607e7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96da55e640 1 -- 172.21.15.62:0/3206798214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607daf40 msgr2=0x55fe607e75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96da55e640 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607daf40 0x55fe607e75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96da55e640 1 -- 172.21.15.62:0/3206798214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe6044f580 con 0x55fe607ea650 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96d955c640 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe607e7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96da55e640 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607ea650 0x55fe607e8030 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f96cc031de0 tx=0x7f96cc031920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c8524c0e087c37 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.127+0000 7f96bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96bd7fa640 1 -- 172.21.15.62:0/3206798214 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc04d030 con 0x55fe607ea650 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96de39a740 1 -- 172.21.15.62:0/3206798214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96cc043070 con 0x55fe607ea650 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96bd7fa640 1 -- 172.21.15.62:0/3206798214 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f96cc004490 con 0x55fe607ea650 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96bd7fa640 1 -- 172.21.15.62:0/3206798214 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f96cc062050 con 0x55fe607ea650 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96de39a740 10 client.16062 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96de39a740 1 -- 172.21.15.62:0/3206798214 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fe60519960 con 0x55fe607ea650 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96bd7fa640 1 -- 172.21.15.62:0/3206798214 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96cc043070 con 0x55fe607ea650 2024-09-15T23:18:43.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96bd7fa640 1 -- 172.21.15.62:0/3206798214 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f96cc030070 con 0x55fe607ea650 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96de39a740 10 client.16062 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96de39a740 10 client.16062 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96de39a740 20 client.16062 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96de39a740 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe607fa0f0 0x55fe607fc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96de39a740 4 client.16062 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.128+0000 7f96de39a740 1 -- 172.21.15.62:0/3206798214 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe6074db90 con 0x55fe607fa0f0 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.129+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe607fa0f0 0x55fe607fc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.129+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe607fa0f0 0x55fe607fc4e0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed0ca5d7fe91b42 server_cookie=65062b7cdc0af75b in_seq=0 out_seq=0 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.129+0000 7f96bd7fa640 10 client.16062 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96bd7fa640 1 -- 172.21.15.62:0/3206798214 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe6074db90 con 0x55fe607fa0f0 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96bd7fa640 10 client.16062 handle_command_reply: tid=0 2024-09-15T23:18:43.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 client.16062 shutdown 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 -- 172.21.15.62:0/3206798214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607ea650 msgr2=0x55fe607e8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607ea650 0x55fe607e8030 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f96cc031de0 tx=0x7f96cc031920 comp rx=0 tx=0).stop 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 -- 172.21.15.62:0/3206798214 shutdown_connections 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe607fa0f0 0x55fe607fc4e0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607ea650 0x55fe607e8030 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96bd7fa640 0 client.16062 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe607e7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 --2- 172.21.15.62:0/3206798214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607daf40 0x55fe607e75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 -- 172.21.15.62:0/3206798214 >> 172.21.15.62:0/3206798214 conn(0x55fe607d75f0 msgr2=0x55fe607dd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 -- 172.21.15.62:0/3206798214 shutdown_connections 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 1 -- 172.21.15.62:0/3206798214 wait complete. 2024-09-15T23:18:43.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.130+0000 7f96de39a740 20 client.16062 trim_cache size 0 max 16384 2024-09-15T23:18:43.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.147+0000 7f96de39a740 1 Processor -- start 2024-09-15T23:18:43.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.147+0000 7f96de39a740 1 -- start start 2024-09-15T23:18:43.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.147+0000 7f96de39a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe60801ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.147+0000 7f96de39a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe60802020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.147+0000 7f96de39a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 0x55fe607f2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.147+0000 7f96de39a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41af400 con 0x55fe607daf40 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.147+0000 7f96de39a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41acca0 con 0x55fe607ea650 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.147+0000 7f96de39a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41aaaf0 con 0x55fe607db860 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d9d5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe60801ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d9d5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe60801ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42030/0 (socket says 172.21.15.62:42030) 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d9d5d640 1 -- 172.21.15.62:0/1367223925 learned_addr learned my addr 172.21.15.62:0/1367223925 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d955c640 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe60802020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96da55e640 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 0x55fe607f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d9d5d640 1 -- 172.21.15.62:0/1367223925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 msgr2=0x55fe60802020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe60802020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d9d5d640 1 -- 172.21.15.62:0/1367223925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 msgr2=0x55fe607f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 0x55fe607f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d9d5d640 1 -- 172.21.15.62:0/1367223925 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe607e6560 con 0x55fe607daf40 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d955c640 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe60802020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe60801ae0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f96d00985b0 tx=0x7f96d000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816a9c3bf6c5ebfd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96bf7fe640 1 -- 172.21.15.62:0/1367223925 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0020040 con 0x55fe607daf40 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96bf7fe640 1 -- 172.21.15.62:0/1367223925 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96d009ff90 con 0x55fe607daf40 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.148+0000 7f96bf7fe640 1 -- 172.21.15.62:0/1367223925 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0086330 con 0x55fe607daf40 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 -- 172.21.15.62:0/1367223925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 msgr2=0x55fe60801ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe60801ae0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f96d00985b0 tx=0x7f96d000bf60 comp rx=0 tx=0).stop 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 -- 172.21.15.62:0/1367223925 shutdown_connections 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 0x55fe607f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe60802020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 --2- 172.21.15.62:0/1367223925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe60801ae0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 -- 172.21.15.62:0/1367223925 >> 172.21.15.62:0/1367223925 conn(0x55fe607d75f0 msgr2=0x55fe607e8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 -- 172.21.15.62:0/1367223925 shutdown_connections 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 -- 172.21.15.62:0/1367223925 wait complete. 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 -- start start 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.149+0000 7f96de39a740 1 Processor -- start 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96de39a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607ef2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96de39a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe607ef7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96de39a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 0x55fe607efd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96de39a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41ab160 con 0x55fe607daf40 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96de39a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41ac5d0 con 0x55fe607ea650 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96de39a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41ab830 con 0x55fe607db860 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96de39a740 10 client.0 _ll_register_callbacks cb 0x7f96cc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:43.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d9d5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607ef2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d9d5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607ef2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42032/0 (socket says 172.21.15.62:42032) 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d9d5d640 1 -- 172.21.15.62:0/2416389670 learned_addr learned my addr 172.21.15.62:0/2416389670 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96da55e640 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 0x55fe607efd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d955c640 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe607ef7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d9d5d640 1 -- 172.21.15.62:0/2416389670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 msgr2=0x55fe607ef7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe607ef7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d9d5d640 1 -- 172.21.15.62:0/2416389670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 msgr2=0x55fe607efd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 0x55fe607efd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d9d5d640 1 -- 172.21.15.62:0/2416389670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe60864ac0 con 0x55fe607daf40 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96d9d5d640 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607ef2b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009610 tx=0x7f96d009bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e27f7968b40ebc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.150+0000 7f96bd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96bd7fa640 1 -- 172.21.15.62:0/2416389670 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0020050 con 0x55fe607daf40 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96bd7fa640 1 -- 172.21.15.62:0/2416389670 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f96d00860d0 con 0x55fe607daf40 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96de39a740 1 -- 172.21.15.62:0/2416389670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96d009ff90 con 0x55fe607daf40 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96bd7fa640 1 -- 172.21.15.62:0/2416389670 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f96d0086310 con 0x55fe607daf40 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96de39a740 10 client.16068 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96de39a740 1 -- 172.21.15.62:0/2416389670 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fe60865c70 con 0x55fe607daf40 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96bd7fa640 1 -- 172.21.15.62:0/2416389670 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96d009ff90 con 0x55fe607daf40 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96bd7fa640 1 -- 172.21.15.62:0/2416389670 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f96d0010460 con 0x55fe607daf40 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96de39a740 10 client.16068 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:43.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96de39a740 10 client.16068 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:18:43.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96de39a740 20 client.16068 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:43.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96de39a740 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe60866310 0x55fe60868730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96de39a740 4 client.16068 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5d3ecd09", "release": true, "await": true}] 2024-09-15T23:18:43.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.151+0000 7f96de39a740 1 -- 172.21.15.62:0/2416389670 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5d3ecd09", "release": true, "await": true}) -- 0x55fe607480a0 con 0x55fe60866310 2024-09-15T23:18:43.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.152+0000 7f96d955c640 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe60866310 0x55fe60868730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.152+0000 7f96d955c640 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe60866310 0x55fe60868730 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb8fd324d0a1c547 server_cookie=8b92ca7ff61063b6 in_seq=0 out_seq=0 2024-09-15T23:18:43.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.152+0000 7f96bd7fa640 10 client.16068 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:43.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.182+0000 7f96bd7fa640 1 -- 172.21.15.62:0/2416389670 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55fe607480a0 con 0x55fe60866310 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.182+0000 7f96bd7fa640 10 client.16068 handle_command_reply: tid=0 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.182+0000 7f96de39a740 1 client.16068 shutdown 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.182+0000 7f96de39a740 1 -- 172.21.15.62:0/2416389670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 msgr2=0x55fe607ef2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.182+0000 7f96de39a740 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607ef2b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f96d0009610 tx=0x7f96d009bd60 comp rx=0 tx=0).stop 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96de39a740 1 -- 172.21.15.62:0/2416389670 shutdown_connections 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96de39a740 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe60866310 0x55fe60868730 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96de39a740 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe607ea650 0x55fe607efd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96bd7fa640 0 client.16068 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96de39a740 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe607db860 0x55fe607ef7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96de39a740 1 --2- 172.21.15.62:0/2416389670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe607daf40 0x55fe607ef2b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96de39a740 1 -- 172.21.15.62:0/2416389670 >> 172.21.15.62:0/2416389670 conn(0x55fe607d75f0 msgr2=0x55fe607dca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96de39a740 1 -- 172.21.15.62:0/2416389670 shutdown_connections 2024-09-15T23:18:43.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96de39a740 1 -- 172.21.15.62:0/2416389670 wait complete. 2024-09-15T23:18:43.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.183+0000 7f96de39a740 20 client.16068 trim_cache size 0 max 16384 2024-09-15T23:18:43.186 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:18:43.186 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 28, 2024-09-15T23:18:43.186 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:18:43.186 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 12, 2024-09-15T23:18:43.186 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:18:43.186 INFO:teuthology.orchestra.run.smithi062.stdout: "5d3ecd09": { 2024-09-15T23:18:43.186 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 12, 2024-09-15T23:18:43.186 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 139.6, 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:18:43.187 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:18:43.188 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:43.188 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:43.188 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:43.188 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:43.188 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:18:43.188 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:18:43.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.184+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f96c00a3240 msgr2=0x7f96c00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.184+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f96c00a3240 0x7f96c00a5700 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f96cc046000 tx=0x7f96cc04b000 comp rx=0 tx=0).stop 2024-09-15T23:18:43.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.184+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4100a60 msgr2=0x7f96d4198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.184+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4100a60 0x7f96d4198470 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f96d000d950 tx=0x7f96d000de10 comp rx=0 tx=0).stop 2024-09-15T23:18:43.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.185+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3069294452 shutdown_connections 2024-09-15T23:18:43.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.185+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f96c00a3240 0x7f96c00a5700 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.185+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d4106a60 0x7f96d419cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.185+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96d4101400 0x7f96d41989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.185+0000 7f96dbfc1640 1 --2- 172.21.15.62:0/3069294452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96d4100a60 0x7f96d4198470 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.185+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3069294452 >> 172.21.15.62:0/3069294452 conn(0x7f96d40fc780 msgr2=0x7f96d40fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.185+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3069294452 shutdown_connections 2024-09-15T23:18:43.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:43.185+0000 7f96dbfc1640 1 -- 172.21.15.62:0/3069294452 wait complete. 2024-09-15T23:18:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.888+0000 7f2528a2e640 1 -- 172.21.15.62:0/2476777780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2524101c80 msgr2=0x7f2524102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.888+0000 7f2528a2e640 1 --2- 172.21.15.62:0/2476777780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2524101c80 0x7f2524102100 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2518009900 tx=0x7f251802a480 comp rx=0 tx=0).stop 2024-09-15T23:18:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.889+0000 7f2528a2e640 1 -- 172.21.15.62:0/2476777780 shutdown_connections 2024-09-15T23:18:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.889+0000 7f2528a2e640 1 --2- 172.21.15.62:0/2476777780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2524102640 0x7f2524108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.889+0000 7f2528a2e640 1 --2- 172.21.15.62:0/2476777780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2524101c80 0x7f2524102100 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.889+0000 7f2528a2e640 1 --2- 172.21.15.62:0/2476777780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2524100f70 0x7f2524101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.889+0000 7f2528a2e640 1 -- 172.21.15.62:0/2476777780 >> 172.21.15.62:0/2476777780 conn(0x7f25240fb930 msgr2=0x7f25240fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.889+0000 7f2528a2e640 1 -- 172.21.15.62:0/2476777780 shutdown_connections 2024-09-15T23:18:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.889+0000 7f2528a2e640 1 -- 172.21.15.62:0/2476777780 wait complete. 2024-09-15T23:18:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.889+0000 7f2528a2e640 1 Processor -- start 2024-09-15T23:18:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.889+0000 7f2528a2e640 1 -- start start 2024-09-15T23:18:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2528a2e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2524100f70 0x7f2524197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2528a2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2524101c80 0x7f2524198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2528a2e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2524102640 0x7f252419f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2528a2e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f252410d820 con 0x7f2524101c80 2024-09-15T23:18:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2528a2e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f252410d6a0 con 0x7f2524102640 2024-09-15T23:18:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2528a2e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f252410d9a0 con 0x7f2524100f70 2024-09-15T23:18:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2521d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2524101c80 0x7f2524198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2522d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2524102640 0x7f252419f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2521d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2524101c80 0x7f2524198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42042/0 (socket says 172.21.15.62:42042) 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2521d9b640 1 -- 172.21.15.62:0/4141996011 learned_addr learned my addr 172.21.15.62:0/4141996011 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f252259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2524100f70 0x7f2524197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2522d9d640 1 -- 172.21.15.62:0/4141996011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2524100f70 msgr2=0x7f2524197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2522d9d640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2524100f70 0x7f2524197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2522d9d640 1 -- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2524101c80 msgr2=0x7f2524198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2522d9d640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2524101c80 0x7f2524198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2522d9d640 1 -- 172.21.15.62:0/4141996011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252419fa30 con 0x7f2524102640 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2521d9b640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2524101c80 0x7f2524198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.890+0000 7f2522d9d640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2524102640 0x7f252419f3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f251c00ec50 tx=0x7f251c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.891+0000 7f25137fe640 1 -- 172.21.15.62:0/4141996011 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c003b70 con 0x7f2524102640 2024-09-15T23:18:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.891+0000 7f25137fe640 1 -- 172.21.15.62:0/4141996011 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f251c003d10 con 0x7f2524102640 2024-09-15T23:18:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.891+0000 7f25137fe640 1 -- 172.21.15.62:0/4141996011 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251c010890 con 0x7f2524102640 2024-09-15T23:18:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.891+0000 7f2528a2e640 1 -- 172.21.15.62:0/4141996011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252419fd20 con 0x7f2524102640 2024-09-15T23:18:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.891+0000 7f2528a2e640 1 -- 172.21.15.62:0/4141996011 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25241a01c0 con 0x7f2524102640 2024-09-15T23:18:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.894+0000 7f252b7cb740 1 Processor -- start 2024-09-15T23:18:43.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f25137fe640 1 -- 172.21.15.62:0/4141996011 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f251c003fc0 con 0x7f2524102640 2024-09-15T23:18:43.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f252b7cb740 1 -- start start 2024-09-15T23:18:43.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f58721b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5877780 0x5582f5879c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f252b7cb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5582f57d12f0 con 0x5582f5871db0 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f252b7cb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5582f5779270 con 0x5582f5877780 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f252b7cb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5582f57cd2b0 con 0x5582f58726f0 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f2521d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f58721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f2521d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f58721b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42046/0 (socket says 172.21.15.62:42046) 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f2521d9b640 1 -- 172.21.15.62:0/1686684521 learned_addr learned my addr 172.21.15.62:0/1686684521 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f252259c640 1 --2- 172.21.15.62:0/1686684521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.895+0000 7f2521d9b640 1 --2- 172.21.15.62:0/1686684521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5877780 0x5582f5879c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f252259c640 1 -- 172.21.15.62:0/1686684521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5877780 msgr2=0x5582f5879c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f252259c640 1 --2- 172.21.15.62:0/1686684521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5877780 0x5582f5879c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f252259c640 1 -- 172.21.15.62:0/1686684521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 msgr2=0x5582f58721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f252259c640 1 --2- 172.21.15.62:0/1686684521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f58721b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f252259c640 1 -- 172.21.15.62:0/1686684521 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582f5778340 con 0x5582f58726f0 2024-09-15T23:18:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f252259c640 1 --2- 172.21.15.62:0/1686684521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58771b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2514009bf0 tx=0x7f2514011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c8f2850c304e26d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f2510ff9640 1 -- 172.21.15.62:0/1686684521 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514004030 con 0x5582f58726f0 2024-09-15T23:18:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f2510ff9640 1 -- 172.21.15.62:0/1686684521 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2514005500 con 0x5582f58726f0 2024-09-15T23:18:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f25137fe640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a33f0 0x7f250c0a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f2510ff9640 1 -- 172.21.15.62:0/1686684521 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25140035f0 con 0x5582f58726f0 2024-09-15T23:18:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f2521d9b640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a33f0 0x7f250c0a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.896+0000 7f25137fe640 1 -- 172.21.15.62:0/4141996011 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f251c0d01f0 con 0x7f2524102640 2024-09-15T23:18:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.897+0000 7f252b7cb740 1 -- 172.21.15.62:0/1686684521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 msgr2=0x5582f58771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.897+0000 7f252b7cb740 1 --2- 172.21.15.62:0/1686684521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58771b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2514009bf0 tx=0x7f2514011810 comp rx=0 tx=0).stop 2024-09-15T23:18:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.897+0000 7f2521d9b640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a33f0 0x7f250c0a58b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f25180096c0 tx=0x7f2518041040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.898+0000 7f252b7cb740 1 -- 172.21.15.62:0/1686684521 shutdown_connections 2024-09-15T23:18:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.898+0000 7f252b7cb740 1 --2- 172.21.15.62:0/1686684521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5877780 0x5582f5879c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.898+0000 7f252b7cb740 1 --2- 172.21.15.62:0/1686684521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58771b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.898+0000 7f252b7cb740 1 --2- 172.21.15.62:0/1686684521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f58721b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.898+0000 7f252b7cb740 1 -- 172.21.15.62:0/1686684521 >> 172.21.15.62:0/1686684521 conn(0x5582f5869010 msgr2=0x5582f5869410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.898+0000 7f252b7cb740 1 -- 172.21.15.62:0/1686684521 shutdown_connections 2024-09-15T23:18:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.898+0000 7f252b7cb740 1 -- 172.21.15.62:0/1686684521 wait complete. 2024-09-15T23:18:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.899+0000 7f252b7cb740 1 -- start start 2024-09-15T23:18:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.899+0000 7f252b7cb740 1 Processor -- start 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f587bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f58726f0 0x5582f587c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 0x5582f5883530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252b7cb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5582f587a7f0 con 0x5582f5871db0 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252b7cb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5582f575a290 con 0x5582f58726f0 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f587bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f2521d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f58726f0 0x5582f587c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f587bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42058/0 (socket says 172.21.15.62:42058) 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f2521d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f58726f0 0x5582f587c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54130/0 (socket says 172.21.15.62:54130) 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252259c640 1 -- 172.21.15.62:0/87480344 learned_addr learned my addr 172.21.15.62:0/87480344 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252b7cb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5582f587a970 con 0x5582f5877780 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f2522d9d640 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 0x5582f5883530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f252b7cb740 10 client.0 _ll_register_callbacks cb 0x5582f578b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f2521d9b640 1 -- 172.21.15.62:0/87480344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 msgr2=0x5582f5883530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f2521d9b640 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 0x5582f5883530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f2521d9b640 1 -- 172.21.15.62:0/87480344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 msgr2=0x5582f587bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f2521d9b640 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f587bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.900+0000 7f2521d9b640 1 -- 172.21.15.62:0/87480344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582f54e4210 con 0x5582f58726f0 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.901+0000 7f2521d9b640 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f58726f0 0x5582f587c4f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2518005ec0 tx=0x7f25180315b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2389efe85ba3b8d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.901+0000 7f2509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.901+0000 7f2509ffb640 1 -- 172.21.15.62:0/87480344 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251804e070 con 0x5582f58726f0 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.901+0000 7f2509ffb640 1 -- 172.21.15.62:0/87480344 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f251803b460 con 0x5582f58726f0 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.901+0000 7f252b7cb740 1 -- 172.21.15.62:0/87480344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2514005500 con 0x5582f58726f0 2024-09-15T23:18:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.901+0000 7f2509ffb640 1 -- 172.21.15.62:0/87480344 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f251805a050 con 0x5582f58726f0 2024-09-15T23:18:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.901+0000 7f252b7cb740 10 client.25712 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.901+0000 7f252b7cb740 1 -- 172.21.15.62:0/87480344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5582f55b0620 con 0x5582f58726f0 2024-09-15T23:18:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.901+0000 7f2509ffb640 1 -- 172.21.15.62:0/87480344 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2514005500 con 0x5582f58726f0 2024-09-15T23:18:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.902+0000 7f2509ffb640 1 -- 172.21.15.62:0/87480344 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f251804a6e0 con 0x5582f58726f0 2024-09-15T23:18:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.903+0000 7f252b7cb740 10 client.25712 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.903+0000 7f252b7cb740 10 client.25712 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.903+0000 7f252b7cb740 20 client.25712 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.903+0000 7f252b7cb740 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5582f5887c20 0x5582f588a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.903+0000 7f252b7cb740 4 client.25712 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.903+0000 7f252b7cb740 1 -- 172.21.15.62:0/87480344 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582f57dea20 con 0x5582f5887c20 2024-09-15T23:18:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.903+0000 7f252259c640 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5582f5887c20 0x5582f588a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.903+0000 7f2509ffb640 10 client.25712 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.903+0000 7f252259c640 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5582f5887c20 0x5582f588a050 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12b583484aa6614d server_cookie=ad68927ea336a55 in_seq=0 out_seq=0 2024-09-15T23:18:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f2509ffb640 1 -- 172.21.15.62:0/87480344 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5582f57dea20 con 0x5582f5887c20 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f2509ffb640 10 client.25712 handle_command_reply: tid=0 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 client.25712 shutdown 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 -- 172.21.15.62:0/87480344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f58726f0 msgr2=0x5582f587c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f58726f0 0x5582f587c4f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2518005ec0 tx=0x7f25180315b0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 -- 172.21.15.62:0/87480344 shutdown_connections 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5582f5887c20 0x5582f588a050 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 0x5582f5883530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f2509ffb640 0 client.25712 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f58726f0 0x5582f587c4f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 --2- 172.21.15.62:0/87480344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5871db0 0x5582f587bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 -- 172.21.15.62:0/87480344 >> 172.21.15.62:0/87480344 conn(0x5582f5869010 msgr2=0x5582f586a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 -- 172.21.15.62:0/87480344 shutdown_connections 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.904+0000 7f252b7cb740 1 -- 172.21.15.62:0/87480344 wait complete. 2024-09-15T23:18:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.905+0000 7f252b7cb740 20 client.25712 trim_cache size 0 max 16384 2024-09-15T23:18:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.927+0000 7f252b7cb740 1 Processor -- start 2024-09-15T23:18:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.928+0000 7f252b7cb740 1 -- start start 2024-09-15T23:18:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.928+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5872190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.928+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58834f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.928+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5877780 0x5582f58679a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.929+0000 7f252b7cb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f25241aada0 con 0x5582f5877780 2024-09-15T23:18:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.929+0000 7f252259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5872190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.929+0000 7f252b7cb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25241a8640 con 0x5582f5871db0 2024-09-15T23:18:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.929+0000 7f252b7cb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f25241a6490 con 0x5582f58726f0 2024-09-15T23:18:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.929+0000 7f252259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5872190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54132/0 (socket says 172.21.15.62:54132) 2024-09-15T23:18:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.929+0000 7f252259c640 1 -- 172.21.15.62:0/1434473984 learned_addr learned my addr 172.21.15.62:0/1434473984 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.929+0000 7f2521d9b640 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58834f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.929+0000 7f2522d9d640 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5877780 0x5582f58679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f2521d9b640 1 -- 172.21.15.62:0/1434473984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 msgr2=0x5582f5872190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f2521d9b640 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5872190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f2521d9b640 1 -- 172.21.15.62:0/1434473984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5877780 msgr2=0x5582f58679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f2521d9b640 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5877780 0x5582f58679a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f2521d9b640 1 -- 172.21.15.62:0/1434473984 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582f587af70 con 0x5582f58726f0 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f2522d9d640 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5877780 0x5582f58679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f2521d9b640 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58834f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f251804aca0 tx=0x7f25180317f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d4debb7d825aae6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f250bfff640 1 -- 172.21.15.62:0/1434473984 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251804e040 con 0x5582f58726f0 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f250bfff640 1 -- 172.21.15.62:0/1434473984 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2518057020 con 0x5582f58726f0 2024-09-15T23:18:43.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.930+0000 7f250bfff640 1 -- 172.21.15.62:0/1434473984 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2518035040 con 0x5582f58726f0 2024-09-15T23:18:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.931+0000 7f252b7cb740 1 -- 172.21.15.62:0/1434473984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 msgr2=0x5582f58834f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.931+0000 7f252b7cb740 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58834f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f251804aca0 tx=0x7f25180317f0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.932+0000 7f252b7cb740 1 -- 172.21.15.62:0/1434473984 shutdown_connections 2024-09-15T23:18:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.932+0000 7f252b7cb740 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f5877780 0x5582f58679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.932+0000 7f252b7cb740 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f58726f0 0x5582f58834f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.932+0000 7f252b7cb740 1 --2- 172.21.15.62:0/1434473984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5872190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.932+0000 7f252b7cb740 1 -- 172.21.15.62:0/1434473984 >> 172.21.15.62:0/1434473984 conn(0x5582f5869010 msgr2=0x5582f58693f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.933+0000 7f252b7cb740 1 -- 172.21.15.62:0/1434473984 shutdown_connections 2024-09-15T23:18:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.933+0000 7f252b7cb740 1 -- 172.21.15.62:0/1434473984 wait complete. 2024-09-15T23:18:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.933+0000 7f252b7cb740 1 -- start start 2024-09-15T23:18:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.933+0000 7f252b7cb740 1 Processor -- start 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.933+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5880f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.933+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f58726f0 0x5582f5881470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.933+0000 7f252b7cb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 0x5582f5882480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252b7cb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f25241a6b00 con 0x5582f58726f0 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252b7cb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25241a7f70 con 0x5582f5871db0 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252b7cb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f25241a71d0 con 0x5582f5877780 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252b7cb740 10 client.0 _ll_register_callbacks cb 0x7f2518009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5880f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5880f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54136/0 (socket says 172.21.15.62:54136) 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252259c640 1 -- 172.21.15.62:0/606102422 learned_addr learned my addr 172.21.15.62:0/606102422 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f2522d9d640 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 0x5582f5882480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f2521d9b640 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f58726f0 0x5582f5881470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252259c640 1 -- 172.21.15.62:0/606102422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 msgr2=0x5582f5882480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252259c640 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 0x5582f5882480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252259c640 1 -- 172.21.15.62:0/606102422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f58726f0 msgr2=0x5582f5881470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252259c640 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f58726f0 0x5582f5881470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252259c640 1 -- 172.21.15.62:0/606102422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582f5882a90 con 0x5582f5871db0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252259c640 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5880f30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2514011d10 tx=0x7f2514004720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a2c15173f8a92cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f2509ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f2509ffb640 1 -- 172.21.15.62:0/606102422 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251401e040 con 0x5582f5871db0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f252b7cb740 1 -- 172.21.15.62:0/606102422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2518057020 con 0x5582f5871db0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.934+0000 7f2509ffb640 1 -- 172.21.15.62:0/606102422 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2514026640 con 0x5582f5871db0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f2509ffb640 1 -- 172.21.15.62:0/606102422 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f251403c050 con 0x5582f5871db0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f252b7cb740 10 client.25718 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f252b7cb740 1 -- 172.21.15.62:0/606102422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5582f58f2cc0 con 0x5582f5871db0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f2509ffb640 1 -- 172.21.15.62:0/606102422 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2518057020 con 0x5582f5871db0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f2509ffb640 1 -- 172.21.15.62:0/606102422 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f2514024040 con 0x5582f5871db0 2024-09-15T23:18:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f252b7cb740 10 client.25718 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:43.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f252b7cb740 10 client.25718 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:43.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f252b7cb740 20 client.25718 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:43.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f252b7cb740 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5582f5887c20 0x5582f58f5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:43.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f252b7cb740 4 client.25718 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:18:43.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f252b7cb740 1 -- 172.21.15.62:0/606102422 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582f57d8d70 con 0x5582f5887c20 2024-09-15T23:18:43.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.935+0000 7f2521d9b640 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5582f5887c20 0x5582f58f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:43.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.936+0000 7f2509ffb640 10 client.25718 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:43.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.936+0000 7f2521d9b640 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5582f5887c20 0x5582f58f5540 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a58e2705991c52b server_cookie=18256d8ca1e12c2e in_seq=0 out_seq=0 2024-09-15T23:18:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f2509ffb640 1 -- 172.21.15.62:0/606102422 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582f57d8d70 con 0x5582f5887c20 2024-09-15T23:18:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f2509ffb640 10 client.25718 handle_command_reply: tid=0 2024-09-15T23:18:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f252b7cb740 1 client.25718 shutdown 2024-09-15T23:18:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f252b7cb740 1 -- 172.21.15.62:0/606102422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 msgr2=0x5582f5880f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f252b7cb740 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5880f30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2514011d10 tx=0x7f2514004720 comp rx=0 tx=0).stop 2024-09-15T23:18:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f252b7cb740 1 -- 172.21.15.62:0/606102422 shutdown_connections 2024-09-15T23:18:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f252b7cb740 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5582f5887c20 0x5582f58f5540 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f252b7cb740 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5582f5877780 0x5582f5882480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f252b7cb740 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5582f58726f0 0x5582f5881470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f252b7cb740 1 --2- 172.21.15.62:0/606102422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5582f5871db0 0x5582f5880f30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f252b7cb740 1 -- 172.21.15.62:0/606102422 >> 172.21.15.62:0/606102422 conn(0x5582f5869010 msgr2=0x5582f58681c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.946+0000 7f2509ffb640 0 client.25718 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.947+0000 7f252b7cb740 1 -- 172.21.15.62:0/606102422 shutdown_connections 2024-09-15T23:18:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.947+0000 7f252b7cb740 1 -- 172.21.15.62:0/606102422 wait complete. 2024-09-15T23:18:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.947+0000 7f252b7cb740 20 client.25718 trim_cache size 0 max 16384 2024-09-15T23:18:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:18:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:18:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:18:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 -- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a33f0 msgr2=0x7f250c0a58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a33f0 0x7f250c0a58b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f25180096c0 tx=0x7f2518041040 comp rx=0 tx=0).stop 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 -- 172.21.15.62:0/4141996011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2524102640 msgr2=0x7f252419f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2524102640 0x7f252419f3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f251c00ec50 tx=0x7f251c00c6a0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 -- 172.21.15.62:0/4141996011 shutdown_connections 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f250c0a33f0 0x7f250c0a58b0 secure :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f25180096c0 tx=0x7f2518041040 comp rx=0 tx=0).stop 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2524102640 0x7f252419f3c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2524101c80 0x7f2524198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 --2- 172.21.15.62:0/4141996011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2524100f70 0x7f2524197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.948+0000 7f2528a2e640 1 -- 172.21.15.62:0/4141996011 >> 172.21.15.62:0/4141996011 conn(0x7f25240fb930 msgr2=0x7f2524105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.949+0000 7f2528a2e640 1 -- 172.21.15.62:0/4141996011 shutdown_connections 2024-09-15T23:18:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:43.949+0000 7f2528a2e640 1 -- 172.21.15.62:0/4141996011 wait complete. 2024-09-15T23:18:43.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:43 smithi062 ceph-mon[30949]: pgmap v443: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.5 MiB/s wr, 350 op/s 2024-09-15T23:18:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:43 smithi067 ceph-mon[35665]: pgmap v443: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.5 MiB/s wr, 350 op/s 2024-09-15T23:18:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:43 smithi148 ceph-mon[35610]: pgmap v443: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.5 MiB/s wr, 350 op/s 2024-09-15T23:18:44.087 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5d3ecd09', total seconds elapsed: 23.3 2024-09-15T23:18:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 145.0 s, 761.6 tps, lat 533.683 ms stddev 3035.682 2024-09-15T23:18:45.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:45 smithi062 ceph-mon[30949]: pgmap v444: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.2 MiB/s wr, 421 op/s 2024-09-15T23:18:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:45 smithi067 ceph-mon[35665]: pgmap v444: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.2 MiB/s wr, 421 op/s 2024-09-15T23:18:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:45 smithi148 ceph-mon[35610]: pgmap v444: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 9.2 MiB/s wr, 421 op/s 2024-09-15T23:18:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:47 smithi062 ceph-mon[30949]: pgmap v445: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 MiB/s wr, 474 op/s 2024-09-15T23:18:48.016 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:18:48.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:47 smithi067 ceph-mon[35665]: pgmap v445: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 MiB/s wr, 474 op/s 2024-09-15T23:18:48.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:47 smithi148 ceph-mon[35610]: pgmap v445: 129 pgs: 129 active+clean; 1.0 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 MiB/s wr, 474 op/s 2024-09-15T23:18:48.427 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.390+0000 7f992e242640 1 -- 172.21.15.62:0/1686831108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102ef0 msgr2=0x7f992810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.390+0000 7f992e242640 1 --2- 172.21.15.62:0/1686831108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102ef0 0x7f992810da40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f992000bc30 tx=0x7f9920009260 comp rx=0 tx=0).stop 2024-09-15T23:18:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.390+0000 7f992e242640 1 -- 172.21.15.62:0/1686831108 shutdown_connections 2024-09-15T23:18:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.390+0000 7f992e242640 1 --2- 172.21.15.62:0/1686831108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102ef0 0x7f992810da40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.390+0000 7f992e242640 1 --2- 172.21.15.62:0/1686831108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9928102570 0x7f99281029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.390+0000 7f992e242640 1 --2- 172.21.15.62:0/1686831108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9928100e20 0x7f9928101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.390+0000 7f992e242640 1 -- 172.21.15.62:0/1686831108 >> 172.21.15.62:0/1686831108 conn(0x7f99280fc7d0 msgr2=0x7f99280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:49.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.390+0000 7f992e242640 1 -- 172.21.15.62:0/1686831108 shutdown_connections 2024-09-15T23:18:49.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.391+0000 7f992e242640 1 -- 172.21.15.62:0/1686831108 wait complete. 2024-09-15T23:18:49.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.391+0000 7f992e242640 1 Processor -- start 2024-09-15T23:18:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f992e242640 1 -- start start 2024-09-15T23:18:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f992e242640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9928100e20 0x7f9928196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f992e242640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102570 0x7f99281968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f992e242640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9928102ef0 0x7f992819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f992e242640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f992810fea0 con 0x7f9928102ef0 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f992e242640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f992810fd20 con 0x7f9928100e20 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f992e242640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9928110020 con 0x7f9928102570 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f9926ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102570 0x7f99281968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f9926ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102570 0x7f99281968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47022/0 (socket says 172.21.15.62:47022) 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f9926ffd640 1 -- 172.21.15.62:0/2872762429 learned_addr learned my addr 172.21.15.62:0/2872762429 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f99277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9928100e20 0x7f9928196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f9927fff640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9928102ef0 0x7f992819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f9926ffd640 1 -- 172.21.15.62:0/2872762429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9928100e20 msgr2=0x7f9928196360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f9926ffd640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9928100e20 0x7f9928196360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f9926ffd640 1 -- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9928102ef0 msgr2=0x7f992819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:49.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f9926ffd640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9928102ef0 0x7f992819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.392+0000 7f9926ffd640 1 -- 172.21.15.62:0/2872762429 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992819b310 con 0x7f9928102570 2024-09-15T23:18:49.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.395+0000 7f9926ffd640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102570 0x7f99281968a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f991c0098a0 tx=0x7f991c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.395+0000 7f9924ff9640 1 -- 172.21.15.62:0/2872762429 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c00ea80 con 0x7f9928102570 2024-09-15T23:18:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.395+0000 7f9924ff9640 1 -- 172.21.15.62:0/2872762429 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f991c005500 con 0x7f9928102570 2024-09-15T23:18:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.395+0000 7f992e242640 1 -- 172.21.15.62:0/2872762429 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f992819b600 con 0x7f9928102570 2024-09-15T23:18:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.395+0000 7f9924ff9640 1 -- 172.21.15.62:0/2872762429 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991c004d70 con 0x7f9928102570 2024-09-15T23:18:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.395+0000 7f992e242640 1 -- 172.21.15.62:0/2872762429 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99281a2ee0 con 0x7f9928102570 2024-09-15T23:18:49.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.398+0000 7f9924ff9640 1 -- 172.21.15.62:0/2872762429 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f991c005020 con 0x7f9928102570 2024-09-15T23:18:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.399+0000 7f9924ff9640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99100a3300 0x7f99100a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.399+0000 7f993061b740 1 Processor -- start 2024-09-15T23:18:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.399+0000 7f9924ff9640 1 -- 172.21.15.62:0/2872762429 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f991c0cf7e0 con 0x7f9928102570 2024-09-15T23:18:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.399+0000 7f993061b740 1 -- start start 2024-09-15T23:18:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.399+0000 7f99277fe640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99100a3300 0x7f99100a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f993061b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 0x564e6f921880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f993061b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f921dc0 0x564e6f92c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f993061b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f92ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f993061b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564e6f8872e0 con 0x564e6f92c680 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f993061b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564e6f82ece0 con 0x564e6f921dc0 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f993061b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564e6f8832a0 con 0x564e6f9214a0 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f9927fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f92ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f9927fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f92ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42120/0 (socket says 172.21.15.62:42120) 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f9927fff640 1 -- 172.21.15.62:0/482929982 learned_addr learned my addr 172.21.15.62:0/482929982 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.400+0000 7f99277fe640 1 --2- 172.21.15.62:0/482929982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 0x564e6f921880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.401+0000 7f9927fff640 1 -- 172.21.15.62:0/482929982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 msgr2=0x564e6f921880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.401+0000 7f9927fff640 1 --2- 172.21.15.62:0/482929982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 0x564e6f921880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.401+0000 7f9927fff640 1 -- 172.21.15.62:0/482929982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f921dc0 msgr2=0x564e6f92c140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.401+0000 7f9927fff640 1 --2- 172.21.15.62:0/482929982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f921dc0 0x564e6f92c140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.401+0000 7f9927fff640 1 -- 172.21.15.62:0/482929982 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e6f813440 con 0x564e6f92c680 2024-09-15T23:18:49.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.401+0000 7f99277fe640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99100a3300 0x7f99100a57c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f99180099e0 tx=0x7f9918009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:49.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.401+0000 7f9927fff640 1 --2- 172.21.15.62:0/482929982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f92ea70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f992003c000 tx=0x7f992000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20beb01f0b3f6ad9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:49.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.402+0000 7f990dffb640 1 -- 172.21.15.62:0/482929982 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992003a040 con 0x564e6f92c680 2024-09-15T23:18:49.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.403+0000 7f990dffb640 1 -- 172.21.15.62:0/482929982 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f992004d070 con 0x564e6f92c680 2024-09-15T23:18:49.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.404+0000 7f993061b740 1 -- 172.21.15.62:0/482929982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 msgr2=0x564e6f92ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:49.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.404+0000 7f993061b740 1 --2- 172.21.15.62:0/482929982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f92ea70 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f992003c000 tx=0x7f992000b8a0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.404+0000 7f993061b740 1 -- 172.21.15.62:0/482929982 shutdown_connections 2024-09-15T23:18:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.404+0000 7f993061b740 1 --2- 172.21.15.62:0/482929982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f92ea70 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.404+0000 7f993061b740 1 --2- 172.21.15.62:0/482929982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f921dc0 0x564e6f92c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.404+0000 7f993061b740 1 --2- 172.21.15.62:0/482929982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 0x564e6f921880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.404+0000 7f993061b740 1 -- 172.21.15.62:0/482929982 >> 172.21.15.62:0/482929982 conn(0x564e6f91da30 msgr2=0x564e6f91de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.404+0000 7f993061b740 1 -- 172.21.15.62:0/482929982 shutdown_connections 2024-09-15T23:18:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.404+0000 7f993061b740 1 -- 172.21.15.62:0/482929982 wait complete. 2024-09-15T23:18:49.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.405+0000 7f993061b740 1 -- start start 2024-09-15T23:18:49.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.405+0000 7f993061b740 1 Processor -- start 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.405+0000 7f993061b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 0x564e6f92bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f993061b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f921dc0 0x564e6f929260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f993061b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f9297a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f993061b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564e6f92f2a0 con 0x564e6f92c680 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f9927fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f9297a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f9927fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f9297a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42132/0 (socket says 172.21.15.62:42132) 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f9927fff640 1 -- 172.21.15.62:0/4061205703 learned_addr learned my addr 172.21.15.62:0/4061205703 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f99277fe640 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 0x564e6f92bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f9926ffd640 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f921dc0 0x564e6f929260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564e6f842050 con 0x564e6f921dc0 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564e6f92f420 con 0x564e6f9214a0 2024-09-15T23:18:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.406+0000 7f993061b740 10 client.0 _ll_register_callbacks cb 0x564e6f882160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:49.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.407+0000 7f99277fe640 1 -- 172.21.15.62:0/4061205703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f921dc0 msgr2=0x564e6f929260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:49.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.407+0000 7f99277fe640 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f921dc0 0x564e6f929260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.407+0000 7f99277fe640 1 -- 172.21.15.62:0/4061205703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 msgr2=0x564e6f9297a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:49.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.407+0000 7f99277fe640 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f9297a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.407+0000 7f99277fe640 1 -- 172.21.15.62:0/4061205703 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e6f5f83c0 con 0x564e6f9214a0 2024-09-15T23:18:49.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.416+0000 7f9906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:18:49.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.416+0000 7f99277fe640 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 0x564e6f92bfb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9918018000 tx=0x7f99180047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffe65c6059dfee05 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:49.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.427+0000 7f9906ffd640 1 -- 172.21.15.62:0/4061205703 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918024030 con 0x564e6f9214a0 2024-09-15T23:18:49.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.427+0000 7f9906ffd640 1 -- 172.21.15.62:0/4061205703 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f991800f040 con 0x564e6f9214a0 2024-09-15T23:18:49.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.427+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f992004d070 con 0x564e6f9214a0 2024-09-15T23:18:49.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.428+0000 7f9906ffd640 1 -- 172.21.15.62:0/4061205703 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9918020670 con 0x564e6f9214a0 2024-09-15T23:18:49.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.429+0000 7f993061b740 10 client.25567 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:49.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.429+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564e6f93ad10 con 0x564e6f9214a0 2024-09-15T23:18:49.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.433+0000 7f9906ffd640 1 -- 172.21.15.62:0/4061205703 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f992004d070 con 0x564e6f9214a0 2024-09-15T23:18:49.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.433+0000 7f9906ffd640 1 -- 172.21.15.62:0/4061205703 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f99180138e0 con 0x564e6f9214a0 2024-09-15T23:18:49.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.435+0000 7f993061b740 10 client.25567 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:49.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.435+0000 7f993061b740 10 client.25567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:49.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.435+0000 7f993061b740 20 client.25567 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:49.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.435+0000 7f993061b740 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564e6f93c220 0x564e6f93e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.435+0000 7f993061b740 4 client.25567 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:49.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.435+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e6f894850 con 0x564e6f93c220 2024-09-15T23:18:49.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.436+0000 7f9926ffd640 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564e6f93c220 0x564e6f93e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.437+0000 7f9926ffd640 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564e6f93c220 0x564e6f93e690 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e97ccd8adb454cd1 server_cookie=8fba2734cde1d34b in_seq=0 out_seq=0 2024-09-15T23:18:49.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.439+0000 7f9906ffd640 10 client.25567 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:49.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.439+0000 7f9906ffd640 1 -- 172.21.15.62:0/4061205703 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564e6f894850 con 0x564e6f93c220 2024-09-15T23:18:49.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.439+0000 7f9906ffd640 10 client.25567 handle_command_reply: tid=0 2024-09-15T23:18:49.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f993061b740 1 client.25567 shutdown 2024-09-15T23:18:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 msgr2=0x564e6f92bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f993061b740 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 0x564e6f92bfb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9918018000 tx=0x7f99180047d0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 shutdown_connections 2024-09-15T23:18:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f993061b740 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564e6f93c220 0x564e6f93e690 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f993061b740 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f92c680 0x564e6f9297a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f993061b740 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f921dc0 0x564e6f929260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f993061b740 1 --2- 172.21.15.62:0/4061205703 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f9214a0 0x564e6f92bfb0 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9918018000 tx=0x7f99180047d0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 >> 172.21.15.62:0/4061205703 conn(0x564e6f91da30 msgr2=0x564e6f923710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:49.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.440+0000 7f9906ffd640 0 client.25567 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.442+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 shutdown_connections 2024-09-15T23:18:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.442+0000 7f993061b740 1 -- 172.21.15.62:0/4061205703 wait complete. 2024-09-15T23:18:49.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.442+0000 7f993061b740 20 client.25567 trim_cache size 0 max 16384 2024-09-15T23:18:49.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.478+0000 7f993061b740 1 Processor -- start 2024-09-15T23:18:49.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.478+0000 7f993061b740 1 -- start start 2024-09-15T23:18:49.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f993061b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 0x564e6f938550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f993061b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f91da30 0x564e6f943f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f993061b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f921dc0 0x564e6f9444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f993061b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f99281af400 con 0x564e6f921dc0 2024-09-15T23:18:49.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f993061b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f99281acca0 con 0x564e6f929900 2024-09-15T23:18:49.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f99277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 0x564e6f938550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f993061b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f99281aaab0 con 0x564e6f91da30 2024-09-15T23:18:49.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f99277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 0x564e6f938550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54168/0 (socket says 172.21.15.62:54168) 2024-09-15T23:18:49.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f9927fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f921dc0 0x564e6f9444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f99277fe640 1 -- 172.21.15.62:0/1025506909 learned_addr learned my addr 172.21.15.62:0/1025506909 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:49.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.479+0000 7f9926ffd640 1 --2- 172.21.15.62:0/1025506909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f91da30 0x564e6f943f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.480+0000 7f99277fe640 1 -- 172.21.15.62:0/1025506909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f91da30 msgr2=0x564e6f943f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:49.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.480+0000 7f99277fe640 1 --2- 172.21.15.62:0/1025506909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f91da30 0x564e6f943f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.480+0000 7f99277fe640 1 -- 172.21.15.62:0/1025506909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f921dc0 msgr2=0x564e6f9444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:49.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.480+0000 7f99277fe640 1 --2- 172.21.15.62:0/1025506909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f921dc0 0x564e6f9444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.480+0000 7f99277fe640 1 -- 172.21.15.62:0/1025506909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e6f93ab20 con 0x564e6f929900 2024-09-15T23:18:49.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.482+0000 7f99277fe640 1 --2- 172.21.15.62:0/1025506909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 0x564e6f938550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f991800e5a0 tx=0x7f9918009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e11324105b0055a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:49.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.485+0000 7f990d7fa640 1 -- 172.21.15.62:0/1025506909 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918020670 con 0x564e6f929900 2024-09-15T23:18:49.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.485+0000 7f990d7fa640 1 -- 172.21.15.62:0/1025506909 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9918037070 con 0x564e6f929900 2024-09-15T23:18:49.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.485+0000 7f990d7fa640 1 -- 172.21.15.62:0/1025506909 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991801fea0 con 0x564e6f929900 2024-09-15T23:18:49.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.486+0000 7f993061b740 1 -- 172.21.15.62:0/1025506909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 msgr2=0x564e6f938550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:49.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.486+0000 7f993061b740 1 --2- 172.21.15.62:0/1025506909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 0x564e6f938550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f991800e5a0 tx=0x7f9918009780 comp rx=0 tx=0).stop 2024-09-15T23:18:49.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.486+0000 7f993061b740 1 -- 172.21.15.62:0/1025506909 shutdown_connections 2024-09-15T23:18:49.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.486+0000 7f993061b740 1 --2- 172.21.15.62:0/1025506909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f921dc0 0x564e6f9444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.486+0000 7f993061b740 1 --2- 172.21.15.62:0/1025506909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f91da30 0x564e6f943f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.486+0000 7f993061b740 1 --2- 172.21.15.62:0/1025506909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 0x564e6f938550 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.486+0000 7f993061b740 1 -- 172.21.15.62:0/1025506909 >> 172.21.15.62:0/1025506909 conn(0x564e6f924c50 msgr2=0x564e6f927b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:49.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.487+0000 7f993061b740 1 -- 172.21.15.62:0/1025506909 shutdown_connections 2024-09-15T23:18:49.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.487+0000 7f993061b740 1 -- 172.21.15.62:0/1025506909 wait complete. 2024-09-15T23:18:49.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.487+0000 7f993061b740 1 -- start start 2024-09-15T23:18:49.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.488+0000 7f993061b740 1 Processor -- start 2024-09-15T23:18:49.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.488+0000 7f993061b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f91da30 0x564e6f930bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.489+0000 7f993061b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f921dc0 0x564e6f931110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.489+0000 7f993061b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 0x564e6f9a5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.489+0000 7f993061b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f99281ab160 con 0x564e6f91da30 2024-09-15T23:18:49.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.489+0000 7f99277fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f91da30 0x564e6f930bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.489+0000 7f99277fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f91da30 0x564e6f930bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42144/0 (socket says 172.21.15.62:42144) 2024-09-15T23:18:49.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.489+0000 7f99277fe640 1 -- 172.21.15.62:0/3886307845 learned_addr learned my addr 172.21.15.62:0/3886307845 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:49.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.489+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f99281ac5d0 con 0x564e6f929900 2024-09-15T23:18:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.489+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f99281ab830 con 0x564e6f921dc0 2024-09-15T23:18:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.489+0000 7f993061b740 10 client.0 _ll_register_callbacks cb 0x7f9918006d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.490+0000 7f99277fe640 1 -- 172.21.15.62:0/3886307845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f921dc0 msgr2=0x564e6f931110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:18:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.490+0000 7f99277fe640 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f921dc0 0x564e6f931110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.490+0000 7f99277fe640 1 -- 172.21.15.62:0/3886307845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 msgr2=0x564e6f9a5f30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:18:49.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.490+0000 7f99277fe640 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 0x564e6f9a5f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.490+0000 7f99277fe640 1 -- 172.21.15.62:0/3886307845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e6f9a6520 con 0x564e6f91da30 2024-09-15T23:18:49.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.491+0000 7f99277fe640 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f91da30 0x564e6f930bd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f991801e660 tx=0x7f991800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4777bd486e7a205d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:49.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.491+0000 7f9906ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:49.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.491+0000 7f9906ffd640 1 -- 172.21.15.62:0/3886307845 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918042070 con 0x564e6f91da30 2024-09-15T23:18:49.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.491+0000 7f9906ffd640 1 -- 172.21.15.62:0/3886307845 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9918013e10 con 0x564e6f91da30 2024-09-15T23:18:49.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.491+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9918037070 con 0x564e6f91da30 2024-09-15T23:18:49.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.491+0000 7f9906ffd640 1 -- 172.21.15.62:0/3886307845 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9918013430 con 0x564e6f91da30 2024-09-15T23:18:49.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.491+0000 7f993061b740 10 client.16104 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:49.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.491+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564e6f9a77f0 con 0x564e6f91da30 2024-09-15T23:18:49.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.493+0000 7f9906ffd640 1 -- 172.21.15.62:0/3886307845 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9918037070 con 0x564e6f91da30 2024-09-15T23:18:49.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.494+0000 7f9906ffd640 1 -- 172.21.15.62:0/3886307845 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f99180138e0 con 0x564e6f91da30 2024-09-15T23:18:49.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.495+0000 7f993061b740 10 client.16104 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:49.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.495+0000 7f993061b740 10 client.16104 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.495+0000 7f993061b740 20 client.16104 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.495+0000 7f993061b740 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564e6f92c680 0x564e6f9a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.495+0000 7f993061b740 4 client.16104 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:18:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.495+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564e6f88ed60 con 0x564e6f92c680 2024-09-15T23:18:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.495+0000 7f9926ffd640 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564e6f92c680 0x564e6f9a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.496+0000 7f9926ffd640 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564e6f92c680 0x564e6f9a9e90 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8d22f783b0fd13a server_cookie=8d229084fbf431c2 in_seq=0 out_seq=0 2024-09-15T23:18:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.496+0000 7f9906ffd640 10 client.16104 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:49.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.496+0000 7f9906ffd640 1 -- 172.21.15.62:0/3886307845 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564e6f88ed60 con 0x564e6f92c680 2024-09-15T23:18:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.496+0000 7f9906ffd640 10 client.16104 handle_command_reply: tid=0 2024-09-15T23:18:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.497+0000 7f993061b740 1 client.16104 shutdown 2024-09-15T23:18:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.497+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f91da30 msgr2=0x564e6f930bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.497+0000 7f993061b740 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f91da30 0x564e6f930bd0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f991801e660 tx=0x7f991800d150 comp rx=0 tx=0).stop 2024-09-15T23:18:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.497+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 shutdown_connections 2024-09-15T23:18:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.497+0000 7f993061b740 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564e6f92c680 0x564e6f9a9e90 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.497+0000 7f993061b740 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564e6f929900 0x564e6f9a5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.497+0000 7f993061b740 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564e6f921dc0 0x564e6f931110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.497+0000 7f993061b740 1 --2- 172.21.15.62:0/3886307845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564e6f91da30 0x564e6f930bd0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.497+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 >> 172.21.15.62:0/3886307845 conn(0x564e6f924c50 msgr2=0x564e6f923710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:49.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.498+0000 7f9906ffd640 0 client.16104 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:49.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.498+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 shutdown_connections 2024-09-15T23:18:49.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.498+0000 7f993061b740 1 -- 172.21.15.62:0/3886307845 wait complete. 2024-09-15T23:18:49.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.498+0000 7f993061b740 20 client.16104 trim_cache size 0 max 16384 2024-09-15T23:18:49.505 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:18:49.505 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:18:49.506 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:18:49.506 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 -- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99100a3300 msgr2=0x7f99100a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99100a3300 0x7f99100a57c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f99180099e0 tx=0x7f9918009110 comp rx=0 tx=0).stop 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 -- 172.21.15.62:0/2872762429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102570 msgr2=0x7f99281968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102570 0x7f99281968a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f991c0098a0 tx=0x7f991c009ca0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 -- 172.21.15.62:0/2872762429 shutdown_connections 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f99100a3300 0x7f99100a57c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9928102ef0 0x7f992819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9928102570 0x7f99281968a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 --2- 172.21.15.62:0/2872762429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9928100e20 0x7f9928196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.499+0000 7f992e242640 1 -- 172.21.15.62:0/2872762429 >> 172.21.15.62:0/2872762429 conn(0x7f99280fc7d0 msgr2=0x7f99281048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.500+0000 7f992e242640 1 -- 172.21.15.62:0/2872762429 shutdown_connections 2024-09-15T23:18:49.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:49.500+0000 7f992e242640 1 -- 172.21.15.62:0/2872762429 wait complete. 2024-09-15T23:18:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 150.0 s, 2954.0 tps, lat 33.796 ms stddev 39.333 2024-09-15T23:18:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:49 smithi062 ceph-mon[30949]: pgmap v446: 129 pgs: 129 active+clean; 1.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 492 op/s 2024-09-15T23:18:50.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:49 smithi067 ceph-mon[35665]: pgmap v446: 129 pgs: 129 active+clean; 1.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 492 op/s 2024-09-15T23:18:50.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:49 smithi148 ceph-mon[35610]: pgmap v446: 129 pgs: 129 active+clean; 1.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 492 op/s 2024-09-15T23:18:50.446 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:18:50.447 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:18:50.889 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:51.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.434+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3870418519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2a810e7b0 msgr2=0x7fb2a8115130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:51.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.434+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3870418519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2a810e7b0 0x7fb2a8115130 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009ca0 tx=0x7fb2a002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.435+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3870418519 shutdown_connections 2024-09-15T23:18:51.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.435+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3870418519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2a810e7b0 0x7fb2a8115130 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.435+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3870418519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2a8109c40 0x7fb2a810e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.435+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3870418519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a8109290 0x7fb2a8109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.435+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3870418519 >> 172.21.15.62:0/3870418519 conn(0x7fb2a8077c10 msgr2=0x7fb2a807a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:51.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.435+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3870418519 shutdown_connections 2024-09-15T23:18:51.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.435+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3870418519 wait complete. 2024-09-15T23:18:51.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.436+0000 7fb2adcc8640 1 Processor -- start 2024-09-15T23:18:51.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2adcc8640 1 -- start start 2024-09-15T23:18:51.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2adcc8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2a8109290 0x7fb2a8073920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:51.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2adcc8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a8109c40 0x7fb2a8073e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:51.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2adcc8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2a810e7b0 0x7fb2a806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:51.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2adcc8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a81185c0 con 0x7fb2a8109290 2024-09-15T23:18:51.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2adcc8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8118440 con 0x7fb2a8109c40 2024-09-15T23:18:51.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2adcc8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8118740 con 0x7fb2a810e7b0 2024-09-15T23:18:51.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2a8109290 0x7fb2a8073920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:51.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2a8109290 0x7fb2a8073920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42168/0 (socket says 172.21.15.62:42168) 2024-09-15T23:18:51.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2a77fe640 1 -- 172.21.15.62:0/3448412503 learned_addr learned my addr 172.21.15.62:0/3448412503 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:51.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.437+0000 7fb2a7fff640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2a810e7b0 0x7fb2a806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.438+0000 7fb2a6ffd640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a8109c40 0x7fb2a8073e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.438+0000 7fb2a7fff640 1 -- 172.21.15.62:0/3448412503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a8109c40 msgr2=0x7fb2a8073e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.438+0000 7fb2a7fff640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a8109c40 0x7fb2a8073e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.438+0000 7fb2a7fff640 1 -- 172.21.15.62:0/3448412503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2a8109290 msgr2=0x7fb2a8073920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.438+0000 7fb2a7fff640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2a8109290 0x7fb2a8073920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.438+0000 7fb2a7fff640 1 -- 172.21.15.62:0/3448412503 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a806d450 con 0x7fb2a810e7b0 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.438+0000 7fb2a7fff640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2a810e7b0 0x7fb2a806cb90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009bf0 tx=0x7fb2a0003e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.439+0000 7fb2a4ff9640 1 -- 172.21.15.62:0/3448412503 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0031650 con 0x7fb2a810e7b0 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.439+0000 7fb2a4ff9640 1 -- 172.21.15.62:0/3448412503 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2a002f380 con 0x7fb2a810e7b0 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.439+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a806d6e0 con 0x7fb2a810e7b0 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.439+0000 7fb2a4ff9640 1 -- 172.21.15.62:0/3448412503 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a002f900 con 0x7fb2a810e7b0 2024-09-15T23:18:51.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.439+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a80716d0 con 0x7fb2a810e7b0 2024-09-15T23:18:51.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.440+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a807c310 con 0x7fb2a810e7b0 2024-09-15T23:18:51.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.444+0000 7fb2a4ff9640 1 -- 172.21.15.62:0/3448412503 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb2a00317f0 con 0x7fb2a810e7b0 2024-09-15T23:18:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.444+0000 7fb2a4ff9640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb2900a3300 0x7fb2900a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.445+0000 7fb2a4ff9640 1 -- 172.21.15.62:0/3448412503 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb2a00f0ed0 con 0x7fb2a810e7b0 2024-09-15T23:18:51.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.445+0000 7fb2a77fe640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb2900a3300 0x7fb2900a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:51.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.446+0000 7fb2a77fe640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb2900a3300 0x7fb2900a57c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb298009cd0 tx=0x7fb298005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:51.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.449+0000 7fb2a4ff9640 1 -- 172.21.15.62:0/3448412503 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb2a00bd870 con 0x7fb2a810e7b0 2024-09-15T23:18:51.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.828+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2a8072220 con 0x7fb2a810e7b0 2024-09-15T23:18:51.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.831+0000 7fb2a4ff9640 1 -- 172.21.15.62:0/3448412503 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11907 (secure 0 0 0) 0x7fb2a00bd170 con 0x7fb2a810e7b0 2024-09-15T23:18:51.833 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:18:51.833 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":28,"btime":"2024-09-15T23:17:33:092139+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:17:32.873645+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:18:51.833 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 28 2024-09-15T23:18:51.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb2900a3300 msgr2=0x7fb2900a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:51.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb2900a3300 0x7fb2900a57c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb298009cd0 tx=0x7fb298005f90 comp rx=0 tx=0).stop 2024-09-15T23:18:51.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2a810e7b0 msgr2=0x7fb2a806cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:51.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2a810e7b0 0x7fb2a806cb90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009bf0 tx=0x7fb2a0003e60 comp rx=0 tx=0).stop 2024-09-15T23:18:51.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 shutdown_connections 2024-09-15T23:18:51.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb2900a3300 0x7fb2900a57c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb2a810e7b0 0x7fb2a806cb90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2a8109c40 0x7fb2a8073e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 --2- 172.21.15.62:0/3448412503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb2a8109290 0x7fb2a8073920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:51.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 >> 172.21.15.62:0/3448412503 conn(0x7fb2a8077c10 msgr2=0x7fb2a810c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:51.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 shutdown_connections 2024-09-15T23:18:51.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:51.835+0000 7fb2adcc8640 1 -- 172.21.15.62:0/3448412503 wait complete. 2024-09-15T23:18:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:51 smithi062 ceph-mon[30949]: pgmap v447: 129 pgs: 129 active+clean; 1.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 11 MiB/s wr, 651 op/s 2024-09-15T23:18:52.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:51 smithi067 ceph-mon[35665]: pgmap v447: 129 pgs: 129 active+clean; 1.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 11 MiB/s wr, 651 op/s 2024-09-15T23:18:52.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:51 smithi148 ceph-mon[35610]: pgmap v447: 129 pgs: 129 active+clean; 1.0 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 11 MiB/s wr, 651 op/s 2024-09-15T23:18:52.620 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:18:52.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:52 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3448412503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:53.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:52 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3448412503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:53.035 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:53.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:52 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3448412503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:18:53.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.609+0000 7f7b10f98640 1 -- 172.21.15.62:0/694554942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b0c1049a0 msgr2=0x7f7b0c1089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:53.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.609+0000 7f7b10f98640 1 --2- 172.21.15.62:0/694554942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b0c1049a0 0x7f7b0c1089f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7b00009860 tx=0x7f7b0002a340 comp rx=0 tx=0).stop 2024-09-15T23:18:53.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.609+0000 7f7b10f98640 1 -- 172.21.15.62:0/694554942 shutdown_connections 2024-09-15T23:18:53.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.609+0000 7f7b10f98640 1 --2- 172.21.15.62:0/694554942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b0c109120 0x7f7b0c110ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.609+0000 7f7b10f98640 1 --2- 172.21.15.62:0/694554942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b0c1049a0 0x7f7b0c1089f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.609+0000 7f7b10f98640 1 --2- 172.21.15.62:0/694554942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c103ff0 0x7f7b0c1043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.609+0000 7f7b10f98640 1 -- 172.21.15.62:0/694554942 >> 172.21.15.62:0/694554942 conn(0x7f7b0c0ffab0 msgr2=0x7f7b0c101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:53.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.609+0000 7f7b10f98640 1 -- 172.21.15.62:0/694554942 shutdown_connections 2024-09-15T23:18:53.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.609+0000 7f7b10f98640 1 -- 172.21.15.62:0/694554942 wait complete. 2024-09-15T23:18:53.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.610+0000 7f7b10f98640 1 Processor -- start 2024-09-15T23:18:53.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.610+0000 7f7b10f98640 1 -- start start 2024-09-15T23:18:53.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b10f98640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b0c103ff0 0x7f7b0c19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b10f98640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c1049a0 0x7f7b0c19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b10f98640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b0c109120 0x7f7b0c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b10f98640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c114140 con 0x7f7b0c103ff0 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b10f98640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c113fc0 con 0x7f7b0c1049a0 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b10f98640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0c1142c0 con 0x7f7b0c109120 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b0c103ff0 0x7f7b0c19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b0c109120 0x7f7b0c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b0c103ff0 0x7f7b0c19a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53882/0 (socket says 172.21.15.62:53882) 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0a59c640 1 -- 172.21.15.62:0/2621759659 learned_addr learned my addr 172.21.15.62:0/2621759659 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c1049a0 0x7f7b0c19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0ad9d640 1 -- 172.21.15.62:0/2621759659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c1049a0 msgr2=0x7f7b0c19ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0ad9d640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c1049a0 0x7f7b0c19ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0ad9d640 1 -- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b0c103ff0 msgr2=0x7f7b0c19a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:53.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0ad9d640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b0c103ff0 0x7f7b0c19a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0ad9d640 1 -- 172.21.15.62:0/2621759659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0c19f5f0 con 0x7f7b0c109120 2024-09-15T23:18:53.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c1049a0 0x7f7b0c19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:53.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.611+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b0c103ff0 0x7f7b0c19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:53.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.612+0000 7f7b0ad9d640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b0c109120 0x7f7b0c19ef10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400efc0 tx=0x7f7b0400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:53.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.612+0000 7f7afb7fe640 1 -- 172.21.15.62:0/2621759659 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b04003970 con 0x7f7b0c109120 2024-09-15T23:18:53.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.612+0000 7f7b10f98640 1 -- 172.21.15.62:0/2621759659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0c19f8e0 con 0x7f7b0c109120 2024-09-15T23:18:53.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.612+0000 7f7b10f98640 1 -- 172.21.15.62:0/2621759659 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0c1a71c0 con 0x7f7b0c109120 2024-09-15T23:18:53.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.612+0000 7f7afb7fe640 1 -- 172.21.15.62:0/2621759659 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b04004020 con 0x7f7b0c109120 2024-09-15T23:18:53.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.612+0000 7f7afb7fe640 1 -- 172.21.15.62:0/2621759659 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0400b9f0 con 0x7f7b0c109120 2024-09-15T23:18:53.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.616+0000 7f7b13371740 1 Processor -- start 2024-09-15T23:18:53.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.617+0000 7f7afb7fe640 1 -- 172.21.15.62:0/2621759659 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7b0401d020 con 0x7f7b0c109120 2024-09-15T23:18:53.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.618+0000 7f7b13371740 1 -- start start 2024-09-15T23:18:53.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.618+0000 7f7b13371740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a315b40 0x559e3a314090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.618+0000 7f7b13371740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a314a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.618+0000 7f7b13371740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a321760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.618+0000 7f7b13371740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559e3a27a2f0 con 0x559e3a31f370 2024-09-15T23:18:53.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.618+0000 7f7b13371740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559e3a221cf0 con 0x559e3a3145d0 2024-09-15T23:18:53.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.618+0000 7f7b13371740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559e3a2762b0 con 0x559e3a315b40 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b09d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a315b40 0x559e3a314090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a314a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a314a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39224/0 (socket says 172.21.15.62:39224) 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b0a59c640 1 -- 172.21.15.62:0/912732402 learned_addr learned my addr 172.21.15.62:0/912732402 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/912732402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a321760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b0a59c640 1 -- 172.21.15.62:0/912732402 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a315b40 msgr2=0x559e3a314090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/912732402 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a315b40 0x559e3a314090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b0a59c640 1 -- 172.21.15.62:0/912732402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 msgr2=0x559e3a321760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/912732402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a321760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b0a59c640 1 -- 172.21.15.62:0/912732402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e3a2063a0 con 0x559e3a3145d0 2024-09-15T23:18:53.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7afb7fe640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7af40a32c0 0x7f7af40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.619+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/912732402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a321760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:53.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.620+0000 7f7afb7fe640 1 -- 172.21.15.62:0/2621759659 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7b040cf8d0 con 0x7f7b0c109120 2024-09-15T23:18:53.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.620+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7af40a32c0 0x7f7af40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.620+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/912732402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a314a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7afc011630 tx=0x7f7afc011af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a84b74a2c281984a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:53.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.620+0000 7f7af8ff9640 1 -- 172.21.15.62:0/912732402 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc01e070 con 0x559e3a3145d0 2024-09-15T23:18:53.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.620+0000 7f7af8ff9640 1 -- 172.21.15.62:0/912732402 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7afc00fbd0 con 0x559e3a3145d0 2024-09-15T23:18:53.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.620+0000 7f7af8ff9640 1 -- 172.21.15.62:0/912732402 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc019d00 con 0x559e3a3145d0 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b13371740 1 -- 172.21.15.62:0/912732402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 msgr2=0x559e3a314a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b13371740 1 --2- 172.21.15.62:0/912732402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a314a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7afc011630 tx=0x7f7afc011af0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7af40a32c0 0x7f7af40a5780 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b0002a930 tx=0x7f7b0002a340 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b13371740 1 -- 172.21.15.62:0/912732402 shutdown_connections 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b13371740 1 --2- 172.21.15.62:0/912732402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a321760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b13371740 1 --2- 172.21.15.62:0/912732402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a314a10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b13371740 1 --2- 172.21.15.62:0/912732402 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a315b40 0x559e3a314090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b13371740 1 -- 172.21.15.62:0/912732402 >> 172.21.15.62:0/912732402 conn(0x559e3a2dd310 msgr2=0x559e3a310c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b13371740 1 -- 172.21.15.62:0/912732402 shutdown_connections 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.621+0000 7f7b13371740 1 -- 172.21.15.62:0/912732402 wait complete. 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.622+0000 7f7b13371740 1 -- start start 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.622+0000 7f7b13371740 1 Processor -- start 2024-09-15T23:18:53.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.622+0000 7f7b13371740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a3145d0 0x559e3a31ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.622+0000 7f7b13371740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 0x559e3a31bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.622+0000 7f7b13371740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 0x559e3a31c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.622+0000 7f7b13371740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559e3a322070 con 0x559e3a3145d0 2024-09-15T23:18:53.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.622+0000 7f7b13371740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559e3a235060 con 0x559e3a315b40 2024-09-15T23:18:53.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.622+0000 7f7b13371740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559e3a3221f0 con 0x559e3a31f370 2024-09-15T23:18:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.622+0000 7f7b13371740 10 client.0 _ll_register_callbacks cb 0x559e39f888c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b0ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 0x559e3a31c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b0ad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 0x559e3a31c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52356/0 (socket says 172.21.15.62:52356) 2024-09-15T23:18:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b0ad9d640 1 -- 172.21.15.62:0/338066878 learned_addr learned my addr 172.21.15.62:0/338066878 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 0x559e3a31bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a3145d0 0x559e3a31ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b09d9b640 1 -- 172.21.15.62:0/338066878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 msgr2=0x559e3a31c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:53.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 0x559e3a31c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b09d9b640 1 -- 172.21.15.62:0/338066878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a3145d0 msgr2=0x559e3a31ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a3145d0 0x559e3a31ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.623+0000 7f7b09d9b640 1 -- 172.21.15.62:0/338066878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e39feb3c0 con 0x559e3a315b40 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.624+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a3145d0 0x559e3a31ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.624+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 0x559e3a31bfa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7b00005940 tx=0x7f7b00005d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3a67e67c0d2d998 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.624+0000 7f7af1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.625+0000 7f7af1ffb640 1 -- 172.21.15.62:0/338066878 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b00037040 con 0x559e3a315b40 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.625+0000 7f7af1ffb640 1 -- 172.21.15.62:0/338066878 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7b00003c40 con 0x559e3a315b40 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.625+0000 7f7b13371740 1 -- 172.21.15.62:0/338066878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7afc00fbd0 con 0x559e3a315b40 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.625+0000 7f7af1ffb640 1 -- 172.21.15.62:0/338066878 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7b0005a050 con 0x559e3a315b40 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.625+0000 7f7b13371740 10 client.25745 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.625+0000 7f7b13371740 1 -- 172.21.15.62:0/338066878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559e3a336280 con 0x559e3a315b40 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.626+0000 7f7af1ffb640 1 -- 172.21.15.62:0/338066878 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7afc00fbd0 con 0x559e3a315b40 2024-09-15T23:18:53.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.626+0000 7f7af1ffb640 1 -- 172.21.15.62:0/338066878 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f7b0004a070 con 0x559e3a315b40 2024-09-15T23:18:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.626+0000 7f7b13371740 10 client.25745 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.626+0000 7f7b13371740 10 client.25745 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:18:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.626+0000 7f7b13371740 20 client.25745 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.626+0000 7f7b13371740 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x559e3a3377e0 0x559e3a339bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.626+0000 7f7b13371740 4 client.25745 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.626+0000 7f7b13371740 1 -- 172.21.15.62:0/338066878 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e3a287860 con 0x559e3a3377e0 2024-09-15T23:18:53.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.626+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x559e3a3377e0 0x559e3a339bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.627+0000 7f7af1ffb640 10 client.25745 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:18:53.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.627+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x559e3a3377e0 0x559e3a339bf0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c8555c2996e02b0 server_cookie=4b2e42e9e79e695b in_seq=0 out_seq=0 2024-09-15T23:18:53.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.628+0000 7f7af1ffb640 1 -- 172.21.15.62:0/338066878 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559e3a287860 con 0x559e3a3377e0 2024-09-15T23:18:53.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.628+0000 7f7af1ffb640 10 client.25745 handle_command_reply: tid=0 2024-09-15T23:18:53.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.628+0000 7f7b13371740 1 client.25745 shutdown 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.629+0000 7f7b13371740 1 -- 172.21.15.62:0/338066878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 msgr2=0x559e3a31bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.629+0000 7f7b13371740 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 0x559e3a31bfa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7b00005940 tx=0x7f7b00005d20 comp rx=0 tx=0).stop 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.630+0000 7f7b13371740 1 -- 172.21.15.62:0/338066878 shutdown_connections 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.630+0000 7f7b13371740 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x559e3a3377e0 0x559e3a339bf0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.630+0000 7f7b13371740 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 0x559e3a31c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.630+0000 7f7b13371740 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 0x559e3a31bfa0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.630+0000 7f7b13371740 1 --2- 172.21.15.62:0/338066878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a3145d0 0x559e3a31ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.630+0000 7f7b13371740 1 -- 172.21.15.62:0/338066878 >> 172.21.15.62:0/338066878 conn(0x559e3a2dd310 msgr2=0x559e3a316930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.630+0000 7f7af1ffb640 0 client.25745 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.631+0000 7f7b13371740 1 -- 172.21.15.62:0/338066878 shutdown_connections 2024-09-15T23:18:53.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.631+0000 7f7b13371740 1 -- 172.21.15.62:0/338066878 wait complete. 2024-09-15T23:18:53.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.631+0000 7f7b13371740 20 client.25745 trim_cache size 0 max 16384 2024-09-15T23:18:53.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.652+0000 7f7b13371740 1 Processor -- start 2024-09-15T23:18:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.652+0000 7f7b13371740 1 -- start start 2024-09-15T23:18:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b13371740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a3145d0 0x559e3a31df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b13371740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 0x559e3a317120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b13371740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a317660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b13371740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec0124e0 con 0x559e3a31f370 2024-09-15T23:18:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b13371740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec00fd80 con 0x559e3a315b40 2024-09-15T23:18:53.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b13371740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec00dbd0 con 0x559e3a3145d0 2024-09-15T23:18:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b0ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a317660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b0ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a317660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53918/0 (socket says 172.21.15.62:53918) 2024-09-15T23:18:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b0ad9d640 1 -- 172.21.15.62:0/1639492852 learned_addr learned my addr 172.21.15.62:0/1639492852 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b0ad9d640 1 -- 172.21.15.62:0/1639492852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a3145d0 msgr2=0x559e3a31df10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:18:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b0ad9d640 1 --2- 172.21.15.62:0/1639492852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a3145d0 0x559e3a31df10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b0ad9d640 1 -- 172.21.15.62:0/1639492852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 msgr2=0x559e3a317120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:18:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b0ad9d640 1 --2- 172.21.15.62:0/1639492852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 0x559e3a317120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.653+0000 7f7b0ad9d640 1 -- 172.21.15.62:0/1639492852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e3a31f020 con 0x559e3a31f370 2024-09-15T23:18:53.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.654+0000 7f7b0ad9d640 1 --2- 172.21.15.62:0/1639492852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a317660 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7b040a42b0 tx=0x7f7b0409b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4bbcd78d8af672 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:53.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.654+0000 7f7af3fff640 1 -- 172.21.15.62:0/1639492852 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0409ff20 con 0x559e3a31f370 2024-09-15T23:18:53.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.654+0000 7f7af3fff640 1 -- 172.21.15.62:0/1639492852 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b040a47e0 con 0x559e3a31f370 2024-09-15T23:18:53.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.654+0000 7f7af3fff640 1 -- 172.21.15.62:0/1639492852 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b040a45b0 con 0x559e3a31f370 2024-09-15T23:18:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.655+0000 7f7b13371740 1 -- 172.21.15.62:0/1639492852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 msgr2=0x559e3a317660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.655+0000 7f7b13371740 1 --2- 172.21.15.62:0/1639492852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a317660 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7b040a42b0 tx=0x7f7b0409b5e0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.656+0000 7f7b13371740 1 -- 172.21.15.62:0/1639492852 shutdown_connections 2024-09-15T23:18:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.656+0000 7f7b13371740 1 --2- 172.21.15.62:0/1639492852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a31f370 0x559e3a317660 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.656+0000 7f7b13371740 1 --2- 172.21.15.62:0/1639492852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a315b40 0x559e3a317120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.656+0000 7f7b13371740 1 --2- 172.21.15.62:0/1639492852 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a3145d0 0x559e3a31df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.656+0000 7f7b13371740 1 -- 172.21.15.62:0/1639492852 >> 172.21.15.62:0/1639492852 conn(0x559e3a2dd310 msgr2=0x559e3a31d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.656+0000 7f7b13371740 1 -- 172.21.15.62:0/1639492852 shutdown_connections 2024-09-15T23:18:53.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.656+0000 7f7b13371740 1 -- 172.21.15.62:0/1639492852 wait complete. 2024-09-15T23:18:53.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.656+0000 7f7b13371740 1 -- start start 2024-09-15T23:18:53.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.657+0000 7f7b13371740 1 Processor -- start 2024-09-15T23:18:53.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.657+0000 7f7b13371740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a3278a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.657+0000 7f7b13371740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a315b40 0x559e3a327de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.657+0000 7f7b0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a3278a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b0a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a3278a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39236/0 (socket says 172.21.15.62:39236) 2024-09-15T23:18:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b0a59c640 1 -- 172.21.15.62:0/200659626 learned_addr learned my addr 172.21.15.62:0/200659626 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b13371740 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 0x559e3a318ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec00e240 con 0x559e3a315b40 2024-09-15T23:18:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec00f6b0 con 0x559e3a3145d0 2024-09-15T23:18:53.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec00e910 con 0x559e3a31f370 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b0ad9d640 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 0x559e3a318ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b13371740 10 client.0 _ll_register_callbacks cb 0x7f7afc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b0a59c640 1 -- 172.21.15.62:0/200659626 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 msgr2=0x559e3a318ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 0x559e3a318ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b0a59c640 1 -- 172.21.15.62:0/200659626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a315b40 msgr2=0x559e3a327de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a315b40 0x559e3a327de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b0a59c640 1 -- 172.21.15.62:0/200659626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e3a3193a0 con 0x559e3a3145d0 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b0a59c640 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a3278a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7afc000c00 tx=0x7f7afc00ffd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=818808d30dd8d667 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7af1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.658+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b040a47e0 con 0x559e3a3145d0 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7af1ffb640 1 -- 172.21.15.62:0/200659626 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7afc01e040 con 0x559e3a3145d0 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7af1ffb640 1 -- 172.21.15.62:0/200659626 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7afc0195f0 con 0x559e3a3145d0 2024-09-15T23:18:53.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7af1ffb640 1 -- 172.21.15.62:0/200659626 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7afc0198b0 con 0x559e3a3145d0 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7b13371740 10 client.25748 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=28}) -- 0x559e3a3a2100 con 0x559e3a3145d0 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7af1ffb640 1 -- 172.21.15.62:0/200659626 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f7afc019a50 con 0x559e3a3145d0 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7b13371740 10 client.25748 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7b13371740 10 client.25748 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7b13371740 20 client.25748 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7b13371740 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x559e3a3a2540 0x559e3a3a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7b13371740 4 client.25748 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.659+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f7b040a47e0 con 0x559e3a3a2540 2024-09-15T23:18:53.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.660+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x559e3a3a2540 0x559e3a3a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.660+0000 7f7b09d9b640 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x559e3a3a2540 0x559e3a3a49c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=baeda74b2550cb42 server_cookie=e68e76dc8c04efde in_seq=0 out_seq=0 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.660+0000 7f7af1ffb640 10 client.25748 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.660+0000 7f7af1ffb640 1 -- 172.21.15.62:0/200659626 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7b040a47e0 con 0x559e3a3a2540 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.660+0000 7f7af1ffb640 10 client.25748 handle_command_reply: tid=0 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.660+0000 7f7b13371740 1 client.25748 shutdown 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.660+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 msgr2=0x559e3a3278a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.660+0000 7f7b13371740 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a3278a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7afc000c00 tx=0x7f7afc00ffd0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 shutdown_connections 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7b13371740 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x559e3a3a2540 0x559e3a3a49c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7b13371740 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e3a31f370 0x559e3a318ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7b13371740 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e3a315b40 0x559e3a327de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7b13371740 1 --2- 172.21.15.62:0/200659626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e3a3145d0 0x559e3a3278a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 >> 172.21.15.62:0/200659626 conn(0x559e3a2dd310 msgr2=0x559e3a31e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7af1ffb640 0 client.25748 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:18:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 shutdown_connections 2024-09-15T23:18:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7b13371740 1 -- 172.21.15.62:0/200659626 wait complete. 2024-09-15T23:18:53.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.661+0000 7f7b13371740 20 client.25748 trim_cache size 0 max 16384 2024-09-15T23:18:53.665 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 -- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7af40a32c0 msgr2=0x7f7af40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7af40a32c0 0x7f7af40a5780 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b0002a930 tx=0x7f7b0002a340 comp rx=0 tx=0).stop 2024-09-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 -- 172.21.15.62:0/2621759659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b0c109120 msgr2=0x7f7b0c19ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b0c109120 0x7f7b0c19ef10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400efc0 tx=0x7f7b0400c800 comp rx=0 tx=0).stop 2024-09-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 -- 172.21.15.62:0/2621759659 shutdown_connections 2024-09-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7af40a32c0 0x7f7af40a5780 secure :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b0002a930 tx=0x7f7b0002a340 comp rx=0 tx=0).stop 2024-09-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b0c109120 0x7f7b0c19ef10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c1049a0 0x7f7b0c19ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 --2- 172.21.15.62:0/2621759659 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b0c103ff0 0x7f7b0c19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.662+0000 7f7b10f98640 1 -- 172.21.15.62:0/2621759659 >> 172.21.15.62:0/2621759659 conn(0x7f7b0c0ffab0 msgr2=0x7f7b0c100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.665+0000 7f7b10f98640 1 -- 172.21.15.62:0/2621759659 shutdown_connections 2024-09-15T23:18:53.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:53.665+0000 7f7b10f98640 1 -- 172.21.15.62:0/2621759659 wait complete. 2024-09-15T23:18:53.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:53 smithi062 ceph-mon[30949]: pgmap v448: 129 pgs: 129 active+clean; 1.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 644 op/s 2024-09-15T23:18:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:53 smithi067 ceph-mon[35665]: pgmap v448: 129 pgs: 129 active+clean; 1.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 644 op/s 2024-09-15T23:18:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:53 smithi148 ceph-mon[35610]: pgmap v448: 129 pgs: 129 active+clean; 1.0 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 644 op/s 2024-09-15T23:18:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 155.0 s, 2082.2 tps, lat 47.245 ms stddev 52.434 2024-09-15T23:18:55.241 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:18:55.655 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:55.930 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:55 smithi062 ceph-mon[30949]: pgmap v449: 129 pgs: 129 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.9 MiB/s wr, 585 op/s 2024-09-15T23:18:56.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:55 smithi067 ceph-mon[35665]: pgmap v449: 129 pgs: 129 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.9 MiB/s wr, 585 op/s 2024-09-15T23:18:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:55 smithi148 ceph-mon[35610]: pgmap v449: 129 pgs: 129 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.9 MiB/s wr, 585 op/s 2024-09-15T23:18:56.842 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:18:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.234+0000 7f3c1bfff640 1 -- 172.21.15.62:0/3394431175 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c200319b0 con 0x7f3c24102570 2024-09-15T23:18:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.235+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/3394431175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c24102570 msgr2=0x7f3c241029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:56.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.235+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/3394431175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c24102570 0x7f3c241029b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3c20009900 tx=0x7f3c2002a4d0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.235+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/3394431175 shutdown_connections 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.235+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/3394431175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c24102ef0 0x7f3c24069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.235+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/3394431175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c24102570 0x7f3c241029b0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.235+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/3394431175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24100e20 0x7f3c24101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.235+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/3394431175 >> 172.21.15.62:0/3394431175 conn(0x7f3c240fc7d0 msgr2=0x7f3c240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.235+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/3394431175 shutdown_connections 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.235+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/3394431175 wait complete. 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.236+0000 7f3c2b9e9640 1 Processor -- start 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c2b9e9640 1 -- start start 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c2b9e9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c24100e20 0x7f3c2410e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c2b9e9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c24102570 0x7f3c2410e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c2b9e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24102ef0 0x7f3c241130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c2b9e9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3c241143f0 con 0x7f3c24102570 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c2b9e9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3c24114270 con 0x7f3c24102ef0 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c2b9e9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3c24114570 con 0x7f3c24100e20 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c29785640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c24100e20 0x7f3c2410e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c29785640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c24100e20 0x7f3c2410e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52366/0 (socket says 172.21.15.62:52366) 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c29785640 1 -- 172.21.15.62:0/605962943 learned_addr learned my addr 172.21.15.62:0/605962943 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c29f86640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24102ef0 0x7f3c241130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.237+0000 7f3c28f84640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c24102570 0x7f3c2410e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.238+0000 7f3c29785640 1 -- 172.21.15.62:0/605962943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24102ef0 msgr2=0x7f3c241130f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.238+0000 7f3c29785640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24102ef0 0x7f3c241130f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.238+0000 7f3c29785640 1 -- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c24102570 msgr2=0x7f3c2410e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.238+0000 7f3c29785640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c24102570 0x7f3c2410e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.238+0000 7f3c29785640 1 -- 172.21.15.62:0/605962943 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c24113890 con 0x7f3c24100e20 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.238+0000 7f3c29785640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c24100e20 0x7f3c2410e2a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00ba50 tx=0x7f3c1c00bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.239+0000 7f3c1a7fc640 1 -- 172.21.15.62:0/605962943 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c019070 con 0x7f3c24100e20 2024-09-15T23:18:56.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.239+0000 7f3c1a7fc640 1 -- 172.21.15.62:0/605962943 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c1c00cc10 con 0x7f3c24100e20 2024-09-15T23:18:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.239+0000 7f3c1a7fc640 1 -- 172.21.15.62:0/605962943 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c0076a0 con 0x7f3c24100e20 2024-09-15T23:18:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.239+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/605962943 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c24113b80 con 0x7f3c24100e20 2024-09-15T23:18:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.240+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/605962943 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c241a0810 con 0x7f3c24100e20 2024-09-15T23:18:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.243+0000 7f3c2ddc2740 1 Processor -- start 2024-09-15T23:18:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.245+0000 7f3c1a7fc640 1 -- 172.21.15.62:0/605962943 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3c1c01c070 con 0x7f3c24100e20 2024-09-15T23:18:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.246+0000 7f3c1a7fc640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3c100a3450 0x7f3c100a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.246+0000 7f3c1a7fc640 1 -- 172.21.15.62:0/605962943 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3c1c0cf1b0 con 0x7f3c24100e20 2024-09-15T23:18:56.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.247+0000 7f3c2ddc2740 1 -- start start 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.247+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd8228a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.247+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 0x55dd8228aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.247+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.247+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55dd821f02f0 con 0x55dd8228a5d0 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.247+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55dd82197cf0 con 0x55dd82295370 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.247+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55dd821ec2b0 con 0x55dd8228bb40 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.249+0000 7f3c29f86640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3c100a3450 0x7f3c100a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.249+0000 7f3c29f86640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.250+0000 7f3c29785640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd8228a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.250+0000 7f3c29785640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd8228a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52376/0 (socket says 172.21.15.62:52376) 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.250+0000 7f3c29785640 1 -- 172.21.15.62:0/3457953985 learned_addr learned my addr 172.21.15.62:0/3457953985 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.250+0000 7f3c28f84640 1 --2- 172.21.15.62:0/3457953985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 0x55dd8228aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.250+0000 7f3c29f86640 1 -- 172.21.15.62:0/3457953985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 msgr2=0x55dd8228a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.250+0000 7f3c29f86640 1 --2- 172.21.15.62:0/3457953985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd8228a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.251+0000 7f3c29f86640 1 -- 172.21.15.62:0/3457953985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 msgr2=0x55dd8228aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.251+0000 7f3c29f86640 1 --2- 172.21.15.62:0/3457953985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 0x55dd8228aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.251+0000 7f3c29f86640 1 -- 172.21.15.62:0/3457953985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd8217c3a0 con 0x55dd82295370 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.251+0000 7f3c29f86640 1 --2- 172.21.15.62:0/3457953985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3c14015750 tx=0x7f3c14015c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fe2c165a08536f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.251+0000 7f3c29f86640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3c100a3450 0x7f3c100a5910 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3c14017750 tx=0x7f3c1400f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.253+0000 7f3c0f7fe640 1 -- 172.21.15.62:0/3457953985 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14027070 con 0x55dd82295370 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.253+0000 7f3c0f7fe640 1 -- 172.21.15.62:0/3457953985 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c14022430 con 0x55dd82295370 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.254+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3457953985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 msgr2=0x55dd82297760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.254+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/3457953985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3c14015750 tx=0x7f3c14015c10 comp rx=0 tx=0).stop 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.256+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3457953985 shutdown_connections 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.256+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/3457953985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.256+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/3457953985 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 0x55dd8228aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.256+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/3457953985 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd8228a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.256+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3457953985 >> 172.21.15.62:0/3457953985 conn(0x55dd82253310 msgr2=0x55dd82286c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.256+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3457953985 shutdown_connections 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.257+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3457953985 wait complete. 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.258+0000 7f3c2ddc2740 1 -- start start 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.258+0000 7f3c2ddc2740 1 Processor -- start 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.259+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd8228a5d0 0x55dd82294e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.259+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd82292130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.259+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd82295370 0x55dd822928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.259+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55dd82298060 con 0x55dd82295370 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.259+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55dd821ab060 con 0x55dd8228a5d0 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.259+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55dd822981e0 con 0x55dd8228bb40 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.259+0000 7f3c2ddc2740 10 client.0 _ll_register_callbacks cb 0x55dd81efe8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.260+0000 7f3c29785640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd8228a5d0 0x55dd82294e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.260+0000 7f3c29f86640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd82295370 0x55dd822928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.260+0000 7f3c29f86640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd82295370 0x55dd822928c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53978/0 (socket says 172.21.15.62:53978) 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.260+0000 7f3c29f86640 1 -- 172.21.15.62:0/988515618 learned_addr learned my addr 172.21.15.62:0/988515618 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.260+0000 7f3c28f84640 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd82292130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.261+0000 7f3c29f86640 1 -- 172.21.15.62:0/988515618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 msgr2=0x55dd82292130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.261+0000 7f3c29f86640 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd82292130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.261+0000 7f3c29f86640 1 -- 172.21.15.62:0/988515618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd8228a5d0 msgr2=0x55dd82294e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.261+0000 7f3c29f86640 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd8228a5d0 0x55dd82294e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.261+0000 7f3c29f86640 1 -- 172.21.15.62:0/988515618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd81f613c0 con 0x55dd82295370 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.261+0000 7f3c28f84640 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd82292130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.264+0000 7f3c29785640 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd8228a5d0 0x55dd82294e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.264+0000 7f3c29f86640 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd82295370 0x55dd822928c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3c140094e0 tx=0x7f3c14021aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d1d2e24d93529b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.264+0000 7f3c0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:18:56.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.265+0000 7f3c0cff9640 1 -- 172.21.15.62:0/988515618 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14032060 con 0x55dd82295370 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.265+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/988515618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dd821fd860 con 0x55dd82295370 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.265+0000 7f3c0cff9640 1 -- 172.21.15.62:0/988515618 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3c14015300 con 0x55dd82295370 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.265+0000 7f3c0cff9640 1 -- 172.21.15.62:0/988515618 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3c14027040 con 0x55dd82295370 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.266+0000 7f3c2ddc2740 10 client.16131 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.266+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/988515618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dd822a3d50 con 0x55dd82295370 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.266+0000 7f3c0cff9640 1 -- 172.21.15.62:0/988515618 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dd821fd860 con 0x55dd82295370 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.267+0000 7f3c0cff9640 1 -- 172.21.15.62:0/988515618 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f3c1402a030 con 0x55dd82295370 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.268+0000 7f3c2ddc2740 10 client.16131 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.268+0000 7f3c2ddc2740 10 client.16131 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.268+0000 7f3c2ddc2740 20 client.16131 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.268+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55dd822a5530 0x55dd822a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.268+0000 7f3c2ddc2740 4 client.16131 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.268+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/988515618 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd821f7d70 con 0x55dd822a5530 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.269+0000 7f3c29785640 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55dd822a5530 0x55dd822a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.272+0000 7f3c29785640 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55dd822a5530 0x55dd822a79d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=17065f70eaaec2ef server_cookie=c44543ccadba986d in_seq=0 out_seq=0 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.272+0000 7f3c0cff9640 10 client.16131 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.273+0000 7f3c0cff9640 1 -- 172.21.15.62:0/988515618 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dd821f7d70 con 0x55dd822a5530 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.273+0000 7f3c0cff9640 10 client.16131 handle_command_reply: tid=0 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.273+0000 7f3c2ddc2740 1 client.16131 shutdown 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.273+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/988515618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd82295370 msgr2=0x55dd822928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.273+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd82295370 0x55dd822928c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f3c140094e0 tx=0x7f3c14021aa0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.274+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/988515618 shutdown_connections 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.274+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55dd822a5530 0x55dd822a79d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.274+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd82295370 0x55dd822928c0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.274+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd82292130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.274+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/988515618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd8228a5d0 0x55dd82294e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.274+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/988515618 >> 172.21.15.62:0/988515618 conn(0x55dd82253310 msgr2=0x55dd8228c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.275+0000 7f3c0cff9640 0 client.16131 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:18:56.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.275+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/988515618 shutdown_connections 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.275+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/988515618 wait complete. 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.275+0000 7f3c2ddc2740 20 client.16131 trim_cache size 0 max 16384 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c2ddc2740 1 Processor -- start 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c2ddc2740 1 -- start start 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 0x55dd822a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd822940a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd8228d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3c08012570 con 0x55dd8228a5d0 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0800fe10 con 0x55dd82295370 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0800dc60 con 0x55dd8228bb40 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c29785640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 0x55dd822a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c29785640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 0x55dd822a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53992/0 (socket says 172.21.15.62:53992) 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c29785640 1 -- 172.21.15.62:0/3007313370 learned_addr learned my addr 172.21.15.62:0/3007313370 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:56.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.300+0000 7f3c28f84640 1 --2- 172.21.15.62:0/3007313370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd822940a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c29f86640 1 --2- 172.21.15.62:0/3007313370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd8228d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c29f86640 1 -- 172.21.15.62:0/3007313370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 msgr2=0x55dd822940a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c29f86640 1 --2- 172.21.15.62:0/3007313370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd822940a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c29f86640 1 -- 172.21.15.62:0/3007313370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 msgr2=0x55dd822a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c29f86640 1 --2- 172.21.15.62:0/3007313370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 0x55dd822a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c29f86640 1 -- 172.21.15.62:0/3007313370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd822a1360 con 0x55dd82295370 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c29f86640 1 --2- 172.21.15.62:0/3007313370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd8228d120 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400b010 tx=0x7f3c140217d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=373767134fc38213 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c0effd640 1 -- 172.21.15.62:0/3007313370 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14032050 con 0x55dd82295370 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c0effd640 1 -- 172.21.15.62:0/3007313370 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c14018ca0 con 0x55dd82295370 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.301+0000 7f3c0effd640 1 -- 172.21.15.62:0/3007313370 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c14021df0 con 0x55dd82295370 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.302+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3007313370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 msgr2=0x55dd8228d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.302+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/3007313370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd8228d120 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400b010 tx=0x7f3c140217d0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.304+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3007313370 shutdown_connections 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.304+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/3007313370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd8228d120 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.304+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/3007313370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228bb40 0x55dd822940a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.304+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/3007313370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228a5d0 0x55dd822a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.304+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3007313370 >> 172.21.15.62:0/3007313370 conn(0x55dd82253310 msgr2=0x55dd82293990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.305+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3007313370 shutdown_connections 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.305+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/3007313370 wait complete. 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.306+0000 7f3c2ddc2740 1 -- start start 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.306+0000 7f3c2ddc2740 1 Processor -- start 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.306+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228a5d0 0x55dd8229cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.306+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228bb40 0x55dd8229d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.306+0000 7f3c2ddc2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0800e2d0 con 0x55dd8228bb40 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0800f740 con 0x55dd82295370 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c2ddc2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0800e9a0 con 0x55dd8228a5d0 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c2ddc2740 10 client.0 _ll_register_callbacks cb 0x7f3c14025070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c28f84640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228bb40 0x55dd8229d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c29785640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228a5d0 0x55dd8229cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c29f86640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c28f84640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228bb40 0x55dd8229d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54004/0 (socket says 172.21.15.62:54004) 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c28f84640 1 -- 172.21.15.62:0/696274230 learned_addr learned my addr 172.21.15.62:0/696274230 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c29785640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228a5d0 0x55dd8229cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52410/0 (socket says 172.21.15.62:52410) 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c29f86640 1 -- 172.21.15.62:0/696274230 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228a5d0 msgr2=0x55dd8229cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c29f86640 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228a5d0 0x55dd8229cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c29f86640 1 -- 172.21.15.62:0/696274230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228bb40 msgr2=0x55dd8229d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c29f86640 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228bb40 0x55dd8229d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c29f86640 1 -- 172.21.15.62:0/696274230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd822984c0 con 0x55dd82295370 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.307+0000 7f3c29f86640 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297ed0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3c14022710 tx=0x7f3c14009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5a210838f4f6bb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c0cff9640 1 -- 172.21.15.62:0/696274230 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1404c060 con 0x55dd82295370 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c0cff9640 1 -- 172.21.15.62:0/696274230 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3c14044050 con 0x55dd82295370 2024-09-15T23:18:56.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/696274230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c14018ca0 con 0x55dd82295370 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c0cff9640 1 -- 172.21.15.62:0/696274230 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3c1404c060 con 0x55dd82295370 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c2ddc2740 10 client.25763 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/696274230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dd8230eb20 con 0x55dd82295370 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c0cff9640 1 -- 172.21.15.62:0/696274230 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c14018ca0 con 0x55dd82295370 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c0cff9640 1 -- 172.21.15.62:0/696274230 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f3c14046070 con 0x55dd82295370 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c2ddc2740 10 client.25763 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c2ddc2740 10 client.25763 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.308+0000 7f3c2ddc2740 20 client.25763 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.309+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55dd8230f1b0 0x55dd823115d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.309+0000 7f3c2ddc2740 4 client.25763 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.309+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/696274230 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dd82174170 con 0x55dd8230f1b0 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.309+0000 7f3c29785640 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55dd8230f1b0 0x55dd823115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.309+0000 7f3c0cff9640 10 client.25763 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.309+0000 7f3c29785640 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55dd8230f1b0 0x55dd823115d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=730c59c99385cf00 server_cookie=4757558250018d5a in_seq=0 out_seq=0 2024-09-15T23:18:56.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.309+0000 7f3c0cff9640 1 -- 172.21.15.62:0/696274230 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dd82174170 con 0x55dd8230f1b0 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.309+0000 7f3c0cff9640 10 client.25763 handle_command_reply: tid=0 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.309+0000 7f3c2ddc2740 1 client.25763 shutdown 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.310+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/696274230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 msgr2=0x55dd82297ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.310+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297ed0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3c14022710 tx=0x7f3c14009bf0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.310+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/696274230 shutdown_connections 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.310+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55dd8230f1b0 0x55dd823115d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.310+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55dd82295370 0x55dd82297ed0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.310+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55dd8228bb40 0x55dd8229d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.310+0000 7f3c2ddc2740 1 --2- 172.21.15.62:0/696274230 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55dd8228a5d0 0x55dd8229cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.310+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/696274230 >> 172.21.15.62:0/696274230 conn(0x55dd82253310 msgr2=0x55dd822941b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.311+0000 7f3c0cff9640 0 client.25763 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.311+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/696274230 shutdown_connections 2024-09-15T23:18:56.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.311+0000 7f3c2ddc2740 1 -- 172.21.15.62:0/696274230 wait complete. 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.311+0000 7f3c2ddc2740 20 client.25763 trim_cache size 0 max 16384 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3c100a3450 msgr2=0x7f3c100a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3c100a3450 0x7f3c100a5910 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3c14017750 tx=0x7f3c1400f000 comp rx=0 tx=0).stop 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/605962943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c24100e20 msgr2=0x7f3c2410e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c24100e20 0x7f3c2410e2a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c00ba50 tx=0x7f3c1c00bf10 comp rx=0 tx=0).stop 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/605962943 shutdown_connections 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3c100a3450 0x7f3c100a5910 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c24102ef0 0x7f3c241130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3c24102570 0x7f3c2410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 --2- 172.21.15.62:0/605962943 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3c24100e20 0x7f3c2410e2a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/605962943 >> 172.21.15.62:0/605962943 conn(0x7f3c240fc7d0 msgr2=0x7f3c24106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/605962943 shutdown_connections 2024-09-15T23:18:56.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:56.312+0000 7f3c2b9e9640 1 -- 172.21.15.62:0/605962943 wait complete. 2024-09-15T23:18:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.102+0000 7f7025835640 1 -- 172.21.15.62:0/2132153664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7020102600 msgr2=0x7f7020108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.102+0000 7f7025835640 1 --2- 172.21.15.62:0/2132153664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7020102600 0x7f7020108e90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f701000d910 tx=0x7f701002c390 comp rx=0 tx=0).stop 2024-09-15T23:18:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.104+0000 7f7025835640 1 -- 172.21.15.62:0/2132153664 shutdown_connections 2024-09-15T23:18:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.104+0000 7f7025835640 1 --2- 172.21.15.62:0/2132153664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7020102600 0x7f7020108e90 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.104+0000 7f7025835640 1 --2- 172.21.15.62:0/2132153664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7020101c40 0x7f70201020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.104+0000 7f7025835640 1 --2- 172.21.15.62:0/2132153664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7020100a50 0x7f7020100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.104+0000 7f7025835640 1 -- 172.21.15.62:0/2132153664 >> 172.21.15.62:0/2132153664 conn(0x7f70200fc1b0 msgr2=0x7f70200fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:57.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.106+0000 7f7025835640 1 -- 172.21.15.62:0/2132153664 shutdown_connections 2024-09-15T23:18:57.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.106+0000 7f7025835640 1 -- 172.21.15.62:0/2132153664 wait complete. 2024-09-15T23:18:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.107+0000 7f7025835640 1 Processor -- start 2024-09-15T23:18:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f7025835640 1 -- start start 2024-09-15T23:18:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f7025835640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7020100a50 0x7f70201a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f7025835640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7020101c40 0x7f70201a7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f7025835640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7020102600 0x7f70201ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f7025835640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f702010b650 con 0x7f7020100a50 2024-09-15T23:18:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f7025835640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f702010b4d0 con 0x7f7020102600 2024-09-15T23:18:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f7025835640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f702010b7d0 con 0x7f7020101c40 2024-09-15T23:18:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f701effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7020100a50 0x7f70201a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f701f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7020102600 0x7f70201ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f701f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7020102600 0x7f70201ae190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39314/0 (socket says 172.21.15.62:39314) 2024-09-15T23:18:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f701f7fe640 1 -- 172.21.15.62:0/4223678061 learned_addr learned my addr 172.21.15.62:0/4223678061 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.108+0000 7f701e7fc640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7020101c40 0x7f70201a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.109+0000 7f701f7fe640 1 -- 172.21.15.62:0/4223678061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7020101c40 msgr2=0x7f70201a7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:57.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.109+0000 7f701f7fe640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7020101c40 0x7f70201a7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.109+0000 7f701f7fe640 1 -- 172.21.15.62:0/4223678061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7020100a50 msgr2=0x7f70201a6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:57.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.109+0000 7f701f7fe640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7020100a50 0x7f70201a6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.109+0000 7f701f7fe640 1 -- 172.21.15.62:0/4223678061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70201ae830 con 0x7f7020102600 2024-09-15T23:18:57.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.110+0000 7f701f7fe640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7020102600 0x7f70201ae190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f701000d860 tx=0x7f70100349c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.110+0000 7f7024833640 1 -- 172.21.15.62:0/4223678061 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7010034d40 con 0x7f7020102600 2024-09-15T23:18:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.110+0000 7f7024833640 1 -- 172.21.15.62:0/4223678061 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7010041070 con 0x7f7020102600 2024-09-15T23:18:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.110+0000 7f7024833640 1 -- 172.21.15.62:0/4223678061 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70100336f0 con 0x7f7020102600 2024-09-15T23:18:57.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.112+0000 7f7025835640 1 -- 172.21.15.62:0/4223678061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70201aeac0 con 0x7f7020102600 2024-09-15T23:18:57.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.112+0000 7f7025835640 1 -- 172.21.15.62:0/4223678061 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70201aef50 con 0x7f7020102600 2024-09-15T23:18:57.118 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.116+0000 7f7024833640 1 -- 172.21.15.62:0/4223678061 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f701000c050 con 0x7f7020102600 2024-09-15T23:18:57.118 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.117+0000 7f7024833640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f70080a3520 0x7f70080a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.118 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.117+0000 7f7024833640 1 -- 172.21.15.62:0/4223678061 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f70100ef3a0 con 0x7f7020102600 2024-09-15T23:18:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.120+0000 7f701effd640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f70080a3520 0x7f70080a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.120+0000 7f70285d0740 1 Processor -- start 2024-09-15T23:18:57.121 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.121+0000 7f70285d0740 1 -- start start 2024-09-15T23:18:57.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.121+0000 7f701effd640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f70080a3520 0x7f70080a59e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f70201a80f0 tx=0x7f70180094f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:57.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.121+0000 7f70285d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6009a0 0x55d70e600da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f70285d0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e605da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f70285d0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 0x55d70e608830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f70285d0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d70e5642f0 con 0x55d70e6012e0 2024-09-15T23:18:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f70285d0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d70e50c270 con 0x55d70e606370 2024-09-15T23:18:57.123 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f70285d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d70e5602b0 con 0x55d70e6009a0 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6009a0 0x55d70e600da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6009a0 0x55d70e600da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52440/0 (socket says 172.21.15.62:52440) 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701effd640 1 -- 172.21.15.62:0/3830130561 learned_addr learned my addr 172.21.15.62:0/3830130561 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701f7fe640 1 --2- 172.21.15.62:0/3830130561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 0x55d70e608830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701e7fc640 1 --2- 172.21.15.62:0/3830130561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e605da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701effd640 1 -- 172.21.15.62:0/3830130561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 msgr2=0x55d70e608830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701effd640 1 --2- 172.21.15.62:0/3830130561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 0x55d70e608830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701effd640 1 -- 172.21.15.62:0/3830130561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 msgr2=0x55d70e605da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701effd640 1 --2- 172.21.15.62:0/3830130561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e605da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.122+0000 7f701effd640 1 -- 172.21.15.62:0/3830130561 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70e50b340 con 0x55d70e6009a0 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.123+0000 7f701effd640 1 --2- 172.21.15.62:0/3830130561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6009a0 0x55d70e600da0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7018017730 tx=0x7f7018011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28652695effc9618 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.123+0000 7f7005ffb640 1 -- 172.21.15.62:0/3830130561 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018027070 con 0x55d70e6009a0 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.123+0000 7f7005ffb640 1 -- 172.21.15.62:0/3830130561 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7018017de0 con 0x55d70e6009a0 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.123+0000 7f7005ffb640 1 -- 172.21.15.62:0/3830130561 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70180166a0 con 0x55d70e6009a0 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.124+0000 7f70285d0740 1 -- 172.21.15.62:0/3830130561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6009a0 msgr2=0x55d70e600da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.124+0000 7f70285d0740 1 --2- 172.21.15.62:0/3830130561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6009a0 0x55d70e600da0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7018017730 tx=0x7f7018011000 comp rx=0 tx=0).stop 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.124+0000 7f70285d0740 1 -- 172.21.15.62:0/3830130561 shutdown_connections 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.124+0000 7f70285d0740 1 --2- 172.21.15.62:0/3830130561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 0x55d70e608830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.124+0000 7f70285d0740 1 --2- 172.21.15.62:0/3830130561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e605da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.124+0000 7f70285d0740 1 --2- 172.21.15.62:0/3830130561 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6009a0 0x55d70e600da0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.124+0000 7f70285d0740 1 -- 172.21.15.62:0/3830130561 >> 172.21.15.62:0/3830130561 conn(0x55d70e5fc010 msgr2=0x55d70e5fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.124+0000 7f70285d0740 1 -- 172.21.15.62:0/3830130561 shutdown_connections 2024-09-15T23:18:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.124+0000 7f70285d0740 1 -- 172.21.15.62:0/3830130561 wait complete. 2024-09-15T23:18:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.125+0000 7f70285d0740 1 -- start start 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.125+0000 7f70285d0740 1 Processor -- start 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f70285d0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 0x55d70e60acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f70285d0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e60b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f70285d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e6121c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f70285d0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d70e609340 con 0x55d70e6012e0 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f70285d0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d70e4ed290 con 0x55d70e6009a0 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f70285d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d70e6094c0 con 0x55d70e606370 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f70285d0740 10 client.0 _ll_register_callbacks cb 0x55d70e51e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f701e7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e60b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f701e7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e60b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54044/0 (socket says 172.21.15.62:54044) 2024-09-15T23:18:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f701e7fc640 1 -- 172.21.15.62:0/1129972618 learned_addr learned my addr 172.21.15.62:0/1129972618 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f701effd640 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 0x55d70e60acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.126+0000 7f701f7fe640 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e6121c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.127+0000 7f701effd640 1 -- 172.21.15.62:0/1129972618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 msgr2=0x55d70e6121c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.127+0000 7f701effd640 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e6121c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.127+0000 7f701effd640 1 -- 172.21.15.62:0/1129972618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 msgr2=0x55d70e60b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.127+0000 7f701effd640 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e60b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.129 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.127+0000 7f701effd640 1 -- 172.21.15.62:0/1129972618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d70e277210 con 0x55d70e6009a0 2024-09-15T23:18:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.128+0000 7f701effd640 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 0x55d70e60acb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7018009d20 tx=0x7f7018022900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a645062d621f87bf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.128+0000 7f6ffeffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:18:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.128+0000 7f6ffeffd640 1 -- 172.21.15.62:0/1129972618 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018027040 con 0x55d70e6009a0 2024-09-15T23:18:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.128+0000 7f6ffeffd640 1 -- 172.21.15.62:0/1129972618 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7018022970 con 0x55d70e6009a0 2024-09-15T23:18:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.128+0000 7f70285d0740 1 -- 172.21.15.62:0/1129972618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7018017de0 con 0x55d70e6009a0 2024-09-15T23:18:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.128+0000 7f6ffeffd640 1 -- 172.21.15.62:0/1129972618 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f701803b050 con 0x55d70e6009a0 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.128+0000 7f70285d0740 10 client.25772 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.128+0000 7f70285d0740 1 -- 172.21.15.62:0/1129972618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d70e343620 con 0x55d70e6009a0 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.129+0000 7f6ffeffd640 1 -- 172.21.15.62:0/1129972618 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7018032070 con 0x55d70e6009a0 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.129+0000 7f6ffeffd640 1 -- 172.21.15.62:0/1129972618 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f701802d4e0 con 0x55d70e6009a0 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.129+0000 7f70285d0740 10 client.25772 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.129+0000 7f70285d0740 10 client.25772 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.129+0000 7f70285d0740 20 client.25772 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.129+0000 7f70285d0740 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d70e6165d0 0x55d70e6189c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.129+0000 7f70285d0740 4 client.25772 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.129+0000 7f70285d0740 1 -- 172.21.15.62:0/1129972618 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7018017de0 con 0x55d70e6165d0 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.129+0000 7f701e7fc640 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d70e6165d0 0x55d70e6189c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.130+0000 7f6ffeffd640 10 client.25772 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.130+0000 7f701e7fc640 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d70e6165d0 0x55d70e6189c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68a081d860d905a6 server_cookie=87a02c17aadf1bf4 in_seq=0 out_seq=0 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f6ffeffd640 1 -- 172.21.15.62:0/1129972618 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f7018017de0 con 0x55d70e6165d0 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f6ffeffd640 10 client.25772 handle_command_reply: tid=0 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 client.25772 shutdown 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 -- 172.21.15.62:0/1129972618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 msgr2=0x55d70e60acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 0x55d70e60acb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7018009d20 tx=0x7f7018022900 comp rx=0 tx=0).stop 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 -- 172.21.15.62:0/1129972618 shutdown_connections 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d70e6165d0 0x55d70e6189c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e6121c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e60b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 --2- 172.21.15.62:0/1129972618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 0x55d70e60acb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f6ffeffd640 0 client.25772 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 -- 172.21.15.62:0/1129972618 >> 172.21.15.62:0/1129972618 conn(0x55d70e5fc010 msgr2=0x55d70e5fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 -- 172.21.15.62:0/1129972618 shutdown_connections 2024-09-15T23:18:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 1 -- 172.21.15.62:0/1129972618 wait complete. 2024-09-15T23:18:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.131+0000 7f70285d0740 20 client.25772 trim_cache size 0 max 16384 2024-09-15T23:18:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.159+0000 7f70285d0740 1 Processor -- start 2024-09-15T23:18:57.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.159+0000 7f70285d0740 1 -- start start 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f70285d0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 0x55d70e600d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f70285d0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e612150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f70285d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e5fb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f70285d0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f70201bbd40 con 0x55d70e6012e0 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f70285d0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70201b95e0 con 0x55d70e6009a0 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f70285d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f70201b7430 con 0x55d70e606370 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701e7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e612150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e5fb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e5fb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52460/0 (socket says 172.21.15.62:52460) 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701f7fe640 1 -- 172.21.15.62:0/1158356942 learned_addr learned my addr 172.21.15.62:0/1158356942 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701effd640 1 --2- 172.21.15.62:0/1158356942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 0x55d70e600d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701f7fe640 1 -- 172.21.15.62:0/1158356942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 msgr2=0x55d70e600d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701f7fe640 1 --2- 172.21.15.62:0/1158356942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 0x55d70e600d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701f7fe640 1 -- 172.21.15.62:0/1158356942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 msgr2=0x55d70e612150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:57.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701f7fe640 1 --2- 172.21.15.62:0/1158356942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e612150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.160+0000 7f701f7fe640 1 -- 172.21.15.62:0/1158356942 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d70e618d30 con 0x55d70e606370 2024-09-15T23:18:57.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.161+0000 7f701f7fe640 1 --2- 172.21.15.62:0/1158356942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e5fb0b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f70100b79b0 tx=0x7f701003cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1519d51ed186cf42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:57.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.161+0000 7f70057fa640 1 -- 172.21.15.62:0/1158356942 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70100b9720 con 0x55d70e606370 2024-09-15T23:18:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.161+0000 7f70057fa640 1 -- 172.21.15.62:0/1158356942 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7010030040 con 0x55d70e606370 2024-09-15T23:18:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.161+0000 7f70057fa640 1 -- 172.21.15.62:0/1158356942 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70100c4250 con 0x55d70e606370 2024-09-15T23:18:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.162+0000 7f70285d0740 1 -- 172.21.15.62:0/1158356942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 msgr2=0x55d70e5fb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:57.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.162+0000 7f70285d0740 1 --2- 172.21.15.62:0/1158356942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e5fb0b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f70100b79b0 tx=0x7f701003cd60 comp rx=0 tx=0).stop 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.162+0000 7f70285d0740 1 -- 172.21.15.62:0/1158356942 shutdown_connections 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.162+0000 7f70285d0740 1 --2- 172.21.15.62:0/1158356942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e606370 0x55d70e5fb0b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.162+0000 7f70285d0740 1 --2- 172.21.15.62:0/1158356942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6012e0 0x55d70e612150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.162+0000 7f70285d0740 1 --2- 172.21.15.62:0/1158356942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e6009a0 0x55d70e600d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.162+0000 7f70285d0740 1 -- 172.21.15.62:0/1158356942 >> 172.21.15.62:0/1158356942 conn(0x55d70e5fc010 msgr2=0x55d70e5fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.162+0000 7f70285d0740 1 -- 172.21.15.62:0/1158356942 shutdown_connections 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.162+0000 7f70285d0740 1 -- 172.21.15.62:0/1158356942 wait complete. 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.163+0000 7f70285d0740 1 -- start start 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.163+0000 7f70285d0740 1 Processor -- start 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.163+0000 7f70285d0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6009a0 0x55d70e60fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f70285d0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6012e0 0x55d70e610060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f70285d0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 0x55d70e611070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f70285d0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f70201b7aa0 con 0x55d70e6009a0 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f70285d0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70201b8f10 con 0x55d70e606370 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f70285d0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f70201b8170 con 0x55d70e6012e0 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6009a0 0x55d70e60fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 0x55d70e611070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6009a0 0x55d70e60fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54056/0 (socket says 172.21.15.62:54056) 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701effd640 1 -- 172.21.15.62:0/3909945376 learned_addr learned my addr 172.21.15.62:0/3909945376 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701e7fc640 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6012e0 0x55d70e610060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f70285d0740 10 client.0 _ll_register_callbacks cb 0x7f7018025070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701f7fe640 1 -- 172.21.15.62:0/3909945376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6012e0 msgr2=0x55d70e610060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701f7fe640 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6012e0 0x55d70e610060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701f7fe640 1 -- 172.21.15.62:0/3909945376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6009a0 msgr2=0x55d70e60fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701f7fe640 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6009a0 0x55d70e60fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701f7fe640 1 -- 172.21.15.62:0/3909945376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d70e611680 con 0x55d70e606370 2024-09-15T23:18:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.164+0000 7f701e7fc640 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6012e0 0x55d70e610060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:18:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.165+0000 7f701f7fe640 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 0x55d70e611070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f701002c980 tx=0x7f7010005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1071b7c4f216a44 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.165+0000 7f6ffeffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:18:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.165+0000 7f6ffeffd640 1 -- 172.21.15.62:0/3909945376 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701000d540 con 0x55d70e606370 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.165+0000 7f6ffeffd640 1 -- 172.21.15.62:0/3909945376 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f70100baa30 con 0x55d70e606370 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.165+0000 7f70285d0740 1 -- 172.21.15.62:0/3909945376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7010030040 con 0x55d70e606370 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.165+0000 7f6ffeffd640 1 -- 172.21.15.62:0/3909945376 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f70100bb5e0 con 0x55d70e606370 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.165+0000 7f70285d0740 10 client.25778 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.165+0000 7f70285d0740 1 -- 172.21.15.62:0/3909945376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d70e681590 con 0x55d70e606370 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.166+0000 7f6ffeffd640 1 -- 172.21.15.62:0/3909945376 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70100bbce0 con 0x55d70e606370 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.166+0000 7f6ffeffd640 1 -- 172.21.15.62:0/3909945376 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f70100d4460 con 0x55d70e606370 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.166+0000 7f70285d0740 10 client.25778 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.166+0000 7f70285d0740 10 client.25778 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.166+0000 7f70285d0740 20 client.25778 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.166+0000 7f70285d0740 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d70e681f10 0x55d70e684330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.166+0000 7f70285d0740 4 client.25778 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:18:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.166+0000 7f70285d0740 1 -- 172.21.15.62:0/3909945376 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7010030040 con 0x55d70e681f10 2024-09-15T23:18:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.166+0000 7f701effd640 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d70e681f10 0x55d70e684330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.167+0000 7f6ffeffd640 10 client.25778 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.167+0000 7f701effd640 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d70e681f10 0x55d70e684330 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9668c23b15aebbe1 server_cookie=622f4619128abb2f in_seq=0 out_seq=0 2024-09-15T23:18:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.178+0000 7f6ffeffd640 1 -- 172.21.15.62:0/3909945376 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7010030040 con 0x55d70e681f10 2024-09-15T23:18:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.178+0000 7f6ffeffd640 10 client.25778 handle_command_reply: tid=0 2024-09-15T23:18:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 client.25778 shutdown 2024-09-15T23:18:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 -- 172.21.15.62:0/3909945376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 msgr2=0x55d70e611070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 0x55d70e611070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f701002c980 tx=0x7f7010005fd0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 -- 172.21.15.62:0/3909945376 shutdown_connections 2024-09-15T23:18:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d70e681f10 0x55d70e684330 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d70e606370 0x55d70e611070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f6ffeffd640 0 client.25778 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d70e6012e0 0x55d70e610060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 --2- 172.21.15.62:0/3909945376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d70e6009a0 0x55d70e60fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 -- 172.21.15.62:0/3909945376 >> 172.21.15.62:0/3909945376 conn(0x55d70e5fc010 msgr2=0x55d70e5fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 -- 172.21.15.62:0/3909945376 shutdown_connections 2024-09-15T23:18:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 1 -- 172.21.15.62:0/3909945376 wait complete. 2024-09-15T23:18:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.179+0000 7f70285d0740 20 client.25778 trim_cache size 0 max 16384 2024-09-15T23:18:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:18:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:18:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:18:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:18:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 -- 172.21.15.62:0/4223678061 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f70080a3520 msgr2=0x7f70080a59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f70080a3520 0x7f70080a59e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f70201a80f0 tx=0x7f70180094f0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 -- 172.21.15.62:0/4223678061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7020102600 msgr2=0x7f70201ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7020102600 0x7f70201ae190 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f701000d860 tx=0x7f70100349c0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 -- 172.21.15.62:0/4223678061 shutdown_connections 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f70080a3520 0x7f70080a59e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7020102600 0x7f70201ae190 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7020101c40 0x7f70201a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 --2- 172.21.15.62:0/4223678061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7020100a50 0x7f70201a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 -- 172.21.15.62:0/4223678061 >> 172.21.15.62:0/4223678061 conn(0x7f70200fc1b0 msgr2=0x7f7020104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 -- 172.21.15.62:0/4223678061 shutdown_connections 2024-09-15T23:18:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:18:57.180+0000 7f7025835640 1 -- 172.21.15.62:0/4223678061 wait complete. 2024-09-15T23:18:57.648 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:18:57.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:57 smithi062 ceph-mon[30949]: pgmap v450: 129 pgs: 129 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.6 MiB/s wr, 573 op/s 2024-09-15T23:18:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:57 smithi067 ceph-mon[35665]: pgmap v450: 129 pgs: 129 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.6 MiB/s wr, 573 op/s 2024-09-15T23:18:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:57 smithi148 ceph-mon[35610]: pgmap v450: 129 pgs: 129 active+clean; 1.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 6.6 MiB/s wr, 573 op/s 2024-09-15T23:18:58.073 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:18:58.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.661+0000 7f509f7fe640 1 -- 172.21.15.62:0/368669983 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50980319b0 con 0x7f50a01091f0 2024-09-15T23:18:58.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.662+0000 7f50a6c38640 1 -- 172.21.15.62:0/368669983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50a01091f0 msgr2=0x7f50a01095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:58.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.662+0000 7f50a6c38640 1 --2- 172.21.15.62:0/368669983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50a01091f0 0x7f50a01095d0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5098009900 tx=0x7f509802a480 comp rx=0 tx=0).stop 2024-09-15T23:18:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.662+0000 7f50a6c38640 1 -- 172.21.15.62:0/368669983 shutdown_connections 2024-09-15T23:18:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.662+0000 7f50a6c38640 1 --2- 172.21.15.62:0/368669983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0114ab0 0x7f50a0116ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.662+0000 7f50a6c38640 1 --2- 172.21.15.62:0/368669983 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50a0109b10 0x7f50a0114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.662+0000 7f50a6c38640 1 --2- 172.21.15.62:0/368669983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50a01091f0 0x7f50a01095d0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.662+0000 7f50a6c38640 1 -- 172.21.15.62:0/368669983 >> 172.21.15.62:0/368669983 conn(0x7f50a006e140 msgr2=0x7f50a006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.662+0000 7f50a6c38640 1 -- 172.21.15.62:0/368669983 shutdown_connections 2024-09-15T23:18:58.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.662+0000 7f50a6c38640 1 -- 172.21.15.62:0/368669983 wait complete. 2024-09-15T23:18:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.663+0000 7f50a6c38640 1 Processor -- start 2024-09-15T23:18:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.663+0000 7f50a6c38640 1 -- start start 2024-09-15T23:18:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.663+0000 7f50a6c38640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50a01091f0 0x7f50a01a2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.664+0000 7f50a6c38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0109b10 0x7f50a01a34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.664+0000 7f50a6c38640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50a0114ab0 0x7f50a01a7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.664+0000 7f50a6c38640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f50a011a7c0 con 0x7f50a01091f0 2024-09-15T23:18:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.664+0000 7f50a6c38640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50a011a640 con 0x7f50a0109b10 2024-09-15T23:18:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.664+0000 7f50a6c38640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f50a011a940 con 0x7f50a0114ab0 2024-09-15T23:18:58.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.664+0000 7f50a51d5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50a0114ab0 0x7f50a01a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.664+0000 7f50a51d5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50a0114ab0 0x7f50a01a7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52488/0 (socket says 172.21.15.62:52488) 2024-09-15T23:18:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.664+0000 7f50a51d5640 1 -- 172.21.15.62:0/2145583497 learned_addr learned my addr 172.21.15.62:0/2145583497 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.665+0000 7f50a51d5640 1 -- 172.21.15.62:0/2145583497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0109b10 msgr2=0x7f50a01a34c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:18:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.665+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0109b10 0x7f50a01a34c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.665+0000 7f50a51d5640 1 -- 172.21.15.62:0/2145583497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50a01091f0 msgr2=0x7f50a01a2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.665+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50a01091f0 0x7f50a01a2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.665+0000 7f50a51d5640 1 -- 172.21.15.62:0/2145583497 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a01a7f30 con 0x7f50a0114ab0 2024-09-15T23:18:58.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.665+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50a0114ab0 0x7f50a01a7850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f509000e940 tx=0x7f509000ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:58.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.667+0000 7f509dffb640 1 -- 172.21.15.62:0/2145583497 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509000ccc0 con 0x7f50a0114ab0 2024-09-15T23:18:58.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.667+0000 7f509dffb640 1 -- 172.21.15.62:0/2145583497 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5090005500 con 0x7f50a0114ab0 2024-09-15T23:18:58.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.667+0000 7f50a6c38640 1 -- 172.21.15.62:0/2145583497 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a01a8220 con 0x7f50a0114ab0 2024-09-15T23:18:58.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.667+0000 7f509dffb640 1 -- 172.21.15.62:0/2145583497 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090010690 con 0x7f50a0114ab0 2024-09-15T23:18:58.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.668+0000 7f50a6c38640 1 -- 172.21.15.62:0/2145583497 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a01afb00 con 0x7f50a0114ab0 2024-09-15T23:18:58.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.671+0000 7f509dffb640 1 -- 172.21.15.62:0/2145583497 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f50900046c0 con 0x7f50a0114ab0 2024-09-15T23:18:58.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.672+0000 7f50a9011740 1 Processor -- start 2024-09-15T23:18:58.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.672+0000 7f50a9011740 1 -- start start 2024-09-15T23:18:58.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.672+0000 7f509dffb640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50800a3410 0x7f50800a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a9011740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4b5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f509dffb640 1 -- 172.21.15.62:0/2145583497 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f50900cffc0 con 0x7f50a0114ab0 2024-09-15T23:18:58.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f509ffff640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50800a3410 0x7f50800a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a9011740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 0x558c6d4b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a9011740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4c2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a9011740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558c6d41b2f0 con 0x558c6d4b55d0 2024-09-15T23:18:58.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a49d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4b5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a49d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4b5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39356/0 (socket says 172.21.15.62:39356) 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a49d4640 1 -- 172.21.15.62:0/2412687062 learned_addr learned my addr 172.21.15.62:0/2412687062 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a9011740 1 -- 172.21.15.62:0/2412687062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558c6d3c2cf0 con 0x558c6d4b6b40 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a9011740 1 -- 172.21.15.62:0/2412687062 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558c6d4172b0 con 0x558c6d4c0370 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f509ffff640 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 0x558c6d4b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f509ffff640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50800a3410 0x7f50800a58d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f50a0072c70 tx=0x7f5088009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.673+0000 7f50a51d5640 1 -- 172.21.15.62:0/2412687062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 msgr2=0x558c6d4b5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.674+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4b5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.674+0000 7f50a51d5640 1 -- 172.21.15.62:0/2412687062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 msgr2=0x558c6d4b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.674+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 0x558c6d4b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.674+0000 7f50a51d5640 1 -- 172.21.15.62:0/2412687062 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6d3a73a0 con 0x558c6d4c0370 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.674+0000 7f50a49d4640 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:58.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.674+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4c2760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f50900cd6e0 tx=0x7f509009be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a848fd1dfec6255 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:58.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.674+0000 7f5096ffd640 1 -- 172.21.15.62:0/2412687062 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509009c200 con 0x558c6d4c0370 2024-09-15T23:18:58.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.674+0000 7f5096ffd640 1 -- 172.21.15.62:0/2412687062 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50900b5070 con 0x558c6d4c0370 2024-09-15T23:18:58.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.674+0000 7f5096ffd640 1 -- 172.21.15.62:0/2412687062 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090098870 con 0x558c6d4c0370 2024-09-15T23:18:58.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.675+0000 7f50a9011740 1 -- 172.21.15.62:0/2412687062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 msgr2=0x558c6d4c2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:58.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.675+0000 7f50a9011740 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4c2760 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f50900cd6e0 tx=0x7f509009be00 comp rx=0 tx=0).stop 2024-09-15T23:18:58.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.676+0000 7f50a9011740 1 -- 172.21.15.62:0/2412687062 shutdown_connections 2024-09-15T23:18:58.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.677+0000 7f50a9011740 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4c2760 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.677+0000 7f50a9011740 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 0x558c6d4b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.677+0000 7f50a9011740 1 --2- 172.21.15.62:0/2412687062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4b5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.677+0000 7f50a9011740 1 -- 172.21.15.62:0/2412687062 >> 172.21.15.62:0/2412687062 conn(0x558c6d47e310 msgr2=0x558c6d4b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:58.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.677+0000 7f50a9011740 1 -- 172.21.15.62:0/2412687062 shutdown_connections 2024-09-15T23:18:58.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.677+0000 7f50a9011740 1 -- 172.21.15.62:0/2412687062 wait complete. 2024-09-15T23:18:58.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.679+0000 7f50a9011740 1 -- start start 2024-09-15T23:18:58.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.679+0000 7f50a9011740 1 Processor -- start 2024-09-15T23:18:58.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.679+0000 7f50a9011740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 0x558c6d4bfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.679+0000 7f50a9011740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.679+0000 7f50a9011740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4bd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a9011740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558c6d4c3070 con 0x558c6d4b55d0 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a9011740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558c6d3d6060 con 0x558c6d4b6b40 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f509ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a51d5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4bd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a49d4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 0x558c6d4bfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a51d5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4bd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52504/0 (socket says 172.21.15.62:52504) 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f509ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bcfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39360/0 (socket says 172.21.15.62:39360) 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a9011740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558c6d4c31f0 con 0x558c6d4c0370 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a51d5640 1 -- 172.21.15.62:0/2705450575 learned_addr learned my addr 172.21.15.62:0/2705450575 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a9011740 10 client.0 _ll_register_callbacks cb 0x558c6d1298c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a51d5640 1 -- 172.21.15.62:0/2705450575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 msgr2=0x558c6d4bcfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bcfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a51d5640 1 -- 172.21.15.62:0/2705450575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 msgr2=0x558c6d4bfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:58.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 0x558c6d4bfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a51d5640 1 -- 172.21.15.62:0/2705450575 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6d18c3c0 con 0x558c6d4c0370 2024-09-15T23:18:58.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f50a51d5640 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4bd570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f50900a4940 tx=0x7f50900cda00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1619836b71ca664 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:58.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.680+0000 7f508ffff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:18:58.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.681+0000 7f508ffff640 1 -- 172.21.15.62:0/2705450575 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50900b9020 con 0x558c6d4c0370 2024-09-15T23:18:58.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.681+0000 7f508ffff640 1 -- 172.21.15.62:0/2705450575 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f50900bc0c0 con 0x558c6d4c0370 2024-09-15T23:18:58.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.681+0000 7f50a9011740 1 -- 172.21.15.62:0/2705450575 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50900b5070 con 0x558c6d4c0370 2024-09-15T23:18:58.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f508ffff640 1 -- 172.21.15.62:0/2705450575 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f50900af050 con 0x558c6d4c0370 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f50a9011740 10 client.25633 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f50a9011740 1 -- 172.21.15.62:0/2705450575 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c6d4c3070 con 0x558c6d4c0370 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f508ffff640 1 -- 172.21.15.62:0/2705450575 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50900b5070 con 0x558c6d4c0370 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f508ffff640 1 -- 172.21.15.62:0/2705450575 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f50900b0240 con 0x558c6d4c0370 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f50a9011740 10 client.25633 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f50a9011740 10 client.25633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f50a9011740 20 client.25633 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f50a9011740 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558c6d4cfc00 0x558c6d4d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f50a9011740 4 client.25633 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f50a9011740 1 -- 172.21.15.62:0/2705450575 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c6d428860 con 0x558c6d4cfc00 2024-09-15T23:18:58.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.685+0000 7f50a49d4640 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558c6d4cfc00 0x558c6d4d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.688+0000 7f50a49d4640 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558c6d4cfc00 0x558c6d4d1ff0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d8b46aecf6e9d4 server_cookie=3505c00f44f22cab in_seq=0 out_seq=0 2024-09-15T23:18:58.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.688+0000 7f508ffff640 10 client.25633 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:58.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.689+0000 7f508ffff640 1 -- 172.21.15.62:0/2705450575 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558c6d428860 con 0x558c6d4cfc00 2024-09-15T23:18:58.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.689+0000 7f508ffff640 10 client.25633 handle_command_reply: tid=0 2024-09-15T23:18:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.689+0000 7f50a9011740 1 client.25633 shutdown 2024-09-15T23:18:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 -- 172.21.15.62:0/2705450575 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 msgr2=0x558c6d4bd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4bd570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f50900a4940 tx=0x7f50900cda00 comp rx=0 tx=0).stop 2024-09-15T23:18:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 -- 172.21.15.62:0/2705450575 shutdown_connections 2024-09-15T23:18:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558c6d4cfc00 0x558c6d4d1ff0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4c0370 0x558c6d4bd570 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bcfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 --2- 172.21.15.62:0/2705450575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4b55d0 0x558c6d4bfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 -- 172.21.15.62:0/2705450575 >> 172.21.15.62:0/2705450575 conn(0x558c6d47e310 msgr2=0x558c6d4b7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f508ffff640 0 client.25633 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 -- 172.21.15.62:0/2705450575 shutdown_connections 2024-09-15T23:18:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 1 -- 172.21.15.62:0/2705450575 wait complete. 2024-09-15T23:18:58.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.690+0000 7f50a9011740 20 client.25633 trim_cache size 0 max 16384 2024-09-15T23:18:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.711+0000 7f50a9011740 1 Processor -- start 2024-09-15T23:18:58.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.711+0000 7f50a9011740 1 -- start start 2024-09-15T23:18:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.711+0000 7f50a9011740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d4cc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.711+0000 7f50a9011740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f50a9011740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 0x558c6d4bbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f50a49d4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d4cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f509ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f50a49d4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d4cc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52512/0 (socket says 172.21.15.62:52512) 2024-09-15T23:18:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f509ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39364/0 (socket says 172.21.15.62:39364) 2024-09-15T23:18:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f50a51d5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 0x558c6d4bbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f50a49d4640 1 -- 172.21.15.62:0/1920714242 learned_addr learned my addr 172.21.15.62:0/1920714242 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f50a51d5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 0x558c6d4bbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54110/0 (socket says 172.21.15.62:54110) 2024-09-15T23:18:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f50a9011740 1 -- 172.21.15.62:0/1920714242 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f50a01b7cd0 con 0x558c6d4c0370 2024-09-15T23:18:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f50a9011740 1 -- 172.21.15.62:0/1920714242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50a01b5570 con 0x558c6d4b6b40 2024-09-15T23:18:58.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.712+0000 7f50a9011740 1 -- 172.21.15.62:0/1920714242 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f50a01b33e0 con 0x558c6d4b55d0 2024-09-15T23:18:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f50a49d4640 1 -- 172.21.15.62:0/1920714242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 msgr2=0x558c6d4bb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f50a49d4640 1 --2- 172.21.15.62:0/1920714242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f50a49d4640 1 -- 172.21.15.62:0/1920714242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 msgr2=0x558c6d4bbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f50a49d4640 1 --2- 172.21.15.62:0/1920714242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 0x558c6d4bbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f50a49d4640 1 -- 172.21.15.62:0/1920714242 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c6d4b6300 con 0x558c6d4b55d0 2024-09-15T23:18:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f509ffff640 1 --2- 172.21.15.62:0/1920714242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:58.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f50a49d4640 1 --2- 172.21.15.62:0/1920714242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d4cc310 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5098005c20 tx=0x7f5098000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69d9925c6ecb590f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:58.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f50967fc640 1 -- 172.21.15.62:0/1920714242 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098035040 con 0x558c6d4b55d0 2024-09-15T23:18:58.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f50967fc640 1 -- 172.21.15.62:0/1920714242 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5098049070 con 0x558c6d4b55d0 2024-09-15T23:18:58.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.713+0000 7f50967fc640 1 -- 172.21.15.62:0/1920714242 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098004b80 con 0x558c6d4b55d0 2024-09-15T23:18:58.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.716+0000 7f50a9011740 1 -- 172.21.15.62:0/1920714242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 msgr2=0x558c6d4cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:58.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.716+0000 7f50a9011740 1 --2- 172.21.15.62:0/1920714242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d4cc310 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5098005c20 tx=0x7f5098000bc0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.716+0000 7f50a9011740 1 -- 172.21.15.62:0/1920714242 shutdown_connections 2024-09-15T23:18:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.716+0000 7f50a9011740 1 --2- 172.21.15.62:0/1920714242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 0x558c6d4bbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.716+0000 7f50a9011740 1 --2- 172.21.15.62:0/1920714242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d4bb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.716+0000 7f50a9011740 1 --2- 172.21.15.62:0/1920714242 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d4cc310 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.716+0000 7f50a9011740 1 -- 172.21.15.62:0/1920714242 >> 172.21.15.62:0/1920714242 conn(0x558c6d47e310 msgr2=0x558c6d4d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:58.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.717+0000 7f50a9011740 1 -- 172.21.15.62:0/1920714242 shutdown_connections 2024-09-15T23:18:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.717+0000 7f50a9011740 1 -- 172.21.15.62:0/1920714242 wait complete. 2024-09-15T23:18:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.718+0000 7f50a9011740 1 -- start start 2024-09-15T23:18:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.718+0000 7f50a9011740 1 Processor -- start 2024-09-15T23:18:58.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.718+0000 7f50a9011740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d532ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a9011740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d532ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a9011740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 0x558c6d4c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a49d4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d532ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a49d4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d532ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52516/0 (socket says 172.21.15.62:52516) 2024-09-15T23:18:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f509ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d532ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f509ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d532ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39366/0 (socket says 172.21.15.62:39366) 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a49d4640 1 -- 172.21.15.62:0/2394327887 learned_addr learned my addr 172.21.15.62:0/2394327887 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f50a01b3a30 con 0x558c6d4c0370 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50a01b4ea0 con 0x558c6d4b6b40 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f50a01b4100 con 0x558c6d4b55d0 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a9011740 10 client.0 _ll_register_callbacks cb 0x7f509009bfd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a49d4640 1 -- 172.21.15.62:0/2394327887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 msgr2=0x558c6d532ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a49d4640 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d532ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a49d4640 1 -- 172.21.15.62:0/2394327887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 msgr2=0x558c6d4c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a49d4640 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 0x558c6d4c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f50a49d4640 1 -- 172.21.15.62:0/2394327887 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c6d4c4110 con 0x558c6d4b55d0 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.719+0000 7f509ffff640 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d532ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.720+0000 7f50a49d4640 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d532ab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f509802ac70 tx=0x7f5098004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2206835aab19eddc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.720+0000 7f508ffff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:18:58.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.720+0000 7f508ffff640 1 -- 172.21.15.62:0/2394327887 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098051020 con 0x558c6d4b55d0 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.720+0000 7f508ffff640 1 -- 172.21.15.62:0/2394327887 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5098005390 con 0x558c6d4b55d0 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.720+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5098049070 con 0x558c6d4b55d0 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.720+0000 7f508ffff640 1 -- 172.21.15.62:0/2394327887 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f509805c050 con 0x558c6d4b55d0 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.720+0000 7f50a9011740 10 client.25639 fetch_fsmap learned FSMap version 28 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.720+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558c6d53a240 con 0x558c6d4b55d0 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.721+0000 7f508ffff640 1 -- 172.21.15.62:0/2394327887 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5098049070 con 0x558c6d4b55d0 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.721+0000 7f508ffff640 1 -- 172.21.15.62:0/2394327887 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f5098031a00 con 0x558c6d4b55d0 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.721+0000 7f50a9011740 10 client.25639 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.721+0000 7f50a9011740 10 client.25639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.721+0000 7f50a9011740 20 client.25639 populate_metadata read hostname 'smithi062' 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.721+0000 7f50a9011740 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558c6d53a910 0x558c6d53cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.721+0000 7f50a9011740 4 client.25639 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:18:58.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.721+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c6d422d70 con 0x558c6d53a910 2024-09-15T23:18:58.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.721+0000 7f509ffff640 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558c6d53a910 0x558c6d53cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:18:58.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.722+0000 7f509ffff640 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558c6d53a910 0x558c6d53cd70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a295ee77b860c901 server_cookie=72c6c9d7e6c19e6a in_seq=0 out_seq=0 2024-09-15T23:18:58.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.722+0000 7f508ffff640 10 client.25639 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:58.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.722+0000 7f508ffff640 1 -- 172.21.15.62:0/2394327887 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c6d422d70 con 0x558c6d53a910 2024-09-15T23:18:58.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.722+0000 7f508ffff640 10 client.25639 handle_command_reply: tid=0 2024-09-15T23:18:58.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.722+0000 7f50a9011740 1 client.25639 shutdown 2024-09-15T23:18:58.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 msgr2=0x558c6d532ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:58.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d532ab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f509802ac70 tx=0x7f5098004610 comp rx=0 tx=0).stop 2024-09-15T23:18:58.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 shutdown_connections 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558c6d53a910 0x558c6d53cd70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558c6d4c0370 0x558c6d4c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558c6d4b6b40 0x558c6d532ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f508ffff640 0 client.25639 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 --2- 172.21.15.62:0/2394327887 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558c6d4b55d0 0x558c6d532ab0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 >> 172.21.15.62:0/2394327887 conn(0x558c6d47e310 msgr2=0x558c6d4bb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 shutdown_connections 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 1 -- 172.21.15.62:0/2394327887 wait complete. 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.723+0000 7f50a9011740 20 client.25639 trim_cache size 0 max 16384 2024-09-15T23:18:58.727 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:18:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 -- 172.21.15.62:0/2145583497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50800a3410 msgr2=0x7f50800a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50800a3410 0x7f50800a58d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f50a0072c70 tx=0x7f5088009500 comp rx=0 tx=0).stop 2024-09-15T23:18:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 -- 172.21.15.62:0/2145583497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50a0114ab0 msgr2=0x7f50a01a7850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:18:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50a0114ab0 0x7f50a01a7850 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f509000e940 tx=0x7f509000ee00 comp rx=0 tx=0).stop 2024-09-15T23:18:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 -- 172.21.15.62:0/2145583497 shutdown_connections 2024-09-15T23:18:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50800a3410 0x7f50800a58d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50a0114ab0 0x7f50a01a7850 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50a0109b10 0x7f50a01a34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 --2- 172.21.15.62:0/2145583497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50a01091f0 0x7f50a01a2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:18:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 -- 172.21.15.62:0/2145583497 >> 172.21.15.62:0/2145583497 conn(0x7f50a006e140 msgr2=0x7f50a00707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:18:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 -- 172.21.15.62:0/2145583497 shutdown_connections 2024-09-15T23:18:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:18:58.724+0000 7f50a6c38640 1 -- 172.21.15.62:0/2145583497 wait complete. 2024-09-15T23:18:58.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:18:58 smithi062 ceph-mon[30949]: pgmap v451: 129 pgs: 129 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.3 MiB/s wr, 546 op/s 2024-09-15T23:18:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:18:58 smithi067 ceph-mon[35665]: pgmap v451: 129 pgs: 129 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.3 MiB/s wr, 546 op/s 2024-09-15T23:18:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:18:58 smithi148 ceph-mon[35610]: pgmap v451: 129 pgs: 129 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.3 MiB/s wr, 546 op/s 2024-09-15T23:18:59.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 160.0 s, 1732.2 tps, lat 58.781 ms stddev 64.826 2024-09-15T23:19:00.893 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:19:00.893 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:19:01.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:00 smithi062 ceph-mon[30949]: pgmap v452: 129 pgs: 129 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 7.7 MiB/s wr, 687 op/s 2024-09-15T23:19:01.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:00 smithi067 ceph-mon[35665]: pgmap v452: 129 pgs: 129 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 7.7 MiB/s wr, 687 op/s 2024-09-15T23:19:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:00 smithi148 ceph-mon[35610]: pgmap v452: 129 pgs: 129 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 7.7 MiB/s wr, 687 op/s 2024-09-15T23:19:01.294 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:03.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:02 smithi062 ceph-mon[30949]: pgmap v453: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.8 MiB/s wr, 616 op/s 2024-09-15T23:19:03.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:02 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:19:03.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:02 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:03.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:02 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:19:03.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:02 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:19:03.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:02 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:03.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:02 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:19:03.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:02 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:02 smithi067 ceph-mon[35665]: pgmap v453: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.8 MiB/s wr, 616 op/s 2024-09-15T23:19:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:02 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:19:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:02 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:02 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:19:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:02 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:19:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:02 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:03.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:02 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:19:03.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:02 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:02 smithi148 ceph-mon[35610]: pgmap v453: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.8 MiB/s wr, 616 op/s 2024-09-15T23:19:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:02 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:19:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:02 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:02 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:19:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:02 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:19:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:02 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:02 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:19:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:02 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:03.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 -- 172.21.15.62:0/3457796652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 msgr2=0x7f91e4107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:03.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/3457796652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 0x7f91e4107b90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009930 tx=0x7f91d802a420 comp rx=0 tx=0).stop 2024-09-15T23:19:03.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 -- 172.21.15.62:0/3457796652 shutdown_connections 2024-09-15T23:19:03.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/3457796652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e41082a0 0x7f91e410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/3457796652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 0x7f91e4107b90 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/3457796652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91e4102d30 0x7f91e4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 -- 172.21.15.62:0/3457796652 >> 172.21.15.62:0/3457796652 conn(0x7f91e40fc7d0 msgr2=0x7f91e40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:03.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 -- 172.21.15.62:0/3457796652 shutdown_connections 2024-09-15T23:19:03.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 -- 172.21.15.62:0/3457796652 wait complete. 2024-09-15T23:19:03.298 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:19:03.298 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:19:03.298 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "6215605b-49ac-4df5-b9f7-ca591acc4e39", 2024-09-15T23:19:03.298 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:19:03.298 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:19:03.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.232+0000 7f91ea1a4640 1 Processor -- start 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91ea1a4640 1 -- start start 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91ea1a4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e4102d30 0x7f91e4100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91ea1a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 0x7f91e4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91ea1a4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91e41082a0 0x7f91e4101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91ea1a4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91e41120a0 con 0x7f91e41036e0 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91ea1a4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4111f20 con 0x7f91e4102d30 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91ea1a4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91e4112220 con 0x7f91e41082a0 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 0x7f91e4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 0x7f91e4100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54140/0 (socket says 172.21.15.62:54140) 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e2ffd640 1 -- 172.21.15.62:0/637686455 learned_addr learned my addr 172.21.15.62:0/637686455 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e37fe640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e4102d30 0x7f91e4100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e3fff640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91e41082a0 0x7f91e4101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e2ffd640 1 -- 172.21.15.62:0/637686455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91e41082a0 msgr2=0x7f91e4101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91e41082a0 0x7f91e4101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e2ffd640 1 -- 172.21.15.62:0/637686455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e4102d30 msgr2=0x7f91e4100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e4102d30 0x7f91e4100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e2ffd640 1 -- 172.21.15.62:0/637686455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e41a6aa0 con 0x7f91e41036e0 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.233+0000 7f91e37fe640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e4102d30 0x7f91e4100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.234+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 0x7f91e4100ed0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f91d8005940 tx=0x7f91d80040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.234+0000 7f91e0ff9640 1 -- 172.21.15.62:0/637686455 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d803f030 con 0x7f91e41036e0 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.234+0000 7f91e0ff9640 1 -- 172.21.15.62:0/637686455 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d80044c0 con 0x7f91e41036e0 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.234+0000 7f91ea1a4640 1 -- 172.21.15.62:0/637686455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e41a6d30 con 0x7f91e41036e0 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.234+0000 7f91e0ff9640 1 -- 172.21.15.62:0/637686455 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8035070 con 0x7f91e41036e0 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.234+0000 7f91ea1a4640 1 -- 172.21.15.62:0/637686455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e41a7200 con 0x7f91e41036e0 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.236+0000 7f91e0ff9640 1 -- 172.21.15.62:0/637686455 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f91d8042030 con 0x7f91e41036e0 2024-09-15T23:19:03.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91ec57d740 1 Processor -- start 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91ec57d740 1 -- start start 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d331b60 0x56096d331f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d332480 0x56096d33c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d33cd40 0x56096d33f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91ec57d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56096d297280 con 0x56096d331b60 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91ec57d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56096d23eb10 con 0x56096d33cd40 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91ec57d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56096d293240 con 0x56096d332480 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e0ff9640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91cc0a32c0 0x7f91cc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d331b60 0x56096d331f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d331b60 0x56096d331f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54154/0 (socket says 172.21.15.62:54154) 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e3fff640 1 -- 172.21.15.62:0/187023607 learned_addr learned my addr 172.21.15.62:0/187023607 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/187023607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d33cd40 0x56096d33f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e0ff9640 1 -- 172.21.15.62:0/637686455 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f91d80eff50 con 0x7f91e41036e0 2024-09-15T23:19:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e3fff640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91cc0a32c0 0x7f91cc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e37fe640 1 --2- 172.21.15.62:0/187023607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d332480 0x56096d33c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e3fff640 1 -- 172.21.15.62:0/187023607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d332480 msgr2=0x56096d33c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e3fff640 1 --2- 172.21.15.62:0/187023607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d332480 0x56096d33c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e3fff640 1 -- 172.21.15.62:0/187023607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d33cd40 msgr2=0x56096d33f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e3fff640 1 --2- 172.21.15.62:0/187023607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d33cd40 0x56096d33f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.237+0000 7f91e3fff640 1 -- 172.21.15.62:0/187023607 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56096d24b1f0 con 0x56096d331b60 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.238+0000 7f91e3fff640 1 --2- 172.21.15.62:0/187023607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d331b60 0x56096d331f40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f91dc005c40 tx=0x7f91dc013f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f312229f0af92a6d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.238+0000 7f91c9ffb640 1 -- 172.21.15.62:0/187023607 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0134e0 con 0x56096d331b60 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.238+0000 7f91e3fff640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91cc0a32c0 0x7f91cc0a5780 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00f850 tx=0x7f91dc01f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.238+0000 7f91c9ffb640 1 -- 172.21.15.62:0/187023607 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91dc022020 con 0x56096d331b60 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.238+0000 7f91c9ffb640 1 -- 172.21.15.62:0/187023607 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc010690 con 0x56096d331b60 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 -- 172.21.15.62:0/187023607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d331b60 msgr2=0x56096d331f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 --2- 172.21.15.62:0/187023607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d331b60 0x56096d331f40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f91dc005c40 tx=0x7f91dc013f50 comp rx=0 tx=0).stop 2024-09-15T23:19:03.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 -- 172.21.15.62:0/187023607 shutdown_connections 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 --2- 172.21.15.62:0/187023607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d33cd40 0x56096d33f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 --2- 172.21.15.62:0/187023607 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d332480 0x56096d33c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 --2- 172.21.15.62:0/187023607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d331b60 0x56096d331f40 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 -- 172.21.15.62:0/187023607 >> 172.21.15.62:0/187023607 conn(0x56096d32e0f0 msgr2=0x56096d32e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 -- 172.21.15.62:0/187023607 shutdown_connections 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 -- 172.21.15.62:0/187023607 wait complete. 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 -- start start 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.239+0000 7f91ec57d740 1 Processor -- start 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 0x56096d339be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d33a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 0x56096d33a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91ec57d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56096d33f910 con 0x56096d332480 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91ec57d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56096d251f10 con 0x56096d331b60 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91ec57d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56096d33fa90 con 0x56096d33cd40 2024-09-15T23:19:03.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91ec57d740 10 client.0 _ll_register_callbacks cb 0x56096d21aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d33a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 0x56096d339be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d33a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54168/0 (socket says 172.21.15.62:54168) 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e2ffd640 1 -- 172.21.15.62:0/53395841 learned_addr learned my addr 172.21.15.62:0/53395841 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 0x56096d33a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e2ffd640 1 -- 172.21.15.62:0/53395841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 msgr2=0x56096d33a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 0x56096d33a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e2ffd640 1 -- 172.21.15.62:0/53395841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 msgr2=0x56096d339be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 0x56096d339be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e2ffd640 1 -- 172.21.15.62:0/53395841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56096d0086a0 con 0x56096d332480 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e37fe640 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 0x56096d339be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d33a120 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009900 tx=0x7f91d80bbde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba4ff416c50ab71 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91c2ffd640 1 -- 172.21.15.62:0/53395841 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8049040 con 0x56096d332480 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91c2ffd640 1 -- 172.21.15.62:0/53395841 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f91d80b8910 con 0x56096d332480 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.240+0000 7f91ec57d740 1 -- 172.21.15.62:0/53395841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91dc022020 con 0x56096d332480 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91c2ffd640 1 -- 172.21.15.62:0/53395841 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f91d80c05a0 con 0x56096d332480 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91ec57d740 10 client.16179 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91ec57d740 1 -- 172.21.15.62:0/53395841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56096d34b4e0 con 0x56096d332480 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91c2ffd640 1 -- 172.21.15.62:0/53395841 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91dc022020 con 0x56096d332480 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91c2ffd640 1 -- 172.21.15.62:0/53395841 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f91d80c0130 con 0x56096d332480 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91ec57d740 10 client.16179 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91ec57d740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91ec57d740 20 client.16179 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91ec57d740 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56096d34c830 0x56096d34ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91ec57d740 4 client.16179 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91ec57d740 1 -- 172.21.15.62:0/53395841 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56096d2a47f0 con 0x56096d34c830 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91e37fe640 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56096d34c830 0x56096d34ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91c2ffd640 10 client.16179 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.241+0000 7f91e37fe640 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56096d34c830 0x56096d34ec80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f84d1031b5941549 server_cookie=2c0b32cbf63bc372 in_seq=0 out_seq=0 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91c2ffd640 1 -- 172.21.15.62:0/53395841 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56096d2a47f0 con 0x56096d34c830 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91c2ffd640 10 client.16179 handle_command_reply: tid=0 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91ec57d740 1 client.16179 shutdown 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91ec57d740 1 -- 172.21.15.62:0/53395841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 msgr2=0x56096d33a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91ec57d740 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d33a120 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f91d8009900 tx=0x7f91d80bbde0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91ec57d740 1 -- 172.21.15.62:0/53395841 shutdown_connections 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91ec57d740 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56096d34c830 0x56096d34ec80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91ec57d740 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 0x56096d33a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91ec57d740 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d33a120 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91ec57d740 1 --2- 172.21.15.62:0/53395841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 0x56096d339be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91c2ffd640 0 client.16179 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.242+0000 7f91ec57d740 1 -- 172.21.15.62:0/53395841 >> 172.21.15.62:0/53395841 conn(0x56096d32e0f0 msgr2=0x56096d32d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.243+0000 7f91ec57d740 1 -- 172.21.15.62:0/53395841 shutdown_connections 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.243+0000 7f91ec57d740 1 -- 172.21.15.62:0/53395841 wait complete. 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.243+0000 7f91ec57d740 20 client.16179 trim_cache size 0 max 16384 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.278+0000 7f91ec57d740 1 Processor -- start 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.278+0000 7f91ec57d740 1 -- start start 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.278+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d331b60 0x56096d3345c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.278+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d332480 0x56096d354590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.279+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d33cd40 0x56096d354ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.279+0000 7f91e37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d331b60 0x56096d3345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.279+0000 7f91e37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d331b60 0x56096d3345c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52570/0 (socket says 172.21.15.62:52570) 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.279+0000 7f91e37fe640 1 -- 172.21.15.62:0/1808679171 learned_addr learned my addr 172.21.15.62:0/1808679171 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.279+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/1808679171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d332480 0x56096d354590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.279+0000 7f91ec57d740 1 -- 172.21.15.62:0/1808679171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91e41b3740 con 0x56096d33cd40 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.279+0000 7f91ec57d740 1 -- 172.21.15.62:0/1808679171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91e41b0fe0 con 0x56096d332480 2024-09-15T23:19:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.279+0000 7f91ec57d740 1 -- 172.21.15.62:0/1808679171 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91e41aee30 con 0x56096d331b60 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.279+0000 7f91e3fff640 1 --2- 172.21.15.62:0/1808679171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d33cd40 0x56096d354ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.280+0000 7f91e3fff640 1 -- 172.21.15.62:0/1808679171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d331b60 msgr2=0x56096d3345c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.280+0000 7f91e3fff640 1 --2- 172.21.15.62:0/1808679171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d331b60 0x56096d3345c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.280+0000 7f91e3fff640 1 -- 172.21.15.62:0/1808679171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d332480 msgr2=0x56096d354590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.280+0000 7f91e3fff640 1 --2- 172.21.15.62:0/1808679171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d332480 0x56096d354590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.280+0000 7f91e3fff640 1 -- 172.21.15.62:0/1808679171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56096d34f300 con 0x56096d33cd40 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.281+0000 7f91e3fff640 1 --2- 172.21.15.62:0/1808679171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d33cd40 0x56096d354ad0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009c20 tx=0x7f91dc005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a84a901ccd6a67f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.281+0000 7f91c97fa640 1 -- 172.21.15.62:0/1808679171 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc034070 con 0x56096d33cd40 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.281+0000 7f91c97fa640 1 -- 172.21.15.62:0/1808679171 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91dc022070 con 0x56096d33cd40 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.281+0000 7f91c97fa640 1 -- 172.21.15.62:0/1808679171 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc010900 con 0x56096d33cd40 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.282+0000 7f91ec57d740 1 -- 172.21.15.62:0/1808679171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d33cd40 msgr2=0x56096d354ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.282+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1808679171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d33cd40 0x56096d354ad0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009c20 tx=0x7f91dc005f70 comp rx=0 tx=0).stop 2024-09-15T23:19:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.283+0000 7f91ec57d740 1 -- 172.21.15.62:0/1808679171 shutdown_connections 2024-09-15T23:19:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.283+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1808679171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d33cd40 0x56096d354ad0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.283+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1808679171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d332480 0x56096d354590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.283+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1808679171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d331b60 0x56096d3345c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.283+0000 7f91ec57d740 1 -- 172.21.15.62:0/1808679171 >> 172.21.15.62:0/1808679171 conn(0x56096d32e0f0 msgr2=0x56096d338cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.285+0000 7f91ec57d740 1 -- 172.21.15.62:0/1808679171 shutdown_connections 2024-09-15T23:19:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.285+0000 7f91ec57d740 1 -- 172.21.15.62:0/1808679171 wait complete. 2024-09-15T23:19:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.286+0000 7f91ec57d740 1 -- start start 2024-09-15T23:19:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.286+0000 7f91ec57d740 1 Processor -- start 2024-09-15T23:19:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.286+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 0x56096d340a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.287+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d340f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.287+0000 7f91ec57d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 0x56096d3414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.287+0000 7f91ec57d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91e41af4a0 con 0x56096d332480 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.287+0000 7f91ec57d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91e41b0910 con 0x56096d331b60 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.287+0000 7f91ec57d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91e41afb70 con 0x56096d33cd40 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.287+0000 7f91ec57d740 10 client.0 _ll_register_callbacks cb 0x7f91d8005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.287+0000 7f91e2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d340f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.287+0000 7f91e2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d340f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54184/0 (socket says 172.21.15.62:54184) 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.287+0000 7f91e2ffd640 1 -- 172.21.15.62:0/1992932281 learned_addr learned my addr 172.21.15.62:0/1992932281 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.288+0000 7f91e37fe640 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 0x56096d340a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.288+0000 7f91e3fff640 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 0x56096d3414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.288+0000 7f91e2ffd640 1 -- 172.21.15.62:0/1992932281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 msgr2=0x56096d3414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.288+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 0x56096d3414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.288+0000 7f91e2ffd640 1 -- 172.21.15.62:0/1992932281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 msgr2=0x56096d340a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.288+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 0x56096d340a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.288+0000 7f91e2ffd640 1 -- 172.21.15.62:0/1992932281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56096d3bed70 con 0x56096d332480 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.288+0000 7f91e2ffd640 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d340f70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f91d80ba550 tx=0x7f91d8031c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3df325bf54d0c69 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.288+0000 7f91c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.289+0000 7f91c2ffd640 1 -- 172.21.15.62:0/1992932281 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8049070 con 0x56096d332480 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.289+0000 7f91c2ffd640 1 -- 172.21.15.62:0/1992932281 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f91d80c0130 con 0x56096d332480 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.289+0000 7f91ec57d740 1 -- 172.21.15.62:0/1992932281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91dc022070 con 0x56096d332480 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.289+0000 7f91c2ffd640 1 -- 172.21.15.62:0/1992932281 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f91d80bc070 con 0x56096d332480 2024-09-15T23:19:03.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.291+0000 7f91ec57d740 10 client.16185 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.291+0000 7f91ec57d740 1 -- 172.21.15.62:0/1992932281 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56096d3c0090 con 0x56096d332480 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.291+0000 7f91c2ffd640 1 -- 172.21.15.62:0/1992932281 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91d80dc2c0 con 0x56096d332480 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.291+0000 7f91c2ffd640 1 -- 172.21.15.62:0/1992932281 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f91d80bb6f0 con 0x56096d332480 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.291+0000 7f91ec57d740 10 client.16185 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.291+0000 7f91ec57d740 10 client.16185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.291+0000 7f91ec57d740 20 client.16185 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.291+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56096d3c0500 0x56096d3c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.292+0000 7f91ec57d740 4 client.16185 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.292+0000 7f91ec57d740 1 -- 172.21.15.62:0/1992932281 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f91dc022070 con 0x56096d3c0500 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.292+0000 7f91e37fe640 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56096d3c0500 0x56096d3c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.292+0000 7f91c2ffd640 10 client.16185 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.292+0000 7f91e37fe640 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56096d3c0500 0x56096d3c2920 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=947f73e3a5274e20 server_cookie=6d7265582cef0bf2 in_seq=0 out_seq=0 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.293+0000 7f91c2ffd640 1 -- 172.21.15.62:0/1992932281 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f91dc022070 con 0x56096d3c0500 2024-09-15T23:19:03.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.293+0000 7f91c2ffd640 10 client.16185 handle_command_reply: tid=0 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.293+0000 7f91ec57d740 1 client.16185 shutdown 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.293+0000 7f91ec57d740 1 -- 172.21.15.62:0/1992932281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 msgr2=0x56096d340f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.293+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d340f70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f91d80ba550 tx=0x7f91d8031c20 comp rx=0 tx=0).stop 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.294+0000 7f91ec57d740 1 -- 172.21.15.62:0/1992932281 shutdown_connections 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.294+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56096d3c0500 0x56096d3c2920 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.294+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56096d33cd40 0x56096d3414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.294+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56096d332480 0x56096d340f70 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.294+0000 7f91ec57d740 1 --2- 172.21.15.62:0/1992932281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56096d331b60 0x56096d340a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.294+0000 7f91ec57d740 1 -- 172.21.15.62:0/1992932281 >> 172.21.15.62:0/1992932281 conn(0x56096d32e0f0 msgr2=0x56096d32ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.294+0000 7f91c2ffd640 0 client.16185 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.296+0000 7f91ec57d740 1 -- 172.21.15.62:0/1992932281 shutdown_connections 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.296+0000 7f91ec57d740 1 -- 172.21.15.62:0/1992932281 wait complete. 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.296+0000 7f91ec57d740 20 client.16185 trim_cache size 0 max 16384 2024-09-15T23:19:03.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 -- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91cc0a32c0 msgr2=0x7f91cc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91cc0a32c0 0x7f91cc0a5780 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f91dc00f850 tx=0x7f91dc01f000 comp rx=0 tx=0).stop 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 -- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 msgr2=0x7f91e4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 0x7f91e4100ed0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f91d8005940 tx=0x7f91d80040c0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 -- 172.21.15.62:0/637686455 shutdown_connections 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91cc0a32c0 0x7f91cc0a5780 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91e41082a0 0x7f91e4101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91e41036e0 0x7f91e4100ed0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 --2- 172.21.15.62:0/637686455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91e4102d30 0x7f91e4100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 -- 172.21.15.62:0/637686455 >> 172.21.15.62:0/637686455 conn(0x7f91e40fc7d0 msgr2=0x7f91e4105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 -- 172.21.15.62:0/637686455 shutdown_connections 2024-09-15T23:19:03.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:02.298+0000 7f91ea1a4640 1 -- 172.21.15.62:0/637686455 wait complete. 2024-09-15T23:19:04.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:03 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:19:04.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:03 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:04.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:03 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:03 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:19:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:03 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:03 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:04.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:03 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:19:04.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:03 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:04.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:03 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 165.0 s, 1741.4 tps, lat 56.872 ms stddev 65.694 2024-09-15T23:19:05.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:04 smithi062 ceph-mon[30949]: pgmap v454: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.9 MiB/s wr, 688 op/s 2024-09-15T23:19:05.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:04 smithi067 ceph-mon[35665]: pgmap v454: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.9 MiB/s wr, 688 op/s 2024-09-15T23:19:05.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:04 smithi148 ceph-mon[35610]: pgmap v454: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 6.9 MiB/s wr, 688 op/s 2024-09-15T23:19:07.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:06 smithi062 ceph-mon[30949]: pgmap v455: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 5.5 MiB/s wr, 665 op/s 2024-09-15T23:19:07.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:06 smithi067 ceph-mon[35665]: pgmap v455: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 5.5 MiB/s wr, 665 op/s 2024-09-15T23:19:07.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:06 smithi148 ceph-mon[35610]: pgmap v455: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 5.5 MiB/s wr, 665 op/s 2024-09-15T23:19:07.508 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:19:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:08.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.527+0000 7f74b6f2a640 1 -- 172.21.15.62:0/3153498905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b0102190 msgr2=0x7f74b01056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:08.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.527+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/3153498905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b0102190 0x7f74b01056d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009900 tx=0x7f74a002a440 comp rx=0 tx=0).stop 2024-09-15T23:19:08.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.527+0000 7f74b6f2a640 1 -- 172.21.15.62:0/3153498905 shutdown_connections 2024-09-15T23:19:08.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.527+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/3153498905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74b0105e00 0x7f74b010d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.527+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/3153498905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b0102190 0x7f74b01056d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.527+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/3153498905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74b01017e0 0x7f74b0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.527+0000 7f74b6f2a640 1 -- 172.21.15.62:0/3153498905 >> 172.21.15.62:0/3153498905 conn(0x7f74b00fba90 msgr2=0x7f74b00fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:08.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.529+0000 7f74b6f2a640 1 -- 172.21.15.62:0/3153498905 shutdown_connections 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.529+0000 7f74b6f2a640 1 -- 172.21.15.62:0/3153498905 wait complete. 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b6f2a640 1 Processor -- start 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b6f2a640 1 -- start start 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b6f2a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74b01017e0 0x7f74b01963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b6f2a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b0102190 0x7f74b01968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b6f2a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74b0105e00 0x7f74b01a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b6f2a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0110ed0 con 0x7f74b01017e0 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b6f2a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0110d50 con 0x7f74b0102190 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b6f2a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0111050 con 0x7f74b0105e00 2024-09-15T23:19:08.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b4cc6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74b01017e0 0x7f74b01963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b4cc6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74b01017e0 0x7f74b01963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48380/0 (socket says 172.21.15.62:48380) 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.530+0000 7f74b4cc6640 1 -- 172.21.15.62:0/731023511 learned_addr learned my addr 172.21.15.62:0/731023511 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.531+0000 7f74affff640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b0102190 0x7f74b01968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.532+0000 7f74b4cc6640 1 -- 172.21.15.62:0/731023511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74b0105e00 msgr2=0x7f74b01a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.531+0000 7f74b54c7640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74b0105e00 0x7f74b01a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.532+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74b0105e00 0x7f74b01a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.532+0000 7f74b4cc6640 1 -- 172.21.15.62:0/731023511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b0102190 msgr2=0x7f74b01968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.532+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b0102190 0x7f74b01968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.532+0000 7f74b4cc6640 1 -- 172.21.15.62:0/731023511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b01a2680 con 0x7f74b01017e0 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.532+0000 7f74affff640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b0102190 0x7f74b01968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:19:08.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.532+0000 7f74b54c7640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74b0105e00 0x7f74b01a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:19:08.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.532+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74b01017e0 0x7f74b01963a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f74a800db80 tx=0x7f74a800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:08.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.535+0000 7f74adffb640 1 -- 172.21.15.62:0/731023511 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a8017070 con 0x7f74b01017e0 2024-09-15T23:19:08.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.535+0000 7f74adffb640 1 -- 172.21.15.62:0/731023511 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74a8003b70 con 0x7f74b01017e0 2024-09-15T23:19:08.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.535+0000 7f74adffb640 1 -- 172.21.15.62:0/731023511 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a80106c0 con 0x7f74b01017e0 2024-09-15T23:19:08.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.535+0000 7f74b6f2a640 1 -- 172.21.15.62:0/731023511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b01a2970 con 0x7f74b01017e0 2024-09-15T23:19:08.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.536+0000 7f74b6f2a640 1 -- 172.21.15.62:0/731023511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b01a2e60 con 0x7f74b01017e0 2024-09-15T23:19:08.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.541+0000 7f74adffb640 1 -- 172.21.15.62:0/731023511 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f74a8004020 con 0x7f74b01017e0 2024-09-15T23:19:08.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.543+0000 7f74adffb640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f749c0a3450 0x7f749c0a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.543+0000 7f74adffb640 1 -- 172.21.15.62:0/731023511 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f74a80cfa90 con 0x7f74b01017e0 2024-09-15T23:19:08.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.546+0000 7f74b9303740 1 Processor -- start 2024-09-15T23:19:08.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.547+0000 7f74affff640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f749c0a3450 0x7f749c0a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.548+0000 7f74affff640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f749c0a3450 0x7f749c0a5910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f74a0000c00 tx=0x7f74a00041a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:08.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.548+0000 7f74b9303740 1 -- start start 2024-09-15T23:19:08.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.548+0000 7f74b9303740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77d54a0 0x55cbf77d5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.548+0000 7f74b9303740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77d5dc0 0x55cbf77e0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.548+0000 7f74b9303740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77e0680 0x55cbf77e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.548+0000 7f74b9303740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55cbf773b2e0 con 0x55cbf77e0680 2024-09-15T23:19:08.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.549+0000 7f74b9303740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55cbf76e2ce0 con 0x55cbf77d5dc0 2024-09-15T23:19:08.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.549+0000 7f74b9303740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55cbf77372a0 con 0x55cbf77d54a0 2024-09-15T23:19:08.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.549+0000 7f74b54c7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77e0680 0x55cbf77e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.549+0000 7f74b54c7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77e0680 0x55cbf77e2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48390/0 (socket says 172.21.15.62:48390) 2024-09-15T23:19:08.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.549+0000 7f74b54c7640 1 -- 172.21.15.62:0/3036864438 learned_addr learned my addr 172.21.15.62:0/3036864438 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:08.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.549+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/3036864438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77d54a0 0x55cbf77d5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.550+0000 7f74b54c7640 1 -- 172.21.15.62:0/3036864438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77d54a0 msgr2=0x55cbf77d5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:08.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.550+0000 7f74b54c7640 1 --2- 172.21.15.62:0/3036864438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77d54a0 0x55cbf77d5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.550+0000 7f74b54c7640 1 -- 172.21.15.62:0/3036864438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77d5dc0 msgr2=0x55cbf77e0140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:19:08.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.550+0000 7f74b54c7640 1 --2- 172.21.15.62:0/3036864438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77d5dc0 0x55cbf77e0140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.550+0000 7f74b54c7640 1 -- 172.21.15.62:0/3036864438 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf76c7440 con 0x55cbf77e0680 2024-09-15T23:19:08.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.550+0000 7f74b54c7640 1 --2- 172.21.15.62:0/3036864438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77e0680 0x55cbf77e2a70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f749800a940 tx=0x7f749800ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8828adec2d2b37 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:08.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.551+0000 7f74a6ffd640 1 -- 172.21.15.62:0/3036864438 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498015070 con 0x55cbf77e0680 2024-09-15T23:19:08.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.551+0000 7f74a6ffd640 1 -- 172.21.15.62:0/3036864438 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749800cc00 con 0x55cbf77e0680 2024-09-15T23:19:08.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.551+0000 7f74a6ffd640 1 -- 172.21.15.62:0/3036864438 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74980186d0 con 0x55cbf77e0680 2024-09-15T23:19:08.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.552+0000 7f74b9303740 1 -- 172.21.15.62:0/3036864438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77e0680 msgr2=0x55cbf77e2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:08.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.552+0000 7f74b9303740 1 --2- 172.21.15.62:0/3036864438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77e0680 0x55cbf77e2a70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f749800a940 tx=0x7f749800ae00 comp rx=0 tx=0).stop 2024-09-15T23:19:08.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.553+0000 7f74b9303740 1 -- 172.21.15.62:0/3036864438 shutdown_connections 2024-09-15T23:19:08.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.553+0000 7f74b9303740 1 --2- 172.21.15.62:0/3036864438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77e0680 0x55cbf77e2a70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.553+0000 7f74b9303740 1 --2- 172.21.15.62:0/3036864438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77d5dc0 0x55cbf77e0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.553+0000 7f74b9303740 1 --2- 172.21.15.62:0/3036864438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77d54a0 0x55cbf77d5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.553+0000 7f74b9303740 1 -- 172.21.15.62:0/3036864438 >> 172.21.15.62:0/3036864438 conn(0x55cbf77d1a30 msgr2=0x55cbf77d1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:08.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.553+0000 7f74b9303740 1 -- 172.21.15.62:0/3036864438 shutdown_connections 2024-09-15T23:19:08.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.554+0000 7f74b9303740 1 -- 172.21.15.62:0/3036864438 wait complete. 2024-09-15T23:19:08.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.554+0000 7f74b9303740 1 -- start start 2024-09-15T23:19:08.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.554+0000 7f74b9303740 1 Processor -- start 2024-09-15T23:19:08.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.555+0000 7f74b9303740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77e00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.555+0000 7f74b9303740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77dd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.555+0000 7f74b9303740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 0x55cbf77fd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.555+0000 7f74b9303740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55cbf77f4260 con 0x55cbf77d54a0 2024-09-15T23:19:08.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.555+0000 7f74b9303740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55cbf76f6050 con 0x55cbf77e0680 2024-09-15T23:19:08.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.555+0000 7f74b9303740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55cbf77f43e0 con 0x55cbf77dda80 2024-09-15T23:19:08.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.555+0000 7f74b9303740 10 client.0 _ll_register_callbacks cb 0x55cbf7736160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:08.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.556+0000 7f74b54c7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 0x55cbf77fd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.557+0000 7f74affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77dd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.557+0000 7f74b54c7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 0x55cbf77fd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:57478/0 (socket says 172.21.15.62:57478) 2024-09-15T23:19:08.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.557+0000 7f74b54c7640 1 -- 172.21.15.62:0/4017539355 learned_addr learned my addr 172.21.15.62:0/4017539355 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:08.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.557+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77e00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.557+0000 7f74b54c7640 1 -- 172.21.15.62:0/4017539355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 msgr2=0x55cbf77dd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:08.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.557+0000 7f74b54c7640 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77dd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.557+0000 7f74b54c7640 1 -- 172.21.15.62:0/4017539355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 msgr2=0x55cbf77e00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:08.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.557+0000 7f74b54c7640 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77e00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.557+0000 7f74b54c7640 1 -- 172.21.15.62:0/4017539355 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf74ac3c0 con 0x55cbf77dda80 2024-09-15T23:19:08.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.558+0000 7f74b54c7640 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 0x55cbf77fd6d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7498008010 tx=0x7f7498011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5ab16044fbb519 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:08.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.560+0000 7f748ffff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:19:08.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.560+0000 7f748ffff640 1 -- 172.21.15.62:0/4017539355 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498029060 con 0x55cbf77dda80 2024-09-15T23:19:08.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.560+0000 7f748ffff640 1 -- 172.21.15.62:0/4017539355 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f749801a450 con 0x55cbf77dda80 2024-09-15T23:19:08.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.560+0000 7f74b9303740 1 -- 172.21.15.62:0/4017539355 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f749800cc00 con 0x55cbf77dda80 2024-09-15T23:19:08.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.564+0000 7f748ffff640 1 -- 172.21.15.62:0/4017539355 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7498015040 con 0x55cbf77dda80 2024-09-15T23:19:08.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.565+0000 7f74b9303740 10 client.25666 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:08.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.565+0000 7f74b9303740 1 -- 172.21.15.62:0/4017539355 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cbf7800120 con 0x55cbf77dda80 2024-09-15T23:19:08.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.565+0000 7f748ffff640 1 -- 172.21.15.62:0/4017539355 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f749800cc00 con 0x55cbf77dda80 2024-09-15T23:19:08.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.565+0000 7f748ffff640 1 -- 172.21.15.62:0/4017539355 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f74980183d0 con 0x55cbf77dda80 2024-09-15T23:19:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.566+0000 7f74b9303740 10 client.25666 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.566+0000 7f74b9303740 10 client.25666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.566+0000 7f74b9303740 20 client.25666 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.566+0000 7f74b9303740 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cbf7801800 0x55cbf7803c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.566+0000 7f74b9303740 4 client.25666 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.566+0000 7f74b9303740 1 -- 172.21.15.62:0/4017539355 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbf7748850 con 0x55cbf7801800 2024-09-15T23:19:08.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.567+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cbf7801800 0x55cbf7803c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.572+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cbf7801800 0x55cbf7803c40 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70314210a30d4598 server_cookie=9875110e7993c19e in_seq=0 out_seq=0 2024-09-15T23:19:08.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.572+0000 7f748ffff640 10 client.25666 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:08.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.573+0000 7f748ffff640 1 -- 172.21.15.62:0/4017539355 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cbf7748850 con 0x55cbf7801800 2024-09-15T23:19:08.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.573+0000 7f748ffff640 10 client.25666 handle_command_reply: tid=0 2024-09-15T23:19:08.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.573+0000 7f74b9303740 1 client.25666 shutdown 2024-09-15T23:19:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.574+0000 7f74b9303740 1 -- 172.21.15.62:0/4017539355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 msgr2=0x55cbf77fd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.574+0000 7f74b9303740 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 0x55cbf77fd6d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7498008010 tx=0x7f7498011b50 comp rx=0 tx=0).stop 2024-09-15T23:19:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.574+0000 7f74b9303740 1 -- 172.21.15.62:0/4017539355 shutdown_connections 2024-09-15T23:19:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.574+0000 7f74b9303740 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cbf7801800 0x55cbf7803c40 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.574+0000 7f74b9303740 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 0x55cbf77fd6d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.574+0000 7f74b9303740 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77dd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.574+0000 7f74b9303740 1 --2- 172.21.15.62:0/4017539355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77e00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.574+0000 7f74b9303740 1 -- 172.21.15.62:0/4017539355 >> 172.21.15.62:0/4017539355 conn(0x55cbf77d1a30 msgr2=0x55cbf77d7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.574+0000 7f748ffff640 0 client.25666 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:08.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.578+0000 7f74b9303740 1 -- 172.21.15.62:0/4017539355 shutdown_connections 2024-09-15T23:19:08.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.578+0000 7f74b9303740 1 -- 172.21.15.62:0/4017539355 wait complete. 2024-09-15T23:19:08.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.579+0000 7f74b9303740 20 client.25666 trim_cache size 0 max 16384 2024-09-15T23:19:08.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.611+0000 7f74b9303740 1 Processor -- start 2024-09-15T23:19:08.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.611+0000 7f74b9303740 1 -- start start 2024-09-15T23:19:08.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.612+0000 7f74b9303740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77d87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.612+0000 7f74b9303740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 0x55cbf77fd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.612+0000 7f74b9303740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77fda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.612+0000 7f74b9303740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01a6c90 con 0x55cbf77d54a0 2024-09-15T23:19:08.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.612+0000 7f74b9303740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01a4530 con 0x55cbf77e0680 2024-09-15T23:19:08.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.612+0000 7f74b9303740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0100090 con 0x55cbf77dda80 2024-09-15T23:19:08.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.612+0000 7f74b54c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77fda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.612+0000 7f74b54c7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77fda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42162/0 (socket says 172.21.15.62:42162) 2024-09-15T23:19:08.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.612+0000 7f74b54c7640 1 -- 172.21.15.62:0/1452377175 learned_addr learned my addr 172.21.15.62:0/1452377175 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:08.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.613+0000 7f74b54c7640 1 -- 172.21.15.62:0/1452377175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 msgr2=0x55cbf77fd4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.613+0000 7f74b54c7640 1 --2- 172.21.15.62:0/1452377175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 0x55cbf77fd4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.613+0000 7f74b54c7640 1 -- 172.21.15.62:0/1452377175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 msgr2=0x55cbf77d87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.613+0000 7f74b54c7640 1 --2- 172.21.15.62:0/1452377175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77d87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.613+0000 7f74b54c7640 1 -- 172.21.15.62:0/1452377175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf77d52d0 con 0x55cbf77e0680 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.615+0000 7f74b54c7640 1 --2- 172.21.15.62:0/1452377175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77fda00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f749800ae40 tx=0x7f7498011700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f9bb6b5b1830938 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.618+0000 7f74a67fc640 1 -- 172.21.15.62:0/1452377175 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498033080 con 0x55cbf77e0680 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.618+0000 7f74a67fc640 1 -- 172.21.15.62:0/1452377175 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7498011960 con 0x55cbf77e0680 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.619+0000 7f74a67fc640 1 -- 172.21.15.62:0/1452377175 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7498018360 con 0x55cbf77e0680 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.619+0000 7f74b9303740 1 -- 172.21.15.62:0/1452377175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 msgr2=0x55cbf77fda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.619+0000 7f74b9303740 1 --2- 172.21.15.62:0/1452377175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77fda00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f749800ae40 tx=0x7f7498011700 comp rx=0 tx=0).stop 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.619+0000 7f74b9303740 1 -- 172.21.15.62:0/1452377175 shutdown_connections 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.619+0000 7f74b9303740 1 --2- 172.21.15.62:0/1452377175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77e0680 0x55cbf77fda00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.619+0000 7f74b9303740 1 --2- 172.21.15.62:0/1452377175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77dda80 0x55cbf77fd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.619+0000 7f74b9303740 1 --2- 172.21.15.62:0/1452377175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77d87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.619+0000 7f74b9303740 1 -- 172.21.15.62:0/1452377175 >> 172.21.15.62:0/1452377175 conn(0x55cbf77d1a30 msgr2=0x55cbf77d80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.620+0000 7f74b9303740 1 -- 172.21.15.62:0/1452377175 shutdown_connections 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.620+0000 7f74b9303740 1 -- 172.21.15.62:0/1452377175 wait complete. 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.620+0000 7f74b9303740 1 -- start start 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.620+0000 7f74b9303740 1 Processor -- start 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.621+0000 7f74b9303740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77f45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.621+0000 7f74b9303740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77dda80 0x55cbf77f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.621+0000 7f74b9303740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77e0680 0x55cbf77f5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.621+0000 7f74b9303740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f74b0073d10 con 0x55cbf77d54a0 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.621+0000 7f74b9303740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01a3e60 con 0x55cbf77dda80 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.621+0000 7f74b9303740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f74b01a30c0 con 0x55cbf77e0680 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.621+0000 7f74b9303740 10 client.0 _ll_register_callbacks cb 0x7f749800a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:08.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.622+0000 7f74affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77dda80 0x55cbf77f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.622+0000 7f74affff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77dda80 0x55cbf77f4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42178/0 (socket says 172.21.15.62:42178) 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.622+0000 7f74affff640 1 -- 172.21.15.62:0/3669048694 learned_addr learned my addr 172.21.15.62:0/3669048694 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.623+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77f45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.623+0000 7f74affff640 1 -- 172.21.15.62:0/3669048694 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77e0680 msgr2=0x55cbf77f5070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.623+0000 7f74affff640 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77e0680 0x55cbf77f5070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.623+0000 7f74affff640 1 -- 172.21.15.62:0/3669048694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 msgr2=0x55cbf77f45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.623+0000 7f74affff640 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77f45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.623+0000 7f74affff640 1 -- 172.21.15.62:0/3669048694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf786a670 con 0x55cbf77dda80 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.623+0000 7f74affff640 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77dda80 0x55cbf77f4b30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009900 tx=0x7f74a0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7baa73b1f8a51b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.624+0000 7f748ffff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.624+0000 7f748ffff640 1 -- 172.21.15.62:0/3669048694 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a004d020 con 0x55cbf77dda80 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.624+0000 7f74b9303740 1 -- 172.21.15.62:0/3669048694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7498011960 con 0x55cbf77dda80 2024-09-15T23:19:08.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.624+0000 7f748ffff640 1 -- 172.21.15.62:0/3669048694 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f74a0004c90 con 0x55cbf77dda80 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.626+0000 7f748ffff640 1 -- 172.21.15.62:0/3669048694 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f74a0058050 con 0x55cbf77dda80 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.626+0000 7f74b9303740 10 client.25823 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.626+0000 7f74b9303740 1 -- 172.21.15.62:0/3669048694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55cbf786b990 con 0x55cbf77dda80 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.628+0000 7f748ffff640 1 -- 172.21.15.62:0/3669048694 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7498011960 con 0x55cbf77dda80 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.628+0000 7f748ffff640 1 -- 172.21.15.62:0/3669048694 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f74a003b770 con 0x55cbf77dda80 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.628+0000 7f74b9303740 10 client.25823 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.628+0000 7f74b9303740 10 client.25823 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.628+0000 7f74b9303740 20 client.25823 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.628+0000 7f74b9303740 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cbf786c1d0 0x55cbf786e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.628+0000 7f74b9303740 4 client.25823 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.628+0000 7f74b9303740 1 -- 172.21.15.62:0/3669048694 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cbf7742d60 con 0x55cbf786c1d0 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.629+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cbf786c1d0 0x55cbf786e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:08.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.631+0000 7f74b4cc6640 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cbf786c1d0 0x55cbf786e5f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2280b595d620a467 server_cookie=9c4224e30402037 in_seq=0 out_seq=0 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.631+0000 7f748ffff640 10 client.25823 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.632+0000 7f748ffff640 1 -- 172.21.15.62:0/3669048694 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cbf7742d60 con 0x55cbf786c1d0 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.632+0000 7f748ffff640 10 client.25823 handle_command_reply: tid=0 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.632+0000 7f74b9303740 1 client.25823 shutdown 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.633+0000 7f74b9303740 1 -- 172.21.15.62:0/3669048694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77dda80 msgr2=0x55cbf77f4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.633+0000 7f74b9303740 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77dda80 0x55cbf77f4b30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f74a0009900 tx=0x7f74a0009260 comp rx=0 tx=0).stop 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.633+0000 7f74b9303740 1 -- 172.21.15.62:0/3669048694 shutdown_connections 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.633+0000 7f74b9303740 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cbf786c1d0 0x55cbf786e5f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.633+0000 7f74b9303740 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cbf77e0680 0x55cbf77f5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.633+0000 7f74b9303740 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cbf77dda80 0x55cbf77f4b30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.633+0000 7f74b9303740 1 --2- 172.21.15.62:0/3669048694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cbf77d54a0 0x55cbf77f45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.633+0000 7f74b9303740 1 -- 172.21.15.62:0/3669048694 >> 172.21.15.62:0/3669048694 conn(0x55cbf77d1a30 msgr2=0x55cbf77df410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.633+0000 7f748ffff640 0 client.25823 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:08.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.634+0000 7f74b9303740 1 -- 172.21.15.62:0/3669048694 shutdown_connections 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.634+0000 7f74b9303740 1 -- 172.21.15.62:0/3669048694 wait complete. 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.634+0000 7f74b9303740 20 client.25823 trim_cache size 0 max 16384 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stdout: "6215605b-49ac-4df5-b9f7-ca591acc4e39": { 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "6215605b-49ac-4df5-b9f7-ca591acc4e39", 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:19:08.643 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:19:08.644 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:19:08.644 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 -- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f749c0a3450 msgr2=0x7f749c0a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f749c0a3450 0x7f749c0a5910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f74a0000c00 tx=0x7f74a00041a0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 -- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74b01017e0 msgr2=0x7f74b01963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74b01017e0 0x7f74b01963a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f74a800db80 tx=0x7f74a800b630 comp rx=0 tx=0).stop 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 -- 172.21.15.62:0/731023511 shutdown_connections 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f749c0a3450 0x7f749c0a5910 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f74b0105e00 0x7f74b01a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74b0102190 0x7f74b01968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 --2- 172.21.15.62:0/731023511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74b01017e0 0x7f74b01963a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 -- 172.21.15.62:0/731023511 >> 172.21.15.62:0/731023511 conn(0x7f74b00fba90 msgr2=0x7f74b01046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:08.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 -- 172.21.15.62:0/731023511 shutdown_connections 2024-09-15T23:19:08.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:08.636+0000 7f74b6f2a640 1 -- 172.21.15.62:0/731023511 wait complete. 2024-09-15T23:19:09.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:08 smithi062 ceph-mon[30949]: pgmap v456: 129 pgs: 129 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 5.2 MiB/s wr, 647 op/s 2024-09-15T23:19:09.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:08 smithi067 ceph-mon[35665]: pgmap v456: 129 pgs: 129 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 5.2 MiB/s wr, 647 op/s 2024-09-15T23:19:09.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:08 smithi148 ceph-mon[35610]: pgmap v456: 129 pgs: 129 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 5.2 MiB/s wr, 647 op/s 2024-09-15T23:19:09.452 INFO:tasks.cephfs.filesystem:scrub status for tag:6215605b-49ac-4df5-b9f7-ca591acc4e39 - {'path': '/', 'tag': '6215605b-49ac-4df5-b9f7-ca591acc4e39', 'options': 'recursive,force'} 2024-09-15T23:19:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 170.0 s, 1630.8 tps, lat 61.894 ms stddev 75.720 2024-09-15T23:19:10.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.331+0000 7ff161f6d640 1 -- 172.21.15.62:0/3760113871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 msgr2=0x7ff15c1020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.331+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3760113871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 0x7ff15c1020c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff150009860 tx=0x7ff15002a340 comp rx=0 tx=0).stop 2024-09-15T23:19:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.331+0000 7ff161f6d640 1 -- 172.21.15.62:0/3760113871 shutdown_connections 2024-09-15T23:19:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.331+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3760113871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff15c102600 0x7ff15c108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.331+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3760113871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 0x7ff15c1020c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.331+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3760113871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff15c100a50 0x7ff15c100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.331+0000 7ff161f6d640 1 -- 172.21.15.62:0/3760113871 >> 172.21.15.62:0/3760113871 conn(0x7ff15c0fc1b0 msgr2=0x7ff15c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.331+0000 7ff161f6d640 1 -- 172.21.15.62:0/3760113871 shutdown_connections 2024-09-15T23:19:10.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.331+0000 7ff161f6d640 1 -- 172.21.15.62:0/3760113871 wait complete. 2024-09-15T23:19:10.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.332+0000 7ff161f6d640 1 Processor -- start 2024-09-15T23:19:10.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.332+0000 7ff161f6d640 1 -- start start 2024-09-15T23:19:10.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.332+0000 7ff161f6d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff15c100a50 0x7ff15c104970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.332+0000 7ff161f6d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 0x7ff15c103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.332+0000 7ff161f6d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff15c102600 0x7ff15c103560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff161f6d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c10c700 con 0x7ff15c100a50 2024-09-15T23:19:10.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff161f6d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c10c580 con 0x7ff15c102600 2024-09-15T23:19:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff15affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 0x7ff15c103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff15affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 0x7ff15c103020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:57510/0 (socket says 172.21.15.62:57510) 2024-09-15T23:19:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff15affd640 1 -- 172.21.15.62:0/3645850314 learned_addr learned my addr 172.21.15.62:0/3645850314 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff161f6d640 1 -- 172.21.15.62:0/3645850314 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c10c880 con 0x7ff15c101c40 2024-09-15T23:19:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff15bfff640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff15c102600 0x7ff15c103560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff15affd640 1 -- 172.21.15.62:0/3645850314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff15c102600 msgr2=0x7ff15c103560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff15affd640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff15c102600 0x7ff15c103560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff15affd640 1 -- 172.21.15.62:0/3645850314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff15c100a50 msgr2=0x7ff15c104970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff15affd640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff15c100a50 0x7ff15c104970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.333+0000 7ff15affd640 1 -- 172.21.15.62:0/3645850314 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c103d80 con 0x7ff15c101c40 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.334+0000 7ff15affd640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 0x7ff15c103020 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff150031000 tx=0x7ff150033d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.334+0000 7ff158ff9640 1 -- 172.21.15.62:0/3645850314 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff150037040 con 0x7ff15c101c40 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.334+0000 7ff158ff9640 1 -- 172.21.15.62:0/3645850314 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff150030850 con 0x7ff15c101c40 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.334+0000 7ff158ff9640 1 -- 172.21.15.62:0/3645850314 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff150004920 con 0x7ff15c101c40 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.334+0000 7ff161f6d640 1 -- 172.21.15.62:0/3645850314 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15c1a1f10 con 0x7ff15c101c40 2024-09-15T23:19:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.334+0000 7ff161f6d640 1 -- 172.21.15.62:0/3645850314 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15c1a2320 con 0x7ff15c101c40 2024-09-15T23:19:10.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.338+0000 7ff158ff9640 1 -- 172.21.15.62:0/3645850314 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff150004ac0 con 0x7ff15c101c40 2024-09-15T23:19:10.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.339+0000 7ff158ff9640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1440a34b0 0x7ff1440a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.339+0000 7ff158ff9640 1 -- 172.21.15.62:0/3645850314 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff1500efaa0 con 0x7ff15c101c40 2024-09-15T23:19:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.339+0000 7ff15b7fe640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1440a34b0 0x7ff1440a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.339+0000 7ff164de5740 1 Processor -- start 2024-09-15T23:19:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.339+0000 7ff164de5740 1 -- start start 2024-09-15T23:19:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff164de5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689929a0 0x564f68992da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff164de5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689932e0 0x564f68997da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff164de5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6899a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff164de5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564f688f62f0 con 0x564f68998370 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff164de5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564f6889e270 con 0x564f689929a0 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff164de5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564f688f22b0 con 0x564f689932e0 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6899a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689932e0 0x564f68997da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6899a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48446/0 (socket says 172.21.15.62:48446) 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689932e0 0x564f68997da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:57522/0 (socket says 172.21.15.62:57522) 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15bfff640 1 -- 172.21.15.62:0/2065535119 learned_addr learned my addr 172.21.15.62:0/2065535119 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15b7fe640 1 --2- 172.21.15.62:0/2065535119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689929a0 0x564f68992da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15b7fe640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1440a34b0 0x7ff1440a5970 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff15c101aa0 tx=0x7ff14c009660 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15affd640 1 -- 172.21.15.62:0/2065535119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689929a0 msgr2=0x564f68992da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15affd640 1 --2- 172.21.15.62:0/2065535119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689929a0 0x564f68992da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15affd640 1 -- 172.21.15.62:0/2065535119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 msgr2=0x564f6899a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15affd640 1 --2- 172.21.15.62:0/2065535119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6899a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15affd640 1 -- 172.21.15.62:0/2065535119 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f6889d340 con 0x564f689932e0 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.340+0000 7ff15bfff640 1 --2- 172.21.15.62:0/2065535119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6899a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.341+0000 7ff15affd640 1 --2- 172.21.15.62:0/2065535119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689932e0 0x564f68997da0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff1500ed590 tx=0x7ff150033f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dcf29c1004231cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.341+0000 7ff141ffb640 1 -- 172.21.15.62:0/2065535119 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1500bc440 con 0x564f689932e0 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.341+0000 7ff141ffb640 1 -- 172.21.15.62:0/2065535119 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1500bbd40 con 0x564f689932e0 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.341+0000 7ff141ffb640 1 -- 172.21.15.62:0/2065535119 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1500b83b0 con 0x564f689932e0 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.342+0000 7ff164de5740 1 -- 172.21.15.62:0/2065535119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689932e0 msgr2=0x564f68997da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.342+0000 7ff164de5740 1 --2- 172.21.15.62:0/2065535119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689932e0 0x564f68997da0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff1500ed590 tx=0x7ff150033f90 comp rx=0 tx=0).stop 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.342+0000 7ff164de5740 1 -- 172.21.15.62:0/2065535119 shutdown_connections 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.342+0000 7ff164de5740 1 --2- 172.21.15.62:0/2065535119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6899a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.342+0000 7ff164de5740 1 --2- 172.21.15.62:0/2065535119 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689932e0 0x564f68997da0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.342+0000 7ff164de5740 1 --2- 172.21.15.62:0/2065535119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689929a0 0x564f68992da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.342+0000 7ff164de5740 1 -- 172.21.15.62:0/2065535119 >> 172.21.15.62:0/2065535119 conn(0x564f6898e010 msgr2=0x564f6898e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.342+0000 7ff164de5740 1 -- 172.21.15.62:0/2065535119 shutdown_connections 2024-09-15T23:19:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.342+0000 7ff164de5740 1 -- 172.21.15.62:0/2065535119 wait complete. 2024-09-15T23:19:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff164de5740 1 -- start start 2024-09-15T23:19:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff164de5740 1 Processor -- start 2024-09-15T23:19:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff164de5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff164de5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f689932e0 0x564f6899cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff164de5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f68998370 0x564f689a3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff164de5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564f6899b100 con 0x564f689932e0 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff164de5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564f6887f290 con 0x564f68998370 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff164de5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564f6899b280 con 0x564f689929a0 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff15b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.343+0000 7ff15affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f689932e0 0x564f6899cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff15bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f68998370 0x564f689a3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff15b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:57524/0 (socket says 172.21.15.62:57524) 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff15b7fe640 1 -- 172.21.15.62:0/1113849925 learned_addr learned my addr 172.21.15.62:0/1113849925 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff164de5740 10 client.0 _ll_register_callbacks cb 0x564f688b0140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff15b7fe640 1 -- 172.21.15.62:0/1113849925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f68998370 msgr2=0x564f689a3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff15b7fe640 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f68998370 0x564f689a3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff15b7fe640 1 -- 172.21.15.62:0/1113849925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f689932e0 msgr2=0x564f6899cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:10.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff15b7fe640 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f689932e0 0x564f6899cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff15b7fe640 1 -- 172.21.15.62:0/1113849925 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f68609210 con 0x564f689929a0 2024-09-15T23:19:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.344+0000 7ff15b7fe640 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899c9b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff14c01d780 tx=0x7ff14c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16acae63eb27edd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.345+0000 7ff13affd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:19:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.345+0000 7ff13affd640 1 -- 172.21.15.62:0/1113849925 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14c010bc0 con 0x564f689929a0 2024-09-15T23:19:10.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.345+0000 7ff13affd640 1 -- 172.21.15.62:0/1113849925 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff14c010d60 con 0x564f689929a0 2024-09-15T23:19:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.346+0000 7ff164de5740 1 -- 172.21.15.62:0/1113849925 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1500bbd40 con 0x564f689929a0 2024-09-15T23:19:10.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.346+0000 7ff13affd640 1 -- 172.21.15.62:0/1113849925 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff14c026440 con 0x564f689929a0 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.347+0000 7ff164de5740 10 client.25678 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.347+0000 7ff164de5740 1 -- 172.21.15.62:0/1113849925 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f686d5620 con 0x564f689929a0 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.347+0000 7ff13affd640 1 -- 172.21.15.62:0/1113849925 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1500bbd40 con 0x564f689929a0 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.347+0000 7ff13affd640 1 -- 172.21.15.62:0/1113849925 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7ff14c024070 con 0x564f689929a0 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.347+0000 7ff164de5740 10 client.25678 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.347+0000 7ff164de5740 10 client.25678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.348+0000 7ff164de5740 20 client.25678 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.348+0000 7ff164de5740 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f689a8610 0x564f689aaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.348+0000 7ff164de5740 4 client.25678 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.348+0000 7ff164de5740 1 -- 172.21.15.62:0/1113849925 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f68903a20 con 0x564f689a8610 2024-09-15T23:19:10.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.348+0000 7ff15affd640 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f689a8610 0x564f689aaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.349+0000 7ff15affd640 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f689a8610 0x564f689aaad0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c0ec938ed3f1661 server_cookie=9a5a06a72f700cca in_seq=0 out_seq=0 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.349+0000 7ff13affd640 10 client.25678 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff13affd640 1 -- 172.21.15.62:0/1113849925 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564f68903a20 con 0x564f689a8610 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff13affd640 10 client.25678 handle_command_reply: tid=0 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 client.25678 shutdown 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 -- 172.21.15.62:0/1113849925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 msgr2=0x564f6899c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899c9b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff14c01d780 tx=0x7ff14c0046c0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 -- 172.21.15.62:0/1113849925 shutdown_connections 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f689a8610 0x564f689aaad0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f68998370 0x564f689a3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f689932e0 0x564f6899cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 --2- 172.21.15.62:0/1113849925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899c9b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 -- 172.21.15.62:0/1113849925 >> 172.21.15.62:0/1113849925 conn(0x564f6898e010 msgr2=0x564f6898f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff13affd640 0 client.25678 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 -- 172.21.15.62:0/1113849925 shutdown_connections 2024-09-15T23:19:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.350+0000 7ff164de5740 1 -- 172.21.15.62:0/1113849925 wait complete. 2024-09-15T23:19:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.351+0000 7ff164de5740 20 client.25678 trim_cache size 0 max 16384 2024-09-15T23:19:10.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff164de5740 1 Processor -- start 2024-09-15T23:19:10.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff164de5740 1 -- start start 2024-09-15T23:19:10.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff164de5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff164de5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff164de5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6898d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff164de5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1af190 con 0x564f68998370 2024-09-15T23:19:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff164de5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1aca30 con 0x564f689932e0 2024-09-15T23:19:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff15affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff15affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42216/0 (socket says 172.21.15.62:42216) 2024-09-15T23:19:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff15affd640 1 -- 172.21.15.62:0/509809304 learned_addr learned my addr 172.21.15.62:0/509809304 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.376+0000 7ff164de5740 1 -- 172.21.15.62:0/509809304 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1aa880 con 0x564f689929a0 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.377+0000 7ff15bfff640 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6898d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.377+0000 7ff15b7fe640 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.377+0000 7ff15bfff640 1 -- 172.21.15.62:0/509809304 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 msgr2=0x564f6899cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.377+0000 7ff15bfff640 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.377+0000 7ff15bfff640 1 -- 172.21.15.62:0/509809304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 msgr2=0x564f689a3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.377+0000 7ff15bfff640 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.377+0000 7ff15bfff640 1 -- 172.21.15.62:0/509809304 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f6899a760 con 0x564f68998370 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.377+0000 7ff15affd640 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.378+0000 7ff15bfff640 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6898d0b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x564f6898ee90 tx=0x7ff154004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383768d89aabd50b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.378+0000 7ff1417fa640 1 -- 172.21.15.62:0/509809304 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154021070 con 0x564f68998370 2024-09-15T23:19:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.378+0000 7ff1417fa640 1 -- 172.21.15.62:0/509809304 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff15401cb40 con 0x564f68998370 2024-09-15T23:19:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.378+0000 7ff1417fa640 1 -- 172.21.15.62:0/509809304 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff154010cf0 con 0x564f68998370 2024-09-15T23:19:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.379+0000 7ff164de5740 1 -- 172.21.15.62:0/509809304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 msgr2=0x564f6898d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.379+0000 7ff164de5740 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6898d0b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x564f6898ee90 tx=0x7ff154004410 comp rx=0 tx=0).stop 2024-09-15T23:19:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.379+0000 7ff164de5740 1 -- 172.21.15.62:0/509809304 shutdown_connections 2024-09-15T23:19:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.379+0000 7ff164de5740 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f6898d0b0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.379+0000 7ff164de5740 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.379+0000 7ff164de5740 1 --2- 172.21.15.62:0/509809304 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f6899cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.379+0000 7ff164de5740 1 -- 172.21.15.62:0/509809304 >> 172.21.15.62:0/509809304 conn(0x564f6898e010 msgr2=0x564f6898c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.379+0000 7ff164de5740 1 -- 172.21.15.62:0/509809304 shutdown_connections 2024-09-15T23:19:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.379+0000 7ff164de5740 1 -- 172.21.15.62:0/509809304 wait complete. 2024-09-15T23:19:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.380+0000 7ff164de5740 1 -- start start 2024-09-15T23:19:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.380+0000 7ff164de5740 1 Processor -- start 2024-09-15T23:19:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.380+0000 7ff164de5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f689a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff164de5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.383 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff164de5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f689a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff164de5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1aaef0 con 0x564f68998370 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff164de5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1ac360 con 0x564f689932e0 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff164de5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c1ab5c0 con 0x564f689929a0 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff164de5740 10 client.0 _ll_register_callbacks cb 0x7ff14c00ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42224/0 (socket says 172.21.15.62:42224) 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f689a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15affd640 1 -- 172.21.15.62:0/1923576261 learned_addr learned my addr 172.21.15.62:0/1923576261 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15bfff640 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f689a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15affd640 1 -- 172.21.15.62:0/1923576261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 msgr2=0x564f689a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15affd640 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f689a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15affd640 1 -- 172.21.15.62:0/1923576261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 msgr2=0x564f689a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:10.384 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15affd640 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f689a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15affd640 1 -- 172.21.15.62:0/1923576261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f689a3450 con 0x564f689932e0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff15affd640 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a1e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff1500bbcd0 tx=0x7ff15002a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f6b0ca85c6e5146 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff13affd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.381+0000 7ff13affd640 1 -- 172.21.15.62:0/1923576261 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1500bc440 con 0x564f689932e0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff13affd640 1 -- 172.21.15.62:0/1923576261 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff1500b83b0 con 0x564f689932e0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff164de5740 1 -- 172.21.15.62:0/1923576261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff15401cb40 con 0x564f689932e0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff13affd640 1 -- 172.21.15.62:0/1923576261 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff150004e20 con 0x564f689932e0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff164de5740 10 client.25838 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff164de5740 1 -- 172.21.15.62:0/1923576261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f68a1bd70 con 0x564f689932e0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff13affd640 1 -- 172.21.15.62:0/1923576261 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff15401cb40 con 0x564f689932e0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff13affd640 1 -- 172.21.15.62:0/1923576261 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7ff15003a070 con 0x564f689932e0 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff164de5740 10 client.25838 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff164de5740 10 client.25838 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:10.385 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff164de5740 20 client.25838 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff164de5740 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f68a1c610 0x564f68a1ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff164de5740 4 client.25838 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:19:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.382+0000 7ff164de5740 1 -- 172.21.15.62:0/1923576261 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564f688fdd70 con 0x564f68a1c610 2024-09-15T23:19:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.383+0000 7ff15b7fe640 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f68a1c610 0x564f68a1ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:10.386 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.384+0000 7ff15b7fe640 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f68a1c610 0x564f68a1ea30 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cbbd080e9fdae02 server_cookie=2ba88d16deec3dbb in_seq=0 out_seq=0 2024-09-15T23:19:10.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.384+0000 7ff13affd640 10 client.25838 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.395+0000 7ff13affd640 1 -- 172.21.15.62:0/1923576261 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564f688fdd70 con 0x564f68a1c610 2024-09-15T23:19:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.395+0000 7ff13affd640 10 client.25838 handle_command_reply: tid=0 2024-09-15T23:19:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 client.25838 shutdown 2024-09-15T23:19:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 -- 172.21.15.62:0/1923576261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 msgr2=0x564f689a1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a1e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff1500bbcd0 tx=0x7ff15002a340 comp rx=0 tx=0).stop 2024-09-15T23:19:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 -- 172.21.15.62:0/1923576261 shutdown_connections 2024-09-15T23:19:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f68a1c610 0x564f68a1ea30 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f68998370 0x564f689a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f689932e0 0x564f689a1e30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff13affd640 0 client.25838 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 --2- 172.21.15.62:0/1923576261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f689929a0 0x564f689a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 -- 172.21.15.62:0/1923576261 >> 172.21.15.62:0/1923576261 conn(0x564f6898e010 msgr2=0x564f6898c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.396+0000 7ff164de5740 1 -- 172.21.15.62:0/1923576261 shutdown_connections 2024-09-15T23:19:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.397+0000 7ff164de5740 1 -- 172.21.15.62:0/1923576261 wait complete. 2024-09-15T23:19:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.397+0000 7ff164de5740 20 client.25838 trim_cache size 0 max 16384 2024-09-15T23:19:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:19:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:19:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:19:10.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:19:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.397+0000 7ff161f6d640 1 -- 172.21.15.62:0/3645850314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1440a34b0 msgr2=0x7ff1440a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.397+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1440a34b0 0x7ff1440a5970 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff15c101aa0 tx=0x7ff14c009660 comp rx=0 tx=0).stop 2024-09-15T23:19:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.397+0000 7ff161f6d640 1 -- 172.21.15.62:0/3645850314 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 msgr2=0x7ff15c103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.397+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 0x7ff15c103020 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff150031000 tx=0x7ff150033d10 comp rx=0 tx=0).stop 2024-09-15T23:19:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.398+0000 7ff161f6d640 1 -- 172.21.15.62:0/3645850314 shutdown_connections 2024-09-15T23:19:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.398+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1440a34b0 0x7ff1440a5970 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.398+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff15c102600 0x7ff15c103560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.398+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff15c101c40 0x7ff15c103020 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.398+0000 7ff161f6d640 1 --2- 172.21.15.62:0/3645850314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff15c100a50 0x7ff15c104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:10.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.398+0000 7ff161f6d640 1 -- 172.21.15.62:0/3645850314 >> 172.21.15.62:0/3645850314 conn(0x7ff15c0fc1b0 msgr2=0x7ff15c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:10.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.399+0000 7ff161f6d640 1 -- 172.21.15.62:0/3645850314 shutdown_connections 2024-09-15T23:19:10.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:10.399+0000 7ff161f6d640 1 -- 172.21.15.62:0/3645850314 wait complete. 2024-09-15T23:19:11.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:10 smithi062 ceph-mon[30949]: pgmap v457: 129 pgs: 129 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.0 MiB/s wr, 781 op/s 2024-09-15T23:19:11.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:10 smithi067 ceph-mon[35665]: pgmap v457: 129 pgs: 129 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.0 MiB/s wr, 781 op/s 2024-09-15T23:19:11.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:10 smithi148 ceph-mon[35610]: pgmap v457: 129 pgs: 129 active+clean; 1.0 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 7.0 MiB/s wr, 781 op/s 2024-09-15T23:19:12.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:12 smithi062 ceph-mon[30949]: pgmap v458: 129 pgs: 129 active+clean; 1.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.6 MiB/s wr, 852 op/s 2024-09-15T23:19:12.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:12 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:19:13.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:12 smithi067 ceph-mon[35665]: pgmap v458: 129 pgs: 129 active+clean; 1.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.6 MiB/s wr, 852 op/s 2024-09-15T23:19:13.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:12 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:19:13.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:12 smithi148 ceph-mon[35610]: pgmap v458: 129 pgs: 129 active+clean; 1.0 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 8.6 MiB/s wr, 852 op/s 2024-09-15T23:19:13.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:12 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:19:14.217 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:13 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:19:14.217 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:13 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:19:14.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:13 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:19:14.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:13 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:19:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:13 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:19:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:13 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:19:14.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 175.0 s, 2188.4 tps, lat 45.279 ms stddev 55.252 2024-09-15T23:19:15.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:14 smithi062 ceph-mon[30949]: pgmap v459: 129 pgs: 129 active+clean; 1.0 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 10 MiB/s wr, 937 op/s 2024-09-15T23:19:15.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:14 smithi067 ceph-mon[35665]: pgmap v459: 129 pgs: 129 active+clean; 1.0 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 10 MiB/s wr, 937 op/s 2024-09-15T23:19:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:14 smithi148 ceph-mon[35610]: pgmap v459: 129 pgs: 129 active+clean; 1.0 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 10 MiB/s wr, 937 op/s 2024-09-15T23:19:17.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:16 smithi062 ceph-mon[30949]: pgmap v460: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.2 MiB/s wr, 798 op/s 2024-09-15T23:19:17.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:16 smithi067 ceph-mon[35665]: pgmap v460: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.2 MiB/s wr, 798 op/s 2024-09-15T23:19:17.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:16 smithi148 ceph-mon[35610]: pgmap v460: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 9.2 MiB/s wr, 798 op/s 2024-09-15T23:19:19.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:18 smithi062 ceph-mon[30949]: pgmap v461: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.2 MiB/s wr, 751 op/s 2024-09-15T23:19:19.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:18 smithi067 ceph-mon[35665]: pgmap v461: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.2 MiB/s wr, 751 op/s 2024-09-15T23:19:19.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:18 smithi148 ceph-mon[35610]: pgmap v461: 129 pgs: 129 active+clean; 1.0 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.2 MiB/s wr, 751 op/s 2024-09-15T23:19:19.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 180.0 s, 2476.2 tps, lat 40.790 ms stddev 45.771 2024-09-15T23:19:21.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:20 smithi062 ceph-mon[30949]: pgmap v462: 129 pgs: 129 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 890 op/s 2024-09-15T23:19:21.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:20 smithi067 ceph-mon[35665]: pgmap v462: 129 pgs: 129 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 890 op/s 2024-09-15T23:19:21.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:20 smithi148 ceph-mon[35610]: pgmap v462: 129 pgs: 129 active+clean; 1.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 11 MiB/s wr, 890 op/s 2024-09-15T23:19:23.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:22 smithi062 ceph-mon[30949]: pgmap v463: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.2 MiB/s wr, 797 op/s 2024-09-15T23:19:23.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:22 smithi067 ceph-mon[35665]: pgmap v463: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.2 MiB/s wr, 797 op/s 2024-09-15T23:19:23.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:22 smithi148 ceph-mon[35610]: pgmap v463: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 9.2 MiB/s wr, 797 op/s 2024-09-15T23:19:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.526+0000 7f1f4c465640 1 -- 172.21.15.62:0/1670579089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f44102600 msgr2=0x7f1f44108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.526+0000 7f1f4c465640 1 --2- 172.21.15.62:0/1670579089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f44102600 0x7f1f44108e90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00bc00 tx=0x7f1f3c02e620 comp rx=0 tx=0).stop 2024-09-15T23:19:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.527+0000 7f1f4c465640 1 -- 172.21.15.62:0/1670579089 shutdown_connections 2024-09-15T23:19:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.527+0000 7f1f4c465640 1 --2- 172.21.15.62:0/1670579089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f44102600 0x7f1f44108e90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.527+0000 7f1f4c465640 1 --2- 172.21.15.62:0/1670579089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f44101c40 0x7f1f441020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.527+0000 7f1f4c465640 1 --2- 172.21.15.62:0/1670579089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f44100a50 0x7f1f44100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.527+0000 7f1f4c465640 1 -- 172.21.15.62:0/1670579089 >> 172.21.15.62:0/1670579089 conn(0x7f1f440fc1b0 msgr2=0x7f1f440fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.527+0000 7f1f4c465640 1 -- 172.21.15.62:0/1670579089 shutdown_connections 2024-09-15T23:19:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.528+0000 7f1f4c465640 1 -- 172.21.15.62:0/1670579089 wait complete. 2024-09-15T23:19:23.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.528+0000 7f1f4c465640 1 Processor -- start 2024-09-15T23:19:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.528+0000 7f1f4c465640 1 -- start start 2024-09-15T23:19:23.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.528+0000 7f1f4c465640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f44100a50 0x7f1f44195b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.528+0000 7f1f4c465640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f44101c40 0x7f1f441960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4c465640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f44102600 0x7f1f4419d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4c465640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4410b5d0 con 0x7f1f44102600 2024-09-15T23:19:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4c465640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4410b450 con 0x7f1f44100a50 2024-09-15T23:19:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4c465640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4410b750 con 0x7f1f44101c40 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4aa02640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f44102600 0x7f1f4419d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4a201640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f44100a50 0x7f1f44195b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4aa02640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f44102600 0x7f1f4419d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41332/0 (socket says 172.21.15.62:41332) 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4aa02640 1 -- 172.21.15.62:0/2138211226 learned_addr learned my addr 172.21.15.62:0/2138211226 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f49a00640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f44101c40 0x7f1f441960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4a201640 1 -- 172.21.15.62:0/2138211226 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f44101c40 msgr2=0x7f1f441960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4a201640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f44101c40 0x7f1f441960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4a201640 1 -- 172.21.15.62:0/2138211226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f44102600 msgr2=0x7f1f4419d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4a201640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f44102600 0x7f1f4419d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.529+0000 7f1f4a201640 1 -- 172.21.15.62:0/2138211226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4419d7a0 con 0x7f1f44100a50 2024-09-15T23:19:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.530+0000 7f1f4a201640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f44100a50 0x7f1f44195b80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800c940 tx=0x7f1f3800cd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.530+0000 7f1f377fe640 1 -- 172.21.15.62:0/2138211226 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38004220 con 0x7f1f44100a50 2024-09-15T23:19:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.530+0000 7f1f377fe640 1 -- 172.21.15.62:0/2138211226 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f380043c0 con 0x7f1f44100a50 2024-09-15T23:19:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.530+0000 7f1f377fe640 1 -- 172.21.15.62:0/2138211226 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3800f450 con 0x7f1f44100a50 2024-09-15T23:19:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.530+0000 7f1f4c465640 1 -- 172.21.15.62:0/2138211226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f4419da90 con 0x7f1f44100a50 2024-09-15T23:19:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.530+0000 7f1f4c465640 1 -- 172.21.15.62:0/2138211226 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f4419df90 con 0x7f1f44100a50 2024-09-15T23:19:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.533+0000 7f1f377fe640 1 -- 172.21.15.62:0/2138211226 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1f3800f5f0 con 0x7f1f44100a50 2024-09-15T23:19:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.533+0000 7f1f4f203740 1 Processor -- start 2024-09-15T23:19:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.534+0000 7f1f4f203740 1 -- start start 2024-09-15T23:19:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f03e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f0435b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f046040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4f203740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56111ef992f0 con 0x56111f03eaf0 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4f203740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56111ef41270 con 0x56111f03e1b0 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4f203740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56111ef952b0 con 0x56111f043b80 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f377fe640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1f300a33f0 0x7f1f300a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4aa02640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f03e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4aa02640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f03e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:55018/0 (socket says 172.21.15.62:55018) 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4aa02640 1 -- 172.21.15.62:0/1126341372 learned_addr learned my addr 172.21.15.62:0/1126341372 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1f300a33f0 0x7f1f300a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f377fe640 1 -- 172.21.15.62:0/2138211226 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1f380cfc30 con 0x7f1f44100a50 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f4a201640 1 --2- 172.21.15.62:0/1126341372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f0435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.535+0000 7f1f49a00640 1 --2- 172.21.15.62:0/1126341372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f046040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f4aa02640 1 -- 172.21.15.62:0/1126341372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 msgr2=0x56111f046040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/1126341372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f046040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f4aa02640 1 -- 172.21.15.62:0/1126341372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 msgr2=0x56111f0435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/1126341372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f0435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f4aa02640 1 -- 172.21.15.62:0/1126341372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56111ef40340 con 0x56111f03e1b0 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/1126341372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f03e5b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c005c40 tx=0x7f1f3c035b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a08b298f9e94e2d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1f300a33f0 0x7f1f300a58b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00bb50 tx=0x7f1f3c047000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f34bfe640 1 -- 172.21.15.62:0/1126341372 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c03a040 con 0x56111f03e1b0 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f34bfe640 1 -- 172.21.15.62:0/1126341372 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3c003e00 con 0x56111f03e1b0 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.536+0000 7f1f34bfe640 1 -- 172.21.15.62:0/1126341372 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c0053b0 con 0x56111f03e1b0 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.537+0000 7f1f4f203740 1 -- 172.21.15.62:0/1126341372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 msgr2=0x56111f03e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.537+0000 7f1f4f203740 1 --2- 172.21.15.62:0/1126341372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f03e5b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c005c40 tx=0x7f1f3c035b00 comp rx=0 tx=0).stop 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.538+0000 7f1f4f203740 1 -- 172.21.15.62:0/1126341372 shutdown_connections 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.538+0000 7f1f4f203740 1 --2- 172.21.15.62:0/1126341372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f046040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.538+0000 7f1f4f203740 1 --2- 172.21.15.62:0/1126341372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f0435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.538+0000 7f1f4f203740 1 --2- 172.21.15.62:0/1126341372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f03e5b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.538+0000 7f1f4f203740 1 -- 172.21.15.62:0/1126341372 >> 172.21.15.62:0/1126341372 conn(0x56111f039820 msgr2=0x56111f039c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.538+0000 7f1f4f203740 1 -- 172.21.15.62:0/1126341372 shutdown_connections 2024-09-15T23:19:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.538+0000 7f1f4f203740 1 -- 172.21.15.62:0/1126341372 wait complete. 2024-09-15T23:19:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.538+0000 7f1f4f203740 1 -- start start 2024-09-15T23:19:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.538+0000 7f1f4f203740 1 Processor -- start 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f048410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f03eaf0 0x56111f048950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f043b80 0x56111f04f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4f203740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56111f046af0 con 0x56111f043b80 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4f203740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56111ef22290 con 0x56111f03e1b0 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4f203740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56111f046c70 con 0x56111f03eaf0 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4f203740 10 client.0 _ll_register_callbacks cb 0x56111ef53140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4aa02640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f043b80 0x56111f04f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4a201640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f048410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4a201640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f048410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:55020/0 (socket says 172.21.15.62:55020) 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4a201640 1 -- 172.21.15.62:0/4180869323 learned_addr learned my addr 172.21.15.62:0/4180869323 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f49a00640 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f03eaf0 0x56111f048950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4a201640 1 -- 172.21.15.62:0/4180869323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f03eaf0 msgr2=0x56111f048950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4a201640 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f03eaf0 0x56111f048950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4a201640 1 -- 172.21.15.62:0/4180869323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f043b80 msgr2=0x56111f04f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4a201640 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f043b80 0x56111f04f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.539+0000 7f1f4a201640 1 -- 172.21.15.62:0/4180869323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56111ecac210 con 0x56111f03e1b0 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.540+0000 7f1f4a201640 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f048410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f3809be00 tx=0x7f1f3809b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8267f44be41191a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.540+0000 7f1f2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.540+0000 7f1f2dffb640 1 -- 172.21.15.62:0/4180869323 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3801d040 con 0x56111f03e1b0 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.540+0000 7f1f4f203740 1 -- 172.21.15.62:0/4180869323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f3c003e00 con 0x56111f03e1b0 2024-09-15T23:19:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.540+0000 7f1f2dffb640 1 -- 172.21.15.62:0/4180869323 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1f380a4850 con 0x56111f03e1b0 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.540+0000 7f1f2dffb640 1 -- 172.21.15.62:0/4180869323 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1f380a0220 con 0x56111f03e1b0 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.540+0000 7f1f4f203740 10 client.25850 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.540+0000 7f1f4f203740 1 -- 172.21.15.62:0/4180869323 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56111ed78620 con 0x56111f03e1b0 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.541+0000 7f1f2dffb640 1 -- 172.21.15.62:0/4180869323 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f3c003e00 con 0x56111f03e1b0 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.541+0000 7f1f2dffb640 1 -- 172.21.15.62:0/4180869323 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f1f380baab0 con 0x56111f03e1b0 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.541+0000 7f1f4f203740 10 client.25850 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.541+0000 7f1f4f203740 10 client.25850 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.541+0000 7f1f4f203740 20 client.25850 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.541+0000 7f1f4f203740 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56111f0540e0 0x56111f056560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.541+0000 7f1f4f203740 4 client.25850 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:23.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.541+0000 7f1f4f203740 1 -- 172.21.15.62:0/4180869323 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56111efa6a20 con 0x56111f0540e0 2024-09-15T23:19:23.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.542+0000 7f1f49a00640 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56111f0540e0 0x56111f056560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.543+0000 7f1f49a00640 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56111f0540e0 0x56111f056560 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=944a201ccbde417 server_cookie=3dd0c916b541a85b in_seq=0 out_seq=0 2024-09-15T23:19:23.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.543+0000 7f1f2dffb640 10 client.25850 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f2dffb640 1 -- 172.21.15.62:0/4180869323 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56111efa6a20 con 0x56111f0540e0 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f2dffb640 10 client.25850 handle_command_reply: tid=0 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f4f203740 1 client.25850 shutdown 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f4f203740 1 -- 172.21.15.62:0/4180869323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 msgr2=0x56111f048410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f4f203740 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f048410 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1f3809be00 tx=0x7f1f3809b890 comp rx=0 tx=0).stop 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f4f203740 1 -- 172.21.15.62:0/4180869323 shutdown_connections 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f4f203740 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56111f0540e0 0x56111f056560 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f4f203740 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f043b80 0x56111f04f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f4f203740 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f03eaf0 0x56111f048950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f4f203740 1 --2- 172.21.15.62:0/4180869323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f048410 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f2dffb640 0 client.25850 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.544+0000 7f1f4f203740 1 -- 172.21.15.62:0/4180869323 >> 172.21.15.62:0/4180869323 conn(0x56111f039820 msgr2=0x56111f0301c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:23.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.545+0000 7f1f4f203740 1 -- 172.21.15.62:0/4180869323 shutdown_connections 2024-09-15T23:19:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.545+0000 7f1f4f203740 1 -- 172.21.15.62:0/4180869323 wait complete. 2024-09-15T23:19:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.545+0000 7f1f4f203740 20 client.25850 trim_cache size 0 max 16384 2024-09-15T23:19:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4f203740 1 Processor -- start 2024-09-15T23:19:23.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4f203740 1 -- start start 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03e1b0 0x56111f0300b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03eaf0 0x56111f04f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f05ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4f203740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1f441aae40 con 0x56111f03e1b0 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4f203740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1f441a86e0 con 0x56111f03eaf0 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4f203740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1f441a6530 con 0x56111f043b80 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4a201640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03e1b0 0x56111f0300b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4a201640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03e1b0 0x56111f0300b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41376/0 (socket says 172.21.15.62:41376) 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4a201640 1 -- 172.21.15.62:0/2586617513 learned_addr learned my addr 172.21.15.62:0/2586617513 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f49a00640 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03eaf0 0x56111f04f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.569+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f05ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f49a00640 1 -- 172.21.15.62:0/2586617513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 msgr2=0x56111f05ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f49a00640 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f05ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f49a00640 1 -- 172.21.15.62:0/2586617513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03e1b0 msgr2=0x56111f0300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f49a00640 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03e1b0 0x56111f0300b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f49a00640 1 -- 172.21.15.62:0/2586617513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56111f056be0 con 0x56111f03eaf0 2024-09-15T23:19:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f4a201640 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03e1b0 0x56111f0300b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f05ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:19:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f49a00640 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03eaf0 0x56111f04f8b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x56111f056e10 tx=0x7f1f40009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75a7c80bdddfb866 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f2ffff640 1 -- 172.21.15.62:0/2586617513 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f400042f0 con 0x56111f03eaf0 2024-09-15T23:19:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f2ffff640 1 -- 172.21.15.62:0/2586617513 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f40004490 con 0x56111f03eaf0 2024-09-15T23:19:23.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.570+0000 7f1f2ffff640 1 -- 172.21.15.62:0/2586617513 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f400035f0 con 0x56111f03eaf0 2024-09-15T23:19:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 -- 172.21.15.62:0/2586617513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03eaf0 msgr2=0x56111f04f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03eaf0 0x56111f04f8b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x56111f056e10 tx=0x7f1f40009260 comp rx=0 tx=0).stop 2024-09-15T23:19:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 -- 172.21.15.62:0/2586617513 shutdown_connections 2024-09-15T23:19:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f05ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03eaf0 0x56111f04f8b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 --2- 172.21.15.62:0/2586617513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03e1b0 0x56111f0300b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 -- 172.21.15.62:0/2586617513 >> 172.21.15.62:0/2586617513 conn(0x56111f039820 msgr2=0x56111f02f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 -- 172.21.15.62:0/2586617513 shutdown_connections 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 -- 172.21.15.62:0/2586617513 wait complete. 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.571+0000 7f1f4f203740 1 -- start start 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4f203740 1 Processor -- start 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f04d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f04d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4f203740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f04e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4f203740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1f441a6ba0 con 0x56111f03eaf0 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4f203740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1f441a8010 con 0x56111f03e1b0 2024-09-15T23:19:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4f203740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1f441a7270 con 0x56111f043b80 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4f203740 10 client.0 _ll_register_callbacks cb 0x7f1f3800ce70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f49a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f04d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4aa02640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f04e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f49a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f04d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41390/0 (socket says 172.21.15.62:41390) 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f49a00640 1 -- 172.21.15.62:0/83461061 learned_addr learned my addr 172.21.15.62:0/83461061 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4a201640 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f04d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4aa02640 1 -- 172.21.15.62:0/83461061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 msgr2=0x56111f04d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f04d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4aa02640 1 -- 172.21.15.62:0/83461061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 msgr2=0x56111f04d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f04d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4aa02640 1 -- 172.21.15.62:0/83461061 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56111f04ed10 con 0x56111f043b80 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f4a201640 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f04d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.572+0000 7f1f49a00640 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f04d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f4aa02640 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f04e700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c02eb40 tx=0x7f1f3c02e620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60f4d3b14f1cbec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f2dffb640 1 -- 172.21.15.62:0/83461061 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c05b070 con 0x56111f043b80 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f4f203740 1 -- 172.21.15.62:0/83461061 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f40004490 con 0x56111f043b80 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f2dffb640 1 -- 172.21.15.62:0/83461061 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1f3c004b40 con 0x56111f043b80 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f2dffb640 1 -- 172.21.15.62:0/83461061 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1f3c067050 con 0x56111f043b80 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f4f203740 10 client.25699 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f4f203740 1 -- 172.21.15.62:0/83461061 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56111f0bf0d0 con 0x56111f043b80 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f2dffb640 1 -- 172.21.15.62:0/83461061 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f40004490 con 0x56111f043b80 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.573+0000 7f1f2dffb640 1 -- 172.21.15.62:0/83461061 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f1f3c04a070 con 0x56111f043b80 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.574+0000 7f1f4f203740 10 client.25699 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.574+0000 7f1f4f203740 10 client.25699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.574+0000 7f1f4f203740 20 client.25699 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.574+0000 7f1f4f203740 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56111f0bf640 0x56111f0c1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.574+0000 7f1f4f203740 4 client.25699 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.574+0000 7f1f4f203740 1 -- 172.21.15.62:0/83461061 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56111efa0d70 con 0x56111f0bf640 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.574+0000 7f1f4a201640 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56111f0bf640 0x56111f0c1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.574+0000 7f1f2dffb640 10 client.25699 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.574+0000 7f1f4a201640 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56111f0bf640 0x56111f0c1a30 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7694c7bf693a7dbc server_cookie=9bf3e01e6030cfd4 in_seq=0 out_seq=0 2024-09-15T23:19:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.596+0000 7f1f2dffb640 1 -- 172.21.15.62:0/83461061 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56111efa0d70 con 0x56111f0bf640 2024-09-15T23:19:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.596+0000 7f1f2dffb640 10 client.25699 handle_command_reply: tid=0 2024-09-15T23:19:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.596+0000 7f1f4f203740 1 client.25699 shutdown 2024-09-15T23:19:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.596+0000 7f1f4f203740 1 -- 172.21.15.62:0/83461061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 msgr2=0x56111f04e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.596+0000 7f1f4f203740 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f04e700 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c02eb40 tx=0x7f1f3c02e620 comp rx=0 tx=0).stop 2024-09-15T23:19:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f4f203740 1 -- 172.21.15.62:0/83461061 shutdown_connections 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f4f203740 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56111f0bf640 0x56111f0c1a30 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f2dffb640 0 client.25699 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f4f203740 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56111f043b80 0x56111f04e700 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f4f203740 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56111f03eaf0 0x56111f04d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f4f203740 1 --2- 172.21.15.62:0/83461061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56111f03e1b0 0x56111f04d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f4f203740 1 -- 172.21.15.62:0/83461061 >> 172.21.15.62:0/83461061 conn(0x56111f039820 msgr2=0x56111f0301c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f4f203740 1 -- 172.21.15.62:0/83461061 shutdown_connections 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f4f203740 1 -- 172.21.15.62:0/83461061 wait complete. 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.597+0000 7f1f4f203740 20 client.25699 trim_cache size 0 max 16384 2024-09-15T23:19:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:19:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:19:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:19:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:19:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 -- 172.21.15.62:0/2138211226 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1f300a33f0 msgr2=0x7f1f300a58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1f300a33f0 0x7f1f300a58b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00bb50 tx=0x7f1f3c047000 comp rx=0 tx=0).stop 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 -- 172.21.15.62:0/2138211226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f44100a50 msgr2=0x7f1f44195b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f44100a50 0x7f1f44195b80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f3800c940 tx=0x7f1f3800cd40 comp rx=0 tx=0).stop 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 -- 172.21.15.62:0/2138211226 shutdown_connections 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1f300a33f0 0x7f1f300a58b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f44102600 0x7f1f4419d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1f44101c40 0x7f1f441960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 --2- 172.21.15.62:0/2138211226 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f44100a50 0x7f1f44195b80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 -- 172.21.15.62:0/2138211226 >> 172.21.15.62:0/2138211226 conn(0x7f1f440fc1b0 msgr2=0x7f1f44104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 -- 172.21.15.62:0/2138211226 shutdown_connections 2024-09-15T23:19:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:23.598+0000 7f1f4c465640 1 -- 172.21.15.62:0/2138211226 wait complete. 2024-09-15T23:19:24.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 185.0 s, 2551.4 tps, lat 38.795 ms stddev 46.074 2024-09-15T23:19:25.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:24 smithi062 ceph-mon[30949]: pgmap v464: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.3 MiB/s wr, 701 op/s 2024-09-15T23:19:25.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:24 smithi067 ceph-mon[35665]: pgmap v464: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.3 MiB/s wr, 701 op/s 2024-09-15T23:19:25.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:24 smithi148 ceph-mon[35610]: pgmap v464: 129 pgs: 129 active+clean; 1.0 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.3 MiB/s wr, 701 op/s 2024-09-15T23:19:27.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:26 smithi062 ceph-mon[30949]: pgmap v465: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.1 MiB/s wr, 680 op/s 2024-09-15T23:19:27.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:26 smithi067 ceph-mon[35665]: pgmap v465: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.1 MiB/s wr, 680 op/s 2024-09-15T23:19:27.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:26 smithi148 ceph-mon[35610]: pgmap v465: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.1 MiB/s wr, 680 op/s 2024-09-15T23:19:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:28 smithi062 ceph-mon[30949]: pgmap v466: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.8 MiB/s wr, 638 op/s 2024-09-15T23:19:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:28 smithi067 ceph-mon[35665]: pgmap v466: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.8 MiB/s wr, 638 op/s 2024-09-15T23:19:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:28 smithi148 ceph-mon[35610]: pgmap v466: 129 pgs: 129 active+clean; 1.1 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.8 MiB/s wr, 638 op/s 2024-09-15T23:19:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 190.0 s, 2038.0 tps, lat 49.397 ms stddev 57.935 2024-09-15T23:19:31.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:30 smithi062 ceph-mon[30949]: pgmap v467: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.9 MiB/s wr, 704 op/s 2024-09-15T23:19:31.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:30 smithi067 ceph-mon[35665]: pgmap v467: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.9 MiB/s wr, 704 op/s 2024-09-15T23:19:31.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:30 smithi148 ceph-mon[35610]: pgmap v467: 129 pgs: 129 active+clean; 1.1 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.9 MiB/s wr, 704 op/s 2024-09-15T23:19:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:32 smithi062 ceph-mon[30949]: pgmap v468: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 686 op/s 2024-09-15T23:19:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:32 smithi067 ceph-mon[35665]: pgmap v468: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 686 op/s 2024-09-15T23:19:33.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:32 smithi148 ceph-mon[35610]: pgmap v468: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.0 MiB/s wr, 686 op/s 2024-09-15T23:19:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 195.0 s, 1625.2 tps, lat 61.511 ms stddev 67.727 2024-09-15T23:19:35.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:34 smithi062 ceph-mon[30949]: pgmap v469: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 742 op/s 2024-09-15T23:19:35.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:34 smithi067 ceph-mon[35665]: pgmap v469: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 742 op/s 2024-09-15T23:19:35.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:34 smithi148 ceph-mon[35610]: pgmap v469: 129 pgs: 129 active+clean; 1.1 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 742 op/s 2024-09-15T23:19:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.723+0000 7f569d8d7640 1 -- 172.21.15.62:0/1245940975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5698102ea0 msgr2=0x7f5698103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.723+0000 7f569d8d7640 1 --2- 172.21.15.62:0/1245940975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5698102ea0 0x7f5698103320 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f568c009900 tx=0x7f568c02a410 comp rx=0 tx=0).stop 2024-09-15T23:19:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.724+0000 7f569d8d7640 1 -- 172.21.15.62:0/1245940975 shutdown_connections 2024-09-15T23:19:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.724+0000 7f569d8d7640 1 --2- 172.21.15.62:0/1245940975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5698103860 0x7f569810a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.724+0000 7f569d8d7640 1 --2- 172.21.15.62:0/1245940975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5698102ea0 0x7f5698103320 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.724+0000 7f569d8d7640 1 --2- 172.21.15.62:0/1245940975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f56980ffe30 0x7f5698100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.724+0000 7f569d8d7640 1 -- 172.21.15.62:0/1245940975 >> 172.21.15.62:0/1245940975 conn(0x7f56980fb930 msgr2=0x7f56980fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.724+0000 7f569d8d7640 1 -- 172.21.15.62:0/1245940975 shutdown_connections 2024-09-15T23:19:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.724+0000 7f569d8d7640 1 -- 172.21.15.62:0/1245940975 wait complete. 2024-09-15T23:19:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.724+0000 7f569d8d7640 1 Processor -- start 2024-09-15T23:19:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f569d8d7640 1 -- start start 2024-09-15T23:19:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f569d8d7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56980ffe30 0x7f5698195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f569d8d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5698102ea0 0x7f5698196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f569d8d7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5698103860 0x7f569819d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f569d8d7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f569810b630 con 0x7f56980ffe30 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f569d8d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f569810b4b0 con 0x7f5698102ea0 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f569d8d7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f569810b7b0 con 0x7f5698103860 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f5696ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56980ffe30 0x7f5698195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f5696ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56980ffe30 0x7f5698195bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56106/0 (socket says 172.21.15.62:56106) 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f5696ffd640 1 -- 172.21.15.62:0/2828744723 learned_addr learned my addr 172.21.15.62:0/2828744723 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f56967fc640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5698102ea0 0x7f5698196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.725+0000 7f56977fe640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5698103860 0x7f569819d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f5696ffd640 1 -- 172.21.15.62:0/2828744723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5698103860 msgr2=0x7f569819d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f5696ffd640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5698103860 0x7f569819d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f5696ffd640 1 -- 172.21.15.62:0/2828744723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5698102ea0 msgr2=0x7f5698196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f5696ffd640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5698102ea0 0x7f5698196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f5696ffd640 1 -- 172.21.15.62:0/2828744723 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569819d7f0 con 0x7f56980ffe30 2024-09-15T23:19:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f56967fc640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5698102ea0 0x7f5698196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f5696ffd640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56980ffe30 0x7f5698195bd0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f568800d950 tx=0x7f568800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f569c8d5640 1 -- 172.21.15.62:0/2828744723 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5688017070 con 0x7f56980ffe30 2024-09-15T23:19:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f569c8d5640 1 -- 172.21.15.62:0/2828744723 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5688009c30 con 0x7f56980ffe30 2024-09-15T23:19:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f569c8d5640 1 -- 172.21.15.62:0/2828744723 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568800f600 con 0x7f56980ffe30 2024-09-15T23:19:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f569d8d7640 1 -- 172.21.15.62:0/2828744723 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f569819dae0 con 0x7f56980ffe30 2024-09-15T23:19:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.726+0000 7f569d8d7640 1 -- 172.21.15.62:0/2828744723 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f569819dfb0 con 0x7f56980ffe30 2024-09-15T23:19:36.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.730+0000 7f56a0672740 1 Processor -- start 2024-09-15T23:19:36.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.731+0000 7f56a0672740 1 -- start start 2024-09-15T23:19:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.731+0000 7f56a0672740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 0x55b0794395b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56a0672740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b079439af0 0x55b07943e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56a0672740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 0x55b079441040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56a0672740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b0793942f0 con 0x55b079439af0 2024-09-15T23:19:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56a0672740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b07933c270 con 0x55b0794391b0 2024-09-15T23:19:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56a0672740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b0793902b0 con 0x55b07943eb80 2024-09-15T23:19:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56977fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 0x55b0794395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56977fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 0x55b0794395b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:37528/0 (socket says 172.21.15.62:37528) 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56977fe640 1 -- 172.21.15.62:0/3533184485 learned_addr learned my addr 172.21.15.62:0/3533184485 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f569c8d5640 1 -- 172.21.15.62:0/2828744723 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f568801a070 con 0x7f56980ffe30 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56967fc640 1 --2- 172.21.15.62:0/3533184485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 0x55b079441040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56977fe640 1 -- 172.21.15.62:0/3533184485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 msgr2=0x55b079441040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56977fe640 1 --2- 172.21.15.62:0/3533184485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 0x55b079441040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56977fe640 1 -- 172.21.15.62:0/3533184485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b079439af0 msgr2=0x55b07943e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56977fe640 1 --2- 172.21.15.62:0/3533184485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b079439af0 0x55b07943e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56977fe640 1 -- 172.21.15.62:0/3533184485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b07933b340 con 0x55b0794391b0 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.732+0000 7f56977fe640 1 --2- 172.21.15.62:0/3533184485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 0x55b0794395b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5690009fd0 tx=0x7f56900118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50873dec7bca0a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.733+0000 7f567dffb640 1 -- 172.21.15.62:0/3533184485 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5690004d20 con 0x55b0794391b0 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.733+0000 7f567dffb640 1 -- 172.21.15.62:0/3533184485 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5690004050 con 0x55b0794391b0 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.733+0000 7f567dffb640 1 -- 172.21.15.62:0/3533184485 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5690005260 con 0x55b0794391b0 2024-09-15T23:19:36.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.733+0000 7f569c8d5640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f56800a33c0 0x7f56800a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.733+0000 7f5696ffd640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f56800a33c0 0x7f56800a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.733+0000 7f569c8d5640 1 -- 172.21.15.62:0/2828744723 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f56880cf600 con 0x7f56980ffe30 2024-09-15T23:19:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.733+0000 7f56a0672740 1 -- 172.21.15.62:0/3533184485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 msgr2=0x55b0794395b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.733+0000 7f56a0672740 1 --2- 172.21.15.62:0/3533184485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 0x55b0794395b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5690009fd0 tx=0x7f56900118d0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.734+0000 7f56a0672740 1 -- 172.21.15.62:0/3533184485 shutdown_connections 2024-09-15T23:19:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.734+0000 7f56a0672740 1 --2- 172.21.15.62:0/3533184485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 0x55b079441040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.734+0000 7f56a0672740 1 --2- 172.21.15.62:0/3533184485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b079439af0 0x55b07943e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.734+0000 7f56a0672740 1 --2- 172.21.15.62:0/3533184485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 0x55b0794395b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.734+0000 7f56a0672740 1 -- 172.21.15.62:0/3533184485 >> 172.21.15.62:0/3533184485 conn(0x55b07942c010 msgr2=0x55b07942c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.734+0000 7f5696ffd640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f56800a33c0 0x7f56800a5880 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5688097db0 tx=0x7f5688086040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.734+0000 7f56a0672740 1 -- 172.21.15.62:0/3533184485 shutdown_connections 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.734+0000 7f56a0672740 1 -- 172.21.15.62:0/3533184485 wait complete. 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.734+0000 7f56a0672740 1 -- start start 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.735+0000 7f56a0672740 1 Processor -- start 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.735+0000 7f56a0672740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.735+0000 7f56a0672740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079443880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.735+0000 7f56a0672740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b07943eb80 0x55b07944a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56967fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079443880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56967fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079443880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:54600/0 (socket says 172.21.15.62:54600) 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56967fc640 1 -- 172.21.15.62:0/253564062 learned_addr learned my addr 172.21.15.62:0/253564062 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56977fe640 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b079441b50 con 0x55b0794391b0 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b07931d290 con 0x55b07943eb80 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b079441cd0 con 0x55b079439af0 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56a0672740 10 client.0 _ll_register_callbacks cb 0x55b07934e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56977fe640 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b07943eb80 0x55b07944a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56977fe640 1 -- 172.21.15.62:0/253564062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 msgr2=0x55b079443880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.736+0000 7f56977fe640 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079443880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f56977fe640 1 -- 172.21.15.62:0/253564062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b07943eb80 msgr2=0x55b07944a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f56977fe640 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b07943eb80 0x55b07944a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f56977fe640 1 -- 172.21.15.62:0/253564062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0790a7210 con 0x55b0794391b0 2024-09-15T23:19:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f56977fe640 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443340 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5690011dd0 tx=0x7f5690004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5b7d9897bb0dcd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f5676ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f5676ffd640 1 -- 172.21.15.62:0/253564062 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5690004200 con 0x55b0794391b0 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f5676ffd640 1 -- 172.21.15.62:0/253564062 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f56900043a0 con 0x55b0794391b0 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5690004050 con 0x55b0794391b0 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f5676ffd640 1 -- 172.21.15.62:0/253564062 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5690035050 con 0x55b0794391b0 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f56a0672740 10 client.16236 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.737+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b079173620 con 0x55b0794391b0 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.738+0000 7f5676ffd640 1 -- 172.21.15.62:0/253564062 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5690004050 con 0x55b0794391b0 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.738+0000 7f5676ffd640 1 -- 172.21.15.62:0/253564062 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f569000f4e0 con 0x55b0794391b0 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.738+0000 7f56a0672740 10 client.16236 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.738+0000 7f56a0672740 10 client.16236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.738+0000 7f56a0672740 20 client.16236 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.738+0000 7f56a0672740 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b07944ed50 0x55b079451180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.738+0000 7f56a0672740 4 client.16236 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.738+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0793a1a20 con 0x55b07944ed50 2024-09-15T23:19:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.738+0000 7f56967fc640 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b07944ed50 0x55b079451180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.739+0000 7f5676ffd640 10 client.16236 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:36.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.739+0000 7f56967fc640 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b07944ed50 0x55b079451180 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58d845d5da421397 server_cookie=ba7486298658d0f1 in_seq=0 out_seq=0 2024-09-15T23:19:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f5676ffd640 1 -- 172.21.15.62:0/253564062 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b0793a1a20 con 0x55b07944ed50 2024-09-15T23:19:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f5676ffd640 10 client.16236 handle_command_reply: tid=0 2024-09-15T23:19:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f56a0672740 1 client.16236 shutdown 2024-09-15T23:19:36.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 msgr2=0x55b079443340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f56a0672740 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443340 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5690011dd0 tx=0x7f5690004870 comp rx=0 tx=0).stop 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 shutdown_connections 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f56a0672740 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b07944ed50 0x55b079451180 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f56a0672740 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b07943eb80 0x55b07944a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f56a0672740 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079443880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f56a0672740 1 --2- 172.21.15.62:0/253564062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443340 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.742+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 >> 172.21.15.62:0/253564062 conn(0x55b07942c010 msgr2=0x55b07942d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.743+0000 7f5676ffd640 0 client.16236 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.743+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 shutdown_connections 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.743+0000 7f56a0672740 1 -- 172.21.15.62:0/253564062 wait complete. 2024-09-15T23:19:36.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.743+0000 7f56a0672740 20 client.16236 trim_cache size 0 max 16384 2024-09-15T23:19:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.767+0000 7f56a0672740 1 Processor -- start 2024-09-15T23:19:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.771+0000 7f56a0672740 1 -- start start 2024-09-15T23:19:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.771+0000 7f56a0672740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.771+0000 7f56a0672740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b079439af0 0x55b07944a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.772+0000 7f56a0672740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 0x55b07942d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.772+0000 7f56a0672740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f56981aad80 con 0x55b0794391b0 2024-09-15T23:19:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.772+0000 7f56a0672740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f56981a8620 con 0x55b079439af0 2024-09-15T23:19:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.772+0000 7f56a0672740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f56981a6470 con 0x55b07943eb80 2024-09-15T23:19:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.772+0000 7f56977fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.772+0000 7f56977fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56142/0 (socket says 172.21.15.62:56142) 2024-09-15T23:19:36.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.772+0000 7f56977fe640 1 -- 172.21.15.62:0/3483528946 learned_addr learned my addr 172.21.15.62:0/3483528946 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:36.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.773+0000 7f56977fe640 1 --2- 172.21.15.62:0/3483528946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 0x55b07942d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.776+0000 7f56967fc640 1 --2- 172.21.15.62:0/3483528946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b079439af0 0x55b07944a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.777+0000 7f56977fe640 1 -- 172.21.15.62:0/3483528946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 msgr2=0x55b07942d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.777+0000 7f56977fe640 1 --2- 172.21.15.62:0/3483528946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 0x55b07942d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.777+0000 7f56977fe640 1 -- 172.21.15.62:0/3483528946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b079439af0 msgr2=0x55b07944a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.777+0000 7f56977fe640 1 --2- 172.21.15.62:0/3483528946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b079439af0 0x55b07944a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.777+0000 7f56977fe640 1 -- 172.21.15.62:0/3483528946 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b07942be30 con 0x55b0794391b0 2024-09-15T23:19:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.781+0000 7f56977fe640 1 --2- 172.21.15.62:0/3483528946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443760 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5690005d70 tx=0x7f5690003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a607ce867cf33de0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.784+0000 7f567d7fa640 1 -- 172.21.15.62:0/3483528946 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5690009b30 con 0x55b0794391b0 2024-09-15T23:19:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.784+0000 7f567d7fa640 1 -- 172.21.15.62:0/3483528946 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f569000f4e0 con 0x55b0794391b0 2024-09-15T23:19:36.785 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.784+0000 7f567d7fa640 1 -- 172.21.15.62:0/3483528946 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56900045f0 con 0x55b0794391b0 2024-09-15T23:19:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.785+0000 7f56a0672740 1 -- 172.21.15.62:0/3483528946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 msgr2=0x55b079443760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.785+0000 7f56a0672740 1 --2- 172.21.15.62:0/3483528946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443760 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5690005d70 tx=0x7f5690003ea0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.787+0000 7f56a0672740 1 -- 172.21.15.62:0/3483528946 shutdown_connections 2024-09-15T23:19:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.787+0000 7f56a0672740 1 --2- 172.21.15.62:0/3483528946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b07943eb80 0x55b07942d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.787+0000 7f56a0672740 1 --2- 172.21.15.62:0/3483528946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b079439af0 0x55b07944a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.787+0000 7f56a0672740 1 --2- 172.21.15.62:0/3483528946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b0794391b0 0x55b079443760 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.787+0000 7f56a0672740 1 -- 172.21.15.62:0/3483528946 >> 172.21.15.62:0/3483528946 conn(0x55b07942c010 msgr2=0x55b07942b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:36.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.795+0000 7f56a0672740 1 -- 172.21.15.62:0/3483528946 shutdown_connections 2024-09-15T23:19:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.797+0000 7f56a0672740 1 -- 172.21.15.62:0/3483528946 wait complete. 2024-09-15T23:19:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.798+0000 7f56a0672740 1 -- start start 2024-09-15T23:19:36.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.798+0000 7f56a0672740 1 Processor -- start 2024-09-15T23:19:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56a0672740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 0x55b079445a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56a0672740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079445f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56a0672740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b07943eb80 0x55b079448f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56a0672740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f56981a6ae0 con 0x55b07943eb80 2024-09-15T23:19:36.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56a0672740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f56981a7f50 con 0x55b0794391b0 2024-09-15T23:19:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56a0672740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f56981a71b0 con 0x55b079439af0 2024-09-15T23:19:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56a0672740 10 client.0 _ll_register_callbacks cb 0x7f568c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56967fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079445f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56967fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079445f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:54614/0 (socket says 172.21.15.62:54614) 2024-09-15T23:19:36.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.802+0000 7f56967fc640 1 -- 172.21.15.62:0/2328779744 learned_addr learned my addr 172.21.15.62:0/2328779744 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.804+0000 7f56967fc640 1 -- 172.21.15.62:0/2328779744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 msgr2=0x55b079445a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.804+0000 7f56967fc640 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 0x55b079445a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.804+0000 7f56967fc640 1 -- 172.21.15.62:0/2328779744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b07943eb80 msgr2=0x55b079448f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.804+0000 7f56967fc640 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b07943eb80 0x55b079448f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.804+0000 7f56967fc640 1 -- 172.21.15.62:0/2328779744 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b079449550 con 0x55b079439af0 2024-09-15T23:19:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.812+0000 7f56967fc640 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079445f70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f568c0314e0 tx=0x7f568c005600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaf49200916b3231 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.812+0000 7f5676ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:19:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.819+0000 7f5676ffd640 1 -- 172.21.15.62:0/2328779744 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568c035040 con 0x55b079439af0 2024-09-15T23:19:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.819+0000 7f5676ffd640 1 -- 172.21.15.62:0/2328779744 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f568c031b80 con 0x55b079439af0 2024-09-15T23:19:36.820 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.819+0000 7f56a0672740 1 -- 172.21.15.62:0/2328779744 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f569000f4e0 con 0x55b079439af0 2024-09-15T23:19:36.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.827+0000 7f5676ffd640 1 -- 172.21.15.62:0/2328779744 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f568c031d80 con 0x55b079439af0 2024-09-15T23:19:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.830+0000 7f56a0672740 10 client.25717 fetch_fsmap learned FSMap version 28 2024-09-15T23:19:36.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.830+0000 7f56a0672740 1 -- 172.21.15.62:0/2328779744 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b07944a960 con 0x55b079439af0 2024-09-15T23:19:36.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.832+0000 7f5676ffd640 1 -- 172.21.15.62:0/2328779744 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f569000f4e0 con 0x55b079439af0 2024-09-15T23:19:36.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.832+0000 7f5676ffd640 1 -- 172.21.15.62:0/2328779744 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 28) ==== 8366+0+0 (secure 0 0 0) 0x7f568c047660 con 0x55b079439af0 2024-09-15T23:19:36.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.832+0000 7f56a0672740 10 client.25717 fetch_fsmap finished waiting for FSMap version 28 2024-09-15T23:19:36.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.832+0000 7f56a0672740 10 client.25717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:36.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.832+0000 7f56a0672740 20 client.25717 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:36.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.832+0000 7f56a0672740 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b0794caf80 0x55b0794cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:36.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.832+0000 7f56a0672740 4 client.25717 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:19:36.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.832+0000 7f56a0672740 1 -- 172.21.15.62:0/2328779744 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b07939bd70 con 0x55b0794caf80 2024-09-15T23:19:36.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.834+0000 7f56977fe640 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b0794caf80 0x55b0794cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:36.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.836+0000 7f56977fe640 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b0794caf80 0x55b0794cd3a0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f930691bcddb2c7 server_cookie=1e55c19426d95332 in_seq=0 out_seq=0 2024-09-15T23:19:36.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.837+0000 7f5676ffd640 10 client.25717 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:36.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.862+0000 7f5676ffd640 1 -- 172.21.15.62:0/2328779744 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b07939bd70 con 0x55b0794caf80 2024-09-15T23:19:36.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.862+0000 7f5676ffd640 10 client.25717 handle_command_reply: tid=0 2024-09-15T23:19:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.863+0000 7f56a0672740 1 client.25717 shutdown 2024-09-15T23:19:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.864+0000 7f56a0672740 1 -- 172.21.15.62:0/2328779744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 msgr2=0x55b079445f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:36.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.864+0000 7f56a0672740 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079445f70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f568c0314e0 tx=0x7f568c005600 comp rx=0 tx=0).stop 2024-09-15T23:19:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.865+0000 7f56a0672740 1 -- 172.21.15.62:0/2328779744 shutdown_connections 2024-09-15T23:19:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.865+0000 7f56a0672740 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b0794caf80 0x55b0794cd3a0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.865+0000 7f56a0672740 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b07943eb80 0x55b079448f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.865+0000 7f56a0672740 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b079439af0 0x55b079445f70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.865+0000 7f56a0672740 1 --2- 172.21.15.62:0/2328779744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0794391b0 0x55b079445a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.865+0000 7f56a0672740 1 -- 172.21.15.62:0/2328779744 >> 172.21.15.62:0/2328779744 conn(0x55b07942c010 msgr2=0x55b07942c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:36.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.865+0000 7f5676ffd640 0 client.25717 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.866+0000 7f56a0672740 1 -- 172.21.15.62:0/2328779744 shutdown_connections 2024-09-15T23:19:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.866+0000 7f56a0672740 1 -- 172.21.15.62:0/2328779744 wait complete. 2024-09-15T23:19:36.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.866+0000 7f56a0672740 20 client.25717 trim_cache size 0 max 16384 2024-09-15T23:19:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:19:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:19:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:19:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:19:36.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 -- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f56800a33c0 msgr2=0x7f56800a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f56800a33c0 0x7f56800a5880 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5688097db0 tx=0x7f5688086040 comp rx=0 tx=0).stop 2024-09-15T23:19:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 -- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56980ffe30 msgr2=0x7f5698195bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56980ffe30 0x7f5698195bd0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f568800d950 tx=0x7f568800de10 comp rx=0 tx=0).stop 2024-09-15T23:19:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 -- 172.21.15.62:0/2828744723 shutdown_connections 2024-09-15T23:19:36.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f56800a33c0 0x7f56800a5880 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5698103860 0x7f569819d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5698102ea0 0x7f5698196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 --2- 172.21.15.62:0/2828744723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56980ffe30 0x7f5698195bd0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 -- 172.21.15.62:0/2828744723 >> 172.21.15.62:0/2828744723 conn(0x7f56980fb930 msgr2=0x7f56981061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 -- 172.21.15.62:0/2828744723 shutdown_connections 2024-09-15T23:19:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:36.867+0000 7f569d8d7640 1 -- 172.21.15.62:0/2828744723 wait complete. 2024-09-15T23:19:37.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:36 smithi062 ceph-mon[30949]: pgmap v470: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 704 op/s 2024-09-15T23:19:37.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:36 smithi067 ceph-mon[35665]: pgmap v470: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 704 op/s 2024-09-15T23:19:37.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:36 smithi148 ceph-mon[35610]: pgmap v470: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 704 op/s 2024-09-15T23:19:39.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:38 smithi062 ceph-mon[30949]: pgmap v471: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.4 MiB/s wr, 578 op/s 2024-09-15T23:19:39.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:38 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:19:39.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:38 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:38 smithi067 ceph-mon[35665]: pgmap v471: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.4 MiB/s wr, 578 op/s 2024-09-15T23:19:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:38 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:19:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:38 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:38 smithi148 ceph-mon[35610]: pgmap v471: 129 pgs: 129 active+clean; 1.1 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.4 MiB/s wr, 578 op/s 2024-09-15T23:19:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:38 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:19:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:38 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:19:39.453 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:19:39.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 200.0 s, 1871.8 tps, lat 53.660 ms stddev 54.774 2024-09-15T23:19:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:40.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:40.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:40.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:19:40.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:40.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:40.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:19:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:19:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:19:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:19:41.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:41 smithi062 ceph-mon[30949]: pgmap v472: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.1 MiB/s wr, 728 op/s 2024-09-15T23:19:41.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:41 smithi067 ceph-mon[35665]: pgmap v472: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.1 MiB/s wr, 728 op/s 2024-09-15T23:19:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:41 smithi148 ceph-mon[35610]: pgmap v472: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.1 MiB/s wr, 728 op/s 2024-09-15T23:19:42.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.062+0000 7f68a087a640 1 -- 172.21.15.62:0/3032351102 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f689c100350 msgr2=0x7f689c1109f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:42.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.062+0000 7f68a087a640 1 --2- 172.21.15.62:0/3032351102 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f689c100350 0x7f689c1109f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6894009cd0 tx=0x7f689402a8f0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.064+0000 7f68a087a640 1 -- 172.21.15.62:0/3032351102 shutdown_connections 2024-09-15T23:19:42.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.064+0000 7f68a087a640 1 --2- 172.21.15.62:0/3032351102 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f689c100350 0x7f689c1109f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.064+0000 7f68a087a640 1 --2- 172.21.15.62:0/3032351102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c0ff9d0 0x7f689c0ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.064+0000 7f68a087a640 1 --2- 172.21.15.62:0/3032351102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689c102800 0x7f689c102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.064+0000 7f68a087a640 1 -- 172.21.15.62:0/3032351102 >> 172.21.15.62:0/3032351102 conn(0x7f689c0fb920 msgr2=0x7f689c0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:42.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.064+0000 7f68a087a640 1 -- 172.21.15.62:0/3032351102 shutdown_connections 2024-09-15T23:19:42.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.064+0000 7f68a087a640 1 -- 172.21.15.62:0/3032351102 wait complete. 2024-09-15T23:19:42.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.066+0000 7f68a087a640 1 Processor -- start 2024-09-15T23:19:42.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.066+0000 7f68a087a640 1 -- start start 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.067+0000 7f68a087a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c0ff9d0 0x7f689c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.067+0000 7f68a087a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f689c100350 0x7f689c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.067+0000 7f68a087a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689c102800 0x7f689c19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.067+0000 7f68a087a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f689c113fc0 con 0x7f689c0ff9d0 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.067+0000 7f68a087a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f689c113e40 con 0x7f689c102800 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.067+0000 7f68a087a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f689c114140 con 0x7f689c100350 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.067+0000 7f689a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f689c100350 0x7f689c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.067+0000 7f689a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f689c100350 0x7f689c19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:54630/0 (socket says 172.21.15.62:54630) 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.067+0000 7f689a59c640 1 -- 172.21.15.62:0/2889415436 learned_addr learned my addr 172.21.15.62:0/2889415436 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.068+0000 7f689ad9d640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c0ff9d0 0x7f689c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.068+0000 7f689b59e640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689c102800 0x7f689c19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.068+0000 7f689ad9d640 1 -- 172.21.15.62:0/2889415436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f689c100350 msgr2=0x7f689c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:42.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.068+0000 7f689ad9d640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f689c100350 0x7f689c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.068+0000 7f689ad9d640 1 -- 172.21.15.62:0/2889415436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689c102800 msgr2=0x7f689c19eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:42.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.068+0000 7f689ad9d640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689c102800 0x7f689c19eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.068+0000 7f689ad9d640 1 -- 172.21.15.62:0/2889415436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689c19f1f0 con 0x7f689c0ff9d0 2024-09-15T23:19:42.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.069+0000 7f689ad9d640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c0ff9d0 0x7f689c19a240 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f688c00e9f0 tx=0x7f688c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:42.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.069+0000 7f6883fff640 1 -- 172.21.15.62:0/2889415436 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c00ccc0 con 0x7f689c0ff9d0 2024-09-15T23:19:42.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.069+0000 7f68a087a640 1 -- 172.21.15.62:0/2889415436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689c19f4e0 con 0x7f689c0ff9d0 2024-09-15T23:19:42.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.069+0000 7f6883fff640 1 -- 172.21.15.62:0/2889415436 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f688c010910 con 0x7f689c0ff9d0 2024-09-15T23:19:42.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.069+0000 7f68a087a640 1 -- 172.21.15.62:0/2889415436 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f689c1a6dc0 con 0x7f689c0ff9d0 2024-09-15T23:19:42.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.071+0000 7f6883fff640 1 -- 172.21.15.62:0/2889415436 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c026060 con 0x7f689c0ff9d0 2024-09-15T23:19:42.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.073+0000 7f68a3454740 1 Processor -- start 2024-09-15T23:19:42.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.073+0000 7f68a3454740 1 -- start start 2024-09-15T23:19:42.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f68a3454740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d3754ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f68a3454740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d37557310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f68a3454740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37559c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f68a3454740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557d374b22e0 con 0x557d3754cf90 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f68a3454740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557d37459ce0 con 0x557d3754c670 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f68a3454740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557d374ae2a0 con 0x557d37557850 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f689b59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d3754ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f689b59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d3754ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:37556/0 (socket says 172.21.15.62:37556) 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f689b59e640 1 -- 172.21.15.62:0/2684657951 learned_addr learned my addr 172.21.15.62:0/2684657951 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f689a59c640 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37559c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.074+0000 7f6883fff640 1 -- 172.21.15.62:0/2889415436 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f688c00f3c0 con 0x7f689c0ff9d0 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.075+0000 7f689a59c640 1 -- 172.21.15.62:0/2684657951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 msgr2=0x557d3754ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.075+0000 7f689a59c640 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d3754ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.075+0000 7f689a59c640 1 -- 172.21.15.62:0/2684657951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 msgr2=0x557d37557310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.075+0000 7f689ad9d640 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d37557310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.075+0000 7f6883fff640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f68840a3280 0x7f68840a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.076+0000 7f6883fff640 1 -- 172.21.15.62:0/2889415436 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f688c0d0130 con 0x7f689c0ff9d0 2024-09-15T23:19:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.076+0000 7f689a59c640 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d37557310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.076+0000 7f689a59c640 1 -- 172.21.15.62:0/2684657951 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d3743e440 con 0x557d37557850 2024-09-15T23:19:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.076+0000 7f689b59e640 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d3754ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.076+0000 7f689ad9d640 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d37557310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:19:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.077+0000 7f689b59e640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f68840a3280 0x7f68840a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.077+0000 7f689a59c640 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37559c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6890009c00 tx=0x7f68900117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c06b407480ff950a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:42.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.077+0000 7f689b59e640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f68840a3280 0x7f68840a5740 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6894004af0 tx=0x7f6894005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:42.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.079+0000 7f68817fa640 1 -- 172.21.15.62:0/2684657951 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6890012450 con 0x557d37557850 2024-09-15T23:19:42.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.079+0000 7f68817fa640 1 -- 172.21.15.62:0/2684657951 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6890012ad0 con 0x557d37557850 2024-09-15T23:19:42.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.079+0000 7f68817fa640 1 -- 172.21.15.62:0/2684657951 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689000fbe0 con 0x557d37557850 2024-09-15T23:19:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.080+0000 7f68a3454740 1 -- 172.21.15.62:0/2684657951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 msgr2=0x557d37559c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.080+0000 7f68a3454740 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37559c40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6890009c00 tx=0x7f68900117b0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.080+0000 7f68a3454740 1 -- 172.21.15.62:0/2684657951 shutdown_connections 2024-09-15T23:19:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.080+0000 7f68a3454740 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37559c40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.080+0000 7f68a3454740 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d37557310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.080+0000 7f68a3454740 1 --2- 172.21.15.62:0/2684657951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d3754ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.080+0000 7f68a3454740 1 -- 172.21.15.62:0/2684657951 >> 172.21.15.62:0/2684657951 conn(0x557d37548c00 msgr2=0x557d37548fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.080+0000 7f68a3454740 1 -- 172.21.15.62:0/2684657951 shutdown_connections 2024-09-15T23:19:42.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.080+0000 7f68a3454740 1 -- 172.21.15.62:0/2684657951 wait complete. 2024-09-15T23:19:42.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.081+0000 7f68a3454740 1 -- start start 2024-09-15T23:19:42.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.081+0000 7f68a3454740 1 Processor -- start 2024-09-15T23:19:42.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.081+0000 7f68a3454740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754c670 0x557d37557240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.081+0000 7f68a3454740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754cf90 0x557d375544f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.081+0000 7f68a3454740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37554ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.081+0000 7f68a3454740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557d3755a420 con 0x557d3754c670 2024-09-15T23:19:42.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.081+0000 7f68a3454740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557d3746d050 con 0x557d3754cf90 2024-09-15T23:19:42.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.081+0000 7f68a3454740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557d3755a5a0 con 0x557d37557850 2024-09-15T23:19:42.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754c670 0x557d37557240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f68a3454740 10 client.0 _ll_register_callbacks cb 0x557d374ad160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754c670 0x557d37557240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56202/0 (socket says 172.21.15.62:56202) 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689ad9d640 1 -- 172.21.15.62:0/4255343780 learned_addr learned my addr 172.21.15.62:0/4255343780 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689b59e640 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37554ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689a59c640 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754cf90 0x557d375544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689b59e640 1 -- 172.21.15.62:0/4255343780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754cf90 msgr2=0x557d375544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689b59e640 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754cf90 0x557d375544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689b59e640 1 -- 172.21.15.62:0/4255343780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754c670 msgr2=0x557d37557240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689b59e640 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754c670 0x557d37557240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689b59e640 1 -- 172.21.15.62:0/4255343780 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d372233c0 con 0x557d37557850 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.082+0000 7f689ad9d640 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754c670 0x557d37557240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.083+0000 7f689b59e640 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37554ae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f689402aee0 tx=0x7f689402a8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dc8fa85d44d59b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:42.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.083+0000 7f687eff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.083+0000 7f687eff5640 1 -- 172.21.15.62:0/4255343780 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6894030b40 con 0x557d37557850 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.083+0000 7f687eff5640 1 -- 172.21.15.62:0/4255343780 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6894030ce0 con 0x557d37557850 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.083+0000 7f68a3454740 1 -- 172.21.15.62:0/4255343780 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6890012ad0 con 0x557d37557850 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.083+0000 7f687eff5640 1 -- 172.21.15.62:0/4255343780 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6894053440 con 0x557d37557850 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.083+0000 7f68a3454740 10 client.25729 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.083+0000 7f68a3454740 1 -- 172.21.15.62:0/4255343780 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557d37565df0 con 0x557d37557850 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.084+0000 7f687eff5640 1 -- 172.21.15.62:0/4255343780 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6890012ad0 con 0x557d37557850 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.084+0000 7f687eff5640 1 -- 172.21.15.62:0/4255343780 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6894051070 con 0x557d37557850 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.084+0000 7f68a3454740 10 client.25729 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.084+0000 7f68a3454740 10 client.25729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.084+0000 7f68a3454740 20 client.25729 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.084+0000 7f68a3454740 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557d37567200 0x557d37569630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.084+0000 7f68a3454740 4 client.25729 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:42.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.084+0000 7f68a3454740 1 -- 172.21.15.62:0/4255343780 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d374bf850 con 0x557d37567200 2024-09-15T23:19:42.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.084+0000 7f689ad9d640 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557d37567200 0x557d37569630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.085+0000 7f689ad9d640 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557d37567200 0x557d37569630 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c08ee3b647d74a9d server_cookie=ad300c09260ed470 in_seq=0 out_seq=0 2024-09-15T23:19:42.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.085+0000 7f687eff5640 10 client.25729 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.085+0000 7f687eff5640 1 -- 172.21.15.62:0/4255343780 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557d374bf850 con 0x557d37567200 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.085+0000 7f687eff5640 10 client.25729 handle_command_reply: tid=0 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.085+0000 7f68a3454740 1 client.25729 shutdown 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.086+0000 7f68a3454740 1 -- 172.21.15.62:0/4255343780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 msgr2=0x557d37554ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.086+0000 7f68a3454740 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37554ae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f689402aee0 tx=0x7f689402a8f0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.086+0000 7f68a3454740 1 -- 172.21.15.62:0/4255343780 shutdown_connections 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.086+0000 7f68a3454740 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557d37567200 0x557d37569630 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.086+0000 7f68a3454740 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d37554ae0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.086+0000 7f68a3454740 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754cf90 0x557d375544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.086+0000 7f68a3454740 1 --2- 172.21.15.62:0/4255343780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754c670 0x557d37557240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.086+0000 7f68a3454740 1 -- 172.21.15.62:0/4255343780 >> 172.21.15.62:0/4255343780 conn(0x557d37548c00 msgr2=0x557d3754e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:42.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.086+0000 7f687eff5640 0 client.25729 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:42.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.087+0000 7f68a3454740 1 -- 172.21.15.62:0/4255343780 shutdown_connections 2024-09-15T23:19:42.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.087+0000 7f68a3454740 1 -- 172.21.15.62:0/4255343780 wait complete. 2024-09-15T23:19:42.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.087+0000 7f68a3454740 20 client.25729 trim_cache size 0 max 16384 2024-09-15T23:19:42.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.113+0000 7f68a3454740 1 Processor -- start 2024-09-15T23:19:42.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.113+0000 7f68a3454740 1 -- start start 2024-09-15T23:19:42.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f68a3454740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d37547d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f68a3454740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3756ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f68a3454740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d3756f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f68a3454740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f689c1af2b0 con 0x557d3754cf90 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f68a3454740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f689c1acb80 con 0x557d3754c670 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f68a3454740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f689c1aa9f0 con 0x557d37557850 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3756ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d37547d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689ad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d37547d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:37582/0 (socket says 172.21.15.62:37582) 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3756ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56216/0 (socket says 172.21.15.62:56216) 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689ad9d640 1 -- 172.21.15.62:0/1917032919 learned_addr learned my addr 172.21.15.62:0/1917032919 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689b59e640 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d3756f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689ad9d640 1 -- 172.21.15.62:0/1917032919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 msgr2=0x557d3756f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689ad9d640 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d3756f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689ad9d640 1 -- 172.21.15.62:0/1917032919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 msgr2=0x557d3756ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689ad9d640 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3756ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689ad9d640 1 -- 172.21.15.62:0/1917032919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d37553520 con 0x557d3754c670 2024-09-15T23:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689b59e640 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d3756f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689a59c640 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3756ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.114+0000 7f689ad9d640 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d37547d20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f688c098d50 tx=0x7f688c0cd780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=521ab9b968c4f6f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.115+0000 7f6880ff9640 1 -- 172.21.15.62:0/1917032919 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c0a0780 con 0x557d3754c670 2024-09-15T23:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.115+0000 7f6880ff9640 1 -- 172.21.15.62:0/1917032919 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f688c0a5040 con 0x557d3754c670 2024-09-15T23:19:42.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.115+0000 7f6880ff9640 1 -- 172.21.15.62:0/1917032919 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c09ba80 con 0x557d3754c670 2024-09-15T23:19:42.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 -- 172.21.15.62:0/1917032919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 msgr2=0x557d37547d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:42.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d37547d20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f688c098d50 tx=0x7f688c0cd780 comp rx=0 tx=0).stop 2024-09-15T23:19:42.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 -- 172.21.15.62:0/1917032919 shutdown_connections 2024-09-15T23:19:42.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d37557850 0x557d3756f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3756ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 --2- 172.21.15.62:0/1917032919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d3754c670 0x557d37547d20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 -- 172.21.15.62:0/1917032919 >> 172.21.15.62:0/1917032919 conn(0x557d37548c00 msgr2=0x557d37543b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:42.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 -- 172.21.15.62:0/1917032919 shutdown_connections 2024-09-15T23:19:42.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 -- 172.21.15.62:0/1917032919 wait complete. 2024-09-15T23:19:42.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.116+0000 7f68a3454740 1 -- start start 2024-09-15T23:19:42.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f68a3454740 1 Processor -- start 2024-09-15T23:19:42.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f68a3454740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d3754c670 0x557d3755dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f68a3454740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3755e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f68a3454740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d37557850 0x557d375caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f68a3454740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f689c1ab040 con 0x557d3754cf90 2024-09-15T23:19:42.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f68a3454740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f689c1ac4b0 con 0x557d37557850 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f68a3454740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f689c1ab710 con 0x557d3754c670 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f68a3454740 10 client.0 _ll_register_callbacks cb 0x7f688c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f689a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3755e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f689b59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d37557850 0x557d375caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f689b59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d37557850 0x557d375caeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:37596/0 (socket says 172.21.15.62:37596) 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.117+0000 7f689b59e640 1 -- 172.21.15.62:0/1854194964 learned_addr learned my addr 172.21.15.62:0/1854194964 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f689ad9d640 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d3754c670 0x557d3755dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f689b59e640 1 -- 172.21.15.62:0/1854194964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d3754c670 msgr2=0x557d3755dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f689b59e640 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d3754c670 0x557d3755dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f689b59e640 1 -- 172.21.15.62:0/1854194964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 msgr2=0x557d3755e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f689b59e640 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3755e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f689b59e640 1 -- 172.21.15.62:0/1854194964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d375cb4f0 con 0x557d37557850 2024-09-15T23:19:42.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f689a59c640 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3755e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f689b59e640 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d37557850 0x557d375caeb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6894005cc0 tx=0x7f689402afb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a160f63747f5c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f687eff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f687eff5640 1 -- 172.21.15.62:0/1854194964 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6894004520 con 0x557d37557850 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f687eff5640 1 -- 172.21.15.62:0/1854194964 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f68940046c0 con 0x557d37557850 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f68a3454740 1 -- 172.21.15.62:0/1854194964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f688c0a5040 con 0x557d37557850 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f687eff5640 1 -- 172.21.15.62:0/1854194964 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6894030b40 con 0x557d37557850 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f68a3454740 10 client.25889 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.118+0000 7f68a3454740 1 -- 172.21.15.62:0/1854194964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557d375d2830 con 0x557d37557850 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.119+0000 7f687eff5640 1 -- 172.21.15.62:0/1854194964 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f688c0a5040 con 0x557d37557850 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.119+0000 7f687eff5640 1 -- 172.21.15.62:0/1854194964 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6894051040 con 0x557d37557850 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.119+0000 7f68a3454740 10 client.25889 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.119+0000 7f68a3454740 10 client.25889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:42.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.119+0000 7f68a3454740 20 client.25889 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:42.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.119+0000 7f68a3454740 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557d375d2fd0 0x557d375d53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:42.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.119+0000 7f68a3454740 4 client.25889 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:19:42.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.119+0000 7f68a3454740 1 -- 172.21.15.62:0/1854194964 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557d374b9d60 con 0x557d375d2fd0 2024-09-15T23:19:42.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.120+0000 7f689ad9d640 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557d375d2fd0 0x557d375d53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:42.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.120+0000 7f689ad9d640 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557d375d2fd0 0x557d375d53f0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a083c64c5492e7ce server_cookie=d5fa80988a3aa18f in_seq=0 out_seq=0 2024-09-15T23:19:42.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.120+0000 7f687eff5640 10 client.25889 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:42.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f687eff5640 1 -- 172.21.15.62:0/1854194964 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557d374b9d60 con 0x557d375d2fd0 2024-09-15T23:19:42.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f687eff5640 10 client.25889 handle_command_reply: tid=0 2024-09-15T23:19:42.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 client.25889 shutdown 2024-09-15T23:19:42.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 -- 172.21.15.62:0/1854194964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d37557850 msgr2=0x557d375caeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d37557850 0x557d375caeb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6894005cc0 tx=0x7f689402afb0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 -- 172.21.15.62:0/1854194964 shutdown_connections 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557d375d2fd0 0x557d375d53f0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f687eff5640 0 client.25889 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d37557850 0x557d375caeb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d3754cf90 0x557d3755e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 --2- 172.21.15.62:0/1854194964 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557d3754c670 0x557d3755dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 -- 172.21.15.62:0/1854194964 >> 172.21.15.62:0/1854194964 conn(0x557d37548c00 msgr2=0x557d37556570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 -- 172.21.15.62:0/1854194964 shutdown_connections 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 1 -- 172.21.15.62:0/1854194964 wait complete. 2024-09-15T23:19:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.121+0000 7f68a3454740 20 client.25889 trim_cache size 0 max 16384 2024-09-15T23:19:42.126 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:19:42.126 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:19:42.126 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:19:42.126 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:19:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 -- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f68840a3280 msgr2=0x7f68840a5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f68840a3280 0x7f68840a5740 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6894004af0 tx=0x7f6894005d80 comp rx=0 tx=0).stop 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 -- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c0ff9d0 msgr2=0x7f689c19a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c0ff9d0 0x7f689c19a240 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f688c00e9f0 tx=0x7f688c00eeb0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 -- 172.21.15.62:0/2889415436 shutdown_connections 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f68840a3280 0x7f68840a5740 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f689c102800 0x7f689c19eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f689c100350 0x7f689c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 --2- 172.21.15.62:0/2889415436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c0ff9d0 0x7f689c19a240 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 -- 172.21.15.62:0/2889415436 >> 172.21.15.62:0/2889415436 conn(0x7f689c0fb920 msgr2=0x7f689c0fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 -- 172.21.15.62:0/2889415436 shutdown_connections 2024-09-15T23:19:42.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:42.122+0000 7f68a087a640 1 -- 172.21.15.62:0/2889415436 wait complete. 2024-09-15T23:19:43.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:43 smithi062 ceph-mon[30949]: pgmap v473: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 727 op/s 2024-09-15T23:19:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:43 smithi067 ceph-mon[35665]: pgmap v473: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 727 op/s 2024-09-15T23:19:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:43 smithi148 ceph-mon[35610]: pgmap v473: 129 pgs: 129 active+clean; 1.1 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 MiB/s wr, 727 op/s 2024-09-15T23:19:43.768 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:19:43.768 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:19:44.209 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:44.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 205.0 s, 1759.0 tps, lat 54.576 ms stddev 67.030 2024-09-15T23:19:44.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.937+0000 7f02b9243640 1 -- 172.21.15.62:0/2529132972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4102d40 msgr2=0x7f02b4103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:44.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.937+0000 7f02b9243640 1 --2- 172.21.15.62:0/2529132972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4102d40 0x7f02b4103120 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009900 tx=0x7f02a402a480 comp rx=0 tx=0).stop 2024-09-15T23:19:44.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.938+0000 7f02b9243640 1 -- 172.21.15.62:0/2529132972 shutdown_connections 2024-09-15T23:19:44.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.938+0000 7f02b9243640 1 --2- 172.21.15.62:0/2529132972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b410c170 0x7f02b410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:44.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.938+0000 7f02b9243640 1 --2- 172.21.15.62:0/2529132972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02b4103660 0x7f02b410bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:44.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.938+0000 7f02b9243640 1 --2- 172.21.15.62:0/2529132972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b4102d40 0x7f02b4103120 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:44.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.938+0000 7f02b9243640 1 -- 172.21.15.62:0/2529132972 >> 172.21.15.62:0/2529132972 conn(0x7f02b40fc7a0 msgr2=0x7f02b40febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:44.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.938+0000 7f02b9243640 1 -- 172.21.15.62:0/2529132972 shutdown_connections 2024-09-15T23:19:44.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.939+0000 7f02b9243640 1 -- 172.21.15.62:0/2529132972 wait complete. 2024-09-15T23:19:44.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.940+0000 7f02b9243640 1 Processor -- start 2024-09-15T23:19:44.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b9243640 1 -- start start 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b9243640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b4102d40 0x7f02b4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b9243640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02b4103660 0x7f02b4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b9243640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b410c170 0x7f02b419cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b9243640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f02b4111f70 con 0x7f02b410c170 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b9243640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f02b4111df0 con 0x7f02b4102d40 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b9243640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f02b41120f0 con 0x7f02b4103660 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b359e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b410c170 0x7f02b419cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b359e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b410c170 0x7f02b419cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52846/0 (socket says 172.21.15.62:52846) 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.941+0000 7f02b359e640 1 -- 172.21.15.62:0/1440814020 learned_addr learned my addr 172.21.15.62:0/1440814020 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.942+0000 7f02b259c640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02b4103660 0x7f02b4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:44.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.942+0000 7f02b2d9d640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b4102d40 0x7f02b4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.942+0000 7f02b359e640 1 -- 172.21.15.62:0/1440814020 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02b4103660 msgr2=0x7f02b4198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.942+0000 7f02b359e640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02b4103660 0x7f02b4198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.942+0000 7f02b359e640 1 -- 172.21.15.62:0/1440814020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b4102d40 msgr2=0x7f02b4198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.942+0000 7f02b359e640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b4102d40 0x7f02b4198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.942+0000 7f02b359e640 1 -- 172.21.15.62:0/1440814020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b419d3e0 con 0x7f02b410c170 2024-09-15T23:19:44.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.942+0000 7f02b2d9d640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b4102d40 0x7f02b4198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:44.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.943+0000 7f02b359e640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b410c170 0x7f02b419cd00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00efd0 tx=0x7f02ac00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:44.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.943+0000 7f029bfff640 1 -- 172.21.15.62:0/1440814020 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac003b70 con 0x7f02b410c170 2024-09-15T23:19:44.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.943+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b419d6d0 con 0x7f02b410c170 2024-09-15T23:19:44.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.943+0000 7f029bfff640 1 -- 172.21.15.62:0/1440814020 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ac003d10 con 0x7f02b410c170 2024-09-15T23:19:44.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.943+0000 7f029bfff640 1 -- 172.21.15.62:0/1440814020 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0109f0 con 0x7f02b410c170 2024-09-15T23:19:44.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.943+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b41008d0 con 0x7f02b410c170 2024-09-15T23:19:44.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.944+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02b4103120 con 0x7f02b410c170 2024-09-15T23:19:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.946+0000 7f029bfff640 1 -- 172.21.15.62:0/1440814020 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f02ac010b90 con 0x7f02b410c170 2024-09-15T23:19:44.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.946+0000 7f029bfff640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f029c0a3340 0x7f029c0a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:44.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.947+0000 7f029bfff640 1 -- 172.21.15.62:0/1440814020 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f02ac0d0420 con 0x7f02b410c170 2024-09-15T23:19:44.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.948+0000 7f02b2d9d640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f029c0a3340 0x7f029c0a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:44.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.950+0000 7f02b2d9d640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f029c0a3340 0x7f029c0a5800 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009230 tx=0x7f02a403c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:44.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:44.954+0000 7f029bfff640 1 -- 172.21.15.62:0/1440814020 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f02ac09cd60 con 0x7f02b410c170 2024-09-15T23:19:45.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:45 smithi062 ceph-mon[30949]: pgmap v474: 129 pgs: 129 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.5 MiB/s wr, 729 op/s 2024-09-15T23:19:45.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.325+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02b4199790 con 0x7f02b410c170 2024-09-15T23:19:45.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.326+0000 7f029bfff640 1 -- 172.21.15.62:0/1440814020 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11906 (secure 0 0 0) 0x7f02b4199790 con 0x7f02b410c170 2024-09-15T23:19:45.327 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:19:45.328 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":29,"btime":"2024-09-15T23:19:37:942586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:19:37.876324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:19:45.328 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.328+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f029c0a3340 msgr2=0x7f029c0a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.328+0000 7f02b9243640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f029c0a3340 0x7f029c0a5800 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009230 tx=0x7f02a403c040 comp rx=0 tx=0).stop 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.328+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b410c170 msgr2=0x7f02b419cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.328+0000 7f02b9243640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b410c170 0x7f02b419cd00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00efd0 tx=0x7f02ac00c6a0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.328+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 shutdown_connections 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.328+0000 7f02b9243640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f029c0a3340 0x7f029c0a5800 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.329+0000 7f02b9243640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f02b410c170 0x7f02b419cd00 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.329+0000 7f02b9243640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f02b4103660 0x7f02b4198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.329+0000 7f02b9243640 1 --2- 172.21.15.62:0/1440814020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02b4102d40 0x7f02b4198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:45.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.329+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 >> 172.21.15.62:0/1440814020 conn(0x7f02b40fc7a0 msgr2=0x7f02b40fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:45.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.329+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 shutdown_connections 2024-09-15T23:19:45.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:45.329+0000 7f02b9243640 1 -- 172.21.15.62:0/1440814020 wait complete. 2024-09-15T23:19:45.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:45 smithi067 ceph-mon[35665]: pgmap v474: 129 pgs: 129 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.5 MiB/s wr, 729 op/s 2024-09-15T23:19:45.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:45 smithi148 ceph-mon[35610]: pgmap v474: 129 pgs: 129 active+clean; 1.1 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.5 MiB/s wr, 729 op/s 2024-09-15T23:19:46.432 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:46 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1440814020' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:19:46.475 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:19:46.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:46 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1440814020' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:19:46.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:46 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1440814020' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:19:46.890 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:47.327 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:47 smithi062 ceph-mon[30949]: pgmap v475: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 MiB/s wr, 747 op/s 2024-09-15T23:19:47.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.471+0000 7f43d874c640 1 -- 172.21.15.62:0/2560247557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43d0102d30 msgr2=0x7f43d0103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:47.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.471+0000 7f43d874c640 1 --2- 172.21.15.62:0/2560247557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43d0102d30 0x7f43d0103110 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009860 tx=0x7f43cc02a3d0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.471+0000 7f43d874c640 1 -- 172.21.15.62:0/2560247557 shutdown_connections 2024-09-15T23:19:47.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.471+0000 7f43d874c640 1 --2- 172.21.15.62:0/2560247557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d010c160 0x7f43d010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.471+0000 7f43d874c640 1 --2- 172.21.15.62:0/2560247557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d0103650 0x7f43d010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.471+0000 7f43d874c640 1 --2- 172.21.15.62:0/2560247557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43d0102d30 0x7f43d0103110 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.471+0000 7f43d874c640 1 -- 172.21.15.62:0/2560247557 >> 172.21.15.62:0/2560247557 conn(0x7f43d00fc7d0 msgr2=0x7f43d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:47.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.471+0000 7f43d874c640 1 -- 172.21.15.62:0/2560247557 shutdown_connections 2024-09-15T23:19:47.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.471+0000 7f43d874c640 1 -- 172.21.15.62:0/2560247557 wait complete. 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.472+0000 7f43d874c640 1 Processor -- start 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.472+0000 7f43d874c640 1 -- start start 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.472+0000 7f43d874c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d0102d30 0x7f43d0100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.472+0000 7f43d874c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d0103650 0x7f43d0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d874c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43d010c160 0x7f43d0101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d874c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f43d0111fa0 con 0x7f43d0102d30 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d874c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43d0111e20 con 0x7f43d0103650 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d874c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f43d0112120 con 0x7f43d010c160 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d5ce7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d0103650 0x7f43d0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d64e8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d0102d30 0x7f43d0100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d5ce7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d0103650 0x7f43d0100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48440/0 (socket says 172.21.15.62:48440) 2024-09-15T23:19:47.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d64e8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d0102d30 0x7f43d0100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52864/0 (socket says 172.21.15.62:52864) 2024-09-15T23:19:47.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d5ce7640 1 -- 172.21.15.62:0/1968801074 learned_addr learned my addr 172.21.15.62:0/1968801074 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:47.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43d010c160 0x7f43d0101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d64e8640 1 -- 172.21.15.62:0/1968801074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43d010c160 msgr2=0x7f43d0101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:47.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d64e8640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43d010c160 0x7f43d0101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d64e8640 1 -- 172.21.15.62:0/1968801074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d0103650 msgr2=0x7f43d0100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:47.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d64e8640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d0103650 0x7f43d0100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.473+0000 7f43d64e8640 1 -- 172.21.15.62:0/1968801074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d01a69a0 con 0x7f43d0102d30 2024-09-15T23:19:47.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.475+0000 7f43d64e8640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d0102d30 0x7f43d0100990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f43cc02a8f0 tx=0x7f43cc030710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:47.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.475+0000 7f43c77fe640 1 -- 172.21.15.62:0/1968801074 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc005260 con 0x7f43d0102d30 2024-09-15T23:19:47.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.475+0000 7f43d874c640 1 -- 172.21.15.62:0/1968801074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d01a6c30 con 0x7f43d0102d30 2024-09-15T23:19:47.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.475+0000 7f43d874c640 1 -- 172.21.15.62:0/1968801074 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d01a7110 con 0x7f43d0102d30 2024-09-15T23:19:47.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.475+0000 7f43c77fe640 1 -- 172.21.15.62:0/1968801074 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43cc0318e0 con 0x7f43d0102d30 2024-09-15T23:19:47.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.475+0000 7f43c77fe640 1 -- 172.21.15.62:0/1968801074 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc004d00 con 0x7f43d0102d30 2024-09-15T23:19:47.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43dab25740 1 Processor -- start 2024-09-15T23:19:47.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43c77fe640 1 -- 172.21.15.62:0/1968801074 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f43cc039040 con 0x7f43d0102d30 2024-09-15T23:19:47.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43dab25740 1 -- start start 2024-09-15T23:19:47.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43dab25740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c478a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43dab25740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 0x557354c48220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43dab25740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c54f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43dab25740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557354ba52f0 con 0x557354c49350 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43dab25740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557354b4ccf0 con 0x557354c52b80 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43dab25740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557354ba12b0 con 0x557354c47de0 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43d6ce9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43d6ce9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c478a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52874/0 (socket says 172.21.15.62:52874) 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43d5ce7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c54f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.479+0000 7f43d6ce9640 1 -- 172.21.15.62:0/582117046 learned_addr learned my addr 172.21.15.62:0/582117046 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43d64e8640 1 --2- 172.21.15.62:0/582117046 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 0x557354c48220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43d6ce9640 1 -- 172.21.15.62:0/582117046 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 msgr2=0x557354c48220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/582117046 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 0x557354c48220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43d6ce9640 1 -- 172.21.15.62:0/582117046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 msgr2=0x557354c54f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/582117046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c54f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43d6ce9640 1 -- 172.21.15.62:0/582117046 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557354b313a0 con 0x557354c49350 2024-09-15T23:19:47.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43d5ce7640 1 --2- 172.21.15.62:0/582117046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c54f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:47.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43c77fe640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43bc0a3390 0x7f43bc0a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43d64e8640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43bc0a3390 0x7f43bc0a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/582117046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c478a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f43c800f480 tx=0x7f43c800f940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8a36abd7ee33b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:47.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43c77fe640 1 -- 172.21.15.62:0/1968801074 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f43cc0f1200 con 0x7f43d0102d30 2024-09-15T23:19:47.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43c4ff9640 1 -- 172.21.15.62:0/582117046 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c801d070 con 0x557354c49350 2024-09-15T23:19:47.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43c4ff9640 1 -- 172.21.15.62:0/582117046 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c800cb80 con 0x557354c49350 2024-09-15T23:19:47.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.480+0000 7f43c4ff9640 1 -- 172.21.15.62:0/582117046 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c8018d50 con 0x557354c49350 2024-09-15T23:19:47.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.481+0000 7f43dab25740 1 -- 172.21.15.62:0/582117046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 msgr2=0x557354c478a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:47.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.481+0000 7f43dab25740 1 --2- 172.21.15.62:0/582117046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c478a0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f43c800f480 tx=0x7f43c800f940 comp rx=0 tx=0).stop 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.481+0000 7f43d64e8640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43bc0a3390 0x7f43bc0a5850 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f43cc004830 tx=0x7f43cc0b9040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.482+0000 7f43dab25740 1 -- 172.21.15.62:0/582117046 shutdown_connections 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.482+0000 7f43dab25740 1 --2- 172.21.15.62:0/582117046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c54f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.482+0000 7f43dab25740 1 --2- 172.21.15.62:0/582117046 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 0x557354c48220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.482+0000 7f43dab25740 1 --2- 172.21.15.62:0/582117046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c478a0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.482+0000 7f43dab25740 1 -- 172.21.15.62:0/582117046 >> 172.21.15.62:0/582117046 conn(0x557354c08310 msgr2=0x557354c44470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.483+0000 7f43dab25740 1 -- 172.21.15.62:0/582117046 shutdown_connections 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.483+0000 7f43dab25740 1 -- 172.21.15.62:0/582117046 wait complete. 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.483+0000 7f43dab25740 1 -- start start 2024-09-15T23:19:47.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.483+0000 7f43dab25740 1 Processor -- start 2024-09-15T23:19:47.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43dab25740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c525a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43dab25740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c4f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43dab25740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 0x557354c4fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43dab25740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557354c55880 con 0x557354c49350 2024-09-15T23:19:47.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43d6ce9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43d6ce9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c525a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48454/0 (socket says 172.21.15.62:48454) 2024-09-15T23:19:47.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43d6ce9640 1 -- 172.21.15.62:0/687247109 learned_addr learned my addr 172.21.15.62:0/687247109 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557354b60060 con 0x557354c47de0 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557354c55a00 con 0x557354c52b80 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43dab25740 10 client.0 _ll_register_callbacks cb 0x5573548b38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 0x557354c4fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43d6ce9640 1 -- 172.21.15.62:0/687247109 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 msgr2=0x557354c4fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 0x557354c4fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43d6ce9640 1 -- 172.21.15.62:0/687247109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 msgr2=0x557354c4f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c4f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.484+0000 7f43d6ce9640 1 -- 172.21.15.62:0/687247109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573549163c0 con 0x557354c47de0 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c525a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f43c8005c40 tx=0x7f43c8005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c37c8816bba63303 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43b9ffb640 1 -- 172.21.15.62:0/687247109 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c801d070 con 0x557354c47de0 2024-09-15T23:19:47.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43b9ffb640 1 -- 172.21.15.62:0/687247109 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f43c8004a40 con 0x557354c47de0 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43c800cb80 con 0x557354c47de0 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43b9ffb640 1 -- 172.21.15.62:0/687247109 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f43c8004c40 con 0x557354c47de0 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43dab25740 10 client.25904 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557354c613c0 con 0x557354c47de0 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43b9ffb640 1 -- 172.21.15.62:0/687247109 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c8004de0 con 0x557354c47de0 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.485+0000 7f43b9ffb640 1 -- 172.21.15.62:0/687247109 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f43c800b040 con 0x557354c47de0 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.486+0000 7f43dab25740 10 client.25904 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.486+0000 7f43dab25740 10 client.25904 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.486+0000 7f43dab25740 20 client.25904 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.486+0000 7f43dab25740 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x557354c62710 0x557354c64b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.486+0000 7f43dab25740 4 client.25904 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.486+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f43c800cb80 con 0x557354c62710 2024-09-15T23:19:47.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.487+0000 7f43d5ce7640 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x557354c62710 0x557354c64b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.488+0000 7f43b9ffb640 10 client.25904 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.488+0000 7f43d5ce7640 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x557354c62710 0x557354c64b00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=399ee93979489586 server_cookie=d0664dba975c4daa in_seq=0 out_seq=0 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.489+0000 7f43b9ffb640 1 -- 172.21.15.62:0/687247109 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f43c800cb80 con 0x557354c62710 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.489+0000 7f43b9ffb640 10 client.25904 handle_command_reply: tid=0 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.489+0000 7f43dab25740 1 client.25904 shutdown 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 msgr2=0x557354c525a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c525a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f43c8005c40 tx=0x7f43c8005bd0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 shutdown_connections 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x557354c62710 0x557354c64b00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 0x557354c4fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c4f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43b9ffb640 0 client.25904 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 --2- 172.21.15.62:0/687247109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c525a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 >> 172.21.15.62:0/687247109 conn(0x557354c08310 msgr2=0x557354c4a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:47.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 shutdown_connections 2024-09-15T23:19:47.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 1 -- 172.21.15.62:0/687247109 wait complete. 2024-09-15T23:19:47.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.490+0000 7f43dab25740 20 client.25904 trim_cache size 0 max 16384 2024-09-15T23:19:47.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.506+0000 7f43dab25740 1 Processor -- start 2024-09-15T23:19:47.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.506+0000 7f43dab25740 1 -- start start 2024-09-15T23:19:47.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.506+0000 7f43dab25740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 0x557354c4e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43dab25740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c3abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43dab25740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c4a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43dab25740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f43b40123d0 con 0x557354c49350 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43dab25740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43b400fca0 con 0x557354c52b80 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43dab25740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f43b400daf0 con 0x557354c47de0 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c4a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c4a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48470/0 (socket says 172.21.15.62:48470) 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 -- 172.21.15.62:0/1385289334 learned_addr learned my addr 172.21.15.62:0/1385289334 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 0x557354c4e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d5ce7640 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c3abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 -- 172.21.15.62:0/1385289334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 msgr2=0x557354c4e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:47.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 0x557354c4e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 -- 172.21.15.62:0/1385289334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 msgr2=0x557354c3abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c3abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 -- 172.21.15.62:0/1385289334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557354c4dda0 con 0x557354c52b80 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d5ce7640 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c3abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.507+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c4a930 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f43c800d5a0 tx=0x7f43c800f940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a7a3dbaf4ee4eee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.508+0000 7f43bbfff640 1 -- 172.21.15.62:0/1385289334 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c801d070 con 0x557354c52b80 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.508+0000 7f43bbfff640 1 -- 172.21.15.62:0/1385289334 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c8003e50 con 0x557354c52b80 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.508+0000 7f43bbfff640 1 -- 172.21.15.62:0/1385289334 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c800d890 con 0x557354c52b80 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.508+0000 7f43dab25740 1 -- 172.21.15.62:0/1385289334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 msgr2=0x557354c4a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.508+0000 7f43dab25740 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c4a930 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f43c800d5a0 tx=0x7f43c800f940 comp rx=0 tx=0).stop 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.509+0000 7f43dab25740 1 -- 172.21.15.62:0/1385289334 shutdown_connections 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.509+0000 7f43dab25740 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c52b80 0x557354c4a930 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.509+0000 7f43dab25740 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c3abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.509+0000 7f43dab25740 1 --2- 172.21.15.62:0/1385289334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c47de0 0x557354c4e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.509+0000 7f43dab25740 1 -- 172.21.15.62:0/1385289334 >> 172.21.15.62:0/1385289334 conn(0x557354c08310 msgr2=0x557354c3a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:47.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.509+0000 7f43dab25740 1 -- 172.21.15.62:0/1385289334 shutdown_connections 2024-09-15T23:19:47.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.509+0000 7f43dab25740 1 -- 172.21.15.62:0/1385289334 wait complete. 2024-09-15T23:19:47.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.509+0000 7f43dab25740 1 -- start start 2024-09-15T23:19:47.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.509+0000 7f43dab25740 1 Processor -- start 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.510+0000 7f43dab25740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c56460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.510+0000 7f43dab25740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c569a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.510+0000 7f43dab25740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 0x557354cc52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.510+0000 7f43dab25740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f43b400e160 con 0x557354c49350 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.510+0000 7f43dab25740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43b400f5d0 con 0x557354c47de0 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.510+0000 7f43d6ce9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c56460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.510+0000 7f43d6ce9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c56460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48472/0 (socket says 172.21.15.62:48472) 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.510+0000 7f43d6ce9640 1 -- 172.21.15.62:0/1562388434 learned_addr learned my addr 172.21.15.62:0/1562388434 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.510+0000 7f43dab25740 1 -- 172.21.15.62:0/1562388434 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f43b400e830 con 0x557354c52b80 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43dab25740 10 client.0 _ll_register_callbacks cb 0x7f43c801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:47.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 0x557354cc52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43d5ce7640 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43d6ce9640 1 -- 172.21.15.62:0/1562388434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 msgr2=0x557354cc52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 0x557354cc52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43d6ce9640 1 -- 172.21.15.62:0/1562388434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 msgr2=0x557354c569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43d6ce9640 1 -- 172.21.15.62:0/1562388434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557354cc5980 con 0x557354c47de0 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43d5ce7640 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43d6ce9640 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c56460 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f43c800d540 tx=0x7f43c800cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=481db5408c57b6b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.511+0000 7f43b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.512+0000 7f43b9ffb640 1 -- 172.21.15.62:0/1562388434 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c803a040 con 0x557354c47de0 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.512+0000 7f43b9ffb640 1 -- 172.21.15.62:0/1562388434 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f43c8015050 con 0x557354c47de0 2024-09-15T23:19:47.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.512+0000 7f43dab25740 1 -- 172.21.15.62:0/1562388434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43c8003e50 con 0x557354c47de0 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.512+0000 7f43b9ffb640 1 -- 172.21.15.62:0/1562388434 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f43c801d040 con 0x557354c47de0 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.512+0000 7f43dab25740 10 client.25910 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.512+0000 7f43dab25740 1 -- 172.21.15.62:0/1562388434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557354ccccf0 con 0x557354c47de0 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.512+0000 7f43b9ffb640 1 -- 172.21.15.62:0/1562388434 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43c8004a00 con 0x557354c47de0 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.513+0000 7f43b9ffb640 1 -- 172.21.15.62:0/1562388434 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f43c800b070 con 0x557354c47de0 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.513+0000 7f43dab25740 10 client.25910 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.513+0000 7f43dab25740 10 client.25910 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.513+0000 7f43dab25740 20 client.25910 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.513+0000 7f43dab25740 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x557354ccd170 0x557354ccf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.513+0000 7f43dab25740 4 client.25910 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.513+0000 7f43dab25740 1 -- 172.21.15.62:0/1562388434 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f43c8003e50 con 0x557354ccd170 2024-09-15T23:19:47.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.513+0000 7f43d5ce7640 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x557354ccd170 0x557354ccf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43d5ce7640 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x557354ccd170 0x557354ccf560 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b0c25e7e0e8c8cec server_cookie=e450d74b5d8d5902 in_seq=0 out_seq=0 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43b9ffb640 10 client.25910 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43b9ffb640 1 -- 172.21.15.62:0/1562388434 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f43c8003e50 con 0x557354ccd170 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43b9ffb640 10 client.25910 handle_command_reply: tid=0 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43dab25740 1 client.25910 shutdown 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43dab25740 1 -- 172.21.15.62:0/1562388434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 msgr2=0x557354c56460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43dab25740 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c56460 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f43c800d540 tx=0x7f43c800cc20 comp rx=0 tx=0).stop 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43dab25740 1 -- 172.21.15.62:0/1562388434 shutdown_connections 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43dab25740 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x557354ccd170 0x557354ccf560 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43dab25740 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557354c52b80 0x557354cc52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43dab25740 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557354c49350 0x557354c569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43dab25740 1 --2- 172.21.15.62:0/1562388434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557354c47de0 0x557354c56460 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43dab25740 1 -- 172.21.15.62:0/1562388434 >> 172.21.15.62:0/1562388434 conn(0x557354c08310 msgr2=0x557354c3ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.514+0000 7f43b9ffb640 0 client.25910 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:19:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.515+0000 7f43dab25740 1 -- 172.21.15.62:0/1562388434 shutdown_connections 2024-09-15T23:19:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.515+0000 7f43dab25740 1 -- 172.21.15.62:0/1562388434 wait complete. 2024-09-15T23:19:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.515+0000 7f43dab25740 20 client.25910 trim_cache size 0 max 16384 2024-09-15T23:19:47.522 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:19:47.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.516+0000 7f43d874c640 1 -- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43bc0a3390 msgr2=0x7f43bc0a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:47.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.516+0000 7f43d874c640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43bc0a3390 0x7f43bc0a5850 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f43cc004830 tx=0x7f43cc0b9040 comp rx=0 tx=0).stop 2024-09-15T23:19:47.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.516+0000 7f43d874c640 1 -- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d0102d30 msgr2=0x7f43d0100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.516+0000 7f43d874c640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d0102d30 0x7f43d0100990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f43cc02a8f0 tx=0x7f43cc030710 comp rx=0 tx=0).stop 2024-09-15T23:19:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.518+0000 7f43d874c640 1 -- 172.21.15.62:0/1968801074 shutdown_connections 2024-09-15T23:19:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.518+0000 7f43d874c640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43bc0a3390 0x7f43bc0a5850 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.518+0000 7f43d874c640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f43d010c160 0x7f43d0101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.518+0000 7f43d874c640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43d0103650 0x7f43d0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.518+0000 7f43d874c640 1 --2- 172.21.15.62:0/1968801074 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43d0102d30 0x7f43d0100990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.518+0000 7f43d874c640 1 -- 172.21.15.62:0/1968801074 >> 172.21.15.62:0/1968801074 conn(0x7f43d00fc7d0 msgr2=0x7f43d00fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.518+0000 7f43d874c640 1 -- 172.21.15.62:0/1968801074 shutdown_connections 2024-09-15T23:19:47.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:47.518+0000 7f43d874c640 1 -- 172.21.15.62:0/1968801074 wait complete. 2024-09-15T23:19:47.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:47 smithi067 ceph-mon[35665]: pgmap v475: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 MiB/s wr, 747 op/s 2024-09-15T23:19:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:47 smithi148 ceph-mon[35610]: pgmap v475: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.3 MiB/s wr, 747 op/s 2024-09-15T23:19:48.365 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:19:48.775 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:49.186 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:49 smithi062 ceph-mon[30949]: pgmap v476: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 715 op/s 2024-09-15T23:19:49.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.334+0000 7f13ebfff640 1 -- 172.21.15.62:0/3146834234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13ec10e060 msgr2=0x7f13ec10e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:49.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.334+0000 7f13ebfff640 1 --2- 172.21.15.62:0/3146834234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13ec10e060 0x7f13ec10e440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f13e0009930 tx=0x7f13e002a440 comp rx=0 tx=0).stop 2024-09-15T23:19:49.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.336+0000 7f13ebfff640 1 -- 172.21.15.62:0/3146834234 shutdown_connections 2024-09-15T23:19:49.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.336+0000 7f13ebfff640 1 --2- 172.21.15.62:0/3146834234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ec108980 0x7f13ec114ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.336+0000 7f13ebfff640 1 --2- 172.21.15.62:0/3146834234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec108020 0x7f13ec108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.336+0000 7f13ebfff640 1 --2- 172.21.15.62:0/3146834234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13ec10e060 0x7f13ec10e440 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.336+0000 7f13ebfff640 1 -- 172.21.15.62:0/3146834234 >> 172.21.15.62:0/3146834234 conn(0x7f13ec077980 msgr2=0x7f13ec079da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:49.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.338+0000 7f13ebfff640 1 -- 172.21.15.62:0/3146834234 shutdown_connections 2024-09-15T23:19:49.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.338+0000 7f13ebfff640 1 -- 172.21.15.62:0/3146834234 wait complete. 2024-09-15T23:19:49.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.339+0000 7f13ebfff640 1 Processor -- start 2024-09-15T23:19:49.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.339+0000 7f13ebfff640 1 -- start start 2024-09-15T23:19:49.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.340+0000 7f13ebfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ec108020 0x7f13ec06d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.340+0000 7f13ebfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec108980 0x7f13ec06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.340+0000 7f13ebfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13ec10e060 0x7f13ec072020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.340+0000 7f13ebfff640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec06bfa0 con 0x7f13ec108980 2024-09-15T23:19:49.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.340+0000 7f13ebfff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec06be20 con 0x7f13ec108020 2024-09-15T23:19:49.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.340+0000 7f13ebfff640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec06c120 con 0x7f13ec10e060 2024-09-15T23:19:49.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.340+0000 7f13eb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13ec10e060 0x7f13ec072020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.341+0000 7f13ea7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec108980 0x7f13ec06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.341+0000 7f13ea7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec108980 0x7f13ec06dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52908/0 (socket says 172.21.15.62:52908) 2024-09-15T23:19:49.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.341+0000 7f13ea7fc640 1 -- 172.21.15.62:0/2118907799 learned_addr learned my addr 172.21.15.62:0/2118907799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:49.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.342+0000 7f13ea7fc640 1 -- 172.21.15.62:0/2118907799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13ec10e060 msgr2=0x7f13ec072020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:49.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.342+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13ec10e060 0x7f13ec072020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.342+0000 7f13ea7fc640 1 -- 172.21.15.62:0/2118907799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ec108020 msgr2=0x7f13ec06d750 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:19:49.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.342+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ec108020 0x7f13ec06d750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.342+0000 7f13ea7fc640 1 -- 172.21.15.62:0/2118907799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec072610 con 0x7f13ec108980 2024-09-15T23:19:49.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.342+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec108980 0x7f13ec06dc90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f13e400ba50 tx=0x7f13e400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:49.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.343+0000 7f13eb7fe640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13ec10e060 0x7f13ec072020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:19:49.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.343+0000 7f13f09ca640 1 -- 172.21.15.62:0/2118907799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4019070 con 0x7f13ec108980 2024-09-15T23:19:49.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.344+0000 7f13f09ca640 1 -- 172.21.15.62:0/2118907799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13e400cbe0 con 0x7f13ec108980 2024-09-15T23:19:49.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.344+0000 7f13ebfff640 1 -- 172.21.15.62:0/2118907799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ec072900 con 0x7f13ec108980 2024-09-15T23:19:49.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.344+0000 7f13f09ca640 1 -- 172.21.15.62:0/2118907799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e40076f0 con 0x7f13ec108980 2024-09-15T23:19:49.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.344+0000 7f13ebfff640 1 -- 172.21.15.62:0/2118907799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ec072e40 con 0x7f13ec108980 2024-09-15T23:19:49.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.348+0000 7f13f3da5740 1 Processor -- start 2024-09-15T23:19:49.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.348+0000 7f13f3da5740 1 -- start start 2024-09-15T23:19:49.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.349+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360adcc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.349+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcc7b0 0x56360adccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.349+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360add7550 0x56360add9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.349+0000 7f13f3da5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56360ad322f0 con 0x56360adcdd20 2024-09-15T23:19:49.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.349+0000 7f13f3da5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56360acd9cf0 con 0x56360adcc7b0 2024-09-15T23:19:49.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.349+0000 7f13f3da5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56360ad2e2b0 con 0x56360add7550 2024-09-15T23:19:49.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.349+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcc7b0 0x56360adccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.349+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcc7b0 0x56360adccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48490/0 (socket says 172.21.15.62:48490) 2024-09-15T23:19:49.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.349+0000 7f13eaffd640 1 -- 172.21.15.62:0/962990204 learned_addr learned my addr 172.21.15.62:0/962990204 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:49.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.350+0000 7f13eb7fe640 1 --2- 172.21.15.62:0/962990204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360adcc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.351+0000 7f13f09ca640 1 -- 172.21.15.62:0/2118907799 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f13e401c070 con 0x7f13ec108980 2024-09-15T23:19:49.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.351+0000 7f13eb7fe640 1 -- 172.21.15.62:0/962990204 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360add7550 msgr2=0x56360add9940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:19:49.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.351+0000 7f13eb7fe640 1 --2- 172.21.15.62:0/962990204 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360add7550 0x56360add9940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.351+0000 7f13eb7fe640 1 -- 172.21.15.62:0/962990204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcc7b0 msgr2=0x56360adccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:49.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.351+0000 7f13eb7fe640 1 --2- 172.21.15.62:0/962990204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcc7b0 0x56360adccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.351+0000 7f13eb7fe640 1 -- 172.21.15.62:0/962990204 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56360acbe3a0 con 0x56360adcdd20 2024-09-15T23:19:49.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.352+0000 7f13eb7fe640 1 --2- 172.21.15.62:0/962990204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360adcc270 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009ba0 tx=0x7f13dc0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b8d7907dc0ff0b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:49.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.352+0000 7f13f09ca640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13d40a3450 0x7f13d40a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.352+0000 7f13d1ffb640 1 -- 172.21.15.62:0/962990204 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc012450 con 0x56360adcdd20 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.352+0000 7f13d1ffb640 1 -- 172.21.15.62:0/962990204 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13dc012ad0 con 0x56360adcdd20 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.352+0000 7f13d1ffb640 1 -- 172.21.15.62:0/962990204 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc00fbb0 con 0x56360adcdd20 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.352+0000 7f13f09ca640 1 -- 172.21.15.62:0/2118907799 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f13e40cf2a0 con 0x7f13ec108980 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.353+0000 7f13eaffd640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13d40a3450 0x7f13d40a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.353+0000 7f13f3da5740 1 -- 172.21.15.62:0/962990204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 msgr2=0x56360adcc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.353+0000 7f13f3da5740 1 --2- 172.21.15.62:0/962990204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360adcc270 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009ba0 tx=0x7f13dc0117b0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.353+0000 7f13eaffd640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13d40a3450 0x7f13d40a5910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f13e0000c00 tx=0x7f13e002a940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.353+0000 7f13f3da5740 1 -- 172.21.15.62:0/962990204 shutdown_connections 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.353+0000 7f13f3da5740 1 --2- 172.21.15.62:0/962990204 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360add7550 0x56360add9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.353+0000 7f13f3da5740 1 --2- 172.21.15.62:0/962990204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcc7b0 0x56360adccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.353+0000 7f13f3da5740 1 --2- 172.21.15.62:0/962990204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360adcc270 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.353+0000 7f13f3da5740 1 -- 172.21.15.62:0/962990204 >> 172.21.15.62:0/962990204 conn(0x56360ad95310 msgr2=0x56360adc8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:49.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.354+0000 7f13f3da5740 1 -- 172.21.15.62:0/962990204 shutdown_connections 2024-09-15T23:19:49.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.354+0000 7f13f3da5740 1 -- 172.21.15.62:0/962990204 wait complete. 2024-09-15T23:19:49.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.354+0000 7f13f3da5740 1 -- start start 2024-09-15T23:19:49.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.355+0000 7f13f3da5740 1 Processor -- start 2024-09-15T23:19:49.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.355+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360add7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.355+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360add42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.355+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360add4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.355+0000 7f13f3da5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56360adda250 con 0x56360adcdd20 2024-09-15T23:19:49.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.355+0000 7f13f3da5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56360aced060 con 0x56360add7550 2024-09-15T23:19:49.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.355+0000 7f13f3da5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56360adda3d0 con 0x56360adcc7b0 2024-09-15T23:19:49.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.355+0000 7f13f3da5740 10 client.0 _ll_register_callbacks cb 0x56360aa408c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:49.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.356+0000 7f13eb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360add4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.356+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360add7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.356+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360add7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:51098/0 (socket says 172.21.15.62:51098) 2024-09-15T23:19:49.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.356+0000 7f13eb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360add4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48498/0 (socket says 172.21.15.62:48498) 2024-09-15T23:19:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.356+0000 7f13ea7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360add42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.356+0000 7f13eaffd640 1 -- 172.21.15.62:0/3979008322 learned_addr learned my addr 172.21.15.62:0/3979008322 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.357+0000 7f13ea7fc640 1 -- 172.21.15.62:0/3979008322 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 msgr2=0x56360add7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.357+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360add7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.357+0000 7f13ea7fc640 1 -- 172.21.15.62:0/3979008322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 msgr2=0x56360add4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.357+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360add4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.357+0000 7f13ea7fc640 1 -- 172.21.15.62:0/3979008322 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56360aaa33c0 con 0x56360adcdd20 2024-09-15T23:19:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.357+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360add42b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x56360adcd250 tx=0x7f13e4000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716daec9ea27a0fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:49.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.357+0000 7f13caffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.358+0000 7f13caffd640 1 -- 172.21.15.62:0/3979008322 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e409bbe0 con 0x56360adcdd20 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.358+0000 7f13caffd640 1 -- 172.21.15.62:0/3979008322 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f13e40a3cc0 con 0x56360adcdd20 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.358+0000 7f13f3da5740 1 -- 172.21.15.62:0/3979008322 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13dc012ad0 con 0x56360adcdd20 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.358+0000 7f13caffd640 1 -- 172.21.15.62:0/3979008322 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f13e409b4e0 con 0x56360adcdd20 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.358+0000 7f13f3da5740 10 client.16284 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.358+0000 7f13f3da5740 1 -- 172.21.15.62:0/3979008322 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56360ade5db0 con 0x56360adcdd20 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.358+0000 7f13caffd640 1 -- 172.21.15.62:0/3979008322 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13dc012ad0 con 0x56360adcdd20 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.359+0000 7f13caffd640 1 -- 172.21.15.62:0/3979008322 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f13e409a3a0 con 0x56360adcdd20 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.359+0000 7f13f3da5740 10 client.16284 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.359+0000 7f13f3da5740 10 client.16284 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.359+0000 7f13f3da5740 20 client.16284 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:49.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.359+0000 7f13f3da5740 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56360ade74c0 0x56360ade98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.359+0000 7f13f3da5740 4 client.16284 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:49.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.359+0000 7f13f3da5740 1 -- 172.21.15.62:0/3979008322 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56360ad3f860 con 0x56360ade74c0 2024-09-15T23:19:49.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.359+0000 7f13eaffd640 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56360ade74c0 0x56360ade98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.360+0000 7f13eaffd640 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56360ade74c0 0x56360ade98e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=31c1441ec7ba444c server_cookie=3cc0e42ac484b70c in_seq=0 out_seq=0 2024-09-15T23:19:49.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.360+0000 7f13caffd640 10 client.16284 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:19:49.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.360+0000 7f13caffd640 1 -- 172.21.15.62:0/3979008322 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56360ad3f860 con 0x56360ade74c0 2024-09-15T23:19:49.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.360+0000 7f13caffd640 10 client.16284 handle_command_reply: tid=0 2024-09-15T23:19:49.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.360+0000 7f13f3da5740 1 client.16284 shutdown 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.361+0000 7f13f3da5740 1 -- 172.21.15.62:0/3979008322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 msgr2=0x56360add42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.361+0000 7f13f3da5740 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360add42b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x56360adcd250 tx=0x7f13e4000f40 comp rx=0 tx=0).stop 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.361+0000 7f13f3da5740 1 -- 172.21.15.62:0/3979008322 shutdown_connections 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.361+0000 7f13f3da5740 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56360ade74c0 0x56360ade98e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.361+0000 7f13f3da5740 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360add4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.361+0000 7f13f3da5740 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360add42b0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.361+0000 7f13f3da5740 1 --2- 172.21.15.62:0/3979008322 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360add7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.361+0000 7f13f3da5740 1 -- 172.21.15.62:0/3979008322 >> 172.21.15.62:0/3979008322 conn(0x56360ad95310 msgr2=0x56360adceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.361+0000 7f13caffd640 0 client.16284 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.362+0000 7f13f3da5740 1 -- 172.21.15.62:0/3979008322 shutdown_connections 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.362+0000 7f13f3da5740 1 -- 172.21.15.62:0/3979008322 wait complete. 2024-09-15T23:19:49.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.362+0000 7f13f3da5740 20 client.16284 trim_cache size 0 max 16384 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.382+0000 7f13f3da5740 1 Processor -- start 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.382+0000 7f13f3da5740 1 -- start start 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.382+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360add4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.382+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcdd20 0x56360ade34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.382+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360add7550 0x56360add3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13f3da5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec1c01a0 con 0x56360add7550 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13f3da5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec1bda70 con 0x56360adcdd20 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13f3da5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec1bb8c0 con 0x56360adcc7b0 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13ea7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcdd20 0x56360ade34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13ea7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcdd20 0x56360ade34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48502/0 (socket says 172.21.15.62:48502) 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13ea7fc640 1 -- 172.21.15.62:0/2301766104 learned_addr learned my addr 172.21.15.62:0/2301766104 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13eb7fe640 1 --2- 172.21.15.62:0/2301766104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360add7550 0x56360add3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13eaffd640 1 --2- 172.21.15.62:0/2301766104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360add4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13ea7fc640 1 -- 172.21.15.62:0/2301766104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 msgr2=0x56360add4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/2301766104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360add4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13ea7fc640 1 -- 172.21.15.62:0/2301766104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360add7550 msgr2=0x56360add3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/2301766104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360add7550 0x56360add3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13ea7fc640 1 -- 172.21.15.62:0/2301766104 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56360adc9b40 con 0x56360adcdd20 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.383+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/2301766104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcdd20 0x56360ade34f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13e409e040 tx=0x7f13e4014e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=142bc6043afbe125 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.384+0000 7f13d17fa640 1 -- 172.21.15.62:0/2301766104 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e40b4170 con 0x56360adcdd20 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.384+0000 7f13d17fa640 1 -- 172.21.15.62:0/2301766104 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13e409bbe0 con 0x56360adcdd20 2024-09-15T23:19:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.384+0000 7f13d17fa640 1 -- 172.21.15.62:0/2301766104 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e409b020 con 0x56360adcdd20 2024-09-15T23:19:49.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.385+0000 7f13f3da5740 1 -- 172.21.15.62:0/2301766104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcdd20 msgr2=0x56360ade34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:49.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.385+0000 7f13f3da5740 1 --2- 172.21.15.62:0/2301766104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcdd20 0x56360ade34f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13e409e040 tx=0x7f13e4014e10 comp rx=0 tx=0).stop 2024-09-15T23:19:49.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.385+0000 7f13f3da5740 1 -- 172.21.15.62:0/2301766104 shutdown_connections 2024-09-15T23:19:49.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.385+0000 7f13f3da5740 1 --2- 172.21.15.62:0/2301766104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360add7550 0x56360add3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.385+0000 7f13f3da5740 1 --2- 172.21.15.62:0/2301766104 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360adcdd20 0x56360ade34f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.385+0000 7f13f3da5740 1 --2- 172.21.15.62:0/2301766104 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360add4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.386+0000 7f13f3da5740 1 -- 172.21.15.62:0/2301766104 >> 172.21.15.62:0/2301766104 conn(0x56360ad95310 msgr2=0x56360add2b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.387+0000 7f13f3da5740 1 -- 172.21.15.62:0/2301766104 shutdown_connections 2024-09-15T23:19:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.387+0000 7f13f3da5740 1 -- 172.21.15.62:0/2301766104 wait complete. 2024-09-15T23:19:49.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.387+0000 7f13f3da5740 1 -- start start 2024-09-15T23:19:49.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.388+0000 7f13f3da5740 1 Processor -- start 2024-09-15T23:19:49.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.388+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360adda800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13f3da5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360addad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360adda800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13eaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360adda800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:51118/0 (socket says 172.21.15.62:51118) 2024-09-15T23:19:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13eaffd640 1 -- 172.21.15.62:0/159929850 learned_addr learned my addr 172.21.15.62:0/159929850 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13f3da5740 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360ae50130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360addad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec1bbf30 con 0x56360adcdd20 2024-09-15T23:19:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec1bd3a0 con 0x56360add7550 2024-09-15T23:19:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec1bc600 con 0x56360adcc7b0 2024-09-15T23:19:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13eb7fe640 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360ae50130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13f3da5740 10 client.0 _ll_register_callbacks cb 0x7f13e4097850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13ea7fc640 1 -- 172.21.15.62:0/159929850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 msgr2=0x56360adda800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360adda800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13ea7fc640 1 -- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 msgr2=0x56360ae50130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360ae50130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13ea7fc640 1 -- 172.21.15.62:0/159929850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56360ae50670 con 0x56360adcdd20 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.389+0000 7f13eb7fe640 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360ae50130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.391+0000 7f13ea7fc640 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360addad40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f13e409e040 tx=0x7f13e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5b9dbc9e351877 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.391+0000 7f13caffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.391+0000 7f13caffd640 1 -- 172.21.15.62:0/159929850 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e409ab30 con 0x56360adcdd20 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.391+0000 7f13caffd640 1 -- 172.21.15.62:0/159929850 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f13e40a3cc0 con 0x56360adcdd20 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.391+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13e409bbe0 con 0x56360adcdd20 2024-09-15T23:19:49.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.391+0000 7f13caffd640 1 -- 172.21.15.62:0/159929850 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f13e409f890 con 0x56360adcdd20 2024-09-15T23:19:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.392+0000 7f13f3da5740 10 client.16290 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:49.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.392+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56360ae51870 con 0x56360adcdd20 2024-09-15T23:19:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.392+0000 7f13caffd640 1 -- 172.21.15.62:0/159929850 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13e409bbe0 con 0x56360adcdd20 2024-09-15T23:19:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.392+0000 7f13caffd640 1 -- 172.21.15.62:0/159929850 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f13e40ad070 con 0x56360adcdd20 2024-09-15T23:19:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.393+0000 7f13f3da5740 10 client.16290 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.393+0000 7f13f3da5740 10 client.16290 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:19:49.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.393+0000 7f13f3da5740 20 client.16290 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.393+0000 7f13f3da5740 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56360ae51d40 0x56360ae54130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.393+0000 7f13f3da5740 4 client.16290 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.393+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56360ad39d70 con 0x56360ae51d40 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.393+0000 7f13eaffd640 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56360ae51d40 0x56360ae54130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.393+0000 7f13caffd640 10 client.16290 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.393+0000 7f13eaffd640 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56360ae51d40 0x56360ae54130 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d9083795f0d54278 server_cookie=e22ad236c9f66161 in_seq=0 out_seq=0 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13caffd640 1 -- 172.21.15.62:0/159929850 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56360ad39d70 con 0x56360ae51d40 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13caffd640 10 client.16290 handle_command_reply: tid=0 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 client.16290 shutdown 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 msgr2=0x56360addad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360addad40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f13e409e040 tx=0x7f13e4004a40 comp rx=0 tx=0).stop 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 shutdown_connections 2024-09-15T23:19:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56360ae51d40 0x56360ae54130 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13caffd640 0 client.16290 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:19:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56360add7550 0x56360ae50130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56360adcdd20 0x56360addad40 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 --2- 172.21.15.62:0/159929850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56360adcc7b0 0x56360adda800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 >> 172.21.15.62:0/159929850 conn(0x56360ad95310 msgr2=0x56360add5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 shutdown_connections 2024-09-15T23:19:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 1 -- 172.21.15.62:0/159929850 wait complete. 2024-09-15T23:19:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.394+0000 7f13f3da5740 20 client.16290 trim_cache size 0 max 16384 2024-09-15T23:19:49.398 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:19:49.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 -- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13d40a3450 msgr2=0x7f13d40a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:49.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13d40a3450 0x7f13d40a5910 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f13e0000c00 tx=0x7f13e002a940 comp rx=0 tx=0).stop 2024-09-15T23:19:49.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 -- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec108980 msgr2=0x7f13ec06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec108980 0x7f13ec06dc90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f13e400ba50 tx=0x7f13e400bf10 comp rx=0 tx=0).stop 2024-09-15T23:19:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 -- 172.21.15.62:0/2118907799 shutdown_connections 2024-09-15T23:19:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f13d40a3450 0x7f13d40a5910 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f13ec10e060 0x7f13ec072020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13ec108980 0x7f13ec06dc90 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 --2- 172.21.15.62:0/2118907799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f13ec108020 0x7f13ec06d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 -- 172.21.15.62:0/2118907799 >> 172.21.15.62:0/2118907799 conn(0x7f13ec077980 msgr2=0x7f13ec0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.395+0000 7f13ebfff640 1 -- 172.21.15.62:0/2118907799 shutdown_connections 2024-09-15T23:19:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:49.397+0000 7f13ebfff640 1 -- 172.21.15.62:0/2118907799 wait complete. 2024-09-15T23:19:49.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:49 smithi067 ceph-mon[35665]: pgmap v476: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 715 op/s 2024-09-15T23:19:49.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:49 smithi148 ceph-mon[35610]: pgmap v476: 129 pgs: 129 active+clean; 1.1 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.4 MiB/s wr, 715 op/s 2024-09-15T23:19:49.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 210.0 s, 1583.2 tps, lat 65.026 ms stddev 77.987 2024-09-15T23:19:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.008+0000 7f0ce589b640 1 -- 172.21.15.62:0/3960801548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce0103710 msgr2=0x7f0ce0109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.008+0000 7f0ce589b640 1 --2- 172.21.15.62:0/3960801548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce0103710 0x7f0ce0109fa0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009900 tx=0x7f0cd802a3f0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.009+0000 7f0ce589b640 1 -- 172.21.15.62:0/3960801548 shutdown_connections 2024-09-15T23:19:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.009+0000 7f0ce589b640 1 --2- 172.21.15.62:0/3960801548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce0103710 0x7f0ce0109fa0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.009+0000 7f0ce589b640 1 --2- 172.21.15.62:0/3960801548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0102d50 0x7f0ce01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.009+0000 7f0ce589b640 1 --2- 172.21.15.62:0/3960801548 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ce0101b60 0x7f0ce0101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.009+0000 7f0ce589b640 1 -- 172.21.15.62:0/3960801548 >> 172.21.15.62:0/3960801548 conn(0x7f0ce00fd2e0 msgr2=0x7f0ce00ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.009+0000 7f0ce589b640 1 -- 172.21.15.62:0/3960801548 shutdown_connections 2024-09-15T23:19:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.009+0000 7f0ce589b640 1 -- 172.21.15.62:0/3960801548 wait complete. 2024-09-15T23:19:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.009+0000 7f0ce589b640 1 Processor -- start 2024-09-15T23:19:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.010+0000 7f0ce589b640 1 -- start start 2024-09-15T23:19:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0ce589b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0101b60 0x7f0ce019e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0ce589b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ce0102d50 0x7f0ce019e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0ce589b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce0103710 0x7f0ce01a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0ce589b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce010d830 con 0x7f0ce0101b60 2024-09-15T23:19:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0ce589b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce010d6b0 con 0x7f0ce0103710 2024-09-15T23:19:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0ce589b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce010d9b0 con 0x7f0ce0102d50 2024-09-15T23:19:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0cde7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ce0102d50 0x7f0ce019e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0cde7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ce0102d50 0x7f0ce019e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:51128/0 (socket says 172.21.15.62:51128) 2024-09-15T23:19:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0cde7fc640 1 -- 172.21.15.62:0/1157974910 learned_addr learned my addr 172.21.15.62:0/1157974910 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0101b60 0x7f0ce019e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0cde7fc640 1 -- 172.21.15.62:0/1157974910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce0103710 msgr2=0x7f0ce01a5970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce0103710 0x7f0ce01a5970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0cde7fc640 1 -- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0101b60 msgr2=0x7f0ce019e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0101b60 0x7f0ce019e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.011+0000 7f0cde7fc640 1 -- 172.21.15.62:0/1157974910 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce01a5fe0 con 0x7f0ce0102d50 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.012+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ce0102d50 0x7f0ce019e900 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000ebc0 tx=0x7f0cd000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.012+0000 7f0ce4899640 1 -- 172.21.15.62:0/1157974910 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0010630 con 0x7f0ce0102d50 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.012+0000 7f0ce589b640 1 -- 172.21.15.62:0/1157974910 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce01a62d0 con 0x7f0ce0102d50 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.013+0000 7f0ce589b640 1 -- 172.21.15.62:0/1157974910 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce01a6870 con 0x7f0ce0102d50 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.013+0000 7f0ce4899640 1 -- 172.21.15.62:0/1157974910 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd0010cb0 con 0x7f0ce0102d50 2024-09-15T23:19:50.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.013+0000 7f0ce4899640 1 -- 172.21.15.62:0/1157974910 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0018bd0 con 0x7f0ce0102d50 2024-09-15T23:19:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.015+0000 7f0ce4899640 1 -- 172.21.15.62:0/1157974910 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0cd00173d0 con 0x7f0ce0102d50 2024-09-15T23:19:50.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.015+0000 7f0ce8633740 1 Processor -- start 2024-09-15T23:19:50.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0ce8633740 1 -- start start 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ad6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7adbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ade830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0ce8633740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5588d7a3a2f0 con 0x5588d7ad69a0 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0ce8633740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5588d79e2270 con 0x5588d7ad72e0 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0ce8633740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5588d7a362b0 con 0x5588d7adc370 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0cdeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7adbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ad6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ad6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52980/0 (socket says 172.21.15.62:52980) 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.016+0000 7f0cdf7fe640 1 -- 172.21.15.62:0/523033273 learned_addr learned my addr 172.21.15.62:0/523033273 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/523033273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ade830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cdeffd640 1 -- 172.21.15.62:0/523033273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 msgr2=0x5588d7ade830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/523033273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ade830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cdeffd640 1 -- 172.21.15.62:0/523033273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 msgr2=0x5588d7ad6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/523033273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ad6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cdeffd640 1 -- 172.21.15.62:0/523033273 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d79e1340 con 0x5588d7ad72e0 2024-09-15T23:19:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0ce4899640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0ccc0a3370 0x7f0ccc0a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/523033273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7adbda0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009b40 tx=0x7f0cc80113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dded86f2b2e0c36 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cd5ffb640 1 -- 172.21.15.62:0/523033273 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8011e10 con 0x5588d7ad72e0 2024-09-15T23:19:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cdf7fe640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0ccc0a3370 0x7f0ccc0a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0ce4899640 1 -- 172.21.15.62:0/1157974910 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0cd00d0220 con 0x7f0ce0102d50 2024-09-15T23:19:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.017+0000 7f0cd5ffb640 1 -- 172.21.15.62:0/523033273 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cc8012930 con 0x5588d7ad72e0 2024-09-15T23:19:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.018+0000 7f0cd5ffb640 1 -- 172.21.15.62:0/523033273 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc800fa50 con 0x5588d7ad72e0 2024-09-15T23:19:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.018+0000 7f0ce8633740 1 -- 172.21.15.62:0/523033273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 msgr2=0x5588d7adbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.018+0000 7f0ce8633740 1 --2- 172.21.15.62:0/523033273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7adbda0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009b40 tx=0x7f0cc80113e0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.018+0000 7f0cdf7fe640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0ccc0a3370 0x7f0ccc0a5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8005c20 tx=0x7f0cd8005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.019+0000 7f0ce8633740 1 -- 172.21.15.62:0/523033273 shutdown_connections 2024-09-15T23:19:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.019+0000 7f0ce8633740 1 --2- 172.21.15.62:0/523033273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ade830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.019+0000 7f0ce8633740 1 --2- 172.21.15.62:0/523033273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7adbda0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.019+0000 7f0ce8633740 1 --2- 172.21.15.62:0/523033273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ad6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.019+0000 7f0ce8633740 1 -- 172.21.15.62:0/523033273 >> 172.21.15.62:0/523033273 conn(0x5588d7ad2010 msgr2=0x5588d7ad2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.020+0000 7f0ce8633740 1 -- 172.21.15.62:0/523033273 shutdown_connections 2024-09-15T23:19:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.021+0000 7f0ce8633740 1 -- 172.21.15.62:0/523033273 wait complete. 2024-09-15T23:19:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0ce8633740 1 -- start start 2024-09-15T23:19:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0ce8633740 1 Processor -- start 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ae0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7ae1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ae8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0ce8633740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5588d7adf3e0 con 0x5588d7ad69a0 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0ce8633740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5588d79c3290 con 0x5588d7ad72e0 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0ce8633740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5588d7adf560 con 0x5588d7adc370 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0ce8633740 10 client.0 _ll_register_callbacks cb 0x5588d79f4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0cdeffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ae0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0cdeffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ae0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52982/0 (socket says 172.21.15.62:52982) 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0cdeffd640 1 -- 172.21.15.62:0/923005536 learned_addr learned my addr 172.21.15.62:0/923005536 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7ae1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.022+0000 7f0cdf7fe640 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ae8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cdeffd640 1 -- 172.21.15.62:0/923005536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 msgr2=0x5588d7ae8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ae8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cdeffd640 1 -- 172.21.15.62:0/923005536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 msgr2=0x5588d7ae1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7ae1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cdeffd640 1 -- 172.21.15.62:0/923005536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d774d210 con 0x5588d7ad69a0 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7ae1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ae0c10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0cc801c040 tx=0x7f0cc8012d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ae3b7878cc823b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cbeffd640 1 -- 172.21.15.62:0/923005536 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8005050 con 0x5588d7ad69a0 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cbeffd640 1 -- 172.21.15.62:0/923005536 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0cc80051f0 con 0x5588d7ad69a0 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0ce8633740 1 -- 172.21.15.62:0/923005536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cc8012930 con 0x5588d7ad69a0 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.023+0000 7f0cbeffd640 1 -- 172.21.15.62:0/923005536 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0cc8032050 con 0x5588d7ad69a0 2024-09-15T23:19:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.024+0000 7f0ce8633740 10 client.16302 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.024+0000 7f0ce8633740 1 -- 172.21.15.62:0/923005536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588d7819620 con 0x5588d7ad69a0 2024-09-15T23:19:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.025+0000 7f0cbeffd640 1 -- 172.21.15.62:0/923005536 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cc8012930 con 0x5588d7ad69a0 2024-09-15T23:19:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.025+0000 7f0cbeffd640 1 -- 172.21.15.62:0/923005536 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0cc8003610 con 0x5588d7ad69a0 2024-09-15T23:19:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.025+0000 7f0ce8633740 10 client.16302 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.025+0000 7f0ce8633740 10 client.16302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.025+0000 7f0ce8633740 20 client.16302 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:50.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.025+0000 7f0ce8633740 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588d7aec600 0x5588d7aee9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.025+0000 7f0ce8633740 4 client.16302 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.025+0000 7f0ce8633740 1 -- 172.21.15.62:0/923005536 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588d7a47a20 con 0x5588d7aec600 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.025+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588d7aec600 0x5588d7aee9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.026+0000 7f0cbeffd640 10 client.16302 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.026+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588d7aec600 0x5588d7aee9f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9a730b3f8439fc7 server_cookie=d20645096b27392b in_seq=0 out_seq=0 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0cbeffd640 1 -- 172.21.15.62:0/923005536 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5588d7a47a20 con 0x5588d7aec600 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0cbeffd640 10 client.16302 handle_command_reply: tid=0 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 client.16302 shutdown 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 -- 172.21.15.62:0/923005536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 msgr2=0x5588d7ae0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ae0c10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f0cc801c040 tx=0x7f0cc8012d10 comp rx=0 tx=0).stop 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 -- 172.21.15.62:0/923005536 shutdown_connections 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588d7aec600 0x5588d7aee9f0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ae8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0cbeffd640 0 client.16302 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7ae1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 --2- 172.21.15.62:0/923005536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad69a0 0x5588d7ae0c10 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 -- 172.21.15.62:0/923005536 >> 172.21.15.62:0/923005536 conn(0x5588d7ad2010 msgr2=0x5588d7ad3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 -- 172.21.15.62:0/923005536 shutdown_connections 2024-09-15T23:19:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.027+0000 7f0ce8633740 1 -- 172.21.15.62:0/923005536 wait complete. 2024-09-15T23:19:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.028+0000 7f0ce8633740 20 client.16302 trim_cache size 0 max 16384 2024-09-15T23:19:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0ce8633740 1 Processor -- start 2024-09-15T23:19:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0ce8633740 1 -- start start 2024-09-15T23:19:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad69a0 0x5588d7ae10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad72e0 0x5588d7ae8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ad2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0ce8633740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc00123c0 con 0x5588d7ad72e0 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0ce8633740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc000fc90 con 0x5588d7ad69a0 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0ce8633740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc000dae0 con 0x5588d7adc370 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cde7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad72e0 0x5588d7ae8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cdeffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad69a0 0x5588d7ae10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ad2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cde7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad72e0 0x5588d7ae8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52998/0 (socket says 172.21.15.62:52998) 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cde7fc640 1 -- 172.21.15.62:0/3423540233 learned_addr learned my addr 172.21.15.62:0/3423540233 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cde7fc640 1 -- 172.21.15.62:0/3423540233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 msgr2=0x5588d7ad2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/3423540233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ad2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cde7fc640 1 -- 172.21.15.62:0/3423540233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad69a0 msgr2=0x5588d7ae10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/3423540233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad69a0 0x5588d7ae10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.055+0000 7f0cde7fc640 1 -- 172.21.15.62:0/3423540233 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588d7adfbc0 con 0x5588d7ad72e0 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.056+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/3423540233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad69a0 0x5588d7ae10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.056+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/3423540233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad72e0 0x5588d7ae8150 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0098e90 tx=0x7f0cd00d2730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9500e074851ee38f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.056+0000 7f0cd57fa640 1 -- 172.21.15.62:0/3423540233 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00cde60 con 0x5588d7ad72e0 2024-09-15T23:19:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.056+0000 7f0cd57fa640 1 -- 172.21.15.62:0/3423540233 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd0085070 con 0x5588d7ad72e0 2024-09-15T23:19:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.056+0000 7f0cd57fa640 1 -- 172.21.15.62:0/3423540233 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0010e60 con 0x5588d7ad72e0 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 -- 172.21.15.62:0/3423540233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad72e0 msgr2=0x5588d7ae8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 --2- 172.21.15.62:0/3423540233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad72e0 0x5588d7ae8150 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0098e90 tx=0x7f0cd00d2730 comp rx=0 tx=0).stop 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 -- 172.21.15.62:0/3423540233 shutdown_connections 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 --2- 172.21.15.62:0/3423540233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7adc370 0x5588d7ad2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 --2- 172.21.15.62:0/3423540233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7ad72e0 0x5588d7ae8150 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 --2- 172.21.15.62:0/3423540233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad69a0 0x5588d7ae10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 -- 172.21.15.62:0/3423540233 >> 172.21.15.62:0/3423540233 conn(0x5588d7ad2010 msgr2=0x5588d7ae09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 -- 172.21.15.62:0/3423540233 shutdown_connections 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 -- 172.21.15.62:0/3423540233 wait complete. 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 -- start start 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 Processor -- start 2024-09-15T23:19:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7ad69a0 0x5588d7ae5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7ae5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.057+0000 7f0ce8633740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7adc370 0x5588d7ae6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0ce8633740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc000e150 con 0x5588d7adc370 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0ce8633740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc000f5c0 con 0x5588d7ad72e0 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7adc370 0x5588d7ae6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0ce8633740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc000e820 con 0x5588d7ad69a0 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7adc370 0x5588d7ae6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53002/0 (socket says 172.21.15.62:53002) 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdf7fe640 1 -- 172.21.15.62:0/627953175 learned_addr learned my addr 172.21.15.62:0/627953175 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7ad69a0 0x5588d7ae5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cde7fc640 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7ae5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0ce8633740 10 client.0 _ll_register_callbacks cb 0x7f0cd00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdf7fe640 1 -- 172.21.15.62:0/627953175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7ad69a0 msgr2=0x5588d7ae5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdf7fe640 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7ad69a0 0x5588d7ae5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdf7fe640 1 -- 172.21.15.62:0/627953175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 msgr2=0x5588d7ae5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdf7fe640 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7ae5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdf7fe640 1 -- 172.21.15.62:0/627953175 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588d7ae6e60 con 0x5588d7adc370 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cdf7fe640 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7adc370 0x5588d7ae6850 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009900 tx=0x7f0cd8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0190c4b388ab080 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cbeffd640 1 -- 172.21.15.62:0/627953175 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8035040 con 0x5588d7adc370 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cbeffd640 1 -- 172.21.15.62:0/627953175 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0cd8004ad0 con 0x5588d7adc370 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0ce8633740 1 -- 172.21.15.62:0/627953175 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cd0085070 con 0x5588d7adc370 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0cbeffd640 1 -- 172.21.15.62:0/627953175 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0cd805d050 con 0x5588d7adc370 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0ce8633740 10 client.16308 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.058+0000 7f0ce8633740 1 -- 172.21.15.62:0/627953175 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588d7ae81c0 con 0x5588d7adc370 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.059+0000 7f0cbeffd640 1 -- 172.21.15.62:0/627953175 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd0085070 con 0x5588d7adc370 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.059+0000 7f0cbeffd640 1 -- 172.21.15.62:0/627953175 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0cd8052020 con 0x5588d7adc370 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.059+0000 7f0ce8633740 10 client.16308 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:50.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.059+0000 7f0ce8633740 10 client.16308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.059+0000 7f0ce8633740 20 client.16308 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.059+0000 7f0ce8633740 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588d7b57810 0x5588d7b59c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.059+0000 7f0ce8633740 4 client.16308 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:19:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.059+0000 7f0ce8633740 1 -- 172.21.15.62:0/627953175 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5588d7a41d70 con 0x5588d7b57810 2024-09-15T23:19:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.060+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588d7b57810 0x5588d7b59c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.060+0000 7f0cbeffd640 10 client.16308 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:50.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.060+0000 7f0cdeffd640 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588d7b57810 0x5588d7b59c30 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b116392e8fb215c7 server_cookie=a9c5dba4d2996662 in_seq=0 out_seq=0 2024-09-15T23:19:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.089+0000 7f0cbeffd640 1 -- 172.21.15.62:0/627953175 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5588d7a41d70 con 0x5588d7b57810 2024-09-15T23:19:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.089+0000 7f0cbeffd640 10 client.16308 handle_command_reply: tid=0 2024-09-15T23:19:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.089+0000 7f0ce8633740 1 client.16308 shutdown 2024-09-15T23:19:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.090+0000 7f0ce8633740 1 -- 172.21.15.62:0/627953175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7adc370 msgr2=0x5588d7ae6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.090+0000 7f0ce8633740 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7adc370 0x5588d7ae6850 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009900 tx=0x7f0cd8009260 comp rx=0 tx=0).stop 2024-09-15T23:19:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.093+0000 7f0ce8633740 1 -- 172.21.15.62:0/627953175 shutdown_connections 2024-09-15T23:19:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.093+0000 7f0ce8633740 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588d7b57810 0x5588d7b59c30 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.093+0000 7f0ce8633740 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588d7adc370 0x5588d7ae6850 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.093+0000 7f0ce8633740 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588d7ad72e0 0x5588d7ae5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.093+0000 7f0ce8633740 1 --2- 172.21.15.62:0/627953175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588d7ad69a0 0x5588d7ae5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.093+0000 7f0ce8633740 1 -- 172.21.15.62:0/627953175 >> 172.21.15.62:0/627953175 conn(0x5588d7ad2010 msgr2=0x5588d7ad09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.093+0000 7f0cbeffd640 0 client.16308 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:50.096 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.095+0000 7f0ce8633740 1 -- 172.21.15.62:0/627953175 shutdown_connections 2024-09-15T23:19:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.098+0000 7f0ce8633740 1 -- 172.21.15.62:0/627953175 wait complete. 2024-09-15T23:19:50.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.102+0000 7f0ce8633740 20 client.16308 trim_cache size 0 max 16384 2024-09-15T23:19:50.108 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:19:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:19:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:19:50.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:19:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.111+0000 7f0ce589b640 1 -- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0ccc0a3370 msgr2=0x7f0ccc0a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.111+0000 7f0ce589b640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0ccc0a3370 0x7f0ccc0a5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8005c20 tx=0x7f0cd8005a20 comp rx=0 tx=0).stop 2024-09-15T23:19:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.111+0000 7f0ce589b640 1 -- 172.21.15.62:0/1157974910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ce0102d50 msgr2=0x7f0ce019e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:50.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.111+0000 7f0ce589b640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ce0102d50 0x7f0ce019e900 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000ebc0 tx=0x7f0cd000c6a0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.114+0000 7f0ce589b640 1 -- 172.21.15.62:0/1157974910 shutdown_connections 2024-09-15T23:19:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.114+0000 7f0ce589b640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0ccc0a3370 0x7f0ccc0a5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.115+0000 7f0ce589b640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce0103710 0x7f0ce01a5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.115+0000 7f0ce589b640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0ce0102d50 0x7f0ce019e900 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.115+0000 7f0ce589b640 1 --2- 172.21.15.62:0/1157974910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ce0101b60 0x7f0ce019e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:50.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.115+0000 7f0ce589b640 1 -- 172.21.15.62:0/1157974910 >> 172.21.15.62:0/1157974910 conn(0x7f0ce00fd2e0 msgr2=0x7f0ce0103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.117+0000 7f0ce589b640 1 -- 172.21.15.62:0/1157974910 shutdown_connections 2024-09-15T23:19:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:19:50.119+0000 7f0ce589b640 1 -- 172.21.15.62:0/1157974910 wait complete. 2024-09-15T23:19:50.353 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:19:50.821 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:51.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:51 smithi062 ceph-mon[30949]: pgmap v477: 129 pgs: 129 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 862 op/s 2024-09-15T23:19:51.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:51 smithi067 ceph-mon[35665]: pgmap v477: 129 pgs: 129 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 862 op/s 2024-09-15T23:19:51.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:51 smithi148 ceph-mon[35610]: pgmap v477: 129 pgs: 129 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 MiB/s wr, 862 op/s 2024-09-15T23:19:53.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.026+0000 7f86b869e640 1 -- 172.21.15.62:0/3670136776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b0109b10 msgr2=0x7f86b0114570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:53.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.026+0000 7f86b869e640 1 --2- 172.21.15.62:0/3670136776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b0109b10 0x7f86b0114570 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f86a0009900 tx=0x7f86a002a4d0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.029+0000 7f86b869e640 1 -- 172.21.15.62:0/3670136776 shutdown_connections 2024-09-15T23:19:53.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.029+0000 7f86b869e640 1 --2- 172.21.15.62:0/3670136776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86b0114ab0 0x7f86b0116ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.029+0000 7f86b869e640 1 --2- 172.21.15.62:0/3670136776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b0109b10 0x7f86b0114570 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.029+0000 7f86b869e640 1 --2- 172.21.15.62:0/3670136776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86b01091f0 0x7f86b01095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.029+0000 7f86b869e640 1 -- 172.21.15.62:0/3670136776 >> 172.21.15.62:0/3670136776 conn(0x7f86b006e140 msgr2=0x7f86b006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:53.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.034+0000 7f86b869e640 1 -- 172.21.15.62:0/3670136776 shutdown_connections 2024-09-15T23:19:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.036+0000 7f86b869e640 1 -- 172.21.15.62:0/3670136776 wait complete. 2024-09-15T23:19:53.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.036+0000 7f86b869e640 1 Processor -- start 2024-09-15T23:19:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.037+0000 7f86b869e640 1 -- start start 2024-09-15T23:19:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.037+0000 7f86b869e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b01091f0 0x7f86b01a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.038+0000 7f86b869e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86b0109b10 0x7f86b019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.038+0000 7f86b869e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86b0114ab0 0x7f86b019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.038+0000 7f86b869e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f86b01186d0 con 0x7f86b01091f0 2024-09-15T23:19:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.038+0000 7f86b869e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f86b0118550 con 0x7f86b0109b10 2024-09-15T23:19:53.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.038+0000 7f86b869e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f86b0118850 con 0x7f86b0114ab0 2024-09-15T23:19:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.038+0000 7f86b643a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b01091f0 0x7f86b01a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.038+0000 7f86b643a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b01091f0 0x7f86b01a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37138/0 (socket says 172.21.15.62:37138) 2024-09-15T23:19:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.038+0000 7f86b643a640 1 -- 172.21.15.62:0/1707766324 learned_addr learned my addr 172.21.15.62:0/1707766324 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.039+0000 7f86b5c39640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86b0109b10 0x7f86b019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.039+0000 7f86b643a640 1 -- 172.21.15.62:0/1707766324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86b0114ab0 msgr2=0x7f86b019f030 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:19:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.039+0000 7f86b643a640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86b0114ab0 0x7f86b019f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.039+0000 7f86b643a640 1 -- 172.21.15.62:0/1707766324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86b0109b10 msgr2=0x7f86b019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.039+0000 7f86b643a640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86b0109b10 0x7f86b019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.039+0000 7f86b643a640 1 -- 172.21.15.62:0/1707766324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b019f6d0 con 0x7f86b01091f0 2024-09-15T23:19:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.040+0000 7f86b643a640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b01091f0 0x7f86b01a5a60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f86a800ba50 tx=0x7f86a800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.041+0000 7f86a77fe640 1 -- 172.21.15.62:0/1707766324 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8019070 con 0x7f86b01091f0 2024-09-15T23:19:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.041+0000 7f86a77fe640 1 -- 172.21.15.62:0/1707766324 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86a800cc10 con 0x7f86b01091f0 2024-09-15T23:19:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.041+0000 7f86a77fe640 1 -- 172.21.15.62:0/1707766324 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a80076c0 con 0x7f86b01091f0 2024-09-15T23:19:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.041+0000 7f86b869e640 1 -- 172.21.15.62:0/1707766324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b01a3590 con 0x7f86b01091f0 2024-09-15T23:19:53.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.041+0000 7f86b869e640 1 -- 172.21.15.62:0/1707766324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b01a3a50 con 0x7f86b01091f0 2024-09-15T23:19:53.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.050+0000 7f86baa77740 1 Processor -- start 2024-09-15T23:19:53.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.050+0000 7f86a77fe640 1 -- 172.21.15.62:0/1707766324 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f86a801c070 con 0x7f86b01091f0 2024-09-15T23:19:53.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.051+0000 7f86a77fe640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f869c0a3460 0x7f869c0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.051+0000 7f86a77fe640 1 -- 172.21.15.62:0/1707766324 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f86a80cf2c0 con 0x7f86b01091f0 2024-09-15T23:19:53.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.052+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f869c0a3460 0x7f869c0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.052+0000 7f86baa77740 1 -- start start 2024-09-15T23:19:53.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.052+0000 7f86baa77740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f76090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.052+0000 7f86baa77740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 0x556cb6f76a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.052+0000 7f86baa77740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6f83760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.053+0000 7f86baa77740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556cb6edc2f0 con 0x556cb6f77b40 2024-09-15T23:19:53.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.053+0000 7f86baa77740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556cb6e83cf0 con 0x556cb6f765d0 2024-09-15T23:19:53.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.053+0000 7f86baa77740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556cb6ed82b0 con 0x556cb6f81370 2024-09-15T23:19:53.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.053+0000 7f86b6c3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6f83760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.053+0000 7f86b6c3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6f83760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:37858/0 (socket says 172.21.15.62:37858) 2024-09-15T23:19:53.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.053+0000 7f86b6c3b640 1 -- 172.21.15.62:0/1622162416 learned_addr learned my addr 172.21.15.62:0/1622162416 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:53.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.054+0000 7f86b643a640 1 --2- 172.21.15.62:0/1622162416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f76090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.054+0000 7f86b6c3b640 1 -- 172.21.15.62:0/1622162416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 msgr2=0x556cb6f76a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:19:53.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.054+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/1622162416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 0x556cb6f76a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.054+0000 7f86b6c3b640 1 -- 172.21.15.62:0/1622162416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 msgr2=0x556cb6f76090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:53.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.054+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/1622162416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f76090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.054+0000 7f86b6c3b640 1 -- 172.21.15.62:0/1622162416 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cb6e683a0 con 0x556cb6f81370 2024-09-15T23:19:53.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.054+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/1622162416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6f83760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f86ac00de50 tx=0x7f86ac0125e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eadf6db239c718b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:53.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.055+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f869c0a3460 0x7f869c0a5920 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f86ac01d000 tx=0x7f86ac01a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:53.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.055+0000 7f86a4ff9640 1 -- 172.21.15.62:0/1622162416 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac00f030 con 0x556cb6f81370 2024-09-15T23:19:53.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.056+0000 7f86a4ff9640 1 -- 172.21.15.62:0/1622162416 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86ac005e50 con 0x556cb6f81370 2024-09-15T23:19:53.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.057+0000 7f86baa77740 1 -- 172.21.15.62:0/1622162416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 msgr2=0x556cb6f83760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:53.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.057+0000 7f86baa77740 1 --2- 172.21.15.62:0/1622162416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6f83760 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f86ac00de50 tx=0x7f86ac0125e0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.057+0000 7f86baa77740 1 -- 172.21.15.62:0/1622162416 shutdown_connections 2024-09-15T23:19:53.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.057+0000 7f86baa77740 1 --2- 172.21.15.62:0/1622162416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6f83760 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.057+0000 7f86baa77740 1 --2- 172.21.15.62:0/1622162416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 0x556cb6f76a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.057+0000 7f86baa77740 1 --2- 172.21.15.62:0/1622162416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f76090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.057+0000 7f86baa77740 1 -- 172.21.15.62:0/1622162416 >> 172.21.15.62:0/1622162416 conn(0x556cb6f3f310 msgr2=0x556cb6f72c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:53.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.058+0000 7f86baa77740 1 -- 172.21.15.62:0/1622162416 shutdown_connections 2024-09-15T23:19:53.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.058+0000 7f86baa77740 1 -- 172.21.15.62:0/1622162416 wait complete. 2024-09-15T23:19:53.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.059+0000 7f86baa77740 1 -- start start 2024-09-15T23:19:53.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.059+0000 7f86baa77740 1 Processor -- start 2024-09-15T23:19:53.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.060+0000 7f86baa77740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 0x556cb6f80c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.060+0000 7f86baa77740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f7dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.060+0000 7f86baa77740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f81370 0x556cb6f7e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.060+0000 7f86baa77740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556cb6f840c0 con 0x556cb6f77b40 2024-09-15T23:19:53.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.060+0000 7f86baa77740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556cb6e97060 con 0x556cb6f81370 2024-09-15T23:19:53.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.060+0000 7f86b6c3b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f81370 0x556cb6f7e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.060+0000 7f86b6c3b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f81370 0x556cb6f7e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:35652/0 (socket says 172.21.15.62:35652) 2024-09-15T23:19:53.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.060+0000 7f86b6c3b640 1 -- 172.21.15.62:0/3136875730 learned_addr learned my addr 172.21.15.62:0/3136875730 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:53.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.061+0000 7f86baa77740 1 -- 172.21.15.62:0/3136875730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556cb6f84240 con 0x556cb6f765d0 2024-09-15T23:19:53.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.061+0000 7f86baa77740 10 client.0 _ll_register_callbacks cb 0x556cb6bea8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:53.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.063+0000 7f86b6c3b640 1 -- 172.21.15.62:0/3136875730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 msgr2=0x556cb6f80c30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:19:53.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.063+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 0x556cb6f80c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.063+0000 7f86b6c3b640 1 -- 172.21.15.62:0/3136875730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 msgr2=0x556cb6f7dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:53.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.063+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f7dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.063+0000 7f86b6c3b640 1 -- 172.21.15.62:0/3136875730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cb6c4d3c0 con 0x556cb6f81370 2024-09-15T23:19:53.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.067+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f81370 0x556cb6f7e420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86ac015ad0 tx=0x7f86ac015b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91050a897fdbc88b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:53.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.067+0000 7f8699ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:19:53.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.069+0000 7f8699ffb640 1 -- 172.21.15.62:0/3136875730 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac027040 con 0x556cb6f81370 2024-09-15T23:19:53.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.069+0000 7f8699ffb640 1 -- 172.21.15.62:0/3136875730 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f86ac021440 con 0x556cb6f81370 2024-09-15T23:19:53.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.069+0000 7f86baa77740 1 -- 172.21.15.62:0/3136875730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86ac005e50 con 0x556cb6f81370 2024-09-15T23:19:53.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.070+0000 7f8699ffb640 1 -- 172.21.15.62:0/3136875730 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86ac00f040 con 0x556cb6f81370 2024-09-15T23:19:53.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f86baa77740 10 client.25946 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f86baa77740 1 -- 172.21.15.62:0/3136875730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556cb6f8fb00 con 0x556cb6f81370 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f8699ffb640 1 -- 172.21.15.62:0/3136875730 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86ac005e50 con 0x556cb6f81370 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f8699ffb640 1 -- 172.21.15.62:0/3136875730 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f86ac015070 con 0x556cb6f81370 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f86baa77740 10 client.25946 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f86baa77740 10 client.25946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f86baa77740 20 client.25946 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f86baa77740 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556cb6f90e50 0x556cb6f93240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f86baa77740 4 client.25946 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.071+0000 7f86baa77740 1 -- 172.21.15.62:0/3136875730 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cb6ee9860 con 0x556cb6f90e50 2024-09-15T23:19:53.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.073+0000 7f86b643a640 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556cb6f90e50 0x556cb6f93240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.075+0000 7f86b643a640 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556cb6f90e50 0x556cb6f93240 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f461cbcdaf93205c server_cookie=8e07eab11b64c765 in_seq=0 out_seq=0 2024-09-15T23:19:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.075+0000 7f8699ffb640 10 client.25946 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.076+0000 7f8699ffb640 1 -- 172.21.15.62:0/3136875730 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556cb6ee9860 con 0x556cb6f90e50 2024-09-15T23:19:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.076+0000 7f8699ffb640 10 client.25946 handle_command_reply: tid=0 2024-09-15T23:19:53.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.076+0000 7f86baa77740 1 client.25946 shutdown 2024-09-15T23:19:53.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.077+0000 7f86baa77740 1 -- 172.21.15.62:0/3136875730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f81370 msgr2=0x556cb6f7e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:53.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.077+0000 7f86baa77740 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f81370 0x556cb6f7e420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86ac015ad0 tx=0x7f86ac015b00 comp rx=0 tx=0).stop 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.078+0000 7f86baa77740 1 -- 172.21.15.62:0/3136875730 shutdown_connections 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.078+0000 7f86baa77740 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556cb6f90e50 0x556cb6f93240 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.078+0000 7f86baa77740 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f81370 0x556cb6f7e420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.078+0000 7f86baa77740 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f7dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.078+0000 7f86baa77740 1 --2- 172.21.15.62:0/3136875730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 0x556cb6f80c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.078+0000 7f86baa77740 1 -- 172.21.15.62:0/3136875730 >> 172.21.15.62:0/3136875730 conn(0x556cb6f3f310 msgr2=0x556cb6f78930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.078+0000 7f8699ffb640 0 client.25946 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.078+0000 7f86baa77740 1 -- 172.21.15.62:0/3136875730 shutdown_connections 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.078+0000 7f86baa77740 1 -- 172.21.15.62:0/3136875730 wait complete. 2024-09-15T23:19:53.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.079+0000 7f86baa77740 20 client.25946 trim_cache size 0 max 16384 2024-09-15T23:19:53.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.103+0000 7f86baa77740 1 Processor -- start 2024-09-15T23:19:53.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.103+0000 7f86baa77740 1 -- start start 2024-09-15T23:19:53.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86baa77740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 0x556cb6f6dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86baa77740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f77b40 0x556cb6f8d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86baa77740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f81370 0x556cb6f7c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86baa77740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f86940124f0 con 0x556cb6f81370 2024-09-15T23:19:53.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86baa77740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f869400fdc0 con 0x556cb6f77b40 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86baa77740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f869400dc10 con 0x556cb6f765d0 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b643a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 0x556cb6f6dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b643a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 0x556cb6f6dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:37874/0 (socket says 172.21.15.62:37874) 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b643a640 1 -- 172.21.15.62:0/828099966 learned_addr learned my addr 172.21.15.62:0/828099966 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b5c39640 1 --2- 172.21.15.62:0/828099966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f77b40 0x556cb6f8d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/828099966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f81370 0x556cb6f7c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b643a640 1 -- 172.21.15.62:0/828099966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f77b40 msgr2=0x556cb6f8d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b643a640 1 --2- 172.21.15.62:0/828099966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f77b40 0x556cb6f8d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b643a640 1 -- 172.21.15.62:0/828099966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f81370 msgr2=0x556cb6f7c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b643a640 1 --2- 172.21.15.62:0/828099966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f81370 0x556cb6f7c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.104+0000 7f86b643a640 1 -- 172.21.15.62:0/828099966 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cb6f72ed0 con 0x556cb6f765d0 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.105+0000 7f86b643a640 1 --2- 172.21.15.62:0/828099966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 0x556cb6f6dc60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f86a8000f80 tx=0x7f86a809e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f04b2a4f6c64b219 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.105+0000 7f869bfff640 1 -- 172.21.15.62:0/828099966 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a809b500 con 0x556cb6f765d0 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.105+0000 7f869bfff640 1 -- 172.21.15.62:0/828099966 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86a809f8b0 con 0x556cb6f765d0 2024-09-15T23:19:53.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.105+0000 7f869bfff640 1 -- 172.21.15.62:0/828099966 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a80a3f40 con 0x556cb6f765d0 2024-09-15T23:19:53.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.107+0000 7f86baa77740 1 -- 172.21.15.62:0/828099966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 msgr2=0x556cb6f6dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:53.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.107+0000 7f86baa77740 1 --2- 172.21.15.62:0/828099966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 0x556cb6f6dc60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f86a8000f80 tx=0x7f86a809e040 comp rx=0 tx=0).stop 2024-09-15T23:19:53.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.107+0000 7f86baa77740 1 -- 172.21.15.62:0/828099966 shutdown_connections 2024-09-15T23:19:53.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.107+0000 7f86baa77740 1 --2- 172.21.15.62:0/828099966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f81370 0x556cb6f7c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.107+0000 7f86baa77740 1 --2- 172.21.15.62:0/828099966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f77b40 0x556cb6f8d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.107+0000 7f86baa77740 1 --2- 172.21.15.62:0/828099966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f765d0 0x556cb6f6dc60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.107+0000 7f86baa77740 1 -- 172.21.15.62:0/828099966 >> 172.21.15.62:0/828099966 conn(0x556cb6f3f310 msgr2=0x556cb6f7dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:53.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.107+0000 7f86baa77740 1 -- 172.21.15.62:0/828099966 shutdown_connections 2024-09-15T23:19:53.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.107+0000 7f86baa77740 1 -- 172.21.15.62:0/828099966 wait complete. 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.108+0000 7f86baa77740 1 -- start start 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.108+0000 7f86baa77740 1 Processor -- start 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.108+0000 7f86baa77740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 0x556cb6f84b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.108+0000 7f86baa77740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f85070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b643a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 0x556cb6f84b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b5c39640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f85070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b5c39640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f85070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37184/0 (socket says 172.21.15.62:37184) 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b5c39640 1 -- 172.21.15.62:0/2707884913 learned_addr learned my addr 172.21.15.62:0/2707884913 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86baa77740 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6ff9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f869400e280 con 0x556cb6f77b40 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f869400f6f0 con 0x556cb6f765d0 2024-09-15T23:19:53.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f869400e950 con 0x556cb6f81370 2024-09-15T23:19:53.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b6c3b640 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6ff9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86baa77740 10 client.0 _ll_register_callbacks cb 0x7f86ac024040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:53.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b5c39640 1 -- 172.21.15.62:0/2707884913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 msgr2=0x556cb6ff9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:53.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b5c39640 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6ff9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b5c39640 1 -- 172.21.15.62:0/2707884913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 msgr2=0x556cb6f84b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:53.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b5c39640 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 0x556cb6f84b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b5c39640 1 -- 172.21.15.62:0/2707884913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cb6ffa050 con 0x556cb6f77b40 2024-09-15T23:19:53.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.109+0000 7f86b643a640 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 0x556cb6f84b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:19:53.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.110+0000 7f86b5c39640 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f85070 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f86a0009900 tx=0x7f86a0004440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2aac469a7f9713 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:53.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.110+0000 7f8699ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:19:53.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.111+0000 7f8699ffb640 1 -- 172.21.15.62:0/2707884913 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a0035040 con 0x556cb6f77b40 2024-09-15T23:19:53.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.111+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86a809f8b0 con 0x556cb6f77b40 2024-09-15T23:19:53.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.111+0000 7f8699ffb640 1 -- 172.21.15.62:0/2707884913 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f86a0005220 con 0x556cb6f77b40 2024-09-15T23:19:53.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.112+0000 7f8699ffb640 1 -- 172.21.15.62:0/2707884913 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86a004d050 con 0x556cb6f77b40 2024-09-15T23:19:53.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.112+0000 7f86baa77740 10 client.16323 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:53.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.112+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556cb6ffb310 con 0x556cb6f77b40 2024-09-15T23:19:53.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.119+0000 7f8699ffb640 1 -- 172.21.15.62:0/2707884913 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86a809f8b0 con 0x556cb6f77b40 2024-09-15T23:19:53.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.119+0000 7f8699ffb640 1 -- 172.21.15.62:0/2707884913 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f86a0031810 con 0x556cb6f77b40 2024-09-15T23:19:53.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.119+0000 7f86baa77740 10 client.16323 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:53.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.119+0000 7f86baa77740 10 client.16323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:53.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.119+0000 7f86baa77740 20 client.16323 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:53.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.119+0000 7f86baa77740 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556cb6ffbad0 0x556cb6ffdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:53.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.120+0000 7f86baa77740 4 client.16323 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:19:53.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.120+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556cb6ee3d70 con 0x556cb6ffbad0 2024-09-15T23:19:53.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.120+0000 7f86b643a640 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556cb6ffbad0 0x556cb6ffdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:53.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.121+0000 7f86b643a640 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556cb6ffbad0 0x556cb6ffdef0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb60c3e0ae389607 server_cookie=58c07822a51e04c1 in_seq=0 out_seq=0 2024-09-15T23:19:53.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.121+0000 7f8699ffb640 10 client.16323 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.121+0000 7f8699ffb640 1 -- 172.21.15.62:0/2707884913 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556cb6ee3d70 con 0x556cb6ffbad0 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.121+0000 7f8699ffb640 10 client.16323 handle_command_reply: tid=0 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.121+0000 7f86baa77740 1 client.16323 shutdown 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 msgr2=0x556cb6f85070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f85070 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f86a0009900 tx=0x7f86a0004440 comp rx=0 tx=0).stop 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 shutdown_connections 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556cb6ffbad0 0x556cb6ffdef0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556cb6f81370 0x556cb6ff9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f8699ffb640 0 client.16323 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556cb6f77b40 0x556cb6f85070 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 --2- 172.21.15.62:0/2707884913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556cb6f765d0 0x556cb6f84b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 >> 172.21.15.62:0/2707884913 conn(0x556cb6f3f310 msgr2=0x556cb6f7f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:53.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 shutdown_connections 2024-09-15T23:19:53.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 1 -- 172.21.15.62:0/2707884913 wait complete. 2024-09-15T23:19:53.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.122+0000 7f86baa77740 20 client.16323 trim_cache size 0 max 16384 2024-09-15T23:19:53.125 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 -- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f869c0a3460 msgr2=0x7f869c0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f869c0a3460 0x7f869c0a5920 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f86ac01d000 tx=0x7f86ac01a000 comp rx=0 tx=0).stop 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 -- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b01091f0 msgr2=0x7f86b01a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b01091f0 0x7f86b01a5a60 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f86a800ba50 tx=0x7f86a800bf10 comp rx=0 tx=0).stop 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 -- 172.21.15.62:0/1707766324 shutdown_connections 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f869c0a3460 0x7f869c0a5920 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86b0114ab0 0x7f86b019f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86b0109b10 0x7f86b019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 --2- 172.21.15.62:0/1707766324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86b01091f0 0x7f86b01a5a60 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 -- 172.21.15.62:0/1707766324 >> 172.21.15.62:0/1707766324 conn(0x7f86b006e140 msgr2=0x7f86b0070a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:53.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 -- 172.21.15.62:0/1707766324 shutdown_connections 2024-09-15T23:19:53.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:53.123+0000 7f86b869e640 1 -- 172.21.15.62:0/1707766324 wait complete. 2024-09-15T23:19:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:53 smithi062 ceph-mon[30949]: pgmap v478: 129 pgs: 129 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 828 op/s 2024-09-15T23:19:53.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:53 smithi067 ceph-mon[35665]: pgmap v478: 129 pgs: 129 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 828 op/s 2024-09-15T23:19:53.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:53 smithi148 ceph-mon[35610]: pgmap v478: 129 pgs: 129 active+clean; 1.1 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 828 op/s 2024-09-15T23:19:54.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 215.0 s, 1793.0 tps, lat 55.427 ms stddev 62.785 2024-09-15T23:19:55.118 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:19:55.119 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:19:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:55 smithi062 ceph-mon[30949]: pgmap v479: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-09-15T23:19:55.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:55 smithi067 ceph-mon[35665]: pgmap v479: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-09-15T23:19:55.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:55 smithi148 ceph-mon[35610]: pgmap v479: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 11 MiB/s wr, 822 op/s 2024-09-15T23:19:55.551 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:56.703 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 13.8 2024-09-15T23:19:56.704 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:19:57.100 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:19:57.100 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:19:57.100 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "16673540-a866-4cf4-a98d-e9b9354d87af", 2024-09-15T23:19:57.101 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:19:57.101 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:19:57.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 -- 172.21.15.62:0/504822838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 msgr2=0x7f91a81021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:57.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 --2- 172.21.15.62:0/504822838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 0x7f91a81021f0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9198009900 tx=0x7f919802a410 comp rx=0 tx=0).stop 2024-09-15T23:19:57.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 -- 172.21.15.62:0/504822838 shutdown_connections 2024-09-15T23:19:57.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 --2- 172.21.15.62:0/504822838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91a8106f40 0x7f91a810eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 --2- 172.21.15.62:0/504822838 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91a81027c0 0x7f91a8106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 --2- 172.21.15.62:0/504822838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 0x7f91a81021f0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 -- 172.21.15.62:0/504822838 >> 172.21.15.62:0/504822838 conn(0x7f91a80fd8d0 msgr2=0x7f91a80ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:57.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 -- 172.21.15.62:0/504822838 shutdown_connections 2024-09-15T23:19:57.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 -- 172.21.15.62:0/504822838 wait complete. 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 Processor -- start 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.212+0000 7f91acc2c640 1 -- start start 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91acc2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 0x7f91a8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91acc2c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91a81027c0 0x7f91a8198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91acc2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91a8106f40 0x7f91a819cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91acc2c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91a8111fc0 con 0x7f91a8101e10 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91acc2c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91a8111e40 con 0x7f91a8106f40 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91acc2c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91a8112140 con 0x7f91a81027c0 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 0x7f91a8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 0x7f91a8198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37204/0 (socket says 172.21.15.62:37204) 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a659c640 1 -- 172.21.15.62:0/1389830851 learned_addr learned my addr 172.21.15.62:0/1389830851 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a6d9d640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91a8106f40 0x7f91a819cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a5d9b640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91a81027c0 0x7f91a8198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a659c640 1 -- 172.21.15.62:0/1389830851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91a81027c0 msgr2=0x7f91a8198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a659c640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91a81027c0 0x7f91a8198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a659c640 1 -- 172.21.15.62:0/1389830851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91a8106f40 msgr2=0x7f91a819cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a659c640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91a8106f40 0x7f91a819cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a659c640 1 -- 172.21.15.62:0/1389830851 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a819d4b0 con 0x7f91a8101e10 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a5d9b640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91a81027c0 0x7f91a8198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.213+0000 7f91a659c640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 0x7f91a8198500 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9198005ab0 tx=0x7f91980042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.214+0000 7f91977fe640 1 -- 172.21.15.62:0/1389830851 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919803f030 con 0x7f91a8101e10 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.214+0000 7f91977fe640 1 -- 172.21.15.62:0/1389830851 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9198003ce0 con 0x7f91a8101e10 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.214+0000 7f91acc2c640 1 -- 172.21.15.62:0/1389830851 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a819d740 con 0x7f91a8101e10 2024-09-15T23:19:57.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.214+0000 7f91977fe640 1 -- 172.21.15.62:0/1389830851 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9198035070 con 0x7f91a8101e10 2024-09-15T23:19:57.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.214+0000 7f91acc2c640 1 -- 172.21.15.62:0/1389830851 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a81a5080 con 0x7f91a8101e10 2024-09-15T23:19:57.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.216+0000 7f91977fe640 1 -- 172.21.15.62:0/1389830851 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9198042030 con 0x7f91a8101e10 2024-09-15T23:19:57.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.216+0000 7f91af005740 1 Processor -- start 2024-09-15T23:19:57.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.217+0000 7f91af005740 1 -- start start 2024-09-15T23:19:57.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91af005740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cb8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91af005740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91af005740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cc8150 0x558585cca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91af005740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558585c1e280 con 0x558585cb8a00 2024-09-15T23:19:57.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91af005740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558585bc5b10 con 0x558585cb9320 2024-09-15T23:19:57.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91af005740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558585c1a240 con 0x558585cc8150 2024-09-15T23:19:57.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91977fe640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91900a3240 0x7f91900a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91a659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91a659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:35698/0 (socket says 172.21.15.62:35698) 2024-09-15T23:19:57.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91a659c640 1 -- 172.21.15.62:0/69250513 learned_addr learned my addr 172.21.15.62:0/69250513 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91a5d9b640 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cc8150 0x558585cca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.218+0000 7f91a6d9d640 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cb8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.219+0000 7f91977fe640 1 -- 172.21.15.62:0/1389830851 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f91980f05e0 con 0x7f91a8101e10 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.219+0000 7f91a6d9d640 1 -- 172.21.15.62:0/69250513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cc8150 msgr2=0x558585cca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.219+0000 7f91a6d9d640 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cc8150 0x558585cca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.219+0000 7f91a6d9d640 1 -- 172.21.15.62:0/69250513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 msgr2=0x558585cc7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.219+0000 7f91a6d9d640 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.219+0000 7f91a6d9d640 1 -- 172.21.15.62:0/69250513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558585bd21f0 con 0x558585cb8a00 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.219+0000 7f91a659c640 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.219+0000 7f91a6d9d640 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cb8de0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f91a000aed0 tx=0x7f91a00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680ce6362d89e095 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.220+0000 7f9194ff9640 1 -- 172.21.15.62:0/69250513 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0003b70 con 0x558585cb8a00 2024-09-15T23:19:57.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.220+0000 7f9194ff9640 1 -- 172.21.15.62:0/69250513 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91a0003d10 con 0x558585cb8a00 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.220+0000 7f9194ff9640 1 -- 172.21.15.62:0/69250513 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0012d40 con 0x558585cb8a00 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.220+0000 7f91a6d9d640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91900a3240 0x7f91900a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91a6d9d640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91900a3240 0x7f91900a5700 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f91a001c040 tx=0x7f91a0012ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 -- 172.21.15.62:0/69250513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 msgr2=0x558585cb8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cb8de0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f91a000aed0 tx=0x7f91a00117b0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 -- 172.21.15.62:0/69250513 shutdown_connections 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cc8150 0x558585cca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 --2- 172.21.15.62:0/69250513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cb8de0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 -- 172.21.15.62:0/69250513 >> 172.21.15.62:0/69250513 conn(0x558585cb50f0 msgr2=0x558585cb54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 -- 172.21.15.62:0/69250513 shutdown_connections 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 -- 172.21.15.62:0/69250513 wait complete. 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 -- start start 2024-09-15T23:19:57.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.221+0000 7f91af005740 1 Processor -- start 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91af005740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cc5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91af005740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91af005740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cc8150 0x558585cc5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91af005740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558585ccad20 con 0x558585cb8a00 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91af005740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558585bd8f10 con 0x558585cb9320 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91af005740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558585ccaea0 con 0x558585cc8150 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91af005740 10 client.0 _ll_register_callbacks cb 0x558585ba1a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91a5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91a5d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:35710/0 (socket says 172.21.15.62:35710) 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91a5d9b640 1 -- 172.21.15.62:0/2930600401 learned_addr learned my addr 172.21.15.62:0/2930600401 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91a659c640 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cc5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91a5d9b640 1 -- 172.21.15.62:0/2930600401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cc8150 msgr2=0x558585cc5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91a5d9b640 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cc8150 0x558585cc5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91a5d9b640 1 -- 172.21.15.62:0/2930600401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 msgr2=0x558585cc5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91a5d9b640 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cc5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.222+0000 7f91a5d9b640 1 -- 172.21.15.62:0/2930600401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55858598f6a0 con 0x558585cb9320 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.223+0000 7f91a5d9b640 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc5560 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f919c00e250 tx=0x7f919c00e710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5facce7f95a099bd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.223+0000 7f918dffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.223+0000 7f918dffb640 1 -- 172.21.15.62:0/2930600401 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c018020 con 0x558585cb9320 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.223+0000 7f918dffb640 1 -- 172.21.15.62:0/2930600401 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f919c013660 con 0x558585cb9320 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.223+0000 7f91af005740 1 -- 172.21.15.62:0/2930600401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91a0003d10 con 0x558585cb9320 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.226+0000 7f918dffb640 1 -- 172.21.15.62:0/2930600401 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f919c023050 con 0x558585cb9320 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.226+0000 7f91af005740 10 client.25961 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:57.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.226+0000 7f91af005740 1 -- 172.21.15.62:0/2930600401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558585cd6930 con 0x558585cb9320 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.226+0000 7f918dffb640 1 -- 172.21.15.62:0/2930600401 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91a0003d10 con 0x558585cb9320 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f918dffb640 1 -- 172.21.15.62:0/2930600401 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f919c0149d0 con 0x558585cb9320 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f91af005740 10 client.25961 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f91af005740 10 client.25961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f91af005740 20 client.25961 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f91af005740 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558585cd7f10 0x558585cda3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f91af005740 4 client.25961 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f91af005740 1 -- 172.21.15.62:0/2930600401 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558585c2b7f0 con 0x558585cd7f10 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f91a659c640 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558585cd7f10 0x558585cda3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f918dffb640 10 client.25961 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.227+0000 7f91a659c640 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558585cd7f10 0x558585cda3b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6390f94577784cf7 server_cookie=d16168775ea0c04e in_seq=0 out_seq=0 2024-09-15T23:19:57.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f918dffb640 1 -- 172.21.15.62:0/2930600401 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558585c2b7f0 con 0x558585cd7f10 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f918dffb640 10 client.25961 handle_command_reply: tid=0 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f91af005740 1 client.25961 shutdown 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f91af005740 1 -- 172.21.15.62:0/2930600401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 msgr2=0x558585cc5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f91af005740 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc5560 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f919c00e250 tx=0x7f919c00e710 comp rx=0 tx=0).stop 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f91af005740 1 -- 172.21.15.62:0/2930600401 shutdown_connections 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f91af005740 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558585cd7f10 0x558585cda3b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f91af005740 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cc8150 0x558585cc5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f918dffb640 0 client.25961 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f91af005740 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb9320 0x558585cc5560 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f91af005740 1 --2- 172.21.15.62:0/2930600401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cc5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.228+0000 7f91af005740 1 -- 172.21.15.62:0/2930600401 >> 172.21.15.62:0/2930600401 conn(0x558585cb50f0 msgr2=0x558585cbae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.229+0000 7f91af005740 1 -- 172.21.15.62:0/2930600401 shutdown_connections 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.229+0000 7f91af005740 1 -- 172.21.15.62:0/2930600401 wait complete. 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.229+0000 7f91af005740 20 client.25961 trim_cache size 0 max 16384 2024-09-15T23:19:57.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.268+0000 7f91af005740 1 Processor -- start 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.268+0000 7f91af005740 1 -- start start 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.268+0000 7f91af005740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb8a00 0x558585cc0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.268+0000 7f91af005740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585cc6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.268+0000 7f91af005740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cc8150 0x558585cdfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.268+0000 7f91af005740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91a81af330 con 0x558585cc8150 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.268+0000 7f91af005740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91a81acc00 con 0x558585cb8a00 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.268+0000 7f91af005740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91a81aaa50 con 0x558585cb9320 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb8a00 0x558585cc0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a5d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585cc6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a5d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585cc6e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:37908/0 (socket says 172.21.15.62:37908) 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a5d9b640 1 -- 172.21.15.62:0/2432698302 learned_addr learned my addr 172.21.15.62:0/2432698302 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a6d9d640 1 --2- 172.21.15.62:0/2432698302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cc8150 0x558585cdfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a659c640 1 -- 172.21.15.62:0/2432698302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 msgr2=0x558585cc6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a659c640 1 --2- 172.21.15.62:0/2432698302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585cc6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a659c640 1 -- 172.21.15.62:0/2432698302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cc8150 msgr2=0x558585cdfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a659c640 1 --2- 172.21.15.62:0/2432698302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cc8150 0x558585cdfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.269+0000 7f91a659c640 1 -- 172.21.15.62:0/2432698302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558585cb4b90 con 0x558585cb8a00 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.272+0000 7f91a659c640 1 --2- 172.21.15.62:0/2432698302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb8a00 0x558585cc0790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f91980b8c80 tx=0x7f91980edbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2708d07ccb8be3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.276+0000 7f918ffff640 1 -- 172.21.15.62:0/2432698302 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9198049040 con 0x558585cb8a00 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.276+0000 7f918ffff640 1 -- 172.21.15.62:0/2432698302 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91980bc820 con 0x558585cb8a00 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.277+0000 7f91af005740 1 -- 172.21.15.62:0/2432698302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb8a00 msgr2=0x558585cc0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.277+0000 7f91af005740 1 --2- 172.21.15.62:0/2432698302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb8a00 0x558585cc0790 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f91980b8c80 tx=0x7f91980edbd0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.281+0000 7f91af005740 1 -- 172.21.15.62:0/2432698302 shutdown_connections 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.281+0000 7f91af005740 1 --2- 172.21.15.62:0/2432698302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cc8150 0x558585cdfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.281+0000 7f91af005740 1 --2- 172.21.15.62:0/2432698302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585cc6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.281+0000 7f91af005740 1 --2- 172.21.15.62:0/2432698302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cb8a00 0x558585cc0790 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.281+0000 7f91af005740 1 -- 172.21.15.62:0/2432698302 >> 172.21.15.62:0/2432698302 conn(0x558585cb50f0 msgr2=0x558585cc6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:57.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.282+0000 7f91af005740 1 -- 172.21.15.62:0/2432698302 shutdown_connections 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.282+0000 7f91af005740 1 -- 172.21.15.62:0/2432698302 wait complete. 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.282+0000 7f91af005740 1 -- start start 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.283+0000 7f91af005740 1 Processor -- start 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.284+0000 7f91af005740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cd0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.284+0000 7f91af005740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585ccbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.284+0000 7f91af005740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cc8150 0x558585ccc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.284+0000 7f91af005740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91a81ab0c0 con 0x558585cb8a00 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.284+0000 7f91af005740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91a81ac530 con 0x558585cc8150 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.284+0000 7f91af005740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91a81ab790 con 0x558585cb9320 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.284+0000 7f91af005740 10 client.0 _ll_register_callbacks cb 0x7f919c00c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.285+0000 7f91a5d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585ccbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.285+0000 7f91a6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cc8150 0x558585ccc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.285+0000 7f91a5d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585ccbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:37914/0 (socket says 172.21.15.62:37914) 2024-09-15T23:19:57.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.285+0000 7f91a5d9b640 1 -- 172.21.15.62:0/1615741605 learned_addr learned my addr 172.21.15.62:0/1615741605 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.286+0000 7f91a659c640 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cd0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.287+0000 7f91a5d9b640 1 -- 172.21.15.62:0/1615741605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cc8150 msgr2=0x558585ccc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.287+0000 7f91a5d9b640 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cc8150 0x558585ccc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.287+0000 7f91a5d9b640 1 -- 172.21.15.62:0/1615741605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 msgr2=0x558585cd0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.287+0000 7f91a5d9b640 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cd0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.287+0000 7f91a5d9b640 1 -- 172.21.15.62:0/1615741605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558585ccc8d0 con 0x558585cb9320 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.287+0000 7f91a5d9b640 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585ccbbb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f919c010010 tx=0x7f919c00c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8adaf659af5d5c58 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.290+0000 7f918dffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.290+0000 7f918dffb640 1 -- 172.21.15.62:0/1615741605 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919c02e060 con 0x558585cb9320 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.290+0000 7f918dffb640 1 -- 172.21.15.62:0/1615741605 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f919c014810 con 0x558585cb9320 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.291+0000 7f91af005740 1 -- 172.21.15.62:0/1615741605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f91980bc820 con 0x558585cb9320 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.296+0000 7f918dffb640 1 -- 172.21.15.62:0/1615741605 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f919c018040 con 0x558585cb9320 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.296+0000 7f91af005740 10 client.25804 fetch_fsmap learned FSMap version 29 2024-09-15T23:19:57.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.296+0000 7f91af005740 1 -- 172.21.15.62:0/1615741605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558585d53d80 con 0x558585cb9320 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.298+0000 7f918dffb640 1 -- 172.21.15.62:0/1615741605 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91980bc820 con 0x558585cb9320 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.298+0000 7f918dffb640 1 -- 172.21.15.62:0/1615741605 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f919c0295a0 con 0x558585cb9320 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.298+0000 7f91af005740 10 client.25804 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.298+0000 7f91af005740 10 client.25804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.298+0000 7f91af005740 20 client.25804 populate_metadata read hostname 'smithi062' 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.298+0000 7f91af005740 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558585d54540 0x558585d56960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.298+0000 7f91af005740 4 client.25804 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.298+0000 7f91af005740 1 -- 172.21.15.62:0/1615741605 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558585c25d00 con 0x558585d54540 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.298+0000 7f91a659c640 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558585d54540 0x558585d56960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:57.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.302+0000 7f91a659c640 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558585d54540 0x558585d56960 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd9a5627e8b6c2f3 server_cookie=ecfa39e10bf6a341 in_seq=0 out_seq=0 2024-09-15T23:19:57.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.302+0000 7f918dffb640 10 client.25804 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:57.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.303+0000 7f918dffb640 1 -- 172.21.15.62:0/1615741605 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558585c25d00 con 0x558585d54540 2024-09-15T23:19:57.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.303+0000 7f918dffb640 10 client.25804 handle_command_reply: tid=0 2024-09-15T23:19:57.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.303+0000 7f91af005740 1 client.25804 shutdown 2024-09-15T23:19:57.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f91af005740 1 -- 172.21.15.62:0/1615741605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 msgr2=0x558585ccbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:57.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f91af005740 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585ccbbb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f919c010010 tx=0x7f919c00c4e0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f91af005740 1 -- 172.21.15.62:0/1615741605 shutdown_connections 2024-09-15T23:19:57.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f91af005740 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558585d54540 0x558585d56960 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f91af005740 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558585cc8150 0x558585ccc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f91af005740 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558585cb9320 0x558585ccbbb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f91af005740 1 --2- 172.21.15.62:0/1615741605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558585cb8a00 0x558585cd0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f91af005740 1 -- 172.21.15.62:0/1615741605 >> 172.21.15.62:0/1615741605 conn(0x558585cb50f0 msgr2=0x558585cb4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f918dffb640 0 client.25804 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.305+0000 7f91af005740 1 -- 172.21.15.62:0/1615741605 shutdown_connections 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.306+0000 7f91af005740 1 -- 172.21.15.62:0/1615741605 wait complete. 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.306+0000 7f91af005740 20 client.25804 trim_cache size 0 max 16384 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.310+0000 7f91acc2c640 1 -- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91900a3240 msgr2=0x7f91900a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.310+0000 7f91acc2c640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91900a3240 0x7f91900a5700 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f91a001c040 tx=0x7f91a0012ee0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.310+0000 7f91acc2c640 1 -- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 msgr2=0x7f91a8198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.310+0000 7f91acc2c640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 0x7f91a8198500 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9198005ab0 tx=0x7f91980042f0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.312+0000 7f91acc2c640 1 -- 172.21.15.62:0/1389830851 shutdown_connections 2024-09-15T23:19:57.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.312+0000 7f91acc2c640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91900a3240 0x7f91900a5700 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.312+0000 7f91acc2c640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91a8106f40 0x7f91a819cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.312+0000 7f91acc2c640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f91a81027c0 0x7f91a8198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.312+0000 7f91acc2c640 1 --2- 172.21.15.62:0/1389830851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91a8101e10 0x7f91a8198500 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:57.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.312+0000 7f91acc2c640 1 -- 172.21.15.62:0/1389830851 >> 172.21.15.62:0/1389830851 conn(0x7f91a80fd8d0 msgr2=0x7f91a80fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:57.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.312+0000 7f91acc2c640 1 -- 172.21.15.62:0/1389830851 shutdown_connections 2024-09-15T23:19:57.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:56.312+0000 7f91acc2c640 1 -- 172.21.15.62:0/1389830851 wait complete. 2024-09-15T23:19:57.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:57 smithi062 ceph-mon[30949]: pgmap v480: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 MiB/s wr, 782 op/s 2024-09-15T23:19:57.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:57 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:19:57.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:57 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:57.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:57 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:19:57.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:57 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:19:57.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:57 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:57.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:57 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:19:57.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:57 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:19:57.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:57 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:57 smithi067 ceph-mon[35665]: pgmap v480: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 MiB/s wr, 782 op/s 2024-09-15T23:19:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:57 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:19:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:57 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:57 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:19:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:57 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:19:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:57 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:57 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:19:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:57 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:19:57.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:57 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:57.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:57 smithi148 ceph-mon[35610]: pgmap v480: 129 pgs: 129 active+clean; 1.1 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 10 MiB/s wr, 782 op/s 2024-09-15T23:19:57.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:57 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:19:57.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:57 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:57.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:57 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:19:57.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:57 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:19:57.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:57 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:57.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:57 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:19:57.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:57 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:19:57.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:57 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:58.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:58 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:58.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:58 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:58.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:58 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:58.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:58 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:58.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:58 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:58.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:58 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:19:59.076 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:19:59.093 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:19:59.256 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:19:59 smithi062 ceph-mon[30949]: pgmap v481: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.6 MiB/s wr, 662 op/s 2024-09-15T23:19:59.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:19:59 smithi067 ceph-mon[35665]: pgmap v481: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.6 MiB/s wr, 662 op/s 2024-09-15T23:19:59.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:19:59 smithi148 ceph-mon[35610]: pgmap v481: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.6 MiB/s wr, 662 op/s 2024-09-15T23:19:59.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 220.0 s, 1263.6 tps, lat 79.808 ms stddev 98.620 2024-09-15T23:19:59.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.749+0000 7fd6d3278640 1 -- 172.21.15.62:0/487332168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6cc105ec0 msgr2=0x7fd6cc106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:59.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.749+0000 7fd6d3278640 1 --2- 172.21.15.62:0/487332168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6cc105ec0 0x7fd6cc106340 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009cd0 tx=0x7fd6c402a910 comp rx=0 tx=0).stop 2024-09-15T23:19:59.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.749+0000 7fd6d3278640 1 -- 172.21.15.62:0/487332168 shutdown_connections 2024-09-15T23:19:59.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.749+0000 7fd6d3278640 1 --2- 172.21.15.62:0/487332168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6cc105ec0 0x7fd6cc106340 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:59.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.749+0000 7fd6d3278640 1 --2- 172.21.15.62:0/487332168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc1011e0 0x7fd6cc105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:59.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.749+0000 7fd6d3278640 1 --2- 172.21.15.62:0/487332168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6cc1008c0 0x7fd6cc100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:59.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.749+0000 7fd6d3278640 1 -- 172.21.15.62:0/487332168 >> 172.21.15.62:0/487332168 conn(0x7fd6cc0fc7d0 msgr2=0x7fd6cc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.751+0000 7fd6d3278640 1 -- 172.21.15.62:0/487332168 shutdown_connections 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.753+0000 7fd6d3278640 1 -- 172.21.15.62:0/487332168 wait complete. 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.754+0000 7fd6d3278640 1 Processor -- start 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.754+0000 7fd6d3278640 1 -- start start 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.754+0000 7fd6d3278640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6cc1008c0 0x7fd6cc10a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.754+0000 7fd6d3278640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc1011e0 0x7fd6cc10a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.754+0000 7fd6d3278640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6cc105ec0 0x7fd6cc072aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.754+0000 7fd6d3278640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc118610 con 0x7fd6cc105ec0 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.754+0000 7fd6d3278640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc118490 con 0x7fd6cc1008c0 2024-09-15T23:19:59.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.754+0000 7fd6d3278640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd6cc118790 con 0x7fd6cc1011e0 2024-09-15T23:19:59.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.756+0000 7fd6d1815640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6cc105ec0 0x7fd6cc072aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:59.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.756+0000 7fd6d1815640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6cc105ec0 0x7fd6cc072aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37238/0 (socket says 172.21.15.62:37238) 2024-09-15T23:19:59.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.756+0000 7fd6d1815640 1 -- 172.21.15.62:0/133227180 learned_addr learned my addr 172.21.15.62:0/133227180 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:19:59.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.757+0000 7fd6d1014640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6cc1008c0 0x7fd6cc10a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:59.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.757+0000 7fd6d0813640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc1011e0 0x7fd6cc10a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:59.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.760+0000 7fd6d1815640 1 -- 172.21.15.62:0/133227180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc1011e0 msgr2=0x7fd6cc10a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:59.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.760+0000 7fd6d1815640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc1011e0 0x7fd6cc10a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:59.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.760+0000 7fd6d1815640 1 -- 172.21.15.62:0/133227180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6cc1008c0 msgr2=0x7fd6cc10a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:19:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.760+0000 7fd6d1815640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6cc1008c0 0x7fd6cc10a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:19:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.760+0000 7fd6d1815640 1 -- 172.21.15.62:0/133227180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6cc073360 con 0x7fd6cc105ec0 2024-09-15T23:19:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.761+0000 7fd6d1815640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6cc105ec0 0x7fd6cc072aa0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4031700 tx=0x7fd6c4031730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:59.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.762+0000 7fd6c27fc640 1 -- 172.21.15.62:0/133227180 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4003b70 con 0x7fd6cc105ec0 2024-09-15T23:19:59.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.762+0000 7fd6c27fc640 1 -- 172.21.15.62:0/133227180 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c4031910 con 0x7fd6cc105ec0 2024-09-15T23:19:59.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.762+0000 7fd6c27fc640 1 -- 172.21.15.62:0/133227180 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c402f9a0 con 0x7fd6cc105ec0 2024-09-15T23:19:59.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.765+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6cc0735f0 con 0x7fd6cc105ec0 2024-09-15T23:19:59.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.765+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6cc1122f0 con 0x7fd6cc105ec0 2024-09-15T23:19:59.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.770+0000 7fd6c27fc640 1 -- 172.21.15.62:0/133227180 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd6c402fb40 con 0x7fd6cc105ec0 2024-09-15T23:19:59.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.770+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6b0005290 con 0x7fd6cc105ec0 2024-09-15T23:19:59.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.771+0000 7fd6c27fc640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd6b80a3450 0x7fd6b80a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:19:59.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.772+0000 7fd6d1014640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd6b80a3450 0x7fd6b80a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:19:59.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.772+0000 7fd6c27fc640 1 -- 172.21.15.62:0/133227180 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd6c40f0f60 con 0x7fd6cc105ec0 2024-09-15T23:19:59.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.776+0000 7fd6d1014640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd6b80a3450 0x7fd6b80a5910 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009bd0 tx=0x7fd6c8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:19:59.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:19:59.782+0000 7fd6c27fc640 1 -- 172.21.15.62:0/133227180 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd6c40bd8a0 con 0x7fd6cc105ec0 2024-09-15T23:19:59.789 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:00.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.167+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6b0005580 con 0x7fd6cc105ec0 2024-09-15T23:20:00.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.172+0000 7fd6c27fc640 1 -- 172.21.15.62:0/133227180 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11906 (secure 0 0 0) 0x7fd6c40bd1a0 con 0x7fd6cc105ec0 2024-09-15T23:20:00.174 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:20:00.174 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":29,"btime":"2024-09-15T23:19:37:942586+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:19:37.876324+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:20:00.174 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 29 2024-09-15T23:20:00.175 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:00 smithi062 ceph-mon[30949]: overall HEALTH_OK 2024-09-15T23:20:00.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd6b80a3450 msgr2=0x7fd6b80a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd6b80a3450 0x7fd6b80a5910 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009bd0 tx=0x7fd6c8005f90 comp rx=0 tx=0).stop 2024-09-15T23:20:00.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6cc105ec0 msgr2=0x7fd6cc072aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6cc105ec0 0x7fd6cc072aa0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4031700 tx=0x7fd6c4031730 comp rx=0 tx=0).stop 2024-09-15T23:20:00.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 shutdown_connections 2024-09-15T23:20:00.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd6b80a3450 0x7fd6b80a5910 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6cc105ec0 0x7fd6cc072aa0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd6cc1011e0 0x7fd6cc10a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 --2- 172.21.15.62:0/133227180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6cc1008c0 0x7fd6cc10a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.177+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 >> 172.21.15.62:0/133227180 conn(0x7fd6cc0fc7d0 msgr2=0x7fd6cc103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:00.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.178+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 shutdown_connections 2024-09-15T23:20:00.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.178+0000 7fd6d3278640 1 -- 172.21.15.62:0/133227180 wait complete. 2024-09-15T23:20:00.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:00 smithi067 ceph-mon[35665]: overall HEALTH_OK 2024-09-15T23:20:00.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:00 smithi148 ceph-mon[35610]: overall HEALTH_OK 2024-09-15T23:20:00.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.860+0000 7fc1d10ff640 1 -- 172.21.15.62:0/4092141040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc102a50 msgr2=0x7fc1cc102e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.860+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/4092141040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc102a50 0x7fc1cc102e30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009900 tx=0x7fc1bc02a410 comp rx=0 tx=0).stop 2024-09-15T23:20:00.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.860+0000 7fc1d10ff640 1 -- 172.21.15.62:0/4092141040 shutdown_connections 2024-09-15T23:20:00.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.860+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/4092141040 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1cc100c80 0x7fc1cc110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.860+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/4092141040 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1cc100100 0x7fc1cc100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.860+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/4092141040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc102a50 0x7fc1cc102e30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.860+0000 7fc1d10ff640 1 -- 172.21.15.62:0/4092141040 >> 172.21.15.62:0/4092141040 conn(0x7fc1cc077230 msgr2=0x7fc1cc0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:00.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.860+0000 7fc1d10ff640 1 -- 172.21.15.62:0/4092141040 shutdown_connections 2024-09-15T23:20:00.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.860+0000 7fc1d10ff640 1 -- 172.21.15.62:0/4092141040 wait complete. 2024-09-15T23:20:00.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.861+0000 7fc1d10ff640 1 Processor -- start 2024-09-15T23:20:00.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.862+0000 7fc1d10ff640 1 -- start start 2024-09-15T23:20:00.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.862+0000 7fc1d10ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc100100 0x7fc1cc19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.862+0000 7fc1d10ff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1cc100c80 0x7fc1cc19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.862+0000 7fc1d10ff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1cc102a50 0x7fc1cc19ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1cad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc100100 0x7fc1cc19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1cad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc100100 0x7fc1cc19a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:35768/0 (socket says 172.21.15.62:35768) 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1cad9d640 1 -- 172.21.15.62:0/1995988321 learned_addr learned my addr 172.21.15.62:0/1995988321 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc114170 con 0x7fc1cc102a50 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc113ff0 con 0x7fc1cc100100 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1142f0 con 0x7fc1cc100c80 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1cb59e640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1cc102a50 0x7fc1cc19ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1ca59c640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1cc100c80 0x7fc1cc19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1cad9d640 1 -- 172.21.15.62:0/1995988321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1cc100c80 msgr2=0x7fc1cc19a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1cc100c80 0x7fc1cc19a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1cad9d640 1 -- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1cc102a50 msgr2=0x7fc1cc19ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1cc102a50 0x7fc1cc19ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.863+0000 7fc1cad9d640 1 -- 172.21.15.62:0/1995988321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1cc19f400 con 0x7fc1cc100100 2024-09-15T23:20:00.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.864+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc100100 0x7fc1cc19a450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009850 tx=0x7fc1bc0305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:00.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.864+0000 7fc1b3fff640 1 -- 172.21.15.62:0/1995988321 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc03f070 con 0x7fc1cc100100 2024-09-15T23:20:00.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.864+0000 7fc1b3fff640 1 -- 172.21.15.62:0/1995988321 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc004460 con 0x7fc1cc100100 2024-09-15T23:20:00.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.864+0000 7fc1b3fff640 1 -- 172.21.15.62:0/1995988321 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc035030 con 0x7fc1cc100100 2024-09-15T23:20:00.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.864+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1cc19f690 con 0x7fc1cc100100 2024-09-15T23:20:00.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.865+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1cc19fa60 con 0x7fc1cc100100 2024-09-15T23:20:00.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.868+0000 7fc1b3fff640 1 -- 172.21.15.62:0/1995988321 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc1bc042070 con 0x7fc1cc100100 2024-09-15T23:20:00.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1d34d8740 1 Processor -- start 2024-09-15T23:20:00.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1d34d8740 1 -- start start 2024-09-15T23:20:00.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad60880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad6b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad6b680 0x55c65ad6da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1b3fff640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1b40a3450 0x7fc1b40a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c65acc62e0 con 0x55c65ad60dc0 2024-09-15T23:20:00.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c65ac6dce0 con 0x55c65ad604a0 2024-09-15T23:20:00.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c65acc22a0 con 0x55c65ad6b680 2024-09-15T23:20:00.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1cad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad6b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1cb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad60880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1cb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad60880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:35774/0 (socket says 172.21.15.62:35774) 2024-09-15T23:20:00.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1cb59e640 1 -- 172.21.15.62:0/858544169 learned_addr learned my addr 172.21.15.62:0/858544169 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:00.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1cb59e640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1b40a3450 0x7fc1b40a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1ca59c640 1 --2- 172.21.15.62:0/858544169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad6b680 0x55c65ad6da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.869+0000 7fc1b3fff640 1 -- 172.21.15.62:0/1995988321 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc1bc0f04f0 con 0x7fc1cc100100 2024-09-15T23:20:00.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.870+0000 7fc1cb59e640 1 -- 172.21.15.62:0/858544169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad6b680 msgr2=0x55c65ad6da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:00.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.870+0000 7fc1cb59e640 1 --2- 172.21.15.62:0/858544169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad6b680 0x55c65ad6da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.870+0000 7fc1cb59e640 1 -- 172.21.15.62:0/858544169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 msgr2=0x55c65ad6b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:00.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.870+0000 7fc1cb59e640 1 --2- 172.21.15.62:0/858544169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad6b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.870+0000 7fc1cb59e640 1 -- 172.21.15.62:0/858544169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65ac52440 con 0x55c65ad604a0 2024-09-15T23:20:00.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.870+0000 7fc1cb59e640 1 --2- 172.21.15.62:0/858544169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad60880 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4005f40 tx=0x7fc1c400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea66f4c4f1075285 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:00.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.870+0000 7fc1cb59e640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1b40a3450 0x7fc1b40a5910 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400fc90 tx=0x7fc1c4018000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:00.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.870+0000 7fc1b17fa640 1 -- 172.21.15.62:0/858544169 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4022070 con 0x55c65ad604a0 2024-09-15T23:20:00.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.870+0000 7fc1b17fa640 1 -- 172.21.15.62:0/858544169 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c4003e50 con 0x55c65ad604a0 2024-09-15T23:20:00.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.871+0000 7fc1b17fa640 1 -- 172.21.15.62:0/858544169 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c40043d0 con 0x55c65ad604a0 2024-09-15T23:20:00.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.873+0000 7fc1d34d8740 1 -- 172.21.15.62:0/858544169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 msgr2=0x55c65ad60880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.873+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/858544169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad60880 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4005f40 tx=0x7fc1c400f790 comp rx=0 tx=0).stop 2024-09-15T23:20:00.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.873+0000 7fc1d34d8740 1 -- 172.21.15.62:0/858544169 shutdown_connections 2024-09-15T23:20:00.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.873+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/858544169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad6b680 0x55c65ad6da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.873+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/858544169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad6b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.873+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/858544169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad60880 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.874+0000 7fc1d34d8740 1 -- 172.21.15.62:0/858544169 >> 172.21.15.62:0/858544169 conn(0x55c65ad5ca30 msgr2=0x55c65ad5ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:00.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.874+0000 7fc1d34d8740 1 -- 172.21.15.62:0/858544169 shutdown_connections 2024-09-15T23:20:00.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.874+0000 7fc1d34d8740 1 -- 172.21.15.62:0/858544169 wait complete. 2024-09-15T23:20:00.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.874+0000 7fc1d34d8740 1 -- start start 2024-09-15T23:20:00.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.875+0000 7fc1d34d8740 1 Processor -- start 2024-09-15T23:20:00.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.875+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad6b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.875+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad60dc0 0x55c65ad682d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.875+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad6b680 0x55c65ad68870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.875+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c65ad6e250 con 0x55c65ad6b680 2024-09-15T23:20:00.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.875+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c65ac81050 con 0x55c65ad604a0 2024-09-15T23:20:00.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.875+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c65ad6e3d0 con 0x55c65ad60dc0 2024-09-15T23:20:00.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.876+0000 7fc1d34d8740 10 client.0 _ll_register_callbacks cb 0x55c65acc1160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:00.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.876+0000 7fc1cad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad6b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.876+0000 7fc1cad9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad6b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:35778/0 (socket says 172.21.15.62:35778) 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.876+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3988816769 learned_addr learned my addr 172.21.15.62:0/3988816769 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.876+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3988816769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad60dc0 msgr2=0x55c65ad682d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.876+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad60dc0 0x55c65ad682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.876+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3988816769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad6b680 msgr2=0x55c65ad68870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.876+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad6b680 0x55c65ad68870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.876+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3988816769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c65aa373c0 con 0x55c65ad604a0 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.877+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad6b020 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc005ab0 tx=0x7fc1bc040f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da3bcfbbb09a2d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.877+0000 7fc1aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.877+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3988816769 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc0d5060 con 0x55c65ad604a0 2024-09-15T23:20:00.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.877+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3988816769 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc1bc0baa30 con 0x55c65ad604a0 2024-09-15T23:20:00.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.877+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3988816769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1c4003e50 con 0x55c65ad604a0 2024-09-15T23:20:00.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3988816769 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1bc0da050 con 0x55c65ad604a0 2024-09-15T23:20:00.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1d34d8740 10 client.25985 fetch_fsmap learned FSMap version 29 2024-09-15T23:20:00.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3988816769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c65ad79c60 con 0x55c65ad604a0 2024-09-15T23:20:00.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3988816769 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c4003e50 con 0x55c65ad604a0 2024-09-15T23:20:00.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3988816769 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc1bc0b8da0 con 0x55c65ad604a0 2024-09-15T23:20:00.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1d34d8740 10 client.25985 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:20:00.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1d34d8740 10 client.25985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:00.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1d34d8740 20 client.25985 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:00.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c65ad7b160 0x55c65ad7d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1d34d8740 4 client.25985 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:00.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3988816769 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c65acd3850 con 0x55c65ad7b160 2024-09-15T23:20:00.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.878+0000 7fc1ca59c640 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c65ad7b160 0x55c65ad7d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.879+0000 7fc1aeff5640 10 client.25985 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:00.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.879+0000 7fc1ca59c640 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c65ad7b160 0x55c65ad7d620 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9e557fa78545ff7 server_cookie=c3d4a6d2ecd949b5 in_seq=0 out_seq=0 2024-09-15T23:20:00.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.882+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3988816769 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c65acd3850 con 0x55c65ad7b160 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.882+0000 7fc1aeff5640 10 client.25985 handle_command_reply: tid=0 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.882+0000 7fc1d34d8740 1 client.25985 shutdown 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3988816769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 msgr2=0x55c65ad6b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad6b020 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc005ab0 tx=0x7fc1bc040f00 comp rx=0 tx=0).stop 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3988816769 shutdown_connections 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c65ad7b160 0x55c65ad7d620 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad6b680 0x55c65ad68870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1aeff5640 0 client.25985 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad60dc0 0x55c65ad682d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3988816769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad604a0 0x55c65ad6b020 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3988816769 >> 172.21.15.62:0/3988816769 conn(0x55c65ad5ca30 msgr2=0x55c65ad62760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:00.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3988816769 shutdown_connections 2024-09-15T23:20:00.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.883+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3988816769 wait complete. 2024-09-15T23:20:00.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.884+0000 7fc1d34d8740 20 client.25985 trim_cache size 0 max 16384 2024-09-15T23:20:00.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1d34d8740 1 Processor -- start 2024-09-15T23:20:00.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1d34d8740 1 -- start start 2024-09-15T23:20:00.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad82f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad83470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 0x55c65ad65550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1b3780 con 0x55c65ad60dc0 2024-09-15T23:20:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1b1050 con 0x55c65ad6b680 2024-09-15T23:20:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1aeea0 con 0x55c65ad604a0 2024-09-15T23:20:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1cad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad82f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.911+0000 7fc1ca59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad83470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cad9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad82f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:37988/0 (socket says 172.21.15.62:37988) 2024-09-15T23:20:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3447866347 learned_addr learned my addr 172.21.15.62:0/3447866347 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cb59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 0x55c65ad65550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3447866347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 msgr2=0x55c65ad65550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3447866347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 0x55c65ad65550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3447866347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 msgr2=0x55c65ad83470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3447866347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad83470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3447866347 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65ad5c2e0 con 0x55c65ad604a0 2024-09-15T23:20:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1ca59c640 1 --2- 172.21.15.62:0/3447866347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad83470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cb59e640 1 --2- 172.21.15.62:0/3447866347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 0x55c65ad65550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3447866347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad82f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0b9100 tx=0x7fc1bc003fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97bec92f7e1bdd6d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1b0ff9640 1 -- 172.21.15.62:0/3447866347 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc0d5020 con 0x55c65ad604a0 2024-09-15T23:20:00.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1b0ff9640 1 -- 172.21.15.62:0/3447866347 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc00b0b0 con 0x55c65ad604a0 2024-09-15T23:20:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.912+0000 7fc1b0ff9640 1 -- 172.21.15.62:0/3447866347 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc049040 con 0x55c65ad604a0 2024-09-15T23:20:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.913+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3447866347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 msgr2=0x55c65ad82f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.913+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3447866347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad82f30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0b9100 tx=0x7fc1bc003fe0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.913+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3447866347 shutdown_connections 2024-09-15T23:20:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.913+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3447866347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 0x55c65ad65550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.913+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3447866347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad83470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.913+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3447866347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad82f30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.913+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3447866347 >> 172.21.15.62:0/3447866347 conn(0x55c65ad5ca30 msgr2=0x55c65ad5bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.913+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3447866347 shutdown_connections 2024-09-15T23:20:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.913+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3447866347 wait complete. 2024-09-15T23:20:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1d34d8740 1 -- start start 2024-09-15T23:20:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1d34d8740 1 Processor -- start 2024-09-15T23:20:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad71440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad71980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1d34d8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 0x55c65ad71ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1af510 con 0x55c65ad60dc0 2024-09-15T23:20:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1b0980 con 0x55c65ad6b680 2024-09-15T23:20:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1d34d8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc1cc1afbe0 con 0x55c65ad604a0 2024-09-15T23:20:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1d34d8740 10 client.0 _ll_register_callbacks cb 0x7fc1bc0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1ca59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad71980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1ca59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad71980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37300/0 (socket says 172.21.15.62:37300) 2024-09-15T23:20:00.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1ca59c640 1 -- 172.21.15.62:0/3744204893 learned_addr learned my addr 172.21.15.62:0/3744204893 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:00.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1cb59e640 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 0x55c65ad71ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.914+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad71440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.915+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3744204893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 msgr2=0x55c65ad71ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:00.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.915+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 0x55c65ad71ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.915+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3744204893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 msgr2=0x55c65ad71980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.915+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad71980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.915+0000 7fc1cad9d640 1 -- 172.21.15.62:0/3744204893 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c65ad67280 con 0x55c65ad604a0 2024-09-15T23:20:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.915+0000 7fc1cad9d640 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad71440 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0b8b90 tx=0x7fc1bc0bc350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70ce7a27443341e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.915+0000 7fc1aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:20:00.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.916+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3744204893 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc0d5070 con 0x55c65ad604a0 2024-09-15T23:20:00.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.916+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3744204893 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc1bc0f0150 con 0x55c65ad604a0 2024-09-15T23:20:00.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.916+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3744204893 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1bc00b0b0 con 0x55c65ad604a0 2024-09-15T23:20:00.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.916+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3744204893 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc1bc049070 con 0x55c65ad604a0 2024-09-15T23:20:00.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1d34d8740 10 client.25825 fetch_fsmap learned FSMap version 29 2024-09-15T23:20:00.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3744204893 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c65ad67280 con 0x55c65ad604a0 2024-09-15T23:20:00.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3744204893 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1bc00b0b0 con 0x55c65ad604a0 2024-09-15T23:20:00.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3744204893 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc1bc0c32d0 con 0x55c65ad604a0 2024-09-15T23:20:00.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1d34d8740 10 client.25825 fetch_fsmap finished waiting for FSMap version 29 2024-09-15T23:20:00.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1d34d8740 10 client.25825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1d34d8740 20 client.25825 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c65adeec20 0x55c65adf1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1d34d8740 4 client.25825 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:20:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3744204893 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c65accdd60 con 0x55c65adeec20 2024-09-15T23:20:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.917+0000 7fc1ca59c640 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c65adeec20 0x55c65adf1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:00.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1aeff5640 10 client.25825 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:00.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1ca59c640 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c65adeec20 0x55c65adf1040 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b5371c5f9588919 server_cookie=475f36445a4bcb76 in_seq=0 out_seq=0 2024-09-15T23:20:00.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1aeff5640 1 -- 172.21.15.62:0/3744204893 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c65accdd60 con 0x55c65adeec20 2024-09-15T23:20:00.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1aeff5640 10 client.25825 handle_command_reply: tid=0 2024-09-15T23:20:00.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1d34d8740 1 client.25825 shutdown 2024-09-15T23:20:00.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3744204893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 msgr2=0x55c65ad71440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad71440 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc0b8b90 tx=0x7fc1bc0bc350 comp rx=0 tx=0).stop 2024-09-15T23:20:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3744204893 shutdown_connections 2024-09-15T23:20:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c65adeec20 0x55c65adf1040 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c65ad6b680 0x55c65ad71ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c65ad60dc0 0x55c65ad71980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1d34d8740 1 --2- 172.21.15.62:0/3744204893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c65ad604a0 0x55c65ad71440 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.918+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3744204893 >> 172.21.15.62:0/3744204893 conn(0x55c65ad5ca30 msgr2=0x55c65ad69b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.919+0000 7fc1aeff5640 0 client.25825 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.919+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3744204893 shutdown_connections 2024-09-15T23:20:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.919+0000 7fc1d34d8740 1 -- 172.21.15.62:0/3744204893 wait complete. 2024-09-15T23:20:00.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.919+0000 7fc1d34d8740 20 client.25825 trim_cache size 0 max 16384 2024-09-15T23:20:00.942 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:20:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:20:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:20:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "16673540-a866-4cf4-a98d-e9b9354d87af": { 2024-09-15T23:20:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:20:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "16673540-a866-4cf4-a98d-e9b9354d87af", 2024-09-15T23:20:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:20:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:00.943 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:20:00.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1b40a3450 msgr2=0x7fc1b40a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1b40a3450 0x7fc1b40a5910 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400fc90 tx=0x7fc1c4018000 comp rx=0 tx=0).stop 2024-09-15T23:20:00.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc100100 msgr2=0x7fc1cc19a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:00.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc100100 0x7fc1cc19a450 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009850 tx=0x7fc1bc0305d0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 shutdown_connections 2024-09-15T23:20:00.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1b40a3450 0x7fc1b40a5910 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1cc102a50 0x7fc1cc19ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc1cc100c80 0x7fc1cc19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 --2- 172.21.15.62:0/1995988321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1cc100100 0x7fc1cc19a450 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.920+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 >> 172.21.15.62:0/1995988321 conn(0x7fc1cc077230 msgr2=0x7fc1cc0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.922+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 shutdown_connections 2024-09-15T23:20:00.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:00.922+0000 7fc1d10ff640 1 -- 172.21.15.62:0/1995988321 wait complete. 2024-09-15T23:20:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:01 smithi148 ceph-mon[35610]: pgmap v482: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.9 MiB/s wr, 828 op/s 2024-09-15T23:20:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:01 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/133227180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:01.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:01 smithi062 ceph-mon[30949]: pgmap v482: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.9 MiB/s wr, 828 op/s 2024-09-15T23:20:01.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:01 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/133227180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:01.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:01 smithi067 ceph-mon[35665]: pgmap v482: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.9 MiB/s wr, 828 op/s 2024-09-15T23:20:01.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:01 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/133227180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:01.736 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 133.8 --await-for 45.0' 2024-09-15T23:20:01.736 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 133.8 --await-for 45.0 2024-09-15T23:20:02.545 INFO:tasks.cephfs.filesystem:scrub status for tag:16673540-a866-4cf4-a98d-e9b9354d87af - {'path': '/', 'tag': '16673540-a866-4cf4-a98d-e9b9354d87af', 'options': 'recursive,force'} 2024-09-15T23:20:02.625 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:03.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.234+0000 7f397b485640 1 -- 172.21.15.62:0/775789161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741012a0 msgr2=0x7f3974105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.234+0000 7f397b485640 1 --2- 172.21.15.62:0/775789161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741012a0 0x7f3974105770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3964009900 tx=0x7f396402a480 comp rx=0 tx=0).stop 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.234+0000 7f397b485640 1 -- 172.21.15.62:0/775789161 shutdown_connections 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.234+0000 7f397b485640 1 --2- 172.21.15.62:0/775789161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974105d90 0x7f3974106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.234+0000 7f397b485640 1 --2- 172.21.15.62:0/775789161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f39741012a0 0x7f3974105770 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.234+0000 7f397b485640 1 --2- 172.21.15.62:0/775789161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3974100980 0x7f3974100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.234+0000 7f397b485640 1 -- 172.21.15.62:0/775789161 >> 172.21.15.62:0/775789161 conn(0x7f39740fc610 msgr2=0x7f39740fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.237+0000 7f397b485640 1 -- 172.21.15.62:0/775789161 shutdown_connections 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.237+0000 7f397b485640 1 -- 172.21.15.62:0/775789161 wait complete. 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.238+0000 7f397b485640 1 Processor -- start 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.239+0000 7f397b485640 1 -- start start 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.239+0000 7f397b485640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974100980 0x7f39741a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.239+0000 7f397b485640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39741012a0 0x7f39741a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.239+0000 7f397b485640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3974105d90 0x7f397419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f397b485640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f39741183a0 con 0x7f3974100980 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f397b485640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3974118220 con 0x7f3974105d90 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f397b485640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3974118520 con 0x7f39741012a0 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f397a483640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974100980 0x7f39741a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f397ac84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3974105d90 0x7f397419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f397ac84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3974105d90 0x7f397419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57282/0 (socket says 172.21.15.62:57282) 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f397ac84640 1 -- 172.21.15.62:0/1702100271 learned_addr learned my addr 172.21.15.62:0/1702100271 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f3979c82640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39741012a0 0x7f39741a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f3979c82640 1 -- 172.21.15.62:0/1702100271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3974105d90 msgr2=0x7f397419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f3979c82640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3974105d90 0x7f397419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f3979c82640 1 -- 172.21.15.62:0/1702100271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974100980 msgr2=0x7f39741a4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f3979c82640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974100980 0x7f39741a4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.240+0000 7f3979c82640 1 -- 172.21.15.62:0/1702100271 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f397419ef40 con 0x7f39741012a0 2024-09-15T23:20:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.241+0000 7f3979c82640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39741012a0 0x7f39741a4b40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3964005ab0 tx=0x7f3964003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.241+0000 7f396b7fe640 1 -- 172.21.15.62:0/1702100271 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396403f070 con 0x7f39741012a0 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.241+0000 7f396b7fe640 1 -- 172.21.15.62:0/1702100271 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3964031c90 con 0x7f39741012a0 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.241+0000 7f396b7fe640 1 -- 172.21.15.62:0/1702100271 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964035030 con 0x7f39741012a0 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.241+0000 7f397b485640 1 -- 172.21.15.62:0/1702100271 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f397419f1d0 con 0x7f39741012a0 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.241+0000 7f397b485640 1 -- 172.21.15.62:0/1702100271 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39741ab3b0 con 0x7f39741012a0 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.245+0000 7f396b7fe640 1 -- 172.21.15.62:0/1702100271 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3964042070 con 0x7f39741012a0 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.245+0000 7f397eac0740 1 Processor -- start 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397eac0740 1 -- start start 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397eac0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0128ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397eac0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e01335a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397eac0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 0x55e1e0135ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397ac84640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0128ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397eac0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e1e008e2a0 con 0x55e1e0128900 2024-09-15T23:20:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397ac84640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0128ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60340/0 (socket says 172.21.15.62:60340) 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397ac84640 1 -- 172.21.15.62:0/1777737744 learned_addr learned my addr 172.21.15.62:0/1777737744 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397eac0740 1 -- 172.21.15.62:0/1777737744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e1e0035b10 con 0x55e1e0133ae0 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.246+0000 7f397eac0740 1 -- 172.21.15.62:0/1777737744 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e1e008a260 con 0x55e1e0129220 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.247+0000 7f397a483640 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e01335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.247+0000 7f3979c82640 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 0x55e1e0135ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.247+0000 7f396b7fe640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39600a3300 0x7f39600a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.247+0000 7f397ac84640 1 -- 172.21.15.62:0/1777737744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 msgr2=0x55e1e01335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.247+0000 7f397ac84640 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e01335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.247+0000 7f397ac84640 1 -- 172.21.15.62:0/1777737744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 msgr2=0x55e1e0135ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.247+0000 7f396b7fe640 1 -- 172.21.15.62:0/1702100271 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f39640f0120 con 0x7f39741012a0 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.248+0000 7f397ac84640 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 0x55e1e0135ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.248+0000 7f397ac84640 1 -- 172.21.15.62:0/1777737744 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1dfdffb90 con 0x55e1e0128900 2024-09-15T23:20:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.248+0000 7f3979c82640 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 0x55e1e0135ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.248+0000 7f397ac84640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39600a3300 0x7f39600a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.248+0000 7f397ac84640 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0128ce0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f396c011470 tx=0x7f396c011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dfc12456838a210 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.249+0000 7f397ac84640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39600a3300 0x7f39600a57c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f396c022000 tx=0x7f396c023000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.250+0000 7f3968ff9640 1 -- 172.21.15.62:0/1777737744 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c01a530 con 0x55e1e0128900 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.250+0000 7f3968ff9640 1 -- 172.21.15.62:0/1777737744 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f396c01ac70 con 0x55e1e0128900 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.250+0000 7f3968ff9640 1 -- 172.21.15.62:0/1777737744 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f396c025b60 con 0x55e1e0128900 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.251+0000 7f397eac0740 1 -- 172.21.15.62:0/1777737744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 msgr2=0x55e1e0128ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.251+0000 7f397eac0740 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0128ce0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f396c011470 tx=0x7f396c011930 comp rx=0 tx=0).stop 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.251+0000 7f397eac0740 1 -- 172.21.15.62:0/1777737744 shutdown_connections 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.251+0000 7f397eac0740 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 0x55e1e0135ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.251+0000 7f397eac0740 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e01335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.251+0000 7f397eac0740 1 --2- 172.21.15.62:0/1777737744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0128ce0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.251+0000 7f397eac0740 1 -- 172.21.15.62:0/1777737744 >> 172.21.15.62:0/1777737744 conn(0x55e1e0124e90 msgr2=0x55e1e0125270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.252+0000 7f397eac0740 1 -- 172.21.15.62:0/1777737744 shutdown_connections 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.252+0000 7f397eac0740 1 -- 172.21.15.62:0/1777737744 wait complete. 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.252+0000 7f397eac0740 1 -- start start 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.252+0000 7f397eac0740 1 Processor -- start 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397eac0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e01309d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397eac0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e0130f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397eac0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 0x55e1e0131450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397eac0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e1e01366b0 con 0x55e1e0128900 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397eac0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e1e00166e0 con 0x55e1e0133ae0 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397eac0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e1e0136830 con 0x55e1e0129220 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397eac0740 10 client.0 _ll_register_callbacks cb 0x55e1e003aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397a483640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e01309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f3979c82640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e0130f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397a483640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e01309d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60348/0 (socket says 172.21.15.62:60348) 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397ac84640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 0x55e1e0131450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397a483640 1 -- 172.21.15.62:0/3316045987 learned_addr learned my addr 172.21.15.62:0/3316045987 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397a483640 1 -- 172.21.15.62:0/3316045987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 msgr2=0x55e1e0130f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397a483640 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e0130f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397a483640 1 -- 172.21.15.62:0/3316045987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 msgr2=0x55e1e0131450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397a483640 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 0x55e1e0131450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f397a483640 1 -- 172.21.15.62:0/3316045987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1e013fba0 con 0x55e1e0128900 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.253+0000 7f3979c82640 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e0130f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.256+0000 7f397a483640 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e01309d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f397000bb00 tx=0x7f3970004540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce90b4b7c253538 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.256+0000 7f395dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.257+0000 7f395dffb640 1 -- 172.21.15.62:0/3316045987 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970003e80 con 0x55e1e0128900 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.257+0000 7f397eac0740 1 -- 172.21.15.62:0/3316045987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f396c01ac70 con 0x55e1e0128900 2024-09-15T23:20:03.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.257+0000 7f395dffb640 1 -- 172.21.15.62:0/3316045987 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3970005020 con 0x55e1e0128900 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.258+0000 7f395dffb640 1 -- 172.21.15.62:0/3316045987 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3970005220 con 0x55e1e0128900 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.258+0000 7f397eac0740 10 client.16374 fetch_fsmap learned FSMap version 30 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.258+0000 7f397eac0740 1 -- 172.21.15.62:0/3316045987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e1e01425e0 con 0x55e1e0128900 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.258+0000 7f395dffb640 1 -- 172.21.15.62:0/3316045987 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f396c01ac70 con 0x55e1e0128900 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.258+0000 7f395dffb640 1 -- 172.21.15.62:0/3316045987 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f397001b630 con 0x55e1e0128900 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.259+0000 7f397eac0740 10 client.16374 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.259+0000 7f397eac0740 10 client.16374 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.259+0000 7f397eac0740 20 client.16374 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.259+0000 7f397eac0740 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e01439f0 0x55e1e0145e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.259+0000 7f397eac0740 4 client.16374 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.259+0000 7f397eac0740 1 -- 172.21.15.62:0/3316045987 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1e009b9d0 con 0x55e1e01439f0 2024-09-15T23:20:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.259+0000 7f3979c82640 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e01439f0 0x55e1e0145e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.259+0000 7f3979c82640 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e01439f0 0x55e1e0145e60 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1312f87cb8a4205a server_cookie=3a23dc89a8391cd4 in_seq=0 out_seq=0 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.259+0000 7f395dffb640 10 client.16374 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.260+0000 7f395dffb640 1 -- 172.21.15.62:0/3316045987 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e1e009b9d0 con 0x55e1e01439f0 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.260+0000 7f395dffb640 10 client.16374 handle_command_reply: tid=0 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.260+0000 7f397eac0740 1 client.16374 shutdown 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.260+0000 7f397eac0740 1 -- 172.21.15.62:0/3316045987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 msgr2=0x55e1e01309d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.260+0000 7f397eac0740 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e01309d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f397000bb00 tx=0x7f3970004540 comp rx=0 tx=0).stop 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.260+0000 7f397eac0740 1 -- 172.21.15.62:0/3316045987 shutdown_connections 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.261+0000 7f397eac0740 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e01439f0 0x55e1e0145e60 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.261+0000 7f397eac0740 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0133ae0 0x55e1e0131450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.261+0000 7f397eac0740 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e0130f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.261+0000 7f397eac0740 1 --2- 172.21.15.62:0/3316045987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e01309d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.261+0000 7f397eac0740 1 -- 172.21.15.62:0/3316045987 >> 172.21.15.62:0/3316045987 conn(0x55e1e0124e90 msgr2=0x55e1e012ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.261+0000 7f395dffb640 0 client.16374 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.261+0000 7f397eac0740 1 -- 172.21.15.62:0/3316045987 shutdown_connections 2024-09-15T23:20:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.261+0000 7f397eac0740 1 -- 172.21.15.62:0/3316045987 wait complete. 2024-09-15T23:20:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.261+0000 7f397eac0740 20 client.16374 trim_cache size 0 max 16384 2024-09-15T23:20:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.272+0000 7f8e783cf640 1 -- 172.21.15.62:0/909731892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e70103710 msgr2=0x7f8e70109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.272+0000 7f8e783cf640 1 --2- 172.21.15.62:0/909731892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e70103710 0x7f8e70109fa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009ca0 tx=0x7f8e6802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.272+0000 7f8e783cf640 1 -- 172.21.15.62:0/909731892 shutdown_connections 2024-09-15T23:20:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.272+0000 7f8e783cf640 1 --2- 172.21.15.62:0/909731892 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e70103710 0x7f8e70109fa0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.272+0000 7f8e783cf640 1 --2- 172.21.15.62:0/909731892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70102d50 0x7f8e701031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.272+0000 7f8e783cf640 1 --2- 172.21.15.62:0/909731892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70101b60 0x7f8e70101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.272+0000 7f8e783cf640 1 -- 172.21.15.62:0/909731892 >> 172.21.15.62:0/909731892 conn(0x7f8e700fd2e0 msgr2=0x7f8e700ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.273+0000 7f8e783cf640 1 -- 172.21.15.62:0/909731892 shutdown_connections 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.273+0000 7f8e783cf640 1 -- 172.21.15.62:0/909731892 wait complete. 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.273+0000 7f8e783cf640 1 Processor -- start 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e783cf640 1 -- start start 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e783cf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101b60 0x7f8e70195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e783cf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70102d50 0x7f8e701960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e783cf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e70103710 0x7f8e7019d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e783cf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7010fa30 con 0x7f8e70101b60 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e783cf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7010f8b0 con 0x7f8e70102d50 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e783cf640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7010fbb0 con 0x7f8e70103710 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e7616b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101b60 0x7f8e70195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e7616b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101b60 0x7f8e70195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60370/0 (socket says 172.21.15.62:60370) 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e7616b640 1 -- 172.21.15.62:0/3409625423 learned_addr learned my addr 172.21.15.62:0/3409625423 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e7596a640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70102d50 0x7f8e701960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e7696c640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e70103710 0x7f8e7019d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e7596a640 1 -- 172.21.15.62:0/3409625423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e70103710 msgr2=0x7f8e7019d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e7596a640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e70103710 0x7f8e7019d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e7596a640 1 -- 172.21.15.62:0/3409625423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101b60 msgr2=0x7f8e70195b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.274+0000 7f8e7596a640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101b60 0x7f8e70195b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.275+0000 7f8e7596a640 1 -- 172.21.15.62:0/3409625423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e701ae8d0 con 0x7f8e70102d50 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.275+0000 7f8e7696c640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e70103710 0x7f8e7019d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.275+0000 7f8e7596a640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70102d50 0x7f8e701960a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000e9f0 tx=0x7f8e6000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.275+0000 7f8e677fe640 1 -- 172.21.15.62:0/3409625423 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6000ccc0 con 0x7f8e70102d50 2024-09-15T23:20:03.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.275+0000 7f8e677fe640 1 -- 172.21.15.62:0/3409625423 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e60010910 con 0x7f8e70102d50 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.275+0000 7f8e783cf640 1 -- 172.21.15.62:0/3409625423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e701aeaa0 con 0x7f8e70102d50 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.275+0000 7f8e677fe640 1 -- 172.21.15.62:0/3409625423 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60018960 con 0x7f8e70102d50 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.275+0000 7f8e783cf640 1 -- 172.21.15.62:0/3409625423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e701aeee0 con 0x7f8e70102d50 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.280+0000 7f8e7b245740 1 Processor -- start 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.280+0000 7f8e677fe640 1 -- 172.21.15.62:0/3409625423 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8e60010ac0 con 0x7f8e70102d50 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.280+0000 7f8e7b245740 1 -- start start 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.280+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90cf9a0 0x5603c90cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.280+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90d4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.280+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90d5370 0x5603c90d7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7b245740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5603c90332f0 con 0x5603c90cf9a0 2024-09-15T23:20:03.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7b245740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5603c8fdb270 con 0x5603c90d5370 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7b245740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5603c902f2b0 con 0x5603c90d02e0 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7616b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90d4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7616b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90d4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33044/0 (socket says 172.21.15.62:33044) 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7616b640 1 -- 172.21.15.62:0/901519547 learned_addr learned my addr 172.21.15.62:0/901519547 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7696c640 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90cf9a0 0x5603c90cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7596a640 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90d5370 0x5603c90d7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7616b640 1 -- 172.21.15.62:0/901519547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90d5370 msgr2=0x5603c90d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7616b640 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90d5370 0x5603c90d7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7616b640 1 -- 172.21.15.62:0/901519547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90cf9a0 msgr2=0x5603c90cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7616b640 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90cf9a0 0x5603c90cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7616b640 1 -- 172.21.15.62:0/901519547 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603c8fda340 con 0x5603c90d02e0 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7696c640 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90cf9a0 0x5603c90cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.281+0000 7f8e7616b640 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90d4da0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009d80 tx=0x7f8e6c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12cb7ee590c273c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.282+0000 7f8e677fe640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e5c0a3460 0x7f8e5c0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.282+0000 7f8e64bfe640 1 -- 172.21.15.62:0/901519547 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c01e070 con 0x5603c90d02e0 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.282+0000 7f8e677fe640 1 -- 172.21.15.62:0/3409625423 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8e600cffe0 con 0x7f8e70102d50 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.282+0000 7f8e64bfe640 1 -- 172.21.15.62:0/901519547 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6c019b10 con 0x5603c90d02e0 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.282+0000 7f8e7696c640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e5c0a3460 0x7f8e5c0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.282+0000 7f8e64bfe640 1 -- 172.21.15.62:0/901519547 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c00eb80 con 0x5603c90d02e0 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7696c640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e5c0a3460 0x7f8e5c0a5920 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009430 tx=0x7f8e68005ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7b245740 1 -- 172.21.15.62:0/901519547 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 msgr2=0x5603c90d4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7b245740 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90d4da0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009d80 tx=0x7f8e6c011810 comp rx=0 tx=0).stop 2024-09-15T23:20:03.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7b245740 1 -- 172.21.15.62:0/901519547 shutdown_connections 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7b245740 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90d5370 0x5603c90d7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7b245740 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90d4da0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7b245740 1 --2- 172.21.15.62:0/901519547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90cf9a0 0x5603c90cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7b245740 1 -- 172.21.15.62:0/901519547 >> 172.21.15.62:0/901519547 conn(0x5603c90cb010 msgr2=0x5603c90cb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7b245740 1 -- 172.21.15.62:0/901519547 shutdown_connections 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.283+0000 7f8e7b245740 1 -- 172.21.15.62:0/901519547 wait complete. 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.284+0000 7f8e7b245740 1 -- start start 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.284+0000 7f8e7b245740 1 Processor -- start 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.284+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90eac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.285+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90eb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.285+0000 7f8e7616b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90eac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.285+0000 7f8e7616b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90eac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57322/0 (socket says 172.21.15.62:57322) 2024-09-15T23:20:03.297 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.285+0000 7f8e7616b640 1 -- 172.21.15.62:0/2668084085 learned_addr learned my addr 172.21.15.62:0/2668084085 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.285+0000 7f8e7596a640 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90eb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.285+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d5370 0x5603c90f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.285+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5603c90e93a0 con 0x5603c90d5370 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.285+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5603c8fbc290 con 0x5603c90cf9a0 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.285+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5603c90e9520 con 0x5603c90d02e0 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e7616b640 1 -- 172.21.15.62:0/2668084085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 msgr2=0x5603c90eb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e7616b640 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90eb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e7616b640 1 -- 172.21.15.62:0/2668084085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d5370 msgr2=0x5603c90f21f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e7616b640 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d5370 0x5603c90f21f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e7616b640 1 -- 172.21.15.62:0/2668084085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603c8d46210 con 0x5603c90cf9a0 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e7b245740 10 client.0 _ll_register_callbacks cb 0x5603c8fed140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e6c019b10 con 0x5603c90cf9a0 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e7616b640 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90eac70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c011d10 tx=0x7f8e6c00e500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de8c85153cbbed7c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.298 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e59ffb640 10 client.26015 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e59ffb640 1 -- 172.21.15.62:0/2668084085 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c027030 con 0x5603c90cf9a0 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.286+0000 7f8e59ffb640 1 -- 172.21.15.62:0/2668084085 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8e6c004020 con 0x5603c90cf9a0 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e59ffb640 1 -- 172.21.15.62:0/2668084085 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8e6c01e040 con 0x5603c90cf9a0 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e7b245740 10 client.26015 fetch_fsmap learned FSMap version 30 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=30}) -- 0x5603c8e12620 con 0x5603c90cf9a0 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e59ffb640 1 -- 172.21.15.62:0/2668084085 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f8e6c004260 con 0x5603c90cf9a0 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e7b245740 10 client.26015 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e7b245740 10 client.26015 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e7b245740 20 client.26015 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5603c90f63a0 0x5603c90f8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e7b245740 4 client.26015 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603c9040a20 con 0x5603c90f63a0 2024-09-15T23:20:03.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.287+0000 7f8e7596a640 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5603c90f63a0 0x5603c90f8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.288+0000 7f8e59ffb640 10 client.26015 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.288+0000 7f8e7596a640 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5603c90f63a0 0x5603c90f8790 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc15f19f38ff96ca server_cookie=d3dea8fd2ed9b9f2 in_seq=0 out_seq=0 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.288+0000 7f8e59ffb640 1 -- 172.21.15.62:0/2668084085 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5603c9040a20 con 0x5603c90f63a0 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.288+0000 7f8e59ffb640 10 client.26015 handle_command_reply: tid=0 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 client.26015 shutdown 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 msgr2=0x5603c90eac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90eac70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c011d10 tx=0x7f8e6c00e500 comp rx=0 tx=0).stop 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 shutdown_connections 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5603c90f63a0 0x5603c90f8790 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d5370 0x5603c90f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e59ffb640 0 client.26015 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90eb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2668084085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90eac70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 >> 172.21.15.62:0/2668084085 conn(0x5603c90cb010 msgr2=0x5603c90cc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 shutdown_connections 2024-09-15T23:20:03.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 1 -- 172.21.15.62:0/2668084085 wait complete. 2024-09-15T23:20:03.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.289+0000 7f8e7b245740 20 client.26015 trim_cache size 0 max 16384 2024-09-15T23:20:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.293+0000 7f397eac0740 1 Processor -- start 2024-09-15T23:20:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.293+0000 7f397eac0740 1 -- start start 2024-09-15T23:20:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.293+0000 7f397eac0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0128900 0x55e1e0132940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.293+0000 7f397eac0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e014b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.293+0000 7f397eac0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0133ae0 0x55e1e014bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.294+0000 7f397a483640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0128900 0x55e1e0132940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.294+0000 7f397a483640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0128900 0x55e1e0132940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57338/0 (socket says 172.21.15.62:57338) 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.294+0000 7f3979c82640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e014b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.294+0000 7f3979c82640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e014b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33058/0 (socket says 172.21.15.62:33058) 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.294+0000 7f397a483640 1 -- 172.21.15.62:0/855258289 learned_addr learned my addr 172.21.15.62:0/855258289 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.294+0000 7f397eac0740 1 -- 172.21.15.62:0/855258289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f39741b7950 con 0x55e1e0133ae0 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.294+0000 7f397eac0740 1 -- 172.21.15.62:0/855258289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f39741b5220 con 0x55e1e0128900 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.294+0000 7f397eac0740 1 -- 172.21.15.62:0/855258289 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f39741b3070 con 0x55e1e0129220 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.295+0000 7f397a483640 1 -- 172.21.15.62:0/855258289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 msgr2=0x55e1e014b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.295+0000 7f397a483640 1 --2- 172.21.15.62:0/855258289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e014b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.295+0000 7f397a483640 1 -- 172.21.15.62:0/855258289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0133ae0 msgr2=0x55e1e014bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.295+0000 7f397a483640 1 --2- 172.21.15.62:0/855258289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0133ae0 0x55e1e014bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.295+0000 7f397a483640 1 -- 172.21.15.62:0/855258289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1e0131d50 con 0x55e1e0128900 2024-09-15T23:20:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.295+0000 7f397a483640 1 --2- 172.21.15.62:0/855258289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0128900 0x55e1e0132940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3970000c00 tx=0x7f3970005cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=204b36d552cd7cfe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.296+0000 7f395ffff640 1 -- 172.21.15.62:0/855258289 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397002c020 con 0x55e1e0128900 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.296+0000 7f395ffff640 1 -- 172.21.15.62:0/855258289 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3970005480 con 0x55e1e0128900 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.296+0000 7f395ffff640 1 -- 172.21.15.62:0/855258289 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3970004e30 con 0x55e1e0128900 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.297+0000 7f397eac0740 1 -- 172.21.15.62:0/855258289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0128900 msgr2=0x55e1e0132940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.297+0000 7f397eac0740 1 --2- 172.21.15.62:0/855258289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0128900 0x55e1e0132940 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3970000c00 tx=0x7f3970005cd0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.297+0000 7f397eac0740 1 -- 172.21.15.62:0/855258289 shutdown_connections 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.297+0000 7f397eac0740 1 --2- 172.21.15.62:0/855258289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0133ae0 0x55e1e014bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.297+0000 7f397eac0740 1 --2- 172.21.15.62:0/855258289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0129220 0x55e1e014b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.297+0000 7f397eac0740 1 --2- 172.21.15.62:0/855258289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0128900 0x55e1e0132940 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.297+0000 7f397eac0740 1 -- 172.21.15.62:0/855258289 >> 172.21.15.62:0/855258289 conn(0x55e1e0124e90 msgr2=0x55e1e0132230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.297+0000 7f397eac0740 1 -- 172.21.15.62:0/855258289 shutdown_connections 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.297+0000 7f397eac0740 1 -- 172.21.15.62:0/855258289 wait complete. 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.298+0000 7f397eac0740 1 -- start start 2024-09-15T23:20:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.298+0000 7f397eac0740 1 Processor -- start 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.298+0000 7f397eac0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0137a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.298+0000 7f397eac0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0129220 0x55e1e0137f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397a483640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0137a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397a483640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0137a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60392/0 (socket says 172.21.15.62:60392) 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397eac0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0133ae0 0x55e1e01384b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f3979c82640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0129220 0x55e1e0137f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f3979c82640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0129220 0x55e1e0137f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57344/0 (socket says 172.21.15.62:57344) 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397a483640 1 -- 172.21.15.62:0/2714875719 learned_addr learned my addr 172.21.15.62:0/2714875719 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f39741b36e0 con 0x55e1e0128900 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f39741b4b50 con 0x55e1e0129220 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f39741b3db0 con 0x55e1e0133ae0 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397ac84640 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0133ae0 0x55e1e01384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397eac0740 10 client.0 _ll_register_callbacks cb 0x7f39700042c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:03.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f3979c82640 1 -- 172.21.15.62:0/2714875719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0133ae0 msgr2=0x55e1e01384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f3979c82640 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0133ae0 0x55e1e01384b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f3979c82640 1 -- 172.21.15.62:0/2714875719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 msgr2=0x55e1e0137a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f3979c82640 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0137a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f3979c82640 1 -- 172.21.15.62:0/2714875719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1e01ad950 con 0x55e1e0129220 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.299+0000 7f397ac84640 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0133ae0 0x55e1e01384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.300+0000 7f3979c82640 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0129220 0x55e1e0137f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f39640ed8f0 tx=0x7f396402a980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df8b2cde5634499b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.300+0000 7f395dffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.300+0000 7f395dffb640 1 -- 172.21.15.62:0/2714875719 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964049040 con 0x55e1e0129220 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.300+0000 7f395dffb640 1 -- 172.21.15.62:0/2714875719 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f39640ba770 con 0x55e1e0129220 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.300+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3970005480 con 0x55e1e0129220 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.300+0000 7f395dffb640 1 -- 172.21.15.62:0/2714875719 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f39640bc470 con 0x55e1e0129220 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.300+0000 7f397eac0740 10 client.26021 fetch_fsmap learned FSMap version 30 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.300+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e1e01aec50 con 0x55e1e0129220 2024-09-15T23:20:03.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.301+0000 7f395dffb640 1 -- 172.21.15.62:0/2714875719 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39640c0770 con 0x55e1e0129220 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.301+0000 7f395dffb640 1 -- 172.21.15.62:0/2714875719 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f396403bb50 con 0x55e1e0129220 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.301+0000 7f397eac0740 10 client.26021 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.301+0000 7f397eac0740 10 client.26021 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.301+0000 7f397eac0740 20 client.26021 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.301+0000 7f397eac0740 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e01af3c0 0x55e1e01b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.301+0000 7f397eac0740 4 client.26021 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.8, "await_for": 45.0}] 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.301+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 133.8, "await_for": 45.0}) -- 0x7f3970005480 con 0x55e1e01af3c0 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.301+0000 7f397a483640 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e01af3c0 0x55e1e01b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.302+0000 7f397a483640 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e01af3c0 0x55e1e01b17e0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f8b41e420fac4a0 server_cookie=e1cf9fdd84d6484a in_seq=0 out_seq=0 2024-09-15T23:20:03.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.302+0000 7f395dffb640 10 client.26021 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:03.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.317+0000 7f8e7b245740 1 Processor -- start 2024-09-15T23:20:03.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.317+0000 7f8e7b245740 1 -- start start 2024-09-15T23:20:03.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.317+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90cfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.317+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90f2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.317+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d5370 0x5603c90eb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.319 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.317+0000 7f8e7b245740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701b3600 con 0x5603c90d5370 2024-09-15T23:20:03.319 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.317+0000 7f8e7b245740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701b0ed0 con 0x5603c90cf9a0 2024-09-15T23:20:03.319 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.317+0000 7f8e7b245740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8e70073fa0 con 0x5603c90d02e0 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7596a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7596a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90f2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33076/0 (socket says 172.21.15.62:33076) 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7596a640 1 -- 172.21.15.62:0/2482992510 learned_addr learned my addr 172.21.15.62:0/2482992510 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7696c640 1 --2- 172.21.15.62:0/2482992510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d5370 0x5603c90eb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7616b640 1 --2- 172.21.15.62:0/2482992510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90cfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7596a640 1 -- 172.21.15.62:0/2482992510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 msgr2=0x5603c90cfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7596a640 1 --2- 172.21.15.62:0/2482992510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90cfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7596a640 1 -- 172.21.15.62:0/2482992510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d5370 msgr2=0x5603c90eb130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7596a640 1 --2- 172.21.15.62:0/2482992510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d5370 0x5603c90eb130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.318+0000 7f8e7596a640 1 -- 172.21.15.62:0/2482992510 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603c90cae30 con 0x5603c90d02e0 2024-09-15T23:20:03.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.319+0000 7f8e7596a640 1 --2- 172.21.15.62:0/2482992510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90f2180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000cf10 tx=0x7f8e600cd5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c23227eeccf1ee20 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.321+0000 7f8e5bfff640 1 -- 172.21.15.62:0/2482992510 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e600a4eb0 con 0x5603c90d02e0 2024-09-15T23:20:03.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.321+0000 7f8e5bfff640 1 -- 172.21.15.62:0/2482992510 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6009a540 con 0x5603c90d02e0 2024-09-15T23:20:03.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.321+0000 7f8e5bfff640 1 -- 172.21.15.62:0/2482992510 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e600a0160 con 0x5603c90d02e0 2024-09-15T23:20:03.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.322+0000 7f8e7b245740 1 -- 172.21.15.62:0/2482992510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 msgr2=0x5603c90f2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.322+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2482992510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90f2180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000cf10 tx=0x7f8e600cd5f0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.323+0000 7f8e7b245740 1 -- 172.21.15.62:0/2482992510 shutdown_connections 2024-09-15T23:20:03.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.323+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2482992510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d5370 0x5603c90eb130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.323+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2482992510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d02e0 0x5603c90f2180 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.323+0000 7f8e7b245740 1 --2- 172.21.15.62:0/2482992510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90cfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.323+0000 7f8e7b245740 1 -- 172.21.15.62:0/2482992510 >> 172.21.15.62:0/2482992510 conn(0x5603c90cb010 msgr2=0x5603c90eaa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.325+0000 7f8e7b245740 1 -- 172.21.15.62:0/2482992510 shutdown_connections 2024-09-15T23:20:03.326 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.325+0000 7f8e7b245740 1 -- 172.21.15.62:0/2482992510 wait complete. 2024-09-15T23:20:03.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.326+0000 7f8e7b245740 1 -- start start 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.326+0000 7f8e7b245740 1 Processor -- start 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.326+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.326+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d02e0 0x5603c90efce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7b245740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d5370 0x5603c90f0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7616b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7616b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90ef7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57364/0 (socket says 172.21.15.62:57364) 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7616b640 1 -- 172.21.15.62:0/4279445997 learned_addr learned my addr 172.21.15.62:0/4279445997 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701af390 con 0x5603c90d02e0 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701b0800 con 0x5603c90cf9a0 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8e701afa60 con 0x5603c90d5370 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7696c640 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d5370 0x5603c90f0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7b245740 10 client.0 _ll_register_callbacks cb 0x7f8e6c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7616b640 1 -- 172.21.15.62:0/4279445997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d5370 msgr2=0x5603c90f0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7616b640 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d5370 0x5603c90f0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7616b640 1 -- 172.21.15.62:0/4279445997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d02e0 msgr2=0x5603c90efce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7616b640 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d02e0 0x5603c90efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.327+0000 7f8e7616b640 1 -- 172.21.15.62:0/4279445997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603c90f1300 con 0x5603c90cf9a0 2024-09-15T23:20:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.328+0000 7f8e7616b640 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90ef7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c019f50 tx=0x7f8e6c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc7a35c404daabc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:03.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.328+0000 7f8e59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.329+0000 7f8e59ffb640 1 -- 172.21.15.62:0/4279445997 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c015020 con 0x5603c90cf9a0 2024-09-15T23:20:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.329+0000 7f8e59ffb640 1 -- 172.21.15.62:0/4279445997 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8e6c005240 con 0x5603c90cf9a0 2024-09-15T23:20:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.330+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e6009a540 con 0x5603c90cf9a0 2024-09-15T23:20:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.330+0000 7f8e59ffb640 1 -- 172.21.15.62:0/4279445997 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8e6c027040 con 0x5603c90cf9a0 2024-09-15T23:20:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.330+0000 7f8e7b245740 10 client.26024 fetch_fsmap learned FSMap version 30 2024-09-15T23:20:03.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.330+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5603c90d7fa0 con 0x5603c90cf9a0 2024-09-15T23:20:03.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.331+0000 7f8e59ffb640 1 -- 172.21.15.62:0/4279445997 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e6c019b10 con 0x5603c90cf9a0 2024-09-15T23:20:03.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.331+0000 7f8e59ffb640 1 -- 172.21.15.62:0/4279445997 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f8e6c004b60 con 0x5603c90cf9a0 2024-09-15T23:20:03.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.331+0000 7f8e7b245740 10 client.26024 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T23:20:03.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.331+0000 7f8e7b245740 10 client.26024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:03.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.331+0000 7f8e7b245740 20 client.26024 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:03.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.331+0000 7f8e7b245740 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5603c90f63a0 0x5603c90da9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:03.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.331+0000 7f8e7b245740 4 client.26024 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:20:03.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.331+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f8e6009a540 con 0x5603c90f63a0 2024-09-15T23:20:03.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.332+0000 7f8e7596a640 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5603c90f63a0 0x5603c90da9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.334+0000 7f8e7596a640 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5603c90f63a0 0x5603c90da9e0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81283f4cee08c1a3 server_cookie=bc81c75fe68c904d in_seq=0 out_seq=0 2024-09-15T23:20:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.334+0000 7f8e59ffb640 10 client.26024 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.371+0000 7f8e59ffb640 1 -- 172.21.15.62:0/4279445997 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f8e6009a540 con 0x5603c90f63a0 2024-09-15T23:20:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.371+0000 7f8e59ffb640 10 client.26024 handle_command_reply: tid=0 2024-09-15T23:20:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.371+0000 7f8e7b245740 1 client.26024 shutdown 2024-09-15T23:20:03.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.371+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 msgr2=0x5603c90ef7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.371+0000 7f8e7b245740 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90ef7a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c019f50 tx=0x7f8e6c000bc0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.372+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 shutdown_connections 2024-09-15T23:20:03.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.372+0000 7f8e7b245740 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5603c90f63a0 0x5603c90da9e0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.372+0000 7f8e7b245740 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5603c90d5370 0x5603c90f0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.372+0000 7f8e7b245740 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5603c90d02e0 0x5603c90efce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.372+0000 7f8e7b245740 1 --2- 172.21.15.62:0/4279445997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5603c90cf9a0 0x5603c90ef7a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.372+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 >> 172.21.15.62:0/4279445997 conn(0x5603c90cb010 msgr2=0x5603c90c99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.373+0000 7f8e59ffb640 0 client.26024 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:03.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.374+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 shutdown_connections 2024-09-15T23:20:03.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.374+0000 7f8e7b245740 1 -- 172.21.15.62:0/4279445997 wait complete. 2024-09-15T23:20:03.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.374+0000 7f8e7b245740 20 client.26024 trim_cache size 0 max 16384 2024-09-15T23:20:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:20:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:20:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:20:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:20:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 -- 172.21.15.62:0/3409625423 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e5c0a3460 msgr2=0x7f8e5c0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e5c0a3460 0x7f8e5c0a5920 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009430 tx=0x7f8e68005ef0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 -- 172.21.15.62:0/3409625423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70102d50 msgr2=0x7f8e701960a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70102d50 0x7f8e701960a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000e9f0 tx=0x7f8e6000eeb0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 -- 172.21.15.62:0/3409625423 shutdown_connections 2024-09-15T23:20:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e5c0a3460 0x7f8e5c0a5920 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e70103710 0x7f8e7019d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e70102d50 0x7f8e701960a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 --2- 172.21.15.62:0/3409625423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101b60 0x7f8e70195b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 -- 172.21.15.62:0/3409625423 >> 172.21.15.62:0/3409625423 conn(0x7f8e700fd2e0 msgr2=0x7f8e7010c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 -- 172.21.15.62:0/3409625423 shutdown_connections 2024-09-15T23:20:03.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:03.376+0000 7f8e783cf640 1 -- 172.21.15.62:0/3409625423 wait complete. 2024-09-15T23:20:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:03 smithi062 ceph-mon[30949]: pgmap v483: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.4 MiB/s wr, 801 op/s 2024-09-15T23:20:03.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:03 smithi067 ceph-mon[35665]: pgmap v483: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.4 MiB/s wr, 801 op/s 2024-09-15T23:20:03.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:03 smithi148 ceph-mon[35610]: pgmap v483: 129 pgs: 129 active+clean; 1.1 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 7.4 MiB/s wr, 801 op/s 2024-09-15T23:20:03.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.560+0000 7f395dffb640 1 -- 172.21.15.62:0/2714875719 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f3970005480 con 0x55e1e01af3c0 2024-09-15T23:20:03.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.560+0000 7f395dffb640 10 client.26021 handle_command_reply: tid=0 2024-09-15T23:20:03.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.560+0000 7f397eac0740 1 client.26021 shutdown 2024-09-15T23:20:03.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.560+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0129220 msgr2=0x55e1e0137f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.560+0000 7f397eac0740 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0129220 0x55e1e0137f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f39640ed8f0 tx=0x7f396402a980 comp rx=0 tx=0).stop 2024-09-15T23:20:03.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 shutdown_connections 2024-09-15T23:20:03.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f397eac0740 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e01af3c0 0x55e1e01b17e0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f395dffb640 0 client.26021 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:03.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f397eac0740 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e0133ae0 0x55e1e01384b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f397eac0740 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e0129220 0x55e1e0137f70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f397eac0740 1 --2- 172.21.15.62:0/2714875719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e0128900 0x55e1e0137a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 >> 172.21.15.62:0/2714875719 conn(0x55e1e0124e90 msgr2=0x55e1e012ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 shutdown_connections 2024-09-15T23:20:03.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f397eac0740 1 -- 172.21.15.62:0/2714875719 wait complete. 2024-09-15T23:20:03.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.561+0000 7f397eac0740 20 client.26021 trim_cache size 0 max 16384 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 30, 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 14, 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "cfb03c62": { 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 14, 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:20:03.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 133.8, 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:03.565 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:03.566 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:20:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 -- 172.21.15.62:0/1702100271 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39600a3300 msgr2=0x7f39600a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39600a3300 0x7f39600a57c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f396c022000 tx=0x7f396c023000 comp rx=0 tx=0).stop 2024-09-15T23:20:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 -- 172.21.15.62:0/1702100271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39741012a0 msgr2=0x7f39741a4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:03.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39741012a0 0x7f39741a4b40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3964005ab0 tx=0x7f3964003b90 comp rx=0 tx=0).stop 2024-09-15T23:20:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 -- 172.21.15.62:0/1702100271 shutdown_connections 2024-09-15T23:20:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f39600a3300 0x7f39600a57c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3974105d90 0x7f397419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f39741012a0 0x7f39741a4b40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 --2- 172.21.15.62:0/1702100271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3974100980 0x7f39741a4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 -- 172.21.15.62:0/1702100271 >> 172.21.15.62:0/1702100271 conn(0x7f39740fc610 msgr2=0x7f397410f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 -- 172.21.15.62:0/1702100271 shutdown_connections 2024-09-15T23:20:03.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:03.562+0000 7f397b485640 1 -- 172.21.15.62:0/1702100271 wait complete. 2024-09-15T23:20:04.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:04 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:20:04.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:04 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:20:04.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:04 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:20:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 225.0 s, 1204.6 tps, lat 66.037 ms stddev 76.364 2024-09-15T23:20:05.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:05 smithi062 ceph-mon[30949]: pgmap v484: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 7.4 MiB/s wr, 791 op/s 2024-09-15T23:20:05.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:05 smithi067 ceph-mon[35665]: pgmap v484: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 7.4 MiB/s wr, 791 op/s 2024-09-15T23:20:05.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:05 smithi148 ceph-mon[35610]: pgmap v484: 129 pgs: 129 active+clean; 1.1 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 7.4 MiB/s wr, 791 op/s 2024-09-15T23:20:07.046 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'cfb03c62', quiesce took 10.3 seconds. Will release after: 3.5 2024-09-15T23:20:07.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:07 smithi062 ceph-mon[30949]: pgmap v485: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.4 MiB/s wr, 842 op/s 2024-09-15T23:20:07.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:07 smithi067 ceph-mon[35665]: pgmap v485: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.4 MiB/s wr, 842 op/s 2024-09-15T23:20:07.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:07 smithi148 ceph-mon[35610]: pgmap v485: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 9.4 MiB/s wr, 842 op/s 2024-09-15T23:20:08.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:08 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:20:08.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:08 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:20:08.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:08 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:20:08.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:08 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:20:08.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:08 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:20:08.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:08 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:20:09.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:09 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:20:09.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:09 smithi062 ceph-mon[30949]: pgmap v486: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 8.2 MiB/s wr, 743 op/s 2024-09-15T23:20:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:09 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:20:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:09 smithi067 ceph-mon[35665]: pgmap v486: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 8.2 MiB/s wr, 743 op/s 2024-09-15T23:20:09.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:09 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:20:09.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:09 smithi148 ceph-mon[35610]: pgmap v486: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 8.2 MiB/s wr, 743 op/s 2024-09-15T23:20:09.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 230.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:20:10.547 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:20:10.818 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.289+0000 7f2f5fd60640 1 -- 172.21.15.62:0/784871331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f581082a0 msgr2=0x7f2f5810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.289+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/784871331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f581082a0 0x7f2f5810ec40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009cd0 tx=0x7f2f5002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.289+0000 7f2f5fd60640 1 -- 172.21.15.62:0/784871331 shutdown_connections 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.289+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/784871331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f581082a0 0x7f2f5810ec40 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.289+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/784871331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581036e0 0x7f2f58107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.289+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/784871331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f58102d30 0x7f2f58103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.289+0000 7f2f5fd60640 1 -- 172.21.15.62:0/784871331 >> 172.21.15.62:0/784871331 conn(0x7f2f580fc7d0 msgr2=0x7f2f580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.290+0000 7f2f5fd60640 1 -- 172.21.15.62:0/784871331 shutdown_connections 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.290+0000 7f2f5fd60640 1 -- 172.21.15.62:0/784871331 wait complete. 2024-09-15T23:20:11.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.290+0000 7f2f5fd60640 1 Processor -- start 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.290+0000 7f2f5fd60640 1 -- start start 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.290+0000 7f2f5fd60640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f58102d30 0x7f2f58198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.290+0000 7f2f5fd60640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f581036e0 0x7f2f58198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5fd60640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581082a0 0x7f2f5819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5fd60640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2f581120a0 con 0x7f2f58102d30 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5fd60640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58111f20 con 0x7f2f581082a0 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5fd60640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58112220 con 0x7f2f581036e0 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5dafc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f58102d30 0x7f2f58198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5dafc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f58102d30 0x7f2f58198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60426/0 (socket says 172.21.15.62:60426) 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5dafc640 1 -- 172.21.15.62:0/2976082382 learned_addr learned my addr 172.21.15.62:0/2976082382 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5e2fd640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581082a0 0x7f2f5819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5d2fb640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f581036e0 0x7f2f58198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:11.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5dafc640 1 -- 172.21.15.62:0/2976082382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f581036e0 msgr2=0x7f2f58198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5dafc640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f581036e0 0x7f2f58198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5dafc640 1 -- 172.21.15.62:0/2976082382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581082a0 msgr2=0x7f2f5819ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5dafc640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581082a0 0x7f2f5819ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5dafc640 1 -- 172.21.15.62:0/2976082382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5819d4e0 con 0x7f2f58102d30 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.291+0000 7f2f5dafc640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f58102d30 0x7f2f58198530 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400e9f0 tx=0x7f2f5400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.292+0000 7f2f4effd640 1 -- 172.21.15.62:0/2976082382 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5400cd80 con 0x7f2f58102d30 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.292+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f5819d7d0 con 0x7f2f58102d30 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.292+0000 7f2f4effd640 1 -- 172.21.15.62:0/2976082382 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f54005500 con 0x7f2f58102d30 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.292+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f581a50b0 con 0x7f2f58102d30 2024-09-15T23:20:11.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.292+0000 7f2f4effd640 1 -- 172.21.15.62:0/2976082382 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54010690 con 0x7f2f58102d30 2024-09-15T23:20:11.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.293+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f3c005290 con 0x7f2f58102d30 2024-09-15T23:20:11.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.294+0000 7f2f4effd640 1 -- 172.21.15.62:0/2976082382 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2f540046c0 con 0x7f2f58102d30 2024-09-15T23:20:11.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.295+0000 7f2f4effd640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f440a3240 0x7f2f440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:11.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.295+0000 7f2f5d2fb640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f440a3240 0x7f2f440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:11.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.295+0000 7f2f4effd640 1 -- 172.21.15.62:0/2976082382 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2f540d01f0 con 0x7f2f58102d30 2024-09-15T23:20:11.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.296+0000 7f2f5d2fb640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f440a3240 0x7f2f440a5700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2f580fede0 tx=0x7f2f4800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:11.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.298+0000 7f2f4effd640 1 -- 172.21.15.62:0/2976082382 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2f540541b0 con 0x7f2f58102d30 2024-09-15T23:20:11.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:11 smithi062 ceph-mon[30949]: pgmap v487: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 11 MiB/s wr, 846 op/s 2024-09-15T23:20:11.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.512+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f3c005580 con 0x7f2f58102d30 2024-09-15T23:20:11.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.513+0000 7f2f4effd640 1 -- 172.21.15.62:0/2976082382 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11904 (secure 0 0 0) 0x7f2f5409cb30 con 0x7f2f58102d30 2024-09-15T23:20:11.515 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:20:11.515 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":30,"btime":"2024-09-15T23:20:03:173251+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:20:02.378764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:20:11.515 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 30 2024-09-15T23:20:11.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.515+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f440a3240 msgr2=0x7f2f440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:11.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.515+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f440a3240 0x7f2f440a5700 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2f580fede0 tx=0x7f2f4800b390 comp rx=0 tx=0).stop 2024-09-15T23:20:11.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.515+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f58102d30 msgr2=0x7f2f58198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:11.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.515+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f58102d30 0x7f2f58198530 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f2f5400e9f0 tx=0x7f2f5400edf0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.516+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 shutdown_connections 2024-09-15T23:20:11.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.516+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f440a3240 0x7f2f440a5700 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.516+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f581082a0 0x7f2f5819ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.516+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f581036e0 0x7f2f58198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.516+0000 7f2f5fd60640 1 --2- 172.21.15.62:0/2976082382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f58102d30 0x7f2f58198530 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:11.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.516+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 >> 172.21.15.62:0/2976082382 conn(0x7f2f580fc7d0 msgr2=0x7f2f58105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:11.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.516+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 shutdown_connections 2024-09-15T23:20:11.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:11.516+0000 7f2f5fd60640 1 -- 172.21.15.62:0/2976082382 wait complete. 2024-09-15T23:20:11.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:11 smithi067 ceph-mon[35665]: pgmap v487: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 11 MiB/s wr, 846 op/s 2024-09-15T23:20:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:11 smithi148 ceph-mon[35610]: pgmap v487: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 11 MiB/s wr, 846 op/s 2024-09-15T23:20:12.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:12 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2976082382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:12.275 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id cfb03c62 --release --await' 2024-09-15T23:20:12.276 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id cfb03c62 --release --await 2024-09-15T23:20:12.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:12 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2976082382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:12.538 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:12.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:12 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2976082382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:12.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 -- 172.21.15.62:0/499104742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80102ef0 msgr2=0x7f3f8010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/499104742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80102ef0 0x7f3f8010da40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3f78009ca0 tx=0x7f3f7802a950 comp rx=0 tx=0).stop 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 -- 172.21.15.62:0/499104742 shutdown_connections 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/499104742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80102ef0 0x7f3f8010da40 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/499104742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f80102570 0x7f3f801029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/499104742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f80100e20 0x7f3f80101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 -- 172.21.15.62:0/499104742 >> 172.21.15.62:0/499104742 conn(0x7f3f800fc7d0 msgr2=0x7f3f800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 -- 172.21.15.62:0/499104742 shutdown_connections 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 -- 172.21.15.62:0/499104742 wait complete. 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 Processor -- start 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 -- start start 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.966+0000 7f3f86ba6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80100e20 0x7f3f8019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f86ba6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f80102570 0x7f3f8019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f86ba6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f80102ef0 0x7f3f8019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f86ba6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3f80110f80 con 0x7f3f80100e20 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f86ba6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3f80110e00 con 0x7f3f80102570 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f86ba6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3f80111100 con 0x7f3f80102ef0 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f84942640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80100e20 0x7f3f8019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f84942640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80100e20 0x7f3f8019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60696/0 (socket says 172.21.15.62:60696) 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f84942640 1 -- 172.21.15.62:0/1882845574 learned_addr learned my addr 172.21.15.62:0/1882845574 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f85143640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f80102ef0 0x7f3f8019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f80102570 0x7f3f8019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f84942640 1 -- 172.21.15.62:0/1882845574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f80102ef0 msgr2=0x7f3f8019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f84942640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f80102ef0 0x7f3f8019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f84942640 1 -- 172.21.15.62:0/1882845574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f80102570 msgr2=0x7f3f8019abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f84942640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f80102570 0x7f3f8019abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f84942640 1 -- 172.21.15.62:0/1882845574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8019f650 con 0x7f3f80100e20 2024-09-15T23:20:12.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f84942640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80100e20 0x7f3f8019a6a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3f7400d770 tx=0x7f3f7400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:12.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f7dffb640 1 -- 172.21.15.62:0/1882845574 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74017070 con 0x7f3f80100e20 2024-09-15T23:20:12.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f86ba6640 1 -- 172.21.15.62:0/1882845574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8019f940 con 0x7f3f80100e20 2024-09-15T23:20:12.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f7dffb640 1 -- 172.21.15.62:0/1882845574 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f74005500 con 0x7f3f80100e20 2024-09-15T23:20:12.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.967+0000 7f3f7dffb640 1 -- 172.21.15.62:0/1882845574 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74004e00 con 0x7f3f80100e20 2024-09-15T23:20:12.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.968+0000 7f3f86ba6640 1 -- 172.21.15.62:0/1882845574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f801a7220 con 0x7f3f80100e20 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.970+0000 7f3f7dffb640 1 -- 172.21.15.62:0/1882845574 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3f74004fa0 con 0x7f3f80100e20 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.970+0000 7f3f88f7f740 1 Processor -- start 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.970+0000 7f3f88f7f740 1 -- start start 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.970+0000 7f3f7dffb640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3f700a3240 0x7f3f700a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.970+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c060 0x564ec141c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec1426d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 0x564ec1429630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564ec1381620 con 0x564ec141c980 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564ec1328e50 con 0x564ec1427240 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564ec137d5e0 con 0x564ec141c060 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7dffb640 1 -- 172.21.15.62:0/1882845574 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3f740cf7d0 con 0x7f3f80100e20 2024-09-15T23:20:12.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f85143640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3f700a3240 0x7f3f700a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec1426d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec1426d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60708/0 (socket says 172.21.15.62:60708) 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7ffff640 1 -- 172.21.15.62:0/2666923796 learned_addr learned my addr 172.21.15.62:0/2666923796 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f85143640 1 --2- 172.21.15.62:0/2666923796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 0x564ec1429630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f84942640 1 --2- 172.21.15.62:0/2666923796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c060 0x564ec141c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7ffff640 1 -- 172.21.15.62:0/2666923796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c060 msgr2=0x564ec141c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/2666923796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c060 0x564ec141c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7ffff640 1 -- 172.21.15.62:0/2666923796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 msgr2=0x564ec1429630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/2666923796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 0x564ec1429630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7ffff640 1 -- 172.21.15.62:0/2666923796 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ec1319d40 con 0x564ec141c980 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f85143640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3f700a3240 0x7f3f700a5700 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3f7802ae70 tx=0x7f3f7802ff20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/2666923796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec1426d00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009ac0 tx=0x7f3f6c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ffa0d715353ba3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:12.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.971+0000 7f3f6affd640 1 -- 172.21.15.62:0/2666923796 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c016070 con 0x564ec141c980 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f6affd640 1 -- 172.21.15.62:0/2666923796 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f6c005500 con 0x564ec141c980 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f6affd640 1 -- 172.21.15.62:0/2666923796 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c011470 con 0x564ec141c980 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f88f7f740 1 -- 172.21.15.62:0/2666923796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 msgr2=0x564ec1426d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/2666923796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec1426d00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009ac0 tx=0x7f3f6c009f80 comp rx=0 tx=0).stop 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f88f7f740 1 -- 172.21.15.62:0/2666923796 shutdown_connections 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/2666923796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 0x564ec1429630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/2666923796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec1426d00 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/2666923796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c060 0x564ec141c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f88f7f740 1 -- 172.21.15.62:0/2666923796 >> 172.21.15.62:0/2666923796 conn(0x564ec14185f0 msgr2=0x564ec14189d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f88f7f740 1 -- 172.21.15.62:0/2666923796 shutdown_connections 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.972+0000 7f3f88f7f740 1 -- 172.21.15.62:0/2666923796 wait complete. 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f88f7f740 1 -- start start 2024-09-15T23:20:12.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f88f7f740 1 Processor -- start 2024-09-15T23:20:12.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c060 0x564ec1424120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 0x564ec1424660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 0x564ec1424ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564ec1429e10 con 0x564ec141c060 2024-09-15T23:20:12.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564ec1309f10 con 0x564ec1427240 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564ec1429f90 con 0x564ec141c980 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f88f7f740 10 client.0 _ll_register_callbacks cb 0x564ec132ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f84942640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c060 0x564ec1424120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f84942640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c060 0x564ec1424120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60718/0 (socket says 172.21.15.62:60718) 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.973+0000 7f3f84942640 1 -- 172.21.15.62:0/1942058568 learned_addr learned my addr 172.21.15.62:0/1942058568 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 0x564ec1424660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f85143640 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 0x564ec1424ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f84942640 1 -- 172.21.15.62:0/1942058568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 msgr2=0x564ec1424660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f84942640 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 0x564ec1424660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f84942640 1 -- 172.21.15.62:0/1942058568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 msgr2=0x564ec1424ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f84942640 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 0x564ec1424ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f84942640 1 -- 172.21.15.62:0/1942058568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ec1090580 con 0x564ec141c060 2024-09-15T23:20:12.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f84942640 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c060 0x564ec1424120 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3f74097e70 tx=0x7f3f740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512e7d334182e542 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f5ffff640 1 -- 172.21.15.62:0/1942058568 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74007560 con 0x564ec141c060 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f5ffff640 1 -- 172.21.15.62:0/1942058568 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3f74009cc0 con 0x564ec141c060 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f88f7f740 1 -- 172.21.15.62:0/1942058568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f6c005500 con 0x564ec141c060 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f5ffff640 1 -- 172.21.15.62:0/1942058568 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3f7401f170 con 0x564ec141c060 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f88f7f740 10 client.16407 fetch_fsmap learned FSMap version 30 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f88f7f740 1 -- 172.21.15.62:0/1942058568 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564ec115a960 con 0x564ec141c060 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.974+0000 7f3f5ffff640 1 -- 172.21.15.62:0/1942058568 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f6c005500 con 0x564ec141c060 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.975+0000 7f3f5ffff640 1 -- 172.21.15.62:0/1942058568 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f3f740b3020 con 0x564ec141c060 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.975+0000 7f3f88f7f740 10 client.16407 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.975+0000 7f3f88f7f740 10 client.16407 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:20:12.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.975+0000 7f3f88f7f740 20 client.16407 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.975+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ec1436d50 0x564ec14391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.975+0000 7f3f88f7f740 4 client.16407 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.975+0000 7f3f88f7f740 1 -- 172.21.15.62:0/1942058568 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ec138eb90 con 0x564ec1436d50 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.975+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ec1436d50 0x564ec14391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ec1436d50 0x564ec14391d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=446305b1708ac68e server_cookie=7e4fcb1757ff851a in_seq=0 out_seq=0 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f5ffff640 10 client.16407 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f5ffff640 1 -- 172.21.15.62:0/1942058568 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564ec138eb90 con 0x564ec1436d50 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f5ffff640 10 client.16407 handle_command_reply: tid=0 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f88f7f740 1 client.16407 shutdown 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f88f7f740 1 -- 172.21.15.62:0/1942058568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c060 msgr2=0x564ec1424120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c060 0x564ec1424120 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3f74097e70 tx=0x7f3f740046c0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f88f7f740 1 -- 172.21.15.62:0/1942058568 shutdown_connections 2024-09-15T23:20:12.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ec1436d50 0x564ec14391d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f5ffff640 0 client.16407 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:12.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec1427240 0x564ec1424ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 0x564ec1424660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/1942058568 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c060 0x564ec1424120 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.976+0000 7f3f88f7f740 1 -- 172.21.15.62:0/1942058568 >> 172.21.15.62:0/1942058568 conn(0x564ec14185f0 msgr2=0x564ec141e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:12.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.977+0000 7f3f88f7f740 1 -- 172.21.15.62:0/1942058568 shutdown_connections 2024-09-15T23:20:12.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.977+0000 7f3f88f7f740 1 -- 172.21.15.62:0/1942058568 wait complete. 2024-09-15T23:20:12.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.977+0000 7f3f88f7f740 20 client.16407 trim_cache size 0 max 16384 2024-09-15T23:20:12.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.993+0000 7f3f88f7f740 1 Processor -- start 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.993+0000 7f3f88f7f740 1 -- start start 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 0x564ec141eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 0x564ec141f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec1427240 0x564ec1417640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801b35e0 con 0x564ec1427240 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801b0eb0 con 0x564ec141c060 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801aed00 con 0x564ec141c980 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f85143640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec1427240 0x564ec1417640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f85143640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec1427240 0x564ec1417640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60728/0 (socket says 172.21.15.62:60728) 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f85143640 1 -- 172.21.15.62:0/3567367656 learned_addr learned my addr 172.21.15.62:0/3567367656 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:12.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f84942640 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 0x564ec141eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 0x564ec141f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f85143640 1 -- 172.21.15.62:0/3567367656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 msgr2=0x564ec141f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f85143640 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 0x564ec141f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f85143640 1 -- 172.21.15.62:0/3567367656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 msgr2=0x564ec141eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f85143640 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 0x564ec141eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f85143640 1 -- 172.21.15.62:0/3567367656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ec1426060 con 0x564ec1427240 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f84942640 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 0x564ec141eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 0x564ec141f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f85143640 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec1427240 0x564ec1417640 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3f78035000 tx=0x7f3f7804a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b07a281305d410b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.994+0000 7f3f6a7fc640 1 -- 172.21.15.62:0/3567367656 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78034040 con 0x564ec1427240 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f6a7fc640 1 -- 172.21.15.62:0/3567367656 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f7804f020 con 0x564ec1427240 2024-09-15T23:20:12.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f6a7fc640 1 -- 172.21.15.62:0/3567367656 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f78030440 con 0x564ec1427240 2024-09-15T23:20:13.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3567367656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec1427240 msgr2=0x564ec1417640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:13.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec1427240 0x564ec1417640 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3f78035000 tx=0x7f3f7804a000 comp rx=0 tx=0).stop 2024-09-15T23:20:13.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3567367656 shutdown_connections 2024-09-15T23:20:13.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec1427240 0x564ec1417640 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec141c980 0x564ec141f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3567367656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 0x564ec141eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3567367656 >> 172.21.15.62:0/3567367656 conn(0x564ec14185f0 msgr2=0x564ec1416f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3567367656 shutdown_connections 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.995+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3567367656 wait complete. 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f88f7f740 1 -- start start 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f88f7f740 1 Processor -- start 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 0x564ec142b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec142bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f88f7f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec1427240 0x564ec149aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801af370 con 0x564ec141c980 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801b07e0 con 0x564ec141c060 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f88f7f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3f801afa40 con 0x564ec1427240 2024-09-15T23:20:13.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f88f7f740 10 client.0 _ll_register_callbacks cb 0x7f3f7409b370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f7ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec142bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f7ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec142bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60736/0 (socket says 172.21.15.62:60736) 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f7ffff640 1 -- 172.21.15.62:0/3363793754 learned_addr learned my addr 172.21.15.62:0/3363793754 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f84942640 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 0x564ec142b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.996+0000 7f3f85143640 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec1427240 0x564ec149aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f7ffff640 1 -- 172.21.15.62:0/3363793754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec1427240 msgr2=0x564ec149aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec1427240 0x564ec149aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f7ffff640 1 -- 172.21.15.62:0/3363793754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 msgr2=0x564ec142b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 0x564ec142b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f7ffff640 1 -- 172.21.15.62:0/3363793754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ec149b0f0 con 0x564ec141c980 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f85143640 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec1427240 0x564ec149aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f7ffff640 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec142bd70 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c000c00 tx=0x7f3f6c009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d365a1fc9d4871d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:13.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f5ffff640 1 -- 172.21.15.62:0/3363793754 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c016040 con 0x564ec141c980 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f5ffff640 1 -- 172.21.15.62:0/3363793754 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3f6c00ecb0 con 0x564ec141c980 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3363793754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f7804f020 con 0x564ec141c980 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f5ffff640 1 -- 172.21.15.62:0/3363793754 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3f6c038050 con 0x564ec141c980 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f88f7f740 10 client.16413 fetch_fsmap learned FSMap version 30 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3363793754 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564ec14a24a0 con 0x564ec141c980 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.997+0000 7f3f5ffff640 1 -- 172.21.15.62:0/3363793754 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f7804f020 con 0x564ec141c980 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f5ffff640 1 -- 172.21.15.62:0/3363793754 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f3f6c00f040 con 0x564ec141c980 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f88f7f740 10 client.16413 fetch_fsmap finished waiting for FSMap version 30 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f88f7f740 10 client.16413 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f88f7f740 20 client.16413 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ec1436d50 0x564ec14a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:13.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f88f7f740 4 client.16413 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "cfb03c62", "release": true, "await": true}] 2024-09-15T23:20:13.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3363793754 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "cfb03c62", "release": true, "await": true}) -- 0x564ec13890a0 con 0x564ec1436d50 2024-09-15T23:20:13.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f84942640 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ec1436d50 0x564ec14a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:13.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f5ffff640 10 client.16413 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:13.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:12.998+0000 7f3f84942640 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ec1436d50 0x564ec14a4b50 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5df8e51d0731785f server_cookie=f9fa2cb4c7355ddf in_seq=0 out_seq=0 2024-09-15T23:20:13.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.020+0000 7f3f5ffff640 1 -- 172.21.15.62:0/3363793754 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x564ec13890a0 con 0x564ec1436d50 2024-09-15T23:20:13.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.020+0000 7f3f5ffff640 10 client.16413 handle_command_reply: tid=0 2024-09-15T23:20:13.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.020+0000 7f3f88f7f740 1 client.16413 shutdown 2024-09-15T23:20:13.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.021+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3363793754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 msgr2=0x564ec142bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:13.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.021+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec142bd70 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c000c00 tx=0x7f3f6c009470 comp rx=0 tx=0).stop 2024-09-15T23:20:13.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.022+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3363793754 shutdown_connections 2024-09-15T23:20:13.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.022+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ec1436d50 0x564ec14a4b50 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.022+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ec1427240 0x564ec149aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.022+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ec141c980 0x564ec142bd70 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.022+0000 7f3f88f7f740 1 --2- 172.21.15.62:0/3363793754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ec141c060 0x564ec142b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.022+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3363793754 >> 172.21.15.62:0/3363793754 conn(0x564ec14185f0 msgr2=0x564ec1417750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:13.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.022+0000 7f3f5ffff640 0 client.16413 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:13.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.022+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3363793754 shutdown_connections 2024-09-15T23:20:13.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.023+0000 7f3f88f7f740 1 -- 172.21.15.62:0/3363793754 wait complete. 2024-09-15T23:20:13.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.023+0000 7f3f88f7f740 20 client.16413 trim_cache size 0 max 16384 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 30, 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 16, 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout: "cfb03c62": { 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 16, 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:20:13.026 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 133.8, 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:20:13.027 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:13.028 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:13.028 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:13.028 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:13.028 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:13.028 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 -- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3f700a3240 msgr2=0x7f3f700a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3f700a3240 0x7f3f700a5700 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3f7802ae70 tx=0x7f3f7802ff20 comp rx=0 tx=0).stop 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 -- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80100e20 msgr2=0x7f3f8019a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80100e20 0x7f3f8019a6a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3f7400d770 tx=0x7f3f7400dc30 comp rx=0 tx=0).stop 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 -- 172.21.15.62:0/1882845574 shutdown_connections 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3f700a3240 0x7f3f700a5700 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3f80102ef0 0x7f3f8019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f80102570 0x7f3f8019abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 --2- 172.21.15.62:0/1882845574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f80100e20 0x7f3f8019a6a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.025+0000 7f3f86ba6640 1 -- 172.21.15.62:0/1882845574 >> 172.21.15.62:0/1882845574 conn(0x7f3f800fc7d0 msgr2=0x7f3f8006a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:13.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.026+0000 7f3f86ba6640 1 -- 172.21.15.62:0/1882845574 shutdown_connections 2024-09-15T23:20:13.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:13.026+0000 7f3f86ba6640 1 -- 172.21.15.62:0/1882845574 wait complete. 2024-09-15T23:20:13.421 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:13 smithi062 ceph-mon[30949]: pgmap v488: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 8.8 MiB/s wr, 620 op/s 2024-09-15T23:20:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:13 smithi067 ceph-mon[35665]: pgmap v488: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 8.8 MiB/s wr, 620 op/s 2024-09-15T23:20:13.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:13 smithi148 ceph-mon[35610]: pgmap v488: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 8.8 MiB/s wr, 620 op/s 2024-09-15T23:20:13.856 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'cfb03c62', total seconds elapsed: 17.2 2024-09-15T23:20:14.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:14 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:20:14.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:14 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:20:14.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:14 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:20:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 235.0 s, 551.8 tps, lat 399.158 ms stddev 1789.079 2024-09-15T23:20:15.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:15 smithi067 ceph-mon[35665]: pgmap v489: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 479 op/s 2024-09-15T23:20:15.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:15 smithi148 ceph-mon[35610]: pgmap v489: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 479 op/s 2024-09-15T23:20:15.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:15 smithi062 ceph-mon[30949]: pgmap v489: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 479 op/s 2024-09-15T23:20:16.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.519+0000 7f8751801640 1 -- 172.21.15.62:0/2429409460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f874c103070 msgr2=0x7f874c103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:16.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.519+0000 7f8751801640 1 --2- 172.21.15.62:0/2429409460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f874c103070 0x7f874c103470 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f873c009900 tx=0x7f873c02a410 comp rx=0 tx=0).stop 2024-09-15T23:20:16.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.519+0000 7f8751801640 1 -- 172.21.15.62:0/2429409460 shutdown_connections 2024-09-15T23:20:16.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.519+0000 7f8751801640 1 --2- 172.21.15.62:0/2429409460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f874c101f60 0x7f874c109dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.519+0000 7f8751801640 1 --2- 172.21.15.62:0/2429409460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874c1015a0 0x7f874c101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.519+0000 7f8751801640 1 --2- 172.21.15.62:0/2429409460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f874c103070 0x7f874c103470 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.519+0000 7f8751801640 1 -- 172.21.15.62:0/2429409460 >> 172.21.15.62:0/2429409460 conn(0x7f874c0fd0f0 msgr2=0x7f874c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:16.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.521+0000 7f8751801640 1 -- 172.21.15.62:0/2429409460 shutdown_connections 2024-09-15T23:20:16.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.521+0000 7f8751801640 1 -- 172.21.15.62:0/2429409460 wait complete. 2024-09-15T23:20:16.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.522+0000 7f8751801640 1 Processor -- start 2024-09-15T23:20:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.523+0000 7f8751801640 1 -- start start 2024-09-15T23:20:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.524+0000 7f8751801640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874c1015a0 0x7f874c195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.524+0000 7f8751801640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f874c101f60 0x7f874c196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.528 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.524+0000 7f8751801640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f874c103070 0x7f874c1ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.524+0000 7f8751801640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f874c10d860 con 0x7f874c103070 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.524+0000 7f8751801640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f874c10d6e0 con 0x7f874c1015a0 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.524+0000 7f8751801640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f874c10d9e0 con 0x7f874c101f60 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.524+0000 7f874b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f874c103070 0x7f874c1ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.524+0000 7f874b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f874c103070 0x7f874c1ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60754/0 (socket says 172.21.15.62:60754) 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.524+0000 7f874b7fe640 1 -- 172.21.15.62:0/3221538419 learned_addr learned my addr 172.21.15.62:0/3221538419 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.525+0000 7f874b7fe640 1 -- 172.21.15.62:0/3221538419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f874c101f60 msgr2=0x7f874c196300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.525+0000 7f874b7fe640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f874c101f60 0x7f874c196300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.525+0000 7f874b7fe640 1 -- 172.21.15.62:0/3221538419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874c1015a0 msgr2=0x7f874c195dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.525+0000 7f874b7fe640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874c1015a0 0x7f874c195dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.525+0000 7f874b7fe640 1 -- 172.21.15.62:0/3221538419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874c19b490 con 0x7f874c103070 2024-09-15T23:20:16.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.525+0000 7f874b7fe640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f874c103070 0x7f874c1ae770 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f874400b970 tx=0x7f874400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.526+0000 7f8733fff640 1 -- 172.21.15.62:0/3221538419 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744019070 con 0x7f874c103070 2024-09-15T23:20:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.526+0000 7f8733fff640 1 -- 172.21.15.62:0/3221538419 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f874400cb40 con 0x7f874c103070 2024-09-15T23:20:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.526+0000 7f8733fff640 1 -- 172.21.15.62:0/3221538419 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87440076a0 con 0x7f874c103070 2024-09-15T23:20:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.527+0000 7f8751801640 1 -- 172.21.15.62:0/3221538419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f874c1aed10 con 0x7f874c103070 2024-09-15T23:20:16.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.527+0000 7f8751801640 1 -- 172.21.15.62:0/3221538419 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f874c1af200 con 0x7f874c103070 2024-09-15T23:20:16.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.531+0000 7f8754670740 1 Processor -- start 2024-09-15T23:20:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.532+0000 7f8733fff640 1 -- 172.21.15.62:0/3221538419 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f87440149b0 con 0x7f874c103070 2024-09-15T23:20:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.533+0000 7f8754670740 1 -- start start 2024-09-15T23:20:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.533+0000 7f8754670740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 0x556de1570f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.533+0000 7f8754670740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1575f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.533+0000 7f8754670740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 0x556de1578a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.533+0000 7f8754670740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556de14d42f0 con 0x556de15714b0 2024-09-15T23:20:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.533+0000 7f8754670740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556de147c270 con 0x556de1576540 2024-09-15T23:20:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.533+0000 7f8754670740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556de14d02b0 con 0x556de1570b70 2024-09-15T23:20:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.533+0000 7f874affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1575f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.533+0000 7f874affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1575f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60756/0 (socket says 172.21.15.62:60756) 2024-09-15T23:20:16.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.534+0000 7f874affd640 1 -- 172.21.15.62:0/4286043373 learned_addr learned my addr 172.21.15.62:0/4286043373 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.534+0000 7f874a7fc640 1 --2- 172.21.15.62:0/4286043373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 0x556de1570f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.534+0000 7f874a7fc640 1 --2- 172.21.15.62:0/4286043373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 0x556de1578a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.538 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.537+0000 7f874affd640 1 -- 172.21.15.62:0/4286043373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 msgr2=0x556de1570f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.537+0000 7f874affd640 1 --2- 172.21.15.62:0/4286043373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 0x556de1570f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.537+0000 7f874affd640 1 -- 172.21.15.62:0/4286043373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 msgr2=0x556de1578a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.537+0000 7f874affd640 1 --2- 172.21.15.62:0/4286043373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 0x556de1578a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.537+0000 7f874affd640 1 -- 172.21.15.62:0/4286043373 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de147b340 con 0x556de15714b0 2024-09-15T23:20:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.538+0000 7f8733fff640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87340a3410 0x7f87340a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.539+0000 7f874affd640 1 --2- 172.21.15.62:0/4286043373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1575f70 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f873c000c00 tx=0x7f873c009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab8abdb995338145 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:16.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.539+0000 7f8733fff640 1 -- 172.21.15.62:0/3221538419 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f87440cf1e0 con 0x7f874c103070 2024-09-15T23:20:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.539+0000 7f87317fa640 1 -- 172.21.15.62:0/4286043373 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c03e070 con 0x556de15714b0 2024-09-15T23:20:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.539+0000 7f87317fa640 1 -- 172.21.15.62:0/4286043373 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f873c004480 con 0x556de15714b0 2024-09-15T23:20:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.539+0000 7f874a7fc640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87340a3410 0x7f87340a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.539+0000 7f87317fa640 1 -- 172.21.15.62:0/4286043373 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c035040 con 0x556de15714b0 2024-09-15T23:20:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f8754670740 1 -- 172.21.15.62:0/4286043373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 msgr2=0x556de1575f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f874a7fc640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87340a3410 0x7f87340a58d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f874c1972d0 tx=0x7f8740004840 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f8754670740 1 --2- 172.21.15.62:0/4286043373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1575f70 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f873c000c00 tx=0x7f873c009680 comp rx=0 tx=0).stop 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f8754670740 1 -- 172.21.15.62:0/4286043373 shutdown_connections 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f8754670740 1 --2- 172.21.15.62:0/4286043373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 0x556de1578a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f8754670740 1 --2- 172.21.15.62:0/4286043373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1575f70 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f8754670740 1 --2- 172.21.15.62:0/4286043373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 0x556de1570f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f8754670740 1 -- 172.21.15.62:0/4286043373 >> 172.21.15.62:0/4286043373 conn(0x556de156c1e0 msgr2=0x556de156c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f8754670740 1 -- 172.21.15.62:0/4286043373 shutdown_connections 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.540+0000 7f8754670740 1 -- 172.21.15.62:0/4286043373 wait complete. 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f8754670740 1 -- start start 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f8754670740 1 Processor -- start 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f8754670740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 0x556de157ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f8754670740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f874a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f874a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60762/0 (socket says 172.21.15.62:60762) 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f874a7fc640 1 -- 172.21.15.62:0/3667616086 learned_addr learned my addr 172.21.15.62:0/3667616086 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f8754670740 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 0x556de15822a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556de1579500 con 0x556de15714b0 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556de145d290 con 0x556de1576540 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556de1579680 con 0x556de1570b70 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f874affd640 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 0x556de157ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f8754670740 10 client.0 _ll_register_callbacks cb 0x556de148e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.541+0000 7f874b7fe640 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 0x556de15822a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f874a7fc640 1 -- 172.21.15.62:0/3667616086 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 msgr2=0x556de157ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f874a7fc640 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 0x556de157ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f874a7fc640 1 -- 172.21.15.62:0/3667616086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 msgr2=0x556de15822a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:16.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f874a7fc640 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 0x556de15822a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f874a7fc640 1 -- 172.21.15.62:0/3667616086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556de11e7210 con 0x556de15714b0 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f874a7fc640 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157b2d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8740017690 tx=0x7f8740006be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791d1164e24fbc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f872eff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f872eff5640 1 -- 172.21.15.62:0/3667616086 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8740027060 con 0x556de15714b0 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f872eff5640 1 -- 172.21.15.62:0/3667616086 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8740015aa0 con 0x556de15714b0 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f873c004480 con 0x556de15714b0 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f872eff5640 1 -- 172.21.15.62:0/3667616086 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f874002e050 con 0x556de15714b0 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f8754670740 10 client.16422 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.542+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556de12b3620 con 0x556de15714b0 2024-09-15T23:20:16.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.543+0000 7f872eff5640 1 -- 172.21.15.62:0/3667616086 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f873c004480 con 0x556de15714b0 2024-09-15T23:20:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.544+0000 7f872eff5640 1 -- 172.21.15.62:0/3667616086 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f8740022400 con 0x556de15714b0 2024-09-15T23:20:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.544+0000 7f8754670740 10 client.16422 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.544+0000 7f8754670740 10 client.16422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.544+0000 7f8754670740 20 client.16422 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.544+0000 7f8754670740 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556de1586a40 0x556de1588e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.544+0000 7f8754670740 4 client.16422 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:16.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.545+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556de14e1a20 con 0x556de1586a40 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.545+0000 7f874affd640 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556de1586a40 0x556de1588e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.546+0000 7f874affd640 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556de1586a40 0x556de1588e30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac4b96977bebcc37 server_cookie=e1510ed89c0acd56 in_seq=0 out_seq=0 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.546+0000 7f872eff5640 10 client.16422 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.546+0000 7f872eff5640 1 -- 172.21.15.62:0/3667616086 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556de14e1a20 con 0x556de1586a40 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.546+0000 7f872eff5640 10 client.16422 handle_command_reply: tid=0 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.546+0000 7f8754670740 1 client.16422 shutdown 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 msgr2=0x556de157b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157b2d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8740017690 tx=0x7f8740006be0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 shutdown_connections 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556de1586a40 0x556de1588e30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f872eff5640 0 client.16422 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1576540 0x556de15822a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157b2d0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 --2- 172.21.15.62:0/3667616086 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1570b70 0x556de157ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 >> 172.21.15.62:0/3667616086 conn(0x556de156c1e0 msgr2=0x556de156d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:16.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 shutdown_connections 2024-09-15T23:20:16.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 1 -- 172.21.15.62:0/3667616086 wait complete. 2024-09-15T23:20:16.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.547+0000 7f8754670740 20 client.16422 trim_cache size 0 max 16384 2024-09-15T23:20:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.570+0000 7f8754670740 1 Processor -- start 2024-09-15T23:20:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.570+0000 7f8754670740 1 -- start start 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.570+0000 7f8754670740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 0x556de156aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.570+0000 7f8754670740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1582260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.570+0000 7f8754670740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de156afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.570+0000 7f8754670740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f874c1bbc90 con 0x556de15714b0 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.570+0000 7f8754670740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f874c1b9560 con 0x556de1570b70 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.570+0000 7f8754670740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f874c1b73b0 con 0x556de1576540 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1582260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 0x556de156aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 0x556de156aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:56126/0 (socket says 172.21.15.62:56126) 2024-09-15T23:20:16.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1582260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60768/0 (socket says 172.21.15.62:60768) 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874affd640 1 -- 172.21.15.62:0/2181063563 learned_addr learned my addr 172.21.15.62:0/2181063563 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874b7fe640 1 --2- 172.21.15.62:0/2181063563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de156afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874a7fc640 1 -- 172.21.15.62:0/2181063563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 msgr2=0x556de156afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874a7fc640 1 --2- 172.21.15.62:0/2181063563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de156afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874a7fc640 1 -- 172.21.15.62:0/2181063563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 msgr2=0x556de156aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874a7fc640 1 --2- 172.21.15.62:0/2181063563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 0x556de156aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874a7fc640 1 -- 172.21.15.62:0/2181063563 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556de156c000 con 0x556de15714b0 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874affd640 1 --2- 172.21.15.62:0/2181063563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 0x556de156aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f874a7fc640 1 --2- 172.21.15.62:0/2181063563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1582260 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f874000d590 tx=0x7f874000d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7427d01372fddf87 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f8730ff9640 1 -- 172.21.15.62:0/2181063563 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8740037070 con 0x556de15714b0 2024-09-15T23:20:16.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f8730ff9640 1 -- 172.21.15.62:0/2181063563 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8740010040 con 0x556de15714b0 2024-09-15T23:20:16.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.571+0000 7f8730ff9640 1 -- 172.21.15.62:0/2181063563 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f874000f050 con 0x556de15714b0 2024-09-15T23:20:16.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.572+0000 7f8754670740 1 -- 172.21.15.62:0/2181063563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 msgr2=0x556de1582260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.572+0000 7f8754670740 1 --2- 172.21.15.62:0/2181063563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1582260 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f874000d590 tx=0x7f874000d9d0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.572+0000 7f8754670740 1 -- 172.21.15.62:0/2181063563 shutdown_connections 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.572+0000 7f8754670740 1 --2- 172.21.15.62:0/2181063563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de156afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.572+0000 7f8754670740 1 --2- 172.21.15.62:0/2181063563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de1582260 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.572+0000 7f8754670740 1 --2- 172.21.15.62:0/2181063563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 0x556de156aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.572+0000 7f8754670740 1 -- 172.21.15.62:0/2181063563 >> 172.21.15.62:0/2181063563 conn(0x556de156c1e0 msgr2=0x556de1572650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.572+0000 7f8754670740 1 -- 172.21.15.62:0/2181063563 shutdown_connections 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.572+0000 7f8754670740 1 -- 172.21.15.62:0/2181063563 wait complete. 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.573+0000 7f8754670740 1 -- start start 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.573+0000 7f8754670740 1 Processor -- start 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.573+0000 7f8754670740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 0x556de157f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.573+0000 7f8754670740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.573+0000 7f8754670740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de1580ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.573+0000 7f8754670740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f874c1b7a20 con 0x556de15714b0 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.573+0000 7f8754670740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f874c1b8e90 con 0x556de1570b70 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.573+0000 7f8754670740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f874c1b80f0 con 0x556de1576540 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.573+0000 7f8754670740 10 client.0 _ll_register_callbacks cb 0x7f874000d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.574+0000 7f874b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de1580ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.574+0000 7f874b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de1580ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53128/0 (socket says 172.21.15.62:53128) 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.574+0000 7f874b7fe640 1 -- 172.21.15.62:0/1438787508 learned_addr learned my addr 172.21.15.62:0/1438787508 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.574+0000 7f874a7fc640 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.574+0000 7f874b7fe640 1 -- 172.21.15.62:0/1438787508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 msgr2=0x556de157f590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.574+0000 7f874b7fe640 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 0x556de157f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.574+0000 7f874b7fe640 1 -- 172.21.15.62:0/1438787508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 msgr2=0x556de157fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.574+0000 7f874b7fe640 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.574+0000 7f874b7fe640 1 -- 172.21.15.62:0/1438787508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556de15810f0 con 0x556de1576540 2024-09-15T23:20:16.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.575+0000 7f874b7fe640 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de1580ae0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f874409b3b0 tx=0x7f8744095f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91868b1aea15679f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.575+0000 7f872eff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:20:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.575+0000 7f872eff5640 1 -- 172.21.15.62:0/1438787508 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87440b3070 con 0x556de1576540 2024-09-15T23:20:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.575+0000 7f8754670740 1 -- 172.21.15.62:0/1438787508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8740010040 con 0x556de1576540 2024-09-15T23:20:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.575+0000 7f872eff5640 1 -- 172.21.15.62:0/1438787508 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f87440b4260 con 0x556de1576540 2024-09-15T23:20:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.575+0000 7f872eff5640 1 -- 172.21.15.62:0/1438787508 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f87440a4090 con 0x556de1576540 2024-09-15T23:20:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.575+0000 7f8754670740 10 client.25891 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.575+0000 7f8754670740 1 -- 172.21.15.62:0/1438787508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556de15824d0 con 0x556de1576540 2024-09-15T23:20:16.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f872eff5640 1 -- 172.21.15.62:0/1438787508 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8740010040 con 0x556de1576540 2024-09-15T23:20:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f872eff5640 1 -- 172.21.15.62:0/1438787508 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f874409a9e0 con 0x556de1576540 2024-09-15T23:20:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f8754670740 10 client.25891 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f8754670740 10 client.25891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f8754670740 20 client.25891 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f8754670740 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556de1586a40 0x556de15f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f8754670740 4 client.25891 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:20:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f8754670740 1 -- 172.21.15.62:0/1438787508 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556de14dbd70 con 0x556de1586a40 2024-09-15T23:20:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f874affd640 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556de1586a40 0x556de15f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.576+0000 7f872eff5640 10 client.25891 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:16.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.577+0000 7f874affd640 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556de1586a40 0x556de15f3d50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28acd77634385721 server_cookie=bea4df767a050c7f in_seq=0 out_seq=0 2024-09-15T23:20:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f872eff5640 1 -- 172.21.15.62:0/1438787508 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556de14dbd70 con 0x556de1586a40 2024-09-15T23:20:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f872eff5640 10 client.25891 handle_command_reply: tid=0 2024-09-15T23:20:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 client.25891 shutdown 2024-09-15T23:20:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 -- 172.21.15.62:0/1438787508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 msgr2=0x556de1580ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de1580ae0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f874409b3b0 tx=0x7f8744095f90 comp rx=0 tx=0).stop 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 -- 172.21.15.62:0/1438787508 shutdown_connections 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556de1586a40 0x556de15f3d50 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556de1576540 0x556de1580ae0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556de15714b0 0x556de157fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 --2- 172.21.15.62:0/1438787508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556de1570b70 0x556de157f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 -- 172.21.15.62:0/1438787508 >> 172.21.15.62:0/1438787508 conn(0x556de156c1e0 msgr2=0x556de156d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f872eff5640 0 client.25891 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 -- 172.21.15.62:0/1438787508 shutdown_connections 2024-09-15T23:20:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 1 -- 172.21.15.62:0/1438787508 wait complete. 2024-09-15T23:20:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.590+0000 7f8754670740 20 client.25891 trim_cache size 0 max 16384 2024-09-15T23:20:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:20:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:20:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:20:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 -- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87340a3410 msgr2=0x7f87340a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87340a3410 0x7f87340a58d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f874c1972d0 tx=0x7f8740004840 comp rx=0 tx=0).stop 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 -- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f874c103070 msgr2=0x7f874c1ae770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f874c103070 0x7f874c1ae770 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f874400b970 tx=0x7f874400be30 comp rx=0 tx=0).stop 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 -- 172.21.15.62:0/3221538419 shutdown_connections 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87340a3410 0x7f87340a58d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f874c103070 0x7f874c1ae770 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f874c101f60 0x7f874c196300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 --2- 172.21.15.62:0/3221538419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f874c1015a0 0x7f874c195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:16.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 -- 172.21.15.62:0/3221538419 >> 172.21.15.62:0/3221538419 conn(0x7f874c0fd0f0 msgr2=0x7f874c108370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 -- 172.21.15.62:0/3221538419 shutdown_connections 2024-09-15T23:20:16.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:16.591+0000 7f8751801640 1 -- 172.21.15.62:0/3221538419 wait complete. 2024-09-15T23:20:17.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:17 smithi067 ceph-mon[35665]: pgmap v490: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.6 MiB/s wr, 411 op/s 2024-09-15T23:20:17.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:17 smithi148 ceph-mon[35610]: pgmap v490: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.6 MiB/s wr, 411 op/s 2024-09-15T23:20:17.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:17 smithi062 ceph-mon[30949]: pgmap v490: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 7.6 MiB/s wr, 411 op/s 2024-09-15T23:20:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:19 smithi067 ceph-mon[35665]: pgmap v491: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.3 MiB/s wr, 272 op/s 2024-09-15T23:20:19.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:19 smithi148 ceph-mon[35610]: pgmap v491: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.3 MiB/s wr, 272 op/s 2024-09-15T23:20:19.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 240.0 s, 1945.6 tps, lat 51.406 ms stddev 66.168 2024-09-15T23:20:19.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:19 smithi062 ceph-mon[30949]: pgmap v491: 129 pgs: 129 active+clean; 1.1 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.3 MiB/s wr, 272 op/s 2024-09-15T23:20:21.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:21 smithi067 ceph-mon[35665]: pgmap v492: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.5 MiB/s wr, 356 op/s 2024-09-15T23:20:21.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:21 smithi148 ceph-mon[35610]: pgmap v492: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.5 MiB/s wr, 356 op/s 2024-09-15T23:20:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:21 smithi062 ceph-mon[30949]: pgmap v492: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 6.5 MiB/s wr, 356 op/s 2024-09-15T23:20:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:23 smithi067 ceph-mon[35665]: pgmap v493: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 4.5 MiB/s wr, 352 op/s 2024-09-15T23:20:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:23 smithi148 ceph-mon[35610]: pgmap v493: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 4.5 MiB/s wr, 352 op/s 2024-09-15T23:20:23.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:23 smithi062 ceph-mon[30949]: pgmap v493: 129 pgs: 129 active+clean; 1.1 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 4.5 MiB/s wr, 352 op/s 2024-09-15T23:20:24.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 245.0 s, 2286.0 tps, lat 44.279 ms stddev 51.331 2024-09-15T23:20:25.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:25 smithi067 ceph-mon[35665]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 5.3 MiB/s wr, 455 op/s 2024-09-15T23:20:25.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:25 smithi148 ceph-mon[35610]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 5.3 MiB/s wr, 455 op/s 2024-09-15T23:20:25.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:25 smithi062 ceph-mon[30949]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 5.3 MiB/s wr, 455 op/s 2024-09-15T23:20:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:27 smithi067 ceph-mon[35665]: pgmap v495: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 6.4 MiB/s wr, 563 op/s 2024-09-15T23:20:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:27 smithi148 ceph-mon[35610]: pgmap v495: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 6.4 MiB/s wr, 563 op/s 2024-09-15T23:20:27.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:27 smithi062 ceph-mon[30949]: pgmap v495: 129 pgs: 129 active+clean; 1.1 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 6.4 MiB/s wr, 563 op/s 2024-09-15T23:20:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:29 smithi067 ceph-mon[35665]: pgmap v496: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 6.4 MiB/s wr, 577 op/s 2024-09-15T23:20:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:29 smithi148 ceph-mon[35610]: pgmap v496: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 6.4 MiB/s wr, 577 op/s 2024-09-15T23:20:29.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 250.0 s, 1999.6 tps, lat 48.912 ms stddev 58.008 2024-09-15T23:20:29.735 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:29 smithi062 ceph-mon[30949]: pgmap v496: 129 pgs: 129 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 6.4 MiB/s wr, 577 op/s 2024-09-15T23:20:29.740 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.736+0000 7f9eb96ac640 1 -- 172.21.15.62:0/369722205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4072d20 msgr2=0x7f9eb4073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.736+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/369722205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4072d20 0x7f9eb4073180 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009900 tx=0x7f9ea402a440 comp rx=0 tx=0).stop 2024-09-15T23:20:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.743+0000 7f9eb96ac640 1 -- 172.21.15.62:0/369722205 shutdown_connections 2024-09-15T23:20:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.743+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/369722205 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9eb40736c0 0x7f9eb4119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.743+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/369722205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4072d20 0x7f9eb4073180 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.743+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/369722205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eb4109700 0x7f9eb4109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.743+0000 7f9eb96ac640 1 -- 172.21.15.62:0/369722205 >> 172.21.15.62:0/369722205 conn(0x7f9eb406ece0 msgr2=0x7f9eb4071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.749+0000 7f9eb96ac640 1 -- 172.21.15.62:0/369722205 shutdown_connections 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.749+0000 7f9eb96ac640 1 -- 172.21.15.62:0/369722205 wait complete. 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.750+0000 7f9eb96ac640 1 Processor -- start 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.750+0000 7f9eb96ac640 1 -- start start 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb96ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eb4072d20 0x7f9eb4115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb96ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9eb40736c0 0x7f9eb4116250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb96ac640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4109700 0x7f9eb4121070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb96ac640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb411c680 con 0x7f9eb4109700 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb96ac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb411c500 con 0x7f9eb4072d20 2024-09-15T23:20:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb96ac640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9eb411c800 con 0x7f9eb40736c0 2024-09-15T23:20:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4109700 0x7f9eb4121070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eb4072d20 0x7f9eb4115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4109700 0x7f9eb4121070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49344/0 (socket says 172.21.15.62:49344) 2024-09-15T23:20:29.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.751+0000 7f9eb37fe640 1 -- 172.21.15.62:0/18927785 learned_addr learned my addr 172.21.15.62:0/18927785 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.752+0000 7f9eb37fe640 1 -- 172.21.15.62:0/18927785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9eb40736c0 msgr2=0x7f9eb4116250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:20:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.752+0000 7f9eb37fe640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9eb40736c0 0x7f9eb4116250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.752+0000 7f9eb37fe640 1 -- 172.21.15.62:0/18927785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eb4072d20 msgr2=0x7f9eb4115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.752+0000 7f9eb37fe640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eb4072d20 0x7f9eb4115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.752+0000 7f9eb37fe640 1 -- 172.21.15.62:0/18927785 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb41193d0 con 0x7f9eb4109700 2024-09-15T23:20:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.752+0000 7f9eb37fe640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4109700 0x7f9eb4121070 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00b3f0 tx=0x7f9e9c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.754+0000 7f9eabfff640 1 -- 172.21.15.62:0/18927785 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c015020 con 0x7f9eb4109700 2024-09-15T23:20:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.754+0000 7f9eabfff640 1 -- 172.21.15.62:0/18927785 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e9c011d10 con 0x7f9eb4109700 2024-09-15T23:20:29.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.754+0000 7f9eabfff640 1 -- 172.21.15.62:0/18927785 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e9c00f2e0 con 0x7f9eb4109700 2024-09-15T23:20:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.755+0000 7f9eb96ac640 1 -- 172.21.15.62:0/18927785 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb4121610 con 0x7f9eb4109700 2024-09-15T23:20:29.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.755+0000 7f9eb96ac640 1 -- 172.21.15.62:0/18927785 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb4121b90 con 0x7f9eb4109700 2024-09-15T23:20:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.763+0000 7f9eabfff640 1 -- 172.21.15.62:0/18927785 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9e9c018020 con 0x7f9eb4109700 2024-09-15T23:20:29.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.769+0000 7f9ebc44c740 1 Processor -- start 2024-09-15T23:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.770+0000 7f9ebc44c740 1 -- start start 2024-09-15T23:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.774+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077a9a0 0x55cfb077ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.774+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077b2e0 0x55cfb077fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.774+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb0782830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.774+0000 7f9ebc44c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55cfb06de2f0 con 0x55cfb077a9a0 2024-09-15T23:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.774+0000 7f9ebc44c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55cfb0686270 con 0x55cfb0780370 2024-09-15T23:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.774+0000 7f9ebc44c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55cfb06da2b0 con 0x55cfb077b2e0 2024-09-15T23:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.775+0000 7f9eabfff640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ea00a3530 0x7f9ea00a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.775+0000 7f9eabfff640 1 -- 172.21.15.62:0/18927785 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e9c0cf330 con 0x7f9eb4109700 2024-09-15T23:20:29.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.775+0000 7f9eb27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb0782830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.775+0000 7f9eb27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb0782830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:55006/0 (socket says 172.21.15.62:55006) 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.775+0000 7f9eb27fc640 1 -- 172.21.15.62:0/1333979813 learned_addr learned my addr 172.21.15.62:0/1333979813 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.775+0000 7f9eb37fe640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ea00a3530 0x7f9ea00a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.775+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/1333979813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077a9a0 0x55cfb077ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.776+0000 7f9eb27fc640 1 -- 172.21.15.62:0/1333979813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077b2e0 msgr2=0x55cfb077fda0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.776+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/1333979813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077b2e0 0x55cfb077fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.776+0000 7f9eb27fc640 1 -- 172.21.15.62:0/1333979813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077a9a0 msgr2=0x55cfb077ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.776+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/1333979813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077a9a0 0x55cfb077ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.776+0000 7f9eb27fc640 1 -- 172.21.15.62:0/1333979813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfb0685340 con 0x55cfb0780370 2024-09-15T23:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.776+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/1333979813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb0782830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4031c70 tx=0x7f9ea4031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e8e5c8fee863baf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:29.780 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.777+0000 7f9eb37fe640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ea00a3530 0x7f9ea00a59f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0979d0 tx=0x7f9e9c087040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:29.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.784+0000 7f9ea97fa640 1 -- 172.21.15.62:0/1333979813 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4035030 con 0x55cfb0780370 2024-09-15T23:20:29.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.784+0000 7f9ea97fa640 1 -- 172.21.15.62:0/1333979813 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea4005050 con 0x55cfb0780370 2024-09-15T23:20:29.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.784+0000 7f9ea97fa640 1 -- 172.21.15.62:0/1333979813 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40049f0 con 0x55cfb0780370 2024-09-15T23:20:29.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.786+0000 7f9ebc44c740 1 -- 172.21.15.62:0/1333979813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 msgr2=0x55cfb0782830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:29.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.786+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/1333979813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb0782830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4031c70 tx=0x7f9ea4031400 comp rx=0 tx=0).stop 2024-09-15T23:20:29.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.787+0000 7f9ebc44c740 1 -- 172.21.15.62:0/1333979813 shutdown_connections 2024-09-15T23:20:29.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.787+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/1333979813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb0782830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.787+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/1333979813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077b2e0 0x55cfb077fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.787+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/1333979813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077a9a0 0x55cfb077ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.787+0000 7f9ebc44c740 1 -- 172.21.15.62:0/1333979813 >> 172.21.15.62:0/1333979813 conn(0x55cfb0776010 msgr2=0x55cfb0776410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:29.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.790+0000 7f9ebc44c740 1 -- 172.21.15.62:0/1333979813 shutdown_connections 2024-09-15T23:20:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.791+0000 7f9ebc44c740 1 -- 172.21.15.62:0/1333979813 wait complete. 2024-09-15T23:20:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.791+0000 7f9ebc44c740 1 -- start start 2024-09-15T23:20:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.791+0000 7f9ebc44c740 1 Processor -- start 2024-09-15T23:20:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.792+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb077a9a0 0x55cfb0784c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.792+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb07851a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.792+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb0780370 0x55cfb078c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.793+0000 7f9eb27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb077a9a0 0x55cfb0784c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.793+0000 7f9eb27fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb077a9a0 0x55cfb0784c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:55012/0 (socket says 172.21.15.62:55012) 2024-09-15T23:20:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.793+0000 7f9eb27fc640 1 -- 172.21.15.62:0/2296637871 learned_addr learned my addr 172.21.15.62:0/2296637871 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.793+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55cfb07832e0 con 0x55cfb077b2e0 2024-09-15T23:20:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.793+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55cfb0667290 con 0x55cfb077a9a0 2024-09-15T23:20:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.793+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55cfb0783460 con 0x55cfb0780370 2024-09-15T23:20:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.793+0000 7f9ebc44c740 10 client.0 _ll_register_callbacks cb 0x55cfb0698140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:29.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.794+0000 7f9eb2ffd640 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb07851a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.794+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb0780370 0x55cfb078c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.794+0000 7f9eb27fc640 1 -- 172.21.15.62:0/2296637871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb0780370 msgr2=0x55cfb078c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.794+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb0780370 0x55cfb078c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.794+0000 7f9eb27fc640 1 -- 172.21.15.62:0/2296637871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 msgr2=0x55cfb07851a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.794+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb07851a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.794+0000 7f9eb27fc640 1 -- 172.21.15.62:0/2296637871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfb03f1210 con 0x55cfb077a9a0 2024-09-15T23:20:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.794+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb077a9a0 0x55cfb0784c60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4001eb0 tx=0x7f9ea4005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee48b70196ae717d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.796+0000 7f9e9a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.796+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/2296637871 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4035070 con 0x55cfb077a9a0 2024-09-15T23:20:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.797+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ea4005050 con 0x55cfb077a9a0 2024-09-15T23:20:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.797+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/2296637871 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9ea4004b40 con 0x55cfb077a9a0 2024-09-15T23:20:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.797+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/2296637871 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9ea4004d40 con 0x55cfb077a9a0 2024-09-15T23:20:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.797+0000 7f9ebc44c740 10 client.26063 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.797+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cfb04bd620 con 0x55cfb077a9a0 2024-09-15T23:20:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.798+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/2296637871 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ea4041070 con 0x55cfb077a9a0 2024-09-15T23:20:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.798+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/2296637871 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f9ea403a040 con 0x55cfb077a9a0 2024-09-15T23:20:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.799+0000 7f9ebc44c740 10 client.26063 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.799+0000 7f9ebc44c740 10 client.26063 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.799+0000 7f9ebc44c740 20 client.26063 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.799+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cfb07904e0 0x55cfb07928d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.799+0000 7f9ebc44c740 4 client.26063 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.799+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9ea4005050 con 0x55cfb07904e0 2024-09-15T23:20:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.799+0000 7f9eb2ffd640 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cfb07904e0 0x55cfb07928d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.799+0000 7f9e9a7fc640 10 client.26063 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.799+0000 7f9eb2ffd640 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cfb07904e0 0x55cfb07928d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa02ca9a327f72bf server_cookie=a0867596b50cb2da in_seq=0 out_seq=0 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/2296637871 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f9ea4005050 con 0x55cfb07904e0 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9e9a7fc640 10 client.26063 handle_command_reply: tid=0 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9ebc44c740 1 client.26063 shutdown 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb077a9a0 msgr2=0x55cfb0784c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb077a9a0 0x55cfb0784c60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4001eb0 tx=0x7f9ea4005b50 comp rx=0 tx=0).stop 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 shutdown_connections 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cfb07904e0 0x55cfb07928d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb0780370 0x55cfb078c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb07851a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/2296637871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb077a9a0 0x55cfb0784c60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 >> 172.21.15.62:0/2296637871 conn(0x55cfb0776010 msgr2=0x55cfb0777820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.800+0000 7f9e9a7fc640 0 client.26063 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.801+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 shutdown_connections 2024-09-15T23:20:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.801+0000 7f9ebc44c740 1 -- 172.21.15.62:0/2296637871 wait complete. 2024-09-15T23:20:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.801+0000 7f9ebc44c740 20 client.26063 trim_cache size 0 max 16384 2024-09-15T23:20:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.826+0000 7f9ebc44c740 1 Processor -- start 2024-09-15T23:20:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.826+0000 7f9ebc44c740 1 -- start start 2024-09-15T23:20:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.826+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb0792720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.826+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb07750b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9ebc44c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e94012530 con 0x55cfb077b2e0 2024-09-15T23:20:29.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9ebc44c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9400fe00 con 0x55cfb0780370 2024-09-15T23:20:29.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9ebc44c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9400dc50 con 0x55cfb077a9a0 2024-09-15T23:20:29.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb0792720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49354/0 (socket says 172.21.15.62:49354) 2024-09-15T23:20:29.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb2ffd640 1 -- 172.21.15.62:0/318236309 learned_addr learned my addr 172.21.15.62:0/318236309 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:29.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb0792720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:55538/0 (socket says 172.21.15.62:55538) 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/318236309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb07750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb2ffd640 1 -- 172.21.15.62:0/318236309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 msgr2=0x55cfb0792720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb2ffd640 1 --2- 172.21.15.62:0/318236309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb0792720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb2ffd640 1 -- 172.21.15.62:0/318236309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 msgr2=0x55cfb07750b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb2ffd640 1 --2- 172.21.15.62:0/318236309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb07750b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb2ffd640 1 -- 172.21.15.62:0/318236309 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfb07847a0 con 0x55cfb077b2e0 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/318236309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb07750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.827+0000 7f9eb2ffd640 1 --2- 172.21.15.62:0/318236309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078c170 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x55cfb077bd10 tx=0x7f9eac00a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29388cc75a63341a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.828+0000 7f9ea8ff9640 1 -- 172.21.15.62:0/318236309 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac003b70 con 0x55cfb077b2e0 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.828+0000 7f9ea8ff9640 1 -- 172.21.15.62:0/318236309 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac003d10 con 0x55cfb077b2e0 2024-09-15T23:20:29.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.828+0000 7f9ea8ff9640 1 -- 172.21.15.62:0/318236309 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eac01b960 con 0x55cfb077b2e0 2024-09-15T23:20:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 -- 172.21.15.62:0/318236309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 msgr2=0x55cfb078c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/318236309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078c170 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x55cfb077bd10 tx=0x7f9eac00a490 comp rx=0 tx=0).stop 2024-09-15T23:20:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 -- 172.21.15.62:0/318236309 shutdown_connections 2024-09-15T23:20:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/318236309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb07750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/318236309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078c170 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/318236309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb0792720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 -- 172.21.15.62:0/318236309 >> 172.21.15.62:0/318236309 conn(0x55cfb0776010 msgr2=0x55cfb07749a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 -- 172.21.15.62:0/318236309 shutdown_connections 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 -- 172.21.15.62:0/318236309 wait complete. 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 -- start start 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.829+0000 7f9ebc44c740 1 Processor -- start 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.830+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb078a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.830+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.830+0000 7f9eb27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb078a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.830+0000 7f9eb27fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb078a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:55548/0 (socket says 172.21.15.62:55548) 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.830+0000 7f9ebc44c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb078b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9eb27fc640 1 -- 172.21.15.62:0/515503306 learned_addr learned my addr 172.21.15.62:0/515503306 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9400e2c0 con 0x55cfb077b2e0 2024-09-15T23:20:29.835 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9eb2ffd640 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9400f730 con 0x55cfb0780370 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9400e990 con 0x55cfb077a9a0 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb078b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9ebc44c740 10 client.0 _ll_register_callbacks cb 0x7f9ea40316b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9eb27fc640 1 -- 172.21.15.62:0/515503306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 msgr2=0x55cfb078b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb078b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9eb27fc640 1 -- 172.21.15.62:0/515503306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 msgr2=0x55cfb078a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.831+0000 7f9eb27fc640 1 -- 172.21.15.62:0/515503306 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfb078bb30 con 0x55cfb077a9a0 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.832+0000 7f9eb27fc640 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb078a020 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4001eb0 tx=0x7f9ea402a440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee24c0cea54d2d9a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.832+0000 7f9e9a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.832+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/515503306 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4035040 con 0x55cfb077a9a0 2024-09-15T23:20:29.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.832+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/515503306 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9ea403cd40 con 0x55cfb077a9a0 2024-09-15T23:20:29.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.832+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9eac003d10 con 0x55cfb077a9a0 2024-09-15T23:20:29.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.832+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/515503306 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9ea4005850 con 0x55cfb077a9a0 2024-09-15T23:20:29.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.832+0000 7f9ebc44c740 10 client.25900 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:29.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.832+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55cfb07fb790 con 0x55cfb077a9a0 2024-09-15T23:20:29.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.833+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/515503306 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eac003d10 con 0x55cfb077a9a0 2024-09-15T23:20:29.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.833+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/515503306 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f9ea4004150 con 0x55cfb077a9a0 2024-09-15T23:20:29.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.833+0000 7f9ebc44c740 10 client.25900 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:29.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.833+0000 7f9ebc44c740 10 client.25900 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:29.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.833+0000 7f9ebc44c740 20 client.25900 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:29.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.833+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cfb07fc110 0x55cfb07fe530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:29.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.833+0000 7f9ebc44c740 4 client.25900 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:20:29.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.833+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cfb06eba20 con 0x55cfb07fc110 2024-09-15T23:20:29.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.833+0000 7f9eb2ffd640 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cfb07fc110 0x55cfb07fe530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:29.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.834+0000 7f9e9a7fc640 10 client.25900 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:29.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.834+0000 7f9eb2ffd640 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cfb07fc110 0x55cfb07fe530 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11c6e0266830d284 server_cookie=a3ba2f2719f54b68 in_seq=0 out_seq=0 2024-09-15T23:20:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9e9a7fc640 1 -- 172.21.15.62:0/515503306 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cfb06eba20 con 0x55cfb07fc110 2024-09-15T23:20:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9e9a7fc640 10 client.25900 handle_command_reply: tid=0 2024-09-15T23:20:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 client.25900 shutdown 2024-09-15T23:20:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 msgr2=0x55cfb078a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb078a020 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4001eb0 tx=0x7f9ea402a440 comp rx=0 tx=0).stop 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 shutdown_connections 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cfb07fc110 0x55cfb07fe530 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cfb0780370 0x55cfb078b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cfb077b2e0 0x55cfb078a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 --2- 172.21.15.62:0/515503306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cfb077a9a0 0x55cfb078a020 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9e9a7fc640 0 client.25900 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 >> 172.21.15.62:0/515503306 conn(0x55cfb0776010 msgr2=0x55cfb07751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 shutdown_connections 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 1 -- 172.21.15.62:0/515503306 wait complete. 2024-09-15T23:20:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.844+0000 7f9ebc44c740 20 client.25900 trim_cache size 0 max 16384 2024-09-15T23:20:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:20:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:20:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:20:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 -- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ea00a3530 msgr2=0x7f9ea00a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ea00a3530 0x7f9ea00a59f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0979d0 tx=0x7f9e9c087040 comp rx=0 tx=0).stop 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 -- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4109700 msgr2=0x7f9eb4121070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4109700 0x7f9eb4121070 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c00b3f0 tx=0x7f9e9c00b8b0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 -- 172.21.15.62:0/18927785 shutdown_connections 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ea00a3530 0x7f9ea00a59f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9eb4109700 0x7f9eb4121070 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9eb40736c0 0x7f9eb4116250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 --2- 172.21.15.62:0/18927785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eb4072d20 0x7f9eb4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 -- 172.21.15.62:0/18927785 >> 172.21.15.62:0/18927785 conn(0x7f9eb406ece0 msgr2=0x7f9eb410c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 -- 172.21.15.62:0/18927785 shutdown_connections 2024-09-15T23:20:29.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:29.845+0000 7f9eb96ac640 1 -- 172.21.15.62:0/18927785 wait complete. 2024-09-15T23:20:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:31 smithi067 ceph-mon[35665]: pgmap v497: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.0 MiB/s wr, 685 op/s 2024-09-15T23:20:31.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:31 smithi148 ceph-mon[35610]: pgmap v497: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.0 MiB/s wr, 685 op/s 2024-09-15T23:20:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:31 smithi062 ceph-mon[30949]: pgmap v497: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 7.0 MiB/s wr, 685 op/s 2024-09-15T23:20:32.546 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:20:32.976 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:33.444 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:33 smithi062 ceph-mon[30949]: pgmap v498: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.7 MiB/s wr, 775 op/s 2024-09-15T23:20:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:33 smithi148 ceph-mon[35610]: pgmap v498: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.7 MiB/s wr, 775 op/s 2024-09-15T23:20:33.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.561+0000 7f29d9e7c640 1 -- 172.21.15.62:0/673255505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4100e20 msgr2=0x7f29d4101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:33.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.561+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/673255505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4100e20 0x7f29d4101200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009890 tx=0x7f29bc02a410 comp rx=0 tx=0).stop 2024-09-15T23:20:33.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.561+0000 7f29d9e7c640 1 -- 172.21.15.62:0/673255505 shutdown_connections 2024-09-15T23:20:33.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.561+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/673255505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29d4102ef0 0x7f29d410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.561+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/673255505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f29d4102570 0x7f29d41029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.561+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/673255505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4100e20 0x7f29d4101200 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.561+0000 7f29d9e7c640 1 -- 172.21.15.62:0/673255505 >> 172.21.15.62:0/673255505 conn(0x7f29d40fc7d0 msgr2=0x7f29d40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:33.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.561+0000 7f29d9e7c640 1 -- 172.21.15.62:0/673255505 shutdown_connections 2024-09-15T23:20:33.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.561+0000 7f29d9e7c640 1 -- 172.21.15.62:0/673255505 wait complete. 2024-09-15T23:20:33.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.562+0000 7f29d9e7c640 1 Processor -- start 2024-09-15T23:20:33.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d9e7c640 1 -- start start 2024-09-15T23:20:33.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d9e7c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f29d4100e20 0x7f29d41974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d9e7c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4102570 0x7f29d41979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d9e7c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29d4102ef0 0x7f29d419bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d9e7c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f29d4110f80 con 0x7f29d4102ef0 2024-09-15T23:20:33.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d9e7c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f29d4110e00 con 0x7f29d4102570 2024-09-15T23:20:33.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d9e7c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f29d4111100 con 0x7f29d4100e20 2024-09-15T23:20:33.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4102570 0x7f29d41979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4102570 0x7f29d41979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46158/0 (socket says 172.21.15.62:46158) 2024-09-15T23:20:33.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.564+0000 7f29d2ffd640 1 -- 172.21.15.62:0/470409700 learned_addr learned my addr 172.21.15.62:0/470409700 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:33.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.565+0000 7f29d3fff640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29d4102ef0 0x7f29d419bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.565+0000 7f29d2ffd640 1 -- 172.21.15.62:0/470409700 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f29d4100e20 msgr2=0x7f29d41974a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:20:33.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.565+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f29d4100e20 0x7f29d41974a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.565+0000 7f29d2ffd640 1 -- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29d4102ef0 msgr2=0x7f29d419bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:33.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.565+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29d4102ef0 0x7f29d419bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.565+0000 7f29d2ffd640 1 -- 172.21.15.62:0/470409700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d419c470 con 0x7f29d4102570 2024-09-15T23:20:33.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.565+0000 7f29d3fff640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29d4102ef0 0x7f29d419bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:33.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.566+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4102570 0x7f29d41979e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f29c400e9f0 tx=0x7f29c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:33.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.566+0000 7f29d0ff9640 1 -- 172.21.15.62:0/470409700 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c400cd80 con 0x7f29d4102570 2024-09-15T23:20:33.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.566+0000 7f29d0ff9640 1 -- 172.21.15.62:0/470409700 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29c4010910 con 0x7f29d4102570 2024-09-15T23:20:33.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.566+0000 7f29d9e7c640 1 -- 172.21.15.62:0/470409700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d419c760 con 0x7f29d4102570 2024-09-15T23:20:33.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.566+0000 7f29d0ff9640 1 -- 172.21.15.62:0/470409700 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c4018990 con 0x7f29d4102570 2024-09-15T23:20:33.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.566+0000 7f29d9e7c640 1 -- 172.21.15.62:0/470409700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d4069f70 con 0x7f29d4102570 2024-09-15T23:20:33.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.568+0000 7f29d0ff9640 1 -- 172.21.15.62:0/470409700 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f29c4010ac0 con 0x7f29d4102570 2024-09-15T23:20:33.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.569+0000 7f29dc255740 1 Processor -- start 2024-09-15T23:20:33.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.569+0000 7f29dc255740 1 -- start start 2024-09-15T23:20:33.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.569+0000 7f29dc255740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201e4a0 0x56528201e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.569+0000 7f29dc255740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282029140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.569+0000 7f29d0ff9640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f29c00a3240 0x7f29c00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.569+0000 7f29dc255740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 0x56528202ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.569+0000 7f29dc255740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565281f842e0 con 0x565282029680 2024-09-15T23:20:33.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.569+0000 7f29dc255740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565281f2bce0 con 0x56528201edc0 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.569+0000 7f29dc255740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x565281f802a0 con 0x56528201e4a0 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282029140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282029140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46170/0 (socket says 172.21.15.62:46170) 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d37fe640 1 -- 172.21.15.62:0/820873648 learned_addr learned my addr 172.21.15.62:0/820873648 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d3fff640 1 --2- 172.21.15.62:0/820873648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201e4a0 0x56528201e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d0ff9640 1 -- 172.21.15.62:0/470409700 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f29c40d0160 con 0x7f29d4102570 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/820873648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 0x56528202ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d3fff640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f29c00a3240 0x7f29c00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d37fe640 1 -- 172.21.15.62:0/820873648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201e4a0 msgr2=0x56528201e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d37fe640 1 --2- 172.21.15.62:0/820873648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201e4a0 0x56528201e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d37fe640 1 -- 172.21.15.62:0/820873648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 msgr2=0x56528202ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d37fe640 1 --2- 172.21.15.62:0/820873648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 0x56528202ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.570+0000 7f29d37fe640 1 -- 172.21.15.62:0/820873648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565281f10440 con 0x56528201edc0 2024-09-15T23:20:33.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.571+0000 7f29d3fff640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f29c00a3240 0x7f29c00a5700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009fd0 tx=0x7f29cc012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:33.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.571+0000 7f29d37fe640 1 --2- 172.21.15.62:0/820873648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282029140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0095e0 tx=0x7f29bc02f300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dda3140e847321f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:33.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.572+0000 7f29c9ffb640 1 -- 172.21.15.62:0/820873648 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc030d70 con 0x56528201edc0 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.572+0000 7f29c9ffb640 1 -- 172.21.15.62:0/820873648 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29bc030880 con 0x56528201edc0 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.572+0000 7f29c9ffb640 1 -- 172.21.15.62:0/820873648 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0049b0 con 0x56528201edc0 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.573+0000 7f29dc255740 1 -- 172.21.15.62:0/820873648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 msgr2=0x565282029140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.573+0000 7f29dc255740 1 --2- 172.21.15.62:0/820873648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282029140 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0095e0 tx=0x7f29bc02f300 comp rx=0 tx=0).stop 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.573+0000 7f29dc255740 1 -- 172.21.15.62:0/820873648 shutdown_connections 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.573+0000 7f29dc255740 1 --2- 172.21.15.62:0/820873648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 0x56528202ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.573+0000 7f29dc255740 1 --2- 172.21.15.62:0/820873648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282029140 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.573+0000 7f29dc255740 1 --2- 172.21.15.62:0/820873648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201e4a0 0x56528201e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.573+0000 7f29dc255740 1 -- 172.21.15.62:0/820873648 >> 172.21.15.62:0/820873648 conn(0x56528201aa30 msgr2=0x56528201ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.574+0000 7f29dc255740 1 -- 172.21.15.62:0/820873648 shutdown_connections 2024-09-15T23:20:33.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.574+0000 7f29dc255740 1 -- 172.21.15.62:0/820873648 wait complete. 2024-09-15T23:20:33.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.574+0000 7f29dc255740 1 -- start start 2024-09-15T23:20:33.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.574+0000 7f29dc255740 1 Processor -- start 2024-09-15T23:20:33.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29dc255740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x565282029030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29dc255740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 0x5652820262e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29d37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x565282029030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29d37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x565282029030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50080/0 (socket says 172.21.15.62:50080) 2024-09-15T23:20:33.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29d37fe640 1 -- 172.21.15.62:0/1551340192 learned_addr learned my addr 172.21.15.62:0/1551340192 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:33.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29dc255740 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565282029680 0x565282026880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56528202c250 con 0x56528201e4a0 2024-09-15T23:20:33.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565281f3f050 con 0x565282029680 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 0x5652820262e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.575+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56528202c3d0 con 0x56528201edc0 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.576+0000 7f29d3fff640 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565282029680 0x565282026880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.576+0000 7f29dc255740 10 client.0 _ll_register_callbacks cb 0x565281f7f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.576+0000 7f29d2ffd640 1 -- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565282029680 msgr2=0x565282026880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.576+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565282029680 0x565282026880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.576+0000 7f29d2ffd640 1 -- 172.21.15.62:0/1551340192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 msgr2=0x565282029030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.576+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x565282029030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.576+0000 7f29d2ffd640 1 -- 172.21.15.62:0/1551340192 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565281cf53c0 con 0x56528201edc0 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.576+0000 7f29d3fff640 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565282029680 0x565282026880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.576+0000 7f29d37fe640 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x565282029030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:33.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.577+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 0x5652820262e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f29c4098800 tx=0x7f29c409cb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31efe5c831ee7a55 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.577+0000 7f29b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.577+0000 7f29b2ffd640 1 -- 172.21.15.62:0/1551340192 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c40a5010 con 0x56528201edc0 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.577+0000 7f29b2ffd640 1 -- 172.21.15.62:0/1551340192 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f29c4018b30 con 0x56528201edc0 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.577+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29bc030880 con 0x56528201edc0 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.577+0000 7f29b2ffd640 1 -- 172.21.15.62:0/1551340192 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f29c4018d30 con 0x56528201edc0 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.577+0000 7f29dc255740 10 client.25909 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.577+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x565282037ca0 con 0x56528201edc0 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29b2ffd640 1 -- 172.21.15.62:0/1551340192 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29bc030880 con 0x56528201edc0 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29b2ffd640 1 -- 172.21.15.62:0/1551340192 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f29c4098280 con 0x56528201edc0 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29dc255740 10 client.25909 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29dc255740 10 client.25909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29dc255740 20 client.25909 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29dc255740 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565282039100 0x56528203b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29dc255740 4 client.25909 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:33.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565281f91850 con 0x565282039100 2024-09-15T23:20:33.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29d37fe640 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565282039100 0x56528203b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29d37fe640 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565282039100 0x56528203b580 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49fedeedcfc14ba6 server_cookie=33f6741a22a7961d in_seq=0 out_seq=0 2024-09-15T23:20:33.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.578+0000 7f29b2ffd640 10 client.25909 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:33.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.579+0000 7f29b2ffd640 1 -- 172.21.15.62:0/1551340192 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565281f91850 con 0x565282039100 2024-09-15T23:20:33.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29b2ffd640 10 client.25909 handle_command_reply: tid=0 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 client.25909 shutdown 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 msgr2=0x5652820262e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 0x5652820262e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f29c4098800 tx=0x7f29c409cb20 comp rx=0 tx=0).stop 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 shutdown_connections 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565282039100 0x56528203b580 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29b2ffd640 0 client.25909 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565282029680 0x565282026880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 0x5652820262e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 --2- 172.21.15.62:0/1551340192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x565282029030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 >> 172.21.15.62:0/1551340192 conn(0x56528201aa30 msgr2=0x565282019b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 shutdown_connections 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 1 -- 172.21.15.62:0/1551340192 wait complete. 2024-09-15T23:20:33.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.580+0000 7f29dc255740 20 client.25909 trim_cache size 0 max 16384 2024-09-15T23:20:33.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.605+0000 7f29dc255740 1 Processor -- start 2024-09-15T23:20:33.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.605+0000 7f29dc255740 1 -- start start 2024-09-15T23:20:33.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.605+0000 7f29dc255740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x5652820354b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.605+0000 7f29dc255740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282025ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.605+0000 7f29dc255740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565282029680 0x565282040e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.605+0000 7f29dc255740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f29d41b37f0 con 0x56528201e4a0 2024-09-15T23:20:33.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.605+0000 7f29dc255740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f29d41b10c0 con 0x56528201edc0 2024-09-15T23:20:33.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.605+0000 7f29dc255740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f29d41aef10 con 0x565282029680 2024-09-15T23:20:33.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.605+0000 7f29d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282025ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282025ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46188/0 (socket says 172.21.15.62:46188) 2024-09-15T23:20:33.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d2ffd640 1 -- 172.21.15.62:0/1835367057 learned_addr learned my addr 172.21.15.62:0/1835367057 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:33.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d3fff640 1 --2- 172.21.15.62:0/1835367057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565282029680 0x565282040e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d37fe640 1 --2- 172.21.15.62:0/1835367057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x5652820354b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d3fff640 1 -- 172.21.15.62:0/1835367057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 msgr2=0x565282025ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:33.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d3fff640 1 --2- 172.21.15.62:0/1835367057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282025ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d3fff640 1 -- 172.21.15.62:0/1835367057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 msgr2=0x5652820354b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:33.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d3fff640 1 --2- 172.21.15.62:0/1835367057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x5652820354b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d3fff640 1 -- 172.21.15.62:0/1835367057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56528201e220 con 0x565282029680 2024-09-15T23:20:33.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.606+0000 7f29d3fff640 1 --2- 172.21.15.62:0/1835367057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565282029680 0x565282040e50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009f90 tx=0x7f29cc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7a8c31ff2a5976c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:33.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29c97fa640 1 -- 172.21.15.62:0/1835367057 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc003970 con 0x565282029680 2024-09-15T23:20:33.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29c97fa640 1 -- 172.21.15.62:0/1835367057 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29cc00e950 con 0x565282029680 2024-09-15T23:20:33.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29c97fa640 1 -- 172.21.15.62:0/1835367057 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc004450 con 0x565282029680 2024-09-15T23:20:33.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29dc255740 1 -- 172.21.15.62:0/1835367057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565282029680 msgr2=0x565282040e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:33.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29dc255740 1 --2- 172.21.15.62:0/1835367057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565282029680 0x565282040e50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009f90 tx=0x7f29cc004a40 comp rx=0 tx=0).stop 2024-09-15T23:20:33.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29dc255740 1 -- 172.21.15.62:0/1835367057 shutdown_connections 2024-09-15T23:20:33.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29dc255740 1 --2- 172.21.15.62:0/1835367057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565282029680 0x565282040e50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29dc255740 1 --2- 172.21.15.62:0/1835367057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201edc0 0x565282025ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29dc255740 1 --2- 172.21.15.62:0/1835367057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56528201e4a0 0x5652820354b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.607+0000 7f29dc255740 1 -- 172.21.15.62:0/1835367057 >> 172.21.15.62:0/1835367057 conn(0x56528201aa30 msgr2=0x5652820253f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:33.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.608+0000 7f29dc255740 1 -- 172.21.15.62:0/1835367057 shutdown_connections 2024-09-15T23:20:33.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.608+0000 7f29dc255740 1 -- 172.21.15.62:0/1835367057 wait complete. 2024-09-15T23:20:33.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.608+0000 7f29dc255740 1 -- start start 2024-09-15T23:20:33.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.608+0000 7f29dc255740 1 Processor -- start 2024-09-15T23:20:33.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29dc255740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201e4a0 0x565282031800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29dc255740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 0x565282031d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29dc255740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 0x565282022b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29dc255740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f29d41af580 con 0x565282029680 2024-09-15T23:20:33.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29dc255740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f29d41b09f0 con 0x56528201e4a0 2024-09-15T23:20:33.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29dc255740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f29d41afc50 con 0x56528201edc0 2024-09-15T23:20:33.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201e4a0 0x565282031800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29dc255740 10 client.0 _ll_register_callbacks cb 0x7f29c4009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:33.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29d37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201e4a0 0x565282031800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46202/0 (socket says 172.21.15.62:46202) 2024-09-15T23:20:33.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29d37fe640 1 -- 172.21.15.62:0/4164881919 learned_addr learned my addr 172.21.15.62:0/4164881919 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:33.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.609+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 0x565282031d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.610+0000 7f29d3fff640 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 0x565282022b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.611+0000 7f29d37fe640 1 -- 172.21.15.62:0/4164881919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 msgr2=0x565282031d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:33.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.611+0000 7f29d37fe640 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 0x565282031d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.611+0000 7f29d37fe640 1 -- 172.21.15.62:0/4164881919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 msgr2=0x565282022b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:33.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.611+0000 7f29d37fe640 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 0x565282022b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.611+0000 7f29d37fe640 1 -- 172.21.15.62:0/4164881919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565282023330 con 0x56528201e4a0 2024-09-15T23:20:33.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.611+0000 7f29d3fff640 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 0x565282022b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:20:33.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.611+0000 7f29d37fe640 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201e4a0 0x565282031800 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f29bc005720 tx=0x7f29bc005610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0283389de01fa4f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:33.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.611+0000 7f29b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:33.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.612+0000 7f29b2ffd640 1 -- 172.21.15.62:0/4164881919 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0044c0 con 0x56528201e4a0 2024-09-15T23:20:33.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.612+0000 7f29b2ffd640 1 -- 172.21.15.62:0/4164881919 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f29bc0305b0 con 0x56528201e4a0 2024-09-15T23:20:33.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.612+0000 7f29dc255740 1 -- 172.21.15.62:0/4164881919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29cc00e950 con 0x56528201e4a0 2024-09-15T23:20:33.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.619+0000 7f29b2ffd640 1 -- 172.21.15.62:0/4164881919 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f29bc005850 con 0x56528201e4a0 2024-09-15T23:20:33.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.620+0000 7f29dc255740 10 client.26084 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:33.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.620+0000 7f29dc255740 1 -- 172.21.15.62:0/4164881919 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5652820ac0c0 con 0x56528201e4a0 2024-09-15T23:20:33.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.622+0000 7f29b2ffd640 1 -- 172.21.15.62:0/4164881919 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29cc00e950 con 0x56528201e4a0 2024-09-15T23:20:33.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.622+0000 7f29b2ffd640 1 -- 172.21.15.62:0/4164881919 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f29bc0524c0 con 0x56528201e4a0 2024-09-15T23:20:33.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.622+0000 7f29dc255740 10 client.26084 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:33.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.622+0000 7f29dc255740 10 client.26084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:33.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.622+0000 7f29dc255740 20 client.26084 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:33.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.622+0000 7f29dc255740 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565282039100 0x5652820ae460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:33.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.622+0000 7f29dc255740 4 client.26084 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:20:33.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.622+0000 7f29dc255740 1 -- 172.21.15.62:0/4164881919 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x565281f8bd60 con 0x565282039100 2024-09-15T23:20:33.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.628+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565282039100 0x5652820ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:33.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.634+0000 7f29d2ffd640 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565282039100 0x5652820ae460 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2b018d5a3a028e0 server_cookie=287af67b9fad1f05 in_seq=0 out_seq=0 2024-09-15T23:20:33.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.635+0000 7f29b2ffd640 10 client.26084 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:33.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.638+0000 7f29b2ffd640 1 -- 172.21.15.62:0/4164881919 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x565281f8bd60 con 0x565282039100 2024-09-15T23:20:33.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.638+0000 7f29b2ffd640 10 client.26084 handle_command_reply: tid=0 2024-09-15T23:20:33.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.638+0000 7f29dc255740 1 client.26084 shutdown 2024-09-15T23:20:33.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.643+0000 7f29dc255740 1 -- 172.21.15.62:0/4164881919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201e4a0 msgr2=0x565282031800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:33.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.643+0000 7f29dc255740 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201e4a0 0x565282031800 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f29bc005720 tx=0x7f29bc005610 comp rx=0 tx=0).stop 2024-09-15T23:20:33.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29dc255740 1 -- 172.21.15.62:0/4164881919 shutdown_connections 2024-09-15T23:20:33.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29dc255740 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565282039100 0x5652820ae460 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29dc255740 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565282029680 0x565282022b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29dc255740 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56528201edc0 0x565282031d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29dc255740 1 --2- 172.21.15.62:0/4164881919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56528201e4a0 0x565282031800 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29b2ffd640 0 client.26084 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:33.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29dc255740 1 -- 172.21.15.62:0/4164881919 >> 172.21.15.62:0/4164881919 conn(0x56528201aa30 msgr2=0x56528201fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:33.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29dc255740 1 -- 172.21.15.62:0/4164881919 shutdown_connections 2024-09-15T23:20:33.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29dc255740 1 -- 172.21.15.62:0/4164881919 wait complete. 2024-09-15T23:20:33.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.646+0000 7f29dc255740 20 client.26084 trim_cache size 0 max 16384 2024-09-15T23:20:33.649 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:20:33.650 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:20:33.650 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:20:33.650 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:20:33.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 -- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f29c00a3240 msgr2=0x7f29c00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:33.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f29c00a3240 0x7f29c00a5700 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009fd0 tx=0x7f29cc012040 comp rx=0 tx=0).stop 2024-09-15T23:20:33.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 -- 172.21.15.62:0/470409700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4102570 msgr2=0x7f29d41979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:33.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4102570 0x7f29d41979e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f29c400e9f0 tx=0x7f29c400eeb0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 -- 172.21.15.62:0/470409700 shutdown_connections 2024-09-15T23:20:33.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f29c00a3240 0x7f29c00a5700 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29d4102ef0 0x7f29d419bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29d4102570 0x7f29d41979e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 --2- 172.21.15.62:0/470409700 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f29d4100e20 0x7f29d41974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:33.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.648+0000 7f29d9e7c640 1 -- 172.21.15.62:0/470409700 >> 172.21.15.62:0/470409700 conn(0x7f29d40fc7d0 msgr2=0x7f29d40fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:33.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.649+0000 7f29d9e7c640 1 -- 172.21.15.62:0/470409700 shutdown_connections 2024-09-15T23:20:33.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:33.649+0000 7f29d9e7c640 1 -- 172.21.15.62:0/470409700 wait complete. 2024-09-15T23:20:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:33 smithi067 ceph-mon[35665]: pgmap v498: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 9.7 MiB/s wr, 775 op/s 2024-09-15T23:20:34.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 255.0 s, 1333.0 tps, lat 76.450 ms stddev 134.469 2024-09-15T23:20:34.595 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:20:34.596 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:20:35.015 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:35.402 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:35 smithi062 ceph-mon[30949]: pgmap v499: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 MiB/s wr, 968 op/s 2024-09-15T23:20:35.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.509+0000 7fc7cd78a640 1 -- 172.21.15.62:0/494171463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7c8105ec0 msgr2=0x7fc7c8106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:35.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.509+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/494171463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7c8105ec0 0x7fc7c8106340 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009cd0 tx=0x7fc7c002a910 comp rx=0 tx=0).stop 2024-09-15T23:20:35.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.509+0000 7fc7cd78a640 1 -- 172.21.15.62:0/494171463 shutdown_connections 2024-09-15T23:20:35.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.509+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/494171463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7c8105ec0 0x7fc7c8106340 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.509+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/494171463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7c81011e0 0x7fc7c8105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.509+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/494171463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c81008c0 0x7fc7c8100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.509+0000 7fc7cd78a640 1 -- 172.21.15.62:0/494171463 >> 172.21.15.62:0/494171463 conn(0x7fc7c80fc7d0 msgr2=0x7fc7c80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.510+0000 7fc7cd78a640 1 -- 172.21.15.62:0/494171463 shutdown_connections 2024-09-15T23:20:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.510+0000 7fc7cd78a640 1 -- 172.21.15.62:0/494171463 wait complete. 2024-09-15T23:20:35.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.510+0000 7fc7cd78a640 1 Processor -- start 2024-09-15T23:20:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.510+0000 7fc7cd78a640 1 -- start start 2024-09-15T23:20:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7cd78a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7c81008c0 0x7fc7c81a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7cd78a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7c81011e0 0x7fc7c81a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7cd78a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c8105ec0 0x7fc7c819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7cd78a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c81186b0 con 0x7fc7c8105ec0 2024-09-15T23:20:35.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7cd78a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c8118530 con 0x7fc7c81011e0 2024-09-15T23:20:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7cd78a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc7c8118830 con 0x7fc7c81008c0 2024-09-15T23:20:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c8105ec0 0x7fc7c819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c67fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7c81011e0 0x7fc7c81a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c8105ec0 0x7fc7c819e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50098/0 (socket says 172.21.15.62:50098) 2024-09-15T23:20:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c77fe640 1 -- 172.21.15.62:0/2117577817 learned_addr learned my addr 172.21.15.62:0/2117577817 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c6ffd640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7c81008c0 0x7fc7c81a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c77fe640 1 -- 172.21.15.62:0/2117577817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7c81008c0 msgr2=0x7fc7c81a4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c77fe640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7c81008c0 0x7fc7c81a4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c77fe640 1 -- 172.21.15.62:0/2117577817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7c81011e0 msgr2=0x7fc7c81a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c77fe640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7c81011e0 0x7fc7c81a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c77fe640 1 -- 172.21.15.62:0/2117577817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7c819f0f0 con 0x7fc7c8105ec0 2024-09-15T23:20:35.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.511+0000 7fc7c67fc640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7c81011e0 0x7fc7c81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:35.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.513+0000 7fc7c77fe640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c8105ec0 0x7fc7c819e9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0005cc0 tx=0x7fc7c0031460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:35.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.514+0000 7fc7affff640 1 -- 172.21.15.62:0/2117577817 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0031850 con 0x7fc7c8105ec0 2024-09-15T23:20:35.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.514+0000 7fc7affff640 1 -- 172.21.15.62:0/2117577817 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7c0004520 con 0x7fc7c8105ec0 2024-09-15T23:20:35.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.514+0000 7fc7affff640 1 -- 172.21.15.62:0/2117577817 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c002fe20 con 0x7fc7c8105ec0 2024-09-15T23:20:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.514+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7c819f380 con 0x7fc7c8105ec0 2024-09-15T23:20:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.514+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7c81ab6d0 con 0x7fc7c8105ec0 2024-09-15T23:20:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.516+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7c806a850 con 0x7fc7c8105ec0 2024-09-15T23:20:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.517+0000 7fc7affff640 1 -- 172.21.15.62:0/2117577817 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc7c0004040 con 0x7fc7c8105ec0 2024-09-15T23:20:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.518+0000 7fc7affff640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7b00a32c0 0x7fc7b00a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.518+0000 7fc7c6ffd640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7b00a32c0 0x7fc7b00a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.518+0000 7fc7affff640 1 -- 172.21.15.62:0/2117577817 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc7c00f13b0 con 0x7fc7c8105ec0 2024-09-15T23:20:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.519+0000 7fc7c6ffd640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7b00a32c0 0x7fc7b00a5780 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc7c819fb00 tx=0x7fc7b80090e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:35.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.524+0000 7fc7affff640 1 -- 172.21.15.62:0/2117577817 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc7c00098b0 con 0x7fc7c8105ec0 2024-09-15T23:20:35.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:35 smithi148 ceph-mon[35610]: pgmap v499: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 MiB/s wr, 968 op/s 2024-09-15T23:20:35.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:35 smithi067 ceph-mon[35665]: pgmap v499: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 MiB/s wr, 968 op/s 2024-09-15T23:20:35.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.909+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7c81ab8a0 con 0x7fc7c8105ec0 2024-09-15T23:20:35.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.910+0000 7fc7affff640 1 -- 172.21.15.62:0/2117577817 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11903 (secure 0 0 0) 0x7fc7c00bdcf0 con 0x7fc7c8105ec0 2024-09-15T23:20:35.911 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:20:35.911 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":31,"btime":"2024-09-15T23:20:13:209709+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:20:12.378892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:20:35.911 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 31 2024-09-15T23:20:35.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7b00a32c0 msgr2=0x7fc7b00a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:35.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7b00a32c0 0x7fc7b00a5780 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc7c819fb00 tx=0x7fc7b80090e0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c8105ec0 msgr2=0x7fc7c819e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:35.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c8105ec0 0x7fc7c819e9c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0005cc0 tx=0x7fc7c0031460 comp rx=0 tx=0).stop 2024-09-15T23:20:35.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 shutdown_connections 2024-09-15T23:20:35.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7b00a32c0 0x7fc7b00a5780 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7c8105ec0 0x7fc7c819e9c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7c81011e0 0x7fc7c81a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 --2- 172.21.15.62:0/2117577817 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7c81008c0 0x7fc7c81a4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:35.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.912+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 >> 172.21.15.62:0/2117577817 conn(0x7fc7c80fc7d0 msgr2=0x7fc7c8103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:35.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.914+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 shutdown_connections 2024-09-15T23:20:35.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:35.914+0000 7fc7cd78a640 1 -- 172.21.15.62:0/2117577817 wait complete. 2024-09-15T23:20:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:36 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2117577817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:36.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:36 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2117577817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:36 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2117577817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:37.190 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:20:37.483 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:37 smithi062 ceph-mon[30949]: pgmap v500: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 968 op/s 2024-09-15T23:20:37.605 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:37.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:37 smithi067 ceph-mon[35665]: pgmap v500: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 968 op/s 2024-09-15T23:20:37.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:37 smithi148 ceph-mon[35610]: pgmap v500: 129 pgs: 129 active+clean; 1.1 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 968 op/s 2024-09-15T23:20:38.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.269+0000 7f6c7f007640 1 -- 172.21.15.62:0/195622700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c78103740 msgr2=0x7f6c78107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:38.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.269+0000 7f6c7f007640 1 --2- 172.21.15.62:0/195622700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c78103740 0x7f6c78107b80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009930 tx=0x7f6c6402a420 comp rx=0 tx=0).stop 2024-09-15T23:20:38.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.269+0000 7f6c7f007640 1 -- 172.21.15.62:0/195622700 shutdown_connections 2024-09-15T23:20:38.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.269+0000 7f6c7f007640 1 --2- 172.21.15.62:0/195622700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c78108290 0x7f6c7810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.269+0000 7f6c7f007640 1 --2- 172.21.15.62:0/195622700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c78103740 0x7f6c78107b80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.269+0000 7f6c7f007640 1 --2- 172.21.15.62:0/195622700 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c78102d90 0x7f6c78103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.269+0000 7f6c7f007640 1 -- 172.21.15.62:0/195622700 >> 172.21.15.62:0/195622700 conn(0x7f6c780fc7d0 msgr2=0x7f6c780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:38.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.269+0000 7f6c7f007640 1 -- 172.21.15.62:0/195622700 shutdown_connections 2024-09-15T23:20:38.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.269+0000 7f6c7f007640 1 -- 172.21.15.62:0/195622700 wait complete. 2024-09-15T23:20:38.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.270+0000 7f6c7f007640 1 Processor -- start 2024-09-15T23:20:38.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7f007640 1 -- start start 2024-09-15T23:20:38.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7f007640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c78102d90 0x7f6c781009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7f007640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c78103740 0x7f6c78100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7f007640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c78108290 0x7f6c78101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7f007640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78112090 con 0x7f6c78103740 2024-09-15T23:20:38.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7f007640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78111f10 con 0x7f6c78102d90 2024-09-15T23:20:38.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7f007640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6c78112210 con 0x7f6c78108290 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7cda3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c78102d90 0x7f6c781009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c78103740 0x7f6c78100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7cda3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c78102d90 0x7f6c781009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46256/0 (socket says 172.21.15.62:46256) 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3139007835 learned_addr learned my addr 172.21.15.62:0/3139007835 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7d5a4640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c78108290 0x7f6c78101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3139007835 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c78108290 msgr2=0x7f6c78101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c78108290 0x7f6c78101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3139007835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c78103740 msgr2=0x7f6c78100f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c78103740 0x7f6c78100f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.271+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3139007835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c781a6b10 con 0x7f6c78102d90 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.272+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c78102d90 0x7f6c781009f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00e9f0 tx=0x7f6c6c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:38.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.272+0000 7f6c75ffb640 1 -- 172.21.15.62:0/3139007835 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c00cd80 con 0x7f6c78102d90 2024-09-15T23:20:38.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.272+0000 7f6c75ffb640 1 -- 172.21.15.62:0/3139007835 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6c010910 con 0x7f6c78102d90 2024-09-15T23:20:38.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.272+0000 7f6c75ffb640 1 -- 172.21.15.62:0/3139007835 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c018990 con 0x7f6c78102d90 2024-09-15T23:20:38.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.272+0000 7f6c7f007640 1 -- 172.21.15.62:0/3139007835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c781a6e00 con 0x7f6c78102d90 2024-09-15T23:20:38.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.272+0000 7f6c7f007640 1 -- 172.21.15.62:0/3139007835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c781a7200 con 0x7f6c78102d90 2024-09-15T23:20:38.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.276+0000 7f6c75ffb640 1 -- 172.21.15.62:0/3139007835 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f6c6c010ac0 con 0x7f6c78102d90 2024-09-15T23:20:38.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.277+0000 7f6c813e0740 1 Processor -- start 2024-09-15T23:20:38.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.277+0000 7f6c813e0740 1 -- start start 2024-09-15T23:20:38.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d053308a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05331220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d0533df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c813e0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556d0528e2f0 con 0x556d0533bb80 2024-09-15T23:20:38.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c813e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556d05235cf0 con 0x556d05330de0 2024-09-15T23:20:38.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c813e0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556d0528a2b0 con 0x556d05332350 2024-09-15T23:20:38.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c7d5a4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d053308a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c7d5a4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d053308a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:46800/0 (socket says 172.21.15.62:46800) 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c7d5a4640 1 -- 172.21.15.62:0/1537573836 learned_addr learned my addr 172.21.15.62:0/1537573836 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c75ffb640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6c680a32c0 0x7f6c680a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c77fff640 1 --2- 172.21.15.62:0/1537573836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d0533df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c7d5a4640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6c680a32c0 0x7f6c680a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c75ffb640 1 -- 172.21.15.62:0/3139007835 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6c6c0d0090 con 0x7f6c78102d90 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c7d5a4640 1 -- 172.21.15.62:0/1537573836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 msgr2=0x556d05331220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c7d5a4640 1 --2- 172.21.15.62:0/1537573836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05331220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c7d5a4640 1 -- 172.21.15.62:0/1537573836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 msgr2=0x556d0533df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c7d5a4640 1 --2- 172.21.15.62:0/1537573836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d0533df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.278+0000 7f6c7d5a4640 1 -- 172.21.15.62:0/1537573836 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d0521a3a0 con 0x556d05332350 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.279+0000 7f6c7d5a4640 1 --2- 172.21.15.62:0/1537573836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d053308a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c700129f0 tx=0x7f6c70012eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=148f75d65ac52a3f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.279+0000 7f6c62ffd640 1 -- 172.21.15.62:0/1537573836 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c70023070 con 0x556d05332350 2024-09-15T23:20:38.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.279+0000 7f6c62ffd640 1 -- 172.21.15.62:0/1537573836 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c70004710 con 0x556d05332350 2024-09-15T23:20:38.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.279+0000 7f6c62ffd640 1 -- 172.21.15.62:0/1537573836 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c7001fa50 con 0x556d05332350 2024-09-15T23:20:38.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c813e0740 1 -- 172.21.15.62:0/1537573836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 msgr2=0x556d053308a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:38.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c813e0740 1 --2- 172.21.15.62:0/1537573836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d053308a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6c700129f0 tx=0x7f6c70012eb0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c7d5a4640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6c680a32c0 0x7f6c680a5780 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6c70022000 tx=0x7f6c70021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:38.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c813e0740 1 -- 172.21.15.62:0/1537573836 shutdown_connections 2024-09-15T23:20:38.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c813e0740 1 --2- 172.21.15.62:0/1537573836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d0533df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c813e0740 1 --2- 172.21.15.62:0/1537573836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05331220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c813e0740 1 --2- 172.21.15.62:0/1537573836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d053308a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c813e0740 1 -- 172.21.15.62:0/1537573836 >> 172.21.15.62:0/1537573836 conn(0x556d052f1310 msgr2=0x556d0532d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c813e0740 1 -- 172.21.15.62:0/1537573836 shutdown_connections 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.280+0000 7f6c813e0740 1 -- 172.21.15.62:0/1537573836 wait complete. 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.281+0000 7f6c813e0740 1 -- start start 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.281+0000 7f6c813e0740 1 Processor -- start 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.281+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d0533b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.281+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d05338850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.281+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d05338e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.281+0000 7f6c813e0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556d0533e880 con 0x556d0533bb80 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.281+0000 7f6c813e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556d05249060 con 0x556d05330de0 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.281+0000 7f6c813e0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556d0533ea00 con 0x556d05332350 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7cda3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d0533b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c813e0740 10 client.0 _ll_register_callbacks cb 0x556d04f9c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7cda3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d0533b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46280/0 (socket says 172.21.15.62:46280) 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3183896227 learned_addr learned my addr 172.21.15.62:0/3183896227 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c77fff640 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d05338850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7d5a4640 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d05338e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3183896227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 msgr2=0x556d05338850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d05338850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3183896227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 msgr2=0x556d05338e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:38.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d05338e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3183896227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d04fff3c0 con 0x556d05330de0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d0533b5a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c09c260 tx=0x7f6c6c09bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2efd555537e5dd11 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c57fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c57fff640 1 -- 172.21.15.62:0/3183896227 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c018d30 con 0x556d05330de0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c57fff640 1 -- 172.21.15.62:0/3183896227 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6c6c09b720 con 0x556d05330de0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.282+0000 7f6c813e0740 1 -- 172.21.15.62:0/3183896227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c70004710 con 0x556d05330de0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.283+0000 7f6c57fff640 1 -- 172.21.15.62:0/3183896227 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c6c0a4f40 con 0x556d05330de0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.283+0000 7f6c813e0740 10 client.26099 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.283+0000 7f6c813e0740 1 -- 172.21.15.62:0/3183896227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556d0534a3a0 con 0x556d05330de0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.283+0000 7f6c57fff640 1 -- 172.21.15.62:0/3183896227 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c70004710 con 0x556d05330de0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.283+0000 7f6c57fff640 1 -- 172.21.15.62:0/3183896227 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6c6c0bc070 con 0x556d05330de0 2024-09-15T23:20:38.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.283+0000 7f6c813e0740 10 client.26099 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.284+0000 7f6c813e0740 10 client.26099 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.284+0000 7f6c813e0740 20 client.26099 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.284+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x556d0534b860 0x556d0534dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.284+0000 7f6c813e0740 4 client.26099 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.284+0000 7f6c813e0740 1 -- 172.21.15.62:0/3183896227 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d0529b860 con 0x556d0534b860 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.284+0000 7f6c77fff640 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x556d0534b860 0x556d0534dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.285+0000 7f6c77fff640 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x556d0534b860 0x556d0534dc50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c28df6cce261d84 server_cookie=a0405a98c0f8eb26 in_seq=0 out_seq=0 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c57fff640 10 client.26099 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c57fff640 1 -- 172.21.15.62:0/3183896227 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556d0529b860 con 0x556d0534b860 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c57fff640 10 client.26099 handle_command_reply: tid=0 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c813e0740 1 client.26099 shutdown 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c813e0740 1 -- 172.21.15.62:0/3183896227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 msgr2=0x556d0533b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d0533b5a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c09c260 tx=0x7f6c6c09bcf0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c813e0740 1 -- 172.21.15.62:0/3183896227 shutdown_connections 2024-09-15T23:20:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x556d0534b860 0x556d0534dc50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c57fff640 0 client.26099 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:20:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d05338e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d05338850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3183896227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d0533b5a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.286+0000 7f6c813e0740 1 -- 172.21.15.62:0/3183896227 >> 172.21.15.62:0/3183896227 conn(0x556d052f1310 msgr2=0x556d05333170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.287+0000 7f6c813e0740 1 -- 172.21.15.62:0/3183896227 shutdown_connections 2024-09-15T23:20:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.287+0000 7f6c813e0740 1 -- 172.21.15.62:0/3183896227 wait complete. 2024-09-15T23:20:38.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.287+0000 7f6c813e0740 20 client.26099 trim_cache size 0 max 16384 2024-09-15T23:20:38.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.310+0000 7f6c813e0740 1 Processor -- start 2024-09-15T23:20:38.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.310+0000 7f6c813e0740 1 -- start start 2024-09-15T23:20:38.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.310+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05337120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.310+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d05332350 0x556d05333930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.310+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d0533bb80 0x556d05333e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.310+0000 7f6c813e0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781af3d0 con 0x556d05332350 2024-09-15T23:20:38.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.310+0000 7f6c813e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781acca0 con 0x556d05330de0 2024-09-15T23:20:38.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.311+0000 7f6c7cda3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05337120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.311+0000 7f6c7cda3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05337120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46284/0 (socket says 172.21.15.62:46284) 2024-09-15T23:20:38.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.311+0000 7f6c7cda3640 1 -- 172.21.15.62:0/844053581 learned_addr learned my addr 172.21.15.62:0/844053581 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:38.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.310+0000 7f6c813e0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781aaaf0 con 0x556d0533bb80 2024-09-15T23:20:38.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.311+0000 7f6c7d5a4640 1 --2- 172.21.15.62:0/844053581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d0533bb80 0x556d05333e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.311+0000 7f6c7cda3640 1 -- 172.21.15.62:0/844053581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d0533bb80 msgr2=0x556d05333e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:38.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.311+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/844053581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d0533bb80 0x556d05333e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.311+0000 7f6c7cda3640 1 -- 172.21.15.62:0/844053581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d05332350 msgr2=0x556d05333930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:38.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.311+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/844053581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d05332350 0x556d05333930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.312+0000 7f6c7cda3640 1 -- 172.21.15.62:0/844053581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d05337800 con 0x556d05330de0 2024-09-15T23:20:38.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.312+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/844053581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05337120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0b7040 tx=0x7f6c6c09a7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a12efb8aa24f480f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:38.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.317+0000 7f6c627fc640 1 -- 172.21.15.62:0/844053581 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c0b50f0 con 0x556d05330de0 2024-09-15T23:20:38.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.317+0000 7f6c627fc640 1 -- 172.21.15.62:0/844053581 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6c0b5290 con 0x556d05330de0 2024-09-15T23:20:38.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.317+0000 7f6c627fc640 1 -- 172.21.15.62:0/844053581 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c018d90 con 0x556d05330de0 2024-09-15T23:20:38.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.318+0000 7f6c813e0740 1 -- 172.21.15.62:0/844053581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 msgr2=0x556d05337120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:38.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.318+0000 7f6c813e0740 1 --2- 172.21.15.62:0/844053581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05337120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0b7040 tx=0x7f6c6c09a7e0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.320+0000 7f6c813e0740 1 -- 172.21.15.62:0/844053581 shutdown_connections 2024-09-15T23:20:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.320+0000 7f6c813e0740 1 --2- 172.21.15.62:0/844053581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d0533bb80 0x556d05333e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.320+0000 7f6c813e0740 1 --2- 172.21.15.62:0/844053581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d05332350 0x556d05333930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.320+0000 7f6c813e0740 1 --2- 172.21.15.62:0/844053581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05337120 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.320+0000 7f6c813e0740 1 -- 172.21.15.62:0/844053581 >> 172.21.15.62:0/844053581 conn(0x556d052f1310 msgr2=0x556d05323d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.320+0000 7f6c813e0740 1 -- 172.21.15.62:0/844053581 shutdown_connections 2024-09-15T23:20:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.321+0000 7f6c813e0740 1 -- 172.21.15.62:0/844053581 wait complete. 2024-09-15T23:20:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.321+0000 7f6c813e0740 1 -- start start 2024-09-15T23:20:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.321+0000 7f6c813e0740 1 Processor -- start 2024-09-15T23:20:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05340670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d05340bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c813e0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d0532c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c7cda3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05340670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c7cda3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05340670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46286/0 (socket says 172.21.15.62:46286) 2024-09-15T23:20:38.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d05340bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d05340bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:46828/0 (socket says 172.21.15.62:46828) 2024-09-15T23:20:38.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c7d5a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d0532c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c7d5a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d0532c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50162/0 (socket says 172.21.15.62:50162) 2024-09-15T23:20:38.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3146757947 learned_addr learned my addr 172.21.15.62:0/3146757947 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:38.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781ab160 con 0x556d0533bb80 2024-09-15T23:20:38.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781ac5d0 con 0x556d05330de0 2024-09-15T23:20:38.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.322+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6c781ab830 con 0x556d05332350 2024-09-15T23:20:38.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.323+0000 7f6c813e0740 10 client.0 _ll_register_callbacks cb 0x7f6c6c09c130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.323+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3146757947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 msgr2=0x556d05340bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.323+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d05340bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.323+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3146757947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 msgr2=0x556d0532c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.323+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d0532c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.323+0000 7f6c7cda3640 1 -- 172.21.15.62:0/3146757947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d053246c0 con 0x556d05330de0 2024-09-15T23:20:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.323+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c6c0b5290 con 0x556d05330de0 2024-09-15T23:20:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.323+0000 7f6c7cda3640 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05340670 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c09c490 tx=0x7f6c6c0a0330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8dc90e78ded93f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.324+0000 7f6c57fff640 10 client.26105 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.324+0000 7f6c57fff640 1 -- 172.21.15.62:0/3146757947 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c0b0050 con 0x556d05330de0 2024-09-15T23:20:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.324+0000 7f6c57fff640 1 -- 172.21.15.62:0/3146757947 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6c6c09b720 con 0x556d05330de0 2024-09-15T23:20:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.324+0000 7f6c57fff640 1 -- 172.21.15.62:0/3146757947 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6c6c0b50f0 con 0x556d05330de0 2024-09-15T23:20:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.324+0000 7f6c813e0740 10 client.26105 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.324+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=31}) -- 0x556d053246c0 con 0x556d05330de0 2024-09-15T23:20:38.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.330+0000 7f6c57fff640 1 -- 172.21.15.62:0/3146757947 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6c6c010430 con 0x556d05330de0 2024-09-15T23:20:38.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.330+0000 7f6c813e0740 10 client.26105 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.330+0000 7f6c813e0740 10 client.26105 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.330+0000 7f6c813e0740 20 client.26105 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.330+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x556d053ae720 0x556d053b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.330+0000 7f6c813e0740 4 client.26105 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.330+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556d05295d70 con 0x556d053ae720 2024-09-15T23:20:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.331+0000 7f6c77fff640 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x556d053ae720 0x556d053b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.333+0000 7f6c77fff640 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x556d053ae720 0x556d053b0b30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9ff26389450291db server_cookie=4263ad04325a59da in_seq=0 out_seq=0 2024-09-15T23:20:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.334+0000 7f6c57fff640 10 client.26105 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:20:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.334+0000 7f6c57fff640 1 -- 172.21.15.62:0/3146757947 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556d05295d70 con 0x556d053ae720 2024-09-15T23:20:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.334+0000 7f6c57fff640 10 client.26105 handle_command_reply: tid=0 2024-09-15T23:20:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.335+0000 7f6c813e0740 1 client.26105 shutdown 2024-09-15T23:20:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.335+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 msgr2=0x556d05340670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.335+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05340670 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c09c490 tx=0x7f6c6c0a0330 comp rx=0 tx=0).stop 2024-09-15T23:20:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.338+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 shutdown_connections 2024-09-15T23:20:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.338+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x556d053ae720 0x556d053b0b30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.338+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556d0533bb80 0x556d0532c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.338+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556d05332350 0x556d05340bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.338+0000 7f6c813e0740 1 --2- 172.21.15.62:0/3146757947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556d05330de0 0x556d05340670 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.338+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 >> 172.21.15.62:0/3146757947 conn(0x556d052f1310 msgr2=0x556d05323d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.338+0000 7f6c57fff640 0 client.26105 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:20:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.341+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 shutdown_connections 2024-09-15T23:20:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.342+0000 7f6c813e0740 1 -- 172.21.15.62:0/3146757947 wait complete. 2024-09-15T23:20:38.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.345+0000 7f6c813e0740 20 client.26105 trim_cache size 0 max 16384 2024-09-15T23:20:38.347 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:20:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 -- 172.21.15.62:0/3139007835 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6c680a32c0 msgr2=0x7f6c680a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6c680a32c0 0x7f6c680a5780 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6c70022000 tx=0x7f6c70021000 comp rx=0 tx=0).stop 2024-09-15T23:20:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 -- 172.21.15.62:0/3139007835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c78102d90 msgr2=0x7f6c781009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c78102d90 0x7f6c781009f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00e9f0 tx=0x7f6c6c00eeb0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 -- 172.21.15.62:0/3139007835 shutdown_connections 2024-09-15T23:20:38.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6c680a32c0 0x7f6c680a5780 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6c78108290 0x7f6c78101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c78103740 0x7f6c78100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 --2- 172.21.15.62:0/3139007835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c78102d90 0x7f6c781009f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.347+0000 7f6c7f007640 1 -- 172.21.15.62:0/3139007835 >> 172.21.15.62:0/3139007835 conn(0x7f6c780fc7d0 msgr2=0x7f6c78105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.348+0000 7f6c7f007640 1 -- 172.21.15.62:0/3139007835 shutdown_connections 2024-09-15T23:20:38.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:38.348+0000 7f6c7f007640 1 -- 172.21.15.62:0/3139007835 wait complete. 2024-09-15T23:20:39.238 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:20:39.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:39 smithi062 ceph-mon[30949]: pgmap v501: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 927 op/s 2024-09-15T23:20:39.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:39 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:39.486 INFO:tasks.quiescer.fs.[cephfs]:Total duration (50.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:20:39.486 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.35 2024-09-15T23:20:39.486 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:20:39.581 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 260.0 s, 1790.3 tps, lat 55.544 ms stddev 63.131 2024-09-15T23:20:39.747 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:39 smithi067 ceph-mon[35665]: pgmap v501: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 927 op/s 2024-09-15T23:20:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:39 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:39.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:39 smithi148 ceph-mon[35610]: pgmap v501: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 927 op/s 2024-09-15T23:20:39.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:39 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:20:40.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.367+0000 7f2906a4d640 1 -- 172.21.15.62:0/237032256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f809c460 msgr2=0x7f28f809c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:40.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.367+0000 7f2906a4d640 1 --2- 172.21.15.62:0/237032256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f809c460 0x7f28f809c8c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f28f001a910 tx=0x7f28f003b450 comp rx=0 tx=0).stop 2024-09-15T23:20:40.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.368+0000 7f2906a4d640 1 -- 172.21.15.62:0/237032256 shutdown_connections 2024-09-15T23:20:40.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.368+0000 7f2906a4d640 1 --2- 172.21.15.62:0/237032256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28f809ce00 0x7f28f80a89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.368+0000 7f2906a4d640 1 --2- 172.21.15.62:0/237032256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f809c460 0x7f28f809c8c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.368+0000 7f2906a4d640 1 --2- 172.21.15.62:0/237032256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28f80a1d70 0x7f28f80a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.368+0000 7f2906a4d640 1 -- 172.21.15.62:0/237032256 >> 172.21.15.62:0/237032256 conn(0x7f28f8009fc0 msgr2=0x7f28f800a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:40.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.368+0000 7f2906a4d640 1 -- 172.21.15.62:0/237032256 shutdown_connections 2024-09-15T23:20:40.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.368+0000 7f2906a4d640 1 -- 172.21.15.62:0/237032256 wait complete. 2024-09-15T23:20:40.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.369+0000 7f2906a4d640 1 Processor -- start 2024-09-15T23:20:40.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.369+0000 7f2906a4d640 1 -- start start 2024-09-15T23:20:40.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.370+0000 7f2906a4d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28f809c460 0x7f28f8014d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.370+0000 7f2906a4d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28f809ce00 0x7f28f800ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.370+0000 7f2906a4d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f80a1d70 0x7f28f800f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.370+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28f809c460 0x7f28f8014d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.370+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28f809c460 0x7f28f8014d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:46846/0 (socket says 172.21.15.62:46846) 2024-09-15T23:20:40.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.370+0000 7f2906a4d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f28f800d8c0 con 0x7f28f80a1d70 2024-09-15T23:20:40.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.370+0000 7f2906a4d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f28f800d740 con 0x7f28f809ce00 2024-09-15T23:20:40.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.370+0000 7f2906a4d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f28f800da40 con 0x7f28f809c460 2024-09-15T23:20:40.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.370+0000 7f2905a4b640 1 -- 172.21.15.62:0/1308767546 learned_addr learned my addr 172.21.15.62:0/1308767546 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:40.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.371+0000 7f290524a640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28f809ce00 0x7f28f800ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.371+0000 7f290624c640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f80a1d70 0x7f28f800f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.371+0000 7f290524a640 1 -- 172.21.15.62:0/1308767546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28f809c460 msgr2=0x7f28f8014d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:40.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.371+0000 7f290524a640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28f809c460 0x7f28f8014d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.371+0000 7f290524a640 1 -- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f80a1d70 msgr2=0x7f28f800f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:40.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.371+0000 7f290524a640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f80a1d70 0x7f28f800f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.371+0000 7f290524a640 1 -- 172.21.15.62:0/1308767546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f800fa80 con 0x7f28f809ce00 2024-09-15T23:20:40.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.372+0000 7f290524a640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28f809ce00 0x7f28f800ee10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f28f003ba40 tx=0x7f28f0004c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:40.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.372+0000 7f28f6ffd640 1 -- 172.21.15.62:0/1308767546 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f004f070 con 0x7f28f809ce00 2024-09-15T23:20:40.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.372+0000 7f28f6ffd640 1 -- 172.21.15.62:0/1308767546 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28f0040c00 con 0x7f28f809ce00 2024-09-15T23:20:40.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.373+0000 7f28f6ffd640 1 -- 172.21.15.62:0/1308767546 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0046030 con 0x7f28f809ce00 2024-09-15T23:20:40.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.373+0000 7f2906a4d640 1 -- 172.21.15.62:0/1308767546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f8147460 con 0x7f28f809ce00 2024-09-15T23:20:40.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.373+0000 7f2906a4d640 1 -- 172.21.15.62:0/1308767546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f8147800 con 0x7f28f809ce00 2024-09-15T23:20:40.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.376+0000 7f290a088740 1 Processor -- start 2024-09-15T23:20:40.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.377+0000 7f290a088740 1 -- start start 2024-09-15T23:20:40.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.377+0000 7f290a088740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029594270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.377+0000 7f290a088740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a0295947b0 0x55a029594bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.377+0000 7f290624c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029594270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.377+0000 7f290624c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029594270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:46848/0 (socket says 172.21.15.62:46848) 2024-09-15T23:20:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.377+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a0295947b0 0x55a029594bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.377+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a0295947b0 0x55a029594bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46308/0 (socket says 172.21.15.62:46308) 2024-09-15T23:20:40.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.378+0000 7f290a088740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a0295a1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.378+0000 7f290a088740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a0294fa2f0 con 0x55a02959f550 2024-09-15T23:20:40.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.378+0000 7f290a088740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a0294a1cf0 con 0x55a0295947b0 2024-09-15T23:20:40.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.378+0000 7f290a088740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a0294f62b0 con 0x55a029595d20 2024-09-15T23:20:40.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.378+0000 7f290624c640 1 -- 172.21.15.62:0/1354181857 learned_addr learned my addr 172.21.15.62:0/1354181857 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:40.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.379+0000 7f290624c640 1 -- 172.21.15.62:0/1354181857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a0295947b0 msgr2=0x55a029594bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:40.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.379+0000 7f290624c640 1 --2- 172.21.15.62:0/1354181857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a0295947b0 0x55a029594bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.379+0000 7f290624c640 1 -- 172.21.15.62:0/1354181857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 msgr2=0x55a0295a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:40.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.379+0000 7f290624c640 1 --2- 172.21.15.62:0/1354181857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a0295a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.379+0000 7f290624c640 1 -- 172.21.15.62:0/1354181857 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0294863a0 con 0x55a029595d20 2024-09-15T23:20:40.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.380+0000 7f28f6ffd640 1 -- 172.21.15.62:0/1308767546 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f28f0052070 con 0x7f28f809ce00 2024-09-15T23:20:40.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.380+0000 7f290624c640 1 --2- 172.21.15.62:0/1354181857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029594270 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2900067bf0 tx=0x7f29000691c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30b58a557a44727a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:40.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.380+0000 7f28ebfff640 1 -- 172.21.15.62:0/1354181857 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290007c020 con 0x55a029595d20 2024-09-15T23:20:40.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.381+0000 7f28ebfff640 1 -- 172.21.15.62:0/1354181857 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290006ccb0 con 0x55a029595d20 2024-09-15T23:20:40.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.381+0000 7f28ebfff640 1 -- 172.21.15.62:0/1354181857 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29000775b0 con 0x55a029595d20 2024-09-15T23:20:40.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.381+0000 7f28f6ffd640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28ec0a33a0 0x7f28ec0a5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.381+0000 7f28f6ffd640 1 -- 172.21.15.62:0/1308767546 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f28f0101110 con 0x7f28f809ce00 2024-09-15T23:20:40.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.382+0000 7f290a088740 1 -- 172.21.15.62:0/1354181857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 msgr2=0x55a029594270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:40.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.382+0000 7f290a088740 1 --2- 172.21.15.62:0/1354181857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029594270 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2900067bf0 tx=0x7f29000691c0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.382+0000 7f290a088740 1 -- 172.21.15.62:0/1354181857 shutdown_connections 2024-09-15T23:20:40.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.382+0000 7f290a088740 1 --2- 172.21.15.62:0/1354181857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a0295a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.382+0000 7f290a088740 1 --2- 172.21.15.62:0/1354181857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a0295947b0 0x55a029594bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.382+0000 7f290a088740 1 --2- 172.21.15.62:0/1354181857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029594270 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.382+0000 7f290a088740 1 -- 172.21.15.62:0/1354181857 >> 172.21.15.62:0/1354181857 conn(0x55a02955d310 msgr2=0x55a029590e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:40.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.383+0000 7f290a088740 1 -- 172.21.15.62:0/1354181857 shutdown_connections 2024-09-15T23:20:40.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.383+0000 7f2905a4b640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28ec0a33a0 0x7f28ec0a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.383+0000 7f290a088740 1 -- 172.21.15.62:0/1354181857 wait complete. 2024-09-15T23:20:40.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.384+0000 7f290a088740 1 -- start start 2024-09-15T23:20:40.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.384+0000 7f290a088740 1 Processor -- start 2024-09-15T23:20:40.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.384+0000 7f2905a4b640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28ec0a33a0 0x7f28ec0a5860 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55a0295952d0 tx=0x7f28fc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:40.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.384+0000 7f290a088740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a029595d20 0x55a02959ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.384+0000 7f290a088740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a02959f550 0x55a02959c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f290a088740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a02959cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f290a088740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a0295a2250 con 0x55a029595d20 2024-09-15T23:20:40.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f290a088740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a0294b5060 con 0x55a02959c780 2024-09-15T23:20:40.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f290a088740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a0295a23d0 con 0x55a02959f550 2024-09-15T23:20:40.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f290a088740 10 client.0 _ll_register_callbacks cb 0x55a0292088c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:40.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a029595d20 0x55a02959ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a029595d20 0x55a02959ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50186/0 (socket says 172.21.15.62:50186) 2024-09-15T23:20:40.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f2905a4b640 1 -- 172.21.15.62:0/219767996 learned_addr learned my addr 172.21.15.62:0/219767996 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:40.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f290524a640 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a02959f550 0x55a02959c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f290624c640 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a02959cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f2905a4b640 1 -- 172.21.15.62:0/219767996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a02959f550 msgr2=0x55a02959c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:40.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f2905a4b640 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a02959f550 0x55a02959c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f2905a4b640 1 -- 172.21.15.62:0/219767996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 msgr2=0x55a02959cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:40.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f2905a4b640 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a02959cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f2905a4b640 1 -- 172.21.15.62:0/219767996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a02926b3c0 con 0x55a029595d20 2024-09-15T23:20:40.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f2905a4b640 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a029595d20 0x55a02959ef40 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00fff0 tx=0x7f28fc004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff4718df2c73eb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:40.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.385+0000 7f28e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:40.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.386+0000 7f28e97fa640 1 -- 172.21.15.62:0/219767996 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc01b050 con 0x55a029595d20 2024-09-15T23:20:40.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.386+0000 7f290a088740 1 -- 172.21.15.62:0/219767996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f290006ccb0 con 0x55a029595d20 2024-09-15T23:20:40.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.386+0000 7f28e97fa640 1 -- 172.21.15.62:0/219767996 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f28fc015820 con 0x55a029595d20 2024-09-15T23:20:40.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.386+0000 7f28e97fa640 1 -- 172.21.15.62:0/219767996 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f28fc02e050 con 0x55a029595d20 2024-09-15T23:20:40.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.386+0000 7f290a088740 10 client.16488 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:40.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.386+0000 7f290a088740 1 -- 172.21.15.62:0/219767996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a0295ae230 con 0x55a029595d20 2024-09-15T23:20:40.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.387+0000 7f28e97fa640 1 -- 172.21.15.62:0/219767996 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290006ccb0 con 0x55a029595d20 2024-09-15T23:20:40.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.387+0000 7f28e97fa640 1 -- 172.21.15.62:0/219767996 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f28fc020a50 con 0x55a029595d20 2024-09-15T23:20:40.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.387+0000 7f290a088740 10 client.16488 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:40.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.387+0000 7f290a088740 10 client.16488 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:20:40.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.387+0000 7f290a088740 20 client.16488 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:40.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.387+0000 7f290a088740 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a0295af690 0x55a0295b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.387+0000 7f290a088740 4 client.16488 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:40.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.387+0000 7f290a088740 1 -- 172.21.15.62:0/219767996 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a029507860 con 0x55a0295af690 2024-09-15T23:20:40.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.387+0000 7f290524a640 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a0295af690 0x55a0295b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.388+0000 7f290524a640 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a0295af690 0x55a0295b1a80 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dce5578136160302 server_cookie=1c023ecb3ec8a7b5 in_seq=0 out_seq=0 2024-09-15T23:20:40.396 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.388+0000 7f28e97fa640 10 client.16488 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:20:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f28e97fa640 1 -- 172.21.15.62:0/219767996 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a029507860 con 0x55a0295af690 2024-09-15T23:20:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f28e97fa640 10 client.16488 handle_command_reply: tid=0 2024-09-15T23:20:40.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 client.16488 shutdown 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 -- 172.21.15.62:0/219767996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a029595d20 msgr2=0x55a02959ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a029595d20 0x55a02959ef40 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00fff0 tx=0x7f28fc004700 comp rx=0 tx=0).stop 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 -- 172.21.15.62:0/219767996 shutdown_connections 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a0295af690 0x55a0295b1a80 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a02959cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a02959f550 0x55a02959c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 --2- 172.21.15.62:0/219767996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a029595d20 0x55a02959ef40 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 -- 172.21.15.62:0/219767996 >> 172.21.15.62:0/219767996 conn(0x55a02955d310 msgr2=0x55a029596ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f28e97fa640 0 client.16488 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 -- 172.21.15.62:0/219767996 shutdown_connections 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.389+0000 7f290a088740 1 -- 172.21.15.62:0/219767996 wait complete. 2024-09-15T23:20:40.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.390+0000 7f290a088740 20 client.16488 trim_cache size 0 max 16384 2024-09-15T23:20:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.416+0000 7f290a088740 1 Processor -- start 2024-09-15T23:20:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.416+0000 7f290a088740 1 -- start start 2024-09-15T23:20:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.417+0000 7f290a088740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029590f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.417+0000 7f290a088740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a029597300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.417+0000 7f290a088740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a029597840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.417+0000 7f290a088740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f28f8153de0 con 0x55a02959f550 2024-09-15T23:20:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.417+0000 7f290a088740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f28f81516b0 con 0x55a02959c780 2024-09-15T23:20:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.417+0000 7f290a088740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f28f814f500 con 0x55a029595d20 2024-09-15T23:20:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.421+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029590f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.421+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029590f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:46856/0 (socket says 172.21.15.62:46856) 2024-09-15T23:20:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.421+0000 7f2905a4b640 1 -- 172.21.15.62:0/4063264206 learned_addr learned my addr 172.21.15.62:0/4063264206 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.421+0000 7f290624c640 1 --2- 172.21.15.62:0/4063264206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a029597840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.421+0000 7f290524a640 1 --2- 172.21.15.62:0/4063264206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a029597300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.425+0000 7f290624c640 1 -- 172.21.15.62:0/4063264206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 msgr2=0x55a029590f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.425+0000 7f290624c640 1 --2- 172.21.15.62:0/4063264206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029590f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.425+0000 7f290624c640 1 -- 172.21.15.62:0/4063264206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 msgr2=0x55a029597300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.425+0000 7f290624c640 1 --2- 172.21.15.62:0/4063264206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a029597300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.425+0000 7f290624c640 1 -- 172.21.15.62:0/4063264206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a02959eb60 con 0x55a02959f550 2024-09-15T23:20:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.428+0000 7f290624c640 1 --2- 172.21.15.62:0/4063264206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a029597840 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f29000518b0 tx=0x7f2900067aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941b19b22dfde6ea server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.430+0000 7f28eb7fe640 1 -- 172.21.15.62:0/4063264206 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290007c040 con 0x55a02959f550 2024-09-15T23:20:40.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.430+0000 7f28eb7fe640 1 -- 172.21.15.62:0/4063264206 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2900081420 con 0x55a02959f550 2024-09-15T23:20:40.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.430+0000 7f28eb7fe640 1 -- 172.21.15.62:0/4063264206 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290006ce60 con 0x55a02959f550 2024-09-15T23:20:40.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.432+0000 7f290a088740 1 -- 172.21.15.62:0/4063264206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 msgr2=0x55a029597840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:40.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.432+0000 7f290a088740 1 --2- 172.21.15.62:0/4063264206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a029597840 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f29000518b0 tx=0x7f2900067aa0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.437+0000 7f290a088740 1 -- 172.21.15.62:0/4063264206 shutdown_connections 2024-09-15T23:20:40.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.437+0000 7f290a088740 1 --2- 172.21.15.62:0/4063264206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a029597840 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.437+0000 7f290a088740 1 --2- 172.21.15.62:0/4063264206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a029597300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.437+0000 7f290a088740 1 --2- 172.21.15.62:0/4063264206 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a029590f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.437+0000 7f290a088740 1 -- 172.21.15.62:0/4063264206 >> 172.21.15.62:0/4063264206 conn(0x55a02955d310 msgr2=0x55a029590800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:40.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.439+0000 7f290a088740 1 -- 172.21.15.62:0/4063264206 shutdown_connections 2024-09-15T23:20:40.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.439+0000 7f290a088740 1 -- 172.21.15.62:0/4063264206 wait complete. 2024-09-15T23:20:40.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.439+0000 7f290a088740 1 -- start start 2024-09-15T23:20:40.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.439+0000 7f290a088740 1 Processor -- start 2024-09-15T23:20:40.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.440+0000 7f290a088740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a0295a7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.440+0000 7f290a088740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a0295a7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.440+0000 7f290a088740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a0295988a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.440+0000 7f290a088740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f28f814fb70 con 0x55a02959f550 2024-09-15T23:20:40.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.440+0000 7f290a088740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f28f8150fe0 con 0x55a02959c780 2024-09-15T23:20:40.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.440+0000 7f290a088740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f28f8150240 con 0x55a029595d20 2024-09-15T23:20:40.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.440+0000 7f290a088740 10 client.0 _ll_register_callbacks cb 0x7f28fc00fca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.442+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a0295a7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.442+0000 7f2905a4b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a0295a7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:46868/0 (socket says 172.21.15.62:46868) 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.442+0000 7f2905a4b640 1 -- 172.21.15.62:0/582740951 learned_addr learned my addr 172.21.15.62:0/582740951 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.445+0000 7f2905a4b640 1 -- 172.21.15.62:0/582740951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 msgr2=0x55a0295a7c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.445+0000 7f2905a4b640 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a0295a7c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.445+0000 7f2905a4b640 1 -- 172.21.15.62:0/582740951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 msgr2=0x55a0295988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.445+0000 7f2905a4b640 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a0295988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.445+0000 7f2905a4b640 1 -- 172.21.15.62:0/582740951 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a029598e60 con 0x55a029595d20 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.447+0000 7f2905a4b640 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a0295a7700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f28fc020ef0 tx=0x7f28fc015320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccfe775bf8a5f75b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:40.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.447+0000 7f28e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.449+0000 7f28e97fa640 1 -- 172.21.15.62:0/582740951 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc01b040 con 0x55a029595d20 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.449+0000 7f290a088740 1 -- 172.21.15.62:0/582740951 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2900081420 con 0x55a029595d20 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.450+0000 7f28e97fa640 1 -- 172.21.15.62:0/582740951 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f28fc004220 con 0x55a029595d20 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.450+0000 7f28e97fa640 1 -- 172.21.15.62:0/582740951 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f28fc043050 con 0x55a029595d20 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f290a088740 10 client.25948 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f290a088740 1 -- 172.21.15.62:0/582740951 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a029619ba0 con 0x55a029595d20 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f28e97fa640 1 -- 172.21.15.62:0/582740951 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2900081420 con 0x55a029595d20 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f28e97fa640 1 -- 172.21.15.62:0/582740951 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f28fc0037d0 con 0x55a029595d20 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f290a088740 10 client.25948 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f290a088740 10 client.25948 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:20:40.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f290a088740 20 client.25948 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:40.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f290a088740 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a02961a320 0x55a02961c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:40.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f290a088740 4 client.25948 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:20:40.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.452+0000 7f290a088740 1 -- 172.21.15.62:0/582740951 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a029501d70 con 0x55a02961a320 2024-09-15T23:20:40.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.454+0000 7f290524a640 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a02961a320 0x55a02961c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:40.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.459+0000 7f290524a640 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a02961a320 0x55a02961c7a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d440ad30736e2ba server_cookie=b9162b07d870535b in_seq=0 out_seq=0 2024-09-15T23:20:40.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.459+0000 7f28e97fa640 10 client.25948 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:20:40.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.459+0000 7f28e97fa640 1 -- 172.21.15.62:0/582740951 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a029501d70 con 0x55a02961a320 2024-09-15T23:20:40.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.460+0000 7f28e97fa640 10 client.25948 handle_command_reply: tid=0 2024-09-15T23:20:40.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.460+0000 7f290a088740 1 client.25948 shutdown 2024-09-15T23:20:40.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.460+0000 7f290a088740 1 -- 172.21.15.62:0/582740951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 msgr2=0x55a0295a7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:40.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.460+0000 7f290a088740 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a0295a7700 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f28fc020ef0 tx=0x7f28fc015320 comp rx=0 tx=0).stop 2024-09-15T23:20:40.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.461+0000 7f290a088740 1 -- 172.21.15.62:0/582740951 shutdown_connections 2024-09-15T23:20:40.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.461+0000 7f290a088740 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a02961a320 0x55a02961c7a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.461+0000 7f290a088740 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a02959f550 0x55a0295988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.461+0000 7f290a088740 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a02959c780 0x55a0295a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.461+0000 7f290a088740 1 --2- 172.21.15.62:0/582740951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a029595d20 0x55a0295a7700 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.461+0000 7f290a088740 1 -- 172.21.15.62:0/582740951 >> 172.21.15.62:0/582740951 conn(0x55a02955d310 msgr2=0x55a02958f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:40.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.461+0000 7f28e97fa640 0 client.25948 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:20:40.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.465+0000 7f290a088740 1 -- 172.21.15.62:0/582740951 shutdown_connections 2024-09-15T23:20:40.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.471+0000 7f290a088740 1 -- 172.21.15.62:0/582740951 wait complete. 2024-09-15T23:20:40.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.472+0000 7f290a088740 20 client.25948 trim_cache size 0 max 16384 2024-09-15T23:20:40.477 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:20:40.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 -- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28ec0a33a0 msgr2=0x7f28ec0a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:40.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28ec0a33a0 0x7f28ec0a5860 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55a0295952d0 tx=0x7f28fc008040 comp rx=0 tx=0).stop 2024-09-15T23:20:40.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 -- 172.21.15.62:0/1308767546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28f809ce00 msgr2=0x7f28f800ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:40.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28f809ce00 0x7f28f800ee10 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f28f003ba40 tx=0x7f28f0004c20 comp rx=0 tx=0).stop 2024-09-15T23:20:40.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 -- 172.21.15.62:0/1308767546 shutdown_connections 2024-09-15T23:20:40.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28ec0a33a0 0x7f28ec0a5860 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f28f80a1d70 0x7f28f800f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f28f809ce00 0x7f28f800ee10 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f28f003ba40 tx=0x7f28f0004c20 comp rx=0 tx=0).stop 2024-09-15T23:20:40.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 --2- 172.21.15.62:0/1308767546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f28f809c460 0x7f28f8014d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:40.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.478+0000 7f2906a4d640 1 -- 172.21.15.62:0/1308767546 >> 172.21.15.62:0/1308767546 conn(0x7f28f8009fc0 msgr2=0x7f28f809b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:40.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.479+0000 7f2906a4d640 1 -- 172.21.15.62:0/1308767546 shutdown_connections 2024-09-15T23:20:40.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:40.480+0000 7f2906a4d640 1 -- 172.21.15.62:0/1308767546 wait complete. 2024-09-15T23:20:40.499 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:40.499 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:40.499 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:20:40.777 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:41.195 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:41.196 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:20:41.197 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:20:41.197 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:20:41.197 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:20:41.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.026+0000 7f4124f42640 1 -- 172.21.15.62:0/3144537937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4120106f10 msgr2=0x7f412010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:41.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.026+0000 7f4124f42640 1 --2- 172.21.15.62:0/3144537937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4120106f10 0x7f412010ea90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4118009ca0 tx=0x7f411802a8d0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.027+0000 7f4124f42640 1 -- 172.21.15.62:0/3144537937 shutdown_connections 2024-09-15T23:20:41.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.027+0000 7f4124f42640 1 --2- 172.21.15.62:0/3144537937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4120106f10 0x7f412010ea90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.027+0000 7f4124f42640 1 --2- 172.21.15.62:0/3144537937 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4120102790 0x7f41201067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.027+0000 7f4124f42640 1 --2- 172.21.15.62:0/3144537937 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4120101de0 0x7f41201021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.027+0000 7f4124f42640 1 -- 172.21.15.62:0/3144537937 >> 172.21.15.62:0/3144537937 conn(0x7f41200fd8e0 msgr2=0x7f41200ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.029+0000 7f4124f42640 1 -- 172.21.15.62:0/3144537937 shutdown_connections 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.034+0000 7f4124f42640 1 -- 172.21.15.62:0/3144537937 wait complete. 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.035+0000 7f4124f42640 1 Processor -- start 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.036+0000 7f4124f42640 1 -- start start 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.036+0000 7f4124f42640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4120101de0 0x7f41201a46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.036+0000 7f4124f42640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4120102790 0x7f41201a4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.036+0000 7f4124f42640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4120106f10 0x7f412019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.036+0000 7f4124f42640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f412010fd80 con 0x7f4120101de0 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.036+0000 7f4124f42640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f412010fc00 con 0x7f4120106f10 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.036+0000 7f4124f42640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f412010ff00 con 0x7f4120102790 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.037+0000 7f411ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4120106f10 0x7f412019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.037+0000 7f411e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4120101de0 0x7f41201a46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:41.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.037+0000 7f411e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4120101de0 0x7f41201a46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50224/0 (socket says 172.21.15.62:50224) 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.037+0000 7f411e59c640 1 -- 172.21.15.62:0/937695302 learned_addr learned my addr 172.21.15.62:0/937695302 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.038+0000 7f411e59c640 1 -- 172.21.15.62:0/937695302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4120102790 msgr2=0x7f41201a4c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.038+0000 7f411e59c640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4120102790 0x7f41201a4c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.038+0000 7f411e59c640 1 -- 172.21.15.62:0/937695302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4120106f10 msgr2=0x7f412019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.038+0000 7f411e59c640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4120106f10 0x7f412019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.038+0000 7f411e59c640 1 -- 172.21.15.62:0/937695302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412019eff0 con 0x7f4120101de0 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.038+0000 7f411e59c640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4120101de0 0x7f41201a46d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f411000b4a0 tx=0x7f411000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.043+0000 7f410f7fe640 1 -- 172.21.15.62:0/937695302 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110014070 con 0x7f4120101de0 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.043+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412019f2e0 con 0x7f4120101de0 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.043+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4120197f80 con 0x7f4120101de0 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.043+0000 7f410f7fe640 1 -- 172.21.15.62:0/937695302 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4110010af0 con 0x7f4120101de0 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.043+0000 7f410f7fe640 1 -- 172.21.15.62:0/937695302 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110017d50 con 0x7f4120101de0 2024-09-15T23:20:41.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.045+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4100005290 con 0x7f4120101de0 2024-09-15T23:20:41.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.052+0000 7f410f7fe640 1 -- 172.21.15.62:0/937695302 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4110017070 con 0x7f4120101de0 2024-09-15T23:20:41.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.053+0000 7f410f7fe640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41080a3460 0x7f41080a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:41.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.053+0000 7f410f7fe640 1 -- 172.21.15.62:0/937695302 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f41100cf1d0 con 0x7f4120101de0 2024-09-15T23:20:41.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.059+0000 7f410f7fe640 1 -- 172.21.15.62:0/937695302 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f411009bb90 con 0x7f4120101de0 2024-09-15T23:20:41.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.060+0000 7f411dd9b640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41080a3460 0x7f41080a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:41.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.070+0000 7f411dd9b640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41080a3460 0x7f41080a5920 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f41200fdd00 tx=0x7f4114005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:41.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.423+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4100005580 con 0x7f4120101de0 2024-09-15T23:20:41.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.424+0000 7f410f7fe640 1 -- 172.21.15.62:0/937695302 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11903 (secure 0 0 0) 0x7f411009b490 con 0x7f4120101de0 2024-09-15T23:20:41.426 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:20:41.426 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":31,"btime":"2024-09-15T23:20:13:209709+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:20:12.378892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:20:41.426 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 31 2024-09-15T23:20:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41080a3460 msgr2=0x7f41080a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41080a3460 0x7f41080a5920 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f41200fdd00 tx=0x7f4114005f90 comp rx=0 tx=0).stop 2024-09-15T23:20:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4120101de0 msgr2=0x7f41201a46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4120101de0 0x7f41201a46d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f411000b4a0 tx=0x7f411000b960 comp rx=0 tx=0).stop 2024-09-15T23:20:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 shutdown_connections 2024-09-15T23:20:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41080a3460 0x7f41080a5920 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4120106f10 0x7f412019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4120102790 0x7f41201a4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 --2- 172.21.15.62:0/937695302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4120101de0 0x7f41201a46d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 >> 172.21.15.62:0/937695302 conn(0x7f41200fd8e0 msgr2=0x7f41200fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.427+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 shutdown_connections 2024-09-15T23:20:41.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:41.428+0000 7f4124f42640 1 -- 172.21.15.62:0/937695302 wait complete. 2024-09-15T23:20:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:41 smithi062 ceph-mon[30949]: pgmap v502: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 MiB/s wr, 961 op/s 2024-09-15T23:20:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:41 smithi067 ceph-mon[35665]: pgmap v502: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 MiB/s wr, 961 op/s 2024-09-15T23:20:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:41 smithi148 ceph-mon[35610]: pgmap v502: 129 pgs: 129 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 20 MiB/s wr, 961 op/s 2024-09-15T23:20:41.813 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:20:42.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:42 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/937695302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:42.533 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0' 2024-09-15T23:20:42.533 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0 2024-09-15T23:20:42.709 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:42.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:42 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/937695302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:42 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/937695302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.001+0000 7fbd46e15640 1 -- 172.21.15.62:0/3901030868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd40073680 msgr2=0x7fbd4010d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.001+0000 7fbd46e15640 1 --2- 172.21.15.62:0/3901030868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd40073680 0x7fbd4010d4f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009890 tx=0x7fbd3c02a380 comp rx=0 tx=0).stop 2024-09-15T23:20:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.003+0000 7fbd46e15640 1 -- 172.21.15.62:0/3901030868 shutdown_connections 2024-09-15T23:20:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.003+0000 7fbd46e15640 1 --2- 172.21.15.62:0/3901030868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4010dbb0 0x7fbd40110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.003+0000 7fbd46e15640 1 --2- 172.21.15.62:0/3901030868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd40073680 0x7fbd4010d4f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.003+0000 7fbd46e15640 1 --2- 172.21.15.62:0/3901030868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd40072cb0 0x7fbd400730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.003+0000 7fbd46e15640 1 -- 172.21.15.62:0/3901030868 >> 172.21.15.62:0/3901030868 conn(0x7fbd4006eb50 msgr2=0x7fbd40070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.007+0000 7fbd46e15640 1 -- 172.21.15.62:0/3901030868 shutdown_connections 2024-09-15T23:20:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.010+0000 7fbd46e15640 1 -- 172.21.15.62:0/3901030868 wait complete. 2024-09-15T23:20:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.010+0000 7fbd46e15640 1 Processor -- start 2024-09-15T23:20:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.010+0000 7fbd46e15640 1 -- start start 2024-09-15T23:20:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd46e15640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40072cb0 0x7fbd401168b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd46e15640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd40073680 0x7fbd40116df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd46e15640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd4010dbb0 0x7fbd40117e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd46e15640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbd40110f30 con 0x7fbd40073680 2024-09-15T23:20:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd46e15640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd40110db0 con 0x7fbd40072cb0 2024-09-15T23:20:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd46e15640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbd401110b0 con 0x7fbd4010dbb0 2024-09-15T23:20:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd46614640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd4010dbb0 0x7fbd40117e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd45612640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd40073680 0x7fbd40116df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd45612640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd40073680 0x7fbd40116df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54658/0 (socket says 172.21.15.62:54658) 2024-09-15T23:20:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.011+0000 7fbd45612640 1 -- 172.21.15.62:0/4049056471 learned_addr learned my addr 172.21.15.62:0/4049056471 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.013+0000 7fbd46614640 1 -- 172.21.15.62:0/4049056471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40072cb0 msgr2=0x7fbd401168b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:20:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.013+0000 7fbd46614640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40072cb0 0x7fbd401168b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.013+0000 7fbd46614640 1 -- 172.21.15.62:0/4049056471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd40073680 msgr2=0x7fbd40116df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.013+0000 7fbd46614640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd40073680 0x7fbd40116df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.013+0000 7fbd46614640 1 -- 172.21.15.62:0/4049056471 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd401184a0 con 0x7fbd4010dbb0 2024-09-15T23:20:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.014+0000 7fbd46614640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd4010dbb0 0x7fbd40117e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009fd0 tx=0x7fbd3800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.016+0000 7fbd36ffd640 1 -- 172.21.15.62:0/4049056471 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3800ce20 con 0x7fbd4010dbb0 2024-09-15T23:20:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.016+0000 7fbd36ffd640 1 -- 172.21.15.62:0/4049056471 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd38005020 con 0x7fbd4010dbb0 2024-09-15T23:20:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.016+0000 7fbd36ffd640 1 -- 172.21.15.62:0/4049056471 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38010720 con 0x7fbd4010dbb0 2024-09-15T23:20:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.017+0000 7fbd46e15640 1 -- 172.21.15.62:0/4049056471 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd40118790 con 0x7fbd4010dbb0 2024-09-15T23:20:43.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.017+0000 7fbd46e15640 1 -- 172.21.15.62:0/4049056471 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd40118c90 con 0x7fbd4010dbb0 2024-09-15T23:20:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.022+0000 7fbd4ae0a740 1 Processor -- start 2024-09-15T23:20:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.022+0000 7fbd4ae0a740 1 -- start start 2024-09-15T23:20:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.022+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea6bb70 0x5579bea6bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.023+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea70f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.023+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea73a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.023+0000 7fbd4ae0a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5579be9cf2f0 con 0x5579bea6c4b0 2024-09-15T23:20:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.023+0000 7fbd4ae0a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5579be977270 con 0x5579bea71540 2024-09-15T23:20:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.023+0000 7fbd4ae0a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5579be9cb2b0 con 0x5579bea6bb70 2024-09-15T23:20:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.023+0000 7fbd45e13640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea70f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.023+0000 7fbd45e13640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea70f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54672/0 (socket says 172.21.15.62:54672) 2024-09-15T23:20:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.023+0000 7fbd45e13640 1 -- 172.21.15.62:0/2414061089 learned_addr learned my addr 172.21.15.62:0/2414061089 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.024+0000 7fbd36ffd640 1 -- 172.21.15.62:0/4049056471 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fbd380108c0 con 0x7fbd4010dbb0 2024-09-15T23:20:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.024+0000 7fbd45612640 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea6bb70 0x5579bea6bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.024+0000 7fbd45612640 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea73a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.030+0000 7fbd45612640 1 -- 172.21.15.62:0/2414061089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 msgr2=0x5579bea73a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.030+0000 7fbd45612640 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea73a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.030+0000 7fbd45612640 1 -- 172.21.15.62:0/2414061089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 msgr2=0x5579bea70f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.031+0000 7fbd45612640 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea70f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.031+0000 7fbd45612640 1 -- 172.21.15.62:0/2414061089 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579be976340 con 0x5579bea6bb70 2024-09-15T23:20:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.031+0000 7fbd36ffd640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd2c0a3450 0x7fbd2c0a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.031+0000 7fbd45612640 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea6bb70 0x5579bea6bf70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c030a60 tx=0x7fbd3c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e325d68d05bde6e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.031+0000 7fbd36ffd640 1 -- 172.21.15.62:0/4049056471 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbd380d0190 con 0x7fbd4010dbb0 2024-09-15T23:20:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.031+0000 7fbd2bfff640 1 -- 172.21.15.62:0/2414061089 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c037030 con 0x5579bea6bb70 2024-09-15T23:20:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.031+0000 7fbd2bfff640 1 -- 172.21.15.62:0/2414061089 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd3c038040 con 0x5579bea6bb70 2024-09-15T23:20:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.032+0000 7fbd45e13640 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea70f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:43.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.032+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2414061089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea6bb70 msgr2=0x5579bea6bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.032+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea6bb70 0x5579bea6bf70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c030a60 tx=0x7fbd3c000bc0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.033+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2414061089 shutdown_connections 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.033+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea73a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.033+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea70f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.033+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2414061089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea6bb70 0x5579bea6bf70 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c030a60 tx=0x7fbd3c000bc0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.033+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2414061089 >> 172.21.15.62:0/2414061089 conn(0x5579bea671e0 msgr2=0x5579bea675e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.033+0000 7fbd45e13640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd2c0a3450 0x7fbd2c0a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.034+0000 7fbd45e13640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd2c0a3450 0x7fbd2c0a5910 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x5579bea68060 tx=0x7fbd3000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.034+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2414061089 shutdown_connections 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.035+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2414061089 wait complete. 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.035+0000 7fbd4ae0a740 1 -- start start 2024-09-15T23:20:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.035+0000 7fbd4ae0a740 1 Processor -- start 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea6c4b0 0x5579bea76220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea71540 0x5579bea76760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea7db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd4ae0a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5579bea74500 con 0x5579bea71540 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd4ae0a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5579be958290 con 0x5579bea6c4b0 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd4ae0a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5579bea74680 con 0x5579bea7b790 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd4ae0a740 10 client.0 _ll_register_callbacks cb 0x5579be989140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd46614640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea7db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd46614640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea7db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56184/0 (socket says 172.21.15.62:56184) 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.036+0000 7fbd46614640 1 -- 172.21.15.62:0/2998498273 learned_addr learned my addr 172.21.15.62:0/2998498273 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.037+0000 7fbd45612640 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea71540 0x5579bea76760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.038+0000 7fbd45612640 1 -- 172.21.15.62:0/2998498273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 msgr2=0x5579bea7db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.038+0000 7fbd45612640 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea7db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.038+0000 7fbd45612640 1 -- 172.21.15.62:0/2998498273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea6c4b0 msgr2=0x5579bea76220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.038+0000 7fbd45612640 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea6c4b0 0x5579bea76220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.038+0000 7fbd45612640 1 -- 172.21.15.62:0/2998498273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579be6e2210 con 0x5579bea71540 2024-09-15T23:20:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.041+0000 7fbd45612640 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea71540 0x5579bea76760 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c037000 tx=0x7fbd3c005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad73d464692d91a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:43.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.041+0000 7fbd297fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.043+0000 7fbd297fa640 1 -- 172.21.15.62:0/2998498273 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c030e00 con 0x5579bea71540 2024-09-15T23:20:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.043+0000 7fbd297fa640 1 -- 172.21.15.62:0/2998498273 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbd3c004040 con 0x5579bea71540 2024-09-15T23:20:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.044+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2998498273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5579be9dca20 con 0x5579bea71540 2024-09-15T23:20:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.044+0000 7fbd297fa640 1 -- 172.21.15.62:0/2998498273 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbd3c005660 con 0x5579bea71540 2024-09-15T23:20:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.044+0000 7fbd4ae0a740 10 client.16506 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.044+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2998498273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5579be7ae620 con 0x5579bea71540 2024-09-15T23:20:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.045+0000 7fbd297fa640 1 -- 172.21.15.62:0/2998498273 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5579be9dca20 con 0x5579bea71540 2024-09-15T23:20:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.045+0000 7fbd297fa640 1 -- 172.21.15.62:0/2998498273 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fbd3c052030 con 0x5579bea71540 2024-09-15T23:20:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.046+0000 7fbd4ae0a740 10 client.16506 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.046+0000 7fbd4ae0a740 10 client.16506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.046+0000 7fbd4ae0a740 20 client.16506 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.046+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5579bea81f10 0x5579bea84300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.046+0000 7fbd4ae0a740 4 client.16506 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.046+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2998498273 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579be9d6d70 con 0x5579bea81f10 2024-09-15T23:20:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.047+0000 7fbd45e13640 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5579bea81f10 0x5579bea84300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.049+0000 7fbd297fa640 10 client.16506 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.049+0000 7fbd45e13640 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5579bea81f10 0x5579bea84300 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c07686f2c9546f81 server_cookie=6f6c64d92cf1ec43 in_seq=0 out_seq=0 2024-09-15T23:20:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.050+0000 7fbd297fa640 1 -- 172.21.15.62:0/2998498273 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5579be9d6d70 con 0x5579bea81f10 2024-09-15T23:20:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.050+0000 7fbd297fa640 10 client.16506 handle_command_reply: tid=0 2024-09-15T23:20:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.050+0000 7fbd4ae0a740 1 client.16506 shutdown 2024-09-15T23:20:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.051+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2998498273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea71540 msgr2=0x5579bea76760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.051+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea71540 0x5579bea76760 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c037000 tx=0x7fbd3c005280 comp rx=0 tx=0).stop 2024-09-15T23:20:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.052+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2998498273 shutdown_connections 2024-09-15T23:20:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.052+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5579bea81f10 0x5579bea84300 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.052+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea7db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.052+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea71540 0x5579bea76760 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.052+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/2998498273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea6c4b0 0x5579bea76220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.052+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2998498273 >> 172.21.15.62:0/2998498273 conn(0x5579bea671e0 msgr2=0x5579bea68a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.052+0000 7fbd297fa640 0 client.16506 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.054+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2998498273 shutdown_connections 2024-09-15T23:20:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.054+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/2998498273 wait complete. 2024-09-15T23:20:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.054+0000 7fbd4ae0a740 20 client.16506 trim_cache size 0 max 16384 2024-09-15T23:20:43.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.101+0000 7fbd4ae0a740 1 Processor -- start 2024-09-15T23:20:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.101+0000 7fbd4ae0a740 1 -- start start 2024-09-15T23:20:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.102+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea83f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.102+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea6d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.102+0000 7fbd45e13640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea83f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.102+0000 7fbd45e13640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea83f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54694/0 (socket says 172.21.15.62:54694) 2024-09-15T23:20:43.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.103+0000 7fbd45e13640 1 -- 172.21.15.62:0/1951094238 learned_addr learned my addr 172.21.15.62:0/1951094238 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:43.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.103+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea6def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.103+0000 7fbd45612640 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea6d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.103+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/1951094238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbd24012450 con 0x5579bea6c4b0 2024-09-15T23:20:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.103+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/1951094238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2400fd20 con 0x5579bea71540 2024-09-15T23:20:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.103+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/1951094238 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2400db70 con 0x5579bea7b790 2024-09-15T23:20:43.105 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.103+0000 7fbd46614640 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea6def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.104+0000 7fbd45e13640 1 -- 172.21.15.62:0/1951094238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 msgr2=0x5579bea6def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.104+0000 7fbd45e13640 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea6def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.104+0000 7fbd45e13640 1 -- 172.21.15.62:0/1951094238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 msgr2=0x5579bea6d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.104+0000 7fbd45e13640 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea6d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.104+0000 7fbd45e13640 1 -- 172.21.15.62:0/1951094238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579bea67000 con 0x5579bea6c4b0 2024-09-15T23:20:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.104+0000 7fbd45e13640 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea83f20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fbd30010cc0 tx=0x7fbd30013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f179705ff19f6594 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.104+0000 7fbd2b7fe640 1 -- 172.21.15.62:0/1951094238 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3002b070 con 0x5579bea6c4b0 2024-09-15T23:20:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.104+0000 7fbd2b7fe640 1 -- 172.21.15.62:0/1951094238 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd30011d50 con 0x5579bea6c4b0 2024-09-15T23:20:43.106 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.104+0000 7fbd2b7fe640 1 -- 172.21.15.62:0/1951094238 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd300255f0 con 0x5579bea6c4b0 2024-09-15T23:20:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.105+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/1951094238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 msgr2=0x5579bea83f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.105+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea83f20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fbd30010cc0 tx=0x7fbd30013000 comp rx=0 tx=0).stop 2024-09-15T23:20:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.105+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/1951094238 shutdown_connections 2024-09-15T23:20:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.105+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea6def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.105+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea6d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.105+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/1951094238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea83f20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.105+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/1951094238 >> 172.21.15.62:0/1951094238 conn(0x5579bea671e0 msgr2=0x5579bea76200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.106+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/1951094238 shutdown_connections 2024-09-15T23:20:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.106+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/1951094238 wait complete. 2024-09-15T23:20:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.106+0000 7fbd4ae0a740 1 -- start start 2024-09-15T23:20:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.106+0000 7fbd4ae0a740 1 Processor -- start 2024-09-15T23:20:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea935f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.108 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea93b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd4ae0a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea94b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd4ae0a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2400e1e0 con 0x5579bea6c4b0 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd4ae0a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2400f650 con 0x5579bea71540 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd4ae0a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2400e8b0 con 0x5579bea7b790 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd4ae0a740 10 client.0 _ll_register_callbacks cb 0x7fbd3000b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45612640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea93b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45612640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea93b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:58714/0 (socket says 172.21.15.62:58714) 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45612640 1 -- 172.21.15.62:0/102275173 learned_addr learned my addr 172.21.15.62:0/102275173 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd46614640 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea94b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45e13640 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea935f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45612640 1 -- 172.21.15.62:0/102275173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 msgr2=0x5579bea94b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45612640 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea94b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45612640 1 -- 172.21.15.62:0/102275173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 msgr2=0x5579bea935f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45612640 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea935f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45612640 1 -- 172.21.15.62:0/102275173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579bea95150 con 0x5579bea71540 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd46614640 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea94b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd45612640 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea93b30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x5579bea77700 tx=0x7fbd3c030740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff600d507897e2db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd297fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.107+0000 7fbd297fa640 1 -- 172.21.15.62:0/102275173 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c05b030 con 0x5579bea71540 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.108+0000 7fbd297fa640 1 -- 172.21.15.62:0/102275173 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbd3c047040 con 0x5579bea71540 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.108+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/102275173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd30011d50 con 0x5579bea71540 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.108+0000 7fbd297fa640 1 -- 172.21.15.62:0/102275173 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbd3c068050 con 0x5579bea71540 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.108+0000 7fbd4ae0a740 10 client.26135 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:43.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.108+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/102275173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5579beaec8c0 con 0x5579bea71540 2024-09-15T23:20:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.120+0000 7fbd297fa640 1 -- 172.21.15.62:0/102275173 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd30011d50 con 0x5579bea71540 2024-09-15T23:20:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.120+0000 7fbd297fa640 1 -- 172.21.15.62:0/102275173 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fbd3c052070 con 0x5579bea71540 2024-09-15T23:20:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.120+0000 7fbd4ae0a740 10 client.26135 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.120+0000 7fbd4ae0a740 10 client.26135 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.120+0000 7fbd4ae0a740 20 client.26135 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.120+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5579beaed060 0x5579beaef480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.120+0000 7fbd4ae0a740 4 client.26135 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:20:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.120+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/102275173 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5579be953900 con 0x5579beaed060 2024-09-15T23:20:43.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.121+0000 7fbd45e13640 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5579beaed060 0x5579beaef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.122+0000 7fbd45e13640 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5579beaed060 0x5579beaef480 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d6bd49eaf2ab10 server_cookie=3bea72324a37e9c in_seq=0 out_seq=0 2024-09-15T23:20:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.122+0000 7fbd297fa640 10 client.26135 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.159+0000 7fbd297fa640 1 -- 172.21.15.62:0/102275173 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5579be953900 con 0x5579beaed060 2024-09-15T23:20:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.159+0000 7fbd297fa640 10 client.26135 handle_command_reply: tid=0 2024-09-15T23:20:43.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.159+0000 7fbd4ae0a740 1 client.26135 shutdown 2024-09-15T23:20:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.161+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/102275173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 msgr2=0x5579bea93b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.161+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea93b30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x5579bea77700 tx=0x7fbd3c030740 comp rx=0 tx=0).stop 2024-09-15T23:20:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.161+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/102275173 shutdown_connections 2024-09-15T23:20:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.162+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5579beaed060 0x5579beaef480 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.162+0000 7fbd297fa640 0 client.26135 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.162+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5579bea7b790 0x5579bea94b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.162+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5579bea71540 0x5579bea93b30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.162+0000 7fbd4ae0a740 1 --2- 172.21.15.62:0/102275173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5579bea6c4b0 0x5579bea935f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.162+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/102275173 >> 172.21.15.62:0/102275173 conn(0x5579bea671e0 msgr2=0x5579bea75fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.162+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/102275173 shutdown_connections 2024-09-15T23:20:43.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.162+0000 7fbd4ae0a740 1 -- 172.21.15.62:0/102275173 wait complete. 2024-09-15T23:20:43.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.164+0000 7fbd4ae0a740 20 client.26135 trim_cache size 0 max 16384 2024-09-15T23:20:43.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:20:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:20:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:20:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:20:43.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.167+0000 7fbd46e15640 1 -- 172.21.15.62:0/4049056471 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd2c0a3450 msgr2=0x7fbd2c0a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.167+0000 7fbd46e15640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd2c0a3450 0x7fbd2c0a5910 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x5579bea68060 tx=0x7fbd3000b390 comp rx=0 tx=0).stop 2024-09-15T23:20:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.167+0000 7fbd46e15640 1 -- 172.21.15.62:0/4049056471 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd4010dbb0 msgr2=0x7fbd40117e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:43.168 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.167+0000 7fbd46e15640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd4010dbb0 0x7fbd40117e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009fd0 tx=0x7fbd3800ef10 comp rx=0 tx=0).stop 2024-09-15T23:20:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.171+0000 7fbd46e15640 1 -- 172.21.15.62:0/4049056471 shutdown_connections 2024-09-15T23:20:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.171+0000 7fbd46e15640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd2c0a3450 0x7fbd2c0a5910 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.171+0000 7fbd46e15640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd4010dbb0 0x7fbd40117e30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.171+0000 7fbd46e15640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd40073680 0x7fbd40116df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.171+0000 7fbd46e15640 1 --2- 172.21.15.62:0/4049056471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd40072cb0 0x7fbd401168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:43.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.171+0000 7fbd46e15640 1 -- 172.21.15.62:0/4049056471 >> 172.21.15.62:0/4049056471 conn(0x7fbd4006eb50 msgr2=0x7fbd400708d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.175+0000 7fbd46e15640 1 -- 172.21.15.62:0/4049056471 shutdown_connections 2024-09-15T23:20:43.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:43.175+0000 7fbd46e15640 1 -- 172.21.15.62:0/4049056471 wait complete. 2024-09-15T23:20:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:43 smithi062 ceph-mon[30949]: pgmap v503: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 964 op/s 2024-09-15T23:20:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:43 smithi067 ceph-mon[35665]: pgmap v503: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 964 op/s 2024-09-15T23:20:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:43 smithi148 ceph-mon[35610]: pgmap v503: 129 pgs: 129 active+clean; 1.1 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 MiB/s wr, 964 op/s 2024-09-15T23:20:44.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 265.0 s, 967.1 tps, lat 101.511 ms stddev 107.621 2024-09-15T23:20:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.233+0000 7f9caa51a640 1 -- 172.21.15.62:0/1802221045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 msgr2=0x7f9c9c0a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.233+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1802221045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 0x7f9c9c0a1320 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4067960 tx=0x7f9ca4096da0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.233+0000 7f9caa51a640 1 -- 172.21.15.62:0/1802221045 shutdown_connections 2024-09-15T23:20:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.233+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1802221045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 0x7f9c9c0a1320 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.233+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1802221045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c094930 0x7f9c9c094d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.233+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1802221045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c9c09a550 0x7f9c9c09a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.233+0000 7f9caa51a640 1 -- 172.21.15.62:0/1802221045 >> 172.21.15.62:0/1802221045 conn(0x7f9c9c00a1f0 msgr2=0x7f9c9c00a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.233+0000 7f9caa51a640 1 -- 172.21.15.62:0/1802221045 shutdown_connections 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.233+0000 7f9caa51a640 1 -- 172.21.15.62:0/1802221045 wait complete. 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.234+0000 7f9caa51a640 1 Processor -- start 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.234+0000 7f9caa51a640 1 -- start start 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9caa51a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c9c094930 0x7f9c9c130640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9caa51a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 0x7f9c9c130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9caa51a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c09a550 0x7f9c9c12a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9caa51a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c0a4760 con 0x7f9c9c0952d0 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9caa51a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c0a45e0 con 0x7f9c9c09a550 2024-09-15T23:20:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9caa51a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c0a48e0 con 0x7f9c9c094930 2024-09-15T23:20:45.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca8d17640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 0x7f9c9c130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca8d17640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 0x7f9c9c130b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54726/0 (socket says 172.21.15.62:54726) 2024-09-15T23:20:45.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca9518640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c9c094930 0x7f9c9c130640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca8d17640 1 -- 172.21.15.62:0/1708151383 learned_addr learned my addr 172.21.15.62:0/1708151383 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:45.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c09a550 0x7f9c9c12a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca8d17640 1 -- 172.21.15.62:0/1708151383 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c9c094930 msgr2=0x7f9c9c130640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:45.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca8d17640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c9c094930 0x7f9c9c130640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca8d17640 1 -- 172.21.15.62:0/1708151383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c09a550 msgr2=0x7f9c9c12a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:45.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca8d17640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c09a550 0x7f9c9c12a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca8d17640 1 -- 172.21.15.62:0/1708151383 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9c12af60 con 0x7f9c9c0952d0 2024-09-15T23:20:45.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca9518640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c9c094930 0x7f9c9c130640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:45.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.235+0000 7f9ca8d17640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 0x7f9c9c130b80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9c940098b0 tx=0x7f9c94009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:45.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.236+0000 7f9c9a7fc640 1 -- 172.21.15.62:0/1708151383 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9400de50 con 0x7f9c9c0952d0 2024-09-15T23:20:45.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.236+0000 7f9c9a7fc640 1 -- 172.21.15.62:0/1708151383 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c94007c80 con 0x7f9c9c0952d0 2024-09-15T23:20:45.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.236+0000 7f9caa51a640 1 -- 172.21.15.62:0/1708151383 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c9c12b250 con 0x7f9c9c0952d0 2024-09-15T23:20:45.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.236+0000 7f9c9a7fc640 1 -- 172.21.15.62:0/1708151383 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94015400 con 0x7f9c9c0952d0 2024-09-15T23:20:45.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.236+0000 7f9caa51a640 1 -- 172.21.15.62:0/1708151383 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c9c004cb0 con 0x7f9c9c0952d0 2024-09-15T23:20:45.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.238+0000 7f9c9a7fc640 1 -- 172.21.15.62:0/1708151383 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9c940077a0 con 0x7f9c9c0952d0 2024-09-15T23:20:45.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9cadb55740 1 Processor -- start 2024-09-15T23:20:45.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9cadb55740 1 -- start start 2024-09-15T23:20:45.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6ed5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6ed5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9c9a7fc640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c900a3470 0x7f9c900a5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ee2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9cadb55740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5635d6e3b2f0 con 0x5635d6ed6cf0 2024-09-15T23:20:45.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9cadb55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5635d6de2cf0 con 0x5635d6ee0520 2024-09-15T23:20:45.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9cadb55740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5635d6e372b0 con 0x5635d6ed5780 2024-09-15T23:20:45.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca9d19640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6ed5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca9d19640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6ed5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54728/0 (socket says 172.21.15.62:54728) 2024-09-15T23:20:45.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca9d19640 1 -- 172.21.15.62:0/3241614434 learned_addr learned my addr 172.21.15.62:0/3241614434 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:45.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9c9a7fc640 1 -- 172.21.15.62:0/1708151383 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9c940cf7a0 con 0x7f9c9c0952d0 2024-09-15T23:20:45.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca9518640 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6ed5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca8d17640 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ee2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca9d19640 1 -- 172.21.15.62:0/3241614434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 msgr2=0x5635d6ed5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:45.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6ed5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca9d19640 1 -- 172.21.15.62:0/3241614434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 msgr2=0x5635d6ee2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:45.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ee2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca9d19640 1 -- 172.21.15.62:0/3241614434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d6dc73a0 con 0x5635d6ed6cf0 2024-09-15T23:20:45.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.239+0000 7f9ca8d17640 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ee2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:20:45.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.240+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c900a3470 0x7f9c900a5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.240+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6ed5240 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4000b60 tx=0x7f9ca4068300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c206b8d3868dde8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:45.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.240+0000 7f9c8f7fe640 1 -- 172.21.15.62:0/3241614434 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4097d70 con 0x5635d6ed6cf0 2024-09-15T23:20:45.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.240+0000 7f9c8f7fe640 1 -- 172.21.15.62:0/3241614434 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca40b0030 con 0x5635d6ed6cf0 2024-09-15T23:20:45.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.240+0000 7f9c8f7fe640 1 -- 172.21.15.62:0/3241614434 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40ac690 con 0x5635d6ed6cf0 2024-09-15T23:20:45.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.240+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c900a3470 0x7f9c900a5930 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40b3000 tx=0x7f9ca40b8000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:45.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 -- 172.21.15.62:0/3241614434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 msgr2=0x5635d6ed5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:45.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6ed5240 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4000b60 tx=0x7f9ca4068300 comp rx=0 tx=0).stop 2024-09-15T23:20:45.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 -- 172.21.15.62:0/3241614434 shutdown_connections 2024-09-15T23:20:45.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ee2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6ed5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 --2- 172.21.15.62:0/3241614434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6ed5240 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 -- 172.21.15.62:0/3241614434 >> 172.21.15.62:0/3241614434 conn(0x5635d6e9e310 msgr2=0x5635d6ed1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:45.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 -- 172.21.15.62:0/3241614434 shutdown_connections 2024-09-15T23:20:45.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 -- 172.21.15.62:0/3241614434 wait complete. 2024-09-15T23:20:45.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.241+0000 7f9cadb55740 1 -- start start 2024-09-15T23:20:45.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.242+0000 7f9cadb55740 1 Processor -- start 2024-09-15T23:20:45.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.242+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6edff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.242+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed6cf0 0x5635d6edd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.242+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6edd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.242+0000 7f9cadb55740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5635d6ee3220 con 0x5635d6ee0520 2024-09-15T23:20:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.242+0000 7f9cadb55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5635d6df6060 con 0x5635d6ed6cf0 2024-09-15T23:20:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.242+0000 7f9cadb55740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5635d6ee33a0 con 0x5635d6ed5780 2024-09-15T23:20:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.242+0000 7f9cadb55740 10 client.0 _ll_register_callbacks cb 0x5635d6b498c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.242+0000 7f9ca9d19640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6edd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.243+0000 7f9ca9d19640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6edd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54738/0 (socket says 172.21.15.62:54738) 2024-09-15T23:20:45.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.243+0000 7f9ca9d19640 1 -- 172.21.15.62:0/646410499 learned_addr learned my addr 172.21.15.62:0/646410499 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:45.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.243+0000 7f9ca9d19640 1 -- 172.21.15.62:0/646410499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 msgr2=0x5635d6edff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:45.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.243+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6edff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.243+0000 7f9ca9d19640 1 -- 172.21.15.62:0/646410499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed6cf0 msgr2=0x5635d6edd200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:20:45.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.243+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed6cf0 0x5635d6edd200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.243+0000 7f9ca9d19640 1 -- 172.21.15.62:0/646410499 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d6bac3c0 con 0x5635d6ee0520 2024-09-15T23:20:45.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.246+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6edd990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40972c0 tx=0x7f9ca4065460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68678c0c2a9a52ce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:45.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.246+0000 7f9c8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:45.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.246+0000 7f9c8cff9640 1 -- 172.21.15.62:0/646410499 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40acaf0 con 0x5635d6ee0520 2024-09-15T23:20:45.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.246+0000 7f9c8cff9640 1 -- 172.21.15.62:0/646410499 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9ca40acc90 con 0x5635d6ee0520 2024-09-15T23:20:45.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.247+0000 7f9cadb55740 1 -- 172.21.15.62:0/646410499 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ca40b0030 con 0x5635d6ee0520 2024-09-15T23:20:45.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.247+0000 7f9c8cff9640 1 -- 172.21.15.62:0/646410499 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9ca40d1050 con 0x5635d6ee0520 2024-09-15T23:20:45.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.252+0000 7f9cadb55740 10 client.16524 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:45.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.252+0000 7f9cadb55740 1 -- 172.21.15.62:0/646410499 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5635d6eeeca0 con 0x5635d6ee0520 2024-09-15T23:20:45.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.253+0000 7f9c8cff9640 1 -- 172.21.15.62:0/646410499 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca40b0030 con 0x5635d6ee0520 2024-09-15T23:20:45.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.253+0000 7f9c8cff9640 1 -- 172.21.15.62:0/646410499 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f9ca40c0450 con 0x5635d6ee0520 2024-09-15T23:20:45.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.253+0000 7f9cadb55740 10 client.16524 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:45.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.253+0000 7f9cadb55740 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:45.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.253+0000 7f9cadb55740 20 client.16524 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:45.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.253+0000 7f9cadb55740 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5635d6eefff0 0x5635d6ef23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.253+0000 7f9cadb55740 4 client.16524 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:45.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.253+0000 7f9cadb55740 1 -- 172.21.15.62:0/646410499 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635d6e48860 con 0x5635d6eefff0 2024-09-15T23:20:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.255+0000 7f9ca9518640 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5635d6eefff0 0x5635d6ef23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.257+0000 7f9ca9518640 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5635d6eefff0 0x5635d6ef23e0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84f7d2fec09d8d00 server_cookie=d21c640cc36afac5 in_seq=0 out_seq=0 2024-09-15T23:20:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.257+0000 7f9c8cff9640 10 client.16524 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.260+0000 7f9c8cff9640 1 -- 172.21.15.62:0/646410499 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5635d6e48860 con 0x5635d6eefff0 2024-09-15T23:20:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.260+0000 7f9c8cff9640 10 client.16524 handle_command_reply: tid=0 2024-09-15T23:20:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.260+0000 7f9cadb55740 1 client.16524 shutdown 2024-09-15T23:20:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.262+0000 7f9cadb55740 1 -- 172.21.15.62:0/646410499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 msgr2=0x5635d6edd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.262+0000 7f9cadb55740 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6edd990 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40972c0 tx=0x7f9ca4065460 comp rx=0 tx=0).stop 2024-09-15T23:20:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.262+0000 7f9cadb55740 1 -- 172.21.15.62:0/646410499 shutdown_connections 2024-09-15T23:20:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.262+0000 7f9cadb55740 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5635d6eefff0 0x5635d6ef23e0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.262+0000 7f9c8cff9640 0 client.16524 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.262+0000 7f9cadb55740 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6edd990 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.262+0000 7f9cadb55740 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed6cf0 0x5635d6edd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.262+0000 7f9cadb55740 1 --2- 172.21.15.62:0/646410499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6edff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.262+0000 7f9cadb55740 1 -- 172.21.15.62:0/646410499 >> 172.21.15.62:0/646410499 conn(0x5635d6e9e310 msgr2=0x5635d6ed7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.263+0000 7f9cadb55740 1 -- 172.21.15.62:0/646410499 shutdown_connections 2024-09-15T23:20:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.263+0000 7f9cadb55740 1 -- 172.21.15.62:0/646410499 wait complete. 2024-09-15T23:20:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.263+0000 7f9cadb55740 20 client.16524 trim_cache size 0 max 16384 2024-09-15T23:20:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.291+0000 7f9cadb55740 1 Processor -- start 2024-09-15T23:20:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.291+0000 7f9cadb55740 1 -- start start 2024-09-15T23:20:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6edcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6edf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ed82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9cadb55740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c142d00 con 0x5635d6ed6cf0 2024-09-15T23:20:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9cadb55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c1405d0 con 0x5635d6ee0520 2024-09-15T23:20:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9cadb55740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c13e420 con 0x5635d6ed5780 2024-09-15T23:20:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9ca9d19640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ed82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9ca9d19640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ed82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:58756/0 (socket says 172.21.15.62:58756) 2024-09-15T23:20:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9ca9d19640 1 -- 172.21.15.62:0/3316311787 learned_addr learned my addr 172.21.15.62:0/3316311787 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9ca8d17640 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6edf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.292+0000 7f9ca9518640 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6edcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.293+0000 7f9ca9518640 1 -- 172.21.15.62:0/3316311787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 msgr2=0x5635d6ed82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.293+0000 7f9ca9518640 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ed82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.293+0000 7f9ca9518640 1 -- 172.21.15.62:0/3316311787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 msgr2=0x5635d6edf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.293+0000 7f9ca9518640 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6edf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.293+0000 7f9ca9518640 1 -- 172.21.15.62:0/3316311787 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d6ed16c0 con 0x5635d6ed5780 2024-09-15T23:20:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.293+0000 7f9ca8d17640 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6edf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.293+0000 7f9ca9518640 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6edcac0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x5635d6de2e40 tx=0x7f9ca001e7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b275e296e90988e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.293+0000 7f9c8effd640 1 -- 172.21.15.62:0/3316311787 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0004220 con 0x5635d6ed5780 2024-09-15T23:20:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9c8effd640 1 -- 172.21.15.62:0/3316311787 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca00043c0 con 0x5635d6ed5780 2024-09-15T23:20:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9c8effd640 1 -- 172.21.15.62:0/3316311787 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0004e10 con 0x5635d6ed5780 2024-09-15T23:20:45.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9cadb55740 1 -- 172.21.15.62:0/3316311787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 msgr2=0x5635d6edcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:45.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9cadb55740 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6edcac0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x5635d6de2e40 tx=0x7f9ca001e7d0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9cadb55740 1 -- 172.21.15.62:0/3316311787 shutdown_connections 2024-09-15T23:20:45.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9cadb55740 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ee0520 0x5635d6ed82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9cadb55740 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ed6cf0 0x5635d6edf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9cadb55740 1 --2- 172.21.15.62:0/3316311787 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed5780 0x5635d6edcac0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9cadb55740 1 -- 172.21.15.62:0/3316311787 >> 172.21.15.62:0/3316311787 conn(0x5635d6e9e310 msgr2=0x5635d6edea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:45.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.294+0000 7f9cadb55740 1 -- 172.21.15.62:0/3316311787 shutdown_connections 2024-09-15T23:20:45.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.295+0000 7f9cadb55740 1 -- 172.21.15.62:0/3316311787 wait complete. 2024-09-15T23:20:45.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.295+0000 7f9cadb55740 1 -- start start 2024-09-15T23:20:45.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.295+0000 7f9cadb55740 1 Processor -- start 2024-09-15T23:20:45.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed5780 0x5635d6ee8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed6cf0 0x5635d6ee8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9cadb55740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6ee5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9cadb55740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c13ea90 con 0x5635d6ee0520 2024-09-15T23:20:45.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9518640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed5780 0x5635d6ee8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9518640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed5780 0x5635d6ee8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:58770/0 (socket says 172.21.15.62:58770) 2024-09-15T23:20:45.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9518640 1 -- 172.21.15.62:0/2251648288 learned_addr learned my addr 172.21.15.62:0/2251648288 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:45.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c13ff00 con 0x5635d6ed5780 2024-09-15T23:20:45.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9c13f160 con 0x5635d6ed6cf0 2024-09-15T23:20:45.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9cadb55740 10 client.0 _ll_register_callbacks cb 0x7f9ca0010d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:45.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6ee5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9518640 1 -- 172.21.15.62:0/2251648288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed6cf0 msgr2=0x5635d6ee8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:45.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9518640 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed6cf0 0x5635d6ee8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9518640 1 -- 172.21.15.62:0/2251648288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 msgr2=0x5635d6ee5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:45.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9518640 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6ee5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9518640 1 -- 172.21.15.62:0/2251648288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d6edb120 con 0x5635d6ed5780 2024-09-15T23:20:45.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9d19640 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6ee5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:45.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9ca9518640 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed5780 0x5635d6ee8660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00108e0 tx=0x7f9ca0005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79c65f34b611697a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:45.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.296+0000 7f9c8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:45.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9c8cff9640 1 -- 172.21.15.62:0/2251648288 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca002b070 con 0x5635d6ed5780 2024-09-15T23:20:45.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ca00043c0 con 0x5635d6ed5780 2024-09-15T23:20:45.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9c8cff9640 1 -- 172.21.15.62:0/2251648288 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9ca0027440 con 0x5635d6ed5780 2024-09-15T23:20:45.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9c8cff9640 1 -- 172.21.15.62:0/2251648288 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9ca003a050 con 0x5635d6ed5780 2024-09-15T23:20:45.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9cadb55740 10 client.26147 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:45.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5635d6f746d0 con 0x5635d6ed5780 2024-09-15T23:20:45.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9c8cff9640 1 -- 172.21.15.62:0/2251648288 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca00043c0 con 0x5635d6ed5780 2024-09-15T23:20:45.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9c8cff9640 1 -- 172.21.15.62:0/2251648288 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f9ca002f500 con 0x5635d6ed5780 2024-09-15T23:20:45.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9cadb55740 10 client.26147 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:45.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9cadb55740 10 client.26147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:45.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9cadb55740 20 client.26147 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:45.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9cadb55740 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5635d6f74d60 0x5635d6f771c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:45.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9cadb55740 4 client.26147 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:20:45.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.297+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5635d6e42d70 con 0x5635d6f74d60 2024-09-15T23:20:45.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.298+0000 7f9ca8d17640 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5635d6f74d60 0x5635d6f771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:45.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.298+0000 7f9ca8d17640 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5635d6f74d60 0x5635d6f771c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b0e4df56408efef server_cookie=94edb69d312bc656 in_seq=0 out_seq=0 2024-09-15T23:20:45.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.298+0000 7f9c8cff9640 10 client.26147 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:45.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9c8cff9640 1 -- 172.21.15.62:0/2251648288 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5635d6e42d70 con 0x5635d6f74d60 2024-09-15T23:20:45.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9c8cff9640 10 client.26147 handle_command_reply: tid=0 2024-09-15T23:20:45.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 client.26147 shutdown 2024-09-15T23:20:45.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed5780 msgr2=0x5635d6ee8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:45.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed5780 0x5635d6ee8660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00108e0 tx=0x7f9ca0005da0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 shutdown_connections 2024-09-15T23:20:45.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5635d6f74d60 0x5635d6f771c0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9c8cff9640 0 client.26147 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:45.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5635d6ee0520 0x5635d6ee5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5635d6ed6cf0 0x5635d6ee8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 --2- 172.21.15.62:0/2251648288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5635d6ed5780 0x5635d6ee8660 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 >> 172.21.15.62:0/2251648288 conn(0x5635d6e9e310 msgr2=0x5635d6edc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:45.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 shutdown_connections 2024-09-15T23:20:45.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 1 -- 172.21.15.62:0/2251648288 wait complete. 2024-09-15T23:20:45.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.299+0000 7f9cadb55740 20 client.26147 trim_cache size 0 max 16384 2024-09-15T23:20:45.169 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:20:45.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.300+0000 7f9caa51a640 1 -- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c900a3470 msgr2=0x7f9c900a5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:45.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.300+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c900a3470 0x7f9c900a5930 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40b3000 tx=0x7f9ca40b8000 comp rx=0 tx=0).stop 2024-09-15T23:20:45.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.300+0000 7f9caa51a640 1 -- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 msgr2=0x7f9c9c130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:45.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.300+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 0x7f9c9c130b80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f9c940098b0 tx=0x7f9c94009d70 comp rx=0 tx=0).stop 2024-09-15T23:20:45.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.300+0000 7f9caa51a640 1 -- 172.21.15.62:0/1708151383 shutdown_connections 2024-09-15T23:20:45.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.301+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c900a3470 0x7f9c900a5930 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.301+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c09a550 0x7f9c9c12a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.301+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c9c0952d0 0x7f9c9c130b80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.301+0000 7f9caa51a640 1 --2- 172.21.15.62:0/1708151383 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c9c094930 0x7f9c9c130640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:45.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.301+0000 7f9caa51a640 1 -- 172.21.15.62:0/1708151383 >> 172.21.15.62:0/1708151383 conn(0x7f9c9c00a1f0 msgr2=0x7f9c9c0931b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:45.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.301+0000 7f9caa51a640 1 -- 172.21.15.62:0/1708151383 shutdown_connections 2024-09-15T23:20:45.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:44.301+0000 7f9caa51a640 1 -- 172.21.15.62:0/1708151383 wait complete. 2024-09-15T23:20:45.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:45 smithi062 ceph-mon[30949]: pgmap v504: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 MiB/s wr, 865 op/s 2024-09-15T23:20:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:45 smithi067 ceph-mon[35665]: pgmap v504: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 MiB/s wr, 865 op/s 2024-09-15T23:20:45.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:45 smithi148 ceph-mon[35610]: pgmap v504: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 MiB/s wr, 865 op/s 2024-09-15T23:20:47.102 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:47.556 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:47 smithi062 ceph-mon[30949]: pgmap v505: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 652 op/s 2024-09-15T23:20:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:47 smithi067 ceph-mon[35665]: pgmap v505: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 652 op/s 2024-09-15T23:20:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:47 smithi148 ceph-mon[35610]: pgmap v505: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.5 MiB/s wr, 652 op/s 2024-09-15T23:20:47.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.976+0000 7f196e371640 1 -- 172.21.15.62:0/2448765459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1968105de0 msgr2=0x7f196810d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:47.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.976+0000 7f196e371640 1 --2- 172.21.15.62:0/2448765459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1968105de0 0x7f196810d960 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1960009ca0 tx=0x7f196002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:20:47.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.976+0000 7f196e371640 1 -- 172.21.15.62:0/2448765459 shutdown_connections 2024-09-15T23:20:47.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.976+0000 7f196e371640 1 --2- 172.21.15.62:0/2448765459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1968105de0 0x7f196810d960 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:47.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.976+0000 7f196e371640 1 --2- 172.21.15.62:0/2448765459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1968101e20 0x7f19681056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:47.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.976+0000 7f196e371640 1 --2- 172.21.15.62:0/2448765459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1968101470 0x7f1968101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:47.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.976+0000 7f196e371640 1 -- 172.21.15.62:0/2448765459 >> 172.21.15.62:0/2448765459 conn(0x7f1968077230 msgr2=0x7f19680fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:47.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.976+0000 7f196e371640 1 -- 172.21.15.62:0/2448765459 shutdown_connections 2024-09-15T23:20:47.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.976+0000 7f196e371640 1 -- 172.21.15.62:0/2448765459 wait complete. 2024-09-15T23:20:47.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.977+0000 7f196e371640 1 Processor -- start 2024-09-15T23:20:47.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.978+0000 7f196e371640 1 -- start start 2024-09-15T23:20:47.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f196e371640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1968101470 0x7f19681079e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:47.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f196e371640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1968101e20 0x7f1968107f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:47.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f196e371640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1968105de0 0x7f1968108460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:47.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f196e371640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1968110e40 con 0x7f1968101e20 2024-09-15T23:20:47.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f196e371640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1968110cc0 con 0x7f1968101470 2024-09-15T23:20:47.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f196e371640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1968110fc0 con 0x7f1968105de0 2024-09-15T23:20:47.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f196c90e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1968105de0 0x7f1968108460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:47.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f196c90e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1968105de0 0x7f1968108460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56308/0 (socket says 172.21.15.62:56308) 2024-09-15T23:20:47.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f196c90e640 1 -- 172.21.15.62:0/2826313730 learned_addr learned my addr 172.21.15.62:0/2826313730 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:47.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.979+0000 7f1967fff640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1968101470 0x7f19681079e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:47.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.980+0000 7f196c90e640 1 -- 172.21.15.62:0/2826313730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1968101470 msgr2=0x7f19681079e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:47.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.980+0000 7f196c90e640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1968101470 0x7f19681079e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:47.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.980+0000 7f196c90e640 1 -- 172.21.15.62:0/2826313730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1968101e20 msgr2=0x7f1968107f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:47.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.980+0000 7f196c90e640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1968101e20 0x7f1968107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:47.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.980+0000 7f196c90e640 1 -- 172.21.15.62:0/2826313730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19681aad00 con 0x7f1968105de0 2024-09-15T23:20:47.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.980+0000 7f196c90e640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1968105de0 0x7f1968108460 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1960031a20 tx=0x7f1960031a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:47.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.981+0000 7f19657fa640 1 -- 172.21.15.62:0/2826313730 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960003b70 con 0x7f1968105de0 2024-09-15T23:20:47.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.981+0000 7f19657fa640 1 -- 172.21.15.62:0/2826313730 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1960003d10 con 0x7f1968105de0 2024-09-15T23:20:47.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.981+0000 7f19657fa640 1 -- 172.21.15.62:0/2826313730 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196002fc80 con 0x7f1968105de0 2024-09-15T23:20:47.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.981+0000 7f196e371640 1 -- 172.21.15.62:0/2826313730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19681aaf90 con 0x7f1968105de0 2024-09-15T23:20:47.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.981+0000 7f196e371640 1 -- 172.21.15.62:0/2826313730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19681ab390 con 0x7f1968105de0 2024-09-15T23:20:47.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.986+0000 7f197074a740 1 Processor -- start 2024-09-15T23:20:47.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.992+0000 7f19657fa640 1 -- 172.21.15.62:0/2826313730 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1960004050 con 0x7f1968105de0 2024-09-15T23:20:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.993+0000 7f19657fa640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19500a3410 0x7f19500a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.994+0000 7f19657fa640 1 -- 172.21.15.62:0/2826313730 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f19600f0b10 con 0x7f1968105de0 2024-09-15T23:20:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.994+0000 7f19677fe640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19500a3410 0x7f19500a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:47.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.995+0000 7f197074a740 1 -- start start 2024-09-15T23:20:47.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.995+0000 7f197074a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f44850 0x562022f44c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:47.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.995+0000 7f197074a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f45170 0x562022f4f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:47.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.995+0000 7f197074a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f51e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:47.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.995+0000 7f197074a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562022ea9160 con 0x562022f45170 2024-09-15T23:20:47.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.995+0000 7f197074a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562022e50720 con 0x562022f4fa30 2024-09-15T23:20:47.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.995+0000 7f197074a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562022ea5120 con 0x562022f44850 2024-09-15T23:20:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f196c90e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f51e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f196c90e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f51e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:58794/0 (socket says 172.21.15.62:58794) 2024-09-15T23:20:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f196c90e640 1 -- 172.21.15.62:0/509317793 learned_addr learned my addr 172.21.15.62:0/509317793 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f19677fe640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19500a3410 0x7f19500a58d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f19681075f0 tx=0x7f195c009590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f1967fff640 1 --2- 172.21.15.62:0/509317793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f44850 0x562022f44c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f196c90e640 1 -- 172.21.15.62:0/509317793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f44850 msgr2=0x562022f44c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f196c90e640 1 --2- 172.21.15.62:0/509317793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f44850 0x562022f44c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:47.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f196c90e640 1 -- 172.21.15.62:0/509317793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f45170 msgr2=0x562022f4f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f196c90e640 1 --2- 172.21.15.62:0/509317793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f45170 0x562022f4f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.996+0000 7f196c90e640 1 -- 172.21.15.62:0/509317793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562022c1aba0 con 0x562022f4fa30 2024-09-15T23:20:47.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:47.997+0000 7f196c90e640 1 --2- 172.21.15.62:0/509317793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f51e20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f196002f8d0 tx=0x7f19600bca90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e439c58988d7fe5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:48.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.008+0000 7f194e7fc640 1 -- 172.21.15.62:0/509317793 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19600bce90 con 0x562022f4fa30 2024-09-15T23:20:48.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.008+0000 7f194e7fc640 1 -- 172.21.15.62:0/509317793 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1960036040 con 0x562022f4fa30 2024-09-15T23:20:48.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.009+0000 7f197074a740 1 -- 172.21.15.62:0/509317793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 msgr2=0x562022f51e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:48.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.009+0000 7f197074a740 1 --2- 172.21.15.62:0/509317793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f51e20 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f196002f8d0 tx=0x7f19600bca90 comp rx=0 tx=0).stop 2024-09-15T23:20:48.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.019+0000 7f197074a740 1 -- 172.21.15.62:0/509317793 shutdown_connections 2024-09-15T23:20:48.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.019+0000 7f197074a740 1 --2- 172.21.15.62:0/509317793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f51e20 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.019+0000 7f197074a740 1 --2- 172.21.15.62:0/509317793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f45170 0x562022f4f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.019+0000 7f197074a740 1 --2- 172.21.15.62:0/509317793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f44850 0x562022f44c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.019+0000 7f197074a740 1 -- 172.21.15.62:0/509317793 >> 172.21.15.62:0/509317793 conn(0x562022f40de0 msgr2=0x562022f411c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:48.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.020+0000 7f197074a740 1 -- 172.21.15.62:0/509317793 shutdown_connections 2024-09-15T23:20:48.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.020+0000 7f197074a740 1 -- 172.21.15.62:0/509317793 wait complete. 2024-09-15T23:20:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.020+0000 7f197074a740 1 -- start start 2024-09-15T23:20:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.021+0000 7f197074a740 1 Processor -- start 2024-09-15T23:20:48.026 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:20:48.027 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:20:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.021+0000 7f197074a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f4c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f197074a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f45170 0x562022f4cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f197074a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 0x562022f4d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f197074a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562022e50720 con 0x562022f44850 2024-09-15T23:20:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f197074a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562022e31900 con 0x562022f45170 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f197074a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562022f523c0 con 0x562022f4fa30 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f197074a740 10 client.0 _ll_register_callbacks cb 0x562022e2f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f19677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f45170 0x562022f4cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f19677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f45170 0x562022f4cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:58796/0 (socket says 172.21.15.62:58796) 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f19677fe640 1 -- 172.21.15.62:0/1172088650 learned_addr learned my addr 172.21.15.62:0/1172088650 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.022+0000 7f1967fff640 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f4c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.023+0000 7f1967fff640 1 -- 172.21.15.62:0/1172088650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 msgr2=0x562022f4d320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.023+0000 7f1967fff640 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 0x562022f4d320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.023+0000 7f1967fff640 1 -- 172.21.15.62:0/1172088650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f45170 msgr2=0x562022f4cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.023+0000 7f1967fff640 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f45170 0x562022f4cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.023+0000 7f1967fff640 1 -- 172.21.15.62:0/1172088650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562022e71c50 con 0x562022f44850 2024-09-15T23:20:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.023+0000 7f1967fff640 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f4c8a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f19580134c0 tx=0x7f1958013980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c183d40265b2b9f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.023+0000 7f19477fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.024+0000 7f19477fe640 1 -- 172.21.15.62:0/1172088650 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195803b070 con 0x562022f44850 2024-09-15T23:20:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.024+0000 7f197074a740 1 -- 172.21.15.62:0/1172088650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1960036040 con 0x562022f44850 2024-09-15T23:20:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.024+0000 7f19477fe640 1 -- 172.21.15.62:0/1172088650 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1958004220 con 0x562022f44850 2024-09-15T23:20:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.024+0000 7f19477fe640 1 -- 172.21.15.62:0/1172088650 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1958004420 con 0x562022f44850 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.026+0000 7f197074a740 10 client.16536 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.026+0000 7f197074a740 1 -- 172.21.15.62:0/1172088650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562022f5b7d0 con 0x562022f44850 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.027+0000 7f19477fe640 1 -- 172.21.15.62:0/1172088650 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1960036040 con 0x562022f44850 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.027+0000 7f19477fe640 1 -- 172.21.15.62:0/1172088650 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f1958003b20 con 0x562022f44850 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.027+0000 7f197074a740 10 client.16536 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.027+0000 7f197074a740 10 client.16536 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.027+0000 7f197074a740 20 client.16536 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.027+0000 7f197074a740 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562022f5d0e0 0x562022f5f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.027+0000 7f197074a740 4 client.16536 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.027+0000 7f197074a740 1 -- 172.21.15.62:0/1172088650 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562022eb6890 con 0x562022f5d0e0 2024-09-15T23:20:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.028+0000 7f19677fe640 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562022f5d0e0 0x562022f5f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.032+0000 7f19677fe640 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562022f5d0e0 0x562022f5f580 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5004d5699a9e549 server_cookie=91d2f8651c19dd51 in_seq=0 out_seq=0 2024-09-15T23:20:48.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.032+0000 7f19477fe640 10 client.16536 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:48.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.034+0000 7f19477fe640 1 -- 172.21.15.62:0/1172088650 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562022eb6890 con 0x562022f5d0e0 2024-09-15T23:20:48.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.034+0000 7f19477fe640 10 client.16536 handle_command_reply: tid=0 2024-09-15T23:20:48.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.034+0000 7f197074a740 1 client.16536 shutdown 2024-09-15T23:20:48.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.035+0000 7f197074a740 1 -- 172.21.15.62:0/1172088650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 msgr2=0x562022f4c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:48.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.035+0000 7f197074a740 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f4c8a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f19580134c0 tx=0x7f1958013980 comp rx=0 tx=0).stop 2024-09-15T23:20:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.040+0000 7f197074a740 1 -- 172.21.15.62:0/1172088650 shutdown_connections 2024-09-15T23:20:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.040+0000 7f197074a740 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562022f5d0e0 0x562022f5f580 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.040+0000 7f197074a740 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 0x562022f4d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.040+0000 7f197074a740 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f45170 0x562022f4cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.040+0000 7f197074a740 1 --2- 172.21.15.62:0/1172088650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f4c8a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.040+0000 7f197074a740 1 -- 172.21.15.62:0/1172088650 >> 172.21.15.62:0/1172088650 conn(0x562022f40de0 msgr2=0x562022f46b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:48.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.040+0000 7f19477fe640 0 client.16536 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:48.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.041+0000 7f197074a740 1 -- 172.21.15.62:0/1172088650 shutdown_connections 2024-09-15T23:20:48.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.045+0000 7f197074a740 1 -- 172.21.15.62:0/1172088650 wait complete. 2024-09-15T23:20:48.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.046+0000 7f197074a740 20 client.16536 trim_cache size 0 max 16384 2024-09-15T23:20:48.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.078+0000 7f197074a740 1 Processor -- start 2024-09-15T23:20:48.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.078+0000 7f197074a740 1 -- start start 2024-09-15T23:20:48.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.078+0000 7f197074a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f44850 0x562022f5f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.078+0000 7f197074a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f45170 0x562022f64e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.078+0000 7f197074a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 0x562022f653d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.078+0000 7f197074a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1954012520 con 0x562022f45170 2024-09-15T23:20:48.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.078+0000 7f197074a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f195400fdf0 con 0x562022f44850 2024-09-15T23:20:48.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.078+0000 7f197074a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f195400dc40 con 0x562022f4fa30 2024-09-15T23:20:48.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.078+0000 7f196c90e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 0x562022f653d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.079+0000 7f196c90e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 0x562022f653d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56326/0 (socket says 172.21.15.62:56326) 2024-09-15T23:20:48.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.079+0000 7f196c90e640 1 -- 172.21.15.62:0/480915293 learned_addr learned my addr 172.21.15.62:0/480915293 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:48.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.079+0000 7f1967fff640 1 --2- 172.21.15.62:0/480915293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f44850 0x562022f5f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.079+0000 7f19677fe640 1 --2- 172.21.15.62:0/480915293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f45170 0x562022f64e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.080+0000 7f196c90e640 1 -- 172.21.15.62:0/480915293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f44850 msgr2=0x562022f5f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.080+0000 7f196c90e640 1 --2- 172.21.15.62:0/480915293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f44850 0x562022f5f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.080+0000 7f196c90e640 1 -- 172.21.15.62:0/480915293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f45170 msgr2=0x562022f64e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.080+0000 7f196c90e640 1 --2- 172.21.15.62:0/480915293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f45170 0x562022f64e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.080+0000 7f196c90e640 1 -- 172.21.15.62:0/480915293 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562022f44fa0 con 0x562022f4fa30 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.080+0000 7f196c90e640 1 --2- 172.21.15.62:0/480915293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 0x562022f653d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1960005e30 tx=0x7f196002f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7b4ea154e17c668 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.080+0000 7f194dffb640 1 -- 172.21.15.62:0/480915293 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19600bce90 con 0x562022f4fa30 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.080+0000 7f194dffb640 1 -- 172.21.15.62:0/480915293 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1960036030 con 0x562022f4fa30 2024-09-15T23:20:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.080+0000 7f194dffb640 1 -- 172.21.15.62:0/480915293 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19600ee340 con 0x562022f4fa30 2024-09-15T23:20:48.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.081+0000 7f197074a740 1 -- 172.21.15.62:0/480915293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 msgr2=0x562022f653d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:48.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.081+0000 7f197074a740 1 --2- 172.21.15.62:0/480915293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 0x562022f653d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1960005e30 tx=0x7f196002f710 comp rx=0 tx=0).stop 2024-09-15T23:20:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.081+0000 7f197074a740 1 -- 172.21.15.62:0/480915293 shutdown_connections 2024-09-15T23:20:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.081+0000 7f197074a740 1 --2- 172.21.15.62:0/480915293 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f4fa30 0x562022f653d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.081+0000 7f197074a740 1 --2- 172.21.15.62:0/480915293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f45170 0x562022f64e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.081+0000 7f197074a740 1 --2- 172.21.15.62:0/480915293 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f44850 0x562022f5f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.081+0000 7f197074a740 1 -- 172.21.15.62:0/480915293 >> 172.21.15.62:0/480915293 conn(0x562022f40de0 msgr2=0x562022f4b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.082+0000 7f197074a740 1 -- 172.21.15.62:0/480915293 shutdown_connections 2024-09-15T23:20:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.082+0000 7f197074a740 1 -- 172.21.15.62:0/480915293 wait complete. 2024-09-15T23:20:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.082+0000 7f197074a740 1 -- start start 2024-09-15T23:20:48.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.082+0000 7f197074a740 1 Processor -- start 2024-09-15T23:20:48.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f197074a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f578d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f197074a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f45170 0x562022f57e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f197074a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f197074a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f195400e2b0 con 0x562022f44850 2024-09-15T23:20:48.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f197074a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f195400f720 con 0x562022f4fa30 2024-09-15T23:20:48.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f197074a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f195400e980 con 0x562022f45170 2024-09-15T23:20:48.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f197074a740 10 client.0 _ll_register_callbacks cb 0x7f1958009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:48.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f1967fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f19677fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f45170 0x562022f57e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f1967fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f578d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54810/0 (socket says 172.21.15.62:54810) 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f1967fff640 1 -- 172.21.15.62:0/2443314984 learned_addr learned my addr 172.21.15.62:0/2443314984 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.083+0000 7f196c90e640 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f196c90e640 1 -- 172.21.15.62:0/2443314984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f45170 msgr2=0x562022f57e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f196c90e640 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f45170 0x562022f57e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f196c90e640 1 -- 172.21.15.62:0/2443314984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 msgr2=0x562022f578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f196c90e640 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f196c90e640 1 -- 172.21.15.62:0/2443314984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562022f52d40 con 0x562022f4fa30 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f1967fff640 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f196c90e640 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f526a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x562022f5cf90 tx=0x7f19600bb190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95074d5450b0fc56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:48.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f19477fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:20:48.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f19477fe640 1 -- 172.21.15.62:0/2443314984 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19600c59c0 con 0x562022f4fa30 2024-09-15T23:20:48.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.084+0000 7f19477fe640 1 -- 172.21.15.62:0/2443314984 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f19600ee340 con 0x562022f4fa30 2024-09-15T23:20:48.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.085+0000 7f197074a740 1 -- 172.21.15.62:0/2443314984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1960036030 con 0x562022f4fa30 2024-09-15T23:20:48.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.085+0000 7f19477fe640 1 -- 172.21.15.62:0/2443314984 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f19600bce90 con 0x562022f4fa30 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.085+0000 7f197074a740 10 client.26165 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.085+0000 7f197074a740 1 -- 172.21.15.62:0/2443314984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562022fc9aa0 con 0x562022f4fa30 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.085+0000 7f19477fe640 1 -- 172.21.15.62:0/2443314984 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1960036030 con 0x562022f4fa30 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.086+0000 7f19477fe640 1 -- 172.21.15.62:0/2443314984 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f19600b8c70 con 0x562022f4fa30 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.086+0000 7f197074a740 10 client.26165 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.086+0000 7f197074a740 10 client.26165 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.086+0000 7f197074a740 20 client.26165 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.086+0000 7f197074a740 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562022fc9f40 0x562022fcc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.086+0000 7f197074a740 4 client.26165 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}] 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.086+0000 7f197074a740 1 -- 172.21.15.62:0/2443314984 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}) -- 0x562022eb0be0 con 0x562022fc9f40 2024-09-15T23:20:48.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.086+0000 7f1967fff640 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562022fc9f40 0x562022fcc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:48.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.087+0000 7f19477fe640 10 client.26165 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:48.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.087+0000 7f1967fff640 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562022fc9f40 0x562022fcc330 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6454943cc84f7e25 server_cookie=4b9dc16b1594fe17 in_seq=0 out_seq=0 2024-09-15T23:20:48.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f19477fe640 1 -- 172.21.15.62:0/2443314984 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x562022eb0be0 con 0x562022fc9f40 2024-09-15T23:20:48.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f19477fe640 10 client.26165 handle_command_reply: tid=0 2024-09-15T23:20:48.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f197074a740 1 client.26165 shutdown 2024-09-15T23:20:48.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f197074a740 1 -- 172.21.15.62:0/2443314984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 msgr2=0x562022f526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f197074a740 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f526a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x562022f5cf90 tx=0x7f19600bb190 comp rx=0 tx=0).stop 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f197074a740 1 -- 172.21.15.62:0/2443314984 shutdown_connections 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f197074a740 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562022fc9f40 0x562022fcc330 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f197074a740 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562022f4fa30 0x562022f526a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f197074a740 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562022f45170 0x562022f57e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f197074a740 1 --2- 172.21.15.62:0/2443314984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562022f44850 0x562022f578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f197074a740 1 -- 172.21.15.62:0/2443314984 >> 172.21.15.62:0/2443314984 conn(0x562022f40de0 msgr2=0x562022f46270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.370+0000 7f19477fe640 0 client.26165 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.371+0000 7f197074a740 1 -- 172.21.15.62:0/2443314984 shutdown_connections 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.371+0000 7f197074a740 1 -- 172.21.15.62:0/2443314984 wait complete. 2024-09-15T23:20:48.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.371+0000 7f197074a740 20 client.26165 trim_cache size 0 max 16384 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 31, 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 18, 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout: "61932369": { 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 18, 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:20:48.374 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 145.3, 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:48.375 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:48.376 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:48.376 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:20:48.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 -- 172.21.15.62:0/2826313730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19500a3410 msgr2=0x7f19500a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:48.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19500a3410 0x7f19500a58d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f19681075f0 tx=0x7f195c009590 comp rx=0 tx=0).stop 2024-09-15T23:20:48.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 -- 172.21.15.62:0/2826313730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1968105de0 msgr2=0x7f1968108460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:48.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1968105de0 0x7f1968108460 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1960031a20 tx=0x7f1960031a50 comp rx=0 tx=0).stop 2024-09-15T23:20:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 -- 172.21.15.62:0/2826313730 shutdown_connections 2024-09-15T23:20:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19500a3410 0x7f19500a58d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1968105de0 0x7f1968108460 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1968101e20 0x7f1968107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 --2- 172.21.15.62:0/2826313730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1968101470 0x7f19681079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.372+0000 7f196e371640 1 -- 172.21.15.62:0/2826313730 >> 172.21.15.62:0/2826313730 conn(0x7f1968077230 msgr2=0x7f1968100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.373+0000 7f196e371640 1 -- 172.21.15.62:0/2826313730 shutdown_connections 2024-09-15T23:20:48.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:48.373+0000 7f196e371640 1 -- 172.21.15.62:0/2826313730 wait complete. 2024-09-15T23:20:48.657 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:49.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.123+0000 7f747aa97640 1 -- 172.21.15.62:0/320752941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474100a80 msgr2=0x7f7474100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:49.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.123+0000 7f747aa97640 1 --2- 172.21.15.62:0/320752941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474100a80 0x7f7474100ee0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f745c009930 tx=0x7f745c02a420 comp rx=0 tx=0).stop 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 -- 172.21.15.62:0/320752941 shutdown_connections 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 --2- 172.21.15.62:0/320752941 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7474101420 0x7f747410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 --2- 172.21.15.62:0/320752941 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474100a80 0x7f7474100ee0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 --2- 172.21.15.62:0/320752941 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474106a80 0x7f7474106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 -- 172.21.15.62:0/320752941 >> 172.21.15.62:0/320752941 conn(0x7f74740fc7a0 msgr2=0x7f74740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 -- 172.21.15.62:0/320752941 shutdown_connections 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 -- 172.21.15.62:0/320752941 wait complete. 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 Processor -- start 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 -- start start 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7474100a80 0x7f74741992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474101420 0x7f74741997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474106a80 0x7f74741a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7474110e80 con 0x7f7474106a80 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7474110d00 con 0x7f7474101420 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.124+0000 7f747aa97640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7474111000 con 0x7f7474100a80 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7479034640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474106a80 0x7f74741a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7479034640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474106a80 0x7f74741a4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54828/0 (socket says 172.21.15.62:54828) 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7479034640 1 -- 172.21.15.62:0/3235055902 learned_addr learned my addr 172.21.15.62:0/3235055902 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7473fff640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474101420 0x7f74741997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7478833640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7474100a80 0x7f74741992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7479034640 1 -- 172.21.15.62:0/3235055902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7474100a80 msgr2=0x7f74741992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7479034640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7474100a80 0x7f74741992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7479034640 1 -- 172.21.15.62:0/3235055902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474101420 msgr2=0x7f74741997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7479034640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474101420 0x7f74741997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7479034640 1 -- 172.21.15.62:0/3235055902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74741a4870 con 0x7f7474106a80 2024-09-15T23:20:49.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7479034640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474106a80 0x7f74741a4330 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f746400cce0 tx=0x7f746400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7471ffb640 1 -- 172.21.15.62:0/3235055902 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746400f450 con 0x7f7474106a80 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7471ffb640 1 -- 172.21.15.62:0/3235055902 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746400fad0 con 0x7f7474106a80 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f747aa97640 1 -- 172.21.15.62:0/3235055902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74741a4a40 con 0x7f7474106a80 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.125+0000 7f7471ffb640 1 -- 172.21.15.62:0/3235055902 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464018c10 con 0x7f7474106a80 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.126+0000 7f747aa97640 1 -- 172.21.15.62:0/3235055902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74741a4f00 con 0x7f7474106a80 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.128+0000 7f7471ffb640 1 -- 172.21.15.62:0/3235055902 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f746400f5f0 con 0x7f7474106a80 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f747ce70740 1 Processor -- start 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f747ce70740 1 -- start start 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f747ce70740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 0x55a424364f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f747ce70740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424365480 0x55a42436f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f747ce70740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a424372130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f747ce70740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a4242ca280 con 0x55a424364b60 2024-09-15T23:20:49.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f747ce70740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a424271b10 con 0x55a424365480 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f747ce70740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a4242c6240 con 0x55a42436fd40 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7473fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 0x55a424364f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7473fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 0x55a424364f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54842/0 (socket says 172.21.15.62:54842) 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7473fff640 1 -- 172.21.15.62:0/504263056 learned_addr learned my addr 172.21.15.62:0/504263056 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7478833640 1 --2- 172.21.15.62:0/504263056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424365480 0x55a42436f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7471ffb640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74600a3240 0x7f74600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7473fff640 1 --2- 172.21.15.62:0/504263056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a424372130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7473fff640 1 -- 172.21.15.62:0/504263056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 msgr2=0x55a424372130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7473fff640 1 --2- 172.21.15.62:0/504263056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a424372130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7473fff640 1 -- 172.21.15.62:0/504263056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424365480 msgr2=0x55a42436f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7473fff640 1 --2- 172.21.15.62:0/504263056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424365480 0x55a42436f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.129+0000 7f7473fff640 1 -- 172.21.15.62:0/504263056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a42427e1f0 con 0x55a424364b60 2024-09-15T23:20:49.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.130+0000 7f7479034640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74600a3240 0x7f74600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.130+0000 7f7471ffb640 1 -- 172.21.15.62:0/3235055902 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f74640d0380 con 0x7f7474106a80 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.130+0000 7f7473fff640 1 --2- 172.21.15.62:0/504263056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 0x55a424364f40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f745c02a940 tx=0x7f745c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26381a87c1f7861 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.130+0000 7f746affd640 1 -- 172.21.15.62:0/504263056 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c044030 con 0x55a424364b60 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.130+0000 7f746affd640 1 -- 172.21.15.62:0/504263056 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f745c005020 con 0x55a424364b60 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.130+0000 7f746affd640 1 -- 172.21.15.62:0/504263056 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c035040 con 0x55a424364b60 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 -- 172.21.15.62:0/504263056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 msgr2=0x55a424364f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 --2- 172.21.15.62:0/504263056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 0x55a424364f40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f745c02a940 tx=0x7f745c000bc0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f7479034640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74600a3240 0x7f74600a5700 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f746400ab20 tx=0x7f746401baf0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 -- 172.21.15.62:0/504263056 shutdown_connections 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 --2- 172.21.15.62:0/504263056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a424372130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 --2- 172.21.15.62:0/504263056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424365480 0x55a42436f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 --2- 172.21.15.62:0/504263056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 0x55a424364f40 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 -- 172.21.15.62:0/504263056 >> 172.21.15.62:0/504263056 conn(0x55a4243610f0 msgr2=0x55a4243614d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 -- 172.21.15.62:0/504263056 shutdown_connections 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 -- 172.21.15.62:0/504263056 wait complete. 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 -- start start 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.131+0000 7f747ce70740 1 Processor -- start 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f747ce70740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 0x55a42436f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f747ce70740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a42436cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f747ce70740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a42436d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f747ce70740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a424372910 con 0x55a424365480 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f747ce70740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a424284f10 con 0x55a424364b60 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f747ce70740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a424372a90 con 0x55a42436fd40 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f747ce70740 10 client.0 _ll_register_callbacks cb 0x55a42424da40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7478833640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a42436cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7478833640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a42436cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54854/0 (socket says 172.21.15.62:54854) 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7478833640 1 -- 172.21.15.62:0/3362642116 learned_addr learned my addr 172.21.15.62:0/3362642116 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7473fff640 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 0x55a42436f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7473fff640 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a42436d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7478833640 1 -- 172.21.15.62:0/3362642116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 msgr2=0x55a42436d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7478833640 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a42436d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7478833640 1 -- 172.21.15.62:0/3362642116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 msgr2=0x55a42436f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7478833640 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 0x55a42436f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7478833640 1 -- 172.21.15.62:0/3362642116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a42403b6a0 con 0x55a424365480 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7478833640 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a42436cae0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f746c00be70 tx=0x7f746c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b312f992793bcd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.132+0000 7f7453fff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f7453fff640 1 -- 172.21.15.62:0/3362642116 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c01f060 con 0x55a424365480 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f7453fff640 1 -- 172.21.15.62:0/3362642116 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f746c0108a0 con 0x55a424365480 2024-09-15T23:20:49.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f747ce70740 1 -- 172.21.15.62:0/3362642116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f745c005020 con 0x55a424365480 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f7453fff640 1 -- 172.21.15.62:0/3362642116 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f746c026050 con 0x55a424365480 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f747ce70740 10 client.16554 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f747ce70740 1 -- 172.21.15.62:0/3362642116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a42437e4a0 con 0x55a424365480 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f7453fff640 1 -- 172.21.15.62:0/3362642116 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f745c005020 con 0x55a424365480 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f7453fff640 1 -- 172.21.15.62:0/3362642116 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f746c019030 con 0x55a424365480 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f747ce70740 10 client.16554 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f747ce70740 10 client.16554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f747ce70740 20 client.16554 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f747ce70740 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a42437f960 0x55a424381db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f747ce70740 4 client.16554 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.133+0000 7f747ce70740 1 -- 172.21.15.62:0/3362642116 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4242d77f0 con 0x55a42437f960 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.134+0000 7f7473fff640 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a42437f960 0x55a424381db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.134+0000 7f7473fff640 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a42437f960 0x55a424381db0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69b248b7c8c0a034 server_cookie=8d8751e42807b49f in_seq=0 out_seq=0 2024-09-15T23:20:49.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.134+0000 7f7453fff640 10 client.16554 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f7453fff640 1 -- 172.21.15.62:0/3362642116 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a4242d77f0 con 0x55a42437f960 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f7453fff640 10 client.16554 handle_command_reply: tid=0 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 client.16554 shutdown 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 -- 172.21.15.62:0/3362642116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 msgr2=0x55a42436cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a42436cae0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f746c00be70 tx=0x7f746c012820 comp rx=0 tx=0).stop 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 -- 172.21.15.62:0/3362642116 shutdown_connections 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a42437f960 0x55a424381db0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a42436d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a42436cae0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 --2- 172.21.15.62:0/3362642116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 0x55a42436f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f7453fff640 0 client.16554 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 -- 172.21.15.62:0/3362642116 >> 172.21.15.62:0/3362642116 conn(0x55a4243610f0 msgr2=0x55a424366e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 -- 172.21.15.62:0/3362642116 shutdown_connections 2024-09-15T23:20:49.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 1 -- 172.21.15.62:0/3362642116 wait complete. 2024-09-15T23:20:49.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.135+0000 7f747ce70740 20 client.16554 trim_cache size 0 max 16384 2024-09-15T23:20:49.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f747ce70740 1 Processor -- start 2024-09-15T23:20:49.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f747ce70740 1 -- start start 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f747ce70740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 0x55a4243876c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f747ce70740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a424365480 0x55a424387c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f747ce70740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a42436fd40 0x55a424369d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f747ce70740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f74741b1490 con 0x55a424364b60 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f747ce70740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74741aed60 con 0x55a42436fd40 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f747ce70740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f74741acbb0 con 0x55a424365480 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f7478833640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a424365480 0x55a424387c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f7478833640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a424365480 0x55a424387c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56384/0 (socket says 172.21.15.62:56384) 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.153+0000 7f7478833640 1 -- 172.21.15.62:0/327895987 learned_addr learned my addr 172.21.15.62:0/327895987 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.154+0000 7f7478833640 1 -- 172.21.15.62:0/327895987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a42436fd40 msgr2=0x55a424369d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.154+0000 7f7478833640 1 --2- 172.21.15.62:0/327895987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a42436fd40 0x55a424369d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.154+0000 7f7478833640 1 -- 172.21.15.62:0/327895987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 msgr2=0x55a4243876c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.154+0000 7f7478833640 1 --2- 172.21.15.62:0/327895987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 0x55a4243876c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.154+0000 7f7478833640 1 -- 172.21.15.62:0/327895987 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a424382430 con 0x55a424365480 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.154+0000 7f7478833640 1 --2- 172.21.15.62:0/327895987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a424365480 0x55a424387c00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f746c00b8b0 tx=0x7f746c00ffb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38dfa25ae0bc4a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.154+0000 7f746a7fc640 1 -- 172.21.15.62:0/327895987 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c02f070 con 0x55a424365480 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f746a7fc640 1 -- 172.21.15.62:0/327895987 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746c005550 con 0x55a424365480 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f746a7fc640 1 -- 172.21.15.62:0/327895987 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c00f920 con 0x55a424365480 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f747ce70740 1 -- 172.21.15.62:0/327895987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a424365480 msgr2=0x55a424387c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f747ce70740 1 --2- 172.21.15.62:0/327895987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a424365480 0x55a424387c00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f746c00b8b0 tx=0x7f746c00ffb0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f747ce70740 1 -- 172.21.15.62:0/327895987 shutdown_connections 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f747ce70740 1 --2- 172.21.15.62:0/327895987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a42436fd40 0x55a424369d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f747ce70740 1 --2- 172.21.15.62:0/327895987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a424365480 0x55a424387c00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f747ce70740 1 --2- 172.21.15.62:0/327895987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424364b60 0x55a4243876c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f747ce70740 1 -- 172.21.15.62:0/327895987 >> 172.21.15.62:0/327895987 conn(0x55a4243610f0 msgr2=0x55a42436b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f747ce70740 1 -- 172.21.15.62:0/327895987 shutdown_connections 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.155+0000 7f747ce70740 1 -- 172.21.15.62:0/327895987 wait complete. 2024-09-15T23:20:49.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f747ce70740 1 -- start start 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f747ce70740 1 Processor -- start 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f747ce70740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 0x55a4243e7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f747ce70740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a4243e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f747ce70740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a4243e4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f747ce70740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f74741ad220 con 0x55a424365480 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f747ce70740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74741ae690 con 0x55a424364b60 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f747ce70740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f74741ad8f0 con 0x55a42436fd40 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f747ce70740 10 client.0 _ll_register_callbacks cb 0x7f746c00bab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f7478833640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a4243e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f7478833640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a4243e7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54872/0 (socket says 172.21.15.62:54872) 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.156+0000 7f7478833640 1 -- 172.21.15.62:0/746336911 learned_addr learned my addr 172.21.15.62:0/746336911 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7473fff640 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a4243e4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7473fff640 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 0x55a4243e7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7473fff640 1 -- 172.21.15.62:0/746336911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 msgr2=0x55a4243e7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7473fff640 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 0x55a4243e7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7473fff640 1 -- 172.21.15.62:0/746336911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 msgr2=0x55a4243e7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7473fff640 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a4243e7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7473fff640 1 -- 172.21.15.62:0/746336911 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4243621a0 con 0x55a42436fd40 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7478833640 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a4243e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7473fff640 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a4243e4660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55a424382630 tx=0x7f745c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8b7de369bdf7da1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7453fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7453fff640 1 -- 172.21.15.62:0/746336911 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c035040 con 0x55a42436fd40 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f747ce70740 1 -- 172.21.15.62:0/746336911 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f746c005550 con 0x55a42436fd40 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7453fff640 1 -- 172.21.15.62:0/746336911 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f745c0042d0 con 0x55a42436fd40 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f7453fff640 1 -- 172.21.15.62:0/746336911 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f745c0044d0 con 0x55a42436fd40 2024-09-15T23:20:49.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f747ce70740 10 client.26008 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.157+0000 7f747ce70740 1 -- 172.21.15.62:0/746336911 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a424360f20 con 0x55a42436fd40 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.158+0000 7f7453fff640 1 -- 172.21.15.62:0/746336911 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746c005550 con 0x55a42436fd40 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.158+0000 7f7453fff640 1 -- 172.21.15.62:0/746336911 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f745c004cd0 con 0x55a42436fd40 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.158+0000 7f747ce70740 10 client.26008 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.158+0000 7f747ce70740 10 client.26008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.158+0000 7f747ce70740 20 client.26008 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.158+0000 7f747ce70740 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a424377580 0x55a424377990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.158+0000 7f747ce70740 4 client.26008 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.158+0000 7f747ce70740 1 -- 172.21.15.62:0/746336911 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a4242d1d00 con 0x55a424377580 2024-09-15T23:20:49.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.158+0000 7f7478833640 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a424377580 0x55a424377990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.159+0000 7f7478833640 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a424377580 0x55a424377990 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed059ec963e4c802 server_cookie=ea290b37b09e66de in_seq=0 out_seq=0 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.159+0000 7f7453fff640 10 client.26008 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f7453fff640 1 -- 172.21.15.62:0/746336911 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a4242d1d00 con 0x55a424377580 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f7453fff640 10 client.26008 handle_command_reply: tid=0 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 client.26008 shutdown 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 -- 172.21.15.62:0/746336911 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 msgr2=0x55a4243e4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a4243e4660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55a424382630 tx=0x7f745c005fd0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 -- 172.21.15.62:0/746336911 shutdown_connections 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a424377580 0x55a424377990 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a42436fd40 0x55a4243e4660 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a424365480 0x55a4243e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 --2- 172.21.15.62:0/746336911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a424364b60 0x55a4243e7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 -- 172.21.15.62:0/746336911 >> 172.21.15.62:0/746336911 conn(0x55a4243610f0 msgr2=0x55a42435fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:49.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f7453fff640 0 client.26008 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:49.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 -- 172.21.15.62:0/746336911 shutdown_connections 2024-09-15T23:20:49.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 1 -- 172.21.15.62:0/746336911 wait complete. 2024-09-15T23:20:49.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.160+0000 7f747ce70740 20 client.26008 trim_cache size 0 max 16384 2024-09-15T23:20:49.168 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:20:49.168 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:20:49.168 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "2d3e0b26-9e50-4216-b137-9f34307e997a", 2024-09-15T23:20:49.168 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:20:49.168 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:20:49.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 -- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74600a3240 msgr2=0x7f74600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:49.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74600a3240 0x7f74600a5700 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f746400ab20 tx=0x7f746401baf0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 -- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474106a80 msgr2=0x7f74741a4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:49.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474106a80 0x7f74741a4330 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f746400cce0 tx=0x7f746400a620 comp rx=0 tx=0).stop 2024-09-15T23:20:49.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 -- 172.21.15.62:0/3235055902 shutdown_connections 2024-09-15T23:20:49.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f74600a3240 0x7f74600a5700 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474106a80 0x7f74741a4330 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474101420 0x7f74741997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 --2- 172.21.15.62:0/3235055902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7474100a80 0x7f74741992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 -- 172.21.15.62:0/3235055902 >> 172.21.15.62:0/3235055902 conn(0x7f74740fc7a0 msgr2=0x7f74740fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 -- 172.21.15.62:0/3235055902 shutdown_connections 2024-09-15T23:20:49.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:49.161+0000 7f747aa97640 1 -- 172.21.15.62:0/3235055902 wait complete. 2024-09-15T23:20:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 270.0 s, 1050.0 tps, lat 73.472 ms stddev 91.109 2024-09-15T23:20:49.576 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:49 smithi067 ceph-mon[35665]: pgmap v506: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 MiB/s wr, 582 op/s 2024-09-15T23:20:49.591 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:49 smithi148 ceph-mon[35610]: pgmap v506: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 MiB/s wr, 582 op/s 2024-09-15T23:20:49.660 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:49 smithi062 ceph-mon[30949]: pgmap v506: 129 pgs: 129 active+clean; 1.1 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.0 MiB/s wr, 582 op/s 2024-09-15T23:20:49.688 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '61932369', quiesce took 10.2 seconds. Will release after: 15.150000000000002 2024-09-15T23:20:50.361 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:20:50.621 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:20:50.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:20:50.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:20:50.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:20:50.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:20:50.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:20:50.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:50 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:20:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:20:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:20:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:20:50.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:20:50.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:50 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:20:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:20:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:20:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:20:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:20:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:50.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:50 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:20:51.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.109+0000 7efe6c971640 1 -- 172.21.15.62:0/2530577681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681036e0 msgr2=0x7efe68107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:51.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.109+0000 7efe6c971640 1 --2- 172.21.15.62:0/2530577681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681036e0 0x7efe68107b90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7efe50009930 tx=0x7efe5002a420 comp rx=0 tx=0).stop 2024-09-15T23:20:51.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.109+0000 7efe6c971640 1 -- 172.21.15.62:0/2530577681 shutdown_connections 2024-09-15T23:20:51.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.109+0000 7efe6c971640 1 --2- 172.21.15.62:0/2530577681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe681082a0 0x7efe6810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.109+0000 7efe6c971640 1 --2- 172.21.15.62:0/2530577681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681036e0 0x7efe68107b90 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.109+0000 7efe6c971640 1 --2- 172.21.15.62:0/2530577681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe68102d30 0x7efe68103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.109+0000 7efe6c971640 1 -- 172.21.15.62:0/2530577681 >> 172.21.15.62:0/2530577681 conn(0x7efe680fc7d0 msgr2=0x7efe680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:51.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.109+0000 7efe6c971640 1 -- 172.21.15.62:0/2530577681 shutdown_connections 2024-09-15T23:20:51.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.110+0000 7efe6c971640 1 -- 172.21.15.62:0/2530577681 wait complete. 2024-09-15T23:20:51.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.110+0000 7efe6c971640 1 Processor -- start 2024-09-15T23:20:51.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.110+0000 7efe6c971640 1 -- start start 2024-09-15T23:20:51.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.110+0000 7efe6c971640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe68102d30 0x7efe68100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.110+0000 7efe6c971640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe681036e0 0x7efe68100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe6c971640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681082a0 0x7efe68101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe6c971640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efe68112090 con 0x7efe681082a0 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe6c971640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efe68111f10 con 0x7efe68102d30 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe6c971640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efe68112210 con 0x7efe681036e0 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681082a0 0x7efe68101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681082a0 0x7efe68101410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54888/0 (socket says 172.21.15.62:54888) 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe66d9d640 1 -- 172.21.15.62:0/3179036803 learned_addr learned my addr 172.21.15.62:0/3179036803 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe6659c640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe68102d30 0x7efe68100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe65d9b640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe681036e0 0x7efe68100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe66d9d640 1 -- 172.21.15.62:0/3179036803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe681036e0 msgr2=0x7efe68100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:51.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe66d9d640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe681036e0 0x7efe68100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe66d9d640 1 -- 172.21.15.62:0/3179036803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe68102d30 msgr2=0x7efe68100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:51.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe66d9d640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe68102d30 0x7efe68100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe66d9d640 1 -- 172.21.15.62:0/3179036803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe681a69f0 con 0x7efe681082a0 2024-09-15T23:20:51.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe6659c640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe68102d30 0x7efe68100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe66d9d640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681082a0 0x7efe68101410 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7efe6000e970 tx=0x7efe6000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe5f7fe640 1 -- 172.21.15.62:0/3179036803 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe6000ccc0 con 0x7efe681082a0 2024-09-15T23:20:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe5f7fe640 1 -- 172.21.15.62:0/3179036803 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe6000ce60 con 0x7efe681082a0 2024-09-15T23:20:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.111+0000 7efe6c971640 1 -- 172.21.15.62:0/3179036803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe681a6ce0 con 0x7efe681082a0 2024-09-15T23:20:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.112+0000 7efe5f7fe640 1 -- 172.21.15.62:0/3179036803 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe60010490 con 0x7efe681082a0 2024-09-15T23:20:51.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.112+0000 7efe6c971640 1 -- 172.21.15.62:0/3179036803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe681a71f0 con 0x7efe681082a0 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.114+0000 7efe5f7fe640 1 -- 172.21.15.62:0/3179036803 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7efe60005020 con 0x7efe681082a0 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.114+0000 7efe6ed4a740 1 Processor -- start 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.114+0000 7efe6ed4a740 1 -- start start 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe5f7fe640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efe540a32c0 0x7efe540a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d84a0 0x55b7990d8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d8dc0 0x55b7990e3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe5f7fe640 1 -- 172.21.15.62:0/3179036803 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7efe600d01e0 con 0x7efe681082a0 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 0x55b7990e5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b79903e2e0 con 0x55b7990d84a0 2024-09-15T23:20:51.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b798fe5ce0 con 0x55b7990d8dc0 2024-09-15T23:20:51.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b79903a2a0 con 0x55b7990e3680 2024-09-15T23:20:51.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe65d9b640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efe540a32c0 0x7efe540a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d84a0 0x55b7990d8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d84a0 0x55b7990d8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54896/0 (socket says 172.21.15.62:54896) 2024-09-15T23:20:51.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe65d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d8dc0 0x55b7990e3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6659c640 1 -- 172.21.15.62:0/427740889 learned_addr learned my addr 172.21.15.62:0/427740889 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe66d9d640 1 --2- 172.21.15.62:0/427740889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 0x55b7990e5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6659c640 1 -- 172.21.15.62:0/427740889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 msgr2=0x55b7990e5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6659c640 1 --2- 172.21.15.62:0/427740889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 0x55b7990e5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6659c640 1 -- 172.21.15.62:0/427740889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d8dc0 msgr2=0x55b7990e3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6659c640 1 --2- 172.21.15.62:0/427740889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d8dc0 0x55b7990e3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe6659c640 1 -- 172.21.15.62:0/427740889 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b798fca440 con 0x55b7990d84a0 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.115+0000 7efe66d9d640 1 --2- 172.21.15.62:0/427740889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 0x55b7990e5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.116+0000 7efe65d9b640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efe540a32c0 0x7efe540a5780 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efe5002a940 tx=0x7efe500040d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.116+0000 7efe6659c640 1 --2- 172.21.15.62:0/427740889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d84a0 0x55b7990d8880 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7efe58007f70 tx=0x7efe5800e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cbfb1bb075ecf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:51.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.116+0000 7efe5cff9640 1 -- 172.21.15.62:0/427740889 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe58014070 con 0x55b7990d84a0 2024-09-15T23:20:51.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.116+0000 7efe5cff9640 1 -- 172.21.15.62:0/427740889 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe58004d20 con 0x55b7990d84a0 2024-09-15T23:20:51.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.116+0000 7efe5cff9640 1 -- 172.21.15.62:0/427740889 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe58005520 con 0x55b7990d84a0 2024-09-15T23:20:51.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 -- 172.21.15.62:0/427740889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d84a0 msgr2=0x55b7990d8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:51.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/427740889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d84a0 0x55b7990d8880 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7efe58007f70 tx=0x7efe5800e7e0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 -- 172.21.15.62:0/427740889 shutdown_connections 2024-09-15T23:20:51.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/427740889 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 0x55b7990e5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/427740889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d8dc0 0x55b7990e3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/427740889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d84a0 0x55b7990d8880 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 -- 172.21.15.62:0/427740889 >> 172.21.15.62:0/427740889 conn(0x55b7990d4a30 msgr2=0x55b7990d4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 -- 172.21.15.62:0/427740889 shutdown_connections 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 -- 172.21.15.62:0/427740889 wait complete. 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 -- start start 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 Processor -- start 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.117+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d8dc0 0x55b7990e01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 0x55b7990e06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b7990e6250 con 0x55b7990d8dc0 2024-09-15T23:20:51.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b798ff9050 con 0x55b7990d84a0 2024-09-15T23:20:51.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b7990e63d0 con 0x55b7990e3680 2024-09-15T23:20:51.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6ed4a740 10 client.0 _ll_register_callbacks cb 0x55b799039160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:51.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe65d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d8dc0 0x55b7990e01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe65d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d8dc0 0x55b7990e01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54904/0 (socket says 172.21.15.62:54904) 2024-09-15T23:20:51.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe65d9b640 1 -- 172.21.15.62:0/3731519784 learned_addr learned my addr 172.21.15.62:0/3731519784 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:51.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6659c640 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:58890/0 (socket says 172.21.15.62:58890) 2024-09-15T23:20:51.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe66d9d640 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 0x55b7990e06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe65d9b640 1 -- 172.21.15.62:0/3731519784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 msgr2=0x55b7990e06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe65d9b640 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 0x55b7990e06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe65d9b640 1 -- 172.21.15.62:0/3731519784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 msgr2=0x55b7990e2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe65d9b640 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe65d9b640 1 -- 172.21.15.62:0/3731519784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b798daf3c0 con 0x55b7990d8dc0 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6659c640 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe65d9b640 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d8dc0 0x55b7990e01b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7efe50033000 tx=0x7efe5002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54caee245a86e1a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe4dffb640 1 -- 172.21.15.62:0/3731519784 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5004e070 con 0x55b7990d8dc0 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe4dffb640 1 -- 172.21.15.62:0/3731519784 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7efe500315b0 con 0x55b7990d8dc0 2024-09-15T23:20:51.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.118+0000 7efe6ed4a740 1 -- 172.21.15.62:0/3731519784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe58004d20 con 0x55b7990d8dc0 2024-09-15T23:20:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe4dffb640 1 -- 172.21.15.62:0/3731519784 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efe5005a050 con 0x55b7990d8dc0 2024-09-15T23:20:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe6ed4a740 10 client.16569 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe6ed4a740 1 -- 172.21.15.62:0/3731519784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b7990f1b60 con 0x55b7990d8dc0 2024-09-15T23:20:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe4dffb640 1 -- 172.21.15.62:0/3731519784 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe58004d20 con 0x55b7990d8dc0 2024-09-15T23:20:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe4dffb640 1 -- 172.21.15.62:0/3731519784 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7efe5003a040 con 0x55b7990d8dc0 2024-09-15T23:20:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe6ed4a740 10 client.16569 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe6ed4a740 10 client.16569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe6ed4a740 20 client.16569 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:51.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b7990f2f70 0x55b7990f5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe6ed4a740 4 client.16569 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe6ed4a740 1 -- 172.21.15.62:0/3731519784 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b79904b850 con 0x55b7990f2f70 2024-09-15T23:20:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.119+0000 7efe6659c640 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b7990f2f70 0x55b7990f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.120+0000 7efe6659c640 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b7990f2f70 0x55b7990f5430 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c292fa27749b413 server_cookie=69e14e02e60dcab8 in_seq=0 out_seq=0 2024-09-15T23:20:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.120+0000 7efe4dffb640 10 client.16569 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.120+0000 7efe4dffb640 1 -- 172.21.15.62:0/3731519784 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b79904b850 con 0x55b7990f2f70 2024-09-15T23:20:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.120+0000 7efe4dffb640 10 client.16569 handle_command_reply: tid=0 2024-09-15T23:20:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 client.16569 shutdown 2024-09-15T23:20:51.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 -- 172.21.15.62:0/3731519784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d8dc0 msgr2=0x55b7990e01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d8dc0 0x55b7990e01b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7efe50033000 tx=0x7efe5002a420 comp rx=0 tx=0).stop 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 -- 172.21.15.62:0/3731519784 shutdown_connections 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b7990f2f70 0x55b7990f5430 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990e3680 0x55b7990e06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d8dc0 0x55b7990e01b0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe4dffb640 0 client.16569 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/3731519784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 -- 172.21.15.62:0/3731519784 >> 172.21.15.62:0/3731519784 conn(0x55b7990d4a30 msgr2=0x55b7990d3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 -- 172.21.15.62:0/3731519784 shutdown_connections 2024-09-15T23:20:51.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 1 -- 172.21.15.62:0/3731519784 wait complete. 2024-09-15T23:20:51.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.121+0000 7efe6ed4a740 20 client.16569 trim_cache size 0 max 16384 2024-09-15T23:20:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6ed4a740 1 Processor -- start 2024-09-15T23:20:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6ed4a740 1 -- start start 2024-09-15T23:20:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990db900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 0x55b7990df190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b3740 con 0x55b7990d4a30 2024-09-15T23:20:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b1010 con 0x55b7990d84a0 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efe681aee60 con 0x55b7990d8dc0 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990ef4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54916/0 (socket says 172.21.15.62:54916) 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.138+0000 7efe6659c640 1 -- 172.21.15.62:0/1238616263 learned_addr learned my addr 172.21.15.62:0/1238616263 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe66d9d640 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 0x55b7990df190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe65d9b640 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990db900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe6659c640 1 -- 172.21.15.62:0/1238616263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 msgr2=0x55b7990df190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe6659c640 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 0x55b7990df190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe6659c640 1 -- 172.21.15.62:0/1238616263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 msgr2=0x55b7990db900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:51.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe6659c640 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990db900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe6659c640 1 -- 172.21.15.62:0/1238616263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7990d8bf0 con 0x55b7990d4a30 2024-09-15T23:20:51.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe66d9d640 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 0x55b7990df190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:20:51.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe6659c640 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990ef4b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7efe58000f80 tx=0x7efe58007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cd6373fb468961 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:51.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe4ffff640 1 -- 172.21.15.62:0/1238616263 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe58014040 con 0x55b7990d4a30 2024-09-15T23:20:51.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe4ffff640 1 -- 172.21.15.62:0/1238616263 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe5802a070 con 0x55b7990d4a30 2024-09-15T23:20:51.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.139+0000 7efe4ffff640 1 -- 172.21.15.62:0/1238616263 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe580059a0 con 0x55b7990d4a30 2024-09-15T23:20:51.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 -- 172.21.15.62:0/1238616263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 msgr2=0x55b7990ef4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:51.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990ef4b0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7efe58000f80 tx=0x7efe58007860 comp rx=0 tx=0).stop 2024-09-15T23:20:51.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 -- 172.21.15.62:0/1238616263 shutdown_connections 2024-09-15T23:20:51.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 0x55b7990df190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990db900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/1238616263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990ef4b0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 -- 172.21.15.62:0/1238616263 >> 172.21.15.62:0/1238616263 conn(0x55b7990e0850 msgr2=0x55b7990dea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:51.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 -- 172.21.15.62:0/1238616263 shutdown_connections 2024-09-15T23:20:51.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 -- 172.21.15.62:0/1238616263 wait complete. 2024-09-15T23:20:51.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 -- start start 2024-09-15T23:20:51.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 Processor -- start 2024-09-15T23:20:51.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990e73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 0x55b7990e7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efe681af4d0 con 0x55b7990d4a30 2024-09-15T23:20:51.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efe681b0940 con 0x55b7990d84a0 2024-09-15T23:20:51.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efe681afba0 con 0x55b7990d8dc0 2024-09-15T23:20:51.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.140+0000 7efe6ed4a740 10 client.0 _ll_register_callbacks cb 0x7efe50009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:51.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990e73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe66d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 0x55b7990e7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990e73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54926/0 (socket says 172.21.15.62:54926) 2024-09-15T23:20:51.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6659c640 1 -- 172.21.15.62:0/2603534520 learned_addr learned my addr 172.21.15.62:0/2603534520 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:51.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe65d9b640 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6659c640 1 -- 172.21.15.62:0/2603534520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 msgr2=0x55b7990e7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:51.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6659c640 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 0x55b7990e7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6659c640 1 -- 172.21.15.62:0/2603534520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 msgr2=0x55b7990e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:51.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6659c640 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6659c640 1 -- 172.21.15.62:0/2603534520 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b79915ccd0 con 0x55b7990d4a30 2024-09-15T23:20:51.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6659c640 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990e73d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7efe58005540 tx=0x7efe58007be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f413cc4aaf52421 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:51.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:51.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe4dffb640 1 -- 172.21.15.62:0/2603534520 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe58010950 con 0x55b7990d4a30 2024-09-15T23:20:51.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe4dffb640 1 -- 172.21.15.62:0/2603534520 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7efe58010af0 con 0x55b7990d4a30 2024-09-15T23:20:51.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe6ed4a740 1 -- 172.21.15.62:0/2603534520 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efe5802a070 con 0x55b7990d4a30 2024-09-15T23:20:51.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.141+0000 7efe4dffb640 1 -- 172.21.15.62:0/2603534520 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7efe580055a0 con 0x55b7990d4a30 2024-09-15T23:20:51.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe6ed4a740 10 client.16575 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:51.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe6ed4a740 1 -- 172.21.15.62:0/2603534520 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b79915dee0 con 0x55b7990d4a30 2024-09-15T23:20:51.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe4dffb640 1 -- 172.21.15.62:0/2603534520 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe5802a070 con 0x55b7990d4a30 2024-09-15T23:20:51.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe4dffb640 1 -- 172.21.15.62:0/2603534520 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7efe58024b40 con 0x55b7990d4a30 2024-09-15T23:20:51.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe6ed4a740 10 client.16575 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:51.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe6ed4a740 10 client.16575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:51.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe6ed4a740 20 client.16575 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:51.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b7990e3680 0x55b799160410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:51.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe6ed4a740 4 client.16575 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:20:51.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe6ed4a740 1 -- 172.21.15.62:0/2603534520 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b799045d60 con 0x55b7990e3680 2024-09-15T23:20:51.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.142+0000 7efe65d9b640 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b7990e3680 0x55b799160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:51.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe65d9b640 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b7990e3680 0x55b799160410 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=659d1bddc4ae1700 server_cookie=46563e4376cbd8c2 in_seq=0 out_seq=0 2024-09-15T23:20:51.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe4dffb640 10 client.16575 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:51.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe4dffb640 1 -- 172.21.15.62:0/2603534520 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b799045d60 con 0x55b7990e3680 2024-09-15T23:20:51.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe4dffb640 10 client.16575 handle_command_reply: tid=0 2024-09-15T23:20:51.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 client.16575 shutdown 2024-09-15T23:20:51.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 -- 172.21.15.62:0/2603534520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 msgr2=0x55b7990e73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:51.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990e73d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7efe58005540 tx=0x7efe58007be0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 -- 172.21.15.62:0/2603534520 shutdown_connections 2024-09-15T23:20:51.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b7990e3680 0x55b799160410 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b7990d8dc0 0x55b7990e7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b7990d84a0 0x55b7990e7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe4dffb640 0 client.16575 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:51.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 --2- 172.21.15.62:0/2603534520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b7990d4a30 0x55b7990e73d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 -- 172.21.15.62:0/2603534520 >> 172.21.15.62:0/2603534520 conn(0x55b7990e0850 msgr2=0x55b7990df2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:51.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 -- 172.21.15.62:0/2603534520 shutdown_connections 2024-09-15T23:20:51.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 1 -- 172.21.15.62:0/2603534520 wait complete. 2024-09-15T23:20:51.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.143+0000 7efe6ed4a740 20 client.16575 trim_cache size 0 max 16384 2024-09-15T23:20:51.157 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:20:51.157 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:20:51.157 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:20:51.157 INFO:teuthology.orchestra.run.smithi062.stdout: "2d3e0b26-9e50-4216-b137-9f34307e997a": { 2024-09-15T23:20:51.157 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:20:51.157 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "2d3e0b26-9e50-4216-b137-9f34307e997a", 2024-09-15T23:20:51.157 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:20:51.158 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:51.158 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:20:51.158 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:20:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 -- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efe540a32c0 msgr2=0x7efe540a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efe540a32c0 0x7efe540a5780 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efe5002a940 tx=0x7efe500040d0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 -- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681082a0 msgr2=0x7efe68101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:51.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681082a0 0x7efe68101410 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7efe6000e970 tx=0x7efe6000ed70 comp rx=0 tx=0).stop 2024-09-15T23:20:51.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 -- 172.21.15.62:0/3179036803 shutdown_connections 2024-09-15T23:20:51.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7efe540a32c0 0x7efe540a5780 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe681082a0 0x7efe68101410 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efe681036e0 0x7efe68100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 --2- 172.21.15.62:0/3179036803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe68102d30 0x7efe68100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:51.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 -- 172.21.15.62:0/3179036803 >> 172.21.15.62:0/3179036803 conn(0x7efe680fc7d0 msgr2=0x7efe68106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:51.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 -- 172.21.15.62:0/3179036803 shutdown_connections 2024-09-15T23:20:51.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:20:51.144+0000 7efe6c971640 1 -- 172.21.15.62:0/3179036803 wait complete. 2024-09-15T23:20:51.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:51 smithi062 ceph-mon[30949]: pgmap v507: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 634 op/s 2024-09-15T23:20:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:51 smithi067 ceph-mon[35665]: pgmap v507: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 634 op/s 2024-09-15T23:20:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:51 smithi148 ceph-mon[35610]: pgmap v507: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 634 op/s 2024-09-15T23:20:51.828 INFO:tasks.cephfs.filesystem:scrub status for tag:2d3e0b26-9e50-4216-b137-9f34307e997a - {'path': '/', 'tag': '2d3e0b26-9e50-4216-b137-9f34307e997a', 'options': 'recursive,force'} 2024-09-15T23:20:53.726 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:53 smithi062 ceph-mon[30949]: pgmap v508: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 5.8 MiB/s wr, 653 op/s 2024-09-15T23:20:53.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:53 smithi067 ceph-mon[35665]: pgmap v508: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 5.8 MiB/s wr, 653 op/s 2024-09-15T23:20:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:53 smithi148 ceph-mon[35610]: pgmap v508: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 5.8 MiB/s wr, 653 op/s 2024-09-15T23:20:54.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 275.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:20:55.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:55 smithi062 ceph-mon[30949]: pgmap v509: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.5 MiB/s wr, 561 op/s 2024-09-15T23:20:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:55 smithi067 ceph-mon[35665]: pgmap v509: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.5 MiB/s wr, 561 op/s 2024-09-15T23:20:55.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:55 smithi148 ceph-mon[35610]: pgmap v509: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.5 MiB/s wr, 561 op/s 2024-09-15T23:20:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.346+0000 7fb92e4af640 1 -- 172.21.15.62:0/2033443219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb928109790 msgr2=0x7fb928109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.346+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2033443219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb928109790 0x7fb928109c10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb920005fd0 tx=0x7fb92002c790 comp rx=0 tx=0).stop 2024-09-15T23:20:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.349+0000 7fb92e4af640 1 -- 172.21.15.62:0/2033443219 shutdown_connections 2024-09-15T23:20:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.349+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2033443219 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92810a150 0x7fb9281109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.349+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2033443219 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb928109790 0x7fb928109c10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.349+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2033443219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb928073730 0x7fb928073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.349+0000 7fb92e4af640 1 -- 172.21.15.62:0/2033443219 >> 172.21.15.62:0/2033443219 conn(0x7fb92806ef20 msgr2=0x7fb928071360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:56.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.353+0000 7fb92e4af640 1 -- 172.21.15.62:0/2033443219 shutdown_connections 2024-09-15T23:20:56.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.354+0000 7fb92e4af640 1 -- 172.21.15.62:0/2033443219 wait complete. 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.354+0000 7fb92e4af640 1 Processor -- start 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.354+0000 7fb92e4af640 1 -- start start 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.355+0000 7fb92e4af640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb928073730 0x7fb92819e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.355+0000 7fb92e4af640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb928109790 0x7fb92819e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.355+0000 7fb92e4af640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92810a150 0x7fb9281a5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.355+0000 7fb92e4af640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb928113e40 con 0x7fb928073730 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.355+0000 7fb92e4af640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb928113cc0 con 0x7fb928109790 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.355+0000 7fb92e4af640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb928113fc0 con 0x7fb92810a150 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.355+0000 7fb927fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb928073730 0x7fb92819e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.356+0000 7fb927fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb928073730 0x7fb92819e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43892/0 (socket says 172.21.15.62:43892) 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.356+0000 7fb927fff640 1 -- 172.21.15.62:0/2938967223 learned_addr learned my addr 172.21.15.62:0/2938967223 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:56.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.356+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92810a150 0x7fb9281a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.356+0000 7fb927fff640 1 -- 172.21.15.62:0/2938967223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92810a150 msgr2=0x7fb9281a5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.356+0000 7fb927fff640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92810a150 0x7fb9281a5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.356+0000 7fb927fff640 1 -- 172.21.15.62:0/2938967223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb928109790 msgr2=0x7fb92819e8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:20:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.356+0000 7fb927fff640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb928109790 0x7fb92819e8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.356+0000 7fb927fff640 1 -- 172.21.15.62:0/2938967223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9281a5fb0 con 0x7fb928073730 2024-09-15T23:20:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.357+0000 7fb927fff640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb928073730 0x7fb92819e390 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb918009fa0 tx=0x7fb91800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.357+0000 7fb9257fa640 1 -- 172.21.15.62:0/2938967223 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91800cd50 con 0x7fb928073730 2024-09-15T23:20:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.357+0000 7fb92e4af640 1 -- 172.21.15.62:0/2938967223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9281a62a0 con 0x7fb928073730 2024-09-15T23:20:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.357+0000 7fb92e4af640 1 -- 172.21.15.62:0/2938967223 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9281a67f0 con 0x7fb928073730 2024-09-15T23:20:56.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.361+0000 7fb931242740 1 Processor -- start 2024-09-15T23:20:56.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.361+0000 7fb931242740 1 -- start start 2024-09-15T23:20:56.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb931242740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 0x557b16f4ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb931242740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4b2e0 0x557b16f4fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb931242740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f52830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb931242740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557b16eae2f0 con 0x557b16f4a9a0 2024-09-15T23:20:56.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb931242740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557b16e56270 con 0x557b16f50370 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb931242740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557b16eaa2b0 con 0x557b16f4b2e0 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb92ca4c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 0x557b16f4ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb92ca4c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 0x557b16f4ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43908/0 (socket says 172.21.15.62:43908) 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb92ca4c640 1 -- 172.21.15.62:0/954518656 learned_addr learned my addr 172.21.15.62:0/954518656 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.363+0000 7fb9277fe640 1 --2- 172.21.15.62:0/954518656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f52830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.362+0000 7fb9257fa640 1 -- 172.21.15.62:0/2938967223 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9180137d0 con 0x7fb928073730 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.363+0000 7fb9257fa640 1 -- 172.21.15.62:0/2938967223 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb918003830 con 0x7fb928073730 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.363+0000 7fb9257fa640 1 -- 172.21.15.62:0/2938967223 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb918022020 con 0x7fb928073730 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.363+0000 7fb9277fe640 1 -- 172.21.15.62:0/954518656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4b2e0 msgr2=0x557b16f4fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.363+0000 7fb9277fe640 1 --2- 172.21.15.62:0/954518656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4b2e0 0x557b16f4fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.363+0000 7fb9277fe640 1 -- 172.21.15.62:0/954518656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 msgr2=0x557b16f4ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:56.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.363+0000 7fb9277fe640 1 --2- 172.21.15.62:0/954518656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 0x557b16f4ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.364+0000 7fb9277fe640 1 -- 172.21.15.62:0/954518656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b16e55340 con 0x557b16f50370 2024-09-15T23:20:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.364+0000 7fb9277fe640 1 --2- 172.21.15.62:0/954518656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f52830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb920000f80 tx=0x7fb920004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6204b1e1bc637c78 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.364+0000 7fb91e7fc640 1 -- 172.21.15.62:0/954518656 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb920065070 con 0x557b16f50370 2024-09-15T23:20:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.364+0000 7fb91e7fc640 1 -- 172.21.15.62:0/954518656 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb92005a030 con 0x557b16f50370 2024-09-15T23:20:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.364+0000 7fb91e7fc640 1 -- 172.21.15.62:0/954518656 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9200042a0 con 0x557b16f50370 2024-09-15T23:20:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.366+0000 7fb931242740 1 -- 172.21.15.62:0/954518656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 msgr2=0x557b16f52830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.366+0000 7fb931242740 1 --2- 172.21.15.62:0/954518656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f52830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb920000f80 tx=0x7fb920004a40 comp rx=0 tx=0).stop 2024-09-15T23:20:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.369+0000 7fb9257fa640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9140a3560 0x7fb9140a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.369+0000 7fb9257fa640 1 -- 172.21.15.62:0/2938967223 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb9180d0a00 con 0x7fb928073730 2024-09-15T23:20:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.371+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9140a3560 0x7fb9140a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.371+0000 7fb931242740 1 -- 172.21.15.62:0/954518656 shutdown_connections 2024-09-15T23:20:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.371+0000 7fb931242740 1 --2- 172.21.15.62:0/954518656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f52830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.371+0000 7fb931242740 1 --2- 172.21.15.62:0/954518656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4b2e0 0x557b16f4fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.371+0000 7fb931242740 1 --2- 172.21.15.62:0/954518656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 0x557b16f4ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.371+0000 7fb931242740 1 -- 172.21.15.62:0/954518656 >> 172.21.15.62:0/954518656 conn(0x557b16f46010 msgr2=0x557b16f46410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:56.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.377+0000 7fb931242740 1 -- 172.21.15.62:0/954518656 shutdown_connections 2024-09-15T23:20:56.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.379+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9140a3560 0x7fb9140a5a20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb91000ac60 tx=0x7fb91000a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:56.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.380+0000 7fb931242740 1 -- 172.21.15.62:0/954518656 wait complete. 2024-09-15T23:20:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.381+0000 7fb931242740 1 -- start start 2024-09-15T23:20:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.381+0000 7fb931242740 1 Processor -- start 2024-09-15T23:20:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.385+0000 7fb931242740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 0x557b16f54b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.385+0000 7fb931242740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 0x557b16f550c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.385+0000 7fb931242740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f50370 0x557b16f5c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.386+0000 7fb931242740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557b16f53180 con 0x557b16f4a9a0 2024-09-15T23:20:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.386+0000 7fb931242740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557b16e37290 con 0x557b16f4b2e0 2024-09-15T23:20:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.386+0000 7fb931242740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557b16f53300 con 0x557b16f50370 2024-09-15T23:20:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.386+0000 7fb931242740 10 client.0 _ll_register_callbacks cb 0x557b16e68140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.387+0000 7fb9277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 0x557b16f550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.387+0000 7fb9277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 0x557b16f550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:46058/0 (socket says 172.21.15.62:46058) 2024-09-15T23:20:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.387+0000 7fb9277fe640 1 -- 172.21.15.62:0/2210996725 learned_addr learned my addr 172.21.15.62:0/2210996725 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.387+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f50370 0x557b16f5c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.388+0000 7fb92ca4c640 1 -- 172.21.15.62:0/2210996725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 msgr2=0x557b16f550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.388+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 0x557b16f550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.388+0000 7fb92ca4c640 1 -- 172.21.15.62:0/2210996725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 msgr2=0x557b16f54b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.388+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 0x557b16f54b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.388+0000 7fb92ca4c640 1 -- 172.21.15.62:0/2210996725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b16bc1210 con 0x557b16f50370 2024-09-15T23:20:56.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.388+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f50370 0x557b16f5c090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb910012cb0 tx=0x7fb9100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7c65d34810b87f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:56.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.390+0000 7fb9077fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:20:56.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.390+0000 7fb9077fe640 1 -- 172.21.15.62:0/2210996725 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb910010b40 con 0x557b16f50370 2024-09-15T23:20:56.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.390+0000 7fb9077fe640 1 -- 172.21.15.62:0/2210996725 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb910010ce0 con 0x557b16f50370 2024-09-15T23:20:56.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.391+0000 7fb931242740 1 -- 172.21.15.62:0/2210996725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb92005a030 con 0x557b16f50370 2024-09-15T23:20:56.393 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.392+0000 7fb9077fe640 1 -- 172.21.15.62:0/2210996725 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb91001f420 con 0x557b16f50370 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.392+0000 7fb931242740 10 client.26026 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.392+0000 7fb931242740 1 -- 172.21.15.62:0/2210996725 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557b16c8d620 con 0x557b16f50370 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.393+0000 7fb9077fe640 1 -- 172.21.15.62:0/2210996725 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb91001faa0 con 0x557b16f50370 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.393+0000 7fb9077fe640 1 -- 172.21.15.62:0/2210996725 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb9100318d0 con 0x557b16f50370 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.393+0000 7fb931242740 10 client.26026 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.393+0000 7fb931242740 10 client.26026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.393+0000 7fb931242740 20 client.26026 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.393+0000 7fb931242740 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557b16f60880 0x557b16f62d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.394+0000 7fb931242740 4 client.26026 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:20:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.394+0000 7fb931242740 1 -- 172.21.15.62:0/2210996725 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb92005a030 con 0x557b16f60880 2024-09-15T23:20:56.396 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.394+0000 7fb927fff640 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557b16f60880 0x557b16f62d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.397+0000 7fb927fff640 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557b16f60880 0x557b16f62d00 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=589d8d24bcc61d31 server_cookie=d9c597df8fd8b4f9 in_seq=0 out_seq=0 2024-09-15T23:20:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.397+0000 7fb9077fe640 10 client.26026 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.398+0000 7fb9077fe640 1 -- 172.21.15.62:0/2210996725 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fb92005a030 con 0x557b16f60880 2024-09-15T23:20:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.398+0000 7fb9077fe640 10 client.26026 handle_command_reply: tid=0 2024-09-15T23:20:56.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.399+0000 7fb931242740 1 client.26026 shutdown 2024-09-15T23:20:56.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.400+0000 7fb931242740 1 -- 172.21.15.62:0/2210996725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f50370 msgr2=0x557b16f5c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.400+0000 7fb931242740 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f50370 0x557b16f5c090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb910012cb0 tx=0x7fb9100046c0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.404+0000 7fb931242740 1 -- 172.21.15.62:0/2210996725 shutdown_connections 2024-09-15T23:20:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.404+0000 7fb931242740 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557b16f60880 0x557b16f62d00 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.404+0000 7fb931242740 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f50370 0x557b16f5c090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.404+0000 7fb931242740 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 0x557b16f550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.404+0000 7fb931242740 1 --2- 172.21.15.62:0/2210996725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4a9a0 0x557b16f54b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.404+0000 7fb9077fe640 0 client.26026 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.404+0000 7fb931242740 1 -- 172.21.15.62:0/2210996725 >> 172.21.15.62:0/2210996725 conn(0x557b16f46010 msgr2=0x557b16f47820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:56.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.408+0000 7fb931242740 1 -- 172.21.15.62:0/2210996725 shutdown_connections 2024-09-15T23:20:56.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.408+0000 7fb931242740 1 -- 172.21.15.62:0/2210996725 wait complete. 2024-09-15T23:20:56.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.409+0000 7fb931242740 20 client.26026 trim_cache size 0 max 16384 2024-09-15T23:20:56.444 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.443+0000 7fb931242740 1 Processor -- start 2024-09-15T23:20:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.444+0000 7fb931242740 1 -- start start 2024-09-15T23:20:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.444+0000 7fb931242740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 0x557b16f4c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.444+0000 7fb931242740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4b2e0 0x557b16f4cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.446 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb931242740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f463f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb9277fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4b2e0 0x557b16f4cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb9277fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4b2e0 0x557b16f4cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43924/0 (socket says 172.21.15.62:43924) 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb927fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 0x557b16f4c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb927fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 0x557b16f4c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:59884/0 (socket says 172.21.15.62:59884) 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb9277fe640 1 -- 172.21.15.62:0/402253258 learned_addr learned my addr 172.21.15.62:0/402253258 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb931242740 1 -- 172.21.15.62:0/402253258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb908012530 con 0x557b16f4b2e0 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb931242740 1 -- 172.21.15.62:0/402253258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb90800fe00 con 0x557b16f50370 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb931242740 1 -- 172.21.15.62:0/402253258 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb90800dc50 con 0x557b16f4a9a0 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/402253258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb92ca4c640 1 -- 172.21.15.62:0/402253258 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 msgr2=0x557b16f4c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/402253258 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 0x557b16f4c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb92ca4c640 1 -- 172.21.15.62:0/402253258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4b2e0 msgr2=0x557b16f4cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:56.447 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/402253258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4b2e0 0x557b16f4cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.448 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.445+0000 7fb92ca4c640 1 -- 172.21.15.62:0/402253258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b16f4c0b0 con 0x557b16f50370 2024-09-15T23:20:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.451+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/402253258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f463f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb910017040 tx=0x7fb91000aa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c06a5192867ce1b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.455+0000 7fb91dffb640 1 -- 172.21.15.62:0/402253258 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91003f070 con 0x557b16f50370 2024-09-15T23:20:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.455+0000 7fb91dffb640 1 -- 172.21.15.62:0/402253258 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9100108c0 con 0x557b16f50370 2024-09-15T23:20:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.456+0000 7fb931242740 1 -- 172.21.15.62:0/402253258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 msgr2=0x557b16f463f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.456+0000 7fb931242740 1 --2- 172.21.15.62:0/402253258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f463f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb910017040 tx=0x7fb91000aa70 comp rx=0 tx=0).stop 2024-09-15T23:20:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.459+0000 7fb931242740 1 -- 172.21.15.62:0/402253258 shutdown_connections 2024-09-15T23:20:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.459+0000 7fb931242740 1 --2- 172.21.15.62:0/402253258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f50370 0x557b16f463f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.459+0000 7fb931242740 1 --2- 172.21.15.62:0/402253258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f4b2e0 0x557b16f4cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.459+0000 7fb931242740 1 --2- 172.21.15.62:0/402253258 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 0x557b16f4c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.461 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.459+0000 7fb931242740 1 -- 172.21.15.62:0/402253258 >> 172.21.15.62:0/402253258 conn(0x557b16f46010 msgr2=0x557b16f54b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:56.465 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.465+0000 7fb931242740 1 -- 172.21.15.62:0/402253258 shutdown_connections 2024-09-15T23:20:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.465+0000 7fb931242740 1 -- 172.21.15.62:0/402253258 wait complete. 2024-09-15T23:20:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.465+0000 7fb931242740 1 -- start start 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.465+0000 7fb931242740 1 Processor -- start 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb931242740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 0x557b16f683e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb931242740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 0x557b16f68920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb931242740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f50370 0x557b16f68e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb931242740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb90800e2c0 con 0x557b16f50370 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb931242740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb90800f730 con 0x557b16f4b2e0 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb931242740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb90800e990 con 0x557b16f4a9a0 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb931242740 10 client.0 _ll_register_callbacks cb 0x7fb910004490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb92ca4c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f50370 0x557b16f68e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb92ca4c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f50370 0x557b16f68e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43928/0 (socket says 172.21.15.62:43928) 2024-09-15T23:20:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.466+0000 7fb92ca4c640 1 -- 172.21.15.62:0/2752644147 learned_addr learned my addr 172.21.15.62:0/2752644147 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:20:56.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.467+0000 7fb927fff640 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 0x557b16f683e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.470 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.469+0000 7fb9277fe640 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 0x557b16f68920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.471+0000 7fb92ca4c640 1 -- 172.21.15.62:0/2752644147 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 msgr2=0x557b16f683e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.471+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 0x557b16f683e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.471+0000 7fb92ca4c640 1 -- 172.21.15.62:0/2752644147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 msgr2=0x557b16f68920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:20:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.471+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 0x557b16f68920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.471+0000 7fb92ca4c640 1 -- 172.21.15.62:0/2752644147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b16fca300 con 0x557b16f50370 2024-09-15T23:20:56.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.471+0000 7fb92ca4c640 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f50370 0x557b16f68e60 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb910006010 tx=0x7fb910031b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8083decca1463d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:20:56.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.472+0000 7fb9077fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:20:56.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.473+0000 7fb9077fe640 1 -- 172.21.15.62:0/2752644147 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb91003f040 con 0x557b16f50370 2024-09-15T23:20:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.473+0000 7fb931242740 1 -- 172.21.15.62:0/2752644147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9100108c0 con 0x557b16f50370 2024-09-15T23:20:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.473+0000 7fb9077fe640 1 -- 172.21.15.62:0/2752644147 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb910010d30 con 0x557b16f50370 2024-09-15T23:20:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.474+0000 7fb9077fe640 1 -- 172.21.15.62:0/2752644147 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb910005040 con 0x557b16f50370 2024-09-15T23:20:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.474+0000 7fb931242740 10 client.16590 fetch_fsmap learned FSMap version 31 2024-09-15T23:20:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.474+0000 7fb931242740 1 -- 172.21.15.62:0/2752644147 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557b16fcb610 con 0x557b16f50370 2024-09-15T23:20:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.475+0000 7fb9077fe640 1 -- 172.21.15.62:0/2752644147 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9100108c0 con 0x557b16f50370 2024-09-15T23:20:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.476+0000 7fb9077fe640 1 -- 172.21.15.62:0/2752644147 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb91004b030 con 0x557b16f50370 2024-09-15T23:20:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.476+0000 7fb931242740 10 client.16590 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:20:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.476+0000 7fb931242740 10 client.16590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:20:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.476+0000 7fb931242740 20 client.16590 populate_metadata read hostname 'smithi062' 2024-09-15T23:20:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.476+0000 7fb931242740 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557b16fcbe80 0x557b16fce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:20:56.478 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.476+0000 7fb931242740 4 client.16590 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:20:56.478 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.476+0000 7fb931242740 1 -- 172.21.15.62:0/2752644147 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557b16ebba20 con 0x557b16fcbe80 2024-09-15T23:20:56.479 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.478+0000 7fb927fff640 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557b16fcbe80 0x557b16fce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:20:56.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.483+0000 7fb927fff640 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557b16fcbe80 0x557b16fce2a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=705d50dabb5aea29 server_cookie=836f772a88358426 in_seq=0 out_seq=0 2024-09-15T23:20:56.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.484+0000 7fb9077fe640 10 client.16590 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb9077fe640 1 -- 172.21.15.62:0/2752644147 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557b16ebba20 con 0x557b16fcbe80 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb9077fe640 10 client.16590 handle_command_reply: tid=0 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 client.16590 shutdown 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 -- 172.21.15.62:0/2752644147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f50370 msgr2=0x557b16f68e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f50370 0x557b16f68e60 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fb910006010 tx=0x7fb910031b70 comp rx=0 tx=0).stop 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 -- 172.21.15.62:0/2752644147 shutdown_connections 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557b16fcbe80 0x557b16fce2a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557b16f50370 0x557b16f68e60 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557b16f4b2e0 0x557b16f68920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb9077fe640 0 client.16590 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:20:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 --2- 172.21.15.62:0/2752644147 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557b16f4a9a0 0x557b16f683e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 -- 172.21.15.62:0/2752644147 >> 172.21.15.62:0/2752644147 conn(0x557b16f46010 msgr2=0x557b16f54930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 -- 172.21.15.62:0/2752644147 shutdown_connections 2024-09-15T23:20:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 1 -- 172.21.15.62:0/2752644147 wait complete. 2024-09-15T23:20:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.810+0000 7fb931242740 20 client.16590 trim_cache size 0 max 16384 2024-09-15T23:20:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:20:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:20:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:20:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:20:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.811+0000 7fb92e4af640 1 -- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9140a3560 msgr2=0x7fb9140a5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.811+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9140a3560 0x7fb9140a5a20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb91000ac60 tx=0x7fb91000a250 comp rx=0 tx=0).stop 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.811+0000 7fb92e4af640 1 -- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb928073730 msgr2=0x7fb92819e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.811+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb928073730 0x7fb92819e390 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb918009fa0 tx=0x7fb91800edd0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.812+0000 7fb92e4af640 1 -- 172.21.15.62:0/2938967223 shutdown_connections 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.812+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9140a3560 0x7fb9140a5a20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.812+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb92810a150 0x7fb9281a5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.812+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb928109790 0x7fb92819e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.812+0000 7fb92e4af640 1 --2- 172.21.15.62:0/2938967223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb928073730 0x7fb92819e390 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.812+0000 7fb92e4af640 1 -- 172.21.15.62:0/2938967223 >> 172.21.15.62:0/2938967223 conn(0x7fb92806ef20 msgr2=0x7fb92810ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.812+0000 7fb92e4af640 1 -- 172.21.15.62:0/2938967223 shutdown_connections 2024-09-15T23:20:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:20:56.812+0000 7fb92e4af640 1 -- 172.21.15.62:0/2938967223 wait complete. 2024-09-15T23:20:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:57 smithi062 ceph-mon[30949]: pgmap v510: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.7 MiB/s wr, 461 op/s 2024-09-15T23:20:57.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:57 smithi067 ceph-mon[35665]: pgmap v510: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.7 MiB/s wr, 461 op/s 2024-09-15T23:20:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:57 smithi148 ceph-mon[35610]: pgmap v510: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 3.7 MiB/s wr, 461 op/s 2024-09-15T23:20:59.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 280.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:20:59.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:20:59 smithi062 ceph-mon[30949]: pgmap v511: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 3.5 MiB/s wr, 384 op/s 2024-09-15T23:20:59.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:20:59 smithi067 ceph-mon[35665]: pgmap v511: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 3.5 MiB/s wr, 384 op/s 2024-09-15T23:20:59.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:20:59 smithi148 ceph-mon[35610]: pgmap v511: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 3.5 MiB/s wr, 384 op/s 2024-09-15T23:21:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:01 smithi062 ceph-mon[30949]: pgmap v512: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 8.9 MiB/s wr, 587 op/s 2024-09-15T23:21:01.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:01 smithi067 ceph-mon[35665]: pgmap v512: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 8.9 MiB/s wr, 587 op/s 2024-09-15T23:21:01.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:01 smithi148 ceph-mon[35610]: pgmap v512: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 8.9 MiB/s wr, 587 op/s 2024-09-15T23:21:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:03 smithi062 ceph-mon[30949]: pgmap v513: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 12 MiB/s wr, 626 op/s 2024-09-15T23:21:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:03 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:21:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:03 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:21:03.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:03 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:21:03.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:03 smithi067 ceph-mon[35665]: pgmap v513: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 12 MiB/s wr, 626 op/s 2024-09-15T23:21:03.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:03 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:21:03.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:03 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:21:03.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:03 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:21:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:03 smithi148 ceph-mon[35610]: pgmap v513: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 12 MiB/s wr, 626 op/s 2024-09-15T23:21:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:03 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:21:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:03 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:21:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:03 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:21:04.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 285.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:21:04.839 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:05.109 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:05.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:05 smithi062 ceph-mon[30949]: pgmap v514: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 11 MiB/s wr, 494 op/s 2024-09-15T23:21:05.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.700+0000 7f4f91d5c640 1 -- 172.21.15.62:0/1194753245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 msgr2=0x7f4f8c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.700+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/1194753245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 0x7f4f8c1029b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f4f74009900 tx=0x7f4f7402a480 comp rx=0 tx=0).stop 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.700+0000 7f4f91d5c640 1 -- 172.21.15.62:0/1194753245 shutdown_connections 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.700+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/1194753245 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.700+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/1194753245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 0x7f4f8c1029b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.700+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/1194753245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f8c100e20 0x7f4f8c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.700+0000 7f4f91d5c640 1 -- 172.21.15.62:0/1194753245 >> 172.21.15.62:0/1194753245 conn(0x7f4f8c0fc7d0 msgr2=0x7f4f8c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.700+0000 7f4f91d5c640 1 -- 172.21.15.62:0/1194753245 shutdown_connections 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.700+0000 7f4f91d5c640 1 -- 172.21.15.62:0/1194753245 wait complete. 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f91d5c640 1 Processor -- start 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f91d5c640 1 -- start start 2024-09-15T23:21:05.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f91d5c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f8c100e20 0x7f4f8c1984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f91d5c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f91d5c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c19cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f91d5c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c1120a0 con 0x7f4f8c102570 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f91d5c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c111f20 con 0x7f4f8c102ef0 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f91d5c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c112220 con 0x7f4f8c100e20 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f83fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f83fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55070/0 (socket says 172.21.15.62:55070) 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f83fff640 1 -- 172.21.15.62:0/3980918989 learned_addr learned my addr 172.21.15.62:0/3980918989 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f8bfff640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c19cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f83fff640 1 -- 172.21.15.62:0/3980918989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f8c100e20 msgr2=0x7f4f8c1984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f83fff640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f8c100e20 0x7f4f8c1984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f83fff640 1 -- 172.21.15.62:0/3980918989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f8c102ef0 msgr2=0x7f4f8c19cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f83fff640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c19cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:05.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.701+0000 7f4f83fff640 1 -- 172.21.15.62:0/3980918989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c19d470 con 0x7f4f8c102570 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.702+0000 7f4f83fff640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006010 tx=0x7f4f74003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.702+0000 7f4f897fa640 1 -- 172.21.15.62:0/3980918989 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7403f070 con 0x7f4f8c102570 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.702+0000 7f4f897fa640 1 -- 172.21.15.62:0/3980918989 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f74031e30 con 0x7f4f8c102570 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.702+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8c19d700 con 0x7f4f8c102570 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.702+0000 7f4f897fa640 1 -- 172.21.15.62:0/3980918989 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f74035030 con 0x7f4f8c102570 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.702+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8c1a5040 con 0x7f4f8c102570 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.703+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8c101200 con 0x7f4f8c102570 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.704+0000 7f4f897fa640 1 -- 172.21.15.62:0/3980918989 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4f74042070 con 0x7f4f8c102570 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.705+0000 7f4f897fa640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f780a3240 0x7f4f780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.705+0000 7f4f8b7fe640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f780a3240 0x7f4f780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.705+0000 7f4f897fa640 1 -- 172.21.15.62:0/3980918989 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4f740f05e0 con 0x7f4f8c102570 2024-09-15T23:21:05.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.705+0000 7f4f8b7fe640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f780a3240 0x7f4f780a5700 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00a990 tx=0x7f4f7c005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:05.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.709+0000 7f4f897fa640 1 -- 172.21.15.62:0/3980918989 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f4f740bcf80 con 0x7f4f8c102570 2024-09-15T23:21:05.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:05 smithi067 ceph-mon[35665]: pgmap v514: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 11 MiB/s wr, 494 op/s 2024-09-15T23:21:05.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:05 smithi148 ceph-mon[35610]: pgmap v514: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 11 MiB/s wr, 494 op/s 2024-09-15T23:21:05.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.920+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f8c1a5270 con 0x7f4f8c102570 2024-09-15T23:21:05.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.921+0000 7f4f897fa640 1 -- 172.21.15.62:0/3980918989 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11903 (secure 0 0 0) 0x7f4f740bc880 con 0x7f4f8c102570 2024-09-15T23:21:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:21:05.922 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":31,"btime":"2024-09-15T23:20:13:209709+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:20:12.378892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:21:05.923 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 31 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.923+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f780a3240 msgr2=0x7f4f780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f780a3240 0x7f4f780a5700 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00a990 tx=0x7f4f7c005c70 comp rx=0 tx=0).stop 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 msgr2=0x7f4f8c198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f4f74006010 tx=0x7f4f74003c80 comp rx=0 tx=0).stop 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 shutdown_connections 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4f780a3240 0x7f4f780a5700 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f8c102ef0 0x7f4f8c19cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f8c102570 0x7f4f8c198a00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 --2- 172.21.15.62:0/3980918989 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4f8c100e20 0x7f4f8c1984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:05.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 >> 172.21.15.62:0/3980918989 conn(0x7f4f8c0fc7d0 msgr2=0x7f4f8c1068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 shutdown_connections 2024-09-15T23:21:05.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:05.924+0000 7f4f91d5c640 1 -- 172.21.15.62:0/3980918989 wait complete. 2024-09-15T23:21:06.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:06 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3980918989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:06.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:06 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3980918989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:06.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:06 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3980918989' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:06.862 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 61932369 --release --await' 2024-09-15T23:21:06.862 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 61932369 --release --await 2024-09-15T23:21:07.120 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:07.550 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:07 smithi062 ceph-mon[30949]: pgmap v515: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 11 MiB/s wr, 455 op/s 2024-09-15T23:21:07.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:07 smithi067 ceph-mon[35665]: pgmap v515: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 11 MiB/s wr, 455 op/s 2024-09-15T23:21:07.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:07 smithi148 ceph-mon[35610]: pgmap v515: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 11 MiB/s wr, 455 op/s 2024-09-15T23:21:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.885+0000 7fdc21360640 1 -- 172.21.15.62:0/3379307331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc1c1027c0 msgr2=0x7fdc1c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.885+0000 7fdc21360640 1 --2- 172.21.15.62:0/3379307331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc1c1027c0 0x7fdc1c106810 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009900 tx=0x7fdc1002a410 comp rx=0 tx=0).stop 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.885+0000 7fdc21360640 1 -- 172.21.15.62:0/3379307331 shutdown_connections 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.885+0000 7fdc21360640 1 --2- 172.21.15.62:0/3379307331 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc1c106f40 0x7fdc1c10eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.885+0000 7fdc21360640 1 --2- 172.21.15.62:0/3379307331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc1c1027c0 0x7fdc1c106810 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.885+0000 7fdc21360640 1 --2- 172.21.15.62:0/3379307331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c101e10 0x7fdc1c1021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.885+0000 7fdc21360640 1 -- 172.21.15.62:0/3379307331 >> 172.21.15.62:0/3379307331 conn(0x7fdc1c0fd8d0 msgr2=0x7fdc1c0ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.885+0000 7fdc21360640 1 -- 172.21.15.62:0/3379307331 shutdown_connections 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.885+0000 7fdc21360640 1 -- 172.21.15.62:0/3379307331 wait complete. 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc21360640 1 Processor -- start 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc21360640 1 -- start start 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc21360640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c101e10 0x7fdc1c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc21360640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc1c1027c0 0x7fdc1c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc21360640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc1c106f40 0x7fdc1c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc21360640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c111fc0 con 0x7fdc1c106f40 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc21360640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c111e40 con 0x7fdc1c101e10 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc21360640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c112140 con 0x7fdc1c1027c0 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc1c106f40 0x7fdc1c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c101e10 0x7fdc1c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc1c106f40 0x7fdc1c074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55074/0 (socket says 172.21.15.62:55074) 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc1b7fe640 1 -- 172.21.15.62:0/2844630406 learned_addr learned my addr 172.21.15.62:0/2844630406 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.886+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c101e10 0x7fdc1c0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:53944/0 (socket says 172.21.15.62:53944) 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc1c1027c0 0x7fdc1c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc1affd640 1 -- 172.21.15.62:0/2844630406 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc1c1027c0 msgr2=0x7fdc1c074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc1affd640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc1c1027c0 0x7fdc1c074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc1affd640 1 -- 172.21.15.62:0/2844630406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc1c106f40 msgr2=0x7fdc1c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:07.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc1affd640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc1c106f40 0x7fdc1c074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc1affd640 1 -- 172.21.15.62:0/2844630406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c1a69a0 con 0x7fdc1c101e10 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc1affd640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c101e10 0x7fdc1c0744f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00d6d0 tx=0x7fdc0c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc03fff640 1 -- 172.21.15.62:0/2844630406 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c017070 con 0x7fdc1c101e10 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc21360640 1 -- 172.21.15.62:0/2844630406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c1a6c90 con 0x7fdc1c101e10 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc03fff640 1 -- 172.21.15.62:0/2844630406 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0c009820 con 0x7fdc1c101e10 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc03fff640 1 -- 172.21.15.62:0/2844630406 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c004cd0 con 0x7fdc1c101e10 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.887+0000 7fdc21360640 1 -- 172.21.15.62:0/2844630406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c1a71a0 con 0x7fdc1c101e10 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc23739740 1 Processor -- start 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc23739740 1 -- start start 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc03fff640 1 -- 172.21.15.62:0/2844630406 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fdc0c0099d0 con 0x7fdc1c101e10 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc23739740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d56440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc23739740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d60d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc23739740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64d63630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc23739740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563c64cbb620 con 0x563c64d61240 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc23739740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563c64c62e50 con 0x563c64d56060 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc23739740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563c64cb75e0 con 0x563c64d56980 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc1a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64d63630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc1a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64d63630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55080/0 (socket says 172.21.15.62:55080) 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc1a7fc640 1 -- 172.21.15.62:0/4097711657 learned_addr learned my addr 172.21.15.62:0/4097711657 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc1b7fe640 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d56440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.890+0000 7fdc1affd640 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d60d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc1a7fc640 1 -- 172.21.15.62:0/4097711657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 msgr2=0x563c64d60d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d60d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc1a7fc640 1 -- 172.21.15.62:0/4097711657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 msgr2=0x563c64d56440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:07.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d56440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc1a7fc640 1 -- 172.21.15.62:0/4097711657 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c64c53d40 con 0x563c64d61240 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc1affd640 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d60d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc03fff640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc040a3240 0x7fdc040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc1b7fe640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc040a3240 0x7fdc040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64d63630 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009900 tx=0x7fdc10031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5752504f5ce545a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc03fff640 1 -- 172.21.15.62:0/2844630406 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdc0c0cf780 con 0x7fdc1c101e10 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc017fa640 1 -- 172.21.15.62:0/4097711657 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10040070 con 0x563c64d61240 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc017fa640 1 -- 172.21.15.62:0/4097711657 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc10005020 con 0x563c64d61240 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.891+0000 7fdc017fa640 1 -- 172.21.15.62:0/4097711657 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10035040 con 0x563c64d61240 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 -- 172.21.15.62:0/4097711657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 msgr2=0x563c64d63630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc1b7fe640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc040a3240 0x7fdc040a5700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdc14011420 tx=0x7fdc1400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64d63630 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009900 tx=0x7fdc10031630 comp rx=0 tx=0).stop 2024-09-15T23:21:07.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 -- 172.21.15.62:0/4097711657 shutdown_connections 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64d63630 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d60d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 --2- 172.21.15.62:0/4097711657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d56440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 -- 172.21.15.62:0/4097711657 >> 172.21.15.62:0/4097711657 conn(0x563c64d525f0 msgr2=0x563c64d529d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 -- 172.21.15.62:0/4097711657 shutdown_connections 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 -- 172.21.15.62:0/4097711657 wait complete. 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 -- start start 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.892+0000 7fdc23739740 1 Processor -- start 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc23739740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d5e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc23739740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56980 0x563c64d5e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc23739740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d61240 0x563c64d5eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc23739740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563c64d63e10 con 0x563c64d56980 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc23739740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563c64c43f10 con 0x563c64d56060 2024-09-15T23:21:07.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc23739740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563c64d63f90 con 0x563c64d61240 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc23739740 10 client.0 _ll_register_callbacks cb 0x563c64c67db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56980 0x563c64d5e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56980 0x563c64d5e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55096/0 (socket says 172.21.15.62:55096) 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1a7fc640 1 -- 172.21.15.62:0/3351344154 learned_addr learned my addr 172.21.15.62:0/3351344154 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1affd640 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d5e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1b7fe640 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d61240 0x563c64d5eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1a7fc640 1 -- 172.21.15.62:0/3351344154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d61240 msgr2=0x563c64d5eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d61240 0x563c64d5eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1a7fc640 1 -- 172.21.15.62:0/3351344154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 msgr2=0x563c64d5e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d5e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1a7fc640 1 -- 172.21.15.62:0/3351344154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c649ca580 con 0x563c64d56980 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56980 0x563c64d5e580 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdc10005940 tx=0x7fdc10005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbbd554ea78eb05 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.893+0000 7fdbfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:07.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdbfeff5640 1 -- 172.21.15.62:0/3351344154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10040070 con 0x563c64d56980 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdbfeff5640 1 -- 172.21.15.62:0/3351344154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdc10004940 con 0x563c64d56980 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdc23739740 1 -- 172.21.15.62:0/3351344154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc10005020 con 0x563c64d56980 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdbfeff5640 1 -- 172.21.15.62:0/3351344154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdc10056050 con 0x563c64d56980 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdc23739740 10 client.16608 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdc23739740 1 -- 172.21.15.62:0/3351344154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563c64a94960 con 0x563c64d56980 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdbfeff5640 1 -- 172.21.15.62:0/3351344154 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc10005020 con 0x563c64d56980 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdbfeff5640 1 -- 172.21.15.62:0/3351344154 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fdc1003cc90 con 0x563c64d56980 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdc23739740 10 client.16608 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdc23739740 10 client.16608 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdc23739740 20 client.16608 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdc23739740 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c64d70b30 0x563c64d72f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdc23739740 4 client.16608 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.894+0000 7fdc23739740 1 -- 172.21.15.62:0/3351344154 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c64cc8b90 con 0x563c64d70b30 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.895+0000 7fdc1affd640 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c64d70b30 0x563c64d72f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.895+0000 7fdc1affd640 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c64d70b30 0x563c64d72f20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87a0f87113abfa54 server_cookie=62c122376057b670 in_seq=0 out_seq=0 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.895+0000 7fdbfeff5640 10 client.16608 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdbfeff5640 1 -- 172.21.15.62:0/3351344154 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563c64cc8b90 con 0x563c64d70b30 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdbfeff5640 10 client.16608 handle_command_reply: tid=0 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 client.16608 shutdown 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 -- 172.21.15.62:0/3351344154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56980 msgr2=0x563c64d5e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56980 0x563c64d5e580 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fdc10005940 tx=0x7fdc10005400 comp rx=0 tx=0).stop 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 -- 172.21.15.62:0/3351344154 shutdown_connections 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c64d70b30 0x563c64d72f20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d61240 0x563c64d5eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56980 0x563c64d5e580 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 --2- 172.21.15.62:0/3351344154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d5e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdbfeff5640 0 client.16608 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:07.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 -- 172.21.15.62:0/3351344154 >> 172.21.15.62:0/3351344154 conn(0x563c64d525f0 msgr2=0x563c64d582d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:07.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 -- 172.21.15.62:0/3351344154 shutdown_connections 2024-09-15T23:21:07.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 1 -- 172.21.15.62:0/3351344154 wait complete. 2024-09-15T23:21:07.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.896+0000 7fdc23739740 20 client.16608 trim_cache size 0 max 16384 2024-09-15T23:21:07.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.916+0000 7fdc23739740 1 Processor -- start 2024-09-15T23:21:07.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.916+0000 7fdc23739740 1 -- start start 2024-09-15T23:21:07.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc23739740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56060 0x563c64d581c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc23739740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d78550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc23739740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d61240 0x563c64d78a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc23739740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1b36c0 con 0x563c64d56060 2024-09-15T23:21:07.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc23739740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1b0f90 con 0x563c64d61240 2024-09-15T23:21:07.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc23739740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1aede0 con 0x563c64d56980 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56060 0x563c64d581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56060 0x563c64d581c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55110/0 (socket says 172.21.15.62:55110) 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1affd640 1 -- 172.21.15.62:0/3631688087 learned_addr learned my addr 172.21.15.62:0/3631688087 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1b7fe640 1 --2- 172.21.15.62:0/3631688087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d61240 0x563c64d78a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1affd640 1 -- 172.21.15.62:0/3631688087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 msgr2=0x563c64d78550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1affd640 1 --2- 172.21.15.62:0/3631688087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d78550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1affd640 1 -- 172.21.15.62:0/3631688087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d61240 msgr2=0x563c64d78a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1affd640 1 --2- 172.21.15.62:0/3631688087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d61240 0x563c64d78a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1affd640 1 -- 172.21.15.62:0/3631688087 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c64d52420 con 0x563c64d56060 2024-09-15T23:21:07.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc1affd640 1 --2- 172.21.15.62:0/3631688087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56060 0x563c64d581c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0984a0 tx=0x7fdc0c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bbf3955dfbaed4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:07.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc00ff9640 1 -- 172.21.15.62:0/3631688087 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c020040 con 0x563c64d56060 2024-09-15T23:21:07.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc00ff9640 1 -- 172.21.15.62:0/3631688087 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0c0cd010 con 0x563c64d56060 2024-09-15T23:21:07.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.917+0000 7fdc00ff9640 1 -- 172.21.15.62:0/3631688087 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c09bd30 con 0x563c64d56060 2024-09-15T23:21:07.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 -- 172.21.15.62:0/3631688087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56060 msgr2=0x563c64d581c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 --2- 172.21.15.62:0/3631688087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56060 0x563c64d581c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0984a0 tx=0x7fdc0c086040 comp rx=0 tx=0).stop 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 -- 172.21.15.62:0/3631688087 shutdown_connections 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 --2- 172.21.15.62:0/3631688087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d61240 0x563c64d78a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 --2- 172.21.15.62:0/3631688087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d78550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 --2- 172.21.15.62:0/3631688087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d56060 0x563c64d581c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 -- 172.21.15.62:0/3631688087 >> 172.21.15.62:0/3631688087 conn(0x563c64d525f0 msgr2=0x563c64d57ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 -- 172.21.15.62:0/3631688087 shutdown_connections 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 -- 172.21.15.62:0/3631688087 wait complete. 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 -- start start 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.918+0000 7fdc23739740 1 Processor -- start 2024-09-15T23:21:07.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc23739740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d65320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc23739740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d65860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc23739740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64dda520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc23739740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1af450 con 0x563c64d61240 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc23739740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1b08c0 con 0x563c64d56060 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc23739740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1afb20 con 0x563c64d56980 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc23739740 10 client.0 _ll_register_callbacks cb 0x7fdc100318e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64dda520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64dda520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55116/0 (socket says 172.21.15.62:55116) 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1b7fe640 1 -- 172.21.15.62:0/2147083592 learned_addr learned my addr 172.21.15.62:0/2147083592 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d65320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d65860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1affd640 1 -- 172.21.15.62:0/2147083592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 msgr2=0x563c64d65860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:07.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1affd640 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d65860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1affd640 1 -- 172.21.15.62:0/2147083592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 msgr2=0x563c64dda520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1affd640 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64dda520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1affd640 1 -- 172.21.15.62:0/2147083592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c64d66140 con 0x563c64d56060 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1b7fe640 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64dda520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdc1affd640 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d65320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0984a0 tx=0x7fdc0c005b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d87d60e43cd65e1b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdbfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.919+0000 7fdbfeff5640 1 -- 172.21.15.62:0/2147083592 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c014030 con 0x563c64d56060 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdbfeff5640 1 -- 172.21.15.62:0/2147083592 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdc0c098140 con 0x563c64d56060 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc23739740 1 -- 172.21.15.62:0/2147083592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc0c0cd010 con 0x563c64d56060 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdbfeff5640 1 -- 172.21.15.62:0/2147083592 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdc0c020050 con 0x563c64d56060 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc23739740 10 client.26231 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc23739740 1 -- 172.21.15.62:0/2147083592 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563c64ddbbb0 con 0x563c64d56060 2024-09-15T23:21:07.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdbfeff5640 1 -- 172.21.15.62:0/2147083592 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0c0cd010 con 0x563c64d56060 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdbfeff5640 1 -- 172.21.15.62:0/2147083592 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fdc0c09af20 con 0x563c64d56060 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc23739740 10 client.26231 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc23739740 10 client.26231 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc23739740 20 client.26231 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc23739740 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c64ddc2d0 0x563c64dde6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc23739740 4 client.26231 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "61932369", "release": true, "await": true}] 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc23739740 1 -- 172.21.15.62:0/2147083592 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "61932369", "release": true, "await": true}) -- 0x563c64cc30a0 con 0x563c64ddc2d0 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.920+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c64ddc2d0 0x563c64dde6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.921+0000 7fdc1a7fc640 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c64ddc2d0 0x563c64dde6c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb1a5eaa141369e6 server_cookie=fd28f15da15cb4fd in_seq=0 out_seq=0 2024-09-15T23:21:07.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.921+0000 7fdbfeff5640 10 client.26231 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:07.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.950+0000 7fdbfeff5640 1 -- 172.21.15.62:0/2147083592 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x563c64cc30a0 con 0x563c64ddc2d0 2024-09-15T23:21:07.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.951+0000 7fdbfeff5640 10 client.26231 handle_command_reply: tid=0 2024-09-15T23:21:07.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.952+0000 7fdc23739740 1 client.26231 shutdown 2024-09-15T23:21:07.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.956+0000 7fdc23739740 1 -- 172.21.15.62:0/2147083592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 msgr2=0x563c64d65320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.956+0000 7fdc23739740 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d65320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0984a0 tx=0x7fdc0c005b10 comp rx=0 tx=0).stop 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.956+0000 7fdc23739740 1 -- 172.21.15.62:0/2147083592 shutdown_connections 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.956+0000 7fdc23739740 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c64ddc2d0 0x563c64dde6c0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.956+0000 7fdc23739740 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c64d61240 0x563c64dda520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.956+0000 7fdc23739740 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c64d56980 0x563c64d65860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.956+0000 7fdbfeff5640 0 client.26231 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.956+0000 7fdc23739740 1 --2- 172.21.15.62:0/2147083592 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c64d56060 0x563c64d65320 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.956+0000 7fdc23739740 1 -- 172.21.15.62:0/2147083592 >> 172.21.15.62:0/2147083592 conn(0x563c64d525f0 msgr2=0x563c64d600c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.957+0000 7fdc23739740 1 -- 172.21.15.62:0/2147083592 shutdown_connections 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.957+0000 7fdc23739740 1 -- 172.21.15.62:0/2147083592 wait complete. 2024-09-15T23:21:07.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.957+0000 7fdc23739740 20 client.26231 trim_cache size 0 max 16384 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 31, 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 20, 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "61932369": { 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 20, 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:21:07.960 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 145.3, 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:07.961 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:07.962 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:07.962 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:21:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 -- 172.21.15.62:0/2844630406 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc040a3240 msgr2=0x7fdc040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc040a3240 0x7fdc040a5700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdc14011420 tx=0x7fdc1400b040 comp rx=0 tx=0).stop 2024-09-15T23:21:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 -- 172.21.15.62:0/2844630406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c101e10 msgr2=0x7fdc1c0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c101e10 0x7fdc1c0744f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00d6d0 tx=0x7fdc0c00db90 comp rx=0 tx=0).stop 2024-09-15T23:21:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 -- 172.21.15.62:0/2844630406 shutdown_connections 2024-09-15T23:21:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdc040a3240 0x7fdc040a5700 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc1c106f40 0x7fdc1c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdc1c1027c0 0x7fdc1c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 --2- 172.21.15.62:0/2844630406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc1c101e10 0x7fdc1c0744f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.961+0000 7fdc21360640 1 -- 172.21.15.62:0/2844630406 >> 172.21.15.62:0/2844630406 conn(0x7fdc1c0fd8d0 msgr2=0x7fdc1c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.963+0000 7fdc21360640 1 -- 172.21.15.62:0/2844630406 shutdown_connections 2024-09-15T23:21:07.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:07.964+0000 7fdc21360640 1 -- 172.21.15.62:0/2844630406 wait complete. 2024-09-15T23:21:09.371 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '61932369', total seconds elapsed: 29.9 2024-09-15T23:21:09.372 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 25.35 2024-09-15T23:21:09.372 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 290.0 s, 653.3 tps, lat 651.465 ms stddev 3390.300 2024-09-15T23:21:09.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:09 smithi062 ceph-mon[30949]: pgmap v516: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 MiB/s wr, 482 op/s 2024-09-15T23:21:09.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:09 smithi067 ceph-mon[35665]: pgmap v516: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 MiB/s wr, 482 op/s 2024-09-15T23:21:09.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:09 smithi148 ceph-mon[35610]: pgmap v516: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 11 MiB/s wr, 482 op/s 2024-09-15T23:21:09.823 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.982+0000 7f2492fa8640 1 -- 172.21.15.62:0/991997377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c100a40 msgr2=0x7f248c100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.982+0000 7f2492fa8640 1 --2- 172.21.15.62:0/991997377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c100a40 0x7f248c100e40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2480009900 tx=0x7f248002a410 comp rx=0 tx=0).stop 2024-09-15T23:21:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.982+0000 7f2492fa8640 1 -- 172.21.15.62:0/991997377 shutdown_connections 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.982+0000 7f2492fa8640 1 --2- 172.21.15.62:0/991997377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c1025f0 0x7f248c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.982+0000 7f2492fa8640 1 --2- 172.21.15.62:0/991997377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c101c30 0x7f248c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.982+0000 7f2492fa8640 1 --2- 172.21.15.62:0/991997377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c100a40 0x7f248c100e40 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.982+0000 7f2492fa8640 1 -- 172.21.15.62:0/991997377 >> 172.21.15.62:0/991997377 conn(0x7f248c0fc1e0 msgr2=0x7f248c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.982+0000 7f2492fa8640 1 -- 172.21.15.62:0/991997377 shutdown_connections 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.982+0000 7f2492fa8640 1 -- 172.21.15.62:0/991997377 wait complete. 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2492fa8640 1 Processor -- start 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2492fa8640 1 -- start start 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2492fa8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c100a40 0x7f248c1049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2492fa8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c101c30 0x7f248c103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2492fa8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c1025f0 0x7f248c103550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2492fa8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f248c10c6f0 con 0x7f248c1025f0 2024-09-15T23:21:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2492fa8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f248c10c570 con 0x7f248c100a40 2024-09-15T23:21:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2492fa8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f248c10c870 con 0x7f248c101c30 2024-09-15T23:21:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2490d44640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c100a40 0x7f248c1049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.983+0000 7f2490d44640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c100a40 0x7f248c1049d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:53978/0 (socket says 172.21.15.62:53978) 2024-09-15T23:21:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.984+0000 7f2491545640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c1025f0 0x7f248c103550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.984+0000 7f2490d44640 1 -- 172.21.15.62:0/2145064861 learned_addr learned my addr 172.21.15.62:0/2145064861 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.984+0000 7f248bfff640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c101c30 0x7f248c103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.984+0000 7f2490d44640 1 -- 172.21.15.62:0/2145064861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c101c30 msgr2=0x7f248c103010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.984+0000 7f2490d44640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c101c30 0x7f248c103010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.984+0000 7f2490d44640 1 -- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c1025f0 msgr2=0x7f248c103550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.984+0000 7f2490d44640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c1025f0 0x7f248c103550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.984+0000 7f2490d44640 1 -- 172.21.15.62:0/2145064861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248c103de0 con 0x7f248c100a40 2024-09-15T23:21:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.985+0000 7f2490d44640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c100a40 0x7f248c1049d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2480031440 tx=0x7f2480032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.985+0000 7f2489ffb640 1 -- 172.21.15.62:0/2145064861 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248003f030 con 0x7f248c100a40 2024-09-15T23:21:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.986+0000 7f2489ffb640 1 -- 172.21.15.62:0/2145064861 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2480003ce0 con 0x7f248c100a40 2024-09-15T23:21:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.986+0000 7f2489ffb640 1 -- 172.21.15.62:0/2145064861 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480035070 con 0x7f248c100a40 2024-09-15T23:21:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.986+0000 7f2492fa8640 1 -- 172.21.15.62:0/2145064861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f248c1a1e40 con 0x7f248c100a40 2024-09-15T23:21:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.986+0000 7f2492fa8640 1 -- 172.21.15.62:0/2145064861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f248c1a2250 con 0x7f248c100a40 2024-09-15T23:21:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.990+0000 7f2495e1d740 1 Processor -- start 2024-09-15T23:21:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.990+0000 7f2495e1d740 1 -- start start 2024-09-15T23:21:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2489ffb640 1 -- 172.21.15.62:0/2145064861 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2480042030 con 0x7f248c100a40 2024-09-15T23:21:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 0x561d229edda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d229f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d229f5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2495e1d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561d229512f0 con 0x561d229ee2e0 2024-09-15T23:21:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2495e1d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d228f9270 con 0x561d229f3370 2024-09-15T23:21:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2495e1d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561d2294d2b0 con 0x561d229ed9a0 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2491545640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 0x561d229edda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f248bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d229f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2490d44640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d229f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2491545640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 0x561d229edda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52464/0 (socket says 172.21.15.62:52464) 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f2491545640 1 -- 172.21.15.62:0/1109019925 learned_addr learned my addr 172.21.15.62:0/1109019925 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f248bfff640 1 -- 172.21.15.62:0/1109019925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 msgr2=0x561d229edda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f248bfff640 1 --2- 172.21.15.62:0/1109019925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 0x561d229edda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f248bfff640 1 -- 172.21.15.62:0/1109019925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 msgr2=0x561d229f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f248bfff640 1 --2- 172.21.15.62:0/1109019925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d229f2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.991+0000 7f248bfff640 1 -- 172.21.15.62:0/1109019925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d228f8340 con 0x561d229f3370 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.992+0000 7f248bfff640 1 --2- 172.21.15.62:0/1109019925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d229f5830 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2478009b40 tx=0x7f24780113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf8bbc7adfda3e39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.992+0000 7f2489ffb640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f247c0a3460 0x7f247c0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.992+0000 7f2476ffd640 1 -- 172.21.15.62:0/1109019925 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478011e10 con 0x561d229f3370 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.992+0000 7f2491545640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f247c0a3460 0x7f247c0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.992+0000 7f2476ffd640 1 -- 172.21.15.62:0/1109019925 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2478012930 con 0x561d229f3370 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.992+0000 7f2476ffd640 1 -- 172.21.15.62:0/1109019925 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247800fa50 con 0x561d229f3370 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.992+0000 7f2489ffb640 1 -- 172.21.15.62:0/2145064861 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f24800f0100 con 0x7f248c100a40 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.993+0000 7f2495e1d740 1 -- 172.21.15.62:0/1109019925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 msgr2=0x561d229f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.993+0000 7f2491545640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f247c0a3460 0x7f247c0a5920 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2484010540 tx=0x7f2484007520 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.993+0000 7f2495e1d740 1 --2- 172.21.15.62:0/1109019925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d229f5830 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2478009b40 tx=0x7f24780113e0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.993+0000 7f2495e1d740 1 -- 172.21.15.62:0/1109019925 shutdown_connections 2024-09-15T23:21:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.993+0000 7f2495e1d740 1 --2- 172.21.15.62:0/1109019925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d229f5830 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.993+0000 7f2495e1d740 1 --2- 172.21.15.62:0/1109019925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d229f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.993+0000 7f2495e1d740 1 --2- 172.21.15.62:0/1109019925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 0x561d229edda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.993+0000 7f2495e1d740 1 -- 172.21.15.62:0/1109019925 >> 172.21.15.62:0/1109019925 conn(0x561d229e9010 msgr2=0x561d229e9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.993+0000 7f2495e1d740 1 -- 172.21.15.62:0/1109019925 shutdown_connections 2024-09-15T23:21:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.994+0000 7f2495e1d740 1 -- 172.21.15.62:0/1109019925 wait complete. 2024-09-15T23:21:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.994+0000 7f2495e1d740 1 -- start start 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.994+0000 7f2495e1d740 1 Processor -- start 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229ed9a0 0x561d22a003b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ee2e0 0x561d22a008f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229f3370 0x561d22a07930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f2495e1d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561d229f6340 con 0x561d229f3370 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f2495e1d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d228da290 con 0x561d229ed9a0 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f2495e1d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561d229f64c0 con 0x561d229ee2e0 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f2495e1d740 10 client.0 _ll_register_callbacks cb 0x561d2290b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f248bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ee2e0 0x561d22a008f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f248bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ee2e0 0x561d22a008f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52476/0 (socket says 172.21.15.62:52476) 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f248bfff640 1 -- 172.21.15.62:0/28915692 learned_addr learned my addr 172.21.15.62:0/28915692 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.995+0000 7f2490d44640 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229ed9a0 0x561d22a003b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f2491545640 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229f3370 0x561d22a07930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f248bfff640 1 -- 172.21.15.62:0/28915692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229ed9a0 msgr2=0x561d22a003b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f248bfff640 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229ed9a0 0x561d22a003b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f248bfff640 1 -- 172.21.15.62:0/28915692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229f3370 msgr2=0x561d22a07930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f248bfff640 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229f3370 0x561d22a07930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f248bfff640 1 -- 172.21.15.62:0/28915692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d22664210 con 0x561d229ee2e0 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f248bfff640 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ee2e0 0x561d22a008f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2478009740 tx=0x7f2478012d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c9a3c61317dc4dd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f246bfff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f246bfff640 1 -- 172.21.15.62:0/28915692 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478005050 con 0x561d229ee2e0 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f246bfff640 1 -- 172.21.15.62:0/28915692 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f24780051f0 con 0x561d229ee2e0 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.996+0000 7f2495e1d740 1 -- 172.21.15.62:0/28915692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2478012930 con 0x561d229ee2e0 2024-09-15T23:21:10.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f246bfff640 1 -- 172.21.15.62:0/28915692 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2478032050 con 0x561d229ee2e0 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f2495e1d740 10 client.26050 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f2495e1d740 1 -- 172.21.15.62:0/28915692 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561d22730620 con 0x561d229ee2e0 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f246bfff640 1 -- 172.21.15.62:0/28915692 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24780053f0 con 0x561d229ee2e0 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f246bfff640 1 -- 172.21.15.62:0/28915692 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f24780037a0 con 0x561d229ee2e0 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f2495e1d740 10 client.26050 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f2495e1d740 10 client.26050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f2495e1d740 20 client.26050 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f2495e1d740 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d22a0beb0 0x561d22a0e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f2495e1d740 4 client.26050 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f2495e1d740 1 -- 172.21.15.62:0/28915692 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2478012930 con 0x561d22a0beb0 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.997+0000 7f2490d44640 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d22a0beb0 0x561d22a0e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.998+0000 7f2490d44640 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d22a0beb0 0x561d22a0e2a0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e3a419e0c6b73fa server_cookie=bc4449dd5d8c80d5 in_seq=0 out_seq=0 2024-09-15T23:21:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.998+0000 7f246bfff640 10 client.26050 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.998+0000 7f246bfff640 1 -- 172.21.15.62:0/28915692 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f2478012930 con 0x561d22a0beb0 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.998+0000 7f246bfff640 10 client.26050 handle_command_reply: tid=0 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.998+0000 7f2495e1d740 1 client.26050 shutdown 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f2495e1d740 1 -- 172.21.15.62:0/28915692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ee2e0 msgr2=0x561d22a008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f2495e1d740 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ee2e0 0x561d22a008f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2478009740 tx=0x7f2478012d10 comp rx=0 tx=0).stop 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f2495e1d740 1 -- 172.21.15.62:0/28915692 shutdown_connections 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f2495e1d740 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d22a0beb0 0x561d22a0e2a0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f2495e1d740 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229f3370 0x561d22a07930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f246bfff640 0 client.26050 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f2495e1d740 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ee2e0 0x561d22a008f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f2495e1d740 1 --2- 172.21.15.62:0/28915692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229ed9a0 0x561d22a003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f2495e1d740 1 -- 172.21.15.62:0/28915692 >> 172.21.15.62:0/28915692 conn(0x561d229e9010 msgr2=0x561d229ea820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:09.999+0000 7f2495e1d740 1 -- 172.21.15.62:0/28915692 shutdown_connections 2024-09-15T23:21:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.000+0000 7f2495e1d740 1 -- 172.21.15.62:0/28915692 wait complete. 2024-09-15T23:21:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.000+0000 7f2495e1d740 20 client.26050 trim_cache size 0 max 16384 2024-09-15T23:21:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.022+0000 7f2495e1d740 1 Processor -- start 2024-09-15T23:21:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.022+0000 7f2495e1d740 1 -- start start 2024-09-15T23:21:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.022+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 0x561d229e80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.022+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d22a078f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d229ea400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f2495e1d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1aac90 con 0x561d229ee2e0 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f2495e1d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1a8560 con 0x561d229f3370 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f2495e1d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1a63b0 con 0x561d229ed9a0 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f248bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d22a078f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f248bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d22a078f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55160/0 (socket says 172.21.15.62:55160) 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f248bfff640 1 -- 172.21.15.62:0/3001093816 learned_addr learned my addr 172.21.15.62:0/3001093816 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f2490d44640 1 --2- 172.21.15.62:0/3001093816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 0x561d229e80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f2491545640 1 --2- 172.21.15.62:0/3001093816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d229ea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f248bfff640 1 -- 172.21.15.62:0/3001093816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 msgr2=0x561d229e80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f248bfff640 1 --2- 172.21.15.62:0/3001093816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 0x561d229e80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f248bfff640 1 -- 172.21.15.62:0/3001093816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 msgr2=0x561d229ea400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f248bfff640 1 --2- 172.21.15.62:0/3001093816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d229ea400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f248bfff640 1 -- 172.21.15.62:0/3001093816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d229ee7c0 con 0x561d229ee2e0 2024-09-15T23:21:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f248bfff640 1 --2- 172.21.15.62:0/3001093816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d22a078f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2478005fc0 tx=0x7f24780095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd051f34427b97b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f24767fc640 1 -- 172.21.15.62:0/3001093816 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247803a070 con 0x561d229ee2e0 2024-09-15T23:21:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f24767fc640 1 -- 172.21.15.62:0/3001093816 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24780099c0 con 0x561d229ee2e0 2024-09-15T23:21:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.023+0000 7f24767fc640 1 -- 172.21.15.62:0/3001093816 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478024940 con 0x561d229ee2e0 2024-09-15T23:21:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.025+0000 7f2495e1d740 1 -- 172.21.15.62:0/3001093816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 msgr2=0x561d22a078f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.025+0000 7f2495e1d740 1 --2- 172.21.15.62:0/3001093816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d22a078f0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2478005fc0 tx=0x7f24780095d0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.025+0000 7f2495e1d740 1 -- 172.21.15.62:0/3001093816 shutdown_connections 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.025+0000 7f2495e1d740 1 --2- 172.21.15.62:0/3001093816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d229ea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.025+0000 7f2495e1d740 1 --2- 172.21.15.62:0/3001093816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ee2e0 0x561d22a078f0 secure :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2478005fc0 tx=0x7f24780095d0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.025+0000 7f2495e1d740 1 --2- 172.21.15.62:0/3001093816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d229ed9a0 0x561d229e80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.025+0000 7f2495e1d740 1 -- 172.21.15.62:0/3001093816 >> 172.21.15.62:0/3001093816 conn(0x561d229e9010 msgr2=0x561d229e79a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.026+0000 7f2495e1d740 1 -- 172.21.15.62:0/3001093816 shutdown_connections 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.026+0000 7f2495e1d740 1 -- 172.21.15.62:0/3001093816 wait complete. 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.026+0000 7f2495e1d740 1 -- start start 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.026+0000 7f2495e1d740 1 Processor -- start 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ed9a0 0x561d22a05490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d22a059d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f2495e1d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d22a069e0 0x561d22a06e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f2495e1d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1a6a20 con 0x561d229ed9a0 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f2495e1d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1a7e90 con 0x561d229f3370 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f2495e1d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1a70f0 con 0x561d22a069e0 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f2495e1d740 10 client.0 _ll_register_callbacks cb 0x7f24800f3ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f248bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d22a059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f2490d44640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ed9a0 0x561d22a05490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f248bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d22a059d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54012/0 (socket says 172.21.15.62:54012) 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f248bfff640 1 -- 172.21.15.62:0/472765401 learned_addr learned my addr 172.21.15.62:0/472765401 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f2491545640 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d22a069e0 0x561d22a06e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f248bfff640 1 -- 172.21.15.62:0/472765401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d22a069e0 msgr2=0x561d22a06e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f248bfff640 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d22a069e0 0x561d22a06e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f248bfff640 1 -- 172.21.15.62:0/472765401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ed9a0 msgr2=0x561d22a05490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:10.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f248bfff640 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ed9a0 0x561d22a05490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f248bfff640 1 -- 172.21.15.62:0/472765401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d22a073d0 con 0x561d229f3370 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f248bfff640 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d22a059d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2478009740 tx=0x7f24780059b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70954142eb8be218 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.027+0000 7f246bfff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f246bfff640 1 -- 172.21.15.62:0/472765401 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247803a040 con 0x561d229f3370 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f246bfff640 1 -- 172.21.15.62:0/472765401 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f24780037f0 con 0x561d229f3370 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f2495e1d740 1 -- 172.21.15.62:0/472765401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24780099c0 con 0x561d229f3370 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f246bfff640 1 -- 172.21.15.62:0/472765401 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2478044050 con 0x561d229f3370 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f2495e1d740 10 client.26249 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f2495e1d740 1 -- 172.21.15.62:0/472765401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561d22a77380 con 0x561d229f3370 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f246bfff640 1 -- 172.21.15.62:0/472765401 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24780099c0 con 0x561d229f3370 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f246bfff640 1 -- 172.21.15.62:0/472765401 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f2478036400 con 0x561d229f3370 2024-09-15T23:21:10.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f2495e1d740 10 client.26249 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f2495e1d740 10 client.26249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f2495e1d740 20 client.26249 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f2495e1d740 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d22a77ca0 0x561d22a7a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f2495e1d740 4 client.26249 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:21:10.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.028+0000 7f2495e1d740 1 -- 172.21.15.62:0/472765401 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d2295ea20 con 0x561d22a77ca0 2024-09-15T23:21:10.034 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.029+0000 7f2490d44640 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d22a77ca0 0x561d22a7a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.034 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.030+0000 7f246bfff640 10 client.26249 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:10.034 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.030+0000 7f2490d44640 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d22a77ca0 0x561d22a7a0c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67cf018fefa4d65a server_cookie=e5fa0164e8fed43b in_seq=0 out_seq=0 2024-09-15T23:21:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f246bfff640 1 -- 172.21.15.62:0/472765401 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d2295ea20 con 0x561d22a77ca0 2024-09-15T23:21:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f246bfff640 10 client.26249 handle_command_reply: tid=0 2024-09-15T23:21:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f2495e1d740 1 client.26249 shutdown 2024-09-15T23:21:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f2495e1d740 1 -- 172.21.15.62:0/472765401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 msgr2=0x561d22a059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f2495e1d740 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d22a059d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2478009740 tx=0x7f24780059b0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f2495e1d740 1 -- 172.21.15.62:0/472765401 shutdown_connections 2024-09-15T23:21:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f2495e1d740 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561d22a77ca0 0x561d22a7a0c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f2495e1d740 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561d22a069e0 0x561d22a06e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f2495e1d740 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d229f3370 0x561d22a059d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f2495e1d740 1 --2- 172.21.15.62:0/472765401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d229ed9a0 0x561d22a05490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.039+0000 7f2495e1d740 1 -- 172.21.15.62:0/472765401 >> 172.21.15.62:0/472765401 conn(0x561d229e9010 msgr2=0x561d229e81c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.040+0000 7f246bfff640 0 client.26249 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.040+0000 7f2495e1d740 1 -- 172.21.15.62:0/472765401 shutdown_connections 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.040+0000 7f2495e1d740 1 -- 172.21.15.62:0/472765401 wait complete. 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.040+0000 7f2495e1d740 20 client.26249 trim_cache size 0 max 16384 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:21:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:21:10.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:21:10.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 -- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f247c0a3460 msgr2=0x7f247c0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f247c0a3460 0x7f247c0a5920 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2484010540 tx=0x7f2484007520 comp rx=0 tx=0).stop 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 -- 172.21.15.62:0/2145064861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c100a40 msgr2=0x7f248c1049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c100a40 0x7f248c1049d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2480031440 tx=0x7f2480032ce0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 -- 172.21.15.62:0/2145064861 shutdown_connections 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f247c0a3460 0x7f247c0a5920 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c1025f0 0x7f248c103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c101c30 0x7f248c103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 --2- 172.21.15.62:0/2145064861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c100a40 0x7f248c1049d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 -- 172.21.15.62:0/2145064861 >> 172.21.15.62:0/2145064861 conn(0x7f248c0fc1e0 msgr2=0x7f248c0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 -- 172.21.15.62:0/2145064861 shutdown_connections 2024-09-15T23:21:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:10.041+0000 7f2492fa8640 1 -- 172.21.15.62:0/2145064861 wait complete. 2024-09-15T23:21:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.462+0000 7fb3d64ad640 1 -- 172.21.15.62:0/522876126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d0102ef0 msgr2=0x7fb3d010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.462+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/522876126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d0102ef0 0x7fb3d010da40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8009cd0 tx=0x7fb3c802ea50 comp rx=0 tx=0).stop 2024-09-15T23:21:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.462+0000 7fb3d64ad640 1 -- 172.21.15.62:0/522876126 shutdown_connections 2024-09-15T23:21:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.462+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/522876126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d0102ef0 0x7fb3d010da40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.462+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/522876126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d0102570 0x7fb3d01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.462+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/522876126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d0100e20 0x7fb3d0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.462+0000 7fb3d64ad640 1 -- 172.21.15.62:0/522876126 >> 172.21.15.62:0/522876126 conn(0x7fb3d00fc7d0 msgr2=0x7fb3d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.462+0000 7fb3d64ad640 1 -- 172.21.15.62:0/522876126 shutdown_connections 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.462+0000 7fb3d64ad640 1 -- 172.21.15.62:0/522876126 wait complete. 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3d64ad640 1 Processor -- start 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3d64ad640 1 -- start start 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3d64ad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d0100e20 0x7fb3d01962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3d64ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d0102570 0x7fb3d01967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3d64ad640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d0102ef0 0x7fb3d019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3d64ad640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d010fe80 con 0x7fb3d0102570 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3d64ad640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d010fd00 con 0x7fb3d0102ef0 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3d64ad640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d0110000 con 0x7fb3d0100e20 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3cf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d0102570 0x7fb3d01967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3cf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d0102570 0x7fb3d01967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55202/0 (socket says 172.21.15.62:55202) 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3d4a4a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d0102ef0 0x7fb3d019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3cf7fe640 1 -- 172.21.15.62:0/2310362298 learned_addr learned my addr 172.21.15.62:0/2310362298 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.463+0000 7fb3cffff640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d0100e20 0x7fb3d01962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.465+0000 7fb3cffff640 1 -- 172.21.15.62:0/2310362298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d0102ef0 msgr2=0x7fb3d019ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.465+0000 7fb3cffff640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d0102ef0 0x7fb3d019ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.465+0000 7fb3cffff640 1 -- 172.21.15.62:0/2310362298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d0102570 msgr2=0x7fb3d01967e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.465+0000 7fb3cffff640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d0102570 0x7fb3d01967e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.465+0000 7fb3cffff640 1 -- 172.21.15.62:0/2310362298 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d019b250 con 0x7fb3d0100e20 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.465+0000 7fb3cffff640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d0100e20 0x7fb3d01962a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400d720 tx=0x7fb3c400dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.466+0000 7fb3cd7fa640 1 -- 172.21.15.62:0/2310362298 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4017070 con 0x7fb3d0100e20 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.466+0000 7fb3cd7fa640 1 -- 172.21.15.62:0/2310362298 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3c400de30 con 0x7fb3d0100e20 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.466+0000 7fb3cd7fa640 1 -- 172.21.15.62:0/2310362298 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40047a0 con 0x7fb3d0100e20 2024-09-15T23:21:10.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.466+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d019b540 con 0x7fb3d0100e20 2024-09-15T23:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.466+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d019b970 con 0x7fb3d0100e20 2024-09-15T23:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.467+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d0101200 con 0x7fb3d0100e20 2024-09-15T23:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.468+0000 7fb3cd7fa640 1 -- 172.21.15.62:0/2310362298 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb3c40099d0 con 0x7fb3d0100e20 2024-09-15T23:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.469+0000 7fb3cd7fa640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3b80a3410 0x7fb3b80a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.469+0000 7fb3cd7fa640 1 -- 172.21.15.62:0/2310362298 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb3c40cf820 con 0x7fb3d0100e20 2024-09-15T23:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.470+0000 7fb3cf7fe640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3b80a3410 0x7fb3b80a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.471+0000 7fb3cf7fe640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3b80a3410 0x7fb3b80a58d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00feca0 tx=0x7fb3c000d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:10.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.474+0000 7fb3cd7fa640 1 -- 172.21.15.62:0/2310362298 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb3c4020020 con 0x7fb3d0100e20 2024-09-15T23:21:10.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.842+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3d0109090 con 0x7fb3d0100e20 2024-09-15T23:21:10.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.843+0000 7fb3cd7fa640 1 -- 172.21.15.62:0/2310362298 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11903 (secure 0 0 0) 0x7fb3c409c160 con 0x7fb3d0100e20 2024-09-15T23:21:10.845 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:21:10.845 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":31,"btime":"2024-09-15T23:20:13:209709+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:20:12.378892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:21:10.845 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 31 2024-09-15T23:21:10.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3b80a3410 msgr2=0x7fb3b80a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3b80a3410 0x7fb3b80a58d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00feca0 tx=0x7fb3c000d040 comp rx=0 tx=0).stop 2024-09-15T23:21:10.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d0100e20 msgr2=0x7fb3d01962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:10.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d0100e20 0x7fb3d01962a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb3c400d720 tx=0x7fb3c400dbe0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 shutdown_connections 2024-09-15T23:21:10.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3b80a3410 0x7fb3b80a58d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3d0102ef0 0x7fb3d019ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3d0102570 0x7fb3d01967e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 --2- 172.21.15.62:0/2310362298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb3d0100e20 0x7fb3d01962a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:10.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.847+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 >> 172.21.15.62:0/2310362298 conn(0x7fb3d00fc7d0 msgr2=0x7fb3d0104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:10.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.848+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 shutdown_connections 2024-09-15T23:21:10.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:10.848+0000 7fb3d64ad640 1 -- 172.21.15.62:0/2310362298 wait complete. 2024-09-15T23:21:11.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:11 smithi062 ceph-mon[30949]: pgmap v517: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 14 MiB/s wr, 725 op/s 2024-09-15T23:21:11.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:11 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2310362298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:11.681 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0' 2024-09-15T23:21:11.699 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 145.35 --await-for 45.0 2024-09-15T23:21:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:11 smithi067 ceph-mon[35665]: pgmap v517: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 14 MiB/s wr, 725 op/s 2024-09-15T23:21:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:11 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2310362298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:11.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:11 smithi148 ceph-mon[35610]: pgmap v517: 129 pgs: 129 active+clean; 1.1 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 14 MiB/s wr, 725 op/s 2024-09-15T23:21:11.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:11 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2310362298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:12.179 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:12.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.818+0000 7f3eb5370640 1 -- 172.21.15.62:0/831287809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3eb0100100 msgr2=0x7f3eb0100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:12.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.818+0000 7f3eb5370640 1 --2- 172.21.15.62:0/831287809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3eb0100100 0x7f3eb0100560 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009860 tx=0x7f3ea402a340 comp rx=0 tx=0).stop 2024-09-15T23:21:12.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.818+0000 7f3eb5370640 1 -- 172.21.15.62:0/831287809 shutdown_connections 2024-09-15T23:21:12.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.818+0000 7f3eb5370640 1 --2- 172.21.15.62:0/831287809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3eb0100aa0 0x7f3eb010ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.818+0000 7f3eb5370640 1 --2- 172.21.15.62:0/831287809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3eb0100100 0x7f3eb0100560 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.818+0000 7f3eb5370640 1 --2- 172.21.15.62:0/831287809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb0106b80 0x7f3eb0106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.818+0000 7f3eb5370640 1 -- 172.21.15.62:0/831287809 >> 172.21.15.62:0/831287809 conn(0x7f3eb0077230 msgr2=0x7f3eb00fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:12.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.818+0000 7f3eb5370640 1 -- 172.21.15.62:0/831287809 shutdown_connections 2024-09-15T23:21:12.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.818+0000 7f3eb5370640 1 -- 172.21.15.62:0/831287809 wait complete. 2024-09-15T23:21:12.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.819+0000 7f3eb5370640 1 Processor -- start 2024-09-15T23:21:12.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.819+0000 7f3eb5370640 1 -- start start 2024-09-15T23:21:12.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eb5370640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3eb0100100 0x7f3eb01079c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eb5370640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb0100aa0 0x7f3eb0107f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eb5370640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3eb0106b80 0x7f3eb0108440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eb5370640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0112050 con 0x7f3eb0100aa0 2024-09-15T23:21:12.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eb5370640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0111ed0 con 0x7f3eb0106b80 2024-09-15T23:21:12.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eb5370640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01121d0 con 0x7f3eb0100100 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3eb0106b80 0x7f3eb0108440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eae7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb0100aa0 0x7f3eb0107f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3eb0106b80 0x7f3eb0108440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:43584/0 (socket says 172.21.15.62:43584) 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.820+0000 7f3eaf7fe640 1 -- 172.21.15.62:0/3198531458 learned_addr learned my addr 172.21.15.62:0/3198531458 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.821+0000 7f3eaf7fe640 1 -- 172.21.15.62:0/3198531458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3eb0100100 msgr2=0x7f3eb01079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.821+0000 7f3eaf7fe640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3eb0100100 0x7f3eb01079c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.821+0000 7f3eaf7fe640 1 -- 172.21.15.62:0/3198531458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb0100aa0 msgr2=0x7f3eb0107f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.821+0000 7f3eaf7fe640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb0100aa0 0x7f3eb0107f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.821+0000 7f3eaf7fe640 1 -- 172.21.15.62:0/3198531458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb019a4d0 con 0x7f3eb0106b80 2024-09-15T23:21:12.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.821+0000 7f3eaf7fe640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3eb0106b80 0x7f3eb0108440 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800eb70 tx=0x7f3ea800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:12.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.821+0000 7f3e97fff640 1 -- 172.21.15.62:0/3198531458 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8004d20 con 0x7f3eb0106b80 2024-09-15T23:21:12.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.821+0000 7f3eb5370640 1 -- 172.21.15.62:0/3198531458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb019a7c0 con 0x7f3eb0106b80 2024-09-15T23:21:12.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.821+0000 7f3eb5370640 1 -- 172.21.15.62:0/3198531458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb019acd0 con 0x7f3eb0106b80 2024-09-15T23:21:12.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.830+0000 7f3e97fff640 1 -- 172.21.15.62:0/3198531458 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ea800ce70 con 0x7f3eb0106b80 2024-09-15T23:21:12.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.831+0000 7f3e97fff640 1 -- 172.21.15.62:0/3198531458 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80107b0 con 0x7f3eb0106b80 2024-09-15T23:21:12.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.839+0000 7f3e97fff640 1 -- 172.21.15.62:0/3198531458 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3ea8004ec0 con 0x7f3eb0106b80 2024-09-15T23:21:12.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.840+0000 7f3e97fff640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e980a3260 0x7f3e980a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.840+0000 7f3e97fff640 1 -- 172.21.15.62:0/3198531458 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3ea80d0020 con 0x7f3eb0106b80 2024-09-15T23:21:12.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.842+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e980a3260 0x7f3e980a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.843+0000 7f3eb7749740 1 Processor -- start 2024-09-15T23:21:12.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.850+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e980a3260 0x7f3e980a5720 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00058c0 tx=0x7f3ea0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:12.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.853+0000 7f3eb7749740 1 -- start start 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.853+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d6dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.853+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6e180 0x55b5e6d78500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.853+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d7ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.853+0000 7f3eb7749740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b5e6cc1160 con 0x55b5e6d6d860 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.853+0000 7f3eb7749740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b5e6c68720 con 0x55b5e6d6e180 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.853+0000 7f3eb7749740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b5e6cbd120 con 0x55b5e6d78a40 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.854+0000 7f3eae7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6e180 0x55b5e6d78500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.854+0000 7f3eae7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6e180 0x55b5e6d78500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:43596/0 (socket says 172.21.15.62:43596) 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.854+0000 7f3eae7fc640 1 -- 172.21.15.62:0/1337709032 learned_addr learned my addr 172.21.15.62:0/1337709032 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.854+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/1337709032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d6dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.855+0000 7f3eae7fc640 1 -- 172.21.15.62:0/1337709032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 msgr2=0x55b5e6d7ae30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.855+0000 7f3eae7fc640 1 --2- 172.21.15.62:0/1337709032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d7ae30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.855+0000 7f3eae7fc640 1 -- 172.21.15.62:0/1337709032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 msgr2=0x55b5e6d6dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:12.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.861+0000 7f3eae7fc640 1 --2- 172.21.15.62:0/1337709032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d6dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.861+0000 7f3eae7fc640 1 -- 172.21.15.62:0/1337709032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e6a32ba0 con 0x55b5e6d6e180 2024-09-15T23:21:12.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.862+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/1337709032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d6dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:12.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.862+0000 7f3eae7fc640 1 --2- 172.21.15.62:0/1337709032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6e180 0x55b5e6d78500 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4000c00 tx=0x7f3ea4030340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=202e38da7474a152 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:12.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.863+0000 7f3e957fa640 1 -- 172.21.15.62:0/1337709032 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4037040 con 0x55b5e6d6e180 2024-09-15T23:21:12.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.863+0000 7f3e957fa640 1 -- 172.21.15.62:0/1337709032 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ea4063070 con 0x55b5e6d6e180 2024-09-15T23:21:12.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.863+0000 7f3e957fa640 1 -- 172.21.15.62:0/1337709032 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea40052a0 con 0x55b5e6d6e180 2024-09-15T23:21:12.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.864+0000 7f3eb7749740 1 -- 172.21.15.62:0/1337709032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6e180 msgr2=0x55b5e6d78500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:12.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.864+0000 7f3eb7749740 1 --2- 172.21.15.62:0/1337709032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6e180 0x55b5e6d78500 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4000c00 tx=0x7f3ea4030340 comp rx=0 tx=0).stop 2024-09-15T23:21:12.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.866+0000 7f3eb7749740 1 -- 172.21.15.62:0/1337709032 shutdown_connections 2024-09-15T23:21:12.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.866+0000 7f3eb7749740 1 --2- 172.21.15.62:0/1337709032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d7ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.866+0000 7f3eb7749740 1 --2- 172.21.15.62:0/1337709032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6e180 0x55b5e6d78500 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.866+0000 7f3eb7749740 1 --2- 172.21.15.62:0/1337709032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d6dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.867+0000 7f3eb7749740 1 -- 172.21.15.62:0/1337709032 >> 172.21.15.62:0/1337709032 conn(0x55b5e6d69df0 msgr2=0x55b5e6d6a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:12.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.868+0000 7f3eb7749740 1 -- 172.21.15.62:0/1337709032 shutdown_connections 2024-09-15T23:21:12.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.869+0000 7f3eb7749740 1 -- 172.21.15.62:0/1337709032 wait complete. 2024-09-15T23:21:12.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.870+0000 7f3eb7749740 1 -- start start 2024-09-15T23:21:12.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.870+0000 7f3eb7749740 1 Processor -- start 2024-09-15T23:21:12.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.875+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d75950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.875+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d6e180 0x55b5e6d75e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.875+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d78a40 0x55b5e6d763d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.875+0000 7f3eb7749740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b5e6d7b610 con 0x55b5e6d6d860 2024-09-15T23:21:12.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.875+0000 7f3eb7749740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b5e6c49900 con 0x55b5e6d78a40 2024-09-15T23:21:12.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.875+0000 7f3eb7749740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b5e6d7b790 con 0x55b5e6d6e180 2024-09-15T23:21:12.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.875+0000 7f3eb7749740 10 client.0 _ll_register_callbacks cb 0x55b5e6c47780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:12.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.876+0000 7f3eae7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d6e180 0x55b5e6d75e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.876+0000 7f3eae7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d6e180 0x55b5e6d75e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:40914/0 (socket says 172.21.15.62:40914) 2024-09-15T23:21:12.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.876+0000 7f3eae7fc640 1 -- 172.21.15.62:0/750630164 learned_addr learned my addr 172.21.15.62:0/750630164 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:12.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.876+0000 7f3eaf7fe640 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d78a40 0x55b5e6d763d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.877+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d75950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.878+0000 7f3eae7fc640 1 -- 172.21.15.62:0/750630164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d78a40 msgr2=0x55b5e6d763d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:12.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.878+0000 7f3eae7fc640 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d78a40 0x55b5e6d763d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.878+0000 7f3eae7fc640 1 -- 172.21.15.62:0/750630164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 msgr2=0x55b5e6d75950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:12.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.878+0000 7f3eae7fc640 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d75950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.878+0000 7f3eae7fc640 1 -- 172.21.15.62:0/750630164 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e6c89c50 con 0x55b5e6d6e180 2024-09-15T23:21:12.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.879+0000 7f3eae7fc640 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d6e180 0x55b5e6d75e90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4005b60 tx=0x7f3ea4005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=119bbfb14a1d9186 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:12.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.879+0000 7f3e92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:21:12.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.879+0000 7f3e92ff5640 1 -- 172.21.15.62:0/750630164 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4037070 con 0x55b5e6d6e180 2024-09-15T23:21:12.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.879+0000 7f3eb7749740 1 -- 172.21.15.62:0/750630164 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ea4063070 con 0x55b5e6d6e180 2024-09-15T23:21:12.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.879+0000 7f3e92ff5640 1 -- 172.21.15.62:0/750630164 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3ea4004030 con 0x55b5e6d6e180 2024-09-15T23:21:12.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.879+0000 7f3e92ff5640 1 -- 172.21.15.62:0/750630164 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3ea4075050 con 0x55b5e6d6e180 2024-09-15T23:21:12.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.882+0000 7f3eb7749740 10 client.26068 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:12.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.882+0000 7f3eb7749740 1 -- 172.21.15.62:0/750630164 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b5e6d7b610 con 0x55b5e6d6e180 2024-09-15T23:21:12.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.882+0000 7f3e92ff5640 1 -- 172.21.15.62:0/750630164 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ea4063070 con 0x55b5e6d6e180 2024-09-15T23:21:12.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.884+0000 7f3e92ff5640 1 -- 172.21.15.62:0/750630164 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f3ea40047d0 con 0x55b5e6d6e180 2024-09-15T23:21:12.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.884+0000 7f3eb7749740 10 client.26068 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:12.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.884+0000 7f3eb7749740 10 client.26068 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:21:12.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.884+0000 7f3eb7749740 20 client.26068 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:12.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.884+0000 7f3eb7749740 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5e6d888a0 0x55b5e6d8acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.884+0000 7f3eb7749740 4 client.26068 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:12.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.884+0000 7f3eb7749740 1 -- 172.21.15.62:0/750630164 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5e6cce890 con 0x55b5e6d888a0 2024-09-15T23:21:12.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.887+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5e6d888a0 0x55b5e6d8acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.892+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5e6d888a0 0x55b5e6d8acc0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3315b21d5e6b4cba server_cookie=8ac5c44f50a185b9 in_seq=0 out_seq=0 2024-09-15T23:21:12.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.892+0000 7f3e92ff5640 10 client.26068 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:12.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.894+0000 7f3e92ff5640 1 -- 172.21.15.62:0/750630164 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5e6cce890 con 0x55b5e6d888a0 2024-09-15T23:21:12.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.894+0000 7f3e92ff5640 10 client.26068 handle_command_reply: tid=0 2024-09-15T23:21:12.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.894+0000 7f3eb7749740 1 client.26068 shutdown 2024-09-15T23:21:12.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.895+0000 7f3eb7749740 1 -- 172.21.15.62:0/750630164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d6e180 msgr2=0x55b5e6d75e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:12.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.895+0000 7f3eb7749740 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d6e180 0x55b5e6d75e90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4005b60 tx=0x7f3ea4005f30 comp rx=0 tx=0).stop 2024-09-15T23:21:12.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.895+0000 7f3eb7749740 1 -- 172.21.15.62:0/750630164 shutdown_connections 2024-09-15T23:21:12.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.895+0000 7f3eb7749740 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5e6d888a0 0x55b5e6d8acc0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.895+0000 7f3eb7749740 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d78a40 0x55b5e6d763d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.895+0000 7f3eb7749740 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d6e180 0x55b5e6d75e90 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4005b60 tx=0x7f3ea4005f30 comp rx=0 tx=0).stop 2024-09-15T23:21:12.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.895+0000 7f3eb7749740 1 --2- 172.21.15.62:0/750630164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d75950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.895+0000 7f3eb7749740 1 -- 172.21.15.62:0/750630164 >> 172.21.15.62:0/750630164 conn(0x55b5e6d69df0 msgr2=0x55b5e6d6fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:12.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.895+0000 7f3e92ff5640 0 client.26068 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:12.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.896+0000 7f3eb7749740 1 -- 172.21.15.62:0/750630164 shutdown_connections 2024-09-15T23:21:12.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.896+0000 7f3eb7749740 1 -- 172.21.15.62:0/750630164 wait complete. 2024-09-15T23:21:12.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.897+0000 7f3eb7749740 20 client.26068 trim_cache size 0 max 16384 2024-09-15T23:21:12.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.926+0000 7f3eb7749740 1 Processor -- start 2024-09-15T23:21:12.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.926+0000 7f3eb7749740 1 -- start start 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.926+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6d860 0x55b5e6d76220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.926+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d905d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.926+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d90b10 0x55b5e6d72bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.926+0000 7f3eb7749740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c012370 con 0x55b5e6d90b10 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.926+0000 7f3eb7749740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c00fc40 con 0x55b5e6d6d860 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.926+0000 7f3eb7749740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c00db40 con 0x55b5e6d78a40 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d90b10 0x55b5e6d72bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d90b10 0x55b5e6d72bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43998/0 (socket says 172.21.15.62:43998) 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaf7fe640 1 -- 172.21.15.62:0/4008384388 learned_addr learned my addr 172.21.15.62:0/4008384388 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/4008384388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6d860 0x55b5e6d76220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaeffd640 1 -- 172.21.15.62:0/4008384388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 msgr2=0x55b5e6d905d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:12.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/4008384388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d905d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaeffd640 1 -- 172.21.15.62:0/4008384388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d90b10 msgr2=0x55b5e6d72bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/4008384388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d90b10 0x55b5e6d72bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaeffd640 1 -- 172.21.15.62:0/4008384388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e6d74900 con 0x55b5e6d6d860 2024-09-15T23:21:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/4008384388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6d860 0x55b5e6d76220 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55b5e6d88c80 tx=0x7f3ea0011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30879c1e6c177165 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3e94ff9640 1 -- 172.21.15.62:0/4008384388 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0005020 con 0x55b5e6d6d860 2024-09-15T23:21:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.927+0000 7f3e94ff9640 1 -- 172.21.15.62:0/4008384388 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ea00047a0 con 0x55b5e6d6d860 2024-09-15T23:21:12.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3e94ff9640 1 -- 172.21.15.62:0/4008384388 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0028610 con 0x55b5e6d6d860 2024-09-15T23:21:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3eb7749740 1 -- 172.21.15.62:0/4008384388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6d860 msgr2=0x55b5e6d76220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3eb7749740 1 --2- 172.21.15.62:0/4008384388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6d860 0x55b5e6d76220 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x55b5e6d88c80 tx=0x7f3ea0011000 comp rx=0 tx=0).stop 2024-09-15T23:21:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3eb7749740 1 -- 172.21.15.62:0/4008384388 shutdown_connections 2024-09-15T23:21:12.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3eb7749740 1 --2- 172.21.15.62:0/4008384388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d90b10 0x55b5e6d72bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3eb7749740 1 --2- 172.21.15.62:0/4008384388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d905d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3eb7749740 1 --2- 172.21.15.62:0/4008384388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d6d860 0x55b5e6d76220 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3eb7749740 1 -- 172.21.15.62:0/4008384388 >> 172.21.15.62:0/4008384388 conn(0x55b5e6d69df0 msgr2=0x55b5e6d779d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3eb7749740 1 -- 172.21.15.62:0/4008384388 shutdown_connections 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.928+0000 7f3eb7749740 1 -- 172.21.15.62:0/4008384388 wait complete. 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.929+0000 7f3eb7749740 1 -- start start 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.929+0000 7f3eb7749740 1 Processor -- start 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.929+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d7ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.929+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d7cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.929+0000 7f3eb7749740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d90b10 0x55b5e6d7d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.929+0000 7f3eb7749740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c00e100 con 0x55b5e6d6d860 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.929+0000 7f3eb7749740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c00f570 con 0x55b5e6d90b10 2024-09-15T23:21:12.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.929+0000 7f3eb7749740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c00e7d0 con 0x55b5e6d78a40 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaeffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d7ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d90b10 0x55b5e6d7d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaeffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d7ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44012/0 (socket says 172.21.15.62:44012) 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d90b10 0x55b5e6d7d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:43626/0 (socket says 172.21.15.62:43626) 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaeffd640 1 -- 172.21.15.62:0/2000248953 learned_addr learned my addr 172.21.15.62:0/2000248953 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eae7fc640 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d7cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eb7749740 10 client.0 _ll_register_callbacks cb 0x7f3ea0014050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaf7fe640 1 -- 172.21.15.62:0/2000248953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 msgr2=0x55b5e6d7cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaf7fe640 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d7cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaf7fe640 1 -- 172.21.15.62:0/2000248953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 msgr2=0x55b5e6d7ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaf7fe640 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d7ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaf7fe640 1 -- 172.21.15.62:0/2000248953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e6d70810 con 0x55b5e6d90b10 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d7ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:12.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3eaf7fe640 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d90b10 0x55b5e6d7d510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8005a00 tx=0x7f3ea809ca20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82781a372c240a45 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.930+0000 7f3e92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.931+0000 7f3e92ff5640 1 -- 172.21.15.62:0/2000248953 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80a4c40 con 0x55b5e6d90b10 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.931+0000 7f3e92ff5640 1 -- 172.21.15.62:0/2000248953 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3ea80bc0b0 con 0x55b5e6d90b10 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.931+0000 7f3eb7749740 1 -- 172.21.15.62:0/2000248953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ea00047a0 con 0x55b5e6d90b10 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.931+0000 7f3e92ff5640 1 -- 172.21.15.62:0/2000248953 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3ea80a4ed0 con 0x55b5e6d90b10 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.931+0000 7f3eb7749740 10 client.26273 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.931+0000 7f3eb7749740 1 -- 172.21.15.62:0/2000248953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b5e6df3930 con 0x55b5e6d90b10 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.931+0000 7f3e92ff5640 1 -- 172.21.15.62:0/2000248953 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ea80bc2b0 con 0x55b5e6d90b10 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.932+0000 7f3e92ff5640 1 -- 172.21.15.62:0/2000248953 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f3ea80aa080 con 0x55b5e6d90b10 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.932+0000 7f3eb7749740 10 client.26273 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.932+0000 7f3eb7749740 10 client.26273 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.932+0000 7f3eb7749740 20 client.26273 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.933+0000 7f3eb7749740 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5e6df40c0 0x55b5e6df64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:12.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.933+0000 7f3eb7749740 4 client.26273 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}] 2024-09-15T23:21:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.933+0000 7f3eb7749740 1 -- 172.21.15.62:0/2000248953 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 145.35, "await_for": 45.0}) -- 0x7f3ea00047a0 con 0x55b5e6df40c0 2024-09-15T23:21:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.933+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5e6df40c0 0x55b5e6df64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.933+0000 7f3eaeffd640 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5e6df40c0 0x55b5e6df64b0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca420f0ac4ce187d server_cookie=315fcd4f4bc1bd6d in_seq=0 out_seq=0 2024-09-15T23:21:12.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:12.934+0000 7f3e92ff5640 10 client.26273 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:13.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.180+0000 7f3e92ff5640 1 -- 172.21.15.62:0/2000248953 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f3ea00047a0 con 0x55b5e6df40c0 2024-09-15T23:21:13.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.180+0000 7f3e92ff5640 10 client.26273 handle_command_reply: tid=0 2024-09-15T23:21:13.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.180+0000 7f3eb7749740 1 client.26273 shutdown 2024-09-15T23:21:13.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 -- 172.21.15.62:0/2000248953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d90b10 msgr2=0x55b5e6d7d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d90b10 0x55b5e6d7d510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8005a00 tx=0x7f3ea809ca20 comp rx=0 tx=0).stop 2024-09-15T23:21:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 -- 172.21.15.62:0/2000248953 shutdown_connections 2024-09-15T23:21:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5e6df40c0 0x55b5e6df64b0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b5e6d90b10 0x55b5e6d7d510 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3e92ff5640 0 client.26273 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:13.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b5e6d78a40 0x55b5e6d7cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:13.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 --2- 172.21.15.62:0/2000248953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5e6d6d860 0x55b5e6d7ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:13.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 -- 172.21.15.62:0/2000248953 >> 172.21.15.62:0/2000248953 conn(0x55b5e6d69df0 msgr2=0x55b5e6d576a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:13.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 -- 172.21.15.62:0/2000248953 shutdown_connections 2024-09-15T23:21:13.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 1 -- 172.21.15.62:0/2000248953 wait complete. 2024-09-15T23:21:13.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.181+0000 7f3eb7749740 20 client.26273 trim_cache size 0 max 16384 2024-09-15T23:21:13.186 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:21:13.186 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 31, 2024-09-15T23:21:13.186 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:21:13.186 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 22, 2024-09-15T23:21:13.186 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:21:13.187 INFO:teuthology.orchestra.run.smithi062.stdout: "9b36dcb4": { 2024-09-15T23:21:13.187 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 22, 2024-09-15T23:21:13.187 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:21:13.187 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:21:13.187 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:21:13.187 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:21:13.187 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:21:13.188 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:21:13.188 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 145.3, 2024-09-15T23:21:13.188 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:21:13.188 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:21:13.188 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:21:13.188 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:21:13.189 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:21:13.189 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:21:13.189 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:13.189 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:13.189 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:13.189 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:13.190 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:13.190 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:21:13.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 -- 172.21.15.62:0/3198531458 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e980a3260 msgr2=0x7f3e980a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:13.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e980a3260 0x7f3e980a5720 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3ea00058c0 tx=0x7f3ea0005850 comp rx=0 tx=0).stop 2024-09-15T23:21:13.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 -- 172.21.15.62:0/3198531458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3eb0106b80 msgr2=0x7f3eb0108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:13.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3eb0106b80 0x7f3eb0108440 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800eb70 tx=0x7f3ea800c6a0 comp rx=0 tx=0).stop 2024-09-15T23:21:13.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 -- 172.21.15.62:0/3198531458 shutdown_connections 2024-09-15T23:21:13.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e980a3260 0x7f3e980a5720 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3eb0106b80 0x7f3eb0108440 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3eb0100aa0 0x7f3eb0107f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 --2- 172.21.15.62:0/3198531458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3eb0100100 0x7f3eb01079c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 -- 172.21.15.62:0/3198531458 >> 172.21.15.62:0/3198531458 conn(0x7f3eb0077230 msgr2=0x7f3eb01027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 -- 172.21.15.62:0/3198531458 shutdown_connections 2024-09-15T23:21:13.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:13.182+0000 7f3eb5370640 1 -- 172.21.15.62:0/3198531458 wait complete. 2024-09-15T23:21:13.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:13 smithi062 ceph-mon[30949]: pgmap v518: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 11 MiB/s wr, 730 op/s 2024-09-15T23:21:13.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:13 smithi067 ceph-mon[35665]: pgmap v518: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 11 MiB/s wr, 730 op/s 2024-09-15T23:21:13.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:13 smithi148 ceph-mon[35610]: pgmap v518: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 11 MiB/s wr, 730 op/s 2024-09-15T23:21:14.571 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 295.0 s, 1014.1 tps, lat 69.694 ms stddev 73.829 2024-09-15T23:21:15.038 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '9b36dcb4', quiesce took 5.7 seconds. Will release after: 19.650000000000002 2024-09-15T23:21:15.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:15 smithi062 ceph-mon[30949]: pgmap v519: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8.4 MiB/s wr, 683 op/s 2024-09-15T23:21:15.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:15 smithi067 ceph-mon[35665]: pgmap v519: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8.4 MiB/s wr, 683 op/s 2024-09-15T23:21:15.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:15 smithi148 ceph-mon[35610]: pgmap v519: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8.4 MiB/s wr, 683 op/s 2024-09-15T23:21:17.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:17 smithi062 ceph-mon[30949]: pgmap v520: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8.7 MiB/s wr, 720 op/s 2024-09-15T23:21:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:17 smithi067 ceph-mon[35665]: pgmap v520: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8.7 MiB/s wr, 720 op/s 2024-09-15T23:21:17.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:17 smithi148 ceph-mon[35610]: pgmap v520: 129 pgs: 129 active+clean; 1.1 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 8.7 MiB/s wr, 720 op/s 2024-09-15T23:21:19.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 300.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:21:19.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:19 smithi062 ceph-mon[30949]: pgmap v521: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.8 MiB/s wr, 731 op/s 2024-09-15T23:21:19.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:19 smithi067 ceph-mon[35665]: pgmap v521: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.8 MiB/s wr, 731 op/s 2024-09-15T23:21:19.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:19 smithi148 ceph-mon[35610]: pgmap v521: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.8 MiB/s wr, 731 op/s 2024-09-15T23:21:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:21 smithi062 ceph-mon[30949]: pgmap v522: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.4 MiB/s wr, 700 op/s 2024-09-15T23:21:21.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:21 smithi067 ceph-mon[35665]: pgmap v522: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.4 MiB/s wr, 700 op/s 2024-09-15T23:21:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:21 smithi148 ceph-mon[35610]: pgmap v522: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 8.4 MiB/s wr, 700 op/s 2024-09-15T23:21:21.830 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:21:22.105 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:22.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.594+0000 7f78773ac640 1 -- 172.21.15.62:0/3370430650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870101df0 msgr2=0x7f78701021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:22.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.594+0000 7f78773ac640 1 --2- 172.21.15.62:0/3370430650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870101df0 0x7f78701021d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f786c009900 tx=0x7f786c02a410 comp rx=0 tx=0).stop 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.594+0000 7f78773ac640 1 -- 172.21.15.62:0/3370430650 shutdown_connections 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.594+0000 7f78773ac640 1 --2- 172.21.15.62:0/3370430650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7870106f20 0x7f787010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.594+0000 7f78773ac640 1 --2- 172.21.15.62:0/3370430650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78701027a0 0x7f78701067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.594+0000 7f78773ac640 1 --2- 172.21.15.62:0/3370430650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870101df0 0x7f78701021d0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.594+0000 7f78773ac640 1 -- 172.21.15.62:0/3370430650 >> 172.21.15.62:0/3370430650 conn(0x7f78700fd8b0 msgr2=0x7f78700ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.594+0000 7f78773ac640 1 -- 172.21.15.62:0/3370430650 shutdown_connections 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.595+0000 7f78773ac640 1 -- 172.21.15.62:0/3370430650 wait complete. 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.595+0000 7f78773ac640 1 Processor -- start 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.595+0000 7f78773ac640 1 -- start start 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f78773ac640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7870101df0 0x7f78701983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f78773ac640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78701027a0 0x7f7870198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f78773ac640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870106f20 0x7f787019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f78773ac640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7870111f80 con 0x7f7870106f20 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f78773ac640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7870111e00 con 0x7f7870101df0 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f78773ac640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7870112100 con 0x7f78701027a0 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875949640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870106f20 0x7f787019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875148640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7870101df0 0x7f78701983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875949640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870106f20 0x7f787019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51534/0 (socket says 172.21.15.62:51534) 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875949640 1 -- 172.21.15.62:0/2488359910 learned_addr learned my addr 172.21.15.62:0/2488359910 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7874947640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78701027a0 0x7f7870198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875949640 1 -- 172.21.15.62:0/2488359910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78701027a0 msgr2=0x7f7870198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875949640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78701027a0 0x7f7870198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875949640 1 -- 172.21.15.62:0/2488359910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7870101df0 msgr2=0x7f78701983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:22.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875949640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7870101df0 0x7f78701983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875949640 1 -- 172.21.15.62:0/2488359910 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f787019d370 con 0x7f7870106f20 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875148640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7870101df0 0x7f78701983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.596+0000 7f7875949640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870106f20 0x7f787019cc90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f786800eaa0 tx=0x7f786800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.597+0000 7f78667fc640 1 -- 172.21.15.62:0/2488359910 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7868004d20 con 0x7f7870106f20 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.597+0000 7f78773ac640 1 -- 172.21.15.62:0/2488359910 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f787019d660 con 0x7f7870106f20 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.597+0000 7f78667fc640 1 -- 172.21.15.62:0/2488359910 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7868005020 con 0x7f7870106f20 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.597+0000 7f78667fc640 1 -- 172.21.15.62:0/2488359910 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78680116d0 con 0x7f7870106f20 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.597+0000 7f78773ac640 1 -- 172.21.15.62:0/2488359910 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78701a4f40 con 0x7f7870106f20 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f78667fc640 1 -- 172.21.15.62:0/2488359910 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f78680051d0 con 0x7f7870106f20 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f7879785740 1 Processor -- start 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f7879785740 1 -- start start 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f7879785740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040c4a0 0x55ac7040c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f7879785740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac7040cdc0 0x55ac70417140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f7879785740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac70417680 0x55ac70419a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f7879785740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ac703722e0 con 0x55ac70417680 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f7879785740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ac70319ce0 con 0x55ac7040cdc0 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f7879785740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ac7036e2a0 con 0x55ac7040c4a0 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.600+0000 7f78667fc640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f785c0a3240 0x7f785c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac70417680 0x55ac70419a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac70417680 0x55ac70419a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51542/0 (socket says 172.21.15.62:51542) 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 -- 172.21.15.62:0/4137697956 learned_addr learned my addr 172.21.15.62:0/4137697956 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040c4a0 0x55ac7040c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7875148640 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac7040cdc0 0x55ac70417140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7875949640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f785c0a3240 0x7f785c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f78667fc640 1 -- 172.21.15.62:0/2488359910 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78680cfd50 con 0x7f7870106f20 2024-09-15T23:21:22.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 -- 172.21.15.62:0/4137697956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040c4a0 msgr2=0x55ac7040c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040c4a0 0x55ac7040c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 -- 172.21.15.62:0/4137697956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac7040cdc0 msgr2=0x55ac70417140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac7040cdc0 0x55ac70417140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 -- 172.21.15.62:0/4137697956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac702fe440 con 0x55ac70417680 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7875148640 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac7040cdc0 0x55ac70417140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7874947640 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac70417680 0x55ac70419a70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f78600117e0 tx=0x7f786000a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ae7628fee8802a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f7875949640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f785c0a3240 0x7f785c0a5700 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f786809bcf0 tx=0x7f7868066ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f785b7fe640 1 -- 172.21.15.62:0/4137697956 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7860005020 con 0x55ac70417680 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f785b7fe640 1 -- 172.21.15.62:0/4137697956 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78600047a0 con 0x55ac70417680 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.601+0000 7f785b7fe640 1 -- 172.21.15.62:0/4137697956 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786000fa30 con 0x55ac70417680 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.602+0000 7f7879785740 1 -- 172.21.15.62:0/4137697956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac70417680 msgr2=0x55ac70419a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.602+0000 7f7879785740 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac70417680 0x55ac70419a70 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f78600117e0 tx=0x7f786000a760 comp rx=0 tx=0).stop 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.602+0000 7f7879785740 1 -- 172.21.15.62:0/4137697956 shutdown_connections 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.602+0000 7f7879785740 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac70417680 0x55ac70419a70 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.602+0000 7f7879785740 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac7040cdc0 0x55ac70417140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.602+0000 7f7879785740 1 --2- 172.21.15.62:0/4137697956 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040c4a0 0x55ac7040c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.602+0000 7f7879785740 1 -- 172.21.15.62:0/4137697956 >> 172.21.15.62:0/4137697956 conn(0x55ac70408a30 msgr2=0x55ac70408e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.602+0000 7f7879785740 1 -- 172.21.15.62:0/4137697956 shutdown_connections 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.602+0000 7f7879785740 1 -- 172.21.15.62:0/4137697956 wait complete. 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7879785740 1 -- start start 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7879785740 1 Processor -- start 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7879785740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac70416fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7879785740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac70414280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7879785740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac704147e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7879785740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ac7041a250 con 0x55ac7040c4a0 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7879785740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ac7032d050 con 0x55ac70417680 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7879785740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ac7041a3d0 con 0x55ac7040cdc0 2024-09-15T23:21:22.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7879785740 10 client.0 _ll_register_callbacks cb 0x55ac7036d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7874947640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac70416fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7874947640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac70416fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51544/0 (socket says 172.21.15.62:51544) 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7874947640 1 -- 172.21.15.62:0/1932462907 learned_addr learned my addr 172.21.15.62:0/1932462907 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.603+0000 7f7875148640 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac70414280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7874947640 1 -- 172.21.15.62:0/1932462907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 msgr2=0x55ac70414280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7874947640 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac70414280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7874947640 1 -- 172.21.15.62:0/1932462907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 msgr2=0x55ac704147e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7874947640 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac704147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7874947640 1 -- 172.21.15.62:0/1932462907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac700e33c0 con 0x55ac7040c4a0 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7875148640 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac70414280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7874947640 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac70416fd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7860004f40 tx=0x7f7860005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89bc5b40b326baf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7858ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:22.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7858ff9640 1 -- 172.21.15.62:0/1932462907 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78600054f0 con 0x55ac7040c4a0 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7858ff9640 1 -- 172.21.15.62:0/1932462907 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f786000f830 con 0x55ac7040c4a0 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7879785740 1 -- 172.21.15.62:0/1932462907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78600047a0 con 0x55ac7040c4a0 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7858ff9640 1 -- 172.21.15.62:0/1932462907 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f786000fa30 con 0x55ac7040c4a0 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7879785740 10 client.16668 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7879785740 1 -- 172.21.15.62:0/1932462907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ac70425c20 con 0x55ac7040c4a0 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.604+0000 7f7858ff9640 1 -- 172.21.15.62:0/1932462907 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78600047a0 con 0x55ac7040c4a0 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.605+0000 7f7858ff9640 1 -- 172.21.15.62:0/1932462907 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f786001ece0 con 0x55ac7040c4a0 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.605+0000 7f7879785740 10 client.16668 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.605+0000 7f7879785740 10 client.16668 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.605+0000 7f7879785740 20 client.16668 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.605+0000 7f7879785740 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ac70426f70 0x55ac70429360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.605+0000 7f7879785740 4 client.16668 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:22.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.605+0000 7f7879785740 1 -- 172.21.15.62:0/1932462907 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac7037f850 con 0x55ac70426f70 2024-09-15T23:21:22.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.606+0000 7f7875148640 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ac70426f70 0x55ac70429360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.606+0000 7f7875148640 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ac70426f70 0x55ac70429360 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c970020c0b701a95 server_cookie=920152d861c91d64 in_seq=0 out_seq=0 2024-09-15T23:21:22.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.606+0000 7f7858ff9640 10 client.16668 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:22.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7858ff9640 1 -- 172.21.15.62:0/1932462907 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ac7037f850 con 0x55ac70426f70 2024-09-15T23:21:22.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7858ff9640 10 client.16668 handle_command_reply: tid=0 2024-09-15T23:21:22.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 client.16668 shutdown 2024-09-15T23:21:22.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 -- 172.21.15.62:0/1932462907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 msgr2=0x55ac70416fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:22.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac70416fd0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7860004f40 tx=0x7f7860005850 comp rx=0 tx=0).stop 2024-09-15T23:21:22.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 -- 172.21.15.62:0/1932462907 shutdown_connections 2024-09-15T23:21:22.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ac70426f70 0x55ac70429360 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac704147e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac70414280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 --2- 172.21.15.62:0/1932462907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac70416fd0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7858ff9640 0 client.16668 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:22.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 -- 172.21.15.62:0/1932462907 >> 172.21.15.62:0/1932462907 conn(0x55ac70408a30 msgr2=0x55ac7040e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:22.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 -- 172.21.15.62:0/1932462907 shutdown_connections 2024-09-15T23:21:22.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 1 -- 172.21.15.62:0/1932462907 wait complete. 2024-09-15T23:21:22.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.607+0000 7f7879785740 20 client.16668 trim_cache size 0 max 16384 2024-09-15T23:21:22.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7879785740 1 Processor -- start 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7879785740 1 -- start start 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7879785740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac704234b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7879785740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac704137e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7879785740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac7042ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7879785740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78701b1540 con 0x55ac7040c4a0 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7879785740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78701aee10 con 0x55ac70417680 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7879785740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78701acc60 con 0x55ac7040cdc0 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7874947640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac704234b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7874947640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac704234b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51554/0 (socket says 172.21.15.62:51554) 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.625+0000 7f7874947640 1 -- 172.21.15.62:0/2543583319 learned_addr learned my addr 172.21.15.62:0/2543583319 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f7874947640 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac7042ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f7875148640 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac704137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f7874947640 1 -- 172.21.15.62:0/2543583319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 msgr2=0x55ac704137e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:22.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f7874947640 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac704137e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f7874947640 1 -- 172.21.15.62:0/2543583319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 msgr2=0x55ac7042ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:22.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f7874947640 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac7042ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f7874947640 1 -- 172.21.15.62:0/2543583319 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac7040d330 con 0x55ac7040c4a0 2024-09-15T23:21:22.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f7875148640 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac704137e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:22.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f7874947640 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac704234b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f78600058c0 tx=0x7f7860004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616cf4836a9f630b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:22.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f785affd640 1 -- 172.21.15.62:0/2543583319 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786000f790 con 0x55ac7040c4a0 2024-09-15T23:21:22.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f785affd640 1 -- 172.21.15.62:0/2543583319 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786000ab80 con 0x55ac7040c4a0 2024-09-15T23:21:22.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.626+0000 7f785affd640 1 -- 172.21.15.62:0/2543583319 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786001cdd0 con 0x55ac7040c4a0 2024-09-15T23:21:22.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 -- 172.21.15.62:0/2543583319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 msgr2=0x55ac704234b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:22.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac704234b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f78600058c0 tx=0x7f7860004f90 comp rx=0 tx=0).stop 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 -- 172.21.15.62:0/2543583319 shutdown_connections 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac7042ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac704137e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 --2- 172.21.15.62:0/2543583319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac704234b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 -- 172.21.15.62:0/2543583319 >> 172.21.15.62:0/2543583319 conn(0x55ac70408a30 msgr2=0x55ac7040e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 -- 172.21.15.62:0/2543583319 shutdown_connections 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 -- 172.21.15.62:0/2543583319 wait complete. 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 -- start start 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.627+0000 7f7879785740 1 Processor -- start 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7879785740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac7041fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7879785740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac7041afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7879785740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac7041b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7879785740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78701ad2d0 con 0x55ac7040c4a0 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7879785740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78701ae740 con 0x55ac70417680 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7879785740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78701ad9a0 con 0x55ac7040cdc0 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7879785740 10 client.0 _ll_register_callbacks cb 0x7f786000ae90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7874947640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac7041fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7874947640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac7041fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51560/0 (socket says 172.21.15.62:51560) 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7874947640 1 -- 172.21.15.62:0/414940717 learned_addr learned my addr 172.21.15.62:0/414940717 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7875148640 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac7041afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7874947640 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac7041b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7875148640 1 -- 172.21.15.62:0/414940717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 msgr2=0x55ac7041b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7875148640 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac7041b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7875148640 1 -- 172.21.15.62:0/414940717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 msgr2=0x55ac7041fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7875148640 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac7041fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7875148640 1 -- 172.21.15.62:0/414940717 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac704096c0 con 0x55ac7040cdc0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.628+0000 7f7874947640 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac7041fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7875148640 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac7041afc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f786c005940 tx=0x7f786c031ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c56142cbd6d222 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7858ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7858ff9640 1 -- 172.21.15.62:0/414940717 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c04a070 con 0x55ac7040cdc0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7879785740 1 -- 172.21.15.62:0/414940717 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f786000ab80 con 0x55ac7040cdc0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7858ff9640 1 -- 172.21.15.62:0/414940717 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f786c004240 con 0x55ac7040cdc0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7858ff9640 1 -- 172.21.15.62:0/414940717 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f786c055050 con 0x55ac7040cdc0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7879785740 10 client.26083 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7879785740 1 -- 172.21.15.62:0/414940717 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ac70491770 con 0x55ac7040cdc0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7858ff9640 1 -- 172.21.15.62:0/414940717 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f786000ab80 con 0x55ac7040cdc0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.629+0000 7f7858ff9640 1 -- 172.21.15.62:0/414940717 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f786c0463a0 con 0x55ac7040cdc0 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7879785740 10 client.26083 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7879785740 10 client.26083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7879785740 20 client.26083 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7879785740 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ac70491ea0 0x55ac704942c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7879785740 4 client.26083 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7879785740 1 -- 172.21.15.62:0/414940717 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ac70379d60 con 0x55ac70491ea0 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7874947640 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ac70491ea0 0x55ac704942c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7874947640 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ac70491ea0 0x55ac704942c0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40365bbc60a900ce server_cookie=50618dad27b3d40e in_seq=0 out_seq=0 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7858ff9640 10 client.26083 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7858ff9640 1 -- 172.21.15.62:0/414940717 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ac70379d60 con 0x55ac70491ea0 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7858ff9640 10 client.26083 handle_command_reply: tid=0 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7879785740 1 client.26083 shutdown 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7879785740 1 -- 172.21.15.62:0/414940717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 msgr2=0x55ac7041afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.630+0000 7f7879785740 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac7041afc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f786c005940 tx=0x7f786c031ad0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7879785740 1 -- 172.21.15.62:0/414940717 shutdown_connections 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7879785740 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ac70491ea0 0x55ac704942c0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7879785740 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac70417680 0x55ac7041b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7879785740 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ac7040cdc0 0x55ac7041afc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7858ff9640 0 client.26083 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7879785740 1 --2- 172.21.15.62:0/414940717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ac7040c4a0 0x55ac7041fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7879785740 1 -- 172.21.15.62:0/414940717 >> 172.21.15.62:0/414940717 conn(0x55ac70408a30 msgr2=0x55ac70407270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7879785740 1 -- 172.21.15.62:0/414940717 shutdown_connections 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7879785740 1 -- 172.21.15.62:0/414940717 wait complete. 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f7879785740 20 client.26083 trim_cache size 0 max 16384 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:21:22.638 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:21:22.639 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 -- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f785c0a3240 msgr2=0x7f785c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f785c0a3240 0x7f785c0a5700 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f786809bcf0 tx=0x7f7868066ed0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 -- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870106f20 msgr2=0x7f787019cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870106f20 0x7f787019cc90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f786800eaa0 tx=0x7f786800eea0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 -- 172.21.15.62:0/2488359910 shutdown_connections 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f785c0a3240 0x7f785c0a5700 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7870106f20 0x7f787019cc90 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78701027a0 0x7f7870198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 --2- 172.21.15.62:0/2488359910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7870101df0 0x7f78701983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.631+0000 7f78773ac640 1 -- 172.21.15.62:0/2488359910 >> 172.21.15.62:0/2488359910 conn(0x7f78700fd8b0 msgr2=0x7f78700feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:22.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.632+0000 7f78773ac640 1 -- 172.21.15.62:0/2488359910 shutdown_connections 2024-09-15T23:21:22.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:22.632+0000 7f78773ac640 1 -- 172.21.15.62:0/2488359910 wait complete. 2024-09-15T23:21:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2567212429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8104fb0 msgr2=0x7fc7e81073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2567212429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8104fb0 0x7fc7e81073a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009900 tx=0x7fc7dc02a410 comp rx=0 tx=0).stop 2024-09-15T23:21:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2567212429 shutdown_connections 2024-09-15T23:21:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2567212429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e81078e0 0x7fc7e8109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2567212429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8104fb0 0x7fc7e81073a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2567212429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7e8102680 0x7fc7e8104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2567212429 >> 172.21.15.62:0/2567212429 conn(0x7fc7e80fc1e0 msgr2=0x7fc7e80fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:23.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2567212429 shutdown_connections 2024-09-15T23:21:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2567212429 wait complete. 2024-09-15T23:21:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 Processor -- start 2024-09-15T23:21:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.138+0000 7fc7f04b7640 1 -- start start 2024-09-15T23:21:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7f04b7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8102680 0x7fc7e8197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7f04b7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7e8104fb0 0x7fc7e81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7f04b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e81078e0 0x7fc7e819f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7f04b7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e810d810 con 0x7fc7e8102680 2024-09-15T23:21:23.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7f04b7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e810d690 con 0x7fc7e81078e0 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7f04b7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e810d990 con 0x7fc7e8104fb0 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7ee253640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8102680 0x7fc7e8197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7ee253640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8102680 0x7fc7e8197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51570/0 (socket says 172.21.15.62:51570) 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7ee253640 1 -- 172.21.15.62:0/2150888800 learned_addr learned my addr 172.21.15.62:0/2150888800 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7eea54640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e81078e0 0x7fc7e819f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7eda52640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7e8104fb0 0x7fc7e81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7ee253640 1 -- 172.21.15.62:0/2150888800 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7e8104fb0 msgr2=0x7fc7e81983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7ee253640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7e8104fb0 0x7fc7e81983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7ee253640 1 -- 172.21.15.62:0/2150888800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e81078e0 msgr2=0x7fc7e819f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7ee253640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e81078e0 0x7fc7e819f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7ee253640 1 -- 172.21.15.62:0/2150888800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e819fa90 con 0x7fc7e8102680 2024-09-15T23:21:23.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7ee253640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8102680 0x7fc7e8197e70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000d770 tx=0x7fc7e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:23.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7e77fe640 1 -- 172.21.15.62:0/2150888800 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0017070 con 0x7fc7e8102680 2024-09-15T23:21:23.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7e77fe640 1 -- 172.21.15.62:0/2150888800 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e0005500 con 0x7fc7e8102680 2024-09-15T23:21:23.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7e77fe640 1 -- 172.21.15.62:0/2150888800 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0004e00 con 0x7fc7e8102680 2024-09-15T23:21:23.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2150888800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e819fd80 con 0x7fc7e8102680 2024-09-15T23:21:23.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.139+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2150888800 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e81a0190 con 0x7fc7e8102680 2024-09-15T23:21:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.142+0000 7fc7e77fe640 1 -- 172.21.15.62:0/2150888800 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc7e0004fa0 con 0x7fc7e8102680 2024-09-15T23:21:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7f3250740 1 Processor -- start 2024-09-15T23:21:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7f3250740 1 -- start start 2024-09-15T23:21:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x56542121dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x565421222da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x565421225830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7f3250740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5654211812f0 con 0x56542121e2e0 2024-09-15T23:21:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7f3250740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565421129270 con 0x56542121d9a0 2024-09-15T23:21:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7f3250740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56542117d2b0 con 0x565421223370 2024-09-15T23:21:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7ee253640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x565421222da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7ee253640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x565421222da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51576/0 (socket says 172.21.15.62:51576) 2024-09-15T23:21:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7ee253640 1 -- 172.21.15.62:0/3106562460 learned_addr learned my addr 172.21.15.62:0/3106562460 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7eda52640 1 --2- 172.21.15.62:0/3106562460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x565421225830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7eea54640 1 --2- 172.21.15.62:0/3106562460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x56542121dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7ee253640 1 -- 172.21.15.62:0/3106562460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 msgr2=0x565421225830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7ee253640 1 --2- 172.21.15.62:0/3106562460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x565421225830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7ee253640 1 -- 172.21.15.62:0/3106562460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 msgr2=0x56542121dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.143+0000 7fc7ee253640 1 --2- 172.21.15.62:0/3106562460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x56542121dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.144+0000 7fc7ee253640 1 -- 172.21.15.62:0/3106562460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565421128340 con 0x56542121e2e0 2024-09-15T23:21:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.144+0000 7fc7e77fe640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7d00a32f0 0x7fc7d00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.144+0000 7fc7eea54640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7d00a32f0 0x7fc7d00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.144+0000 7fc7e77fe640 1 -- 172.21.15.62:0/2150888800 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc7e00cf710 con 0x7fc7e8102680 2024-09-15T23:21:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.144+0000 7fc7ee253640 1 --2- 172.21.15.62:0/3106562460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x565421222da0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00d5c10 tx=0x7fc7e00d7690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83614e7a454b6f92 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.144+0000 7fc7e4bfe640 1 -- 172.21.15.62:0/3106562460 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e0020070 con 0x56542121e2e0 2024-09-15T23:21:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.144+0000 7fc7e4bfe640 1 -- 172.21.15.62:0/3106562460 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7e00046c0 con 0x56542121e2e0 2024-09-15T23:21:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.144+0000 7fc7e4bfe640 1 -- 172.21.15.62:0/3106562460 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7e009b4f0 con 0x56542121e2e0 2024-09-15T23:21:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7eea54640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7d00a32f0 0x7fc7d00a57b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8005fd0 tx=0x7fc7d80096a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:23.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 -- 172.21.15.62:0/3106562460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 msgr2=0x565421222da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 --2- 172.21.15.62:0/3106562460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x565421222da0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00d5c10 tx=0x7fc7e00d7690 comp rx=0 tx=0).stop 2024-09-15T23:21:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 -- 172.21.15.62:0/3106562460 shutdown_connections 2024-09-15T23:21:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 --2- 172.21.15.62:0/3106562460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x565421225830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 --2- 172.21.15.62:0/3106562460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x565421222da0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 --2- 172.21.15.62:0/3106562460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x56542121dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 -- 172.21.15.62:0/3106562460 >> 172.21.15.62:0/3106562460 conn(0x565421219010 msgr2=0x565421219410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 -- 172.21.15.62:0/3106562460 shutdown_connections 2024-09-15T23:21:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 -- 172.21.15.62:0/3106562460 wait complete. 2024-09-15T23:21:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 -- start start 2024-09-15T23:21:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.145+0000 7fc7f3250740 1 Processor -- start 2024-09-15T23:21:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x565421227b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x5654212280d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x56542122f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7f3250740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565421226340 con 0x56542121e2e0 2024-09-15T23:21:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7f3250740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56542110a290 con 0x56542121d9a0 2024-09-15T23:21:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7f3250740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5654212264c0 con 0x565421223370 2024-09-15T23:21:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7f3250740 10 client.0 _ll_register_callbacks cb 0x56542113b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eda52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x5654212280d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eda52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x5654212280d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51582/0 (socket says 172.21.15.62:51582) 2024-09-15T23:21:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eda52640 1 -- 172.21.15.62:0/371976799 learned_addr learned my addr 172.21.15.62:0/371976799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7ee253640 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x565421227b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eea54640 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x56542122f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eda52640 1 -- 172.21.15.62:0/371976799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 msgr2=0x56542122f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:23.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eda52640 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x56542122f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eda52640 1 -- 172.21.15.62:0/371976799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 msgr2=0x565421227b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eda52640 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x565421227b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eda52640 1 -- 172.21.15.62:0/371976799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565420e94210 con 0x56542121e2e0 2024-09-15T23:21:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7ee253640 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x565421227b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:23.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7eda52640 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x5654212280d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc031440 tx=0x7fc7dc005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7b6e3b3b19d1bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7cdffb640 1 -- 172.21.15.62:0/371976799 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc035040 con 0x56542121e2e0 2024-09-15T23:21:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7cdffb640 1 -- 172.21.15.62:0/371976799 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc7dc0318b0 con 0x56542121e2e0 2024-09-15T23:21:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.146+0000 7fc7f3250740 1 -- 172.21.15.62:0/371976799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56542118ea20 con 0x56542121e2e0 2024-09-15T23:21:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7cdffb640 1 -- 172.21.15.62:0/371976799 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc7dc04e050 con 0x56542121e2e0 2024-09-15T23:21:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7f3250740 10 client.16686 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7f3250740 1 -- 172.21.15.62:0/371976799 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x565420f60620 con 0x56542121e2e0 2024-09-15T23:21:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7cdffb640 1 -- 172.21.15.62:0/371976799 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56542118ea20 con 0x56542121e2e0 2024-09-15T23:21:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7cdffb640 1 -- 172.21.15.62:0/371976799 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fc7dc0035f0 con 0x56542121e2e0 2024-09-15T23:21:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7f3250740 10 client.16686 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7f3250740 10 client.16686 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:23.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7f3250740 20 client.16686 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7f3250740 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5654212337b0 0x565421235be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7f3250740 4 client.16686 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7f3250740 1 -- 172.21.15.62:0/371976799 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565421188d70 con 0x5654212337b0 2024-09-15T23:21:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.147+0000 7fc7ee253640 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5654212337b0 0x565421235be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.148+0000 7fc7cdffb640 10 client.16686 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:23.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.148+0000 7fc7ee253640 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5654212337b0 0x565421235be0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d33dbc75a74ed5d server_cookie=d2b82f98299bf292 in_seq=0 out_seq=0 2024-09-15T23:21:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.148+0000 7fc7cdffb640 1 -- 172.21.15.62:0/371976799 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565421188d70 con 0x5654212337b0 2024-09-15T23:21:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.148+0000 7fc7cdffb640 10 client.16686 handle_command_reply: tid=0 2024-09-15T23:21:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.148+0000 7fc7f3250740 1 client.16686 shutdown 2024-09-15T23:21:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.148+0000 7fc7f3250740 1 -- 172.21.15.62:0/371976799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 msgr2=0x5654212280d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.148+0000 7fc7f3250740 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x5654212280d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc031440 tx=0x7fc7dc005270 comp rx=0 tx=0).stop 2024-09-15T23:21:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7f3250740 1 -- 172.21.15.62:0/371976799 shutdown_connections 2024-09-15T23:21:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7f3250740 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5654212337b0 0x565421235be0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7f3250740 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x56542122f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7f3250740 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x5654212280d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7f3250740 1 --2- 172.21.15.62:0/371976799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x565421227b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7cdffb640 0 client.16686 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7f3250740 1 -- 172.21.15.62:0/371976799 >> 172.21.15.62:0/371976799 conn(0x565421219010 msgr2=0x56542121a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7f3250740 1 -- 172.21.15.62:0/371976799 shutdown_connections 2024-09-15T23:21:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7f3250740 1 -- 172.21.15.62:0/371976799 wait complete. 2024-09-15T23:21:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.149+0000 7fc7f3250740 20 client.16686 trim_cache size 0 max 16384 2024-09-15T23:21:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7f3250740 1 Processor -- start 2024-09-15T23:21:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7f3250740 1 -- start start 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x565421219f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x56542121a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7f3250740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81acf80 con 0x56542121e2e0 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7f3250740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81aa850 con 0x56542121d9a0 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7f3250740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81a86a0 con 0x565421223370 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7eda52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7eda52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51584/0 (socket says 172.21.15.62:51584) 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.163+0000 7fc7eda52640 1 -- 172.21.15.62:0/1414499518 learned_addr learned my addr 172.21.15.62:0/1414499518 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7eea54640 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x56542121a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7ee253640 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x565421219f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7eda52640 1 -- 172.21.15.62:0/1414499518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 msgr2=0x56542121a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7eda52640 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x56542121a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7eda52640 1 -- 172.21.15.62:0/1414499518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 msgr2=0x565421219f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7eda52640 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x565421219f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7eda52640 1 -- 172.21.15.62:0/1414499518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56542121e7c0 con 0x56542121e2e0 2024-09-15T23:21:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7eea54640 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x56542121a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7eda52640 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122f0d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0039c0 tx=0x7fc7dc003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e673c3e1dd385f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7cffff640 1 -- 172.21.15.62:0/1414499518 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc035030 con 0x56542121e2e0 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7cffff640 1 -- 172.21.15.62:0/1414499518 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7dc040070 con 0x56542121e2e0 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7cffff640 1 -- 172.21.15.62:0/1414499518 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc031df0 con 0x56542121e2e0 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7f3250740 1 -- 172.21.15.62:0/1414499518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 msgr2=0x56542122f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.164+0000 7fc7f3250740 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122f0d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc0039c0 tx=0x7fc7dc003e90 comp rx=0 tx=0).stop 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 -- 172.21.15.62:0/1414499518 shutdown_connections 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565421223370 0x56542121a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122f0d0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 --2- 172.21.15.62:0/1414499518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56542121d9a0 0x565421219f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 -- 172.21.15.62:0/1414499518 >> 172.21.15.62:0/1414499518 conn(0x565421219010 msgr2=0x56542121f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 -- 172.21.15.62:0/1414499518 shutdown_connections 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 -- 172.21.15.62:0/1414499518 wait complete. 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 -- start start 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 Processor -- start 2024-09-15T23:21:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56542121d9a0 0x56542122c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565421223370 0x56542122d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81a8d10 con 0x56542121e2e0 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81aa180 con 0x565421223370 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e81a93e0 con 0x56542121d9a0 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7f3250740 10 client.0 _ll_register_callbacks cb 0x7fc7dc009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7eda52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7eda52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51596/0 (socket says 172.21.15.62:51596) 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7eda52640 1 -- 172.21.15.62:0/706740114 learned_addr learned my addr 172.21.15.62:0/706740114 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7ee253640 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56542121d9a0 0x56542122c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7eda52640 1 -- 172.21.15.62:0/706740114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56542121d9a0 msgr2=0x56542122c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7eda52640 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56542121d9a0 0x56542122c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7eda52640 1 -- 172.21.15.62:0/706740114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565421223370 msgr2=0x56542122d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7eea54640 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565421223370 0x56542122d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.165+0000 7fc7eda52640 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565421223370 0x56542122d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7eda52640 1 -- 172.21.15.62:0/706740114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56542122dde0 con 0x56542121e2e0 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7ee253640 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56542121d9a0 0x56542122c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7eda52640 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122c810 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009900 tx=0x7fc7dc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bce1478be0c969f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7cdffb640 1 -- 172.21.15.62:0/706740114 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc035070 con 0x56542121e2e0 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7cdffb640 1 -- 172.21.15.62:0/706740114 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc7dc032c80 con 0x56542121e2e0 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7f3250740 1 -- 172.21.15.62:0/706740114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7dc040070 con 0x56542121e2e0 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7cdffb640 1 -- 172.21.15.62:0/706740114 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc7dc06a050 con 0x56542121e2e0 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7f3250740 10 client.16692 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:23.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7f3250740 1 -- 172.21.15.62:0/706740114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56542122f060 con 0x56542121e2e0 2024-09-15T23:21:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7cdffb640 1 -- 172.21.15.62:0/706740114 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7dc040070 con 0x56542121e2e0 2024-09-15T23:21:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7cdffb640 1 -- 172.21.15.62:0/706740114 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fc7dc004ca0 con 0x56542121e2e0 2024-09-15T23:21:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7f3250740 10 client.16692 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7f3250740 10 client.16692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:23.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7f3250740 20 client.16692 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.166+0000 7fc7f3250740 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56542129e800 0x5654212a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.167+0000 7fc7f3250740 4 client.16692 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:21:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.167+0000 7fc7f3250740 1 -- 172.21.15.62:0/706740114 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565421105900 con 0x56542129e800 2024-09-15T23:21:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.167+0000 7fc7ee253640 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56542129e800 0x5654212a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.167+0000 7fc7ee253640 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56542129e800 0x5654212a0bf0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee91834eec4528af server_cookie=66915ba987edecf3 in_seq=0 out_seq=0 2024-09-15T23:21:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.167+0000 7fc7cdffb640 10 client.16692 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7cdffb640 1 -- 172.21.15.62:0/706740114 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565421105900 con 0x56542129e800 2024-09-15T23:21:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7cdffb640 10 client.16692 handle_command_reply: tid=0 2024-09-15T23:21:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7f3250740 1 client.16692 shutdown 2024-09-15T23:21:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7f3250740 1 -- 172.21.15.62:0/706740114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 msgr2=0x56542122c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7f3250740 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122c810 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009900 tx=0x7fc7dc003f10 comp rx=0 tx=0).stop 2024-09-15T23:21:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7f3250740 1 -- 172.21.15.62:0/706740114 shutdown_connections 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7f3250740 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56542129e800 0x5654212a0bf0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7f3250740 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565421223370 0x56542122d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7f3250740 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56542121e2e0 0x56542122c810 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7cdffb640 0 client.16692 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7f3250740 1 --2- 172.21.15.62:0/706740114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56542121d9a0 0x56542122c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.176+0000 7fc7f3250740 1 -- 172.21.15.62:0/706740114 >> 172.21.15.62:0/706740114 conn(0x565421219010 msgr2=0x565421219f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f3250740 1 -- 172.21.15.62:0/706740114 shutdown_connections 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f3250740 1 -- 172.21.15.62:0/706740114 wait complete. 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f3250740 20 client.16692 trim_cache size 0 max 16384 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:21:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:21:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:21:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:21:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7d00a32f0 msgr2=0x7fc7d00a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7d00a32f0 0x7fc7d00a57b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc7d8005fd0 tx=0x7fc7d80096a0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8102680 msgr2=0x7fc7e8197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8102680 0x7fc7e8197e70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000d770 tx=0x7fc7e000dc30 comp rx=0 tx=0).stop 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2150888800 shutdown_connections 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc7d00a32f0 0x7fc7d00a57b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7e81078e0 0x7fc7e819f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc7e8104fb0 0x7fc7e81983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 --2- 172.21.15.62:0/2150888800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7e8102680 0x7fc7e8197e70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2150888800 >> 172.21.15.62:0/2150888800 conn(0x7fc7e80fc1e0 msgr2=0x7fc7e8103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2150888800 shutdown_connections 2024-09-15T23:21:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:23.177+0000 7fc7f04b7640 1 -- 172.21.15.62:0/2150888800 wait complete. 2024-09-15T23:21:23.406 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:21:23.406 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:23.708 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:23.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:23 smithi062 ceph-mon[30949]: pgmap v523: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.6 MiB/s wr, 416 op/s 2024-09-15T23:21:23.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:23 smithi067 ceph-mon[35665]: pgmap v523: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.6 MiB/s wr, 416 op/s 2024-09-15T23:21:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:23 smithi148 ceph-mon[35610]: pgmap v523: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.6 MiB/s wr, 416 op/s 2024-09-15T23:21:24.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 -- 172.21.15.62:0/1607120257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04101de0 msgr2=0x7f8d041021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:24.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/1607120257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04101de0 0x7f8d041021c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009ca0 tx=0x7f8cfc02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 -- 172.21.15.62:0/1607120257 shutdown_connections 2024-09-15T23:21:24.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/1607120257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04106f10 0x7f8d0410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/1607120257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d04102790 0x7f8d041067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/1607120257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04101de0 0x7f8d041021c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 -- 172.21.15.62:0/1607120257 >> 172.21.15.62:0/1607120257 conn(0x7f8d040fd8c0 msgr2=0x7f8d040ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 -- 172.21.15.62:0/1607120257 shutdown_connections 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 -- 172.21.15.62:0/1607120257 wait complete. 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.170+0000 7f8d0b10f640 1 Processor -- start 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d0b10f640 1 -- start start 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d0b10f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04101de0 0x7f8d041984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d0b10f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04102790 0x7f8d041989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d0b10f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d04106f10 0x7f8d0419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d0b10f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8d04111fc0 con 0x7f8d04102790 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d0b10f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d04111e40 con 0x7f8d04101de0 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d0b10f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8d04112140 con 0x7f8d04106f10 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d03fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04102790 0x7f8d041989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:24.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d03fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04102790 0x7f8d041989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51602/0 (socket says 172.21.15.62:51602) 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d03fff640 1 -- 172.21.15.62:0/2759497299 learned_addr learned my addr 172.21.15.62:0/2759497299 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d08eab640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04101de0 0x7f8d041984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d096ac640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d04106f10 0x7f8d0419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d03fff640 1 -- 172.21.15.62:0/2759497299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d04106f10 msgr2=0x7f8d0419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d03fff640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d04106f10 0x7f8d0419cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d03fff640 1 -- 172.21.15.62:0/2759497299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04101de0 msgr2=0x7f8d041984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d03fff640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04101de0 0x7f8d041984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d03fff640 1 -- 172.21.15.62:0/2759497299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0419d450 con 0x7f8d04102790 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.171+0000 7f8d08eab640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04101de0 0x7f8d041984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.172+0000 7f8d03fff640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04102790 0x7f8d041989e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400d950 tx=0x7f8cf400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.172+0000 7f8d01ffb640 1 -- 172.21.15.62:0/2759497299 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4017070 con 0x7f8d04102790 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.172+0000 7f8d01ffb640 1 -- 172.21.15.62:0/2759497299 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cf4009c30 con 0x7f8d04102790 2024-09-15T23:21:24.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.172+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0419d740 con 0x7f8d04102790 2024-09-15T23:21:24.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.172+0000 7f8d01ffb640 1 -- 172.21.15.62:0/2759497299 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf400f6b0 con 0x7f8d04102790 2024-09-15T23:21:24.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.172+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d041a5020 con 0x7f8d04102790 2024-09-15T23:21:24.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.173+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d041021c0 con 0x7f8d04102790 2024-09-15T23:21:24.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.174+0000 7f8d01ffb640 1 -- 172.21.15.62:0/2759497299 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8cf40109d0 con 0x7f8d04102790 2024-09-15T23:21:24.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.175+0000 7f8d01ffb640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8cf00a3240 0x7f8cf00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:24.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.175+0000 7f8d08eab640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8cf00a3240 0x7f8cf00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:24.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.175+0000 7f8d01ffb640 1 -- 172.21.15.62:0/2759497299 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8cf40cf9d0 con 0x7f8d04102790 2024-09-15T23:21:24.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.175+0000 7f8d08eab640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8cf00a3240 0x7f8cf00a5700 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc005e30 tx=0x7f8cfc000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:24.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.179+0000 7f8d01ffb640 1 -- 172.21.15.62:0/2759497299 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8cf4009de0 con 0x7f8d04102790 2024-09-15T23:21:24.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.392+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d04109b60 con 0x7f8d04102790 2024-09-15T23:21:24.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.392+0000 7f8d01ffb640 1 -- 172.21.15.62:0/2759497299 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11903 (secure 0 0 0) 0x7f8cf409c370 con 0x7f8d04102790 2024-09-15T23:21:24.394 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:21:24.394 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":31,"btime":"2024-09-15T23:20:13:209709+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:20:12.378892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:21:24.394 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 31 2024-09-15T23:21:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8cf00a3240 msgr2=0x7f8cf00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8cf00a3240 0x7f8cf00a5700 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc005e30 tx=0x7f8cfc000f40 comp rx=0 tx=0).stop 2024-09-15T23:21:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04102790 msgr2=0x7f8d041989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:24.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04102790 0x7f8d041989e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400d950 tx=0x7f8cf400de10 comp rx=0 tx=0).stop 2024-09-15T23:21:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 shutdown_connections 2024-09-15T23:21:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8cf00a3240 0x7f8cf00a5700 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8d04106f10 0x7f8d0419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d04102790 0x7f8d041989e0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 --2- 172.21.15.62:0/2759497299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d04101de0 0x7f8d041984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 >> 172.21.15.62:0/2759497299 conn(0x7f8d040fd8c0 msgr2=0x7f8d040fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 shutdown_connections 2024-09-15T23:21:24.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:24.396+0000 7f8d0b10f640 1 -- 172.21.15.62:0/2759497299 wait complete. 2024-09-15T23:21:24.573 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 305.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:21:24.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:24 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2759497299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:24.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:24 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2759497299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:24.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:24 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2759497299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:25.076 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:21:25.334 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:25.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:25 smithi062 ceph-mon[30949]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-09-15T23:21:25.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:25 smithi067 ceph-mon[35665]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-09-15T23:21:25.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:25 smithi148 ceph-mon[35610]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 MiB/s wr, 173 op/s 2024-09-15T23:21:25.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.813+0000 7f5a67543640 1 -- 172.21.15.62:0/701292326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 msgr2=0x7f5a6010d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.813+0000 7f5a67543640 1 --2- 172.21.15.62:0/701292326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 0x7f5a6010d930 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009cd0 tx=0x7f5a5802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.813+0000 7f5a67543640 1 -- 172.21.15.62:0/701292326 shutdown_connections 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.813+0000 7f5a67543640 1 --2- 172.21.15.62:0/701292326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 0x7f5a6010d930 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.813+0000 7f5a67543640 1 --2- 172.21.15.62:0/701292326 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a60100a60 0x7f5a60100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.813+0000 7f5a67543640 1 --2- 172.21.15.62:0/701292326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a60106a60 0x7f5a60106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.813+0000 7f5a67543640 1 -- 172.21.15.62:0/701292326 >> 172.21.15.62:0/701292326 conn(0x7f5a600fc780 msgr2=0x7f5a600feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.813+0000 7f5a67543640 1 -- 172.21.15.62:0/701292326 shutdown_connections 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.813+0000 7f5a67543640 1 -- 172.21.15.62:0/701292326 wait complete. 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a67543640 1 Processor -- start 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a67543640 1 -- start start 2024-09-15T23:21:25.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a67543640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a60100a60 0x7f5a60102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a67543640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 0x7f5a60102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a67543640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a60106a60 0x7f5a601030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a67543640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5a60110ed0 con 0x7f5a60101400 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a67543640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5a60110d50 con 0x7f5a60100a60 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a67543640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5a60111050 con 0x7f5a60106a60 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a64ade640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 0x7f5a60102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a64ade640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 0x7f5a60102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51620/0 (socket says 172.21.15.62:51620) 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a64ade640 1 -- 172.21.15.62:0/3593423724 learned_addr learned my addr 172.21.15.62:0/3593423724 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a652df640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a60100a60 0x7f5a60102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a65ae0640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a60106a60 0x7f5a601030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.814+0000 7f5a64ade640 1 -- 172.21.15.62:0/3593423724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a60106a60 msgr2=0x7f5a601030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a64ade640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a60106a60 0x7f5a601030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a64ade640 1 -- 172.21.15.62:0/3593423724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a60100a60 msgr2=0x7f5a60102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a64ade640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a60100a60 0x7f5a60102630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a64ade640 1 -- 172.21.15.62:0/3593423724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a601a68c0 con 0x7f5a60101400 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a65ae0640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a60106a60 0x7f5a601030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a64ade640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 0x7f5a60102b70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000b4a0 tx=0x7f5a5000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a567fc640 1 -- 172.21.15.62:0/3593423724 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a50015020 con 0x7f5a60101400 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a67543640 1 -- 172.21.15.62:0/3593423724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a601a6bb0 con 0x7f5a60101400 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a567fc640 1 -- 172.21.15.62:0/3593423724 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a50011d10 con 0x7f5a60101400 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a567fc640 1 -- 172.21.15.62:0/3593423724 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5000f2e0 con 0x7f5a60101400 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.815+0000 7f5a67543640 1 -- 172.21.15.62:0/3593423724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a601a70a0 con 0x7f5a60101400 2024-09-15T23:21:25.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.817+0000 7f5a567fc640 1 -- 172.21.15.62:0/3593423724 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f5a50018020 con 0x7f5a60101400 2024-09-15T23:21:25.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a6991c740 1 Processor -- start 2024-09-15T23:21:25.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a6991c740 1 -- start start 2024-09-15T23:21:25.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd05d0 0x5638a1bd0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bdd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a6991c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5638a1b362f0 con 0x5638a1bd1b40 2024-09-15T23:21:25.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a6991c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5638a1addcf0 con 0x5638a1bdb370 2024-09-15T23:21:25.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a6991c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5638a1b322b0 con 0x5638a1bd05d0 2024-09-15T23:21:25.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a567fc640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5a4c0a3240 0x7f5a4c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a65ae0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a65ae0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51622/0 (socket says 172.21.15.62:51622) 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a65ae0640 1 -- 172.21.15.62:0/3018526499 learned_addr learned my addr 172.21.15.62:0/3018526499 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a652df640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd05d0 0x5638a1bd0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a567fc640 1 -- 172.21.15.62:0/3593423724 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5a500cf5d0 con 0x7f5a60101400 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a64ade640 1 --2- 172.21.15.62:0/3018526499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bdd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.818+0000 7f5a65ae0640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5a4c0a3240 0x7f5a4c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a652df640 1 -- 172.21.15.62:0/3018526499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 msgr2=0x5638a1bdd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a652df640 1 --2- 172.21.15.62:0/3018526499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bdd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a652df640 1 -- 172.21.15.62:0/3018526499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 msgr2=0x5638a1bd0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a652df640 1 --2- 172.21.15.62:0/3018526499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a652df640 1 -- 172.21.15.62:0/3018526499 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638a1ac23a0 con 0x5638a1bd05d0 2024-09-15T23:21:25.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a64ade640 1 --2- 172.21.15.62:0/3018526499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bdd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:25.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a65ae0640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5a4c0a3240 0x7f5a4c0a5700 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5a58006010 tx=0x7f5a58009800 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:25.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a652df640 1 --2- 172.21.15.62:0/3018526499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd05d0 0x5638a1bd0a10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00abf0 tx=0x7f5a5c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59c6b2dd20f86f37 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:25.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a4b7fe640 1 -- 172.21.15.62:0/3018526499 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c005220 con 0x5638a1bd05d0 2024-09-15T23:21:25.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a4b7fe640 1 -- 172.21.15.62:0/3018526499 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a5c004c80 con 0x5638a1bd05d0 2024-09-15T23:21:25.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.819+0000 7f5a4b7fe640 1 -- 172.21.15.62:0/3018526499 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c018800 con 0x5638a1bd05d0 2024-09-15T23:21:25.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 -- 172.21.15.62:0/3018526499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd05d0 msgr2=0x5638a1bd0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3018526499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd05d0 0x5638a1bd0a10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c00abf0 tx=0x7f5a5c00f790 comp rx=0 tx=0).stop 2024-09-15T23:21:25.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 -- 172.21.15.62:0/3018526499 shutdown_connections 2024-09-15T23:21:25.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3018526499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bdd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3018526499 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd05d0 0x5638a1bd0a10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3018526499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 -- 172.21.15.62:0/3018526499 >> 172.21.15.62:0/3018526499 conn(0x5638a1b99310 msgr2=0x5638a1bccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:25.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 -- 172.21.15.62:0/3018526499 shutdown_connections 2024-09-15T23:21:25.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 -- 172.21.15.62:0/3018526499 wait complete. 2024-09-15T23:21:25.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 -- start start 2024-09-15T23:21:25.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.820+0000 7f5a6991c740 1 Processor -- start 2024-09-15T23:21:25.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd05d0 0x5638a1bdaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 0x5638a1bd8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a6991c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5638a1bde070 con 0x5638a1bd1b40 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a6991c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5638a1af1060 con 0x5638a1bd05d0 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a6991c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5638a1bde1f0 con 0x5638a1bdb370 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a6991c740 10 client.0 _ll_register_callbacks cb 0x5638a18448c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a64ade640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a64ade640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51634/0 (socket says 172.21.15.62:51634) 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a64ade640 1 -- 172.21.15.62:0/1163728677 learned_addr learned my addr 172.21.15.62:0/1163728677 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a652df640 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd05d0 0x5638a1bdaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a65ae0640 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 0x5638a1bd8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a64ade640 1 -- 172.21.15.62:0/1163728677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 msgr2=0x5638a1bd8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a64ade640 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 0x5638a1bd8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a64ade640 1 -- 172.21.15.62:0/1163728677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd05d0 msgr2=0x5638a1bdaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:25.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a64ade640 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd05d0 0x5638a1bdaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a64ade640 1 -- 172.21.15.62:0/1163728677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638a18a73c0 con 0x5638a1bd1b40 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a64ade640 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd7f50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5a500cd0c0 tx=0x7f5a5009b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43eef052e8462b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.821+0000 7f5a48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a48ff9640 1 -- 172.21.15.62:0/1163728677 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5009bf70 con 0x5638a1bd1b40 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a48ff9640 1 -- 172.21.15.62:0/1163728677 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5a50010510 con 0x5638a1bd1b40 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a6991c740 1 -- 172.21.15.62:0/1163728677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a5c004c80 con 0x5638a1bd1b40 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a48ff9640 1 -- 172.21.15.62:0/1163728677 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5a5009b870 con 0x5638a1bd1b40 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a6991c740 10 client.16710 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a6991c740 1 -- 172.21.15.62:0/1163728677 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5638a1be9ab0 con 0x5638a1bd1b40 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a48ff9640 1 -- 172.21.15.62:0/1163728677 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a5c004c80 con 0x5638a1bd1b40 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a48ff9640 1 -- 172.21.15.62:0/1163728677 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5a500ba080 con 0x5638a1bd1b40 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a6991c740 10 client.16710 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:25.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a6991c740 10 client.16710 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a6991c740 20 client.16710 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a6991c740 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5638a1beaf70 0x5638a1bed3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a6991c740 4 client.16710 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a6991c740 1 -- 172.21.15.62:0/1163728677 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638a1b43860 con 0x5638a1beaf70 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.822+0000 7f5a652df640 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5638a1beaf70 0x5638a1bed3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.823+0000 7f5a48ff9640 10 client.16710 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.823+0000 7f5a652df640 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5638a1beaf70 0x5638a1bed3a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce735779b31dd9e0 server_cookie=a031b7fdbf889949 in_seq=0 out_seq=0 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a48ff9640 1 -- 172.21.15.62:0/1163728677 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5638a1b43860 con 0x5638a1beaf70 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a48ff9640 10 client.16710 handle_command_reply: tid=0 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 client.16710 shutdown 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 -- 172.21.15.62:0/1163728677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 msgr2=0x5638a1bd7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd7f50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f5a500cd0c0 tx=0x7f5a5009b640 comp rx=0 tx=0).stop 2024-09-15T23:21:25.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 -- 172.21.15.62:0/1163728677 shutdown_connections 2024-09-15T23:21:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5638a1beaf70 0x5638a1bed3a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 0x5638a1bd8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd1b40 0x5638a1bd7f50 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a48ff9640 0 client.16710 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:21:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 --2- 172.21.15.62:0/1163728677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd05d0 0x5638a1bdaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 -- 172.21.15.62:0/1163728677 >> 172.21.15.62:0/1163728677 conn(0x5638a1b99310 msgr2=0x5638a1bd2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 -- 172.21.15.62:0/1163728677 shutdown_connections 2024-09-15T23:21:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 1 -- 172.21.15.62:0/1163728677 wait complete. 2024-09-15T23:21:25.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.824+0000 7f5a6991c740 20 client.16710 trim_cache size 0 max 16384 2024-09-15T23:21:25.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a6991c740 1 Processor -- start 2024-09-15T23:21:25.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a6991c740 1 -- start start 2024-09-15T23:21:25.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1bd9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd1b40 0x5638a1bd3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bd3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a6991c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5a601b35f0 con 0x5638a1bd05d0 2024-09-15T23:21:25.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a6991c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5a601b0ec0 con 0x5638a1bdb370 2024-09-15T23:21:25.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a6991c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5a601aed10 con 0x5638a1bd1b40 2024-09-15T23:21:25.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a652df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1bd9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a652df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1bd9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51644/0 (socket says 172.21.15.62:51644) 2024-09-15T23:21:25.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.837+0000 7f5a652df640 1 -- 172.21.15.62:0/2901465234 learned_addr learned my addr 172.21.15.62:0/2901465234 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:25.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a65ae0640 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bd3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a652df640 1 -- 172.21.15.62:0/2901465234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd1b40 msgr2=0x5638a1bd3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:25.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a64ade640 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd1b40 0x5638a1bd3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a652df640 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd1b40 0x5638a1bd3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a652df640 1 -- 172.21.15.62:0/2901465234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 msgr2=0x5638a1bd3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:25.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a652df640 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bd3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a652df640 1 -- 172.21.15.62:0/2901465234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638a1bcddd0 con 0x5638a1bd05d0 2024-09-15T23:21:25.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a65ae0640 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bd3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:25.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a652df640 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1bd9ec0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c004820 tx=0x7f5a5c00a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c95d3c78ec4481 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:25.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a4affd640 1 -- 172.21.15.62:0/2901465234 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c033070 con 0x5638a1bd05d0 2024-09-15T23:21:25.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a4affd640 1 -- 172.21.15.62:0/2901465234 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a5c02e950 con 0x5638a1bd05d0 2024-09-15T23:21:25.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.838+0000 7f5a4affd640 1 -- 172.21.15.62:0/2901465234 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c018750 con 0x5638a1bd05d0 2024-09-15T23:21:25.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 -- 172.21.15.62:0/2901465234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 msgr2=0x5638a1bd9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1bd9ec0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c004820 tx=0x7f5a5c00a630 comp rx=0 tx=0).stop 2024-09-15T23:21:25.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 -- 172.21.15.62:0/2901465234 shutdown_connections 2024-09-15T23:21:25.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bdb370 0x5638a1bd3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bd1b40 0x5638a1bd3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 --2- 172.21.15.62:0/2901465234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1bd9ec0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 -- 172.21.15.62:0/2901465234 >> 172.21.15.62:0/2901465234 conn(0x5638a1b99310 msgr2=0x5638a1bd97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:25.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 -- 172.21.15.62:0/2901465234 shutdown_connections 2024-09-15T23:21:25.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 -- 172.21.15.62:0/2901465234 wait complete. 2024-09-15T23:21:25.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 -- start start 2024-09-15T23:21:25.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.839+0000 7f5a6991c740 1 Processor -- start 2024-09-15T23:21:25.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1be3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd1b40 0x5638a1bd4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a6991c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 0x5638a1bd50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a6991c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5a601af380 con 0x5638a1bd05d0 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a6991c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5a601b07f0 con 0x5638a1bd1b40 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a6991c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5a601afa50 con 0x5638a1bdb370 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a6991c740 10 client.0 _ll_register_callbacks cb 0x7f5a5000bf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a652df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1be3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a652df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1be3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51654/0 (socket says 172.21.15.62:51654) 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a652df640 1 -- 172.21.15.62:0/3522254675 learned_addr learned my addr 172.21.15.62:0/3522254675 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a64ade640 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd1b40 0x5638a1bd4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a65ae0640 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 0x5638a1bd50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a652df640 1 -- 172.21.15.62:0/3522254675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 msgr2=0x5638a1bd50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a652df640 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 0x5638a1bd50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a652df640 1 -- 172.21.15.62:0/3522254675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd1b40 msgr2=0x5638a1bd4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a652df640 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd1b40 0x5638a1bd4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a652df640 1 -- 172.21.15.62:0/3522254675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638a1bd5790 con 0x5638a1bd05d0 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a652df640 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1be3aa0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c01a040 tx=0x7f5a5c005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582da3ddd80d5938 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.840+0000 7f5a48ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a48ff9640 1 -- 172.21.15.62:0/3522254675 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5c033030 con 0x5638a1bd05d0 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a48ff9640 1 -- 172.21.15.62:0/3522254675 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5a5c02e740 con 0x5638a1bd05d0 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a6991c740 1 -- 172.21.15.62:0/3522254675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a5c02e950 con 0x5638a1bd05d0 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a48ff9640 1 -- 172.21.15.62:0/3522254675 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5a5c02eb00 con 0x5638a1bd05d0 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a6991c740 10 client.16716 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:25.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a6991c740 1 -- 172.21.15.62:0/3522254675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5638a1c55b00 con 0x5638a1bd05d0 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a48ff9640 1 -- 172.21.15.62:0/3522254675 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a5c02e950 con 0x5638a1bd05d0 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a48ff9640 1 -- 172.21.15.62:0/3522254675 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f5a5c010550 con 0x5638a1bd05d0 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a6991c740 10 client.16716 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a6991c740 10 client.16716 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a6991c740 20 client.16716 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5638a1c56100 0x5638a1c58520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a6991c740 4 client.16716 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.841+0000 7f5a6991c740 1 -- 172.21.15.62:0/3522254675 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5638a1b3dd70 con 0x5638a1c56100 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a64ade640 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5638a1c56100 0x5638a1c58520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a64ade640 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5638a1c56100 0x5638a1c58520 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e9727b1d901b709f server_cookie=43e3a4ac9d132b99 in_seq=0 out_seq=0 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a48ff9640 10 client.16716 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a48ff9640 1 -- 172.21.15.62:0/3522254675 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5638a1b3dd70 con 0x5638a1c56100 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a48ff9640 10 client.16716 handle_command_reply: tid=0 2024-09-15T23:21:25.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a6991c740 1 client.16716 shutdown 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a6991c740 1 -- 172.21.15.62:0/3522254675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 msgr2=0x5638a1be3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1be3aa0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c01a040 tx=0x7f5a5c005c50 comp rx=0 tx=0).stop 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a6991c740 1 -- 172.21.15.62:0/3522254675 shutdown_connections 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5638a1c56100 0x5638a1c58520 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a48ff9640 0 client.16716 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638a1bdb370 0x5638a1bd50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638a1bd1b40 0x5638a1bd4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a6991c740 1 --2- 172.21.15.62:0/3522254675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638a1bd05d0 0x5638a1be3aa0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.842+0000 7f5a6991c740 1 -- 172.21.15.62:0/3522254675 >> 172.21.15.62:0/3522254675 conn(0x5638a1b99310 msgr2=0x5638a1be6300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:25.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a6991c740 1 -- 172.21.15.62:0/3522254675 shutdown_connections 2024-09-15T23:21:25.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a6991c740 1 -- 172.21.15.62:0/3522254675 wait complete. 2024-09-15T23:21:25.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a6991c740 20 client.16716 trim_cache size 0 max 16384 2024-09-15T23:21:25.859 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:21:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 -- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5a4c0a3240 msgr2=0x7f5a4c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5a4c0a3240 0x7f5a4c0a5700 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5a58006010 tx=0x7f5a58009800 comp rx=0 tx=0).stop 2024-09-15T23:21:25.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 -- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 msgr2=0x7f5a60102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 0x7f5a60102b70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000b4a0 tx=0x7f5a5000b8a0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 -- 172.21.15.62:0/3593423724 shutdown_connections 2024-09-15T23:21:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5a4c0a3240 0x7f5a4c0a5700 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5a60106a60 0x7f5a601030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5a60101400 0x7f5a60102b70 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 --2- 172.21.15.62:0/3593423724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a60100a60 0x7f5a60102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 -- 172.21.15.62:0/3593423724 >> 172.21.15.62:0/3593423724 conn(0x7f5a600fc780 msgr2=0x7f5a60104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 -- 172.21.15.62:0/3593423724 shutdown_connections 2024-09-15T23:21:25.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:25.843+0000 7f5a67543640 1 -- 172.21.15.62:0/3593423724 wait complete. 2024-09-15T23:21:26.538 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:21:26.794 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:27.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 -- 172.21.15.62:0/3370243951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470100ca0 msgr2=0x7f6470106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:27.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 --2- 172.21.15.62:0/3370243951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470100ca0 0x7f6470106840 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6464009930 tx=0x7f646402a420 comp rx=0 tx=0).stop 2024-09-15T23:21:27.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 -- 172.21.15.62:0/3370243951 shutdown_connections 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 --2- 172.21.15.62:0/3370243951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6470106f50 0x7f647010eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 --2- 172.21.15.62:0/3370243951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470100ca0 0x7f6470106840 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 --2- 172.21.15.62:0/3370243951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f64701002f0 0x7f64701006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 -- 172.21.15.62:0/3370243951 >> 172.21.15.62:0/3370243951 conn(0x7f6470077210 msgr2=0x7f64700fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 -- 172.21.15.62:0/3370243951 shutdown_connections 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 -- 172.21.15.62:0/3370243951 wait complete. 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 Processor -- start 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.281+0000 7f6476916640 1 -- start start 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6476916640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64701002f0 0x7f6470198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6476916640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6470100ca0 0x7f6470198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6476916640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470106f50 0x7f647019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6476916640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6470111fa0 con 0x7f6470106f50 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6476916640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6470111e20 con 0x7f64701002f0 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6476916640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6470112120 con 0x7f6470100ca0 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6474eb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470106f50 0x7f647019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6474eb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470106f50 0x7f647019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51676/0 (socket says 172.21.15.62:51676) 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6474eb3640 1 -- 172.21.15.62:0/2490132344 learned_addr learned my addr 172.21.15.62:0/2490132344 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f646ffff640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64701002f0 0x7f6470198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f646f7fe640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6470100ca0 0x7f6470198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6474eb3640 1 -- 172.21.15.62:0/2490132344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6470100ca0 msgr2=0x7f6470198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6474eb3640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6470100ca0 0x7f6470198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6474eb3640 1 -- 172.21.15.62:0/2490132344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64701002f0 msgr2=0x7f6470198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6474eb3640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64701002f0 0x7f6470198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f6474eb3640 1 -- 172.21.15.62:0/2490132344 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647019d4b0 con 0x7f6470106f50 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f646f7fe640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6470100ca0 0x7f6470198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.282+0000 7f646ffff640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64701002f0 0x7f6470198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.283+0000 7f6474eb3640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470106f50 0x7f647019cdd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f646800eaa0 tx=0x7f646800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.283+0000 7f646d7fa640 1 -- 172.21.15.62:0/2490132344 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646800cdb0 con 0x7f6470106f50 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.283+0000 7f646d7fa640 1 -- 172.21.15.62:0/2490132344 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6468005020 con 0x7f6470106f50 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.283+0000 7f6476916640 1 -- 172.21.15.62:0/2490132344 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f647019d7a0 con 0x7f6470106f50 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.283+0000 7f646d7fa640 1 -- 172.21.15.62:0/2490132344 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64680116b0 con 0x7f6470106f50 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.283+0000 7f6476916640 1 -- 172.21.15.62:0/2490132344 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64701a5080 con 0x7f6470106f50 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.285+0000 7f646d7fa640 1 -- 172.21.15.62:0/2490132344 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f6468005560 con 0x7f6470106f50 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.285+0000 7f6478cef740 1 Processor -- start 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f6478cef740 1 -- start start 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f6478cef740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 0x5654a3d83090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f6478cef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d83a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f6478cef740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3d90760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f6478cef740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5654a3ce92f0 con 0x5654a3d8e370 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f6478cef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5654a3c90cf0 con 0x5654a3d835d0 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f6478cef740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5654a3ce52b0 con 0x5654a3d84b40 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646d7fa640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f64580a3240 0x7f64580a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3d90760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3d90760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51678/0 (socket says 172.21.15.62:51678) 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646f7fe640 1 -- 172.21.15.62:0/1347465519 learned_addr learned my addr 172.21.15.62:0/1347465519 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f6474eb3640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f64580a3240 0x7f64580a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646ffff640 1 --2- 172.21.15.62:0/1347465519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d83a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646f7fe640 1 --2- 172.21.15.62:0/1347465519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 0x5654a3d83090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646d7fa640 1 -- 172.21.15.62:0/2490132344 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f64680cff30 con 0x7f6470106f50 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646f7fe640 1 -- 172.21.15.62:0/1347465519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 msgr2=0x5654a3d83090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646f7fe640 1 --2- 172.21.15.62:0/1347465519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 0x5654a3d83090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646f7fe640 1 -- 172.21.15.62:0/1347465519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 msgr2=0x5654a3d83a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646f7fe640 1 --2- 172.21.15.62:0/1347465519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d83a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.286+0000 7f646f7fe640 1 -- 172.21.15.62:0/1347465519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654a3c753a0 con 0x5654a3d8e370 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.287+0000 7f6474eb3640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f64580a3240 0x7f64580a5700 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f646809c160 tx=0x7f6468066b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.287+0000 7f646f7fe640 1 --2- 172.21.15.62:0/1347465519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3d90760 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6464009930 tx=0x7f6464005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a92dc89d5790ad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.287+0000 7f64567fc640 1 -- 172.21.15.62:0/1347465519 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464035040 con 0x5654a3d8e370 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.287+0000 7f64567fc640 1 -- 172.21.15.62:0/1347465519 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64640319f0 con 0x5654a3d8e370 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.287+0000 7f64567fc640 1 -- 172.21.15.62:0/1347465519 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464003e30 con 0x5654a3d8e370 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 -- 172.21.15.62:0/1347465519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 msgr2=0x5654a3d90760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 --2- 172.21.15.62:0/1347465519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3d90760 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6464009930 tx=0x7f6464005b50 comp rx=0 tx=0).stop 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 -- 172.21.15.62:0/1347465519 shutdown_connections 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 --2- 172.21.15.62:0/1347465519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3d90760 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 --2- 172.21.15.62:0/1347465519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d83a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 --2- 172.21.15.62:0/1347465519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 0x5654a3d83090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 -- 172.21.15.62:0/1347465519 >> 172.21.15.62:0/1347465519 conn(0x5654a3d4c310 msgr2=0x5654a3d7fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 -- 172.21.15.62:0/1347465519 shutdown_connections 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 -- 172.21.15.62:0/1347465519 wait complete. 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 -- start start 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.288+0000 7f6478cef740 1 Processor -- start 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f6478cef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d8dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f6478cef740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d84b40 0x5654a3d8b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f6478cef740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 0x5654a3d8b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f6478cef740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5654a3d91010 con 0x5654a3d84b40 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f6478cef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5654a3ca4060 con 0x5654a3d835d0 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f6478cef740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5654a3d91190 con 0x5654a3d8e370 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f6478cef740 10 client.0 _ll_register_callbacks cb 0x5654a39f78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:27.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d84b40 0x5654a3d8b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d84b40 0x5654a3d8b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51684/0 (socket says 172.21.15.62:51684) 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646ffff640 1 -- 172.21.15.62:0/160287000 learned_addr learned my addr 172.21.15.62:0/160287000 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646f7fe640 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d8dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646f7fe640 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 0x5654a3d8b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646ffff640 1 -- 172.21.15.62:0/160287000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 msgr2=0x5654a3d8b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646ffff640 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 0x5654a3d8b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646ffff640 1 -- 172.21.15.62:0/160287000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 msgr2=0x5654a3d8dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646ffff640 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d8dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646ffff640 1 -- 172.21.15.62:0/160287000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654a3a5a3c0 con 0x5654a3d84b40 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f646ffff640 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d84b40 0x5654a3d8b020 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f646000bfb0 tx=0x7f64600128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3553c4efc7fe7760 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f644f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f644f7fe640 1 -- 172.21.15.62:0/160287000 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460005220 con 0x5654a3d84b40 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f644f7fe640 1 -- 172.21.15.62:0/160287000 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f64600053c0 con 0x5654a3d84b40 2024-09-15T23:21:27.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.289+0000 7f6478cef740 1 -- 172.21.15.62:0/160287000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64640319f0 con 0x5654a3d84b40 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f644f7fe640 1 -- 172.21.15.62:0/160287000 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6460025050 con 0x5654a3d84b40 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f6478cef740 10 client.16728 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f6478cef740 1 -- 172.21.15.62:0/160287000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5654a3d9cb10 con 0x5654a3d84b40 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f644f7fe640 1 -- 172.21.15.62:0/160287000 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64640319f0 con 0x5654a3d84b40 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f644f7fe640 1 -- 172.21.15.62:0/160287000 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6460019030 con 0x5654a3d84b40 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f6478cef740 10 client.16728 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f6478cef740 10 client.16728 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f6478cef740 20 client.16728 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f6478cef740 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5654a3d9e070 0x5654a3da0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f6478cef740 4 client.16728 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f6478cef740 1 -- 172.21.15.62:0/160287000 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654a3cf6860 con 0x5654a3d9e070 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.290+0000 7f646f7fe640 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5654a3d9e070 0x5654a3da0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.291+0000 7f644f7fe640 10 client.16728 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:21:27.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.291+0000 7f646f7fe640 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5654a3d9e070 0x5654a3da0490 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=71b0a78a42ce0a66 server_cookie=5a24c5145f7bd87d in_seq=0 out_seq=0 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f644f7fe640 1 -- 172.21.15.62:0/160287000 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5654a3cf6860 con 0x5654a3d9e070 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f644f7fe640 10 client.16728 handle_command_reply: tid=0 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 client.16728 shutdown 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 -- 172.21.15.62:0/160287000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d84b40 msgr2=0x5654a3d8b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d84b40 0x5654a3d8b020 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f646000bfb0 tx=0x7f64600128e0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 -- 172.21.15.62:0/160287000 shutdown_connections 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5654a3d9e070 0x5654a3da0490 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 0x5654a3d8b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d84b40 0x5654a3d8b020 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f644f7fe640 0 client.16728 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 --2- 172.21.15.62:0/160287000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d8dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 -- 172.21.15.62:0/160287000 >> 172.21.15.62:0/160287000 conn(0x5654a3d4c310 msgr2=0x5654a3d85960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 -- 172.21.15.62:0/160287000 shutdown_connections 2024-09-15T23:21:27.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 1 -- 172.21.15.62:0/160287000 wait complete. 2024-09-15T23:21:27.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.292+0000 7f6478cef740 20 client.16728 trim_cache size 0 max 16384 2024-09-15T23:21:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f6478cef740 1 Processor -- start 2024-09-15T23:21:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f6478cef740 1 -- start start 2024-09-15T23:21:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f6478cef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d8aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f6478cef740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 0x5654a3d9a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f6478cef740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3da5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f6478cef740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f64701b1580 con 0x5654a3d8e370 2024-09-15T23:21:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f6478cef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64701aee50 con 0x5654a3d835d0 2024-09-15T23:21:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f6478cef740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f64701acca0 con 0x5654a3d84b40 2024-09-15T23:21:27.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f646f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3da5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f646f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3da5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51688/0 (socket says 172.21.15.62:51688) 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f646f7fe640 1 -- 172.21.15.62:0/3126319805 learned_addr learned my addr 172.21.15.62:0/3126319805 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.305+0000 7f646f7fe640 1 --2- 172.21.15.62:0/3126319805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d8aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f646ffff640 1 --2- 172.21.15.62:0/3126319805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 0x5654a3d9a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f646f7fe640 1 -- 172.21.15.62:0/3126319805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 msgr2=0x5654a3d9a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f646f7fe640 1 --2- 172.21.15.62:0/3126319805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 0x5654a3d9a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f646f7fe640 1 -- 172.21.15.62:0/3126319805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 msgr2=0x5654a3d8aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f646f7fe640 1 --2- 172.21.15.62:0/3126319805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d8aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f646f7fe640 1 -- 172.21.15.62:0/3126319805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654a3d80960 con 0x5654a3d8e370 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f646f7fe640 1 --2- 172.21.15.62:0/3126319805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3da5d60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6464053fd0 tx=0x7f6464040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3316e5c07c30a2d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f6455ffb640 1 -- 172.21.15.62:0/3126319805 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464035040 con 0x5654a3d8e370 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f6455ffb640 1 -- 172.21.15.62:0/3126319805 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6464031910 con 0x5654a3d8e370 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.306+0000 7f6455ffb640 1 -- 172.21.15.62:0/3126319805 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464053570 con 0x5654a3d8e370 2024-09-15T23:21:27.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 -- 172.21.15.62:0/3126319805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 msgr2=0x5654a3da5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 --2- 172.21.15.62:0/3126319805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3da5d60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6464053fd0 tx=0x7f6464040040 comp rx=0 tx=0).stop 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 -- 172.21.15.62:0/3126319805 shutdown_connections 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 --2- 172.21.15.62:0/3126319805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d8e370 0x5654a3da5d60 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 --2- 172.21.15.62:0/3126319805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d84b40 0x5654a3d9a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 --2- 172.21.15.62:0/3126319805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d835d0 0x5654a3d8aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 -- 172.21.15.62:0/3126319805 >> 172.21.15.62:0/3126319805 conn(0x5654a3d4c310 msgr2=0x5654a3d8a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 -- 172.21.15.62:0/3126319805 shutdown_connections 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 -- 172.21.15.62:0/3126319805 wait complete. 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 -- start start 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.307+0000 7f6478cef740 1 Processor -- start 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f6478cef740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d835d0 0x5654a3d88d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f6478cef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d84b40 0x5654a3d963d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f6478cef740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 0x5654a3d969a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f6478cef740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f64701ad310 con 0x5654a3d835d0 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f6478cef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64701ae780 con 0x5654a3d84b40 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f6478cef740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f64701ad9e0 con 0x5654a3d8e370 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f6478cef740 10 client.0 _ll_register_callbacks cb 0x7f646000b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d835d0 0x5654a3d88d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d835d0 0x5654a3d88d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51698/0 (socket says 172.21.15.62:51698) 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 -- 172.21.15.62:0/2689725845 learned_addr learned my addr 172.21.15.62:0/2689725845 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646ffff640 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d84b40 0x5654a3d963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 0x5654a3d969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 -- 172.21.15.62:0/2689725845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 msgr2=0x5654a3d969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 0x5654a3d969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 -- 172.21.15.62:0/2689725845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d84b40 msgr2=0x5654a3d963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d84b40 0x5654a3d963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 -- 172.21.15.62:0/2689725845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654a3d9a7f0 con 0x5654a3d835d0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f646f7fe640 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d835d0 0x5654a3d88d70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6464005ab0 tx=0x7f6464005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689210cbf8d2f27a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f644f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f644f7fe640 1 -- 172.21.15.62:0/2689725845 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464035040 con 0x5654a3d835d0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f644f7fe640 1 -- 172.21.15.62:0/2689725845 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f64640535d0 con 0x5654a3d835d0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.308+0000 7f6478cef740 1 -- 172.21.15.62:0/2689725845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6464031910 con 0x5654a3d835d0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f644f7fe640 1 -- 172.21.15.62:0/2689725845 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f64640537f0 con 0x5654a3d835d0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f6478cef740 10 client.16734 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f6478cef740 1 -- 172.21.15.62:0/2689725845 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5654a3d7f4f0 con 0x5654a3d835d0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f644f7fe640 1 -- 172.21.15.62:0/2689725845 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6464031910 con 0x5654a3d835d0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f644f7fe640 1 -- 172.21.15.62:0/2689725845 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f646405d020 con 0x5654a3d835d0 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f6478cef740 10 client.16734 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f6478cef740 10 client.16734 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f6478cef740 20 client.16734 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:27.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f6478cef740 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5654a3e11a40 0x5654a3e13e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f6478cef740 4 client.16734 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f6478cef740 1 -- 172.21.15.62:0/2689725845 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5654a3cf0d70 con 0x5654a3e11a40 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.309+0000 7f646ffff640 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5654a3e11a40 0x5654a3e13e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f646ffff640 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5654a3e11a40 0x5654a3e13e30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=47a446fdc5d1cf9 server_cookie=b4c89c2a69e9aa6a in_seq=0 out_seq=0 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f644f7fe640 10 client.16734 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f644f7fe640 1 -- 172.21.15.62:0/2689725845 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5654a3cf0d70 con 0x5654a3e11a40 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f644f7fe640 10 client.16734 handle_command_reply: tid=0 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f6478cef740 1 client.16734 shutdown 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f6478cef740 1 -- 172.21.15.62:0/2689725845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d835d0 msgr2=0x5654a3d88d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f6478cef740 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d835d0 0x5654a3d88d70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6464005ab0 tx=0x7f6464005a20 comp rx=0 tx=0).stop 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f6478cef740 1 -- 172.21.15.62:0/2689725845 shutdown_connections 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f6478cef740 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x5654a3e11a40 0x5654a3e13e30 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f6478cef740 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5654a3d8e370 0x5654a3d969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f6478cef740 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5654a3d84b40 0x5654a3d963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f6478cef740 1 --2- 172.21.15.62:0/2689725845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654a3d835d0 0x5654a3d88d70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f6478cef740 1 -- 172.21.15.62:0/2689725845 >> 172.21.15.62:0/2689725845 conn(0x5654a3d4c310 msgr2=0x5654a3d7e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.310+0000 7f644f7fe640 0 client.16734 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:21:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6478cef740 1 -- 172.21.15.62:0/2689725845 shutdown_connections 2024-09-15T23:21:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6478cef740 1 -- 172.21.15.62:0/2689725845 wait complete. 2024-09-15T23:21:27.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6478cef740 20 client.16734 trim_cache size 0 max 16384 2024-09-15T23:21:27.319 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 -- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f64580a3240 msgr2=0x7f64580a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f64580a3240 0x7f64580a5700 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f646809c160 tx=0x7f6468066b60 comp rx=0 tx=0).stop 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 -- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470106f50 msgr2=0x7f647019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470106f50 0x7f647019cdd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f646800eaa0 tx=0x7f646800ef60 comp rx=0 tx=0).stop 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 -- 172.21.15.62:0/2490132344 shutdown_connections 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f64580a3240 0x7f64580a5700 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6470106f50 0x7f647019cdd0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6470100ca0 0x7f6470198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 --2- 172.21.15.62:0/2490132344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64701002f0 0x7f6470198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 -- 172.21.15.62:0/2490132344 >> 172.21.15.62:0/2490132344 conn(0x7f6470077210 msgr2=0x7f6470105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 -- 172.21.15.62:0/2490132344 shutdown_connections 2024-09-15T23:21:27.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:27.311+0000 7f6476916640 1 -- 172.21.15.62:0/2490132344 wait complete. 2024-09-15T23:21:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:27 smithi062 ceph-mon[30949]: pgmap v525: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 382 KiB/s wr, 55 op/s 2024-09-15T23:21:27.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:27 smithi067 ceph-mon[35665]: pgmap v525: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 382 KiB/s wr, 55 op/s 2024-09-15T23:21:27.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:27 smithi148 ceph-mon[35610]: pgmap v525: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 382 KiB/s wr, 55 op/s 2024-09-15T23:21:27.998 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:21:28.255 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:28.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.698+0000 7fcb50926640 1 -- 172.21.15.62:0/1810919317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 msgr2=0x7fcb4c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:28.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.698+0000 7fcb50926640 1 --2- 172.21.15.62:0/1810919317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 0x7fcb4c1021c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c009900 tx=0x7fcb3c02a3f0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.698+0000 7fcb50926640 1 -- 172.21.15.62:0/1810919317 shutdown_connections 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.698+0000 7fcb50926640 1 --2- 172.21.15.62:0/1810919317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb4c106f10 0x7fcb4c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.698+0000 7fcb50926640 1 --2- 172.21.15.62:0/1810919317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c102790 0x7fcb4c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.698+0000 7fcb50926640 1 --2- 172.21.15.62:0/1810919317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 0x7fcb4c1021c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.698+0000 7fcb50926640 1 -- 172.21.15.62:0/1810919317 >> 172.21.15.62:0/1810919317 conn(0x7fcb4c0fd8c0 msgr2=0x7fcb4c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.698+0000 7fcb50926640 1 -- 172.21.15.62:0/1810919317 shutdown_connections 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.699+0000 7fcb50926640 1 -- 172.21.15.62:0/1810919317 wait complete. 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.699+0000 7fcb50926640 1 Processor -- start 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.699+0000 7fcb50926640 1 -- start start 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb50926640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 0x7fcb4c19a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb50926640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb4c102790 0x7fcb4c19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb50926640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c106f10 0x7fcb4c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb50926640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c111fc0 con 0x7fcb4c101de0 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb50926640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c111e40 con 0x7fcb4c106f10 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb50926640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c112140 con 0x7fcb4c102790 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 0x7fcb4c19a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 0x7fcb4c19a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51718/0 (socket says 172.21.15.62:51718) 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4a59c640 1 -- 172.21.15.62:0/505494451 learned_addr learned my addr 172.21.15.62:0/505494451 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c106f10 0x7fcb4c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb4c102790 0x7fcb4c19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4a59c640 1 -- 172.21.15.62:0/505494451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb4c102790 msgr2=0x7fcb4c19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4a59c640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb4c102790 0x7fcb4c19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4a59c640 1 -- 172.21.15.62:0/505494451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c106f10 msgr2=0x7fcb4c19edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4a59c640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c106f10 0x7fcb4c19edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4a59c640 1 -- 172.21.15.62:0/505494451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb4c19f4d0 con 0x7fcb4c101de0 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb4c102790 0x7fcb4c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb4a59c640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 0x7fcb4c19a520 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c006010 tx=0x7fcb3c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.700+0000 7fcb437fe640 1 -- 172.21.15.62:0/505494451 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c03f070 con 0x7fcb4c101de0 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.701+0000 7fcb437fe640 1 -- 172.21.15.62:0/505494451 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb3c031c90 con 0x7fcb4c101de0 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.701+0000 7fcb50926640 1 -- 172.21.15.62:0/505494451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb4c19f760 con 0x7fcb4c101de0 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.701+0000 7fcb437fe640 1 -- 172.21.15.62:0/505494451 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3c035030 con 0x7fcb4c101de0 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.701+0000 7fcb50926640 1 -- 172.21.15.62:0/505494451 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb4c1a70a0 con 0x7fcb4c101de0 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb437fe640 1 -- 172.21.15.62:0/505494451 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fcb3c042070 con 0x7fcb4c101de0 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb52cff740 1 Processor -- start 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb52cff740 1 -- start start 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbfb40 0x56265adbe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 0x56265adbea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adc9370 0x56265adcb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb52cff740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56265ad242f0 con 0x56265adbfb40 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb52cff740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56265accbcf0 con 0x56265adc9370 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb52cff740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56265ad202b0 con 0x56265adbe5d0 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb4ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbfb40 0x56265adbe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb4ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbfb40 0x56265adbe090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51722/0 (socket says 172.21.15.62:51722) 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1116968674 learned_addr learned my addr 172.21.15.62:0/1116968674 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/1116968674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adc9370 0x56265adcb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.704+0000 7fcb4a59c640 1 --2- 172.21.15.62:0/1116968674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 0x56265adbea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1116968674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 msgr2=0x56265adbea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1116968674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 0x56265adbea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1116968674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adc9370 msgr2=0x56265adcb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1116968674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adc9370 0x56265adcb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1116968674 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56265acb03a0 con 0x56265adbfb40 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb437fe640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb380a3240 0x7fcb380a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1116968674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbfb40 0x56265adbe090 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fcb44007ec0 tx=0x7fcb4400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6880fd6536f5a1a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb4a59c640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb380a3240 0x7fcb380a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb40ff9640 1 -- 172.21.15.62:0/1116968674 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44010840 con 0x56265adbfb40 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb40ff9640 1 -- 172.21.15.62:0/1116968674 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb440183f0 con 0x56265adbfb40 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb40ff9640 1 -- 172.21.15.62:0/1116968674 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44017600 con 0x56265adbfb40 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.705+0000 7fcb437fe640 1 -- 172.21.15.62:0/505494451 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcb3c0f05a0 con 0x7fcb4c101de0 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb4a59c640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb380a3240 0x7fcb380a5700 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0ee090 tx=0x7fcb3c0f5040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb52cff740 1 -- 172.21.15.62:0/1116968674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbfb40 msgr2=0x56265adbe090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1116968674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbfb40 0x56265adbe090 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fcb44007ec0 tx=0x7fcb4400f790 comp rx=0 tx=0).stop 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb52cff740 1 -- 172.21.15.62:0/1116968674 shutdown_connections 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1116968674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adc9370 0x56265adcb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1116968674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 0x56265adbea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1116968674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbfb40 0x56265adbe090 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb52cff740 1 -- 172.21.15.62:0/1116968674 >> 172.21.15.62:0/1116968674 conn(0x56265ad87310 msgr2=0x56265adbac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb52cff740 1 -- 172.21.15.62:0/1116968674 shutdown_connections 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.706+0000 7fcb52cff740 1 -- 172.21.15.62:0/1116968674 wait complete. 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb52cff740 1 -- start start 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb52cff740 1 Processor -- start 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 0x56265adc8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 0x56265adc6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb52cff740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56265adcc070 con 0x56265adc9370 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb52cff740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56265acdf060 con 0x56265adbfb40 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb52cff740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56265adcc1f0 con 0x56265adbe5d0 2024-09-15T23:21:28.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb52cff740 10 client.0 _ll_register_callbacks cb 0x56265aa328c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb4ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb4ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51734/0 (socket says 172.21.15.62:51734) 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.707+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1852149325 learned_addr learned my addr 172.21.15.62:0/1852149325 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 0x56265adc6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 0x56265adc8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1852149325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 msgr2=0x56265adc8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 0x56265adc8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1852149325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 msgr2=0x56265adc6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 0x56265adc6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1852149325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56265aa953c0 con 0x56265adc9370 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 0x56265adc6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc6650 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcb440078b0 tx=0x7fcb44012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873daa7ba612d077 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:28.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1852149325 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4402c060 con 0x56265adc9370 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1852149325 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcb44004c40 con 0x56265adc9370 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb52cff740 1 -- 172.21.15.62:0/1852149325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb440183f0 con 0x56265adc9370 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1852149325 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcb44033050 con 0x56265adc9370 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb52cff740 10 client.16746 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb52cff740 1 -- 172.21.15.62:0/1852149325 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56265add7b40 con 0x56265adc9370 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.708+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1852149325 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb440183f0 con 0x56265adc9370 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1852149325 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fcb44010640 con 0x56265adc9370 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb52cff740 10 client.16746 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb52cff740 10 client.16746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb52cff740 20 client.16746 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56265add8fa0 0x56265addb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb52cff740 4 client.16746 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb52cff740 1 -- 172.21.15.62:0/1852149325 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56265ad31860 con 0x56265add8fa0 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56265add8fa0 0x56265addb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb31ffb640 10 client.16746 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.709+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56265add8fa0 0x56265addb390 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c93529f7dddb6891 server_cookie=f04277d8768e36aa in_seq=0 out_seq=0 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1852149325 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56265ad31860 con 0x56265add8fa0 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb31ffb640 10 client.16746 handle_command_reply: tid=0 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 client.16746 shutdown 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 -- 172.21.15.62:0/1852149325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 msgr2=0x56265adc6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc6650 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcb440078b0 tx=0x7fcb44012040 comp rx=0 tx=0).stop 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 -- 172.21.15.62:0/1852149325 shutdown_connections 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56265add8fa0 0x56265addb390 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc6650 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 0x56265adc6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1852149325 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbe5d0 0x56265adc8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb31ffb640 0 client.16746 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:28.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 -- 172.21.15.62:0/1852149325 >> 172.21.15.62:0/1852149325 conn(0x56265ad87310 msgr2=0x56265adc0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:28.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 -- 172.21.15.62:0/1852149325 shutdown_connections 2024-09-15T23:21:28.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.710+0000 7fcb52cff740 1 -- 172.21.15.62:0/1852149325 wait complete. 2024-09-15T23:21:28.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.711+0000 7fcb52cff740 20 client.16746 trim_cache size 0 max 16384 2024-09-15T23:21:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.725+0000 7fcb52cff740 1 Processor -- start 2024-09-15T23:21:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.725+0000 7fcb52cff740 1 -- start start 2024-09-15T23:21:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.725+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbe5d0 0x56265adbae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 0x56265add5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adc9370 0x56265ade0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb52cff740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c1b35f0 con 0x56265adbe5d0 2024-09-15T23:21:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb52cff740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c1b0ec0 con 0x56265adbfb40 2024-09-15T23:21:28.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb52cff740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c1aed10 con 0x56265adc9370 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb4ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbe5d0 0x56265adbae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb4ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbe5d0 0x56265adbae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51738/0 (socket says 172.21.15.62:51738) 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/339903222 learned_addr learned my addr 172.21.15.62:0/339903222 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/339903222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 0x56265add5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/339903222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adc9370 msgr2=0x56265ade0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/339903222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adc9370 0x56265ade0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/339903222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 msgr2=0x56265add5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/339903222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 0x56265add5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/339903222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56265adbf760 con 0x56265adbe5d0 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/339903222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbe5d0 0x56265adbae10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fcb44000f80 tx=0x7fcb44017420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf414891b13cec4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.726+0000 7fcb33fff640 1 -- 172.21.15.62:0/339903222 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4402c040 con 0x56265adbe5d0 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb33fff640 1 -- 172.21.15.62:0/339903222 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb4400fc90 con 0x56265adbe5d0 2024-09-15T23:21:28.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb33fff640 1 -- 172.21.15.62:0/339903222 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb44017a60 con 0x56265adbe5d0 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb52cff740 1 -- 172.21.15.62:0/339903222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbe5d0 msgr2=0x56265adbae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb52cff740 1 --2- 172.21.15.62:0/339903222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbe5d0 0x56265adbae10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fcb44000f80 tx=0x7fcb44017420 comp rx=0 tx=0).stop 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb52cff740 1 -- 172.21.15.62:0/339903222 shutdown_connections 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb52cff740 1 --2- 172.21.15.62:0/339903222 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adc9370 0x56265ade0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb52cff740 1 --2- 172.21.15.62:0/339903222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbfb40 0x56265add5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb52cff740 1 --2- 172.21.15.62:0/339903222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adbe5d0 0x56265adbae10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb52cff740 1 -- 172.21.15.62:0/339903222 >> 172.21.15.62:0/339903222 conn(0x56265ad87310 msgr2=0x56265adba700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb52cff740 1 -- 172.21.15.62:0/339903222 shutdown_connections 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.727+0000 7fcb52cff740 1 -- 172.21.15.62:0/339903222 wait complete. 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb52cff740 1 -- start start 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb52cff740 1 Processor -- start 2024-09-15T23:21:28.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbe5d0 0x56265adc2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbfb40 0x56265adc3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb52cff740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb52cff740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c1af380 con 0x56265adc9370 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb52cff740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c1b07f0 con 0x56265adbe5d0 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb52cff740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4c1afa50 con 0x56265adbfb40 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb52cff740 10 client.0 _ll_register_callbacks cb 0x7fcb3400b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb4ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb4ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51746/0 (socket says 172.21.15.62:51746) 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1388264812 learned_addr learned my addr 172.21.15.62:0/1388264812 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbe5d0 0x56265adc2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbfb40 0x56265adc3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1388264812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbfb40 msgr2=0x56265adc3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.728+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbfb40 0x56265adc3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1388264812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbe5d0 msgr2=0x56265adc2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbe5d0 0x56265adc2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb4ad9d640 1 -- 172.21.15.62:0/1388264812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56265ae42900 con 0x56265adc9370 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb4ad9d640 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc3a50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcb44007920 tx=0x7fcb440109a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53335117cf5614e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb31ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1388264812 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4402c030 con 0x56265adc9370 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1388264812 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcb4403c070 con 0x56265adc9370 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb52cff740 1 -- 172.21.15.62:0/1388264812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb4400fc90 con 0x56265adc9370 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1388264812 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcb440177e0 con 0x56265adc9370 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb52cff740 10 client.16752 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb52cff740 1 -- 172.21.15.62:0/1388264812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56265ae43b40 con 0x56265adc9370 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1388264812 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb4400fc90 con 0x56265adc9370 2024-09-15T23:21:28.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.729+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1388264812 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fcb44018960 con 0x56265adc9370 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb52cff740 10 client.16752 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb52cff740 10 client.16752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb52cff740 20 client.16752 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56265ae441a0 0x56265ae46590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb52cff740 4 client.16752 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb52cff740 1 -- 172.21.15.62:0/1388264812 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56265ad2bd70 con 0x56265ae441a0 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56265ae441a0 0x56265ae46590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb49d9b640 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56265ae441a0 0x56265ae46590 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17224f0e3e5f9f13 server_cookie=f61d60862ce688f8 in_seq=0 out_seq=0 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb31ffb640 10 client.16752 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb31ffb640 1 -- 172.21.15.62:0/1388264812 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56265ad2bd70 con 0x56265ae441a0 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb31ffb640 10 client.16752 handle_command_reply: tid=0 2024-09-15T23:21:28.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.730+0000 7fcb52cff740 1 client.16752 shutdown 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 -- 172.21.15.62:0/1388264812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 msgr2=0x56265adc3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc3a50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcb44007920 tx=0x7fcb440109a0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 -- 172.21.15.62:0/1388264812 shutdown_connections 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56265ae441a0 0x56265ae46590 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56265adc9370 0x56265adc3a50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56265adbfb40 0x56265adc3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb31ffb640 0 client.16752 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 --2- 172.21.15.62:0/1388264812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56265adbe5d0 0x56265adc2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 -- 172.21.15.62:0/1388264812 >> 172.21.15.62:0/1388264812 conn(0x56265ad87310 msgr2=0x56265adc78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 -- 172.21.15.62:0/1388264812 shutdown_connections 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 1 -- 172.21.15.62:0/1388264812 wait complete. 2024-09-15T23:21:28.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb52cff740 20 client.16752 trim_cache size 0 max 16384 2024-09-15T23:21:28.746 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:21:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.731+0000 7fcb50926640 1 -- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb380a3240 msgr2=0x7fcb380a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb380a3240 0x7fcb380a5700 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0ee090 tx=0x7fcb3c0f5040 comp rx=0 tx=0).stop 2024-09-15T23:21:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 -- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 msgr2=0x7fcb4c19a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:28.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 0x7fcb4c19a520 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c006010 tx=0x7fcb3c003ba0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 -- 172.21.15.62:0/505494451 shutdown_connections 2024-09-15T23:21:28.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb380a3240 0x7fcb380a5700 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb4c106f10 0x7fcb4c19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb4c102790 0x7fcb4c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 --2- 172.21.15.62:0/505494451 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb4c101de0 0x7fcb4c19a520 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:28.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 -- 172.21.15.62:0/505494451 >> 172.21.15.62:0/505494451 conn(0x7fcb4c0fd8c0 msgr2=0x7fcb4c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:28.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 -- 172.21.15.62:0/505494451 shutdown_connections 2024-09-15T23:21:28.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:28.732+0000 7fcb50926640 1 -- 172.21.15.62:0/505494451 wait complete. 2024-09-15T23:21:29.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 310.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:21:29.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:29 smithi062 ceph-mon[30949]: pgmap v526: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 82 KiB/s wr, 22 op/s 2024-09-15T23:21:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:29 smithi067 ceph-mon[35665]: pgmap v526: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 82 KiB/s wr, 22 op/s 2024-09-15T23:21:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:29 smithi148 ceph-mon[35610]: pgmap v526: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 82 KiB/s wr, 22 op/s 2024-09-15T23:21:30.461 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:21:30.461 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:21:30.734 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:31.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.207+0000 7f8780957640 1 -- 172.21.15.62:0/3468096398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c103660 msgr2=0x7f877c10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:31.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.207+0000 7f8780957640 1 --2- 172.21.15.62:0/3468096398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c103660 0x7f877c10bb50 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8770009900 tx=0x7f877002a480 comp rx=0 tx=0).stop 2024-09-15T23:21:31.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.207+0000 7f8780957640 1 -- 172.21.15.62:0/3468096398 shutdown_connections 2024-09-15T23:21:31.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.207+0000 7f8780957640 1 --2- 172.21.15.62:0/3468096398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c10c170 0x7f877c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.207+0000 7f8780957640 1 --2- 172.21.15.62:0/3468096398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c103660 0x7f877c10bb50 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.207+0000 7f8780957640 1 --2- 172.21.15.62:0/3468096398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f877c102d40 0x7f877c103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.207+0000 7f8780957640 1 -- 172.21.15.62:0/3468096398 >> 172.21.15.62:0/3468096398 conn(0x7f877c0fc7a0 msgr2=0x7f877c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:31.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.207+0000 7f8780957640 1 -- 172.21.15.62:0/3468096398 shutdown_connections 2024-09-15T23:21:31.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.207+0000 7f8780957640 1 -- 172.21.15.62:0/3468096398 wait complete. 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.208+0000 7f8780957640 1 Processor -- start 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.208+0000 7f8780957640 1 -- start start 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.208+0000 7f8780957640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c102d40 0x7f877c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f8780957640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f877c103660 0x7f877c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f8780957640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c10c170 0x7f877c19cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f8780957640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f877c111f70 con 0x7f877c102d40 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f8780957640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f877c111df0 con 0x7f877c10c170 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f8780957640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f877c1120f0 con 0x7f877c103660 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c102d40 0x7f877c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c102d40 0x7f877c198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51766/0 (socket says 172.21.15.62:51766) 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877a59c640 1 -- 172.21.15.62:0/267445552 learned_addr learned my addr 172.21.15.62:0/267445552 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877ad9d640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c10c170 0x7f877c19cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f8779d9b640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f877c103660 0x7f877c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877a59c640 1 -- 172.21.15.62:0/267445552 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f877c103660 msgr2=0x7f877c198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877a59c640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f877c103660 0x7f877c198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877a59c640 1 -- 172.21.15.62:0/267445552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c10c170 msgr2=0x7f877c19cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877a59c640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c10c170 0x7f877c19cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877a59c640 1 -- 172.21.15.62:0/267445552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877c19d3e0 con 0x7f877c102d40 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f877a59c640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c102d40 0x7f877c198430 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f876c00d950 tx=0x7f876c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f876b7fe640 1 -- 172.21.15.62:0/267445552 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c017070 con 0x7f877c102d40 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f8780957640 1 -- 172.21.15.62:0/267445552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f877c19d6d0 con 0x7f877c102d40 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f876b7fe640 1 -- 172.21.15.62:0/267445552 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f876c009c30 con 0x7f877c102d40 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f876b7fe640 1 -- 172.21.15.62:0/267445552 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c00f6b0 con 0x7f877c102d40 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.209+0000 7f8780957640 1 -- 172.21.15.62:0/267445552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f877c1a4fb0 con 0x7f877c102d40 2024-09-15T23:21:31.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.212+0000 7f876b7fe640 1 -- 172.21.15.62:0/267445552 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f876c0109d0 con 0x7f877c102d40 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.212+0000 7f8782d30740 1 Processor -- start 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.212+0000 7f8782d30740 1 -- start start 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f8782d30740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c90787f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f8782d30740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90788480 0x555c90792800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f8782d30740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90792d40 0x555c90795130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f8782d30740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555c906ed280 con 0x555c90788480 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f8782d30740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555c90694b10 con 0x555c90787b60 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f8782d30740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555c906e9240 con 0x555c90792d40 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f876b7fe640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87640a3240 0x7f87640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90788480 0x555c90792800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90788480 0x555c90792800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51770/0 (socket says 172.21.15.62:51770) 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877a59c640 1 -- 172.21.15.62:0/628215170 learned_addr learned my addr 172.21.15.62:0/628215170 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:31.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877ad9d640 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c90787f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877ad9d640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87640a3240 0x7f87640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f8779d9b640 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90792d40 0x555c90795130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f876b7fe640 1 -- 172.21.15.62:0/267445552 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f876c0cf7f0 con 0x7f877c102d40 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877a59c640 1 -- 172.21.15.62:0/628215170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90792d40 msgr2=0x555c90795130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877a59c640 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90792d40 0x555c90795130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877a59c640 1 -- 172.21.15.62:0/628215170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 msgr2=0x555c90787f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877a59c640 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c90787f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877a59c640 1 -- 172.21.15.62:0/628215170 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c906a11f0 con 0x555c90788480 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.213+0000 7f877ad9d640 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c90787f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.214+0000 7f877a59c640 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90788480 0x555c90792800 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f876c05c680 tx=0x7f876c00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39aed14b21e1316 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.214+0000 7f877ad9d640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87640a3240 0x7f87640a5700 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8774005fd0 tx=0x7f8774009600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:31.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.214+0000 7f8768ff9640 1 -- 172.21.15.62:0/628215170 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c020070 con 0x555c90788480 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.214+0000 7f8768ff9640 1 -- 172.21.15.62:0/628215170 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f876c009de0 con 0x555c90788480 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.214+0000 7f8768ff9640 1 -- 172.21.15.62:0/628215170 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c0981b0 con 0x555c90788480 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.214+0000 7f8782d30740 1 -- 172.21.15.62:0/628215170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90788480 msgr2=0x555c90792800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.214+0000 7f8782d30740 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90788480 0x555c90792800 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f876c05c680 tx=0x7f876c00bf60 comp rx=0 tx=0).stop 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 -- 172.21.15.62:0/628215170 shutdown_connections 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90792d40 0x555c90795130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90788480 0x555c90792800 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 --2- 172.21.15.62:0/628215170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c90787f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 -- 172.21.15.62:0/628215170 >> 172.21.15.62:0/628215170 conn(0x555c907840f0 msgr2=0x555c907844d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 -- 172.21.15.62:0/628215170 shutdown_connections 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 -- 172.21.15.62:0/628215170 wait complete. 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 -- start start 2024-09-15T23:21:31.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 Processor -- start 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9078fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 0x555c90790150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90790690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555c90795910 con 0x555c90792d40 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555c906a7f10 con 0x555c90787b60 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555c90795a90 con 0x555c90788480 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f8782d30740 10 client.0 _ll_register_callbacks cb 0x555c90670a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.215+0000 7f877ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90790690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f877ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90790690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51786/0 (socket says 172.21.15.62:51786) 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f877ad9d640 1 -- 172.21.15.62:0/2534142920 learned_addr learned my addr 172.21.15.62:0/2534142920 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f8779d9b640 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 0x555c90790150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f877a59c640 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9078fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f877ad9d640 1 -- 172.21.15.62:0/2534142920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 msgr2=0x555c90790150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f877ad9d640 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 0x555c90790150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f877ad9d640 1 -- 172.21.15.62:0/2534142920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 msgr2=0x555c9078fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f877ad9d640 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9078fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f877ad9d640 1 -- 172.21.15.62:0/2534142920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c9045e6a0 con 0x555c90792d40 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f877ad9d640 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90790690 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8774003c20 tx=0x7f8774004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77eb76e427e90ece server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f8761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f8761ffb640 1 -- 172.21.15.62:0/2534142920 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774025030 con 0x555c90792d40 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f8761ffb640 1 -- 172.21.15.62:0/2534142920 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f877401b070 con 0x555c90792d40 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f8782d30740 1 -- 172.21.15.62:0/2534142920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f876c009de0 con 0x555c90792d40 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.216+0000 7f8761ffb640 1 -- 172.21.15.62:0/2534142920 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8774032050 con 0x555c90792d40 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8782d30740 10 client.16764 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:31.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8782d30740 1 -- 172.21.15.62:0/2534142920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555c907a1560 con 0x555c90792d40 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8761ffb640 1 -- 172.21.15.62:0/2534142920 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f876c009de0 con 0x555c90792d40 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8761ffb640 1 -- 172.21.15.62:0/2534142920 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f877401e050 con 0x555c90792d40 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8782d30740 10 client.16764 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8782d30740 10 client.16764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8782d30740 20 client.16764 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8782d30740 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c907a2970 0x555c907a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8782d30740 4 client.16764 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f8782d30740 1 -- 172.21.15.62:0/2534142920 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c906fa7f0 con 0x555c907a2970 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.217+0000 7f877a59c640 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c907a2970 0x555c907a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.218+0000 7f8761ffb640 10 client.16764 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.218+0000 7f877a59c640 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c907a2970 0x555c907a4dd0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8e7bd3068cf3bf7 server_cookie=d9d01968732b419e in_seq=0 out_seq=0 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.218+0000 7f8761ffb640 1 -- 172.21.15.62:0/2534142920 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555c906fa7f0 con 0x555c907a2970 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.218+0000 7f8761ffb640 10 client.16764 handle_command_reply: tid=0 2024-09-15T23:21:31.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.218+0000 7f8782d30740 1 client.16764 shutdown 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.218+0000 7f8782d30740 1 -- 172.21.15.62:0/2534142920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 msgr2=0x555c90790690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.218+0000 7f8782d30740 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90790690 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8774003c20 tx=0x7f8774004ab0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8782d30740 1 -- 172.21.15.62:0/2534142920 shutdown_connections 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8782d30740 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c907a2970 0x555c907a4dd0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8782d30740 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90790690 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8782d30740 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 0x555c90790150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8761ffb640 0 client.16764 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8782d30740 1 --2- 172.21.15.62:0/2534142920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9078fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8782d30740 1 -- 172.21.15.62:0/2534142920 >> 172.21.15.62:0/2534142920 conn(0x555c907840f0 msgr2=0x555c90789e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8782d30740 1 -- 172.21.15.62:0/2534142920 shutdown_connections 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8782d30740 1 -- 172.21.15.62:0/2534142920 wait complete. 2024-09-15T23:21:31.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.219+0000 7f8782d30740 20 client.16764 trim_cache size 0 max 16384 2024-09-15T23:21:31.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.236+0000 7f8782d30740 1 Processor -- start 2024-09-15T23:21:31.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.236+0000 7f8782d30740 1 -- start start 2024-09-15T23:21:31.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.236+0000 7f8782d30740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90787b60 0x555c9078a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.236+0000 7f8782d30740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90788480 0x555c9078b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.236+0000 7f8782d30740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90791a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.236+0000 7f8782d30740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f877c1b14d0 con 0x555c90792d40 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.236+0000 7f8782d30740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f877c1aeda0 con 0x555c90788480 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.236+0000 7f8782d30740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f877c1acbf0 con 0x555c90787b60 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90791a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90791a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51792/0 (socket says 172.21.15.62:51792) 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877ad9d640 1 -- 172.21.15.62:0/2669759776 learned_addr learned my addr 172.21.15.62:0/2669759776 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f8779d9b640 1 --2- 172.21.15.62:0/2669759776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90788480 0x555c9078b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877a59c640 1 --2- 172.21.15.62:0/2669759776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90787b60 0x555c9078a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877ad9d640 1 -- 172.21.15.62:0/2669759776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90787b60 msgr2=0x555c9078a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877ad9d640 1 --2- 172.21.15.62:0/2669759776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90787b60 0x555c9078a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877ad9d640 1 -- 172.21.15.62:0/2669759776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90788480 msgr2=0x555c9078b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:31.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877ad9d640 1 --2- 172.21.15.62:0/2669759776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90788480 0x555c9078b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877ad9d640 1 -- 172.21.15.62:0/2669759776 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c90783f20 con 0x555c90792d40 2024-09-15T23:21:31.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f877ad9d640 1 --2- 172.21.15.62:0/2669759776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90791a70 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8774003bf0 tx=0x7f8774003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae18879ae094516 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:31.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f8763fff640 1 -- 172.21.15.62:0/2669759776 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774025030 con 0x555c90792d40 2024-09-15T23:21:31.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f8763fff640 1 -- 172.21.15.62:0/2669759776 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8774005bd0 con 0x555c90792d40 2024-09-15T23:21:31.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.237+0000 7f8763fff640 1 -- 172.21.15.62:0/2669759776 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774004260 con 0x555c90792d40 2024-09-15T23:21:31.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 -- 172.21.15.62:0/2669759776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 msgr2=0x555c90791a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:31.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 --2- 172.21.15.62:0/2669759776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90791a70 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8774003bf0 tx=0x7f8774003f80 comp rx=0 tx=0).stop 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 -- 172.21.15.62:0/2669759776 shutdown_connections 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 --2- 172.21.15.62:0/2669759776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90791a70 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 --2- 172.21.15.62:0/2669759776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90788480 0x555c9078b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 --2- 172.21.15.62:0/2669759776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90787b60 0x555c9078a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 -- 172.21.15.62:0/2669759776 >> 172.21.15.62:0/2669759776 conn(0x555c907840f0 msgr2=0x555c90791360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 -- 172.21.15.62:0/2669759776 shutdown_connections 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 -- 172.21.15.62:0/2669759776 wait complete. 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 -- start start 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.238+0000 7f8782d30740 1 Processor -- start 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8782d30740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9079b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8782d30740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 0x555c907966c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8782d30740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90796c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8782d30740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f877c1ad260 con 0x555c90792d40 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8782d30740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f877c1ae6d0 con 0x555c90787b60 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8782d30740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f877c1ad930 con 0x555c90788480 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8782d30740 10 client.0 _ll_register_callbacks cb 0x7f8774012470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9079b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90796c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9079b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:55268/0 (socket says 172.21.15.62:55268) 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877a59c640 1 -- 172.21.15.62:0/845550016 learned_addr learned my addr 172.21.15.62:0/845550016 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8779d9b640 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 0x555c907966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877a59c640 1 -- 172.21.15.62:0/845550016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 msgr2=0x555c907966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877a59c640 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 0x555c907966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877a59c640 1 -- 172.21.15.62:0/845550016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 msgr2=0x555c90796c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:31.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877a59c640 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90796c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877a59c640 1 -- 172.21.15.62:0/845550016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c90797400 con 0x555c90787b60 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877ad9d640 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90796c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f877a59c640 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9079b390 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f876c0058c0 tx=0x7f876c09a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de6142f00d61df9e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8761ffb640 1 -- 172.21.15.62:0/845550016 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876c020070 con 0x555c90787b60 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8761ffb640 1 -- 172.21.15.62:0/845550016 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f876c0981b0 con 0x555c90787b60 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.239+0000 7f8782d30740 1 -- 172.21.15.62:0/845550016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8774005bd0 con 0x555c90787b60 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8761ffb640 1 -- 172.21.15.62:0/845550016 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f876c0a4060 con 0x555c90787b60 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8782d30740 10 client.26366 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8782d30740 1 -- 172.21.15.62:0/845550016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555c9080d670 con 0x555c90787b60 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8761ffb640 1 -- 172.21.15.62:0/845550016 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8774005bd0 con 0x555c90787b60 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8761ffb640 1 -- 172.21.15.62:0/845550016 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f876c086110 con 0x555c90787b60 2024-09-15T23:21:31.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8782d30740 10 client.26366 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8782d30740 10 client.26366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8782d30740 20 client.26366 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8782d30740 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c9080da40 0x555c9080fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8782d30740 4 client.26366 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8782d30740 1 -- 172.21.15.62:0/845550016 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555c906f4d00 con 0x555c9080da40 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.240+0000 7f8779d9b640 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c9080da40 0x555c9080fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.241+0000 7f8779d9b640 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c9080da40 0x555c9080fe30 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1682e7e57e81d195 server_cookie=d456408383f55ace in_seq=0 out_seq=0 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.241+0000 7f8761ffb640 10 client.26366 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8761ffb640 1 -- 172.21.15.62:0/845550016 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555c906f4d00 con 0x555c9080da40 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8761ffb640 10 client.26366 handle_command_reply: tid=0 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 client.26366 shutdown 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 -- 172.21.15.62:0/845550016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 msgr2=0x555c9079b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:31.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9079b390 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f876c0058c0 tx=0x7f876c09a3f0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 -- 172.21.15.62:0/845550016 shutdown_connections 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c9080da40 0x555c9080fe30 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c90792d40 0x555c90796c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c90788480 0x555c907966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 --2- 172.21.15.62:0/845550016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c90787b60 0x555c9079b390 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 -- 172.21.15.62:0/845550016 >> 172.21.15.62:0/845550016 conn(0x555c907840f0 msgr2=0x555c90783270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8761ffb640 0 client.26366 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 -- 172.21.15.62:0/845550016 shutdown_connections 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 1 -- 172.21.15.62:0/845550016 wait complete. 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.242+0000 7f8782d30740 20 client.26366 trim_cache size 0 max 16384 2024-09-15T23:21:31.249 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:21:31.250 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:21:31.250 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "34def3c8-2740-4d8c-8cf4-d7b922968dca", 2024-09-15T23:21:31.250 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:21:31.250 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:21:31.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 -- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87640a3240 msgr2=0x7f87640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87640a3240 0x7f87640a5700 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8774005fd0 tx=0x7f8774009600 comp rx=0 tx=0).stop 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 -- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c102d40 msgr2=0x7f877c198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c102d40 0x7f877c198430 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f876c00d950 tx=0x7f876c00de10 comp rx=0 tx=0).stop 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 -- 172.21.15.62:0/267445552 shutdown_connections 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f87640a3240 0x7f87640a5700 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f877c10c170 0x7f877c19cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f877c103660 0x7f877c198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 --2- 172.21.15.62:0/267445552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c102d40 0x7f877c198430 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 -- 172.21.15.62:0/267445552 >> 172.21.15.62:0/267445552 conn(0x7f877c0fc7a0 msgr2=0x7f877c0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 -- 172.21.15.62:0/267445552 shutdown_connections 2024-09-15T23:21:31.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:31.243+0000 7f8780957640 1 -- 172.21.15.62:0/267445552 wait complete. 2024-09-15T23:21:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:31 smithi067 ceph-mon[35665]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.3 KiB/s wr, 11 op/s 2024-09-15T23:21:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:31 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:21:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:31 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:31 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:21:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:31 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:31.525 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:21:31 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: 2024-09-15T23:21:31.258+0000 7ff293927640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000f [...2,head] /volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans/ auth{2=1} v1983 ap=3 f(v4 m2024-09-15T23:20:21.776008+0000 6=6+0) n(v16 rc2024-09-15T23:21:11.944219+0000 b1515520 7=6+1) (inest sync->mix g=2) (ipolicy sync r=1) (iquiesce lock x=1 by request(mds.0:1524 nref=3)) | dirtyscattered=0 request=1 lock=2 dirfrag=1 caps=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55598f2f4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v282>,<0x10000000005/data v121>,<0x100000001fb/postgres v25>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v43>,<0x10000000001/sv_1 v57>,<0x10000000000/qa v44>,<0x1/volumes v43>]//[]","memoryvalue":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v1983>,<0x10000000005/data v411>,<0x100000001fb/postgres v252>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v282>,<0x10000000001/sv_1 v280>,<0x10000000000/qa v286>,<0x1/volumes v264>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-09-15T23:19:34.177764+0000 5=5+0)","ondisk_value.rstat":"n(v0 rc2024-09-15T23:21:11.944219+0000 b1515520 7=6+1)","memory_value.dirstat":"f(v4 m2024-09-15T23:20:21.776008+0000 6=6+0)","memory_value.rstat":"n(v16 rc2024-09-15T23:21:11.944219+0000 b1515520 7=6+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-09-15T23:21:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:31 smithi062 ceph-mon[30949]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.3 KiB/s wr, 11 op/s 2024-09-15T23:21:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:31 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:21:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:31 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:31 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:21:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:31 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:31 smithi148 ceph-mon[35610]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 7.3 KiB/s wr, 11 op/s 2024-09-15T23:21:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:31 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:21:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:31 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:31 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:21:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:31 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:32.023 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:21:32.278 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:32.725 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:32 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:21:32.725 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:32 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:32.725 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:32 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:21:32.725 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:32 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:21:32.726 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:32 smithi062 ceph-mon[30949]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:21:32.726 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:32 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:32.726 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:32 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:32.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.722+0000 7f66ee916640 1 -- 172.21.15.62:0/900693458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8101420 msgr2=0x7f66e810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.722+0000 7f66ee916640 1 --2- 172.21.15.62:0/900693458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8101420 0x7f66e810d950 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009cd0 tx=0x7f66e002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.722+0000 7f66ee916640 1 -- 172.21.15.62:0/900693458 shutdown_connections 2024-09-15T23:21:32.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.722+0000 7f66ee916640 1 --2- 172.21.15.62:0/900693458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8101420 0x7f66e810d950 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.722+0000 7f66ee916640 1 --2- 172.21.15.62:0/900693458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e8100a80 0x7f66e8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.722+0000 7f66ee916640 1 --2- 172.21.15.62:0/900693458 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66e8106a80 0x7f66e8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.722+0000 7f66ee916640 1 -- 172.21.15.62:0/900693458 >> 172.21.15.62:0/900693458 conn(0x7f66e80fc7a0 msgr2=0x7f66e80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:32.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.723+0000 7f66ee916640 1 -- 172.21.15.62:0/900693458 shutdown_connections 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.723+0000 7f66ee916640 1 -- 172.21.15.62:0/900693458 wait complete. 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.724+0000 7f66ee916640 1 Processor -- start 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.724+0000 7f66ee916640 1 -- start start 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.724+0000 7f66ee916640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66e8100a80 0x7f66e8199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.724+0000 7f66ee916640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e8101420 0x7f66e8199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.724+0000 7f66ee916640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8106a80 0x7f66e81a41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.724+0000 7f66ee916640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f66e8110ea0 con 0x7f66e8106a80 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.724+0000 7f66ee916640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66e8110d20 con 0x7f66e8101420 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.724+0000 7f66ee916640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f66e8111020 con 0x7f66e8100a80 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66eceb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8106a80 0x7f66e81a41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66eceb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8106a80 0x7f66e81a41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57632/0 (socket says 172.21.15.62:57632) 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66eceb3640 1 -- 172.21.15.62:0/1384059672 learned_addr learned my addr 172.21.15.62:0/1384059672 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:32.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66e77fe640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e8101420 0x7f66e8199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66e7fff640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66e8100a80 0x7f66e8199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66e77fe640 1 -- 172.21.15.62:0/1384059672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66e8100a80 msgr2=0x7f66e8199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66e77fe640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66e8100a80 0x7f66e8199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66e77fe640 1 -- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8106a80 msgr2=0x7f66e81a41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66e77fe640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8106a80 0x7f66e81a41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66e77fe640 1 -- 172.21.15.62:0/1384059672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e81a4710 con 0x7f66e8101420 2024-09-15T23:21:32.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66eceb3640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8106a80 0x7f66e81a41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:32.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.725+0000 7f66e77fe640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e8101420 0x7f66e8199690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00d6d0 tx=0x7f66dc00dad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:32.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.726+0000 7f66e57fa640 1 -- 172.21.15.62:0/1384059672 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc017070 con 0x7f66e8101420 2024-09-15T23:21:32.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.726+0000 7f66ee916640 1 -- 172.21.15.62:0/1384059672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e81a48e0 con 0x7f66e8101420 2024-09-15T23:21:32.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.726+0000 7f66ee916640 1 -- 172.21.15.62:0/1384059672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e81a4e20 con 0x7f66e8101420 2024-09-15T23:21:32.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.726+0000 7f66e57fa640 1 -- 172.21.15.62:0/1384059672 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66dc0047a0 con 0x7f66e8101420 2024-09-15T23:21:32.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.726+0000 7f66e57fa640 1 -- 172.21.15.62:0/1384059672 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc020060 con 0x7f66e8101420 2024-09-15T23:21:32.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.728+0000 7f66e57fa640 1 -- 172.21.15.62:0/1384059672 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f66dc009b90 con 0x7f66e8101420 2024-09-15T23:21:32.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.728+0000 7f66f0cef740 1 Processor -- start 2024-09-15T23:21:32.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.728+0000 7f66f0cef740 1 -- start start 2024-09-15T23:21:32.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a911880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a91c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a91ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66f0cef740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a45a8772e0 con 0x55a45a9114a0 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66f0cef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a45a81ece0 con 0x55a45a91c680 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66f0cef740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a45a8732a0 con 0x55a45a911dc0 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66eceb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a911880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66eceb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a911880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57636/0 (socket says 172.21.15.62:57636) 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66eceb3640 1 -- 172.21.15.62:0/2547390286 learned_addr learned my addr 172.21.15.62:0/2547390286 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66e77fe640 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a91ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66e7fff640 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a91c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66e57fa640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66d00a3240 0x7f66d00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66e77fe640 1 -- 172.21.15.62:0/2547390286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 msgr2=0x55a45a91c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66e77fe640 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a91c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66e77fe640 1 -- 172.21.15.62:0/2547390286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 msgr2=0x55a45a911880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:32.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66e77fe640 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a911880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66e77fe640 1 -- 172.21.15.62:0/2547390286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a45a803440 con 0x55a45a91c680 2024-09-15T23:21:32.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66eceb3640 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a911880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:32.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.729+0000 7f66eceb3640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66d00a3240 0x7f66d00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.730+0000 7f66e57fa640 1 -- 172.21.15.62:0/1384059672 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f66dc0cf4a0 con 0x7f66e8101420 2024-09-15T23:21:32.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.730+0000 7f66e77fe640 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a91ea70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0ccb90 tx=0x7f66dc066120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eca1b5f726e7b66 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:32.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.730+0000 7f66ce7fc640 1 -- 172.21.15.62:0/2547390286 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc09b220 con 0x55a45a91c680 2024-09-15T23:21:32.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.730+0000 7f66ce7fc640 1 -- 172.21.15.62:0/2547390286 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66dc09bde0 con 0x55a45a91c680 2024-09-15T23:21:32.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.730+0000 7f66ce7fc640 1 -- 172.21.15.62:0/2547390286 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc066190 con 0x55a45a91c680 2024-09-15T23:21:32.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 -- 172.21.15.62:0/2547390286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 msgr2=0x55a45a91ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66eceb3640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66d00a3240 0x7f66d00a5700 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009a60 tx=0x7f66e0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a91ea70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0ccb90 tx=0x7f66dc066120 comp rx=0 tx=0).stop 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 -- 172.21.15.62:0/2547390286 shutdown_connections 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a91ea70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a91c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2547390286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a911880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 -- 172.21.15.62:0/2547390286 >> 172.21.15.62:0/2547390286 conn(0x55a45a90da30 msgr2=0x55a45a90de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 -- 172.21.15.62:0/2547390286 shutdown_connections 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 -- 172.21.15.62:0/2547390286 wait complete. 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 -- start start 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.731+0000 7f66f0cef740 1 Processor -- start 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 0x55a45a91beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a919160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a91c680 0x55a45a9196a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66f0cef740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a45a923600 con 0x55a45a91c680 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66f0cef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a45a832050 con 0x55a45a9114a0 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66f0cef740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a45a923780 con 0x55a45a911dc0 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66f0cef740 10 client.0 _ll_register_callbacks cb 0x55a45a872160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66eceb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a91c680 0x55a45a9196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66eceb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a91c680 0x55a45a9196a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57644/0 (socket says 172.21.15.62:57644) 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66eceb3640 1 -- 172.21.15.62:0/2199168724 learned_addr learned my addr 172.21.15.62:0/2199168724 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66e7fff640 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 0x55a45a91beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66e77fe640 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a919160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66eceb3640 1 -- 172.21.15.62:0/2199168724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 msgr2=0x55a45a919160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66eceb3640 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a919160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66eceb3640 1 -- 172.21.15.62:0/2199168724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 msgr2=0x55a45a91beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66eceb3640 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 0x55a45a91beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66eceb3640 1 -- 172.21.15.62:0/2199168724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a45a5e83c0 con 0x55a45a91c680 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66eceb3640 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a91c680 0x55a45a9196a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f66e0005cc0 tx=0x7f66e002a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58316e28c9e52f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66c77fe640 1 -- 172.21.15.62:0/2199168724 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0030870 con 0x55a45a91c680 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66c77fe640 1 -- 172.21.15.62:0/2199168724 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f66e0030a10 con 0x55a45a91c680 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.732+0000 7f66f0cef740 1 -- 172.21.15.62:0/2199168724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66dc09bde0 con 0x55a45a91c680 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66c77fe640 1 -- 172.21.15.62:0/2199168724 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f66e0030c10 con 0x55a45a91c680 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66f0cef740 10 client.16782 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66f0cef740 1 -- 172.21.15.62:0/2199168724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a45a92ef40 con 0x55a45a91c680 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66c77fe640 1 -- 172.21.15.62:0/2199168724 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66dc09bde0 con 0x55a45a91c680 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66c77fe640 1 -- 172.21.15.62:0/2199168724 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f66e0030070 con 0x55a45a91c680 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66f0cef740 10 client.16782 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:32.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66f0cef740 10 client.16782 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66f0cef740 20 client.16782 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a45a930450 0x55a45a932890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66f0cef740 4 client.16782 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66f0cef740 1 -- 172.21.15.62:0/2199168724 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a45a884850 con 0x55a45a930450 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.733+0000 7f66e7fff640 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a45a930450 0x55a45a932890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.734+0000 7f66e7fff640 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a45a930450 0x55a45a932890 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9725dd99915a0174 server_cookie=227d4c2716309ef3 in_seq=0 out_seq=0 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.734+0000 7f66c77fe640 10 client.16782 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66c77fe640 1 -- 172.21.15.62:0/2199168724 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a45a884850 con 0x55a45a930450 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66c77fe640 10 client.16782 handle_command_reply: tid=0 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 client.16782 shutdown 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 -- 172.21.15.62:0/2199168724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a91c680 msgr2=0x55a45a9196a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a91c680 0x55a45a9196a0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f66e0005cc0 tx=0x7f66e002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 -- 172.21.15.62:0/2199168724 shutdown_connections 2024-09-15T23:21:32.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a45a930450 0x55a45a932890 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a91c680 0x55a45a9196a0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a919160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 --2- 172.21.15.62:0/2199168724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 0x55a45a91beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 -- 172.21.15.62:0/2199168724 >> 172.21.15.62:0/2199168724 conn(0x55a45a90da30 msgr2=0x55a45a913760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66c77fe640 0 client.16782 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 -- 172.21.15.62:0/2199168724 shutdown_connections 2024-09-15T23:21:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 1 -- 172.21.15.62:0/2199168724 wait complete. 2024-09-15T23:21:32.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.735+0000 7f66f0cef740 20 client.16782 trim_cache size 0 max 16384 2024-09-15T23:21:32.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.752+0000 7f66f0cef740 1 Processor -- start 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.752+0000 7f66f0cef740 1 -- start start 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a914900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a91b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a938160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66f0cef740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f66d40123e0 con 0x55a45a9114a0 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66f0cef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66d400fcb0 con 0x55a45a91c680 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66f0cef740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f66d400db00 con 0x55a45a911dc0 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a914900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a914900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57646/0 (socket says 172.21.15.62:57646) 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e7fff640 1 -- 172.21.15.62:0/4230591861 learned_addr learned my addr 172.21.15.62:0/4230591861 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66eceb3640 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a938160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e77fe640 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a91b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e7fff640 1 -- 172.21.15.62:0/4230591861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 msgr2=0x55a45a91b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e7fff640 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a91b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e7fff640 1 -- 172.21.15.62:0/4230591861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 msgr2=0x55a45a938160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e7fff640 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a938160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e7fff640 1 -- 172.21.15.62:0/4230591861 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a45a90d860 con 0x55a45a9114a0 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66eceb3640 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a938160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66e7fff640 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a914900 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009a10 tx=0x7f66d8009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13133540d7cdfd02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.753+0000 7f66cdffb640 1 -- 172.21.15.62:0/4230591861 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8025070 con 0x55a45a9114a0 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66cdffb640 1 -- 172.21.15.62:0/4230591861 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66d801e480 con 0x55a45a9114a0 2024-09-15T23:21:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66cdffb640 1 -- 172.21.15.62:0/4230591861 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8014650 con 0x55a45a9114a0 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66f0cef740 1 -- 172.21.15.62:0/4230591861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 msgr2=0x55a45a914900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66f0cef740 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a914900 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009a10 tx=0x7f66d8009110 comp rx=0 tx=0).stop 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66f0cef740 1 -- 172.21.15.62:0/4230591861 shutdown_connections 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66f0cef740 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a91c680 0x55a45a938160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66f0cef740 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a911dc0 0x55a45a91b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66f0cef740 1 --2- 172.21.15.62:0/4230591861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a9114a0 0x55a45a914900 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66f0cef740 1 -- 172.21.15.62:0/4230591861 >> 172.21.15.62:0/4230591861 conn(0x55a45a90da30 msgr2=0x55a45a91a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66f0cef740 1 -- 172.21.15.62:0/4230591861 shutdown_connections 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.754+0000 7f66f0cef740 1 -- 172.21.15.62:0/4230591861 wait complete. 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66f0cef740 1 -- start start 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66f0cef740 1 Processor -- start 2024-09-15T23:21:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 0x55a45a91d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a911dc0 0x55a45a91dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66f0cef740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a91c680 0x55a45a91e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66f0cef740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f66d400e170 con 0x55a45a911dc0 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66f0cef740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66d400f5e0 con 0x55a45a9114a0 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66f0cef740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f66d400e840 con 0x55a45a91c680 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66f0cef740 10 client.0 _ll_register_callbacks cb 0x7f66e0009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66e77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a911dc0 0x55a45a91dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66e77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a911dc0 0x55a45a91dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57652/0 (socket says 172.21.15.62:57652) 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66e77fe640 1 -- 172.21.15.62:0/1873701802 learned_addr learned my addr 172.21.15.62:0/1873701802 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66e7fff640 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 0x55a45a91d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.755+0000 7f66eceb3640 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a91c680 0x55a45a91e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66e77fe640 1 -- 172.21.15.62:0/1873701802 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a91c680 msgr2=0x55a45a91e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66e77fe640 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a91c680 0x55a45a91e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66e77fe640 1 -- 172.21.15.62:0/1873701802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 msgr2=0x55a45a91d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66e77fe640 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 0x55a45a91d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66e77fe640 1 -- 172.21.15.62:0/1873701802 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a45a9270a0 con 0x55a45a911dc0 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66e77fe640 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a911dc0 0x55a45a91dd90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f66dc09b670 tx=0x7f66dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cfe2e4a1329f0d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66c77fe640 1 -- 172.21.15.62:0/1873701802 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc0671b0 con 0x55a45a911dc0 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66c77fe640 1 -- 172.21.15.62:0/1873701802 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f66dc0cf160 con 0x55a45a911dc0 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66f0cef740 1 -- 172.21.15.62:0/1873701802 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66d801e480 con 0x55a45a911dc0 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66c77fe640 1 -- 172.21.15.62:0/1873701802 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f66dc09b040 con 0x55a45a911dc0 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66f0cef740 10 client.16788 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66f0cef740 1 -- 172.21.15.62:0/1873701802 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55a45a91efb0 con 0x55a45a911dc0 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.756+0000 7f66c77fe640 1 -- 172.21.15.62:0/1873701802 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66d801e480 con 0x55a45a911dc0 2024-09-15T23:21:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66c77fe640 1 -- 172.21.15.62:0/1873701802 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f66dc0ccc40 con 0x55a45a911dc0 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66f0cef740 10 client.16788 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66f0cef740 10 client.16788 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66f0cef740 20 client.16788 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66f0cef740 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a45a91f400 0x55a45a9217f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66f0cef740 4 client.16788 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66f0cef740 1 -- 172.21.15.62:0/1873701802 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a45a87ed60 con 0x55a45a91f400 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66e7fff640 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a45a91f400 0x55a45a9217f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66e7fff640 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a45a91f400 0x55a45a9217f0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30119b51e792904f server_cookie=3607e4ba28544822 in_seq=0 out_seq=0 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.757+0000 7f66c77fe640 10 client.16788 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66c77fe640 1 -- 172.21.15.62:0/1873701802 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a45a87ed60 con 0x55a45a91f400 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66c77fe640 10 client.16788 handle_command_reply: tid=0 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 client.16788 shutdown 2024-09-15T23:21:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 -- 172.21.15.62:0/1873701802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a911dc0 msgr2=0x55a45a91dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a911dc0 0x55a45a91dd90 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f66dc09b670 tx=0x7f66dc0046c0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 -- 172.21.15.62:0/1873701802 shutdown_connections 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55a45a91f400 0x55a45a9217f0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a45a91c680 0x55a45a91e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a45a911dc0 0x55a45a91dd90 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66c77fe640 0 client.16788 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 --2- 172.21.15.62:0/1873701802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a45a9114a0 0x55a45a91d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 -- 172.21.15.62:0/1873701802 >> 172.21.15.62:0/1873701802 conn(0x55a45a90da30 msgr2=0x55a45a912ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 -- 172.21.15.62:0/1873701802 shutdown_connections 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 1 -- 172.21.15.62:0/1873701802 wait complete. 2024-09-15T23:21:32.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.758+0000 7f66f0cef740 20 client.16788 trim_cache size 0 max 16384 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout: "34def3c8-2740-4d8c-8cf4-d7b922968dca": { 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "34def3c8-2740-4d8c-8cf4-d7b922968dca", 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:32.766 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:21:32.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 -- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66d00a3240 msgr2=0x7f66d00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66d00a3240 0x7f66d00a5700 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009a60 tx=0x7f66e0005fd0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 -- 172.21.15.62:0/1384059672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e8101420 msgr2=0x7f66e8199690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:32.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e8101420 0x7f66e8199690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00d6d0 tx=0x7f66dc00dad0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 -- 172.21.15.62:0/1384059672 shutdown_connections 2024-09-15T23:21:32.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f66d00a3240 0x7f66d00a5700 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f66e8106a80 0x7f66e81a41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66e8101420 0x7f66e8199690 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 --2- 172.21.15.62:0/1384059672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66e8100a80 0x7f66e8199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:32.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 -- 172.21.15.62:0/1384059672 >> 172.21.15.62:0/1384059672 conn(0x7f66e80fc7a0 msgr2=0x7f66e80fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:32.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 -- 172.21.15.62:0/1384059672 shutdown_connections 2024-09-15T23:21:32.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:32.759+0000 7f66ee916640 1 -- 172.21.15.62:0/1384059672 wait complete. 2024-09-15T23:21:32.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:32 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:21:32.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:32 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:32.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:32 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:21:32.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:32 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:21:32.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:32 smithi067 ceph-mon[35665]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:21:32.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:32 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:32.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:32 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:32 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:21:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:32 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:32 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:21:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:32 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:21:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:32 smithi148 ceph-mon[35610]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:21:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:32 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:32.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:32 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:21:33.466 INFO:tasks.cephfs.filesystem:scrub status for tag:34def3c8-2740-4d8c-8cf4-d7b922968dca - {'path': '/', 'tag': '34def3c8-2740-4d8c-8cf4-d7b922968dca', 'options': 'recursive,force'} 2024-09-15T23:21:33.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:33 smithi062 ceph-mon[30949]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.3 KiB/s wr, 28 op/s 2024-09-15T23:21:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:33 smithi067 ceph-mon[35665]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.3 KiB/s wr, 28 op/s 2024-09-15T23:21:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:33 smithi148 ceph-mon[35610]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.3 KiB/s wr, 28 op/s 2024-09-15T23:21:34.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 315.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:21:34.689 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:21:34.959 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.440+0000 7fd96bef4640 1 -- 172.21.15.62:0/4124026459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 msgr2=0x7fd9641067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.440+0000 7fd96bef4640 1 --2- 172.21.15.62:0/4124026459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 0x7fd9641067d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd958009900 tx=0x7fd95802a410 comp rx=0 tx=0).stop 2024-09-15T23:21:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.440+0000 7fd96bef4640 1 -- 172.21.15.62:0/4124026459 shutdown_connections 2024-09-15T23:21:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.440+0000 7fd96bef4640 1 --2- 172.21.15.62:0/4124026459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd964106f00 0x7fd96410ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.440+0000 7fd96bef4640 1 --2- 172.21.15.62:0/4124026459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 0x7fd9641067d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.440+0000 7fd96bef4640 1 --2- 172.21.15.62:0/4124026459 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd964101dd0 0x7fd9641021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.440+0000 7fd96bef4640 1 -- 172.21.15.62:0/4124026459 >> 172.21.15.62:0/4124026459 conn(0x7fd9640fd8b0 msgr2=0x7fd9640ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.440+0000 7fd96bef4640 1 -- 172.21.15.62:0/4124026459 shutdown_connections 2024-09-15T23:21:35.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.440+0000 7fd96bef4640 1 -- 172.21.15.62:0/4124026459 wait complete. 2024-09-15T23:21:35.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96bef4640 1 Processor -- start 2024-09-15T23:21:35.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96bef4640 1 -- start start 2024-09-15T23:21:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96bef4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd964101dd0 0x7fd9641a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96bef4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 0x7fd9641a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96bef4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd964106f00 0x7fd96419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96bef4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641185d0 con 0x7fd964102780 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96bef4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd964118450 con 0x7fd964101dd0 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96bef4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd964118750 con 0x7fd964106f00 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96948f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 0x7fd9641a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96948f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 0x7fd9641a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57656/0 (socket says 172.21.15.62:57656) 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd96948f640 1 -- 172.21.15.62:0/437768490 learned_addr learned my addr 172.21.15.62:0/437768490 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.441+0000 7fd969c90640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd964101dd0 0x7fd9641a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd96a491640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd964106f00 0x7fd96419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd96948f640 1 -- 172.21.15.62:0/437768490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd964106f00 msgr2=0x7fd96419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd96948f640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd964106f00 0x7fd96419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd96948f640 1 -- 172.21.15.62:0/437768490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd964101dd0 msgr2=0x7fd9641a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd96948f640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd964101dd0 0x7fd9641a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd96948f640 1 -- 172.21.15.62:0/437768490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96419f070 con 0x7fd964102780 2024-09-15T23:21:35.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd96948f640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 0x7fd9641a4c20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd958005ab0 tx=0x7fd958003bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:35.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd962ffd640 1 -- 172.21.15.62:0/437768490 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95803f070 con 0x7fd964102780 2024-09-15T23:21:35.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd962ffd640 1 -- 172.21.15.62:0/437768490 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd95803b460 con 0x7fd964102780 2024-09-15T23:21:35.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96419f300 con 0x7fd964102780 2024-09-15T23:21:35.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd962ffd640 1 -- 172.21.15.62:0/437768490 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958035030 con 0x7fd964102780 2024-09-15T23:21:35.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.442+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9641ab500 con 0x7fd964102780 2024-09-15T23:21:35.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.443+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9641021b0 con 0x7fd964102780 2024-09-15T23:21:35.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.445+0000 7fd962ffd640 1 -- 172.21.15.62:0/437768490 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd958042070 con 0x7fd964102780 2024-09-15T23:21:35.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.445+0000 7fd962ffd640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd94c0a3310 0x7fd94c0a57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:35.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.445+0000 7fd969c90640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd94c0a3310 0x7fd94c0a57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:35.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.445+0000 7fd962ffd640 1 -- 172.21.15.62:0/437768490 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd9580f0490 con 0x7fd964102780 2024-09-15T23:21:35.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.446+0000 7fd969c90640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd94c0a3310 0x7fd94c0a57d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd95c0058c0 tx=0x7fd95c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:35.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.448+0000 7fd962ffd640 1 -- 172.21.15.62:0/437768490 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd9580bce30 con 0x7fd964102780 2024-09-15T23:21:35.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.661+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9641ab790 con 0x7fd964102780 2024-09-15T23:21:35.664 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:35 smithi062 ceph-mon[30949]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 7.2 KiB/s wr, 65 op/s 2024-09-15T23:21:35.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.662+0000 7fd962ffd640 1 -- 172.21.15.62:0/437768490 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11903 (secure 0 0 0) 0x7fd9580bc730 con 0x7fd964102780 2024-09-15T23:21:35.665 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:21:35.665 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":31,"btime":"2024-09-15T23:20:13:209709+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:20:12.378892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:21:35.665 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 31 2024-09-15T23:21:35.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd94c0a3310 msgr2=0x7fd94c0a57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:35.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd94c0a3310 0x7fd94c0a57d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd95c0058c0 tx=0x7fd95c00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 msgr2=0x7fd9641a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:35.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 0x7fd9641a4c20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd958005ab0 tx=0x7fd958003bb0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 shutdown_connections 2024-09-15T23:21:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd94c0a3310 0x7fd94c0a57d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd964106f00 0x7fd96419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd964102780 0x7fd9641a4c20 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 --2- 172.21.15.62:0/437768490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd964101dd0 0x7fd9641a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 >> 172.21.15.62:0/437768490 conn(0x7fd9640fd8b0 msgr2=0x7fd9640fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 shutdown_connections 2024-09-15T23:21:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:35.664+0000 7fd96bef4640 1 -- 172.21.15.62:0/437768490 wait complete. 2024-09-15T23:21:35.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:35 smithi067 ceph-mon[35665]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 7.2 KiB/s wr, 65 op/s 2024-09-15T23:21:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:35 smithi148 ceph-mon[35610]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 7.2 KiB/s wr, 65 op/s 2024-09-15T23:21:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.271+0000 7fb040042640 1 -- 172.21.15.62:0/2541554264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 msgr2=0x7fb0381020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.271+0000 7fb040042640 1 --2- 172.21.15.62:0/2541554264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 0x7fb0381020c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fb030009ca0 tx=0x7fb03002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 -- 172.21.15.62:0/2541554264 shutdown_connections 2024-09-15T23:21:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 --2- 172.21.15.62:0/2541554264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb038102600 0x7fb038108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 --2- 172.21.15.62:0/2541554264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 0x7fb0381020c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 --2- 172.21.15.62:0/2541554264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038100a50 0x7fb038100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 -- 172.21.15.62:0/2541554264 >> 172.21.15.62:0/2541554264 conn(0x7fb0380fc1b0 msgr2=0x7fb0380fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 -- 172.21.15.62:0/2541554264 shutdown_connections 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 -- 172.21.15.62:0/2541554264 wait complete. 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 Processor -- start 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 -- start start 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb038100a50 0x7fb038195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.272+0000 7fb040042640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 0x7fb038196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb040042640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038102600 0x7fb03819d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb040042640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb03810b610 con 0x7fb038101c40 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb040042640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb03810b490 con 0x7fb038102600 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb040042640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb03810b790 con 0x7fb038100a50 2024-09-15T23:21:36.277 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03d5dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 0x7fb038196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03ddde640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb038100a50 0x7fb038195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03d5dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 0x7fb038196100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57674/0 (socket says 172.21.15.62:57674) 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03d5dd640 1 -- 172.21.15.62:0/2703820585 learned_addr learned my addr 172.21.15.62:0/2703820585 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03e5df640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038102600 0x7fb03819d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03ddde640 1 -- 172.21.15.62:0/2703820585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038102600 msgr2=0x7fb03819d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03ddde640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038102600 0x7fb03819d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03ddde640 1 -- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 msgr2=0x7fb038196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03ddde640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 0x7fb038196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03ddde640 1 -- 172.21.15.62:0/2703820585 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03819d7e0 con 0x7fb038100a50 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 0x7fb038196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb03ddde640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb038100a50 0x7fb038195bc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb034010aa0 tx=0x7fb034010f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.273+0000 7fb02affd640 1 -- 172.21.15.62:0/2703820585 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03401a030 con 0x7fb038100a50 2024-09-15T23:21:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.274+0000 7fb040042640 1 -- 172.21.15.62:0/2703820585 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb03819dad0 con 0x7fb038100a50 2024-09-15T23:21:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.274+0000 7fb02affd640 1 -- 172.21.15.62:0/2703820585 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb03400e940 con 0x7fb038100a50 2024-09-15T23:21:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.274+0000 7fb02affd640 1 -- 172.21.15.62:0/2703820585 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034003be0 con 0x7fb038100a50 2024-09-15T23:21:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.274+0000 7fb040042640 1 -- 172.21.15.62:0/2703820585 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb03819e020 con 0x7fb038100a50 2024-09-15T23:21:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.276+0000 7fb042dd2740 1 Processor -- start 2024-09-15T23:21:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.276+0000 7fb042dd2740 1 -- start start 2024-09-15T23:21:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.276+0000 7fb02affd640 1 -- 172.21.15.62:0/2703820585 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb03401d070 con 0x7fb038100a50 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.276+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef4fd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.276+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef5021b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.276+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef502780 0x55d9ef504c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.276+0000 7fb042dd2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9ef45c2f0 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.276+0000 7fb042dd2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d9ef404270 con 0x55d9ef4fd6f0 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.276+0000 7fb042dd2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d9ef4582b0 con 0x55d9ef502780 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03e5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef4fd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03e5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef4fd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57688/0 (socket says 172.21.15.62:57688) 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03e5df640 1 -- 172.21.15.62:0/3504517811 learned_addr learned my addr 172.21.15.62:0/3504517811 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef502780 0x55d9ef504c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03ddde640 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef5021b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03e5df640 1 -- 172.21.15.62:0/3504517811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef502780 msgr2=0x55d9ef504c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03e5df640 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef502780 0x55d9ef504c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03e5df640 1 -- 172.21.15.62:0/3504517811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 msgr2=0x55d9ef5021b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03e5df640 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef5021b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03e5df640 1 -- 172.21.15.62:0/3504517811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9ef403340 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03ddde640 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef5021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03e5df640 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef4fd1b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009ca0 tx=0x7fb02c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc047975a37652c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb02affd640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0240a32f0 0x7fb0240a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb023fff640 1 -- 172.21.15.62:0/3504517811 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c01e070 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb023fff640 1 -- 172.21.15.62:0/3504517811 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb02c009e50 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb023fff640 1 -- 172.21.15.62:0/3504517811 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c00eac0 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.277+0000 7fb03ddde640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0240a32f0 0x7fb0240a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb02affd640 1 -- 172.21.15.62:0/2703820585 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb0340cf920 con 0x7fb038100a50 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb042dd2740 1 -- 172.21.15.62:0/3504517811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 msgr2=0x55d9ef4fd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb03ddde640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0240a32f0 0x7fb0240a57b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb0340cd410 tx=0x7fb0340035f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:36.287 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb042dd2740 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef4fd1b0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009ca0 tx=0x7fb02c011810 comp rx=0 tx=0).stop 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb042dd2740 1 -- 172.21.15.62:0/3504517811 shutdown_connections 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb042dd2740 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef502780 0x55d9ef504c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb042dd2740 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef5021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb042dd2740 1 --2- 172.21.15.62:0/3504517811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef4fd1b0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb042dd2740 1 -- 172.21.15.62:0/3504517811 >> 172.21.15.62:0/3504517811 conn(0x55d9ef4f8420 msgr2=0x55d9ef4f8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb042dd2740 1 -- 172.21.15.62:0/3504517811 shutdown_connections 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.278+0000 7fb042dd2740 1 -- 172.21.15.62:0/3504517811 wait complete. 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb042dd2740 1 -- start start 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb042dd2740 1 Processor -- start 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef507000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef507540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 0x55d9ef50e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb042dd2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9ef505750 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.288 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb042dd2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d9ef3e5290 con 0x55d9ef502780 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb042dd2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d9ef5058d0 con 0x55d9ef4fd6f0 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb042dd2740 10 client.0 _ll_register_callbacks cb 0x55d9ef416140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb03e5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef507000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb03e5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef507000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57704/0 (socket says 172.21.15.62:57704) 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb03e5df640 1 -- 172.21.15.62:0/1930248338 learned_addr learned my addr 172.21.15.62:0/1930248338 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.279+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef507540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb03e5df640 1 -- 172.21.15.62:0/1930248338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fd6f0 msgr2=0x55d9ef507540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb03e5df640 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef507540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb03e5df640 1 -- 172.21.15.62:0/1930248338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 msgr2=0x55d9ef50e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb03e5df640 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 0x55d9ef50e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb03e5df640 1 -- 172.21.15.62:0/1930248338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9ef16f210 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef507540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:36.289 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb03e5df640 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef507000 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009c70 tx=0x7fb02c009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5058d064ac51c89 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb0217fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb0217fa640 1 -- 172.21.15.62:0/1930248338 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c028030 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb0217fa640 1 -- 172.21.15.62:0/1930248338 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb02c00ea30 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb042dd2740 1 -- 172.21.15.62:0/1930248338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb02c009e50 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb0217fa640 1 -- 172.21.15.62:0/1930248338 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb02c032050 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb042dd2740 10 client.16806 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb042dd2740 1 -- 172.21.15.62:0/1930248338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d9ef23b620 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.280+0000 7fb0217fa640 1 -- 172.21.15.62:0/1930248338 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb02c009e50 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb0217fa640 1 -- 172.21.15.62:0/1930248338 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb02c019430 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb042dd2740 10 client.16806 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb042dd2740 10 client.16806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb042dd2740 20 client.16806 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9ef5129c0 0x55d9ef514de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb042dd2740 4 client.16806 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:36.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb042dd2740 1 -- 172.21.15.62:0/1930248338 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9ef469a20 con 0x55d9ef5129c0 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9ef5129c0 0x55d9ef514de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb0217fa640 10 client.16806 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.281+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9ef5129c0 0x55d9ef514de0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6bce2625a755524 server_cookie=953ad0b6bbd05e84 in_seq=0 out_seq=0 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.282+0000 7fb0217fa640 1 -- 172.21.15.62:0/1930248338 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d9ef469a20 con 0x55d9ef5129c0 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.282+0000 7fb0217fa640 10 client.16806 handle_command_reply: tid=0 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.282+0000 7fb042dd2740 1 client.16806 shutdown 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.282+0000 7fb042dd2740 1 -- 172.21.15.62:0/1930248338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 msgr2=0x55d9ef507000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.282+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef507000 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009c70 tx=0x7fb02c009930 comp rx=0 tx=0).stop 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb042dd2740 1 -- 172.21.15.62:0/1930248338 shutdown_connections 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9ef5129c0 0x55d9ef514de0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 0x55d9ef50e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef507540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1930248338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef507000 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb0217fa640 0 client.16806 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb042dd2740 1 -- 172.21.15.62:0/1930248338 >> 172.21.15.62:0/1930248338 conn(0x55d9ef4f8420 msgr2=0x55d9ef4f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb042dd2740 1 -- 172.21.15.62:0/1930248338 shutdown_connections 2024-09-15T23:21:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb042dd2740 1 -- 172.21.15.62:0/1930248338 wait complete. 2024-09-15T23:21:36.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.283+0000 7fb042dd2740 20 client.16806 trim_cache size 0 max 16384 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb042dd2740 1 Processor -- start 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb042dd2740 1 -- start start 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef5075d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 0x55d9ef4f99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb042dd2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb0381aac90 con 0x55d9ef4fd6f0 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb042dd2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb0381a8560 con 0x55d9ef502780 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb042dd2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb0381a63b0 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb03d5dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb03d5dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57718/0 (socket says 172.21.15.62:57718) 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb03d5dd640 1 -- 172.21.15.62:0/1658808484 learned_addr learned my addr 172.21.15.62:0/1658808484 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:36.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb03e5df640 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef5075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb03d5dd640 1 -- 172.21.15.62:0/1658808484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 msgr2=0x55d9ef5075d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.297+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef5075d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb03d5dd640 1 -- 172.21.15.62:0/1658808484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 msgr2=0x55d9ef4f99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb03e5df640 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 0x55d9ef4f99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 0x55d9ef4f99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb03d5dd640 1 -- 172.21.15.62:0/1658808484 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9ef505630 con 0x55d9ef4fd6f0 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb03e5df640 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 0x55d9ef4f99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50e4a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb030031ae0 tx=0x7fb030004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4fb09e746add25 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb0237fe640 1 -- 172.21.15.62:0/1658808484 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03002f610 con 0x55d9ef4fd6f0 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb0237fe640 1 -- 172.21.15.62:0/1658808484 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb030036040 con 0x55d9ef4fd6f0 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb0237fe640 1 -- 172.21.15.62:0/1658808484 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb030030bb0 con 0x55d9ef4fd6f0 2024-09-15T23:21:36.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb042dd2740 1 -- 172.21.15.62:0/1658808484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fd6f0 msgr2=0x55d9ef50e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.298+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50e4a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb030031ae0 tx=0x7fb030004a40 comp rx=0 tx=0).stop 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 -- 172.21.15.62:0/1658808484 shutdown_connections 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef502780 0x55d9ef4f99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50e4a0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 --2- 172.21.15.62:0/1658808484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef5075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 -- 172.21.15.62:0/1658808484 >> 172.21.15.62:0/1658808484 conn(0x55d9ef4f8420 msgr2=0x55d9ef4f92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 -- 172.21.15.62:0/1658808484 shutdown_connections 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 -- 172.21.15.62:0/1658808484 wait complete. 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 -- start start 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 Processor -- start 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef50bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef502780 0x55d9ef50d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb0381a6a20 con 0x55d9ef502780 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb0381a7e90 con 0x55d9ef4fd6f0 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb0381a70f0 con 0x55d9ef4fcdb0 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.299+0000 7fb042dd2740 10 client.0 _ll_register_callbacks cb 0x7fb02c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef502780 0x55d9ef50d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef502780 0x55d9ef50d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57722/0 (socket says 172.21.15.62:57722) 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 -- 172.21.15.62:0/932437641 learned_addr learned my addr 172.21.15.62:0/932437641 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef50bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 -- 172.21.15.62:0/932437641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 msgr2=0x55d9ef50bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef50bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 -- 172.21.15.62:0/932437641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 msgr2=0x55d9ef50c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 -- 172.21.15.62:0/932437641 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9ef50d7f0 con 0x55d9ef502780 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb03e5df640 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef502780 0x55d9ef50d1e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb02c006010 tx=0x7fb02c009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3b1d52df1ac03c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb0217fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb0217fa640 1 -- 172.21.15.62:0/932437641 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c028040 con 0x55d9ef502780 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb0217fa640 1 -- 172.21.15.62:0/932437641 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb02c0053f0 con 0x55d9ef502780 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb042dd2740 1 -- 172.21.15.62:0/932437641 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb030036040 con 0x55d9ef502780 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb0217fa640 1 -- 172.21.15.62:0/932437641 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb02c01e040 con 0x55d9ef502780 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb042dd2740 10 client.16812 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.300+0000 7fb042dd2740 1 -- 172.21.15.62:0/932437641 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d9ef57d7b0 con 0x55d9ef502780 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.301+0000 7fb0217fa640 1 -- 172.21.15.62:0/932437641 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb030036040 con 0x55d9ef502780 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.301+0000 7fb0217fa640 1 -- 172.21.15.62:0/932437641 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7fb02c0035f0 con 0x55d9ef502780 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.301+0000 7fb042dd2740 10 client.16812 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.301+0000 7fb042dd2740 10 client.16812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.301+0000 7fb042dd2740 20 client.16812 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.301+0000 7fb042dd2740 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9ef57e130 0x55d9ef580550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.301+0000 7fb042dd2740 4 client.16812 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:21:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.301+0000 7fb042dd2740 1 -- 172.21.15.62:0/932437641 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d9ef463d70 con 0x55d9ef57e130 2024-09-15T23:21:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.301+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9ef57e130 0x55d9ef580550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.302+0000 7fb03d5dd640 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9ef57e130 0x55d9ef580550 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4375eba757addf11 server_cookie=192bfc3cc8ca7c1b in_seq=0 out_seq=0 2024-09-15T23:21:36.309 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.302+0000 7fb0217fa640 10 client.16812 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb0217fa640 1 -- 172.21.15.62:0/932437641 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d9ef463d70 con 0x55d9ef57e130 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb0217fa640 10 client.16812 handle_command_reply: tid=0 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 client.16812 shutdown 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 -- 172.21.15.62:0/932437641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef502780 msgr2=0x55d9ef50d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef502780 0x55d9ef50d1e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb02c006010 tx=0x7fb02c009930 comp rx=0 tx=0).stop 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 -- 172.21.15.62:0/932437641 shutdown_connections 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9ef57e130 0x55d9ef580550 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9ef502780 0x55d9ef50d1e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9ef4fd6f0 0x55d9ef50c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb0217fa640 0 client.16812 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 --2- 172.21.15.62:0/932437641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9ef4fcdb0 0x55d9ef50bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 -- 172.21.15.62:0/932437641 >> 172.21.15.62:0/932437641 conn(0x55d9ef4f8420 msgr2=0x55d9ef4f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:36.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 -- 172.21.15.62:0/932437641 shutdown_connections 2024-09-15T23:21:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.312+0000 7fb042dd2740 1 -- 172.21.15.62:0/932437641 wait complete. 2024-09-15T23:21:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb042dd2740 20 client.16812 trim_cache size 0 max 16384 2024-09-15T23:21:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:21:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:21:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:21:36.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:21:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 -- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0240a32f0 msgr2=0x7fb0240a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0240a32f0 0x7fb0240a57b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb0340cd410 tx=0x7fb0340035f0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 -- 172.21.15.62:0/2703820585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb038100a50 msgr2=0x7fb038195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb038100a50 0x7fb038195bc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb034010aa0 tx=0x7fb034010f60 comp rx=0 tx=0).stop 2024-09-15T23:21:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 -- 172.21.15.62:0/2703820585 shutdown_connections 2024-09-15T23:21:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0240a32f0 0x7fb0240a57b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb038102600 0x7fb03819d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb038101c40 0x7fb038196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 --2- 172.21.15.62:0/2703820585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb038100a50 0x7fb038195bc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 -- 172.21.15.62:0/2703820585 >> 172.21.15.62:0/2703820585 conn(0x7fb0380fc1b0 msgr2=0x7fb038104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 -- 172.21.15.62:0/2703820585 shutdown_connections 2024-09-15T23:21:36.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:36.313+0000 7fb040042640 1 -- 172.21.15.62:0/2703820585 wait complete. 2024-09-15T23:21:36.514 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 9b36dcb4 --release --await' 2024-09-15T23:21:36.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 9b36dcb4 --release --await 2024-09-15T23:21:36.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:36 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/437768490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:36.770 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:21:36.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:36 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/437768490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:36.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:36 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/437768490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:21:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 -- 172.21.15.62:0/85464872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b781027a0 msgr2=0x7f6b781067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/85464872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b781027a0 0x7f6b781067f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009900 tx=0x7f6b6c02a3f0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 -- 172.21.15.62:0/85464872 shutdown_connections 2024-09-15T23:21:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/85464872 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b78106f20 0x7f6b7810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/85464872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b781027a0 0x7f6b781067f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/85464872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b78101df0 0x7f6b781021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 -- 172.21.15.62:0/85464872 >> 172.21.15.62:0/85464872 conn(0x7f6b780fd8f0 msgr2=0x7f6b780ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 -- 172.21.15.62:0/85464872 shutdown_connections 2024-09-15T23:21:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 -- 172.21.15.62:0/85464872 wait complete. 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 Processor -- start 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.246+0000 7f6b7dcad640 1 -- start start 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b7dcad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78101df0 0x7f6b780744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b7dcad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b781027a0 0x7f6b78074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b7dcad640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b78106f20 0x7f6b78074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b7dcad640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6b78111fd0 con 0x7f6b78101df0 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b7dcad640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b78111e50 con 0x7f6b78106f20 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b7dcad640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6b78112150 con 0x7f6b781027a0 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b777fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78101df0 0x7f6b780744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b777fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78101df0 0x7f6b780744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57742/0 (socket says 172.21.15.62:57742) 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b777fe640 1 -- 172.21.15.62:0/737594866 learned_addr learned my addr 172.21.15.62:0/737594866 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:37.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b77fff640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b78106f20 0x7f6b78074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b781027a0 0x7f6b78074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b777fe640 1 -- 172.21.15.62:0/737594866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b781027a0 msgr2=0x7f6b78074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b777fe640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b781027a0 0x7f6b78074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b777fe640 1 -- 172.21.15.62:0/737594866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b78106f20 msgr2=0x7f6b78074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b777fe640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b78106f20 0x7f6b78074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b777fe640 1 -- 172.21.15.62:0/737594866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b781a6850 con 0x7f6b78101df0 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b781027a0 0x7f6b78074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.247+0000 7f6b777fe640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78101df0 0x7f6b780744f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c920 tx=0x7f6b6800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.248+0000 7f6b74ff9640 1 -- 172.21.15.62:0/737594866 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68007c70 con 0x7f6b78101df0 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.248+0000 7f6b74ff9640 1 -- 172.21.15.62:0/737594866 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b68007e10 con 0x7f6b78101df0 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.248+0000 7f6b74ff9640 1 -- 172.21.15.62:0/737594866 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68018890 con 0x7f6b78101df0 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.248+0000 7f6b7dcad640 1 -- 172.21.15.62:0/737594866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b781a6b40 con 0x7f6b78101df0 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.248+0000 7f6b7dcad640 1 -- 172.21.15.62:0/737594866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b781a70a0 con 0x7f6b78101df0 2024-09-15T23:21:37.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.250+0000 7f6b74ff9640 1 -- 172.21.15.62:0/737594866 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f6b68018a70 con 0x7f6b78101df0 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.250+0000 7f6b80086740 1 Processor -- start 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.250+0000 7f6b80086740 1 -- start start 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.250+0000 7f6b80086740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d0a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.250+0000 7f6b80086740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 0x562d19d14d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.250+0000 7f6b80086740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d15240 0x562d19d17630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.250+0000 7f6b80086740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562d19c6f620 con 0x562d19d0a060 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.250+0000 7f6b80086740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562d19c16e50 con 0x562d19d15240 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.250+0000 7f6b80086740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562d19c6b5e0 con 0x562d19d0a980 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d0a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d0a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57746/0 (socket says 172.21.15.62:57746) 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b77fff640 1 -- 172.21.15.62:0/2432508764 learned_addr learned my addr 172.21.15.62:0/2432508764 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b777fe640 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 0x562d19d14d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d15240 0x562d19d17630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b77fff640 1 -- 172.21.15.62:0/2432508764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 msgr2=0x562d19d14d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 0x562d19d14d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b77fff640 1 -- 172.21.15.62:0/2432508764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d15240 msgr2=0x562d19d17630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d15240 0x562d19d17630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b77fff640 1 -- 172.21.15.62:0/2432508764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d19c07d40 con 0x562d19d0a060 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d15240 0x562d19d17630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b777fe640 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 0x562d19d14d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b74ff9640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b600a32c0 0x7f6b600a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b777fe640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b600a32c0 0x7f6b600a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d0a440 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009f70 tx=0x7f6b70010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2105f4279e2c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b74ff9640 1 -- 172.21.15.62:0/737594866 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6b680cfd50 con 0x7f6b78101df0 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b5dffb640 1 -- 172.21.15.62:0/2432508764 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70016070 con 0x562d19d0a060 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b5dffb640 1 -- 172.21.15.62:0/2432508764 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b70004d20 con 0x562d19d0a060 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.251+0000 7f6b5dffb640 1 -- 172.21.15.62:0/2432508764 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70012950 con 0x562d19d0a060 2024-09-15T23:21:37.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b777fe640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b600a32c0 0x7f6b600a5780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6b680cd840 tx=0x7f6b68000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 -- 172.21.15.62:0/2432508764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 msgr2=0x562d19d0a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d0a440 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009f70 tx=0x7f6b70010800 comp rx=0 tx=0).stop 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 -- 172.21.15.62:0/2432508764 shutdown_connections 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d15240 0x562d19d17630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 0x562d19d14d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 --2- 172.21.15.62:0/2432508764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d0a440 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 -- 172.21.15.62:0/2432508764 >> 172.21.15.62:0/2432508764 conn(0x562d19d065f0 msgr2=0x562d19d069d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 -- 172.21.15.62:0/2432508764 shutdown_connections 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 -- 172.21.15.62:0/2432508764 wait complete. 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.252+0000 7f6b80086740 1 -- start start 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b80086740 1 Processor -- start 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b80086740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d121d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b80086740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 0x562d19d12710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b80086740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d15240 0x562d19d12c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b80086740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562d19d17e10 con 0x562d19d0a060 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b80086740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562d19bf7f10 con 0x562d19d0a980 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b80086740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562d19d17f90 con 0x562d19d15240 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b80086740 10 client.0 _ll_register_callbacks cb 0x562d19c1bdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d121d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d121d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57758/0 (socket says 172.21.15.62:57758) 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b77fff640 1 -- 172.21.15.62:0/1832343426 learned_addr learned my addr 172.21.15.62:0/1832343426 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 0x562d19d12710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b77fff640 1 -- 172.21.15.62:0/1832343426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d15240 msgr2=0x562d19d12c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b77fff640 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d15240 0x562d19d12c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b77fff640 1 -- 172.21.15.62:0/1832343426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 msgr2=0x562d19d12710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b77fff640 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 0x562d19d12710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.253+0000 7f6b77fff640 1 -- 172.21.15.62:0/1832343426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d1997e580 con 0x562d19d0a060 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b77fff640 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d121d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6b70006010 tx=0x7f6b70009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d495ace20e37dc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b56ffd640 1 -- 172.21.15.62:0/1832343426 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70016070 con 0x562d19d0a060 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b56ffd640 1 -- 172.21.15.62:0/1832343426 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6b700053e0 con 0x562d19d0a060 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b80086740 1 -- 172.21.15.62:0/1832343426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b70004d20 con 0x562d19d0a060 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b56ffd640 1 -- 172.21.15.62:0/1832343426 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6b7002f050 con 0x562d19d0a060 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b80086740 10 client.16824 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b80086740 1 -- 172.21.15.62:0/1832343426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562d19a48960 con 0x562d19d0a060 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b56ffd640 1 -- 172.21.15.62:0/1832343426 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b70004d20 con 0x562d19d0a060 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b56ffd640 1 -- 172.21.15.62:0/1832343426 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6b7000f040 con 0x562d19d0a060 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b80086740 10 client.16824 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b80086740 10 client.16824 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:21:37.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b80086740 20 client.16824 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b80086740 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562d19d24dc0 0x562d19d271b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b80086740 4 client.16824 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.254+0000 7f6b80086740 1 -- 172.21.15.62:0/1832343426 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d19c7cb90 con 0x562d19d24dc0 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.255+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562d19d24dc0 0x562d19d271b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.255+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562d19d24dc0 0x562d19d271b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c570c037c510b4c server_cookie=96bdbe2b2b2ba13e in_seq=0 out_seq=0 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.255+0000 7f6b56ffd640 10 client.16824 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b56ffd640 1 -- 172.21.15.62:0/1832343426 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562d19c7cb90 con 0x562d19d24dc0 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b56ffd640 10 client.16824 handle_command_reply: tid=0 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 client.16824 shutdown 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 -- 172.21.15.62:0/1832343426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 msgr2=0x562d19d121d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d121d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f6b70006010 tx=0x7f6b70009cb0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 -- 172.21.15.62:0/1832343426 shutdown_connections 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562d19d24dc0 0x562d19d271b0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b56ffd640 0 client.16824 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d15240 0x562d19d12c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 0x562d19d12710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 --2- 172.21.15.62:0/1832343426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d0a060 0x562d19d121d0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 -- 172.21.15.62:0/1832343426 >> 172.21.15.62:0/1832343426 conn(0x562d19d065f0 msgr2=0x562d19d05750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 -- 172.21.15.62:0/1832343426 shutdown_connections 2024-09-15T23:21:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 1 -- 172.21.15.62:0/1832343426 wait complete. 2024-09-15T23:21:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.256+0000 7f6b80086740 20 client.16824 trim_cache size 0 max 16384 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.273+0000 7f6b80086740 1 Processor -- start 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.273+0000 7f6b80086740 1 -- start start 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b80086740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a060 0x562d19d05640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b80086740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 0x562d19d0f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b80086740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d0f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b80086740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6b781b35e0 con 0x562d19d15240 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b80086740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b781b0eb0 con 0x562d19d0a980 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b80086740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6b781aed00 con 0x562d19d0a060 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d0f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d0f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57770/0 (socket says 172.21.15.62:57770) 2024-09-15T23:21:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 -- 172.21.15.62:0/2713281988 learned_addr learned my addr 172.21.15.62:0/2713281988 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 0x562d19d0f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a060 0x562d19d05640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 -- 172.21.15.62:0/2713281988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a060 msgr2=0x562d19d05640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a060 0x562d19d05640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 -- 172.21.15.62:0/2713281988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 msgr2=0x562d19d0f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 0x562d19d0f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 -- 172.21.15.62:0/2713281988 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d19d07280 con 0x562d19d15240 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 0x562d19d0f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.274+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d0f990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6b70002230 tx=0x7f6b70009730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c488f186e686af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b5d7fa640 1 -- 172.21.15.62:0/2713281988 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70016070 con 0x562d19d15240 2024-09-15T23:21:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b5d7fa640 1 -- 172.21.15.62:0/2713281988 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b70022070 con 0x562d19d15240 2024-09-15T23:21:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b5d7fa640 1 -- 172.21.15.62:0/2713281988 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70012980 con 0x562d19d15240 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b80086740 1 -- 172.21.15.62:0/2713281988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 msgr2=0x562d19d0f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b80086740 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d0f990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f6b70002230 tx=0x7f6b70009730 comp rx=0 tx=0).stop 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b80086740 1 -- 172.21.15.62:0/2713281988 shutdown_connections 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b80086740 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d0f990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b80086740 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a980 0x562d19d0f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b80086740 1 --2- 172.21.15.62:0/2713281988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a060 0x562d19d05640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b80086740 1 -- 172.21.15.62:0/2713281988 >> 172.21.15.62:0/2713281988 conn(0x562d19d065f0 msgr2=0x562d19d04f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b80086740 1 -- 172.21.15.62:0/2713281988 shutdown_connections 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.275+0000 7f6b80086740 1 -- 172.21.15.62:0/2713281988 wait complete. 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b80086740 1 -- start start 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b80086740 1 Processor -- start 2024-09-15T23:21:37.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b80086740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a060 0x562d19d288b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b80086740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 0x562d19d28df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b80086740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d88740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b80086740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6b781af370 con 0x562d19d15240 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b80086740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b781b07e0 con 0x562d19d0a060 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b80086740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6b781afa40 con 0x562d19d0a980 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b80086740 10 client.0 _ll_register_callbacks cb 0x7f6b70009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d88740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d88740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57774/0 (socket says 172.21.15.62:57774) 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.276+0000 7f6b77fff640 1 -- 172.21.15.62:0/2561233015 learned_addr learned my addr 172.21.15.62:0/2561233015 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a060 0x562d19d288b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 0x562d19d28df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b77fff640 1 -- 172.21.15.62:0/2561233015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 msgr2=0x562d19d28df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:37.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 0x562d19d28df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b77fff640 1 -- 172.21.15.62:0/2561233015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a060 msgr2=0x562d19d288b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a060 0x562d19d288b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b77fff640 1 -- 172.21.15.62:0/2561233015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d19d88db0 con 0x562d19d15240 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b77fff640 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d88740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6b70000f80 tx=0x7f6b70004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ed01805e4aff0a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b56ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b56ffd640 1 -- 172.21.15.62:0/2561233015 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7003a040 con 0x562d19d15240 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b80086740 1 -- 172.21.15.62:0/2561233015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b70022070 con 0x562d19d15240 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b56ffd640 1 -- 172.21.15.62:0/2561233015 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6b700053e0 con 0x562d19d15240 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b56ffd640 1 -- 172.21.15.62:0/2561233015 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6b70016040 con 0x562d19d15240 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b80086740 10 client.16830 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.277+0000 7f6b80086740 1 -- 172.21.15.62:0/2561233015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562d19d90170 con 0x562d19d15240 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b56ffd640 1 -- 172.21.15.62:0/2561233015 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b70022070 con 0x562d19d15240 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b56ffd640 1 -- 172.21.15.62:0/2561233015 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f6b70040070 con 0x562d19d15240 2024-09-15T23:21:37.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b80086740 10 client.16830 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:37.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b80086740 10 client.16830 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:21:37.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b80086740 20 client.16830 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:37.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b80086740 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562d19d90780 0x562d19d92ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:37.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b80086740 4 client.16830 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "9b36dcb4", "release": true, "await": true}] 2024-09-15T23:21:37.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b80086740 1 -- 172.21.15.62:0/2561233015 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "9b36dcb4", "release": true, "await": true}) -- 0x562d19c770a0 con 0x562d19d90780 2024-09-15T23:21:37.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562d19d90780 0x562d19d92ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:37.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b56ffd640 10 client.16830 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:37.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.278+0000 7f6b76ffd640 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562d19d90780 0x562d19d92ba0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=181aee753fcf68d8 server_cookie=881d674ae45d93d8 in_seq=0 out_seq=0 2024-09-15T23:21:37.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.304+0000 7f6b56ffd640 1 -- 172.21.15.62:0/2561233015 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x562d19c770a0 con 0x562d19d90780 2024-09-15T23:21:37.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.304+0000 7f6b56ffd640 10 client.16830 handle_command_reply: tid=0 2024-09-15T23:21:37.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.304+0000 7f6b80086740 1 client.16830 shutdown 2024-09-15T23:21:37.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.308+0000 7f6b80086740 1 -- 172.21.15.62:0/2561233015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 msgr2=0x562d19d88740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:37.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.308+0000 7f6b80086740 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d88740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6b70000f80 tx=0x7f6b70004f00 comp rx=0 tx=0).stop 2024-09-15T23:21:37.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.308+0000 7f6b80086740 1 -- 172.21.15.62:0/2561233015 shutdown_connections 2024-09-15T23:21:37.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.308+0000 7f6b80086740 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562d19d90780 0x562d19d92ba0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.309+0000 7f6b80086740 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562d19d15240 0x562d19d88740 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.309+0000 7f6b80086740 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562d19d0a980 0x562d19d28df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.309+0000 7f6b80086740 1 --2- 172.21.15.62:0/2561233015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562d19d0a060 0x562d19d288b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.309+0000 7f6b80086740 1 -- 172.21.15.62:0/2561233015 >> 172.21.15.62:0/2561233015 conn(0x562d19d065f0 msgr2=0x562d19d0c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:37.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.310+0000 7f6b56ffd640 0 client.16830 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:37.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.310+0000 7f6b80086740 1 -- 172.21.15.62:0/2561233015 shutdown_connections 2024-09-15T23:21:37.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.310+0000 7f6b80086740 1 -- 172.21.15.62:0/2561233015 wait complete. 2024-09-15T23:21:37.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.310+0000 7f6b80086740 20 client.16830 trim_cache size 0 max 16384 2024-09-15T23:21:37.315 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 31, 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 24, 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "9b36dcb4": { 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 24, 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:21:37.316 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 145.3, 2024-09-15T23:21:37.317 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:21:37.317 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:21:37.317 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:21:37.317 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:21:37.317 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:21:37.317 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:21:37.317 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:37.320 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:37.320 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:37.320 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:37.320 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:21:37.321 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:21:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.313+0000 7f6b7dcad640 1 -- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b600a32c0 msgr2=0x7f6b600a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.313+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b600a32c0 0x7f6b600a5780 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6b680cd840 tx=0x7f6b68000bc0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.313+0000 7f6b7dcad640 1 -- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78101df0 msgr2=0x7f6b780744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.313+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78101df0 0x7f6b780744f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c920 tx=0x7f6b6800cde0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.316+0000 7f6b7dcad640 1 -- 172.21.15.62:0/737594866 shutdown_connections 2024-09-15T23:21:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.316+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b600a32c0 0x7f6b600a5780 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.316+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b78106f20 0x7f6b78074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.316+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b781027a0 0x7f6b78074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.316+0000 7f6b7dcad640 1 --2- 172.21.15.62:0/737594866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b78101df0 0x7f6b780744f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.316+0000 7f6b7dcad640 1 -- 172.21.15.62:0/737594866 >> 172.21.15.62:0/737594866 conn(0x7f6b780fd8f0 msgr2=0x7f6b780febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.319+0000 7f6b7dcad640 1 -- 172.21.15.62:0/737594866 shutdown_connections 2024-09-15T23:21:37.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:21:37.319+0000 7f6b7dcad640 1 -- 172.21.15.62:0/737594866 wait complete. 2024-09-15T23:21:37.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:37 smithi062 ceph-mon[30949]: pgmap v530: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 6.8 KiB/s wr, 96 op/s 2024-09-15T23:21:37.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:37 smithi067 ceph-mon[35665]: pgmap v530: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 6.8 KiB/s wr, 96 op/s 2024-09-15T23:21:37.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:37 smithi148 ceph-mon[35610]: pgmap v530: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 6.8 KiB/s wr, 96 op/s 2024-09-15T23:21:38.050 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '9b36dcb4', total seconds elapsed: 28.7 2024-09-15T23:21:39.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 320.0 s, 185.0 tps, lat 2850.330 ms stddev 7551.031 2024-09-15T23:21:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:39 smithi062 ceph-mon[30949]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.4 KiB/s wr, 97 op/s 2024-09-15T23:21:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:39 smithi067 ceph-mon[35665]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.4 KiB/s wr, 97 op/s 2024-09-15T23:21:39.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:39 smithi148 ceph-mon[35610]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 8.4 KiB/s wr, 97 op/s 2024-09-15T23:21:40.734 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:40 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:40.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:40 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:40.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:40 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:21:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:41 smithi062 ceph-mon[30949]: pgmap v532: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.3 MiB/s wr, 179 op/s 2024-09-15T23:21:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:21:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:41 smithi067 ceph-mon[35665]: pgmap v532: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.3 MiB/s wr, 179 op/s 2024-09-15T23:21:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:21:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:41 smithi148 ceph-mon[35610]: pgmap v532: 129 pgs: 129 active+clean; 1.1 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 1.3 MiB/s wr, 179 op/s 2024-09-15T23:21:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:21:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:21:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:21:42.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:42 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:21:42.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:42 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:21:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:42 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:21:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:43 smithi062 ceph-mon[30949]: pgmap v533: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 5.3 MiB/s wr, 347 op/s 2024-09-15T23:21:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:43 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:21:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:43 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:21:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:43 smithi067 ceph-mon[35665]: pgmap v533: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 5.3 MiB/s wr, 347 op/s 2024-09-15T23:21:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:43 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:21:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:43 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:21:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:43 smithi148 ceph-mon[35610]: pgmap v533: 129 pgs: 129 active+clean; 1.1 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 5.3 MiB/s wr, 347 op/s 2024-09-15T23:21:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:43 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:21:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:43 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:21:44.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 325.0 s, 1091.6 tps, lat 89.710 ms stddev 115.457 2024-09-15T23:21:45.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:45 smithi062 ceph-mon[30949]: pgmap v534: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 10 MiB/s wr, 524 op/s 2024-09-15T23:21:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:45 smithi067 ceph-mon[35665]: pgmap v534: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 10 MiB/s wr, 524 op/s 2024-09-15T23:21:45.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:45 smithi148 ceph-mon[35610]: pgmap v534: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 10 MiB/s wr, 524 op/s 2024-09-15T23:21:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:47 smithi062 ceph-mon[30949]: pgmap v535: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 12 MiB/s wr, 553 op/s 2024-09-15T23:21:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:47 smithi067 ceph-mon[35665]: pgmap v535: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 12 MiB/s wr, 553 op/s 2024-09-15T23:21:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:47 smithi148 ceph-mon[35610]: pgmap v535: 129 pgs: 129 active+clean; 1.2 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 12 MiB/s wr, 553 op/s 2024-09-15T23:21:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.464+0000 7f69d6542640 1 -- 172.21.15.62:0/474118334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 msgr2=0x7f69d0108610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.464+0000 7f69d6542640 1 --2- 172.21.15.62:0/474118334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 0x7f69d0108610 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f69c8009be0 tx=0x7f69c802a710 comp rx=0 tx=0).stop 2024-09-15T23:21:49.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.465+0000 7f69d6542640 1 -- 172.21.15.62:0/474118334 shutdown_connections 2024-09-15T23:21:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.465+0000 7f69d6542640 1 --2- 172.21.15.62:0/474118334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 0x7f69d0108610 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.465+0000 7f69d6542640 1 --2- 172.21.15.62:0/474118334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69d01013c0 0x7f69d0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.465+0000 7f69d6542640 1 --2- 172.21.15.62:0/474118334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d01001d0 0x7f69d01005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.465+0000 7f69d6542640 1 -- 172.21.15.62:0/474118334 >> 172.21.15.62:0/474118334 conn(0x7f69d00fb930 msgr2=0x7f69d00fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.468+0000 7f69d6542640 1 -- 172.21.15.62:0/474118334 shutdown_connections 2024-09-15T23:21:49.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.468+0000 7f69d6542640 1 -- 172.21.15.62:0/474118334 wait complete. 2024-09-15T23:21:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.469+0000 7f69d6542640 1 Processor -- start 2024-09-15T23:21:49.470 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.469+0000 7f69d6542640 1 -- start start 2024-09-15T23:21:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69d6542640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d01001d0 0x7f69d0073700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69d6542640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69d01013c0 0x7f69d0073c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69d6542640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 0x7f69d0071d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69d6542640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f69d006c660 con 0x7f69d01013c0 2024-09-15T23:21:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69d6542640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69d006c4e0 con 0x7f69d01001d0 2024-09-15T23:21:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69d4adf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 0x7f69d0071d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69d4adf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 0x7f69d0071d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:40472/0 (socket says 172.21.15.62:40472) 2024-09-15T23:21:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69d4adf640 1 -- 172.21.15.62:0/3334824833 learned_addr learned my addr 172.21.15.62:0/3334824833 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69d6542640 1 -- 172.21.15.62:0/3334824833 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f69d006c7e0 con 0x7f69d0101d80 2024-09-15T23:21:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.470+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69d01013c0 0x7f69d0073c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.471+0000 7f69d4adf640 1 -- 172.21.15.62:0/3334824833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d01001d0 msgr2=0x7f69d0073700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.471+0000 7f69cffff640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d01001d0 0x7f69d0073700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.471+0000 7f69d4adf640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d01001d0 0x7f69d0073700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.471+0000 7f69d4adf640 1 -- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69d01013c0 msgr2=0x7f69d0073c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.471+0000 7f69d4adf640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69d01013c0 0x7f69d0073c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.471+0000 7f69d4adf640 1 -- 172.21.15.62:0/3334824833 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d00724c0 con 0x7f69d0101d80 2024-09-15T23:21:49.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.471+0000 7f69d4adf640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 0x7f69d0071d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f69c8034040 tx=0x7f69c8032f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:49.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.476+0000 7f69cd7fa640 1 -- 172.21.15.62:0/3334824833 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c8042030 con 0x7f69d0101d80 2024-09-15T23:21:49.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.476+0000 7f69cd7fa640 1 -- 172.21.15.62:0/3334824833 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c8003970 con 0x7f69d0101d80 2024-09-15T23:21:49.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.476+0000 7f69cd7fa640 1 -- 172.21.15.62:0/3334824833 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c8036040 con 0x7f69d0101d80 2024-09-15T23:21:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.477+0000 7f69d6542640 1 -- 172.21.15.62:0/3334824833 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d0072750 con 0x7f69d0101d80 2024-09-15T23:21:49.478 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.477+0000 7f69d6542640 1 -- 172.21.15.62:0/3334824833 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d01a6940 con 0x7f69d0101d80 2024-09-15T23:21:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.481+0000 7f69cd7fa640 1 -- 172.21.15.62:0/3334824833 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f69c8045030 con 0x7f69d0101d80 2024-09-15T23:21:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.481+0000 7f69d92d9740 1 Processor -- start 2024-09-15T23:21:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.481+0000 7f69d92d9740 1 -- start start 2024-09-15T23:21:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.482+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ac9a0 0x5638f43acda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.482+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.482+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.482+0000 7f69d92d9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5638f43102f0 con 0x5638f43ac9a0 2024-09-15T23:21:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.482+0000 7f69d92d9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5638f42b8270 con 0x5638f43b2370 2024-09-15T23:21:49.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.482+0000 7f69d92d9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5638f430c2b0 con 0x5638f43ad2e0 2024-09-15T23:21:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43b1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:40480/0 (socket says 172.21.15.62:40480) 2024-09-15T23:21:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cffff640 1 -- 172.21.15.62:0/3255492815 learned_addr learned my addr 172.21.15.62:0/3255492815 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:49.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cd7fa640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f69b80a34b0 0x7f69b80a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cffff640 1 -- 172.21.15.62:0/3255492815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 msgr2=0x5638f43b4830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:21:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cffff640 1 --2- 172.21.15.62:0/3255492815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43b4830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cffff640 1 -- 172.21.15.62:0/3255492815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ac9a0 msgr2=0x5638f43acda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cffff640 1 --2- 172.21.15.62:0/3255492815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ac9a0 0x5638f43acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cffff640 1 -- 172.21.15.62:0/3255492815 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638f42b7340 con 0x5638f43ad2e0 2024-09-15T23:21:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cd7fa640 1 -- 172.21.15.62:0/3334824833 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f69c80f3bf0 con 0x7f69d0101d80 2024-09-15T23:21:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69d4adf640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f69b80a34b0 0x7f69b80a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.483+0000 7f69cffff640 1 --2- 172.21.15.62:0/3255492815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43b1da0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f69c400a8a0 tx=0x7f69c400ad60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51c43f148f0313e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.484+0000 7f69b67fc640 1 -- 172.21.15.62:0/3255492815 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4015070 con 0x5638f43ad2e0 2024-09-15T23:21:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.484+0000 7f69b67fc640 1 -- 172.21.15.62:0/3255492815 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c400cb40 con 0x5638f43ad2e0 2024-09-15T23:21:49.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.484+0000 7f69b67fc640 1 -- 172.21.15.62:0/3255492815 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c40185c0 con 0x5638f43ad2e0 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.484+0000 7f69d92d9740 1 -- 172.21.15.62:0/3255492815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 msgr2=0x5638f43b1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.484+0000 7f69d4adf640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f69b80a34b0 0x7f69b80a5970 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f69c80bc2f0 tx=0x7f69c802a710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.484+0000 7f69d92d9740 1 --2- 172.21.15.62:0/3255492815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43b1da0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f69c400a8a0 tx=0x7f69c400ad60 comp rx=0 tx=0).stop 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.485+0000 7f69d92d9740 1 -- 172.21.15.62:0/3255492815 shutdown_connections 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.485+0000 7f69d92d9740 1 --2- 172.21.15.62:0/3255492815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.485+0000 7f69d92d9740 1 --2- 172.21.15.62:0/3255492815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43b1da0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.485+0000 7f69d92d9740 1 --2- 172.21.15.62:0/3255492815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ac9a0 0x5638f43acda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.485+0000 7f69d92d9740 1 -- 172.21.15.62:0/3255492815 >> 172.21.15.62:0/3255492815 conn(0x5638f43a8010 msgr2=0x5638f43a8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.485+0000 7f69d92d9740 1 -- 172.21.15.62:0/3255492815 shutdown_connections 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.485+0000 7f69d92d9740 1 -- 172.21.15.62:0/3255492815 wait complete. 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.485+0000 7f69d92d9740 1 -- start start 2024-09-15T23:21:49.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.485+0000 7f69d92d9740 1 Processor -- start 2024-09-15T23:21:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 0x5638f43b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43be1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69d92d9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5638f43b53e0 con 0x5638f43ad2e0 2024-09-15T23:21:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69d92d9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5638f4299290 con 0x5638f43b2370 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69d92d9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5638f43b5560 con 0x5638f43ac9a0 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69d92d9740 10 client.0 _ll_register_callbacks cb 0x5638f42ca140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69cf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43be1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69cf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:40486/0 (socket says 172.21.15.62:40486) 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69cf7fe640 1 -- 172.21.15.62:0/2297876567 learned_addr learned my addr 172.21.15.62:0/2297876567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.486+0000 7f69cffff640 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 0x5638f43b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69cf7fe640 1 -- 172.21.15.62:0/2297876567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 msgr2=0x5638f43be1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43be1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69cf7fe640 1 -- 172.21.15.62:0/2297876567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 msgr2=0x5638f43b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 0x5638f43b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69cf7fe640 1 -- 172.21.15.62:0/2297876567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638f4023210 con 0x5638f43ac9a0 2024-09-15T23:21:49.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b6c60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f69c000a5c0 tx=0x7f69c00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d836f2899e01e684 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69af7fe640 1 -- 172.21.15.62:0/2297876567 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c000dc70 con 0x5638f43ac9a0 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69d92d9740 1 -- 172.21.15.62:0/2297876567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69c400cb40 con 0x5638f43ac9a0 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.487+0000 7f69af7fe640 1 -- 172.21.15.62:0/2297876567 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f69c0005020 con 0x5638f43ac9a0 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69af7fe640 1 -- 172.21.15.62:0/2297876567 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f69c0025050 con 0x5638f43ac9a0 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69d92d9740 10 client.26206 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69d92d9740 1 -- 172.21.15.62:0/2297876567 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5638f40ef620 con 0x5638f43ac9a0 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69af7fe640 1 -- 172.21.15.62:0/2297876567 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c400cb40 con 0x5638f43ac9a0 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69af7fe640 1 -- 172.21.15.62:0/2297876567 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f69c000b040 con 0x5638f43ac9a0 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69d92d9740 10 client.26206 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69d92d9740 10 client.26206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69d92d9740 20 client.26206 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69d92d9740 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638f43c28c0 0x5638f43c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.491 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69d92d9740 4 client.26206 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:21:49.492 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.488+0000 7f69d92d9740 1 -- 172.21.15.62:0/2297876567 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638f431da20 con 0x5638f43c28c0 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.489+0000 7f69cffff640 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638f43c28c0 0x5638f43c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.489+0000 7f69af7fe640 10 client.26206 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.489+0000 7f69cffff640 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638f43c28c0 0x5638f43c4cb0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=178c155b7d6e660 server_cookie=924c2ed9460138fb in_seq=0 out_seq=0 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69af7fe640 1 -- 172.21.15.62:0/2297876567 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5638f431da20 con 0x5638f43c28c0 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69af7fe640 10 client.26206 handle_command_reply: tid=0 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 client.26206 shutdown 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 -- 172.21.15.62:0/2297876567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 msgr2=0x5638f43b6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b6c60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f69c000a5c0 tx=0x7f69c00046c0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 -- 172.21.15.62:0/2297876567 shutdown_connections 2024-09-15T23:21:49.493 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638f43c28c0 0x5638f43c4cb0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43be1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 0x5638f43b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 --2- 172.21.15.62:0/2297876567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b6c60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 -- 172.21.15.62:0/2297876567 >> 172.21.15.62:0/2297876567 conn(0x5638f43a8010 msgr2=0x5638f43a9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69af7fe640 0 client.26206 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 -- 172.21.15.62:0/2297876567 shutdown_connections 2024-09-15T23:21:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 1 -- 172.21.15.62:0/2297876567 wait complete. 2024-09-15T23:21:49.494 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.490+0000 7f69d92d9740 20 client.26206 trim_cache size 0 max 16384 2024-09-15T23:21:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69d92d9740 1 Processor -- start 2024-09-15T23:21:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69d92d9740 1 -- start start 2024-09-15T23:21:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 0x5638f43be1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69d92d9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01b3620 con 0x5638f43ad2e0 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69cf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69cf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:40498/0 (socket says 172.21.15.62:40498) 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69cf7fe640 1 -- 172.21.15.62:0/3137029766 learned_addr learned my addr 172.21.15.62:0/3137029766 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69cffff640 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 0x5638f43be1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69d92d9740 1 -- 172.21.15.62:0/3137029766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01b0ef0 con 0x5638f43b2370 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69d92d9740 1 -- 172.21.15.62:0/3137029766 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01aed40 con 0x5638f43ac9a0 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.512+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.513+0000 7f69cf7fe640 1 -- 172.21.15.62:0/3137029766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 msgr2=0x5638f43b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.513+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.513+0000 7f69cf7fe640 1 -- 172.21.15.62:0/3137029766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 msgr2=0x5638f43be1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.513+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 0x5638f43be1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.513+0000 7f69cf7fe640 1 -- 172.21.15.62:0/3137029766 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638f43adf70 con 0x5638f43ac9a0 2024-09-15T23:21:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.513+0000 7f69cffff640 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 0x5638f43be1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.513+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b7120 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f69c000e910 tx=0x7f69c0005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c0cd7e52e334474 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.513+0000 7f69b5ffb640 1 -- 172.21.15.62:0/3137029766 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c002f070 con 0x5638f43ac9a0 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.514+0000 7f69b5ffb640 1 -- 172.21.15.62:0/3137029766 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c001d460 con 0x5638f43ac9a0 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.514+0000 7f69b5ffb640 1 -- 172.21.15.62:0/3137029766 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c00054b0 con 0x5638f43ac9a0 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.515+0000 7f69d92d9740 1 -- 172.21.15.62:0/3137029766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 msgr2=0x5638f43b7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.515+0000 7f69d92d9740 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b7120 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f69c000e910 tx=0x7f69c0005850 comp rx=0 tx=0).stop 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.515+0000 7f69d92d9740 1 -- 172.21.15.62:0/3137029766 shutdown_connections 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.515+0000 7f69d92d9740 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43b2370 0x5638f43b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.515+0000 7f69d92d9740 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43ad2e0 0x5638f43be1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.515+0000 7f69d92d9740 1 --2- 172.21.15.62:0/3137029766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ac9a0 0x5638f43b7120 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.515+0000 7f69d92d9740 1 -- 172.21.15.62:0/3137029766 >> 172.21.15.62:0/3137029766 conn(0x5638f43a8010 msgr2=0x5638f43b6a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.515+0000 7f69d92d9740 1 -- 172.21.15.62:0/3137029766 shutdown_connections 2024-09-15T23:21:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.515+0000 7f69d92d9740 1 -- 172.21.15.62:0/3137029766 wait complete. 2024-09-15T23:21:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.516+0000 7f69d92d9740 1 -- start start 2024-09-15T23:21:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.516+0000 7f69d92d9740 1 Processor -- start 2024-09-15T23:21:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.516+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43ac9a0 0x5638f43bb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.516+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69d92d9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43b2370 0x5638f43bcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69d92d9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01af3b0 con 0x5638f43b2370 2024-09-15T23:21:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69d92d9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01b0820 con 0x5638f43ac9a0 2024-09-15T23:21:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69d92d9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f69d01afa80 con 0x5638f43ad2e0 2024-09-15T23:21:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69d92d9740 10 client.0 _ll_register_callbacks cb 0x7f69c000aa30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:21:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43ac9a0 0x5638f43bb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43ac9a0 0x5638f43bb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:45730/0 (socket says 172.21.15.62:45730) 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cf7fe640 1 -- 172.21.15.62:0/4260939426 learned_addr learned my addr 172.21.15.62:0/4260939426 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43b2370 0x5638f43bcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cffff640 1 -- 172.21.15.62:0/4260939426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43ac9a0 msgr2=0x5638f43bb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cffff640 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43ac9a0 0x5638f43bb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cffff640 1 -- 172.21.15.62:0/4260939426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43b2370 msgr2=0x5638f43bcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cffff640 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43b2370 0x5638f43bcc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cffff640 1 -- 172.21.15.62:0/4260939426 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638f43bd260 con 0x5638f43ad2e0 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43ac9a0 0x5638f43bb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69cffff640 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43bbc90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f69c400a7c0 tx=0x7f69c400a5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e672578e1fdeb790 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:21:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.517+0000 7f69af7fe640 1 -- 172.21.15.62:0/4260939426 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4015040 con 0x5638f43ad2e0 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69af7fe640 1 -- 172.21.15.62:0/4260939426 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f69c400c940 con 0x5638f43ad2e0 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69d92d9740 1 -- 172.21.15.62:0/4260939426 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69c001d460 con 0x5638f43ad2e0 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69af7fe640 1 -- 172.21.15.62:0/4260939426 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f69c400cb40 con 0x5638f43ad2e0 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69d92d9740 10 client.26212 fetch_fsmap learned FSMap version 31 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69d92d9740 1 -- 172.21.15.62:0/4260939426 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5638f442d550 con 0x5638f43ad2e0 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69af7fe640 1 -- 172.21.15.62:0/4260939426 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69c40118d0 con 0x5638f43ad2e0 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69af7fe640 1 -- 172.21.15.62:0/4260939426 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 31) ==== 8354+0+0 (secure 0 0 0) 0x7f69c4003690 con 0x5638f43ad2e0 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69d92d9740 10 client.26212 fetch_fsmap finished waiting for FSMap version 31 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69d92d9740 10 client.26212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69d92d9740 20 client.26212 populate_metadata read hostname 'smithi062' 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.518+0000 7f69d92d9740 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638f442da60 0x5638f442fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.519+0000 7f69d92d9740 4 client.26212 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.519+0000 7f69d92d9740 1 -- 172.21.15.62:0/4260939426 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f69c001d460 con 0x5638f442da60 2024-09-15T23:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.519+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638f442da60 0x5638f442fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:21:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.520+0000 7f69af7fe640 10 client.26212 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.520+0000 7f69cf7fe640 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638f442da60 0x5638f442fe50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2985902a5961d807 server_cookie=9646f491bfe26e in_seq=0 out_seq=0 2024-09-15T23:21:49.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.556+0000 7f69af7fe640 1 -- 172.21.15.62:0/4260939426 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f69c001d460 con 0x5638f442da60 2024-09-15T23:21:49.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.557+0000 7f69af7fe640 10 client.26212 handle_command_reply: tid=0 2024-09-15T23:21:49.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.558+0000 7f69d92d9740 1 client.26212 shutdown 2024-09-15T23:21:49.562 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.561+0000 7f69d92d9740 1 -- 172.21.15.62:0/4260939426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 msgr2=0x5638f43bbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.562 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.561+0000 7f69d92d9740 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43bbc90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f69c400a7c0 tx=0x7f69c400a5e0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.561+0000 7f69d92d9740 1 -- 172.21.15.62:0/4260939426 shutdown_connections 2024-09-15T23:21:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.561+0000 7f69d92d9740 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638f442da60 0x5638f442fe50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.562+0000 7f69af7fe640 0 client.26212 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:21:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.562+0000 7f69d92d9740 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638f43b2370 0x5638f43bcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.562+0000 7f69d92d9740 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638f43ad2e0 0x5638f43bbc90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.562+0000 7f69d92d9740 1 --2- 172.21.15.62:0/4260939426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638f43ac9a0 0x5638f43bb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.562+0000 7f69d92d9740 1 -- 172.21.15.62:0/4260939426 >> 172.21.15.62:0/4260939426 conn(0x5638f43a8010 msgr2=0x5638f43a69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:49.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.563+0000 7f69d92d9740 1 -- 172.21.15.62:0/4260939426 shutdown_connections 2024-09-15T23:21:49.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.564+0000 7f69d92d9740 1 -- 172.21.15.62:0/4260939426 wait complete. 2024-09-15T23:21:49.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.564+0000 7f69d92d9740 20 client.26212 trim_cache size 0 max 16384 2024-09-15T23:21:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:21:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:21:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:21:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:21:49.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.566+0000 7f69d6542640 1 -- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f69b80a34b0 msgr2=0x7f69b80a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.566+0000 7f69d6542640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f69b80a34b0 0x7f69b80a5970 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f69c80bc2f0 tx=0x7f69c802a710 comp rx=0 tx=0).stop 2024-09-15T23:21:49.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.566+0000 7f69d6542640 1 -- 172.21.15.62:0/3334824833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 msgr2=0x7f69d0071d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:21:49.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.566+0000 7f69d6542640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 0x7f69d0071d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f69c8034040 tx=0x7f69c8032f60 comp rx=0 tx=0).stop 2024-09-15T23:21:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.567+0000 7f69d6542640 1 -- 172.21.15.62:0/3334824833 shutdown_connections 2024-09-15T23:21:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.567+0000 7f69d6542640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f69b80a34b0 0x7f69b80a5970 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.567+0000 7f69d6542640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f69d0101d80 0x7f69d0071d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.567+0000 7f69d6542640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f69d01013c0 0x7f69d0073c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.567+0000 7f69d6542640 1 --2- 172.21.15.62:0/3334824833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d01001d0 0x7f69d0073700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:21:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.567+0000 7f69d6542640 1 -- 172.21.15.62:0/3334824833 >> 172.21.15.62:0/3334824833 conn(0x7f69d00fb930 msgr2=0x7f69d0104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:21:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.568+0000 7f69d6542640 1 -- 172.21.15.62:0/3334824833 shutdown_connections 2024-09-15T23:21:49.572 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 330.0 s, 1504.5 tps, lat 68.939 ms stddev 90.328 2024-09-15T23:21:49.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:21:49.571+0000 7f69d6542640 1 -- 172.21.15.62:0/3334824833 wait complete. 2024-09-15T23:21:49.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:49 smithi062 ceph-mon[30949]: pgmap v536: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 15 MiB/s wr, 592 op/s 2024-09-15T23:21:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:49 smithi067 ceph-mon[35665]: pgmap v536: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 15 MiB/s wr, 592 op/s 2024-09-15T23:21:49.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:49 smithi148 ceph-mon[35610]: pgmap v536: 129 pgs: 129 active+clean; 1.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 15 MiB/s wr, 592 op/s 2024-09-15T23:21:51.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:51 smithi062 ceph-mon[30949]: pgmap v537: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 20 MiB/s wr, 721 op/s 2024-09-15T23:21:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:51 smithi067 ceph-mon[35665]: pgmap v537: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 20 MiB/s wr, 721 op/s 2024-09-15T23:21:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:51 smithi148 ceph-mon[35610]: pgmap v537: 129 pgs: 129 active+clean; 1.2 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 20 MiB/s wr, 721 op/s 2024-09-15T23:21:53.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:53 smithi067 ceph-mon[35665]: pgmap v538: 129 pgs: 129 active+clean; 1.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 24 MiB/s wr, 763 op/s 2024-09-15T23:21:53.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:53 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:21:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:53 smithi148 ceph-mon[35610]: pgmap v538: 129 pgs: 129 active+clean; 1.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 24 MiB/s wr, 763 op/s 2024-09-15T23:21:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:53 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:21:53.829 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:53 smithi062 ceph-mon[30949]: pgmap v538: 129 pgs: 129 active+clean; 1.2 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 24 MiB/s wr, 763 op/s 2024-09-15T23:21:53.829 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:53 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:21:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 335.0 s, 1545.7 tps, lat 64.958 ms stddev 76.973 2024-09-15T23:21:55.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:55 smithi062 ceph-mon[30949]: pgmap v539: 129 pgs: 129 active+clean; 1.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 727 op/s 2024-09-15T23:21:56.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:55 smithi067 ceph-mon[35665]: pgmap v539: 129 pgs: 129 active+clean; 1.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 727 op/s 2024-09-15T23:21:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:55 smithi148 ceph-mon[35610]: pgmap v539: 129 pgs: 129 active+clean; 1.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 25 MiB/s wr, 727 op/s 2024-09-15T23:21:57.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:57 smithi062 ceph-mon[30949]: pgmap v540: 129 pgs: 129 active+clean; 1.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 MiB/s wr, 594 op/s 2024-09-15T23:21:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:57 smithi067 ceph-mon[35665]: pgmap v540: 129 pgs: 129 active+clean; 1.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 MiB/s wr, 594 op/s 2024-09-15T23:21:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:57 smithi148 ceph-mon[35610]: pgmap v540: 129 pgs: 129 active+clean; 1.3 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 MiB/s wr, 594 op/s 2024-09-15T23:21:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 340.0 s, 1077.0 tps, lat 90.786 ms stddev 103.083 2024-09-15T23:21:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:21:59 smithi062 ceph-mon[30949]: pgmap v541: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 MiB/s wr, 591 op/s 2024-09-15T23:22:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:21:59 smithi067 ceph-mon[35665]: pgmap v541: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 MiB/s wr, 591 op/s 2024-09-15T23:22:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:21:59 smithi148 ceph-mon[35610]: pgmap v541: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 MiB/s wr, 591 op/s 2024-09-15T23:22:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:01 smithi062 ceph-mon[30949]: pgmap v542: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 MiB/s wr, 648 op/s 2024-09-15T23:22:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:01 smithi067 ceph-mon[35665]: pgmap v542: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 MiB/s wr, 648 op/s 2024-09-15T23:22:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:01 smithi148 ceph-mon[35610]: pgmap v542: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 23 MiB/s wr, 648 op/s 2024-09-15T23:22:02.738 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.733+0000 7f03c535f640 1 -- 172.21.15.62:0/112683617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 msgr2=0x7f03c00750b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.733+0000 7f03c535f640 1 --2- 172.21.15.62:0/112683617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 0x7f03c00750b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f03b0009900 tx=0x7f03b002a440 comp rx=0 tx=0).stop 2024-09-15T23:22:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.737+0000 7f03c535f640 1 -- 172.21.15.62:0/112683617 shutdown_connections 2024-09-15T23:22:02.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.737+0000 7f03c535f640 1 --2- 172.21.15.62:0/112683617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c010e130 0x7f03c0110520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.737+0000 7f03c535f640 1 --2- 172.21.15.62:0/112683617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 0x7f03c00750b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.737+0000 7f03c535f640 1 --2- 172.21.15.62:0/112683617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c0076230 0x7f03c0074680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.737+0000 7f03c535f640 1 -- 172.21.15.62:0/112683617 >> 172.21.15.62:0/112683617 conn(0x7f03c00fd2e0 msgr2=0x7f03c00ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.738+0000 7f03c535f640 1 -- 172.21.15.62:0/112683617 shutdown_connections 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.739+0000 7f03c535f640 1 -- 172.21.15.62:0/112683617 wait complete. 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.740+0000 7f03c535f640 1 Processor -- start 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.740+0000 7f03c535f640 1 -- start start 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03c535f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 0x7f03c0072e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03c535f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c0076230 0x7f03c0073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03beffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 0x7f03c0072e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03beffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 0x7f03c0072e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:42044/0 (socket says 172.21.15.62:42044) 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03c535f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c010e130 0x7f03c010b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03beffd640 1 -- 172.21.15.62:0/4094215241 learned_addr learned my addr 172.21.15.62:0/4094215241 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f03c006d510 con 0x7f03c0076230 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f03c006d390 con 0x7f03c010e130 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f03c006d690 con 0x7f03c0074c50 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03bf7fe640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c010e130 0x7f03c010b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.741+0000 7f03be7fc640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c0076230 0x7f03c0073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03beffd640 1 -- 172.21.15.62:0/4094215241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c010e130 msgr2=0x7f03c010b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03beffd640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c010e130 0x7f03c010b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03beffd640 1 -- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c0076230 msgr2=0x7f03c0073340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03beffd640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c0076230 0x7f03c0073340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03beffd640 1 -- 172.21.15.62:0/4094215241 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c010be90 con 0x7f03c0074c50 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03be7fc640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c0076230 0x7f03c0073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:02.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03bf7fe640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c010e130 0x7f03c010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03beffd640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 0x7f03c0072e00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f03b800eab0 tx=0x7f03b800ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03b7fff640 1 -- 172.21.15.62:0/4094215241 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b800ce40 con 0x7f03c0074c50 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03b7fff640 1 -- 172.21.15.62:0/4094215241 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03b8011950 con 0x7f03c0074c50 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c010c180 con 0x7f03c0074c50 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03b7fff640 1 -- 172.21.15.62:0/4094215241 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b8010ae0 con 0x7f03c0074c50 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.742+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c010c6a0 con 0x7f03c0074c50 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.746+0000 7f03b7fff640 1 -- 172.21.15.62:0/4094215241 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f03b8011b00 con 0x7f03c0074c50 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.748+0000 7f03b7fff640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f03ac0a33d0 0x7f03ac0a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.748+0000 7f03b7fff640 1 -- 172.21.15.62:0/4094215241 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f03b80d03e0 con 0x7f03c0074c50 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.748+0000 7f03be7fc640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f03ac0a33d0 0x7f03ac0a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.749+0000 7f03c81d4740 1 Processor -- start 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03be7fc640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f03ac0a33d0 0x7f03ac0a5890 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f03b0000c00 tx=0x7f03b0031540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03c81d4740 1 -- start start 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528bcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528bf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03c81d4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d95281b2f0 con 0x55d9528b82e0 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03c81d4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d9527c3270 con 0x55d9528b79a0 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03c81d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d9528172b0 con 0x55d9528bd370 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03be7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528bcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528b7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36418/0 (socket says 172.21.15.62:36418) 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03beffd640 1 -- 172.21.15.62:0/2734583056 learned_addr learned my addr 172.21.15.62:0/2734583056 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03bf7fe640 1 --2- 172.21.15.62:0/2734583056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528bf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03beffd640 1 -- 172.21.15.62:0/2734583056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 msgr2=0x55d9528bf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03beffd640 1 --2- 172.21.15.62:0/2734583056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528bf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03beffd640 1 -- 172.21.15.62:0/2734583056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 msgr2=0x55d9528bcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:02.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03beffd640 1 --2- 172.21.15.62:0/2734583056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528bcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03beffd640 1 -- 172.21.15.62:0/2734583056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9527c2340 con 0x55d9528b79a0 2024-09-15T23:22:02.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.750+0000 7f03bf7fe640 1 --2- 172.21.15.62:0/2734583056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:02.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.751+0000 7f03beffd640 1 --2- 172.21.15.62:0/2734583056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528b7da0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f03b800ea80 tx=0x7f03b809bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=825308c5297fa92b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:02.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.751+0000 7f03b57fa640 1 -- 172.21.15.62:0/2734583056 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b80b5070 con 0x55d9528b79a0 2024-09-15T23:22:02.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.751+0000 7f03b57fa640 1 -- 172.21.15.62:0/2734583056 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03b80a09d0 con 0x55d9528b79a0 2024-09-15T23:22:02.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.751+0000 7f03b57fa640 1 -- 172.21.15.62:0/2734583056 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b809c880 con 0x55d9528b79a0 2024-09-15T23:22:02.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.752+0000 7f03c81d4740 1 -- 172.21.15.62:0/2734583056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 msgr2=0x55d9528b7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.752+0000 7f03c81d4740 1 --2- 172.21.15.62:0/2734583056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528b7da0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f03b800ea80 tx=0x7f03b809bf80 comp rx=0 tx=0).stop 2024-09-15T23:22:02.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.752+0000 7f03c81d4740 1 -- 172.21.15.62:0/2734583056 shutdown_connections 2024-09-15T23:22:02.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.752+0000 7f03c81d4740 1 --2- 172.21.15.62:0/2734583056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.752+0000 7f03c81d4740 1 --2- 172.21.15.62:0/2734583056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528bcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.752+0000 7f03c81d4740 1 --2- 172.21.15.62:0/2734583056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528b7da0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.753+0000 7f03c81d4740 1 -- 172.21.15.62:0/2734583056 >> 172.21.15.62:0/2734583056 conn(0x55d9528b3010 msgr2=0x55d9528b3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:02.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.753+0000 7f03c81d4740 1 -- 172.21.15.62:0/2734583056 shutdown_connections 2024-09-15T23:22:02.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.753+0000 7f03c81d4740 1 -- 172.21.15.62:0/2734583056 wait complete. 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.753+0000 7f03c81d4740 1 -- start start 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.753+0000 7f03c81d4740 1 Processor -- start 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.753+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528c21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528c9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36430/0 (socket says 172.21.15.62:36430) 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03beffd640 1 -- 172.21.15.62:0/1640417150 learned_addr learned my addr 172.21.15.62:0/1640417150 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d9528c0340 con 0x55d9528b82e0 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d9527a4290 con 0x55d9528b79a0 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d9528c04c0 con 0x55d9528bd370 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03be7fc640 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528c21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.775 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03bf7fe640 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03c81d4740 10 client.0 _ll_register_callbacks cb 0x55d9527d5140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03beffd640 1 -- 172.21.15.62:0/1640417150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 msgr2=0x55d9528c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03beffd640 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03beffd640 1 -- 172.21.15.62:0/1640417150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 msgr2=0x55d9528c21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03beffd640 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528c21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03beffd640 1 -- 172.21.15.62:0/1640417150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d95252e210 con 0x55d9528b79a0 2024-09-15T23:22:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03bf7fe640 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03beffd640 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c1c80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f03b80a4d60 tx=0x7f03b809cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c885400a264cff10 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:02.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:02.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03a67fc640 1 -- 172.21.15.62:0/1640417150 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b80b9020 con 0x55d9528b79a0 2024-09-15T23:22:02.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.754+0000 7f03a67fc640 1 -- 172.21.15.62:0/1640417150 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f03b80a5000 con 0x55d9528b79a0 2024-09-15T23:22:02.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03b80a09d0 con 0x55d9528b79a0 2024-09-15T23:22:02.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03a67fc640 1 -- 172.21.15.62:0/1640417150 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f03b80b5070 con 0x55d9528b79a0 2024-09-15T23:22:02.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03c81d4740 10 client.26426 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:02.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d9525fa620 con 0x55d9528b79a0 2024-09-15T23:22:02.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03a67fc640 1 -- 172.21.15.62:0/1640417150 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03b80cdc10 con 0x55d9528b79a0 2024-09-15T23:22:02.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03a67fc640 1 -- 172.21.15.62:0/1640417150 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f03b807d020 con 0x55d9528b79a0 2024-09-15T23:22:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03c81d4740 10 client.26426 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03c81d4740 10 client.26426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03c81d4740 20 client.26426 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03c81d4740 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9528cd4c0 0x55d9528cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03c81d4740 4 client.26426 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f03b80a09d0 con 0x55d9528cd4c0 2024-09-15T23:22:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.755+0000 7f03be7fc640 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9528cd4c0 0x55d9528cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.756+0000 7f03a67fc640 10 client.26426 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.756+0000 7f03be7fc640 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9528cd4c0 0x55d9528cf8b0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5d33e24a7df6bbf server_cookie=623e3b77be6d040f in_seq=0 out_seq=0 2024-09-15T23:22:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.756+0000 7f03a67fc640 1 -- 172.21.15.62:0/1640417150 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f03b80a09d0 con 0x55d9528cd4c0 2024-09-15T23:22:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.756+0000 7f03a67fc640 10 client.26426 handle_command_reply: tid=0 2024-09-15T23:22:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.756+0000 7f03c81d4740 1 client.26426 shutdown 2024-09-15T23:22:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 msgr2=0x55d9528c1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c1c80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f03b80a4d60 tx=0x7f03b809cbd0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 shutdown_connections 2024-09-15T23:22:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9528cd4c0 0x55d9528cf8b0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03a67fc640 0 client.26426 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528bd370 0x55d9528c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528c21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 --2- 172.21.15.62:0/1640417150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c1c80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 >> 172.21.15.62:0/1640417150 conn(0x55d9528b3010 msgr2=0x55d9528b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 shutdown_connections 2024-09-15T23:22:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 1 -- 172.21.15.62:0/1640417150 wait complete. 2024-09-15T23:22:02.780 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.757+0000 7f03c81d4740 20 client.26426 trim_cache size 0 max 16384 2024-09-15T23:22:02.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.777+0000 7f03c81d4740 1 Processor -- start 2024-09-15T23:22:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.777+0000 7f03c81d4740 1 -- start start 2024-09-15T23:22:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.777+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b3010 0x55d9528c2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.777+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b79a0 0x55d9528c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b3010 0x55d9528c2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b3010 0x55d9528c2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36432/0 (socket says 172.21.15.62:36432) 2024-09-15T23:22:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03be7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b79a0 0x55d9528c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03beffd640 1 -- 172.21.15.62:0/3444466988 learned_addr learned my addr 172.21.15.62:0/3444466988 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03c81d4740 1 -- 172.21.15.62:0/3444466988 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f03c01c4520 con 0x55d9528b82e0 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03c81d4740 1 -- 172.21.15.62:0/3444466988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f03c01c1df0 con 0x55d9528b3010 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03c81d4740 1 -- 172.21.15.62:0/3444466988 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f03c01bfc40 con 0x55d9528b79a0 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03bf7fe640 1 --2- 172.21.15.62:0/3444466988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03beffd640 1 -- 172.21.15.62:0/3444466988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b79a0 msgr2=0x55d9528c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03beffd640 1 --2- 172.21.15.62:0/3444466988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b79a0 0x55d9528c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03beffd640 1 -- 172.21.15.62:0/3444466988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 msgr2=0x55d9528b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03beffd640 1 --2- 172.21.15.62:0/3444466988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03beffd640 1 -- 172.21.15.62:0/3444466988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9528bc870 con 0x55d9528b3010 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.778+0000 7f03beffd640 1 --2- 172.21.15.62:0/3444466988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b3010 0x55d9528c2250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f03b800c700 tx=0x7f03b809ab30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=631d5386283964a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:02.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.779+0000 7f03b4ff9640 1 -- 172.21.15.62:0/3444466988 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b80a5290 con 0x55d9528b3010 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.779+0000 7f03b4ff9640 1 -- 172.21.15.62:0/3444466988 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03b8098410 con 0x55d9528b3010 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.779+0000 7f03b4ff9640 1 -- 172.21.15.62:0/3444466988 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b80b0e30 con 0x55d9528b3010 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.780+0000 7f03c81d4740 1 -- 172.21.15.62:0/3444466988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b3010 msgr2=0x55d9528c2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.780+0000 7f03c81d4740 1 --2- 172.21.15.62:0/3444466988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b3010 0x55d9528c2250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f03b800c700 tx=0x7f03b809ab30 comp rx=0 tx=0).stop 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.780+0000 7f03c81d4740 1 -- 172.21.15.62:0/3444466988 shutdown_connections 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.780+0000 7f03c81d4740 1 --2- 172.21.15.62:0/3444466988 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b82e0 0x55d9528b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.780+0000 7f03c81d4740 1 --2- 172.21.15.62:0/3444466988 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b79a0 0x55d9528c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.780+0000 7f03c81d4740 1 --2- 172.21.15.62:0/3444466988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b3010 0x55d9528c2250 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.780+0000 7f03c81d4740 1 -- 172.21.15.62:0/3444466988 >> 172.21.15.62:0/3444466988 conn(0x55d9528cfc20 msgr2=0x55d9528b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.781+0000 7f03c81d4740 1 -- 172.21.15.62:0/3444466988 shutdown_connections 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.781+0000 7f03c81d4740 1 -- 172.21.15.62:0/3444466988 wait complete. 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.781+0000 7f03c81d4740 1 -- start start 2024-09-15T23:22:02.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.781+0000 7f03c81d4740 1 Processor -- start 2024-09-15T23:22:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.781+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b3010 0x55d9528c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.781+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.781+0000 7f03c81d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b82e0 0x55d9528c8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03c81d4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f03c01c02b0 con 0x55d9528b3010 2024-09-15T23:22:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03c81d4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f03c01c1720 con 0x55d9528b79a0 2024-09-15T23:22:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03c81d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f03c01c0980 con 0x55d9528b82e0 2024-09-15T23:22:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03c81d4740 10 client.0 _ll_register_callbacks cb 0x7f03b80b3070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03beffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b3010 0x55d9528c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03beffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b3010 0x55d9528c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54688/0 (socket says 172.21.15.62:54688) 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03beffd640 1 -- 172.21.15.62:0/457527992 learned_addr learned my addr 172.21.15.62:0/457527992 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03be7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03bf7fe640 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b82e0 0x55d9528c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03be7fc640 1 -- 172.21.15.62:0/457527992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b82e0 msgr2=0x55d9528c8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03be7fc640 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b82e0 0x55d9528c8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03be7fc640 1 -- 172.21.15.62:0/457527992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b3010 msgr2=0x55d9528c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03be7fc640 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b3010 0x55d9528c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03be7fc640 1 -- 172.21.15.62:0/457527992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9528c8c30 con 0x55d9528b79a0 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03be7fc640 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c7610 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f03b002aa30 tx=0x7f03b002a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35a79406c331195c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.782+0000 7f03a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03a67fc640 1 -- 172.21.15.62:0/457527992 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b0035050 con 0x55d9528b79a0 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03a67fc640 1 -- 172.21.15.62:0/457527992 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f03b0005040 con 0x55d9528b79a0 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03c81d4740 1 -- 172.21.15.62:0/457527992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03b8098410 con 0x55d9528b79a0 2024-09-15T23:22:02.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03a67fc640 1 -- 172.21.15.62:0/457527992 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f03b0063050 con 0x55d9528b79a0 2024-09-15T23:22:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03c81d4740 10 client.26432 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03c81d4740 1 -- 172.21.15.62:0/457527992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d952938950 con 0x55d9528b79a0 2024-09-15T23:22:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03a67fc640 1 -- 172.21.15.62:0/457527992 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03b00046c0 con 0x55d9528b79a0 2024-09-15T23:22:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03a67fc640 1 -- 172.21.15.62:0/457527992 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f03b0003710 con 0x55d9528b79a0 2024-09-15T23:22:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03c81d4740 10 client.26432 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03c81d4740 10 client.26432 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03c81d4740 20 client.26432 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:02.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03c81d4740 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9528bd370 0x55d95293b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03c81d4740 4 client.26432 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:22:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.783+0000 7f03c81d4740 1 -- 172.21.15.62:0/457527992 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f03b8098410 con 0x55d9528bd370 2024-09-15T23:22:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.784+0000 7f03beffd640 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9528bd370 0x55d95293b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.784+0000 7f03a67fc640 10 client.26432 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:02.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.784+0000 7f03beffd640 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9528bd370 0x55d95293b3a0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2c857ccd2da5ee6 server_cookie=decfa231a807ed7e in_seq=0 out_seq=0 2024-09-15T23:22:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.805+0000 7f03a67fc640 1 -- 172.21.15.62:0/457527992 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f03b8098410 con 0x55d9528bd370 2024-09-15T23:22:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.805+0000 7f03a67fc640 10 client.26432 handle_command_reply: tid=0 2024-09-15T23:22:02.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.805+0000 7f03c81d4740 1 client.26432 shutdown 2024-09-15T23:22:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.805+0000 7f03c81d4740 1 -- 172.21.15.62:0/457527992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 msgr2=0x55d9528c7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.806+0000 7f03c81d4740 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c7610 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f03b002aa30 tx=0x7f03b002a440 comp rx=0 tx=0).stop 2024-09-15T23:22:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.806+0000 7f03c81d4740 1 -- 172.21.15.62:0/457527992 shutdown_connections 2024-09-15T23:22:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.806+0000 7f03c81d4740 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d9528bd370 0x55d95293b3a0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.806+0000 7f03c81d4740 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d9528b82e0 0x55d9528c8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.806+0000 7f03c81d4740 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d9528b79a0 0x55d9528c7610 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f03b002aa30 tx=0x7f03b002a440 comp rx=0 tx=0).stop 2024-09-15T23:22:02.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.806+0000 7f03c81d4740 1 --2- 172.21.15.62:0/457527992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d9528b3010 0x55d9528c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.806+0000 7f03c81d4740 1 -- 172.21.15.62:0/457527992 >> 172.21.15.62:0/457527992 conn(0x55d9528cfc20 msgr2=0x55d9528b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.806+0000 7f03a67fc640 0 client.26432 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:02.808 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.807+0000 7f03c81d4740 1 -- 172.21.15.62:0/457527992 shutdown_connections 2024-09-15T23:22:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c81d4740 1 -- 172.21.15.62:0/457527992 wait complete. 2024-09-15T23:22:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c81d4740 20 client.26432 trim_cache size 0 max 16384 2024-09-15T23:22:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:22:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:22:02.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:22:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:22:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f03ac0a33d0 msgr2=0x7f03ac0a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.810 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f03ac0a33d0 0x7f03ac0a5890 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f03b0000c00 tx=0x7f03b0031540 comp rx=0 tx=0).stop 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 msgr2=0x7f03c0072e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 0x7f03c0072e00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f03b800eab0 tx=0x7f03b800ef70 comp rx=0 tx=0).stop 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 shutdown_connections 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f03ac0a33d0 0x7f03ac0a5890 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03c010e130 0x7f03c010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03c0076230 0x7f03c0073340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 --2- 172.21.15.62:0/4094215241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f03c0074c50 0x7f03c0072e00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.808+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 >> 172.21.15.62:0/4094215241 conn(0x7f03c00fd2e0 msgr2=0x7f03c01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.809+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 shutdown_connections 2024-09-15T23:22:02.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:02.809+0000 7f03c535f640 1 -- 172.21.15.62:0/4094215241 wait complete. 2024-09-15T23:22:03.467 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:22:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:03.982 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:03 smithi062 ceph-mon[30949]: pgmap v543: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 21 MiB/s wr, 616 op/s 2024-09-15T23:22:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:03 smithi067 ceph-mon[35665]: pgmap v543: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 21 MiB/s wr, 616 op/s 2024-09-15T23:22:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:03 smithi148 ceph-mon[35610]: pgmap v543: 129 pgs: 129 active+clean; 1.3 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 21 MiB/s wr, 616 op/s 2024-09-15T23:22:04.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.428+0000 7fe9d68e7640 1 -- 172.21.15.62:0/3624781728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d0109450 msgr2=0x7fe9d0110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.428+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/3624781728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d0109450 0x7fe9d0110250 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe9c001a910 tx=0x7fe9c003b470 comp rx=0 tx=0).stop 2024-09-15T23:22:04.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.433+0000 7fe9d68e7640 1 -- 172.21.15.62:0/3624781728 shutdown_connections 2024-09-15T23:22:04.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.433+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/3624781728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d011b140 0x7fe9d011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.433+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/3624781728 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9d0109830 0x7fe9d0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.433+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/3624781728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d0109450 0x7fe9d0110250 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.433+0000 7fe9d68e7640 1 -- 172.21.15.62:0/3624781728 >> 172.21.15.62:0/3624781728 conn(0x7fe9d006e140 msgr2=0x7fe9d006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:04.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.434+0000 7fe9d68e7640 1 -- 172.21.15.62:0/3624781728 shutdown_connections 2024-09-15T23:22:04.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.434+0000 7fe9d68e7640 1 -- 172.21.15.62:0/3624781728 wait complete. 2024-09-15T23:22:04.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.435+0000 7fe9d68e7640 1 Processor -- start 2024-09-15T23:22:04.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.435+0000 7fe9d68e7640 1 -- start start 2024-09-15T23:22:04.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.435+0000 7fe9d68e7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d0109450 0x7fe9d011a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.435+0000 7fe9d68e7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d0109830 0x7fe9d0116840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.435+0000 7fe9d68e7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9d011b140 0x7fe9d0116d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.435+0000 7fe9d68e7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d0113300 con 0x7fe9d0109830 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.435+0000 7fe9d68e7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d0113180 con 0x7fe9d0109450 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.435+0000 7fe9d68e7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d0113480 con 0x7fe9d011b140 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.436+0000 7fe9d4e84640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9d011b140 0x7fe9d0116d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.436+0000 7fe9d4e84640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9d011b140 0x7fe9d0116d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:42110/0 (socket says 172.21.15.62:42110) 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.436+0000 7fe9d4e84640 1 -- 172.21.15.62:0/2588347717 learned_addr learned my addr 172.21.15.62:0/2588347717 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.438+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d0109830 0x7fe9d0116840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.438+0000 7fe9d4e84640 1 -- 172.21.15.62:0/2588347717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d0109450 msgr2=0x7fe9d011a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.438+0000 7fe9d4e84640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d0109450 0x7fe9d011a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.438+0000 7fe9d4e84640 1 -- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d0109830 msgr2=0x7fe9d0116840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.438+0000 7fe9d4e84640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d0109830 0x7fe9d0116840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.438+0000 7fe9d4e84640 1 -- 172.21.15.62:0/2588347717 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d0117370 con 0x7fe9d011b140 2024-09-15T23:22:04.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.439+0000 7fe9d4e84640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9d011b140 0x7fe9d0116d80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800cae0 tx=0x7fe9c800a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:04.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.441+0000 7fe9cd7fa640 1 -- 172.21.15.62:0/2588347717 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c800ae30 con 0x7fe9d011b140 2024-09-15T23:22:04.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.441+0000 7fe9d68e7640 1 -- 172.21.15.62:0/2588347717 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9d0117660 con 0x7fe9d011b140 2024-09-15T23:22:04.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.442+0000 7fe9d68e7640 1 -- 172.21.15.62:0/2588347717 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9d007a7c0 con 0x7fe9d011b140 2024-09-15T23:22:04.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.442+0000 7fe9cd7fa640 1 -- 172.21.15.62:0/2588347717 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c8005020 con 0x7fe9d011b140 2024-09-15T23:22:04.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.442+0000 7fe9cd7fa640 1 -- 172.21.15.62:0/2588347717 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c800f690 con 0x7fe9d011b140 2024-09-15T23:22:04.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.446+0000 7fe9d8cc0740 1 Processor -- start 2024-09-15T23:22:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.448+0000 7fe9d8cc0740 1 -- start start 2024-09-15T23:22:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.448+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdae880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.448+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.448+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdbba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.448+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bc4cd142e0 con 0x55bc4cdae4a0 2024-09-15T23:22:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.448+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55bc4ccbbce0 con 0x55bc4cdb9680 2024-09-15T23:22:04.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.448+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bc4cd102a0 con 0x55bc4cdaedc0 2024-09-15T23:22:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.451+0000 7fe9cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdbba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.452+0000 7fe9cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdbba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36486/0 (socket says 172.21.15.62:36486) 2024-09-15T23:22:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.452+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/2435786860 learned_addr learned my addr 172.21.15.62:0/2435786860 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:04.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.452+0000 7fe9cffff640 1 --2- 172.21.15.62:0/2435786860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.452+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/2435786860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 msgr2=0x55bc4cdb9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.452+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/2435786860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.453+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/2435786860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdae4a0 msgr2=0x55bc4cdae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.453+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/2435786860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.453+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/2435786860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc4cca0440 con 0x55bc4cdb9680 2024-09-15T23:22:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.451+0000 7fe9cd7fa640 1 -- 172.21.15.62:0/2588347717 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe9c800f950 con 0x7fe9d011b140 2024-09-15T23:22:04.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.453+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/2435786860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdbba70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400cce0 tx=0x7fe9c40117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1231ac30ad78ebd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:04.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.455+0000 7fe9cd7fa640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9b80a3450 0x7fe9b80a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.455+0000 7fe9b67fc640 1 -- 172.21.15.62:0/2435786860 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c4004030 con 0x55bc4cdb9680 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.455+0000 7fe9b67fc640 1 -- 172.21.15.62:0/2435786860 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c40041d0 con 0x55bc4cdb9680 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.455+0000 7fe9cffff640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9b80a3450 0x7fe9b80a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.456+0000 7fe9cd7fa640 1 -- 172.21.15.62:0/2588347717 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe9c80d0100 con 0x7fe9d011b140 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.456+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2435786860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 msgr2=0x55bc4cdbba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.456+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2435786860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdbba70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe9c400cce0 tx=0x7fe9c40117b0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.456+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2435786860 shutdown_connections 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.456+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2435786860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdbba70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.456+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2435786860 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.456+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2435786860 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.456+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2435786860 >> 172.21.15.62:0/2435786860 conn(0x55bc4cdaaa30 msgr2=0x55bc4cdaae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:04.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.461+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2435786860 shutdown_connections 2024-09-15T23:22:04.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.462+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2435786860 wait complete. 2024-09-15T23:22:04.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.462+0000 7fe9d8cc0740 1 -- start start 2024-09-15T23:22:04.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.462+0000 7fe9d8cc0740 1 Processor -- start 2024-09-15T23:22:04.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.463+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdb8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.463+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.463+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdb6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.463+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bc4cdbc2a0 con 0x55bc4cdaedc0 2024-09-15T23:22:04.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.463+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55bc4cccf050 con 0x55bc4cdb9680 2024-09-15T23:22:04.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.463+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bc4cdbc420 con 0x55bc4cdae4a0 2024-09-15T23:22:04.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.463+0000 7fe9d8cc0740 10 client.0 _ll_register_callbacks cb 0x55bc4cd0f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:04.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.464+0000 7fe9cffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdb8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.464+0000 7fe9cffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdb8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:42126/0 (socket says 172.21.15.62:42126) 2024-09-15T23:22:04.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.464+0000 7fe9cffff640 1 -- 172.21.15.62:0/3712484865 learned_addr learned my addr 172.21.15.62:0/3712484865 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:04.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.464+0000 7fe9d4e84640 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdb6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.466+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.467+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/3712484865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdae4a0 msgr2=0x55bc4cdb8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:04.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.467+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdb8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.467+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/3712484865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 msgr2=0x55bc4cdb6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:04.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.467+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdb6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.467+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/3712484865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc4ca853c0 con 0x55bc4cdaedc0 2024-09-15T23:22:04.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.467+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb6220 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4005780 tx=0x7fe9c400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c92175ea430cdd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:04.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.469+0000 7fe9af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:22:04.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.469+0000 7fe9af7fe640 1 -- 172.21.15.62:0/3712484865 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c400c930 con 0x55bc4cdaedc0 2024-09-15T23:22:04.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.469+0000 7fe9af7fe640 1 -- 172.21.15.62:0/3712484865 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe9c4004030 con 0x55bc4cdaedc0 2024-09-15T23:22:04.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.470+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/3712484865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bc4cd21850 con 0x55bc4cdaedc0 2024-09-15T23:22:04.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.472+0000 7fe9cffff640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9b80a3450 0x7fe9b80a5910 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0003820 tx=0x7fe9c0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:04.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.472+0000 7fe9af7fe640 1 -- 172.21.15.62:0/3712484865 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe9c402c050 con 0x55bc4cdaedc0 2024-09-15T23:22:04.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.472+0000 7fe9d8cc0740 10 client.16869 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:04.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.472+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/3712484865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bc4cdc7d80 con 0x55bc4cdaedc0 2024-09-15T23:22:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.475+0000 7fe9af7fe640 1 -- 172.21.15.62:0/3712484865 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55bc4cd21850 con 0x55bc4cdaedc0 2024-09-15T23:22:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.475+0000 7fe9af7fe640 1 -- 172.21.15.62:0/3712484865 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7fe9c4004b10 con 0x55bc4cdaedc0 2024-09-15T23:22:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.476+0000 7fe9d8cc0740 10 client.16869 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.476+0000 7fe9d8cc0740 10 client.16869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.476+0000 7fe9d8cc0740 20 client.16869 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:04.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.476+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bc4cdc9690 0x55bc4cdcbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.476+0000 7fe9d8cc0740 4 client.16869 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:04.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.476+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/3712484865 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc4cd1bd60 con 0x55bc4cdc9690 2024-09-15T23:22:04.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.482+0000 7fe9cffff640 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bc4cdc9690 0x55bc4cdcbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.490+0000 7fe9af7fe640 10 client.16869 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:04.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.490+0000 7fe9cffff640 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bc4cdc9690 0x55bc4cdcbb10 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=211672e88d4ad5f2 server_cookie=71a18a040253adf5 in_seq=0 out_seq=0 2024-09-15T23:22:04.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.491+0000 7fe9af7fe640 1 -- 172.21.15.62:0/3712484865 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bc4cd1bd60 con 0x55bc4cdc9690 2024-09-15T23:22:04.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.491+0000 7fe9af7fe640 10 client.16869 handle_command_reply: tid=0 2024-09-15T23:22:04.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.491+0000 7fe9d8cc0740 1 client.16869 shutdown 2024-09-15T23:22:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.495+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/3712484865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaedc0 msgr2=0x55bc4cdb6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.495+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb6220 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe9c4005780 tx=0x7fe9c400f4a0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.495+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/3712484865 shutdown_connections 2024-09-15T23:22:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.495+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bc4cdc9690 0x55bc4cdcbb10 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.495+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdb9680 0x55bc4cdb6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.495+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb6220 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.495+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/3712484865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdb8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.495+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/3712484865 >> 172.21.15.62:0/3712484865 conn(0x55bc4cdaaa30 msgr2=0x55bc4cdb0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:04.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.495+0000 7fe9af7fe640 0 client.16869 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:04.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.496+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/3712484865 shutdown_connections 2024-09-15T23:22:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.499+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/3712484865 wait complete. 2024-09-15T23:22:04.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.500+0000 7fe9d8cc0740 20 client.16869 trim_cache size 0 max 16384 2024-09-15T23:22:04.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.533+0000 7fe9d8cc0740 1 Processor -- start 2024-09-15T23:22:04.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.533+0000 7fe9d8cc0740 1 -- start start 2024-09-15T23:22:04.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.533+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdc55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.533+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdb61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.533+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.533+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc012520 con 0x55bc4cdae4a0 2024-09-15T23:22:04.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.533+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc00fdf0 con 0x55bc4cdaaa30 2024-09-15T23:22:04.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.533+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc00dc40 con 0x55bc4cdaedc0 2024-09-15T23:22:04.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9d4e84640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdc55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9d4e84640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:42134/0 (socket says 172.21.15.62:42134) 2024-09-15T23:22:04.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9d4e84640 1 -- 172.21.15.62:0/566799478 learned_addr learned my addr 172.21.15.62:0/566799478 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:04.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdc55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36506/0 (socket says 172.21.15.62:36506) 2024-09-15T23:22:04.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9cffff640 1 -- 172.21.15.62:0/566799478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 msgr2=0x55bc4cdb17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:04.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9cffff640 1 --2- 172.21.15.62:0/566799478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9cffff640 1 -- 172.21.15.62:0/566799478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdae4a0 msgr2=0x55bc4cdb61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:04.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9cffff640 1 --2- 172.21.15.62:0/566799478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdb61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.534+0000 7fe9cffff640 1 -- 172.21.15.62:0/566799478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc4cdb4770 con 0x55bc4cdaaa30 2024-09-15T23:22:04.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.535+0000 7fe9cffff640 1 --2- 172.21.15.62:0/566799478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdc55e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55bc4cdc9a70 tx=0x7fe9c0059000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa1c4c22c727bdce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:04.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.535+0000 7fe9b5ffb640 1 -- 172.21.15.62:0/566799478 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0056040 con 0x55bc4cdaaa30 2024-09-15T23:22:04.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.535+0000 7fe9b5ffb640 1 -- 172.21.15.62:0/566799478 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c0069070 con 0x55bc4cdaaa30 2024-09-15T23:22:04.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.535+0000 7fe9b5ffb640 1 -- 172.21.15.62:0/566799478 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c0063720 con 0x55bc4cdaaa30 2024-09-15T23:22:04.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.536+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/566799478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdaaa30 msgr2=0x55bc4cdc55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.536+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/566799478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdc55e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x55bc4cdc9a70 tx=0x7fe9c0059000 comp rx=0 tx=0).stop 2024-09-15T23:22:04.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.536+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/566799478 shutdown_connections 2024-09-15T23:22:04.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.536+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/566799478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4cdb17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.536+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/566799478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdb61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.536+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/566799478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdc55e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.536+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/566799478 >> 172.21.15.62:0/566799478 conn(0x55bc4cdb68c0 msgr2=0x55bc4cdb1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:04.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.536+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/566799478 shutdown_connections 2024-09-15T23:22:04.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.536+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/566799478 wait complete. 2024-09-15T23:22:04.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.537+0000 7fe9d8cc0740 1 -- start start 2024-09-15T23:22:04.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.537+0000 7fe9d8cc0740 1 Processor -- start 2024-09-15T23:22:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdbccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdbd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9d8cc0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4ce323a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc00e2b0 con 0x55bc4cdaaa30 2024-09-15T23:22:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc00f720 con 0x55bc4cdae4a0 2024-09-15T23:22:04.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9d8cc0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe9bc00e980 con 0x55bc4cdaedc0 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdbd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9d8cc0740 10 client.0 _ll_register_callbacks cb 0x7fe9c400cbb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9cffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdbccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdbd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36510/0 (socket says 172.21.15.62:36510) 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/2137143764 learned_addr learned my addr 172.21.15.62:0/2137143764 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9d4e84640 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4ce323a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/2137143764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 msgr2=0x55bc4ce323a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4ce323a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/2137143764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaaa30 msgr2=0x55bc4cdbccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdbccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.538+0000 7fe9cf7fe640 1 -- 172.21.15.62:0/2137143764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc4ce32910 con 0x55bc4cdae4a0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.539+0000 7fe9cf7fe640 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdbd1f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55bc4cdb5ab0 tx=0x7fe9c4005640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7774f4ca611424d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.539+0000 7fe9af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.539+0000 7fe9af7fe640 1 -- 172.21.15.62:0/2137143764 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c400c930 con 0x55bc4cdae4a0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.539+0000 7fe9af7fe640 1 -- 172.21.15.62:0/2137143764 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe9c400fd50 con 0x55bc4cdae4a0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.539+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2137143764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9c0069070 con 0x55bc4cdae4a0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.539+0000 7fe9af7fe640 1 -- 172.21.15.62:0/2137143764 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe9c403e050 con 0x55bc4cdae4a0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.539+0000 7fe9d8cc0740 10 client.26447 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.539+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2137143764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bc4ce33c20 con 0x55bc4cdae4a0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.540+0000 7fe9af7fe640 1 -- 172.21.15.62:0/2137143764 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9c4004b60 con 0x55bc4cdae4a0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.540+0000 7fe9af7fe640 1 -- 172.21.15.62:0/2137143764 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7fe9c4039030 con 0x55bc4cdae4a0 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.540+0000 7fe9d8cc0740 10 client.26447 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.540+0000 7fe9d8cc0740 10 client.26447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:04.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.540+0000 7fe9d8cc0740 20 client.26447 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.540+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bc4cdb9680 0x55bc4ce363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.540+0000 7fe9d8cc0740 4 client.26447 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:22:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.540+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2137143764 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe9c0069070 con 0x55bc4cdb9680 2024-09-15T23:22:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.540+0000 7fe9cffff640 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bc4cdb9680 0x55bc4ce363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.541+0000 7fe9cffff640 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bc4cdb9680 0x55bc4ce363a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cac921cb8a88974 server_cookie=cb2c1366c8ae7005 in_seq=0 out_seq=0 2024-09-15T23:22:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.541+0000 7fe9af7fe640 10 client.26447 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.541+0000 7fe9af7fe640 1 -- 172.21.15.62:0/2137143764 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fe9c0069070 con 0x55bc4cdb9680 2024-09-15T23:22:04.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.541+0000 7fe9af7fe640 10 client.26447 handle_command_reply: tid=0 2024-09-15T23:22:04.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.541+0000 7fe9d8cc0740 1 client.26447 shutdown 2024-09-15T23:22:04.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.542+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2137143764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdae4a0 msgr2=0x55bc4cdbd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.542+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdbd1f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x55bc4cdb5ab0 tx=0x7fe9c4005640 comp rx=0 tx=0).stop 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.542+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2137143764 shutdown_connections 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.542+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bc4cdb9680 0x55bc4ce363a0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.542+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bc4cdaedc0 0x55bc4ce323a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.542+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bc4cdae4a0 0x55bc4cdbd1f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.542+0000 7fe9d8cc0740 1 --2- 172.21.15.62:0/2137143764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bc4cdaaa30 0x55bc4cdbccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.542+0000 7fe9af7fe640 0 client.26447 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.542+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2137143764 >> 172.21.15.62:0/2137143764 conn(0x55bc4cdb68c0 msgr2=0x55bc4cdb18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.543+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2137143764 shutdown_connections 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.543+0000 7fe9d8cc0740 1 -- 172.21.15.62:0/2137143764 wait complete. 2024-09-15T23:22:04.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.543+0000 7fe9d8cc0740 20 client.26447 trim_cache size 0 max 16384 2024-09-15T23:22:04.549 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:22:04.549 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:22:04.549 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:22:04.549 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:22:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.544+0000 7fe9d68e7640 1 -- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9b80a3450 msgr2=0x7fe9b80a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.544+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9b80a3450 0x7fe9b80a5910 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0003820 tx=0x7fe9c0000bc0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.544+0000 7fe9d68e7640 1 -- 172.21.15.62:0/2588347717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9d011b140 msgr2=0x7fe9d0116d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.544+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9d011b140 0x7fe9d0116d80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800cae0 tx=0x7fe9c800a250 comp rx=0 tx=0).stop 2024-09-15T23:22:04.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.545+0000 7fe9d68e7640 1 -- 172.21.15.62:0/2588347717 shutdown_connections 2024-09-15T23:22:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.545+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9b80a3450 0x7fe9b80a5910 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.545+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9d011b140 0x7fe9d0116d80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.545+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9d0109830 0x7fe9d0116840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.545+0000 7fe9d68e7640 1 --2- 172.21.15.62:0/2588347717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d0109450 0x7fe9d011a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.545+0000 7fe9d68e7640 1 -- 172.21.15.62:0/2588347717 >> 172.21.15.62:0/2588347717 conn(0x7fe9d006e140 msgr2=0x7fe9d00722e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.545+0000 7fe9d68e7640 1 -- 172.21.15.62:0/2588347717 shutdown_connections 2024-09-15T23:22:04.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:04.545+0000 7fe9d68e7640 1 -- 172.21.15.62:0/2588347717 wait complete. 2024-09-15T23:22:04.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 345.0 s, 1130.6 tps, lat 89.065 ms stddev 104.928 2024-09-15T23:22:05.707 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:22:05.707 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:22:05.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:05 smithi062 ceph-mon[30949]: pgmap v544: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 MiB/s wr, 627 op/s 2024-09-15T23:22:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:05 smithi067 ceph-mon[35665]: pgmap v544: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 MiB/s wr, 627 op/s 2024-09-15T23:22:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:05 smithi148 ceph-mon[35610]: pgmap v544: 129 pgs: 129 active+clean; 1.3 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 MiB/s wr, 627 op/s 2024-09-15T23:22:06.147 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:06.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.896+0000 7fd27ccd8640 1 -- 172.21.15.62:0/1172148308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd278109450 msgr2=0x7fd278110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:06.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.896+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/1172148308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd278109450 0x7fd278110250 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd27001a910 tx=0x7fd27003b450 comp rx=0 tx=0).stop 2024-09-15T23:22:06.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.899+0000 7fd27ccd8640 1 -- 172.21.15.62:0/1172148308 shutdown_connections 2024-09-15T23:22:06.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.899+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/1172148308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd27811b140 0x7fd27811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:06.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.899+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/1172148308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd278109830 0x7fd278109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:06.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.899+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/1172148308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd278109450 0x7fd278110250 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:06.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.899+0000 7fd27ccd8640 1 -- 172.21.15.62:0/1172148308 >> 172.21.15.62:0/1172148308 conn(0x7fd27806e140 msgr2=0x7fd27806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:06.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.906+0000 7fd27ccd8640 1 -- 172.21.15.62:0/1172148308 shutdown_connections 2024-09-15T23:22:06.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.906+0000 7fd27ccd8640 1 -- 172.21.15.62:0/1172148308 wait complete. 2024-09-15T23:22:06.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.907+0000 7fd27ccd8640 1 Processor -- start 2024-09-15T23:22:06.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.909+0000 7fd27ccd8640 1 -- start start 2024-09-15T23:22:06.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.909+0000 7fd27ccd8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd278109450 0x7fd278116850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:06.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.909+0000 7fd27ccd8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278109830 0x7fd278116d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:06.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.909+0000 7fd27ccd8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd27811b140 0x7fd27811a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:06.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.909+0000 7fd27ccd8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd278113300 con 0x7fd278109830 2024-09-15T23:22:06.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.909+0000 7fd27ccd8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd278113180 con 0x7fd278109450 2024-09-15T23:22:06.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.909+0000 7fd27ccd8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd278113480 con 0x7fd27811b140 2024-09-15T23:22:06.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.911+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278109830 0x7fd278116d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:06.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.911+0000 7fd275d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278109830 0x7fd278116d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54748/0 (socket says 172.21.15.62:54748) 2024-09-15T23:22:06.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.911+0000 7fd275d9b640 1 -- 172.21.15.62:0/271130869 learned_addr learned my addr 172.21.15.62:0/271130869 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:06.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.912+0000 7fd27659c640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd278109450 0x7fd278116850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:06.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.912+0000 7fd276d9d640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd27811b140 0x7fd27811a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:06.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.913+0000 7fd27659c640 1 -- 172.21.15.62:0/271130869 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd27811b140 msgr2=0x7fd27811a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:06.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.913+0000 7fd27659c640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd27811b140 0x7fd27811a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:06.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.913+0000 7fd27659c640 1 -- 172.21.15.62:0/271130869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278109830 msgr2=0x7fd278116d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:06.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.913+0000 7fd27659c640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278109830 0x7fd278116d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:06.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.913+0000 7fd27659c640 1 -- 172.21.15.62:0/271130869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27811a7f0 con 0x7fd278109450 2024-09-15T23:22:06.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.913+0000 7fd27659c640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd278109450 0x7fd278116850 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd270044040 tx=0x7fd2700050b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:06.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.914+0000 7fd26f7fe640 1 -- 172.21.15.62:0/271130869 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270051070 con 0x7fd278109450 2024-09-15T23:22:06.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.914+0000 7fd26f7fe640 1 -- 172.21.15.62:0/271130869 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd270040c00 con 0x7fd278109450 2024-09-15T23:22:06.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.914+0000 7fd26f7fe640 1 -- 172.21.15.62:0/271130869 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270042930 con 0x7fd278109450 2024-09-15T23:22:06.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.916+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27811aa20 con 0x7fd278109450 2024-09-15T23:22:06.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.916+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27811af10 con 0x7fd278109450 2024-09-15T23:22:06.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.919+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd27806f6c0 con 0x7fd278109450 2024-09-15T23:22:06.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.924+0000 7fd26f7fe640 1 -- 172.21.15.62:0/271130869 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd270054070 con 0x7fd278109450 2024-09-15T23:22:06.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.925+0000 7fd26f7fe640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2640a3450 0x7fd2640a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:06.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.925+0000 7fd26f7fe640 1 -- 172.21.15.62:0/271130869 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd2701017c0 con 0x7fd278109450 2024-09-15T23:22:06.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.928+0000 7fd26f7fe640 1 -- 172.21.15.62:0/271130869 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd2700ce2a0 con 0x7fd278109450 2024-09-15T23:22:06.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.929+0000 7fd275d9b640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2640a3450 0x7fd2640a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:06.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:06.930+0000 7fd275d9b640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2640a3450 0x7fd2640a5910 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd278071980 tx=0x7fd26000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:07.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.320+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd27810d3b0 con 0x7fd278109450 2024-09-15T23:22:07.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.321+0000 7fd26f7fe640 1 -- 172.21.15.62:0/271130869 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11904 (secure 0 0 0) 0x7fd27810d3b0 con 0x7fd278109450 2024-09-15T23:22:07.324 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:22:07.324 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":32,"btime":"2024-09-15T23:21:52:478522+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:21:52.380746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:22:07.325 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 32 2024-09-15T23:22:07.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.330+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2640a3450 msgr2=0x7fd2640a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:07.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.330+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2640a3450 0x7fd2640a5910 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd278071980 tx=0x7fd26000a380 comp rx=0 tx=0).stop 2024-09-15T23:22:07.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.330+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd278109450 msgr2=0x7fd278116850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:07.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.330+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd278109450 0x7fd278116850 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd270044040 tx=0x7fd2700050b0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.332+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 shutdown_connections 2024-09-15T23:22:07.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.332+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2640a3450 0x7fd2640a5910 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.332+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd27811b140 0x7fd27811a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.332+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd278109830 0x7fd278116d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.332+0000 7fd27ccd8640 1 --2- 172.21.15.62:0/271130869 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd278109450 0x7fd278116850 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd270044040 tx=0x7fd2700050b0 comp rx=0 tx=0).stop 2024-09-15T23:22:07.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.332+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 >> 172.21.15.62:0/271130869 conn(0x7fd27806e140 msgr2=0x7fd2780722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:07.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.339+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 shutdown_connections 2024-09-15T23:22:07.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:07.340+0000 7fd27ccd8640 1 -- 172.21.15.62:0/271130869 wait complete. 2024-09-15T23:22:07.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:07 smithi062 ceph-mon[30949]: pgmap v545: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 544 op/s 2024-09-15T23:22:07.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:07 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/271130869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:08.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:07 smithi067 ceph-mon[35665]: pgmap v545: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 544 op/s 2024-09-15T23:22:08.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:07 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/271130869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:07 smithi148 ceph-mon[35610]: pgmap v545: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 MiB/s wr, 544 op/s 2024-09-15T23:22:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:07 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/271130869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:09.234 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:22:09.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 350.0 s, 926.4 tps, lat 108.971 ms stddev 143.493 2024-09-15T23:22:09.617 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:09.928 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:09 smithi062 ceph-mon[30949]: pgmap v546: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 16 MiB/s wr, 544 op/s 2024-09-15T23:22:10.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:09 smithi067 ceph-mon[35665]: pgmap v546: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 16 MiB/s wr, 544 op/s 2024-09-15T23:22:10.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:09 smithi148 ceph-mon[35610]: pgmap v546: 129 pgs: 129 active+clean; 1.3 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 16 MiB/s wr, 544 op/s 2024-09-15T23:22:10.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.197+0000 7f919b2bd640 1 -- 172.21.15.62:0/460008722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941009b0 msgr2=0x7f9194100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.197+0000 7f919b2bd640 1 --2- 172.21.15.62:0/460008722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941009b0 0x7f9194100d90 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9190009900 tx=0x7f919002a410 comp rx=0 tx=0).stop 2024-09-15T23:22:10.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.197+0000 7f919b2bd640 1 -- 172.21.15.62:0/460008722 shutdown_connections 2024-09-15T23:22:10.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.197+0000 7f919b2bd640 1 --2- 172.21.15.62:0/460008722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f919410e4d0 0x7f91941108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.197+0000 7f919b2bd640 1 --2- 172.21.15.62:0/460008722 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9194101360 0x7f919410df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.197+0000 7f919b2bd640 1 --2- 172.21.15.62:0/460008722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941009b0 0x7f9194100d90 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.197+0000 7f919b2bd640 1 -- 172.21.15.62:0/460008722 >> 172.21.15.62:0/460008722 conn(0x7f91940fc7d0 msgr2=0x7f91940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.197+0000 7f919b2bd640 1 -- 172.21.15.62:0/460008722 shutdown_connections 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.197+0000 7f919b2bd640 1 -- 172.21.15.62:0/460008722 wait complete. 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.198+0000 7f919b2bd640 1 Processor -- start 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.199+0000 7f919b2bd640 1 -- start start 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.200+0000 7f919b2bd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941009b0 0x7f9194107630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.200+0000 7f919b2bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9194101360 0x7f9194107b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.200+0000 7f919b2bd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f919410e4d0 0x7f9194102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.200+0000 7f919b2bd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9194114230 con 0x7f91941009b0 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919b2bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91941140b0 con 0x7f9194101360 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919985a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f919410e4d0 0x7f9194102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919985a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f919410e4d0 0x7f9194102b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:42176/0 (socket says 172.21.15.62:42176) 2024-09-15T23:22:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919985a640 1 -- 172.21.15.62:0/1763747345 learned_addr learned my addr 172.21.15.62:0/1763747345 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919b2bd640 1 -- 172.21.15.62:0/1763747345 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f91941143b0 con 0x7f919410e4d0 2024-09-15T23:22:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919985a640 1 -- 172.21.15.62:0/1763747345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9194101360 msgr2=0x7f9194107b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:22:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919985a640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9194101360 0x7f9194107b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919985a640 1 -- 172.21.15.62:0/1763747345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941009b0 msgr2=0x7f9194107630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919985a640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941009b0 0x7f9194107630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.201+0000 7f919985a640 1 -- 172.21.15.62:0/1763747345 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91941033d0 con 0x7f919410e4d0 2024-09-15T23:22:10.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.202+0000 7f919985a640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f919410e4d0 0x7f9194102b10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f918c00eb10 tx=0x7f918c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:10.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.202+0000 7f918a7fc640 1 -- 172.21.15.62:0/1763747345 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c004b20 con 0x7f919410e4d0 2024-09-15T23:22:10.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.203+0000 7f919b2bd640 1 -- 172.21.15.62:0/1763747345 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91941036c0 con 0x7f919410e4d0 2024-09-15T23:22:10.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.203+0000 7f919b2bd640 1 -- 172.21.15.62:0/1763747345 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f919410d520 con 0x7f919410e4d0 2024-09-15T23:22:10.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.203+0000 7f918a7fc640 1 -- 172.21.15.62:0/1763747345 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f918c004cc0 con 0x7f919410e4d0 2024-09-15T23:22:10.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.203+0000 7f918a7fc640 1 -- 172.21.15.62:0/1763747345 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c010790 con 0x7f919410e4d0 2024-09-15T23:22:10.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.206+0000 7f918a7fc640 1 -- 172.21.15.62:0/1763747345 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f918c005020 con 0x7f919410e4d0 2024-09-15T23:22:10.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.207+0000 7f918a7fc640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91800a32c0 0x7f91800a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.207+0000 7f918a7fc640 1 -- 172.21.15.62:0/1763747345 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f918c0d0200 con 0x7f919410e4d0 2024-09-15T23:22:10.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.207+0000 7f9199059640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91800a32c0 0x7f91800a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.208+0000 7f919d696740 1 Processor -- start 2024-09-15T23:22:10.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.208+0000 7f919d696740 1 -- start start 2024-09-15T23:22:10.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.208+0000 7f9199059640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91800a32c0 0x7f91800a5780 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f919002a930 tx=0x7f9190000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:10.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.208+0000 7f919d696740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a727683b40 0x55a727682090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.208+0000 7f919d696740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a727682a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f9199059640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a727683b40 0x55a727682090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f9199059640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a727683b40 0x55a727682090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54782/0 (socket says 172.21.15.62:54782) 2024-09-15T23:22:10.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f9199059640 1 -- 172.21.15.62:0/3731948803 learned_addr learned my addr 172.21.15.62:0/3731948803 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:10.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f919d696740 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a72768d370 0x55a72768f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f919d696740 1 -- 172.21.15.62:0/3731948803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a7275e82f0 con 0x55a727683b40 2024-09-15T23:22:10.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f919d696740 1 -- 172.21.15.62:0/3731948803 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a72758fcf0 con 0x55a7276825d0 2024-09-15T23:22:10.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f919d696740 1 -- 172.21.15.62:0/3731948803 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a7275e42b0 con 0x55a72768d370 2024-09-15T23:22:10.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f9198858640 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a727682a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f919985a640 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a72768d370 0x55a72768f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f9199059640 1 -- 172.21.15.62:0/3731948803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a72768d370 msgr2=0x55a72768f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:10.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f9199059640 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a72768d370 0x55a72768f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f9199059640 1 -- 172.21.15.62:0/3731948803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 msgr2=0x55a727682a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:10.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f9199059640 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a727682a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.209+0000 7f9199059640 1 -- 172.21.15.62:0/3731948803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7275743a0 con 0x55a727683b40 2024-09-15T23:22:10.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.210+0000 7f9198858640 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a727682a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:10.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.210+0000 7f9199059640 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a727683b40 0x55a727682090 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9190031850 tx=0x7f919003a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b149537144e7f65a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:10.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.210+0000 7f917f7fe640 1 -- 172.21.15.62:0/3731948803 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190040070 con 0x55a727683b40 2024-09-15T23:22:10.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.210+0000 7f917f7fe640 1 -- 172.21.15.62:0/3731948803 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f919004c070 con 0x55a727683b40 2024-09-15T23:22:10.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.210+0000 7f917f7fe640 1 -- 172.21.15.62:0/3731948803 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190035040 con 0x55a727683b40 2024-09-15T23:22:10.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.211+0000 7f919d696740 1 -- 172.21.15.62:0/3731948803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a727683b40 msgr2=0x55a727682090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.211+0000 7f919d696740 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a727683b40 0x55a727682090 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9190031850 tx=0x7f919003a000 comp rx=0 tx=0).stop 2024-09-15T23:22:10.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.211+0000 7f919d696740 1 -- 172.21.15.62:0/3731948803 shutdown_connections 2024-09-15T23:22:10.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.211+0000 7f919d696740 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a72768d370 0x55a72768f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.211+0000 7f919d696740 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a727682a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.211+0000 7f919d696740 1 --2- 172.21.15.62:0/3731948803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a727683b40 0x55a727682090 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.211+0000 7f919d696740 1 -- 172.21.15.62:0/3731948803 >> 172.21.15.62:0/3731948803 conn(0x55a72764b310 msgr2=0x55a72767ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:10.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.211+0000 7f919d696740 1 -- 172.21.15.62:0/3731948803 shutdown_connections 2024-09-15T23:22:10.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.211+0000 7f919d696740 1 -- 172.21.15.62:0/3731948803 wait complete. 2024-09-15T23:22:10.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.212+0000 7f919d696740 1 -- start start 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.212+0000 7f919d696740 1 Processor -- start 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919d696740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72768cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919d696740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a727689f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919d696740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72768a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919d696740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a727690010 con 0x55a72768d370 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919d696740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a7275a3060 con 0x55a7276825d0 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919d696740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a727690190 con 0x55a727683b40 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919d696740 10 client.0 _ll_register_callbacks cb 0x55a7272f68c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919985a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72768a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f9198858640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a727689f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f9199059640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72768cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919985a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72768a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54798/0 (socket says 172.21.15.62:54798) 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919985a640 1 -- 172.21.15.62:0/4179976380 learned_addr learned my addr 172.21.15.62:0/4179976380 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919985a640 1 -- 172.21.15.62:0/4179976380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 msgr2=0x55a727689f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919985a640 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a727689f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919985a640 1 -- 172.21.15.62:0/4179976380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 msgr2=0x55a72768cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919985a640 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72768cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f919985a640 1 -- 172.21.15.62:0/4179976380 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7273593c0 con 0x55a72768d370 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f9198858640 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a727689f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.213+0000 7f9199059640 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72768cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.214+0000 7f919985a640 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72768a4f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f918c0049d0 tx=0x7f918c09bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e38734e175286935 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:10.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.214+0000 7f917cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.214+0000 7f917cff9640 1 -- 172.21.15.62:0/4179976380 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c0b4070 con 0x55a72768d370 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.214+0000 7f917cff9640 1 -- 172.21.15.62:0/4179976380 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f918c0b5260 con 0x55a72768d370 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.214+0000 7f919d696740 1 -- 172.21.15.62:0/4179976380 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f919004c070 con 0x55a72768d370 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.214+0000 7f917cff9640 1 -- 172.21.15.62:0/4179976380 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f918c016530 con 0x55a72768d370 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.214+0000 7f919d696740 10 client.16887 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.214+0000 7f919d696740 1 -- 172.21.15.62:0/4179976380 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a7276929f0 con 0x55a72768d370 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.215+0000 7f917cff9640 1 -- 172.21.15.62:0/4179976380 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f918c07e070 con 0x55a72768d370 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.215+0000 7f917cff9640 1 -- 172.21.15.62:0/4179976380 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f918c09b8f0 con 0x55a72768d370 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.215+0000 7f919d696740 10 client.16887 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.215+0000 7f919d696740 10 client.16887 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.215+0000 7f919d696740 20 client.16887 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.215+0000 7f919d696740 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a727694440 0x55a7276968a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.215+0000 7f919d696740 4 client.16887 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.215+0000 7f919d696740 1 -- 172.21.15.62:0/4179976380 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7275f5860 con 0x55a727694440 2024-09-15T23:22:10.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.215+0000 7f9199059640 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a727694440 0x55a7276968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.216+0000 7f917cff9640 10 client.16887 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.216+0000 7f9199059640 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a727694440 0x55a7276968a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=33d747d6a5fe3ffa server_cookie=5aa57c8d6293ccfb in_seq=0 out_seq=0 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.216+0000 7f917cff9640 1 -- 172.21.15.62:0/4179976380 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a7275f5860 con 0x55a727694440 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.216+0000 7f917cff9640 10 client.16887 handle_command_reply: tid=0 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.216+0000 7f919d696740 1 client.16887 shutdown 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 -- 172.21.15.62:0/4179976380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 msgr2=0x55a72768a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72768a4f0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f918c0049d0 tx=0x7f918c09bec0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 -- 172.21.15.62:0/4179976380 shutdown_connections 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a727694440 0x55a7276968a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72768a4f0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a727689f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 --2- 172.21.15.62:0/4179976380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72768cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 -- 172.21.15.62:0/4179976380 >> 172.21.15.62:0/4179976380 conn(0x55a72764b310 msgr2=0x55a727684930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:10.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f917cff9640 0 client.16887 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:22:10.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 -- 172.21.15.62:0/4179976380 shutdown_connections 2024-09-15T23:22:10.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 1 -- 172.21.15.62:0/4179976380 wait complete. 2024-09-15T23:22:10.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.217+0000 7f919d696740 20 client.16887 trim_cache size 0 max 16384 2024-09-15T23:22:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.236+0000 7f919d696740 1 Processor -- start 2024-09-15T23:22:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.236+0000 7f919d696740 1 -- start start 2024-09-15T23:22:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.236+0000 7f919d696740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72768bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.236+0000 7f919d696740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a72769c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.236+0000 7f919d696740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72769c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.236+0000 7f919d696740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f91780123e0 con 0x55a72768d370 2024-09-15T23:22:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.236+0000 7f919d696740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f917800fcb0 con 0x55a7276825d0 2024-09-15T23:22:10.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.236+0000 7f919d696740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f917800db00 con 0x55a727683b40 2024-09-15T23:22:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.238+0000 7f919985a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72769c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.238+0000 7f919985a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72769c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54814/0 (socket says 172.21.15.62:54814) 2024-09-15T23:22:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.238+0000 7f919985a640 1 -- 172.21.15.62:0/2861921540 learned_addr learned my addr 172.21.15.62:0/2861921540 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.238+0000 7f919985a640 1 -- 172.21.15.62:0/2861921540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 msgr2=0x55a72769c1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.238+0000 7f919985a640 1 --2- 172.21.15.62:0/2861921540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a72769c1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.239+0000 7f919985a640 1 -- 172.21.15.62:0/2861921540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 msgr2=0x55a72768bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.239+0000 7f919985a640 1 --2- 172.21.15.62:0/2861921540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72768bf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.239+0000 7f919985a640 1 -- 172.21.15.62:0/2861921540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a72768d050 con 0x55a72768d370 2024-09-15T23:22:10.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.239+0000 7f919985a640 1 --2- 172.21.15.62:0/2861921540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72769c6f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f918c098900 tx=0x7f918c0093a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e9cd408da0c41b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.239+0000 7f917effd640 1 -- 172.21.15.62:0/2861921540 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c09c270 con 0x55a72768d370 2024-09-15T23:22:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.239+0000 7f917effd640 1 -- 172.21.15.62:0/2861921540 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f918c07e070 con 0x55a72768d370 2024-09-15T23:22:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.240+0000 7f917effd640 1 -- 172.21.15.62:0/2861921540 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c09b8f0 con 0x55a72768d370 2024-09-15T23:22:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.240+0000 7f919d696740 1 -- 172.21.15.62:0/2861921540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 msgr2=0x55a72769c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.240+0000 7f919d696740 1 --2- 172.21.15.62:0/2861921540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72769c6f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f918c098900 tx=0x7f918c0093a0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.248+0000 7f919d696740 1 -- 172.21.15.62:0/2861921540 shutdown_connections 2024-09-15T23:22:10.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.248+0000 7f919d696740 1 --2- 172.21.15.62:0/2861921540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a72769c6f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.248+0000 7f919d696740 1 --2- 172.21.15.62:0/2861921540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a72769c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.248+0000 7f919d696740 1 --2- 172.21.15.62:0/2861921540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72768bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.248+0000 7f919d696740 1 -- 172.21.15.62:0/2861921540 >> 172.21.15.62:0/2861921540 conn(0x55a72764b310 msgr2=0x55a72768b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:10.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.255+0000 7f919d696740 1 -- 172.21.15.62:0/2861921540 shutdown_connections 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.256+0000 7f919d696740 1 -- 172.21.15.62:0/2861921540 wait complete. 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.256+0000 7f919d696740 1 -- start start 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.256+0000 7f919d696740 1 Processor -- start 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.257+0000 7f919d696740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72769c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.257+0000 7f919d696740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a727699190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.257+0000 7f919d696740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a7276996d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.257+0000 7f919d696740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f917800e170 con 0x55a72768d370 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.257+0000 7f919d696740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f917800f5e0 con 0x55a7276825d0 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.257+0000 7f919d696740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f917800e840 con 0x55a727683b40 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.257+0000 7f919d696740 10 client.0 _ll_register_callbacks cb 0x7f918c005bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:10.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.258+0000 7f919985a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a7276996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.258+0000 7f919985a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a7276996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54818/0 (socket says 172.21.15.62:54818) 2024-09-15T23:22:10.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.258+0000 7f919985a640 1 -- 172.21.15.62:0/2756860271 learned_addr learned my addr 172.21.15.62:0/2756860271 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:10.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.260+0000 7f9198858640 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a727699190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.260+0000 7f919985a640 1 -- 172.21.15.62:0/2756860271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 msgr2=0x55a727699190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:10.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.260+0000 7f919985a640 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a727699190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.260+0000 7f919985a640 1 -- 172.21.15.62:0/2756860271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 msgr2=0x55a72769c790 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:10.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.260+0000 7f919985a640 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72769c790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.260+0000 7f919985a640 1 -- 172.21.15.62:0/2756860271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a727699d70 con 0x55a72768d370 2024-09-15T23:22:10.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.268+0000 7f919985a640 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a7276996d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f918c0a4be0 tx=0x7f918c09b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf00b8373486051 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:10.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.268+0000 7f917cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:22:10.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.268+0000 7f917cff9640 1 -- 172.21.15.62:0/2756860271 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c09c270 con 0x55a72768d370 2024-09-15T23:22:10.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.268+0000 7f919d696740 1 -- 172.21.15.62:0/2756860271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f918c07e070 con 0x55a72768d370 2024-09-15T23:22:10.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.268+0000 7f917cff9640 1 -- 172.21.15.62:0/2756860271 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f918c0b5260 con 0x55a72768d370 2024-09-15T23:22:10.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.272+0000 7f917cff9640 1 -- 172.21.15.62:0/2756860271 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f918c0a0850 con 0x55a72768d370 2024-09-15T23:22:10.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.273+0000 7f919d696740 10 client.16893 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:10.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.273+0000 7f919d696740 1 -- 172.21.15.62:0/2756860271 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a727729530 con 0x55a72768d370 2024-09-15T23:22:10.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.274+0000 7f917cff9640 1 -- 172.21.15.62:0/2756860271 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f918c07e070 con 0x55a72768d370 2024-09-15T23:22:10.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.274+0000 7f917cff9640 1 -- 172.21.15.62:0/2756860271 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f918c0a4c80 con 0x55a72768d370 2024-09-15T23:22:10.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.274+0000 7f919d696740 10 client.16893 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:10.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.274+0000 7f919d696740 10 client.16893 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:22:10.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.274+0000 7f919d696740 20 client.16893 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:10.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.274+0000 7f919d696740 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a727729960 0x55a72772bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:10.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.274+0000 7f919d696740 4 client.16893 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:22:10.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.274+0000 7f919d696740 1 -- 172.21.15.62:0/2756860271 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a7275efd70 con 0x55a727729960 2024-09-15T23:22:10.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.281+0000 7f9199059640 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a727729960 0x55a72772bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:10.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.293+0000 7f917cff9640 10 client.16893 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:22:10.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.293+0000 7f9199059640 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a727729960 0x55a72772bd80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a7f90951fdc46131 server_cookie=b9d674d475c94ebd in_seq=0 out_seq=0 2024-09-15T23:22:10.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f917cff9640 1 -- 172.21.15.62:0/2756860271 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a7275efd70 con 0x55a727729960 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f917cff9640 10 client.16893 handle_command_reply: tid=0 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f919d696740 1 client.16893 shutdown 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f919d696740 1 -- 172.21.15.62:0/2756860271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 msgr2=0x55a7276996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f919d696740 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a7276996d0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f918c0a4be0 tx=0x7f918c09b7d0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f919d696740 1 -- 172.21.15.62:0/2756860271 shutdown_connections 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f919d696740 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a727729960 0x55a72772bd80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f919d696740 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a72768d370 0x55a7276996d0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f919d696740 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a727683b40 0x55a727699190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f919d696740 1 --2- 172.21.15.62:0/2756860271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a7276825d0 0x55a72769c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f919d696740 1 -- 172.21.15.62:0/2756860271 >> 172.21.15.62:0/2756860271 conn(0x55a72764b310 msgr2=0x55a72767dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.298+0000 7f917cff9640 0 client.16893 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:22:10.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.303+0000 7f919d696740 1 -- 172.21.15.62:0/2756860271 shutdown_connections 2024-09-15T23:22:10.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.303+0000 7f919d696740 1 -- 172.21.15.62:0/2756860271 wait complete. 2024-09-15T23:22:10.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.303+0000 7f919d696740 20 client.16893 trim_cache size 0 max 16384 2024-09-15T23:22:10.305 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:22:10.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.307+0000 7f919b2bd640 1 -- 172.21.15.62:0/1763747345 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91800a32c0 msgr2=0x7f91800a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.307+0000 7f919b2bd640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91800a32c0 0x7f91800a5780 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f919002a930 tx=0x7f9190000bc0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.307+0000 7f919b2bd640 1 -- 172.21.15.62:0/1763747345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f919410e4d0 msgr2=0x7f9194102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:10.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.307+0000 7f919b2bd640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f919410e4d0 0x7f9194102b10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f918c00eb10 tx=0x7f918c00c2d0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.308+0000 7f919b2bd640 1 -- 172.21.15.62:0/1763747345 shutdown_connections 2024-09-15T23:22:10.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.308+0000 7f919b2bd640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f91800a32c0 0x7f91800a5780 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.308+0000 7f919b2bd640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f919410e4d0 0x7f9194102b10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.308+0000 7f919b2bd640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9194101360 0x7f9194107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.308+0000 7f919b2bd640 1 --2- 172.21.15.62:0/1763747345 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91941009b0 0x7f9194107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:10.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.308+0000 7f919b2bd640 1 -- 172.21.15.62:0/1763747345 >> 172.21.15.62:0/1763747345 conn(0x7f91940fc7d0 msgr2=0x7f91940febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:10.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.312+0000 7f919b2bd640 1 -- 172.21.15.62:0/1763747345 shutdown_connections 2024-09-15T23:22:10.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:10.313+0000 7f919b2bd640 1 -- 172.21.15.62:0/1763747345 wait complete. 2024-09-15T23:22:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:11 smithi062 ceph-mon[30949]: pgmap v547: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 602 op/s 2024-09-15T23:22:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:11 smithi067 ceph-mon[35665]: pgmap v547: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 602 op/s 2024-09-15T23:22:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:11 smithi148 ceph-mon[35610]: pgmap v547: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 15 MiB/s wr, 602 op/s 2024-09-15T23:22:12.126 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:22:12.587 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:13.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.126+0000 7f3126214640 1 -- 172.21.15.62:0/4187367308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100e10 msgr2=0x7f312010d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.126+0000 7f3126214640 1 --2- 172.21.15.62:0/4187367308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100e10 0x7f312010d070 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3118009ca0 tx=0x7f311802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.127+0000 7f3126214640 1 -- 172.21.15.62:0/4187367308 shutdown_connections 2024-09-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.127+0000 7f3126214640 1 --2- 172.21.15.62:0/4187367308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100e10 0x7f312010d070 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.127+0000 7f3126214640 1 --2- 172.21.15.62:0/4187367308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120100490 0x7f31201008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.127+0000 7f3126214640 1 --2- 172.21.15.62:0/4187367308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3120106250 0x7f3120106630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.127+0000 7f3126214640 1 -- 172.21.15.62:0/4187367308 >> 172.21.15.62:0/4187367308 conn(0x7f31200fba90 msgr2=0x7f31200fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.127+0000 7f3126214640 1 -- 172.21.15.62:0/4187367308 shutdown_connections 2024-09-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.127+0000 7f3126214640 1 -- 172.21.15.62:0/4187367308 wait complete. 2024-09-15T23:22:13.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.127+0000 7f3126214640 1 Processor -- start 2024-09-15T23:22:13.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f3126214640 1 -- start start 2024-09-15T23:22:13.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f3126214640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100490 0x7f31201991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f3126214640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120100e10 0x7f3120199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f3126214640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3120106250 0x7f31201a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f3126214640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3120110e60 con 0x7f3120100490 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f3126214640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3120110ce0 con 0x7f3120100e10 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f3126214640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3120110fe0 con 0x7f3120106250 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f311f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100490 0x7f31201991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f311f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100490 0x7f31201991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47838/0 (socket says 172.21.15.62:47838) 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f311f7fe640 1 -- 172.21.15.62:0/1285548986 learned_addr learned my addr 172.21.15.62:0/1285548986 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f311effd640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120100e10 0x7f3120199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.128+0000 7f311ffff640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3120106250 0x7f31201a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311f7fe640 1 -- 172.21.15.62:0/1285548986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3120106250 msgr2=0x7f31201a4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311f7fe640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3120106250 0x7f31201a4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311f7fe640 1 -- 172.21.15.62:0/1285548986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120100e10 msgr2=0x7f3120199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311f7fe640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120100e10 0x7f3120199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311f7fe640 1 -- 172.21.15.62:0/1285548986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31201a47b0 con 0x7f3120100490 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311effd640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120100e10 0x7f3120199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311f7fe640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100490 0x7f31201991f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f311000e9f0 tx=0x7f311000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311cff9640 1 -- 172.21.15.62:0/1285548986 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311000cd80 con 0x7f3120100490 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311cff9640 1 -- 172.21.15.62:0/1285548986 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3110010910 con 0x7f3120100490 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f311cff9640 1 -- 172.21.15.62:0/1285548986 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110018990 con 0x7f3120100490 2024-09-15T23:22:13.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f3126214640 1 -- 172.21.15.62:0/1285548986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3120196fe0 con 0x7f3120100490 2024-09-15T23:22:13.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.129+0000 7f3126214640 1 -- 172.21.15.62:0/1285548986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3120197570 con 0x7f3120100490 2024-09-15T23:22:13.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.134+0000 7f311cff9640 1 -- 172.21.15.62:0/1285548986 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3110010ac0 con 0x7f3120100490 2024-09-15T23:22:13.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.135+0000 7f311cff9640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31080a32c0 0x7f31080a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.135+0000 7f31285ed740 1 Processor -- start 2024-09-15T23:22:13.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.135+0000 7f311cff9640 1 -- 172.21.15.62:0/1285548986 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f31100cfe60 con 0x7f3120100490 2024-09-15T23:22:13.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.135+0000 7f311effd640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31080a32c0 0x7f31080a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.135+0000 7f31285ed740 1 -- start start 2024-09-15T23:22:13.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.135+0000 7f31285ed740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390bf8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f31285ed740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390bf8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f31285ed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390c05760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f31285ed740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f390b5e2f0 con 0x55f390bf85d0 2024-09-15T23:22:13.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f31285ed740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f390b05cf0 con 0x55f390bf9b40 2024-09-15T23:22:13.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f31285ed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f390b5a2b0 con 0x55f390c03370 2024-09-15T23:22:13.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390bf8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390bf8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40966/0 (socket says 172.21.15.62:40966) 2024-09-15T23:22:13.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311f7fe640 1 -- 172.21.15.62:0/2729128708 learned_addr learned my addr 172.21.15.62:0/2729128708 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:13.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311effd640 1 --2- 172.21.15.62:0/2729128708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390bf8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311ffff640 1 --2- 172.21.15.62:0/2729128708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390c05760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311f7fe640 1 -- 172.21.15.62:0/2729128708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 msgr2=0x55f390c05760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:13.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311f7fe640 1 --2- 172.21.15.62:0/2729128708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390c05760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311f7fe640 1 -- 172.21.15.62:0/2729128708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 msgr2=0x55f390bf8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:13.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311f7fe640 1 --2- 172.21.15.62:0/2729128708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390bf8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311f7fe640 1 -- 172.21.15.62:0/2729128708 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f390aea3a0 con 0x55f390bf9b40 2024-09-15T23:22:13.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311effd640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31080a32c0 0x7f31080a5780 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f31140095f0 tx=0x7f3114012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:13.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.136+0000 7f311f7fe640 1 --2- 172.21.15.62:0/2729128708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390bf8090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f311009c140 tx=0x7f311009bbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d359a7de78d4cb19 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:13.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.137+0000 7f3105ffb640 1 -- 172.21.15.62:0/2729128708 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311009a4b0 con 0x55f390bf9b40 2024-09-15T23:22:13.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.137+0000 7f3105ffb640 1 -- 172.21.15.62:0/2729128708 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31100bc070 con 0x55f390bf9b40 2024-09-15T23:22:13.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.137+0000 7f3105ffb640 1 -- 172.21.15.62:0/2729128708 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110018b30 con 0x55f390bf9b40 2024-09-15T23:22:13.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 -- 172.21.15.62:0/2729128708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 msgr2=0x55f390bf8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 --2- 172.21.15.62:0/2729128708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390bf8090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f311009c140 tx=0x7f311009bbe0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 -- 172.21.15.62:0/2729128708 shutdown_connections 2024-09-15T23:22:13.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 --2- 172.21.15.62:0/2729128708 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390c05760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 --2- 172.21.15.62:0/2729128708 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390bf8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 --2- 172.21.15.62:0/2729128708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390bf8090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 -- 172.21.15.62:0/2729128708 >> 172.21.15.62:0/2729128708 conn(0x55f390bc1310 msgr2=0x55f390bf4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:13.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 -- 172.21.15.62:0/2729128708 shutdown_connections 2024-09-15T23:22:13.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 -- 172.21.15.62:0/2729128708 wait complete. 2024-09-15T23:22:13.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.138+0000 7f31285ed740 1 -- start start 2024-09-15T23:22:13.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f31285ed740 1 Processor -- start 2024-09-15T23:22:13.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f31285ed740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390c02d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f31285ed740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390c00090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f31285ed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390c006f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f311f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390c02d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f311f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390c02d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47860/0 (socket says 172.21.15.62:47860) 2024-09-15T23:22:13.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f31285ed740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f390c0e820 con 0x55f390bf85d0 2024-09-15T23:22:13.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f311f7fe640 1 -- 172.21.15.62:0/925173542 learned_addr learned my addr 172.21.15.62:0/925173542 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:13.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f390b19060 con 0x55f390bf9b40 2024-09-15T23:22:13.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f390c0e9a0 con 0x55f390c03370 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f311effd640 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390c00090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.139+0000 7f31285ed740 10 client.0 _ll_register_callbacks cb 0x55f39086c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f311ffff640 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390c006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f311effd640 1 -- 172.21.15.62:0/925173542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 msgr2=0x55f390c006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f311effd640 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390c006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f311effd640 1 -- 172.21.15.62:0/925173542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 msgr2=0x55f390c02d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f311effd640 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390c02d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f311effd640 1 -- 172.21.15.62:0/925173542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3908cf3c0 con 0x55f390bf9b40 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f311f7fe640 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390c02d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f311effd640 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390c00090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f312010e2c0 tx=0x7f31140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79771e761106f4f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f30feffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:13.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f30feffd640 1 -- 172.21.15.62:0/925173542 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311400bd00 con 0x55f390bf9b40 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f30feffd640 1 -- 172.21.15.62:0/925173542 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3114026480 con 0x55f390bf9b40 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.140+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31100bc070 con 0x55f390bf9b40 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f30feffd640 1 -- 172.21.15.62:0/925173542 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3114034050 con 0x55f390bf9b40 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f31285ed740 10 client.26468 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f390c1a3e0 con 0x55f390bf9b40 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f30feffd640 1 -- 172.21.15.62:0/925173542 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31100bc070 con 0x55f390bf9b40 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f30feffd640 1 -- 172.21.15.62:0/925173542 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f3114025610 con 0x55f390bf9b40 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f31285ed740 10 client.26468 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f31285ed740 10 client.26468 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f31285ed740 20 client.26468 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f31285ed740 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f390c1b8a0 0x55f390c1dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f31285ed740 4 client.26468 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.141+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f390b6b860 con 0x55f390c1b8a0 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.142+0000 7f311f7fe640 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f390c1b8a0 0x55f390c1dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.142+0000 7f311f7fe640 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f390c1b8a0 0x55f390c1dd20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c613978d485e384 server_cookie=e4d301769ec1cc9b in_seq=0 out_seq=0 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.144+0000 7f30feffd640 10 client.26468 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.144+0000 7f30feffd640 1 -- 172.21.15.62:0/925173542 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f390b6b860 con 0x55f390c1b8a0 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.144+0000 7f30feffd640 10 client.26468 handle_command_reply: tid=0 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.144+0000 7f31285ed740 1 client.26468 shutdown 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.144+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 msgr2=0x55f390c00090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.144+0000 7f31285ed740 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390c00090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f312010e2c0 tx=0x7f31140046c0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 shutdown_connections 2024-09-15T23:22:13.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f31285ed740 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f390c1b8a0 0x55f390c1dd20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f30feffd640 0 client.26468 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:22:13.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f31285ed740 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390c006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f31285ed740 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf9b40 0x55f390c00090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f31285ed740 1 --2- 172.21.15.62:0/925173542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf85d0 0x55f390c02d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 >> 172.21.15.62:0/925173542 conn(0x55f390bc1310 msgr2=0x55f390bfa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:13.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 shutdown_connections 2024-09-15T23:22:13.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f31285ed740 1 -- 172.21.15.62:0/925173542 wait complete. 2024-09-15T23:22:13.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.145+0000 7f31285ed740 20 client.26468 trim_cache size 0 max 16384 2024-09-15T23:22:13.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.162+0000 7f31285ed740 1 Processor -- start 2024-09-15T23:22:13.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.162+0000 7f31285ed740 1 -- start start 2024-09-15T23:22:13.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.162+0000 7f31285ed740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390bff950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.162+0000 7f31285ed740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf9b40 0x55f390bfb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.162+0000 7f31285ed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390bfb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.162+0000 7f31285ed740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f31201af430 con 0x55f390bf9b40 2024-09-15T23:22:13.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.162+0000 7f31285ed740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f31201acd00 con 0x55f390bf85d0 2024-09-15T23:22:13.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.162+0000 7f31285ed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f31201aab50 con 0x55f390c03370 2024-09-15T23:22:13.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390bfb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390bfb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48728/0 (socket says 172.21.15.62:48728) 2024-09-15T23:22:13.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311ffff640 1 -- 172.21.15.62:0/1622634721 learned_addr learned my addr 172.21.15.62:0/1622634721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:13.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311f7fe640 1 --2- 172.21.15.62:0/1622634721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390bff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311ffff640 1 -- 172.21.15.62:0/1622634721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 msgr2=0x55f390bff950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:13.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311ffff640 1 --2- 172.21.15.62:0/1622634721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390bff950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311ffff640 1 -- 172.21.15.62:0/1622634721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf9b40 msgr2=0x55f390bfb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:13.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311ffff640 1 --2- 172.21.15.62:0/1622634721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf9b40 0x55f390bfb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311ffff640 1 -- 172.21.15.62:0/1622634721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f390bff040 con 0x55f390c03370 2024-09-15T23:22:13.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.163+0000 7f311f7fe640 1 --2- 172.21.15.62:0/1622634721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390bff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:13.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.164+0000 7f311ffff640 1 --2- 172.21.15.62:0/1622634721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390bfb660 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3118009ca0 tx=0x7f3118004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88edb42837e8e38f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:13.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.164+0000 7f31057fa640 1 -- 172.21.15.62:0/1622634721 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118003970 con 0x55f390c03370 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.164+0000 7f31057fa640 1 -- 172.21.15.62:0/1622634721 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f311804b070 con 0x55f390c03370 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.164+0000 7f31057fa640 1 -- 172.21.15.62:0/1622634721 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118004440 con 0x55f390c03370 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.165+0000 7f31285ed740 1 -- 172.21.15.62:0/1622634721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 msgr2=0x55f390bfb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.166+0000 7f31285ed740 1 --2- 172.21.15.62:0/1622634721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390bfb660 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3118009ca0 tx=0x7f3118004a80 comp rx=0 tx=0).stop 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.166+0000 7f31285ed740 1 -- 172.21.15.62:0/1622634721 shutdown_connections 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.166+0000 7f31285ed740 1 --2- 172.21.15.62:0/1622634721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390c03370 0x55f390bfb660 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.166+0000 7f31285ed740 1 --2- 172.21.15.62:0/1622634721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390bf9b40 0x55f390bfb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.166+0000 7f31285ed740 1 --2- 172.21.15.62:0/1622634721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390bff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.166+0000 7f31285ed740 1 -- 172.21.15.62:0/1622634721 >> 172.21.15.62:0/1622634721 conn(0x55f390bc1310 msgr2=0x55f390c02110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.166+0000 7f31285ed740 1 -- 172.21.15.62:0/1622634721 shutdown_connections 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.166+0000 7f31285ed740 1 -- 172.21.15.62:0/1622634721 wait complete. 2024-09-15T23:22:13.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.166+0000 7f31285ed740 1 -- start start 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f31285ed740 1 Processor -- start 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f31285ed740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390c14570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f31285ed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390bf9b40 0x55f390c0f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f31285ed740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390c03370 0x55f390c0ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f31285ed740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f31201ab1c0 con 0x55f390c03370 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f31285ed740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f31201ac630 con 0x55f390bf85d0 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f31285ed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f31201ab890 con 0x55f390bf9b40 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f31285ed740 10 client.0 _ll_register_callbacks cb 0x7f3114010450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f311f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390c14570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f311f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390c14570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40996/0 (socket says 172.21.15.62:40996) 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f311f7fe640 1 -- 172.21.15.62:0/3553746574 learned_addr learned my addr 172.21.15.62:0/3553746574 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f311effd640 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390bf9b40 0x55f390c0f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.167+0000 7f311ffff640 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390c03370 0x55f390c0ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.168+0000 7f311effd640 1 -- 172.21.15.62:0/3553746574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 msgr2=0x55f390c14570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.168+0000 7f311effd640 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390c14570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.168+0000 7f311effd640 1 -- 172.21.15.62:0/3553746574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390c03370 msgr2=0x55f390c0ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.168+0000 7f311effd640 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390c03370 0x55f390c0ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.168+0000 7f311effd640 1 -- 172.21.15.62:0/3553746574 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f390c00a60 con 0x55f390bf9b40 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.168+0000 7f311effd640 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390bf9b40 0x55f390c0f8a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55f390bf3380 tx=0x7f3114010000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d46dfaf9af3be4a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.168+0000 7f30feffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.168+0000 7f30feffd640 1 -- 172.21.15.62:0/3553746574 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311403c070 con 0x55f390bf9b40 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.168+0000 7f30feffd640 1 -- 172.21.15.62:0/3553746574 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3114025c70 con 0x55f390bf9b40 2024-09-15T23:22:13.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.169+0000 7f31285ed740 1 -- 172.21.15.62:0/3553746574 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f311804b070 con 0x55f390bf9b40 2024-09-15T23:22:13.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.173+0000 7f30feffd640 1 -- 172.21.15.62:0/3553746574 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f311400b8e0 con 0x55f390bf9b40 2024-09-15T23:22:13.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.173+0000 7f31285ed740 10 client.26284 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:13.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.173+0000 7f31285ed740 1 -- 172.21.15.62:0/3553746574 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f390c86560 con 0x55f390bf9b40 2024-09-15T23:22:13.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.179+0000 7f30feffd640 1 -- 172.21.15.62:0/3553746574 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f311402b070 con 0x55f390bf9b40 2024-09-15T23:22:13.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.180+0000 7f30feffd640 1 -- 172.21.15.62:0/3553746574 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f3114004640 con 0x55f390bf9b40 2024-09-15T23:22:13.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.180+0000 7f31285ed740 10 client.26284 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:13.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.180+0000 7f31285ed740 10 client.26284 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:22:13.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.180+0000 7f31285ed740 20 client.26284 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:13.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.180+0000 7f31285ed740 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f390c1b8a0 0x55f390c88cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:13.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.180+0000 7f31285ed740 4 client.26284 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:22:13.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.180+0000 7f31285ed740 1 -- 172.21.15.62:0/3553746574 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f311804b070 con 0x55f390c1b8a0 2024-09-15T23:22:13.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.187+0000 7f311f7fe640 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f390c1b8a0 0x55f390c88cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:13.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.204+0000 7f311f7fe640 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f390c1b8a0 0x55f390c88cc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc90b750afa69117 server_cookie=78662de29705eeb7 in_seq=0 out_seq=0 2024-09-15T23:22:13.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.205+0000 7f30feffd640 10 client.26284 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:22:13.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.206+0000 7f30feffd640 1 -- 172.21.15.62:0/3553746574 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f311804b070 con 0x55f390c1b8a0 2024-09-15T23:22:13.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.206+0000 7f30feffd640 10 client.26284 handle_command_reply: tid=0 2024-09-15T23:22:13.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.207+0000 7f31285ed740 1 client.26284 shutdown 2024-09-15T23:22:13.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.210+0000 7f31285ed740 1 -- 172.21.15.62:0/3553746574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390bf9b40 msgr2=0x55f390c0f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.210+0000 7f31285ed740 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390bf9b40 0x55f390c0f8a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55f390bf3380 tx=0x7f3114010000 comp rx=0 tx=0).stop 2024-09-15T23:22:13.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.211+0000 7f31285ed740 1 -- 172.21.15.62:0/3553746574 shutdown_connections 2024-09-15T23:22:13.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.211+0000 7f31285ed740 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f390c1b8a0 0x55f390c88cc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.211+0000 7f31285ed740 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f390c03370 0x55f390c0ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.211+0000 7f31285ed740 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f390bf9b40 0x55f390c0f8a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.211+0000 7f31285ed740 1 --2- 172.21.15.62:0/3553746574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f390bf85d0 0x55f390c14570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.211+0000 7f30feffd640 0 client.26284 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:22:13.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.211+0000 7f31285ed740 1 -- 172.21.15.62:0/3553746574 >> 172.21.15.62:0/3553746574 conn(0x55f390bc1310 msgr2=0x55f390bf34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:13.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.215+0000 7f31285ed740 1 -- 172.21.15.62:0/3553746574 shutdown_connections 2024-09-15T23:22:13.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.215+0000 7f31285ed740 1 -- 172.21.15.62:0/3553746574 wait complete. 2024-09-15T23:22:13.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.216+0000 7f31285ed740 20 client.26284 trim_cache size 0 max 16384 2024-09-15T23:22:13.217 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:22:13.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 -- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31080a32c0 msgr2=0x7f31080a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31080a32c0 0x7f31080a5780 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f31140095f0 tx=0x7f3114012040 comp rx=0 tx=0).stop 2024-09-15T23:22:13.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 -- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100490 msgr2=0x7f31201991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:13.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100490 0x7f31201991f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f311000e9f0 tx=0x7f311000eeb0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 -- 172.21.15.62:0/1285548986 shutdown_connections 2024-09-15T23:22:13.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f31080a32c0 0x7f31080a5780 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3120106250 0x7f31201a4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3120100e10 0x7f3120199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 --2- 172.21.15.62:0/1285548986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3120100490 0x7f31201991f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:13.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.218+0000 7f3126214640 1 -- 172.21.15.62:0/1285548986 >> 172.21.15.62:0/1285548986 conn(0x7f31200fba90 msgr2=0x7f31200fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:13.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.219+0000 7f3126214640 1 -- 172.21.15.62:0/1285548986 shutdown_connections 2024-09-15T23:22:13.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:13.219+0000 7f3126214640 1 -- 172.21.15.62:0/1285548986 wait complete. 2024-09-15T23:22:13.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:13 smithi062 ceph-mon[30949]: pgmap v548: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 636 op/s 2024-09-15T23:22:14.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:13 smithi067 ceph-mon[35665]: pgmap v548: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 636 op/s 2024-09-15T23:22:14.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:13 smithi148 ceph-mon[35610]: pgmap v548: 129 pgs: 129 active+clean; 1.3 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 636 op/s 2024-09-15T23:22:14.276 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:22:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 355.0 s, 1137.0 tps, lat 88.179 ms stddev 119.228 2024-09-15T23:22:14.663 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.271+0000 7f9e9f069640 1 -- 172.21.15.62:0/2702501032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e98102570 msgr2=0x7f9e981029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.271+0000 7f9e9f069640 1 --2- 172.21.15.62:0/2702501032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e98102570 0x7f9e981029b0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009900 tx=0x7f9e8402a480 comp rx=0 tx=0).stop 2024-09-15T23:22:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.271+0000 7f9e9f069640 1 -- 172.21.15.62:0/2702501032 shutdown_connections 2024-09-15T23:22:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.271+0000 7f9e9f069640 1 --2- 172.21.15.62:0/2702501032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e98102ef0 0x7f9e9810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.271+0000 7f9e9f069640 1 --2- 172.21.15.62:0/2702501032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e98102570 0x7f9e981029b0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.271+0000 7f9e9f069640 1 --2- 172.21.15.62:0/2702501032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e98100e20 0x7f9e98101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.271+0000 7f9e9f069640 1 -- 172.21.15.62:0/2702501032 >> 172.21.15.62:0/2702501032 conn(0x7f9e980fc7d0 msgr2=0x7f9e980febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:15.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.271+0000 7f9e9f069640 1 -- 172.21.15.62:0/2702501032 shutdown_connections 2024-09-15T23:22:15.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.271+0000 7f9e9f069640 1 -- 172.21.15.62:0/2702501032 wait complete. 2024-09-15T23:22:15.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.272+0000 7f9e9f069640 1 Processor -- start 2024-09-15T23:22:15.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.272+0000 7f9e9f069640 1 -- start start 2024-09-15T23:22:15.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.272+0000 7f9e9f069640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e98100e20 0x7f9e98107ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.272+0000 7f9e9f069640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e98102570 0x7f9e98108020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.272+0000 7f9e9f069640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e98102ef0 0x7f9e98108560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.272+0000 7f9e9f069640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e98110f50 con 0x7f9e98100e20 2024-09-15T23:22:15.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.272+0000 7f9e9f069640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e98110dd0 con 0x7f9e98102ef0 2024-09-15T23:22:15.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.272+0000 7f9e9f069640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981110d0 con 0x7f9e98102570 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e9d606640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e98102ef0 0x7f9e98108560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e9d606640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e98102ef0 0x7f9e98108560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41022/0 (socket says 172.21.15.62:41022) 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e9d606640 1 -- 172.21.15.62:0/3136864982 learned_addr learned my addr 172.21.15.62:0/3136864982 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e98102570 0x7f9e98108020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e9ce05640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e98100e20 0x7f9e98107ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e97fff640 1 -- 172.21.15.62:0/3136864982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e98102ef0 msgr2=0x7f9e98108560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e97fff640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e98102ef0 0x7f9e98108560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e97fff640 1 -- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e98100e20 msgr2=0x7f9e98107ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e97fff640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e98100e20 0x7f9e98107ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e97fff640 1 -- 172.21.15.62:0/3136864982 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e981a69f0 con 0x7f9e98102570 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e9d606640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e98102ef0 0x7f9e98108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:15.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.273+0000 7f9e97fff640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e98102570 0x7f9e98108020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9e84000c00 tx=0x7f9e84003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.274+0000 7f9e95ffb640 1 -- 172.21.15.62:0/3136864982 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8403f070 con 0x7f9e98102570 2024-09-15T23:22:15.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.274+0000 7f9e95ffb640 1 -- 172.21.15.62:0/3136864982 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e84031c20 con 0x7f9e98102570 2024-09-15T23:22:15.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.274+0000 7f9e95ffb640 1 -- 172.21.15.62:0/3136864982 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84035030 con 0x7f9e98102570 2024-09-15T23:22:15.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.274+0000 7f9e9f069640 1 -- 172.21.15.62:0/3136864982 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e98198290 con 0x7f9e98102570 2024-09-15T23:22:15.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.274+0000 7f9e9f069640 1 -- 172.21.15.62:0/3136864982 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e98198660 con 0x7f9e98102570 2024-09-15T23:22:15.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.277+0000 7f9e95ffb640 1 -- 172.21.15.62:0/3136864982 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9e84042070 con 0x7f9e98102570 2024-09-15T23:22:15.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.278+0000 7f9e95ffb640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e880a32c0 0x7f9e880a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.278+0000 7f9e95ffb640 1 -- 172.21.15.62:0/3136864982 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e840f0290 con 0x7f9e98102570 2024-09-15T23:22:15.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.278+0000 7f9e9ce05640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e880a32c0 0x7f9e880a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9ea1442740 1 Processor -- start 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9ea1442740 1 -- start start 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9e9ce05640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e880a32c0 0x7f9e880a5780 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0095f0 tx=0x7f9e8c016040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560160913b40 0x560160912090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x560160912a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 0x56016091f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9ea1442740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5601608782f0 con 0x560160913b40 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9ea1442740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56016081fcf0 con 0x56016091d370 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9ea1442740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5601608742b0 con 0x5601609125d0 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x560160912a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x560160912a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48772/0 (socket says 172.21.15.62:48772) 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9e97fff640 1 -- 172.21.15.62:0/2473513079 learned_addr learned my addr 172.21.15.62:0/2473513079 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9e9ce05640 1 --2- 172.21.15.62:0/2473513079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560160913b40 0x560160912090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.279+0000 7f9e9d606640 1 --2- 172.21.15.62:0/2473513079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 0x56016091f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.280+0000 7f9e97fff640 1 -- 172.21.15.62:0/2473513079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 msgr2=0x56016091f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.280+0000 7f9e97fff640 1 --2- 172.21.15.62:0/2473513079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 0x56016091f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.280+0000 7f9e97fff640 1 -- 172.21.15.62:0/2473513079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560160913b40 msgr2=0x560160912090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.280+0000 7f9e97fff640 1 --2- 172.21.15.62:0/2473513079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560160913b40 0x560160912090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.280+0000 7f9e97fff640 1 -- 172.21.15.62:0/2473513079 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601608043a0 con 0x5601609125d0 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.280+0000 7f9e97fff640 1 --2- 172.21.15.62:0/2473513079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x560160912a10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9e840c4c10 tx=0x7f9e84003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e23750a5324578df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.280+0000 7f9e82ffd640 1 -- 172.21.15.62:0/2473513079 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e840d5070 con 0x5601609125d0 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.280+0000 7f9e82ffd640 1 -- 172.21.15.62:0/2473513079 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e840bc580 con 0x5601609125d0 2024-09-15T23:22:15.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.280+0000 7f9e82ffd640 1 -- 172.21.15.62:0/2473513079 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84049050 con 0x5601609125d0 2024-09-15T23:22:15.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.281+0000 7f9ea1442740 1 -- 172.21.15.62:0/2473513079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 msgr2=0x560160912a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.281+0000 7f9ea1442740 1 --2- 172.21.15.62:0/2473513079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x560160912a10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9e840c4c10 tx=0x7f9e84003f90 comp rx=0 tx=0).stop 2024-09-15T23:22:15.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.281+0000 7f9ea1442740 1 -- 172.21.15.62:0/2473513079 shutdown_connections 2024-09-15T23:22:15.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.281+0000 7f9ea1442740 1 --2- 172.21.15.62:0/2473513079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 0x56016091f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.281+0000 7f9ea1442740 1 --2- 172.21.15.62:0/2473513079 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x560160912a10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.281+0000 7f9ea1442740 1 --2- 172.21.15.62:0/2473513079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560160913b40 0x560160912090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.281+0000 7f9ea1442740 1 -- 172.21.15.62:0/2473513079 >> 172.21.15.62:0/2473513079 conn(0x5601608db310 msgr2=0x56016090ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.281+0000 7f9ea1442740 1 -- 172.21.15.62:0/2473513079 shutdown_connections 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.281+0000 7f9ea1442740 1 -- 172.21.15.62:0/2473513079 wait complete. 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.282+0000 7f9ea1442740 1 -- start start 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.282+0000 7f9ea1442740 1 Processor -- start 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x56016091cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560160913b40 0x56016091a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56016091d370 0x56016091a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e9ce05640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x56016091cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e9ce05640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x56016091cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48782/0 (socket says 172.21.15.62:48782) 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560160913b40 0x56016091a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560160913b40 0x56016091a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41032/0 (socket says 172.21.15.62:41032) 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e9ce05640 1 -- 172.21.15.62:0/977900605 learned_addr learned my addr 172.21.15.62:0/977900605 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:15.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56016091fe30 con 0x56016091d370 2024-09-15T23:22:15.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560160833060 con 0x560160913b40 2024-09-15T23:22:15.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56016091ffb0 con 0x5601609125d0 2024-09-15T23:22:15.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9ea1442740 10 client.0 _ll_register_callbacks cb 0x5601605868c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:15.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e9ce05640 1 -- 172.21.15.62:0/977900605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560160913b40 msgr2=0x56016091a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e9ce05640 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560160913b40 0x56016091a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e9ce05640 1 -- 172.21.15.62:0/977900605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56016091d370 msgr2=0x56016091a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e9ce05640 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56016091d370 0x56016091a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.283+0000 7f9e9ce05640 1 -- 172.21.15.62:0/977900605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601605e93c0 con 0x5601609125d0 2024-09-15T23:22:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.284+0000 7f9e9ce05640 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x56016091cdd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00bed0 tx=0x7f9e8c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb97205a8f7dc14 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.284+0000 7f9e77fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:22:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.285+0000 7f9e77fff640 1 -- 172.21.15.62:0/977900605 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c004020 con 0x5601609125d0 2024-09-15T23:22:15.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.285+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e840bc580 con 0x5601609125d0 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.285+0000 7f9e77fff640 1 -- 172.21.15.62:0/977900605 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e8c0041c0 con 0x5601609125d0 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.285+0000 7f9e77fff640 1 -- 172.21.15.62:0/977900605 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e8c043050 con 0x5601609125d0 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.285+0000 7f9ea1442740 10 client.26296 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.285+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56016092b8d0 con 0x5601609125d0 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.286+0000 7f9e77fff640 1 -- 172.21.15.62:0/977900605 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e8c004400 con 0x5601609125d0 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.286+0000 7f9e77fff640 1 -- 172.21.15.62:0/977900605 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f9e8c003890 con 0x5601609125d0 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.286+0000 7f9ea1442740 10 client.26296 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.286+0000 7f9ea1442740 10 client.26296 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.286+0000 7f9ea1442740 20 client.26296 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.286+0000 7f9ea1442740 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56016092cd90 0x56016092f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.286+0000 7f9ea1442740 4 client.26296 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.287+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9e840bc580 con 0x56016092cd90 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.287+0000 7f9e97fff640 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56016092cd90 0x56016092f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.287+0000 7f9e77fff640 10 client.26296 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.287+0000 7f9e97fff640 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56016092cd90 0x56016092f1e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efe5878f4cbb6631 server_cookie=60c5ca0daf92fbe9 in_seq=0 out_seq=0 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9e77fff640 1 -- 172.21.15.62:0/977900605 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f9e840bc580 con 0x56016092cd90 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9e77fff640 10 client.26296 handle_command_reply: tid=0 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9ea1442740 1 client.26296 shutdown 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 msgr2=0x56016091cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9ea1442740 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x56016091cdd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c00bed0 tx=0x7f9e8c0046c0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 shutdown_connections 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9ea1442740 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56016092cd90 0x56016092f1e0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9ea1442740 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56016091d370 0x56016091a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9ea1442740 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560160913b40 0x56016091a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9ea1442740 1 --2- 172.21.15.62:0/977900605 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601609125d0 0x56016091cdd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.288+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 >> 172.21.15.62:0/977900605 conn(0x5601608db310 msgr2=0x560160914960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.289+0000 7f9e77fff640 0 client.26296 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.289+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 shutdown_connections 2024-09-15T23:22:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.289+0000 7f9ea1442740 1 -- 172.21.15.62:0/977900605 wait complete. 2024-09-15T23:22:15.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.289+0000 7f9ea1442740 20 client.26296 trim_cache size 0 max 16384 2024-09-15T23:22:15.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.306+0000 7f9ea1442740 1 Processor -- start 2024-09-15T23:22:15.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9ea1442740 1 -- start start 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601609125d0 0x56016090e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x56016090ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56016091d370 0x560160934ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9ea1442740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981af500 con 0x56016091d370 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9ea1442740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981acdd0 con 0x5601609125d0 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x56016090ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x56016090ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48792/0 (socket says 172.21.15.62:48792) 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9e97fff640 1 -- 172.21.15.62:0/4209363446 learned_addr learned my addr 172.21.15.62:0/4209363446 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9e9ce05640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601609125d0 0x56016090e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.307+0000 7f9e9d606640 1 --2- 172.21.15.62:0/4209363446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56016091d370 0x560160934ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9ea1442740 1 -- 172.21.15.62:0/4209363446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981aac20 con 0x560160913b40 2024-09-15T23:22:15.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9e97fff640 1 -- 172.21.15.62:0/4209363446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601609125d0 msgr2=0x56016090e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9e97fff640 1 --2- 172.21.15.62:0/4209363446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601609125d0 0x56016090e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9e97fff640 1 -- 172.21.15.62:0/4209363446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56016091d370 msgr2=0x560160934ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9e97fff640 1 --2- 172.21.15.62:0/4209363446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56016091d370 0x560160934ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9e97fff640 1 -- 172.21.15.62:0/4209363446 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56016090f180 con 0x560160913b40 2024-09-15T23:22:15.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9e97fff640 1 --2- 172.21.15.62:0/4209363446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x56016090ec40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9e840b8f50 tx=0x7f9e84038fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cecbdcd698f2fe1d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9e827fc640 1 -- 172.21.15.62:0/4209363446 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e840da070 con 0x560160913b40 2024-09-15T23:22:15.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9e827fc640 1 -- 172.21.15.62:0/4209363446 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e840c0880 con 0x560160913b40 2024-09-15T23:22:15.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.308+0000 7f9e827fc640 1 -- 172.21.15.62:0/4209363446 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84049040 con 0x560160913b40 2024-09-15T23:22:15.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.309+0000 7f9ea1442740 1 -- 172.21.15.62:0/4209363446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 msgr2=0x56016090ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.309+0000 7f9ea1442740 1 --2- 172.21.15.62:0/4209363446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x56016090ec40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9e840b8f50 tx=0x7f9e84038fd0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.310+0000 7f9ea1442740 1 -- 172.21.15.62:0/4209363446 shutdown_connections 2024-09-15T23:22:15.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.310+0000 7f9ea1442740 1 --2- 172.21.15.62:0/4209363446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56016091d370 0x560160934ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.310+0000 7f9ea1442740 1 --2- 172.21.15.62:0/4209363446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x56016090ec40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.310+0000 7f9ea1442740 1 --2- 172.21.15.62:0/4209363446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601609125d0 0x56016090e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.310+0000 7f9ea1442740 1 -- 172.21.15.62:0/4209363446 >> 172.21.15.62:0/4209363446 conn(0x5601608db310 msgr2=0x5601609152c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:15.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.310+0000 7f9ea1442740 1 -- 172.21.15.62:0/4209363446 shutdown_connections 2024-09-15T23:22:15.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.311+0000 7f9ea1442740 1 -- 172.21.15.62:0/4209363446 wait complete. 2024-09-15T23:22:15.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.311+0000 7f9ea1442740 1 -- start start 2024-09-15T23:22:15.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.312+0000 7f9ea1442740 1 Processor -- start 2024-09-15T23:22:15.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.312+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601609125d0 0x560160925200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x560160925740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9ea1442740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 0x5601609221d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9e9ce05640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601609125d0 0x560160925200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9e9ce05640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601609125d0 0x560160925200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47926/0 (socket says 172.21.15.62:47926) 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x560160925740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9e9ce05640 1 -- 172.21.15.62:0/1206001262 learned_addr learned my addr 172.21.15.62:0/1206001262 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9e97fff640 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x560160925740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48796/0 (socket says 172.21.15.62:48796) 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981ab290 con 0x5601609125d0 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981ac700 con 0x56016091d370 2024-09-15T23:22:15.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981ab960 con 0x560160913b40 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9e9d606640 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 0x5601609221d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.313+0000 7f9ea1442740 10 client.0 _ll_register_callbacks cb 0x7f9e8c00b610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e9d606640 1 -- 172.21.15.62:0/1206001262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 msgr2=0x560160925740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e9d606640 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x560160925740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e9d606640 1 -- 172.21.15.62:0/1206001262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601609125d0 msgr2=0x560160925200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e9d606640 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601609125d0 0x560160925200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e9d606640 1 -- 172.21.15.62:0/1206001262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560160925e50 con 0x56016091d370 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e97fff640 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x560160925740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e9d606640 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 0x5601609221d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009aa0 tx=0x7f9e90010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b598d0e3b118f66 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e77fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:15.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e77fff640 1 -- 172.21.15.62:0/1206001262 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e90021060 con 0x56016091d370 2024-09-15T23:22:15.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e840c0880 con 0x56016091d370 2024-09-15T23:22:15.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e77fff640 1 -- 172.21.15.62:0/1206001262 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e9000e7c0 con 0x56016091d370 2024-09-15T23:22:15.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9e77fff640 1 -- 172.21.15.62:0/1206001262 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e9000e9c0 con 0x56016091d370 2024-09-15T23:22:15.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9ea1442740 10 client.26489 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:15.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.314+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560160998130 con 0x56016091d370 2024-09-15T23:22:15.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.315+0000 7f9e77fff640 1 -- 172.21.15.62:0/1206001262 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e840c0880 con 0x56016091d370 2024-09-15T23:22:15.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.316+0000 7f9e77fff640 1 -- 172.21.15.62:0/1206001262 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f9e9001b070 con 0x56016091d370 2024-09-15T23:22:15.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.319+0000 7f9ea1442740 10 client.26489 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:15.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.319+0000 7f9ea1442740 10 client.26489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:15.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.319+0000 7f9ea1442740 20 client.26489 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:15.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.319+0000 7f9ea1442740 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5601609987b0 0x56016099abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.319+0000 7f9ea1442740 4 client.26489 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:22:15.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.319+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560160885860 con 0x5601609987b0 2024-09-15T23:22:15.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.322+0000 7f9e9ce05640 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5601609987b0 0x56016099abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.327+0000 7f9e9ce05640 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5601609987b0 0x56016099abd0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7961efddb40fb4ed server_cookie=a5fa3c8c0753dbe9 in_seq=0 out_seq=0 2024-09-15T23:22:15.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.327+0000 7f9e77fff640 10 client.26489 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:15.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.332+0000 7f9e77fff640 1 -- 172.21.15.62:0/1206001262 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560160885860 con 0x5601609987b0 2024-09-15T23:22:15.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.332+0000 7f9e77fff640 10 client.26489 handle_command_reply: tid=0 2024-09-15T23:22:15.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.332+0000 7f9ea1442740 1 client.26489 shutdown 2024-09-15T23:22:15.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.336+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 msgr2=0x5601609221d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.336+0000 7f9ea1442740 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 0x5601609221d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9e90009aa0 tx=0x7f9e90010800 comp rx=0 tx=0).stop 2024-09-15T23:22:15.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.340+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 shutdown_connections 2024-09-15T23:22:15.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.340+0000 7f9ea1442740 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5601609987b0 0x56016099abd0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.340+0000 7f9e77fff640 0 client.26489 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:15.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.340+0000 7f9ea1442740 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56016091d370 0x5601609221d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.340+0000 7f9ea1442740 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560160913b40 0x560160925740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.340+0000 7f9ea1442740 1 --2- 172.21.15.62:0/1206001262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601609125d0 0x560160925200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.340+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 >> 172.21.15.62:0/1206001262 conn(0x5601608db310 msgr2=0x56016090e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:15.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.344+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 shutdown_connections 2024-09-15T23:22:15.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.345+0000 7f9ea1442740 1 -- 172.21.15.62:0/1206001262 wait complete. 2024-09-15T23:22:15.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.345+0000 7f9ea1442740 20 client.26489 trim_cache size 0 max 16384 2024-09-15T23:22:15.347 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:22:15.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 -- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e880a32c0 msgr2=0x7f9e880a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e880a32c0 0x7f9e880a5780 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0095f0 tx=0x7f9e8c016040 comp rx=0 tx=0).stop 2024-09-15T23:22:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 -- 172.21.15.62:0/3136864982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e98102570 msgr2=0x7f9e98108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e98102570 0x7f9e98108020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9e84000c00 tx=0x7f9e84003b90 comp rx=0 tx=0).stop 2024-09-15T23:22:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 -- 172.21.15.62:0/3136864982 shutdown_connections 2024-09-15T23:22:15.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e880a32c0 0x7f9e880a5780 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e98102ef0 0x7f9e98108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e98102570 0x7f9e98108020 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9e84000c00 tx=0x7f9e84003b90 comp rx=0 tx=0).stop 2024-09-15T23:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 --2- 172.21.15.62:0/3136864982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e98100e20 0x7f9e98107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 -- 172.21.15.62:0/3136864982 >> 172.21.15.62:0/3136864982 conn(0x7f9e980fc7d0 msgr2=0x7f9e98103880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 -- 172.21.15.62:0/3136864982 shutdown_connections 2024-09-15T23:22:15.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:15.347+0000 7f9e9f069640 1 -- 172.21.15.62:0/3136864982 wait complete. 2024-09-15T23:22:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.939+0000 7f1964065640 1 -- 172.21.15.62:0/901668297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c104f00 msgr2=0x7f195c1072f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.939+0000 7f1964065640 1 --2- 172.21.15.62:0/901668297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c104f00 0x7f195c1072f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1950009900 tx=0x7f195002a410 comp rx=0 tx=0).stop 2024-09-15T23:22:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.939+0000 7f1964065640 1 -- 172.21.15.62:0/901668297 shutdown_connections 2024-09-15T23:22:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.939+0000 7f1964065640 1 --2- 172.21.15.62:0/901668297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f195c107830 0x7f195c109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.939+0000 7f1964065640 1 --2- 172.21.15.62:0/901668297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c104f00 0x7f195c1072f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.939+0000 7f1964065640 1 --2- 172.21.15.62:0/901668297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f195c1025d0 0x7f195c1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.939+0000 7f1964065640 1 -- 172.21.15.62:0/901668297 >> 172.21.15.62:0/901668297 conn(0x7f195c0fc1b0 msgr2=0x7f195c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.939+0000 7f1964065640 1 -- 172.21.15.62:0/901668297 shutdown_connections 2024-09-15T23:22:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.939+0000 7f1964065640 1 -- 172.21.15.62:0/901668297 wait complete. 2024-09-15T23:22:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.941+0000 7f1964065640 1 Processor -- start 2024-09-15T23:22:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.941+0000 7f1964065640 1 -- start start 2024-09-15T23:22:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.941+0000 7f1964065640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c1025d0 0x7f195c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.944 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.941+0000 7f1964065640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f195c104f00 0x7f195c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.941+0000 7f1964065640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f195c107830 0x7f195c19f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.941+0000 7f1961e01640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c1025d0 0x7f195c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961e01640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c1025d0 0x7f195c197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41052/0 (socket says 172.21.15.62:41052) 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f195c104f00 0x7f195c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961e01640 1 -- 172.21.15.62:0/2268867151 learned_addr learned my addr 172.21.15.62:0/2268867151 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961600640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f195c104f00 0x7f195c1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47946/0 (socket says 172.21.15.62:47946) 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f195c10d7e0 con 0x7f195c104f00 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f195c10d660 con 0x7f195c1025d0 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f195c10d960 con 0x7f195c107830 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1962602640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f195c107830 0x7f195c19f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961e01640 1 -- 172.21.15.62:0/2268867151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f195c107830 msgr2=0x7f195c19f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961e01640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f195c107830 0x7f195c19f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961e01640 1 -- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f195c104f00 msgr2=0x7f195c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961e01640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f195c104f00 0x7f195c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961e01640 1 -- 172.21.15.62:0/2268867151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195c19fab0 con 0x7f195c1025d0 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961600640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f195c104f00 0x7f195c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1962602640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f195c107830 0x7f195c19f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1961e01640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c1025d0 0x7f195c197e90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f194c00d950 tx=0x7f194c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f195affd640 1 -- 172.21.15.62:0/2268867151 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c017030 con 0x7f195c1025d0 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f195affd640 1 -- 172.21.15.62:0/2268867151 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f194c004020 con 0x7f195c1025d0 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195c19fda0 con 0x7f195c1025d0 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.942+0000 7f195affd640 1 -- 172.21.15.62:0/2268867151 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0047a0 con 0x7f195c1025d0 2024-09-15T23:22:15.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.943+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195c1a0260 con 0x7f195c1025d0 2024-09-15T23:22:15.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.945+0000 7f195affd640 1 -- 172.21.15.62:0/2268867151 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f194c0041d0 con 0x7f195c1025d0 2024-09-15T23:22:15.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.946+0000 7f1966e03740 1 Processor -- start 2024-09-15T23:22:15.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.946+0000 7f1966e03740 1 -- start start 2024-09-15T23:22:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.956+0000 7f1966e03740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7276da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.956+0000 7f1966e03740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd72772e0 0x558cd727bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.956+0000 7f1966e03740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 0x558cd727e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.956+0000 7f1966e03740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558cd71da2f0 con 0x558cd72769a0 2024-09-15T23:22:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.956+0000 7f1966e03740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558cd7182270 con 0x558cd727c370 2024-09-15T23:22:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.959+0000 7f1962602640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7276da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.959+0000 7f1962602640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7276da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47954/0 (socket says 172.21.15.62:47954) 2024-09-15T23:22:15.966 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.959+0000 7f1962602640 1 -- 172.21.15.62:0/3372058398 learned_addr learned my addr 172.21.15.62:0/3372058398 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.959+0000 7f1966e03740 1 -- 172.21.15.62:0/3372058398 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558cd71d62b0 con 0x558cd72772e0 2024-09-15T23:22:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.959+0000 7f1961600640 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 0x558cd727e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.964+0000 7f1961e01640 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd72772e0 0x558cd727bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.968+0000 7f195affd640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19440abc00 0x7f19440ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.968+0000 7f1962602640 1 -- 172.21.15.62:0/3372058398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd72772e0 msgr2=0x558cd727bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.968+0000 7f1962602640 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd72772e0 0x558cd727bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.968+0000 7f1962602640 1 -- 172.21.15.62:0/3372058398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 msgr2=0x558cd727e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.968+0000 7f1962602640 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 0x558cd727e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.968+0000 7f1962602640 1 -- 172.21.15.62:0/3372058398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cd7181340 con 0x558cd72769a0 2024-09-15T23:22:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.968+0000 7f195affd640 1 -- 172.21.15.62:0/2268867151 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f194c0cf800 con 0x7f195c1025d0 2024-09-15T23:22:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.968+0000 7f1961600640 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 0x558cd727e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.974+0000 7f1962602640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19440abc00 0x7f19440ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.977 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.976+0000 7f1962602640 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7276da0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1954010fb0 tx=0x7f19540099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4578329b16a53fa9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.977+0000 7f1943fff640 1 -- 172.21.15.62:0/3372058398 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954022070 con 0x558cd72769a0 2024-09-15T23:22:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.977+0000 7f1943fff640 1 -- 172.21.15.62:0/3372058398 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f195401c960 con 0x558cd72769a0 2024-09-15T23:22:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.977+0000 7f1943fff640 1 -- 172.21.15.62:0/3372058398 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195401a8b0 con 0x558cd72769a0 2024-09-15T23:22:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.978+0000 7f1966e03740 1 -- 172.21.15.62:0/3372058398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 msgr2=0x558cd7276da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.978+0000 7f1966e03740 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7276da0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1954010fb0 tx=0x7f19540099d0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:15 smithi062 ceph-mon[30949]: pgmap v549: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 639 op/s 2024-09-15T23:22:15.982 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.981+0000 7f1962602640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19440abc00 0x7f19440ae0c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f195401f000 tx=0x7f1954020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.982+0000 7f1966e03740 1 -- 172.21.15.62:0/3372058398 shutdown_connections 2024-09-15T23:22:15.983 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.982+0000 7f1966e03740 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 0x558cd727e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.982+0000 7f1966e03740 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd72772e0 0x558cd727bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.982+0000 7f1966e03740 1 --2- 172.21.15.62:0/3372058398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7276da0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.984 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.982+0000 7f1966e03740 1 -- 172.21.15.62:0/3372058398 >> 172.21.15.62:0/3372058398 conn(0x558cd7272010 msgr2=0x558cd7272410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:15.986 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.985+0000 7f1966e03740 1 -- 172.21.15.62:0/3372058398 shutdown_connections 2024-09-15T23:22:15.989 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.988+0000 7f1966e03740 1 -- 172.21.15.62:0/3372058398 wait complete. 2024-09-15T23:22:15.989 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.988+0000 7f1966e03740 1 -- start start 2024-09-15T23:22:15.990 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.989+0000 7f1966e03740 1 Processor -- start 2024-09-15T23:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.989+0000 7f1966e03740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7280c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.990+0000 7f1966e03740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd72772e0 0x558cd7281180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.990+0000 7f1966e03740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 0x558cd72881c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.990+0000 7f1966e03740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558cd727f390 con 0x558cd72769a0 2024-09-15T23:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.990+0000 7f1966e03740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558cd7163290 con 0x558cd727c370 2024-09-15T23:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.990+0000 7f1966e03740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558cd727f510 con 0x558cd72772e0 2024-09-15T23:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.990+0000 7f1966e03740 10 client.0 _ll_register_callbacks cb 0x558cd7194140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.991+0000 7f1961e01640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7280c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.991+0000 7f1961e01640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7280c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47968/0 (socket says 172.21.15.62:47968) 2024-09-15T23:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.991+0000 7f1961e01640 1 -- 172.21.15.62:0/2178496795 learned_addr learned my addr 172.21.15.62:0/2178496795 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.991+0000 7f1961e01640 1 -- 172.21.15.62:0/2178496795 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd72772e0 msgr2=0x558cd7281180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.991+0000 7f1961e01640 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd72772e0 0x558cd7281180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.991+0000 7f1961e01640 1 -- 172.21.15.62:0/2178496795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 msgr2=0x558cd72881c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.991+0000 7f1961e01640 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 0x558cd72881c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.991+0000 7f1961e01640 1 -- 172.21.15.62:0/2178496795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cd6eed210 con 0x558cd72769a0 2024-09-15T23:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.992+0000 7f1961e01640 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7280c40 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f194c09b9d0 tx=0x7f194c09b600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20856c7dad483af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.992+0000 7f19417fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:22:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.993+0000 7f19417fa640 1 -- 172.21.15.62:0/2178496795 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c09fdf0 con 0x558cd72769a0 2024-09-15T23:22:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.993+0000 7f1966e03740 1 -- 172.21.15.62:0/2178496795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f195401c960 con 0x558cd72769a0 2024-09-15T23:22:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.993+0000 7f19417fa640 1 -- 172.21.15.62:0/2178496795 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f194c086070 con 0x558cd72769a0 2024-09-15T23:22:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.994+0000 7f19417fa640 1 -- 172.21.15.62:0/2178496795 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f194c0a46b0 con 0x558cd72769a0 2024-09-15T23:22:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.994+0000 7f1966e03740 10 client.16935 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.994+0000 7f1966e03740 1 -- 172.21.15.62:0/2178496795 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558cd6fb9620 con 0x558cd72769a0 2024-09-15T23:22:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.995+0000 7f19417fa640 1 -- 172.21.15.62:0/2178496795 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f195401c960 con 0x558cd72769a0 2024-09-15T23:22:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.995+0000 7f19417fa640 1 -- 172.21.15.62:0/2178496795 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f194c0cfbb0 con 0x558cd72769a0 2024-09-15T23:22:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.995+0000 7f1966e03740 10 client.16935 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.995+0000 7f1966e03740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.995+0000 7f1966e03740 20 client.16935 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.995+0000 7f1966e03740 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558cd728cce0 0x558cd728f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.995+0000 7f1966e03740 4 client.16935 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.995+0000 7f1966e03740 1 -- 172.21.15.62:0/2178496795 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cd71e7a20 con 0x558cd728cce0 2024-09-15T23:22:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:15.997+0000 7f1961600640 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558cd728cce0 0x558cd728f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.000+0000 7f1961600640 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558cd728cce0 0x558cd728f180 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92eb2832cd8e19fd server_cookie=da27ef5a7f09efec in_seq=0 out_seq=0 2024-09-15T23:22:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.000+0000 7f19417fa640 10 client.16935 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.002+0000 7f19417fa640 1 -- 172.21.15.62:0/2178496795 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558cd71e7a20 con 0x558cd728cce0 2024-09-15T23:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.002+0000 7f19417fa640 10 client.16935 handle_command_reply: tid=0 2024-09-15T23:22:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.003+0000 7f1966e03740 1 client.16935 shutdown 2024-09-15T23:22:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.006+0000 7f1966e03740 1 -- 172.21.15.62:0/2178496795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 msgr2=0x558cd7280c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.006+0000 7f1966e03740 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7280c40 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f194c09b9d0 tx=0x7f194c09b600 comp rx=0 tx=0).stop 2024-09-15T23:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.009+0000 7f1966e03740 1 -- 172.21.15.62:0/2178496795 shutdown_connections 2024-09-15T23:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.009+0000 7f1966e03740 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558cd728cce0 0x558cd728f180 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.009+0000 7f1966e03740 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd727c370 0x558cd72881c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.009+0000 7f1966e03740 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd72772e0 0x558cd7281180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.009+0000 7f1966e03740 1 --2- 172.21.15.62:0/2178496795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7280c40 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.009+0000 7f1966e03740 1 -- 172.21.15.62:0/2178496795 >> 172.21.15.62:0/2178496795 conn(0x558cd7272010 msgr2=0x558cd7273820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.009+0000 7f19417fa640 0 client.16935 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.010+0000 7f1966e03740 1 -- 172.21.15.62:0/2178496795 shutdown_connections 2024-09-15T23:22:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.010+0000 7f1966e03740 1 -- 172.21.15.62:0/2178496795 wait complete. 2024-09-15T23:22:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.010+0000 7f1966e03740 20 client.16935 trim_cache size 0 max 16384 2024-09-15T23:22:16.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:15 smithi067 ceph-mon[35665]: pgmap v549: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 639 op/s 2024-09-15T23:22:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.034+0000 7f1966e03740 1 Processor -- start 2024-09-15T23:22:16.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.034+0000 7f1966e03740 1 -- start start 2024-09-15T23:22:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.034+0000 7f1966e03740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7281100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1966e03740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72772e0 0x558cd7288180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1966e03740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 0x558cd7272f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1966e03740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1aaf70 con 0x558cd72769a0 2024-09-15T23:22:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1966e03740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1a8840 con 0x558cd72772e0 2024-09-15T23:22:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1966e03740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1a6690 con 0x558cd727c370 2024-09-15T23:22:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1961600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72772e0 0x558cd7288180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1961600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72772e0 0x558cd7288180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41084/0 (socket says 172.21.15.62:41084) 2024-09-15T23:22:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1961600640 1 -- 172.21.15.62:0/3049440276 learned_addr learned my addr 172.21.15.62:0/3049440276 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1962602640 1 --2- 172.21.15.62:0/3049440276 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 0x558cd7272f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.035+0000 7f1961e01640 1 --2- 172.21.15.62:0/3049440276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7281100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.036+0000 7f1962602640 1 -- 172.21.15.62:0/3049440276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72772e0 msgr2=0x558cd7288180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.036+0000 7f1962602640 1 --2- 172.21.15.62:0/3049440276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72772e0 0x558cd7288180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.036+0000 7f1962602640 1 -- 172.21.15.62:0/3049440276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 msgr2=0x558cd7281100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.036+0000 7f1962602640 1 --2- 172.21.15.62:0/3049440276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7281100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.036+0000 7f1962602640 1 -- 172.21.15.62:0/3049440276 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cd727f680 con 0x558cd727c370 2024-09-15T23:22:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.036+0000 7f1962602640 1 --2- 172.21.15.62:0/3049440276 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 0x558cd7272f20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1954009ed0 tx=0x7f195401e030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ee70164a47f84bc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.036+0000 7f19437fe640 1 -- 172.21.15.62:0/3049440276 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195402d070 con 0x558cd727c370 2024-09-15T23:22:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.036+0000 7f19437fe640 1 -- 172.21.15.62:0/3049440276 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f195401ad70 con 0x558cd727c370 2024-09-15T23:22:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.036+0000 7f19437fe640 1 -- 172.21.15.62:0/3049440276 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954022030 con 0x558cd727c370 2024-09-15T23:22:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.037+0000 7f1966e03740 1 -- 172.21.15.62:0/3049440276 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 msgr2=0x558cd7272f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.037+0000 7f1966e03740 1 --2- 172.21.15.62:0/3049440276 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 0x558cd7272f20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1954009ed0 tx=0x7f195401e030 comp rx=0 tx=0).stop 2024-09-15T23:22:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.037+0000 7f1966e03740 1 -- 172.21.15.62:0/3049440276 shutdown_connections 2024-09-15T23:22:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.037+0000 7f1966e03740 1 --2- 172.21.15.62:0/3049440276 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 0x558cd7272f20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.037+0000 7f1966e03740 1 --2- 172.21.15.62:0/3049440276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72772e0 0x558cd7288180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.037+0000 7f1966e03740 1 --2- 172.21.15.62:0/3049440276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72769a0 0x558cd7281100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.037+0000 7f1966e03740 1 -- 172.21.15.62:0/3049440276 >> 172.21.15.62:0/3049440276 conn(0x558cd7272010 msgr2=0x558cd72809f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.037+0000 7f1966e03740 1 -- 172.21.15.62:0/3049440276 shutdown_connections 2024-09-15T23:22:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.037+0000 7f1966e03740 1 -- 172.21.15.62:0/3049440276 wait complete. 2024-09-15T23:22:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.038+0000 7f1966e03740 1 -- start start 2024-09-15T23:22:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.038+0000 7f1966e03740 1 Processor -- start 2024-09-15T23:22:16.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:15 smithi148 ceph-mon[35610]: pgmap v549: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 639 op/s 2024-09-15T23:22:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.038+0000 7f1966e03740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72769a0 0x558cd7288150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1966e03740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72772e0 0x558cd7284c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1961e01640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72769a0 0x558cd7288150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1961e01640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72769a0 0x558cd7288150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41086/0 (socket says 172.21.15.62:41086) 2024-09-15T23:22:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1961e01640 1 -- 172.21.15.62:0/4283532135 learned_addr learned my addr 172.21.15.62:0/4283532135 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1961600640 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72772e0 0x558cd7284c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1966e03740 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 0x558cd72852c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1a6d00 con 0x558cd72772e0 2024-09-15T23:22:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1a8170 con 0x558cd72769a0 2024-09-15T23:22:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1a73d0 con 0x558cd727c370 2024-09-15T23:22:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1962602640 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 0x558cd72852c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1966e03740 10 client.0 _ll_register_callbacks cb 0x7f194c09b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1961e01640 1 -- 172.21.15.62:0/4283532135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 msgr2=0x558cd72852c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1961e01640 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 0x558cd72852c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1961e01640 1 -- 172.21.15.62:0/4283532135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72772e0 msgr2=0x558cd7284c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1961e01640 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72772e0 0x558cd7284c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.039+0000 7f1961e01640 1 -- 172.21.15.62:0/4283532135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cd72f5b20 con 0x558cd72769a0 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.040+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f195401ad70 con 0x558cd72769a0 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.040+0000 7f1961e01640 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72769a0 0x558cd7288150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f194c005fd0 tx=0x7f194c0982e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf96761600122d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.040+0000 7f19417fa640 10 client.26504 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.040+0000 7f19417fa640 1 -- 172.21.15.62:0/4283532135 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c09fdf0 con 0x558cd72769a0 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.040+0000 7f19417fa640 1 -- 172.21.15.62:0/4283532135 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f194c0a4220 con 0x558cd72769a0 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.040+0000 7f19417fa640 1 -- 172.21.15.62:0/4283532135 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f194c0a4440 con 0x558cd72769a0 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f1966e03740 10 client.26504 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=32}) -- 0x558cd72f6d30 con 0x558cd72769a0 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f19417fa640 1 -- 172.21.15.62:0/4283532135 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f194c0ba1b0 con 0x558cd72769a0 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f1966e03740 10 client.26504 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f1966e03740 10 client.26504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f1966e03740 20 client.26504 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f1966e03740 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558cd72f7470 0x558cd72f98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f1966e03740 4 client.26504 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:22:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558cd71e1d70 con 0x558cd72f7470 2024-09-15T23:22:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.041+0000 7f1961600640 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558cd72f7470 0x558cd72f98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.042+0000 7f19417fa640 10 client.26504 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.042+0000 7f1961600640 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558cd72f7470 0x558cd72f98b0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b9373d8ace84823 server_cookie=92c99127938d8e07 in_seq=0 out_seq=0 2024-09-15T23:22:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.062+0000 7f19417fa640 1 -- 172.21.15.62:0/4283532135 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558cd71e1d70 con 0x558cd72f7470 2024-09-15T23:22:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.062+0000 7f19417fa640 10 client.26504 handle_command_reply: tid=0 2024-09-15T23:22:16.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.062+0000 7f1966e03740 1 client.26504 shutdown 2024-09-15T23:22:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.064+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72769a0 msgr2=0x558cd7288150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:16.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.064+0000 7f1966e03740 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72769a0 0x558cd7288150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f194c005fd0 tx=0x7f194c0982e0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.064+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 shutdown_connections 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.064+0000 7f1966e03740 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558cd72f7470 0x558cd72f98b0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.064+0000 7f1966e03740 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558cd727c370 0x558cd72852c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.064+0000 7f1966e03740 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558cd72772e0 0x558cd7284c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.064+0000 7f1966e03740 1 --2- 172.21.15.62:0/4283532135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558cd72769a0 0x558cd7288150 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.064+0000 7f19417fa640 0 client.26504 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.064+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 >> 172.21.15.62:0/4283532135 conn(0x558cd7272010 msgr2=0x558cd72709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.065+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 shutdown_connections 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.065+0000 7f1966e03740 1 -- 172.21.15.62:0/4283532135 wait complete. 2024-09-15T23:22:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.065+0000 7f1966e03740 20 client.26504 trim_cache size 0 max 16384 2024-09-15T23:22:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:22:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:22:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:22:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:22:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19440abc00 msgr2=0x7f19440ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19440abc00 0x7f19440ae0c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f195401f000 tx=0x7f1954020000 comp rx=0 tx=0).stop 2024-09-15T23:22:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c1025d0 msgr2=0x7f195c197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c1025d0 0x7f195c197e90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f194c00d950 tx=0x7f194c00de10 comp rx=0 tx=0).stop 2024-09-15T23:22:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 shutdown_connections 2024-09-15T23:22:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f19440abc00 0x7f19440ae0c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f195c107830 0x7f195c19f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f195c104f00 0x7f195c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 --2- 172.21.15.62:0/2268867151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f195c1025d0 0x7f195c197e90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 >> 172.21.15.62:0/2268867151 conn(0x7f195c0fc1b0 msgr2=0x7f195c102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 shutdown_connections 2024-09-15T23:22:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:16.066+0000 7f1964065640 1 -- 172.21.15.62:0/2268867151 wait complete. 2024-09-15T23:22:17.128 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:22:17.128 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:22:17.541 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:17.964 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:17 smithi062 ceph-mon[30949]: pgmap v550: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 MiB/s wr, 581 op/s 2024-09-15T23:22:18.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:17 smithi067 ceph-mon[35665]: pgmap v550: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 MiB/s wr, 581 op/s 2024-09-15T23:22:18.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:17 smithi148 ceph-mon[35610]: pgmap v550: 129 pgs: 129 active+clean; 1.4 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 MiB/s wr, 581 op/s 2024-09-15T23:22:18.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.101+0000 7fb1ca090640 1 -- 172.21.15.62:0/2915662241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4102570 msgr2=0x7fb1c41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:18.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.101+0000 7fb1ca090640 1 --2- 172.21.15.62:0/2915662241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4102570 0x7fb1c41029b0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009ca0 tx=0x7fb1bc02a960 comp rx=0 tx=0).stop 2024-09-15T23:22:18.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.102+0000 7fb1ca090640 1 -- 172.21.15.62:0/2915662241 shutdown_connections 2024-09-15T23:22:18.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.102+0000 7fb1ca090640 1 --2- 172.21.15.62:0/2915662241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1c4102ef0 0x7fb1c4069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.102+0000 7fb1ca090640 1 --2- 172.21.15.62:0/2915662241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c4102570 0x7fb1c41029b0 secure :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009ca0 tx=0x7fb1bc02a960 comp rx=0 tx=0).stop 2024-09-15T23:22:18.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.102+0000 7fb1ca090640 1 --2- 172.21.15.62:0/2915662241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c4100e20 0x7fb1c4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.102+0000 7fb1ca090640 1 -- 172.21.15.62:0/2915662241 >> 172.21.15.62:0/2915662241 conn(0x7fb1c40fc7d0 msgr2=0x7fb1c40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:18.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.104+0000 7fb1ca090640 1 -- 172.21.15.62:0/2915662241 shutdown_connections 2024-09-15T23:22:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.116+0000 7fb1ca090640 1 -- 172.21.15.62:0/2915662241 wait complete. 2024-09-15T23:22:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.116+0000 7fb1ca090640 1 Processor -- start 2024-09-15T23:22:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.117+0000 7fb1ca090640 1 -- start start 2024-09-15T23:22:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.117+0000 7fb1ca090640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1c4100e20 0x7fb1c4073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.117+0000 7fb1ca090640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c4102ef0 0x7fb1c406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.117+0000 7fb1ca090640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c406d120 0x7fb1c406d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.117+0000 7fb1ca090640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c4114390 con 0x7fb1c406d120 2024-09-15T23:22:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.117+0000 7fb1ca090640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c4114210 con 0x7fb1c4102ef0 2024-09-15T23:22:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.117+0000 7fb1ca090640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c4114510 con 0x7fb1c4100e20 2024-09-15T23:22:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.117+0000 7fb1c2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c4102ef0 0x7fb1c406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.118+0000 7fb1c2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c4102ef0 0x7fb1c406cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41096/0 (socket says 172.21.15.62:41096) 2024-09-15T23:22:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.118+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/1526944799 learned_addr learned my addr 172.21.15.62:0/1526944799 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:18.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.118+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/1526944799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1c4100e20 msgr2=0x7fb1c4073b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:22:18.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.118+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1c4100e20 0x7fb1c4073b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.118+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/1526944799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c406d120 msgr2=0x7fb1c406d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:18.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.118+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c406d120 0x7fb1c406d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.118+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/1526944799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c4071750 con 0x7fb1c4102ef0 2024-09-15T23:22:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.118+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c4102ef0 0x7fb1c406cbe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc000f80 tx=0x7fb1bc02f5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.120+0000 7fb1c0ff9640 1 -- 172.21.15.62:0/1526944799 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc034040 con 0x7fb1c4102ef0 2024-09-15T23:22:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.120+0000 7fb1c0ff9640 1 -- 172.21.15.62:0/1526944799 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1bc030b40 con 0x7fb1c4102ef0 2024-09-15T23:22:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.120+0000 7fb1c0ff9640 1 -- 172.21.15.62:0/1526944799 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc04c450 con 0x7fb1c4102ef0 2024-09-15T23:22:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.120+0000 7fb1ca090640 1 -- 172.21.15.62:0/1526944799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c40719e0 con 0x7fb1c4102ef0 2024-09-15T23:22:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.120+0000 7fb1ca090640 1 -- 172.21.15.62:0/1526944799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c4071e70 con 0x7fb1c4102ef0 2024-09-15T23:22:18.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.126+0000 7fb1c0ff9640 1 -- 172.21.15.62:0/1526944799 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb1bc037020 con 0x7fb1c4102ef0 2024-09-15T23:22:18.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.128+0000 7fb1c0ff9640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1ac0a33d0 0x7fb1ac0a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.128+0000 7fb1cc469740 1 Processor -- start 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.128+0000 7fb1cc469740 1 -- start start 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.128+0000 7fb1c0ff9640 1 -- 172.21.15.62:0/1526944799 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb1bc0f9f00 con 0x7fb1c4102ef0 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.128+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa570eb90 0x560fa570ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.128+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5719830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.128+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5719d70 0x560fa571c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1cc469740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560fa5674280 con 0x560fa5719d70 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1cc469740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560fa561bb10 con 0x560fa570f4b0 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1cc469740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560fa5670240 con 0x560fa570eb90 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1c2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5719830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1c2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5719830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41100/0 (socket says 172.21.15.62:41100) 2024-09-15T23:22:18.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/765668390 learned_addr learned my addr 172.21.15.62:0/765668390 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1c3fff640 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5719d70 0x560fa571c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1c37fe640 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa570eb90 0x560fa570ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/765668390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa570eb90 msgr2=0x560fa570ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa570eb90 0x560fa570ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/765668390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5719d70 msgr2=0x560fa571c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.129+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5719d70 0x560fa571c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.130+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/765668390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa56281f0 con 0x560fa570f4b0 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.130+0000 7fb1c3fff640 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5719d70 0x560fa571c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.130+0000 7fb1c37fe640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1ac0a33d0 0x7fb1ac0a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.130+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5719830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc02aed0 tx=0x7fb1bc02a960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d9f9a339c0c439b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:18.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.132+0000 7fb1a9ffb640 1 -- 172.21.15.62:0/765668390 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc02fdc0 con 0x560fa570f4b0 2024-09-15T23:22:18.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.132+0000 7fb1a9ffb640 1 -- 172.21.15.62:0/765668390 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1bc0c61f0 con 0x560fa570f4b0 2024-09-15T23:22:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.133+0000 7fb1cc469740 1 -- 172.21.15.62:0/765668390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 msgr2=0x560fa5719830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.133+0000 7fb1cc469740 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5719830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc02aed0 tx=0x7fb1bc02a960 comp rx=0 tx=0).stop 2024-09-15T23:22:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.133+0000 7fb1cc469740 1 -- 172.21.15.62:0/765668390 shutdown_connections 2024-09-15T23:22:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.133+0000 7fb1cc469740 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5719d70 0x560fa571c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.133+0000 7fb1cc469740 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5719830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.133+0000 7fb1cc469740 1 --2- 172.21.15.62:0/765668390 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa570eb90 0x560fa570ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.133+0000 7fb1cc469740 1 -- 172.21.15.62:0/765668390 >> 172.21.15.62:0/765668390 conn(0x560fa570b120 msgr2=0x560fa570b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:18.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.134+0000 7fb1c37fe640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1ac0a33d0 0x7fb1ac0a5890 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4008fd0 tx=0x7fb1b4006860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:18.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.134+0000 7fb1cc469740 1 -- 172.21.15.62:0/765668390 shutdown_connections 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.134+0000 7fb1cc469740 1 -- 172.21.15.62:0/765668390 wait complete. 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.135+0000 7fb1cc469740 1 -- start start 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.135+0000 7fb1cc469740 1 Processor -- start 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa570eb90 0x560fa5716c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5717150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5717690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1cc469740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560fa561bb10 con 0x560fa570eb90 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1cc469740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560fa562ef10 con 0x560fa570f4b0 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1cc469740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560fa571c700 con 0x560fa5719d70 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1cc469740 10 client.0 _ll_register_callbacks cb 0x560fa55f7a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1c3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5717690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1c2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5717150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1c3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5717690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48882/0 (socket says 172.21.15.62:48882) 2024-09-15T23:22:18.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.136+0000 7fb1c3fff640 1 -- 172.21.15.62:0/1642603877 learned_addr learned my addr 172.21.15.62:0/1642603877 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:18.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.138+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/1642603877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 msgr2=0x560fa5717690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:18.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.138+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5717690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.138+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/1642603877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa570eb90 msgr2=0x560fa5716c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:18.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.138+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa570eb90 0x560fa5716c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.138+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/1642603877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa53e56a0 con 0x560fa570f4b0 2024-09-15T23:22:18.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.138+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5717150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x560fa570bf80 tx=0x7fb1bc02fdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96421aab42c3ddbf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:18.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.138+0000 7fb1a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:18.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.139+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/1642603877 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0cedb0 con 0x560fa570f4b0 2024-09-15T23:22:18.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.139+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/1642603877 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb1bc0c2860 con 0x560fa570f4b0 2024-09-15T23:22:18.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.140+0000 7fb1cc469740 1 -- 172.21.15.62:0/1642603877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1bc0c61f0 con 0x560fa570f4b0 2024-09-15T23:22:18.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.142+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/1642603877 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1bc0c44f0 con 0x560fa570f4b0 2024-09-15T23:22:18.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.142+0000 7fb1cc469740 10 client.26513 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:18.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.142+0000 7fb1cc469740 1 -- 172.21.15.62:0/1642603877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560fa571e290 con 0x560fa570f4b0 2024-09-15T23:22:18.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.143+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/1642603877 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1bc0c5640 con 0x560fa570f4b0 2024-09-15T23:22:18.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.144+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/1642603877 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7fb1bc054030 con 0x560fa570f4b0 2024-09-15T23:22:18.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.144+0000 7fb1cc469740 10 client.26513 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:18.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.144+0000 7fb1cc469740 10 client.26513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:18.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.144+0000 7fb1cc469740 20 client.26513 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:18.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.144+0000 7fb1cc469740 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560fa571f740 0x560fa5721b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.144+0000 7fb1cc469740 4 client.26513 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:18.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.144+0000 7fb1cc469740 1 -- 172.21.15.62:0/1642603877 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb1bc0c61f0 con 0x560fa571f740 2024-09-15T23:22:18.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.144+0000 7fb1c37fe640 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560fa571f740 0x560fa5721b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.145+0000 7fb1c37fe640 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560fa571f740 0x560fa5721b60 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbcc90c3f738f27b server_cookie=e4908d890a39c130 in_seq=0 out_seq=0 2024-09-15T23:22:18.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.145+0000 7fb1a2ffd640 10 client.26513 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.146+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/1642603877 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fb1bc0c61f0 con 0x560fa571f740 2024-09-15T23:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.146+0000 7fb1a2ffd640 10 client.26513 handle_command_reply: tid=0 2024-09-15T23:22:18.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.146+0000 7fb1cc469740 1 client.26513 shutdown 2024-09-15T23:22:18.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.148+0000 7fb1cc469740 1 -- 172.21.15.62:0/1642603877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 msgr2=0x560fa5717150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:18.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.148+0000 7fb1cc469740 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5717150 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x560fa570bf80 tx=0x7fb1bc02fdc0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.148+0000 7fb1cc469740 1 -- 172.21.15.62:0/1642603877 shutdown_connections 2024-09-15T23:22:18.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.148+0000 7fb1cc469740 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560fa571f740 0x560fa5721b60 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.148+0000 7fb1cc469740 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5717690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.148+0000 7fb1cc469740 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570f4b0 0x560fa5717150 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x560fa570bf80 tx=0x7fb1bc02fdc0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.148+0000 7fb1cc469740 1 --2- 172.21.15.62:0/1642603877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa570eb90 0x560fa5716c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.148+0000 7fb1cc469740 1 -- 172.21.15.62:0/1642603877 >> 172.21.15.62:0/1642603877 conn(0x560fa570b120 msgr2=0x560fa570a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:18.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.149+0000 7fb1a2ffd640 0 client.26513 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:18.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.149+0000 7fb1cc469740 1 -- 172.21.15.62:0/1642603877 shutdown_connections 2024-09-15T23:22:18.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.149+0000 7fb1cc469740 1 -- 172.21.15.62:0/1642603877 wait complete. 2024-09-15T23:22:18.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.149+0000 7fb1cc469740 20 client.26513 trim_cache size 0 max 16384 2024-09-15T23:22:18.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1cc469740 1 Processor -- start 2024-09-15T23:22:18.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1cc469740 1 -- start start 2024-09-15T23:22:18.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57174e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5713e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa57143b0 0x560fa57147f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1cc469740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c4122540 con 0x560fa57143b0 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1cc469740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c411fe10 con 0x560fa570eb90 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1cc469740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c411dce0 con 0x560fa5719d70 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1c2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5713e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa57143b0 0x560fa57147f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1c2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5713e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48896/0 (socket says 172.21.15.62:48896) 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa57143b0 0x560fa57147f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48026/0 (socket says 172.21.15.62:48026) 2024-09-15T23:22:18.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.182+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/3119391301 learned_addr learned my addr 172.21.15.62:0/3119391301 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.183+0000 7fb1c37fe640 1 --2- 172.21.15.62:0/3119391301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57174e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.183+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/3119391301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 msgr2=0x560fa57174e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.183+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/3119391301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57174e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.183+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/3119391301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa57143b0 msgr2=0x560fa57147f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.183+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/3119391301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa57143b0 0x560fa57147f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.183+0000 7fb1c2ffd640 1 -- 172.21.15.62:0/3119391301 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fa570af50 con 0x560fa5719d70 2024-09-15T23:22:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.183+0000 7fb1c3fff640 1 --2- 172.21.15.62:0/3119391301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa57143b0 0x560fa57147f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:18.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.183+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/3119391301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5713e70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0c6180 tx=0x7fb1bc02a960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7712e88b98785d35 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:18.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.183+0000 7fb1a97fa640 1 -- 172.21.15.62:0/3119391301 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0c61f0 con 0x560fa5719d70 2024-09-15T23:22:18.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.184+0000 7fb1a97fa640 1 -- 172.21.15.62:0/3119391301 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1bc0c2860 con 0x560fa5719d70 2024-09-15T23:22:18.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.184+0000 7fb1a97fa640 1 -- 172.21.15.62:0/3119391301 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0f7d40 con 0x560fa5719d70 2024-09-15T23:22:18.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.185+0000 7fb1cc469740 1 -- 172.21.15.62:0/3119391301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 msgr2=0x560fa5713e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:18.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.185+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3119391301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5713e70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0c6180 tx=0x7fb1bc02a960 comp rx=0 tx=0).stop 2024-09-15T23:22:18.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.185+0000 7fb1cc469740 1 -- 172.21.15.62:0/3119391301 shutdown_connections 2024-09-15T23:22:18.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.185+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3119391301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa57143b0 0x560fa57147f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.185+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3119391301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa5719d70 0x560fa5713e70 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0c6180 tx=0x7fb1bc02a960 comp rx=0 tx=0).stop 2024-09-15T23:22:18.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.185+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3119391301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57174e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.185+0000 7fb1cc469740 1 -- 172.21.15.62:0/3119391301 >> 172.21.15.62:0/3119391301 conn(0x560fa570b120 msgr2=0x560fa5718b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:18.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.186+0000 7fb1cc469740 1 -- 172.21.15.62:0/3119391301 shutdown_connections 2024-09-15T23:22:18.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.186+0000 7fb1cc469740 1 -- 172.21.15.62:0/3119391301 wait complete. 2024-09-15T23:22:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.186+0000 7fb1cc469740 1 -- start start 2024-09-15T23:22:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.187+0000 7fb1cc469740 1 Processor -- start 2024-09-15T23:22:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.187+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57263a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.187+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa57143b0 0x560fa57268e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.187+0000 7fb1cc469740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5726e20 0x560fa57b5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.187+0000 7fb1cc469740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c411e330 con 0x560fa5726e20 2024-09-15T23:22:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.187+0000 7fb1cc469740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c411f740 con 0x560fa570eb90 2024-09-15T23:22:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.187+0000 7fb1cc469740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c411e9a0 con 0x560fa57143b0 2024-09-15T23:22:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.188+0000 7fb1c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57263a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.188+0000 7fb1c37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57263a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41122/0 (socket says 172.21.15.62:41122) 2024-09-15T23:22:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.188+0000 7fb1c37fe640 1 -- 172.21.15.62:0/3673075448 learned_addr learned my addr 172.21.15.62:0/3673075448 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.188+0000 7fb1c3fff640 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5726e20 0x560fa57b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.188+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa57143b0 0x560fa57268e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.188+0000 7fb1cc469740 10 client.0 _ll_register_callbacks cb 0x7fb1bc0c2490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.189+0000 7fb1c37fe640 1 -- 172.21.15.62:0/3673075448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa57143b0 msgr2=0x560fa57268e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.189+0000 7fb1c37fe640 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa57143b0 0x560fa57268e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.189+0000 7fb1c37fe640 1 -- 172.21.15.62:0/3673075448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5726e20 msgr2=0x560fa57b5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.189+0000 7fb1c37fe640 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5726e20 0x560fa57b5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.189+0000 7fb1c37fe640 1 -- 172.21.15.62:0/3673075448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fa57b5f40 con 0x560fa570eb90 2024-09-15T23:22:18.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.190+0000 7fb1c37fe640 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57263a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x560fa57223b0 tx=0x7fb1b4004700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78a22d345c04b2a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.191+0000 7fb1a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.191+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/3673075448 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4008850 con 0x560fa570eb90 2024-09-15T23:22:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.191+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/3673075448 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb1b4022470 con 0x560fa570eb90 2024-09-15T23:22:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.191+0000 7fb1cc469740 1 -- 172.21.15.62:0/3673075448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1bc0c2860 con 0x560fa570eb90 2024-09-15T23:22:18.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.191+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/3673075448 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1b40089f0 con 0x560fa570eb90 2024-09-15T23:22:18.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.191+0000 7fb1cc469740 10 client.26519 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:18.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.191+0000 7fb1cc469740 1 -- 172.21.15.62:0/3673075448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560fa57b7260 con 0x560fa570eb90 2024-09-15T23:22:18.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.192+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/3673075448 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1bc0c2860 con 0x560fa570eb90 2024-09-15T23:22:18.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.192+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/3673075448 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7fb1bc0ca170 con 0x560fa570eb90 2024-09-15T23:22:18.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.192+0000 7fb1cc469740 10 client.26519 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:18.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.192+0000 7fb1cc469740 10 client.26519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:18.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.192+0000 7fb1cc469740 20 client.26519 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:18.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.192+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560fa57b7a40 0x560fa57b9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:18.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.192+0000 7fb1cc469740 4 client.26519 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:22:18.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.192+0000 7fb1cc469740 1 -- 172.21.15.62:0/3673075448 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560fa56817f0 con 0x560fa57b7a40 2024-09-15T23:22:18.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.193+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560fa57b7a40 0x560fa57b9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:18.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.194+0000 7fb1c2ffd640 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560fa57b7a40 0x560fa57b9e60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3669d17f60cb8878 server_cookie=2b35a1456f07de36 in_seq=0 out_seq=0 2024-09-15T23:22:18.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.194+0000 7fb1a2ffd640 10 client.26519 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1a2ffd640 1 -- 172.21.15.62:0/3673075448 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560fa56817f0 con 0x560fa57b7a40 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1a2ffd640 10 client.26519 handle_command_reply: tid=0 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 client.26519 shutdown 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 -- 172.21.15.62:0/3673075448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 msgr2=0x560fa57263a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57263a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x560fa57223b0 tx=0x7fb1b4004700 comp rx=0 tx=0).stop 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 -- 172.21.15.62:0/3673075448 shutdown_connections 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560fa57b7a40 0x560fa57b9e60 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560fa5726e20 0x560fa57b5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560fa57143b0 0x560fa57268e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 --2- 172.21.15.62:0/3673075448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560fa570eb90 0x560fa57263a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 -- 172.21.15.62:0/3673075448 >> 172.21.15.62:0/3673075448 conn(0x560fa570b120 msgr2=0x560fa57173c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1a2ffd640 0 client.26519 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:18.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 -- 172.21.15.62:0/3673075448 shutdown_connections 2024-09-15T23:22:18.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 1 -- 172.21.15.62:0/3673075448 wait complete. 2024-09-15T23:22:18.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.195+0000 7fb1cc469740 20 client.26519 trim_cache size 0 max 16384 2024-09-15T23:22:18.198 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:22:18.198 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:22:18.198 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "6a6d391f-b2f0-42f1-aae9-a5d6735b3d04", 2024-09-15T23:22:18.198 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:22:18.198 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.196+0000 7fb1ca090640 1 -- 172.21.15.62:0/1526944799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1ac0a33d0 msgr2=0x7fb1ac0a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.196+0000 7fb1ca090640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1ac0a33d0 0x7fb1ac0a5890 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4008fd0 tx=0x7fb1b4006860 comp rx=0 tx=0).stop 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.196+0000 7fb1ca090640 1 -- 172.21.15.62:0/1526944799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c4102ef0 msgr2=0x7fb1c406cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.196+0000 7fb1ca090640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c4102ef0 0x7fb1c406cbe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc000f80 tx=0x7fb1bc02f5e0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.197+0000 7fb1ca090640 1 -- 172.21.15.62:0/1526944799 shutdown_connections 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.197+0000 7fb1ca090640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1ac0a33d0 0x7fb1ac0a5890 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.197+0000 7fb1ca090640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1c406d120 0x7fb1c406d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.197+0000 7fb1ca090640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c4102ef0 0x7fb1c406cbe0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.197+0000 7fb1ca090640 1 --2- 172.21.15.62:0/1526944799 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1c4100e20 0x7fb1c4073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:18.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.197+0000 7fb1ca090640 1 -- 172.21.15.62:0/1526944799 >> 172.21.15.62:0/1526944799 conn(0x7fb1c40fc7d0 msgr2=0x7fb1c40ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:18.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.197+0000 7fb1ca090640 1 -- 172.21.15.62:0/1526944799 shutdown_connections 2024-09-15T23:22:18.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:18.197+0000 7fb1ca090640 1 -- 172.21.15.62:0/1526944799 wait complete. 2024-09-15T23:22:18.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:18 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:22:18.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:18 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:18.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:18 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:22:18.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:18 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:22:18.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:18 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:18.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:18 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:22:18.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:18 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:18.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:18 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:18 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:22:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:18 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:18 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:22:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:18 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:22:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:18 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:18 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:22:19.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:18 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:18 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:18 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:22:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:18 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:18 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:22:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:18 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:22:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:18 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:18 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:22:19.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:18 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:18 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.059 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:22:19.488 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:19.573 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 360.0 s, 1046.6 tps, lat 94.454 ms stddev 113.299 2024-09-15T23:22:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:19 smithi067 ceph-mon[35665]: pgmap v551: 129 pgs: 129 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 14 MiB/s wr, 640 op/s 2024-09-15T23:22:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:19 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:22:19.925 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:19 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.940 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:19 smithi148 ceph-mon[35610]: pgmap v551: 129 pgs: 129 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 14 MiB/s wr, 640 op/s 2024-09-15T23:22:19.940 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:19 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:22:19.940 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:19 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:19 smithi062 ceph-mon[30949]: pgmap v551: 129 pgs: 129 active+clean; 1.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 14 MiB/s wr, 640 op/s 2024-09-15T23:22:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:19 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:22:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:19 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:22:21.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.306+0000 7f5ddf49f640 1 -- 172.21.15.62:0/487189326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd8102ef0 msgr2=0x7f5dd810da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:21.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.306+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/487189326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd8102ef0 0x7f5dd810da40 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009ca0 tx=0x7f5dd002a950 comp rx=0 tx=0).stop 2024-09-15T23:22:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.306+0000 7f5ddf49f640 1 -- 172.21.15.62:0/487189326 shutdown_connections 2024-09-15T23:22:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.306+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/487189326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd8102ef0 0x7f5dd810da40 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.306+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/487189326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dd8102570 0x7f5dd81029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.306+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/487189326 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5dd8100e20 0x7f5dd8101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.306+0000 7f5ddf49f640 1 -- 172.21.15.62:0/487189326 >> 172.21.15.62:0/487189326 conn(0x7f5dd80fc7d0 msgr2=0x7f5dd80febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.306+0000 7f5ddf49f640 1 -- 172.21.15.62:0/487189326 shutdown_connections 2024-09-15T23:22:21.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.306+0000 7f5ddf49f640 1 -- 172.21.15.62:0/487189326 wait complete. 2024-09-15T23:22:21.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.307+0000 7f5ddf49f640 1 Processor -- start 2024-09-15T23:22:21.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.308+0000 7f5ddf49f640 1 -- start start 2024-09-15T23:22:21.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.308+0000 7f5ddf49f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd8100e20 0x7f5dd80fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.308+0000 7f5ddf49f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dd8102570 0x7f5dd80fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.308+0000 7f5ddf49f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5dd8102ef0 0x7f5dd80fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.308+0000 7f5ddf49f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8110f80 con 0x7f5dd8100e20 2024-09-15T23:22:21.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.308+0000 7f5ddf49f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8110e00 con 0x7f5dd8102570 2024-09-15T23:22:21.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.308+0000 7f5ddf49f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8111100 con 0x7f5dd8102ef0 2024-09-15T23:22:21.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddca3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dd8102570 0x7f5dd80fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddd23b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd8100e20 0x7f5dd80fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddca3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dd8102570 0x7f5dd80fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41142/0 (socket says 172.21.15.62:41142) 2024-09-15T23:22:21.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddca3a640 1 -- 172.21.15.62:0/2108924160 learned_addr learned my addr 172.21.15.62:0/2108924160 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddda3c640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5dd8102ef0 0x7f5dd80fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddca3a640 1 -- 172.21.15.62:0/2108924160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5dd8102ef0 msgr2=0x7f5dd80fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5dd8102ef0 0x7f5dd80fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddca3a640 1 -- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd8100e20 msgr2=0x7f5dd80fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd8100e20 0x7f5dd80fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddca3a640 1 -- 172.21.15.62:0/2108924160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd81a69c0 con 0x7f5dd8102570 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd8100e20 0x7f5dd80fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dd8102570 0x7f5dd80fd1c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800d950 tx=0x7f5dc800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5dce7fc640 1 -- 172.21.15.62:0/2108924160 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8017030 con 0x7f5dd8102570 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5ddf49f640 1 -- 172.21.15.62:0/2108924160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd81a6cb0 con 0x7f5dd8102570 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.309+0000 7f5dce7fc640 1 -- 172.21.15.62:0/2108924160 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dc8004020 con 0x7f5dd8102570 2024-09-15T23:22:21.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.310+0000 7f5dce7fc640 1 -- 172.21.15.62:0/2108924160 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc80047a0 con 0x7f5dd8102570 2024-09-15T23:22:21.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.310+0000 7f5ddf49f640 1 -- 172.21.15.62:0/2108924160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd81a71e0 con 0x7f5dd8102570 2024-09-15T23:22:21.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.312+0000 7f5dce7fc640 1 -- 172.21.15.62:0/2108924160 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f5dc80041d0 con 0x7f5dd8102570 2024-09-15T23:22:21.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.312+0000 7f5de1878740 1 Processor -- start 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.312+0000 7f5de1878740 1 -- start start 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.312+0000 7f5de1878740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d004a0 0x558644d00880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.312+0000 7f5de1878740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d0b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5de1878740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d0b680 0x558644d0da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5de1878740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558644c662e0 con 0x558644d0b680 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5de1878740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558644c0dce0 con 0x558644d004a0 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5de1878740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558644c622a0 con 0x558644d00dc0 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5ddda3c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d004a0 0x558644d00880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5ddda3c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d004a0 0x558644d00880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41154/0 (socket says 172.21.15.62:41154) 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5ddd23b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d0b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5ddd23b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d0b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48922/0 (socket says 172.21.15.62:48922) 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5ddd23b640 1 -- 172.21.15.62:0/3206665589 learned_addr learned my addr 172.21.15.62:0/3206665589 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:21.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.313+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/3206665589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d0b680 0x558644d0da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.314+0000 7f5ddd23b640 1 -- 172.21.15.62:0/3206665589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d004a0 msgr2=0x558644d00880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.314+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3206665589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d004a0 0x558644d00880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.314+0000 7f5ddd23b640 1 -- 172.21.15.62:0/3206665589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d0b680 msgr2=0x558644d0da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.314+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3206665589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d0b680 0x558644d0da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.314+0000 7f5ddd23b640 1 -- 172.21.15.62:0/3206665589 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558644bf2440 con 0x558644d00dc0 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.314+0000 7f5dce7fc640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5dc40a3240 0x7f5dc40a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.314+0000 7f5dce7fc640 1 -- 172.21.15.62:0/2108924160 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5dc80cf5d0 con 0x7f5dd8102570 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.315+0000 7f5ddda3c640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5dc40a3240 0x7f5dc40a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.315+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3206665589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d0b140 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40115c0 tx=0x7f5dd4011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5d9ab6793a4ddda server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.315+0000 7f5dc37fe640 1 -- 172.21.15.62:0/3206665589 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400f640 con 0x558644d00dc0 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.315+0000 7f5dc37fe640 1 -- 172.21.15.62:0/3206665589 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd400fcc0 con 0x558644d00dc0 2024-09-15T23:22:21.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.316+0000 7f5dc37fe640 1 -- 172.21.15.62:0/3206665589 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4019e10 con 0x558644d00dc0 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 -- 172.21.15.62:0/3206665589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 msgr2=0x558644d0b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5ddda3c640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5dc40a3240 0x7f5dc40a5700 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0005e30 tx=0x7f5dd0005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 --2- 172.21.15.62:0/3206665589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d0b140 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40115c0 tx=0x7f5dd4011a80 comp rx=0 tx=0).stop 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 -- 172.21.15.62:0/3206665589 shutdown_connections 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 --2- 172.21.15.62:0/3206665589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d0b680 0x558644d0da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 --2- 172.21.15.62:0/3206665589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d0b140 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 --2- 172.21.15.62:0/3206665589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d004a0 0x558644d00880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 -- 172.21.15.62:0/3206665589 >> 172.21.15.62:0/3206665589 conn(0x558644cfca30 msgr2=0x558644cfce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 -- 172.21.15.62:0/3206665589 shutdown_connections 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 -- 172.21.15.62:0/3206665589 wait complete. 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 -- start start 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.317+0000 7f5de1878740 1 Processor -- start 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5de1878740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d0af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5de1878740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d08240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5de1878740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d0b680 0x558644d08780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5de1878740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558644d0e250 con 0x558644d004a0 2024-09-15T23:22:21.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5de1878740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558644c21050 con 0x558644d00dc0 2024-09-15T23:22:21.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5de1878740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558644d0e3d0 con 0x558644d0b680 2024-09-15T23:22:21.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddca3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d08240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5de1878740 10 client.0 _ll_register_callbacks cb 0x558644c61160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:21.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddca3a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d08240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41170/0 (socket says 172.21.15.62:41170) 2024-09-15T23:22:21.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddca3a640 1 -- 172.21.15.62:0/555841236 learned_addr learned my addr 172.21.15.62:0/555841236 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d0af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddda3c640 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d0b680 0x558644d08780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddca3a640 1 -- 172.21.15.62:0/555841236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d0b680 msgr2=0x558644d08780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d0b680 0x558644d08780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddca3a640 1 -- 172.21.15.62:0/555841236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 msgr2=0x558644d0af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d0af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddca3a640 1 -- 172.21.15.62:0/555841236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586449d73c0 con 0x558644d00dc0 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d08240 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5dc809b510 tx=0x7f5dc8005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b8776c0c6ed74ae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5dc0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:21.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/555841236 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc809b350 con 0x558644d00dc0 2024-09-15T23:22:21.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5de1878740 1 -- 172.21.15.62:0/555841236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dd400fcc0 con 0x558644d00dc0 2024-09-15T23:22:21.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.318+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/555841236 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5dc8097e80 con 0x558644d00dc0 2024-09-15T23:22:21.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.319+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/555841236 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5dc809b170 con 0x558644d00dc0 2024-09-15T23:22:21.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.319+0000 7f5de1878740 10 client.26531 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:21.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.319+0000 7f5de1878740 1 -- 172.21.15.62:0/555841236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558644d22490 con 0x558644d00dc0 2024-09-15T23:22:21.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.319+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/555841236 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd400fcc0 con 0x558644d00dc0 2024-09-15T23:22:21.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.319+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/555841236 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f5dc8084b60 con 0x558644d00dc0 2024-09-15T23:22:21.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.319+0000 7f5de1878740 10 client.26531 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.319+0000 7f5de1878740 10 client.26531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.319+0000 7f5de1878740 20 client.26531 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.319+0000 7f5de1878740 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558644d23db0 0x558644d26250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.320+0000 7f5de1878740 4 client.26531 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.320+0000 7f5de1878740 1 -- 172.21.15.62:0/555841236 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558644c73850 con 0x558644d23db0 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.320+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558644d23db0 0x558644d26250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.320+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558644d23db0 0x558644d26250 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbe5879e44c21d3e server_cookie=42dcccbb50982a3c in_seq=0 out_seq=0 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.320+0000 7f5dc0ff9640 10 client.26531 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/555841236 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558644c73850 con 0x558644d23db0 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5dc0ff9640 10 client.26531 handle_command_reply: tid=0 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5de1878740 1 client.26531 shutdown 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5de1878740 1 -- 172.21.15.62:0/555841236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 msgr2=0x558644d08240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5de1878740 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d08240 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5dc809b510 tx=0x7f5dc8005dc0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5de1878740 1 -- 172.21.15.62:0/555841236 shutdown_connections 2024-09-15T23:22:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5de1878740 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558644d23db0 0x558644d26250 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5de1878740 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d0b680 0x558644d08780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5de1878740 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d08240 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5de1878740 1 --2- 172.21.15.62:0/555841236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d0af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5dc0ff9640 0 client.26531 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.321+0000 7f5de1878740 1 -- 172.21.15.62:0/555841236 >> 172.21.15.62:0/555841236 conn(0x558644cfca30 msgr2=0x558644d02710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.322+0000 7f5de1878740 1 -- 172.21.15.62:0/555841236 shutdown_connections 2024-09-15T23:22:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.322+0000 7f5de1878740 1 -- 172.21.15.62:0/555841236 wait complete. 2024-09-15T23:22:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.322+0000 7f5de1878740 20 client.26531 trim_cache size 0 max 16384 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.343+0000 7f5de1878740 1 Processor -- start 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.343+0000 7f5de1878740 1 -- start start 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5de1878740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644cfca30 0x558644d0a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5de1878740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d2bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5de1878740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d2c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5de1878740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd81af2e0 con 0x558644d004a0 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5de1878740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd81acbb0 con 0x558644cfca30 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5de1878740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd81aaa00 con 0x558644d00dc0 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddca3a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d2bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddca3a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d2bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48082/0 (socket says 172.21.15.62:48082) 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddca3a640 1 -- 172.21.15.62:0/3566996712 learned_addr learned my addr 172.21.15.62:0/3566996712 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3566996712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644cfca30 0x558644d0a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddda3c640 1 --2- 172.21.15.62:0/3566996712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d2c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddd23b640 1 -- 172.21.15.62:0/3566996712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 msgr2=0x558644d2c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3566996712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d2c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddd23b640 1 -- 172.21.15.62:0/3566996712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 msgr2=0x558644d2bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3566996712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d2bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.344+0000 7f5ddd23b640 1 -- 172.21.15.62:0/3566996712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558644cfc4d0 con 0x558644cfca30 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.345+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3566996712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644cfca30 0x558644d0a1b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009ce0 tx=0x7f5dd40199b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dc59d89babd8815 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.345+0000 7f5dc2ffd640 1 -- 172.21.15.62:0/3566996712 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4031070 con 0x558644cfca30 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.345+0000 7f5dc2ffd640 1 -- 172.21.15.62:0/3566996712 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dd4019530 con 0x558644cfca30 2024-09-15T23:22:21.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.345+0000 7f5dc2ffd640 1 -- 172.21.15.62:0/3566996712 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400f640 con 0x558644cfca30 2024-09-15T23:22:21.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.345+0000 7f5de1878740 1 -- 172.21.15.62:0/3566996712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644cfca30 msgr2=0x558644d0a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.345+0000 7f5de1878740 1 --2- 172.21.15.62:0/3566996712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644cfca30 0x558644d0a1b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009ce0 tx=0x7f5dd40199b0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 -- 172.21.15.62:0/3566996712 shutdown_connections 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 --2- 172.21.15.62:0/3566996712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644d00dc0 0x558644d2c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 --2- 172.21.15.62:0/3566996712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d2bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 --2- 172.21.15.62:0/3566996712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644cfca30 0x558644d0a1b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 -- 172.21.15.62:0/3566996712 >> 172.21.15.62:0/3566996712 conn(0x558644d088e0 msgr2=0x558644d09aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 -- 172.21.15.62:0/3566996712 shutdown_connections 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 -- 172.21.15.62:0/3566996712 wait complete. 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 -- start start 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 Processor -- start 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644cfca30 0x558644d1c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d1c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.346+0000 7f5de1878740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d19820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5de1878740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd81ab070 con 0x558644d004a0 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5de1878740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd81ac4e0 con 0x558644d00dc0 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5de1878740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd81ab740 con 0x558644cfca30 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddca3a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d1c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddca3a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d1c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48094/0 (socket says 172.21.15.62:48094) 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddca3a640 1 -- 172.21.15.62:0/3377064456 learned_addr learned my addr 172.21.15.62:0/3377064456 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644cfca30 0x558644d1c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddda3c640 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d19820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5de1878740 10 client.0 _ll_register_callbacks cb 0x7f5dc80051a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddca3a640 1 -- 172.21.15.62:0/3377064456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644cfca30 msgr2=0x558644d1c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644cfca30 0x558644d1c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddca3a640 1 -- 172.21.15.62:0/3377064456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 msgr2=0x558644d19820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:21.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d19820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddca3a640 1 -- 172.21.15.62:0/3377064456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558644d1a050 con 0x558644d004a0 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644cfca30 0x558644d1c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.347+0000 7f5ddca3a640 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d1c7c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80a3f50 tx=0x7f5dc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae9a3f8e671f0afb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.348+0000 7f5dc0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.348+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/3377064456 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc809b350 con 0x558644d004a0 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.348+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/3377064456 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5dc809ae60 con 0x558644d004a0 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.348+0000 7f5de1878740 1 -- 172.21.15.62:0/3377064456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dd4019530 con 0x558644d004a0 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.349+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/3377064456 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5dc80b9050 con 0x558644d004a0 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.349+0000 7f5de1878740 10 client.16968 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.349+0000 7f5de1878740 1 -- 172.21.15.62:0/3377064456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558644d8e740 con 0x558644d004a0 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.350+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/3377064456 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5dc8084d70 con 0x558644d004a0 2024-09-15T23:22:21.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.350+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/3377064456 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f5dc809cce0 con 0x558644d004a0 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.350+0000 7f5de1878740 10 client.16968 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.350+0000 7f5de1878740 10 client.16968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.350+0000 7f5de1878740 20 client.16968 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.350+0000 7f5de1878740 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558644d0b680 0x558644d90f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.350+0000 7f5de1878740 4 client.16968 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.350+0000 7f5de1878740 1 -- 172.21.15.62:0/3377064456 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f5dd4019530 con 0x558644d0b680 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.350+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558644d0b680 0x558644d90f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5ddd23b640 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558644d0b680 0x558644d90f30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a84ed7cb7f3a3d28 server_cookie=1247581fcab82da7 in_seq=0 out_seq=0 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5dc0ff9640 10 client.16968 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5dc0ff9640 1 -- 172.21.15.62:0/3377064456 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f5dd4019530 con 0x558644d0b680 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5dc0ff9640 10 client.16968 handle_command_reply: tid=0 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 client.16968 shutdown 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 -- 172.21.15.62:0/3377064456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 msgr2=0x558644d1c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:21.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d1c7c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5dc80a3f50 tx=0x7f5dc80046c0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 -- 172.21.15.62:0/3377064456 shutdown_connections 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558644d0b680 0x558644d90f30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558644d00dc0 0x558644d19820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558644d004a0 0x558644d1c7c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 --2- 172.21.15.62:0/3377064456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558644cfca30 0x558644d1c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5dc0ff9640 0 client.16968 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 -- 172.21.15.62:0/3377064456 >> 172.21.15.62:0/3377064456 conn(0x558644d088e0 msgr2=0x558644d066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 -- 172.21.15.62:0/3377064456 shutdown_connections 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 1 -- 172.21.15.62:0/3377064456 wait complete. 2024-09-15T23:22:21.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.351+0000 7f5de1878740 20 client.16968 trim_cache size 0 max 16384 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout: "6a6d391f-b2f0-42f1-aae9-a5d6735b3d04": { 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "6a6d391f-b2f0-42f1-aae9-a5d6735b3d04", 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:22:21.359 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:22:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 -- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5dc40a3240 msgr2=0x7f5dc40a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5dc40a3240 0x7f5dc40a5700 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0005e30 tx=0x7f5dd0005c50 comp rx=0 tx=0).stop 2024-09-15T23:22:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 -- 172.21.15.62:0/2108924160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dd8102570 msgr2=0x7f5dd80fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dd8102570 0x7f5dd80fd1c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800d950 tx=0x7f5dc800de10 comp rx=0 tx=0).stop 2024-09-15T23:22:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 -- 172.21.15.62:0/2108924160 shutdown_connections 2024-09-15T23:22:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5dc40a3240 0x7f5dc40a5700 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5dd8102ef0 0x7f5dd80fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dd8102570 0x7f5dd80fd1c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 --2- 172.21.15.62:0/2108924160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dd8100e20 0x7f5dd80fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:21.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 -- 172.21.15.62:0/2108924160 >> 172.21.15.62:0/2108924160 conn(0x7f5dd80fc7d0 msgr2=0x7f5dd806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:21.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 -- 172.21.15.62:0/2108924160 shutdown_connections 2024-09-15T23:22:21.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:21.352+0000 7f5ddf49f640 1 -- 172.21.15.62:0/2108924160 wait complete. 2024-09-15T23:22:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:21 smithi062 ceph-mon[30949]: pgmap v552: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 15 MiB/s wr, 750 op/s 2024-09-15T23:22:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:21 smithi067 ceph-mon[35665]: pgmap v552: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 15 MiB/s wr, 750 op/s 2024-09-15T23:22:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:21 smithi148 ceph-mon[35610]: pgmap v552: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 15 MiB/s wr, 750 op/s 2024-09-15T23:22:22.236 INFO:tasks.cephfs.filesystem:scrub status for tag:6a6d391f-b2f0-42f1-aae9-a5d6735b3d04 - {'path': '/', 'tag': '6a6d391f-b2f0-42f1-aae9-a5d6735b3d04', 'options': 'recursive,force'} 2024-09-15T23:22:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:23 smithi062 ceph-mon[30949]: pgmap v553: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 15 MiB/s wr, 774 op/s 2024-09-15T23:22:24.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:23 smithi067 ceph-mon[35665]: pgmap v553: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 15 MiB/s wr, 774 op/s 2024-09-15T23:22:24.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:23 smithi148 ceph-mon[35610]: pgmap v553: 129 pgs: 129 active+clean; 1.4 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 15 MiB/s wr, 774 op/s 2024-09-15T23:22:24.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 365.0 s, 1097.2 tps, lat 90.329 ms stddev 110.677 2024-09-15T23:22:25.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:25 smithi062 ceph-mon[30949]: pgmap v554: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 13 MiB/s wr, 707 op/s 2024-09-15T23:22:26.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:25 smithi067 ceph-mon[35665]: pgmap v554: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 13 MiB/s wr, 707 op/s 2024-09-15T23:22:26.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:25 smithi148 ceph-mon[35610]: pgmap v554: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 13 MiB/s wr, 707 op/s 2024-09-15T23:22:27.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:27 smithi062 ceph-mon[30949]: pgmap v555: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 12 MiB/s wr, 649 op/s 2024-09-15T23:22:28.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:27 smithi067 ceph-mon[35665]: pgmap v555: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 12 MiB/s wr, 649 op/s 2024-09-15T23:22:28.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:27 smithi148 ceph-mon[35610]: pgmap v555: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 12 MiB/s wr, 649 op/s 2024-09-15T23:22:29.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.196+0000 7f50e645e640 1 -- 172.21.15.62:0/1966875625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 msgr2=0x7f50e01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.196+0000 7f50e645e640 1 --2- 172.21.15.62:0/1966875625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 0x7f50e01031d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009900 tx=0x7f50d402a410 comp rx=0 tx=0).stop 2024-09-15T23:22:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.196+0000 7f50e645e640 1 -- 172.21.15.62:0/1966875625 shutdown_connections 2024-09-15T23:22:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.196+0000 7f50e645e640 1 --2- 172.21.15.62:0/1966875625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e0103710 0x7f50e0109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.196+0000 7f50e645e640 1 --2- 172.21.15.62:0/1966875625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 0x7f50e01031d0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.196+0000 7f50e645e640 1 --2- 172.21.15.62:0/1966875625 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e0101b60 0x7f50e0101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.196+0000 7f50e645e640 1 -- 172.21.15.62:0/1966875625 >> 172.21.15.62:0/1966875625 conn(0x7f50e00fd2e0 msgr2=0x7f50e00ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.196+0000 7f50e645e640 1 -- 172.21.15.62:0/1966875625 shutdown_connections 2024-09-15T23:22:29.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.196+0000 7f50e645e640 1 -- 172.21.15.62:0/1966875625 wait complete. 2024-09-15T23:22:29.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.197+0000 7f50e645e640 1 Processor -- start 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.197+0000 7f50e645e640 1 -- start start 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e645e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e0101b60 0x7f50e0197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e645e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 0x7f50e0198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e645e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e0103710 0x7f50e019f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e645e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f50e010d7e0 con 0x7f50e0102d50 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e645e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50e010d660 con 0x7f50e0103710 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e645e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f50e010d960 con 0x7f50e0101b60 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50df7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 0x7f50e0198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50dffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e0101b60 0x7f50e0197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50df7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 0x7f50e0198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56636/0 (socket says 172.21.15.62:56636) 2024-09-15T23:22:29.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50df7fe640 1 -- 172.21.15.62:0/2268698581 learned_addr learned my addr 172.21.15.62:0/2268698581 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50dffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e0101b60 0x7f50e0197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:59672/0 (socket says 172.21.15.62:59672) 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e49fb640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e0103710 0x7f50e019f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e49fb640 1 -- 172.21.15.62:0/2268698581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e0101b60 msgr2=0x7f50e0197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e49fb640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e0101b60 0x7f50e0197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e49fb640 1 -- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 msgr2=0x7f50e0198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e49fb640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 0x7f50e0198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50e49fb640 1 -- 172.21.15.62:0/2268698581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e019f9f0 con 0x7f50e0103710 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.198+0000 7f50df7fe640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 0x7f50e0198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.199+0000 7f50e49fb640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e0103710 0x7f50e019f380 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f50d8009fd0 tx=0x7f50d800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.199+0000 7f50dd7fa640 1 -- 172.21.15.62:0/2268698581 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d800ce20 con 0x7f50e0103710 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.199+0000 7f50e645e640 1 -- 172.21.15.62:0/2268698581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e019fce0 con 0x7f50e0103710 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.199+0000 7f50dd7fa640 1 -- 172.21.15.62:0/2268698581 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50d8005020 con 0x7f50e0103710 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.199+0000 7f50dd7fa640 1 -- 172.21.15.62:0/2268698581 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d8010720 con 0x7f50e0103710 2024-09-15T23:22:29.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.199+0000 7f50e645e640 1 -- 172.21.15.62:0/2268698581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e01a01d0 con 0x7f50e0103710 2024-09-15T23:22:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.202+0000 7f50e91fa740 1 Processor -- start 2024-09-15T23:22:29.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.203+0000 7f50e91fa740 1 -- start start 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.203+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3b9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.203+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3bee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.203+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.203+0000 7f50e91fa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ea4b31f2f0 con 0x55ea4b3b99a0 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.203+0000 7f50e91fa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ea4b2c7270 con 0x55ea4b3bf3a0 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.203+0000 7f50e91fa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ea4b31b2b0 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.204+0000 7f50dd7fa640 1 -- 172.21.15.62:0/2268698581 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f50d8005540 con 0x7f50e0103710 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.204+0000 7f50dffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3bee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.204+0000 7f50df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.204+0000 7f50dffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3bee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:59674/0 (socket says 172.21.15.62:59674) 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.204+0000 7f50dffff640 1 -- 172.21.15.62:0/1461185016 learned_addr learned my addr 172.21.15.62:0/1461185016 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.204+0000 7f50df7fe640 1 --2- 172.21.15.62:0/1461185016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3b9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.205+0000 7f50dffff640 1 -- 172.21.15.62:0/1461185016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3bf3a0 msgr2=0x55ea4b3c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:29.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.205+0000 7f50dffff640 1 --2- 172.21.15.62:0/1461185016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3c1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.205+0000 7f50dffff640 1 -- 172.21.15.62:0/1461185016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 msgr2=0x55ea4b3b9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.205+0000 7f50dffff640 1 --2- 172.21.15.62:0/1461185016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3b9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.205+0000 7f50dffff640 1 -- 172.21.15.62:0/1461185016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea4b2c6340 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.205+0000 7f50dffff640 1 --2- 172.21.15.62:0/1461185016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3bee60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009ca0 tx=0x7f50d0033820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d612350db26cec1f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.206+0000 7f50c67fc640 1 -- 172.21.15.62:0/1461185016 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0004020 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.206+0000 7f50c67fc640 1 -- 172.21.15.62:0/1461185016 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50d00041c0 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.206+0000 7f50c67fc640 1 -- 172.21.15.62:0/1461185016 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d003b780 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.206+0000 7f50dd7fa640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50c80a33f0 0x7f50c80a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50dd7fa640 1 -- 172.21.15.62:0/2268698581 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f50d80d0350 con 0x7f50e0103710 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50df7fe640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50c80a33f0 0x7f50c80a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50e91fa740 1 -- 172.21.15.62:0/1461185016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3ba2e0 msgr2=0x55ea4b3bee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1461185016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3bee60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009ca0 tx=0x7f50d0033820 comp rx=0 tx=0).stop 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50e91fa740 1 -- 172.21.15.62:0/1461185016 shutdown_connections 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1461185016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3c1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1461185016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3bee60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1461185016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3b9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50e91fa740 1 -- 172.21.15.62:0/1461185016 >> 172.21.15.62:0/1461185016 conn(0x55ea4b3b6f90 msgr2=0x55ea4b3b7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50e91fa740 1 -- 172.21.15.62:0/1461185016 shutdown_connections 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.207+0000 7f50e91fa740 1 -- 172.21.15.62:0/1461185016 wait complete. 2024-09-15T23:22:29.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e91fa740 1 -- start start 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e91fa740 1 Processor -- start 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50df7fe640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50c80a33f0 0x7f50c80a58b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f50d40096c0 tx=0x7f50d40055d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3c6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3cd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e91fa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ea4b3c23c0 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e91fa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ea4b2a8290 con 0x55ea4b3b99a0 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e91fa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ea4b3c2540 con 0x55ea4b3bf3a0 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e91fa740 10 client.0 _ll_register_callbacks cb 0x55ea4b2d9140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50df7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50df7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3c6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56652/0 (socket says 172.21.15.62:56652) 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50df7fe640 1 -- 172.21.15.62:0/1147655678 learned_addr learned my addr 172.21.15.62:0/1147655678 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.208+0000 7f50e49fb640 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50dffff640 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50df7fe640 1 -- 172.21.15.62:0/1147655678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 msgr2=0x55ea4b3cd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50df7fe640 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3cd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50df7fe640 1 -- 172.21.15.62:0/1147655678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3b99a0 msgr2=0x55ea4b3c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50df7fe640 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50df7fe640 1 -- 172.21.15.62:0/1147655678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea4b032210 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50e49fb640 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50df7fe640 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3c6130 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f50d4005880 tx=0x7f50d402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26f8328045c155f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1147655678 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4035020 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50e91fa740 1 -- 172.21.15.62:0/1147655678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50d00041c0 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.213 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.209+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1147655678 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f50d4005020 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.210+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1147655678 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f50d40046c0 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.210+0000 7f50e91fa740 10 client.16980 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.210+0000 7f50e91fa740 1 -- 172.21.15.62:0/1147655678 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ea4b0fe620 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.210+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1147655678 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50d00041c0 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.210+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1147655678 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f50d403a040 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.210+0000 7f50e91fa740 10 client.16980 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.210+0000 7f50e91fa740 10 client.16980 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.210+0000 7f50e91fa740 20 client.16980 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.210+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ea4b3d1dc0 0x55ea4b3d4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.211+0000 7f50e91fa740 4 client.16980 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.211+0000 7f50e91fa740 1 -- 172.21.15.62:0/1147655678 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea4b32ca20 con 0x55ea4b3d1dc0 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.211+0000 7f50dffff640 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ea4b3d1dc0 0x55ea4b3d4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.211+0000 7f50dffff640 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ea4b3d1dc0 0x55ea4b3d4240 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80bd173079889378 server_cookie=6abac6506d7a3a7e in_seq=0 out_seq=0 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.211+0000 7f50bf7fe640 10 client.16980 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1147655678 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ea4b32ca20 con 0x55ea4b3d1dc0 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50bf7fe640 10 client.16980 handle_command_reply: tid=0 2024-09-15T23:22:29.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 client.16980 shutdown 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 -- 172.21.15.62:0/1147655678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3ba2e0 msgr2=0x55ea4b3c6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3c6130 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f50d4005880 tx=0x7f50d402a410 comp rx=0 tx=0).stop 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 -- 172.21.15.62:0/1147655678 shutdown_connections 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ea4b3d1dc0 0x55ea4b3d4240 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3cd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3c6130 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1147655678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 -- 172.21.15.62:0/1147655678 >> 172.21.15.62:0/1147655678 conn(0x55ea4b3b6f90 msgr2=0x55ea4b3b87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50bf7fe640 0 client.16980 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 -- 172.21.15.62:0/1147655678 shutdown_connections 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 1 -- 172.21.15.62:0/1147655678 wait complete. 2024-09-15T23:22:29.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.212+0000 7f50e91fa740 20 client.16980 trim_cache size 0 max 16384 2024-09-15T23:22:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.239+0000 7f50e91fa740 1 Processor -- start 2024-09-15T23:22:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.239+0000 7f50e91fa740 1 -- start start 2024-09-15T23:22:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.240+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3c4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.240+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3cd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.240+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.240+0000 7f50e91fa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01aac30 con 0x55ea4b3bf3a0 2024-09-15T23:22:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.240+0000 7f50e91fa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01a8500 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.240+0000 7f50e91fa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01a6350 con 0x55ea4b3b99a0 2024-09-15T23:22:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.241+0000 7f50df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.245 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.241+0000 7f50df7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:53934/0 (socket says 172.21.15.62:53934) 2024-09-15T23:22:29.246 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.241+0000 7f50dffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.246 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.241+0000 7f50df7fe640 1 -- 172.21.15.62:0/3266924187 learned_addr learned my addr 172.21.15.62:0/3266924187 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:29.246 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.242+0000 7f50e49fb640 1 --2- 172.21.15.62:0/3266924187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.246 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.244+0000 7f50dffff640 1 -- 172.21.15.62:0/3266924187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 msgr2=0x55ea4b3cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:29.246 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.244+0000 7f50dffff640 1 --2- 172.21.15.62:0/3266924187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.247 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.244+0000 7f50dffff640 1 -- 172.21.15.62:0/3266924187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3bf3a0 msgr2=0x55ea4b3c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:29.247 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.244+0000 7f50dffff640 1 --2- 172.21.15.62:0/3266924187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.247 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.244+0000 7f50dffff640 1 -- 172.21.15.62:0/3266924187 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea4b3c3de0 con 0x55ea4b3b99a0 2024-09-15T23:22:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.247+0000 7f50dffff640 1 --2- 172.21.15.62:0/3266924187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3c4160 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009f50 tx=0x7f50d00053d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6e4bc820a986015 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.248+0000 7f50c5ffb640 1 -- 172.21.15.62:0/3266924187 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0005540 con 0x55ea4b3b99a0 2024-09-15T23:22:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.248+0000 7f50c5ffb640 1 -- 172.21.15.62:0/3266924187 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50d0003ad0 con 0x55ea4b3b99a0 2024-09-15T23:22:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.248+0000 7f50c5ffb640 1 -- 172.21.15.62:0/3266924187 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d00476c0 con 0x55ea4b3b99a0 2024-09-15T23:22:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.249+0000 7f50e91fa740 1 -- 172.21.15.62:0/3266924187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3b99a0 msgr2=0x55ea4b3c4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.249+0000 7f50e91fa740 1 --2- 172.21.15.62:0/3266924187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3c4160 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009f50 tx=0x7f50d00053d0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.257+0000 7f50e91fa740 1 -- 172.21.15.62:0/3266924187 shutdown_connections 2024-09-15T23:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.257+0000 7f50e91fa740 1 --2- 172.21.15.62:0/3266924187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b3c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.257+0000 7f50e91fa740 1 --2- 172.21.15.62:0/3266924187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b3cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.257+0000 7f50e91fa740 1 --2- 172.21.15.62:0/3266924187 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3c4160 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.257+0000 7f50e91fa740 1 -- 172.21.15.62:0/3266924187 >> 172.21.15.62:0/3266924187 conn(0x55ea4b3b6f90 msgr2=0x55ea4b3bd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.264+0000 7f50e91fa740 1 -- 172.21.15.62:0/3266924187 shutdown_connections 2024-09-15T23:22:29.269 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.268+0000 7f50e91fa740 1 -- 172.21.15.62:0/3266924187 wait complete. 2024-09-15T23:22:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.268+0000 7f50e91fa740 1 -- start start 2024-09-15T23:22:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.268+0000 7f50e91fa740 1 Processor -- start 2024-09-15T23:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.270+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.270+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b438ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.270+0000 7f50e91fa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b43b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.270+0000 7f50e91fa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01a69c0 con 0x55ea4b3b99a0 2024-09-15T23:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.270+0000 7f50e91fa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01a7e30 con 0x55ea4b3ba2e0 2024-09-15T23:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.270+0000 7f50e91fa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f50e01a7090 con 0x55ea4b3bf3a0 2024-09-15T23:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.270+0000 7f50e91fa740 10 client.0 _ll_register_callbacks cb 0x7f50d0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.272+0000 7f50dffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.272+0000 7f50dffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56660/0 (socket says 172.21.15.62:56660) 2024-09-15T23:22:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.272+0000 7f50dffff640 1 -- 172.21.15.62:0/1273937639 learned_addr learned my addr 172.21.15.62:0/1273937639 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.272+0000 7f50e49fb640 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b43b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.274+0000 7f50df7fe640 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b438ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.274+0000 7f50dffff640 1 -- 172.21.15.62:0/1273937639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 msgr2=0x55ea4b43b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.274+0000 7f50dffff640 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b43b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.274+0000 7f50dffff640 1 -- 172.21.15.62:0/1273937639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 msgr2=0x55ea4b438ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.274+0000 7f50dffff640 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b438ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.274+0000 7f50dffff640 1 -- 172.21.15.62:0/1273937639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea4b44c660 con 0x55ea4b3b99a0 2024-09-15T23:22:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.282+0000 7f50dffff640 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3cb0f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009a30 tx=0x7f50d0005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f4ef5e8f4ce1c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.283+0000 7f50bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:22:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.283+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1273937639 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d00035f0 con 0x55ea4b3b99a0 2024-09-15T23:22:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.283+0000 7f50e91fa740 1 -- 172.21.15.62:0/1273937639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50d0003ad0 con 0x55ea4b3b99a0 2024-09-15T23:22:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.283+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1273937639 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f50d0003790 con 0x55ea4b3b99a0 2024-09-15T23:22:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.285+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1273937639 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f50d0047600 con 0x55ea4b3b99a0 2024-09-15T23:22:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.289+0000 7f50e91fa740 10 client.16986 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:29.290 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.289+0000 7f50e91fa740 1 -- 172.21.15.62:0/1273937639 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ea4b44d970 con 0x55ea4b3b99a0 2024-09-15T23:22:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.292+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1273937639 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50d0003ad0 con 0x55ea4b3b99a0 2024-09-15T23:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.292+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1273937639 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f50d005b8d0 con 0x55ea4b3b99a0 2024-09-15T23:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.292+0000 7f50e91fa740 10 client.16986 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.292+0000 7f50e91fa740 10 client.16986 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.292+0000 7f50e91fa740 20 client.16986 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.292+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ea4b44e1b0 0x55ea4b4505d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.293+0000 7f50e91fa740 4 client.16986 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.293+0000 7f50e91fa740 1 -- 172.21.15.62:0/1273937639 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ea4b326d70 con 0x55ea4b44e1b0 2024-09-15T23:22:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.294+0000 7f50df7fe640 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ea4b44e1b0 0x55ea4b4505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.299+0000 7f50df7fe640 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ea4b44e1b0 0x55ea4b4505d0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=636b09ec83069216 server_cookie=6b8c1c3e48f59a7b in_seq=0 out_seq=0 2024-09-15T23:22:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.299+0000 7f50bf7fe640 10 client.16986 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.373+0000 7f50bf7fe640 1 -- 172.21.15.62:0/1273937639 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ea4b326d70 con 0x55ea4b44e1b0 2024-09-15T23:22:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.373+0000 7f50bf7fe640 10 client.16986 handle_command_reply: tid=0 2024-09-15T23:22:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.374+0000 7f50e91fa740 1 client.16986 shutdown 2024-09-15T23:22:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.375+0000 7f50e91fa740 1 -- 172.21.15.62:0/1273937639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 msgr2=0x55ea4b3cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.375+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3cb0f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009a30 tx=0x7f50d0005da0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.375+0000 7f50e91fa740 1 -- 172.21.15.62:0/1273937639 shutdown_connections 2024-09-15T23:22:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.376+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ea4b44e1b0 0x55ea4b4505d0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.376+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ea4b3bf3a0 0x55ea4b43b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.376+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ea4b3ba2e0 0x55ea4b438ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.376+0000 7f50bf7fe640 0 client.16986 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.376+0000 7f50e91fa740 1 --2- 172.21.15.62:0/1273937639 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ea4b3b99a0 0x55ea4b3cb0f0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.376+0000 7f50e91fa740 1 -- 172.21.15.62:0/1273937639 >> 172.21.15.62:0/1273937639 conn(0x55ea4b3b6f90 msgr2=0x55ea4b3c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.376+0000 7f50e91fa740 1 -- 172.21.15.62:0/1273937639 shutdown_connections 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.376+0000 7f50e91fa740 1 -- 172.21.15.62:0/1273937639 wait complete. 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.376+0000 7f50e91fa740 20 client.16986 trim_cache size 0 max 16384 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:22:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:22:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 -- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50c80a33f0 msgr2=0x7f50c80a58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50c80a33f0 0x7f50c80a58b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f50d40096c0 tx=0x7f50d40055d0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 -- 172.21.15.62:0/2268698581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e0103710 msgr2=0x7f50e019f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e0103710 0x7f50e019f380 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f50d8009fd0 tx=0x7f50d800ef10 comp rx=0 tx=0).stop 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 -- 172.21.15.62:0/2268698581 shutdown_connections 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f50c80a33f0 0x7f50c80a58b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e0103710 0x7f50e019f380 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50e0102d50 0x7f50e0198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 --2- 172.21.15.62:0/2268698581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f50e0101b60 0x7f50e0197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 -- 172.21.15.62:0/2268698581 >> 172.21.15.62:0/2268698581 conn(0x7f50e00fd2e0 msgr2=0x7f50e0106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:29.380 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 -- 172.21.15.62:0/2268698581 shutdown_connections 2024-09-15T23:22:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:29.377+0000 7f50e645e640 1 -- 172.21.15.62:0/2268698581 wait complete. 2024-09-15T23:22:29.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 370.0 s, 1278.4 tps, lat 80.091 ms stddev 93.547 2024-09-15T23:22:29.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:29 smithi062 ceph-mon[30949]: pgmap v556: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 12 MiB/s wr, 684 op/s 2024-09-15T23:22:30.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:29 smithi067 ceph-mon[35665]: pgmap v556: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 12 MiB/s wr, 684 op/s 2024-09-15T23:22:30.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:29 smithi148 ceph-mon[35610]: pgmap v556: 129 pgs: 129 active+clean; 1.4 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 12 MiB/s wr, 684 op/s 2024-09-15T23:22:31.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:31 smithi062 ceph-mon[30949]: pgmap v557: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 14 MiB/s wr, 718 op/s 2024-09-15T23:22:32.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:31 smithi067 ceph-mon[35665]: pgmap v557: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 14 MiB/s wr, 718 op/s 2024-09-15T23:22:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:31 smithi148 ceph-mon[35610]: pgmap v557: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 14 MiB/s wr, 718 op/s 2024-09-15T23:22:32.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:32 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:22:33.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:32 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:22:33.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:32 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:22:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:33 smithi062 ceph-mon[30949]: pgmap v558: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 14 MiB/s wr, 665 op/s 2024-09-15T23:22:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:33 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:22:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:33 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:22:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:33 smithi067 ceph-mon[35665]: pgmap v558: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 14 MiB/s wr, 665 op/s 2024-09-15T23:22:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:33 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:22:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:33 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:22:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:33 smithi148 ceph-mon[35610]: pgmap v558: 129 pgs: 129 active+clean; 1.4 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 14 MiB/s wr, 665 op/s 2024-09-15T23:22:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:33 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:22:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:33 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:22:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 375.0 s, 1107.0 tps, lat 89.329 ms stddev 106.475 2024-09-15T23:22:35.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:35 smithi062 ceph-mon[30949]: pgmap v559: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 MiB/s wr, 604 op/s 2024-09-15T23:22:36.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:35 smithi067 ceph-mon[35665]: pgmap v559: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 MiB/s wr, 604 op/s 2024-09-15T23:22:36.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:35 smithi148 ceph-mon[35610]: pgmap v559: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 MiB/s wr, 604 op/s 2024-09-15T23:22:37.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:37 smithi062 ceph-mon[30949]: pgmap v560: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 MiB/s wr, 550 op/s 2024-09-15T23:22:38.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:37 smithi067 ceph-mon[35665]: pgmap v560: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 MiB/s wr, 550 op/s 2024-09-15T23:22:38.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:37 smithi148 ceph-mon[35610]: pgmap v560: 129 pgs: 129 active+clean; 1.4 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 MiB/s wr, 550 op/s 2024-09-15T23:22:39.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 380.0 s, 809.8 tps, lat 122.636 ms stddev 214.739 2024-09-15T23:22:39.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:39 smithi062 ceph-mon[30949]: pgmap v561: 129 pgs: 129 active+clean; 1.5 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 603 op/s 2024-09-15T23:22:40.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:39 smithi067 ceph-mon[35665]: pgmap v561: 129 pgs: 129 active+clean; 1.5 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 603 op/s 2024-09-15T23:22:40.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:39 smithi148 ceph-mon[35610]: pgmap v561: 129 pgs: 129 active+clean; 1.5 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 603 op/s 2024-09-15T23:22:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:41 smithi062 ceph-mon[30949]: pgmap v562: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 MiB/s wr, 1.12k op/s 2024-09-15T23:22:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:22:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:41.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:41 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:41 smithi067 ceph-mon[35665]: pgmap v562: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 MiB/s wr, 1.12k op/s 2024-09-15T23:22:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:22:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:41 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:41 smithi148 ceph-mon[35610]: pgmap v562: 129 pgs: 129 active+clean; 1.5 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 18 MiB/s wr, 1.12k op/s 2024-09-15T23:22:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:22:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:41 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.544+0000 7f5b3a18d640 1 -- 172.21.15.62:0/1414384790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 msgr2=0x7f5b34109c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.544+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/1414384790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 0x7f5b34109c10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0074e0 tx=0x7f5b1c02b0a0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.547+0000 7f5b3a18d640 1 -- 172.21.15.62:0/1414384790 shutdown_connections 2024-09-15T23:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.547+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/1414384790 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b3410a150 0x7f5b341109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.547+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/1414384790 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 0x7f5b34109c10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.547+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/1414384790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b34073730 0x7f5b34073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.547+0000 7f5b3a18d640 1 -- 172.21.15.62:0/1414384790 >> 172.21.15.62:0/1414384790 conn(0x7f5b3406ef20 msgr2=0x7f5b34071360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:42.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.551+0000 7f5b3a18d640 1 -- 172.21.15.62:0/1414384790 shutdown_connections 2024-09-15T23:22:42.557 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.554+0000 7f5b3a18d640 1 -- 172.21.15.62:0/1414384790 wait complete. 2024-09-15T23:22:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.555+0000 7f5b3a18d640 1 Processor -- start 2024-09-15T23:22:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.555+0000 7f5b3a18d640 1 -- start start 2024-09-15T23:22:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.555+0000 7f5b3a18d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b34073730 0x7f5b341a6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.555+0000 7f5b3a18d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 0x7f5b341a70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.555+0000 7f5b3a18d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b3410a150 0x7f5b341ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.555+0000 7f5b3a18d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5b34113440 con 0x7f5b3410a150 2024-09-15T23:22:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.555+0000 7f5b3a18d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b341132c0 con 0x7f5b34109790 2024-09-15T23:22:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.555+0000 7f5b3a18d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5b341135c0 con 0x7f5b34073730 2024-09-15T23:22:42.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 0x7f5b341a70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 0x7f5b341a70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36180/0 (socket says 172.21.15.62:36180) 2024-09-15T23:22:42.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b32ffd640 1 -- 172.21.15.62:0/3626860330 learned_addr learned my addr 172.21.15.62:0/3626860330 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:42.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b33fff640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b3410a150 0x7f5b341ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b32ffd640 1 -- 172.21.15.62:0/3626860330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b34073730 msgr2=0x7f5b341a6bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:22:42.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b34073730 0x7f5b341a6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b32ffd640 1 -- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b3410a150 msgr2=0x7f5b341ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b3410a150 0x7f5b341ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b32ffd640 1 -- 172.21.15.62:0/3626860330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b34117480 con 0x7f5b34109790 2024-09-15T23:22:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.556+0000 7f5b33fff640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b3410a150 0x7f5b341ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.557+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 0x7f5b341a70f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0053e0 tx=0x7f5b1c009fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:42.561 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.559+0000 7f5b30ff9640 1 -- 172.21.15.62:0/3626860330 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c041070 con 0x7f5b34109790 2024-09-15T23:22:42.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.559+0000 7f5b3a18d640 1 -- 172.21.15.62:0/3626860330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b34117710 con 0x7f5b34109790 2024-09-15T23:22:42.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.559+0000 7f5b3a18d640 1 -- 172.21.15.62:0/3626860330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b34117c60 con 0x7f5b34109790 2024-09-15T23:22:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.563+0000 7f5b3d000740 1 Processor -- start 2024-09-15T23:22:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.564+0000 7f5b30ff9640 1 -- 172.21.15.62:0/3626860330 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b1c007ce0 con 0x7f5b34109790 2024-09-15T23:22:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.564+0000 7f5b30ff9640 1 -- 172.21.15.62:0/3626860330 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00a930 con 0x7f5b34109790 2024-09-15T23:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.568+0000 7f5b3d000740 1 -- start start 2024-09-15T23:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.569+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b19a0 0x563c178b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.569+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b22e0 0x563c178b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.569+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c178b9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.569+0000 7f5b3d000740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563c178152f0 con 0x563c178b22e0 2024-09-15T23:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.569+0000 7f5b3d000740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563c177bd270 con 0x563c178b19a0 2024-09-15T23:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.569+0000 7f5b3d000740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563c178112b0 con 0x563c178b7370 2024-09-15T23:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.569+0000 7f5b30ff9640 1 -- 172.21.15.62:0/3626860330 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f5b1c044070 con 0x7f5b34109790 2024-09-15T23:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.570+0000 7f5b33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b19a0 0x563c178b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.570+0000 7f5b33fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b19a0 0x563c178b1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36196/0 (socket says 172.21.15.62:36196) 2024-09-15T23:22:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.570+0000 7f5b33fff640 1 -- 172.21.15.62:0/3463500890 learned_addr learned my addr 172.21.15.62:0/3463500890 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.570+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c178b9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.571+0000 7f5b30ff9640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b200a34c0 0x7f5b200a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.571+0000 7f5b30ff9640 1 -- 172.21.15.62:0/3626860330 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5b1c0efdb0 con 0x7f5b34109790 2024-09-15T23:22:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.571+0000 7f5b33fff640 1 -- 172.21.15.62:0/3463500890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 msgr2=0x563c178b9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.571+0000 7f5b33fff640 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c178b9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.571+0000 7f5b33fff640 1 -- 172.21.15.62:0/3463500890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b22e0 msgr2=0x563c178b6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.571+0000 7f5b337fe640 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b22e0 0x563c178b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.572+0000 7f5b33fff640 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b22e0 0x563c178b6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.572+0000 7f5b33fff640 1 -- 172.21.15.62:0/3463500890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c177bc340 con 0x563c178b19a0 2024-09-15T23:22:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.572+0000 7f5b337fe640 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b22e0 0x563c178b6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:42.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.572+0000 7f5b33fff640 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b19a0 0x563c178b1da0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00bb60 tx=0x7f5b2c00d710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60b0a707d88aca17 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.575+0000 7f5b29ffb640 1 -- 172.21.15.62:0/3463500890 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c00be60 con 0x563c178b19a0 2024-09-15T23:22:42.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.575+0000 7f5b29ffb640 1 -- 172.21.15.62:0/3463500890 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b2c017b00 con 0x563c178b19a0 2024-09-15T23:22:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.575+0000 7f5b29ffb640 1 -- 172.21.15.62:0/3463500890 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c00fca0 con 0x563c178b19a0 2024-09-15T23:22:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.576+0000 7f5b33fff640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b200a34c0 0x7f5b200a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.576+0000 7f5b3d000740 1 -- 172.21.15.62:0/3463500890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b19a0 msgr2=0x563c178b1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.576+0000 7f5b3d000740 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b19a0 0x563c178b1da0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00bb60 tx=0x7f5b2c00d710 comp rx=0 tx=0).stop 2024-09-15T23:22:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.577+0000 7f5b3d000740 1 -- 172.21.15.62:0/3463500890 shutdown_connections 2024-09-15T23:22:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.577+0000 7f5b3d000740 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c178b9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.577+0000 7f5b3d000740 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b22e0 0x563c178b6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.577+0000 7f5b3d000740 1 --2- 172.21.15.62:0/3463500890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b19a0 0x563c178b1da0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.577+0000 7f5b3d000740 1 -- 172.21.15.62:0/3463500890 >> 172.21.15.62:0/3463500890 conn(0x563c178ad010 msgr2=0x563c178ad410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.577+0000 7f5b33fff640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b200a34c0 0x7f5b200a5980 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00bb60 tx=0x7f5b2c00d710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.581+0000 7f5b3d000740 1 -- 172.21.15.62:0/3463500890 shutdown_connections 2024-09-15T23:22:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.583+0000 7f5b3d000740 1 -- 172.21.15.62:0/3463500890 wait complete. 2024-09-15T23:22:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.583+0000 7f5b3d000740 1 -- start start 2024-09-15T23:22:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.584+0000 7f5b3d000740 1 Processor -- start 2024-09-15T23:22:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.584+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 0x563c178bbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.584+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178bc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.584+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c178c3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.584+0000 7f5b3d000740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563c178ba340 con 0x563c178b19a0 2024-09-15T23:22:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.584+0000 7f5b3d000740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563c1779e290 con 0x563c178b22e0 2024-09-15T23:22:42.587 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.584+0000 7f5b3d000740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563c178ba4c0 con 0x563c178b7370 2024-09-15T23:22:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b3d000740 10 client.0 _ll_register_callbacks cb 0x563c177cf140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178bc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b33fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c178c3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178bc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36206/0 (socket says 172.21.15.62:36206) 2024-09-15T23:22:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b32ffd640 1 -- 172.21.15.62:0/4056195143 learned_addr learned my addr 172.21.15.62:0/4056195143 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b32ffd640 1 -- 172.21.15.62:0/4056195143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 msgr2=0x563c178c3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c178c3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b32ffd640 1 -- 172.21.15.62:0/4056195143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 msgr2=0x563c178bbbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:22:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 0x563c178bbbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.585+0000 7f5b32ffd640 1 -- 172.21.15.62:0/4056195143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c17528210 con 0x563c178b22e0 2024-09-15T23:22:42.589 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.587+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178bc130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0b84e0 tx=0x7f5b1c039000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c592e57c47e824fe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.588+0000 7f5b12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.588+0000 7f5b12ffd640 1 -- 172.21.15.62:0/4056195143 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0dc0b0 con 0x563c178b22e0 2024-09-15T23:22:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.589+0000 7f5b3d000740 1 -- 172.21.15.62:0/4056195143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b2c017b00 con 0x563c178b22e0 2024-09-15T23:22:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.589+0000 7f5b12ffd640 1 -- 172.21.15.62:0/4056195143 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5b1c00a040 con 0x563c178b22e0 2024-09-15T23:22:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.590+0000 7f5b12ffd640 1 -- 172.21.15.62:0/4056195143 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b1c0bc080 con 0x563c178b22e0 2024-09-15T23:22:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.594+0000 7f5b3d000740 10 client.26567 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.594+0000 7f5b3d000740 1 -- 172.21.15.62:0/4056195143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563c175f4620 con 0x563c178b22e0 2024-09-15T23:22:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.595+0000 7f5b12ffd640 1 -- 172.21.15.62:0/4056195143 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b2c017b00 con 0x563c178b22e0 2024-09-15T23:22:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.595+0000 7f5b12ffd640 1 -- 172.21.15.62:0/4056195143 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f5b1c0bc2e0 con 0x563c178b22e0 2024-09-15T23:22:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.595+0000 7f5b3d000740 10 client.26567 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.595+0000 7f5b3d000740 10 client.26567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.595+0000 7f5b3d000740 20 client.26567 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.595+0000 7f5b3d000740 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c178c7ab0 0x563c178c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.595+0000 7f5b3d000740 4 client.26567 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.595+0000 7f5b3d000740 1 -- 172.21.15.62:0/4056195143 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c17822a20 con 0x563c178c7ab0 2024-09-15T23:22:42.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.596+0000 7f5b337fe640 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c178c7ab0 0x563c178c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.597+0000 7f5b337fe640 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c178c7ab0 0x563c178c9f70 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd3fad340a0d2077 server_cookie=c01e31627c02c1e1 in_seq=0 out_seq=0 2024-09-15T23:22:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.597+0000 7f5b12ffd640 10 client.26567 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.602+0000 7f5b12ffd640 1 -- 172.21.15.62:0/4056195143 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563c17822a20 con 0x563c178c7ab0 2024-09-15T23:22:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.602+0000 7f5b12ffd640 10 client.26567 handle_command_reply: tid=0 2024-09-15T23:22:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.602+0000 7f5b3d000740 1 client.26567 shutdown 2024-09-15T23:22:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.603+0000 7f5b3d000740 1 -- 172.21.15.62:0/4056195143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 msgr2=0x563c178bc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.603+0000 7f5b3d000740 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178bc130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0b84e0 tx=0x7f5b1c039000 comp rx=0 tx=0).stop 2024-09-15T23:22:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.604+0000 7f5b3d000740 1 -- 172.21.15.62:0/4056195143 shutdown_connections 2024-09-15T23:22:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.604+0000 7f5b3d000740 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c178c7ab0 0x563c178c9f70 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.604+0000 7f5b3d000740 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c178c3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.604+0000 7f5b3d000740 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178bc130 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.604+0000 7f5b3d000740 1 --2- 172.21.15.62:0/4056195143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 0x563c178bbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.604+0000 7f5b3d000740 1 -- 172.21.15.62:0/4056195143 >> 172.21.15.62:0/4056195143 conn(0x563c178ad010 msgr2=0x563c178ae820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:42.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.604+0000 7f5b12ffd640 0 client.26567 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.605+0000 7f5b3d000740 1 -- 172.21.15.62:0/4056195143 shutdown_connections 2024-09-15T23:22:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.605+0000 7f5b3d000740 1 -- 172.21.15.62:0/4056195143 wait complete. 2024-09-15T23:22:42.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.605+0000 7f5b3d000740 20 client.26567 trim_cache size 0 max 16384 2024-09-15T23:22:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b3d000740 1 Processor -- start 2024-09-15T23:22:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b3d000740 1 -- start start 2024-09-15T23:22:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b19a0 0x563c178adec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b7370 0x563c178ae400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b337fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b19a0 0x563c178adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b337fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b19a0 0x563c178adec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:57320/0 (socket says 172.21.15.62:57320) 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b33fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b7370 0x563c178ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b33fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b7370 0x563c178ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41980/0 (socket says 172.21.15.62:41980) 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36210/0 (socket says 172.21.15.62:36210) 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b337fe640 1 -- 172.21.15.62:0/3962493908 learned_addr learned my addr 172.21.15.62:0/3962493908 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b3d000740 1 -- 172.21.15.62:0/3962493908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5b14012530 con 0x563c178b7370 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b3d000740 1 -- 172.21.15.62:0/3962493908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1400fe00 con 0x563c178b22e0 2024-09-15T23:22:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.635+0000 7f5b3d000740 1 -- 172.21.15.62:0/3962493908 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1400dc50 con 0x563c178b19a0 2024-09-15T23:22:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.636+0000 7f5b32ffd640 1 -- 172.21.15.62:0/3962493908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b19a0 msgr2=0x563c178adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.636+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/3962493908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b19a0 0x563c178adec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.636+0000 7f5b32ffd640 1 -- 172.21.15.62:0/3962493908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b7370 msgr2=0x563c178ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.636+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/3962493908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b7370 0x563c178ae400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.636+0000 7f5b32ffd640 1 -- 172.21.15.62:0/3962493908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c178b2f70 con 0x563c178b22e0 2024-09-15T23:22:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.636+0000 7f5b337fe640 1 --2- 172.21.15.62:0/3962493908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b19a0 0x563c178adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.636+0000 7f5b32ffd640 1 --2- 172.21.15.62:0/3962493908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c3130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0074b0 tx=0x7f5b1c02b0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da7bdd8b8a2fdd6c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.636+0000 7f5b297fa640 1 -- 172.21.15.62:0/3962493908 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0bbe50 con 0x563c178b22e0 2024-09-15T23:22:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.637+0000 7f5b297fa640 1 -- 172.21.15.62:0/3962493908 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b1c0dc0b0 con 0x563c178b22e0 2024-09-15T23:22:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.637+0000 7f5b297fa640 1 -- 172.21.15.62:0/3962493908 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0c0430 con 0x563c178b22e0 2024-09-15T23:22:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.637+0000 7f5b3d000740 1 -- 172.21.15.62:0/3962493908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 msgr2=0x563c178c3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.637+0000 7f5b3d000740 1 --2- 172.21.15.62:0/3962493908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c3130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0074b0 tx=0x7f5b1c02b0a0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.638+0000 7f5b3d000740 1 -- 172.21.15.62:0/3962493908 shutdown_connections 2024-09-15T23:22:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.638+0000 7f5b3d000740 1 --2- 172.21.15.62:0/3962493908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b7370 0x563c178ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.638+0000 7f5b3d000740 1 --2- 172.21.15.62:0/3962493908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c3130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.638+0000 7f5b3d000740 1 --2- 172.21.15.62:0/3962493908 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b19a0 0x563c178adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.638+0000 7f5b3d000740 1 -- 172.21.15.62:0/3962493908 >> 172.21.15.62:0/3962493908 conn(0x563c178ad010 msgr2=0x563c178b3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.638+0000 7f5b3d000740 1 -- 172.21.15.62:0/3962493908 shutdown_connections 2024-09-15T23:22:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.638+0000 7f5b3d000740 1 -- 172.21.15.62:0/3962493908 wait complete. 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.638+0000 7f5b3d000740 1 -- start start 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.639+0000 7f5b3d000740 1 Processor -- start 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.639+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 0x563c178c0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b3d000740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c17930840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b3d000740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1400e2c0 con 0x563c178b19a0 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b3d000740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1400f730 con 0x563c178b22e0 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b3d000740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1400e990 con 0x563c178b7370 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b32ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b33fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c17930840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b33fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c17930840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:57330/0 (socket says 172.21.15.62:57330) 2024-09-15T23:22:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b33fff640 1 -- 172.21.15.62:0/1013014952 learned_addr learned my addr 172.21.15.62:0/1013014952 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b3d000740 10 client.0 _ll_register_callbacks cb 0x7f5b1c00aad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.640+0000 7f5b337fe640 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 0x563c178c0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.641+0000 7f5b33fff640 1 -- 172.21.15.62:0/1013014952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 msgr2=0x563c178c0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.641+0000 7f5b33fff640 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.641+0000 7f5b33fff640 1 -- 172.21.15.62:0/1013014952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 msgr2=0x563c178c0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.641+0000 7f5b33fff640 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 0x563c178c0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.641+0000 7f5b33fff640 1 -- 172.21.15.62:0/1013014952 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c17930e50 con 0x563c178b7370 2024-09-15T23:22:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.641+0000 7f5b337fe640 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 0x563c178c0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.641+0000 7f5b33fff640 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c17930840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0056f0 tx=0x7f5b2c00b470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de68c52b6731b5b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.643+0000 7f5b12ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:22:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.643+0000 7f5b12ffd640 1 -- 172.21.15.62:0/1013014952 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c00b690 con 0x563c178b7370 2024-09-15T23:22:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.643+0000 7f5b12ffd640 1 -- 172.21.15.62:0/1013014952 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5b2c017d80 con 0x563c178b7370 2024-09-15T23:22:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.643+0000 7f5b3d000740 1 -- 172.21.15.62:0/1013014952 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b1c0dc0b0 con 0x563c178b7370 2024-09-15T23:22:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.643+0000 7f5b12ffd640 1 -- 172.21.15.62:0/1013014952 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b2c042050 con 0x563c178b7370 2024-09-15T23:22:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.644+0000 7f5b3d000740 10 client.26371 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:42.646 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.644+0000 7f5b3d000740 1 -- 172.21.15.62:0/1013014952 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563c179321b0 con 0x563c178b7370 2024-09-15T23:22:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.644+0000 7f5b12ffd640 1 -- 172.21.15.62:0/1013014952 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b1c0dc0b0 con 0x563c178b7370 2024-09-15T23:22:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.644+0000 7f5b12ffd640 1 -- 172.21.15.62:0/1013014952 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f5b2c00f5f0 con 0x563c178b7370 2024-09-15T23:22:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.645+0000 7f5b3d000740 10 client.26371 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.645+0000 7f5b3d000740 10 client.26371 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.645+0000 7f5b3d000740 20 client.26371 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.645+0000 7f5b3d000740 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c17932b60 0x563c17934f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.645+0000 7f5b3d000740 4 client.26371 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:22:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.645+0000 7f5b3d000740 1 -- 172.21.15.62:0/1013014952 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563c1781cd70 con 0x563c17932b60 2024-09-15T23:22:42.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.645+0000 7f5b337fe640 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c17932b60 0x563c17934f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.645+0000 7f5b12ffd640 10 client.26371 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:42.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.646+0000 7f5b337fe640 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c17932b60 0x563c17934f80 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f09a58e488424f5 server_cookie=249467f5041914a4 in_seq=0 out_seq=0 2024-09-15T23:22:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.751+0000 7f5b12ffd640 1 -- 172.21.15.62:0/1013014952 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563c1781cd70 con 0x563c17932b60 2024-09-15T23:22:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.751+0000 7f5b12ffd640 10 client.26371 handle_command_reply: tid=0 2024-09-15T23:22:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.751+0000 7f5b3d000740 1 client.26371 shutdown 2024-09-15T23:22:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 -- 172.21.15.62:0/1013014952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 msgr2=0x563c17930840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c17930840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c0056f0 tx=0x7f5b2c00b470 comp rx=0 tx=0).stop 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 -- 172.21.15.62:0/1013014952 shutdown_connections 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563c17932b60 0x563c17934f80 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563c178b7370 0x563c17930840 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563c178b22e0 0x563c178c0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 --2- 172.21.15.62:0/1013014952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563c178b19a0 0x563c178c0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 -- 172.21.15.62:0/1013014952 >> 172.21.15.62:0/1013014952 conn(0x563c178ad010 msgr2=0x563c178bbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b12ffd640 0 client.26371 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 -- 172.21.15.62:0/1013014952 shutdown_connections 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 1 -- 172.21.15.62:0/1013014952 wait complete. 2024-09-15T23:22:42.755 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.752+0000 7f5b3d000740 20 client.26371 trim_cache size 0 max 16384 2024-09-15T23:22:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:22:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:22:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:22:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:22:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 -- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b200a34c0 msgr2=0x7f5b200a5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b200a34c0 0x7f5b200a5980 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00bb60 tx=0x7f5b2c00d710 comp rx=0 tx=0).stop 2024-09-15T23:22:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 -- 172.21.15.62:0/3626860330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 msgr2=0x7f5b341a70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 0x7f5b341a70f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0053e0 tx=0x7f5b1c009fb0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 -- 172.21.15.62:0/3626860330 shutdown_connections 2024-09-15T23:22:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b200a34c0 0x7f5b200a5980 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b3410a150 0x7f5b341ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b34109790 0x7f5b341a70f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 --2- 172.21.15.62:0/3626860330 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b34073730 0x7f5b341a6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.753+0000 7f5b3a18d640 1 -- 172.21.15.62:0/3626860330 >> 172.21.15.62:0/3626860330 conn(0x7f5b3406ef20 msgr2=0x7f5b34115eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.754+0000 7f5b3a18d640 1 -- 172.21.15.62:0/3626860330 shutdown_connections 2024-09-15T23:22:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:42.754+0000 7f5b3a18d640 1 -- 172.21.15.62:0/3626860330 wait complete. 2024-09-15T23:22:42.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:42 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi067 to 176.0M 2024-09-15T23:22:42.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:42 smithi062 ceph-mon[30949]: Unable to set osd_memory_target on smithi067 to 184598220: error parsing value: Value '184598220' is below minimum 939524096 2024-09-15T23:22:42.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:42.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:22:42.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:22:42.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:22:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:42 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi067 to 176.0M 2024-09-15T23:22:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:42 smithi067 ceph-mon[35665]: Unable to set osd_memory_target on smithi067 to 184598220: error parsing value: Value '184598220' is below minimum 939524096 2024-09-15T23:22:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:22:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:22:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:22:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:42 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi067 to 176.0M 2024-09-15T23:22:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:42 smithi148 ceph-mon[35610]: Unable to set osd_memory_target on smithi067 to 184598220: error parsing value: Value '184598220' is below minimum 939524096 2024-09-15T23:22:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:22:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:22:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:22:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:22:43.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:43 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi062 to 176.0M 2024-09-15T23:22:43.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:43 smithi062 ceph-mon[30949]: Unable to set osd_memory_target on smithi062 to 184587468: error parsing value: Value '184587468' is below minimum 939524096 2024-09-15T23:22:43.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:43 smithi062 ceph-mon[30949]: pgmap v563: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 22 MiB/s wr, 1.56k op/s 2024-09-15T23:22:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:43 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi062 to 176.0M 2024-09-15T23:22:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:43 smithi067 ceph-mon[35665]: Unable to set osd_memory_target on smithi062 to 184587468: error parsing value: Value '184587468' is below minimum 939524096 2024-09-15T23:22:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:43 smithi067 ceph-mon[35665]: pgmap v563: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 22 MiB/s wr, 1.56k op/s 2024-09-15T23:22:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:43 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi062 to 176.0M 2024-09-15T23:22:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:43 smithi148 ceph-mon[35610]: Unable to set osd_memory_target on smithi062 to 184587468: error parsing value: Value '184587468' is below minimum 939524096 2024-09-15T23:22:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:43 smithi148 ceph-mon[35610]: pgmap v563: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 22 MiB/s wr, 1.56k op/s 2024-09-15T23:22:44.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 385.0 s, 1156.6 tps, lat 87.186 ms stddev 108.812 2024-09-15T23:22:45.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:45 smithi062 ceph-mon[30949]: pgmap v564: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 21 MiB/s wr, 1.59k op/s 2024-09-15T23:22:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:45 smithi067 ceph-mon[35665]: pgmap v564: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 21 MiB/s wr, 1.59k op/s 2024-09-15T23:22:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:45 smithi148 ceph-mon[35610]: pgmap v564: 129 pgs: 129 active+clean; 1.5 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 21 MiB/s wr, 1.59k op/s 2024-09-15T23:22:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:47 smithi062 ceph-mon[30949]: pgmap v565: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 21 MiB/s wr, 1.59k op/s 2024-09-15T23:22:48.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:47 smithi067 ceph-mon[35665]: pgmap v565: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 21 MiB/s wr, 1.59k op/s 2024-09-15T23:22:48.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:47 smithi148 ceph-mon[35610]: pgmap v565: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 21 MiB/s wr, 1.59k op/s 2024-09-15T23:22:49.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 390.0 s, 1020.4 tps, lat 97.915 ms stddev 115.744 2024-09-15T23:22:50.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:49 smithi067 ceph-mon[35665]: pgmap v566: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 21 MiB/s wr, 1.66k op/s 2024-09-15T23:22:50.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:49 smithi148 ceph-mon[35610]: pgmap v566: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 21 MiB/s wr, 1.66k op/s 2024-09-15T23:22:50.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:49 smithi062 ceph-mon[30949]: pgmap v566: 129 pgs: 129 active+clean; 1.5 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 21 MiB/s wr, 1.66k op/s 2024-09-15T23:22:52.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:51 smithi067 ceph-mon[35665]: pgmap v567: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 21 MiB/s wr, 1.68k op/s 2024-09-15T23:22:52.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:51 smithi148 ceph-mon[35610]: pgmap v567: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 21 MiB/s wr, 1.68k op/s 2024-09-15T23:22:52.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:51 smithi062 ceph-mon[30949]: pgmap v567: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 21 MiB/s wr, 1.68k op/s 2024-09-15T23:22:52.237 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:22:52.706 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:53.118 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:52 smithi062 ceph-mon[30949]: pgmap v568: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 14 MiB/s wr, 1.17k op/s 2024-09-15T23:22:53.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.252+0000 7f42e72dc640 1 -- 172.21.15.62:0/2267027442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e010c1b0 msgr2=0x7f42e010e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:53.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.252+0000 7f42e72dc640 1 --2- 172.21.15.62:0/2267027442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e010c1b0 0x7f42e010e620 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f42d8009ca0 tx=0x7f42d802a950 comp rx=0 tx=0).stop 2024-09-15T23:22:53.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.252+0000 7f42e72dc640 1 -- 172.21.15.62:0/2267027442 shutdown_connections 2024-09-15T23:22:53.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.252+0000 7f42e72dc640 1 --2- 172.21.15.62:0/2267027442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e010c1b0 0x7f42e010e620 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.252+0000 7f42e72dc640 1 --2- 172.21.15.62:0/2267027442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e01036f0 0x7f42e010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.252+0000 7f42e72dc640 1 --2- 172.21.15.62:0/2267027442 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42e0102dd0 0x7f42e01031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.252+0000 7f42e72dc640 1 -- 172.21.15.62:0/2267027442 >> 172.21.15.62:0/2267027442 conn(0x7f42e00fc7d0 msgr2=0x7f42e00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:53.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.252+0000 7f42e72dc640 1 -- 172.21.15.62:0/2267027442 shutdown_connections 2024-09-15T23:22:53.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.252+0000 7f42e72dc640 1 -- 172.21.15.62:0/2267027442 wait complete. 2024-09-15T23:22:53.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.253+0000 7f42e72dc640 1 Processor -- start 2024-09-15T23:22:53.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.253+0000 7f42e72dc640 1 -- start start 2024-09-15T23:22:53.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.254+0000 7f42e72dc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42e0102dd0 0x7f42e0198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.254+0000 7f42e72dc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e01036f0 0x7f42e0198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.254+0000 7f42e5078640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42e0102dd0 0x7f42e0198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.254+0000 7f42e5078640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42e0102dd0 0x7f42e0198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34528/0 (socket says 172.21.15.62:34528) 2024-09-15T23:22:53.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.254+0000 7f42e5078640 1 -- 172.21.15.62:0/3158037990 learned_addr learned my addr 172.21.15.62:0/3158037990 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:53.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.254+0000 7f42e4877640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e01036f0 0x7f42e0198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.254+0000 7f42e72dc640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e010c1b0 0x7f42e019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.254+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f42e0111f80 con 0x7f42e010c1b0 2024-09-15T23:22:53.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.255+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f42e0111e00 con 0x7f42e01036f0 2024-09-15T23:22:53.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.255+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f42e0112100 con 0x7f42e0102dd0 2024-09-15T23:22:53.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.256+0000 7f42e5879640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e010c1b0 0x7f42e019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.256+0000 7f42e4877640 1 -- 172.21.15.62:0/3158037990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42e0102dd0 msgr2=0x7f42e0198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.256+0000 7f42e4877640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42e0102dd0 0x7f42e0198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.256+0000 7f42e4877640 1 -- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e010c1b0 msgr2=0x7f42e019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.256+0000 7f42e4877640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e010c1b0 0x7f42e019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.256+0000 7f42e4877640 1 -- 172.21.15.62:0/3158037990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e019d3b0 con 0x7f42e01036f0 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.257+0000 7f42e4877640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e01036f0 0x7f42e0198940 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009f10 tx=0x7f42d000ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.257+0000 7f42d67fc640 1 -- 172.21.15.62:0/3158037990 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d0020070 con 0x7f42e01036f0 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.257+0000 7f42d67fc640 1 -- 172.21.15.62:0/3158037990 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42d000ccc0 con 0x7f42e01036f0 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.257+0000 7f42d67fc640 1 -- 172.21.15.62:0/3158037990 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d001bab0 con 0x7f42e01036f0 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.257+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e019d6a0 con 0x7f42e01036f0 2024-09-15T23:22:53.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.258+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e0100960 con 0x7f42e01036f0 2024-09-15T23:22:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.269+0000 7f42e96b5740 1 Processor -- start 2024-09-15T23:22:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.269+0000 7f42d67fc640 1 -- 172.21.15.62:0/3158037990 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f42d0023070 con 0x7f42e01036f0 2024-09-15T23:22:53.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.270+0000 7f42d67fc640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42cc0a3340 0x7f42cc0a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.271+0000 7f42d67fc640 1 -- 172.21.15.62:0/3158037990 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f42d00cf950 con 0x7f42e01036f0 2024-09-15T23:22:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e96b5740 1 -- start start 2024-09-15T23:22:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 0x5558e9580090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e958a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e958d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e96b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5558e94dd2e0 con 0x5558e95805d0 2024-09-15T23:22:53.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e96b5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5558e9484ce0 con 0x5558e957fcb0 2024-09-15T23:22:53.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e96b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5558e94d92a0 con 0x5558e958ae90 2024-09-15T23:22:53.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:52 smithi067 ceph-mon[35665]: pgmap v568: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 14 MiB/s wr, 1.17k op/s 2024-09-15T23:22:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e5879640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e958d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e5879640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42cc0a3340 0x7f42cc0a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e5879640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e958d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34538/0 (socket says 172.21.15.62:34538) 2024-09-15T23:22:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.272+0000 7f42e5879640 1 -- 172.21.15.62:0/3329847611 learned_addr learned my addr 172.21.15.62:0/3329847611 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.274+0000 7f42e4877640 1 --2- 172.21.15.62:0/3329847611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e958a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.275+0000 7f42e5078640 1 --2- 172.21.15.62:0/3329847611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 0x5558e9580090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.275+0000 7f42e5879640 1 -- 172.21.15.62:0/3329847611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 msgr2=0x5558e9580090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:53.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.275+0000 7f42e5879640 1 --2- 172.21.15.62:0/3329847611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 0x5558e9580090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.275+0000 7f42e5879640 1 -- 172.21.15.62:0/3329847611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 msgr2=0x5558e958a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:53.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.275+0000 7f42e5879640 1 --2- 172.21.15.62:0/3329847611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e958a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.275+0000 7f42e5879640 1 -- 172.21.15.62:0/3329847611 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e9469440 con 0x5558e958ae90 2024-09-15T23:22:53.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.278+0000 7f42e5879640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42cc0a3340 0x7f42cc0a5800 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f42d8009a60 tx=0x7f42d8004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:53.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.278+0000 7f42e5879640 1 --2- 172.21.15.62:0/3329847611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e958d280 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f42d8035000 tx=0x7f42d8047000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=203171ed9b981b79 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:53.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.282+0000 7f42cb7fe640 1 -- 172.21.15.62:0/3329847611 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8040030 con 0x5558e958ae90 2024-09-15T23:22:53.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.282+0000 7f42cb7fe640 1 -- 172.21.15.62:0/3329847611 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42d8003d70 con 0x5558e958ae90 2024-09-15T23:22:53.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.282+0000 7f42cb7fe640 1 -- 172.21.15.62:0/3329847611 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8043040 con 0x5558e958ae90 2024-09-15T23:22:53.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.285+0000 7f42e96b5740 1 -- 172.21.15.62:0/3329847611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 msgr2=0x5558e958d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:53.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.285+0000 7f42e96b5740 1 --2- 172.21.15.62:0/3329847611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e958d280 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f42d8035000 tx=0x7f42d8047000 comp rx=0 tx=0).stop 2024-09-15T23:22:53.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.288+0000 7f42e96b5740 1 -- 172.21.15.62:0/3329847611 shutdown_connections 2024-09-15T23:22:53.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:52 smithi148 ceph-mon[35610]: pgmap v568: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 14 MiB/s wr, 1.17k op/s 2024-09-15T23:22:53.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.288+0000 7f42e96b5740 1 --2- 172.21.15.62:0/3329847611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e958d280 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.288+0000 7f42e96b5740 1 --2- 172.21.15.62:0/3329847611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e958a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.288+0000 7f42e96b5740 1 --2- 172.21.15.62:0/3329847611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 0x5558e9580090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.288+0000 7f42e96b5740 1 -- 172.21.15.62:0/3329847611 >> 172.21.15.62:0/3329847611 conn(0x5558e957c240 msgr2=0x5558e957c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:53.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.294+0000 7f42e96b5740 1 -- 172.21.15.62:0/3329847611 shutdown_connections 2024-09-15T23:22:53.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.294+0000 7f42e96b5740 1 -- 172.21.15.62:0/3329847611 wait complete. 2024-09-15T23:22:53.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.295+0000 7f42e96b5740 1 -- start start 2024-09-15T23:22:53.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.295+0000 7f42e96b5740 1 Processor -- start 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.297+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 0x5558e958a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.297+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e9587ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.297+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 0x5558e9588080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.297+0000 7f42e96b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5558e958da60 con 0x5558e957fcb0 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.297+0000 7f42e96b5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5558e9498050 con 0x5558e958ae90 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.297+0000 7f42e96b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5558e958dbe0 con 0x5558e95805d0 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.297+0000 7f42e96b5740 10 client.0 _ll_register_callbacks cb 0x5558e94d8160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.300+0000 7f42e4877640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e9587ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.300+0000 7f42e4877640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e9587ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34540/0 (socket says 172.21.15.62:34540) 2024-09-15T23:22:53.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.300+0000 7f42e4877640 1 -- 172.21.15.62:0/1150753948 learned_addr learned my addr 172.21.15.62:0/1150753948 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:53.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.301+0000 7f42e5879640 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 0x5558e9588080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.304+0000 7f42e5078640 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 0x5558e958a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.306+0000 7f42e4877640 1 -- 172.21.15.62:0/1150753948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 msgr2=0x5558e9588080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:53.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.306+0000 7f42e4877640 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 0x5558e9588080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.306+0000 7f42e4877640 1 -- 172.21.15.62:0/1150753948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 msgr2=0x5558e958a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:53.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.306+0000 7f42e4877640 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 0x5558e958a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.306+0000 7f42e4877640 1 -- 172.21.15.62:0/1150753948 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558e924e3c0 con 0x5558e95805d0 2024-09-15T23:22:53.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.306+0000 7f42e4877640 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e9587ae0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f42d001b4f0 tx=0x7f42d009b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f085415bb70f870 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:53.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.307+0000 7f42c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:22:53.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.310+0000 7f42c8ff9640 1 -- 172.21.15.62:0/1150753948 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d00a4570 con 0x5558e95805d0 2024-09-15T23:22:53.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.310+0000 7f42c8ff9640 1 -- 172.21.15.62:0/1150753948 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f42d009b090 con 0x5558e95805d0 2024-09-15T23:22:53.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.310+0000 7f42e96b5740 1 -- 172.21.15.62:0/1150753948 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42d8003d70 con 0x5558e95805d0 2024-09-15T23:22:53.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.311+0000 7f42c8ff9640 1 -- 172.21.15.62:0/1150753948 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f42d00a4800 con 0x5558e95805d0 2024-09-15T23:22:53.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.313+0000 7f42e96b5740 10 client.26383 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:53.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.313+0000 7f42e96b5740 1 -- 172.21.15.62:0/1150753948 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5558e95aa530 con 0x5558e95805d0 2024-09-15T23:22:53.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.313+0000 7f42c8ff9640 1 -- 172.21.15.62:0/1150753948 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42d8003d70 con 0x5558e95805d0 2024-09-15T23:22:53.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.315+0000 7f42c8ff9640 1 -- 172.21.15.62:0/1150753948 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f42d00860b0 con 0x5558e95805d0 2024-09-15T23:22:53.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.315+0000 7f42e96b5740 10 client.26383 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:53.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.315+0000 7f42e96b5740 10 client.26383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:53.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.315+0000 7f42e96b5740 20 client.26383 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:53.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.315+0000 7f42e96b5740 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5558e95abd70 0x5558e95ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.315+0000 7f42e96b5740 4 client.26383 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:53.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.315+0000 7f42e96b5740 1 -- 172.21.15.62:0/1150753948 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558e94ea850 con 0x5558e95abd70 2024-09-15T23:22:53.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.316+0000 7f42e5078640 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5558e95abd70 0x5558e95ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.321+0000 7f42e5078640 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5558e95abd70 0x5558e95ae1d0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dda230478f9a40a5 server_cookie=a423330b754e5961 in_seq=0 out_seq=0 2024-09-15T23:22:53.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.321+0000 7f42c8ff9640 10 client.26383 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.322+0000 7f42c8ff9640 1 -- 172.21.15.62:0/1150753948 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5558e94ea850 con 0x5558e95abd70 2024-09-15T23:22:53.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.322+0000 7f42c8ff9640 10 client.26383 handle_command_reply: tid=0 2024-09-15T23:22:53.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.324+0000 7f42e96b5740 1 client.26383 shutdown 2024-09-15T23:22:53.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.325+0000 7f42e96b5740 1 -- 172.21.15.62:0/1150753948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 msgr2=0x5558e9587ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:53.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.325+0000 7f42e96b5740 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e9587ae0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f42d001b4f0 tx=0x7f42d009b840 comp rx=0 tx=0).stop 2024-09-15T23:22:53.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.325+0000 7f42e96b5740 1 -- 172.21.15.62:0/1150753948 shutdown_connections 2024-09-15T23:22:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.325+0000 7f42e96b5740 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5558e95abd70 0x5558e95ae1d0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.325+0000 7f42e96b5740 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 0x5558e9588080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.325+0000 7f42e96b5740 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e9587ae0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.325+0000 7f42e96b5740 1 --2- 172.21.15.62:0/1150753948 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 0x5558e958a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.325+0000 7f42e96b5740 1 -- 172.21.15.62:0/1150753948 >> 172.21.15.62:0/1150753948 conn(0x5558e957c240 msgr2=0x5558e9572b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.325+0000 7f42c8ff9640 0 client.26383 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.326+0000 7f42e96b5740 1 -- 172.21.15.62:0/1150753948 shutdown_connections 2024-09-15T23:22:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.326+0000 7f42e96b5740 1 -- 172.21.15.62:0/1150753948 wait complete. 2024-09-15T23:22:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.326+0000 7f42e96b5740 20 client.26383 trim_cache size 0 max 16384 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e96b5740 1 Processor -- start 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e96b5740 1 -- start start 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 0x5558e9587ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e95b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e95b4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e96b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f42e01a6a70 con 0x5558e95805d0 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e96b5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f42e00737d0 con 0x5558e957fcb0 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e96b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f42e00716d0 con 0x5558e958ae90 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e4877640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e95b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e4877640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e95b3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52774/0 (socket says 172.21.15.62:52774) 2024-09-15T23:22:53.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e4877640 1 -- 172.21.15.62:0/2340098524 learned_addr learned my addr 172.21.15.62:0/2340098524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e5078640 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 0x5558e9587ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.357+0000 7f42e5879640 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e95b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42e4877640 1 -- 172.21.15.62:0/2340098524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 msgr2=0x5558e95b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42e4877640 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e95b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42e4877640 1 -- 172.21.15.62:0/2340098524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 msgr2=0x5558e9587ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42e4877640 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 0x5558e9587ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42e4877640 1 -- 172.21.15.62:0/2340098524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e957cdf0 con 0x5558e95805d0 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42e5078640 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 0x5558e9587ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42e4877640 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e95b3ae0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f42d009be00 tx=0x7f42d009b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837d0bf37f45614e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42caffd640 1 -- 172.21.15.62:0/2340098524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d009ab50 con 0x5558e95805d0 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42caffd640 1 -- 172.21.15.62:0/2340098524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42d00860b0 con 0x5558e95805d0 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.358+0000 7f42caffd640 1 -- 172.21.15.62:0/2340098524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d009ba10 con 0x5558e95805d0 2024-09-15T23:22:53.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.359+0000 7f42e96b5740 1 -- 172.21.15.62:0/2340098524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 msgr2=0x5558e95b3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:53.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.359+0000 7f42e96b5740 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e95b3ae0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f42d009be00 tx=0x7f42d009b8b0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.359+0000 7f42e96b5740 1 -- 172.21.15.62:0/2340098524 shutdown_connections 2024-09-15T23:22:53.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.359+0000 7f42e96b5740 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e958ae90 0x5558e95b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.359+0000 7f42e96b5740 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e95805d0 0x5558e95b3ae0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.359+0000 7f42e96b5740 1 --2- 172.21.15.62:0/2340098524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e957fcb0 0x5558e9587ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.360+0000 7f42e96b5740 1 -- 172.21.15.62:0/2340098524 >> 172.21.15.62:0/2340098524 conn(0x5558e957c240 msgr2=0x5558e95873a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:53.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.360+0000 7f42e96b5740 1 -- 172.21.15.62:0/2340098524 shutdown_connections 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.360+0000 7f42e96b5740 1 -- 172.21.15.62:0/2340098524 wait complete. 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.360+0000 7f42e96b5740 1 -- start start 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.361+0000 7f42e96b5740 1 Processor -- start 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.361+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 0x5558e9593060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e95935a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e5078640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 0x5558e9593060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e96b5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 0x5558e95842f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e96b5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f42e0071c90 con 0x5558e957fcb0 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e96b5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f42e0073100 con 0x5558e958ae90 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e96b5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f42e0072360 con 0x5558e95805d0 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e4877640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e95935a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e4877640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e95935a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34554/0 (socket says 172.21.15.62:34554) 2024-09-15T23:22:53.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e4877640 1 -- 172.21.15.62:0/4141047979 learned_addr learned my addr 172.21.15.62:0/4141047979 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e5879640 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 0x5558e95842f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e96b5740 10 client.0 _ll_register_callbacks cb 0x7f42d0009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e4877640 1 -- 172.21.15.62:0/4141047979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 msgr2=0x5558e95842f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e4877640 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 0x5558e95842f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e4877640 1 -- 172.21.15.62:0/4141047979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 msgr2=0x5558e9593060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e4877640 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 0x5558e9593060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e4877640 1 -- 172.21.15.62:0/4141047979 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558e958aba0 con 0x5558e95805d0 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.362+0000 7f42e5078640 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 0x5558e9593060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.363+0000 7f42e4877640 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e95935a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f42d009bbd0 tx=0x7f42d00a4370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e76142e3a77086f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.363+0000 7f42c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.363+0000 7f42c8ff9640 1 -- 172.21.15.62:0/4141047979 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d007f070 con 0x5558e95805d0 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.363+0000 7f42c8ff9640 1 -- 172.21.15.62:0/4141047979 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f42d00ba4a0 con 0x5558e95805d0 2024-09-15T23:22:53.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.363+0000 7f42e96b5740 1 -- 172.21.15.62:0/4141047979 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42d00860b0 con 0x5558e95805d0 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.363+0000 7f42c8ff9640 1 -- 172.21.15.62:0/4141047979 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f42d009ba10 con 0x5558e95805d0 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42e96b5740 10 client.26389 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42e96b5740 1 -- 172.21.15.62:0/4141047979 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5558e9616260 con 0x5558e95805d0 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42c8ff9640 1 -- 172.21.15.62:0/4141047979 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42d00860b0 con 0x5558e95805d0 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42c8ff9640 1 -- 172.21.15.62:0/4141047979 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f42d007a870 con 0x5558e95805d0 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42e96b5740 10 client.26389 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42e96b5740 10 client.26389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42e96b5740 20 client.26389 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42e96b5740 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5558e9616790 0x5558e9618bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42e96b5740 4 client.26389 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.364+0000 7f42e96b5740 1 -- 172.21.15.62:0/4141047979 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5558e94e4d60 con 0x5558e9616790 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.365+0000 7f42e5078640 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5558e9616790 0x5558e9618bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.365+0000 7f42e5078640 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5558e9616790 0x5558e9618bb0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=822ca7217a70c541 server_cookie=c7cc33453f467e60 in_seq=0 out_seq=0 2024-09-15T23:22:53.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.365+0000 7f42c8ff9640 10 client.26389 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:53.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.365+0000 7f42c8ff9640 1 -- 172.21.15.62:0/4141047979 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5558e94e4d60 con 0x5558e9616790 2024-09-15T23:22:53.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.365+0000 7f42c8ff9640 10 client.26389 handle_command_reply: tid=0 2024-09-15T23:22:53.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.365+0000 7f42e96b5740 1 client.26389 shutdown 2024-09-15T23:22:53.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.365+0000 7f42e96b5740 1 -- 172.21.15.62:0/4141047979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 msgr2=0x5558e95935a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:53.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.365+0000 7f42e96b5740 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e95935a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f42d009bbd0 tx=0x7f42d00a4370 comp rx=0 tx=0).stop 2024-09-15T23:22:53.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.366+0000 7f42e96b5740 1 -- 172.21.15.62:0/4141047979 shutdown_connections 2024-09-15T23:22:53.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.366+0000 7f42e96b5740 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5558e9616790 0x5558e9618bb0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.366+0000 7f42e96b5740 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558e958ae90 0x5558e95842f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.366+0000 7f42e96b5740 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5558e95805d0 0x5558e95935a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.366+0000 7f42c8ff9640 0 client.26389 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.366+0000 7f42e96b5740 1 --2- 172.21.15.62:0/4141047979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5558e957fcb0 0x5558e9593060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.366+0000 7f42e96b5740 1 -- 172.21.15.62:0/4141047979 >> 172.21.15.62:0/4141047979 conn(0x5558e957c240 msgr2=0x5558e9589b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:53.371 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:22:53.371 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:22:53.371 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:22:53.371 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.366+0000 7f42e96b5740 1 -- 172.21.15.62:0/4141047979 shutdown_connections 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.368+0000 7f42e96b5740 1 -- 172.21.15.62:0/4141047979 wait complete. 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.368+0000 7f42e96b5740 20 client.26389 trim_cache size 0 max 16384 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42cc0a3340 msgr2=0x7f42cc0a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42cc0a3340 0x7f42cc0a5800 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f42d8009a60 tx=0x7f42d8004a40 comp rx=0 tx=0).stop 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e01036f0 msgr2=0x7f42e0198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e01036f0 0x7f42e0198940 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009f10 tx=0x7f42d000ed10 comp rx=0 tx=0).stop 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 shutdown_connections 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f42cc0a3340 0x7f42cc0a5800 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f42e010c1b0 0x7f42e019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42e01036f0 0x7f42e0198940 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 --2- 172.21.15.62:0/3158037990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f42e0102dd0 0x7f42e0198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 >> 172.21.15.62:0/3158037990 conn(0x7f42e00fc7d0 msgr2=0x7f42e00fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 shutdown_connections 2024-09-15T23:22:53.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:53.369+0000 7f42e72dc640 1 -- 172.21.15.62:0/3158037990 wait complete. 2024-09-15T23:22:54.229 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:22:54.230 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:22:54.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 395.0 s, 946.0 tps, lat 105.032 ms stddev 118.892 2024-09-15T23:22:54.703 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:55.170 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:54 smithi062 ceph-mon[30949]: pgmap v569: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.6 MiB/s wr, 653 op/s 2024-09-15T23:22:55.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:54 smithi067 ceph-mon[35665]: pgmap v569: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.6 MiB/s wr, 653 op/s 2024-09-15T23:22:55.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.286+0000 7f0a9ac11640 1 -- 172.21.15.62:0/449108636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94104990 msgr2=0x7f0a941089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.286+0000 7f0a9ac11640 1 --2- 172.21.15.62:0/449108636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94104990 0x7f0a941089e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0a80009900 tx=0x7f0a8002a410 comp rx=0 tx=0).stop 2024-09-15T23:22:55.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.286+0000 7f0a9ac11640 1 -- 172.21.15.62:0/449108636 shutdown_connections 2024-09-15T23:22:55.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.286+0000 7f0a9ac11640 1 --2- 172.21.15.62:0/449108636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a94109110 0x7f0a94110c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.286+0000 7f0a9ac11640 1 --2- 172.21.15.62:0/449108636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94104990 0x7f0a941089e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.286+0000 7f0a9ac11640 1 --2- 172.21.15.62:0/449108636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a94103fe0 0x7f0a941043c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.286+0000 7f0a9ac11640 1 -- 172.21.15.62:0/449108636 >> 172.21.15.62:0/449108636 conn(0x7f0a940ffa60 msgr2=0x7f0a94101e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:55.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.286+0000 7f0a9ac11640 1 -- 172.21.15.62:0/449108636 shutdown_connections 2024-09-15T23:22:55.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.286+0000 7f0a9ac11640 1 -- 172.21.15.62:0/449108636 wait complete. 2024-09-15T23:22:55.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.287+0000 7f0a9ac11640 1 Processor -- start 2024-09-15T23:22:55.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.287+0000 7f0a9ac11640 1 -- start start 2024-09-15T23:22:55.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:54 smithi148 ceph-mon[35610]: pgmap v569: 129 pgs: 129 active+clean; 1.5 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.6 MiB/s wr, 653 op/s 2024-09-15T23:22:55.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a9ac11640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a94103fe0 0x7f0a9419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a9ac11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a94104990 0x7f0a9419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a9ac11640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94109110 0x7f0a9419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a989ad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a94103fe0 0x7f0a9419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a991ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94109110 0x7f0a9419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a991ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94109110 0x7f0a9419ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49218/0 (socket says 172.21.15.62:49218) 2024-09-15T23:22:55.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a93fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a94104990 0x7f0a9419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a989ad640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a94103fe0 0x7f0a9419a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34568/0 (socket says 172.21.15.62:34568) 2024-09-15T23:22:55.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a991ae640 1 -- 172.21.15.62:0/2730035975 learned_addr learned my addr 172.21.15.62:0/2730035975 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:55.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a9ac11640 1 -- 172.21.15.62:0/2730035975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0a94114130 con 0x7f0a94104990 2024-09-15T23:22:55.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a9ac11640 1 -- 172.21.15.62:0/2730035975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0a94113fb0 con 0x7f0a94109110 2024-09-15T23:22:55.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.288+0000 7f0a9ac11640 1 -- 172.21.15.62:0/2730035975 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0a941142b0 con 0x7f0a94103fe0 2024-09-15T23:22:55.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.290+0000 7f0a93fff640 1 -- 172.21.15.62:0/2730035975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a94103fe0 msgr2=0x7f0a9419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.290+0000 7f0a93fff640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a94103fe0 0x7f0a9419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.290+0000 7f0a93fff640 1 -- 172.21.15.62:0/2730035975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94109110 msgr2=0x7f0a9419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.290+0000 7f0a93fff640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94109110 0x7f0a9419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.290+0000 7f0a93fff640 1 -- 172.21.15.62:0/2730035975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9419f620 con 0x7f0a94104990 2024-09-15T23:22:55.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.290+0000 7f0a991ae640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94109110 0x7f0a9419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:55.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.290+0000 7f0a989ad640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a94103fe0 0x7f0a9419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:22:55.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.291+0000 7f0a93fff640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a94104990 0x7f0a9419abb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0a80005940 tx=0x7f0a800040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:55.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.291+0000 7f0a91ffb640 1 -- 172.21.15.62:0/2730035975 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8003f030 con 0x7f0a94104990 2024-09-15T23:22:55.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.291+0000 7f0a9ac11640 1 -- 172.21.15.62:0/2730035975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9419f8b0 con 0x7f0a94104990 2024-09-15T23:22:55.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.291+0000 7f0a9ac11640 1 -- 172.21.15.62:0/2730035975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a941a71f0 con 0x7f0a94104990 2024-09-15T23:22:55.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.291+0000 7f0a91ffb640 1 -- 172.21.15.62:0/2730035975 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a800044c0 con 0x7f0a94104990 2024-09-15T23:22:55.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.291+0000 7f0a91ffb640 1 -- 172.21.15.62:0/2730035975 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80049060 con 0x7f0a94104990 2024-09-15T23:22:55.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.292+0000 7f0a7f7fe640 1 -- 172.21.15.62:0/2730035975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a74005290 con 0x7f0a94104990 2024-09-15T23:22:55.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.294+0000 7f0a91ffb640 1 -- 172.21.15.62:0/2730035975 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0a80042030 con 0x7f0a94104990 2024-09-15T23:22:55.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.295+0000 7f0a91ffb640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a840a32c0 0x7f0a840a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.295+0000 7f0a91ffb640 1 -- 172.21.15.62:0/2730035975 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0a800f02e0 con 0x7f0a94104990 2024-09-15T23:22:55.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.295+0000 7f0a989ad640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a840a32c0 0x7f0a840a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.295+0000 7f0a989ad640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a840a32c0 0x7f0a840a5780 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009b50 tx=0x7f0a88005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:55.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.299+0000 7f0a91ffb640 1 -- 172.21.15.62:0/2730035975 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0a800bcc20 con 0x7f0a94104990 2024-09-15T23:22:55.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.679+0000 7f0a7f7fe640 1 -- 172.21.15.62:0/2730035975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a74005580 con 0x7f0a94104990 2024-09-15T23:22:55.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.680+0000 7f0a91ffb640 1 -- 172.21.15.62:0/2730035975 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11904 (secure 0 0 0) 0x7f0a800bc520 con 0x7f0a94104990 2024-09-15T23:22:55.682 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:22:55.682 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":32,"btime":"2024-09-15T23:21:52:478522+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:21:52.380746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:22:55.682 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 32 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 -- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a840a32c0 msgr2=0x7f0a840a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a840a32c0 0x7f0a840a5780 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009b50 tx=0x7f0a88005f90 comp rx=0 tx=0).stop 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 -- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a94104990 msgr2=0x7f0a9419abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a94104990 0x7f0a9419abb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0a80005940 tx=0x7f0a800040c0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 -- 172.21.15.62:0/2730035975 shutdown_connections 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0a840a32c0 0x7f0a840a5780 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a94109110 0x7f0a9419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a94104990 0x7f0a9419abb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 --2- 172.21.15.62:0/2730035975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0a94103fe0 0x7f0a9419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 -- 172.21.15.62:0/2730035975 >> 172.21.15.62:0/2730035975 conn(0x7f0a940ffa60 msgr2=0x7f0a94100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 -- 172.21.15.62:0/2730035975 shutdown_connections 2024-09-15T23:22:55.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:55.683+0000 7f0a7f7fe640 1 -- 172.21.15.62:0/2730035975 wait complete. 2024-09-15T23:22:55.923 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.917+0000 7f456d0ed640 1 -- 172.21.15.62:0/3652333575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4568072d20 msgr2=0x7f4568073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.917+0000 7f456d0ed640 1 --2- 172.21.15.62:0/3652333575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4568072d20 0x7f4568073180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4558009ca0 tx=0x7f455802a8d0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.921+0000 7f456d0ed640 1 -- 172.21.15.62:0/3652333575 shutdown_connections 2024-09-15T23:22:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.921+0000 7f456d0ed640 1 --2- 172.21.15.62:0/3652333575 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45680736c0 0x7f4568119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.921+0000 7f456d0ed640 1 --2- 172.21.15.62:0/3652333575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4568072d20 0x7f4568073180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.921+0000 7f456d0ed640 1 --2- 172.21.15.62:0/3652333575 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4568109700 0x7f4568109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.921+0000 7f456d0ed640 1 -- 172.21.15.62:0/3652333575 >> 172.21.15.62:0/3652333575 conn(0x7f456806ece0 msgr2=0x7f4568071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.928+0000 7f456d0ed640 1 -- 172.21.15.62:0/3652333575 shutdown_connections 2024-09-15T23:22:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.933+0000 7f456d0ed640 1 -- 172.21.15.62:0/3652333575 wait complete. 2024-09-15T23:22:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.934+0000 7f456d0ed640 1 Processor -- start 2024-09-15T23:22:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.934+0000 7f456d0ed640 1 -- start start 2024-09-15T23:22:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.934+0000 7f456d0ed640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4568072d20 0x7f45681158a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.934+0000 7f456d0ed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45680736c0 0x7f4568115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.934+0000 7f456d0ed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4568109700 0x7f45681b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.934+0000 7f456d0ed640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f456811c700 con 0x7f45680736c0 2024-09-15T23:22:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.934+0000 7f456d0ed640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f456811c580 con 0x7f4568109700 2024-09-15T23:22:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.934+0000 7f456d0ed640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f456811c880 con 0x7f4568072d20 2024-09-15T23:22:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.935+0000 7f4566d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4568072d20 0x7f45681158a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.935+0000 7f4566d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4568072d20 0x7f45681158a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34592/0 (socket says 172.21.15.62:34592) 2024-09-15T23:22:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.935+0000 7f4566d9d640 1 -- 172.21.15.62:0/2033237840 learned_addr learned my addr 172.21.15.62:0/2033237840 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.935+0000 7f456759e640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4568109700 0x7f45681b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.937+0000 7f4566d9d640 1 -- 172.21.15.62:0/2033237840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4568109700 msgr2=0x7f45681b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.937+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4568109700 0x7f45681b6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.937+0000 7f4566d9d640 1 -- 172.21.15.62:0/2033237840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45680736c0 msgr2=0x7f4568115de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.937+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45680736c0 0x7f4568115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.937+0000 7f4566d9d640 1 -- 172.21.15.62:0/2033237840 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45681164c0 con 0x7f4568072d20 2024-09-15T23:22:55.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.939+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4568072d20 0x7f45681158a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f456000b4a0 tx=0x7f456000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:55.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.939+0000 7f455ffff640 1 -- 172.21.15.62:0/2033237840 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560014070 con 0x7f4568072d20 2024-09-15T23:22:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.940+0000 7f456d0ed640 1 -- 172.21.15.62:0/2033237840 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45681b7470 con 0x7f4568072d20 2024-09-15T23:22:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.940+0000 7f456d0ed640 1 -- 172.21.15.62:0/2033237840 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45681b7a40 con 0x7f4568072d20 2024-09-15T23:22:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.940+0000 7f455ffff640 1 -- 172.21.15.62:0/2033237840 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4560010af0 con 0x7f4568072d20 2024-09-15T23:22:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.940+0000 7f455ffff640 1 -- 172.21.15.62:0/2033237840 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560017d10 con 0x7f4568072d20 2024-09-15T23:22:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.946+0000 7f455ffff640 1 -- 172.21.15.62:0/2033237840 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4560017070 con 0x7f4568072d20 2024-09-15T23:22:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.948+0000 7f456fe89740 1 Processor -- start 2024-09-15T23:22:55.949 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.948+0000 7f456fe89740 1 -- start start 2024-09-15T23:22:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f456fe89740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f456fe89740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f456fe89740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b5370 0x55e1e80b7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f456fe89740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e1e80132f0 con 0x55e1e80b02e0 2024-09-15T23:22:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f456fe89740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e1e7fbb270 con 0x55e1e80b5370 2024-09-15T23:22:55.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f456fe89740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e1e800f2b0 con 0x55e1e80af9a0 2024-09-15T23:22:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f4566d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f4566d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80b4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52838/0 (socket says 172.21.15.62:52838) 2024-09-15T23:22:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f4566d9d640 1 -- 172.21.15.62:0/2346788308 learned_addr learned my addr 172.21.15.62:0/2346788308 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.949+0000 7f456759e640 1 --2- 172.21.15.62:0/2346788308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.950+0000 7f455ffff640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f45540a3530 0x7f45540a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.951 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.950+0000 7f455ffff640 1 -- 172.21.15.62:0/2033237840 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f45600cff90 con 0x7f4568072d20 2024-09-15T23:22:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.950+0000 7f4566d9d640 1 -- 172.21.15.62:0/2346788308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 msgr2=0x55e1e80afda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.950+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2346788308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80afda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.950+0000 7f4566d9d640 1 -- 172.21.15.62:0/2346788308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b5370 msgr2=0x55e1e80b7830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:22:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.950+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2346788308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b5370 0x55e1e80b7830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.950+0000 7f4566d9d640 1 -- 172.21.15.62:0/2346788308 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1e7fba340 con 0x55e1e80b02e0 2024-09-15T23:22:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.951+0000 7f456759e640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f45540a3530 0x7f45540a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.951+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2346788308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80b4da0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f456009bed0 tx=0x7f45600047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee2ef580e14e103 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.952+0000 7f456759e640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f45540a3530 0x7f45540a59f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4550006fd0 tx=0x7f4550008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.952+0000 7f455d7fa640 1 -- 172.21.15.62:0/2346788308 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456001e070 con 0x55e1e80b02e0 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.952+0000 7f455d7fa640 1 -- 172.21.15.62:0/2346788308 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f456009c8d0 con 0x55e1e80b02e0 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.952+0000 7f455d7fa640 1 -- 172.21.15.62:0/2346788308 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456009a4d0 con 0x55e1e80b02e0 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.953+0000 7f456fe89740 1 -- 172.21.15.62:0/2346788308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 msgr2=0x55e1e80b4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.953+0000 7f456fe89740 1 --2- 172.21.15.62:0/2346788308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80b4da0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f456009bed0 tx=0x7f45600047d0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.953+0000 7f456fe89740 1 -- 172.21.15.62:0/2346788308 shutdown_connections 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.953+0000 7f456fe89740 1 --2- 172.21.15.62:0/2346788308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b5370 0x55e1e80b7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.953+0000 7f456fe89740 1 --2- 172.21.15.62:0/2346788308 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80b4da0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.953+0000 7f456fe89740 1 --2- 172.21.15.62:0/2346788308 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80afda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.953+0000 7f456fe89740 1 -- 172.21.15.62:0/2346788308 >> 172.21.15.62:0/2346788308 conn(0x55e1e80ab010 msgr2=0x55e1e80ab410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.954+0000 7f456fe89740 1 -- 172.21.15.62:0/2346788308 shutdown_connections 2024-09-15T23:22:55.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.954+0000 7f456fe89740 1 -- 172.21.15.62:0/2346788308 wait complete. 2024-09-15T23:22:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.954+0000 7f456fe89740 1 -- start start 2024-09-15T23:22:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.955+0000 7f456fe89740 1 Processor -- start 2024-09-15T23:22:55.956 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.955+0000 7f456fe89740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80b9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.955+0000 7f456fe89740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80ba160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f4566d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80b9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f4566d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80b9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34606/0 (socket says 172.21.15.62:34606) 2024-09-15T23:22:55.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f4566d9d640 1 -- 172.21.15.62:0/2220119387 learned_addr learned my addr 172.21.15.62:0/2220119387 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f456fe89740 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b5370 0x55e1e80c11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e1e80b82e0 con 0x55e1e80b02e0 2024-09-15T23:22:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e1e7f9c290 con 0x55e1e80b5370 2024-09-15T23:22:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e1e80b8460 con 0x55e1e80af9a0 2024-09-15T23:22:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f456fe89740 10 client.0 _ll_register_callbacks cb 0x55e1e7fcd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f4566d9d640 1 -- 172.21.15.62:0/2220119387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b5370 msgr2=0x55e1e80c11a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b5370 0x55e1e80c11a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.956+0000 7f4566d9d640 1 -- 172.21.15.62:0/2220119387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 msgr2=0x55e1e80ba160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:22:55.958 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.957+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80ba160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.959 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.957+0000 7f4566d9d640 1 -- 172.21.15.62:0/2220119387 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1e7d26210 con 0x55e1e80af9a0 2024-09-15T23:22:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.957+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80b9c20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4560098630 tx=0x7f45600a0580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22f26c83a79b4fa8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:55.961 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.957+0000 7f454e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:22:55.961 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.958+0000 7f454e7fc640 1 -- 172.21.15.62:0/2220119387 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456001e070 con 0x55e1e80af9a0 2024-09-15T23:22:55.961 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.958+0000 7f454e7fc640 1 -- 172.21.15.62:0/2220119387 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f45600cc780 con 0x55e1e80af9a0 2024-09-15T23:22:55.961 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.958+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f456009c8d0 con 0x55e1e80af9a0 2024-09-15T23:22:55.967 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.964+0000 7f454e7fc640 1 -- 172.21.15.62:0/2220119387 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f456009c1d0 con 0x55e1e80af9a0 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.964+0000 7f456fe89740 10 client.26401 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.964+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e1e7df2620 con 0x55e1e80af9a0 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.964+0000 7f454e7fc640 1 -- 172.21.15.62:0/2220119387 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45600b50d0 con 0x55e1e80af9a0 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.965+0000 7f454e7fc640 1 -- 172.21.15.62:0/2220119387 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f45600bc0b0 con 0x55e1e80af9a0 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.965+0000 7f456fe89740 10 client.26401 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.965+0000 7f456fe89740 10 client.26401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.965+0000 7f456fe89740 20 client.26401 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.965+0000 7f456fe89740 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e80c5480 0x55e1e80c7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.965+0000 7f456fe89740 4 client.26401 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.965+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f456009c8d0 con 0x55e1e80c5480 2024-09-15T23:22:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.965+0000 7f456659c640 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e80c5480 0x55e1e80c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.966+0000 7f456659c640 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e80c5480 0x55e1e80c7870 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e16d9e9c41ed0b1 server_cookie=de9691c264201e88 in_seq=0 out_seq=0 2024-09-15T23:22:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.966+0000 7f454e7fc640 10 client.26401 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.967+0000 7f454e7fc640 1 -- 172.21.15.62:0/2220119387 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f456009c8d0 con 0x55e1e80c5480 2024-09-15T23:22:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.967+0000 7f454e7fc640 10 client.26401 handle_command_reply: tid=0 2024-09-15T23:22:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.967+0000 7f456fe89740 1 client.26401 shutdown 2024-09-15T23:22:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 msgr2=0x55e1e80b9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80b9c20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4560098630 tx=0x7f45600a0580 comp rx=0 tx=0).stop 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 shutdown_connections 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e80c5480 0x55e1e80c7870 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b5370 0x55e1e80c11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80ba160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 --2- 172.21.15.62:0/2220119387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80af9a0 0x55e1e80b9c20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 >> 172.21.15.62:0/2220119387 conn(0x55e1e80ab010 msgr2=0x55e1e80ac820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f454e7fc640 0 client.26401 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 shutdown_connections 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 1 -- 172.21.15.62:0/2220119387 wait complete. 2024-09-15T23:22:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:55.969+0000 7f456fe89740 20 client.26401 trim_cache size 0 max 16384 2024-09-15T23:22:56.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.005+0000 7f456fe89740 1 Processor -- start 2024-09-15T23:22:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.007+0000 7f456fe89740 1 -- start start 2024-09-15T23:22:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.007+0000 7f456fe89740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80af9a0 0x55e1e80b99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.007+0000 7f456fe89740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b02e0 0x55e1e80c1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.007+0000 7f456fe89740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80b9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.007+0000 7f456fe89740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f45480124e0 con 0x55e1e80af9a0 2024-09-15T23:22:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.007+0000 7f456fe89740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f454800fdb0 con 0x55e1e80b02e0 2024-09-15T23:22:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.007+0000 7f456fe89740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f454800dc00 con 0x55e1e80b5370 2024-09-15T23:22:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.008+0000 7f456759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80b9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.008+0000 7f4566d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80af9a0 0x55e1e80b99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.008+0000 7f456759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80b9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34622/0 (socket says 172.21.15.62:34622) 2024-09-15T23:22:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.008+0000 7f456759e640 1 -- 172.21.15.62:0/2607826886 learned_addr learned my addr 172.21.15.62:0/2607826886 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.009+0000 7f456759e640 1 -- 172.21.15.62:0/2607826886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b02e0 msgr2=0x55e1e80c1160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.009+0000 7f456759e640 1 --2- 172.21.15.62:0/2607826886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b02e0 0x55e1e80c1160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.009+0000 7f456759e640 1 -- 172.21.15.62:0/2607826886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80af9a0 msgr2=0x55e1e80b99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.009+0000 7f456759e640 1 --2- 172.21.15.62:0/2607826886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80af9a0 0x55e1e80b99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.009+0000 7f456759e640 1 -- 172.21.15.62:0/2607826886 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1e80b9760 con 0x55e1e80b5370 2024-09-15T23:22:56.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.010+0000 7f456759e640 1 --2- 172.21.15.62:0/2607826886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80b9f10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4550012c50 tx=0x7f4550015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b6af28447055c95 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:56.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.010+0000 7f455cff9640 1 -- 172.21.15.62:0/2607826886 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550012e90 con 0x55e1e80b5370 2024-09-15T23:22:56.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.010+0000 7f455cff9640 1 -- 172.21.15.62:0/2607826886 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4550027070 con 0x55e1e80b5370 2024-09-15T23:22:56.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.012+0000 7f456fe89740 1 -- 172.21.15.62:0/2607826886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 msgr2=0x55e1e80b9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:56.013 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.012+0000 7f456fe89740 1 --2- 172.21.15.62:0/2607826886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80b9f10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4550012c50 tx=0x7f4550015000 comp rx=0 tx=0).stop 2024-09-15T23:22:56.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.014+0000 7f456fe89740 1 -- 172.21.15.62:0/2607826886 shutdown_connections 2024-09-15T23:22:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.014+0000 7f456fe89740 1 --2- 172.21.15.62:0/2607826886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80b9f10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.014+0000 7f456fe89740 1 --2- 172.21.15.62:0/2607826886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80b02e0 0x55e1e80c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.014+0000 7f456fe89740 1 --2- 172.21.15.62:0/2607826886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80af9a0 0x55e1e80b99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.014+0000 7f456fe89740 1 -- 172.21.15.62:0/2607826886 >> 172.21.15.62:0/2607826886 conn(0x55e1e80ab010 msgr2=0x55e1e80b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:56.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.023+0000 7f456fe89740 1 -- 172.21.15.62:0/2607826886 shutdown_connections 2024-09-15T23:22:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.024+0000 7f456fe89740 1 -- 172.21.15.62:0/2607826886 wait complete. 2024-09-15T23:22:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f456fe89740 1 -- start start 2024-09-15T23:22:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f456fe89740 1 Processor -- start 2024-09-15T23:22:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f456fe89740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80af9a0 0x55e1e80be610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f456fe89740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80beb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f456fe89740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80bfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f456fe89740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f454800e270 con 0x55e1e80b02e0 2024-09-15T23:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f456fe89740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f454800f6e0 con 0x55e1e80af9a0 2024-09-15T23:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f456fe89740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f454800e940 con 0x55e1e80b5370 2024-09-15T23:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f456fe89740 10 client.0 _ll_register_callbacks cb 0x7f4550009350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f4566d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80af9a0 0x55e1e80be610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f4566d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80af9a0 0x55e1e80be610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49270/0 (socket says 172.21.15.62:49270) 2024-09-15T23:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.025+0000 7f4566d9d640 1 -- 172.21.15.62:0/2342518875 learned_addr learned my addr 172.21.15.62:0/2342518875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.026+0000 7f456759e640 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80bfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:56.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.027+0000 7f4566d9d640 1 -- 172.21.15.62:0/2342518875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 msgr2=0x55e1e80bfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:56.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.027+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80bfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.027+0000 7f4566d9d640 1 -- 172.21.15.62:0/2342518875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 msgr2=0x55e1e80beb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:22:56.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.027+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80beb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.027+0000 7f4566d9d640 1 -- 172.21.15.62:0/2342518875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1e80c0170 con 0x55e1e80af9a0 2024-09-15T23:22:56.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.028+0000 7f4566d9d640 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80af9a0 0x55e1e80be610 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f45600a00d0 tx=0x7f45600a01c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e8f835725b2d354 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:56.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.028+0000 7f454e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:22:56.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.029+0000 7f454e7fc640 1 -- 172.21.15.62:0/2342518875 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456001e070 con 0x55e1e80af9a0 2024-09-15T23:22:56.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.029+0000 7f454e7fc640 1 -- 172.21.15.62:0/2342518875 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f45600ccaa0 con 0x55e1e80af9a0 2024-09-15T23:22:56.034 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.030+0000 7f456fe89740 1 -- 172.21.15.62:0/2342518875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4550027070 con 0x55e1e80af9a0 2024-09-15T23:22:56.034 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.033+0000 7f454e7fc640 1 -- 172.21.15.62:0/2342518875 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f456009bd10 con 0x55e1e80af9a0 2024-09-15T23:22:56.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.035+0000 7f456fe89740 10 client.26603 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:56.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.035+0000 7f456fe89740 1 -- 172.21.15.62:0/2342518875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e1e80c1480 con 0x55e1e80af9a0 2024-09-15T23:22:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.037+0000 7f454e7fc640 1 -- 172.21.15.62:0/2342518875 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4550027070 con 0x55e1e80af9a0 2024-09-15T23:22:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.037+0000 7f454e7fc640 1 -- 172.21.15.62:0/2342518875 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f4560017360 con 0x55e1e80af9a0 2024-09-15T23:22:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.037+0000 7f456fe89740 10 client.26603 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.037+0000 7f456fe89740 10 client.26603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:22:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.037+0000 7f456fe89740 20 client.26603 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.037+0000 7f456fe89740 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e8130530 0x55e1e8132950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.037+0000 7f456fe89740 4 client.26603 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:22:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.037+0000 7f456fe89740 1 -- 172.21.15.62:0/2342518875 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e1e8020a20 con 0x55e1e8130530 2024-09-15T23:22:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.039+0000 7f456659c640 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e8130530 0x55e1e8132950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.045+0000 7f456659c640 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e8130530 0x55e1e8132950 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e22a42a7007a189 server_cookie=8560714667d9e496 in_seq=0 out_seq=0 2024-09-15T23:22:56.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.046+0000 7f454e7fc640 10 client.26603 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:56.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.069+0000 7f454e7fc640 1 -- 172.21.15.62:0/2342518875 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e1e8020a20 con 0x55e1e8130530 2024-09-15T23:22:56.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.069+0000 7f454e7fc640 10 client.26603 handle_command_reply: tid=0 2024-09-15T23:22:56.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.069+0000 7f456fe89740 1 client.26603 shutdown 2024-09-15T23:22:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 -- 172.21.15.62:0/2342518875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80af9a0 msgr2=0x55e1e80be610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80af9a0 0x55e1e80be610 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f45600a00d0 tx=0x7f45600a01c0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 -- 172.21.15.62:0/2342518875 shutdown_connections 2024-09-15T23:22:56.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55e1e8130530 0x55e1e8132950 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e1e80b5370 0x55e1e80bfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1e80b02e0 0x55e1e80beb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 --2- 172.21.15.62:0/2342518875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1e80af9a0 0x55e1e80be610 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 -- 172.21.15.62:0/2342518875 >> 172.21.15.62:0/2342518875 conn(0x55e1e80ab010 msgr2=0x55e1e80b99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f454e7fc640 0 client.26603 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 -- 172.21.15.62:0/2342518875 shutdown_connections 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 1 -- 172.21.15.62:0/2342518875 wait complete. 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.071+0000 7f456fe89740 20 client.26603 trim_cache size 0 max 16384 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:22:56.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:22:56.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:22:56.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:22:56.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 -- 172.21.15.62:0/2033237840 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f45540a3530 msgr2=0x7f45540a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:56.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f45540a3530 0x7f45540a59f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4550006fd0 tx=0x7f4550008040 comp rx=0 tx=0).stop 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 -- 172.21.15.62:0/2033237840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4568072d20 msgr2=0x7f45681158a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4568072d20 0x7f45681158a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f456000b4a0 tx=0x7f456000b960 comp rx=0 tx=0).stop 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 -- 172.21.15.62:0/2033237840 shutdown_connections 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f45540a3530 0x7f45540a59f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4568109700 0x7f45681b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45680736c0 0x7f4568115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 --2- 172.21.15.62:0/2033237840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4568072d20 0x7f45681158a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 -- 172.21.15.62:0/2033237840 >> 172.21.15.62:0/2033237840 conn(0x7f456806ece0 msgr2=0x7f4568071710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 -- 172.21.15.62:0/2033237840 shutdown_connections 2024-09-15T23:22:56.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:22:56.072+0000 7f456d0ed640 1 -- 172.21.15.62:0/2033237840 wait complete. 2024-09-15T23:22:56.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:55 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2730035975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:56.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:55 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2730035975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:56.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:55 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2730035975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:22:56.824 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:22:57.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:56 smithi062 ceph-mon[30949]: pgmap v570: 129 pgs: 129 active+clean; 1.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.4 MiB/s wr, 562 op/s 2024-09-15T23:22:57.258 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:57.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:56 smithi067 ceph-mon[35665]: pgmap v570: 129 pgs: 129 active+clean; 1.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.4 MiB/s wr, 562 op/s 2024-09-15T23:22:57.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:56 smithi148 ceph-mon[35610]: pgmap v570: 129 pgs: 129 active+clean; 1.5 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.4 MiB/s wr, 562 op/s 2024-09-15T23:22:57.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.852+0000 7fb897057640 1 -- 172.21.15.62:0/2388471196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb890100e20 msgr2=0x7fb890101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.852+0000 7fb897057640 1 --2- 172.21.15.62:0/2388471196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb890100e20 0x7fb890101200 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009900 tx=0x7fb87c02a480 comp rx=0 tx=0).stop 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.853+0000 7fb897057640 1 -- 172.21.15.62:0/2388471196 shutdown_connections 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.853+0000 7fb897057640 1 --2- 172.21.15.62:0/2388471196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb890102ef0 0x7fb89010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.853+0000 7fb897057640 1 --2- 172.21.15.62:0/2388471196 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb890102570 0x7fb8901029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.853+0000 7fb897057640 1 --2- 172.21.15.62:0/2388471196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb890100e20 0x7fb890101200 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.853+0000 7fb897057640 1 -- 172.21.15.62:0/2388471196 >> 172.21.15.62:0/2388471196 conn(0x7fb8900fc7d0 msgr2=0x7fb8900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.853+0000 7fb897057640 1 -- 172.21.15.62:0/2388471196 shutdown_connections 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.853+0000 7fb897057640 1 -- 172.21.15.62:0/2388471196 wait complete. 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.853+0000 7fb897057640 1 Processor -- start 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.853+0000 7fb897057640 1 -- start start 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb897057640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb890100e20 0x7fb890073a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb897057640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb890102570 0x7fb89006dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb897057640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb890102ef0 0x7fb89006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb897057640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb890110fa0 con 0x7fb890102ef0 2024-09-15T23:22:57.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb897057640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb890110e20 con 0x7fb890100e20 2024-09-15T23:22:57.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb897057640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb890111120 con 0x7fb890102570 2024-09-15T23:22:57.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb88ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb890102570 0x7fb89006dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb894df3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb890100e20 0x7fb890073a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb88ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb890102570 0x7fb89006dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34656/0 (socket says 172.21.15.62:34656) 2024-09-15T23:22:57.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb88ffff640 1 -- 172.21.15.62:0/730832475 learned_addr learned my addr 172.21.15.62:0/730832475 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:57.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.854+0000 7fb8955f4640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb890102ef0 0x7fb89006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.855+0000 7fb8955f4640 1 -- 172.21.15.62:0/730832475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb890102570 msgr2=0x7fb89006dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.855+0000 7fb8955f4640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb890102570 0x7fb89006dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.855+0000 7fb8955f4640 1 -- 172.21.15.62:0/730832475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb890100e20 msgr2=0x7fb890073a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.855+0000 7fb8955f4640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb890100e20 0x7fb890073a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.855+0000 7fb8955f4640 1 -- 172.21.15.62:0/730832475 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89006e660 con 0x7fb890102ef0 2024-09-15T23:22:57.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.855+0000 7fb8955f4640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb890102ef0 0x7fb89006e020 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb888009fd0 tx=0x7fb88800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:57.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.856+0000 7fb88dffb640 1 -- 172.21.15.62:0/730832475 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88800ce20 con 0x7fb890102ef0 2024-09-15T23:22:57.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.856+0000 7fb88dffb640 1 -- 172.21.15.62:0/730832475 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb888005020 con 0x7fb890102ef0 2024-09-15T23:22:57.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.857+0000 7fb897057640 1 -- 172.21.15.62:0/730832475 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8901ad490 con 0x7fb890102ef0 2024-09-15T23:22:57.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.857+0000 7fb88dffb640 1 -- 172.21.15.62:0/730832475 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888010720 con 0x7fb890102ef0 2024-09-15T23:22:57.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.857+0000 7fb897057640 1 -- 172.21.15.62:0/730832475 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8901ad830 con 0x7fb890102ef0 2024-09-15T23:22:57.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.860+0000 7fb899430740 1 Processor -- start 2024-09-15T23:22:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.861+0000 7fb88dffb640 1 -- 172.21.15.62:0/730832475 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb888005540 con 0x7fb890102ef0 2024-09-15T23:22:57.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.861+0000 7fb899430740 1 -- start start 2024-09-15T23:22:57.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.861+0000 7fb899430740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e519b40 0x55a27e518090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.861+0000 7fb899430740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e5185d0 0x55a27e518a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.861+0000 7fb899430740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 0x55a27e525760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.861+0000 7fb899430740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a27e47e2f0 con 0x55a27e523370 2024-09-15T23:22:57.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.861+0000 7fb899430740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a27e425cf0 con 0x55a27e519b40 2024-09-15T23:22:57.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.861+0000 7fb899430740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a27e47a2b0 con 0x55a27e5185d0 2024-09-15T23:22:57.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.862+0000 7fb88ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e519b40 0x55a27e518090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.862+0000 7fb894df3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e5185d0 0x55a27e518a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.862+0000 7fb894df3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e5185d0 0x55a27e518a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34666/0 (socket says 172.21.15.62:34666) 2024-09-15T23:22:57.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.862+0000 7fb894df3640 1 -- 172.21.15.62:0/789718550 learned_addr learned my addr 172.21.15.62:0/789718550 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:57.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.862+0000 7fb88ffff640 1 --2- 172.21.15.62:0/789718550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 0x55a27e525760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.863+0000 7fb894df3640 1 -- 172.21.15.62:0/789718550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e519b40 msgr2=0x55a27e518090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:57.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.863+0000 7fb894df3640 1 --2- 172.21.15.62:0/789718550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e519b40 0x55a27e518090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.863+0000 7fb894df3640 1 -- 172.21.15.62:0/789718550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 msgr2=0x55a27e525760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:57.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.863+0000 7fb894df3640 1 --2- 172.21.15.62:0/789718550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 0x55a27e525760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.863+0000 7fb894df3640 1 -- 172.21.15.62:0/789718550 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27e40a3a0 con 0x55a27e5185d0 2024-09-15T23:22:57.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.863+0000 7fb88ffff640 1 --2- 172.21.15.62:0/789718550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e519b40 0x55a27e518090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:22:57.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.863+0000 7fb894df3640 1 --2- 172.21.15.62:0/789718550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e5185d0 0x55a27e518a10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009230 tx=0x7fb87c005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30b6fc92b824af1d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:57.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.863+0000 7fb88dffb640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8800a32c0 0x7fb8800a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.864+0000 7fb88dffb640 1 -- 172.21.15.62:0/730832475 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb8880d0250 con 0x7fb890102ef0 2024-09-15T23:22:57.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.864+0000 7fb88ffff640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8800a32c0 0x7fb8800a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.864+0000 7fb87affd640 1 -- 172.21.15.62:0/789718550 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c035040 con 0x55a27e5185d0 2024-09-15T23:22:57.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.864+0000 7fb87affd640 1 -- 172.21.15.62:0/789718550 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb87c047070 con 0x55a27e5185d0 2024-09-15T23:22:57.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.864+0000 7fb87affd640 1 -- 172.21.15.62:0/789718550 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c004a30 con 0x55a27e5185d0 2024-09-15T23:22:57.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.866+0000 7fb899430740 1 -- 172.21.15.62:0/789718550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e5185d0 msgr2=0x55a27e518a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.866+0000 7fb899430740 1 --2- 172.21.15.62:0/789718550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e5185d0 0x55a27e518a10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb87c009230 tx=0x7fb87c005e00 comp rx=0 tx=0).stop 2024-09-15T23:22:57.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.866+0000 7fb88ffff640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8800a32c0 0x7fb8800a5780 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb884009fd0 tx=0x7fb884013040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:57.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.867+0000 7fb899430740 1 -- 172.21.15.62:0/789718550 shutdown_connections 2024-09-15T23:22:57.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.867+0000 7fb899430740 1 --2- 172.21.15.62:0/789718550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 0x55a27e525760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.867+0000 7fb899430740 1 --2- 172.21.15.62:0/789718550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e5185d0 0x55a27e518a10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.867+0000 7fb899430740 1 --2- 172.21.15.62:0/789718550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e519b40 0x55a27e518090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.867+0000 7fb899430740 1 -- 172.21.15.62:0/789718550 >> 172.21.15.62:0/789718550 conn(0x55a27e4e1310 msgr2=0x55a27e514c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:57.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.867+0000 7fb899430740 1 -- 172.21.15.62:0/789718550 shutdown_connections 2024-09-15T23:22:57.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.867+0000 7fb899430740 1 -- 172.21.15.62:0/789718550 wait complete. 2024-09-15T23:22:57.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.868+0000 7fb899430740 1 -- start start 2024-09-15T23:22:57.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.868+0000 7fb899430740 1 Processor -- start 2024-09-15T23:22:57.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb899430740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 0x55a27e522db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb899430740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e520060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb899430740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 0x55a27e5207f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb894df3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 0x55a27e522db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb899430740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a27e526010 con 0x55a27e523370 2024-09-15T23:22:57.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb88ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e520060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb899430740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a27e439060 con 0x55a27e5185d0 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb899430740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a27e526190 con 0x55a27e519b40 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb894df3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 0x55a27e522db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49304/0 (socket says 172.21.15.62:49304) 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb894df3640 1 -- 172.21.15.62:0/324505306 learned_addr learned my addr 172.21.15.62:0/324505306 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb8955f4640 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 0x55a27e5207f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.869+0000 7fb899430740 10 client.0 _ll_register_callbacks cb 0x55a27e18c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.870+0000 7fb8955f4640 1 -- 172.21.15.62:0/324505306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 msgr2=0x55a27e520060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.870+0000 7fb8955f4640 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e520060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.870+0000 7fb8955f4640 1 -- 172.21.15.62:0/324505306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 msgr2=0x55a27e522db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:57.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.870+0000 7fb8955f4640 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 0x55a27e522db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.870+0000 7fb8955f4640 1 -- 172.21.15.62:0/324505306 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27e1ef3c0 con 0x55a27e523370 2024-09-15T23:22:57.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.870+0000 7fb8955f4640 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 0x55a27e5207f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb88800c760 tx=0x7fb88809ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20081f676aa1ca7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:57.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.870+0000 7fb86ffff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:22:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.871+0000 7fb86ffff640 1 -- 172.21.15.62:0/324505306 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880a5160 con 0x55a27e523370 2024-09-15T23:22:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.871+0000 7fb899430740 1 -- 172.21.15.62:0/324505306 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb87c047070 con 0x55a27e523370 2024-09-15T23:22:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.871+0000 7fb86ffff640 1 -- 172.21.15.62:0/324505306 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb88809c750 con 0x55a27e523370 2024-09-15T23:22:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.871+0000 7fb86ffff640 1 -- 172.21.15.62:0/324505306 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb88809c950 con 0x55a27e523370 2024-09-15T23:22:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.871+0000 7fb899430740 10 client.17034 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.871+0000 7fb899430740 1 -- 172.21.15.62:0/324505306 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a27e531b70 con 0x55a27e523370 2024-09-15T23:22:57.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.871+0000 7fb86ffff640 1 -- 172.21.15.62:0/324505306 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8880b5260 con 0x55a27e523370 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.872+0000 7fb86ffff640 1 -- 172.21.15.62:0/324505306 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7fb88809b940 con 0x55a27e523370 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.872+0000 7fb899430740 10 client.17034 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.872+0000 7fb899430740 10 client.17034 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.872+0000 7fb899430740 20 client.17034 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.872+0000 7fb899430740 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a27e533030 0x55a27e535420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.872+0000 7fb899430740 4 client.17034 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.872+0000 7fb899430740 1 -- 172.21.15.62:0/324505306 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb87c047070 con 0x55a27e533030 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.872+0000 7fb894df3640 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a27e533030 0x55a27e535420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.873+0000 7fb894df3640 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a27e533030 0x55a27e535420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fa2a914c18abf85c server_cookie=607984b312099ff2 in_seq=0 out_seq=0 2024-09-15T23:22:57.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.873+0000 7fb86ffff640 10 client.17034 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:22:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.873+0000 7fb86ffff640 1 -- 172.21.15.62:0/324505306 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fb87c047070 con 0x55a27e533030 2024-09-15T23:22:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.873+0000 7fb86ffff640 10 client.17034 handle_command_reply: tid=0 2024-09-15T23:22:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb899430740 1 client.17034 shutdown 2024-09-15T23:22:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb899430740 1 -- 172.21.15.62:0/324505306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 msgr2=0x55a27e5207f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb899430740 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 0x55a27e5207f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb88800c760 tx=0x7fb88809ccb0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb899430740 1 -- 172.21.15.62:0/324505306 shutdown_connections 2024-09-15T23:22:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb899430740 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a27e533030 0x55a27e535420 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb899430740 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e523370 0x55a27e5207f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb899430740 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e520060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb899430740 1 --2- 172.21.15.62:0/324505306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 0x55a27e522db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb899430740 1 -- 172.21.15.62:0/324505306 >> 172.21.15.62:0/324505306 conn(0x55a27e4e1310 msgr2=0x55a27e51a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.874+0000 7fb86ffff640 0 client.17034 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:22:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.875+0000 7fb899430740 1 -- 172.21.15.62:0/324505306 shutdown_connections 2024-09-15T23:22:57.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.875+0000 7fb899430740 1 -- 172.21.15.62:0/324505306 wait complete. 2024-09-15T23:22:57.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.875+0000 7fb899430740 20 client.17034 trim_cache size 0 max 16384 2024-09-15T23:22:57.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.917+0000 7fb899430740 1 Processor -- start 2024-09-15T23:22:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.920+0000 7fb899430740 1 -- start start 2024-09-15T23:22:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.921+0000 7fb899430740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e5185d0 0x55a27e514d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.921+0000 7fb899430740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e513bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.921+0000 7fb899430740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e523370 0x55a27e51b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.921+0000 7fb899430740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901b7c00 con 0x55a27e5185d0 2024-09-15T23:22:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.921+0000 7fb899430740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901b54d0 con 0x55a27e523370 2024-09-15T23:22:57.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.921+0000 7fb899430740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901b3320 con 0x55a27e519b40 2024-09-15T23:22:57.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.923+0000 7fb88ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e513bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.923+0000 7fb88ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e513bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34674/0 (socket says 172.21.15.62:34674) 2024-09-15T23:22:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.923+0000 7fb88ffff640 1 -- 172.21.15.62:0/1060169620 learned_addr learned my addr 172.21.15.62:0/1060169620 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.923+0000 7fb8955f4640 1 --2- 172.21.15.62:0/1060169620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e523370 0x55a27e51b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.926+0000 7fb894df3640 1 --2- 172.21.15.62:0/1060169620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e5185d0 0x55a27e514d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.926+0000 7fb88ffff640 1 -- 172.21.15.62:0/1060169620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e523370 msgr2=0x55a27e51b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.926+0000 7fb88ffff640 1 --2- 172.21.15.62:0/1060169620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e523370 0x55a27e51b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.926+0000 7fb88ffff640 1 -- 172.21.15.62:0/1060169620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e5185d0 msgr2=0x55a27e514d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.926+0000 7fb88ffff640 1 --2- 172.21.15.62:0/1060169620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e5185d0 0x55a27e514d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.926+0000 7fb88ffff640 1 -- 172.21.15.62:0/1060169620 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a27e515960 con 0x55a27e519b40 2024-09-15T23:22:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.926+0000 7fb88ffff640 1 --2- 172.21.15.62:0/1060169620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e513bf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb89006f2a0 tx=0x7fb8840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96deea882e760b42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.934+0000 7fb87a7fc640 1 -- 172.21.15.62:0/1060169620 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88402f070 con 0x55a27e519b40 2024-09-15T23:22:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.934+0000 7fb87a7fc640 1 -- 172.21.15.62:0/1060169620 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb884020aa0 con 0x55a27e519b40 2024-09-15T23:22:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.934+0000 7fb87a7fc640 1 -- 172.21.15.62:0/1060169620 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88402aac0 con 0x55a27e519b40 2024-09-15T23:22:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.935+0000 7fb899430740 1 -- 172.21.15.62:0/1060169620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 msgr2=0x55a27e513bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.935+0000 7fb899430740 1 --2- 172.21.15.62:0/1060169620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e513bf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb89006f2a0 tx=0x7fb8840046c0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.939+0000 7fb899430740 1 -- 172.21.15.62:0/1060169620 shutdown_connections 2024-09-15T23:22:57.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.939+0000 7fb899430740 1 --2- 172.21.15.62:0/1060169620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e523370 0x55a27e51b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.939+0000 7fb899430740 1 --2- 172.21.15.62:0/1060169620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e519b40 0x55a27e513bf0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.939+0000 7fb899430740 1 --2- 172.21.15.62:0/1060169620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e5185d0 0x55a27e514d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.939+0000 7fb899430740 1 -- 172.21.15.62:0/1060169620 >> 172.21.15.62:0/1060169620 conn(0x55a27e4e1310 msgr2=0x55a27e5134e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:57.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.940+0000 7fb899430740 1 -- 172.21.15.62:0/1060169620 shutdown_connections 2024-09-15T23:22:57.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.940+0000 7fb899430740 1 -- 172.21.15.62:0/1060169620 wait complete. 2024-09-15T23:22:57.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.940+0000 7fb899430740 1 -- start start 2024-09-15T23:22:57.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.941+0000 7fb899430740 1 Processor -- start 2024-09-15T23:22:57.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.942+0000 7fb899430740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 0x55a27e52b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.942+0000 7fb899430740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e519b40 0x55a27e52bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.942+0000 7fb899430740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e523370 0x55a27e528420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.942+0000 7fb899430740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901b3990 con 0x55a27e519b40 2024-09-15T23:22:57.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.943+0000 7fb8955f4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e523370 0x55a27e528420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.943+0000 7fb8955f4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e523370 0x55a27e528420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34686/0 (socket says 172.21.15.62:34686) 2024-09-15T23:22:57.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.943+0000 7fb8955f4640 1 -- 172.21.15.62:0/1697103290 learned_addr learned my addr 172.21.15.62:0/1697103290 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:57.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.943+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901b4e00 con 0x55a27e5185d0 2024-09-15T23:22:57.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.943+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb8901b4060 con 0x55a27e523370 2024-09-15T23:22:57.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.943+0000 7fb899430740 10 client.0 _ll_register_callbacks cb 0x7fb888005830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:57.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.944+0000 7fb8955f4640 1 -- 172.21.15.62:0/1697103290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 msgr2=0x55a27e52b630 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:57.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.944+0000 7fb8955f4640 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 0x55a27e52b630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.944+0000 7fb8955f4640 1 -- 172.21.15.62:0/1697103290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e519b40 msgr2=0x55a27e52bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:57.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.944+0000 7fb8955f4640 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e519b40 0x55a27e52bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.945+0000 7fb8955f4640 1 -- 172.21.15.62:0/1697103290 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a27e515dd0 con 0x55a27e523370 2024-09-15T23:22:57.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.947+0000 7fb8955f4640 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e523370 0x55a27e528420 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb8880b7040 tx=0x7fb8880d0160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baf612dcf6cb57d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:57.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.947+0000 7fb86ffff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:22:57.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.948+0000 7fb86ffff640 1 -- 172.21.15.62:0/1697103290 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88807d030 con 0x55a27e523370 2024-09-15T23:22:57.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.949+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb884020aa0 con 0x55a27e523370 2024-09-15T23:22:57.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.949+0000 7fb86ffff640 1 -- 172.21.15.62:0/1697103290 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb88809c750 con 0x55a27e523370 2024-09-15T23:22:57.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.953+0000 7fb86ffff640 1 -- 172.21.15.62:0/1697103290 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb8880a5160 con 0x55a27e523370 2024-09-15T23:22:57.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.954+0000 7fb899430740 10 client.26425 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:57.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.954+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a27e59e530 con 0x55a27e523370 2024-09-15T23:22:57.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.960+0000 7fb86ffff640 1 -- 172.21.15.62:0/1697103290 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb884020aa0 con 0x55a27e523370 2024-09-15T23:22:57.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.960+0000 7fb86ffff640 1 -- 172.21.15.62:0/1697103290 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7fb8880b5260 con 0x55a27e523370 2024-09-15T23:22:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.965+0000 7fb899430740 10 client.26425 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.965+0000 7fb899430740 10 client.26425 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:22:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.965+0000 7fb899430740 20 client.26425 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.965+0000 7fb899430740 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a27e59ec80 0x55a27e5a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.965+0000 7fb899430740 4 client.26425 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:22:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.965+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a27e48b860 con 0x55a27e59ec80 2024-09-15T23:22:57.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.965+0000 7fb894df3640 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a27e59ec80 0x55a27e5a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:57.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.969+0000 7fb894df3640 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a27e59ec80 0x55a27e5a10e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=865d2e298590e9dd server_cookie=e2cf835dc0388c50 in_seq=0 out_seq=0 2024-09-15T23:22:57.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.969+0000 7fb86ffff640 10 client.26425 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:22:57.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.970+0000 7fb86ffff640 1 -- 172.21.15.62:0/1697103290 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a27e48b860 con 0x55a27e59ec80 2024-09-15T23:22:57.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.970+0000 7fb86ffff640 10 client.26425 handle_command_reply: tid=0 2024-09-15T23:22:57.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.971+0000 7fb899430740 1 client.26425 shutdown 2024-09-15T23:22:57.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.976+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e523370 msgr2=0x55a27e528420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.976+0000 7fb899430740 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e523370 0x55a27e528420 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb8880b7040 tx=0x7fb8880d0160 comp rx=0 tx=0).stop 2024-09-15T23:22:57.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.977+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 shutdown_connections 2024-09-15T23:22:57.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.977+0000 7fb899430740 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55a27e59ec80 0x55a27e5a10e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.977+0000 7fb899430740 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a27e523370 0x55a27e528420 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.977+0000 7fb899430740 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a27e519b40 0x55a27e52bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.977+0000 7fb899430740 1 --2- 172.21.15.62:0/1697103290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a27e5185d0 0x55a27e52b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.977+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 >> 172.21.15.62:0/1697103290 conn(0x55a27e4e1310 msgr2=0x55a27e513d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:57.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.978+0000 7fb86ffff640 0 client.26425 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:22:57.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.984+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 shutdown_connections 2024-09-15T23:22:57.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.984+0000 7fb899430740 1 -- 172.21.15.62:0/1697103290 wait complete. 2024-09-15T23:22:57.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.984+0000 7fb899430740 20 client.26425 trim_cache size 0 max 16384 2024-09-15T23:22:57.986 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:22:57.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 -- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8800a32c0 msgr2=0x7fb8800a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8800a32c0 0x7fb8800a5780 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb884009fd0 tx=0x7fb884013040 comp rx=0 tx=0).stop 2024-09-15T23:22:57.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 -- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb890102ef0 msgr2=0x7fb89006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:57.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb890102ef0 0x7fb89006e020 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fb888009fd0 tx=0x7fb88800ef10 comp rx=0 tx=0).stop 2024-09-15T23:22:57.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 -- 172.21.15.62:0/730832475 shutdown_connections 2024-09-15T23:22:57.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8800a32c0 0x7fb8800a5780 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb890102ef0 0x7fb89006e020 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb890102570 0x7fb89006dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 --2- 172.21.15.62:0/730832475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb890100e20 0x7fb890073a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:57.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 -- 172.21.15.62:0/730832475 >> 172.21.15.62:0/730832475 conn(0x7fb8900fc7d0 msgr2=0x7fb890103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:57.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 -- 172.21.15.62:0/730832475 shutdown_connections 2024-09-15T23:22:57.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:57.985+0000 7fb897057640 1 -- 172.21.15.62:0/730832475 wait complete. 2024-09-15T23:22:58.725 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:22:59.156 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:22:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:22:58 smithi067 ceph-mon[35665]: pgmap v571: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.9 MiB/s wr, 581 op/s 2024-09-15T23:22:59.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:22:58 smithi148 ceph-mon[35610]: pgmap v571: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.9 MiB/s wr, 581 op/s 2024-09-15T23:22:59.338 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:22:58 smithi062 ceph-mon[30949]: pgmap v571: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.9 MiB/s wr, 581 op/s 2024-09-15T23:22:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 400.0 s, 1170.8 tps, lat 85.910 ms stddev 98.495 2024-09-15T23:22:59.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.785+0000 7f41d691a640 1 -- 172.21.15.62:0/2354204856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41d0109120 msgr2=0x7f41d0110ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.785+0000 7f41d691a640 1 --2- 172.21.15.62:0/2354204856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41d0109120 0x7f41d0110ca0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009be0 tx=0x7f41c802a6c0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.786+0000 7f41d691a640 1 -- 172.21.15.62:0/2354204856 shutdown_connections 2024-09-15T23:22:59.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.786+0000 7f41d691a640 1 --2- 172.21.15.62:0/2354204856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41d0109120 0x7f41d0110ca0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.786+0000 7f41d691a640 1 --2- 172.21.15.62:0/2354204856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41d01049a0 0x7f41d01089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.786+0000 7f41d691a640 1 --2- 172.21.15.62:0/2354204856 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d0103ff0 0x7f41d01043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.786+0000 7f41d691a640 1 -- 172.21.15.62:0/2354204856 >> 172.21.15.62:0/2354204856 conn(0x7f41d00ffab0 msgr2=0x7f41d0101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:59.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.786+0000 7f41d691a640 1 -- 172.21.15.62:0/2354204856 shutdown_connections 2024-09-15T23:22:59.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.786+0000 7f41d691a640 1 -- 172.21.15.62:0/2354204856 wait complete. 2024-09-15T23:22:59.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.787+0000 7f41d691a640 1 Processor -- start 2024-09-15T23:22:59.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.787+0000 7f41d691a640 1 -- start start 2024-09-15T23:22:59.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41d691a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41d0103ff0 0x7f41d019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41d691a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d01049a0 0x7f41d019abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41d691a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41d0109120 0x7f41d019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41d691a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f41d0114130 con 0x7f41d0109120 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41d691a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f41d0113fb0 con 0x7f41d0103ff0 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41d691a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f41d01142b0 con 0x7f41d01049a0 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41d4eb7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41d0109120 0x7f41d019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41d4eb7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41d0109120 0x7f41d019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52930/0 (socket says 172.21.15.62:52930) 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41d4eb7640 1 -- 172.21.15.62:0/2999845622 learned_addr learned my addr 172.21.15.62:0/2999845622 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41cffff640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41d0103ff0 0x7f41d019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.788+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d01049a0 0x7f41d019abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41cf7fe640 1 -- 172.21.15.62:0/2999845622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41d0103ff0 msgr2=0x7f41d019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41d0103ff0 0x7f41d019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41cf7fe640 1 -- 172.21.15.62:0/2999845622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41d0109120 msgr2=0x7f41d019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41d0109120 0x7f41d019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41cf7fe640 1 -- 172.21.15.62:0/2999845622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d019f660 con 0x7f41d01049a0 2024-09-15T23:22:59.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d01049a0 0x7f41d019abf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f41c400e9f0 tx=0x7f41c400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:59.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41cd7fa640 1 -- 172.21.15.62:0/2999845622 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c400ccc0 con 0x7f41d01049a0 2024-09-15T23:22:59.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41cd7fa640 1 -- 172.21.15.62:0/2999845622 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c4010910 con 0x7f41d01049a0 2024-09-15T23:22:59.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41cd7fa640 1 -- 172.21.15.62:0/2999845622 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4018980 con 0x7f41d01049a0 2024-09-15T23:22:59.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.789+0000 7f41d691a640 1 -- 172.21.15.62:0/2999845622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41d019f950 con 0x7f41d01049a0 2024-09-15T23:22:59.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.790+0000 7f41d691a640 1 -- 172.21.15.62:0/2999845622 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41d01a7230 con 0x7f41d01049a0 2024-09-15T23:22:59.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.793+0000 7f41cd7fa640 1 -- 172.21.15.62:0/2999845622 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f41c4010ac0 con 0x7f41d01049a0 2024-09-15T23:22:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.794+0000 7f41d8cf3740 1 Processor -- start 2024-09-15T23:22:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.794+0000 7f41d8cf3740 1 -- start start 2024-09-15T23:22:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.794+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f4e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.794+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 0x561b85f4ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 0x561b85f63f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561b85eb42f0 con 0x561b85f61b80 2024-09-15T23:22:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 0x561b85f4ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41cffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 0x561b85f4ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49370/0 (socket says 172.21.15.62:49370) 2024-09-15T23:22:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41cffff640 1 -- 172.21.15.62:0/2537908146 learned_addr learned my addr 172.21.15.62:0/2537908146 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41cd7fa640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41b80a32c0 0x7f41b80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41d4eb7640 1 --2- 172.21.15.62:0/2537908146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f4e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41d8cf3740 1 -- 172.21.15.62:0/2537908146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561b85e5bcf0 con 0x561b85f4e5d0 2024-09-15T23:22:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41d8cf3740 1 -- 172.21.15.62:0/2537908146 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561b85eb02b0 con 0x561b85f4fb40 2024-09-15T23:22:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41cd7fa640 1 -- 172.21.15.62:0/2999845622 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f41c40d00c0 con 0x7f41d01049a0 2024-09-15T23:22:59.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/2537908146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 0x561b85f63f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41d4eb7640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41b80a32c0 0x7f41b80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41d4eb7640 1 -- 172.21.15.62:0/2537908146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 msgr2=0x561b85f4ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.795+0000 7f41d4eb7640 1 --2- 172.21.15.62:0/2537908146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 0x561b85f4ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.796+0000 7f41d4eb7640 1 -- 172.21.15.62:0/2537908146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 msgr2=0x561b85f63f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.796+0000 7f41d4eb7640 1 --2- 172.21.15.62:0/2537908146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 0x561b85f63f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.796+0000 7f41d4eb7640 1 -- 172.21.15.62:0/2537908146 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b85e403a0 con 0x561b85f4fb40 2024-09-15T23:22:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.798+0000 7f41d4eb7640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41b80a32c0 0x7f41b80a5780 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f41c8046000 tx=0x7f41c8048000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:59.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.807+0000 7f41d4eb7640 1 --2- 172.21.15.62:0/2537908146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f4e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41c8004af0 tx=0x7f41c80303a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57bcbddcf1b8597a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:59.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.811+0000 7f41b67fc640 1 -- 172.21.15.62:0/2537908146 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c804f070 con 0x561b85f4fb40 2024-09-15T23:22:59.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.811+0000 7f41b67fc640 1 -- 172.21.15.62:0/2537908146 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c8030710 con 0x561b85f4fb40 2024-09-15T23:22:59.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.812+0000 7f41d8cf3740 1 -- 172.21.15.62:0/2537908146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 msgr2=0x561b85f4e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.812+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/2537908146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f4e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f41c8004af0 tx=0x7f41c80303a0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.816+0000 7f41d8cf3740 1 -- 172.21.15.62:0/2537908146 shutdown_connections 2024-09-15T23:22:59.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.816+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/2537908146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 0x561b85f63f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.816+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/2537908146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 0x561b85f4ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.816+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/2537908146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f4e090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.816+0000 7f41d8cf3740 1 -- 172.21.15.62:0/2537908146 >> 172.21.15.62:0/2537908146 conn(0x561b85f17310 msgr2=0x561b85f4ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:59.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.823+0000 7f41d8cf3740 1 -- 172.21.15.62:0/2537908146 shutdown_connections 2024-09-15T23:22:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.825+0000 7f41d8cf3740 1 -- 172.21.15.62:0/2537908146 wait complete. 2024-09-15T23:22:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.825+0000 7f41d8cf3740 1 -- start start 2024-09-15T23:22:59.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.826+0000 7f41d8cf3740 1 Processor -- start 2024-09-15T23:22:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.826+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f61450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.826+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f5e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.827+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f61b80 0x561b85f5ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.827+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561b85f64870 con 0x561b85f4e5d0 2024-09-15T23:22:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.827+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561b85e6f060 con 0x561b85f61b80 2024-09-15T23:22:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.827+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561b85f649f0 con 0x561b85f4fb40 2024-09-15T23:22:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.827+0000 7f41d8cf3740 10 client.0 _ll_register_callbacks cb 0x561b85bc28c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:59.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.828+0000 7f41cffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f61450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.828+0000 7f41cffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f61450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52954/0 (socket says 172.21.15.62:52954) 2024-09-15T23:22:59.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.828+0000 7f41cffff640 1 -- 172.21.15.62:0/55082087 learned_addr learned my addr 172.21.15.62:0/55082087 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.829+0000 7f41cffff640 1 -- 172.21.15.62:0/55082087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 msgr2=0x561b85f5e750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.829+0000 7f41cffff640 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f5e750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.829+0000 7f41cffff640 1 -- 172.21.15.62:0/55082087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f61b80 msgr2=0x561b85f5ec90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:22:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.829+0000 7f41cffff640 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f61b80 0x561b85f5ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.829+0000 7f41cffff640 1 -- 172.21.15.62:0/55082087 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b85c253c0 con 0x561b85f4e5d0 2024-09-15T23:22:59.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.829+0000 7f41cffff640 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f61450 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f41c0013710 tx=0x7f41c0013bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed49178afe56671 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:59.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.829+0000 7f41af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:22:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.830+0000 7f41af7fe640 1 -- 172.21.15.62:0/55082087 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c0004440 con 0x561b85f4e5d0 2024-09-15T23:22:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.830+0000 7f41d8cf3740 1 -- 172.21.15.62:0/55082087 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41c8030710 con 0x561b85f4e5d0 2024-09-15T23:22:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.830+0000 7f41af7fe640 1 -- 172.21.15.62:0/55082087 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f41c00119c0 con 0x561b85f4e5d0 2024-09-15T23:22:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.830+0000 7f41af7fe640 1 -- 172.21.15.62:0/55082087 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f41c0011c00 con 0x561b85f4e5d0 2024-09-15T23:22:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.830+0000 7f41d8cf3740 10 client.17046 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.830+0000 7f41d8cf3740 1 -- 172.21.15.62:0/55082087 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561b85f70330 con 0x561b85f4e5d0 2024-09-15T23:22:59.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.831+0000 7f41af7fe640 1 -- 172.21.15.62:0/55082087 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c8030710 con 0x561b85f4e5d0 2024-09-15T23:22:59.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.832+0000 7f41af7fe640 1 -- 172.21.15.62:0/55082087 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f41c001a070 con 0x561b85f4e5d0 2024-09-15T23:22:59.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.832+0000 7f41d8cf3740 10 client.17046 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:59.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.832+0000 7f41d8cf3740 10 client.17046 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:22:59.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.832+0000 7f41d8cf3740 20 client.17046 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:59.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.832+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x561b85f717f0 0x561b85f73c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.832+0000 7f41d8cf3740 4 client.17046 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:22:59.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.832+0000 7f41d8cf3740 1 -- 172.21.15.62:0/55082087 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b85ec1860 con 0x561b85f717f0 2024-09-15T23:22:59.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.833+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x561b85f717f0 0x561b85f73c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.834+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x561b85f717f0 0x561b85f73c60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ebec9614434e4f0 server_cookie=4bac0d636a971d7b in_seq=0 out_seq=0 2024-09-15T23:22:59.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.834+0000 7f41af7fe640 10 client.17046 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:22:59.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.835+0000 7f41af7fe640 1 -- 172.21.15.62:0/55082087 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561b85ec1860 con 0x561b85f717f0 2024-09-15T23:22:59.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.835+0000 7f41af7fe640 10 client.17046 handle_command_reply: tid=0 2024-09-15T23:22:59.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.835+0000 7f41d8cf3740 1 client.17046 shutdown 2024-09-15T23:22:59.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.842+0000 7f41d8cf3740 1 -- 172.21.15.62:0/55082087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 msgr2=0x561b85f61450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.842+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f61450 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f41c0013710 tx=0x7f41c0013bd0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.852+0000 7f41d8cf3740 1 -- 172.21.15.62:0/55082087 shutdown_connections 2024-09-15T23:22:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.852+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x561b85f717f0 0x561b85f73c60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.852+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f61b80 0x561b85f5ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.852+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f5e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.852+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/55082087 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f61450 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.852+0000 7f41af7fe640 0 client.17046 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:22:59.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.852+0000 7f41d8cf3740 1 -- 172.21.15.62:0/55082087 >> 172.21.15.62:0/55082087 conn(0x561b85f17310 msgr2=0x561b85f50960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:59.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.854+0000 7f41d8cf3740 1 -- 172.21.15.62:0/55082087 shutdown_connections 2024-09-15T23:22:59.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.857+0000 7f41d8cf3740 1 -- 172.21.15.62:0/55082087 wait complete. 2024-09-15T23:22:59.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.858+0000 7f41d8cf3740 20 client.17046 trim_cache size 0 max 16384 2024-09-15T23:22:59.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.889+0000 7f41d8cf3740 1 Processor -- start 2024-09-15T23:22:59.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41d8cf3740 1 -- start start 2024-09-15T23:22:59.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f6db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4fb40 0x561b85f79530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f61b80 0x561b85f79a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f41d01af270 con 0x561b85f4e5d0 2024-09-15T23:22:59.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f41d01acb40 con 0x561b85f4fb40 2024-09-15T23:22:59.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f41d01aaa60 con 0x561b85f61b80 2024-09-15T23:22:59.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4fb40 0x561b85f79530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41cf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4fb40 0x561b85f79530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49394/0 (socket says 172.21.15.62:49394) 2024-09-15T23:22:59.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41cf7fe640 1 -- 172.21.15.62:0/728436182 learned_addr learned my addr 172.21.15.62:0/728436182 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:59.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.890+0000 7f41d4eb7640 1 --2- 172.21.15.62:0/728436182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f61b80 0x561b85f79a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.891+0000 7f41cffff640 1 --2- 172.21.15.62:0/728436182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f6db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.891+0000 7f41cf7fe640 1 -- 172.21.15.62:0/728436182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f61b80 msgr2=0x561b85f79a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:59.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.891+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/728436182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f61b80 0x561b85f79a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.891+0000 7f41cf7fe640 1 -- 172.21.15.62:0/728436182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 msgr2=0x561b85f6db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:59.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.891+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/728436182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f6db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.891+0000 7f41cf7fe640 1 -- 172.21.15.62:0/728436182 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b85f4bbc0 con 0x561b85f4fb40 2024-09-15T23:22:59.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.893+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/728436182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4fb40 0x561b85f79530 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f41c400c6a0 tx=0x7f41c40bc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4adf4fbe6a5cdbf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:59.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.894+0000 7f41b5ffb640 1 -- 172.21.15.62:0/728436182 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c40a0710 con 0x561b85f4fb40 2024-09-15T23:22:59.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.894+0000 7f41b5ffb640 1 -- 172.21.15.62:0/728436182 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c40a4fd0 con 0x561b85f4fb40 2024-09-15T23:22:59.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.894+0000 7f41b5ffb640 1 -- 172.21.15.62:0/728436182 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c409baa0 con 0x561b85f4fb40 2024-09-15T23:22:59.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.895+0000 7f41d8cf3740 1 -- 172.21.15.62:0/728436182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4fb40 msgr2=0x561b85f79530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.895+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/728436182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4fb40 0x561b85f79530 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f41c400c6a0 tx=0x7f41c40bc040 comp rx=0 tx=0).stop 2024-09-15T23:22:59.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.897+0000 7f41d8cf3740 1 -- 172.21.15.62:0/728436182 shutdown_connections 2024-09-15T23:22:59.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.897+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/728436182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f61b80 0x561b85f79a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.897+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/728436182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4fb40 0x561b85f79530 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.897+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/728436182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f4e5d0 0x561b85f6db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.897+0000 7f41d8cf3740 1 -- 172.21.15.62:0/728436182 >> 172.21.15.62:0/728436182 conn(0x561b85f17310 msgr2=0x561b85f5d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:59.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.903+0000 7f41d8cf3740 1 -- 172.21.15.62:0/728436182 shutdown_connections 2024-09-15T23:22:59.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.903+0000 7f41d8cf3740 1 -- 172.21.15.62:0/728436182 wait complete. 2024-09-15T23:22:59.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.903+0000 7f41d8cf3740 1 -- start start 2024-09-15T23:22:59.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.904+0000 7f41d8cf3740 1 Processor -- start 2024-09-15T23:22:59.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.904+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 0x561b85f51450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.905+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f51990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.905+0000 7f41d8cf3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 0x561b85f59200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.905+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f41d01ab020 con 0x561b85f61b80 2024-09-15T23:22:59.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.905+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f41d01ac470 con 0x561b85f4e5d0 2024-09-15T23:22:59.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.905+0000 7f41d8cf3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f41d01ab6d0 con 0x561b85f4fb40 2024-09-15T23:22:59.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.905+0000 7f41d8cf3740 10 client.0 _ll_register_callbacks cb 0x7f41c0009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:22:59.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.906+0000 7f41d4eb7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 0x561b85f59200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.906+0000 7f41d4eb7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 0x561b85f59200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52966/0 (socket says 172.21.15.62:52966) 2024-09-15T23:22:59.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.906+0000 7f41d4eb7640 1 -- 172.21.15.62:0/3719042027 learned_addr learned my addr 172.21.15.62:0/3719042027 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:22:59.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.909+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f51990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.909+0000 7f41cffff640 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 0x561b85f51450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.911+0000 7f41cf7fe640 1 -- 172.21.15.62:0/3719042027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 msgr2=0x561b85f51450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:59.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.911+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 0x561b85f51450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.912+0000 7f41cf7fe640 1 -- 172.21.15.62:0/3719042027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 msgr2=0x561b85f59200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:22:59.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.912+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 0x561b85f59200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.912+0000 7f41cf7fe640 1 -- 172.21.15.62:0/3719042027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b85f59a00 con 0x561b85f4fb40 2024-09-15T23:22:59.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.912+0000 7f41cf7fe640 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f51990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x561b85f64840 tx=0x7f41c40cd7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9865686cf1c25e5f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:22:59.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.912+0000 7f41af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:22:59.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.912+0000 7f41af7fe640 1 -- 172.21.15.62:0/3719042027 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c40989d0 con 0x561b85f4fb40 2024-09-15T23:22:59.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.913+0000 7f41d8cf3740 1 -- 172.21.15.62:0/3719042027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41c40a4fd0 con 0x561b85f4fb40 2024-09-15T23:22:59.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.913+0000 7f41af7fe640 1 -- 172.21.15.62:0/3719042027 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f41c409b7b0 con 0x561b85f4fb40 2024-09-15T23:22:59.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.916+0000 7f41af7fe640 1 -- 172.21.15.62:0/3719042027 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f41c409b9b0 con 0x561b85f4fb40 2024-09-15T23:22:59.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.917+0000 7f41d8cf3740 10 client.26440 fetch_fsmap learned FSMap version 32 2024-09-15T23:22:59.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.917+0000 7f41d8cf3740 1 -- 172.21.15.62:0/3719042027 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561b85f67d70 con 0x561b85f4fb40 2024-09-15T23:22:59.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.917+0000 7f41af7fe640 1 -- 172.21.15.62:0/3719042027 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41c40a4fd0 con 0x561b85f4fb40 2024-09-15T23:22:59.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.918+0000 7f41af7fe640 1 -- 172.21.15.62:0/3719042027 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f41c4010430 con 0x561b85f4fb40 2024-09-15T23:22:59.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.918+0000 7f41d8cf3740 10 client.26440 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:22:59.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.918+0000 7f41d8cf3740 10 client.26440 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:22:59.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.918+0000 7f41d8cf3740 20 client.26440 populate_metadata read hostname 'smithi062' 2024-09-15T23:22:59.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.918+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x561b85fe55e0 0x561b85fe79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:22:59.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.918+0000 7f41d8cf3740 4 client.26440 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:22:59.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.918+0000 7f41d8cf3740 1 -- 172.21.15.62:0/3719042027 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b85ebbd70 con 0x561b85fe55e0 2024-09-15T23:22:59.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.919+0000 7f41cffff640 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x561b85fe55e0 0x561b85fe79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:22:59.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.928+0000 7f41af7fe640 10 client.26440 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:22:59.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.928+0000 7f41cffff640 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x561b85fe55e0 0x561b85fe79d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dbf1d002d15b1bf7 server_cookie=5a93ad3dc058316e in_seq=0 out_seq=0 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.928+0000 7f41af7fe640 1 -- 172.21.15.62:0/3719042027 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b85ebbd70 con 0x561b85fe55e0 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.928+0000 7f41af7fe640 10 client.26440 handle_command_reply: tid=0 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.928+0000 7f41d8cf3740 1 client.26440 shutdown 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.929+0000 7f41d8cf3740 1 -- 172.21.15.62:0/3719042027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 msgr2=0x561b85f51990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.929+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f51990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x561b85f64840 tx=0x7f41c40cd7b0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.929+0000 7f41d8cf3740 1 -- 172.21.15.62:0/3719042027 shutdown_connections 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.929+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x561b85fe55e0 0x561b85fe79d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.929+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b85f61b80 0x561b85f59200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.929+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b85f4fb40 0x561b85f51990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.929+0000 7f41d8cf3740 1 --2- 172.21.15.62:0/3719042027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b85f4e5d0 0x561b85f51450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.929+0000 7f41d8cf3740 1 -- 172.21.15.62:0/3719042027 >> 172.21.15.62:0/3719042027 conn(0x561b85f17310 msgr2=0x561b85f49d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:59.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.929+0000 7f41af7fe640 0 client.26440 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:22:59.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.932+0000 7f41d8cf3740 1 -- 172.21.15.62:0/3719042027 shutdown_connections 2024-09-15T23:22:59.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.932+0000 7f41d8cf3740 1 -- 172.21.15.62:0/3719042027 wait complete. 2024-09-15T23:22:59.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.932+0000 7f41d8cf3740 20 client.26440 trim_cache size 0 max 16384 2024-09-15T23:22:59.935 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:22:59.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 -- 172.21.15.62:0/2999845622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41b80a32c0 msgr2=0x7f41b80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41b80a32c0 0x7f41b80a5780 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f41c8046000 tx=0x7f41c8048000 comp rx=0 tx=0).stop 2024-09-15T23:22:59.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 -- 172.21.15.62:0/2999845622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d01049a0 msgr2=0x7f41d019abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:22:59.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d01049a0 0x7f41d019abf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f41c400e9f0 tx=0x7f41c400eeb0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 -- 172.21.15.62:0/2999845622 shutdown_connections 2024-09-15T23:22:59.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f41b80a32c0 0x7f41b80a5780 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f41d0109120 0x7f41d019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f41d01049a0 0x7f41d019abf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 --2- 172.21.15.62:0/2999845622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41d0103ff0 0x7f41d019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:22:59.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.935+0000 7f41d691a640 1 -- 172.21.15.62:0/2999845622 >> 172.21.15.62:0/2999845622 conn(0x7f41d00ffab0 msgr2=0x7f41d0100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:22:59.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.936+0000 7f41d691a640 1 -- 172.21.15.62:0/2999845622 shutdown_connections 2024-09-15T23:22:59.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:22:59.936+0000 7f41d691a640 1 -- 172.21.15.62:0/2999845622 wait complete. 2024-09-15T23:23:01.046 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:23:01.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:00 smithi067 ceph-mon[35665]: pgmap v572: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 608 op/s 2024-09-15T23:23:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:00 smithi148 ceph-mon[35610]: pgmap v572: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 608 op/s 2024-09-15T23:23:01.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:00 smithi062 ceph-mon[30949]: pgmap v572: 129 pgs: 129 active+clean; 1.5 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 608 op/s 2024-09-15T23:23:01.553 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:03.068 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:02 smithi062 ceph-mon[30949]: pgmap v573: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.0 MiB/s wr, 591 op/s 2024-09-15T23:23:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:03 smithi067 ceph-mon[35665]: pgmap v573: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.0 MiB/s wr, 591 op/s 2024-09-15T23:23:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:02 smithi148 ceph-mon[35610]: pgmap v573: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.0 MiB/s wr, 591 op/s 2024-09-15T23:23:03.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.018+0000 7fa3ae65d640 1 -- 172.21.15.62:0/3128991766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a8108da0 msgr2=0x7fa3a810cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.018+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/3128991766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a8108da0 0x7fa3a810cdf0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fa39c009860 tx=0x7fa39c02a380 comp rx=0 tx=0).stop 2024-09-15T23:23:03.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.025+0000 7fa3ae65d640 1 -- 172.21.15.62:0/3128991766 shutdown_connections 2024-09-15T23:23:03.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.026+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/3128991766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a810d520 0x7fa3a81150a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.026+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/3128991766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a8108da0 0x7fa3a810cdf0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.026+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/3128991766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3a81083f0 0x7fa3a81087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.026+0000 7fa3ae65d640 1 -- 172.21.15.62:0/3128991766 >> 172.21.15.62:0/3128991766 conn(0x7fa3a8077af0 msgr2=0x7fa3a8079f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.029+0000 7fa3ae65d640 1 -- 172.21.15.62:0/3128991766 shutdown_connections 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.033+0000 7fa3ae65d640 1 -- 172.21.15.62:0/3128991766 wait complete. 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.033+0000 7fa3ae65d640 1 Processor -- start 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.033+0000 7fa3ae65d640 1 -- start start 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.034+0000 7fa3ae65d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a81083f0 0x7fa3a8073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.034+0000 7fa3ae65d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3a8108da0 0x7fa3a806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.034+0000 7fa3ae65d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a810d520 0x7fa3a806d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.034+0000 7fa3ae65d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a8118570 con 0x7fa3a810d520 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.034+0000 7fa3ae65d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a81183f0 con 0x7fa3a8108da0 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.034+0000 7fa3ae65d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a81186f0 con 0x7fa3a81083f0 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.036+0000 7fa3acbfa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a810d520 0x7fa3a806d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.036+0000 7fa3acbfa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a810d520 0x7fa3a806d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41740/0 (socket says 172.21.15.62:41740) 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.036+0000 7fa3acbfa640 1 -- 172.21.15.62:0/1707478733 learned_addr learned my addr 172.21.15.62:0/1707478733 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.036+0000 7fa3a7fff640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a81083f0 0x7fa3a8073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.036+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3a8108da0 0x7fa3a806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.039+0000 7fa3acbfa640 1 -- 172.21.15.62:0/1707478733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a81083f0 msgr2=0x7fa3a8073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.039+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a81083f0 0x7fa3a8073ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.039+0000 7fa3acbfa640 1 -- 172.21.15.62:0/1707478733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3a8108da0 msgr2=0x7fa3a806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.039+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3a8108da0 0x7fa3a806cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.039+0000 7fa3acbfa640 1 -- 172.21.15.62:0/1707478733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a806d730 con 0x7fa3a810d520 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.042+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a810d520 0x7fa3a806d090 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000cad0 tx=0x7fa3a000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.047+0000 7fa3a57fa640 1 -- 172.21.15.62:0/1707478733 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000ad60 con 0x7fa3a810d520 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.047+0000 7fa3a57fa640 1 -- 172.21.15.62:0/1707478733 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3a0005020 con 0x7fa3a810d520 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.047+0000 7fa3a57fa640 1 -- 172.21.15.62:0/1707478733 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000f6b0 con 0x7fa3a810d520 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.047+0000 7fa3ae65d640 1 -- 172.21.15.62:0/1707478733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a80715f0 con 0x7fa3a810d520 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.047+0000 7fa3ae65d640 1 -- 172.21.15.62:0/1707478733 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a8071ab0 con 0x7fa3a810d520 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.054+0000 7fa3a57fa640 1 -- 172.21.15.62:0/1707478733 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa3a0005560 con 0x7fa3a810d520 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.054+0000 7fa3b0a36740 1 Processor -- start 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.055+0000 7fa3a57fa640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3900a3450 0x7fa3900a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.057+0000 7fa3a57fa640 1 -- 172.21.15.62:0/1707478733 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa3a00cfb70 con 0x7fa3a810d520 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.059+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3900a3450 0x7fa3900a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3b0a36740 1 -- start start 2024-09-15T23:23:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8636b40 0x55f4c8635090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 0x55f4c8635a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c8642760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3b0a36740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f4c859b2f0 con 0x55f4c8640370 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3b0a36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f4c8542cf0 con 0x55f4c8636b40 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3b0a36740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f4c85972b0 con 0x55f4c86355d0 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3acbfa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c8642760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3acbfa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c8642760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41744/0 (socket says 172.21.15.62:41744) 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.064+0000 7fa3acbfa640 1 -- 172.21.15.62:0/4115323828 learned_addr learned my addr 172.21.15.62:0/4115323828 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.068+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/4115323828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 0x55f4c8635a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.068+0000 7fa3acbfa640 1 -- 172.21.15.62:0/4115323828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 msgr2=0x55f4c8635a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.068+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/4115323828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 0x55f4c8635a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.068+0000 7fa3acbfa640 1 -- 172.21.15.62:0/4115323828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8636b40 msgr2=0x55f4c8635090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.068+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/4115323828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8636b40 0x55f4c8635090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.068+0000 7fa3acbfa640 1 -- 172.21.15.62:0/4115323828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4c85273a0 con 0x55f4c8640370 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.068+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3900a3450 0x7fa3900a5910 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0058c0 tx=0x7fa39c030c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.072+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/4115323828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c8642760 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0098210 tx=0x7fa3a009c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37e2ae83270ca23 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.072+0000 7fa38e7fc640 1 -- 172.21.15.62:0/4115323828 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a001d030 con 0x55f4c8640370 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.073+0000 7fa38e7fc640 1 -- 172.21.15.62:0/4115323828 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3a009f030 con 0x55f4c8640370 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.073+0000 7fa38e7fc640 1 -- 172.21.15.62:0/4115323828 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a009b200 con 0x55f4c8640370 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.074+0000 7fa3b0a36740 1 -- 172.21.15.62:0/4115323828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 msgr2=0x55f4c8642760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.074+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/4115323828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c8642760 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0098210 tx=0x7fa3a009c5f0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.077+0000 7fa3b0a36740 1 -- 172.21.15.62:0/4115323828 shutdown_connections 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.077+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/4115323828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c8642760 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.077+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/4115323828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 0x55f4c8635a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.077+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/4115323828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8636b40 0x55f4c8635090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.077+0000 7fa3b0a36740 1 -- 172.21.15.62:0/4115323828 >> 172.21.15.62:0/4115323828 conn(0x55f4c85fe310 msgr2=0x55f4c8631c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.083+0000 7fa3b0a36740 1 -- 172.21.15.62:0/4115323828 shutdown_connections 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.086+0000 7fa3b0a36740 1 -- 172.21.15.62:0/4115323828 wait complete. 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.087+0000 7fa3b0a36740 1 -- start start 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.088+0000 7fa3b0a36740 1 Processor -- start 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.088+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c863fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c863d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3a7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c863fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3a7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c863fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52230/0 (socket says 172.21.15.62:52230) 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3a7fff640 1 -- 172.21.15.62:0/2795446772 learned_addr learned my addr 172.21.15.62:0/2795446772 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8640370 0x55f4c863d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f4c8643070 con 0x55f4c8636b40 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f4c8556060 con 0x55f4c86355d0 2024-09-15T23:23:03.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8640370 0x55f4c863d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f4c86431f0 con 0x55f4c8640370 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.089+0000 7fa3b0a36740 10 client.0 _ll_register_callbacks cb 0x55f4c82a98c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.091+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c863d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.094+0000 7fa3a7fff640 1 -- 172.21.15.62:0/2795446772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8640370 msgr2=0x55f4c863d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.094+0000 7fa3a7fff640 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8640370 0x55f4c863d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.094+0000 7fa3a7fff640 1 -- 172.21.15.62:0/2795446772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 msgr2=0x55f4c863d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.094+0000 7fa3a7fff640 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c863d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.094+0000 7fa3a7fff640 1 -- 172.21.15.62:0/2795446772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4c830c3c0 con 0x55f4c86355d0 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.094+0000 7fa3a7fff640 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c863fde0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa3980116c0 tx=0x7fa398011b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e893d7a0e217783e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.094+0000 7fa3877fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.095+0000 7fa3877fe640 1 -- 172.21.15.62:0/2795446772 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39800fdf0 con 0x55f4c86355d0 2024-09-15T23:23:03.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.095+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3a009f030 con 0x55f4c86355d0 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.095+0000 7fa3877fe640 1 -- 172.21.15.62:0/2795446772 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa398004030 con 0x55f4c86355d0 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.095+0000 7fa3877fe640 1 -- 172.21.15.62:0/2795446772 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa398024050 con 0x55f4c86355d0 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.095+0000 7fa3b0a36740 10 client.26642 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.095+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f4c864ec00 con 0x55f4c86355d0 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.096+0000 7fa3877fe640 1 -- 172.21.15.62:0/2795446772 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3a009f030 con 0x55f4c86355d0 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.096+0000 7fa3877fe640 1 -- 172.21.15.62:0/2795446772 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7fa398019420 con 0x55f4c86355d0 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.097+0000 7fa3b0a36740 10 client.26642 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.097+0000 7fa3b0a36740 10 client.26642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.097+0000 7fa3b0a36740 20 client.26642 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.097+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f4c86504e0 0x55f4c8652960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.097+0000 7fa3b0a36740 4 client.26642 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.097+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4c85a8860 con 0x55f4c86504e0 2024-09-15T23:23:03.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.099+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f4c86504e0 0x55f4c8652960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.100+0000 7fa3877fe640 10 client.26642 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.100+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f4c86504e0 0x55f4c8652960 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0c4e2bd18f3a844 server_cookie=b2fa2429bcd3bd9 in_seq=0 out_seq=0 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3877fe640 1 -- 172.21.15.62:0/2795446772 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f4c85a8860 con 0x55f4c86504e0 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3877fe640 10 client.26642 handle_command_reply: tid=0 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 client.26642 shutdown 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 msgr2=0x55f4c863fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c863fde0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa3980116c0 tx=0x7fa398011b80 comp rx=0 tx=0).stop 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 shutdown_connections 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f4c86504e0 0x55f4c8652960 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8640370 0x55f4c863d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c863d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2795446772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c863fde0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 >> 172.21.15.62:0/2795446772 conn(0x55f4c85fe310 msgr2=0x55f4c8637990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3877fe640 0 client.26642 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 shutdown_connections 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2795446772 wait complete. 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.101+0000 7fa3b0a36740 20 client.26642 trim_cache size 0 max 16384 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.114+0000 7fa3b0a36740 1 Processor -- start 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.114+0000 7fa3b0a36740 1 -- start start 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 0x55f4c864c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c8658270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8640370 0x55f4c86587b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3b0a36740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a81c8c80 con 0x55f4c8636b40 2024-09-15T23:23:03.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3b0a36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a81c6550 con 0x55f4c8640370 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3b0a36740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a81c43c0 con 0x55f4c86355d0 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c8658270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3a77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c8658270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41764/0 (socket says 172.21.15.62:41764) 2024-09-15T23:23:03.925 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3a77fe640 1 -- 172.21.15.62:0/2748934478 learned_addr learned my addr 172.21.15.62:0/2748934478 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8640370 0x55f4c86587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3a7fff640 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 0x55f4c864c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3acbfa640 1 -- 172.21.15.62:0/2748934478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 msgr2=0x55f4c864c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 0x55f4c864c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3acbfa640 1 -- 172.21.15.62:0/2748934478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 msgr2=0x55f4c8658270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c8658270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3acbfa640 1 -- 172.21.15.62:0/2748934478 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4c864c850 con 0x55f4c8640370 2024-09-15T23:23:03.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c8658270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3a7fff640 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 0x55f4c864c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa3acbfa640 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8640370 0x55f4c86587b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0098210 tx=0x7fa3a00a02a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c01eb1c93de2bc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa38dffb640 1 -- 172.21.15.62:0/2748934478 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a001d040 con 0x55f4c8640370 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.115+0000 7fa38dffb640 1 -- 172.21.15.62:0/2748934478 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3a009f050 con 0x55f4c8640370 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa38dffb640 1 -- 172.21.15.62:0/2748934478 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a009b460 con 0x55f4c8640370 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2748934478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8640370 msgr2=0x55f4c86587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8640370 0x55f4c86587b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0098210 tx=0x7fa3a00a02a0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2748934478 shutdown_connections 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c8640370 0x55f4c86587b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8636b40 0x55f4c8658270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/2748934478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c86355d0 0x55f4c864c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2748934478 >> 172.21.15.62:0/2748934478 conn(0x55f4c85fe310 msgr2=0x55f4c86382c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:03.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2748934478 shutdown_connections 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.116+0000 7fa3b0a36740 1 -- 172.21.15.62:0/2748934478 wait complete. 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3b0a36740 1 -- start start 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3b0a36740 1 Processor -- start 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c8643bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8636b40 0x55f4c8644100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3b0a36740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c86b3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3b0a36740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a81c4a10 con 0x55f4c8640370 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3b0a36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a81c5e80 con 0x55f4c86355d0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3b0a36740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a81c50e0 con 0x55f4c8636b40 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3b0a36740 10 client.0 _ll_register_callbacks cb 0x7fa3980097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3a7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c8643bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3a7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c8643bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52246/0 (socket says 172.21.15.62:52246) 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3acbfa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c86b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.117+0000 7fa3a7fff640 1 -- 172.21.15.62:0/925597313 learned_addr learned my addr 172.21.15.62:0/925597313 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8636b40 0x55f4c8644100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3a7fff640 1 -- 172.21.15.62:0/925597313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8636b40 msgr2=0x55f4c8644100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3a7fff640 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8636b40 0x55f4c8644100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3a7fff640 1 -- 172.21.15.62:0/925597313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 msgr2=0x55f4c86b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3a7fff640 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c86b3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3a7fff640 1 -- 172.21.15.62:0/925597313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4c86b38b0 con 0x55f4c86355d0 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8636b40 0x55f4c8644100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3a7fff640 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c8643bc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa398000c00 tx=0x7fa398005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea18d1385e402618 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3877fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3877fe640 1 -- 172.21.15.62:0/925597313 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa398019420 con 0x55f4c86355d0 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3877fe640 1 -- 172.21.15.62:0/925597313 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa39800fbf0 con 0x55f4c86355d0 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3b0a36740 1 -- 172.21.15.62:0/925597313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3a009f050 con 0x55f4c86355d0 2024-09-15T23:23:03.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3877fe640 1 -- 172.21.15.62:0/925597313 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa398037050 con 0x55f4c86355d0 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3b0a36740 10 client.26648 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3b0a36740 1 -- 172.21.15.62:0/925597313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f4c86babd0 con 0x55f4c86355d0 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.118+0000 7fa3877fe640 1 -- 172.21.15.62:0/925597313 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3a009f050 con 0x55f4c86355d0 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3877fe640 1 -- 172.21.15.62:0/925597313 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7fa398004680 con 0x55f4c86355d0 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3b0a36740 10 client.26648 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3b0a36740 10 client.26648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3b0a36740 20 client.26648 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f4c86bb350 0x55f4c86bd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3b0a36740 4 client.26648 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3b0a36740 1 -- 172.21.15.62:0/925597313 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4c85a2d70 con 0x55f4c86bb350 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f4c86bb350 0x55f4c86bd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3877fe640 10 client.26648 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.119+0000 7fa3a77fe640 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f4c86bb350 0x55f4c86bd770 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b92c9b355a540919 server_cookie=e2be55ab70deff25 in_seq=0 out_seq=0 2024-09-15T23:23:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3877fe640 1 -- 172.21.15.62:0/925597313 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4c85a2d70 con 0x55f4c86bb350 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3877fe640 10 client.26648 handle_command_reply: tid=0 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 client.26648 shutdown 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 -- 172.21.15.62:0/925597313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 msgr2=0x55f4c8643bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c8643bc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa398000c00 tx=0x7fa398005da0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 -- 172.21.15.62:0/925597313 shutdown_connections 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f4c86bb350 0x55f4c86bd770 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f4c8640370 0x55f4c86b3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f4c8636b40 0x55f4c8644100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 --2- 172.21.15.62:0/925597313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f4c86355d0 0x55f4c8643bc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3877fe640 0 client.26648 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 -- 172.21.15.62:0/925597313 >> 172.21.15.62:0/925597313 conn(0x55f4c85fe310 msgr2=0x55f4c863b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 -- 172.21.15.62:0/925597313 shutdown_connections 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 1 -- 172.21.15.62:0/925597313 wait complete. 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3b0a36740 20 client.26648 trim_cache size 0 max 16384 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3ae65d640 1 -- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3900a3450 msgr2=0x7fa3900a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3900a3450 0x7fa3900a5910 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0058c0 tx=0x7fa39c030c90 comp rx=0 tx=0).stop 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3ae65d640 1 -- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a810d520 msgr2=0x7fa3a806d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.120+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a810d520 0x7fa3a806d090 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000cad0 tx=0x7fa3a000ced0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.121+0000 7fa3ae65d640 1 -- 172.21.15.62:0/1707478733 shutdown_connections 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.121+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3900a3450 0x7fa3900a5910 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.121+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa3a810d520 0x7fa3a806d090 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.121+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3a8108da0 0x7fa3a806cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.121+0000 7fa3ae65d640 1 --2- 172.21.15.62:0/1707478733 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa3a81083f0 0x7fa3a8073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.121+0000 7fa3ae65d640 1 -- 172.21.15.62:0/1707478733 >> 172.21.15.62:0/1707478733 conn(0x7fa3a8077af0 msgr2=0x7fa3a8078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.121+0000 7fa3ae65d640 1 -- 172.21.15.62:0/1707478733 shutdown_connections 2024-09-15T23:23:03.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:03.121+0000 7fa3ae65d640 1 -- 172.21.15.62:0/1707478733 wait complete. 2024-09-15T23:23:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 405.0 s, 789.6 tps, lat 124.502 ms stddev 140.699 2024-09-15T23:23:05.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:05 smithi067 ceph-mon[35665]: pgmap v574: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.0 MiB/s wr, 549 op/s 2024-09-15T23:23:05.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:05 smithi148 ceph-mon[35610]: pgmap v574: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.0 MiB/s wr, 549 op/s 2024-09-15T23:23:05.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:05 smithi062 ceph-mon[30949]: pgmap v574: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.0 MiB/s wr, 549 op/s 2024-09-15T23:23:06.562 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:23:06.563 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:23:07.036 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:07.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:07 smithi067 ceph-mon[35665]: pgmap v575: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.7 MiB/s wr, 524 op/s 2024-09-15T23:23:07.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:07 smithi148 ceph-mon[35610]: pgmap v575: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.7 MiB/s wr, 524 op/s 2024-09-15T23:23:07.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:07 smithi062 ceph-mon[30949]: pgmap v575: 129 pgs: 129 active+clean; 1.5 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.7 MiB/s wr, 524 op/s 2024-09-15T23:23:08.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.942+0000 7f5b2a81a640 1 -- 172.21.15.62:0/3500405060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24105d90 msgr2=0x7f5b24106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.942+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/3500405060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24105d90 0x7f5b24106220 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009ca0 tx=0x7f5b1c02a970 comp rx=0 tx=0).stop 2024-09-15T23:23:08.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.943+0000 7f5b2a81a640 1 -- 172.21.15.62:0/3500405060 shutdown_connections 2024-09-15T23:23:08.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.943+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/3500405060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24105d90 0x7f5b24106220 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.943+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/3500405060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b241012a0 0x7f5b24105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.943+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/3500405060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b24100980 0x7f5b24100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.943+0000 7f5b2a81a640 1 -- 172.21.15.62:0/3500405060 >> 172.21.15.62:0/3500405060 conn(0x7f5b240fc610 msgr2=0x7f5b240fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.946+0000 7f5b2a81a640 1 -- 172.21.15.62:0/3500405060 shutdown_connections 2024-09-15T23:23:08.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.947+0000 7f5b2a81a640 1 -- 172.21.15.62:0/3500405060 wait complete. 2024-09-15T23:23:08.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.947+0000 7f5b2a81a640 1 Processor -- start 2024-09-15T23:23:08.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b2a81a640 1 -- start start 2024-09-15T23:23:08.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b2a81a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b24100980 0x7f5b24114b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b2a81a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b241012a0 0x7f5b2410ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b2a81a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24105d90 0x7f5b2410f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b29818640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b24100980 0x7f5b24114b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b29818640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b24100980 0x7f5b24114b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41790/0 (socket says 172.21.15.62:41790) 2024-09-15T23:23:08.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b2a81a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5b24107790 con 0x7f5b24100980 2024-09-15T23:23:08.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b2a81a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b24107610 con 0x7f5b24105d90 2024-09-15T23:23:08.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b2a81a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5b24107910 con 0x7f5b241012a0 2024-09-15T23:23:08.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b29818640 1 -- 172.21.15.62:0/2329460262 learned_addr learned my addr 172.21.15.62:0/2329460262 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:08.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.948+0000 7f5b2a019640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24105d90 0x7f5b2410f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.951+0000 7f5b29017640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b241012a0 0x7f5b2410ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.951+0000 7f5b2a019640 1 -- 172.21.15.62:0/2329460262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b241012a0 msgr2=0x7f5b2410ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.951+0000 7f5b2a019640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b241012a0 0x7f5b2410ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.951+0000 7f5b2a019640 1 -- 172.21.15.62:0/2329460262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b24100980 msgr2=0x7f5b24114b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.951+0000 7f5b2a019640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b24100980 0x7f5b24114b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.951+0000 7f5b2a019640 1 -- 172.21.15.62:0/2329460262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2410f6e0 con 0x7f5b24105d90 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.952+0000 7f5b2a019640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24105d90 0x7f5b2410f120 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009ca0 tx=0x7f5b1c003e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.952+0000 7f5b1affd640 1 -- 172.21.15.62:0/2329460262 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c047070 con 0x7f5b24105d90 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.952+0000 7f5b1affd640 1 -- 172.21.15.62:0/2329460262 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b1c02f8a0 con 0x7f5b24105d90 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.952+0000 7f5b1affd640 1 -- 172.21.15.62:0/2329460262 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c034070 con 0x7f5b24105d90 2024-09-15T23:23:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.953+0000 7f5b2a81a640 1 -- 172.21.15.62:0/2329460262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2410f970 con 0x7f5b24105d90 2024-09-15T23:23:08.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.953+0000 7f5b2a81a640 1 -- 172.21.15.62:0/2329460262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b241b38b0 con 0x7f5b24105d90 2024-09-15T23:23:08.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.956+0000 7f5b1affd640 1 -- 172.21.15.62:0/2329460262 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f5b1c04a070 con 0x7f5b24105d90 2024-09-15T23:23:08.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.957+0000 7f5b2de55740 1 Processor -- start 2024-09-15T23:23:08.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.957+0000 7f5b1affd640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b100a33d0 0x7f5b100a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.957+0000 7f5b1affd640 1 -- 172.21.15.62:0/2329460262 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5b1c0f0370 con 0x7f5b24105d90 2024-09-15T23:23:08.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.957+0000 7f5b2de55740 1 -- start start 2024-09-15T23:23:08.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.957+0000 7f5b29017640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b100a33d0 0x7f5b100a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.957+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c58ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c74a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 0x5595d0c77350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b2de55740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5595d0bbe280 con 0x5595d0c59530 2024-09-15T23:23:08.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b2de55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5595d0b65b10 con 0x5595d0c74f60 2024-09-15T23:23:08.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b2de55740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5595d0bba240 con 0x5595d0c58c10 2024-09-15T23:23:08.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b29818640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c58ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b29818640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c58ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43490/0 (socket says 172.21.15.62:43490) 2024-09-15T23:23:08.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b29818640 1 -- 172.21.15.62:0/845355151 learned_addr learned my addr 172.21.15.62:0/845355151 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:08.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b29017640 1 --2- 172.21.15.62:0/845355151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c74a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b2a019640 1 --2- 172.21.15.62:0/845355151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 0x5595d0c77350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b29818640 1 -- 172.21.15.62:0/845355151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 msgr2=0x5595d0c77350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.958+0000 7f5b29818640 1 --2- 172.21.15.62:0/845355151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 0x5595d0c77350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.959+0000 7f5b29818640 1 -- 172.21.15.62:0/845355151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 msgr2=0x5595d0c74a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.959+0000 7f5b29818640 1 --2- 172.21.15.62:0/845355151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c74a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.959+0000 7f5b29818640 1 -- 172.21.15.62:0/845355151 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595d0b721f0 con 0x5595d0c58c10 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.959+0000 7f5b29017640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b100a33d0 0x7f5b100a5890 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5b241103a0 tx=0x7f5b20014950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.959+0000 7f5b29818640 1 --2- 172.21.15.62:0/845355151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c58ff0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009b00 tx=0x7f5b1400e1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=523d4f85a56ba80b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.959+0000 7f5b0ffff640 1 -- 172.21.15.62:0/845355151 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b14017070 con 0x5595d0c58c10 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.959+0000 7f5b0ffff640 1 -- 172.21.15.62:0/845355151 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b1400ec10 con 0x5595d0c58c10 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.959+0000 7f5b0ffff640 1 -- 172.21.15.62:0/845355151 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b14012630 con 0x5595d0c58c10 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.960+0000 7f5b2de55740 1 -- 172.21.15.62:0/845355151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 msgr2=0x5595d0c58ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.960+0000 7f5b2de55740 1 --2- 172.21.15.62:0/845355151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c58ff0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009b00 tx=0x7f5b1400e1d0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.960+0000 7f5b2de55740 1 -- 172.21.15.62:0/845355151 shutdown_connections 2024-09-15T23:23:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.960+0000 7f5b2de55740 1 --2- 172.21.15.62:0/845355151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 0x5595d0c77350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.960+0000 7f5b2de55740 1 --2- 172.21.15.62:0/845355151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c74a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.960+0000 7f5b2de55740 1 --2- 172.21.15.62:0/845355151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c58ff0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.960+0000 7f5b2de55740 1 -- 172.21.15.62:0/845355151 >> 172.21.15.62:0/845355151 conn(0x5595d0c55300 msgr2=0x5595d0c556e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.961+0000 7f5b2de55740 1 -- 172.21.15.62:0/845355151 shutdown_connections 2024-09-15T23:23:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.961+0000 7f5b2de55740 1 -- 172.21.15.62:0/845355151 wait complete. 2024-09-15T23:23:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.961+0000 7f5b2de55740 1 -- start start 2024-09-15T23:23:08.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.961+0000 7f5b2de55740 1 Processor -- start 2024-09-15T23:23:08.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c71f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c72450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 0x5595d0c72990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b2de55740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5595d0c77b30 con 0x5595d0c59530 2024-09-15T23:23:08.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b2de55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5595d0b78f10 con 0x5595d0c74f60 2024-09-15T23:23:08.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b2de55740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5595d0c77cb0 con 0x5595d0c58c10 2024-09-15T23:23:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b2de55740 10 client.0 _ll_register_callbacks cb 0x5595d0b41a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b29818640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c71f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b29818640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c71f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43498/0 (socket says 172.21.15.62:43498) 2024-09-15T23:23:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b29818640 1 -- 172.21.15.62:0/3713805224 learned_addr learned my addr 172.21.15.62:0/3713805224 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.962+0000 7f5b29017640 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c72450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.963+0000 7f5b2a019640 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 0x5595d0c72990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.963+0000 7f5b2a019640 1 -- 172.21.15.62:0/3713805224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 msgr2=0x5595d0c71f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.963+0000 7f5b2a019640 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c71f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.963+0000 7f5b2a019640 1 -- 172.21.15.62:0/3713805224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 msgr2=0x5595d0c72450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.963+0000 7f5b2a019640 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c72450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.963+0000 7f5b2a019640 1 -- 172.21.15.62:0/3713805224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595d092f6a0 con 0x5595d0c74f60 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.963+0000 7f5b2a019640 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 0x5595d0c72990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0edac0 tx=0x7f5b1c0bc280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=249d4bc5133e5ecd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.963+0000 7f5b0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.964+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/3713805224 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0d5060 con 0x5595d0c74f60 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.964+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/3713805224 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5b1c0ba9c0 con 0x5595d0c74f60 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.964+0000 7f5b2de55740 1 -- 172.21.15.62:0/3713805224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b1400ec10 con 0x5595d0c74f60 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.964+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/3713805224 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b1c0c09c0 con 0x5595d0c74f60 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.964+0000 7f5b2de55740 10 client.26660 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:08.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.964+0000 7f5b2de55740 1 -- 172.21.15.62:0/3713805224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5595d0c77b30 con 0x5595d0c74f60 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.964+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/3713805224 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b1400ec10 con 0x5595d0c74f60 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.964+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/3713805224 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f5b1c041030 con 0x5595d0c74f60 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.965+0000 7f5b2de55740 10 client.26660 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.965+0000 7f5b2de55740 10 client.26660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.965+0000 7f5b2de55740 20 client.26660 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.965+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5595d0c84da0 0x5595d0c871b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.965+0000 7f5b2de55740 4 client.26660 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.965+0000 7f5b2de55740 1 -- 172.21.15.62:0/3713805224 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595d0bcb7f0 con 0x5595d0c84da0 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.965+0000 7f5b29818640 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5595d0c84da0 0x5595d0c871b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.966+0000 7f5b0d7fa640 10 client.26660 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:08.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.966+0000 7f5b29818640 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5595d0c84da0 0x5595d0c871b0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f67f234a4fe38b9 server_cookie=b168d72b6e6c383e in_seq=0 out_seq=0 2024-09-15T23:23:08.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.966+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/3713805224 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5595d0bcb7f0 con 0x5595d0c84da0 2024-09-15T23:23:08.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.966+0000 7f5b0d7fa640 10 client.26660 handle_command_reply: tid=0 2024-09-15T23:23:08.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.966+0000 7f5b2de55740 1 client.26660 shutdown 2024-09-15T23:23:08.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b2de55740 1 -- 172.21.15.62:0/3713805224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 msgr2=0x5595d0c72990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:08.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 0x5595d0c72990 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0edac0 tx=0x7f5b1c0bc280 comp rx=0 tx=0).stop 2024-09-15T23:23:08.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b2de55740 1 -- 172.21.15.62:0/3713805224 shutdown_connections 2024-09-15T23:23:08.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5595d0c84da0 0x5595d0c871b0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c74f60 0x5595d0c72990 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c72450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3713805224 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c71f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:08.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b2de55740 1 -- 172.21.15.62:0/3713805224 >> 172.21.15.62:0/3713805224 conn(0x5595d0c55300 msgr2=0x5595d0c5b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:08.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b0d7fa640 0 client.26660 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:08.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.967+0000 7f5b2de55740 1 -- 172.21.15.62:0/3713805224 shutdown_connections 2024-09-15T23:23:08.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.968+0000 7f5b2de55740 1 -- 172.21.15.62:0/3713805224 wait complete. 2024-09-15T23:23:08.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.968+0000 7f5b2de55740 20 client.26660 trim_cache size 0 max 16384 2024-09-15T23:23:08.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.992+0000 7f5b2de55740 1 Processor -- start 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.992+0000 7f5b2de55740 1 -- start start 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c58c10 0x5595d0c727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c54300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c74f60 0x5595d0c54840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b2de55740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5b241b7910 con 0x5595d0c59530 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b2de55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b241b51e0 con 0x5595d0c58c10 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b2de55740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5b24110150 con 0x5595d0c74f60 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b2a019640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c74f60 0x5595d0c54840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b29818640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c58c10 0x5595d0c727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b29818640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c58c10 0x5595d0c727e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52306/0 (socket says 172.21.15.62:52306) 2024-09-15T23:23:08.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b29818640 1 -- 172.21.15.62:0/3400114256 learned_addr learned my addr 172.21.15.62:0/3400114256 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:09.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.993+0000 7f5b29017640 1 --2- 172.21.15.62:0/3400114256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c54300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b2a019640 1 -- 172.21.15.62:0/3400114256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c58c10 msgr2=0x5595d0c727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b2a019640 1 --2- 172.21.15.62:0/3400114256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c58c10 0x5595d0c727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b2a019640 1 -- 172.21.15.62:0/3400114256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 msgr2=0x5595d0c54300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b2a019640 1 --2- 172.21.15.62:0/3400114256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c54300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b2a019640 1 -- 172.21.15.62:0/3400114256 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595d0c6cd30 con 0x5595d0c74f60 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b29017640 1 --2- 172.21.15.62:0/3400114256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c54300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b2a019640 1 --2- 172.21.15.62:0/3400114256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c74f60 0x5595d0c54840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0ba970 tx=0x7f5b1c02a970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b457a27cf97d33 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b0f7fe640 1 -- 172.21.15.62:0/3400114256 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0c5280 con 0x5595d0c74f60 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b0f7fe640 1 -- 172.21.15.62:0/3400114256 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b1c0ba9c0 con 0x5595d0c74f60 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.994+0000 7f5b0f7fe640 1 -- 172.21.15.62:0/3400114256 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0d0b30 con 0x5595d0c74f60 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.995+0000 7f5b2de55740 1 -- 172.21.15.62:0/3400114256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c74f60 msgr2=0x5595d0c54840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.995+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3400114256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c74f60 0x5595d0c54840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0ba970 tx=0x7f5b1c02a970 comp rx=0 tx=0).stop 2024-09-15T23:23:09.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.995+0000 7f5b2de55740 1 -- 172.21.15.62:0/3400114256 shutdown_connections 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.995+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3400114256 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c74f60 0x5595d0c54840 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.995+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3400114256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c59530 0x5595d0c54300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.995+0000 7f5b2de55740 1 --2- 172.21.15.62:0/3400114256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c58c10 0x5595d0c727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.995+0000 7f5b2de55740 1 -- 172.21.15.62:0/3400114256 >> 172.21.15.62:0/3400114256 conn(0x5595d0c55300 msgr2=0x5595d0c53bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.995+0000 7f5b2de55740 1 -- 172.21.15.62:0/3400114256 shutdown_connections 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.995+0000 7f5b2de55740 1 -- 172.21.15.62:0/3400114256 wait complete. 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.996+0000 7f5b2de55740 1 -- start start 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.996+0000 7f5b2de55740 1 Processor -- start 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.996+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c79080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.996+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c59530 0x5595d0c795c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.996+0000 7f5b2de55740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c74f60 0x5595d0ceea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.996+0000 7f5b2de55740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5b24073fe0 con 0x5595d0c74f60 2024-09-15T23:23:09.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.996+0000 7f5b2de55740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b241b4b10 con 0x5595d0c59530 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.996+0000 7f5b2de55740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5b241b3d70 con 0x5595d0c58c10 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.996+0000 7f5b2de55740 10 client.0 _ll_register_callbacks cb 0x7f5b1c0b82a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b2a019640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c74f60 0x5595d0ceea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b29017640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c59530 0x5595d0c795c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b2a019640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c74f60 0x5595d0ceea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41830/0 (socket says 172.21.15.62:41830) 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b2a019640 1 -- 172.21.15.62:0/2944573218 learned_addr learned my addr 172.21.15.62:0/2944573218 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b29818640 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c79080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b29017640 1 -- 172.21.15.62:0/2944573218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 msgr2=0x5595d0c79080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b29017640 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c79080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b29017640 1 -- 172.21.15.62:0/2944573218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c74f60 msgr2=0x5595d0ceea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b29017640 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c74f60 0x5595d0ceea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b29017640 1 -- 172.21.15.62:0/2944573218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595d0c79ec0 con 0x5595d0c59530 2024-09-15T23:23:09.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b29017640 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c59530 0x5595d0c795c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x5595d0c717a0 tx=0x7f5b20004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98212b59f2c7fd9b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.997+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/2944573218 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20011440 con 0x5595d0c59530 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b2de55740 1 -- 172.21.15.62:0/2944573218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b1c0ba9c0 con 0x5595d0c59530 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/2944573218 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5b200115e0 con 0x5595d0c59530 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/2944573218 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5b20033050 con 0x5595d0c59530 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b2de55740 10 client.26666 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b2de55740 1 -- 172.21.15.62:0/2944573218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5595d0cf0190 con 0x5595d0c59530 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/2944573218 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b1c0ba9c0 con 0x5595d0c59530 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/2944573218 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f5b20020b70 con 0x5595d0c59530 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b2de55740 10 client.26666 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b2de55740 10 client.26666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:09.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b2de55740 20 client.26666 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:09.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b2de55740 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5595d0cf0920 0x5595d0cf2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b2de55740 4 client.26666 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:23:09.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b2de55740 1 -- 172.21.15.62:0/2944573218 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5595d0bc5d00 con 0x5595d0cf0920 2024-09-15T23:23:09.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.998+0000 7f5b29818640 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5595d0cf0920 0x5595d0cf2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.999+0000 7f5b0d7fa640 10 client.26666 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:09.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:08.999+0000 7f5b29818640 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5595d0cf0920 0x5595d0cf2d40 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b2afb9a992dcbcc server_cookie=3118073cfca7044a in_seq=0 out_seq=0 2024-09-15T23:23:09.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.000+0000 7f5b0d7fa640 1 -- 172.21.15.62:0/2944573218 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5595d0bc5d00 con 0x5595d0cf0920 2024-09-15T23:23:09.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.000+0000 7f5b0d7fa640 10 client.26666 handle_command_reply: tid=0 2024-09-15T23:23:09.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.000+0000 7f5b2de55740 1 client.26666 shutdown 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 -- 172.21.15.62:0/2944573218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c59530 msgr2=0x5595d0c795c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c59530 0x5595d0c795c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x5595d0c717a0 tx=0x7f5b20004730 comp rx=0 tx=0).stop 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 -- 172.21.15.62:0/2944573218 shutdown_connections 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5595d0cf0920 0x5595d0cf2d40 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d0c74f60 0x5595d0ceea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5595d0c59530 0x5595d0c795c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 --2- 172.21.15.62:0/2944573218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5595d0c58c10 0x5595d0c79080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 -- 172.21.15.62:0/2944573218 >> 172.21.15.62:0/2944573218 conn(0x5595d0c55300 msgr2=0x5595d0c54410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b0d7fa640 0 client.26666 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 -- 172.21.15.62:0/2944573218 shutdown_connections 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 1 -- 172.21.15.62:0/2944573218 wait complete. 2024-09-15T23:23:09.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.001+0000 7f5b2de55740 20 client.26666 trim_cache size 0 max 16384 2024-09-15T23:23:09.008 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:23:09.008 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:23:09.008 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "bfb45eee-321f-41df-945a-41b64bea6e42", 2024-09-15T23:23:09.008 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:23:09.008 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 -- 172.21.15.62:0/2329460262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b100a33d0 msgr2=0x7f5b100a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b100a33d0 0x7f5b100a5890 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5b241103a0 tx=0x7f5b20014950 comp rx=0 tx=0).stop 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 -- 172.21.15.62:0/2329460262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24105d90 msgr2=0x7f5b2410f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24105d90 0x7f5b2410f120 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009ca0 tx=0x7f5b1c003e60 comp rx=0 tx=0).stop 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 -- 172.21.15.62:0/2329460262 shutdown_connections 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b100a33d0 0x7f5b100a5890 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24105d90 0x7f5b2410f120 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b241012a0 0x7f5b2410ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 --2- 172.21.15.62:0/2329460262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b24100980 0x7f5b24114b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 -- 172.21.15.62:0/2329460262 >> 172.21.15.62:0/2329460262 conn(0x7f5b240fc610 msgr2=0x7f5b240fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.002+0000 7f5b2a81a640 1 -- 172.21.15.62:0/2329460262 shutdown_connections 2024-09-15T23:23:09.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:09.006+0000 7f5b2a81a640 1 -- 172.21.15.62:0/2329460262 wait complete. 2024-09-15T23:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.194+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/3135217158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e181025f0 msgr2=0x7f9e18108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.194+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/3135217158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e181025f0 0x7f9e18108e80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9e10009ca0 tx=0x7f9e1002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.195+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/3135217158 shutdown_connections 2024-09-15T23:23:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.195+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/3135217158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e181025f0 0x7f9e18108e80 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.195+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/3135217158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e18101c30 0x7f9e181020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.195+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/3135217158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e18100a40 0x7f9e18100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.195+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/3135217158 >> 172.21.15.62:0/3135217158 conn(0x7f9e180fc1e0 msgr2=0x7f9e180fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.195+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/3135217158 shutdown_connections 2024-09-15T23:23:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.195+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/3135217158 wait complete. 2024-09-15T23:23:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.195+0000 7f9e1d1d6640 1 Processor -- start 2024-09-15T23:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e1d1d6640 1 -- start start 2024-09-15T23:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e1d1d6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e18100a40 0x7f9e181990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e1d1d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e18101c30 0x7f9e18199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e1d1d6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e181025f0 0x7f9e18195c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e18100a40 0x7f9e181990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e18100a40 0x7f9e181990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43518/0 (socket says 172.21.15.62:43518) 2024-09-15T23:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e16d9d640 1 -- 172.21.15.62:0/2068851167 learned_addr learned my addr 172.21.15.62:0/2068851167 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1810b600 con 0x7f9e181025f0 2024-09-15T23:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1810b480 con 0x7f9e18101c30 2024-09-15T23:23:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.196+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1810b780 con 0x7f9e18100a40 2024-09-15T23:23:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.197+0000 7f9e1759e640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e181025f0 0x7f9e18195c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.198+0000 7f9e1659c640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e18101c30 0x7f9e18199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.198+0000 7f9e1759e640 1 -- 172.21.15.62:0/2068851167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e18100a40 msgr2=0x7f9e181990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.198+0000 7f9e1759e640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e18100a40 0x7f9e181990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.198+0000 7f9e1759e640 1 -- 172.21.15.62:0/2068851167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e18101c30 msgr2=0x7f9e18199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.198+0000 7f9e1759e640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e18101c30 0x7f9e18199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.198+0000 7f9e1759e640 1 -- 172.21.15.62:0/2068851167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e18196510 con 0x7f9e181025f0 2024-09-15T23:23:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.199+0000 7f9e1759e640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e181025f0 0x7f9e18195c50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9e10031ae0 tx=0x7f9e100319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.199+0000 7f9dfffff640 1 -- 172.21.15.62:0/2068851167 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10003b70 con 0x7f9e181025f0 2024-09-15T23:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.199+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e181a63b0 con 0x7f9e181025f0 2024-09-15T23:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.199+0000 7f9dfffff640 1 -- 172.21.15.62:0/2068851167 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e10003d10 con 0x7f9e181025f0 2024-09-15T23:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.199+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e181a67b0 con 0x7f9e181025f0 2024-09-15T23:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.200+0000 7f9dfffff640 1 -- 172.21.15.62:0/2068851167 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1002fad0 con 0x7f9e181025f0 2024-09-15T23:23:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9dfffff640 1 -- 172.21.15.62:0/2068851167 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9e100045b0 con 0x7f9e181025f0 2024-09-15T23:23:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e2004a740 1 Processor -- start 2024-09-15T23:23:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e2004a740 1 -- start start 2024-09-15T23:23:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99a9a0 0x5600ba99ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba99fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e2004a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5600ba8fe2f0 con 0x5600ba99a9a0 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e2004a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5600ba8a6270 con 0x5600ba99b2e0 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e2004a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5600ba8fa2b0 con 0x5600ba9a0370 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e16d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba99fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e16d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba99fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52348/0 (socket says 172.21.15.62:52348) 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e16d9d640 1 -- 172.21.15.62:0/3397549731 learned_addr learned my addr 172.21.15.62:0/3397549731 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.203+0000 7f9e1659c640 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99a9a0 0x5600ba99ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9e16d9d640 1 -- 172.21.15.62:0/3397549731 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 msgr2=0x5600ba9a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9e1659c640 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9a2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9e16d9d640 1 -- 172.21.15.62:0/3397549731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99a9a0 msgr2=0x5600ba99ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99a9a0 0x5600ba99ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9e16d9d640 1 -- 172.21.15.62:0/3397549731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600ba8a5340 con 0x5600ba99b2e0 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9e1659c640 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba99fda0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009d70 tx=0x7f9e08011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bee4784817adeb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9dfffff640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e000a3440 0x7f9e000a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9dfd7fa640 1 -- 172.21.15.62:0/3397549731 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08004220 con 0x5600ba99b2e0 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9dfd7fa640 1 -- 172.21.15.62:0/3397549731 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e080043c0 con 0x5600ba99b2e0 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9dfd7fa640 1 -- 172.21.15.62:0/3397549731 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e080198f0 con 0x5600ba99b2e0 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9e1659c640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e000a3440 0x7f9e000a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.204+0000 7f9dfffff640 1 -- 172.21.15.62:0/2068851167 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e100f1060 con 0x7f9e181025f0 2024-09-15T23:23:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e2004a740 1 -- 172.21.15.62:0/3397549731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 msgr2=0x5600ba99fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e2004a740 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba99fda0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009d70 tx=0x7f9e08011810 comp rx=0 tx=0).stop 2024-09-15T23:23:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e1659c640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e000a3440 0x7f9e000a5900 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00d6f0 tx=0x7f9e0c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e2004a740 1 -- 172.21.15.62:0/3397549731 shutdown_connections 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e2004a740 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e2004a740 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba99fda0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e2004a740 1 --2- 172.21.15.62:0/3397549731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99a9a0 0x5600ba99ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e2004a740 1 -- 172.21.15.62:0/3397549731 >> 172.21.15.62:0/3397549731 conn(0x5600ba996010 msgr2=0x5600ba996410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e2004a740 1 -- 172.21.15.62:0/3397549731 shutdown_connections 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.205+0000 7f9e2004a740 1 -- 172.21.15.62:0/3397549731 wait complete. 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e2004a740 1 -- start start 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e2004a740 1 Processor -- start 2024-09-15T23:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99a9a0 0x5600ba9a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba9a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba9a0370 0x5600ba9ac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e2004a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5600ba9a3340 con 0x5600ba9a0370 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e2004a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5600ba887290 con 0x5600ba99b2e0 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e2004a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5600ba9a34c0 con 0x5600ba99a9a0 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e2004a740 10 client.0 _ll_register_callbacks cb 0x5600ba8b8140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99a9a0 0x5600ba9a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99a9a0 0x5600ba9a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43528/0 (socket says 172.21.15.62:43528) 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.206+0000 7f9e16d9d640 1 -- 172.21.15.62:0/914737867 learned_addr learned my addr 172.21.15.62:0/914737867 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.208+0000 7f9e16d9d640 1 -- 172.21.15.62:0/914737867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 msgr2=0x5600ba9a4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.208+0000 7f9e1659c640 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba9a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.208+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba9a4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.208+0000 7f9e16d9d640 1 -- 172.21.15.62:0/914737867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba9a0370 msgr2=0x5600ba9ac000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.208+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba9a0370 0x5600ba9ac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.208+0000 7f9e16d9d640 1 -- 172.21.15.62:0/914737867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600ba611210 con 0x5600ba99a9a0 2024-09-15T23:23:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.208+0000 7f9e1659c640 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba9a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.212+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99a9a0 0x5600ba9a4a80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9e0801b040 tx=0x7f9e08003e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2587600d1f7636b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.213+0000 7f9dfaff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:23:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.215+0000 7f9dfaff5640 1 -- 172.21.15.62:0/914737867 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0804a070 con 0x5600ba99a9a0 2024-09-15T23:23:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.215+0000 7f9dfaff5640 1 -- 172.21.15.62:0/914737867 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e08003ab0 con 0x5600ba99a9a0 2024-09-15T23:23:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.215+0000 7f9e2004a740 1 -- 172.21.15.62:0/914737867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e080043c0 con 0x5600ba99a9a0 2024-09-15T23:23:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.221+0000 7f9dfaff5640 1 -- 172.21.15.62:0/914737867 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e08005040 con 0x5600ba99a9a0 2024-09-15T23:23:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.222+0000 7f9e2004a740 10 client.26473 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.222+0000 7f9e2004a740 1 -- 172.21.15.62:0/914737867 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5600ba6dd620 con 0x5600ba99a9a0 2024-09-15T23:23:09.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:09 smithi062 ceph-mon[30949]: pgmap v576: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.6 MiB/s wr, 576 op/s 2024-09-15T23:23:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.228+0000 7f9dfaff5640 1 -- 172.21.15.62:0/914737867 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e080043c0 con 0x5600ba99a9a0 2024-09-15T23:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.228+0000 7f9dfaff5640 1 -- 172.21.15.62:0/914737867 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f9e08019650 con 0x5600ba99a9a0 2024-09-15T23:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.228+0000 7f9e2004a740 10 client.26473 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.228+0000 7f9e2004a740 10 client.26473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.228+0000 7f9e2004a740 20 client.26473 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.228+0000 7f9e2004a740 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5600ba9ada50 0x5600ba9afed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.228+0000 7f9e2004a740 4 client.26473 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.228+0000 7f9e2004a740 1 -- 172.21.15.62:0/914737867 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600ba90ba20 con 0x5600ba9ada50 2024-09-15T23:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.229+0000 7f9e1659c640 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5600ba9ada50 0x5600ba9afed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.250 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.246+0000 7f9e1659c640 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5600ba9ada50 0x5600ba9afed0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b903030a056fc9db server_cookie=94a6439e0cabc9b1 in_seq=0 out_seq=0 2024-09-15T23:23:09.250 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.247+0000 7f9dfaff5640 10 client.26473 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.251+0000 7f9dfaff5640 1 -- 172.21.15.62:0/914737867 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5600ba90ba20 con 0x5600ba9ada50 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.251+0000 7f9dfaff5640 10 client.26473 handle_command_reply: tid=0 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.251+0000 7f9e2004a740 1 client.26473 shutdown 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.253+0000 7f9e2004a740 1 -- 172.21.15.62:0/914737867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99a9a0 msgr2=0x5600ba9a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.253+0000 7f9e2004a740 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99a9a0 0x5600ba9a4a80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9e0801b040 tx=0x7f9e08003e40 comp rx=0 tx=0).stop 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.253+0000 7f9e2004a740 1 -- 172.21.15.62:0/914737867 shutdown_connections 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.253+0000 7f9e2004a740 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5600ba9ada50 0x5600ba9afed0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.253+0000 7f9e2004a740 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba9a0370 0x5600ba9ac000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.253+0000 7f9e2004a740 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba99b2e0 0x5600ba9a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.253+0000 7f9e2004a740 1 --2- 172.21.15.62:0/914737867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99a9a0 0x5600ba9a4a80 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9e0801b040 tx=0x7f9e08003e40 comp rx=0 tx=0).stop 2024-09-15T23:23:09.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.253+0000 7f9e2004a740 1 -- 172.21.15.62:0/914737867 >> 172.21.15.62:0/914737867 conn(0x5600ba996010 msgr2=0x5600ba997820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.258 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.255+0000 7f9dfaff5640 0 client.26473 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:09.259 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.258+0000 7f9e2004a740 1 -- 172.21.15.62:0/914737867 shutdown_connections 2024-09-15T23:23:09.262 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.261+0000 7f9e2004a740 1 -- 172.21.15.62:0/914737867 wait complete. 2024-09-15T23:23:09.262 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.261+0000 7f9e2004a740 20 client.26473 trim_cache size 0 max 16384 2024-09-15T23:23:09.274 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:23:09 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: 2024-09-15T23:23:09.059+0000 7ff293927640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000f [...2,head] /volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans/ auth{2=1} v2989 ap=2 f(v7 m2024-09-15T23:22:40.590845+0000 1=1+0) n(v19 rc2024-09-15T23:22:40.590845+0000 2=1+1) (inest sync->mix g=2) mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55598f2f4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v282>,<0x10000000005/data v121>,<0x100000001fb/postgres v25>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v43>,<0x10000000001/sv_1 v57>,<0x10000000000/qa v44>,<0x1/volumes v43>]//[]","memoryvalue":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v2989>,<0x10000000005/data v557>,<0x100000001fb/postgres v359>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v390>,<0x10000000001/sv_1 v383>,<0x10000000000/qa v382>,<0x1/volumes v360>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-09-15T23:22:40.543846+0000 8=8+0)","ondisk_value.rstat":"n(v0 rc2024-09-15T23:22:40.590845+0000 2=1+1)","memory_value.dirstat":"f(v7 m2024-09-15T23:22:40.590845+0000 1=1+0)","memory_value.rstat":"n(v19 rc2024-09-15T23:22:40.590845+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-09-15T23:23:09.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:09 smithi067 ceph-mon[35665]: pgmap v576: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.6 MiB/s wr, 576 op/s 2024-09-15T23:23:09.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:09 smithi148 ceph-mon[35610]: pgmap v576: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.6 MiB/s wr, 576 op/s 2024-09-15T23:23:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.300+0000 7f9e2004a740 1 Processor -- start 2024-09-15T23:23:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.303+0000 7f9e2004a740 1 -- start start 2024-09-15T23:23:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.303+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99b2e0 0x5600ba9a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.303+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.304+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba9bc490 0x5600ba9bc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.304+0000 7f9e2004a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e181aad80 con 0x5600ba99b2e0 2024-09-15T23:23:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.304+0000 7f9e2004a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e181a8650 con 0x5600ba9bc490 2024-09-15T23:23:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.304+0000 7f9e2004a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e18073e60 con 0x5600ba9a0370 2024-09-15T23:23:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.306+0000 7f9e1659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.306+0000 7f9e1659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43530/0 (socket says 172.21.15.62:43530) 2024-09-15T23:23:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.306+0000 7f9e1659c640 1 -- 172.21.15.62:0/921904416 learned_addr learned my addr 172.21.15.62:0/921904416 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.306+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/921904416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99b2e0 0x5600ba9a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.308+0000 7f9e1659c640 1 -- 172.21.15.62:0/921904416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba9bc490 msgr2=0x5600ba9bc920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:23:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.308+0000 7f9e1659c640 1 --2- 172.21.15.62:0/921904416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba9bc490 0x5600ba9bc920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.308+0000 7f9e1659c640 1 -- 172.21.15.62:0/921904416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99b2e0 msgr2=0x5600ba9a4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.308+0000 7f9e1659c640 1 --2- 172.21.15.62:0/921904416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99b2e0 0x5600ba9a4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.308+0000 7f9e1659c640 1 -- 172.21.15.62:0/921904416 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600ba99a520 con 0x5600ba9a0370 2024-09-15T23:23:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.310+0000 7f9e1659c640 1 --2- 172.21.15.62:0/921904416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9abfc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x5600ba9a5f60 tx=0x7f9e0c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dcadff8ed418930 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.311+0000 7f9dfcff9640 1 -- 172.21.15.62:0/921904416 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c01aaa0 con 0x5600ba9a0370 2024-09-15T23:23:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.311+0000 7f9dfcff9640 1 -- 172.21.15.62:0/921904416 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e0c034070 con 0x5600ba9a0370 2024-09-15T23:23:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.312+0000 7f9e2004a740 1 -- 172.21.15.62:0/921904416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 msgr2=0x5600ba9abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.312+0000 7f9e2004a740 1 --2- 172.21.15.62:0/921904416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9abfc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x5600ba9a5f60 tx=0x7f9e0c018000 comp rx=0 tx=0).stop 2024-09-15T23:23:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.314+0000 7f9e2004a740 1 -- 172.21.15.62:0/921904416 shutdown_connections 2024-09-15T23:23:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.314+0000 7f9e2004a740 1 --2- 172.21.15.62:0/921904416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba9bc490 0x5600ba9bc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.314+0000 7f9e2004a740 1 --2- 172.21.15.62:0/921904416 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba9a0370 0x5600ba9abfc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.314+0000 7f9e2004a740 1 --2- 172.21.15.62:0/921904416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba99b2e0 0x5600ba9a4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.314+0000 7f9e2004a740 1 -- 172.21.15.62:0/921904416 >> 172.21.15.62:0/921904416 conn(0x5600ba996010 msgr2=0x5600ba9a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.319 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.319+0000 7f9e2004a740 1 -- 172.21.15.62:0/921904416 shutdown_connections 2024-09-15T23:23:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.319+0000 7f9e2004a740 1 -- 172.21.15.62:0/921904416 wait complete. 2024-09-15T23:23:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e2004a740 1 -- start start 2024-09-15T23:23:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e2004a740 1 Processor -- start 2024-09-15T23:23:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99b2e0 0x5600ba9abeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba9a0370 0x5600ba9a8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e2004a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba9bc490 0x5600ba9a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.321 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e2004a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e181a6b30 con 0x5600ba9a0370 2024-09-15T23:23:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e2004a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e181a7f80 con 0x5600ba9bc490 2024-09-15T23:23:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e2004a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e181a71e0 con 0x5600ba99b2e0 2024-09-15T23:23:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e2004a740 10 client.0 _ll_register_callbacks cb 0x7f9e08004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.320+0000 7f9e16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99b2e0 0x5600ba9abeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.322 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.321+0000 7f9e16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99b2e0 0x5600ba9abeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43536/0 (socket says 172.21.15.62:43536) 2024-09-15T23:23:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.321+0000 7f9e16d9d640 1 -- 172.21.15.62:0/1136230656 learned_addr learned my addr 172.21.15.62:0/1136230656 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.321+0000 7f9e1659c640 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba9a0370 0x5600ba9a8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.321+0000 7f9e1759e640 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba9bc490 0x5600ba9a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.321+0000 7f9e16d9d640 1 -- 172.21.15.62:0/1136230656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba9bc490 msgr2=0x5600ba9a8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.321+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba9bc490 0x5600ba9a8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.321+0000 7f9e16d9d640 1 -- 172.21.15.62:0/1136230656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba9a0370 msgr2=0x5600ba9a8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.321+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba9a0370 0x5600ba9a8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.321+0000 7f9e16d9d640 1 -- 172.21.15.62:0/1136230656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600baa1a2e0 con 0x5600ba99b2e0 2024-09-15T23:23:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.322+0000 7f9e16d9d640 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99b2e0 0x5600ba9abeb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9e08003e10 tx=0x7f9e08004500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=976d5710b92d106 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.323+0000 7f9dfaff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:23:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.323+0000 7f9dfaff5640 1 -- 172.21.15.62:0/1136230656 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0804a040 con 0x5600ba99b2e0 2024-09-15T23:23:09.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.323+0000 7f9dfaff5640 1 -- 172.21.15.62:0/1136230656 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e08019630 con 0x5600ba99b2e0 2024-09-15T23:23:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.324+0000 7f9e2004a740 1 -- 172.21.15.62:0/1136230656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e0c034070 con 0x5600ba99b2e0 2024-09-15T23:23:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.324+0000 7f9dfaff5640 1 -- 172.21.15.62:0/1136230656 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e080051e0 con 0x5600ba99b2e0 2024-09-15T23:23:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.327+0000 7f9e2004a740 10 client.26479 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.327+0000 7f9e2004a740 1 -- 172.21.15.62:0/1136230656 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5600baa1b5c0 con 0x5600ba99b2e0 2024-09-15T23:23:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.328+0000 7f9dfaff5640 1 -- 172.21.15.62:0/1136230656 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e0c034070 con 0x5600ba99b2e0 2024-09-15T23:23:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.328+0000 7f9dfaff5640 1 -- 172.21.15.62:0/1136230656 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f9e08004c80 con 0x5600ba99b2e0 2024-09-15T23:23:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.328+0000 7f9e2004a740 10 client.26479 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.328+0000 7f9e2004a740 10 client.26479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.328+0000 7f9e2004a740 20 client.26479 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.328+0000 7f9e2004a740 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5600baa1bfd0 0x5600baa1e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.328+0000 7f9e2004a740 4 client.26479 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:23:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.328+0000 7f9e2004a740 1 -- 172.21.15.62:0/1136230656 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600ba905d70 con 0x5600baa1bfd0 2024-09-15T23:23:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.330+0000 7f9e1659c640 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5600baa1bfd0 0x5600baa1e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.331+0000 7f9e1659c640 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5600baa1bfd0 0x5600baa1e3f0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5150e4bd66707149 server_cookie=4f24570cf90e2889 in_seq=0 out_seq=0 2024-09-15T23:23:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.331+0000 7f9dfaff5640 10 client.26479 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:09.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9dfaff5640 1 -- 172.21.15.62:0/1136230656 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600ba905d70 con 0x5600baa1bfd0 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9dfaff5640 10 client.26479 handle_command_reply: tid=0 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 client.26479 shutdown 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 -- 172.21.15.62:0/1136230656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99b2e0 msgr2=0x5600ba9abeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99b2e0 0x5600ba9abeb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9e08003e10 tx=0x7f9e08004500 comp rx=0 tx=0).stop 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 -- 172.21.15.62:0/1136230656 shutdown_connections 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5600baa1bfd0 0x5600baa1e3f0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600ba9bc490 0x5600ba9a8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5600ba9a0370 0x5600ba9a8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 --2- 172.21.15.62:0/1136230656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5600ba99b2e0 0x5600ba9abeb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 -- 172.21.15.62:0/1136230656 >> 172.21.15.62:0/1136230656 conn(0x5600ba996010 msgr2=0x5600ba9a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9dfaff5640 0 client.26479 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:09.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 -- 172.21.15.62:0/1136230656 shutdown_connections 2024-09-15T23:23:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 1 -- 172.21.15.62:0/1136230656 wait complete. 2024-09-15T23:23:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.367+0000 7f9e2004a740 20 client.26479 trim_cache size 0 max 16384 2024-09-15T23:23:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:23:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:23:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:23:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:23:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e000a3440 msgr2=0x7f9e000a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e000a3440 0x7f9e000a5900 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00d6f0 tx=0x7f9e0c005f90 comp rx=0 tx=0).stop 2024-09-15T23:23:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e181025f0 msgr2=0x7f9e18195c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e181025f0 0x7f9e18195c50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9e10031ae0 tx=0x7f9e100319f0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 shutdown_connections 2024-09-15T23:23:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e000a3440 0x7f9e000a5900 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e181025f0 0x7f9e18195c50 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e18101c30 0x7f9e18199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 --2- 172.21.15.62:0/2068851167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e18100a40 0x7f9e181990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.368+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 >> 172.21.15.62:0/2068851167 conn(0x7f9e180fc1e0 msgr2=0x7f9e18104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.369+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 shutdown_connections 2024-09-15T23:23:09.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:09.369+0000 7f9e1d1d6640 1 -- 172.21.15.62:0/2068851167 wait complete. 2024-09-15T23:23:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 410.0 s, 590.8 tps, lat 168.778 ms stddev 203.297 2024-09-15T23:23:09.848 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:23:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:23:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:23:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:23:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:23:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:23:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:23:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:10 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:23:10.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:23:10.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:23:10.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:23:10.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:23:10.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:23:10.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:10 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:23:10.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:23:10.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:23:10.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:23:10.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:23:10.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:23:10.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:10 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:10.300 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:10.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.921+0000 7f9dfc782640 1 -- 172.21.15.62:0/3396969784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df4109830 msgr2=0x7f9df4109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.921+0000 7f9dfc782640 1 --2- 172.21.15.62:0/3396969784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df4109830 0x7f9df4109c90 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009900 tx=0x7f9df002a420 comp rx=0 tx=0).stop 2024-09-15T23:23:10.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.923+0000 7f9dfc782640 1 -- 172.21.15.62:0/3396969784 shutdown_connections 2024-09-15T23:23:10.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.923+0000 7f9dfc782640 1 --2- 172.21.15.62:0/3396969784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df411b140 0x7f9df411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.923+0000 7f9dfc782640 1 --2- 172.21.15.62:0/3396969784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df4109830 0x7f9df4109c90 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.923+0000 7f9dfc782640 1 --2- 172.21.15.62:0/3396969784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9df4109450 0x7f9df4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.923+0000 7f9dfc782640 1 -- 172.21.15.62:0/3396969784 >> 172.21.15.62:0/3396969784 conn(0x7f9df406e140 msgr2=0x7f9df406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.924+0000 7f9dfc782640 1 -- 172.21.15.62:0/3396969784 shutdown_connections 2024-09-15T23:23:10.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.927+0000 7f9dfc782640 1 -- 172.21.15.62:0/3396969784 wait complete. 2024-09-15T23:23:10.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.927+0000 7f9dfc782640 1 Processor -- start 2024-09-15T23:23:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.931+0000 7f9dfc782640 1 -- start start 2024-09-15T23:23:10.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.931+0000 7f9dfc782640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df4109450 0x7f9df41a72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.931+0000 7f9dfc782640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df4109830 0x7f9df41a7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.931+0000 7f9dfc782640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9df411b140 0x7f9df41abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.931+0000 7f9dfc782640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9df4120dc0 con 0x7f9df4109450 2024-09-15T23:23:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.931+0000 7f9dfc782640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9df4120c40 con 0x7f9df4109830 2024-09-15T23:23:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.931+0000 7f9dfc782640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9df4120f40 con 0x7f9df411b140 2024-09-15T23:23:10.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.932+0000 7f9dfad1f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9df411b140 0x7f9df41abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.932+0000 7f9dfad1f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9df411b140 0x7f9df41abb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43544/0 (socket says 172.21.15.62:43544) 2024-09-15T23:23:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.932+0000 7f9dfad1f640 1 -- 172.21.15.62:0/1176043473 learned_addr learned my addr 172.21.15.62:0/1176043473 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.932+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df4109830 0x7f9df41a7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.932+0000 7f9dfa51e640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df4109450 0x7f9df41a72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.933+0000 7f9dfa51e640 1 -- 172.21.15.62:0/1176043473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9df411b140 msgr2=0x7f9df41abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.933+0000 7f9dfa51e640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9df411b140 0x7f9df41abb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.933+0000 7f9dfa51e640 1 -- 172.21.15.62:0/1176043473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df4109830 msgr2=0x7f9df41a7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.933+0000 7f9dfa51e640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df4109830 0x7f9df41a7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.933+0000 7f9dfa51e640 1 -- 172.21.15.62:0/1176043473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df41ac270 con 0x7f9df4109450 2024-09-15T23:23:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.934+0000 7f9dfa51e640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df4109450 0x7f9df41a72c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00eb00 tx=0x7f9dec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.934+0000 7f9deb7fe640 1 -- 172.21.15.62:0/1176043473 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec010430 con 0x7f9df4109450 2024-09-15T23:23:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.935+0000 7f9deb7fe640 1 -- 172.21.15.62:0/1176043473 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dec010ab0 con 0x7f9df4109450 2024-09-15T23:23:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.935+0000 7f9dfc782640 1 -- 172.21.15.62:0/1176043473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df41ac560 con 0x7f9df4109450 2024-09-15T23:23:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.935+0000 7f9dfc782640 1 -- 172.21.15.62:0/1176043473 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df41b3e40 con 0x7f9df4109450 2024-09-15T23:23:10.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.935+0000 7f9deb7fe640 1 -- 172.21.15.62:0/1176043473 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec017b70 con 0x7f9df4109450 2024-09-15T23:23:10.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.939+0000 7f9dfeb5b740 1 Processor -- start 2024-09-15T23:23:10.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.940+0000 7f9dfeb5b740 1 -- start start 2024-09-15T23:23:10.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0a3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a3dc0 0x55c1db0ae140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0ae680 0x55c1db0b0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c1db0092e0 con 0x55c1db0a3dc0 2024-09-15T23:23:10.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c1dafb0ce0 con 0x55c1db0ae680 2024-09-15T23:23:10.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c1db0052a0 con 0x55c1db0a34a0 2024-09-15T23:23:10.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9dfad1f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9df9d1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0ae680 0x55c1db0b0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9df9d1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0ae680 0x55c1db0b0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52388/0 (socket says 172.21.15.62:52388) 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9dfad1f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0a3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43554/0 (socket says 172.21.15.62:43554) 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.941+0000 7f9df9d1d640 1 -- 172.21.15.62:0/3423513765 learned_addr learned my addr 172.21.15.62:0/3423513765 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.942+0000 7f9df9d1d640 1 -- 172.21.15.62:0/3423513765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 msgr2=0x55c1db0a3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.942+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/3423513765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0a3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.942+0000 7f9df9d1d640 1 -- 172.21.15.62:0/3423513765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a3dc0 msgr2=0x55c1db0ae140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.942+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/3423513765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a3dc0 0x55c1db0ae140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.942+0000 7f9df9d1d640 1 -- 172.21.15.62:0/3423513765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1daf95440 con 0x55c1db0ae680 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.942+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/3423513765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0ae680 0x55c1db0b0a70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9df0005ab0 tx=0x7f9df0005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78f1f9b3c8bb0e56 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.943+0000 7f9de8ff9640 1 -- 172.21.15.62:0/3423513765 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0036070 con 0x55c1db0ae680 2024-09-15T23:23:10.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.943+0000 7f9de8ff9640 1 -- 172.21.15.62:0/3423513765 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9df0031da0 con 0x55c1db0ae680 2024-09-15T23:23:10.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.943+0000 7f9de8ff9640 1 -- 172.21.15.62:0/3423513765 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0003c20 con 0x55c1db0ae680 2024-09-15T23:23:10.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.944+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3423513765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0ae680 msgr2=0x55c1db0b0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.944+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3423513765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0ae680 0x55c1db0b0a70 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9df0005ab0 tx=0x7f9df0005dd0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.944+0000 7f9deb7fe640 1 -- 172.21.15.62:0/1176043473 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9dec017d10 con 0x7f9df4109450 2024-09-15T23:23:10.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.944+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3423513765 shutdown_connections 2024-09-15T23:23:10.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.944+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3423513765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0ae680 0x55c1db0b0a70 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.944+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3423513765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a3dc0 0x55c1db0ae140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.944+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3423513765 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.944+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3423513765 >> 172.21.15.62:0/3423513765 conn(0x55c1db09fa30 msgr2=0x55c1db09fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.945+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3423513765 shutdown_connections 2024-09-15T23:23:10.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.946+0000 7f9deb7fe640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9de00a3320 0x7f9de00a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.946+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9de00a3320 0x7f9de00a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.946+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3423513765 wait complete. 2024-09-15T23:23:10.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.946+0000 7f9deb7fe640 1 -- 172.21.15.62:0/1176043473 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9dec0d04a0 con 0x7f9df4109450 2024-09-15T23:23:10.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.946+0000 7f9dfeb5b740 1 -- start start 2024-09-15T23:23:10.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.946+0000 7f9dfeb5b740 1 Processor -- start 2024-09-15T23:23:10.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0ab3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0abae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c1db0b1250 con 0x55c1db0ae680 2024-09-15T23:23:10.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c1dafc4050 con 0x55c1db0a3dc0 2024-09-15T23:23:10.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c1db0b13d0 con 0x55c1db0a34a0 2024-09-15T23:23:10.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfeb5b740 10 client.0 _ll_register_callbacks cb 0x55c1db004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:10.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfad1f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0abae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfa51e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfad1f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0abae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41908/0 (socket says 172.21.15.62:41908) 2024-09-15T23:23:10.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.947+0000 7f9dfad1f640 1 -- 172.21.15.62:0/2427295888 learned_addr learned my addr 172.21.15.62:0/2427295888 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.948+0000 7f9dfad1f640 1 -- 172.21.15.62:0/2427295888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 msgr2=0x55c1db0ae0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.948+0000 7f9dfad1f640 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0ae0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.948+0000 7f9dfad1f640 1 -- 172.21.15.62:0/2427295888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 msgr2=0x55c1db0ab3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.948+0000 7f9dfad1f640 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0ab3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.948+0000 7f9dfad1f640 1 -- 172.21.15.62:0/2427295888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1dad7a3c0 con 0x55c1db0ae680 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.948+0000 7f9dfad1f640 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0abae0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9de4011580 tx=0x7f9de4011a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf4d4077d47de3e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.948+0000 7f9dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.948+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9de00a3320 0x7f9de00a57e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9df0034040 tx=0x7f9df003e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.949+0000 7f9dddffb640 1 -- 172.21.15.62:0/2427295888 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4019070 con 0x55c1db0ae680 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.949+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/2427295888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9df0031da0 con 0x55c1db0ae680 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.949+0000 7f9dddffb640 1 -- 172.21.15.62:0/2427295888 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9de4015640 con 0x55c1db0ae680 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.949+0000 7f9dddffb640 1 -- 172.21.15.62:0/2427295888 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9de4025050 con 0x55c1db0ae680 2024-09-15T23:23:10.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.949+0000 7f9dfeb5b740 10 client.17109 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:10.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.949+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/2427295888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c1db0bce00 con 0x55c1db0ae680 2024-09-15T23:23:10.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.950+0000 7f9dddffb640 1 -- 172.21.15.62:0/2427295888 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9df0031da0 con 0x55c1db0ae680 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.950+0000 7f9dddffb640 1 -- 172.21.15.62:0/2427295888 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f9de4013a50 con 0x55c1db0ae680 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.950+0000 7f9dfeb5b740 10 client.17109 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.950+0000 7f9dfeb5b740 10 client.17109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.950+0000 7f9dfeb5b740 20 client.17109 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.950+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1db0be700 0x55c1db0c0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.950+0000 7f9dfeb5b740 4 client.17109 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.950+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/2427295888 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1db016850 con 0x55c1db0be700 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.950+0000 7f9dfa51e640 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1db0be700 0x55c1db0c0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.951+0000 7f9dddffb640 10 client.17109 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:10.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.951+0000 7f9dfa51e640 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1db0be700 0x55c1db0c0b60 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=308f216a4779dd65 server_cookie=3ab8fa3c21860a86 in_seq=0 out_seq=0 2024-09-15T23:23:10.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.952+0000 7f9dddffb640 1 -- 172.21.15.62:0/2427295888 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c1db016850 con 0x55c1db0be700 2024-09-15T23:23:10.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.952+0000 7f9dddffb640 10 client.17109 handle_command_reply: tid=0 2024-09-15T23:23:10.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.952+0000 7f9dfeb5b740 1 client.17109 shutdown 2024-09-15T23:23:10.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.952+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/2427295888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 msgr2=0x55c1db0abae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.952+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0abae0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9de4011580 tx=0x7f9de4011a40 comp rx=0 tx=0).stop 2024-09-15T23:23:10.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/2427295888 shutdown_connections 2024-09-15T23:23:10.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1db0be700 0x55c1db0c0b60 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0abae0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0ab3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/2427295888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0ae0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/2427295888 >> 172.21.15.62:0/2427295888 conn(0x55c1db09fa30 msgr2=0x55c1db0a5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dddffb640 0 client.17109 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:10.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/2427295888 shutdown_connections 2024-09-15T23:23:10.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/2427295888 wait complete. 2024-09-15T23:23:10.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.953+0000 7f9dfeb5b740 20 client.17109 trim_cache size 0 max 16384 2024-09-15T23:23:10.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.980+0000 7f9dfeb5b740 1 Processor -- start 2024-09-15T23:23:10.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfeb5b740 1 -- start start 2024-09-15T23:23:10.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0ba510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0aa380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd80124b0 con 0x55c1db0ae680 2024-09-15T23:23:10.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd800fd80 con 0x55c1db0a3dc0 2024-09-15T23:23:10.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd800dbd0 con 0x55c1db0a34a0 2024-09-15T23:23:10.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfa51e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0ba510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfad1f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfad1f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41922/0 (socket says 172.21.15.62:41922) 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9dfad1f640 1 -- 172.21.15.62:0/4206632847 learned_addr learned my addr 172.21.15.62:0/4206632847 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.981+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/4206632847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9dfad1f640 1 -- 172.21.15.62:0/4206632847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 msgr2=0x55c1db0ba510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9dfad1f640 1 --2- 172.21.15.62:0/4206632847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0ba510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9dfad1f640 1 -- 172.21.15.62:0/4206632847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 msgr2=0x55c1db0aa380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9dfad1f640 1 --2- 172.21.15.62:0/4206632847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0aa380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9dfad1f640 1 -- 172.21.15.62:0/4206632847 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1db0ab340 con 0x55c1db0ae680 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/4206632847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0aa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9dfad1f640 1 --2- 172.21.15.62:0/4206632847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0c6470 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9de4004700 tx=0x7f9de4013f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c337bedbf69933 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9ddffff640 1 -- 172.21.15.62:0/4206632847 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4019070 con 0x55c1db0ae680 2024-09-15T23:23:10.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9ddffff640 1 -- 172.21.15.62:0/4206632847 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de4014620 con 0x55c1db0ae680 2024-09-15T23:23:10.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.982+0000 7f9ddffff640 1 -- 172.21.15.62:0/4206632847 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4014e90 con 0x55c1db0ae680 2024-09-15T23:23:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.983+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/4206632847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 msgr2=0x55c1db0c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.983+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/4206632847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0c6470 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9de4004700 tx=0x7f9de4013f90 comp rx=0 tx=0).stop 2024-09-15T23:23:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.983+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/4206632847 shutdown_connections 2024-09-15T23:23:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.983+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/4206632847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0ae680 0x55c1db0c6470 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.983+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/4206632847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0aa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.983+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/4206632847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0a34a0 0x55c1db0ba510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.983+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/4206632847 >> 172.21.15.62:0/4206632847 conn(0x55c1db09fa30 msgr2=0x55c1db0a9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.983+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/4206632847 shutdown_connections 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.983+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/4206632847 wait complete. 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfeb5b740 1 -- start start 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfeb5b740 1 Processor -- start 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a34a0 0x55c1db0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfeb5b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0ae680 0x55c1db0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd800e240 con 0x55c1db0a34a0 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd800f6b0 con 0x55c1db0a3dc0 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfeb5b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd800e910 con 0x55c1db0ae680 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfeb5b740 10 client.0 _ll_register_callbacks cb 0x7f9de40013a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:10.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.984+0000 7f9dfa51e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a34a0 0x55c1db0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dfa51e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a34a0 0x55c1db0b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41926/0 (socket says 172.21.15.62:41926) 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dfad1f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0ae680 0x55c1db0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dfa51e640 1 -- 172.21.15.62:0/3058892159 learned_addr learned my addr 172.21.15.62:0/3058892159 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dfa51e640 1 -- 172.21.15.62:0/3058892159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0ae680 msgr2=0x55c1db0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dfa51e640 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0ae680 0x55c1db0b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dfa51e640 1 -- 172.21.15.62:0/3058892159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 msgr2=0x55c1db0b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dfa51e640 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dfa51e640 1 -- 172.21.15.62:0/3058892159 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1db1275b0 con 0x55c1db0a34a0 2024-09-15T23:23:10.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dfa51e640 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a34a0 0x55c1db0b3550 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9dec018000 tx=0x7f9dec00cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739f67bafe2a6be6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.985+0000 7f9dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.986+0000 7f9dddffb640 1 -- 172.21.15.62:0/3058892159 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec0a0a90 con 0x55c1db0a34a0 2024-09-15T23:23:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.986+0000 7f9dddffb640 1 -- 172.21.15.62:0/3058892159 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9dec0173c0 con 0x55c1db0a34a0 2024-09-15T23:23:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.986+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3058892159 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9de4014620 con 0x55c1db0a34a0 2024-09-15T23:23:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.986+0000 7f9dddffb640 1 -- 172.21.15.62:0/3058892159 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9dec0a5350 con 0x55c1db0a34a0 2024-09-15T23:23:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.986+0000 7f9dfeb5b740 10 client.17115 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.986+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3058892159 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c1db1288e0 con 0x55c1db0a34a0 2024-09-15T23:23:10.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.987+0000 7f9dddffb640 1 -- 172.21.15.62:0/3058892159 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dec0d0160 con 0x55c1db0a34a0 2024-09-15T23:23:10.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9dddffb640 1 -- 172.21.15.62:0/3058892159 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f9dec09bb30 con 0x55c1db0a34a0 2024-09-15T23:23:10.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9dfeb5b740 10 client.17115 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:10.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9dfeb5b740 10 client.17115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:10.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9dfeb5b740 20 client.17115 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:10.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1db128fb0 0x55c1db12b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:10.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9dfeb5b740 4 client.17115 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:23:10.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3058892159 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f9de4014620 con 0x55c1db128fb0 2024-09-15T23:23:10.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1db128fb0 0x55c1db12b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9dddffb640 10 client.17115 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.988+0000 7f9df9d1d640 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1db128fb0 0x55c1db12b3d0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d90b8115a06ec02 server_cookie=27df9daa74527efc in_seq=0 out_seq=0 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dddffb640 1 -- 172.21.15.62:0/3058892159 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f9de4014620 con 0x55c1db128fb0 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dddffb640 10 client.17115 handle_command_reply: tid=0 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 client.17115 shutdown 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3058892159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a34a0 msgr2=0x55c1db0b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a34a0 0x55c1db0b3550 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f9dec018000 tx=0x7f9dec00cf10 comp rx=0 tx=0).stop 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3058892159 shutdown_connections 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1db128fb0 0x55c1db12b3d0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c1db0ae680 0x55c1db0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c1db0a3dc0 0x55c1db0b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 --2- 172.21.15.62:0/3058892159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c1db0a34a0 0x55c1db0b3550 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3058892159 >> 172.21.15.62:0/3058892159 conn(0x55c1db09fa30 msgr2=0x55c1db0aa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dddffb640 0 client.17115 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:10.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3058892159 shutdown_connections 2024-09-15T23:23:10.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 1 -- 172.21.15.62:0/3058892159 wait complete. 2024-09-15T23:23:10.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.989+0000 7f9dfeb5b740 20 client.17115 trim_cache size 0 max 16384 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout: "bfb45eee-321f-41df-945a-41b64bea6e42": { 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "bfb45eee-321f-41df-945a-41b64bea6e42", 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:10.997 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:23:10.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 -- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9de00a3320 msgr2=0x7f9de00a57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9de00a3320 0x7f9de00a57e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9df0034040 tx=0x7f9df003e040 comp rx=0 tx=0).stop 2024-09-15T23:23:10.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 -- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df4109450 msgr2=0x7f9df41a72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df4109450 0x7f9df41a72c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00eb00 tx=0x7f9dec00c2d0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 -- 172.21.15.62:0/1176043473 shutdown_connections 2024-09-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9de00a3320 0x7f9de00a57e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9df411b140 0x7f9df41abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df4109830 0x7f9df41a7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 --2- 172.21.15.62:0/1176043473 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9df4109450 0x7f9df41a72c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 -- 172.21.15.62:0/1176043473 >> 172.21.15.62:0/1176043473 conn(0x7f9df406e140 msgr2=0x7f9df411bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 -- 172.21.15.62:0/1176043473 shutdown_connections 2024-09-15T23:23:10.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:10.991+0000 7f9dfc782640 1 -- 172.21.15.62:0/1176043473 wait complete. 2024-09-15T23:23:11.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:11 smithi062 ceph-mon[30949]: pgmap v577: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.3 MiB/s wr, 608 op/s 2024-09-15T23:23:11.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:11 smithi067 ceph-mon[35665]: pgmap v577: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.3 MiB/s wr, 608 op/s 2024-09-15T23:23:11.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:11 smithi148 ceph-mon[35610]: pgmap v577: 129 pgs: 129 active+clean; 1.5 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 6.3 MiB/s wr, 608 op/s 2024-09-15T23:23:11.836 INFO:tasks.cephfs.filesystem:scrub status for tag:bfb45eee-321f-41df-945a-41b64bea6e42 - {'path': '/', 'tag': 'bfb45eee-321f-41df-945a-41b64bea6e42', 'options': 'recursive,force'} 2024-09-15T23:23:12.208 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 23.7 2024-09-15T23:23:12.209 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:23:12.633 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:13.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:13 smithi148 ceph-mon[35610]: pgmap v578: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 5.8 MiB/s wr, 624 op/s 2024-09-15T23:23:13.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:13 smithi062 ceph-mon[30949]: pgmap v578: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 5.8 MiB/s wr, 624 op/s 2024-09-15T23:23:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:13 smithi067 ceph-mon[35665]: pgmap v578: 129 pgs: 129 active+clean; 1.5 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 5.8 MiB/s wr, 624 op/s 2024-09-15T23:23:14.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.168+0000 7f8223d6f640 1 -- 172.21.15.62:0/1668000873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f821c106a70 msgr2=0x7f821c106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.169+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1668000873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f821c106a70 0x7f821c106e50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8214009cd0 tx=0x7f821402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.169+0000 7f8223d6f640 1 -- 172.21.15.62:0/1668000873 shutdown_connections 2024-09-15T23:23:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.169+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1668000873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c101410 0x7f821c10d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.169+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1668000873 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f821c100a70 0x7f821c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.169+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1668000873 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f821c106a70 0x7f821c106e50 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.169+0000 7f8223d6f640 1 -- 172.21.15.62:0/1668000873 >> 172.21.15.62:0/1668000873 conn(0x7f821c0fc7d0 msgr2=0x7f821c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.169+0000 7f8223d6f640 1 -- 172.21.15.62:0/1668000873 shutdown_connections 2024-09-15T23:23:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.169+0000 7f8223d6f640 1 -- 172.21.15.62:0/1668000873 wait complete. 2024-09-15T23:23:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f8223d6f640 1 Processor -- start 2024-09-15T23:23:14.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f8223d6f640 1 -- start start 2024-09-15T23:23:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f8223d6f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f821c100a70 0x7f821c199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f8223d6f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c101410 0x7f821c199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f8223d6f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f821c106a70 0x7f821c1a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f8223d6f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f821c110e30 con 0x7f821c100a70 2024-09-15T23:23:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f8223d6f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f821c110cb0 con 0x7f821c101410 2024-09-15T23:23:14.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f8223d6f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f821c110fb0 con 0x7f821c106a70 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f8221b0b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f821c100a70 0x7f821c199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f822130a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c101410 0x7f821c199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f822130a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c101410 0x7f821c199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49142/0 (socket says 172.21.15.62:49142) 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f822130a640 1 -- 172.21.15.62:0/1140592497 learned_addr learned my addr 172.21.15.62:0/1140592497 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.171+0000 7f822230c640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f821c106a70 0x7f821c1a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.172+0000 7f822130a640 1 -- 172.21.15.62:0/1140592497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f821c106a70 msgr2=0x7f821c1a4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.172+0000 7f822130a640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f821c106a70 0x7f821c1a4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.172+0000 7f822130a640 1 -- 172.21.15.62:0/1140592497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f821c100a70 msgr2=0x7f821c199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.172+0000 7f822130a640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f821c100a70 0x7f821c199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.172+0000 7f822130a640 1 -- 172.21.15.62:0/1140592497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821c1a48c0 con 0x7f821c101410 2024-09-15T23:23:14.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.172+0000 7f822130a640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c101410 0x7f821c199840 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f821800d950 tx=0x7f821800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.172+0000 7f8212ffd640 1 -- 172.21.15.62:0/1140592497 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218017070 con 0x7f821c101410 2024-09-15T23:23:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.172+0000 7f8212ffd640 1 -- 172.21.15.62:0/1140592497 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8218009c30 con 0x7f821c101410 2024-09-15T23:23:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.172+0000 7f8212ffd640 1 -- 172.21.15.62:0/1140592497 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821800f6b0 con 0x7f821c101410 2024-09-15T23:23:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.173+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f821c1a4a90 con 0x7f821c101410 2024-09-15T23:23:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.173+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821c1a4f50 con 0x7f821c101410 2024-09-15T23:23:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.175+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f821c100ed0 con 0x7f821c101410 2024-09-15T23:23:14.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.175+0000 7f8212ffd640 1 -- 172.21.15.62:0/1140592497 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f821800f850 con 0x7f821c101410 2024-09-15T23:23:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.176+0000 7f8212ffd640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82080a3390 0x7f82080a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.176+0000 7f8212ffd640 1 -- 172.21.15.62:0/1140592497 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f82180cf7d0 con 0x7f821c101410 2024-09-15T23:23:14.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.176+0000 7f8221b0b640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82080a3390 0x7f82080a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:14.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.177+0000 7f8221b0b640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82080a3390 0x7f82080a5850 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8214006010 tx=0x7f8214009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:14.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.180+0000 7f8212ffd640 1 -- 172.21.15.62:0/1140592497 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8218009de0 con 0x7f821c101410 2024-09-15T23:23:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 415.0 s, 1071.2 tps, lat 96.522 ms stddev 136.367 2024-09-15T23:23:14.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.596+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f821c06ad80 con 0x7f821c101410 2024-09-15T23:23:14.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.598+0000 7f8212ffd640 1 -- 172.21.15.62:0/1140592497 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11904 (secure 0 0 0) 0x7f821809bac0 con 0x7f821c101410 2024-09-15T23:23:14.599 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:23:14.600 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":32,"btime":"2024-09-15T23:21:52:478522+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:21:52.380746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:23:14.600 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 32 2024-09-15T23:23:14.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82080a3390 msgr2=0x7f82080a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:14.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82080a3390 0x7f82080a5850 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8214006010 tx=0x7f8214009360 comp rx=0 tx=0).stop 2024-09-15T23:23:14.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c101410 msgr2=0x7f821c199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:14.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c101410 0x7f821c199840 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f821800d950 tx=0x7f821800de10 comp rx=0 tx=0).stop 2024-09-15T23:23:14.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 shutdown_connections 2024-09-15T23:23:14.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82080a3390 0x7f82080a5850 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f821c106a70 0x7f821c1a4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f821c101410 0x7f821c199840 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 --2- 172.21.15.62:0/1140592497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f821c100a70 0x7f821c199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:14.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 >> 172.21.15.62:0/1140592497 conn(0x7f821c0fc7d0 msgr2=0x7f821c0fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:14.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.601+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 shutdown_connections 2024-09-15T23:23:14.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:14.602+0000 7f8223d6f640 1 -- 172.21.15.62:0/1140592497 wait complete. 2024-09-15T23:23:15.227 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:15 smithi062 ceph-mon[30949]: pgmap v579: 129 pgs: 129 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 5.9 MiB/s wr, 654 op/s 2024-09-15T23:23:15.228 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:15 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1140592497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:15 smithi148 ceph-mon[35610]: pgmap v579: 129 pgs: 129 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 5.9 MiB/s wr, 654 op/s 2024-09-15T23:23:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:15 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1140592497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:15.460 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 143.7 --await-for 45.0' 2024-09-15T23:23:15.461 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 143.7 --await-for 45.0 2024-09-15T23:23:15.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:15 smithi067 ceph-mon[35665]: pgmap v579: 129 pgs: 129 active+clean; 1.5 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 5.9 MiB/s wr, 654 op/s 2024-09-15T23:23:15.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:15 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1140592497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:15.915 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:16.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.527+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/3219444163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c741082a0 msgr2=0x7f2c7410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.527+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/3219444163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c741082a0 0x7f2c7410ec40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c009ca0 tx=0x7f2c6c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.527+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/3219444163 shutdown_connections 2024-09-15T23:23:16.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.527+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/3219444163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c741082a0 0x7f2c7410ec40 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.527+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/3219444163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c741036e0 0x7f2c74107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.527+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/3219444163 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c74102d30 0x7f2c74103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.527+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/3219444163 >> 172.21.15.62:0/3219444163 conn(0x7f2c740fc7d0 msgr2=0x7f2c740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:16.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.527+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/3219444163 shutdown_connections 2024-09-15T23:23:16.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.527+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/3219444163 wait complete. 2024-09-15T23:23:16.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.528+0000 7f2c7c1f2640 1 Processor -- start 2024-09-15T23:23:16.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.528+0000 7f2c7c1f2640 1 -- start start 2024-09-15T23:23:16.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.528+0000 7f2c7c1f2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c74102d30 0x7f2c7410ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.528+0000 7f2c7c1f2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c741036e0 0x7f2c7410bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.529+0000 7f2c7c1f2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c741082a0 0x7f2c7410c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.529+0000 7f2c7978d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c741036e0 0x7f2c7410bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.529+0000 7f2c7978d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c741036e0 0x7f2c7410bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:54364/0 (socket says 172.21.15.62:54364) 2024-09-15T23:23:16.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.529+0000 7f2c7978d640 1 -- 172.21.15.62:0/2158801328 learned_addr learned my addr 172.21.15.62:0/2158801328 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:16.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.529+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c741082a0 0x7f2c7410c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.529+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2c74114290 con 0x7f2c74102d30 2024-09-15T23:23:16.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.529+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c74114110 con 0x7f2c741082a0 2024-09-15T23:23:16.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.529+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2c74114410 con 0x7f2c741036e0 2024-09-15T23:23:16.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.531+0000 7f2c7a78f640 1 -- 172.21.15.62:0/2158801328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c741036e0 msgr2=0x7f2c7410bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.531+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c741036e0 0x7f2c7410bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.531+0000 7f2c7a78f640 1 -- 172.21.15.62:0/2158801328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c74102d30 msgr2=0x7f2c7410ea90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:23:16.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.531+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c74102d30 0x7f2c7410ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.531+0000 7f2c7a78f640 1 -- 172.21.15.62:0/2158801328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7410c8f0 con 0x7f2c741082a0 2024-09-15T23:23:16.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.531+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c741082a0 0x7f2c7410c080 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c02ade0 tx=0x7f2c6c0319f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:16.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.532+0000 7f2c6affd640 1 -- 172.21.15.62:0/2158801328 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c003b70 con 0x7f2c741082a0 2024-09-15T23:23:16.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.532+0000 7f2c6affd640 1 -- 172.21.15.62:0/2158801328 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c6c003d10 con 0x7f2c741082a0 2024-09-15T23:23:16.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.532+0000 7f2c6affd640 1 -- 172.21.15.62:0/2158801328 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c02fc40 con 0x7f2c741082a0 2024-09-15T23:23:16.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.532+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c741ab010 con 0x7f2c741082a0 2024-09-15T23:23:16.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.532+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c741ab4d0 con 0x7f2c741082a0 2024-09-15T23:23:16.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.535+0000 7f2c6affd640 1 -- 172.21.15.62:0/2158801328 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2c6c004050 con 0x7f2c741082a0 2024-09-15T23:23:16.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.536+0000 7f2c7e5cb740 1 Processor -- start 2024-09-15T23:23:16.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.536+0000 7f2c6affd640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c600a3390 0x7f2c600a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.536+0000 7f2c6affd640 1 -- 172.21.15.62:0/2158801328 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2c6c0f0b50 con 0x7f2c741082a0 2024-09-15T23:23:16.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.536+0000 7f2c7e5cb740 1 -- start start 2024-09-15T23:23:16.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.536+0000 7f2c7978d640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c600a3390 0x7f2c600a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eacb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 0x557849eb7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557849e122a0 con 0x557849eb7940 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557849db9b10 con 0x557849eac760 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557849e0e260 con 0x557849ead080 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c79f8e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eacb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c79f8e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eacb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49168/0 (socket says 172.21.15.62:49168) 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c79f8e640 1 -- 172.21.15.62:0/3283330793 learned_addr learned my addr 172.21.15.62:0/3283330793 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7978d640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c600a3390 0x7f2c600a5850 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2c7410d300 tx=0x7f2c64005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7978d640 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 0x557849eb7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7978d640 1 -- 172.21.15.62:0/3283330793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 msgr2=0x557849eacb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:16.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7978d640 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eacb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7978d640 1 -- 172.21.15.62:0/3283330793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 msgr2=0x557849eb9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:16.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7978d640 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7978d640 1 -- 172.21.15.62:0/3283330793 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557849b83b90 con 0x557849ead080 2024-09-15T23:23:16.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.537+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:16.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.538+0000 7f2c7978d640 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 0x557849eb7400 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2c64004270 tx=0x7f2c64014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=579073713ad5d3ba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:16.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.538+0000 7f2c5ffff640 1 -- 172.21.15.62:0/3283330793 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6401e420 con 0x557849ead080 2024-09-15T23:23:16.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.538+0000 7f2c5ffff640 1 -- 172.21.15.62:0/3283330793 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c6401eaa0 con 0x557849ead080 2024-09-15T23:23:16.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.538+0000 7f2c5ffff640 1 -- 172.21.15.62:0/3283330793 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6401db30 con 0x557849ead080 2024-09-15T23:23:16.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.539+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3283330793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 msgr2=0x557849eb7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.539+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 0x557849eb7400 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2c64004270 tx=0x7f2c64014000 comp rx=0 tx=0).stop 2024-09-15T23:23:16.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.539+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3283330793 shutdown_connections 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.540+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.540+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 0x557849eb7400 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.540+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3283330793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eacb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.540+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3283330793 >> 172.21.15.62:0/3283330793 conn(0x557849ea8cf0 msgr2=0x557849ea90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.540+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3283330793 shutdown_connections 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.540+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3283330793 wait complete. 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.540+0000 7f2c7e5cb740 1 -- start start 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.540+0000 7f2c7e5cb740 1 Processor -- start 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eb4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 0x557849eb4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557849eba510 con 0x557849eb7940 2024-09-15T23:23:16.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557849d9a6e0 con 0x557849eac760 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557849eba690 con 0x557849ead080 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7e5cb740 10 client.0 _ll_register_callbacks cb 0x557849dbea80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7a78f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7a78f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54290/0 (socket says 172.21.15.62:54290) 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7a78f640 1 -- 172.21.15.62:0/4221626202 learned_addr learned my addr 172.21.15.62:0/4221626202 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.541+0000 7f2c7978d640 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 0x557849eb4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c7a78f640 1 -- 172.21.15.62:0/4221626202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 msgr2=0x557849eb4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 0x557849eb4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c7a78f640 1 -- 172.21.15.62:0/4221626202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 msgr2=0x557849eb4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c79f8e640 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eb4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eb4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c7a78f640 1 -- 172.21.15.62:0/4221626202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557849ec3a00 con 0x557849eb7940 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c79f8e640 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eb4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:16.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb5290 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c006010 tx=0x7f2c6c0bcb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da18894b997a41 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/4221626202 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c0bcf10 con 0x557849eb7940 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/4221626202 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2c6c004730 con 0x557849eb7940 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.542+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/4221626202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c6401eaa0 con 0x557849eb7940 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.543+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/4221626202 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2c6c0c12c0 con 0x557849eb7940 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.543+0000 7f2c7e5cb740 10 client.17130 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.543+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/4221626202 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557849eba510 con 0x557849eb7940 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/4221626202 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c6c036040 con 0x557849eb7940 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/4221626202 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f2c6c0c56f0 con 0x557849eb7940 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c7e5cb740 10 client.17130 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c7e5cb740 10 client.17130 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c7e5cb740 20 client.17130 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557849ec7820 0x557849ec9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c7e5cb740 4 client.17130 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:16.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/4221626202 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2c6401eaa0 con 0x557849ec7820 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c79f8e640 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557849ec7820 0x557849ec9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c5d7fa640 10 client.17130 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.544+0000 7f2c79f8e640 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557849ec7820 0x557849ec9c10 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cb9905a8282cb3f server_cookie=ea5572177618c3cc in_seq=0 out_seq=0 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.545+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/4221626202 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f2c6401eaa0 con 0x557849ec7820 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.545+0000 7f2c5d7fa640 10 client.17130 handle_command_reply: tid=0 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.545+0000 7f2c7e5cb740 1 client.17130 shutdown 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.545+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/4221626202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 msgr2=0x557849eb5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.545+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb5290 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c006010 tx=0x7f2c6c0bcb10 comp rx=0 tx=0).stop 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/4221626202 shutdown_connections 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557849ec7820 0x557849ec9c10 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c5d7fa640 0 client.17130 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849eb5290 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849ead080 0x557849eb4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/4221626202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849eac760 0x557849eb4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/4221626202 >> 172.21.15.62:0/4221626202 conn(0x557849ea8cf0 msgr2=0x557849eae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:16.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/4221626202 shutdown_connections 2024-09-15T23:23:16.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/4221626202 wait complete. 2024-09-15T23:23:16.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.546+0000 7f2c7e5cb740 20 client.17130 trim_cache size 0 max 16384 2024-09-15T23:23:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.576+0000 7f2c7e5cb740 1 Processor -- start 2024-09-15T23:23:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.579+0000 7f2c7e5cb740 1 -- start start 2024-09-15T23:23:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.580+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eac760 0x557849eb66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.580+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 0x557849eaf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.580+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849ecf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.581+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741b7b00 con 0x557849eb7940 2024-09-15T23:23:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.581+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741b53d0 con 0x557849ead080 2024-09-15T23:23:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.581+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741b3220 con 0x557849eac760 2024-09-15T23:23:16.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.584+0000 7f2c7a78f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849ecf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.584+0000 7f2c7a78f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849ecf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54294/0 (socket says 172.21.15.62:54294) 2024-09-15T23:23:16.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.584+0000 7f2c7a78f640 1 -- 172.21.15.62:0/3134293771 learned_addr learned my addr 172.21.15.62:0/3134293771 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:16.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.585+0000 7f2c79f8e640 1 --2- 172.21.15.62:0/3134293771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eac760 0x557849eb66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.585+0000 7f2c7978d640 1 --2- 172.21.15.62:0/3134293771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 0x557849eaf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.586+0000 7f2c7a78f640 1 -- 172.21.15.62:0/3134293771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eac760 msgr2=0x557849eb66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:16.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.586+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/3134293771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eac760 0x557849eb66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.586+0000 7f2c7a78f640 1 -- 172.21.15.62:0/3134293771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 msgr2=0x557849eaf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:16.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.586+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/3134293771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 0x557849eaf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.586+0000 7f2c7a78f640 1 -- 172.21.15.62:0/3134293771 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557849ea8b20 con 0x557849eb7940 2024-09-15T23:23:16.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.587+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/3134293771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849ecf4f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c009bf0 tx=0x7f2c6c0bb420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c947fe9408c66a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:16.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.590+0000 7f2c5f7fe640 1 -- 172.21.15.62:0/3134293771 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c0d8070 con 0x557849eb7940 2024-09-15T23:23:16.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.590+0000 7f2c5f7fe640 1 -- 172.21.15.62:0/3134293771 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c6c036040 con 0x557849eb7940 2024-09-15T23:23:16.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.590+0000 7f2c5f7fe640 1 -- 172.21.15.62:0/3134293771 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c0b9580 con 0x557849eb7940 2024-09-15T23:23:16.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.594+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3134293771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 msgr2=0x557849ecf4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.594+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3134293771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849ecf4f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c009bf0 tx=0x7f2c6c0bb420 comp rx=0 tx=0).stop 2024-09-15T23:23:16.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.601+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3134293771 shutdown_connections 2024-09-15T23:23:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.601+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3134293771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eb7940 0x557849ecf4f0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.601+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3134293771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 0x557849eaf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.601+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3134293771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eac760 0x557849eb66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.601+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3134293771 >> 172.21.15.62:0/3134293771 conn(0x557849ea8cf0 msgr2=0x557849ea7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:16.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.605+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3134293771 shutdown_connections 2024-09-15T23:23:16.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.609+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3134293771 wait complete. 2024-09-15T23:23:16.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.609+0000 7f2c7e5cb740 1 -- start start 2024-09-15T23:23:16.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.609+0000 7f2c7e5cb740 1 Processor -- start 2024-09-15T23:23:16.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eac760 0x557849ec0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 0x557849ebc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7e5cb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eb7940 0x557849ebca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741b3890 con 0x557849eac760 2024-09-15T23:23:16.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741b4d00 con 0x557849ead080 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7e5cb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2c741b3f60 con 0x557849eb7940 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7978d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 0x557849ebc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7978d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 0x557849ebc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49208/0 (socket says 172.21.15.62:49208) 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7978d640 1 -- 172.21.15.62:0/3184226916 learned_addr learned my addr 172.21.15.62:0/3184226916 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eb7940 0x557849ebca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.610+0000 7f2c7e5cb740 10 client.0 _ll_register_callbacks cb 0x7f2c6c0bcce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.611+0000 7f2c7a78f640 1 -- 172.21.15.62:0/3184226916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 msgr2=0x557849ebc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.611+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 0x557849ebc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.611+0000 7f2c7a78f640 1 -- 172.21.15.62:0/3184226916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eac760 msgr2=0x557849ec0120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.611+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eac760 0x557849ec0120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.611+0000 7f2c7a78f640 1 -- 172.21.15.62:0/3184226916 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557849ea4260 con 0x557849eb7940 2024-09-15T23:23:16.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.617+0000 7f2c7a78f640 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eb7940 0x557849ebca50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0ee640 tx=0x7f2c6c004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7ce82512e64de53 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:16.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.618+0000 7f2c5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:23:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.622+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/3184226916 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c0d8040 con 0x557849eb7940 2024-09-15T23:23:16.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.622+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/3184226916 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2c6c0b9580 con 0x557849eb7940 2024-09-15T23:23:16.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.623+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3184226916 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c6c036040 con 0x557849eb7940 2024-09-15T23:23:16.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.626+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/3184226916 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2c6c0bc360 con 0x557849eb7940 2024-09-15T23:23:16.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.627+0000 7f2c7e5cb740 10 client.26521 fetch_fsmap learned FSMap version 32 2024-09-15T23:23:16.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.627+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3184226916 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557849ea4260 con 0x557849eb7940 2024-09-15T23:23:16.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.628+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/3184226916 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c6c0bc500 con 0x557849eb7940 2024-09-15T23:23:16.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.628+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/3184226916 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 32) ==== 8358+0+0 (secure 0 0 0) 0x7f2c6c0c2450 con 0x557849eb7940 2024-09-15T23:23:16.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.629+0000 7f2c7e5cb740 10 client.26521 fetch_fsmap finished waiting for FSMap version 32 2024-09-15T23:23:16.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.629+0000 7f2c7e5cb740 10 client.26521 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:23:16.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.629+0000 7f2c7e5cb740 20 client.26521 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:16.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.629+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557849ec7820 0x557849f355b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:16.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.629+0000 7f2c7e5cb740 4 client.26521 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.7, "await_for": 45.0}] 2024-09-15T23:23:16.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.629+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3184226916 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 143.7, "await_for": 45.0}) -- 0x7f2c6c036040 con 0x557849ec7820 2024-09-15T23:23:16.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.629+0000 7f2c79f8e640 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557849ec7820 0x557849f355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.638+0000 7f2c79f8e640 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557849ec7820 0x557849f355b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9593a71c4b3448b6 server_cookie=2ee48ec40320440e in_seq=0 out_seq=0 2024-09-15T23:23:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.638+0000 7f2c5d7fa640 10 client.26521 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:16.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c5d7fa640 1 -- 172.21.15.62:0/3184226916 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f2c6c036040 con 0x557849ec7820 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c5d7fa640 10 client.26521 handle_command_reply: tid=0 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 client.26521 shutdown 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3184226916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eb7940 msgr2=0x557849ebca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eb7940 0x557849ebca50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0ee640 tx=0x7f2c6c004960 comp rx=0 tx=0).stop 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3184226916 shutdown_connections 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557849ec7820 0x557849f355b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557849eb7940 0x557849ebca50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557849ead080 0x557849ebc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c5d7fa640 0 client.26521 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 --2- 172.21.15.62:0/3184226916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557849eac760 0x557849ec0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3184226916 >> 172.21.15.62:0/3184226916 conn(0x557849ea8cf0 msgr2=0x557849ea7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:16.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3184226916 shutdown_connections 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 1 -- 172.21.15.62:0/3184226916 wait complete. 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.991+0000 7f2c7e5cb740 20 client.26521 trim_cache size 0 max 16384 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 32, 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 26, 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stdout: "87dc2717": { 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 26, 2024-09-15T23:23:16.994 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 143.7, 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:23:16.995 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:16.996 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:16.996 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:16.996 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:16.996 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:16.996 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:23:16.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c600a3390 msgr2=0x7f2c600a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c600a3390 0x7f2c600a5850 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2c7410d300 tx=0x7f2c64005f90 comp rx=0 tx=0).stop 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c741082a0 msgr2=0x7f2c7410c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c741082a0 0x7f2c7410c080 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c02ade0 tx=0x7f2c6c0319f0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 shutdown_connections 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c600a3390 0x7f2c600a5850 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c741082a0 0x7f2c7410c080 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c741036e0 0x7f2c7410bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 --2- 172.21.15.62:0/2158801328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c74102d30 0x7f2c7410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 >> 172.21.15.62:0/2158801328 conn(0x7f2c740fc7d0 msgr2=0x7f2c74105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 shutdown_connections 2024-09-15T23:23:16.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:16.992+0000 7f2c7c1f2640 1 -- 172.21.15.62:0/2158801328 wait complete. 2024-09-15T23:23:17.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:17 smithi148 ceph-mon[35610]: pgmap v580: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.9 MiB/s wr, 631 op/s 2024-09-15T23:23:17.364 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:17 smithi062 ceph-mon[30949]: pgmap v580: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.9 MiB/s wr, 631 op/s 2024-09-15T23:23:17.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:17 smithi067 ceph-mon[35665]: pgmap v580: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 5.9 MiB/s wr, 631 op/s 2024-09-15T23:23:17.797 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '87dc2717', quiesce took 5.6 seconds. Will release after: 18.1 2024-09-15T23:23:19.326 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:19 smithi067 ceph-mon[35665]: pgmap v581: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.9 MiB/s wr, 720 op/s 2024-09-15T23:23:19.327 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:19 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:23:19.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:19 smithi062 ceph-mon[30949]: pgmap v581: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.9 MiB/s wr, 720 op/s 2024-09-15T23:23:19.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:19 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:23:19.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:19 smithi148 ceph-mon[35610]: pgmap v581: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 6.9 MiB/s wr, 720 op/s 2024-09-15T23:23:19.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:19 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:23:19.572 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 420.0 s, 673.0 tps, lat 66.395 ms stddev 80.293 2024-09-15T23:23:21.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:21 smithi062 ceph-mon[30949]: pgmap v582: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 6.1 MiB/s wr, 681 op/s 2024-09-15T23:23:21.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:21 smithi067 ceph-mon[35665]: pgmap v582: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 6.1 MiB/s wr, 681 op/s 2024-09-15T23:23:21.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:21 smithi148 ceph-mon[35610]: pgmap v582: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 6.1 MiB/s wr, 681 op/s 2024-09-15T23:23:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.470+0000 7f3298ee4640 1 -- 172.21.15.62:0/900542361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32941078a0 msgr2=0x7f3294109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.470+0000 7f3298ee4640 1 --2- 172.21.15.62:0/900542361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32941078a0 0x7f3294109cb0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f328c009ca0 tx=0x7f328c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.471+0000 7f3298ee4640 1 -- 172.21.15.62:0/900542361 shutdown_connections 2024-09-15T23:23:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.471+0000 7f3298ee4640 1 --2- 172.21.15.62:0/900542361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32941078a0 0x7f3294109cb0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.471+0000 7f3298ee4640 1 --2- 172.21.15.62:0/900542361 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3294104f70 0x7f3294107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.471+0000 7f3298ee4640 1 --2- 172.21.15.62:0/900542361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3294102640 0x7f3294104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.471+0000 7f3298ee4640 1 -- 172.21.15.62:0/900542361 >> 172.21.15.62:0/900542361 conn(0x7f32940fc1e0 msgr2=0x7f32940fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.471+0000 7f3298ee4640 1 -- 172.21.15.62:0/900542361 shutdown_connections 2024-09-15T23:23:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.471+0000 7f3298ee4640 1 -- 172.21.15.62:0/900542361 wait complete. 2024-09-15T23:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3298ee4640 1 Processor -- start 2024-09-15T23:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3298ee4640 1 -- start start 2024-09-15T23:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3298ee4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3294102640 0x7f329419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3298ee4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3294104f70 0x7f329419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3298ee4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32941078a0 0x7f32941a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3298ee4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f329410d800 con 0x7f3294104f70 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3298ee4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f329410d680 con 0x7f32941078a0 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3298ee4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f329410d980 con 0x7f3294102640 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3291d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3294104f70 0x7f329419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3291d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3294104f70 0x7f329419a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52700/0 (socket says 172.21.15.62:52700) 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3291d9b640 1 -- 172.21.15.62:0/644350305 learned_addr learned my addr 172.21.15.62:0/644350305 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f329259c640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3294102640 0x7f329419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3292d9d640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32941078a0 0x7f32941a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3291d9b640 1 -- 172.21.15.62:0/644350305 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3294102640 msgr2=0x7f329419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3291d9b640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3294102640 0x7f329419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3291d9b640 1 -- 172.21.15.62:0/644350305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32941078a0 msgr2=0x7f32941a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3291d9b640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32941078a0 0x7f32941a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.476 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3291d9b640 1 -- 172.21.15.62:0/644350305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32941a1c70 con 0x7f3294104f70 2024-09-15T23:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f329259c640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3294102640 0x7f329419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3292d9d640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32941078a0 0x7f32941a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.472+0000 7f3291d9b640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3294104f70 0x7f329419a590 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3288005fd0 tx=0x7f328800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.473+0000 7f32837fe640 1 -- 172.21.15.62:0/644350305 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288017070 con 0x7f3294104f70 2024-09-15T23:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.473+0000 7f32837fe640 1 -- 172.21.15.62:0/644350305 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3288009c30 con 0x7f3294104f70 2024-09-15T23:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.473+0000 7f3298ee4640 1 -- 172.21.15.62:0/644350305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32941a1f60 con 0x7f3294104f70 2024-09-15T23:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.473+0000 7f32837fe640 1 -- 172.21.15.62:0/644350305 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3288010460 con 0x7f3294104f70 2024-09-15T23:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.473+0000 7f3298ee4640 1 -- 172.21.15.62:0/644350305 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32941a2420 con 0x7f3294104f70 2024-09-15T23:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.475+0000 7f32837fe640 1 -- 172.21.15.62:0/644350305 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3288003e70 con 0x7f3294104f70 2024-09-15T23:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.475+0000 7f329bc87740 1 Processor -- start 2024-09-15T23:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f329bc87740 1 -- start start 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f329bc87740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089369a0 0x560108936da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f329bc87740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089372e0 0x56010893bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f32837fe640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f327c0a32f0 0x7f327c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f329bc87740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 0x56010893e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f329bc87740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56010889a2f0 con 0x5601089369a0 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f329bc87740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560108842270 con 0x5601089372e0 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f329bc87740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5601088962b0 con 0x56010893c370 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f3292d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089369a0 0x560108936da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f3291d9b640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f327c0a32f0 0x7f327c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f3292d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089369a0 0x560108936da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52712/0 (socket says 172.21.15.62:52712) 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f3292d9d640 1 -- 172.21.15.62:0/507051392 learned_addr learned my addr 172.21.15.62:0/507051392 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f32837fe640 1 -- 172.21.15.62:0/644350305 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f32880cfa00 con 0x7f3294104f70 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f329259c640 1 --2- 172.21.15.62:0/507051392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089372e0 0x56010893bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.476+0000 7f3292d9d640 1 --2- 172.21.15.62:0/507051392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 0x56010893e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3292d9d640 1 -- 172.21.15.62:0/507051392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 msgr2=0x56010893e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3292d9d640 1 --2- 172.21.15.62:0/507051392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 0x56010893e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3292d9d640 1 -- 172.21.15.62:0/507051392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089372e0 msgr2=0x56010893bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3292d9d640 1 --2- 172.21.15.62:0/507051392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089372e0 0x56010893bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3292d9d640 1 -- 172.21.15.62:0/507051392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560108841340 con 0x5601089369a0 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3291d9b640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f327c0a32f0 0x7f327c0a57b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f32880980a0 tx=0x7f3288084f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3292d9d640 1 --2- 172.21.15.62:0/507051392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089369a0 0x560108936da0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f328c004af0 tx=0x7f328c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1ce23cf4215851 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3280ff9640 1 -- 172.21.15.62:0/507051392 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c03e070 con 0x5601089369a0 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3280ff9640 1 -- 172.21.15.62:0/507051392 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f328c036050 con 0x5601089369a0 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.477+0000 7f3280ff9640 1 -- 172.21.15.62:0/507051392 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c004380 con 0x5601089369a0 2024-09-15T23:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 -- 172.21.15.62:0/507051392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089369a0 msgr2=0x560108936da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 --2- 172.21.15.62:0/507051392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089369a0 0x560108936da0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f328c004af0 tx=0x7f328c005ef0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 -- 172.21.15.62:0/507051392 shutdown_connections 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 --2- 172.21.15.62:0/507051392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 0x56010893e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 --2- 172.21.15.62:0/507051392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089372e0 0x56010893bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 --2- 172.21.15.62:0/507051392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089369a0 0x560108936da0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 -- 172.21.15.62:0/507051392 >> 172.21.15.62:0/507051392 conn(0x560108932010 msgr2=0x560108932410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 -- 172.21.15.62:0/507051392 shutdown_connections 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 -- 172.21.15.62:0/507051392 wait complete. 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 -- start start 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 Processor -- start 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x560108940a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.478+0000 7f329bc87740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601089372e0 0x560108940f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f329bc87740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x560108947f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f329bc87740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56010893f160 con 0x56010893c370 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f329bc87740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560108823290 con 0x5601089369a0 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f329bc87740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56010893f2e0 con 0x5601089372e0 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f329bc87740 10 client.0 _ll_register_callbacks cb 0x560108854140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x560108947f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x560108947f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52718/0 (socket says 172.21.15.62:52718) 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 -- 172.21.15.62:0/3433248486 learned_addr learned my addr 172.21.15.62:0/3433248486 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x560108940a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 -- 172.21.15.62:0/3433248486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601089372e0 msgr2=0x560108940f50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601089372e0 0x560108940f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 -- 172.21.15.62:0/3433248486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 msgr2=0x560108940a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x560108940a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 -- 172.21.15.62:0/3433248486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601085ad210 con 0x56010893c370 2024-09-15T23:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3292d9d640 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x560108947f90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f328c000f80 tx=0x7f328c031ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3d927557bf3581 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3279ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3279ffb640 1 -- 172.21.15.62:0/3433248486 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c03e040 con 0x56010893c370 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3279ffb640 1 -- 172.21.15.62:0/3433248486 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f328c003970 con 0x56010893c370 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f329bc87740 1 -- 172.21.15.62:0/3433248486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f328c036050 con 0x56010893c370 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.479+0000 7f3279ffb640 1 -- 172.21.15.62:0/3433248486 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f328c003b70 con 0x56010893c370 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f329bc87740 10 client.17145 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f329bc87740 1 -- 172.21.15.62:0/3433248486 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560108679620 con 0x56010893c370 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f3279ffb640 1 -- 172.21.15.62:0/3433248486 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f328c036050 con 0x56010893c370 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f3279ffb640 1 -- 172.21.15.62:0/3433248486 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f328c047020 con 0x56010893c370 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f329bc87740 10 client.17145 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f329bc87740 10 client.17145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f329bc87740 20 client.17145 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f329bc87740 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56010894c380 0x56010894e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f329bc87740 4 client.17145 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f329bc87740 1 -- 172.21.15.62:0/3433248486 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601088a7a20 con 0x56010894c380 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.480+0000 7f329259c640 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56010894c380 0x56010894e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329259c640 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56010894c380 0x56010894e770 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d42ee040d2517fed server_cookie=a1eb16229a4ac0c0 in_seq=0 out_seq=0 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f3279ffb640 10 client.17145 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f3279ffb640 1 -- 172.21.15.62:0/3433248486 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5601088a7a20 con 0x56010894c380 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f3279ffb640 10 client.17145 handle_command_reply: tid=0 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 client.17145 shutdown 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 -- 172.21.15.62:0/3433248486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 msgr2=0x560108947f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x560108947f90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f328c000f80 tx=0x7f328c031ed0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 -- 172.21.15.62:0/3433248486 shutdown_connections 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56010894c380 0x56010894e770 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x560108947f90 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601089372e0 0x560108940f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f3279ffb640 0 client.17145 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 --2- 172.21.15.62:0/3433248486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x560108940a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 -- 172.21.15.62:0/3433248486 >> 172.21.15.62:0/3433248486 conn(0x560108932010 msgr2=0x560108933820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.481+0000 7f329bc87740 1 -- 172.21.15.62:0/3433248486 shutdown_connections 2024-09-15T23:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.482+0000 7f329bc87740 1 -- 172.21.15.62:0/3433248486 wait complete. 2024-09-15T23:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.482+0000 7f329bc87740 20 client.17145 trim_cache size 0 max 16384 2024-09-15T23:23:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329bc87740 1 Processor -- start 2024-09-15T23:23:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329bc87740 1 -- start start 2024-09-15T23:23:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329bc87740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x5601089309a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329bc87740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089372e0 0x560108947f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329bc87740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 0x560108930ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329bc87740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f32941af190 con 0x5601089372e0 2024-09-15T23:23:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329bc87740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f32941aca60 con 0x5601089369a0 2024-09-15T23:23:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329bc87740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f32941aa8b0 con 0x56010893c370 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089372e0 0x560108947f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089372e0 0x560108947f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52720/0 (socket says 172.21.15.62:52720) 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f329259c640 1 -- 172.21.15.62:0/2947155367 learned_addr learned my addr 172.21.15.62:0/2947155367 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.496+0000 7f3292d9d640 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x5601089309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f3292d9d640 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 0x560108930ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f329259c640 1 -- 172.21.15.62:0/2947155367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 msgr2=0x560108930ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f329259c640 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 0x560108930ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f329259c640 1 -- 172.21.15.62:0/2947155367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 msgr2=0x5601089309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f329259c640 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x5601089309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f329259c640 1 -- 172.21.15.62:0/2947155367 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56010894edf0 con 0x5601089372e0 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f3292d9d640 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x5601089309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f329259c640 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089372e0 0x560108947f00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x560108937d10 tx=0x7f328400b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0e268eb7d3ba70 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f327bfff640 1 -- 172.21.15.62:0/2947155367 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284004030 con 0x5601089372e0 2024-09-15T23:23:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f327bfff640 1 -- 172.21.15.62:0/2947155367 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32840041d0 con 0x5601089372e0 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.497+0000 7f327bfff640 1 -- 172.21.15.62:0/2947155367 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284003e60 con 0x5601089372e0 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 -- 172.21.15.62:0/2947155367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089372e0 msgr2=0x560108947f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089372e0 0x560108947f00 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x560108937d10 tx=0x7f328400b5a0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 -- 172.21.15.62:0/2947155367 shutdown_connections 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56010893c370 0x560108930ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601089372e0 0x560108947f00 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 --2- 172.21.15.62:0/2947155367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x5601089309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 -- 172.21.15.62:0/2947155367 >> 172.21.15.62:0/2947155367 conn(0x560108932010 msgr2=0x5601089384c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 -- 172.21.15.62:0/2947155367 shutdown_connections 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 -- 172.21.15.62:0/2947155367 wait complete. 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 -- start start 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 Processor -- start 2024-09-15T23:23:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x560108945310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601089372e0 0x560108945850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x5601089467f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f32941aaf20 con 0x56010893c370 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f32941ac390 con 0x5601089369a0 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f32941ab5f0 con 0x5601089372e0 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f329bc87740 10 client.0 _ll_register_callbacks cb 0x7f328400b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f3292d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x5601089467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f3292d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x5601089467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52726/0 (socket says 172.21.15.62:52726) 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f3292d9d640 1 -- 172.21.15.62:0/2854430665 learned_addr learned my addr 172.21.15.62:0/2854430665 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.498+0000 7f3292d9d640 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x560108945310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f329259c640 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601089372e0 0x560108945850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3292d9d640 1 -- 172.21.15.62:0/2854430665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601089372e0 msgr2=0x560108945850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:22.506 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3292d9d640 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601089372e0 0x560108945850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.506 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3292d9d640 1 -- 172.21.15.62:0/2854430665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 msgr2=0x560108945310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:22.506 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3292d9d640 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x560108945310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3292d9d640 1 -- 172.21.15.62:0/2854430665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560108946e00 con 0x56010893c370 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3292d9d640 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x5601089467f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f328c006010 tx=0x7f328c02af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca820aa9153c123 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3279ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3279ffb640 1 -- 172.21.15.62:0/2854430665 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c03e070 con 0x56010893c370 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3279ffb640 1 -- 172.21.15.62:0/2854430665 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f328c031760 con 0x56010893c370 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f329bc87740 1 -- 172.21.15.62:0/2854430665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32840041d0 con 0x56010893c370 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3279ffb640 1 -- 172.21.15.62:0/2854430665 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f328c04d040 con 0x56010893c370 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f329bc87740 10 client.17151 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f329bc87740 1 -- 172.21.15.62:0/2854430665 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x560108948080 con 0x56010893c370 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.499+0000 7f3279ffb640 1 -- 172.21.15.62:0/2854430665 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32840041d0 con 0x56010893c370 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f3279ffb640 1 -- 172.21.15.62:0/2854430665 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f328c063060 con 0x56010893c370 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f329bc87740 10 client.17151 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f329bc87740 10 client.17151 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f329bc87740 20 client.17151 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f329bc87740 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5601089b7490 0x5601089b98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f329bc87740 4 client.17151 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f329bc87740 1 -- 172.21.15.62:0/2854430665 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5601088a1d70 con 0x5601089b7490 2024-09-15T23:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f329259c640 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5601089b7490 0x5601089b98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f3279ffb640 10 client.17151 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.500+0000 7f329259c640 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5601089b7490 0x5601089b98b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=498d2cb080b160c server_cookie=f305cdf8b817c065 in_seq=0 out_seq=0 2024-09-15T23:23:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.514+0000 7f3279ffb640 1 -- 172.21.15.62:0/2854430665 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5601088a1d70 con 0x5601089b7490 2024-09-15T23:23:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.514+0000 7f3279ffb640 10 client.17151 handle_command_reply: tid=0 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.514+0000 7f329bc87740 1 client.17151 shutdown 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 -- 172.21.15.62:0/2854430665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 msgr2=0x5601089467f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x5601089467f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f328c006010 tx=0x7f328c02af60 comp rx=0 tx=0).stop 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 -- 172.21.15.62:0/2854430665 shutdown_connections 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5601089b7490 0x5601089b98b0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56010893c370 0x5601089467f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5601089372e0 0x560108945850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f3279ffb640 0 client.17151 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 --2- 172.21.15.62:0/2854430665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5601089369a0 0x560108945310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 -- 172.21.15.62:0/2854430665 >> 172.21.15.62:0/2854430665 conn(0x560108932010 msgr2=0x560108932e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 -- 172.21.15.62:0/2854430665 shutdown_connections 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 1 -- 172.21.15.62:0/2854430665 wait complete. 2024-09-15T23:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f329bc87740 20 client.17151 trim_cache size 0 max 16384 2024-09-15T23:23:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:23:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:23:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:23:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:23:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f3298ee4640 1 -- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f327c0a32f0 msgr2=0x7f327c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.515+0000 7f3298ee4640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f327c0a32f0 0x7f327c0a57b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f32880980a0 tx=0x7f3288084f90 comp rx=0 tx=0).stop 2024-09-15T23:23:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 -- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3294104f70 msgr2=0x7f329419a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3294104f70 0x7f329419a590 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3288005fd0 tx=0x7f328800dde0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 -- 172.21.15.62:0/644350305 shutdown_connections 2024-09-15T23:23:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f327c0a32f0 0x7f327c0a57b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32941078a0 0x7f32941a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3294104f70 0x7f329419a590 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 --2- 172.21.15.62:0/644350305 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3294102640 0x7f329419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 -- 172.21.15.62:0/644350305 >> 172.21.15.62:0/644350305 conn(0x7f32940fc1e0 msgr2=0x7f3294101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 -- 172.21.15.62:0/644350305 shutdown_connections 2024-09-15T23:23:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:22.516+0000 7f3298ee4640 1 -- 172.21.15.62:0/644350305 wait complete. 2024-09-15T23:23:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:23 smithi062 ceph-mon[30949]: pgmap v583: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 5.1 MiB/s wr, 566 op/s 2024-09-15T23:23:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:23 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:23:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:23 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:23:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:23 smithi067 ceph-mon[35665]: pgmap v583: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 5.1 MiB/s wr, 566 op/s 2024-09-15T23:23:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:23 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:23:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:23 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:23:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:23 smithi148 ceph-mon[35610]: pgmap v583: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 5.1 MiB/s wr, 566 op/s 2024-09-15T23:23:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:23 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:23:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:23 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:23:24.422 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:24 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:23:24.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:24 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:23:24.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:24 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:23:24.573 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 425.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:23:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:25 smithi062 ceph-mon[30949]: pgmap v584: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 4.2 MiB/s wr, 417 op/s 2024-09-15T23:23:25.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:25 smithi067 ceph-mon[35665]: pgmap v584: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 4.2 MiB/s wr, 417 op/s 2024-09-15T23:23:25.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:25 smithi148 ceph-mon[35610]: pgmap v584: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 4.2 MiB/s wr, 417 op/s 2024-09-15T23:23:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:27 smithi062 ceph-mon[30949]: pgmap v585: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 3.2 MiB/s wr, 284 op/s 2024-09-15T23:23:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:27 smithi067 ceph-mon[35665]: pgmap v585: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 3.2 MiB/s wr, 284 op/s 2024-09-15T23:23:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:27 smithi148 ceph-mon[35610]: pgmap v585: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 3.2 MiB/s wr, 284 op/s 2024-09-15T23:23:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:29 smithi062 ceph-mon[30949]: pgmap v586: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-09-15T23:23:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:29 smithi067 ceph-mon[35665]: pgmap v586: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-09-15T23:23:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:29 smithi148 ceph-mon[35610]: pgmap v586: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 2.9 MiB/s wr, 271 op/s 2024-09-15T23:23:29.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 430.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:23:31.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:31 smithi062 ceph-mon[30949]: pgmap v587: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 3.1 MiB/s wr, 317 op/s 2024-09-15T23:23:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:31 smithi067 ceph-mon[35665]: pgmap v587: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 3.1 MiB/s wr, 317 op/s 2024-09-15T23:23:31.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:31 smithi148 ceph-mon[35610]: pgmap v587: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 3.1 MiB/s wr, 317 op/s 2024-09-15T23:23:33.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:33 smithi062 ceph-mon[30949]: pgmap v588: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.5 MiB/s wr, 353 op/s 2024-09-15T23:23:33.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:33 smithi067 ceph-mon[35665]: pgmap v588: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.5 MiB/s wr, 353 op/s 2024-09-15T23:23:33.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:33 smithi148 ceph-mon[35610]: pgmap v588: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 3.5 MiB/s wr, 353 op/s 2024-09-15T23:23:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 435.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:23:35.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:35 smithi062 ceph-mon[30949]: pgmap v589: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.3 MiB/s wr, 328 op/s 2024-09-15T23:23:35.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:35 smithi067 ceph-mon[35665]: pgmap v589: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.3 MiB/s wr, 328 op/s 2024-09-15T23:23:35.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:35 smithi148 ceph-mon[35610]: pgmap v589: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.3 MiB/s wr, 328 op/s 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.615+0000 7ff3cc791640 1 -- 172.21.15.62:0/358098720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 msgr2=0x7ff3c4100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.615+0000 7ff3cc791640 1 --2- 172.21.15.62:0/358098720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 0x7ff3c4100e60 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009900 tx=0x7ff3c002a410 comp rx=0 tx=0).stop 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.615+0000 7ff3cc791640 1 -- 172.21.15.62:0/358098720 shutdown_connections 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.615+0000 7ff3cc791640 1 --2- 172.21.15.62:0/358098720 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3c4102610 0x7ff3c4108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.615+0000 7ff3cc791640 1 --2- 172.21.15.62:0/358098720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c4101c50 0x7ff3c41020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.615+0000 7ff3cc791640 1 --2- 172.21.15.62:0/358098720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 0x7ff3c4100e60 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.615+0000 7ff3cc791640 1 -- 172.21.15.62:0/358098720 >> 172.21.15.62:0/358098720 conn(0x7ff3c40fc1e0 msgr2=0x7ff3c40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.615+0000 7ff3cc791640 1 -- 172.21.15.62:0/358098720 shutdown_connections 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.615+0000 7ff3cc791640 1 -- 172.21.15.62:0/358098720 wait complete. 2024-09-15T23:23:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3cc791640 1 Processor -- start 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3cc791640 1 -- start start 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3cc791640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 0x7ff3c4197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3cc791640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3c4101c50 0x7ff3c4198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3cc791640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c4102610 0x7ff3c419f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3cc791640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c410c6c0 con 0x7ff3c4100a60 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3cc791640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c410c540 con 0x7ff3c4102610 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3cc791640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c410c840 con 0x7ff3c4101c50 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3ca52d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 0x7ff3c4197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3ca52d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 0x7ff3c4197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60094/0 (socket says 172.21.15.62:60094) 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.616+0000 7ff3ca52d640 1 -- 172.21.15.62:0/4134340636 learned_addr learned my addr 172.21.15.62:0/4134340636 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3c9d2c640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3c4101c50 0x7ff3c4198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c4102610 0x7ff3c419f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3ca52d640 1 -- 172.21.15.62:0/4134340636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3c4101c50 msgr2=0x7ff3c4198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3c4101c50 0x7ff3c4198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3ca52d640 1 -- 172.21.15.62:0/4134340636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c4102610 msgr2=0x7ff3c419f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c4102610 0x7ff3c419f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3ca52d640 1 -- 172.21.15.62:0/4134340636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c419fa50 con 0x7ff3c4100a60 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 0x7ff3c4197e30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0031610 tx=0x7ff3c00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3bb7fe640 1 -- 172.21.15.62:0/4134340636 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c003f030 con 0x7ff3c4100a60 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3cc791640 1 -- 172.21.15.62:0/4134340636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c419fce0 con 0x7ff3c4100a60 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3bb7fe640 1 -- 172.21.15.62:0/4134340636 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c0003ce0 con 0x7ff3c4100a60 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3bb7fe640 1 -- 172.21.15.62:0/4134340636 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0035070 con 0x7ff3c4100a60 2024-09-15T23:23:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.617+0000 7ff3cc791640 1 -- 172.21.15.62:0/4134340636 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c41a01d0 con 0x7ff3c4100a60 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.620+0000 7ff3bb7fe640 1 -- 172.21.15.62:0/4134340636 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff3c0042030 con 0x7ff3c4100a60 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.620+0000 7ff3cf60b740 1 Processor -- start 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.620+0000 7ff3cf60b740 1 -- start start 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e59a0 0x555c2a1e5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 0x555c2a1eada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 0x555c2a1ed830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555c2a1492f0 con 0x555c2a1e59a0 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555c2a0f1270 con 0x555c2a1eb370 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555c2a1452b0 con 0x555c2a1e62e0 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e59a0 0x555c2a1e5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3bb7fe640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3b00a32f0 0x7ff3b00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e59a0 0x555c2a1e5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60100/0 (socket says 172.21.15.62:60100) 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 -- 172.21.15.62:0/1791017584 learned_addr learned my addr 172.21.15.62:0/1791017584 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3c9d2c640 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 0x555c2a1ed830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 0x555c2a1eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3b00a32f0 0x7ff3b00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3bb7fe640 1 -- 172.21.15.62:0/4134340636 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff3c00f0230 con 0x7ff3c4100a60 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 -- 172.21.15.62:0/1791017584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 msgr2=0x555c2a1eada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 0x555c2a1eada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 -- 172.21.15.62:0/1791017584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 msgr2=0x555c2a1ed830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 0x555c2a1ed830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 -- 172.21.15.62:0/1791017584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c2a0f0340 con 0x555c2a1e59a0 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 0x555c2a1eada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3c9d2c640 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 0x555c2a1ed830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.621+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e59a0 0x555c2a1e5da0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0149c0 tx=0x7ff3bc014e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96dcb2370b000f30 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.622+0000 7ff3b8ff9640 1 -- 172.21.15.62:0/1791017584 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc01eb40 con 0x555c2a1e59a0 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.622+0000 7ff3b8ff9640 1 -- 172.21.15.62:0/1791017584 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3bc023070 con 0x555c2a1e59a0 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.622+0000 7ff3b8ff9640 1 -- 172.21.15.62:0/1791017584 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc013b00 con 0x555c2a1e59a0 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.622+0000 7ff3cf60b740 1 -- 172.21.15.62:0/1791017584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e59a0 msgr2=0x555c2a1e5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.622+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3b00a32f0 0x7ff3b00a57b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc022000 tx=0x7ff3bc020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.622+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e59a0 0x555c2a1e5da0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0149c0 tx=0x7ff3bc014e80 comp rx=0 tx=0).stop 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 -- 172.21.15.62:0/1791017584 shutdown_connections 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 0x555c2a1ed830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 0x555c2a1eada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/1791017584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e59a0 0x555c2a1e5da0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 -- 172.21.15.62:0/1791017584 >> 172.21.15.62:0/1791017584 conn(0x555c2a1e1010 msgr2=0x555c2a1e1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 -- 172.21.15.62:0/1791017584 shutdown_connections 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 -- 172.21.15.62:0/1791017584 wait complete. 2024-09-15T23:23:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 -- start start 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 Processor -- start 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 0x555c2a1efbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e62e0 0x555c2a1f0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 0x555c2a1f7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555c2a1ee340 con 0x555c2a1e62e0 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555c2a0d2290 con 0x555c2a1eb370 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555c2a1ee4c0 con 0x555c2a1e59a0 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3cf60b740 10 client.0 _ll_register_callbacks cb 0x555c2a103140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.623+0000 7ff3c9d2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e62e0 0x555c2a1f0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3c9d2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e62e0 0x555c2a1f0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60108/0 (socket says 172.21.15.62:60108) 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3c9d2c640 1 -- 172.21.15.62:0/403070466 learned_addr learned my addr 172.21.15.62:0/403070466 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 0x555c2a1efbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 0x555c2a1f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3c9d2c640 1 -- 172.21.15.62:0/403070466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 msgr2=0x555c2a1efbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3c9d2c640 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 0x555c2a1efbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3c9d2c640 1 -- 172.21.15.62:0/403070466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 msgr2=0x555c2a1f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3c9d2c640 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 0x555c2a1f7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3c9d2c640 1 -- 172.21.15.62:0/403070466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c29e5c210 con 0x555c2a1e62e0 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3c9d2c640 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e62e0 0x555c2a1f0130 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400bad0 tx=0x7ff3b400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bdf5a1079d2c2d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3adffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3adffb640 1 -- 172.21.15.62:0/403070466 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4011440 con 0x555c2a1e62e0 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3adffb640 1 -- 172.21.15.62:0/403070466 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff3b40115e0 con 0x555c2a1e62e0 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3cf60b740 1 -- 172.21.15.62:0/403070466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3bc023070 con 0x555c2a1e62e0 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3adffb640 1 -- 172.21.15.62:0/403070466 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff3b40117e0 con 0x555c2a1e62e0 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3cf60b740 10 client.17163 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.624+0000 7ff3cf60b740 1 -- 172.21.15.62:0/403070466 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555c29f28620 con 0x555c2a1e62e0 2024-09-15T23:23:35.631 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.625+0000 7ff3adffb640 1 -- 172.21.15.62:0/403070466 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3bc023070 con 0x555c2a1e62e0 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.625+0000 7ff3adffb640 1 -- 172.21.15.62:0/403070466 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff3b401b9f0 con 0x555c2a1e62e0 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.625+0000 7ff3cf60b740 10 client.17163 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.625+0000 7ff3cf60b740 10 client.17163 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.625+0000 7ff3cf60b740 20 client.17163 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.625+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c2a1fb980 0x555c2a1fdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.625+0000 7ff3cf60b740 4 client.17163 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.625+0000 7ff3cf60b740 1 -- 172.21.15.62:0/403070466 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c2a156a20 con 0x555c2a1fb980 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.625+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c2a1fb980 0x555c2a1fdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.626+0000 7ff3adffb640 10 client.17163 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:35.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.626+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c2a1fb980 0x555c2a1fdda0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b979df747a96009e server_cookie=4ee12fb088807207 in_seq=0 out_seq=0 2024-09-15T23:23:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.626+0000 7ff3adffb640 1 -- 172.21.15.62:0/403070466 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555c2a156a20 con 0x555c2a1fb980 2024-09-15T23:23:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.626+0000 7ff3adffb640 10 client.17163 handle_command_reply: tid=0 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.626+0000 7ff3cf60b740 1 client.17163 shutdown 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 -- 172.21.15.62:0/403070466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e62e0 msgr2=0x555c2a1f0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e62e0 0x555c2a1f0130 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400bad0 tx=0x7ff3b400bf90 comp rx=0 tx=0).stop 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 -- 172.21.15.62:0/403070466 shutdown_connections 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c2a1fb980 0x555c2a1fdda0 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1eb370 0x555c2a1f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1e62e0 0x555c2a1f0130 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/403070466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 0x555c2a1efbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3adffb640 0 client.17163 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 -- 172.21.15.62:0/403070466 >> 172.21.15.62:0/403070466 conn(0x555c2a1e1010 msgr2=0x555c2a1e2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 -- 172.21.15.62:0/403070466 shutdown_connections 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 1 -- 172.21.15.62:0/403070466 wait complete. 2024-09-15T23:23:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.627+0000 7ff3cf60b740 20 client.17163 trim_cache size 0 max 16384 2024-09-15T23:23:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.641+0000 7ff3cf60b740 1 Processor -- start 2024-09-15T23:23:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.641+0000 7ff3cf60b740 1 -- start start 2024-09-15T23:23:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e59a0 0x555c2a1e1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 0x555c2a1f7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1e23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41acf20 con 0x555c2a1eb370 2024-09-15T23:23:35.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41aa7f0 con 0x555c2a1e59a0 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41a8640 con 0x555c2a1e62e0 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cad2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1e23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cad2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1e23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60122/0 (socket says 172.21.15.62:60122) 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cad2e640 1 -- 172.21.15.62:0/3596379781 learned_addr learned my addr 172.21.15.62:0/3596379781 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3c9d2c640 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 0x555c2a1f7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e59a0 0x555c2a1e1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cad2e640 1 -- 172.21.15.62:0/3596379781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 msgr2=0x555c2a1f7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 0x555c2a1f7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cad2e640 1 -- 172.21.15.62:0/3596379781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e59a0 msgr2=0x555c2a1e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e59a0 0x555c2a1e1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cad2e640 1 -- 172.21.15.62:0/3596379781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c2a1f8610 con 0x555c2a1eb370 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e59a0 0x555c2a1e1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:35.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1e23d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc013fd0 tx=0x7ff3bc012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cfb9ee5c30d16df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.642+0000 7ff3affff640 1 -- 172.21.15.62:0/3596379781 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc037070 con 0x555c2a1eb370 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3affff640 1 -- 172.21.15.62:0/3596379781 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3bc023070 con 0x555c2a1eb370 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3affff640 1 -- 172.21.15.62:0/3596379781 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc01eba0 con 0x555c2a1eb370 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 -- 172.21.15.62:0/3596379781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 msgr2=0x555c2a1e23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1e23d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc013fd0 tx=0x7ff3bc012040 comp rx=0 tx=0).stop 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 -- 172.21.15.62:0/3596379781 shutdown_connections 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1e23d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e62e0 0x555c2a1f7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/3596379781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e59a0 0x555c2a1e1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 -- 172.21.15.62:0/3596379781 >> 172.21.15.62:0/3596379781 conn(0x555c2a1e1010 msgr2=0x555c2a1efbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 -- 172.21.15.62:0/3596379781 shutdown_connections 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 -- 172.21.15.62:0/3596379781 wait complete. 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.643+0000 7ff3cf60b740 1 -- start start 2024-09-15T23:23:35.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cf60b740 1 Processor -- start 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 0x555c2a1f4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e62e0 0x555c2a1f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cf60b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41a8cb0 con 0x555c2a1eb370 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41aa120 con 0x555c2a1e62e0 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cf60b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c41a9380 con 0x555c2a1e59a0 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cf60b740 10 client.0 _ll_register_callbacks cb 0x7ff3b4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cad2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cad2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1f5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60124/0 (socket says 172.21.15.62:60124) 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cad2e640 1 -- 172.21.15.62:0/243450483 learned_addr learned my addr 172.21.15.62:0/243450483 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 0x555c2a1f4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3c9d2c640 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e62e0 0x555c2a1f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cad2e640 1 -- 172.21.15.62:0/243450483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 msgr2=0x555c2a1f4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 0x555c2a1f4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cad2e640 1 -- 172.21.15.62:0/243450483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e62e0 msgr2=0x555c2a1f4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e62e0 0x555c2a1f4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cad2e640 1 -- 172.21.15.62:0/243450483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c2a1f63e0 con 0x555c2a1eb370 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3cad2e640 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1f5dd0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0149c0 tx=0x7ff3bc005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c1e39da9870857 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.644+0000 7ff3adffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3adffb640 1 -- 172.21.15.62:0/243450483 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc040020 con 0x555c2a1eb370 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3adffb640 1 -- 172.21.15.62:0/243450483 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff3bc01eb40 con 0x555c2a1eb370 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3cf60b740 1 -- 172.21.15.62:0/243450483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3bc023070 con 0x555c2a1eb370 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3adffb640 1 -- 172.21.15.62:0/243450483 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff3bc037070 con 0x555c2a1eb370 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3cf60b740 10 client.17169 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3cf60b740 1 -- 172.21.15.62:0/243450483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x555c2a266680 con 0x555c2a1eb370 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3adffb640 1 -- 172.21.15.62:0/243450483 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3bc023070 con 0x555c2a1eb370 2024-09-15T23:23:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3adffb640 1 -- 172.21.15.62:0/243450483 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7ff3bc00f040 con 0x555c2a1eb370 2024-09-15T23:23:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3cf60b740 10 client.17169 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3cf60b740 10 client.17169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3cf60b740 20 client.17169 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c2a266e50 0x555c2a269270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3cf60b740 4 client.17169 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:23:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.645+0000 7ff3cf60b740 1 -- 172.21.15.62:0/243450483 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555c2a150d70 con 0x555c2a266e50 2024-09-15T23:23:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.646+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c2a266e50 0x555c2a269270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.646+0000 7ff3ca52d640 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c2a266e50 0x555c2a269270 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f7b3ab5c8abd535 server_cookie=6d20f087e69cd77f in_seq=0 out_seq=0 2024-09-15T23:23:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.646+0000 7ff3adffb640 10 client.17169 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3adffb640 1 -- 172.21.15.62:0/243450483 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555c2a150d70 con 0x555c2a266e50 2024-09-15T23:23:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3adffb640 10 client.17169 handle_command_reply: tid=0 2024-09-15T23:23:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 client.17169 shutdown 2024-09-15T23:23:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 -- 172.21.15.62:0/243450483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 msgr2=0x555c2a1f5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1f5dd0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0149c0 tx=0x7ff3bc005b50 comp rx=0 tx=0).stop 2024-09-15T23:23:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 -- 172.21.15.62:0/243450483 shutdown_connections 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555c2a266e50 0x555c2a269270 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c2a1eb370 0x555c2a1f5dd0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c2a1e62e0 0x555c2a1f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3adffb640 0 client.17169 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 --2- 172.21.15.62:0/243450483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555c2a1e59a0 0x555c2a1f4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 -- 172.21.15.62:0/243450483 >> 172.21.15.62:0/243450483 conn(0x555c2a1e1010 msgr2=0x555c2a1df9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 -- 172.21.15.62:0/243450483 shutdown_connections 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 1 -- 172.21.15.62:0/243450483 wait complete. 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cf60b740 20 client.17169 trim_cache size 0 max 16384 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:23:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:23:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:23:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:23:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 -- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3b00a32f0 msgr2=0x7ff3b00a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3b00a32f0 0x7ff3b00a57b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc022000 tx=0x7ff3bc020000 comp rx=0 tx=0).stop 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 -- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 msgr2=0x7ff3c4197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 0x7ff3c4197e30 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0031610 tx=0x7ff3c00043b0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 -- 172.21.15.62:0/4134340636 shutdown_connections 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3b00a32f0 0x7ff3b00a57b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3c4102610 0x7ff3c419f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3c4101c50 0x7ff3c4198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 --2- 172.21.15.62:0/4134340636 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3c4100a60 0x7ff3c4197e30 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 -- 172.21.15.62:0/4134340636 >> 172.21.15.62:0/4134340636 conn(0x7ff3c40fc1e0 msgr2=0x7ff3c40fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 -- 172.21.15.62:0/4134340636 shutdown_connections 2024-09-15T23:23:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:35.660+0000 7ff3cc791640 1 -- 172.21.15.62:0/4134340636 wait complete. 2024-09-15T23:23:35.898 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:23:36.161 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:36.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.618+0000 7ff50b6ae640 1 -- 172.21.15.62:0/4211111445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff504100b20 msgr2=0x7ff504100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.618+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/4211111445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff504100b20 0x7ff504100f00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff500009900 tx=0x7ff50002a410 comp rx=0 tx=0).stop 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.618+0000 7ff50b6ae640 1 -- 172.21.15.62:0/4211111445 shutdown_connections 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.618+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/4211111445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff50410e500 0x7ff504110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.618+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/4211111445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff504101440 0x7ff50410df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.618+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/4211111445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff504100b20 0x7ff504100f00 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.618+0000 7ff50b6ae640 1 -- 172.21.15.62:0/4211111445 >> 172.21.15.62:0/4211111445 conn(0x7ff5040fc7d0 msgr2=0x7ff5040febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.618+0000 7ff50b6ae640 1 -- 172.21.15.62:0/4211111445 shutdown_connections 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.618+0000 7ff50b6ae640 1 -- 172.21.15.62:0/4211111445 wait complete. 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff50b6ae640 1 Processor -- start 2024-09-15T23:23:36.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff50b6ae640 1 -- start start 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff50b6ae640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff504100b20 0x7ff5041a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff50b6ae640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504101440 0x7ff5041a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff50b6ae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff50410e500 0x7ff50419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff50b6ae640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff504114260 con 0x7ff50410e500 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff50b6ae640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff5041140e0 con 0x7ff504100b20 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff50b6ae640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff5041143e0 con 0x7ff504101440 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff509c4b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff50410e500 0x7ff50419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff509c4b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff50410e500 0x7ff50419a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60148/0 (socket says 172.21.15.62:60148) 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff509c4b640 1 -- 172.21.15.62:0/281649094 learned_addr learned my addr 172.21.15.62:0/281649094 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff50944a640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff504100b20 0x7ff5041a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:36.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.619+0000 7ff508c49640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504101440 0x7ff5041a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff509c4b640 1 -- 172.21.15.62:0/281649094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504101440 msgr2=0x7ff5041a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff509c4b640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504101440 0x7ff5041a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff509c4b640 1 -- 172.21.15.62:0/281649094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff504100b20 msgr2=0x7ff5041a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff509c4b640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff504100b20 0x7ff5041a0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff509c4b640 1 -- 172.21.15.62:0/281649094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50419ad00 con 0x7ff50410e500 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff508c49640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504101440 0x7ff5041a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff509c4b640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff50410e500 0x7ff50419a5d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00e9d0 tx=0x7ff4fc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff4fa7fc640 1 -- 172.21.15.62:0/281649094 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc00cd80 con 0x7ff50410e500 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff50419aff0 con 0x7ff50410e500 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff4fa7fc640 1 -- 172.21.15.62:0/281649094 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4fc005020 con 0x7ff50410e500 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff4fa7fc640 1 -- 172.21.15.62:0/281649094 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0116d0 con 0x7ff50410e500 2024-09-15T23:23:36.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.620+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5041a72e0 con 0x7ff50410e500 2024-09-15T23:23:36.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.621+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff50406a850 con 0x7ff50410e500 2024-09-15T23:23:36.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.622+0000 7ff4fa7fc640 1 -- 172.21.15.62:0/281649094 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff4fc0051d0 con 0x7ff50410e500 2024-09-15T23:23:36.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.623+0000 7ff4fa7fc640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4f00a3240 0x7ff4f00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:36.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.624+0000 7ff50944a640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4f00a3240 0x7ff4f00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:36.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.624+0000 7ff4fa7fc640 1 -- 172.21.15.62:0/281649094 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff4fc0d02d0 con 0x7ff50410e500 2024-09-15T23:23:36.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.624+0000 7ff50944a640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4f00a3240 0x7ff4f00a5700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff500005ab0 tx=0x7ff500005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:36.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.628+0000 7ff4fa7fc640 1 -- 172.21.15.62:0/281649094 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff4fc016030 con 0x7ff50410e500 2024-09-15T23:23:36.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.840+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff504100f00 con 0x7ff50410e500 2024-09-15T23:23:36.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.841+0000 7ff4fa7fc640 1 -- 172.21.15.62:0/281649094 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11906 (secure 0 0 0) 0x7ff4fc0bc030 con 0x7ff50410e500 2024-09-15T23:23:36.844 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:23:36.844 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":33,"btime":"2024-09-15T23:23:18:032879+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:23:17.383471+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:23:36.844 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 33 2024-09-15T23:23:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4f00a3240 msgr2=0x7ff4f00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4f00a3240 0x7ff4f00a5700 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff500005ab0 tx=0x7ff500005a20 comp rx=0 tx=0).stop 2024-09-15T23:23:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff50410e500 msgr2=0x7ff50419a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff50410e500 0x7ff50419a5d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00e9d0 tx=0x7ff4fc00ee90 comp rx=0 tx=0).stop 2024-09-15T23:23:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 shutdown_connections 2024-09-15T23:23:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4f00a3240 0x7ff4f00a5700 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff50410e500 0x7ff50419a5d0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff504101440 0x7ff5041a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 --2- 172.21.15.62:0/281649094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff504100b20 0x7ff5041a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 >> 172.21.15.62:0/281649094 conn(0x7ff5040fc7d0 msgr2=0x7ff5040febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 shutdown_connections 2024-09-15T23:23:36.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:36.844+0000 7ff50b6ae640 1 -- 172.21.15.62:0/281649094 wait complete. 2024-09-15T23:23:37.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:37 smithi062 ceph-mon[30949]: pgmap v590: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.3 MiB/s wr, 327 op/s 2024-09-15T23:23:37.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:37 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/281649094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:37 smithi067 ceph-mon[35665]: pgmap v590: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.3 MiB/s wr, 327 op/s 2024-09-15T23:23:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:37 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/281649094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:37 smithi148 ceph-mon[35610]: pgmap v590: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 3.3 MiB/s wr, 327 op/s 2024-09-15T23:23:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:37 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/281649094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:37.577 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 87dc2717 --release --await' 2024-09-15T23:23:37.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 87dc2717 --release --await 2024-09-15T23:23:37.838 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:38.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.324+0000 7f2a3e458640 1 -- 172.21.15.62:0/866605377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 msgr2=0x7f2a38100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:38.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.324+0000 7f2a3e458640 1 --2- 172.21.15.62:0/866605377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 0x7f2a38100480 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009900 tx=0x7f2a2802a410 comp rx=0 tx=0).stop 2024-09-15T23:23:38.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.324+0000 7f2a3e458640 1 -- 172.21.15.62:0/866605377 shutdown_connections 2024-09-15T23:23:38.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.324+0000 7f2a3e458640 1 --2- 172.21.15.62:0/866605377 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a38104e80 0x7f2a38105330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.324+0000 7f2a3e458640 1 --2- 172.21.15.62:0/866605377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a3806a370 0x7f2a381009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.324+0000 7f2a3e458640 1 --2- 172.21.15.62:0/866605377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 0x7f2a38100480 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.324+0000 7f2a3e458640 1 -- 172.21.15.62:0/866605377 >> 172.21.15.62:0/866605377 conn(0x7f2a38077210 msgr2=0x7f2a380fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:38.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.324+0000 7f2a3e458640 1 -- 172.21.15.62:0/866605377 shutdown_connections 2024-09-15T23:23:38.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.324+0000 7f2a3e458640 1 -- 172.21.15.62:0/866605377 wait complete. 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.325+0000 7f2a3e458640 1 Processor -- start 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.325+0000 7f2a3e458640 1 -- start start 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a3e458640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 0x7f2a381983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a3e458640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a3806a370 0x7f2a381988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a3e458640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a38104e80 0x7f2a3819cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a3e458640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2a38110eb0 con 0x7f2a38069f00 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a3e458640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2a38110d30 con 0x7f2a3806a370 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a3e458640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2a38111030 con 0x7f2a38104e80 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a37fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 0x7f2a381983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a37fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 0x7f2a381983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60170/0 (socket says 172.21.15.62:60170) 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a37fff640 1 -- 172.21.15.62:0/2895726912 learned_addr learned my addr 172.21.15.62:0/2895726912 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a3806a370 0x7f2a381988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a38104e80 0x7f2a3819cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a37fff640 1 -- 172.21.15.62:0/2895726912 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a38104e80 msgr2=0x7f2a3819cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a37fff640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a38104e80 0x7f2a3819cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a37fff640 1 -- 172.21.15.62:0/2895726912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a3806a370 msgr2=0x7f2a381988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a37fff640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a3806a370 0x7f2a381988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a37fff640 1 -- 172.21.15.62:0/2895726912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3819d350 con 0x7f2a38069f00 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.326+0000 7f2a37fff640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 0x7f2a381983a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2a28006010 tx=0x7f2a280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.327+0000 7f2a357fa640 1 -- 172.21.15.62:0/2895726912 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2803f030 con 0x7f2a38069f00 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.327+0000 7f2a357fa640 1 -- 172.21.15.62:0/2895726912 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a28003ce0 con 0x7f2a38069f00 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.327+0000 7f2a3e458640 1 -- 172.21.15.62:0/2895726912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3819d5e0 con 0x7f2a38069f00 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.327+0000 7f2a357fa640 1 -- 172.21.15.62:0/2895726912 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28035070 con 0x7f2a38069f00 2024-09-15T23:23:38.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.327+0000 7f2a3e458640 1 -- 172.21.15.62:0/2895726912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a381a4f20 con 0x7f2a38069f00 2024-09-15T23:23:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.329+0000 7f2a357fa640 1 -- 172.21.15.62:0/2895726912 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2a28042030 con 0x7f2a38069f00 2024-09-15T23:23:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.329+0000 7f2a40831740 1 Processor -- start 2024-09-15T23:23:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.329+0000 7f2a40831740 1 -- start start 2024-09-15T23:23:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a357fa640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2a200a3240 0x7f2a200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a40831740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31abf3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a40831740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 0x55f31ac02110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a40831740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac04a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a40831740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f31ab58620 con 0x55f31ac02650 2024-09-15T23:23:38.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a40831740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f31aaffe50 con 0x55f31abf2f80 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a40831740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f31ab545e0 con 0x55f31abf38a0 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2a200a3240 0x7f2a200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a357fa640 1 -- 172.21.15.62:0/2895726912 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2a280f0270 con 0x7f2a38069f00 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a3c9f5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac04a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a3c9f5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac04a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60174/0 (socket says 172.21.15.62:60174) 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a3c9f5640 1 -- 172.21.15.62:0/4054944429 learned_addr learned my addr 172.21.15.62:0/4054944429 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a37fff640 1 --2- 172.21.15.62:0/4054944429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31abf3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a3c9f5640 1 -- 172.21.15.62:0/4054944429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 msgr2=0x55f31ac02110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/4054944429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 0x55f31ac02110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a3c9f5640 1 -- 172.21.15.62:0/4054944429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 msgr2=0x55f31abf3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/4054944429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31abf3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a3c9f5640 1 -- 172.21.15.62:0/4054944429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f31aaf0d40 con 0x55f31ac02650 2024-09-15T23:23:38.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.330+0000 7f2a37fff640 1 --2- 172.21.15.62:0/4054944429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31abf3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2a200a3240 0x7f2a200a5700 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2a30012c00 tx=0x7f2a30009e00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/4054944429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac04a40 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2a30010800 tx=0x7f2a3001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae01449f727e3448 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a1e7fc640 1 -- 172.21.15.62:0/4054944429 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30024070 con 0x55f31ac02650 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a1e7fc640 1 -- 172.21.15.62:0/4054944429 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a3000e820 con 0x55f31ac02650 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a1e7fc640 1 -- 172.21.15.62:0/4054944429 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30005390 con 0x55f31ac02650 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a40831740 1 -- 172.21.15.62:0/4054944429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 msgr2=0x55f31ac04a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a40831740 1 --2- 172.21.15.62:0/4054944429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac04a40 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2a30010800 tx=0x7f2a3001c000 comp rx=0 tx=0).stop 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a40831740 1 -- 172.21.15.62:0/4054944429 shutdown_connections 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a40831740 1 --2- 172.21.15.62:0/4054944429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac04a40 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.331+0000 7f2a40831740 1 --2- 172.21.15.62:0/4054944429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 0x55f31ac02110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 --2- 172.21.15.62:0/4054944429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31abf3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 -- 172.21.15.62:0/4054944429 >> 172.21.15.62:0/4054944429 conn(0x55f31abef5f0 msgr2=0x55f31abef9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 -- 172.21.15.62:0/4054944429 shutdown_connections 2024-09-15T23:23:38.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 -- 172.21.15.62:0/4054944429 wait complete. 2024-09-15T23:23:38.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 -- start start 2024-09-15T23:23:38.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 Processor -- start 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf2f80 0x55f31abff5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf38a0 0x55f31abffb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31ac02650 0x55f31ac00070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f31ac05160 con 0x55f31abf38a0 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f31aae0f10 con 0x55f31ac02650 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f31ac052e0 con 0x55f31abf2f80 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.332+0000 7f2a40831740 10 client.0 _ll_register_callbacks cb 0x55f31ab04db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a377fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf38a0 0x55f31abffb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a377fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf38a0 0x55f31abffb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60176/0 (socket says 172.21.15.62:60176) 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a377fe640 1 -- 172.21.15.62:0/3581605198 learned_addr learned my addr 172.21.15.62:0/3581605198 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31ac02650 0x55f31ac00070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a37fff640 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf2f80 0x55f31abff5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a377fe640 1 -- 172.21.15.62:0/3581605198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf2f80 msgr2=0x55f31abff5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a377fe640 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf2f80 0x55f31abff5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a377fe640 1 -- 172.21.15.62:0/3581605198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31ac02650 msgr2=0x55f31ac00070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a377fe640 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31ac02650 0x55f31ac00070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a377fe640 1 -- 172.21.15.62:0/3581605198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f31a867580 con 0x55f31abf38a0 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a377fe640 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf38a0 0x55f31abffb30 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00bed0 tx=0x7f2a2c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10aa93759ac41336 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a177fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a177fe640 1 -- 172.21.15.62:0/3581605198 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c005220 con 0x55f31abf38a0 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a177fe640 1 -- 172.21.15.62:0/3581605198 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2a2c0053c0 con 0x55f31abf38a0 2024-09-15T23:23:38.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a40831740 1 -- 172.21.15.62:0/3581605198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a3000e820 con 0x55f31abf38a0 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a177fe640 1 -- 172.21.15.62:0/3581605198 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2a2c024050 con 0x55f31abf38a0 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a40831740 10 client.17187 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.333+0000 7f2a40831740 1 -- 172.21.15.62:0/3581605198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f31a931960 con 0x55f31abf38a0 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.334+0000 7f2a177fe640 1 -- 172.21.15.62:0/3581605198 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a3000e820 con 0x55f31abf38a0 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.334+0000 7f2a177fe640 1 -- 172.21.15.62:0/3581605198 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f2a2c0118c0 con 0x55f31abf38a0 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.334+0000 7f2a40831740 10 client.17187 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.334+0000 7f2a40831740 10 client.17187 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.334+0000 7f2a40831740 20 client.17187 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.334+0000 7f2a40831740 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f31ac12330 0x55f31ac14750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.334+0000 7f2a40831740 4 client.17187 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.334+0000 7f2a40831740 1 -- 172.21.15.62:0/3581605198 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f31ab65b90 con 0x55f31ac12330 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.334+0000 7f2a37fff640 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f31ac12330 0x55f31ac14750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a37fff640 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f31ac12330 0x55f31ac14750 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e18b818d95b6958 server_cookie=45538aacca1da912 in_seq=0 out_seq=0 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a177fe640 10 client.17187 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a177fe640 1 -- 172.21.15.62:0/3581605198 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f31ab65b90 con 0x55f31ac12330 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a177fe640 10 client.17187 handle_command_reply: tid=0 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a40831740 1 client.17187 shutdown 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a40831740 1 -- 172.21.15.62:0/3581605198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf38a0 msgr2=0x55f31abffb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a40831740 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf38a0 0x55f31abffb30 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c00bed0 tx=0x7f2a2c0107a0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a40831740 1 -- 172.21.15.62:0/3581605198 shutdown_connections 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a40831740 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f31ac12330 0x55f31ac14750 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.335+0000 7f2a177fe640 0 client.17187 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.336+0000 7f2a40831740 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31ac02650 0x55f31ac00070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.336+0000 7f2a40831740 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf38a0 0x55f31abffb30 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.336+0000 7f2a40831740 1 --2- 172.21.15.62:0/3581605198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf2f80 0x55f31abff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.336+0000 7f2a40831740 1 -- 172.21.15.62:0/3581605198 >> 172.21.15.62:0/3581605198 conn(0x55f31abef5f0 msgr2=0x55f31abf5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.336+0000 7f2a40831740 1 -- 172.21.15.62:0/3581605198 shutdown_connections 2024-09-15T23:23:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.336+0000 7f2a40831740 1 -- 172.21.15.62:0/3581605198 wait complete. 2024-09-15T23:23:38.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.336+0000 7f2a40831740 20 client.17187 trim_cache size 0 max 16384 2024-09-15T23:23:38.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a40831740 1 Processor -- start 2024-09-15T23:23:38.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a40831740 1 -- start start 2024-09-15T23:23:38.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a40831740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31ac01560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a40831740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 0x55f31ac1a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a40831740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac1a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a40831740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2a381af2d0 con 0x55f31ac02650 2024-09-15T23:23:38.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a40831740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2a381acba0 con 0x55f31abf2f80 2024-09-15T23:23:38.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a40831740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2a381aa9f0 con 0x55f31abf38a0 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a3c9f5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac1a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a3c9f5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac1a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60192/0 (socket says 172.21.15.62:60192) 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a3c9f5640 1 -- 172.21.15.62:0/2740438134 learned_addr learned my addr 172.21.15.62:0/2740438134 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a37fff640 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31ac01560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a377fe640 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 0x55f31ac1a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a3c9f5640 1 -- 172.21.15.62:0/2740438134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 msgr2=0x55f31ac1a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 0x55f31ac1a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a3c9f5640 1 -- 172.21.15.62:0/2740438134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 msgr2=0x55f31ac01560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31ac01560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a3c9f5640 1 -- 172.21.15.62:0/2740438134 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f31abf4490 con 0x55f31ac02650 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a37fff640 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31ac01560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:38.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.353+0000 7f2a377fe640 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 0x55f31ac1a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac1a580 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2a30010800 tx=0x7f2a30009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72821b61ada08c62 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a1dffb640 1 -- 172.21.15.62:0/2740438134 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30024050 con 0x55f31ac02650 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a1dffb640 1 -- 172.21.15.62:0/2740438134 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a30003e50 con 0x55f31ac02650 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a1dffb640 1 -- 172.21.15.62:0/2740438134 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a300052b0 con 0x55f31ac02650 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a40831740 1 -- 172.21.15.62:0/2740438134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 msgr2=0x55f31ac1a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a40831740 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac1a580 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2a30010800 tx=0x7f2a30009e70 comp rx=0 tx=0).stop 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a40831740 1 -- 172.21.15.62:0/2740438134 shutdown_connections 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a40831740 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31ac02650 0x55f31ac1a580 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a40831740 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31abf38a0 0x55f31ac1a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a40831740 1 --2- 172.21.15.62:0/2740438134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf2f80 0x55f31ac01560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a40831740 1 -- 172.21.15.62:0/2740438134 >> 172.21.15.62:0/2740438134 conn(0x55f31abef5f0 msgr2=0x55f31ac00e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:38.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a40831740 1 -- 172.21.15.62:0/2740438134 shutdown_connections 2024-09-15T23:23:38.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.354+0000 7f2a40831740 1 -- 172.21.15.62:0/2740438134 wait complete. 2024-09-15T23:23:38.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.355+0000 7f2a40831740 1 -- start start 2024-09-15T23:23:38.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.355+0000 7f2a40831740 1 Processor -- start 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.355+0000 7f2a40831740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf2f80 0x55f31ac0aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.355+0000 7f2a40831740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf38a0 0x55f31ac07590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.355+0000 7f2a40831740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31ac02650 0x55f31ac07ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.355+0000 7f2a40831740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2a381ab060 con 0x55f31abf2f80 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.355+0000 7f2a40831740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2a381ac4d0 con 0x55f31abf38a0 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.355+0000 7f2a40831740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2a381ab730 con 0x55f31ac02650 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.355+0000 7f2a40831740 10 client.0 _ll_register_callbacks cb 0x7f2a2c00bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a37fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf2f80 0x55f31ac0aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a37fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf2f80 0x55f31ac0aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:60206/0 (socket says 172.21.15.62:60206) 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a37fff640 1 -- 172.21.15.62:0/3474953098 learned_addr learned my addr 172.21.15.62:0/3474953098 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:38.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a377fe640 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf38a0 0x55f31ac07590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a3c9f5640 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31ac02650 0x55f31ac07ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a37fff640 1 -- 172.21.15.62:0/3474953098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31ac02650 msgr2=0x55f31ac07ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a37fff640 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31ac02650 0x55f31ac07ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a37fff640 1 -- 172.21.15.62:0/3474953098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf38a0 msgr2=0x55f31ac07590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a37fff640 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf38a0 0x55f31ac07590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a37fff640 1 -- 172.21.15.62:0/3474953098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f31ac08090 con 0x55f31abf2f80 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a377fe640 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf38a0 0x55f31ac07590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a37fff640 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf2f80 0x55f31ac0aa90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2a280c4cb0 tx=0x7f2a280f0190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d656aee050964a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a177fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a177fe640 1 -- 172.21.15.62:0/3474953098 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28049040 con 0x55f31abf2f80 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a177fe640 1 -- 172.21.15.62:0/3474953098 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2a280b8c90 con 0x55f31abf2f80 2024-09-15T23:23:38.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a40831740 1 -- 172.21.15.62:0/3474953098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a30003e50 con 0x55f31abf2f80 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a177fe640 1 -- 172.21.15.62:0/3474953098 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2a280bc620 con 0x55f31abf2f80 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a40831740 10 client.17193 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.356+0000 7f2a40831740 1 -- 172.21.15.62:0/3474953098 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55f31ac7ddd0 con 0x55f31abf2f80 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.357+0000 7f2a177fe640 1 -- 172.21.15.62:0/3474953098 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a30003e50 con 0x55f31abf2f80 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.357+0000 7f2a177fe640 1 -- 172.21.15.62:0/3474953098 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f2a280bc880 con 0x55f31abf2f80 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.357+0000 7f2a40831740 10 client.17193 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.357+0000 7f2a40831740 10 client.17193 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.357+0000 7f2a40831740 20 client.17193 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.357+0000 7f2a40831740 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f31ac12330 0x55f31ac80530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.357+0000 7f2a40831740 4 client.17193 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "87dc2717", "release": true, "await": true}] 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.357+0000 7f2a40831740 1 -- 172.21.15.62:0/3474953098 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "87dc2717", "release": true, "await": true}) -- 0x55f31ab600a0 con 0x55f31ac12330 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.357+0000 7f2a377fe640 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f31ac12330 0x55f31ac80530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:38.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.358+0000 7f2a177fe640 10 client.17193 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:38.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.358+0000 7f2a377fe640 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f31ac12330 0x55f31ac80530 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=604be29d2ae78367 server_cookie=6c1d6ef8504cf47a in_seq=0 out_seq=0 2024-09-15T23:23:38.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.386+0000 7f2a177fe640 1 -- 172.21.15.62:0/3474953098 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55f31ab600a0 con 0x55f31ac12330 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.386+0000 7f2a177fe640 10 client.17193 handle_command_reply: tid=0 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.386+0000 7f2a40831740 1 client.17193 shutdown 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 -- 172.21.15.62:0/3474953098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf2f80 msgr2=0x55f31ac0aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf2f80 0x55f31ac0aa90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2a280c4cb0 tx=0x7f2a280f0190 comp rx=0 tx=0).stop 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 -- 172.21.15.62:0/3474953098 shutdown_connections 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55f31ac12330 0x55f31ac80530 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f31ac02650 0x55f31ac07ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f31abf38a0 0x55f31ac07590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a177fe640 0 client.17193 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 --2- 172.21.15.62:0/3474953098 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f31abf2f80 0x55f31ac0aa90 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 -- 172.21.15.62:0/3474953098 >> 172.21.15.62:0/3474953098 conn(0x55f31abef5f0 msgr2=0x55f31abee750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:38.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 -- 172.21.15.62:0/3474953098 shutdown_connections 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 1 -- 172.21.15.62:0/3474953098 wait complete. 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.387+0000 7f2a40831740 20 client.17193 trim_cache size 0 max 16384 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 33, 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 28, 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stdout: "87dc2717": { 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 28, 2024-09-15T23:23:38.391 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 143.7, 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:23:38.392 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:38.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:38.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:38.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:38.393 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:23:38.393 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:23:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.389+0000 7f2a3e458640 1 -- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2a200a3240 msgr2=0x7f2a200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.389+0000 7f2a3e458640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2a200a3240 0x7f2a200a5700 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f2a30012c00 tx=0x7f2a30009e00 comp rx=0 tx=0).stop 2024-09-15T23:23:38.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.389+0000 7f2a3e458640 1 -- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 msgr2=0x7f2a381983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.389+0000 7f2a3e458640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 0x7f2a381983a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f2a28006010 tx=0x7f2a280042f0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.391+0000 7f2a3e458640 1 -- 172.21.15.62:0/2895726912 shutdown_connections 2024-09-15T23:23:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.391+0000 7f2a3e458640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2a200a3240 0x7f2a200a5700 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.391+0000 7f2a3e458640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2a38104e80 0x7f2a3819cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.391+0000 7f2a3e458640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a3806a370 0x7f2a381988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.391+0000 7f2a3e458640 1 --2- 172.21.15.62:0/2895726912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a38069f00 0x7f2a381983a0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:38.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.391+0000 7f2a3e458640 1 -- 172.21.15.62:0/2895726912 >> 172.21.15.62:0/2895726912 conn(0x7f2a38077210 msgr2=0x7f2a38107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:38.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.394+0000 7f2a3e458640 1 -- 172.21.15.62:0/2895726912 shutdown_connections 2024-09-15T23:23:38.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:38.394+0000 7f2a3e458640 1 -- 172.21.15.62:0/2895726912 wait complete. 2024-09-15T23:23:39.134 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '87dc2717', total seconds elapsed: 26.9 2024-09-15T23:23:39.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:39 smithi062 ceph-mon[30949]: pgmap v591: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.2 MiB/s wr, 330 op/s 2024-09-15T23:23:39.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:39 smithi067 ceph-mon[35665]: pgmap v591: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.2 MiB/s wr, 330 op/s 2024-09-15T23:23:39.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:39 smithi148 ceph-mon[35610]: pgmap v591: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 3.2 MiB/s wr, 330 op/s 2024-09-15T23:23:39.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 440.0 s, 222.0 tps, lat 2040.794 ms stddev 6185.117 2024-09-15T23:23:41.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:41 smithi062 ceph-mon[30949]: pgmap v592: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 2.9 MiB/s wr, 309 op/s 2024-09-15T23:23:41.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:41 smithi067 ceph-mon[35665]: pgmap v592: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 2.9 MiB/s wr, 309 op/s 2024-09-15T23:23:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:41 smithi148 ceph-mon[35610]: pgmap v592: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 2.9 MiB/s wr, 309 op/s 2024-09-15T23:23:41.837 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:23:42.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:42 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:23:42.306 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:42.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:42 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:23:42.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:42 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.950+0000 7f3283cd0640 1 -- 172.21.15.62:0/884774237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c109450 msgr2=0x7f327c110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.950+0000 7f3283cd0640 1 --2- 172.21.15.62:0/884774237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c109450 0x7f327c110250 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3278009900 tx=0x7f327802a3f0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.951+0000 7f3283cd0640 1 -- 172.21.15.62:0/884774237 shutdown_connections 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.951+0000 7f3283cd0640 1 --2- 172.21.15.62:0/884774237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f327c11b140 0x7f327c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.951+0000 7f3283cd0640 1 --2- 172.21.15.62:0/884774237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f327c109830 0x7f327c109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.951+0000 7f3283cd0640 1 --2- 172.21.15.62:0/884774237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c109450 0x7f327c110250 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3278009900 tx=0x7f327802a3f0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.951+0000 7f3283cd0640 1 -- 172.21.15.62:0/884774237 >> 172.21.15.62:0/884774237 conn(0x7f327c06e140 msgr2=0x7f327c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:42.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.954+0000 7f3283cd0640 1 -- 172.21.15.62:0/884774237 shutdown_connections 2024-09-15T23:23:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.955+0000 7f3283cd0640 1 -- 172.21.15.62:0/884774237 wait complete. 2024-09-15T23:23:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.956+0000 7f3283cd0640 1 Processor -- start 2024-09-15T23:23:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.956+0000 7f3283cd0640 1 -- start start 2024-09-15T23:23:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.956+0000 7f3283cd0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f327c109830 0x7f327c11a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.956+0000 7f3283cd0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c11b140 0x7f327c113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.956+0000 7f3283cd0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f327c113f40 0x7f327c1143f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.956+0000 7f3283cd0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f327c120df0 con 0x7f327c109830 2024-09-15T23:23:42.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.956+0000 7f3283cd0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f327c120c70 con 0x7f327c11b140 2024-09-15T23:23:42.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.956+0000 7f3283cd0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f327c120f70 con 0x7f327c113f40 2024-09-15T23:23:42.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.957+0000 7f328126b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c11b140 0x7f327c113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.957+0000 7f328126b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c11b140 0x7f327c113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:56898/0 (socket says 172.21.15.62:56898) 2024-09-15T23:23:42.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.957+0000 7f328126b640 1 -- 172.21.15.62:0/3522352243 learned_addr learned my addr 172.21.15.62:0/3522352243 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:42.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.958+0000 7f328126b640 1 -- 172.21.15.62:0/3522352243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f327c113f40 msgr2=0x7f327c1143f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.958+0000 7f328126b640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f327c113f40 0x7f327c1143f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.958+0000 7f328126b640 1 -- 172.21.15.62:0/3522352243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f327c109830 msgr2=0x7f327c11a970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.958+0000 7f328126b640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f327c109830 0x7f327c11a970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.958+0000 7f328126b640 1 -- 172.21.15.62:0/3522352243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327c118600 con 0x7f327c11b140 2024-09-15T23:23:42.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.958+0000 7f328126b640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c11b140 0x7f327c113a00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f326c00d950 tx=0x7f326c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.961+0000 7f3272ffd640 1 -- 172.21.15.62:0/3522352243 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c017070 con 0x7f327c11b140 2024-09-15T23:23:42.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.961+0000 7f3283cd0640 1 -- 172.21.15.62:0/3522352243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f327c1188f0 con 0x7f327c11b140 2024-09-15T23:23:42.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.961+0000 7f3283cd0640 1 -- 172.21.15.62:0/3522352243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f327c118e30 con 0x7f327c11b140 2024-09-15T23:23:42.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.963+0000 7f3272ffd640 1 -- 172.21.15.62:0/3522352243 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c009c30 con 0x7f327c11b140 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.963+0000 7f3272ffd640 1 -- 172.21.15.62:0/3522352243 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c00f6b0 con 0x7f327c11b140 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.966+0000 7f3272ffd640 1 -- 172.21.15.62:0/3522352243 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f326c00f8d0 con 0x7f327c11b140 2024-09-15T23:23:42.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.967+0000 7f3272ffd640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32680a3410 0x7f32680a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.967+0000 7f3272ffd640 1 -- 172.21.15.62:0/3522352243 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f326c0d0940 con 0x7f327c11b140 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.969+0000 7f32860a9740 1 Processor -- start 2024-09-15T23:23:42.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.969+0000 7f32860a9740 1 -- start start 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f32860a9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 0x563778f07880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f32860a9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f07dc0 0x563778f12140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f32860a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f12680 0x563778f14a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f32860a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563778e6d2e0 con 0x563778f12680 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f32860a9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563778e14ce0 con 0x563778f07dc0 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f32860a9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563778e692a0 con 0x563778f074a0 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f328126b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f07dc0 0x563778f12140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f328126b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f07dc0 0x563778f12140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:56910/0 (socket says 172.21.15.62:56910) 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f328126b640 1 -- 172.21.15.62:0/2438598739 learned_addr learned my addr 172.21.15.62:0/2438598739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f328226d640 1 --2- 172.21.15.62:0/2438598739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f12680 0x563778f14a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f328126b640 1 -- 172.21.15.62:0/2438598739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 msgr2=0x563778f07880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:42.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f328126b640 1 --2- 172.21.15.62:0/2438598739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 0x563778f07880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.970+0000 7f328126b640 1 -- 172.21.15.62:0/2438598739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f12680 msgr2=0x563778f14a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.971+0000 7f328126b640 1 --2- 172.21.15.62:0/2438598739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f12680 0x563778f14a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.971+0000 7f328126b640 1 -- 172.21.15.62:0/2438598739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563778df9440 con 0x563778f07dc0 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.971+0000 7f328226d640 1 --2- 172.21.15.62:0/2438598739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f12680 0x563778f14a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.971+0000 7f328126b640 1 --2- 172.21.15.62:0/2438598739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f07dc0 0x563778f12140 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f326c009610 tx=0x7f326c09d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e9b87384f2b294 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.971+0000 7f3267fff640 1 -- 172.21.15.62:0/2438598739 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c020040 con 0x563778f07dc0 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.971+0000 7f3267fff640 1 -- 172.21.15.62:0/2438598739 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c0a57f0 con 0x563778f07dc0 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.972+0000 7f3281a6c640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32680a3410 0x7f32680a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.972+0000 7f32860a9740 1 -- 172.21.15.62:0/2438598739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f07dc0 msgr2=0x563778f12140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.972+0000 7f32860a9740 1 --2- 172.21.15.62:0/2438598739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f07dc0 0x563778f12140 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f326c009610 tx=0x7f326c09d3b0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.972+0000 7f32860a9740 1 -- 172.21.15.62:0/2438598739 shutdown_connections 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.972+0000 7f32860a9740 1 --2- 172.21.15.62:0/2438598739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f12680 0x563778f14a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.972+0000 7f32860a9740 1 --2- 172.21.15.62:0/2438598739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f07dc0 0x563778f12140 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.972+0000 7f32860a9740 1 --2- 172.21.15.62:0/2438598739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 0x563778f07880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.972+0000 7f32860a9740 1 -- 172.21.15.62:0/2438598739 >> 172.21.15.62:0/2438598739 conn(0x563778f03a30 msgr2=0x563778f03e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:42.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.977+0000 7f32860a9740 1 -- 172.21.15.62:0/2438598739 shutdown_connections 2024-09-15T23:23:42.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.977+0000 7f32860a9740 1 -- 172.21.15.62:0/2438598739 wait complete. 2024-09-15T23:23:42.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.978+0000 7f32860a9740 1 -- start start 2024-09-15T23:23:42.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.978+0000 7f32860a9740 1 Processor -- start 2024-09-15T23:23:42.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f32860a9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 0x563778f11d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f32860a9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f12680 0x563778f0f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f32860a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f0fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f32860a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563778f152f0 con 0x563778f0f580 2024-09-15T23:23:42.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f32860a9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563778e28050 con 0x563778f12680 2024-09-15T23:23:42.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f32860a9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563778f15470 con 0x563778f074a0 2024-09-15T23:23:42.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f32860a9740 10 client.0 _ll_register_callbacks cb 0x563778e68160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:42.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f328226d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f0fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f328226d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f0fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52004/0 (socket says 172.21.15.62:52004) 2024-09-15T23:23:42.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.982+0000 7f328226d640 1 -- 172.21.15.62:0/3333789932 learned_addr learned my addr 172.21.15.62:0/3333789932 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:42.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.983+0000 7f328226d640 1 -- 172.21.15.62:0/3333789932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 msgr2=0x563778f11d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:42.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.983+0000 7f328126b640 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f12680 0x563778f0f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.983+0000 7f328226d640 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 0x563778f11d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.983+0000 7f328226d640 1 -- 172.21.15.62:0/3333789932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f12680 msgr2=0x563778f0f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:42.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.983+0000 7f328226d640 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f12680 0x563778f0f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:42.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.983+0000 7f328226d640 1 -- 172.21.15.62:0/3333789932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563778bde3c0 con 0x563778f0f580 2024-09-15T23:23:42.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.983+0000 7f328226d640 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f0fa00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f3274009f50 tx=0x7f327400f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b51ef6d7335f47b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.983+0000 7f32657fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:42.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.985+0000 7f3281a6c640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32680a3410 0x7f32680a58d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f327802a9e0 tx=0x7f3278005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:42.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.985+0000 7f32657fa640 1 -- 172.21.15.62:0/3333789932 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274003b70 con 0x563778f0f580 2024-09-15T23:23:42.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.985+0000 7f32860a9740 1 -- 172.21.15.62:0/3333789932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563778e7a850 con 0x563778f0f580 2024-09-15T23:23:42.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.985+0000 7f32657fa640 1 -- 172.21.15.62:0/3333789932 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3274003d10 con 0x563778f0f580 2024-09-15T23:23:42.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.985+0000 7f32657fa640 1 -- 172.21.15.62:0/3333789932 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3274022050 con 0x563778f0f580 2024-09-15T23:23:42.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.985+0000 7f32860a9740 10 client.17199 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:42.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.985+0000 7f32860a9740 1 -- 172.21.15.62:0/3333789932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563778f20fd0 con 0x563778f0f580 2024-09-15T23:23:42.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.986+0000 7f32657fa640 1 -- 172.21.15.62:0/3333789932 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563778e7a850 con 0x563778f0f580 2024-09-15T23:23:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.986+0000 7f32657fa640 1 -- 172.21.15.62:0/3333789932 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f32740119b0 con 0x563778f0f580 2024-09-15T23:23:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.987+0000 7f32860a9740 10 client.17199 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.987+0000 7f32860a9740 10 client.17199 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.987+0000 7f32860a9740 20 client.17199 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.987+0000 7f32860a9740 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563778f22450 0x563778f248d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.987+0000 7f32860a9740 4 client.17199 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.987+0000 7f32860a9740 1 -- 172.21.15.62:0/3333789932 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563778e74d60 con 0x563778f22450 2024-09-15T23:23:42.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.993+0000 7f3281a6c640 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563778f22450 0x563778f248d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:42.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.997+0000 7f32657fa640 10 client.17199 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:42.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.997+0000 7f3281a6c640 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563778f22450 0x563778f248d0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d73f0e437c9a3f3 server_cookie=94ec69ab866bb260 in_seq=0 out_seq=0 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.998+0000 7f32657fa640 1 -- 172.21.15.62:0/3333789932 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563778e74d60 con 0x563778f22450 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.998+0000 7f32657fa640 10 client.17199 handle_command_reply: tid=0 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.998+0000 7f32860a9740 1 client.17199 shutdown 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.998+0000 7f32860a9740 1 -- 172.21.15.62:0/3333789932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 msgr2=0x563778f0fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.998+0000 7f32860a9740 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f0fa00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f3274009f50 tx=0x7f327400f7f0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.998+0000 7f32860a9740 1 -- 172.21.15.62:0/3333789932 shutdown_connections 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.998+0000 7f32860a9740 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563778f22450 0x563778f248d0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.999+0000 7f32860a9740 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f0fa00 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.999+0000 7f32860a9740 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f12680 0x563778f0f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.999+0000 7f32860a9740 1 --2- 172.21.15.62:0/3333789932 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 0x563778f11d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.999+0000 7f32860a9740 1 -- 172.21.15.62:0/3333789932 >> 172.21.15.62:0/3333789932 conn(0x563778f03a30 msgr2=0x563778f08f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:43.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.999+0000 7f32657fa640 0 client.17199 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:43.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:42.999+0000 7f32860a9740 1 -- 172.21.15.62:0/3333789932 shutdown_connections 2024-09-15T23:23:43.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.001+0000 7f32860a9740 1 -- 172.21.15.62:0/3333789932 wait complete. 2024-09-15T23:23:43.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.001+0000 7f32860a9740 20 client.17199 trim_cache size 0 max 16384 2024-09-15T23:23:43.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.032+0000 7f32860a9740 1 Processor -- start 2024-09-15T23:23:43.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.034+0000 7f32860a9740 1 -- start start 2024-09-15T23:23:43.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.034+0000 7f32860a9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 0x563778f1e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:43.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.034+0000 7f32860a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f13a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:43.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.034+0000 7f32860a9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f12680 0x563778f13fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:43.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.034+0000 7f32860a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3260012420 con 0x563778f0f580 2024-09-15T23:23:43.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.034+0000 7f328126b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f13a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:43.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.034+0000 7f328126b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f13a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52016/0 (socket says 172.21.15.62:52016) 2024-09-15T23:23:43.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.034+0000 7f328126b640 1 -- 172.21.15.62:0/832706918 learned_addr learned my addr 172.21.15.62:0/832706918 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:43.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.035+0000 7f32860a9740 1 -- 172.21.15.62:0/832706918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f326000fcf0 con 0x563778f12680 2024-09-15T23:23:43.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.035+0000 7f32860a9740 1 -- 172.21.15.62:0/832706918 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f326000db40 con 0x563778f074a0 2024-09-15T23:23:43.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.035+0000 7f328126b640 1 -- 172.21.15.62:0/832706918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 msgr2=0x563778f1e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:43.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.035+0000 7f328126b640 1 --2- 172.21.15.62:0/832706918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 0x563778f1e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.035+0000 7f328126b640 1 -- 172.21.15.62:0/832706918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f12680 msgr2=0x563778f13fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:43.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.035+0000 7f328126b640 1 --2- 172.21.15.62:0/832706918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f12680 0x563778f13fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.035+0000 7f328126b640 1 -- 172.21.15.62:0/832706918 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563778f12100 con 0x563778f0f580 2024-09-15T23:23:43.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.036+0000 7f328126b640 1 --2- 172.21.15.62:0/832706918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f13a70 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f326c005fd0 tx=0x7f326c00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e176464cb22d3104 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:43.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.039+0000 7f32677fe640 1 -- 172.21.15.62:0/832706918 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c0b9020 con 0x563778f0f580 2024-09-15T23:23:43.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.039+0000 7f32677fe640 1 -- 172.21.15.62:0/832706918 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c0ce170 con 0x563778f0f580 2024-09-15T23:23:43.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.039+0000 7f32677fe640 1 -- 172.21.15.62:0/832706918 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c020040 con 0x563778f0f580 2024-09-15T23:23:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.040+0000 7f32860a9740 1 -- 172.21.15.62:0/832706918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 msgr2=0x563778f13a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.040+0000 7f32860a9740 1 --2- 172.21.15.62:0/832706918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f13a70 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f326c005fd0 tx=0x7f326c00bf60 comp rx=0 tx=0).stop 2024-09-15T23:23:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.041+0000 7f32860a9740 1 -- 172.21.15.62:0/832706918 shutdown_connections 2024-09-15T23:23:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.041+0000 7f32860a9740 1 --2- 172.21.15.62:0/832706918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f12680 0x563778f13fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.041+0000 7f32860a9740 1 --2- 172.21.15.62:0/832706918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f0f580 0x563778f13a70 secure :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f326c005fd0 tx=0x7f326c00bf60 comp rx=0 tx=0).stop 2024-09-15T23:23:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.041+0000 7f32860a9740 1 --2- 172.21.15.62:0/832706918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f074a0 0x563778f1e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.041+0000 7f32860a9740 1 -- 172.21.15.62:0/832706918 >> 172.21.15.62:0/832706918 conn(0x563778f03a30 msgr2=0x563778f09fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.047+0000 7f32860a9740 1 -- 172.21.15.62:0/832706918 shutdown_connections 2024-09-15T23:23:43.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.048+0000 7f32860a9740 1 -- 172.21.15.62:0/832706918 wait complete. 2024-09-15T23:23:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.048+0000 7f32860a9740 1 -- start start 2024-09-15T23:23:43.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.048+0000 7f32860a9740 1 Processor -- start 2024-09-15T23:23:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.049+0000 7f32860a9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f074a0 0x563778f1ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.049+0000 7f32860a9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f12680 0x563778f1b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.049+0000 7f32860a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f158c0 0x563778f15d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.049+0000 7f32860a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f326000e1b0 con 0x563778f158c0 2024-09-15T23:23:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.049+0000 7f328126b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f12680 0x563778f1b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.049+0000 7f328126b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f12680 0x563778f1b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35858/0 (socket says 172.21.15.62:35858) 2024-09-15T23:23:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.049+0000 7f328126b640 1 -- 172.21.15.62:0/3201787758 learned_addr learned my addr 172.21.15.62:0/3201787758 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.049+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f326000f620 con 0x563778f074a0 2024-09-15T23:23:43.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.050+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f326000e880 con 0x563778f12680 2024-09-15T23:23:43.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.050+0000 7f32860a9740 10 client.0 _ll_register_callbacks cb 0x7f327c110870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:43.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.051+0000 7f3281a6c640 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f074a0 0x563778f1ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:43.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.052+0000 7f328226d640 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f158c0 0x563778f15d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:43.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.055+0000 7f3281a6c640 1 -- 172.21.15.62:0/3201787758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f12680 msgr2=0x563778f1b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:43.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.055+0000 7f3281a6c640 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f12680 0x563778f1b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.055+0000 7f3281a6c640 1 -- 172.21.15.62:0/3201787758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f158c0 msgr2=0x563778f15d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:43.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.055+0000 7f3281a6c640 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f158c0 0x563778f15d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.055+0000 7f3281a6c640 1 -- 172.21.15.62:0/3201787758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563778f16570 con 0x563778f074a0 2024-09-15T23:23:43.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.059+0000 7f3281a6c640 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f074a0 0x563778f1ab40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3278008000 tx=0x7f3278000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1601644512ef4f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:43.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.060+0000 7f32657fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:43.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.061+0000 7f32657fa640 1 -- 172.21.15.62:0/3201787758 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3278035040 con 0x563778f074a0 2024-09-15T23:23:43.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.061+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f326c0ce170 con 0x563778f074a0 2024-09-15T23:23:43.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.061+0000 7f32657fa640 1 -- 172.21.15.62:0/3201787758 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3278003c00 con 0x563778f074a0 2024-09-15T23:23:43.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.063+0000 7f32657fa640 1 -- 172.21.15.62:0/3201787758 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3278003e00 con 0x563778f074a0 2024-09-15T23:23:43.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.066+0000 7f32860a9740 10 client.26777 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:43.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.066+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563778f8e240 con 0x563778f074a0 2024-09-15T23:23:43.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.070+0000 7f32657fa640 1 -- 172.21.15.62:0/3201787758 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c0ce170 con 0x563778f074a0 2024-09-15T23:23:43.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.070+0000 7f32657fa640 1 -- 172.21.15.62:0/3201787758 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f3278050070 con 0x563778f074a0 2024-09-15T23:23:43.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.072+0000 7f32860a9740 10 client.26777 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:43.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.072+0000 7f32860a9740 10 client.26777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:43.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.072+0000 7f32860a9740 20 client.26777 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:43.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.072+0000 7f32860a9740 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563778f8e970 0x563778f90d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:43.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.072+0000 7f32860a9740 4 client.26777 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:23:43.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.072+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563778f0abc0 con 0x563778f8e970 2024-09-15T23:23:43.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.072+0000 7f328126b640 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563778f8e970 0x563778f90d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:43.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.074+0000 7f328126b640 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563778f8e970 0x563778f90d90 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=437aefcc6a4000c8 server_cookie=b0072baf0d6fef56 in_seq=0 out_seq=0 2024-09-15T23:23:43.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.074+0000 7f32657fa640 10 client.26777 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:43.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.077+0000 7f32657fa640 1 -- 172.21.15.62:0/3201787758 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563778f0abc0 con 0x563778f8e970 2024-09-15T23:23:43.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.077+0000 7f32657fa640 10 client.26777 handle_command_reply: tid=0 2024-09-15T23:23:43.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.078+0000 7f32860a9740 1 client.26777 shutdown 2024-09-15T23:23:43.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.082+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f074a0 msgr2=0x563778f1ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:43.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.082+0000 7f32860a9740 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f074a0 0x563778f1ab40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3278008000 tx=0x7f3278000bc0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.085+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 shutdown_connections 2024-09-15T23:23:43.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.085+0000 7f32860a9740 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563778f8e970 0x563778f90d90 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.085+0000 7f32860a9740 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563778f158c0 0x563778f15d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.085+0000 7f32860a9740 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563778f12680 0x563778f1b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.085+0000 7f32860a9740 1 --2- 172.21.15.62:0/3201787758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563778f074a0 0x563778f1ab40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.085+0000 7f32657fa640 0 client.26777 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:43.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.085+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 >> 172.21.15.62:0/3201787758 conn(0x563778f03a30 msgr2=0x563778f0d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:43.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.086+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 shutdown_connections 2024-09-15T23:23:43.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.090+0000 7f32860a9740 1 -- 172.21.15.62:0/3201787758 wait complete. 2024-09-15T23:23:43.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.094+0000 7f32860a9740 20 client.26777 trim_cache size 0 max 16384 2024-09-15T23:23:43.096 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:23:43.096 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:23:43.096 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:23:43.096 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:23:43.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 -- 172.21.15.62:0/3522352243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32680a3410 msgr2=0x7f32680a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:43.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32680a3410 0x7f32680a58d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f327802a9e0 tx=0x7f3278005fd0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 -- 172.21.15.62:0/3522352243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c11b140 msgr2=0x7f327c113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:43.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c11b140 0x7f327c113a00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f326c00d950 tx=0x7f326c00de10 comp rx=0 tx=0).stop 2024-09-15T23:23:43.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 -- 172.21.15.62:0/3522352243 shutdown_connections 2024-09-15T23:23:43.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f32680a3410 0x7f32680a58d0 secure :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f327802a9e0 tx=0x7f3278005fd0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f327c113f40 0x7f327c1143f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f327c11b140 0x7f327c113a00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 --2- 172.21.15.62:0/3522352243 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f327c109830 0x7f327c11a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:43.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 -- 172.21.15.62:0/3522352243 >> 172.21.15.62:0/3522352243 conn(0x7f327c06e140 msgr2=0x7f327c10ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:43.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.095+0000 7f3283cd0640 1 -- 172.21.15.62:0/3522352243 shutdown_connections 2024-09-15T23:23:43.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:43.096+0000 7f3283cd0640 1 -- 172.21.15.62:0/3522352243 wait complete. 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: pgmap v593: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 2.2 MiB/s wr, 209 op/s 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: Adjusting osd_memory_target on smithi148 to 176.0M 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: Unable to set osd_memory_target on smithi148 to 184591052: error parsing value: Value '184591052' is below minimum 939524096 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:23:43.468 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:23:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: pgmap v593: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 2.2 MiB/s wr, 209 op/s 2024-09-15T23:23:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: Adjusting osd_memory_target on smithi148 to 176.0M 2024-09-15T23:23:43.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: Unable to set osd_memory_target on smithi148 to 184591052: error parsing value: Value '184591052' is below minimum 939524096 2024-09-15T23:23:43.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:23:43.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:23:43.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:23:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: pgmap v593: 129 pgs: 129 active+clean; 1.5 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 2.2 MiB/s wr, 209 op/s 2024-09-15T23:23:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-09-15T23:23:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: Adjusting osd_memory_target on smithi148 to 176.0M 2024-09-15T23:23:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: Unable to set osd_memory_target on smithi148 to 184591052: error parsing value: Value '184591052' is below minimum 939524096 2024-09-15T23:23:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:23:43.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:23:43.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:23:43.930 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:23:43.930 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:23:44.437 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:44.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 445.0 s, 898.6 tps, lat 111.613 ms stddev 142.789 2024-09-15T23:23:45.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.044+0000 7fd4cad59640 1 -- 172.21.15.62:0/2312182397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c4109450 msgr2=0x7fd4c4110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:45.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.044+0000 7fd4cad59640 1 --2- 172.21.15.62:0/2312182397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c4109450 0x7fd4c4110250 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009900 tx=0x7fd4b402a440 comp rx=0 tx=0).stop 2024-09-15T23:23:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.044+0000 7fd4cad59640 1 -- 172.21.15.62:0/2312182397 shutdown_connections 2024-09-15T23:23:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.044+0000 7fd4cad59640 1 --2- 172.21.15.62:0/2312182397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c411b140 0x7fd4c411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.044+0000 7fd4cad59640 1 --2- 172.21.15.62:0/2312182397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c4109830 0x7fd4c4109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.044+0000 7fd4cad59640 1 --2- 172.21.15.62:0/2312182397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c4109450 0x7fd4c4110250 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.044+0000 7fd4cad59640 1 -- 172.21.15.62:0/2312182397 >> 172.21.15.62:0/2312182397 conn(0x7fd4c406e140 msgr2=0x7fd4c406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.045+0000 7fd4cad59640 1 -- 172.21.15.62:0/2312182397 shutdown_connections 2024-09-15T23:23:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.045+0000 7fd4cad59640 1 -- 172.21.15.62:0/2312182397 wait complete. 2024-09-15T23:23:45.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.045+0000 7fd4cad59640 1 Processor -- start 2024-09-15T23:23:45.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4cad59640 1 -- start start 2024-09-15T23:23:45.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4cad59640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c4109450 0x7fd4c41186f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4cad59640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c4109830 0x7fd4c4111780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4cad59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c411b140 0x7fd4c4111cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4cad59640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4120e10 con 0x7fd4c4109830 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4cad59640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4120c90 con 0x7fd4c411b140 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4cad59640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c4120f90 con 0x7fd4c4109450 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4c8af5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c4109450 0x7fd4c41186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4c8af5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c4109450 0x7fd4c41186f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35880/0 (socket says 172.21.15.62:35880) 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c4109830 0x7fd4c4111780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4c8af5640 1 -- 172.21.15.62:0/3323571508 learned_addr learned my addr 172.21.15.62:0/3323571508 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:45.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4c92f6640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c411b140 0x7fd4c4111cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:45.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4c8af5640 1 -- 172.21.15.62:0/3323571508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c411b140 msgr2=0x7fd4c4111cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:45.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.046+0000 7fd4c8af5640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c411b140 0x7fd4c4111cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:45.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.047+0000 7fd4c8af5640 1 -- 172.21.15.62:0/3323571508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c4109830 msgr2=0x7fd4c4111780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:45.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.047+0000 7fd4c8af5640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c4109830 0x7fd4c4111780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:45.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.047+0000 7fd4c8af5640 1 -- 172.21.15.62:0/3323571508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c4112280 con 0x7fd4c4109450 2024-09-15T23:23:45.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.047+0000 7fd4c8af5640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c4109450 0x7fd4c41186f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd4b40058c0 tx=0x7fd4b4005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:45.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.047+0000 7fd4c1ffb640 1 -- 172.21.15.62:0/3323571508 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4040030 con 0x7fd4c4109450 2024-09-15T23:23:45.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.047+0000 7fd4c1ffb640 1 -- 172.21.15.62:0/3323571508 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4b4003e30 con 0x7fd4c4109450 2024-09-15T23:23:45.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.047+0000 7fd4c1ffb640 1 -- 172.21.15.62:0/3323571508 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b403cb20 con 0x7fd4c4109450 2024-09-15T23:23:45.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.049+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c4112510 con 0x7fd4c4109450 2024-09-15T23:23:45.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.050+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c41163f0 con 0x7fd4c4109450 2024-09-15T23:23:45.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.053+0000 7fd4c1ffb640 1 -- 172.21.15.62:0/3323571508 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd4b4043030 con 0x7fd4c4109450 2024-09-15T23:23:45.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.053+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c4073b00 con 0x7fd4c4109450 2024-09-15T23:23:45.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.054+0000 7fd4c1ffb640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3380 0x7fd4ac0a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:45.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.054+0000 7fd4c1ffb640 1 -- 172.21.15.62:0/3323571508 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd4b40f08c0 con 0x7fd4c4109450 2024-09-15T23:23:45.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.059+0000 7fd4c1ffb640 1 -- 172.21.15.62:0/3323571508 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd4b40bd200 con 0x7fd4c4109450 2024-09-15T23:23:45.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.062+0000 7fd4c3fff640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3380 0x7fd4ac0a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:45.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.070+0000 7fd4c3fff640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3380 0x7fd4ac0a5840 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4112f40 tx=0x7fd4bc0072f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:45.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:45 smithi062 ceph-mon[30949]: pgmap v594: 129 pgs: 129 active+clean; 1.5 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-09-15T23:23:45.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.426+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4c4112cb0 con 0x7fd4c4109450 2024-09-15T23:23:45.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.428+0000 7fd4c1ffb640 1 -- 172.21.15.62:0/3323571508 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11906 (secure 0 0 0) 0x7fd4b40bcb00 con 0x7fd4c4109450 2024-09-15T23:23:45.430 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:23:45.430 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":33,"btime":"2024-09-15T23:23:18:032879+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:23:17.383471+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:23:45.431 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 33 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3380 msgr2=0x7fd4ac0a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3380 0x7fd4ac0a5840 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4112f40 tx=0x7fd4bc0072f0 comp rx=0 tx=0).stop 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c4109450 msgr2=0x7fd4c41186f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c4109450 0x7fd4c41186f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd4b40058c0 tx=0x7fd4b4005230 comp rx=0 tx=0).stop 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 shutdown_connections 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3380 0x7fd4ac0a5840 secure :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd4c4112f40 tx=0x7fd4bc0072f0 comp rx=0 tx=0).stop 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c411b140 0x7fd4c4111cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c4109830 0x7fd4c4111780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 --2- 172.21.15.62:0/3323571508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c4109450 0x7fd4c41186f0 secure :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd4b40058c0 tx=0x7fd4b4005230 comp rx=0 tx=0).stop 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 >> 172.21.15.62:0/3323571508 conn(0x7fd4c406e140 msgr2=0x7fd4c411be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.432+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 shutdown_connections 2024-09-15T23:23:45.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:45.433+0000 7fd4cad59640 1 -- 172.21.15.62:0/3323571508 wait complete. 2024-09-15T23:23:45.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:45 smithi067 ceph-mon[35665]: pgmap v594: 129 pgs: 129 active+clean; 1.5 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-09-15T23:23:45.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:45 smithi148 ceph-mon[35610]: pgmap v594: 129 pgs: 129 active+clean; 1.5 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 1.8 MiB/s wr, 186 op/s 2024-09-15T23:23:46.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:46 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3323571508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:46.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:46 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3323571508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:46.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:46 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3323571508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:23:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:47 smithi062 ceph-mon[30949]: pgmap v595: 129 pgs: 129 active+clean; 1.5 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-09-15T23:23:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:47 smithi067 ceph-mon[35665]: pgmap v595: 129 pgs: 129 active+clean; 1.5 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-09-15T23:23:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:47 smithi148 ceph-mon[35610]: pgmap v595: 129 pgs: 129 active+clean; 1.5 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 2.1 MiB/s wr, 224 op/s 2024-09-15T23:23:48.563 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:23:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.818+0000 7f3635e1e640 1 -- 172.21.15.62:0/1381543549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3630072d20 msgr2=0x7f3630073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.818+0000 7f3635e1e640 1 --2- 172.21.15.62:0/1381543549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3630072d20 0x7f3630073180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3620009900 tx=0x7f362002a420 comp rx=0 tx=0).stop 2024-09-15T23:23:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.822+0000 7f3635e1e640 1 -- 172.21.15.62:0/1381543549 shutdown_connections 2024-09-15T23:23:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.822+0000 7f3635e1e640 1 --2- 172.21.15.62:0/1381543549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36300736c0 0x7f3630119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.822+0000 7f3635e1e640 1 --2- 172.21.15.62:0/1381543549 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3630072d20 0x7f3630073180 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.822+0000 7f3635e1e640 1 --2- 172.21.15.62:0/1381543549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3630109700 0x7f3630109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.822+0000 7f3635e1e640 1 -- 172.21.15.62:0/1381543549 >> 172.21.15.62:0/1381543549 conn(0x7f363006ece0 msgr2=0x7f3630071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.825+0000 7f3635e1e640 1 -- 172.21.15.62:0/1381543549 shutdown_connections 2024-09-15T23:23:48.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.827+0000 7f3635e1e640 1 -- 172.21.15.62:0/1381543549 wait complete. 2024-09-15T23:23:48.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.828+0000 7f3635e1e640 1 Processor -- start 2024-09-15T23:23:48.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.831+0000 7f3635e1e640 1 -- start start 2024-09-15T23:23:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.831+0000 7f3635e1e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3630072d20 0x7f36301a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.831+0000 7f3635e1e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36300736c0 0x7f36301a7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.831+0000 7f3635e1e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3630109700 0x7f36301ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.831+0000 7f3635e1e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f363011c680 con 0x7f3630109700 2024-09-15T23:23:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.831+0000 7f3635e1e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f363011c500 con 0x7f36300736c0 2024-09-15T23:23:48.834 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.831+0000 7f3635e1e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f363011c800 con 0x7f3630072d20 2024-09-15T23:23:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.834+0000 7f362effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36300736c0 0x7f36301a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.834+0000 7f362effd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36300736c0 0x7f36301a7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:56994/0 (socket says 172.21.15.62:56994) 2024-09-15T23:23:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.834+0000 7f362effd640 1 -- 172.21.15.62:0/2532693724 learned_addr learned my addr 172.21.15.62:0/2532693724 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.835+0000 7f362effd640 1 -- 172.21.15.62:0/2532693724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3630072d20 msgr2=0x7f36301a6c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:23:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.835+0000 7f362effd640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3630072d20 0x7f36301a6c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.835+0000 7f362effd640 1 -- 172.21.15.62:0/2532693724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3630109700 msgr2=0x7f36301ae1b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:23:48.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.835+0000 7f362effd640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3630109700 0x7f36301ae1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.835+0000 7f362effd640 1 -- 172.21.15.62:0/2532693724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36301ae850 con 0x7f36300736c0 2024-09-15T23:23:48.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.836+0000 7f362effd640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36300736c0 0x7f36301a7180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3620031610 tx=0x7f3620032cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:48.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.837+0000 7f362cff9640 1 -- 172.21.15.62:0/2532693724 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f362003f030 con 0x7f36300736c0 2024-09-15T23:23:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.838+0000 7f362cff9640 1 -- 172.21.15.62:0/2532693724 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3620003ce0 con 0x7f36300736c0 2024-09-15T23:23:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.838+0000 7f3635e1e640 1 -- 172.21.15.62:0/2532693724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36301aeae0 con 0x7f36300736c0 2024-09-15T23:23:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.838+0000 7f3635e1e640 1 -- 172.21.15.62:0/2532693724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36301af050 con 0x7f36300736c0 2024-09-15T23:23:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.838+0000 7f362cff9640 1 -- 172.21.15.62:0/2532693724 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620035070 con 0x7f36300736c0 2024-09-15T23:23:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.843+0000 7f362cff9640 1 -- 172.21.15.62:0/2532693724 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3620042030 con 0x7f36300736c0 2024-09-15T23:23:48.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.846+0000 7f362cff9640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36180a3530 0x7f36180a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.846+0000 7f362f7fe640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36180a3530 0x7f36180a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.846+0000 7f3638c90740 1 Processor -- start 2024-09-15T23:23:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.846+0000 7f3638c90740 1 -- start start 2024-09-15T23:23:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f3638c90740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 0x564214d0dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f3638c90740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d12da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f3638c90740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d15830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f3638c90740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564214c712f0 con 0x564214d0e2e0 2024-09-15T23:23:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f3638c90740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564214c19270 con 0x564214d0d9a0 2024-09-15T23:23:48.849 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f3638c90740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564214c6d2b0 con 0x564214d13370 2024-09-15T23:23:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f362cff9640 1 -- 172.21.15.62:0/2532693724 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f36200f1430 con 0x7f36300736c0 2024-09-15T23:23:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 0x564214d0dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 0x564214d0dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57000/0 (socket says 172.21.15.62:57000) 2024-09-15T23:23:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.847+0000 7f362f7fe640 1 -- 172.21.15.62:0/2456067664 learned_addr learned my addr 172.21.15.62:0/2456067664 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.848+0000 7f362f7fe640 1 -- 172.21.15.62:0/2456067664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 msgr2=0x564214d15830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:23:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.848+0000 7f362f7fe640 1 --2- 172.21.15.62:0/2456067664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d15830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.848+0000 7f362f7fe640 1 -- 172.21.15.62:0/2456067664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 msgr2=0x564214d12da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.848+0000 7f362f7fe640 1 --2- 172.21.15.62:0/2456067664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d12da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.848+0000 7f362f7fe640 1 -- 172.21.15.62:0/2456067664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564214c18340 con 0x564214d0d9a0 2024-09-15T23:23:48.851 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.848+0000 7f362f7fe640 1 --2- 172.21.15.62:0/2456067664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 0x564214d0dda0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3628015870 tx=0x7f3628015960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f83b74ec2e08a2a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.849+0000 7f362f7fe640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36180a3530 0x7f36180a59f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3628015e60 tx=0x7f3628019000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.850+0000 7f3615ffb640 1 -- 172.21.15.62:0/2456067664 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628004b20 con 0x564214d0d9a0 2024-09-15T23:23:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.850+0000 7f3615ffb640 1 -- 172.21.15.62:0/2456067664 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3628003e50 con 0x564214d0d9a0 2024-09-15T23:23:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.850+0000 7f3615ffb640 1 -- 172.21.15.62:0/2456067664 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628004590 con 0x564214d0d9a0 2024-09-15T23:23:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.851+0000 7f3638c90740 1 -- 172.21.15.62:0/2456067664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 msgr2=0x564214d0dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:48.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.851+0000 7f3638c90740 1 --2- 172.21.15.62:0/2456067664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 0x564214d0dda0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3628015870 tx=0x7f3628015960 comp rx=0 tx=0).stop 2024-09-15T23:23:48.858 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.856+0000 7f3638c90740 1 -- 172.21.15.62:0/2456067664 shutdown_connections 2024-09-15T23:23:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.856+0000 7f3638c90740 1 --2- 172.21.15.62:0/2456067664 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d15830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.856+0000 7f3638c90740 1 --2- 172.21.15.62:0/2456067664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d12da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.856+0000 7f3638c90740 1 --2- 172.21.15.62:0/2456067664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 0x564214d0dda0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.856+0000 7f3638c90740 1 -- 172.21.15.62:0/2456067664 >> 172.21.15.62:0/2456067664 conn(0x564214d09010 msgr2=0x564214d09410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:48.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.860+0000 7f3638c90740 1 -- 172.21.15.62:0/2456067664 shutdown_connections 2024-09-15T23:23:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.860+0000 7f3638c90740 1 -- 172.21.15.62:0/2456067664 wait complete. 2024-09-15T23:23:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.860+0000 7f3638c90740 1 -- start start 2024-09-15T23:23:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.861+0000 7f3638c90740 1 Processor -- start 2024-09-15T23:23:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.861+0000 7f3638c90740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 0x564214d17c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.861+0000 7f3638c90740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d18160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.861+0000 7f3638c90740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d1f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.861+0000 7f3638c90740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564214d16340 con 0x564214d0e2e0 2024-09-15T23:23:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.861+0000 7f3638c90740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564214bfa290 con 0x564214d0d9a0 2024-09-15T23:23:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.861+0000 7f3638c90740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564214d164c0 con 0x564214d13370 2024-09-15T23:23:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.861+0000 7f3638c90740 10 client.0 _ll_register_callbacks cb 0x564214c2b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.862+0000 7f362effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d18160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.862+0000 7f362effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d18160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52072/0 (socket says 172.21.15.62:52072) 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.862+0000 7f362effd640 1 -- 172.21.15.62:0/2696042842 learned_addr learned my addr 172.21.15.62:0/2696042842 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.863+0000 7f362effd640 1 -- 172.21.15.62:0/2696042842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 msgr2=0x564214d1f1a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.863+0000 7f362effd640 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d1f1a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.863+0000 7f362effd640 1 -- 172.21.15.62:0/2696042842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 msgr2=0x564214d17c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.863+0000 7f362effd640 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 0x564214d17c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.863+0000 7f362effd640 1 -- 172.21.15.62:0/2696042842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564214984210 con 0x564214d0e2e0 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.863+0000 7f362effd640 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d18160 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3620006010 tx=0x7f362002a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff274b70dd06870 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.863+0000 7f360effd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.863+0000 7f360effd640 1 -- 172.21.15.62:0/2696042842 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620032e60 con 0x564214d0e2e0 2024-09-15T23:23:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.864+0000 7f360effd640 1 -- 172.21.15.62:0/2696042842 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f36200b9c50 con 0x564214d0e2e0 2024-09-15T23:23:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.864+0000 7f3638c90740 1 -- 172.21.15.62:0/2696042842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3628003e50 con 0x564214d0e2e0 2024-09-15T23:23:48.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.864+0000 7f360effd640 1 -- 172.21.15.62:0/2696042842 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f362004b300 con 0x564214d0e2e0 2024-09-15T23:23:48.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.864+0000 7f3638c90740 10 client.17211 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:48.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.864+0000 7f3638c90740 1 -- 172.21.15.62:0/2696042842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564214a50620 con 0x564214d0e2e0 2024-09-15T23:23:48.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.865+0000 7f360effd640 1 -- 172.21.15.62:0/2696042842 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3628003e50 con 0x564214d0e2e0 2024-09-15T23:23:48.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.865+0000 7f360effd640 1 -- 172.21.15.62:0/2696042842 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f362003b4f0 con 0x564214d0e2e0 2024-09-15T23:23:48.867 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.865+0000 7f3638c90740 10 client.17211 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:48.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.865+0000 7f3638c90740 10 client.17211 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:48.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.865+0000 7f3638c90740 20 client.17211 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:48.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.865+0000 7f3638c90740 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564214d23a80 0x564214d25f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.865+0000 7f3638c90740 4 client.17211 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:48.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.865+0000 7f3638c90740 1 -- 172.21.15.62:0/2696042842 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564214c7ea20 con 0x564214d23a80 2024-09-15T23:23:48.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.866+0000 7f362f7fe640 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564214d23a80 0x564214d25f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.867+0000 7f362f7fe640 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564214d23a80 0x564214d25f00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1878507aa43148d server_cookie=a9674771d16c7643 in_seq=0 out_seq=0 2024-09-15T23:23:48.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.867+0000 7f360effd640 10 client.17211 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.868+0000 7f360effd640 1 -- 172.21.15.62:0/2696042842 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564214c7ea20 con 0x564214d23a80 2024-09-15T23:23:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.869+0000 7f360effd640 10 client.17211 handle_command_reply: tid=0 2024-09-15T23:23:48.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.869+0000 7f3638c90740 1 client.17211 shutdown 2024-09-15T23:23:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.870+0000 7f3638c90740 1 -- 172.21.15.62:0/2696042842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 msgr2=0x564214d18160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.870+0000 7f3638c90740 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d18160 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3620006010 tx=0x7f362002a420 comp rx=0 tx=0).stop 2024-09-15T23:23:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f3638c90740 1 -- 172.21.15.62:0/2696042842 shutdown_connections 2024-09-15T23:23:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f3638c90740 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564214d23a80 0x564214d25f00 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f3638c90740 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d1f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f3638c90740 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d18160 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f3638c90740 1 --2- 172.21.15.62:0/2696042842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0d9a0 0x564214d17c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f3638c90740 1 -- 172.21.15.62:0/2696042842 >> 172.21.15.62:0/2696042842 conn(0x564214d09010 msgr2=0x564214d0a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f360effd640 0 client.17211 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f3638c90740 1 -- 172.21.15.62:0/2696042842 shutdown_connections 2024-09-15T23:23:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f3638c90740 1 -- 172.21.15.62:0/2696042842 wait complete. 2024-09-15T23:23:48.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.871+0000 7f3638c90740 20 client.17211 trim_cache size 0 max 16384 2024-09-15T23:23:48.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.905+0000 7f3638c90740 1 Processor -- start 2024-09-15T23:23:48.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.905+0000 7f3638c90740 1 -- start start 2024-09-15T23:23:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f3638c90740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d0d9a0 0x564214d0f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f3638c90740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d0fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f3638c90740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d13370 0x564214d1f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d0d9a0 0x564214d0f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d0d9a0 0x564214d0f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35926/0 (socket says 172.21.15.62:35926) 2024-09-15T23:23:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f362f7fe640 1 -- 172.21.15.62:0/4277327455 learned_addr learned my addr 172.21.15.62:0/4277327455 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f3638c90740 1 -- 172.21.15.62:0/4277327455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f361c012490 con 0x564214d0e2e0 2024-09-15T23:23:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f3638c90740 1 -- 172.21.15.62:0/4277327455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f361c00fd60 con 0x564214d13370 2024-09-15T23:23:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f362ffff640 1 --2- 172.21.15.62:0/4277327455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d13370 0x564214d1f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.906+0000 7f3638c90740 1 -- 172.21.15.62:0/4277327455 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f361c00dbb0 con 0x564214d0d9a0 2024-09-15T23:23:48.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.907+0000 7f362effd640 1 --2- 172.21.15.62:0/4277327455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d0fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.908+0000 7f362effd640 1 -- 172.21.15.62:0/4277327455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d0d9a0 msgr2=0x564214d0f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.908+0000 7f362effd640 1 --2- 172.21.15.62:0/4277327455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d0d9a0 0x564214d0f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.908+0000 7f362effd640 1 -- 172.21.15.62:0/4277327455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d13370 msgr2=0x564214d1f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.908+0000 7f362effd640 1 --2- 172.21.15.62:0/4277327455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d13370 0x564214d1f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.908+0000 7f362effd640 1 -- 172.21.15.62:0/4277327455 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564214d157b0 con 0x564214d0e2e0 2024-09-15T23:23:48.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.908+0000 7f362effd640 1 --2- 172.21.15.62:0/4277327455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d0fa00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f36200b9a40 tx=0x7f3620005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2523c355894be09c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:48.911 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.909+0000 7f36157fa640 1 -- 172.21.15.62:0/4277327455 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36200bdd70 con 0x564214d0e2e0 2024-09-15T23:23:48.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.909+0000 7f36157fa640 1 -- 172.21.15.62:0/4277327455 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36200bd670 con 0x564214d0e2e0 2024-09-15T23:23:48.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.910+0000 7f3638c90740 1 -- 172.21.15.62:0/4277327455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 msgr2=0x564214d0fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:48.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.910+0000 7f3638c90740 1 --2- 172.21.15.62:0/4277327455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d0fa00 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f36200b9a40 tx=0x7f3620005d50 comp rx=0 tx=0).stop 2024-09-15T23:23:48.918 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.916+0000 7f3638c90740 1 -- 172.21.15.62:0/4277327455 shutdown_connections 2024-09-15T23:23:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.916+0000 7f3638c90740 1 --2- 172.21.15.62:0/4277327455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d13370 0x564214d1f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.916+0000 7f3638c90740 1 --2- 172.21.15.62:0/4277327455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0e2e0 0x564214d0fa00 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.916+0000 7f3638c90740 1 --2- 172.21.15.62:0/4277327455 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d0d9a0 0x564214d0f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.919 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.916+0000 7f3638c90740 1 -- 172.21.15.62:0/4277327455 >> 172.21.15.62:0/4277327455 conn(0x564214d09010 msgr2=0x564214d17c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:48.922 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.918+0000 7f3638c90740 1 -- 172.21.15.62:0/4277327455 shutdown_connections 2024-09-15T23:23:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.920+0000 7f3638c90740 1 -- 172.21.15.62:0/4277327455 wait complete. 2024-09-15T23:23:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.920+0000 7f3638c90740 1 -- start start 2024-09-15T23:23:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.921+0000 7f3638c90740 1 Processor -- start 2024-09-15T23:23:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.922+0000 7f3638c90740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0d9a0 0x564214d1be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.922+0000 7f3638c90740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0e2e0 0x564214d1c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.922+0000 7f3638c90740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d8c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.922+0000 7f3638c90740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f361c00e220 con 0x564214d0d9a0 2024-09-15T23:23:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.922+0000 7f3638c90740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f361c00f690 con 0x564214d0e2e0 2024-09-15T23:23:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.922+0000 7f3638c90740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f361c00e8f0 con 0x564214d13370 2024-09-15T23:23:48.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.922+0000 7f3638c90740 10 client.0 _ll_register_callbacks cb 0x7f36200b91c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:48.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.925+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0d9a0 0x564214d1be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.925+0000 7f362f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0d9a0 0x564214d1be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52092/0 (socket says 172.21.15.62:52092) 2024-09-15T23:23:48.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.925+0000 7f362f7fe640 1 -- 172.21.15.62:0/981996919 learned_addr learned my addr 172.21.15.62:0/981996919 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:48.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.926+0000 7f362ffff640 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d8c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.926+0000 7f362f7fe640 1 -- 172.21.15.62:0/981996919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 msgr2=0x564214d8c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.927+0000 7f362f7fe640 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d8c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.927+0000 7f362f7fe640 1 -- 172.21.15.62:0/981996919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0e2e0 msgr2=0x564214d1c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.927+0000 7f362f7fe640 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0e2e0 0x564214d1c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.927+0000 7f362f7fe640 1 -- 172.21.15.62:0/981996919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564214d1f450 con 0x564214d0d9a0 2024-09-15T23:23:48.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.927+0000 7f362f7fe640 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0d9a0 0x564214d1be20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f362801f040 tx=0x7f36280157c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebbb4938fad7427 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:48.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.929+0000 7f360effd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:23:48.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.929+0000 7f360effd640 1 -- 172.21.15.62:0/981996919 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3628004c00 con 0x564214d0d9a0 2024-09-15T23:23:48.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.929+0000 7f3638c90740 1 -- 172.21.15.62:0/981996919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36200bd670 con 0x564214d0d9a0 2024-09-15T23:23:48.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.929+0000 7f360effd640 1 -- 172.21.15.62:0/981996919 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3628004da0 con 0x564214d0d9a0 2024-09-15T23:23:48.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.936+0000 7f360effd640 1 -- 172.21.15.62:0/981996919 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f362803c050 con 0x564214d0d9a0 2024-09-15T23:23:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.936+0000 7f3638c90740 10 client.17217 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.936+0000 7f3638c90740 1 -- 172.21.15.62:0/981996919 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x564214d9ee00 con 0x564214d0d9a0 2024-09-15T23:23:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.940+0000 7f360effd640 1 -- 172.21.15.62:0/981996919 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36200bd670 con 0x564214d0d9a0 2024-09-15T23:23:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.940+0000 7f360effd640 1 -- 172.21.15.62:0/981996919 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f362801f450 con 0x564214d0d9a0 2024-09-15T23:23:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.940+0000 7f3638c90740 10 client.17217 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.940+0000 7f3638c90740 10 client.17217 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.940+0000 7f3638c90740 20 client.17217 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.940+0000 7f3638c90740 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564214d9f790 0x564214da1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.940+0000 7f3638c90740 4 client.17217 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:23:48.943 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.940+0000 7f3638c90740 1 -- 172.21.15.62:0/981996919 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564214c78d70 con 0x564214d9f790 2024-09-15T23:23:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.942+0000 7f362effd640 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564214d9f790 0x564214da1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.943+0000 7f362effd640 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564214d9f790 0x564214da1bb0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3174501a509d6d30 server_cookie=218d909ba3ec052a in_seq=0 out_seq=0 2024-09-15T23:23:48.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.945+0000 7f360effd640 10 client.17217 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.990+0000 7f360effd640 1 -- 172.21.15.62:0/981996919 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564214c78d70 con 0x564214d9f790 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.990+0000 7f360effd640 10 client.17217 handle_command_reply: tid=0 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.990+0000 7f3638c90740 1 client.17217 shutdown 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 -- 172.21.15.62:0/981996919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0d9a0 msgr2=0x564214d1be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0d9a0 0x564214d1be20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f362801f040 tx=0x7f36280157c0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 -- 172.21.15.62:0/981996919 shutdown_connections 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564214d9f790 0x564214da1bb0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564214d13370 0x564214d8c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564214d0e2e0 0x564214d1c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 --2- 172.21.15.62:0/981996919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564214d0d9a0 0x564214d1be20 secure :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f362801f040 tx=0x7f36280157c0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 -- 172.21.15.62:0/981996919 >> 172.21.15.62:0/981996919 conn(0x564214d09010 msgr2=0x564214d17bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f360effd640 0 client.17217 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 -- 172.21.15.62:0/981996919 shutdown_connections 2024-09-15T23:23:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 1 -- 172.21.15.62:0/981996919 wait complete. 2024-09-15T23:23:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.991+0000 7f3638c90740 20 client.17217 trim_cache size 0 max 16384 2024-09-15T23:23:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:23:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:23:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:23:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.992+0000 7f3635e1e640 1 -- 172.21.15.62:0/2532693724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36180a3530 msgr2=0x7f36180a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.992+0000 7f3635e1e640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36180a3530 0x7f36180a59f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3628015e60 tx=0x7f3628019000 comp rx=0 tx=0).stop 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.992+0000 7f3635e1e640 1 -- 172.21.15.62:0/2532693724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36300736c0 msgr2=0x7f36301a7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.992+0000 7f3635e1e640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36300736c0 0x7f36301a7180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3620031610 tx=0x7f3620032cf0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.993+0000 7f3635e1e640 1 -- 172.21.15.62:0/2532693724 shutdown_connections 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.993+0000 7f3635e1e640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36180a3530 0x7f36180a59f0 secure :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3628015e60 tx=0x7f3628019000 comp rx=0 tx=0).stop 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.993+0000 7f3635e1e640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3630109700 0x7f36301ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.993+0000 7f3635e1e640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36300736c0 0x7f36301a7180 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.993+0000 7f3635e1e640 1 --2- 172.21.15.62:0/2532693724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3630072d20 0x7f36301a6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.993+0000 7f3635e1e640 1 -- 172.21.15.62:0/2532693724 >> 172.21.15.62:0/2532693724 conn(0x7f363006ece0 msgr2=0x7f3630071720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.995+0000 7f3635e1e640 1 -- 172.21.15.62:0/2532693724 shutdown_connections 2024-09-15T23:23:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:23:48.995+0000 7f3635e1e640 1 -- 172.21.15.62:0/2532693724 wait complete. 2024-09-15T23:23:49.035 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:49.365 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:49 smithi062 ceph-mon[30949]: pgmap v596: 129 pgs: 129 active+clean; 1.5 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 3.3 MiB/s wr, 343 op/s 2024-09-15T23:23:49.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:49 smithi067 ceph-mon[35665]: pgmap v596: 129 pgs: 129 active+clean; 1.5 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 3.3 MiB/s wr, 343 op/s 2024-09-15T23:23:49.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:49 smithi148 ceph-mon[35610]: pgmap v596: 129 pgs: 129 active+clean; 1.5 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 3.3 MiB/s wr, 343 op/s 2024-09-15T23:23:49.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 450.0 s, 1013.4 tps, lat 99.766 ms stddev 121.419 2024-09-15T23:23:49.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.625+0000 7f1318669640 1 -- 172.21.15.62:0/1382044205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 msgr2=0x7f131007b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.625+0000 7f1318669640 1 --2- 172.21.15.62:0/1382044205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 0x7f131007b3c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1308009ca0 tx=0x7f130802a8f0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.627+0000 7f1318669640 1 -- 172.21.15.62:0/1382044205 shutdown_connections 2024-09-15T23:23:49.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.627+0000 7f1318669640 1 --2- 172.21.15.62:0/1382044205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 0x7f131007b3c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.627+0000 7f1318669640 1 --2- 172.21.15.62:0/1382044205 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1310109e00 0x7f13100742d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.627+0000 7f1318669640 1 --2- 172.21.15.62:0/1382044205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1310109450 0x7f1310109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.627+0000 7f1318669640 1 -- 172.21.15.62:0/1382044205 >> 172.21.15.62:0/1382044205 conn(0x7f131006e140 msgr2=0x7f131006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.628+0000 7f1318669640 1 -- 172.21.15.62:0/1382044205 shutdown_connections 2024-09-15T23:23:49.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.628+0000 7f1318669640 1 -- 172.21.15.62:0/1382044205 wait complete. 2024-09-15T23:23:49.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.629+0000 7f1318669640 1 Processor -- start 2024-09-15T23:23:49.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1318669640 1 -- start start 2024-09-15T23:23:49.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1318669640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 0x7f131007b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1318669640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1310109450 0x7f1310076110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1318669640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1310109e00 0x7f1310076650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1318669640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1310082ba0 con 0x7f1310109450 2024-09-15T23:23:49.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1318669640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1310082a20 con 0x7f1310074990 2024-09-15T23:23:49.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1318669640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1310082d20 con 0x7f1310109e00 2024-09-15T23:23:49.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1316405640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 0x7f131007b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1316405640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 0x7f131007b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57018/0 (socket says 172.21.15.62:57018) 2024-09-15T23:23:49.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.630+0000 7f1316405640 1 -- 172.21.15.62:0/1111465158 learned_addr learned my addr 172.21.15.62:0/1111465158 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:49.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.635+0000 7f1315c04640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1310109450 0x7f1310076110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.635+0000 7f1316c06640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1310109e00 0x7f1310076650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.635+0000 7f1316405640 1 -- 172.21.15.62:0/1111465158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1310109e00 msgr2=0x7f1310076650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.635+0000 7f1316405640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1310109e00 0x7f1310076650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.635+0000 7f1316405640 1 -- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1310109450 msgr2=0x7f1310076110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.635+0000 7f1316405640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1310109450 0x7f1310076110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.635+0000 7f1316405640 1 -- 172.21.15.62:0/1111465158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1310076d80 con 0x7f1310074990 2024-09-15T23:23:49.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.636+0000 7f1316405640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 0x7f131007b080 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f130c00e9f0 tx=0x7f130c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.636+0000 7f13077fe640 1 -- 172.21.15.62:0/1111465158 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c00cd80 con 0x7f1310074990 2024-09-15T23:23:49.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.637+0000 7f1318669640 1 -- 172.21.15.62:0/1111465158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1310137290 con 0x7f1310074990 2024-09-15T23:23:49.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.637+0000 7f1318669640 1 -- 172.21.15.62:0/1111465158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13101377d0 con 0x7f1310074990 2024-09-15T23:23:49.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.637+0000 7f13077fe640 1 -- 172.21.15.62:0/1111465158 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f130c010910 con 0x7f1310074990 2024-09-15T23:23:49.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.637+0000 7f13077fe640 1 -- 172.21.15.62:0/1111465158 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c018990 con 0x7f1310074990 2024-09-15T23:23:49.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.645+0000 7f131aa42740 1 Processor -- start 2024-09-15T23:23:49.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.646+0000 7f131aa42740 1 -- start start 2024-09-15T23:23:49.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.646+0000 7f131aa42740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e5b40 0x5625e46e4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.646+0000 7f131aa42740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e45d0 0x5625e46e4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.646+0000 7f131aa42740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e46f1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.646+0000 7f131aa42740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5625e464a2f0 con 0x5625e46e5b40 2024-09-15T23:23:49.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.646+0000 7f131aa42740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5625e45f1cf0 con 0x5625e46ef370 2024-09-15T23:23:49.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.646+0000 7f131aa42740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5625e46462b0 con 0x5625e46e45d0 2024-09-15T23:23:49.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.647+0000 7f1315c04640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e46f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.647+0000 7f1315c04640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e46f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57022/0 (socket says 172.21.15.62:57022) 2024-09-15T23:23:49.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.647+0000 7f1315c04640 1 -- 172.21.15.62:0/4280445081 learned_addr learned my addr 172.21.15.62:0/4280445081 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:49.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.648+0000 7f1315c04640 1 -- 172.21.15.62:0/4280445081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e45d0 msgr2=0x5625e46e4a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:49.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.648+0000 7f1315c04640 1 --2- 172.21.15.62:0/4280445081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e45d0 0x5625e46e4a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.648+0000 7f1315c04640 1 -- 172.21.15.62:0/4280445081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e5b40 msgr2=0x5625e46e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.648+0000 7f1315c04640 1 --2- 172.21.15.62:0/4280445081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e5b40 0x5625e46e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.648+0000 7f1315c04640 1 -- 172.21.15.62:0/4280445081 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625e45d63a0 con 0x5625e46ef370 2024-09-15T23:23:49.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.648+0000 7f1315c04640 1 --2- 172.21.15.62:0/4280445081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e46f1760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f130000bb60 tx=0x7f130000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d7a829f966dcba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.648+0000 7f13077fe640 1 -- 172.21.15.62:0/1111465158 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f130c018c10 con 0x7f1310074990 2024-09-15T23:23:49.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.649+0000 7f1304ff9640 1 -- 172.21.15.62:0/4280445081 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300015070 con 0x5625e46ef370 2024-09-15T23:23:49.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.649+0000 7f1304ff9640 1 -- 172.21.15.62:0/4280445081 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f130000be50 con 0x5625e46ef370 2024-09-15T23:23:49.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.649+0000 7f13077fe640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12fc0a3450 0x7f12fc0a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.650+0000 7f13077fe640 1 -- 172.21.15.62:0/1111465158 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f130c0cffe0 con 0x7f1310074990 2024-09-15T23:23:49.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.650+0000 7f131aa42740 1 -- 172.21.15.62:0/4280445081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 msgr2=0x5625e46f1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.650+0000 7f131aa42740 1 --2- 172.21.15.62:0/4280445081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e46f1760 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f130000bb60 tx=0x7f130000f730 comp rx=0 tx=0).stop 2024-09-15T23:23:49.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.651+0000 7f1316c06640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12fc0a3450 0x7f12fc0a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.652+0000 7f1316c06640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12fc0a3450 0x7f12fc0a5910 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1308006010 tx=0x7f1308005fa0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.654+0000 7f131aa42740 1 -- 172.21.15.62:0/4280445081 shutdown_connections 2024-09-15T23:23:49.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.654+0000 7f131aa42740 1 --2- 172.21.15.62:0/4280445081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e46f1760 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.654+0000 7f131aa42740 1 --2- 172.21.15.62:0/4280445081 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e45d0 0x5625e46e4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.654+0000 7f131aa42740 1 --2- 172.21.15.62:0/4280445081 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e5b40 0x5625e46e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.654+0000 7f131aa42740 1 -- 172.21.15.62:0/4280445081 >> 172.21.15.62:0/4280445081 conn(0x5625e46ad310 msgr2=0x5625e46e0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.654+0000 7f131aa42740 1 -- 172.21.15.62:0/4280445081 shutdown_connections 2024-09-15T23:23:49.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.654+0000 7f131aa42740 1 -- 172.21.15.62:0/4280445081 wait complete. 2024-09-15T23:23:49.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.654+0000 7f131aa42740 1 -- start start 2024-09-15T23:23:49.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.654+0000 7f131aa42740 1 Processor -- start 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.655+0000 7f131aa42740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e45d0 0x5625e46eecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.655+0000 7f131aa42740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e5b40 0x5625e46ebf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.655+0000 7f131aa42740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e46ec4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.655+0000 7f131aa42740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5625e46f20c0 con 0x5625e46ef370 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.655+0000 7f131aa42740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5625e4605060 con 0x5625e46e5b40 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.655+0000 7f131aa42740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5625e46f2240 con 0x5625e46e45d0 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.655+0000 7f131aa42740 10 client.0 _ll_register_callbacks cb 0x5625e43588c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.656+0000 7f1316c06640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e46ec4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.656+0000 7f1316c06640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e46ec4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52114/0 (socket says 172.21.15.62:52114) 2024-09-15T23:23:49.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.656+0000 7f1316c06640 1 -- 172.21.15.62:0/3269130525 learned_addr learned my addr 172.21.15.62:0/3269130525 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:49.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.660+0000 7f1315c04640 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e5b40 0x5625e46ebf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.660+0000 7f1316405640 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e45d0 0x5625e46eecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.661+0000 7f1315c04640 1 -- 172.21.15.62:0/3269130525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e45d0 msgr2=0x5625e46eecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.661+0000 7f1315c04640 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e45d0 0x5625e46eecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.661+0000 7f1315c04640 1 -- 172.21.15.62:0/3269130525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 msgr2=0x5625e46ec4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.661+0000 7f1315c04640 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e46ec4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.661+0000 7f1315c04640 1 -- 172.21.15.62:0/3269130525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625e43bb3c0 con 0x5625e46e5b40 2024-09-15T23:23:49.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.661+0000 7f1315c04640 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e5b40 0x5625e46ebf60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f130000fc30 tx=0x7f130000bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d441666bd254496e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.662+0000 7f12f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:49.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.662+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3269130525 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300015070 con 0x5625e46e5b40 2024-09-15T23:23:49.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.662+0000 7f131aa42740 1 -- 172.21.15.62:0/3269130525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f130000be50 con 0x5625e46e5b40 2024-09-15T23:23:49.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.662+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3269130525 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f13000041f0 con 0x5625e46e5b40 2024-09-15T23:23:49.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.663+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3269130525 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f130002c050 con 0x5625e46e5b40 2024-09-15T23:23:49.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.663+0000 7f131aa42740 10 client.26804 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:49.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.663+0000 7f131aa42740 1 -- 172.21.15.62:0/3269130525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5625e471fba0 con 0x5625e46e5b40 2024-09-15T23:23:49.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.664+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3269130525 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f130000be50 con 0x5625e46e5b40 2024-09-15T23:23:49.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.664+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3269130525 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f1300004a20 con 0x5625e46e5b40 2024-09-15T23:23:49.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.664+0000 7f131aa42740 10 client.26804 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:49.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.664+0000 7f131aa42740 10 client.26804 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:23:49.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.664+0000 7f131aa42740 20 client.26804 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:49.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.664+0000 7f131aa42740 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5625e4721340 0x5625e4723760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.664+0000 7f131aa42740 4 client.26804 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:49.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.664+0000 7f131aa42740 1 -- 172.21.15.62:0/3269130525 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625e4657860 con 0x5625e4721340 2024-09-15T23:23:49.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.667+0000 7f1316405640 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5625e4721340 0x5625e4723760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.671+0000 7f12f9ffb640 10 client.26804 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:23:49.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.671+0000 7f1316405640 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5625e4721340 0x5625e4723760 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aadc850e3bd75c7f server_cookie=e552fc64c1281fea in_seq=0 out_seq=0 2024-09-15T23:23:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3269130525 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5625e4657860 con 0x5625e4721340 2024-09-15T23:23:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f12f9ffb640 10 client.26804 handle_command_reply: tid=0 2024-09-15T23:23:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f131aa42740 1 client.26804 shutdown 2024-09-15T23:23:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f131aa42740 1 -- 172.21.15.62:0/3269130525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e5b40 msgr2=0x5625e46ebf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f131aa42740 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e5b40 0x5625e46ebf60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f130000fc30 tx=0x7f130000bd90 comp rx=0 tx=0).stop 2024-09-15T23:23:49.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f131aa42740 1 -- 172.21.15.62:0/3269130525 shutdown_connections 2024-09-15T23:23:49.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f131aa42740 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5625e4721340 0x5625e4723760 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f131aa42740 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e46ec4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f131aa42740 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e5b40 0x5625e46ebf60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.672+0000 7f131aa42740 1 --2- 172.21.15.62:0/3269130525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e45d0 0x5625e46eecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.673+0000 7f131aa42740 1 -- 172.21.15.62:0/3269130525 >> 172.21.15.62:0/3269130525 conn(0x5625e46ad310 msgr2=0x5625e46e6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.673+0000 7f12f9ffb640 0 client.26804 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:23:49.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.673+0000 7f131aa42740 1 -- 172.21.15.62:0/3269130525 shutdown_connections 2024-09-15T23:23:49.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.673+0000 7f131aa42740 1 -- 172.21.15.62:0/3269130525 wait complete. 2024-09-15T23:23:49.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.673+0000 7f131aa42740 20 client.26804 trim_cache size 0 max 16384 2024-09-15T23:23:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.698+0000 7f131aa42740 1 Processor -- start 2024-09-15T23:23:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.698+0000 7f131aa42740 1 -- start start 2024-09-15T23:23:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.698+0000 7f131aa42740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e45d0 0x5625e46fb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.699+0000 7f131aa42740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 0x5625e4729030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.699+0000 7f131aa42740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e4729570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.699+0000 7f131aa42740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f12f4012560 con 0x5625e46e45d0 2024-09-15T23:23:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.699+0000 7f131aa42740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12f400fe30 con 0x5625e46ef370 2024-09-15T23:23:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.699+0000 7f131aa42740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f12f400dc80 con 0x5625e46e5b40 2024-09-15T23:23:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.699+0000 7f1316c06640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e4729570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.699+0000 7f1316c06640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e4729570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57050/0 (socket says 172.21.15.62:57050) 2024-09-15T23:23:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.699+0000 7f1316c06640 1 -- 172.21.15.62:0/2804964324 learned_addr learned my addr 172.21.15.62:0/2804964324 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.699+0000 7f1315c04640 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 0x5625e4729030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.700+0000 7f1316405640 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e45d0 0x5625e46fb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f1316c06640 1 -- 172.21.15.62:0/2804964324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 msgr2=0x5625e4729030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f1316c06640 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 0x5625e4729030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f1316c06640 1 -- 172.21.15.62:0/2804964324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e45d0 msgr2=0x5625e46fb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f1316c06640 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e45d0 0x5625e46fb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f1316c06640 1 -- 172.21.15.62:0/2804964324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625e46e4d50 con 0x5625e46ef370 2024-09-15T23:23:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f1316405640 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e45d0 0x5625e46fb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:49.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f1315c04640 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 0x5625e4729030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:49.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f1316c06640 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e4729570 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f130802aee0 tx=0x7f130803e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e255f4ea693043f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f12fbfff640 1 -- 172.21.15.62:0/2804964324 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130802f840 con 0x5625e46ef370 2024-09-15T23:23:49.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f12fbfff640 1 -- 172.21.15.62:0/2804964324 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1308036040 con 0x5625e46ef370 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.701+0000 7f12fbfff640 1 -- 172.21.15.62:0/2804964324 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1308030e40 con 0x5625e46ef370 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.702+0000 7f131aa42740 1 -- 172.21.15.62:0/2804964324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 msgr2=0x5625e4729570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.702+0000 7f131aa42740 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e4729570 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f130802aee0 tx=0x7f130803e000 comp rx=0 tx=0).stop 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.702+0000 7f131aa42740 1 -- 172.21.15.62:0/2804964324 shutdown_connections 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.702+0000 7f131aa42740 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46ef370 0x5625e4729570 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.702+0000 7f131aa42740 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 0x5625e4729030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.702+0000 7f131aa42740 1 --2- 172.21.15.62:0/2804964324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46e45d0 0x5625e46fb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.702+0000 7f131aa42740 1 -- 172.21.15.62:0/2804964324 >> 172.21.15.62:0/2804964324 conn(0x5625e46ad310 msgr2=0x5625e46ea830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.702+0000 7f131aa42740 1 -- 172.21.15.62:0/2804964324 shutdown_connections 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.702+0000 7f131aa42740 1 -- 172.21.15.62:0/2804964324 wait complete. 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f131aa42740 1 -- start start 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f131aa42740 1 Processor -- start 2024-09-15T23:23:49.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f131aa42740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e45d0 0x5625e46f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f131aa42740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 0x5625e46f3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f131aa42740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e4789ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f131aa42740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f12f400e2f0 con 0x5625e46ef370 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f131aa42740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f12f400f760 con 0x5625e46e45d0 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f131aa42740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f12f400e9c0 con 0x5625e46e5b40 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f131aa42740 10 client.0 _ll_register_callbacks cb 0x7f130000b9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f1316405640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e45d0 0x5625e46f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f1316c06640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e4789ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f1315c04640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 0x5625e46f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f1316405640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e45d0 0x5625e46f2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57052/0 (socket says 172.21.15.62:57052) 2024-09-15T23:23:49.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.703+0000 7f1316405640 1 -- 172.21.15.62:0/3156661446 learned_addr learned my addr 172.21.15.62:0/3156661446 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:49.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.704+0000 7f1316405640 1 -- 172.21.15.62:0/3156661446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 msgr2=0x5625e46f3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.704+0000 7f1316405640 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 0x5625e46f3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.704+0000 7f1316405640 1 -- 172.21.15.62:0/3156661446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 msgr2=0x5625e4789ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:49.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.704+0000 7f1316405640 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e4789ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.704+0000 7f1316405640 1 -- 172.21.15.62:0/3156661446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625e478a530 con 0x5625e46e45d0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.704+0000 7f1316c06640 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e4789ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.704+0000 7f1316405640 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e45d0 0x5625e46f2ad0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f130c098bf0 tx=0x7f130c0cd5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=444be8d79618e172 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.704+0000 7f12f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.704+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3156661446 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c0bae60 con 0x5625e46e45d0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.705+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3156661446 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f130c098890 con 0x5625e46e45d0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.705+0000 7f131aa42740 1 -- 172.21.15.62:0/3156661446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1308036040 con 0x5625e46e45d0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.705+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3156661446 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f130c0a0150 con 0x5625e46e45d0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.705+0000 7f131aa42740 10 client.26810 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.705+0000 7f131aa42740 1 -- 172.21.15.62:0/3156661446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5625e478b870 con 0x5625e46e45d0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.705+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3156661446 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1308036040 con 0x5625e46e45d0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3156661446 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f130c0a4a70 con 0x5625e46e45d0 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 10 client.26810 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 10 client.26810 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:23:49.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 20 client.26810 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5625e478bf90 0x5625e478e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 4 client.26810 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 1 -- 172.21.15.62:0/3156661446 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5625e4651d70 con 0x5625e478bf90 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f1315c04640 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5625e478bf90 0x5625e478e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f1315c04640 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5625e478bf90 0x5625e478e380 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=525ec1eccca1b099 server_cookie=c1fc7d75c732bbef in_seq=0 out_seq=0 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f12f9ffb640 10 client.26810 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f12f9ffb640 1 -- 172.21.15.62:0/3156661446 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5625e4651d70 con 0x5625e478bf90 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f12f9ffb640 10 client.26810 handle_command_reply: tid=0 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 1 client.26810 shutdown 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 1 -- 172.21.15.62:0/3156661446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e45d0 msgr2=0x5625e46f2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e45d0 0x5625e46f2ad0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f130c098bf0 tx=0x7f130c0cd5d0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 1 -- 172.21.15.62:0/3156661446 shutdown_connections 2024-09-15T23:23:49.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5625e478bf90 0x5625e478e380 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625e46ef370 0x5625e4789ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.707+0000 7f131aa42740 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625e46e5b40 0x5625e46f3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.708+0000 7f12f9ffb640 0 client.26810 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:23:49.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.708+0000 7f131aa42740 1 --2- 172.21.15.62:0/3156661446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625e46e45d0 0x5625e46f2ad0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.708+0000 7f131aa42740 1 -- 172.21.15.62:0/3156661446 >> 172.21.15.62:0/3156661446 conn(0x5625e46ad310 msgr2=0x5625e46ed7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.708+0000 7f131aa42740 1 -- 172.21.15.62:0/3156661446 shutdown_connections 2024-09-15T23:23:49.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.708+0000 7f131aa42740 1 -- 172.21.15.62:0/3156661446 wait complete. 2024-09-15T23:23:49.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.708+0000 7f131aa42740 20 client.26810 trim_cache size 0 max 16384 2024-09-15T23:23:49.713 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:23:49.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 -- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12fc0a3450 msgr2=0x7f12fc0a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12fc0a3450 0x7f12fc0a5910 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1308006010 tx=0x7f1308005fa0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 -- 172.21.15.62:0/1111465158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 msgr2=0x7f131007b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:49.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 0x7f131007b080 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f130c00e9f0 tx=0x7f130c00eeb0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 -- 172.21.15.62:0/1111465158 shutdown_connections 2024-09-15T23:23:49.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f12fc0a3450 0x7f12fc0a5910 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1310109e00 0x7f1310076650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1310109450 0x7f1310076110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 --2- 172.21.15.62:0/1111465158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1310074990 0x7f131007b080 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:49.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 -- 172.21.15.62:0/1111465158 >> 172.21.15.62:0/1111465158 conn(0x7f131006e140 msgr2=0x7f13100707a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:49.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 -- 172.21.15.62:0/1111465158 shutdown_connections 2024-09-15T23:23:49.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:49.709+0000 7f1318669640 1 -- 172.21.15.62:0/1111465158 wait complete. 2024-09-15T23:23:50.542 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:23:50.970 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:51.367 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:51 smithi062 ceph-mon[30949]: pgmap v597: 129 pgs: 129 active+clean; 1.5 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.0 MiB/s wr, 484 op/s 2024-09-15T23:23:51.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.519+0000 7f2afbc71640 1 -- 172.21.15.62:0/753246506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2af4103670 msgr2=0x7f2af410bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.519+0000 7f2afbc71640 1 --2- 172.21.15.62:0/753246506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2af4103670 0x7f2af410bb60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2ae400dd10 tx=0x7f2ae4032a30 comp rx=0 tx=0).stop 2024-09-15T23:23:51.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.520+0000 7f2afbc71640 1 -- 172.21.15.62:0/753246506 shutdown_connections 2024-09-15T23:23:51.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.520+0000 7f2afbc71640 1 --2- 172.21.15.62:0/753246506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2af410c180 0x7f2af410e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.520+0000 7f2afbc71640 1 --2- 172.21.15.62:0/753246506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2af4103670 0x7f2af410bb60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.520+0000 7f2afbc71640 1 --2- 172.21.15.62:0/753246506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2af4102d50 0x7f2af4103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.520+0000 7f2afbc71640 1 -- 172.21.15.62:0/753246506 >> 172.21.15.62:0/753246506 conn(0x7f2af40fc7d0 msgr2=0x7f2af40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:51.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.521+0000 7f2afbc71640 1 -- 172.21.15.62:0/753246506 shutdown_connections 2024-09-15T23:23:51.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.521+0000 7f2afbc71640 1 -- 172.21.15.62:0/753246506 wait complete. 2024-09-15T23:23:51.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:51 smithi067 ceph-mon[35665]: pgmap v597: 129 pgs: 129 active+clean; 1.5 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.0 MiB/s wr, 484 op/s 2024-09-15T23:23:51.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.522+0000 7f2afbc71640 1 Processor -- start 2024-09-15T23:23:51.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.522+0000 7f2afbc71640 1 -- start start 2024-09-15T23:23:51.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.522+0000 7f2afbc71640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2af4102d50 0x7f2af41a7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.522+0000 7f2afbc71640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2af4103670 0x7f2af41a7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.522+0000 7f2afbc71640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2af410c180 0x7f2af41abbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.522+0000 7f2afbc71640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2af4120e10 con 0x7f2af4103670 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.522+0000 7f2afbc71640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2af4120c90 con 0x7f2af410c180 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.522+0000 7f2afbc71640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2af4120f90 con 0x7f2af4102d50 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2af4102d50 0x7f2af41a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2af4102d50 0x7f2af41a7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35980/0 (socket says 172.21.15.62:35980) 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2af9a0d640 1 -- 172.21.15.62:0/3788841003 learned_addr learned my addr 172.21.15.62:0/3788841003 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2afa20e640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2af410c180 0x7f2af41abbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2af920c640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2af4103670 0x7f2af41a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2af9a0d640 1 -- 172.21.15.62:0/3788841003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2af410c180 msgr2=0x7f2af41abbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2af410c180 0x7f2af41abbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2af9a0d640 1 -- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2af4103670 msgr2=0x7f2af41a7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2af4103670 0x7f2af41a7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.523+0000 7f2af9a0d640 1 -- 172.21.15.62:0/3788841003 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af41ac2c0 con 0x7f2af4102d50 2024-09-15T23:23:51.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.525+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2af4102d50 0x7f2af41a7310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2af000b4a0 tx=0x7f2af000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:51.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.526+0000 7f2aeaffd640 1 -- 172.21.15.62:0/3788841003 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0014070 con 0x7f2af4102d50 2024-09-15T23:23:51.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.526+0000 7f2aeaffd640 1 -- 172.21.15.62:0/3788841003 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2af0010af0 con 0x7f2af4102d50 2024-09-15T23:23:51.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.526+0000 7f2aeaffd640 1 -- 172.21.15.62:0/3788841003 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0017d10 con 0x7f2af4102d50 2024-09-15T23:23:51.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.526+0000 7f2afbc71640 1 -- 172.21.15.62:0/3788841003 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af41ac5b0 con 0x7f2af4102d50 2024-09-15T23:23:51.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.526+0000 7f2afbc71640 1 -- 172.21.15.62:0/3788841003 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af4115620 con 0x7f2af4102d50 2024-09-15T23:23:51.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.530+0000 7f2aeaffd640 1 -- 172.21.15.62:0/3788841003 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2af0017070 con 0x7f2af4102d50 2024-09-15T23:23:51.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.531+0000 7f2aeaffd640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ae00a3450 0x7f2ae00a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.531+0000 7f2aeaffd640 1 -- 172.21.15.62:0/3788841003 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2af00cee20 con 0x7f2af4102d50 2024-09-15T23:23:51.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.536+0000 7f2af920c640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ae00a3450 0x7f2ae00a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.536+0000 7f2afe04a740 1 Processor -- start 2024-09-15T23:23:51.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:51 smithi148 ceph-mon[35610]: pgmap v597: 129 pgs: 129 active+clean; 1.5 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.0 MiB/s wr, 484 op/s 2024-09-15T23:23:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.547+0000 7f2afe04a740 1 -- start start 2024-09-15T23:23:51.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.547+0000 7f2af920c640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ae00a3450 0x7f2ae00a5910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4005ab0 tx=0x7f2ae400f040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:51.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.548+0000 7f2afe04a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafc9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.548+0000 7f2afe04a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafc9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.548+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafc9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.548+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafc9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57090/0 (socket says 172.21.15.62:57090) 2024-09-15T23:23:51.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.548+0000 7f2af9a0d640 1 -- 172.21.15.62:0/3388234976 learned_addr learned my addr 172.21.15.62:0/3388234976 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:51.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.548+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 0x55f1dafd6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.548+0000 7f2afe04a740 1 -- 172.21.15.62:0/3388234976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f1daf2f2f0 con 0x55f1dafd4370 2024-09-15T23:23:51.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.548+0000 7f2afe04a740 1 -- 172.21.15.62:0/3388234976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f1daed6cf0 con 0x55f1dafcab40 2024-09-15T23:23:51.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.549+0000 7f2afe04a740 1 -- 172.21.15.62:0/3388234976 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f1daf2b2b0 con 0x55f1dafc95d0 2024-09-15T23:23:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.552+0000 7f2afa20e640 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 0x55f1dafd6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.552+0000 7f2af920c640 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafc9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.553+0000 7f2af920c640 1 -- 172.21.15.62:0/3388234976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 msgr2=0x55f1dafc9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.553+0000 7f2af920c640 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafc9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.553+0000 7f2af920c640 1 -- 172.21.15.62:0/3388234976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 msgr2=0x55f1dafd6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.553+0000 7f2af920c640 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 0x55f1dafd6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.553+0000 7f2af920c640 1 -- 172.21.15.62:0/3388234976 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1daebb3a0 con 0x55f1dafc95d0 2024-09-15T23:23:51.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.553+0000 7f2af920c640 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafc9a10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4033020 tx=0x7f2ae4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7745309f61e36f31 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.553+0000 7f2adffff640 1 -- 172.21.15.62:0/3388234976 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae403b070 con 0x55f1dafc95d0 2024-09-15T23:23:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.554+0000 7f2adffff640 1 -- 172.21.15.62:0/3388234976 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ae4005510 con 0x55f1dafc95d0 2024-09-15T23:23:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.554+0000 7f2adffff640 1 -- 172.21.15.62:0/3388234976 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae400ca80 con 0x55f1dafc95d0 2024-09-15T23:23:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.554+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafc9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.555+0000 7f2afe04a740 1 -- 172.21.15.62:0/3388234976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 msgr2=0x55f1dafc9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.555+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafc9a10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4033020 tx=0x7f2ae4014000 comp rx=0 tx=0).stop 2024-09-15T23:23:51.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.559+0000 7f2afe04a740 1 -- 172.21.15.62:0/3388234976 shutdown_connections 2024-09-15T23:23:51.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.559+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 0x55f1dafd6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.559+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafc9a10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.559+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3388234976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafc9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.559+0000 7f2afe04a740 1 -- 172.21.15.62:0/3388234976 >> 172.21.15.62:0/3388234976 conn(0x55f1daf92310 msgr2=0x55f1dafc5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:51.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.560+0000 7f2afe04a740 1 -- 172.21.15.62:0/3388234976 shutdown_connections 2024-09-15T23:23:51.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.564+0000 7f2afe04a740 1 -- 172.21.15.62:0/3388234976 wait complete. 2024-09-15T23:23:51.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.564+0000 7f2afe04a740 1 -- start start 2024-09-15T23:23:51.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.565+0000 7f2afe04a740 1 Processor -- start 2024-09-15T23:23:51.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.565+0000 7f2afe04a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafd3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.566+0000 7f2afe04a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafd1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.566+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafd3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.566+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafd3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36000/0 (socket says 172.21.15.62:36000) 2024-09-15T23:23:51.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.566+0000 7f2af9a0d640 1 -- 172.21.15.62:0/3105572113 learned_addr learned my addr 172.21.15.62:0/3105572113 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:51.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.566+0000 7f2af920c640 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafd1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.566+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 0x55f1dafd1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.566+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55f1dafd6fb0 con 0x55f1dafd4370 2024-09-15T23:23:51.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.567+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55f1daeea060 con 0x55f1dafcab40 2024-09-15T23:23:51.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.567+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55f1dafd7130 con 0x55f1dafc95d0 2024-09-15T23:23:51.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.567+0000 7f2af9a0d640 1 -- 172.21.15.62:0/3105572113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 msgr2=0x55f1dafd1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:51.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.567+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafd1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.567+0000 7f2af9a0d640 1 -- 172.21.15.62:0/3105572113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 msgr2=0x55f1dafd1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:51.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.568+0000 7f2afa20e640 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 0x55f1dafd1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.568+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 0x55f1dafd1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.568+0000 7f2af9a0d640 1 -- 172.21.15.62:0/3105572113 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1daca03c0 con 0x55f1dafc95d0 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.568+0000 7f2afa20e640 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 0x55f1dafd1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.568+0000 7f2af920c640 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafd1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.568+0000 7f2afe04a740 10 client.0 _ll_register_callbacks cb 0x55f1dac3d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.568+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ae4005510 con 0x55f1dafc95d0 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.569+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafd3d60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2af009aff0 tx=0x7f2af009b060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c1ea9a0d1acd739 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.569+0000 7f2add7fa640 10 client.26617 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.569+0000 7f2add7fa640 1 -- 172.21.15.62:0/3105572113 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af00b4060 con 0x55f1dafc95d0 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.569+0000 7f2add7fa640 1 -- 172.21.15.62:0/3105572113 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2af009f0d0 con 0x55f1dafc95d0 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.569+0000 7f2add7fa640 1 -- 172.21.15.62:0/3105572113 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2af001e070 con 0x55f1dafc95d0 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.569+0000 7f2afe04a740 10 client.26617 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:51.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.570+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55f1dafe2890 con 0x55f1dafc95d0 2024-09-15T23:23:51.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.571+0000 7f2add7fa640 1 -- 172.21.15.62:0/3105572113 <== mon.1 v2:172.21.15.148:3300/0 4 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f2af0010610 con 0x55f1dafc95d0 2024-09-15T23:23:51.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.571+0000 7f2afe04a740 10 client.26617 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:51.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.571+0000 7f2afe04a740 10 client.26617 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:23:51.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.571+0000 7f2afe04a740 20 client.26617 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:51.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.571+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f1dafe3c80 0x55f1dafe6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.571+0000 7f2afe04a740 4 client.26617 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:51.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.571+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1daf3c860 con 0x55f1dafe3c80 2024-09-15T23:23:51.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.575+0000 7f2af920c640 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f1dafe3c80 0x55f1dafe6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.576+0000 7f2af920c640 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f1dafe3c80 0x55f1dafe6110 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f61784952bcc05d server_cookie=9cc5969d6d328e61 in_seq=0 out_seq=0 2024-09-15T23:23:51.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.576+0000 7f2add7fa640 10 client.26617 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:23:51.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.577+0000 7f2add7fa640 1 -- 172.21.15.62:0/3105572113 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f1daf3c860 con 0x55f1dafe3c80 2024-09-15T23:23:51.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.577+0000 7f2add7fa640 10 client.26617 handle_command_reply: tid=0 2024-09-15T23:23:51.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.577+0000 7f2afe04a740 1 client.26617 shutdown 2024-09-15T23:23:51.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.581+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 msgr2=0x55f1dafd3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.581+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafd3d60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2af009aff0 tx=0x7f2af009b060 comp rx=0 tx=0).stop 2024-09-15T23:23:51.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.581+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 shutdown_connections 2024-09-15T23:23:51.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.581+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f1dafe3c80 0x55f1dafe6110 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.581+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafd4370 0x55f1dafd1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.581+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafcab40 0x55f1dafd1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.581+0000 7f2afe04a740 1 --2- 172.21.15.62:0/3105572113 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafc95d0 0x55f1dafd3d60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.581+0000 7f2add7fa640 0 client.26617 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:23:51.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.581+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 >> 172.21.15.62:0/3105572113 conn(0x55f1daf92310 msgr2=0x55f1dafcb9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:51.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.582+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 shutdown_connections 2024-09-15T23:23:51.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.582+0000 7f2afe04a740 1 -- 172.21.15.62:0/3105572113 wait complete. 2024-09-15T23:23:51.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.582+0000 7f2afe04a740 20 client.26617 trim_cache size 0 max 16384 2024-09-15T23:23:51.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afe04a740 1 Processor -- start 2024-09-15T23:23:51.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afe04a740 1 -- start start 2024-09-15T23:23:51.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afe04a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafcc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afe04a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 0x55f1dafcc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afe04a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1dafeba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afe04a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad80124a0 con 0x55f1dafc95d0 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afe04a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad800fd70 con 0x55f1dafd4370 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afe04a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad800dbc0 con 0x55f1dafcab40 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafcc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afa20e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1dafeba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafcc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52202/0 (socket says 172.21.15.62:52202) 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2afa20e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1dafeba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57110/0 (socket says 172.21.15.62:57110) 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2af9a0d640 1 -- 172.21.15.62:0/88823993 learned_addr learned my addr 172.21.15.62:0/88823993 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:51.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.612+0000 7f2af920c640 1 --2- 172.21.15.62:0/88823993 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 0x55f1dafcc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.613+0000 7f2af9a0d640 1 -- 172.21.15.62:0/88823993 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 msgr2=0x55f1dafcc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.613+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/88823993 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 0x55f1dafcc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.613+0000 7f2af9a0d640 1 -- 172.21.15.62:0/88823993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 msgr2=0x55f1dafeba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.613+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/88823993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1dafeba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.613+0000 7f2af9a0d640 1 -- 172.21.15.62:0/88823993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1dafc6dd0 con 0x55f1dafc95d0 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.613+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/88823993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafcc120 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2af009ad60 tx=0x7f2af001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee74aee1233b76c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.613+0000 7f2adf7fe640 1 -- 172.21.15.62:0/88823993 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af0097860 con 0x55f1dafc95d0 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.614+0000 7f2adf7fe640 1 -- 172.21.15.62:0/88823993 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ae4005510 con 0x55f1dafc95d0 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.614+0000 7f2adf7fe640 1 -- 172.21.15.62:0/88823993 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af009a4b0 con 0x55f1dafc95d0 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.615+0000 7f2afe04a740 1 -- 172.21.15.62:0/88823993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 msgr2=0x55f1dafcc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.615+0000 7f2afe04a740 1 --2- 172.21.15.62:0/88823993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafcc120 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f2af009ad60 tx=0x7f2af001e000 comp rx=0 tx=0).stop 2024-09-15T23:23:51.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.616+0000 7f2afe04a740 1 -- 172.21.15.62:0/88823993 shutdown_connections 2024-09-15T23:23:51.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.616+0000 7f2afe04a740 1 --2- 172.21.15.62:0/88823993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1dafeba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.616+0000 7f2afe04a740 1 --2- 172.21.15.62:0/88823993 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 0x55f1dafcc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.616+0000 7f2afe04a740 1 --2- 172.21.15.62:0/88823993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafcc120 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.616+0000 7f2afe04a740 1 -- 172.21.15.62:0/88823993 >> 172.21.15.62:0/88823993 conn(0x55f1daf92310 msgr2=0x55f1dafc4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:51.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.617+0000 7f2afe04a740 1 -- 172.21.15.62:0/88823993 shutdown_connections 2024-09-15T23:23:51.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.617+0000 7f2afe04a740 1 -- 172.21.15.62:0/88823993 wait complete. 2024-09-15T23:23:51.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.617+0000 7f2afe04a740 1 -- start start 2024-09-15T23:23:51.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.617+0000 7f2afe04a740 1 Processor -- start 2024-09-15T23:23:51.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.618+0000 7f2afe04a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafd8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.618+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafd8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.618+0000 7f2af9a0d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafd8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52218/0 (socket says 172.21.15.62:52218) 2024-09-15T23:23:51.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.618+0000 7f2afe04a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 0x55f1dafd8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.618+0000 7f2af9a0d640 1 -- 172.21.15.62:0/105154713 learned_addr learned my addr 172.21.15.62:0/105154713 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:51.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.618+0000 7f2afe04a740 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1db04d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2af920c640 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 0x55f1dafd8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad800e230 con 0x55f1dafc95d0 2024-09-15T23:23:51.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afa20e640 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1db04d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad800f6a0 con 0x55f1dafd4370 2024-09-15T23:23:51.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad800e900 con 0x55f1dafcab40 2024-09-15T23:23:51.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afe04a740 10 client.0 _ll_register_callbacks cb 0x7f2af009ae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afa20e640 1 -- 172.21.15.62:0/105154713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 msgr2=0x55f1dafd8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afa20e640 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 0x55f1dafd8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afa20e640 1 -- 172.21.15.62:0/105154713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 msgr2=0x55f1dafd8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afa20e640 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafd8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afa20e640 1 -- 172.21.15.62:0/105154713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1db04df30 con 0x55f1dafd4370 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ae4005510 con 0x55f1dafd4370 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.619+0000 7f2afa20e640 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1db04d9f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55f1dafca070 tx=0x7f2aec004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79859d85003c5c57 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.620+0000 7f2add7fa640 10 client.26825 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.620+0000 7f2add7fa640 1 -- 172.21.15.62:0/105154713 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec004b20 con 0x55f1dafd4370 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.620+0000 7f2add7fa640 1 -- 172.21.15.62:0/105154713 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2aec004cc0 con 0x55f1dafd4370 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.620+0000 7f2add7fa640 1 -- 172.21.15.62:0/105154713 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2aec012720 con 0x55f1dafd4370 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.620+0000 7f2afe04a740 10 client.26825 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.620+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55f1db04f110 con 0x55f1dafd4370 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.620+0000 7f2add7fa640 1 -- 172.21.15.62:0/105154713 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f2aec0128c0 con 0x55f1dafd4370 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.620+0000 7f2afe04a740 10 client.26825 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.621+0000 7f2afe04a740 10 client.26825 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.621+0000 7f2afe04a740 20 client.26825 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.621+0000 7f2afe04a740 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f1dafe3c80 0x55f1db051730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.621+0000 7f2afe04a740 4 client.26825 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.621+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1daf36d70 con 0x55f1dafe3c80 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.621+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f1dafe3c80 0x55f1db051730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.621+0000 7f2af9a0d640 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f1dafe3c80 0x55f1db051730 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=36fd3813f8e52d7c server_cookie=e83d019c59790b72 in_seq=0 out_seq=0 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.621+0000 7f2add7fa640 10 client.26825 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.622+0000 7f2add7fa640 1 -- 172.21.15.62:0/105154713 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1daf36d70 con 0x55f1dafe3c80 2024-09-15T23:23:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.622+0000 7f2add7fa640 10 client.26825 handle_command_reply: tid=0 2024-09-15T23:23:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.622+0000 7f2afe04a740 1 client.26825 shutdown 2024-09-15T23:23:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.622+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 msgr2=0x55f1db04d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.622+0000 7f2afe04a740 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1db04d9f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55f1dafca070 tx=0x7f2aec004410 comp rx=0 tx=0).stop 2024-09-15T23:23:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.623+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 shutdown_connections 2024-09-15T23:23:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.623+0000 7f2afe04a740 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55f1dafe3c80 0x55f1db051730 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.623+0000 7f2afe04a740 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f1dafd4370 0x55f1db04d9f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.623+0000 7f2afe04a740 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55f1dafcab40 0x55f1dafd8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.623+0000 7f2afe04a740 1 --2- 172.21.15.62:0/105154713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f1dafc95d0 0x55f1dafd8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.623+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 >> 172.21.15.62:0/105154713 conn(0x55f1daf92310 msgr2=0x55f1dafd2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.623+0000 7f2add7fa640 0 client.26825 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:23:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.623+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 shutdown_connections 2024-09-15T23:23:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.624+0000 7f2afe04a740 1 -- 172.21.15.62:0/105154713 wait complete. 2024-09-15T23:23:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.625+0000 7f2afe04a740 20 client.26825 trim_cache size 0 max 16384 2024-09-15T23:23:51.627 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:23:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 -- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ae00a3450 msgr2=0x7f2ae00a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ae00a3450 0x7f2ae00a5910 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4005ab0 tx=0x7f2ae400f040 comp rx=0 tx=0).stop 2024-09-15T23:23:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 -- 172.21.15.62:0/3788841003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2af4102d50 msgr2=0x7f2af41a7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2af4102d50 0x7f2af41a7310 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2af000b4a0 tx=0x7f2af000b960 comp rx=0 tx=0).stop 2024-09-15T23:23:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 -- 172.21.15.62:0/3788841003 shutdown_connections 2024-09-15T23:23:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ae00a3450 0x7f2ae00a5910 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2af410c180 0x7f2af41abbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2af4103670 0x7f2af41a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 --2- 172.21.15.62:0/3788841003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2af4102d50 0x7f2af41a7310 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:51.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 -- 172.21.15.62:0/3788841003 >> 172.21.15.62:0/3788841003 conn(0x7f2af40fc7d0 msgr2=0x7f2af40fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:51.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 -- 172.21.15.62:0/3788841003 shutdown_connections 2024-09-15T23:23:51.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:51.626+0000 7f2afbc71640 1 -- 172.21.15.62:0/3788841003 wait complete. 2024-09-15T23:23:52.485 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:23:52.922 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:53.353 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:53 smithi062 ceph-mon[30949]: pgmap v598: 129 pgs: 129 active+clean; 1.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 556 op/s 2024-09-15T23:23:53.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:53 smithi067 ceph-mon[35665]: pgmap v598: 129 pgs: 129 active+clean; 1.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 556 op/s 2024-09-15T23:23:53.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:53 smithi148 ceph-mon[35610]: pgmap v598: 129 pgs: 129 active+clean; 1.5 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 5.5 MiB/s wr, 556 op/s 2024-09-15T23:23:53.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.668+0000 7f4d11d52640 1 -- 172.21.15.62:0/269765858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0c1009a0 msgr2=0x7f4d0c100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.668+0000 7f4d11d52640 1 --2- 172.21.15.62:0/269765858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0c1009a0 0x7f4d0c100d80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009900 tx=0x7f4d0002a4b0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.671+0000 7f4d11d52640 1 -- 172.21.15.62:0/269765858 shutdown_connections 2024-09-15T23:23:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.671+0000 7f4d11d52640 1 --2- 172.21.15.62:0/269765858 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0c105db0 0x7f4d0c106240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.671+0000 7f4d11d52640 1 --2- 172.21.15.62:0/269765858 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d0c1012c0 0x7f4d0c105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.671+0000 7f4d11d52640 1 --2- 172.21.15.62:0/269765858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0c1009a0 0x7f4d0c100d80 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.671+0000 7f4d11d52640 1 -- 172.21.15.62:0/269765858 >> 172.21.15.62:0/269765858 conn(0x7f4d0c0fc630 msgr2=0x7f4d0c0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:53.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.672+0000 7f4d11d52640 1 -- 172.21.15.62:0/269765858 shutdown_connections 2024-09-15T23:23:53.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.672+0000 7f4d11d52640 1 -- 172.21.15.62:0/269765858 wait complete. 2024-09-15T23:23:53.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.673+0000 7f4d11d52640 1 Processor -- start 2024-09-15T23:23:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.675+0000 7f4d11d52640 1 -- start start 2024-09-15T23:23:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.675+0000 7f4d11d52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0c1009a0 0x7f4d0c072b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.676+0000 7f4d11d52640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0c1012c0 0x7f4d0c0730d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.676+0000 7f4d11d52640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d0c105db0 0x7f4d0c109e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.676+0000 7f4d11d52640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c118410 con 0x7f4d0c1009a0 2024-09-15T23:23:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.676+0000 7f4d11d52640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c118290 con 0x7f4d0c1012c0 2024-09-15T23:23:53.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.676+0000 7f4d11d52640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c118590 con 0x7f4d0c105db0 2024-09-15T23:23:53.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.678+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0c1009a0 0x7f4d0c072b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.678+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0c1009a0 0x7f4d0c072b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33112/0 (socket says 172.21.15.62:33112) 2024-09-15T23:23:53.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.678+0000 7f4d0b7fe640 1 -- 172.21.15.62:0/2069886129 learned_addr learned my addr 172.21.15.62:0/2069886129 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:53.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.678+0000 7f4d0affd640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0c1012c0 0x7f4d0c0730d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.678+0000 7f4d0affd640 1 -- 172.21.15.62:0/2069886129 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d0c105db0 msgr2=0x7f4d0c109e30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:23:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.678+0000 7f4d0affd640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d0c105db0 0x7f4d0c109e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.678+0000 7f4d0affd640 1 -- 172.21.15.62:0/2069886129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0c1009a0 msgr2=0x7f4d0c072b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.678+0000 7f4d0affd640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0c1009a0 0x7f4d0c072b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.678+0000 7f4d0affd640 1 -- 172.21.15.62:0/2069886129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0c10a510 con 0x7f4d0c1012c0 2024-09-15T23:23:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.679+0000 7f4d0affd640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0c1012c0 0x7f4d0c0730d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400cab0 tx=0x7f4d0400cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.679+0000 7f4d08ff9640 1 -- 172.21.15.62:0/2069886129 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0400ae20 con 0x7f4d0c1012c0 2024-09-15T23:23:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.679+0000 7f4d08ff9640 1 -- 172.21.15.62:0/2069886129 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d0400f930 con 0x7f4d0c1012c0 2024-09-15T23:23:53.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.679+0000 7f4d08ff9640 1 -- 172.21.15.62:0/2069886129 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d04018b70 con 0x7f4d0c1012c0 2024-09-15T23:23:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.682+0000 7f4d11d52640 1 -- 172.21.15.62:0/2069886129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d0c10a800 con 0x7f4d0c1012c0 2024-09-15T23:23:53.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.682+0000 7f4d11d52640 1 -- 172.21.15.62:0/2069886129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d0c112050 con 0x7f4d0c1012c0 2024-09-15T23:23:53.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.690+0000 7f4d1412b740 1 Processor -- start 2024-09-15T23:23:53.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.692+0000 7f4d08ff9640 1 -- 172.21.15.62:0/2069886129 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4d04018d10 con 0x7f4d0c1012c0 2024-09-15T23:23:53.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.692+0000 7f4d1412b740 1 -- start start 2024-09-15T23:23:53.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.695+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6dccf0 0x558d6e6db240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.695+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6dbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.695+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6e6520 0x558d6e6e8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.695+0000 7f4d1412b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558d6e6412f0 con 0x558d6e6db780 2024-09-15T23:23:53.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.695+0000 7f4d1412b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558d6e5e8cf0 con 0x558d6e6dccf0 2024-09-15T23:23:53.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.695+0000 7f4d1412b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558d6e63d2b0 con 0x558d6e6e6520 2024-09-15T23:23:53.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.696+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6dccf0 0x558d6e6db240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.696+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6dccf0 0x558d6e6db240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36452/0 (socket says 172.21.15.62:36452) 2024-09-15T23:23:53.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.696+0000 7f4d0bfff640 1 -- 172.21.15.62:0/966528218 learned_addr learned my addr 172.21.15.62:0/966528218 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:53.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.696+0000 7f4d0bfff640 1 -- 172.21.15.62:0/966528218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6e6520 msgr2=0x558d6e6e8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:53.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.697+0000 7f4d08ff9640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4cf40a3300 0x7f4cf40a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.697+0000 7f4d08ff9640 1 -- 172.21.15.62:0/2069886129 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4d040cfed0 con 0x7f4d0c1012c0 2024-09-15T23:23:53.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.697+0000 7f4d0affd640 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6e6520 0x558d6e6e8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.698+0000 7f4d0b7fe640 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6dbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.699+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6e6520 0x558d6e6e8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.699+0000 7f4d0bfff640 1 -- 172.21.15.62:0/966528218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 msgr2=0x558d6e6dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:53.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.699+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6dbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.699+0000 7f4d0bfff640 1 -- 172.21.15.62:0/966528218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d6e5cd3a0 con 0x558d6e6dccf0 2024-09-15T23:23:53.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.699+0000 7f4d0b7fe640 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6dbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:53.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.699+0000 7f4d0affd640 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6e6520 0x558d6e6e8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:23:53.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.700+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6dccf0 0x558d6e6db240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00a900 tx=0x7f4cfc00adc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc099279e0c7f2e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:53.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.700+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4cf40a3300 0x7f4cf40a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.701+0000 7f4cf1ffb640 1 -- 172.21.15.62:0/966528218 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc00f650 con 0x558d6e6dccf0 2024-09-15T23:23:53.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.701+0000 7f4cf1ffb640 1 -- 172.21.15.62:0/966528218 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cfc00fd50 con 0x558d6e6dccf0 2024-09-15T23:23:53.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.701+0000 7f4cf1ffb640 1 -- 172.21.15.62:0/966528218 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc00e3d0 con 0x558d6e6dccf0 2024-09-15T23:23:53.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.702+0000 7f4d1412b740 1 -- 172.21.15.62:0/966528218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6dccf0 msgr2=0x558d6e6db240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.702+0000 7f4d1412b740 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6dccf0 0x558d6e6db240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00a900 tx=0x7f4cfc00adc0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.702+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4cf40a3300 0x7f4cf40a57c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc01bfd0 tx=0x7f4cfc00eae0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:53.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.703+0000 7f4d1412b740 1 -- 172.21.15.62:0/966528218 shutdown_connections 2024-09-15T23:23:53.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.703+0000 7f4d1412b740 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6e6520 0x558d6e6e8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.703+0000 7f4d1412b740 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6dbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.703+0000 7f4d1412b740 1 --2- 172.21.15.62:0/966528218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6dccf0 0x558d6e6db240 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.703+0000 7f4d1412b740 1 -- 172.21.15.62:0/966528218 >> 172.21.15.62:0/966528218 conn(0x558d6e6a4310 msgr2=0x558d6e6d7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:53.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.705+0000 7f4d1412b740 1 -- 172.21.15.62:0/966528218 shutdown_connections 2024-09-15T23:23:53.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.707+0000 7f4d1412b740 1 -- 172.21.15.62:0/966528218 wait complete. 2024-09-15T23:23:53.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.708+0000 7f4d1412b740 1 -- start start 2024-09-15T23:23:53.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.708+0000 7f4d1412b740 1 Processor -- start 2024-09-15T23:23:53.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6e5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6e3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6e3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d1412b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558d6e6e9220 con 0x558d6e6db780 2024-09-15T23:23:53.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d1412b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558d6e5fc060 con 0x558d6e6e6520 2024-09-15T23:23:53.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d1412b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558d6e6e93a0 con 0x558d6e6dccf0 2024-09-15T23:23:53.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d1412b740 10 client.0 _ll_register_callbacks cb 0x558d6e34f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:53.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6e3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d0bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6e3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36456/0 (socket says 172.21.15.62:36456) 2024-09-15T23:23:53.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d0bfff640 1 -- 172.21.15.62:0/4282097013 learned_addr learned my addr 172.21.15.62:0/4282097013 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d0bfff640 1 -- 172.21.15.62:0/4282097013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 msgr2=0x558d6e6e3120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6e3120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.710+0000 7f4d0bfff640 1 -- 172.21.15.62:0/4282097013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 msgr2=0x558d6e6e5e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.711+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6e5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.711+0000 7f4d0bfff640 1 -- 172.21.15.62:0/4282097013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d6e3b23c0 con 0x558d6e6e6520 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.711+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6e3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00ae00 tx=0x7f4cfc013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d6583e231d6cfd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.711+0000 7f4ceaffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.711+0000 7f4ceaffd640 1 -- 172.21.15.62:0/4282097013 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cfc035060 con 0x558d6e6e6520 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.711+0000 7f4ceaffd640 1 -- 172.21.15.62:0/4282097013 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4cfc01b620 con 0x558d6e6e6520 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.711+0000 7f4d1412b740 1 -- 172.21.15.62:0/4282097013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4cfc00fd50 con 0x558d6e6e6520 2024-09-15T23:23:53.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.712+0000 7f4ceaffd640 1 -- 172.21.15.62:0/4282097013 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4cfc03c050 con 0x558d6e6e6520 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.712+0000 7f4d1412b740 10 client.26834 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.712+0000 7f4d1412b740 1 -- 172.21.15.62:0/4282097013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558d6e6f4ba0 con 0x558d6e6e6520 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.712+0000 7f4ceaffd640 1 -- 172.21.15.62:0/4282097013 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cfc00fd50 con 0x558d6e6e6520 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.712+0000 7f4ceaffd640 1 -- 172.21.15.62:0/4282097013 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4cfc00f450 con 0x558d6e6e6520 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.712+0000 7f4d1412b740 10 client.26834 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.712+0000 7f4d1412b740 10 client.26834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.713+0000 7f4d1412b740 20 client.26834 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.713+0000 7f4d1412b740 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558d6e6f6060 0x558d6e6f8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.713+0000 7f4d1412b740 4 client.26834 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.713+0000 7f4d1412b740 1 -- 172.21.15.62:0/4282097013 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d6e64e860 con 0x558d6e6f6060 2024-09-15T23:23:53.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.713+0000 7f4d0b7fe640 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558d6e6f6060 0x558d6e6f8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.714+0000 7f4d0b7fe640 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558d6e6f6060 0x558d6e6f8450 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4468b156590391a0 server_cookie=77628cce708c21e5 in_seq=0 out_seq=0 2024-09-15T23:23:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.714+0000 7f4ceaffd640 10 client.26834 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4ceaffd640 1 -- 172.21.15.62:0/4282097013 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558d6e64e860 con 0x558d6e6f6060 2024-09-15T23:23:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4ceaffd640 10 client.26834 handle_command_reply: tid=0 2024-09-15T23:23:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 client.26834 shutdown 2024-09-15T23:23:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 -- 172.21.15.62:0/4282097013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 msgr2=0x558d6e6e3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6e3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc00ae00 tx=0x7f4cfc013000 comp rx=0 tx=0).stop 2024-09-15T23:23:53.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 -- 172.21.15.62:0/4282097013 shutdown_connections 2024-09-15T23:23:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558d6e6f6060 0x558d6e6f8450 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4ceaffd640 0 client.26834 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6e3680 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6e3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 --2- 172.21.15.62:0/4282097013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6e5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 -- 172.21.15.62:0/4282097013 >> 172.21.15.62:0/4282097013 conn(0x558d6e6a4310 msgr2=0x558d6e6ddae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 -- 172.21.15.62:0/4282097013 shutdown_connections 2024-09-15T23:23:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 1 -- 172.21.15.62:0/4282097013 wait complete. 2024-09-15T23:23:53.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.715+0000 7f4d1412b740 20 client.26834 trim_cache size 0 max 16384 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.736+0000 7f4d1412b740 1 Processor -- start 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.736+0000 7f4d1412b740 1 -- start start 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.736+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6db780 0x558d6e6d6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.736+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6de2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6e6520 0x558d6e6de810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d1412b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c1c8a70 con 0x558d6e6e6520 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d1412b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c1c6340 con 0x558d6e6db780 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d1412b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c1c4190 con 0x558d6e6dccf0 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6db780 0x558d6e6d6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6db780 0x558d6e6d6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36460/0 (socket says 172.21.15.62:36460) 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0b7fe640 1 -- 172.21.15.62:0/238350326 learned_addr learned my addr 172.21.15.62:0/238350326 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:53.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0affd640 1 --2- 172.21.15.62:0/238350326 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6de2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0b7fe640 1 -- 172.21.15.62:0/238350326 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 msgr2=0x558d6e6de2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:53.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0b7fe640 1 --2- 172.21.15.62:0/238350326 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6de2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0b7fe640 1 -- 172.21.15.62:0/238350326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6e6520 msgr2=0x558d6e6de810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:53.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0b7fe640 1 --2- 172.21.15.62:0/238350326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6e6520 0x558d6e6de810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0b7fe640 1 -- 172.21.15.62:0/238350326 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d6e6d8f80 con 0x558d6e6db780 2024-09-15T23:23:53.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.737+0000 7f4d0b7fe640 1 --2- 172.21.15.62:0/238350326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6db780 0x558d6e6d6dc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009230 tx=0x7f4d00009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fda0d5f69e0707d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:53.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.738+0000 7f4cf17fa640 1 -- 172.21.15.62:0/238350326 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00049070 con 0x558d6e6db780 2024-09-15T23:23:53.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.738+0000 7f4cf17fa640 1 -- 172.21.15.62:0/238350326 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d000043c0 con 0x558d6e6db780 2024-09-15T23:23:53.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.738+0000 7f4cf17fa640 1 -- 172.21.15.62:0/238350326 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00005100 con 0x558d6e6db780 2024-09-15T23:23:53.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.738+0000 7f4d1412b740 1 -- 172.21.15.62:0/238350326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6db780 msgr2=0x558d6e6d6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.738+0000 7f4d1412b740 1 --2- 172.21.15.62:0/238350326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6db780 0x558d6e6d6dc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4d00009230 tx=0x7f4d00009650 comp rx=0 tx=0).stop 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.738+0000 7f4d1412b740 1 -- 172.21.15.62:0/238350326 shutdown_connections 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.738+0000 7f4d1412b740 1 --2- 172.21.15.62:0/238350326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6e6520 0x558d6e6de810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.738+0000 7f4d1412b740 1 --2- 172.21.15.62:0/238350326 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6de2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.739+0000 7f4d1412b740 1 --2- 172.21.15.62:0/238350326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6db780 0x558d6e6d6dc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.739+0000 7f4d1412b740 1 -- 172.21.15.62:0/238350326 >> 172.21.15.62:0/238350326 conn(0x558d6e6a4310 msgr2=0x558d6e6d66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.739+0000 7f4d1412b740 1 -- 172.21.15.62:0/238350326 shutdown_connections 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.739+0000 7f4d1412b740 1 -- 172.21.15.62:0/238350326 wait complete. 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.739+0000 7f4d1412b740 1 -- start start 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.739+0000 7f4d1412b740 1 Processor -- start 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6eeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6e9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6eeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d1412b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6ea430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6eeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33154/0 (socket says 172.21.15.62:33154) 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0b7fe640 1 -- 172.21.15.62:0/2398558600 learned_addr learned my addr 172.21.15.62:0/2398558600 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c1c4800 con 0x558d6e6db780 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c1c5c70 con 0x558d6e6e6520 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c1c4ed0 con 0x558d6e6dccf0 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d1412b740 10 client.0 _ll_register_callbacks cb 0x7f4cfc00a750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0affd640 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6e9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6ea430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0affd640 1 -- 172.21.15.62:0/2398558600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 msgr2=0x558d6e6ea430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0affd640 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6ea430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0affd640 1 -- 172.21.15.62:0/2398558600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 msgr2=0x558d6e6eeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0affd640 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6eeb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0affd640 1 -- 172.21.15.62:0/2398558600 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d6e6eaa20 con 0x558d6e6dccf0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0bfff640 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4d0affd640 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6e9ef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4d040cd620 tx=0x7f4d0409bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b72fdd847248e7c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.740+0000 7f4ceaffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.741+0000 7f4ceaffd640 1 -- 172.21.15.62:0/2398558600 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d040a0520 con 0x558d6e6dccf0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.741+0000 7f4ceaffd640 1 -- 172.21.15.62:0/2398558600 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4d040a0160 con 0x558d6e6dccf0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.741+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d000043c0 con 0x558d6e6dccf0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.741+0000 7f4ceaffd640 1 -- 172.21.15.62:0/2398558600 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4d040a4de0 con 0x558d6e6dccf0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.741+0000 7f4d1412b740 10 client.26629 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.741+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558d6e760860 con 0x558d6e6dccf0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.741+0000 7f4ceaffd640 1 -- 172.21.15.62:0/2398558600 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d000043c0 con 0x558d6e6dccf0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.741+0000 7f4ceaffd640 1 -- 172.21.15.62:0/2398558600 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4d04096ce0 con 0x558d6e6dccf0 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.742+0000 7f4d1412b740 10 client.26629 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.742+0000 7f4d1412b740 10 client.26629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:53.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.742+0000 7f4d1412b740 20 client.26629 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.742+0000 7f4d1412b740 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558d6e6f6060 0x558d6e763030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.742+0000 7f4d1412b740 4 client.26629 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.742+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558d6e648d70 con 0x558d6e6f6060 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.742+0000 7f4d0b7fe640 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558d6e6f6060 0x558d6e763030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.742+0000 7f4d0b7fe640 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558d6e6f6060 0x558d6e763030 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7a04b0efac11878 server_cookie=4cc202dff78eca15 in_seq=0 out_seq=0 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.742+0000 7f4ceaffd640 10 client.26629 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4ceaffd640 1 -- 172.21.15.62:0/2398558600 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558d6e648d70 con 0x558d6e6f6060 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4ceaffd640 10 client.26629 handle_command_reply: tid=0 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 client.26629 shutdown 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 msgr2=0x558d6e6e9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6e9ef0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4d040cd620 tx=0x7f4d0409bd80 comp rx=0 tx=0).stop 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 shutdown_connections 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558d6e6f6060 0x558d6e763030 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d6e6e6520 0x558d6e6ea430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558d6e6dccf0 0x558d6e6e9ef0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4ceaffd640 0 client.26629 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 --2- 172.21.15.62:0/2398558600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558d6e6db780 0x558d6e6eeb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 >> 172.21.15.62:0/2398558600 conn(0x558d6e6a4310 msgr2=0x558d6e6d6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 shutdown_connections 2024-09-15T23:23:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 1 -- 172.21.15.62:0/2398558600 wait complete. 2024-09-15T23:23:53.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.743+0000 7f4d1412b740 20 client.26629 trim_cache size 0 max 16384 2024-09-15T23:23:53.749 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:23:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 -- 172.21.15.62:0/2069886129 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4cf40a3300 msgr2=0x7f4cf40a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4cf40a3300 0x7f4cf40a57c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc01bfd0 tx=0x7f4cfc00eae0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 -- 172.21.15.62:0/2069886129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0c1012c0 msgr2=0x7f4d0c0730d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0c1012c0 0x7f4d0c0730d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4d0400cab0 tx=0x7f4d0400cf70 comp rx=0 tx=0).stop 2024-09-15T23:23:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 -- 172.21.15.62:0/2069886129 shutdown_connections 2024-09-15T23:23:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4cf40a3300 0x7f4cf40a57c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4d0c105db0 0x7f4d0c109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d0c1012c0 0x7f4d0c0730d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 --2- 172.21.15.62:0/2069886129 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4d0c1009a0 0x7f4d0c072b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 -- 172.21.15.62:0/2069886129 >> 172.21.15.62:0/2069886129 conn(0x7f4d0c0fc630 msgr2=0x7f4d0c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 -- 172.21.15.62:0/2069886129 shutdown_connections 2024-09-15T23:23:53.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:53.744+0000 7f4d11d52640 1 -- 172.21.15.62:0/2069886129 wait complete. 2024-09-15T23:23:54.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 455.0 s, 1142.4 tps, lat 86.626 ms stddev 105.091 2024-09-15T23:23:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:55 smithi062 ceph-mon[30949]: pgmap v599: 129 pgs: 129 active+clean; 1.5 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 MiB/s wr, 583 op/s 2024-09-15T23:23:55.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:55 smithi067 ceph-mon[35665]: pgmap v599: 129 pgs: 129 active+clean; 1.5 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 MiB/s wr, 583 op/s 2024-09-15T23:23:55.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:55 smithi148 ceph-mon[35610]: pgmap v599: 129 pgs: 129 active+clean; 1.5 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.9 MiB/s wr, 583 op/s 2024-09-15T23:23:55.553 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:23:55.554 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:23:56.057 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:56.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.642+0000 7f902d6f8640 1 -- 172.21.15.62:0/69044899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f902811b140 msgr2=0x7f902811d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:56.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.642+0000 7f902d6f8640 1 --2- 172.21.15.62:0/69044899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f902811b140 0x7f902811d530 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9018009be0 tx=0x7f901802a6e0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.643+0000 7f9026ffd640 1 --2- 172.21.15.62:0/69044899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028109450 0x7f9028110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:23:56.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.645+0000 7f902d6f8640 1 -- 172.21.15.62:0/69044899 shutdown_connections 2024-09-15T23:23:56.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.645+0000 7f902d6f8640 1 --2- 172.21.15.62:0/69044899 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f902811b140 0x7f902811d530 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.645+0000 7f902d6f8640 1 --2- 172.21.15.62:0/69044899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9028109830 0x7f9028109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.645+0000 7f902d6f8640 1 --2- 172.21.15.62:0/69044899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028109450 0x7f9028110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.645+0000 7f902d6f8640 1 -- 172.21.15.62:0/69044899 >> 172.21.15.62:0/69044899 conn(0x7f902806e140 msgr2=0x7f902806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:56.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.645+0000 7f902d6f8640 1 -- 172.21.15.62:0/69044899 shutdown_connections 2024-09-15T23:23:56.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.645+0000 7f902d6f8640 1 -- 172.21.15.62:0/69044899 wait complete. 2024-09-15T23:23:56.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.646+0000 7f902d6f8640 1 Processor -- start 2024-09-15T23:23:56.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.646+0000 7f902d6f8640 1 -- start start 2024-09-15T23:23:56.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.647+0000 7f902d6f8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028109450 0x7f90281a71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.647+0000 7f902d6f8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9028109830 0x7f90281a7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.647+0000 7f9026ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028109450 0x7f90281a71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.647+0000 7f9026ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028109450 0x7f90281a71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36474/0 (socket says 172.21.15.62:36474) 2024-09-15T23:23:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.648+0000 7f9026ffd640 1 -- 172.21.15.62:0/350991063 learned_addr learned my addr 172.21.15.62:0/350991063 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.648+0000 7f902d6f8640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f902811b140 0x7f90281abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.648+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9028120e00 con 0x7f902811b140 2024-09-15T23:23:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.648+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9028120c80 con 0x7f9028109450 2024-09-15T23:23:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.648+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9028120f80 con 0x7f9028109830 2024-09-15T23:23:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.649+0000 7f90267fc640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9028109830 0x7f90281a7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.649+0000 7f90277fe640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f902811b140 0x7f90281abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.649+0000 7f90267fc640 1 -- 172.21.15.62:0/350991063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028109450 msgr2=0x7f90281a71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:56.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.649+0000 7f90267fc640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028109450 0x7f90281a71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.649+0000 7f90267fc640 1 -- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f902811b140 msgr2=0x7f90281abab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:56.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.649+0000 7f90267fc640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f902811b140 0x7f90281abab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.649+0000 7f90267fc640 1 -- 172.21.15.62:0/350991063 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90281ac190 con 0x7f9028109830 2024-09-15T23:23:56.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.650+0000 7f90267fc640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9028109830 0x7f90281a7720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f901000d720 tx=0x7f901000dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:56.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.650+0000 7f901ffff640 1 -- 172.21.15.62:0/350991063 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9010017070 con 0x7f9028109830 2024-09-15T23:23:56.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.650+0000 7f901ffff640 1 -- 172.21.15.62:0/350991063 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f901000de30 con 0x7f9028109830 2024-09-15T23:23:56.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.650+0000 7f901ffff640 1 -- 172.21.15.62:0/350991063 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90100047a0 con 0x7f9028109830 2024-09-15T23:23:56.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.650+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90281ac480 con 0x7f9028109830 2024-09-15T23:23:56.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.650+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90281ac8b0 con 0x7f9028109830 2024-09-15T23:23:56.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.653+0000 7f901ffff640 1 -- 172.21.15.62:0/350991063 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9010004940 con 0x7f9028109830 2024-09-15T23:23:56.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.656+0000 7f901ffff640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f90140a3410 0x7f90140a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.656+0000 7f901ffff640 1 -- 172.21.15.62:0/350991063 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f90100cf5b0 con 0x7f9028109830 2024-09-15T23:23:56.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.656+0000 7f902fad1740 1 Processor -- start 2024-09-15T23:23:56.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.657+0000 7f902fad1740 1 -- start start 2024-09-15T23:23:56.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.657+0000 7f902fad1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2bec6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.657+0000 7f902fad1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bec7480 0x55bf2bed1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.657+0000 7f902fad1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bed4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.657+0000 7f9026ffd640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f90140a3410 0x7f90140a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.657+0000 7f902fad1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bf2be2c280 con 0x55bf2bed1d40 2024-09-15T23:23:56.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.657+0000 7f902fad1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55bf2bdd3b10 con 0x55bf2bec7480 2024-09-15T23:23:56.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.657+0000 7f902fad1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bf2be28240 con 0x55bf2bec6b60 2024-09-15T23:23:56.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.658+0000 7f90267fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bec7480 0x55bf2bed1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.658+0000 7f90277fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bed4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.658+0000 7f9026ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2bec6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.658+0000 7f90277fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bed4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33174/0 (socket says 172.21.15.62:33174) 2024-09-15T23:23:56.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.658+0000 7f90277fe640 1 -- 172.21.15.62:0/1276689672 learned_addr learned my addr 172.21.15.62:0/1276689672 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:56.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.658+0000 7f9026ffd640 1 -- 172.21.15.62:0/1276689672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bec7480 msgr2=0x55bf2bed1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:56.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.658+0000 7f9026ffd640 1 --2- 172.21.15.62:0/1276689672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bec7480 0x55bf2bed1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.658+0000 7f9026ffd640 1 -- 172.21.15.62:0/1276689672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 msgr2=0x55bf2bed4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:56.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.658+0000 7f9026ffd640 1 --2- 172.21.15.62:0/1276689672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bed4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.659+0000 7f9026ffd640 1 -- 172.21.15.62:0/1276689672 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf2bde01f0 con 0x55bf2bec6b60 2024-09-15T23:23:56.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.659+0000 7f9026ffd640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f90140a3410 0x7f90140a58d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9020005fd0 tx=0x7f902000bf20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:56.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.659+0000 7f9026ffd640 1 --2- 172.21.15.62:0/1276689672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2bec6f40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9020003a80 tx=0x7f902001e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e85c81768109050 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:56.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.660+0000 7f901d7fa640 1 -- 172.21.15.62:0/1276689672 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902001f650 con 0x55bf2bec6b60 2024-09-15T23:23:56.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.660+0000 7f901d7fa640 1 -- 172.21.15.62:0/1276689672 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9020024070 con 0x55bf2bec6b60 2024-09-15T23:23:56.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.661+0000 7f901d7fa640 1 -- 172.21.15.62:0/1276689672 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902001eb00 con 0x55bf2bec6b60 2024-09-15T23:23:56.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.663+0000 7f902fad1740 1 -- 172.21.15.62:0/1276689672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 msgr2=0x55bf2bec6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:56.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.663+0000 7f902fad1740 1 --2- 172.21.15.62:0/1276689672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2bec6f40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9020003a80 tx=0x7f902001e000 comp rx=0 tx=0).stop 2024-09-15T23:23:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.665+0000 7f902fad1740 1 -- 172.21.15.62:0/1276689672 shutdown_connections 2024-09-15T23:23:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.665+0000 7f902fad1740 1 --2- 172.21.15.62:0/1276689672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bed4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.665+0000 7f902fad1740 1 --2- 172.21.15.62:0/1276689672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bec7480 0x55bf2bed1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.665+0000 7f902fad1740 1 --2- 172.21.15.62:0/1276689672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2bec6f40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.665+0000 7f902fad1740 1 -- 172.21.15.62:0/1276689672 >> 172.21.15.62:0/1276689672 conn(0x55bf2bec30f0 msgr2=0x55bf2bec34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:56.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.668+0000 7f902fad1740 1 -- 172.21.15.62:0/1276689672 shutdown_connections 2024-09-15T23:23:56.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.669+0000 7f902fad1740 1 -- 172.21.15.62:0/1276689672 wait complete. 2024-09-15T23:23:56.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.669+0000 7f902fad1740 1 -- start start 2024-09-15T23:23:56.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.669+0000 7f902fad1740 1 Processor -- start 2024-09-15T23:23:56.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.670+0000 7f902fad1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2becebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.670+0000 7f902fad1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec7480 0x55bf2becf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.670+0000 7f902fad1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2becf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f902fad1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bf2bed4910 con 0x55bf2bec7480 2024-09-15T23:23:56.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f902fad1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55bf2bde6f10 con 0x55bf2bed1d40 2024-09-15T23:23:56.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f902fad1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bf2bed4a90 con 0x55bf2bec6b60 2024-09-15T23:23:56.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f90277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2becf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f90277fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2becf650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36490/0 (socket says 172.21.15.62:36490) 2024-09-15T23:23:56.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f90277fe640 1 -- 172.21.15.62:0/427405018 learned_addr learned my addr 172.21.15.62:0/427405018 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:56.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f90277fe640 1 -- 172.21.15.62:0/427405018 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 msgr2=0x55bf2becebd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:56.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f90277fe640 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2becebd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f90277fe640 1 -- 172.21.15.62:0/427405018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec7480 msgr2=0x55bf2becf110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:23:56.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f90277fe640 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec7480 0x55bf2becf110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.671+0000 7f90277fe640 1 -- 172.21.15.62:0/427405018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf2bb9d6a0 con 0x55bf2bed1d40 2024-09-15T23:23:56.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.672+0000 7f902fad1740 10 client.0 _ll_register_callbacks cb 0x55bf2bdafa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:56.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.672+0000 7f902fad1740 1 -- 172.21.15.62:0/427405018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9020024070 con 0x55bf2bed1d40 2024-09-15T23:23:56.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.672+0000 7f90277fe640 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2becf650 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f90180099a0 tx=0x7f9018030c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa841e0ce59412db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:56.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.673+0000 7f900e7fc640 10 client.26849 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.673+0000 7f900e7fc640 1 -- 172.21.15.62:0/427405018 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9018036040 con 0x55bf2bed1d40 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.673+0000 7f900e7fc640 1 -- 172.21.15.62:0/427405018 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f90180303a0 con 0x55bf2bed1d40 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.673+0000 7f900e7fc640 1 -- 172.21.15.62:0/427405018 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f90180305a0 con 0x55bf2bed1d40 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.674+0000 7f902fad1740 10 client.26849 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.674+0000 7f902fad1740 1 -- 172.21.15.62:0/427405018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=33}) -- 0x55bf2bee03c0 con 0x55bf2bed1d40 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.674+0000 7f900e7fc640 1 -- 172.21.15.62:0/427405018 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f9018003970 con 0x55bf2bed1d40 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.674+0000 7f902fad1740 10 client.26849 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.674+0000 7f902fad1740 10 client.26849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.674+0000 7f902fad1740 20 client.26849 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.674+0000 7f902fad1740 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bf2bee1dc0 0x55bf2bee4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.674+0000 7f902fad1740 4 client.26849 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.674+0000 7f902fad1740 1 -- 172.21.15.62:0/427405018 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf2be397f0 con 0x55bf2bee1dc0 2024-09-15T23:23:56.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.675+0000 7f9026ffd640 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bf2bee1dc0 0x55bf2bee4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.675+0000 7f9026ffd640 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bf2bee1dc0 0x55bf2bee4280 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee81aae64bf320ea server_cookie=33f1f0c152235504 in_seq=0 out_seq=0 2024-09-15T23:23:56.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.675+0000 7f900e7fc640 10 client.26849 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:56.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.676+0000 7f900e7fc640 1 -- 172.21.15.62:0/427405018 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bf2be397f0 con 0x55bf2bee1dc0 2024-09-15T23:23:56.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.676+0000 7f900e7fc640 10 client.26849 handle_command_reply: tid=0 2024-09-15T23:23:56.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.676+0000 7f902fad1740 1 client.26849 shutdown 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 -- 172.21.15.62:0/427405018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 msgr2=0x55bf2becf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2becf650 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f90180099a0 tx=0x7f9018030c60 comp rx=0 tx=0).stop 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 -- 172.21.15.62:0/427405018 shutdown_connections 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bf2bee1dc0 0x55bf2bee4280 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2becf650 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec7480 0x55bf2becf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 --2- 172.21.15.62:0/427405018 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2becebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 -- 172.21.15.62:0/427405018 >> 172.21.15.62:0/427405018 conn(0x55bf2bec30f0 msgr2=0x55bf2bec8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f900e7fc640 0 client.26849 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:56.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 -- 172.21.15.62:0/427405018 shutdown_connections 2024-09-15T23:23:56.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.678+0000 7f902fad1740 1 -- 172.21.15.62:0/427405018 wait complete. 2024-09-15T23:23:56.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.679+0000 7f902fad1740 20 client.26849 trim_cache size 0 max 16384 2024-09-15T23:23:56.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.710+0000 7f902fad1740 1 Processor -- start 2024-09-15T23:23:56.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.711+0000 7f902fad1740 1 -- start start 2024-09-15T23:23:56.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.711+0000 7f902fad1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2becf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.711+0000 7f902fad1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bec7480 0x55bf2bed0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.711+0000 7f902fad1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bee9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f902fad1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f90080123d0 con 0x55bf2bed1d40 2024-09-15T23:23:56.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f90277fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bee9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f9026ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2becf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f90277fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bee9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33182/0 (socket says 172.21.15.62:33182) 2024-09-15T23:23:56.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f90277fe640 1 -- 172.21.15.62:0/3340705809 learned_addr learned my addr 172.21.15.62:0/3340705809 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:56.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f902fad1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f900800fca0 con 0x55bf2bec7480 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f902fad1740 1 -- 172.21.15.62:0/3340705809 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f900800daf0 con 0x55bf2bec6b60 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f90277fe640 1 -- 172.21.15.62:0/3340705809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 msgr2=0x55bf2becf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f90277fe640 1 --2- 172.21.15.62:0/3340705809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2becf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f90277fe640 1 -- 172.21.15.62:0/3340705809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bec7480 msgr2=0x55bf2bed0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f90277fe640 1 --2- 172.21.15.62:0/3340705809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bec7480 0x55bf2bed0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f90277fe640 1 -- 172.21.15.62:0/3340705809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf2bec72b0 con 0x55bf2bed1d40 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.712+0000 7f90277fe640 1 --2- 172.21.15.62:0/3340705809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bee9b90 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f90180314d0 tx=0x7f9018040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31b4fb89fb42fec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.713+0000 7f901cff9640 1 -- 172.21.15.62:0/3340705809 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9018036040 con 0x55bf2bed1d40 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.713+0000 7f901cff9640 1 -- 172.21.15.62:0/3340705809 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9020024070 con 0x55bf2bed1d40 2024-09-15T23:23:56.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.713+0000 7f901cff9640 1 -- 172.21.15.62:0/3340705809 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9018003f70 con 0x55bf2bed1d40 2024-09-15T23:23:56.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.714+0000 7f902fad1740 1 -- 172.21.15.62:0/3340705809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 msgr2=0x55bf2bee9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:56.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.714+0000 7f902fad1740 1 --2- 172.21.15.62:0/3340705809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bee9b90 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f90180314d0 tx=0x7f9018040000 comp rx=0 tx=0).stop 2024-09-15T23:23:56.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.714+0000 7f902fad1740 1 -- 172.21.15.62:0/3340705809 shutdown_connections 2024-09-15T23:23:56.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.714+0000 7f902fad1740 1 --2- 172.21.15.62:0/3340705809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bed1d40 0x55bf2bee9b90 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.714+0000 7f902fad1740 1 --2- 172.21.15.62:0/3340705809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bec7480 0x55bf2bed0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.714+0000 7f902fad1740 1 --2- 172.21.15.62:0/3340705809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec6b60 0x55bf2becf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.714+0000 7f902fad1740 1 -- 172.21.15.62:0/3340705809 >> 172.21.15.62:0/3340705809 conn(0x55bf2bec30f0 msgr2=0x55bf2bed0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:56.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.714+0000 7f902fad1740 1 -- 172.21.15.62:0/3340705809 shutdown_connections 2024-09-15T23:23:56.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.714+0000 7f902fad1740 1 -- 172.21.15.62:0/3340705809 wait complete. 2024-09-15T23:23:56.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.715+0000 7f902fad1740 1 -- start start 2024-09-15T23:23:56.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.715+0000 7f902fad1740 1 Processor -- start 2024-09-15T23:23:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f902fad1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec6b60 0x55bf2beda4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f902fad1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec7480 0x55bf2bed57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f902fad1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2bed5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f902fad1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f900800e160 con 0x55bf2bec6b60 2024-09-15T23:23:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f902fad1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f900800f5d0 con 0x55bf2bed1d40 2024-09-15T23:23:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f902fad1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f900800e830 con 0x55bf2bec7480 2024-09-15T23:23:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f902fad1740 10 client.0 _ll_register_callbacks cb 0x7f9018009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:23:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f9026ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec6b60 0x55bf2beda4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f9026ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec6b60 0x55bf2beda4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33186/0 (socket says 172.21.15.62:33186) 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f9026ffd640 1 -- 172.21.15.62:0/2444577667 learned_addr learned my addr 172.21.15.62:0/2444577667 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f90277fe640 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2bed5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.716+0000 7f90267fc640 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec7480 0x55bf2bed57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f90277fe640 1 -- 172.21.15.62:0/2444577667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec7480 msgr2=0x55bf2bed57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f90277fe640 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec7480 0x55bf2bed57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f90277fe640 1 -- 172.21.15.62:0/2444577667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec6b60 msgr2=0x55bf2beda4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f90277fe640 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec6b60 0x55bf2beda4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f90277fe640 1 -- 172.21.15.62:0/2444577667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf2bed6510 con 0x55bf2bed1d40 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f9026ffd640 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec6b60 0x55bf2beda4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f90277fe640 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2bed5d30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9018003ea0 tx=0x7f9018030f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=476c3b5487a6b3c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f900e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f900e7fc640 1 -- 172.21.15.62:0/2444577667 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9018036040 con 0x55bf2bed1d40 2024-09-15T23:23:56.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f902fad1740 1 -- 172.21.15.62:0/2444577667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9020024070 con 0x55bf2bed1d40 2024-09-15T23:23:56.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.717+0000 7f900e7fc640 1 -- 172.21.15.62:0/2444577667 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f90180303a0 con 0x55bf2bed1d40 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.719+0000 7f900e7fc640 1 -- 172.21.15.62:0/2444577667 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9018031ca0 con 0x55bf2bed1d40 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.719+0000 7f902fad1740 10 client.26852 fetch_fsmap learned FSMap version 33 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.719+0000 7f902fad1740 1 -- 172.21.15.62:0/2444577667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55bf2bf4cbb0 con 0x55bf2bed1d40 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.720+0000 7f900e7fc640 1 -- 172.21.15.62:0/2444577667 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9020024070 con 0x55bf2bed1d40 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.720+0000 7f900e7fc640 1 -- 172.21.15.62:0/2444577667 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f9018047470 con 0x55bf2bed1d40 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.720+0000 7f902fad1740 10 client.26852 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.720+0000 7f902fad1740 10 client.26852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.720+0000 7f902fad1740 20 client.26852 populate_metadata read hostname 'smithi062' 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.720+0000 7f902fad1740 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bf2bf4d260 0x55bf2bf4f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.720+0000 7f902fad1740 4 client.26852 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:23:56.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.720+0000 7f902fad1740 1 -- 172.21.15.62:0/2444577667 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bf2be33d00 con 0x55bf2bf4d260 2024-09-15T23:23:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.720+0000 7f9026ffd640 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bf2bf4d260 0x55bf2bf4f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:23:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.721+0000 7f9026ffd640 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bf2bf4d260 0x55bf2bf4f680 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42afc797b9f3834b server_cookie=c8214175de1f6a3c in_seq=0 out_seq=0 2024-09-15T23:23:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.721+0000 7f900e7fc640 10 client.26852 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f900e7fc640 1 -- 172.21.15.62:0/2444577667 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bf2be33d00 con 0x55bf2bf4d260 2024-09-15T23:23:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f900e7fc640 10 client.26852 handle_command_reply: tid=0 2024-09-15T23:23:56.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f902fad1740 1 client.26852 shutdown 2024-09-15T23:23:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f902fad1740 1 -- 172.21.15.62:0/2444577667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 msgr2=0x55bf2bed5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f902fad1740 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2bed5d30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9018003ea0 tx=0x7f9018030f90 comp rx=0 tx=0).stop 2024-09-15T23:23:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f902fad1740 1 -- 172.21.15.62:0/2444577667 shutdown_connections 2024-09-15T23:23:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f902fad1740 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bf2bf4d260 0x55bf2bf4f680 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f902fad1740 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bf2bed1d40 0x55bf2bed5d30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f902fad1740 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bf2bec7480 0x55bf2bed57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f902fad1740 1 --2- 172.21.15.62:0/2444577667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bf2bec6b60 0x55bf2beda4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f900e7fc640 0 client.26852 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.722+0000 7f902fad1740 1 -- 172.21.15.62:0/2444577667 >> 172.21.15.62:0/2444577667 conn(0x55bf2bec30f0 msgr2=0x55bf2bec8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.723+0000 7f902fad1740 1 -- 172.21.15.62:0/2444577667 shutdown_connections 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.723+0000 7f902fad1740 1 -- 172.21.15.62:0/2444577667 wait complete. 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.723+0000 7f902fad1740 20 client.26852 trim_cache size 0 max 16384 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "aef06d6b-db0d-4829-802a-06a263ffb4d7", 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:23:56.725 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:23:56.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.723+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f90140a3410 msgr2=0x7f90140a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:56.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.723+0000 7f902d6f8640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f90140a3410 0x7f90140a58d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9020005fd0 tx=0x7f902000bf20 comp rx=0 tx=0).stop 2024-09-15T23:23:56.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.723+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9028109830 msgr2=0x7f90281a7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:23:56.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.723+0000 7f902d6f8640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9028109830 0x7f90281a7720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f901000d720 tx=0x7f901000dbe0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.724+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 shutdown_connections 2024-09-15T23:23:56.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.724+0000 7f902d6f8640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f90140a3410 0x7f90140a58d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.724+0000 7f902d6f8640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f902811b140 0x7f90281abab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.724+0000 7f902d6f8640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9028109830 0x7f90281a7720 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.724+0000 7f902d6f8640 1 --2- 172.21.15.62:0/350991063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9028109450 0x7f90281a71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:23:56.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.724+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 >> 172.21.15.62:0/350991063 conn(0x7f902806e140 msgr2=0x7f902811bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:23:56.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.724+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 shutdown_connections 2024-09-15T23:23:56.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:23:56.724+0000 7f902d6f8640 1 -- 172.21.15.62:0/350991063 wait complete. 2024-09-15T23:23:57.024 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:23:56 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: 2024-09-15T23:23:56.767+0000 7ff293927640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000f [...2,head] /volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans/ auth{2=1} v3099 ap=2 f(v7 m2024-09-15T23:22:40.590845+0000 1=1+0) n(v20 rc2024-09-15T23:22:40.590845+0000 b212992 2=1+1) (inest sync->mix g=2) mcw={2=AsLsXsFs} | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55598f2f4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v282>,<0x10000000005/data v121>,<0x100000001fb/postgres v25>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v43>,<0x10000000001/sv_1 v57>,<0x10000000000/qa v44>,<0x1/volumes v43>]//[]","memoryvalue":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v3099>,<0x10000000005/data v609>,<0x100000001fb/postgres v398>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v432>,<0x10000000001/sv_1 v424>,<0x10000000000/qa v425>,<0x1/volumes v393>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-09-15T23:22:40.543846+0000 8=8+0)","ondisk_value.rstat":"n(v0 rc2024-09-15T23:22:40.590845+0000 b212992 2=1+1)","memory_value.dirstat":"f(v7 m2024-09-15T23:22:40.590845+0000 1=1+0)","memory_value.rstat":"n(v20 rc2024-09-15T23:22:40.590845+0000 b212992 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-09-15T23:23:57.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:57 smithi062 ceph-mon[30949]: pgmap v600: 129 pgs: 129 active+clean; 1.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 540 op/s 2024-09-15T23:23:57.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:57 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:23:57.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:57 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:57 smithi067 ceph-mon[35665]: pgmap v600: 129 pgs: 129 active+clean; 1.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 540 op/s 2024-09-15T23:23:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:57 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:23:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:57 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:57.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:57 smithi148 ceph-mon[35610]: pgmap v600: 129 pgs: 129 active+clean; 1.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 540 op/s 2024-09-15T23:23:57.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:57 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:23:57.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:57 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:58 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:23:58.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:58 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:58 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:23:58.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:58 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:23:58.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:58 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:58 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:23:58.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:58 smithi062 ceph-mon[30949]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:23:58.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:58 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:58 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:58 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:23:58.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:58 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:58 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:23:58.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:58 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:23:58.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:58 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:58 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:23:58.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:58 smithi067 ceph-mon[35665]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:23:58.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:58 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:58 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:58 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:23:58.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:58 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:58 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:23:58.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:58 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:23:58.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:58 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:58 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:23:58.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:58 smithi148 ceph-mon[35610]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:23:58.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:58 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:58.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:58 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:23:59.244 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:23:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 460.0 s, 821.2 tps, lat 120.888 ms stddev 143.311 2024-09-15T23:23:59.712 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:23:59.901 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:23:59 smithi062 ceph-mon[30949]: pgmap v601: 129 pgs: 129 active+clean; 1.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.2 MiB/s wr, 611 op/s 2024-09-15T23:24:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:23:59 smithi067 ceph-mon[35665]: pgmap v601: 129 pgs: 129 active+clean; 1.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.2 MiB/s wr, 611 op/s 2024-09-15T23:24:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:23:59 smithi148 ceph-mon[35610]: pgmap v601: 129 pgs: 129 active+clean; 1.5 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 6.2 MiB/s wr, 611 op/s 2024-09-15T23:24:00.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.337+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1074679248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 msgr2=0x7fe4dc109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.337+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1074679248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 0x7fe4dc109830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8008570 tx=0x7fe4d802e390 comp rx=0 tx=0).stop 2024-09-15T23:24:00.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.338+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1074679248 shutdown_connections 2024-09-15T23:24:00.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.338+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1074679248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4dc074990 0x7fe4dc07b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.338+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1074679248 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4dc109e00 0x7fe4dc0742d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.338+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1074679248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 0x7fe4dc109830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.338+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1074679248 >> 172.21.15.62:0/1074679248 conn(0x7fe4dc06e140 msgr2=0x7fe4dc06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:00.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.348+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1074679248 shutdown_connections 2024-09-15T23:24:00.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.350+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1074679248 wait complete. 2024-09-15T23:24:00.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.351+0000 7fe4e3c77640 1 Processor -- start 2024-09-15T23:24:00.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.351+0000 7fe4e3c77640 1 -- start start 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.356+0000 7fe4e3c77640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4dc074990 0x7fe4dc07b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.356+0000 7fe4e3c77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 0x7fe4dc076190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.356+0000 7fe4e3c77640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4dc109e00 0x7fe4dc0766d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.356+0000 7fe4e3c77640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc07e790 con 0x7fe4dc109e00 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.356+0000 7fe4e3c77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc07e610 con 0x7fe4dc109450 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.356+0000 7fe4e3c77640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc07e910 con 0x7fe4dc074990 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.357+0000 7fe4e1212640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 0x7fe4dc076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.358+0000 7fe4e1212640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 0x7fe4dc076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36528/0 (socket says 172.21.15.62:36528) 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.358+0000 7fe4e1212640 1 -- 172.21.15.62:0/1328190357 learned_addr learned my addr 172.21.15.62:0/1328190357 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:00.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.359+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4dc074990 0x7fe4dc07b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.359+0000 7fe4e1212640 1 -- 172.21.15.62:0/1328190357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4dc074990 msgr2=0x7fe4dc07b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.359+0000 7fe4e1212640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4dc074990 0x7fe4dc07b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.359+0000 7fe4e1212640 1 -- 172.21.15.62:0/1328190357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4dc109e00 msgr2=0x7fe4dc0766d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.359+0000 7fe4e1212640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4dc109e00 0x7fe4dc0766d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.359+0000 7fe4e1212640 1 -- 172.21.15.62:0/1328190357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4dc076f00 con 0x7fe4dc109450 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.360+0000 7fe4e1212640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 0x7fe4dc076190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc005d00 tx=0x7fe4cc00ad60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.360+0000 7fe4d2ffd640 1 -- 172.21.15.62:0/1328190357 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc019070 con 0x7fe4dc109450 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.360+0000 7fe4d2ffd640 1 -- 172.21.15.62:0/1328190357 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc014410 con 0x7fe4dc109450 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.360+0000 7fe4d2ffd640 1 -- 172.21.15.62:0/1328190357 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc013610 con 0x7fe4dc109450 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.361+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1328190357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4dc137290 con 0x7fe4dc109450 2024-09-15T23:24:00.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.361+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1328190357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4dc137780 con 0x7fe4dc109450 2024-09-15T23:24:00.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.365+0000 7fe4d2ffd640 1 -- 172.21.15.62:0/1328190357 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe4cc01c070 con 0x7fe4dc109450 2024-09-15T23:24:00.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.367+0000 7fe4e6050740 1 Processor -- start 2024-09-15T23:24:00.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.368+0000 7fe4d2ffd640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4c80a3450 0x7fe4c80a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.368+0000 7fe4d2ffd640 1 -- 172.21.15.62:0/1328190357 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe4cc0cff60 con 0x7fe4dc109450 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.370+0000 7fe4e6050740 1 -- start start 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.370+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 0x565429f4b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.370+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f4bdc0 0x565429f56140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.370+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f56680 0x565429f58a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.370+0000 7fe4e6050740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565429eb12e0 con 0x565429f4bdc0 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.370+0000 7fe4e6050740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565429e58ce0 con 0x565429f56680 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.370+0000 7fe4e6050740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x565429ead2a0 con 0x565429f4b4a0 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.372+0000 7fe4e1212640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f4bdc0 0x565429f56140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.372+0000 7fe4e1212640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f4bdc0 0x565429f56140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33198/0 (socket says 172.21.15.62:33198) 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.372+0000 7fe4e1212640 1 -- 172.21.15.62:0/3565343069 learned_addr learned my addr 172.21.15.62:0/3565343069 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:00.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.374+0000 7fe4e2214640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4c80a3450 0x7fe4c80a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.374+0000 7fe4e1212640 1 -- 172.21.15.62:0/3565343069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 msgr2=0x565429f4b880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:24:00.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.374+0000 7fe4e1212640 1 --2- 172.21.15.62:0/3565343069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 0x565429f4b880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.374+0000 7fe4e1212640 1 -- 172.21.15.62:0/3565343069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f56680 msgr2=0x565429f58a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:24:00.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.374+0000 7fe4e1212640 1 --2- 172.21.15.62:0/3565343069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f56680 0x565429f58a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.374+0000 7fe4e1212640 1 -- 172.21.15.62:0/3565343069 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565429e3d440 con 0x565429f4bdc0 2024-09-15T23:24:00.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.377+0000 7fe4e1212640 1 --2- 172.21.15.62:0/3565343069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f4bdc0 0x565429f56140 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc007590 tx=0x7fe4cc015fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b70dbc06f18652 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:00.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.377+0000 7fe4c7fff640 1 -- 172.21.15.62:0/3565343069 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc0a4e10 con 0x565429f4bdc0 2024-09-15T23:24:00.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.377+0000 7fe4c7fff640 1 -- 172.21.15.62:0/3565343069 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc09a4a0 con 0x565429f4bdc0 2024-09-15T23:24:00.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.377+0000 7fe4c7fff640 1 -- 172.21.15.62:0/3565343069 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc0a4980 con 0x565429f4bdc0 2024-09-15T23:24:00.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.378+0000 7fe4e2214640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4c80a3450 0x7fe4c80a5910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x565429f48db0 tx=0x7fe4d4005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:00.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.379+0000 7fe4e6050740 1 -- 172.21.15.62:0/3565343069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f4bdc0 msgr2=0x565429f56140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.379+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3565343069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f4bdc0 0x565429f56140 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc007590 tx=0x7fe4cc015fc0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.380+0000 7fe4e6050740 1 -- 172.21.15.62:0/3565343069 shutdown_connections 2024-09-15T23:24:00.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.380+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3565343069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f56680 0x565429f58a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.380+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3565343069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f4bdc0 0x565429f56140 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.380+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3565343069 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 0x565429f4b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.380+0000 7fe4e6050740 1 -- 172.21.15.62:0/3565343069 >> 172.21.15.62:0/3565343069 conn(0x565429f47a30 msgr2=0x565429f47e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.384+0000 7fe4e6050740 1 -- 172.21.15.62:0/3565343069 shutdown_connections 2024-09-15T23:24:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.385+0000 7fe4e6050740 1 -- 172.21.15.62:0/3565343069 wait complete. 2024-09-15T23:24:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.386+0000 7fe4e6050740 1 -- start start 2024-09-15T23:24:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.386+0000 7fe4e6050740 1 Processor -- start 2024-09-15T23:24:00.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f56010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f532c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f53840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e6050740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565429f59250 con 0x565429f56680 2024-09-15T23:24:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e6050740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565429e6c050 con 0x565429f4b4a0 2024-09-15T23:24:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e6050740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x565429f593d0 con 0x565429f4bdc0 2024-09-15T23:24:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e6050740 10 client.0 _ll_register_callbacks cb 0x565429eac160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e2214640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f53840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e2214640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f53840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33204/0 (socket says 172.21.15.62:33204) 2024-09-15T23:24:00.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.387+0000 7fe4e2214640 1 -- 172.21.15.62:0/2358659317 learned_addr learned my addr 172.21.15.62:0/2358659317 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:00.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.389+0000 7fe4e1212640 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.389+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f56010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.389+0000 7fe4e2214640 1 -- 172.21.15.62:0/2358659317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 msgr2=0x565429f532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:00.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.389+0000 7fe4e2214640 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.389+0000 7fe4e2214640 1 -- 172.21.15.62:0/2358659317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 msgr2=0x565429f56010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:00.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.389+0000 7fe4e2214640 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f56010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.389+0000 7fe4e2214640 1 -- 172.21.15.62:0/2358659317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565429c223c0 con 0x565429f56680 2024-09-15T23:24:00.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.389+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f56010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:00.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.395+0000 7fe4e2214640 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f53840 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4011c60 tx=0x7fe4d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597ab43427b0570f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:00.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.395+0000 7fe4c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:00.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.397+0000 7fe4c57fa640 1 -- 172.21.15.62:0/2358659317 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4022020 con 0x565429f56680 2024-09-15T23:24:00.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.397+0000 7fe4e6050740 1 -- 172.21.15.62:0/2358659317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4cc09a4a0 con 0x565429f56680 2024-09-15T23:24:00.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.397+0000 7fe4c57fa640 1 -- 172.21.15.62:0/2358659317 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe4d4009d00 con 0x565429f56680 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.401+0000 7fe4c57fa640 1 -- 172.21.15.62:0/2358659317 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe4d402d050 con 0x565429f56680 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.401+0000 7fe4e6050740 10 client.17277 fetch_fsmap learned FSMap version 33 2024-09-15T23:24:00.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.401+0000 7fe4e6050740 1 -- 172.21.15.62:0/2358659317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x565429f64d80 con 0x565429f56680 2024-09-15T23:24:00.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.402+0000 7fe4c57fa640 1 -- 172.21.15.62:0/2358659317 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc09a4a0 con 0x565429f56680 2024-09-15T23:24:00.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.402+0000 7fe4c57fa640 1 -- 172.21.15.62:0/2358659317 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fe4d400f040 con 0x565429f56680 2024-09-15T23:24:00.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.402+0000 7fe4e6050740 10 client.17277 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:24:00.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.402+0000 7fe4e6050740 10 client.17277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:00.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.402+0000 7fe4e6050740 20 client.17277 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:00.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.402+0000 7fe4e6050740 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565429f666b0 0x565429f68b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.402+0000 7fe4e6050740 4 client.17277 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:00.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.402+0000 7fe4e6050740 1 -- 172.21.15.62:0/2358659317 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565429ebe850 con 0x565429f666b0 2024-09-15T23:24:00.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.402+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565429f666b0 0x565429f68b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.403+0000 7fe4c57fa640 10 client.17277 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:00.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.404+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565429f666b0 0x565429f68b10 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5691dd6e2a7aecd7 server_cookie=d1f86db851f888b3 in_seq=0 out_seq=0 2024-09-15T23:24:00.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.407+0000 7fe4c57fa640 1 -- 172.21.15.62:0/2358659317 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x565429ebe850 con 0x565429f666b0 2024-09-15T23:24:00.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.407+0000 7fe4c57fa640 10 client.17277 handle_command_reply: tid=0 2024-09-15T23:24:00.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.407+0000 7fe4e6050740 1 client.17277 shutdown 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 -- 172.21.15.62:0/2358659317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 msgr2=0x565429f53840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f53840 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4011c60 tx=0x7fe4d4004ab0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 -- 172.21.15.62:0/2358659317 shutdown_connections 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565429f666b0 0x565429f68b10 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f53840 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 --2- 172.21.15.62:0/2358659317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f56010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4c57fa640 0 client.17277 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 -- 172.21.15.62:0/2358659317 >> 172.21.15.62:0/2358659317 conn(0x565429f47a30 msgr2=0x565429f4d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 -- 172.21.15.62:0/2358659317 shutdown_connections 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 1 -- 172.21.15.62:0/2358659317 wait complete. 2024-09-15T23:24:00.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.409+0000 7fe4e6050740 20 client.17277 trim_cache size 0 max 16384 2024-09-15T23:24:00.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.438+0000 7fe4e6050740 1 Processor -- start 2024-09-15T23:24:00.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.438+0000 7fe4e6050740 1 -- start start 2024-09-15T23:24:00.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 0x565429f62540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4bdc0 0x565429f4e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f6e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e6050740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c0012430 con 0x565429f56680 2024-09-15T23:24:00.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e6050740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c000fd00 con 0x565429f4bdc0 2024-09-15T23:24:00.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e6050740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c000db50 con 0x565429f4b4a0 2024-09-15T23:24:00.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e1a13640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 0x565429f62540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e1a13640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 0x565429f62540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36544/0 (socket says 172.21.15.62:36544) 2024-09-15T23:24:00.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e1a13640 1 -- 172.21.15.62:0/1795343969 learned_addr learned my addr 172.21.15.62:0/1795343969 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:00.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e1212640 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4bdc0 0x565429f4e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.439+0000 7fe4e2214640 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f6e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4e1a13640 1 -- 172.21.15.62:0/1795343969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4bdc0 msgr2=0x565429f4e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:00.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4bdc0 0x565429f4e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4e1a13640 1 -- 172.21.15.62:0/1795343969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 msgr2=0x565429f6e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:00.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f6e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.443 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4e1a13640 1 -- 172.21.15.62:0/1795343969 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565429f4b2d0 con 0x565429f4b4a0 2024-09-15T23:24:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4e1212640 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4bdc0 0x565429f4e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 0x565429f62540 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8005f30 tx=0x7fe4d8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9d1e67bbfa131b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4c77fe640 1 -- 172.21.15.62:0/1795343969 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8004040 con 0x565429f4b4a0 2024-09-15T23:24:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4c77fe640 1 -- 172.21.15.62:0/1795343969 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d80041e0 con 0x565429f4b4a0 2024-09-15T23:24:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.440+0000 7fe4c77fe640 1 -- 172.21.15.62:0/1795343969 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8003bc0 con 0x565429f4b4a0 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.441+0000 7fe4e6050740 1 -- 172.21.15.62:0/1795343969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 msgr2=0x565429f62540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.441+0000 7fe4e6050740 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 0x565429f62540 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8005f30 tx=0x7fe4d8005850 comp rx=0 tx=0).stop 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.441+0000 7fe4e6050740 1 -- 172.21.15.62:0/1795343969 shutdown_connections 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.441+0000 7fe4e6050740 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f6e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.441+0000 7fe4e6050740 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4bdc0 0x565429f4e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.441+0000 7fe4e6050740 1 --2- 172.21.15.62:0/1795343969 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4b4a0 0x565429f62540 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.441+0000 7fe4e6050740 1 -- 172.21.15.62:0/1795343969 >> 172.21.15.62:0/1795343969 conn(0x565429f47a30 msgr2=0x565429f4e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.441+0000 7fe4e6050740 1 -- 172.21.15.62:0/1795343969 shutdown_connections 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.441+0000 7fe4e6050740 1 -- 172.21.15.62:0/1795343969 wait complete. 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.442+0000 7fe4e6050740 1 -- start start 2024-09-15T23:24:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.442+0000 7fe4e6050740 1 Processor -- start 2024-09-15T23:24:00.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.442+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f5ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.442+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f5a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.442+0000 7fe4e6050740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f5a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.442+0000 7fe4e6050740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c000e1c0 con 0x565429f56680 2024-09-15T23:24:00.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.442+0000 7fe4e6050740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c000f630 con 0x565429f4b4a0 2024-09-15T23:24:00.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.442+0000 7fe4e6050740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c000e890 con 0x565429f4bdc0 2024-09-15T23:24:00.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.442+0000 7fe4e6050740 10 client.0 _ll_register_callbacks cb 0x7fe4d40094e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:00.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e1212640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f5a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e2214640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f5a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e1212640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f5a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36546/0 (socket says 172.21.15.62:36546) 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e1212640 1 -- 172.21.15.62:0/3416864519 learned_addr learned my addr 172.21.15.62:0/3416864519 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f5ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e2214640 1 -- 172.21.15.62:0/3416864519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 msgr2=0x565429f5a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e2214640 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f5a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e2214640 1 -- 172.21.15.62:0/3416864519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 msgr2=0x565429f5ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e2214640 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f5ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e2214640 1 -- 172.21.15.62:0/3416864519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565429f5adb0 con 0x565429f56680 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e1212640 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f5a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:00.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f5ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:00.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4e2214640 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f5a580 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400ebc0 tx=0x7fe4d4009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db71cd22aaf2104 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:00.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:00.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.443+0000 7fe4c57fa640 1 -- 172.21.15.62:0/3416864519 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4022040 con 0x565429f56680 2024-09-15T23:24:00.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.444+0000 7fe4c57fa640 1 -- 172.21.15.62:0/3416864519 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe4d401d5e0 con 0x565429f56680 2024-09-15T23:24:00.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.444+0000 7fe4e6050740 1 -- 172.21.15.62:0/3416864519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d80041e0 con 0x565429f56680 2024-09-15T23:24:00.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.444+0000 7fe4c57fa640 1 -- 172.21.15.62:0/3416864519 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe4d4045050 con 0x565429f56680 2024-09-15T23:24:00.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.444+0000 7fe4e6050740 10 client.17283 fetch_fsmap learned FSMap version 33 2024-09-15T23:24:00.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.444+0000 7fe4e6050740 1 -- 172.21.15.62:0/3416864519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x565429fd12d0 con 0x565429f56680 2024-09-15T23:24:00.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.447+0000 7fe4c57fa640 1 -- 172.21.15.62:0/3416864519 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d4017030 con 0x565429f56680 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.447+0000 7fe4c57fa640 1 -- 172.21.15.62:0/3416864519 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7fe4d401e460 con 0x565429f56680 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.447+0000 7fe4e6050740 10 client.17283 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.447+0000 7fe4e6050740 10 client.17283 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.447+0000 7fe4e6050740 20 client.17283 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.447+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565429fd1970 0x565429fd3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.447+0000 7fe4e6050740 4 client.17283 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.447+0000 7fe4e6050740 1 -- 172.21.15.62:0/3416864519 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe4d80041e0 con 0x565429fd1970 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.447+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565429fd1970 0x565429fd3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4c57fa640 10 client.17283 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e1a13640 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565429fd1970 0x565429fd3d90 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=451e025a55e416c server_cookie=866e76a362a6098d in_seq=0 out_seq=0 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4c57fa640 1 -- 172.21.15.62:0/3416864519 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fe4d80041e0 con 0x565429fd1970 2024-09-15T23:24:00.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4c57fa640 10 client.17283 handle_command_reply: tid=0 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e6050740 1 client.17283 shutdown 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e6050740 1 -- 172.21.15.62:0/3416864519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 msgr2=0x565429f5a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f5a580 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400ebc0 tx=0x7fe4d4009fb0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e6050740 1 -- 172.21.15.62:0/3416864519 shutdown_connections 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x565429fd1970 0x565429fd3d90 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4c57fa640 0 client.17283 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565429f56680 0x565429f5a580 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565429f4bdc0 0x565429f5a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e6050740 1 --2- 172.21.15.62:0/3416864519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565429f4b4a0 0x565429f5ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.448+0000 7fe4e6050740 1 -- 172.21.15.62:0/3416864519 >> 172.21.15.62:0/3416864519 conn(0x565429f47a30 msgr2=0x565429f4d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e6050740 1 -- 172.21.15.62:0/3416864519 shutdown_connections 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e6050740 1 -- 172.21.15.62:0/3416864519 wait complete. 2024-09-15T23:24:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e6050740 20 client.17283 trim_cache size 0 max 16384 2024-09-15T23:24:00.455 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:24:00.455 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:24:00.455 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:24:00.455 INFO:teuthology.orchestra.run.smithi062.stdout: "aef06d6b-db0d-4829-802a-06a263ffb4d7": { 2024-09-15T23:24:00.455 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:24:00.455 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "aef06d6b-db0d-4829-802a-06a263ffb4d7", 2024-09-15T23:24:00.455 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:24:00.455 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:00.456 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:00.456 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:24:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1328190357 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4c80a3450 msgr2=0x7fe4c80a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4c80a3450 0x7fe4c80a5910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x565429f48db0 tx=0x7fe4d4005bd0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1328190357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 msgr2=0x7fe4dc076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 0x7fe4dc076190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc005d00 tx=0x7fe4cc00ad60 comp rx=0 tx=0).stop 2024-09-15T23:24:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1328190357 shutdown_connections 2024-09-15T23:24:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4c80a3450 0x7fe4c80a5910 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4dc109e00 0x7fe4dc0766d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4dc109450 0x7fe4dc076190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 --2- 172.21.15.62:0/1328190357 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4dc074990 0x7fe4dc07b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.449+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1328190357 >> 172.21.15.62:0/1328190357 conn(0x7fe4dc06e140 msgr2=0x7fe4dc0724c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.450+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1328190357 shutdown_connections 2024-09-15T23:24:00.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:00.450+0000 7fe4e3c77640 1 -- 172.21.15.62:0/1328190357 wait complete. 2024-09-15T23:24:01.270 INFO:tasks.cephfs.filesystem:scrub status for tag:aef06d6b-db0d-4829-802a-06a263ffb4d7 - {'path': '/', 'tag': 'aef06d6b-db0d-4829-802a-06a263ffb4d7', 'options': 'recursive,force'} 2024-09-15T23:24:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:01 smithi062 ceph-mon[30949]: pgmap v602: 129 pgs: 129 active+clean; 1.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 608 op/s 2024-09-15T23:24:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:01 smithi067 ceph-mon[35665]: pgmap v602: 129 pgs: 129 active+clean; 1.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 608 op/s 2024-09-15T23:24:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:01 smithi148 ceph-mon[35610]: pgmap v602: 129 pgs: 129 active+clean; 1.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 5.9 MiB/s wr, 608 op/s 2024-09-15T23:24:02.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.119+0000 7f433b8b7640 1 -- 172.21.15.62:0/3780141628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334102630 msgr2=0x7f4334104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.119+0000 7f433b8b7640 1 --2- 172.21.15.62:0/3780141628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334102630 0x7f4334104a20 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f4330009900 tx=0x7f433002a410 comp rx=0 tx=0).stop 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.119+0000 7f433b8b7640 1 -- 172.21.15.62:0/3780141628 shutdown_connections 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.119+0000 7f433b8b7640 1 --2- 172.21.15.62:0/3780141628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4334107890 0x7f4334109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.119+0000 7f433b8b7640 1 --2- 172.21.15.62:0/3780141628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4334104f60 0x7f4334107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.119+0000 7f433b8b7640 1 --2- 172.21.15.62:0/3780141628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334102630 0x7f4334104a20 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.119+0000 7f433b8b7640 1 -- 172.21.15.62:0/3780141628 >> 172.21.15.62:0/3780141628 conn(0x7f43340fc1b0 msgr2=0x7f43340fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.119+0000 7f433b8b7640 1 -- 172.21.15.62:0/3780141628 shutdown_connections 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.119+0000 7f433b8b7640 1 -- 172.21.15.62:0/3780141628 wait complete. 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f433b8b7640 1 Processor -- start 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f433b8b7640 1 -- start start 2024-09-15T23:24:02.134 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f433b8b7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4334102630 0x7f4334197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f433b8b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4334104f60 0x7f4334198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f433b8b7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334107890 0x7f433419f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f433b8b7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f433410d870 con 0x7f4334107890 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f433b8b7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f433410d6f0 con 0x7f4334104f60 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f433b8b7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f433410d9f0 con 0x7f4334102630 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4339e54640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334107890 0x7f433419f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4339e54640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334107890 0x7f433419f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33254/0 (socket says 172.21.15.62:33254) 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4339e54640 1 -- 172.21.15.62:0/1456235041 learned_addr learned my addr 172.21.15.62:0/1456235041 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4339653640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4334102630 0x7f4334197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4338e52640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4334104f60 0x7f4334198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4339e54640 1 -- 172.21.15.62:0/1456235041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4334102630 msgr2=0x7f4334197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4339e54640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4334102630 0x7f4334197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4339e54640 1 -- 172.21.15.62:0/1456235041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4334104f60 msgr2=0x7f4334198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4339e54640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4334104f60 0x7f4334198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.120+0000 7f4339e54640 1 -- 172.21.15.62:0/1456235041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f433419fa40 con 0x7f4334107890 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.121+0000 7f4339e54640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334107890 0x7f433419f3d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f432c00dac0 tx=0x7f432c00df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.121+0000 7f432a7fc640 1 -- 172.21.15.62:0/1456235041 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c017070 con 0x7f4334107890 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.121+0000 7f432a7fc640 1 -- 172.21.15.62:0/1456235041 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f432c003e50 con 0x7f4334107890 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.121+0000 7f432a7fc640 1 -- 172.21.15.62:0/1456235041 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f432c0106c0 con 0x7f4334107890 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.121+0000 7f433b8b7640 1 -- 172.21.15.62:0/1456235041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f433419fd30 con 0x7f4334107890 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.121+0000 7f433b8b7640 1 -- 172.21.15.62:0/1456235041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43341a0220 con 0x7f4334107890 2024-09-15T23:24:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.124+0000 7f432a7fc640 1 -- 172.21.15.62:0/1456235041 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f432c003970 con 0x7f4334107890 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.124+0000 7f433e652740 1 Processor -- start 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.124+0000 7f433e652740 1 -- start start 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.124+0000 7f433e652740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 0x56512ea031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.124+0000 7f433e652740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.124+0000 7f433e652740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 0x56512ea0ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.125+0000 7f433e652740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56512e9622f0 con 0x56512ea02db0 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.125+0000 7f433e652740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56512e90a270 con 0x56512ea08780 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.125+0000 7f433e652740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56512e95e2b0 con 0x56512ea036f0 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.125+0000 7f4338e52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 0x56512ea031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.125+0000 7f4338e52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 0x56512ea031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33264/0 (socket says 172.21.15.62:33264) 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.125+0000 7f4338e52640 1 -- 172.21.15.62:0/424237699 learned_addr learned my addr 172.21.15.62:0/424237699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.125+0000 7f4338e52640 1 --2- 172.21.15.62:0/424237699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 0x56512ea0ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.125+0000 7f4339653640 1 --2- 172.21.15.62:0/424237699 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f432a7fc640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43200a3370 0x7f43200a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f4339653640 1 -- 172.21.15.62:0/424237699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 msgr2=0x56512ea0ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f4339653640 1 --2- 172.21.15.62:0/424237699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 0x56512ea0ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f4339653640 1 -- 172.21.15.62:0/424237699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 msgr2=0x56512ea031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f4339653640 1 --2- 172.21.15.62:0/424237699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 0x56512ea031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f4339653640 1 -- 172.21.15.62:0/424237699 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56512e909340 con 0x56512ea036f0 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f432a7fc640 1 -- 172.21.15.62:0/1456235041 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f432c0cf820 con 0x7f4334107890 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f4339653640 1 --2- 172.21.15.62:0/424237699 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea081b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f433002a930 tx=0x7f4330004190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7c05969eb5dcfe7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f431f7fe640 1 -- 172.21.15.62:0/424237699 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330035040 con 0x56512ea036f0 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.126+0000 7f431f7fe640 1 -- 172.21.15.62:0/424237699 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4330005520 con 0x56512ea036f0 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f431f7fe640 1 -- 172.21.15.62:0/424237699 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330031910 con 0x56512ea036f0 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f4339e54640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43200a3370 0x7f43200a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f433e652740 1 -- 172.21.15.62:0/424237699 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 msgr2=0x56512ea081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f433e652740 1 --2- 172.21.15.62:0/424237699 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea081b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f433002a930 tx=0x7f4330004190 comp rx=0 tx=0).stop 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f4339e54640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43200a3370 0x7f43200a5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f432c0a4200 tx=0x7f432c009520 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f433e652740 1 -- 172.21.15.62:0/424237699 shutdown_connections 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f433e652740 1 --2- 172.21.15.62:0/424237699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 0x56512ea0ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f433e652740 1 --2- 172.21.15.62:0/424237699 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea081b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f433e652740 1 --2- 172.21.15.62:0/424237699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 0x56512ea031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f433e652740 1 -- 172.21.15.62:0/424237699 >> 172.21.15.62:0/424237699 conn(0x56512e9fa010 msgr2=0x56512e9fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f433e652740 1 -- 172.21.15.62:0/424237699 shutdown_connections 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.127+0000 7f433e652740 1 -- 172.21.15.62:0/424237699 wait complete. 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f433e652740 1 -- start start 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f433e652740 1 Processor -- start 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f433e652740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 0x56512ea0d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f433e652740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 0x56512ea0d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f433e652740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512ea145a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f433e652740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56512ea0b790 con 0x56512ea08780 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f433e652740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56512e8eb290 con 0x56512ea036f0 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f433e652740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56512ea0b910 con 0x56512ea02db0 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f433e652740 10 client.0 _ll_register_callbacks cb 0x56512e91c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f4339653640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 0x56512ea0d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f4338e52640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 0x56512ea0d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f4339653640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 0x56512ea0d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36592/0 (socket says 172.21.15.62:36592) 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f4338e52640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 0x56512ea0d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36574/0 (socket says 172.21.15.62:36574) 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.128+0000 7f4339653640 1 -- 172.21.15.62:0/1993972893 learned_addr learned my addr 172.21.15.62:0/1993972893 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f4338e52640 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512ea145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f4338e52640 1 -- 172.21.15.62:0/1993972893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 msgr2=0x56512ea0d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f4338e52640 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 0x56512ea0d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f4338e52640 1 -- 172.21.15.62:0/1993972893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 msgr2=0x56512ea0d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:02.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f4338e52640 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 0x56512ea0d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f4338e52640 1 -- 172.21.15.62:0/1993972893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56512e675210 con 0x56512ea08780 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f4338e52640 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512ea145a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4324009fd0 tx=0x7f4324005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db8090443d14f88 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f431cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f431cff9640 1 -- 172.21.15.62:0/1993972893 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324011a20 con 0x56512ea08780 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f433e652740 1 -- 172.21.15.62:0/1993972893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4330005520 con 0x56512ea08780 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f431cff9640 1 -- 172.21.15.62:0/1993972893 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4324011bc0 con 0x56512ea08780 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f431cff9640 1 -- 172.21.15.62:0/1993972893 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f4324011dc0 con 0x56512ea08780 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f433e652740 10 client.17295 fetch_fsmap learned FSMap version 33 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.129+0000 7f433e652740 1 -- 172.21.15.62:0/1993972893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56512e741620 con 0x56512ea08780 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.130+0000 7f431cff9640 1 -- 172.21.15.62:0/1993972893 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4330005520 con 0x56512ea08780 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.130+0000 7f431cff9640 1 -- 172.21.15.62:0/1993972893 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4324022410 con 0x56512ea08780 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.130+0000 7f433e652740 10 client.17295 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.130+0000 7f433e652740 10 client.17295 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.130+0000 7f433e652740 20 client.17295 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.130+0000 7f433e652740 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56512ea18cb0 0x56512ea1b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.130+0000 7f433e652740 4 client.17295 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.130+0000 7f433e652740 1 -- 172.21.15.62:0/1993972893 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56512e96fa20 con 0x56512ea18cb0 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.130+0000 7f4339653640 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56512ea18cb0 0x56512ea1b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.131+0000 7f431cff9640 10 client.17295 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.131+0000 7f4339653640 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56512ea18cb0 0x56512ea1b130 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f84654f1c1dbbef server_cookie=7266ac169bbeff98 in_seq=0 out_seq=0 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.131+0000 7f431cff9640 1 -- 172.21.15.62:0/1993972893 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56512e96fa20 con 0x56512ea18cb0 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.131+0000 7f431cff9640 10 client.17295 handle_command_reply: tid=0 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.131+0000 7f433e652740 1 client.17295 shutdown 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.131+0000 7f433e652740 1 -- 172.21.15.62:0/1993972893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 msgr2=0x56512ea145a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.131+0000 7f433e652740 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512ea145a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f4324009fd0 tx=0x7f4324005850 comp rx=0 tx=0).stop 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f433e652740 1 -- 172.21.15.62:0/1993972893 shutdown_connections 2024-09-15T23:24:02.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f433e652740 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56512ea18cb0 0x56512ea1b130 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f433e652740 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512ea145a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f433e652740 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 0x56512ea0d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f433e652740 1 --2- 172.21.15.62:0/1993972893 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 0x56512ea0d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f433e652740 1 -- 172.21.15.62:0/1993972893 >> 172.21.15.62:0/1993972893 conn(0x56512e9fa010 msgr2=0x56512e9fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:02.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f431cff9640 0 client.17295 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:02.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f433e652740 1 -- 172.21.15.62:0/1993972893 shutdown_connections 2024-09-15T23:24:02.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f433e652740 1 -- 172.21.15.62:0/1993972893 wait complete. 2024-09-15T23:24:02.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.132+0000 7f433e652740 20 client.17295 trim_cache size 0 max 16384 2024-09-15T23:24:02.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f433e652740 1 Processor -- start 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f433e652740 1 -- start start 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f433e652740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 0x56512e9fa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f433e652740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 0x56512ea14530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f433e652740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512e9fa930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f433e652740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f43341aaca0 con 0x56512ea08780 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f433e652740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43341a8570 con 0x56512ea036f0 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f433e652740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f43341a63c0 con 0x56512ea02db0 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f4338e52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512e9fa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f4338e52640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512e9fa930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33280/0 (socket says 172.21.15.62:33280) 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f4338e52640 1 -- 172.21.15.62:0/4239541300 learned_addr learned my addr 172.21.15.62:0/4239541300 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f4338e52640 1 --2- 172.21.15.62:0/4239541300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 0x56512e9fa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.151+0000 7f4339653640 1 --2- 172.21.15.62:0/4239541300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 0x56512ea14530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.152+0000 7f4338e52640 1 -- 172.21.15.62:0/4239541300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 msgr2=0x56512e9fa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.152+0000 7f4338e52640 1 --2- 172.21.15.62:0/4239541300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 0x56512e9fa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.152+0000 7f4338e52640 1 -- 172.21.15.62:0/4239541300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 msgr2=0x56512ea14530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.152+0000 7f4338e52640 1 --2- 172.21.15.62:0/4239541300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 0x56512ea14530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.152+0000 7f4338e52640 1 -- 172.21.15.62:0/4239541300 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56512e9f52d0 con 0x56512ea08780 2024-09-15T23:24:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.152+0000 7f4338e52640 1 --2- 172.21.15.62:0/4239541300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512e9fa930 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f4324001eb0 tx=0x7f4324009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9fd8cf90289539 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.152+0000 7f431effd640 1 -- 172.21.15.62:0/4239541300 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43240132d0 con 0x56512ea08780 2024-09-15T23:24:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.152+0000 7f431effd640 1 -- 172.21.15.62:0/4239541300 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4324011a20 con 0x56512ea08780 2024-09-15T23:24:02.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.152+0000 7f431effd640 1 -- 172.21.15.62:0/4239541300 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324005070 con 0x56512ea08780 2024-09-15T23:24:02.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.153+0000 7f433e652740 1 -- 172.21.15.62:0/4239541300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 msgr2=0x56512e9fa930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.153+0000 7f433e652740 1 --2- 172.21.15.62:0/4239541300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512e9fa930 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f4324001eb0 tx=0x7f4324009bd0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.154+0000 7f433e652740 1 -- 172.21.15.62:0/4239541300 shutdown_connections 2024-09-15T23:24:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.154+0000 7f433e652740 1 --2- 172.21.15.62:0/4239541300 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea08780 0x56512e9fa930 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.154+0000 7f433e652740 1 --2- 172.21.15.62:0/4239541300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea036f0 0x56512ea14530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.154+0000 7f433e652740 1 --2- 172.21.15.62:0/4239541300 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea02db0 0x56512e9fa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.154+0000 7f433e652740 1 -- 172.21.15.62:0/4239541300 >> 172.21.15.62:0/4239541300 conn(0x56512e9fa010 msgr2=0x56512ea0d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.154+0000 7f433e652740 1 -- 172.21.15.62:0/4239541300 shutdown_connections 2024-09-15T23:24:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.154+0000 7f433e652740 1 -- 172.21.15.62:0/4239541300 wait complete. 2024-09-15T23:24:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.155+0000 7f433e652740 1 -- start start 2024-09-15T23:24:02.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.155+0000 7f433e652740 1 Processor -- start 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.156+0000 7f433e652740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 0x56512ea11250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.156+0000 7f433e652740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea11790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.156+0000 7f433e652740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 0x56512ea81a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.156+0000 7f433e652740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f43341a6a30 con 0x56512ea02db0 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.156+0000 7f433e652740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f43341a7ea0 con 0x56512ea08780 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.156+0000 7f4339653640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea11790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.156+0000 7f4339653640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea11790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36590/0 (socket says 172.21.15.62:36590) 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.156+0000 7f4339653640 1 -- 172.21.15.62:0/3605268691 learned_addr learned my addr 172.21.15.62:0/3605268691 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.156+0000 7f4338e52640 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 0x56512ea81a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.157+0000 7f433e652740 1 -- 172.21.15.62:0/3605268691 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f43341a7100 con 0x56512ea036f0 2024-09-15T23:24:02.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.157+0000 7f433e652740 10 client.0 _ll_register_callbacks cb 0x7f4324013220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:02.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.161+0000 7f4338e52640 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 0x56512ea11250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.161+0000 7f4339653640 1 -- 172.21.15.62:0/3605268691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 msgr2=0x56512ea81a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.161+0000 7f4339653640 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 0x56512ea81a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.161+0000 7f4339653640 1 -- 172.21.15.62:0/3605268691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 msgr2=0x56512ea11250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.161+0000 7f4339653640 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 0x56512ea11250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.161+0000 7f4339653640 1 -- 172.21.15.62:0/3605268691 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56512ea11e30 con 0x56512ea036f0 2024-09-15T23:24:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.162+0000 7f4339653640 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea11790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4330006010 tx=0x7f43300040f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcf6c6fb672b899 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:02.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.162+0000 7f431cff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:24:02.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.163+0000 7f431cff9640 1 -- 172.21.15.62:0/3605268691 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4330035040 con 0x56512ea036f0 2024-09-15T23:24:02.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.163+0000 7f431cff9640 1 -- 172.21.15.62:0/3605268691 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f43300318b0 con 0x56512ea036f0 2024-09-15T23:24:02.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.166+0000 7f433e652740 1 -- 172.21.15.62:0/3605268691 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4324011a20 con 0x56512ea036f0 2024-09-15T23:24:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.168+0000 7f431cff9640 1 -- 172.21.15.62:0/3605268691 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f433005f050 con 0x56512ea036f0 2024-09-15T23:24:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.168+0000 7f433e652740 10 client.26674 fetch_fsmap learned FSMap version 33 2024-09-15T23:24:02.169 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.168+0000 7f433e652740 1 -- 172.21.15.62:0/3605268691 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x56512ea83150 con 0x56512ea036f0 2024-09-15T23:24:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.169+0000 7f431cff9640 1 -- 172.21.15.62:0/3605268691 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43300035f0 con 0x56512ea036f0 2024-09-15T23:24:02.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.169+0000 7f431cff9640 1 -- 172.21.15.62:0/3605268691 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 33) ==== 8362+0+0 (secure 0 0 0) 0x7f4330058070 con 0x56512ea036f0 2024-09-15T23:24:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.169+0000 7f433e652740 10 client.26674 fetch_fsmap finished waiting for FSMap version 33 2024-09-15T23:24:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.169+0000 7f433e652740 10 client.26674 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.169+0000 7f433e652740 20 client.26674 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.169+0000 7f433e652740 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56512ea83780 0x56512ea85b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.170+0000 7f433e652740 4 client.26674 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:24:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.170+0000 7f433e652740 1 -- 172.21.15.62:0/3605268691 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f4324011a20 con 0x56512ea83780 2024-09-15T23:24:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.170+0000 7f4338e52640 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56512ea83780 0x56512ea85b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:02.171 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.171+0000 7f431cff9640 10 client.26674 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:02.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.171+0000 7f4338e52640 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56512ea83780 0x56512ea85b70 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ad9a4b94234f7ba server_cookie=f0f8d782fe3a1e2e in_seq=0 out_seq=0 2024-09-15T23:24:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.333+0000 7f431cff9640 1 -- 172.21.15.62:0/3605268691 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f4324011a20 con 0x56512ea83780 2024-09-15T23:24:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.333+0000 7f431cff9640 10 client.26674 handle_command_reply: tid=0 2024-09-15T23:24:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.333+0000 7f433e652740 1 client.26674 shutdown 2024-09-15T23:24:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.333+0000 7f433e652740 1 -- 172.21.15.62:0/3605268691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 msgr2=0x56512ea11790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.333+0000 7f433e652740 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea11790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4330006010 tx=0x7f43300040f0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.333+0000 7f433e652740 1 -- 172.21.15.62:0/3605268691 shutdown_connections 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.333+0000 7f433e652740 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56512ea83780 0x56512ea85b70 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.334+0000 7f433e652740 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56512ea08780 0x56512ea81a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.334+0000 7f433e652740 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56512ea036f0 0x56512ea11790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.334+0000 7f433e652740 1 --2- 172.21.15.62:0/3605268691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56512ea02db0 0x56512ea11250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.334+0000 7f433e652740 1 -- 172.21.15.62:0/3605268691 >> 172.21.15.62:0/3605268691 conn(0x56512e9fa010 msgr2=0x56512e9f89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.334+0000 7f431cff9640 0 client.26674 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.334+0000 7f433e652740 1 -- 172.21.15.62:0/3605268691 shutdown_connections 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.334+0000 7f433e652740 1 -- 172.21.15.62:0/3605268691 wait complete. 2024-09-15T23:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.334+0000 7f433e652740 20 client.26674 trim_cache size 0 max 16384 2024-09-15T23:24:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:24:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:24:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:24:02.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 -- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43200a3370 msgr2=0x7f43200a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43200a3370 0x7f43200a5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f432c0a4200 tx=0x7f432c009520 comp rx=0 tx=0).stop 2024-09-15T23:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 -- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334107890 msgr2=0x7f433419f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334107890 0x7f433419f3d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f432c00dac0 tx=0x7f432c00df80 comp rx=0 tx=0).stop 2024-09-15T23:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 -- 172.21.15.62:0/1456235041 shutdown_connections 2024-09-15T23:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f43200a3370 0x7f43200a5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4334107890 0x7f433419f3d0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4334104f60 0x7f4334198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 --2- 172.21.15.62:0/1456235041 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4334102630 0x7f4334197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:02.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 -- 172.21.15.62:0/1456235041 >> 172.21.15.62:0/1456235041 conn(0x7f43340fc1b0 msgr2=0x7f4334102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:02.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 -- 172.21.15.62:0/1456235041 shutdown_connections 2024-09-15T23:24:02.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:02.335+0000 7f433b8b7640 1 -- 172.21.15.62:0/1456235041 wait complete. 2024-09-15T23:24:03.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:03 smithi062 ceph-mon[30949]: pgmap v603: 129 pgs: 129 active+clean; 1.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 5.5 MiB/s wr, 649 op/s 2024-09-15T23:24:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:03 smithi067 ceph-mon[35665]: pgmap v603: 129 pgs: 129 active+clean; 1.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 5.5 MiB/s wr, 649 op/s 2024-09-15T23:24:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:03 smithi148 ceph-mon[35610]: pgmap v603: 129 pgs: 129 active+clean; 1.5 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 5.5 MiB/s wr, 649 op/s 2024-09-15T23:24:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 465.0 s, 931.0 tps, lat 109.012 ms stddev 132.764 2024-09-15T23:24:05.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:05 smithi062 ceph-mon[30949]: pgmap v604: 129 pgs: 129 active+clean; 1.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 5.5 MiB/s wr, 637 op/s 2024-09-15T23:24:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:05 smithi067 ceph-mon[35665]: pgmap v604: 129 pgs: 129 active+clean; 1.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 5.5 MiB/s wr, 637 op/s 2024-09-15T23:24:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:05 smithi148 ceph-mon[35610]: pgmap v604: 129 pgs: 129 active+clean; 1.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 5.5 MiB/s wr, 637 op/s 2024-09-15T23:24:07.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:07 smithi062 ceph-mon[30949]: pgmap v605: 129 pgs: 129 active+clean; 1.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 4.6 MiB/s wr, 577 op/s 2024-09-15T23:24:07.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:07 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:24:08.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:07 smithi067 ceph-mon[35665]: pgmap v605: 129 pgs: 129 active+clean; 1.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 4.6 MiB/s wr, 577 op/s 2024-09-15T23:24:08.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:07 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:24:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:07 smithi148 ceph-mon[35610]: pgmap v605: 129 pgs: 129 active+clean; 1.5 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 4.6 MiB/s wr, 577 op/s 2024-09-15T23:24:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:07 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:24:08.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:08 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:24:08.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:08 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:24:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:08 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:24:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:08 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:24:09.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:08 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:24:09.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:08 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:24:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 470.0 s, 1056.4 tps, lat 94.634 ms stddev 104.752 2024-09-15T23:24:09.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:09 smithi062 ceph-mon[30949]: pgmap v606: 129 pgs: 129 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.6 MiB/s wr, 656 op/s 2024-09-15T23:24:10.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:09 smithi067 ceph-mon[35665]: pgmap v606: 129 pgs: 129 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.6 MiB/s wr, 656 op/s 2024-09-15T23:24:10.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:09 smithi148 ceph-mon[35610]: pgmap v606: 129 pgs: 129 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.6 MiB/s wr, 656 op/s 2024-09-15T23:24:11.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:11 smithi062 ceph-mon[30949]: pgmap v607: 129 pgs: 129 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 5.3 MiB/s wr, 626 op/s 2024-09-15T23:24:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:11 smithi067 ceph-mon[35665]: pgmap v607: 129 pgs: 129 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 5.3 MiB/s wr, 626 op/s 2024-09-15T23:24:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:11 smithi148 ceph-mon[35610]: pgmap v607: 129 pgs: 129 active+clean; 1.5 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 5.3 MiB/s wr, 626 op/s 2024-09-15T23:24:13.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:13 smithi062 ceph-mon[30949]: pgmap v608: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.2 MiB/s wr, 667 op/s 2024-09-15T23:24:13.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:13 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:24:14.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:13 smithi067 ceph-mon[35665]: pgmap v608: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.2 MiB/s wr, 667 op/s 2024-09-15T23:24:14.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:13 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:24:14.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:13 smithi148 ceph-mon[35610]: pgmap v608: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 6.2 MiB/s wr, 667 op/s 2024-09-15T23:24:14.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:13 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:24:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 475.0 s, 923.6 tps, lat 106.774 ms stddev 123.708 2024-09-15T23:24:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.497+0000 7f599f71a640 1 -- 172.21.15.62:0/1883632382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5998072d20 msgr2=0x7f5998073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.497+0000 7f599f71a640 1 --2- 172.21.15.62:0/1883632382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5998072d20 0x7f5998073180 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5994009900 tx=0x7f599402a440 comp rx=0 tx=0).stop 2024-09-15T23:24:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.498+0000 7f599f71a640 1 -- 172.21.15.62:0/1883632382 shutdown_connections 2024-09-15T23:24:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.498+0000 7f599f71a640 1 --2- 172.21.15.62:0/1883632382 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59980736c0 0x7f5998119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.498+0000 7f599f71a640 1 --2- 172.21.15.62:0/1883632382 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5998072d20 0x7f5998073180 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.498+0000 7f599f71a640 1 --2- 172.21.15.62:0/1883632382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5998109700 0x7f5998109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.498+0000 7f599f71a640 1 -- 172.21.15.62:0/1883632382 >> 172.21.15.62:0/1883632382 conn(0x7f599806ece0 msgr2=0x7f5998071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.499+0000 7f599f71a640 1 -- 172.21.15.62:0/1883632382 shutdown_connections 2024-09-15T23:24:15.501 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.500+0000 7f599f71a640 1 -- 172.21.15.62:0/1883632382 wait complete. 2024-09-15T23:24:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599f71a640 1 Processor -- start 2024-09-15T23:24:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599f71a640 1 -- start start 2024-09-15T23:24:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599f71a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5998072d20 0x7f59981a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599f71a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59980736c0 0x7f59981a7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599f71a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5998109700 0x7f59981ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599f71a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f599811c700 con 0x7f59980736c0 2024-09-15T23:24:15.502 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599f71a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f599811c580 con 0x7f5998072d20 2024-09-15T23:24:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599f71a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f599811c880 con 0x7f5998109700 2024-09-15T23:24:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599ccb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59980736c0 0x7f59981a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599ccb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59980736c0 0x7f59981a7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34856/0 (socket says 172.21.15.62:34856) 2024-09-15T23:24:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.501+0000 7f599ccb5640 1 -- 172.21.15.62:0/1355689123 learned_addr learned my addr 172.21.15.62:0/1355689123 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.502+0000 7f599ccb5640 1 -- 172.21.15.62:0/1355689123 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5998109700 msgr2=0x7f59981ae1b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:24:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.502+0000 7f599ccb5640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5998109700 0x7f59981ae1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.502+0000 7f599ccb5640 1 -- 172.21.15.62:0/1355689123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5998072d20 msgr2=0x7f59981a6c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:24:15.503 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.502+0000 7f599ccb5640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5998072d20 0x7f59981a6c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.502+0000 7f599ccb5640 1 -- 172.21.15.62:0/1355689123 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59981ae850 con 0x7f59980736c0 2024-09-15T23:24:15.504 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.503+0000 7f599ccb5640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59980736c0 0x7f59981a7180 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f599402aa30 tx=0x7f59940038f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.504+0000 7f598e7fc640 1 -- 172.21.15.62:0/1355689123 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599403f030 con 0x7f59980736c0 2024-09-15T23:24:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.504+0000 7f598e7fc640 1 -- 172.21.15.62:0/1355689123 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5994003d20 con 0x7f59980736c0 2024-09-15T23:24:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.504+0000 7f598e7fc640 1 -- 172.21.15.62:0/1355689123 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5994035070 con 0x7f59980736c0 2024-09-15T23:24:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.504+0000 7f599f71a640 1 -- 172.21.15.62:0/1355689123 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59981aeae0 con 0x7f59980736c0 2024-09-15T23:24:15.505 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.504+0000 7f599f71a640 1 -- 172.21.15.62:0/1355689123 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59981aef80 con 0x7f59980736c0 2024-09-15T23:24:15.510 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.508+0000 7f59a24ae740 1 Processor -- start 2024-09-15T23:24:15.512 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.511+0000 7f59a24ae740 1 -- start start 2024-09-15T23:24:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.511+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7ddf1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.511+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf22e0 0x562e7ddf6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.511+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7ddf9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.511+0000 7f59a24ae740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562e7dd552f0 con 0x562e7ddf7370 2024-09-15T23:24:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.511+0000 7f59a24ae740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562e7dcfd270 con 0x562e7ddf22e0 2024-09-15T23:24:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.511+0000 7f59a24ae740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562e7dd512b0 con 0x562e7ddf19a0 2024-09-15T23:24:15.513 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.512+0000 7f599dcb7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7ddf1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.512+0000 7f599dcb7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7ddf1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36308/0 (socket says 172.21.15.62:36308) 2024-09-15T23:24:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.512+0000 7f599dcb7640 1 -- 172.21.15.62:0/219123146 learned_addr learned my addr 172.21.15.62:0/219123146 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:15.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.512+0000 7f599d4b6640 1 --2- 172.21.15.62:0/219123146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf22e0 0x562e7ddf6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.514+0000 7f599ccb5640 1 --2- 172.21.15.62:0/219123146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7ddf9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.514+0000 7f598e7fc640 1 -- 172.21.15.62:0/1355689123 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f5994042030 con 0x7f59980736c0 2024-09-15T23:24:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.514+0000 7f599d4b6640 1 -- 172.21.15.62:0/219123146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 msgr2=0x562e7ddf1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.514+0000 7f599d4b6640 1 --2- 172.21.15.62:0/219123146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7ddf1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.514+0000 7f599d4b6640 1 -- 172.21.15.62:0/219123146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 msgr2=0x562e7ddf9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.514+0000 7f599d4b6640 1 --2- 172.21.15.62:0/219123146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7ddf9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.514+0000 7f599d4b6640 1 -- 172.21.15.62:0/219123146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e7dcfc340 con 0x562e7ddf22e0 2024-09-15T23:24:15.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.515+0000 7f598e7fc640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59840a3520 0x7f59840a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.516+0000 7f599d4b6640 1 --2- 172.21.15.62:0/219123146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf22e0 0x562e7ddf6da0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f59880096b0 tx=0x7f5988009b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aed8ab95ea721be6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:15.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.516+0000 7f598e7fc640 1 -- 172.21.15.62:0/1355689123 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f59940f2bc0 con 0x7f59980736c0 2024-09-15T23:24:15.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.517+0000 7f599dcb7640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59840a3520 0x7f59840a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.519+0000 7f59837fe640 1 -- 172.21.15.62:0/219123146 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988015020 con 0x562e7ddf22e0 2024-09-15T23:24:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.519+0000 7f599dcb7640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59840a3520 0x7f59840a59e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5990009fc0 tx=0x7f599000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.519+0000 7f59837fe640 1 -- 172.21.15.62:0/219123146 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f598800bda0 con 0x562e7ddf22e0 2024-09-15T23:24:15.520 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.519+0000 7f59837fe640 1 -- 172.21.15.62:0/219123146 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988010380 con 0x562e7ddf22e0 2024-09-15T23:24:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.520+0000 7f59a24ae740 1 -- 172.21.15.62:0/219123146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf22e0 msgr2=0x562e7ddf6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.520+0000 7f59a24ae740 1 --2- 172.21.15.62:0/219123146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf22e0 0x562e7ddf6da0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f59880096b0 tx=0x7f5988009b70 comp rx=0 tx=0).stop 2024-09-15T23:24:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.520+0000 7f59a24ae740 1 -- 172.21.15.62:0/219123146 shutdown_connections 2024-09-15T23:24:15.521 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.520+0000 7f59a24ae740 1 --2- 172.21.15.62:0/219123146 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7ddf9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.520+0000 7f59a24ae740 1 --2- 172.21.15.62:0/219123146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf22e0 0x562e7ddf6da0 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f59880096b0 tx=0x7f5988009b70 comp rx=0 tx=0).stop 2024-09-15T23:24:15.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.520+0000 7f59a24ae740 1 --2- 172.21.15.62:0/219123146 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7ddf1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.522 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.520+0000 7f59a24ae740 1 -- 172.21.15.62:0/219123146 >> 172.21.15.62:0/219123146 conn(0x562e7dded010 msgr2=0x562e7dded410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:15.526 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.526+0000 7f59a24ae740 1 -- 172.21.15.62:0/219123146 shutdown_connections 2024-09-15T23:24:15.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.526+0000 7f59a24ae740 1 -- 172.21.15.62:0/219123146 wait complete. 2024-09-15T23:24:15.527 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.527+0000 7f59a24ae740 1 -- start start 2024-09-15T23:24:15.528 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.527+0000 7f59a24ae740 1 Processor -- start 2024-09-15T23:24:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.528+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de0cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.528+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7de0d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.528+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7de12410 0x562e7de14800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.528+0000 7f59a24ae740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562e7de0b350 con 0x562e7ddf7370 2024-09-15T23:24:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.528+0000 7f59a24ae740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562e7dcde290 con 0x562e7de12410 2024-09-15T23:24:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.528+0000 7f59a24ae740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562e7de0b4d0 con 0x562e7ddf19a0 2024-09-15T23:24:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.528+0000 7f59a24ae740 10 client.0 _ll_register_callbacks cb 0x562e7dd0f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:15.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.529+0000 7f599d4b6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de0cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.529+0000 7f599ccb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7de0d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.529+0000 7f599d4b6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de0cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36318/0 (socket says 172.21.15.62:36318) 2024-09-15T23:24:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.529+0000 7f599d4b6640 1 -- 172.21.15.62:0/1670025210 learned_addr learned my addr 172.21.15.62:0/1670025210 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.529+0000 7f599ccb5640 1 -- 172.21.15.62:0/1670025210 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 msgr2=0x562e7de0cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.529+0000 7f599ccb5640 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de0cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.529+0000 7f599ccb5640 1 -- 172.21.15.62:0/1670025210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7de12410 msgr2=0x562e7de14800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:24:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.529+0000 7f599ccb5640 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7de12410 0x562e7de14800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.529+0000 7f599ccb5640 1 -- 172.21.15.62:0/1670025210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e7da68210 con 0x562e7ddf7370 2024-09-15T23:24:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.530+0000 7f599ccb5640 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7de0d3e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f59940bb1d0 tx=0x7f599403bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61021c9e1f42cb6c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.530+0000 7f5980ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.530+0000 7f5980ff9640 1 -- 172.21.15.62:0/1670025210 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5994009520 con 0x562e7ddf7370 2024-09-15T23:24:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.530+0000 7f5980ff9640 1 -- 172.21.15.62:0/1670025210 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f59940316e0 con 0x562e7ddf7370 2024-09-15T23:24:15.532 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.531+0000 7f59a24ae740 1 -- 172.21.15.62:0/1670025210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f598800bda0 con 0x562e7ddf7370 2024-09-15T23:24:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.531+0000 7f5980ff9640 1 -- 172.21.15.62:0/1670025210 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f59940bf500 con 0x562e7ddf7370 2024-09-15T23:24:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.531+0000 7f59a24ae740 10 client.17310 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.531+0000 7f59a24ae740 1 -- 172.21.15.62:0/1670025210 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562e7db34620 con 0x562e7ddf7370 2024-09-15T23:24:15.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.532+0000 7f5980ff9640 1 -- 172.21.15.62:0/1670025210 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f598800bda0 con 0x562e7ddf7370 2024-09-15T23:24:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.532+0000 7f5980ff9640 1 -- 172.21.15.62:0/1670025210 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f59940be250 con 0x562e7ddf7370 2024-09-15T23:24:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.532+0000 7f59a24ae740 10 client.17310 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.532+0000 7f59a24ae740 10 client.17310 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.532+0000 7f59a24ae740 20 client.17310 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.532+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562e7de19110 0x562e7de1b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.532+0000 7f59a24ae740 4 client.17310 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.532+0000 7f59a24ae740 1 -- 172.21.15.62:0/1670025210 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e7dd62a20 con 0x562e7de19110 2024-09-15T23:24:15.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.533+0000 7f599d4b6640 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562e7de19110 0x562e7de1b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.538+0000 7f599d4b6640 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562e7de19110 0x562e7de1b590 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8285419deb4679b server_cookie=6c66538432526797 in_seq=0 out_seq=0 2024-09-15T23:24:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.538+0000 7f5980ff9640 10 client.17310 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f5980ff9640 1 -- 172.21.15.62:0/1670025210 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562e7dd62a20 con 0x562e7de19110 2024-09-15T23:24:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f5980ff9640 10 client.17310 handle_command_reply: tid=0 2024-09-15T23:24:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f59a24ae740 1 client.17310 shutdown 2024-09-15T23:24:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f59a24ae740 1 -- 172.21.15.62:0/1670025210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 msgr2=0x562e7de0d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7de0d3e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f59940bb1d0 tx=0x7f599403bf10 comp rx=0 tx=0).stop 2024-09-15T23:24:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f59a24ae740 1 -- 172.21.15.62:0/1670025210 shutdown_connections 2024-09-15T23:24:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562e7de19110 0x562e7de1b590 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7de12410 0x562e7de14800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7de0d3e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1670025210 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de0cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f59a24ae740 1 -- 172.21.15.62:0/1670025210 >> 172.21.15.62:0/1670025210 conn(0x562e7dded010 msgr2=0x562e7ddee820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.539+0000 7f5980ff9640 0 client.17310 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.540+0000 7f59a24ae740 1 -- 172.21.15.62:0/1670025210 shutdown_connections 2024-09-15T23:24:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.540+0000 7f59a24ae740 1 -- 172.21.15.62:0/1670025210 wait complete. 2024-09-15T23:24:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.540+0000 7f59a24ae740 20 client.17310 trim_cache size 0 max 16384 2024-09-15T23:24:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.564+0000 7f59a24ae740 1 Processor -- start 2024-09-15T23:24:15.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.564+0000 7f59a24ae740 1 -- start start 2024-09-15T23:24:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.565+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf19a0 0x562e7ddec0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.565+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7ddec5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.565+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7de12410 0x562e7ddee400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.565+0000 7f599d4b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf19a0 0x562e7ddec0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.565+0000 7f599d4b6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf19a0 0x562e7ddec0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:47998/0 (socket says 172.21.15.62:47998) 2024-09-15T23:24:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.565+0000 7f599d4b6640 1 -- 172.21.15.62:0/3359135773 learned_addr learned my addr 172.21.15.62:0/3359135773 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.565+0000 7f59a24ae740 1 -- 172.21.15.62:0/3359135773 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f59981b3490 con 0x562e7ddf7370 2024-09-15T23:24:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.566+0000 7f599dcb7640 1 --2- 172.21.15.62:0/3359135773 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7de12410 0x562e7ddee400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.566+0000 7f59a24ae740 1 -- 172.21.15.62:0/3359135773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f59981b0d60 con 0x562e7ddf19a0 2024-09-15T23:24:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.566+0000 7f59a24ae740 1 -- 172.21.15.62:0/3359135773 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5998118c60 con 0x562e7de12410 2024-09-15T23:24:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.566+0000 7f599d4b6640 1 -- 172.21.15.62:0/3359135773 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7de12410 msgr2=0x562e7ddee400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.566+0000 7f599d4b6640 1 --2- 172.21.15.62:0/3359135773 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7de12410 0x562e7ddee400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.566+0000 7f599d4b6640 1 -- 172.21.15.62:0/3359135773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 msgr2=0x562e7ddec5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.566+0000 7f599d4b6640 1 --2- 172.21.15.62:0/3359135773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7ddec5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.566+0000 7f599d4b6640 1 -- 172.21.15.62:0/3359135773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e7ddece30 con 0x562e7ddf19a0 2024-09-15T23:24:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.567+0000 7f599d4b6640 1 --2- 172.21.15.62:0/3359135773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf19a0 0x562e7ddec0b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f59880096b0 tx=0x7f5988009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2029f27e433736b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.568+0000 7f5982ffd640 1 -- 172.21.15.62:0/3359135773 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988015050 con 0x562e7ddf19a0 2024-09-15T23:24:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.568+0000 7f5982ffd640 1 -- 172.21.15.62:0/3359135773 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f598800b660 con 0x562e7ddf19a0 2024-09-15T23:24:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.568+0000 7f5982ffd640 1 -- 172.21.15.62:0/3359135773 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59880256a0 con 0x562e7ddf19a0 2024-09-15T23:24:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.571+0000 7f59a24ae740 1 -- 172.21.15.62:0/3359135773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf19a0 msgr2=0x562e7ddec0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.571+0000 7f59a24ae740 1 --2- 172.21.15.62:0/3359135773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf19a0 0x562e7ddec0b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f59880096b0 tx=0x7f5988009040 comp rx=0 tx=0).stop 2024-09-15T23:24:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.575+0000 7f59a24ae740 1 -- 172.21.15.62:0/3359135773 shutdown_connections 2024-09-15T23:24:15.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.575+0000 7f59a24ae740 1 --2- 172.21.15.62:0/3359135773 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7de12410 0x562e7ddee400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.575+0000 7f59a24ae740 1 --2- 172.21.15.62:0/3359135773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7ddec5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.575+0000 7f59a24ae740 1 --2- 172.21.15.62:0/3359135773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7ddf19a0 0x562e7ddec0b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.575+0000 7f59a24ae740 1 -- 172.21.15.62:0/3359135773 >> 172.21.15.62:0/3359135773 conn(0x562e7dded010 msgr2=0x562e7ddeb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:15.586 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.585+0000 7f59a24ae740 1 -- 172.21.15.62:0/3359135773 shutdown_connections 2024-09-15T23:24:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.588+0000 7f59a24ae740 1 -- 172.21.15.62:0/3359135773 wait complete. 2024-09-15T23:24:15.590 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.589+0000 7f59a24ae740 1 -- start start 2024-09-15T23:24:15.590 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.589+0000 7f59a24ae740 1 Processor -- start 2024-09-15T23:24:15.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.590+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de2a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.590+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7de2add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.590+0000 7f59a24ae740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7de12410 0x562e7de2bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.590+0000 7f59a24ae740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f59981af2a0 con 0x562e7ddf7370 2024-09-15T23:24:15.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.590+0000 7f59a24ae740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f59981b0690 con 0x562e7de12410 2024-09-15T23:24:15.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.590+0000 7f59a24ae740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f59981af8f0 con 0x562e7ddf19a0 2024-09-15T23:24:15.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.591+0000 7f59a24ae740 10 client.0 _ll_register_callbacks cb 0x7f59940bb010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:15.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.591+0000 7f599d4b6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de2a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.591+0000 7f599d4b6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de2a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36346/0 (socket says 172.21.15.62:36346) 2024-09-15T23:24:15.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.591+0000 7f599d4b6640 1 -- 172.21.15.62:0/1547148508 learned_addr learned my addr 172.21.15.62:0/1547148508 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:15.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.592+0000 7f599d4b6640 1 -- 172.21.15.62:0/1547148508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7de12410 msgr2=0x562e7de2bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:15.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.592+0000 7f599d4b6640 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7de12410 0x562e7de2bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.592+0000 7f599d4b6640 1 -- 172.21.15.62:0/1547148508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 msgr2=0x562e7de2add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:15.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.592+0000 7f599d4b6640 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7de2add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.592+0000 7f599d4b6640 1 -- 172.21.15.62:0/1547148508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e7de2c3f0 con 0x562e7ddf19a0 2024-09-15T23:24:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.596+0000 7f599d4b6640 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de2a890 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5988011ac0 tx=0x7f5988011af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=334a7f3372a0d74c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:15.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.596+0000 7f5980ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:24:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.599+0000 7f5980ff9640 1 -- 172.21.15.62:0/1547148508 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5988037060 con 0x562e7ddf19a0 2024-09-15T23:24:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.599+0000 7f59a24ae740 1 -- 172.21.15.62:0/1547148508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f598800b660 con 0x562e7ddf19a0 2024-09-15T23:24:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.599+0000 7f5980ff9640 1 -- 172.21.15.62:0/1547148508 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f598800b810 con 0x562e7ddf19a0 2024-09-15T23:24:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.600+0000 7f5980ff9640 1 -- 172.21.15.62:0/1547148508 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f598803e050 con 0x562e7ddf19a0 2024-09-15T23:24:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.602+0000 7f59a24ae740 10 client.26689 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:15.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.602+0000 7f59a24ae740 1 -- 172.21.15.62:0/1547148508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562e7de2d6e0 con 0x562e7ddf19a0 2024-09-15T23:24:15.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.603+0000 7f5980ff9640 1 -- 172.21.15.62:0/1547148508 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5988032460 con 0x562e7ddf19a0 2024-09-15T23:24:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.603+0000 7f5980ff9640 1 -- 172.21.15.62:0/1547148508 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f5988025440 con 0x562e7ddf19a0 2024-09-15T23:24:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.603+0000 7f59a24ae740 10 client.26689 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.603+0000 7f59a24ae740 10 client.26689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.604+0000 7f59a24ae740 20 client.26689 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.604+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562e7de84100 0x562e7de86520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.604+0000 7f59a24ae740 4 client.26689 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:24:15.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.604+0000 7f59a24ae740 1 -- 172.21.15.62:0/1547148508 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f598800b660 con 0x562e7de84100 2024-09-15T23:24:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.607+0000 7f599ccb5640 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562e7de84100 0x562e7de86520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:15.615 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.614+0000 7f599ccb5640 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562e7de84100 0x562e7de86520 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=560e6386bd1838e5 server_cookie=19e8f653b3648c83 in_seq=0 out_seq=0 2024-09-15T23:24:15.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.615+0000 7f5980ff9640 10 client.26689 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.648+0000 7f5980ff9640 1 -- 172.21.15.62:0/1547148508 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f598800b660 con 0x562e7de84100 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.648+0000 7f5980ff9640 10 client.26689 handle_command_reply: tid=0 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.648+0000 7f59a24ae740 1 client.26689 shutdown 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.648+0000 7f59a24ae740 1 -- 172.21.15.62:0/1547148508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 msgr2=0x562e7de2a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.648+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de2a890 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5988011ac0 tx=0x7f5988011af0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.648+0000 7f59a24ae740 1 -- 172.21.15.62:0/1547148508 shutdown_connections 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.648+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562e7de84100 0x562e7de86520 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.648+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562e7de12410 0x562e7de2bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.649+0000 7f5980ff9640 0 client.26689 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.649+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562e7ddf7370 0x562e7de2add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.649+0000 7f59a24ae740 1 --2- 172.21.15.62:0/1547148508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562e7ddf19a0 0x562e7de2a890 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.649+0000 7f59a24ae740 1 -- 172.21.15.62:0/1547148508 >> 172.21.15.62:0/1547148508 conn(0x562e7dded010 msgr2=0x562e7ddec1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.649+0000 7f59a24ae740 1 -- 172.21.15.62:0/1547148508 shutdown_connections 2024-09-15T23:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.649+0000 7f59a24ae740 1 -- 172.21.15.62:0/1547148508 wait complete. 2024-09-15T23:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.649+0000 7f59a24ae740 20 client.26689 trim_cache size 0 max 16384 2024-09-15T23:24:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:24:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:24:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:24:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.650+0000 7f599f71a640 1 -- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59840a3520 msgr2=0x7f59840a59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.650+0000 7f599f71a640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59840a3520 0x7f59840a59e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5990009fc0 tx=0x7f599000b040 comp rx=0 tx=0).stop 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.650+0000 7f599f71a640 1 -- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59980736c0 msgr2=0x7f59981a7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.650+0000 7f599f71a640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59980736c0 0x7f59981a7180 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f599402aa30 tx=0x7f59940038f0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.651+0000 7f599f71a640 1 -- 172.21.15.62:0/1355689123 shutdown_connections 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.651+0000 7f599f71a640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f59840a3520 0x7f59840a59e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.651+0000 7f599f71a640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5998109700 0x7f59981ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.651+0000 7f599f71a640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59980736c0 0x7f59981a7180 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.651+0000 7f599f71a640 1 --2- 172.21.15.62:0/1355689123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5998072d20 0x7f59981a6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.651+0000 7f599f71a640 1 -- 172.21.15.62:0/1355689123 >> 172.21.15.62:0/1355689123 conn(0x7f599806ece0 msgr2=0x7f5998071640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.651+0000 7f599f71a640 1 -- 172.21.15.62:0/1355689123 shutdown_connections 2024-09-15T23:24:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:15.651+0000 7f599f71a640 1 -- 172.21.15.62:0/1355689123 wait complete. 2024-09-15T23:24:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:15 smithi062 ceph-mon[30949]: pgmap v609: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.5 MiB/s wr, 566 op/s 2024-09-15T23:24:16.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:15 smithi067 ceph-mon[35665]: pgmap v609: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.5 MiB/s wr, 566 op/s 2024-09-15T23:24:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:15 smithi148 ceph-mon[35610]: pgmap v609: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.5 MiB/s wr, 566 op/s 2024-09-15T23:24:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:17 smithi062 ceph-mon[30949]: pgmap v610: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.5 MiB/s wr, 541 op/s 2024-09-15T23:24:18.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:17 smithi067 ceph-mon[35665]: pgmap v610: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.5 MiB/s wr, 541 op/s 2024-09-15T23:24:18.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:17 smithi148 ceph-mon[35610]: pgmap v610: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.5 MiB/s wr, 541 op/s 2024-09-15T23:24:19.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 480.0 s, 853.0 tps, lat 118.629 ms stddev 135.479 2024-09-15T23:24:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:19 smithi062 ceph-mon[30949]: pgmap v611: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 6.6 MiB/s wr, 647 op/s 2024-09-15T23:24:20.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:19 smithi067 ceph-mon[35665]: pgmap v611: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 6.6 MiB/s wr, 647 op/s 2024-09-15T23:24:20.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:19 smithi148 ceph-mon[35610]: pgmap v611: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 6.6 MiB/s wr, 647 op/s 2024-09-15T23:24:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:21 smithi062 ceph-mon[30949]: pgmap v612: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.0 MiB/s wr, 615 op/s 2024-09-15T23:24:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:21 smithi067 ceph-mon[35665]: pgmap v612: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.0 MiB/s wr, 615 op/s 2024-09-15T23:24:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:21 smithi148 ceph-mon[35610]: pgmap v612: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 6.0 MiB/s wr, 615 op/s 2024-09-15T23:24:23.149 INFO:tasks.quiescer.fs.[cephfs]:Total duration (44.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:24:23.149 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-09-15T23:24:23.150 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:23.633 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:23.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:23 smithi062 ceph-mon[30949]: pgmap v613: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 6.4 MiB/s wr, 655 op/s 2024-09-15T23:24:24.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:23 smithi067 ceph-mon[35665]: pgmap v613: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 6.4 MiB/s wr, 655 op/s 2024-09-15T23:24:24.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:23 smithi148 ceph-mon[35610]: pgmap v613: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 6.4 MiB/s wr, 655 op/s 2024-09-15T23:24:24.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.445+0000 7f74796ff640 1 -- 172.21.15.62:0/2961738427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474109450 msgr2=0x7f7474110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.445+0000 7f74796ff640 1 --2- 172.21.15.62:0/2961738427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474109450 0x7f7474110250 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7468009860 tx=0x7f746802a360 comp rx=0 tx=0).stop 2024-09-15T23:24:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.450+0000 7f74796ff640 1 -- 172.21.15.62:0/2961738427 shutdown_connections 2024-09-15T23:24:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.450+0000 7f74796ff640 1 --2- 172.21.15.62:0/2961738427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f747411b140 0x7f747411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.450+0000 7f74796ff640 1 --2- 172.21.15.62:0/2961738427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7474109830 0x7f7474109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.450+0000 7f74796ff640 1 --2- 172.21.15.62:0/2961738427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474109450 0x7f7474110250 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.450+0000 7f74796ff640 1 -- 172.21.15.62:0/2961738427 >> 172.21.15.62:0/2961738427 conn(0x7f747406e140 msgr2=0x7f747406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:24.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.460+0000 7f74796ff640 1 -- 172.21.15.62:0/2961738427 shutdown_connections 2024-09-15T23:24:24.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.463+0000 7f74796ff640 1 -- 172.21.15.62:0/2961738427 wait complete. 2024-09-15T23:24:24.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.464+0000 7f74796ff640 1 Processor -- start 2024-09-15T23:24:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.474+0000 7f74796ff640 1 -- start start 2024-09-15T23:24:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.474+0000 7f74796ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474109450 0x7f747411a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.474+0000 7f74796ff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109830 0x7f7474114960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.474+0000 7f74796ff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f747411b140 0x7f7474114ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.474+0000 7f74727fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109830 0x7f7474114960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74727fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109830 0x7f7474114960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52534/0 (socket says 172.21.15.62:52534) 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74727fc640 1 -- 172.21.15.62:0/2365471807 learned_addr learned my addr 172.21.15.62:0/2365471807 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7474111370 con 0x7f7474109830 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f74741111f0 con 0x7f7474109450 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f74741114f0 con 0x7f747411b140 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74737fe640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f747411b140 0x7f7474114ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74737fe640 1 -- 172.21.15.62:0/2365471807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474109450 msgr2=0x7f747411a8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74737fe640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474109450 0x7f747411a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74737fe640 1 -- 172.21.15.62:0/2365471807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109830 msgr2=0x7f7474114960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74737fe640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109830 0x7f7474114960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74737fe640 1 -- 172.21.15.62:0/2365471807 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7474115620 con 0x7f747411b140 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.475+0000 7f74737fe640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f747411b140 0x7f7474114ea0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f746400b410 tx=0x7f746400b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:24.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.476+0000 7f745bfff640 1 -- 172.21.15.62:0/2365471807 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464015020 con 0x7f747411b140 2024-09-15T23:24:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.476+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74741bc130 con 0x7f747411b140 2024-09-15T23:24:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.476+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74741bc5b0 con 0x7f747411b140 2024-09-15T23:24:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.477+0000 7f745bfff640 1 -- 172.21.15.62:0/2365471807 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7464011d10 con 0x7f747411b140 2024-09-15T23:24:24.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.477+0000 7f745bfff640 1 -- 172.21.15.62:0/2365471807 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746400f2e0 con 0x7f747411b140 2024-09-15T23:24:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.483+0000 7f745bfff640 1 -- 172.21.15.62:0/2365471807 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7464018020 con 0x7f747411b140 2024-09-15T23:24:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.484+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7460005290 con 0x7f747411b140 2024-09-15T23:24:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.484+0000 7f745bfff640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f745c0a3450 0x7f745c0a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.484+0000 7f745bfff640 1 -- 172.21.15.62:0/2365471807 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f74640cf2b0 con 0x7f747411b140 2024-09-15T23:24:24.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.484+0000 7f7472ffd640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f745c0a3450 0x7f745c0a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:24.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.490+0000 7f7472ffd640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f745c0a3450 0x7f745c0a5910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f74680095f0 tx=0x7f7468005760 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:24.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.495+0000 7f745bfff640 1 -- 172.21.15.62:0/2365471807 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f746401f020 con 0x7f747411b140 2024-09-15T23:24:24.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 485.0 s, 693.0 tps, lat 141.999 ms stddev 160.031 2024-09-15T23:24:24.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.881+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7460005580 con 0x7f747411b140 2024-09-15T23:24:24.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.884+0000 7f745bfff640 1 -- 172.21.15.62:0/2365471807 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11904 (secure 0 0 0) 0x7f746409bbf0 con 0x7f747411b140 2024-09-15T23:24:24.886 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:24:24.887 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":34,"btime":"2024-09-15T23:24:12:668733+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:12.388349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:24:24.887 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 34 2024-09-15T23:24:24.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.893+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f745c0a3450 msgr2=0x7f745c0a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.893+0000 7f74796ff640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f745c0a3450 0x7f745c0a5910 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f74680095f0 tx=0x7f7468005760 comp rx=0 tx=0).stop 2024-09-15T23:24:24.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.893+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f747411b140 msgr2=0x7f7474114ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:24.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.893+0000 7f74796ff640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f747411b140 0x7f7474114ea0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f746400b410 tx=0x7f746400b8d0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.894+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 shutdown_connections 2024-09-15T23:24:24.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.894+0000 7f74796ff640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f745c0a3450 0x7f745c0a5910 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.894+0000 7f74796ff640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f747411b140 0x7f7474114ea0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.894+0000 7f74796ff640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109830 0x7f7474114960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.894+0000 7f74796ff640 1 --2- 172.21.15.62:0/2365471807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7474109450 0x7f747411a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:24.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.894+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 >> 172.21.15.62:0/2365471807 conn(0x7f747406e140 msgr2=0x7f7474072380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:24.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.896+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 shutdown_connections 2024-09-15T23:24:24.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:24.897+0000 7f74796ff640 1 -- 172.21.15.62:0/2365471807 wait complete. 2024-09-15T23:24:25.729 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0' 2024-09-15T23:24:25.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0 2024-09-15T23:24:25.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:25 smithi062 ceph-mon[30949]: pgmap v614: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 5.4 MiB/s wr, 590 op/s 2024-09-15T23:24:25.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:25 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2365471807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:26.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:25 smithi067 ceph-mon[35665]: pgmap v614: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 5.4 MiB/s wr, 590 op/s 2024-09-15T23:24:26.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:25 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2365471807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:26.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:25 smithi148 ceph-mon[35610]: pgmap v614: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 5.4 MiB/s wr, 590 op/s 2024-09-15T23:24:26.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:25 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2365471807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:26.184 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:26.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.775+0000 7f9df0cb2640 1 -- 172.21.15.62:0/913205088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dec11b140 msgr2=0x7f9dec11d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:26.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.775+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/913205088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dec11b140 0x7f9dec11d530 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009ca0 tx=0x7f9de402a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.775+0000 7f9df0cb2640 1 -- 172.21.15.62:0/913205088 shutdown_connections 2024-09-15T23:24:26.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.775+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/913205088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dec11b140 0x7f9dec11d530 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.775+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/913205088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dec109830 0x7f9dec109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.775+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/913205088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dec109450 0x7f9dec110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.775+0000 7f9df0cb2640 1 -- 172.21.15.62:0/913205088 >> 172.21.15.62:0/913205088 conn(0x7f9dec06e140 msgr2=0x7f9dec06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:26.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.778+0000 7f9df0cb2640 1 -- 172.21.15.62:0/913205088 shutdown_connections 2024-09-15T23:24:26.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.778+0000 7f9df0cb2640 1 -- 172.21.15.62:0/913205088 wait complete. 2024-09-15T23:24:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.780+0000 7f9df0cb2640 1 Processor -- start 2024-09-15T23:24:26.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.780+0000 7f9df0cb2640 1 -- start start 2024-09-15T23:24:26.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.780+0000 7f9df0cb2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dec109450 0x7f9dec1a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9df0cb2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dec109830 0x7f9dec1a7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9df0cb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dec11b140 0x7f9dec1ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9df0cb2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec120da0 con 0x7f9dec109830 2024-09-15T23:24:26.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9de9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dec109830 0x7f9dec1a7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9dead9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dec11b140 0x7f9dec1ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9dead9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dec11b140 0x7f9dec1ab810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48754/0 (socket says 172.21.15.62:48754) 2024-09-15T23:24:26.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9dead9d640 1 -- 172.21.15.62:0/461175848 learned_addr learned my addr 172.21.15.62:0/461175848 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:26.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9de9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dec109830 0x7f9dec1a7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52556/0 (socket says 172.21.15.62:52556) 2024-09-15T23:24:26.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec120c20 con 0x7f9dec11b140 2024-09-15T23:24:26.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.781+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec120f20 con 0x7f9dec109450 2024-09-15T23:24:26.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.782+0000 7f9de9d9b640 1 -- 172.21.15.62:0/461175848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dec109450 msgr2=0x7f9dec1a6f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:24:26.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.782+0000 7f9de9d9b640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dec109450 0x7f9dec1a6f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.782+0000 7f9de9d9b640 1 -- 172.21.15.62:0/461175848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dec11b140 msgr2=0x7f9dec1ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:26.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.782+0000 7f9de9d9b640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dec11b140 0x7f9dec1ab810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.782+0000 7f9de9d9b640 1 -- 172.21.15.62:0/461175848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dec1abef0 con 0x7f9dec109830 2024-09-15T23:24:26.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.782+0000 7f9de9d9b640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dec109830 0x7f9dec1a7480 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00d6d0 tx=0x7f9ddc00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.786+0000 7f9ddb7fe640 1 -- 172.21.15.62:0/461175848 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc017070 con 0x7f9dec109830 2024-09-15T23:24:26.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.786+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dec1ac1e0 con 0x7f9dec109830 2024-09-15T23:24:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.786+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dec1b3ac0 con 0x7f9dec109830 2024-09-15T23:24:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.786+0000 7f9ddb7fe640 1 -- 172.21.15.62:0/461175848 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ddc009d00 con 0x7f9dec109830 2024-09-15T23:24:26.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.786+0000 7f9ddb7fe640 1 -- 172.21.15.62:0/461175848 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc020060 con 0x7f9dec109830 2024-09-15T23:24:26.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.791+0000 7f9df308b740 1 Processor -- start 2024-09-15T23:24:26.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.793+0000 7f9ddb7fe640 1 -- 172.21.15.62:0/461175848 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9ddc005430 con 0x7f9dec109830 2024-09-15T23:24:26.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.793+0000 7f9df308b740 1 -- start start 2024-09-15T23:24:26.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.793+0000 7f9df308b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52d99c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.793+0000 7f9df308b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d9a170 0x560f52da44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9dead9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52d99c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9dead9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52d99c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48764/0 (socket says 172.21.15.62:48764) 2024-09-15T23:24:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9df308b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 0x560f52da6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9dea59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d9a170 0x560f52da44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9df308b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560f52cfe160 con 0x560f52da4a30 2024-09-15T23:24:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9df308b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560f52ca5720 con 0x560f52d99850 2024-09-15T23:24:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9df308b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560f52cfa120 con 0x560f52d9a170 2024-09-15T23:24:26.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9dead9d640 1 -- 172.21.15.62:0/1112901742 learned_addr learned my addr 172.21.15.62:0/1112901742 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:26.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9dead9d640 1 -- 172.21.15.62:0/1112901742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d9a170 msgr2=0x560f52da44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:26.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9dead9d640 1 --2- 172.21.15.62:0/1112901742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d9a170 0x560f52da44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9dead9d640 1 -- 172.21.15.62:0/1112901742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 msgr2=0x560f52da6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:26.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9dead9d640 1 --2- 172.21.15.62:0/1112901742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 0x560f52da6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.796+0000 7f9dead9d640 1 -- 172.21.15.62:0/1112901742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f52a6fba0 con 0x560f52d99850 2024-09-15T23:24:26.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.797+0000 7f9ddb7fe640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9dd40a3300 0x7f9dd40a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.798+0000 7f9ddb7fe640 1 -- 172.21.15.62:0/461175848 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9ddc0cf680 con 0x7f9dec109830 2024-09-15T23:24:26.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.798+0000 7f9dead9d640 1 --2- 172.21.15.62:0/1112901742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52d99c30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009a10 tx=0x7f9de402f310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1d218f68984b20e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.800+0000 7f9dd8ff9640 1 -- 172.21.15.62:0/1112901742 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4031db0 con 0x560f52d99850 2024-09-15T23:24:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.800+0000 7f9dd8ff9640 1 -- 172.21.15.62:0/1112901742 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de4036030 con 0x560f52d99850 2024-09-15T23:24:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.800+0000 7f9dd8ff9640 1 -- 172.21.15.62:0/1112901742 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de40316b0 con 0x560f52d99850 2024-09-15T23:24:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.800+0000 7f9dea59c640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9dd40a3300 0x7f9dd40a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.801+0000 7f9df308b740 1 -- 172.21.15.62:0/1112901742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 msgr2=0x560f52d99c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.801+0000 7f9df308b740 1 --2- 172.21.15.62:0/1112901742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52d99c30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009a10 tx=0x7f9de402f310 comp rx=0 tx=0).stop 2024-09-15T23:24:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.801+0000 7f9df308b740 1 -- 172.21.15.62:0/1112901742 shutdown_connections 2024-09-15T23:24:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.801+0000 7f9df308b740 1 --2- 172.21.15.62:0/1112901742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 0x560f52da6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.801+0000 7f9df308b740 1 --2- 172.21.15.62:0/1112901742 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d9a170 0x560f52da44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.801+0000 7f9df308b740 1 --2- 172.21.15.62:0/1112901742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52d99c30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.801+0000 7f9df308b740 1 -- 172.21.15.62:0/1112901742 >> 172.21.15.62:0/1112901742 conn(0x560f52d95de0 msgr2=0x560f52d961c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:26.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.807+0000 7f9df308b740 1 -- 172.21.15.62:0/1112901742 shutdown_connections 2024-09-15T23:24:26.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.809+0000 7f9dea59c640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9dd40a3300 0x7f9dd40a57c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x560f52d96c40 tx=0x7f9de000a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:26.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.809+0000 7f9df308b740 1 -- 172.21.15.62:0/1112901742 wait complete. 2024-09-15T23:24:26.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.809+0000 7f9df308b740 1 -- start start 2024-09-15T23:24:26.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.809+0000 7f9df308b740 1 Processor -- start 2024-09-15T23:24:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9df308b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d99850 0x560f52da1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9df308b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d9a170 0x560f52da1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9df308b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52da4a30 0x560f52da2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9df308b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560f52da7480 con 0x560f52d99850 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9df308b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560f52c86900 con 0x560f52d9a170 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9df308b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560f52da7600 con 0x560f52da4a30 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9df308b740 10 client.0 _ll_register_callbacks cb 0x560f52c84780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9de9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d9a170 0x560f52da1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9dea59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d99850 0x560f52da1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9de9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d9a170 0x560f52da1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48780/0 (socket says 172.21.15.62:48780) 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9de9d9b640 1 -- 172.21.15.62:0/1769769411 learned_addr learned my addr 172.21.15.62:0/1769769411 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9dead9d640 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52da4a30 0x560f52da2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9de9d9b640 1 -- 172.21.15.62:0/1769769411 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52da4a30 msgr2=0x560f52da2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:26.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9de9d9b640 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52da4a30 0x560f52da2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9de9d9b640 1 -- 172.21.15.62:0/1769769411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d99850 msgr2=0x560f52da1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9de9d9b640 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d99850 0x560f52da1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.810+0000 7f9de9d9b640 1 -- 172.21.15.62:0/1769769411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f52cc6c50 con 0x560f52d9a170 2024-09-15T23:24:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.811+0000 7f9de9d9b640 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d9a170 0x560f52da1ec0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0982f0 tx=0x7f9ddc005860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=555d94260fdda325 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.812+0000 7f9dd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:24:26.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.812+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/1769769411 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc0a4590 con 0x560f52d9a170 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.812+0000 7f9df308b740 1 -- 172.21.15.62:0/1769769411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9de4036030 con 0x560f52d9a170 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.812+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/1769769411 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9ddc0046c0 con 0x560f52d9a170 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/1769769411 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9ddc099c20 con 0x560f52d9a170 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9df308b740 10 client.26906 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9df308b740 1 -- 172.21.15.62:0/1769769411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560f52dc4290 con 0x560f52d9a170 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/1769769411 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de4036030 con 0x560f52d9a170 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/1769769411 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f9ddc0cfb30 con 0x560f52d9a170 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9df308b740 10 client.26906 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9df308b740 10 client.26906 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9df308b740 20 client.26906 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9df308b740 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560f52dc5c70 0x560f52dc80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9df308b740 4 client.26906 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:26.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.813+0000 7f9df308b740 1 -- 172.21.15.62:0/1769769411 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f52d0b890 con 0x560f52dc5c70 2024-09-15T23:24:26.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.816+0000 7f9dea59c640 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560f52dc5c70 0x560f52dc80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.822+0000 7f9dea59c640 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560f52dc5c70 0x560f52dc80d0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccabdb5f1af4762 server_cookie=95f4fa6834f64112 in_seq=0 out_seq=0 2024-09-15T23:24:26.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.822+0000 7f9dd1ffb640 10 client.26906 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:26.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/1769769411 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560f52d0b890 con 0x560f52dc5c70 2024-09-15T23:24:26.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9dd1ffb640 10 client.26906 handle_command_reply: tid=0 2024-09-15T23:24:26.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 client.26906 shutdown 2024-09-15T23:24:26.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 -- 172.21.15.62:0/1769769411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d9a170 msgr2=0x560f52da1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:26.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d9a170 0x560f52da1ec0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0982f0 tx=0x7f9ddc005860 comp rx=0 tx=0).stop 2024-09-15T23:24:26.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 -- 172.21.15.62:0/1769769411 shutdown_connections 2024-09-15T23:24:26.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560f52dc5c70 0x560f52dc80d0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52da4a30 0x560f52da2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d9a170 0x560f52da1ec0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 --2- 172.21.15.62:0/1769769411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d99850 0x560f52da1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 -- 172.21.15.62:0/1769769411 >> 172.21.15.62:0/1769769411 conn(0x560f52d95de0 msgr2=0x560f52d9bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:26.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9dd1ffb640 0 client.26906 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:26.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 -- 172.21.15.62:0/1769769411 shutdown_connections 2024-09-15T23:24:26.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.823+0000 7f9df308b740 1 -- 172.21.15.62:0/1769769411 wait complete. 2024-09-15T23:24:26.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.824+0000 7f9df308b740 20 client.26906 trim_cache size 0 max 16384 2024-09-15T23:24:26.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9df308b740 1 Processor -- start 2024-09-15T23:24:26.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9df308b740 1 -- start start 2024-09-15T23:24:26.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9df308b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d99850 0x560f52da3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9df308b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d9a170 0x560f52da0250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9dea59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d99850 0x560f52da3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9df308b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52da4a30 0x560f52da2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9dea59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d99850 0x560f52da3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36470/0 (socket says 172.21.15.62:36470) 2024-09-15T23:24:26.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9de9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d9a170 0x560f52da0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9df308b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc012500 con 0x560f52d9a170 2024-09-15T23:24:26.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9de9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d9a170 0x560f52da0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52574/0 (socket says 172.21.15.62:52574) 2024-09-15T23:24:26.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9df308b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc00fdd0 con 0x560f52da4a30 2024-09-15T23:24:26.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9df308b740 1 -- 172.21.15.62:0/973918110 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc00dc20 con 0x560f52d99850 2024-09-15T23:24:26.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9de9d9b640 1 -- 172.21.15.62:0/973918110 learned_addr learned my addr 172.21.15.62:0/973918110 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:26.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.855+0000 7f9dead9d640 1 --2- 172.21.15.62:0/973918110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52da4a30 0x560f52da2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.856+0000 7f9de9d9b640 1 -- 172.21.15.62:0/973918110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d99850 msgr2=0x560f52da3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:26.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.856+0000 7f9de9d9b640 1 --2- 172.21.15.62:0/973918110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d99850 0x560f52da3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.856+0000 7f9de9d9b640 1 -- 172.21.15.62:0/973918110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52da4a30 msgr2=0x560f52da2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:26.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.856+0000 7f9de9d9b640 1 --2- 172.21.15.62:0/973918110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52da4a30 0x560f52da2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.856+0000 7f9de9d9b640 1 -- 172.21.15.62:0/973918110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f52d99680 con 0x560f52d9a170 2024-09-15T23:24:26.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.856+0000 7f9de9d9b640 1 --2- 172.21.15.62:0/973918110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d9a170 0x560f52da0250 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00d6a0 tx=0x7f9ddc097cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eca074ee8243d21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:26.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.857+0000 7f9dd3fff640 1 -- 172.21.15.62:0/973918110 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc0a4590 con 0x560f52d9a170 2024-09-15T23:24:26.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.857+0000 7f9dd3fff640 1 -- 172.21.15.62:0/973918110 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ddc004dc0 con 0x560f52d9a170 2024-09-15T23:24:26.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.857+0000 7f9dd3fff640 1 -- 172.21.15.62:0/973918110 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc0cfae0 con 0x560f52d9a170 2024-09-15T23:24:26.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.858+0000 7f9df308b740 1 -- 172.21.15.62:0/973918110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d9a170 msgr2=0x560f52da0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:26.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.858+0000 7f9df308b740 1 --2- 172.21.15.62:0/973918110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d9a170 0x560f52da0250 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00d6a0 tx=0x7f9ddc097cf0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.858+0000 7f9df308b740 1 -- 172.21.15.62:0/973918110 shutdown_connections 2024-09-15T23:24:26.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.858+0000 7f9df308b740 1 --2- 172.21.15.62:0/973918110 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52da4a30 0x560f52da2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.858+0000 7f9df308b740 1 --2- 172.21.15.62:0/973918110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52d9a170 0x560f52da0250 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.858+0000 7f9df308b740 1 --2- 172.21.15.62:0/973918110 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d99850 0x560f52da3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.858+0000 7f9df308b740 1 -- 172.21.15.62:0/973918110 >> 172.21.15.62:0/973918110 conn(0x560f52d95de0 msgr2=0x560f52d94ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:26.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.859+0000 7f9df308b740 1 -- 172.21.15.62:0/973918110 shutdown_connections 2024-09-15T23:24:26.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.859+0000 7f9df308b740 1 -- 172.21.15.62:0/973918110 wait complete. 2024-09-15T23:24:26.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.860+0000 7f9df308b740 1 -- start start 2024-09-15T23:24:26.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.860+0000 7f9df308b740 1 Processor -- start 2024-09-15T23:24:26.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9df308b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52dac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9df308b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d9a170 0x560f52dace20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9df308b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 0x560f52da98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9dea59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52dac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9df308b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc00e290 con 0x560f52da4a30 2024-09-15T23:24:26.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9dea59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52dac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48806/0 (socket says 172.21.15.62:48806) 2024-09-15T23:24:26.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9df308b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc00f700 con 0x560f52d99850 2024-09-15T23:24:26.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9dea59c640 1 -- 172.21.15.62:0/67241685 learned_addr learned my addr 172.21.15.62:0/67241685 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:26.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9dead9d640 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 0x560f52da98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9df308b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc00e960 con 0x560f52d9a170 2024-09-15T23:24:26.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9df308b740 10 client.0 _ll_register_callbacks cb 0x7f9ddc009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:26.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9dead9d640 1 -- 172.21.15.62:0/67241685 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d9a170 msgr2=0x560f52dace20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:24:26.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9dead9d640 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d9a170 0x560f52dace20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9dead9d640 1 -- 172.21.15.62:0/67241685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 msgr2=0x560f52dac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:26.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9dead9d640 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52dac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:26.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.861+0000 7f9dead9d640 1 -- 172.21.15.62:0/67241685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f52d913f0 con 0x560f52da4a30 2024-09-15T23:24:26.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.862+0000 7f9dead9d640 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 0x560f52da98b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009a60 tx=0x7f9de4003c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1bb13fffb8c2e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:26.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.862+0000 7f9dd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:26.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.862+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/67241685 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de4055070 con 0x560f52da4a30 2024-09-15T23:24:26.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.862+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/67241685 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9de4030a00 con 0x560f52da4a30 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.862+0000 7f9df308b740 1 -- 172.21.15.62:0/67241685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ddc004dc0 con 0x560f52da4a30 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.862+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/67241685 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f9de404a040 con 0x560f52da4a30 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.863+0000 7f9df308b740 10 client.17328 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.863+0000 7f9df308b740 1 -- 172.21.15.62:0/67241685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x560f52e30cc0 con 0x560f52da4a30 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.863+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/67241685 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de4036040 con 0x560f52da4a30 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.863+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/67241685 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f9de4043070 con 0x560f52da4a30 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.863+0000 7f9df308b740 10 client.17328 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.863+0000 7f9df308b740 10 client.17328 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.863+0000 7f9df308b740 20 client.17328 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.864+0000 7f9df308b740 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560f52e31120 0x560f52e33510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.864+0000 7f9df308b740 4 client.17328 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}] 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.864+0000 7f9df308b740 1 -- 172.21.15.62:0/67241685 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}) -- 0x7f9ddc004dc0 con 0x560f52e31120 2024-09-15T23:24:26.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.864+0000 7f9dea59c640 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560f52e31120 0x560f52e33510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:26.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.864+0000 7f9dea59c640 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560f52e31120 0x560f52e33510 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58ee850a4308365b server_cookie=696bc0181c738351 in_seq=0 out_seq=0 2024-09-15T23:24:26.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:26.864+0000 7f9dd1ffb640 10 client.17328 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:27.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.226+0000 7f9dd1ffb640 1 -- 172.21.15.62:0/67241685 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f9ddc004dc0 con 0x560f52e31120 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.226+0000 7f9dd1ffb640 10 client.17328 handle_command_reply: tid=0 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.226+0000 7f9df308b740 1 client.17328 shutdown 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 -- 172.21.15.62:0/67241685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 msgr2=0x560f52da98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 0x560f52da98b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009a60 tx=0x7f9de4003c60 comp rx=0 tx=0).stop 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 -- 172.21.15.62:0/67241685 shutdown_connections 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560f52e31120 0x560f52e33510 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f52da4a30 0x560f52da98b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9dd1ffb640 0 client.17328 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560f52d9a170 0x560f52dace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 --2- 172.21.15.62:0/67241685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560f52d99850 0x560f52dac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 -- 172.21.15.62:0/67241685 >> 172.21.15.62:0/67241685 conn(0x560f52d95de0 msgr2=0x560f52d9b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 -- 172.21.15.62:0/67241685 shutdown_connections 2024-09-15T23:24:27.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 1 -- 172.21.15.62:0/67241685 wait complete. 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.227+0000 7f9df308b740 20 client.17328 trim_cache size 0 max 16384 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 34, 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 30, 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stdout: "83c3dbcf": { 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 30, 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:24:27.230 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 142.3, 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:27.231 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:27.232 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:27.232 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:24:27.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9dd40a3300 msgr2=0x7f9dd40a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:27.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9dd40a3300 0x7f9dd40a57c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x560f52d96c40 tx=0x7f9de000a250 comp rx=0 tx=0).stop 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dec109830 msgr2=0x7f9dec1a7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dec109830 0x7f9dec1a7480 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00d6d0 tx=0x7f9ddc00db90 comp rx=0 tx=0).stop 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 shutdown_connections 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9dd40a3300 0x7f9dd40a57c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dec11b140 0x7f9dec1ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dec109830 0x7f9dec1a7480 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 --2- 172.21.15.62:0/461175848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9dec109450 0x7f9dec1a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 >> 172.21.15.62:0/461175848 conn(0x7f9dec06e140 msgr2=0x7f9dec11bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 shutdown_connections 2024-09-15T23:24:27.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:27.228+0000 7f9df0cb2640 1 -- 172.21.15.62:0/461175848 wait complete. 2024-09-15T23:24:27.968 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:27 smithi062 ceph-mon[30949]: pgmap v615: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 5.3 MiB/s wr, 554 op/s 2024-09-15T23:24:27.998 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '83c3dbcf', quiesce took 4.8 seconds. Will release after: 17.55 2024-09-15T23:24:28.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:27 smithi067 ceph-mon[35665]: pgmap v615: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 5.3 MiB/s wr, 554 op/s 2024-09-15T23:24:28.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:27 smithi148 ceph-mon[35610]: pgmap v615: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 5.3 MiB/s wr, 554 op/s 2024-09-15T23:24:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.757+0000 7f1480fcf640 1 -- 172.21.15.62:0/2623473847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c100f80 msgr2=0x7f147c101380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.757+0000 7f1480fcf640 1 --2- 172.21.15.62:0/2623473847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c100f80 0x7f147c101380 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f146c009900 tx=0x7f146c02a480 comp rx=0 tx=0).stop 2024-09-15T23:24:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.757+0000 7f1480fcf640 1 -- 172.21.15.62:0/2623473847 shutdown_connections 2024-09-15T23:24:28.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.757+0000 7f1480fcf640 1 --2- 172.21.15.62:0/2623473847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f147c102650 0x7f147c108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.757+0000 7f1480fcf640 1 --2- 172.21.15.62:0/2623473847 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f147c101c90 0x7f147c102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.757+0000 7f1480fcf640 1 --2- 172.21.15.62:0/2623473847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c100f80 0x7f147c101380 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.757+0000 7f1480fcf640 1 -- 172.21.15.62:0/2623473847 >> 172.21.15.62:0/2623473847 conn(0x7f147c0fb980 msgr2=0x7f147c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.757+0000 7f1480fcf640 1 -- 172.21.15.62:0/2623473847 shutdown_connections 2024-09-15T23:24:28.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.757+0000 7f1480fcf640 1 -- 172.21.15.62:0/2623473847 wait complete. 2024-09-15T23:24:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1480fcf640 1 Processor -- start 2024-09-15T23:24:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1480fcf640 1 -- start start 2024-09-15T23:24:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1480fcf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f147c100f80 0x7f147c10b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1480fcf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f147c101c90 0x7f147c109a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1480fcf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c102650 0x7f147c109fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1480fcf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f147c10d800 con 0x7f147c102650 2024-09-15T23:24:28.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1480fcf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f147c10d680 con 0x7f147c101c90 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1480fcf640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f147c10d980 con 0x7f147c100f80 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f147ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c102650 0x7f147c109fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f147ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c102650 0x7f147c109fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52600/0 (socket says 172.21.15.62:52600) 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f147ad9d640 1 -- 172.21.15.62:0/3981872334 learned_addr learned my addr 172.21.15.62:0/3981872334 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f147a59c640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f147c100f80 0x7f147c10b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1479d9b640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f147c101c90 0x7f147c109a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f147ad9d640 1 -- 172.21.15.62:0/3981872334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f147c100f80 msgr2=0x7f147c10b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f147ad9d640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f147c100f80 0x7f147c10b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f147ad9d640 1 -- 172.21.15.62:0/3981872334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f147c101c90 msgr2=0x7f147c109a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f147ad9d640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f147c101c90 0x7f147c109a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.763 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f147ad9d640 1 -- 172.21.15.62:0/3981872334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147c10a830 con 0x7f147c102650 2024-09-15T23:24:28.763 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.758+0000 7f1479d9b640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f147c101c90 0x7f147c109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:28.763 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.759+0000 7f147ad9d640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c102650 0x7f147c109fa0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f147400eaa0 tx=0x7f147400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.763 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.759+0000 7f146b7fe640 1 -- 172.21.15.62:0/3981872334 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147400cdb0 con 0x7f147c102650 2024-09-15T23:24:28.763 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.759+0000 7f146b7fe640 1 -- 172.21.15.62:0/3981872334 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1474005020 con 0x7f147c102650 2024-09-15T23:24:28.763 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.759+0000 7f1480fcf640 1 -- 172.21.15.62:0/3981872334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f147c1a1e90 con 0x7f147c102650 2024-09-15T23:24:28.763 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.759+0000 7f146b7fe640 1 -- 172.21.15.62:0/3981872334 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14740116b0 con 0x7f147c102650 2024-09-15T23:24:28.763 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.759+0000 7f1480fcf640 1 -- 172.21.15.62:0/3981872334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f147c1a22a0 con 0x7f147c102650 2024-09-15T23:24:28.768 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f146b7fe640 1 -- 172.21.15.62:0/3981872334 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1474005560 con 0x7f147c102650 2024-09-15T23:24:28.768 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1483e45740 1 Processor -- start 2024-09-15T23:24:28.768 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1483e45740 1 -- start start 2024-09-15T23:24:28.768 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1483e45740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0c0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.768 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1483e45740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 0x555ecc0c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.768 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1483e45740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0c8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1483e45740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555ecc0242f0 con 0x555ecc0c09a0 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1483e45740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555ecbfcc270 con 0x555ecc0c12e0 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1483e45740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555ecc0202b0 con 0x555ecc0c6370 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1479d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0c0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1479d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0c0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52612/0 (socket says 172.21.15.62:52612) 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.762+0000 7f1479d9b640 1 -- 172.21.15.62:0/1593402489 learned_addr learned my addr 172.21.15.62:0/1593402489 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f147a59c640 1 --2- 172.21.15.62:0/1593402489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 0x555ecc0c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1479d9b640 1 --2- 172.21.15.62:0/1593402489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f146b7fe640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14640a32f0 0x7f14640a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1479d9b640 1 -- 172.21.15.62:0/1593402489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 msgr2=0x555ecc0c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f147ad9d640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14640a32f0 0x7f14640a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1479d9b640 1 --2- 172.21.15.62:0/1593402489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1479d9b640 1 -- 172.21.15.62:0/1593402489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 msgr2=0x555ecc0c5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1479d9b640 1 --2- 172.21.15.62:0/1593402489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 0x555ecc0c5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1479d9b640 1 -- 172.21.15.62:0/1593402489 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ecbfcb340 con 0x555ecc0c09a0 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f146b7fe640 1 -- 172.21.15.62:0/3981872334 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f14740cff00 con 0x7f147c102650 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1479d9b640 1 --2- 172.21.15.62:0/1593402489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0c0da0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1470012900 tx=0x7f1470008f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29b40a672f95c8f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1468ff9640 1 -- 172.21.15.62:0/1593402489 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470010b80 con 0x555ecc0c09a0 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1468ff9640 1 -- 172.21.15.62:0/1593402489 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1470010d20 con 0x555ecc0c09a0 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.763+0000 7f1468ff9640 1 -- 172.21.15.62:0/1593402489 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470005360 con 0x555ecc0c09a0 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f1483e45740 1 -- 172.21.15.62:0/1593402489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 msgr2=0x555ecc0c0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f147ad9d640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14640a32f0 0x7f14640a57b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1474098600 tx=0x7f14740bc040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f1483e45740 1 --2- 172.21.15.62:0/1593402489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0c0da0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1470012900 tx=0x7f1470008f90 comp rx=0 tx=0).stop 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f1483e45740 1 -- 172.21.15.62:0/1593402489 shutdown_connections 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f1483e45740 1 --2- 172.21.15.62:0/1593402489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.770 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f1483e45740 1 --2- 172.21.15.62:0/1593402489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 0x555ecc0c5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f1483e45740 1 --2- 172.21.15.62:0/1593402489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0c0da0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f1483e45740 1 -- 172.21.15.62:0/1593402489 >> 172.21.15.62:0/1593402489 conn(0x555ecc0bc010 msgr2=0x555ecc0bc410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f1483e45740 1 -- 172.21.15.62:0/1593402489 shutdown_connections 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.764+0000 7f1483e45740 1 -- 172.21.15.62:0/1593402489 wait complete. 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1483e45740 1 -- start start 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1483e45740 1 Processor -- start 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1483e45740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1483e45740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 0x555ecc0cb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1483e45740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0d2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1483e45740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555ecc0c9340 con 0x555ecc0c09a0 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1483e45740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555ecbfad290 con 0x555ecc0c12e0 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1483e45740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555ecc0c94c0 con 0x555ecc0c6370 2024-09-15T23:24:28.771 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1483e45740 10 client.0 _ll_register_callbacks cb 0x555ecbfde140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1479d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1479d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52628/0 (socket says 172.21.15.62:52628) 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1479d9b640 1 -- 172.21.15.62:0/1665490273 learned_addr learned my addr 172.21.15.62:0/1665490273 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f147a59c640 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 0x555ecc0cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1479d9b640 1 -- 172.21.15.62:0/1665490273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 msgr2=0x555ecc0d2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1479d9b640 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0d2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1479d9b640 1 -- 172.21.15.62:0/1665490273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 msgr2=0x555ecc0cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1479d9b640 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 0x555ecc0cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.765+0000 7f1479d9b640 1 -- 172.21.15.62:0/1665490273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ecbd37210 con 0x555ecc0c09a0 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.766+0000 7f1479d9b640 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cab00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1470008970 tx=0x7f1470012e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8f7c8ae4378c4a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.766+0000 7f1461ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.766+0000 7f1461ffb640 1 -- 172.21.15.62:0/1665490273 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470012570 con 0x555ecc0c09a0 2024-09-15T23:24:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.766+0000 7f1483e45740 1 -- 172.21.15.62:0/1665490273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1470010d20 con 0x555ecc0c09a0 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.766+0000 7f1461ffb640 1 -- 172.21.15.62:0/1665490273 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1470005100 con 0x555ecc0c09a0 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.766+0000 7f1461ffb640 1 -- 172.21.15.62:0/1665490273 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f1470034050 con 0x555ecc0c09a0 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.766+0000 7f1483e45740 10 client.17340 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.766+0000 7f1483e45740 1 -- 172.21.15.62:0/1665490273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555ecbe03620 con 0x555ecc0c09a0 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.766+0000 7f1461ffb640 1 -- 172.21.15.62:0/1665490273 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1470010d20 con 0x555ecc0c09a0 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f1461ffb640 1 -- 172.21.15.62:0/1665490273 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f147001eb80 con 0x555ecc0c09a0 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f1483e45740 10 client.17340 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f1483e45740 10 client.17340 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f1483e45740 20 client.17340 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f1483e45740 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555ecc0d6560 0x555ecc0d89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f1483e45740 4 client.17340 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f1483e45740 1 -- 172.21.15.62:0/1665490273 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ecc031a20 con 0x555ecc0d6560 2024-09-15T23:24:28.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f147a59c640 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555ecc0d6560 0x555ecc0d89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f1461ffb640 10 client.17340 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:28.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.767+0000 7f147a59c640 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555ecc0d6560 0x555ecc0d89a0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed21fd1ef05dcd56 server_cookie=96472f615dd4fc60 in_seq=0 out_seq=0 2024-09-15T23:24:28.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1461ffb640 1 -- 172.21.15.62:0/1665490273 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555ecc031a20 con 0x555ecc0d6560 2024-09-15T23:24:28.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1461ffb640 10 client.17340 handle_command_reply: tid=0 2024-09-15T23:24:28.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 client.17340 shutdown 2024-09-15T23:24:28.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 -- 172.21.15.62:0/1665490273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 msgr2=0x555ecc0cab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cab00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f1470008970 tx=0x7f1470012e80 comp rx=0 tx=0).stop 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 -- 172.21.15.62:0/1665490273 shutdown_connections 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555ecc0d6560 0x555ecc0d89a0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0d2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c12e0 0x555ecc0cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 --2- 172.21.15.62:0/1665490273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cab00 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1461ffb640 0 client.17340 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 -- 172.21.15.62:0/1665490273 >> 172.21.15.62:0/1665490273 conn(0x555ecc0bc010 msgr2=0x555ecc0bd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.768+0000 7f1483e45740 1 -- 172.21.15.62:0/1665490273 shutdown_connections 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.769+0000 7f1483e45740 1 -- 172.21.15.62:0/1665490273 wait complete. 2024-09-15T23:24:28.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.769+0000 7f1483e45740 20 client.17340 trim_cache size 0 max 16384 2024-09-15T23:24:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1483e45740 1 Processor -- start 2024-09-15T23:24:28.785 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1483e45740 1 -- start start 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1483e45740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1483e45740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c12e0 0x555ecc0d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1483e45740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c6370 0x555ecc0ba9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1483e45740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f147c1af090 con 0x555ecc0c09a0 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1483e45740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f147c1ac960 con 0x555ecc0c6370 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1483e45740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f147c1aa7b0 con 0x555ecc0c12e0 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1479d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1479d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52636/0 (socket says 172.21.15.62:52636) 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.783+0000 7f1479d9b640 1 -- 172.21.15.62:0/2310784508 learned_addr learned my addr 172.21.15.62:0/2310784508 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f147a59c640 1 --2- 172.21.15.62:0/2310784508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c12e0 0x555ecc0d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f1479d9b640 1 -- 172.21.15.62:0/2310784508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c12e0 msgr2=0x555ecc0d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f1479d9b640 1 --2- 172.21.15.62:0/2310784508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c12e0 0x555ecc0d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f1479d9b640 1 -- 172.21.15.62:0/2310784508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c6370 msgr2=0x555ecc0ba9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f1479d9b640 1 --2- 172.21.15.62:0/2310784508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c6370 0x555ecc0ba9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f1479d9b640 1 -- 172.21.15.62:0/2310784508 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ecc0c9220 con 0x555ecc0c09a0 2024-09-15T23:24:28.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f1479d9b640 1 --2- 172.21.15.62:0/2310784508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cafc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f14700297d0 tx=0x7f1470008fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b91602ddc3ebbd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f1463fff640 1 -- 172.21.15.62:0/2310784508 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470004d50 con 0x555ecc0c09a0 2024-09-15T23:24:28.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f1463fff640 1 -- 172.21.15.62:0/2310784508 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1470025070 con 0x555ecc0c09a0 2024-09-15T23:24:28.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.784+0000 7f1463fff640 1 -- 172.21.15.62:0/2310784508 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470005000 con 0x555ecc0c09a0 2024-09-15T23:24:28.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 -- 172.21.15.62:0/2310784508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 msgr2=0x555ecc0cafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.790 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 --2- 172.21.15.62:0/2310784508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cafc0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f14700297d0 tx=0x7f1470008fa0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 -- 172.21.15.62:0/2310784508 shutdown_connections 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 --2- 172.21.15.62:0/2310784508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c6370 0x555ecc0ba9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 --2- 172.21.15.62:0/2310784508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c12e0 0x555ecc0d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 --2- 172.21.15.62:0/2310784508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c09a0 0x555ecc0cafc0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 -- 172.21.15.62:0/2310784508 >> 172.21.15.62:0/2310784508 conn(0x555ecc0bc010 msgr2=0x555ecc0ca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 -- 172.21.15.62:0/2310784508 shutdown_connections 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 -- 172.21.15.62:0/2310784508 wait complete. 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 -- start start 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 Processor -- start 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c09a0 0x555ecc0cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c12e0 0x555ecc0cf770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0d0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f147c1aae20 con 0x555ecc0c12e0 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f147c1ac290 con 0x555ecc0c09a0 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f147c1ab4f0 con 0x555ecc0c6370 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f1483e45740 10 client.0 _ll_register_callbacks cb 0x7f146c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f147a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c12e0 0x555ecc0cf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f147a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c12e0 0x555ecc0cf770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52652/0 (socket says 172.21.15.62:52652) 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.785+0000 7f147a59c640 1 -- 172.21.15.62:0/765688531 learned_addr learned my addr 172.21.15.62:0/765688531 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1479d9b640 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1479d9b640 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c09a0 0x555ecc0cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f147a59c640 1 -- 172.21.15.62:0/765688531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 msgr2=0x555ecc0d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f147a59c640 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0d0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f147a59c640 1 -- 172.21.15.62:0/765688531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c09a0 msgr2=0x555ecc0cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f147a59c640 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c09a0 0x555ecc0cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f147a59c640 1 -- 172.21.15.62:0/765688531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ecc0d0d40 con 0x555ecc0c12e0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1479d9b640 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c09a0 0x555ecc0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f147a59c640 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c12e0 0x555ecc0cf770 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f146c0314e0 tx=0x7f146c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f411a1e392f37f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1461ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1461ffb640 1 -- 172.21.15.62:0/765688531 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c04a070 con 0x555ecc0c12e0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1461ffb640 1 -- 172.21.15.62:0/765688531 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f146c005040 con 0x555ecc0c12e0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1483e45740 1 -- 172.21.15.62:0/765688531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1470025070 con 0x555ecc0c12e0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1461ffb640 1 -- 172.21.15.62:0/765688531 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f146c053050 con 0x555ecc0c12e0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1483e45740 10 client.17346 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.786+0000 7f1483e45740 1 -- 172.21.15.62:0/765688531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x555ecc0d2120 con 0x555ecc0c12e0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1461ffb640 1 -- 172.21.15.62:0/765688531 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1470025070 con 0x555ecc0c12e0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1461ffb640 1 -- 172.21.15.62:0/765688531 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f146c04d020 con 0x555ecc0c12e0 2024-09-15T23:24:28.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1483e45740 10 client.17346 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1483e45740 10 client.17346 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1483e45740 20 client.17346 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1483e45740 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555ecc1416d0 0x555ecc143af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1483e45740 4 client.17346 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:24:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1483e45740 1 -- 172.21.15.62:0/765688531 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555ecc02bd70 con 0x555ecc1416d0 2024-09-15T23:24:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1479d9b640 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555ecc1416d0 0x555ecc143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1461ffb640 10 client.17346 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:28.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.787+0000 7f1479d9b640 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555ecc1416d0 0x555ecc143af0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=525b364ffb29fe33 server_cookie=a8ef10d8aabc628e in_seq=0 out_seq=0 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1461ffb640 1 -- 172.21.15.62:0/765688531 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555ecc02bd70 con 0x555ecc1416d0 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1461ffb640 10 client.17346 handle_command_reply: tid=0 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 client.17346 shutdown 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 -- 172.21.15.62:0/765688531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c12e0 msgr2=0x555ecc0cf770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c12e0 0x555ecc0cf770 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f146c0314e0 tx=0x7f146c005c00 comp rx=0 tx=0).stop 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 -- 172.21.15.62:0/765688531 shutdown_connections 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555ecc1416d0 0x555ecc143af0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555ecc0c6370 0x555ecc0d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555ecc0c12e0 0x555ecc0cf770 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1461ffb640 0 client.17346 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 --2- 172.21.15.62:0/765688531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555ecc0c09a0 0x555ecc0cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 -- 172.21.15.62:0/765688531 >> 172.21.15.62:0/765688531 conn(0x555ecc0bc010 msgr2=0x555ecc0bcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 -- 172.21.15.62:0/765688531 shutdown_connections 2024-09-15T23:24:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 1 -- 172.21.15.62:0/765688531 wait complete. 2024-09-15T23:24:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.800+0000 7f1483e45740 20 client.17346 trim_cache size 0 max 16384 2024-09-15T23:24:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:24:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:24:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:24:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:24:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 -- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14640a32f0 msgr2=0x7f14640a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14640a32f0 0x7f14640a57b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1474098600 tx=0x7f14740bc040 comp rx=0 tx=0).stop 2024-09-15T23:24:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 -- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c102650 msgr2=0x7f147c109fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c102650 0x7f147c109fa0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f147400eaa0 tx=0x7f147400ef60 comp rx=0 tx=0).stop 2024-09-15T23:24:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 -- 172.21.15.62:0/3981872334 shutdown_connections 2024-09-15T23:24:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f14640a32f0 0x7f14640a57b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f147c102650 0x7f147c109fa0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f147c101c90 0x7f147c109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 --2- 172.21.15.62:0/3981872334 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f147c100f80 0x7f147c10b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 -- 172.21.15.62:0/3981872334 >> 172.21.15.62:0/3981872334 conn(0x7f147c0fb980 msgr2=0x7f147c104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 -- 172.21.15.62:0/3981872334 shutdown_connections 2024-09-15T23:24:28.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:28.801+0000 7f1480fcf640 1 -- 172.21.15.62:0/3981872334 wait complete. 2024-09-15T23:24:29.572 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 490.0 s, 370.0 tps, lat 147.614 ms stddev 173.186 2024-09-15T23:24:29.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:29 smithi062 ceph-mon[30949]: pgmap v616: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.7 MiB/s wr, 613 op/s 2024-09-15T23:24:30.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:29 smithi067 ceph-mon[35665]: pgmap v616: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.7 MiB/s wr, 613 op/s 2024-09-15T23:24:30.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:29 smithi148 ceph-mon[35610]: pgmap v616: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 5.7 MiB/s wr, 613 op/s 2024-09-15T23:24:31.272 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:24:31.565 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:31.946 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:31 smithi062 ceph-mon[30949]: pgmap v617: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 4.5 MiB/s wr, 511 op/s 2024-09-15T23:24:32.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:31 smithi067 ceph-mon[35665]: pgmap v617: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 4.5 MiB/s wr, 511 op/s 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 -- 172.21.15.62:0/378154192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 msgr2=0x7f60ac1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 --2- 172.21.15.62:0/378154192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 0x7f60ac1029b0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009900 tx=0x7f60a002a480 comp rx=0 tx=0).stop 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 -- 172.21.15.62:0/378154192 shutdown_connections 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 --2- 172.21.15.62:0/378154192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ac102ef0 0x7f60ac10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 --2- 172.21.15.62:0/378154192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 0x7f60ac1029b0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 --2- 172.21.15.62:0/378154192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ac100e20 0x7f60ac101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 -- 172.21.15.62:0/378154192 >> 172.21.15.62:0/378154192 conn(0x7f60ac0fc7d0 msgr2=0x7f60ac0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 -- 172.21.15.62:0/378154192 shutdown_connections 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 -- 172.21.15.62:0/378154192 wait complete. 2024-09-15T23:24:32.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.029+0000 7f60b14e1640 1 Processor -- start 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60b14e1640 1 -- start start 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60b14e1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ac100e20 0x7f60ac1984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60b14e1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 0x7f60ac1989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60b14e1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ac102ef0 0x7f60ac19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60b14e1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac110f80 con 0x7f60ac102570 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60b14e1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac110e00 con 0x7f60ac100e20 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60b14e1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac111100 con 0x7f60ac102ef0 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 0x7f60ac1989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 0x7f60ac1989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52666/0 (socket says 172.21.15.62:52666) 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60aa7fc640 1 -- 172.21.15.62:0/1233411130 learned_addr learned my addr 172.21.15.62:0/1233411130 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60aaffd640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ac100e20 0x7f60ac1984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60ab7fe640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ac102ef0 0x7f60ac19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60aa7fc640 1 -- 172.21.15.62:0/1233411130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ac102ef0 msgr2=0x7f60ac19cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ac102ef0 0x7f60ac19cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60aa7fc640 1 -- 172.21.15.62:0/1233411130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ac100e20 msgr2=0x7f60ac1984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ac100e20 0x7f60ac1984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.030+0000 7f60aa7fc640 1 -- 172.21.15.62:0/1233411130 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ac19d460 con 0x7f60ac102570 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.031+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 0x7f60ac1989f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f60a0006010 tx=0x7f60a0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.031+0000 7f6093fff640 1 -- 172.21.15.62:0/1233411130 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a003f070 con 0x7f60ac102570 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.031+0000 7f60b14e1640 1 -- 172.21.15.62:0/1233411130 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ac19d6f0 con 0x7f60ac102570 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.031+0000 7f6093fff640 1 -- 172.21.15.62:0/1233411130 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60a0031c30 con 0x7f60ac102570 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.031+0000 7f6093fff640 1 -- 172.21.15.62:0/1233411130 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0035030 con 0x7f60ac102570 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.031+0000 7f60b14e1640 1 -- 172.21.15.62:0/1233411130 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ac1a5030 con 0x7f60ac102570 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.033+0000 7f6093fff640 1 -- 172.21.15.62:0/1233411130 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f60a0042070 con 0x7f60ac102570 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60b38ba740 1 Processor -- start 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60b38ba740 1 -- start start 2024-09-15T23:24:32.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaeb3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaec2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaec4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60b38ba740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5589eae192e0 con 0x5589eaec2a90 2024-09-15T23:24:32.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60b38ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5589eadc0ce0 con 0x5589eaeb3cc0 2024-09-15T23:24:32.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60b38ba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5589eae152a0 con 0x5589eaeb33a0 2024-09-15T23:24:32.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaec4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f6093fff640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f60940a3240 0x7f60940a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaec4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52678/0 (socket says 172.21.15.62:52678) 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60aa7fc640 1 -- 172.21.15.62:0/3004229193 learned_addr learned my addr 172.21.15.62:0/3004229193 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60aaffd640 1 --2- 172.21.15.62:0/3004229193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaec2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60ab7fe640 1 --2- 172.21.15.62:0/3004229193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaeb3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f60ab7fe640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f60940a3240 0x7f60940a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.034+0000 7f6093fff640 1 -- 172.21.15.62:0/1233411130 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f60a00f02e0 con 0x7f60ac102570 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60aa7fc640 1 -- 172.21.15.62:0/3004229193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 msgr2=0x5589eaeb3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/3004229193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaeb3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60aa7fc640 1 -- 172.21.15.62:0/3004229193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 msgr2=0x5589eaec2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/3004229193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaec2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60aa7fc640 1 -- 172.21.15.62:0/3004229193 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589eada5440 con 0x5589eaec2a90 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/3004229193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaec4e80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f60a00bc5c0 tx=0x7f60a002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4528d5a4e3223c96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60ab7fe640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f60940a3240 0x7f60940a5700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f60a4005c40 tx=0x7f60a4009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:32.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60917fa640 1 -- 172.21.15.62:0/3004229193 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0049030 con 0x5589eaec2a90 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60917fa640 1 -- 172.21.15.62:0/3004229193 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60a00f0130 con 0x5589eaec2a90 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.035+0000 7f60917fa640 1 -- 172.21.15.62:0/3004229193 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a00ba760 con 0x5589eaec2a90 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 -- 172.21.15.62:0/3004229193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 msgr2=0x5589eaec4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 --2- 172.21.15.62:0/3004229193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaec4e80 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f60a00bc5c0 tx=0x7f60a002a480 comp rx=0 tx=0).stop 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 -- 172.21.15.62:0/3004229193 shutdown_connections 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 --2- 172.21.15.62:0/3004229193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaec4e80 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 --2- 172.21.15.62:0/3004229193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaec2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 --2- 172.21.15.62:0/3004229193 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaeb3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 -- 172.21.15.62:0/3004229193 >> 172.21.15.62:0/3004229193 conn(0x5589eaeafa30 msgr2=0x5589eaeafe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 -- 172.21.15.62:0/3004229193 shutdown_connections 2024-09-15T23:24:32.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 -- 172.21.15.62:0/3004229193 wait complete. 2024-09-15T23:24:32.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 -- start start 2024-09-15T23:24:32.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.036+0000 7f60b38ba740 1 Processor -- start 2024-09-15T23:24:32.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaec24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaebf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaebfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60b38ba740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5589eaec5480 con 0x5589eaec2a90 2024-09-15T23:24:32.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60b38ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5589eadd4050 con 0x5589eaeb3cc0 2024-09-15T23:24:32.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60b38ba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5589eaec5600 con 0x5589eaeb33a0 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60b38ba740 10 client.0 _ll_register_callbacks cb 0x5589eae14160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaebfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60ab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaebfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52680/0 (socket says 172.21.15.62:52680) 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60ab7fe640 1 -- 172.21.15.62:0/123853317 learned_addr learned my addr 172.21.15.62:0/123853317 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaebf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60aaffd640 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaec24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60aa7fc640 1 -- 172.21.15.62:0/123853317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 msgr2=0x5589eaec24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaec24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60aa7fc640 1 -- 172.21.15.62:0/123853317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 msgr2=0x5589eaebfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaebfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60aa7fc640 1 -- 172.21.15.62:0/123853317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589eab8a3c0 con 0x5589eaeb3cc0 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.037+0000 7f60ab7fe640 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaebfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaebf750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f60a00bc5c0 tx=0x7f60a00b8280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54e255ed5b66bcd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:32.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f608eff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f608eff5640 1 -- 172.21.15.62:0/123853317 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a00d4020 con 0x5589eaeb3cc0 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60b38ba740 1 -- 172.21.15.62:0/123853317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60a00f0130 con 0x5589eaeb3cc0 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f608eff5640 1 -- 172.21.15.62:0/123853317 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f60a00b8ca0 con 0x5589eaeb3cc0 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f608eff5640 1 -- 172.21.15.62:0/123853317 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f60a0049050 con 0x5589eaeb3cc0 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60b38ba740 10 client.26936 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60b38ba740 1 -- 172.21.15.62:0/123853317 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5589eaec5480 con 0x5589eaeb3cc0 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f608eff5640 1 -- 172.21.15.62:0/123853317 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60a00f0130 con 0x5589eaeb3cc0 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f608eff5640 1 -- 172.21.15.62:0/123853317 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f60a00c4df0 con 0x5589eaeb3cc0 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60b38ba740 10 client.26936 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60b38ba740 10 client.26936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60b38ba740 20 client.26936 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60b38ba740 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589eaed2080 0x5589eaed4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60b38ba740 4 client.26936 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:32.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.038+0000 7f60b38ba740 1 -- 172.21.15.62:0/123853317 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589eae26850 con 0x5589eaed2080 2024-09-15T23:24:32.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.039+0000 7f60aaffd640 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589eaed2080 0x5589eaed4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.049 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:31 smithi148 ceph-mon[35610]: pgmap v617: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 4.5 MiB/s wr, 511 op/s 2024-09-15T23:24:32.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.039+0000 7f60aaffd640 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589eaed2080 0x5589eaed4470 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e69053c280987c5c server_cookie=cf3ea771f7cf2206 in_seq=0 out_seq=0 2024-09-15T23:24:32.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.039+0000 7f608eff5640 10 client.26936 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:32.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f608eff5640 1 -- 172.21.15.62:0/123853317 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5589eae26850 con 0x5589eaed2080 2024-09-15T23:24:32.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f608eff5640 10 client.26936 handle_command_reply: tid=0 2024-09-15T23:24:32.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 client.26936 shutdown 2024-09-15T23:24:32.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 -- 172.21.15.62:0/123853317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 msgr2=0x5589eaebf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaebf750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f60a00bc5c0 tx=0x7f60a00b8280 comp rx=0 tx=0).stop 2024-09-15T23:24:32.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 -- 172.21.15.62:0/123853317 shutdown_connections 2024-09-15T23:24:32.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589eaed2080 0x5589eaed4470 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaec2a90 0x5589eaebfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaebf750 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 --2- 172.21.15.62:0/123853317 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaec24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f608eff5640 0 client.26936 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:32.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 -- 172.21.15.62:0/123853317 >> 172.21.15.62:0/123853317 conn(0x5589eaeafa30 msgr2=0x5589eaeb5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:32.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 -- 172.21.15.62:0/123853317 shutdown_connections 2024-09-15T23:24:32.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 1 -- 172.21.15.62:0/123853317 wait complete. 2024-09-15T23:24:32.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.040+0000 7f60b38ba740 20 client.26936 trim_cache size 0 max 16384 2024-09-15T23:24:32.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.057+0000 7f60b38ba740 1 Processor -- start 2024-09-15T23:24:32.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.057+0000 7f60b38ba740 1 -- start start 2024-09-15T23:24:32.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.057+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb33a0 0x5589eaece650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.057+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaed9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.057+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaec2a90 0x5589eaeda260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.057+0000 7f60b38ba740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1af250 con 0x5589eaeb33a0 2024-09-15T23:24:32.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.057+0000 7f60b38ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1acb20 con 0x5589eaeb3cc0 2024-09-15T23:24:32.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.057+0000 7f60b38ba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1aa970 con 0x5589eaec2a90 2024-09-15T23:24:32.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb33a0 0x5589eaece650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb33a0 0x5589eaece650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52690/0 (socket says 172.21.15.62:52690) 2024-09-15T23:24:32.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aaffd640 1 -- 172.21.15.62:0/1024111724 learned_addr learned my addr 172.21.15.62:0/1024111724 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:32.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaed9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60ab7fe640 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaec2a90 0x5589eaeda260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aaffd640 1 -- 172.21.15.62:0/1024111724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaec2a90 msgr2=0x5589eaeda260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aaffd640 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaec2a90 0x5589eaeda260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aaffd640 1 -- 172.21.15.62:0/1024111724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 msgr2=0x5589eaed9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aaffd640 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaed9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aaffd640 1 -- 172.21.15.62:0/1024111724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589eaeb3ab0 con 0x5589eaeb33a0 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaed9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60ab7fe640 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaec2a90 0x5589eaeda260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f60aaffd640 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb33a0 0x5589eaece650 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x5589eaed4cc0 tx=0x7f609c00a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9145af69a6ea118d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f6090ff9640 1 -- 172.21.15.62:0/1024111724 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c005220 con 0x5589eaeb33a0 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f6090ff9640 1 -- 172.21.15.62:0/1024111724 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f609c004c80 con 0x5589eaeb33a0 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.058+0000 7f6090ff9640 1 -- 172.21.15.62:0/1024111724 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c0269d0 con 0x5589eaeb33a0 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 -- 172.21.15.62:0/1024111724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb33a0 msgr2=0x5589eaece650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb33a0 0x5589eaece650 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x5589eaed4cc0 tx=0x7f609c00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 -- 172.21.15.62:0/1024111724 shutdown_connections 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaec2a90 0x5589eaeda260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaeb3cc0 0x5589eaed9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1024111724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb33a0 0x5589eaece650 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 -- 172.21.15.62:0/1024111724 >> 172.21.15.62:0/1024111724 conn(0x5589eaeafa30 msgr2=0x5589eaebad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 -- 172.21.15.62:0/1024111724 shutdown_connections 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 -- 172.21.15.62:0/1024111724 wait complete. 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 -- start start 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.059+0000 7f60b38ba740 1 Processor -- start 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaec5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb3cc0 0x5589eaec6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60b38ba740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaec2a90 0x5589eaf34cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60b38ba740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1aafe0 con 0x5589eaeb3cc0 2024-09-15T23:24:32.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60b38ba740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1ac450 con 0x5589eaec2a90 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60b38ba740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f60ac1ab6b0 con 0x5589eaeb33a0 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60b38ba740 10 client.0 _ll_register_callbacks cb 0x7f609c00a590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb3cc0 0x5589eaec6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aa7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb3cc0 0x5589eaec6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52692/0 (socket says 172.21.15.62:52692) 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aa7fc640 1 -- 172.21.15.62:0/1332895261 learned_addr learned my addr 172.21.15.62:0/1332895261 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60ab7fe640 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaec2a90 0x5589eaf34cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aaffd640 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaec5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aa7fc640 1 -- 172.21.15.62:0/1332895261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 msgr2=0x5589eaec5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaec5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aa7fc640 1 -- 172.21.15.62:0/1332895261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaec2a90 msgr2=0x5589eaf34cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaec2a90 0x5589eaf34cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aa7fc640 1 -- 172.21.15.62:0/1332895261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589eaf35350 con 0x5589eaeb3cc0 2024-09-15T23:24:32.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f60aa7fc640 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb3cc0 0x5589eaec6330 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f60a00b8a90 tx=0x7f60a00bc070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9634249619cdbaf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:32.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.060+0000 7f608eff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:32.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f608eff5640 1 -- 172.21.15.62:0/1332895261 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a00d4040 con 0x5589eaeb3cc0 2024-09-15T23:24:32.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f608eff5640 1 -- 172.21.15.62:0/1332895261 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f60a003b460 con 0x5589eaeb3cc0 2024-09-15T23:24:32.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f60b38ba740 1 -- 172.21.15.62:0/1332895261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f609c004c80 con 0x5589eaeb3cc0 2024-09-15T23:24:32.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f608eff5640 1 -- 172.21.15.62:0/1332895261 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f60a00f0130 con 0x5589eaeb3cc0 2024-09-15T23:24:32.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f60b38ba740 10 client.17364 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:32.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f60b38ba740 1 -- 172.21.15.62:0/1332895261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5589eaf35350 con 0x5589eaeb3cc0 2024-09-15T23:24:32.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f608eff5640 1 -- 172.21.15.62:0/1332895261 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f609c004c80 con 0x5589eaeb3cc0 2024-09-15T23:24:32.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f608eff5640 1 -- 172.21.15.62:0/1332895261 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f60a00c4d60 con 0x5589eaeb3cc0 2024-09-15T23:24:32.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f60b38ba740 10 client.17364 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:32.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f60b38ba740 10 client.17364 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:32.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f60b38ba740 20 client.17364 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:32.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589eaf3ca00 0x5589eaf3ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:32.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f60b38ba740 4 client.17364 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:24:32.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.061+0000 7f60b38ba740 1 -- 172.21.15.62:0/1332895261 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5589eae20d60 con 0x5589eaf3ca00 2024-09-15T23:24:32.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60aaffd640 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589eaf3ca00 0x5589eaf3ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:32.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60aaffd640 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589eaf3ca00 0x5589eaf3ee20 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a9ea8c102394066 server_cookie=774948e4fb7e03cc in_seq=0 out_seq=0 2024-09-15T23:24:32.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f608eff5640 10 client.17364 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:32.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f608eff5640 1 -- 172.21.15.62:0/1332895261 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5589eae20d60 con 0x5589eaf3ca00 2024-09-15T23:24:32.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f608eff5640 10 client.17364 handle_command_reply: tid=0 2024-09-15T23:24:32.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 client.17364 shutdown 2024-09-15T23:24:32.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 -- 172.21.15.62:0/1332895261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb3cc0 msgr2=0x5589eaec6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb3cc0 0x5589eaec6330 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f60a00b8a90 tx=0x7f60a00bc070 comp rx=0 tx=0).stop 2024-09-15T23:24:32.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 -- 172.21.15.62:0/1332895261 shutdown_connections 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589eaf3ca00 0x5589eaf3ee20 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589eaec2a90 0x5589eaf34cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f608eff5640 0 client.17364 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589eaeb3cc0 0x5589eaec6330 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 --2- 172.21.15.62:0/1332895261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589eaeb33a0 0x5589eaec5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 -- 172.21.15.62:0/1332895261 >> 172.21.15.62:0/1332895261 conn(0x5589eaeafa30 msgr2=0x5589eaeae350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.062+0000 7f60b38ba740 1 -- 172.21.15.62:0/1332895261 shutdown_connections 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b38ba740 1 -- 172.21.15.62:0/1332895261 wait complete. 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b38ba740 20 client.17364 trim_cache size 0 max 16384 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:24:32.071 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:24:32.072 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:24:32.072 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:24:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 -- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f60940a3240 msgr2=0x7f60940a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f60940a3240 0x7f60940a5700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f60a4005c40 tx=0x7f60a4009500 comp rx=0 tx=0).stop 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 -- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 msgr2=0x7f60ac1989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 0x7f60ac1989f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f60a0006010 tx=0x7f60a0003c10 comp rx=0 tx=0).stop 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 -- 172.21.15.62:0/1233411130 shutdown_connections 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f60940a3240 0x7f60940a5700 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f60ac102ef0 0x7f60ac19cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ac102570 0x7f60ac1989f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 --2- 172.21.15.62:0/1233411130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60ac100e20 0x7f60ac1984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 -- 172.21.15.62:0/1233411130 >> 172.21.15.62:0/1233411130 conn(0x7f60ac0fc7d0 msgr2=0x7f60ac06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 -- 172.21.15.62:0/1233411130 shutdown_connections 2024-09-15T23:24:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:32.063+0000 7f60b14e1640 1 -- 172.21.15.62:0/1233411130 wait complete. 2024-09-15T23:24:33.273 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:24:33.276 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:33.799 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:33 smithi062 ceph-mon[30949]: pgmap v618: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 4.1 MiB/s wr, 455 op/s 2024-09-15T23:24:34.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:33 smithi067 ceph-mon[35665]: pgmap v618: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 4.1 MiB/s wr, 455 op/s 2024-09-15T23:24:34.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:33 smithi148 ceph-mon[35610]: pgmap v618: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 4.1 MiB/s wr, 455 op/s 2024-09-15T23:24:34.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.312+0000 7f27f64bd640 1 -- 172.21.15.62:0/3993871650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0106f10 msgr2=0x7f27f010ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.312+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3993871650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0106f10 0x7f27f010ea90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f27e8009ca0 tx=0x7f27e802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.312+0000 7f27f64bd640 1 -- 172.21.15.62:0/3993871650 shutdown_connections 2024-09-15T23:24:34.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.312+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3993871650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0106f10 0x7f27f010ea90 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.312+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3993871650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27f0102790 0x7f27f01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.312+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3993871650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f0101de0 0x7f27f01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.312+0000 7f27f64bd640 1 -- 172.21.15.62:0/3993871650 >> 172.21.15.62:0/3993871650 conn(0x7f27f00fd8c0 msgr2=0x7f27f00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.312+0000 7f27f64bd640 1 -- 172.21.15.62:0/3993871650 shutdown_connections 2024-09-15T23:24:34.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.312+0000 7f27f64bd640 1 -- 172.21.15.62:0/3993871650 wait complete. 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27f64bd640 1 Processor -- start 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27f64bd640 1 -- start start 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27f64bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f0101de0 0x7f27f0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27f64bd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0102790 0x7f27f01989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27f64bd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27f0106f10 0x7f27f019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27f64bd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0111f70 con 0x7f27f0102790 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27f64bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27f0111df0 con 0x7f27f0101de0 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27f64bd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f27f01120f0 con 0x7f27f0106f10 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27ef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0102790 0x7f27f01989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:34.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27ef7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0102790 0x7f27f01989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54432/0 (socket says 172.21.15.62:54432) 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27ef7fe640 1 -- 172.21.15.62:0/3371075056 learned_addr learned my addr 172.21.15.62:0/3371075056 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27f4a5a640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27f0106f10 0x7f27f019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.313+0000 7f27effff640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f0101de0 0x7f27f0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27ef7fe640 1 -- 172.21.15.62:0/3371075056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27f0106f10 msgr2=0x7f27f019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27ef7fe640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27f0106f10 0x7f27f019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27ef7fe640 1 -- 172.21.15.62:0/3371075056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f0101de0 msgr2=0x7f27f0198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27ef7fe640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f0101de0 0x7f27f0198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27ef7fe640 1 -- 172.21.15.62:0/3371075056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f019d420 con 0x7f27f0102790 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27effff640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f0101de0 0x7f27f0198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27ef7fe640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0102790 0x7f27f01989b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f27e400d770 tx=0x7f27e400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27ed7fa640 1 -- 172.21.15.62:0/3371075056 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e4017070 con 0x7f27f0102790 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27ed7fa640 1 -- 172.21.15.62:0/3371075056 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e4005500 con 0x7f27f0102790 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f019d710 con 0x7f27f0102790 2024-09-15T23:24:34.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27ed7fa640 1 -- 172.21.15.62:0/3371075056 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e4004e00 con 0x7f27f0102790 2024-09-15T23:24:34.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.314+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f01a4ff0 con 0x7f27f0102790 2024-09-15T23:24:34.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.315+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f01021c0 con 0x7f27f0102790 2024-09-15T23:24:34.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.316+0000 7f27ed7fa640 1 -- 172.21.15.62:0/3371075056 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f27e4004fa0 con 0x7f27f0102790 2024-09-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.317+0000 7f27ed7fa640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d80a3240 0x7f27d80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.317+0000 7f27effff640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d80a3240 0x7f27d80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.317+0000 7f27ed7fa640 1 -- 172.21.15.62:0/3371075056 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f27e40cf6b0 con 0x7f27f0102790 2024-09-15T23:24:34.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.318+0000 7f27effff640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d80a3240 0x7f27d80a5700 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009c20 tx=0x7f27e0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:34.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.321+0000 7f27ed7fa640 1 -- 172.21.15.62:0/3371075056 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f27e409c050 con 0x7f27f0102790 2024-09-15T23:24:34.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.535+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27f010a150 con 0x7f27f0102790 2024-09-15T23:24:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.535+0000 7f27ed7fa640 1 -- 172.21.15.62:0/3371075056 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11904 (secure 0 0 0) 0x7f27e409b950 con 0x7f27f0102790 2024-09-15T23:24:34.537 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:24:34.537 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":34,"btime":"2024-09-15T23:24:12:668733+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:12.388349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:24:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 34 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d80a3240 msgr2=0x7f27d80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d80a3240 0x7f27d80a5700 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009c20 tx=0x7f27e0005f90 comp rx=0 tx=0).stop 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0102790 msgr2=0x7f27f01989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0102790 0x7f27f01989b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f27e400d770 tx=0x7f27e400dc30 comp rx=0 tx=0).stop 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 shutdown_connections 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d80a3240 0x7f27d80a5700 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27f0106f10 0x7f27f019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27f0102790 0x7f27f01989b0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 --2- 172.21.15.62:0/3371075056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f0101de0 0x7f27f0198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 >> 172.21.15.62:0/3371075056 conn(0x7f27f00fd8c0 msgr2=0x7f27f00fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 shutdown_connections 2024-09-15T23:24:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:34.538+0000 7f27f64bd640 1 -- 172.21.15.62:0/3371075056 wait complete. 2024-09-15T23:24:34.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 495.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:24:34.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:34 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3371075056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:35.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:34 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3371075056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:35.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:34 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3371075056' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:35.204 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:24:35.459 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:35.882 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:35 smithi062 ceph-mon[30949]: pgmap v619: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.880+0000 7f8e4266f640 1 -- 172.21.15.62:0/1608810961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e3c102d90 msgr2=0x7f8e3c103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.880+0000 7f8e4266f640 1 --2- 172.21.15.62:0/1608810961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e3c102d90 0x7f8e3c103170 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009930 tx=0x7f8e2c02a420 comp rx=0 tx=0).stop 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.880+0000 7f8e4266f640 1 -- 172.21.15.62:0/1608810961 shutdown_connections 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.880+0000 7f8e4266f640 1 --2- 172.21.15.62:0/1608810961 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e3c108290 0x7f8e3c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.880+0000 7f8e4266f640 1 --2- 172.21.15.62:0/1608810961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3c103740 0x7f8e3c107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.880+0000 7f8e4266f640 1 --2- 172.21.15.62:0/1608810961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e3c102d90 0x7f8e3c103170 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.880+0000 7f8e4266f640 1 -- 172.21.15.62:0/1608810961 >> 172.21.15.62:0/1608810961 conn(0x7f8e3c0fc7d0 msgr2=0x7f8e3c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.880+0000 7f8e4266f640 1 -- 172.21.15.62:0/1608810961 shutdown_connections 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.880+0000 7f8e4266f640 1 -- 172.21.15.62:0/1608810961 wait complete. 2024-09-15T23:24:35.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.881+0000 7f8e4266f640 1 Processor -- start 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.881+0000 7f8e4266f640 1 -- start start 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.881+0000 7f8e4266f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3c102d90 0x7f8e3c1009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.881+0000 7f8e4266f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e3c103740 0x7f8e3c100f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.881+0000 7f8e4266f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e3c108290 0x7f8e3c101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.881+0000 7f8e4266f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c1120b0 con 0x7f8e3c103740 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.881+0000 7f8e4266f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c111f30 con 0x7f8e3c102d90 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.881+0000 7f8e4266f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c112230 con 0x7f8e3c108290 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3c102d90 0x7f8e3c1009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e3c103740 0x7f8e3c100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3c102d90 0x7f8e3c1009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34702/0 (socket says 172.21.15.62:34702) 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3bfff640 1 -- 172.21.15.62:0/4206558674 learned_addr learned my addr 172.21.15.62:0/4206558674 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e40c0c640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e3c108290 0x7f8e3c101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3b7fe640 1 -- 172.21.15.62:0/4206558674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e3c108290 msgr2=0x7f8e3c101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e3c108290 0x7f8e3c101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3b7fe640 1 -- 172.21.15.62:0/4206558674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3c102d90 msgr2=0x7f8e3c1009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3c102d90 0x7f8e3c1009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3b7fe640 1 -- 172.21.15.62:0/4206558674 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3c1a6a80 con 0x7f8e3c103740 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3c102d90 0x7f8e3c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e3c103740 0x7f8e3c100f30 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000d6a0 tx=0x7f8e3000db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e397fa640 1 -- 172.21.15.62:0/4206558674 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30017070 con 0x7f8e3c103740 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e397fa640 1 -- 172.21.15.62:0/4206558674 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e30009820 con 0x7f8e3c103740 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e4266f640 1 -- 172.21.15.62:0/4206558674 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e3c1a6d70 con 0x7f8e3c103740 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e397fa640 1 -- 172.21.15.62:0/4206558674 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30004d00 con 0x7f8e3c103740 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.882+0000 7f8e4266f640 1 -- 172.21.15.62:0/4206558674 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e3c1a7250 con 0x7f8e3c103740 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.884+0000 7f8e397fa640 1 -- 172.21.15.62:0/4206558674 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8e300099d0 con 0x7f8e3c103740 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.885+0000 7f8e44a48740 1 Processor -- start 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.885+0000 7f8e44a48740 1 -- start start 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e397fa640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e240a3240 0x7f8e240a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee7995090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 0x562ee7995a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee79a0370 0x562ee79a2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e44a48740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562ee78fb2f0 con 0x562ee7996b40 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e44a48740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562ee78a2cf0 con 0x562ee79955d0 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e44a48740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562ee78f72b0 con 0x562ee79a0370 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e40c0c640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e240a3240 0x7f8e240a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e397fa640 1 -- 172.21.15.62:0/4206558674 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8e300cf5b0 con 0x7f8e3c103740 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee7995090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee7995090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54452/0 (socket says 172.21.15.62:54452) 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3bfff640 1 -- 172.21.15.62:0/2195705646 learned_addr learned my addr 172.21.15.62:0/2195705646 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/2195705646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 0x562ee7995a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3bfff640 1 -- 172.21.15.62:0/2195705646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee79a0370 msgr2=0x562ee79a2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/2195705646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee79a0370 0x562ee79a2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3bfff640 1 -- 172.21.15.62:0/2195705646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 msgr2=0x562ee7995a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/2195705646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 0x562ee7995a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3bfff640 1 -- 172.21.15.62:0/2195705646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ee78873a0 con 0x562ee7996b40 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e40c0c640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e240a3240 0x7f8e240a5700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400ad70 tx=0x7f8e3400f790 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.886+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/2195705646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee7995090 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c004560 tx=0x7f8e2c004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e49b10c76eea55 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e227fc640 1 -- 172.21.15.62:0/2195705646 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c035040 con 0x562ee7996b40 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e227fc640 1 -- 172.21.15.62:0/2195705646 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e2c004c40 con 0x562ee7996b40 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e227fc640 1 -- 172.21.15.62:0/2195705646 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c003c30 con 0x562ee7996b40 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e44a48740 1 -- 172.21.15.62:0/2195705646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 msgr2=0x562ee7995090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e44a48740 1 --2- 172.21.15.62:0/2195705646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee7995090 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c004560 tx=0x7f8e2c004590 comp rx=0 tx=0).stop 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e44a48740 1 -- 172.21.15.62:0/2195705646 shutdown_connections 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e44a48740 1 --2- 172.21.15.62:0/2195705646 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee79a0370 0x562ee79a2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e44a48740 1 --2- 172.21.15.62:0/2195705646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 0x562ee7995a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e44a48740 1 --2- 172.21.15.62:0/2195705646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee7995090 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e44a48740 1 -- 172.21.15.62:0/2195705646 >> 172.21.15.62:0/2195705646 conn(0x562ee795e310 msgr2=0x562ee7991c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:35.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e44a48740 1 -- 172.21.15.62:0/2195705646 shutdown_connections 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.887+0000 7f8e44a48740 1 -- 172.21.15.62:0/2195705646 wait complete. 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e44a48740 1 -- start start 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e44a48740 1 Processor -- start 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee79955d0 0x562ee799fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee799d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 0x562ee799d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e44a48740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562ee79a3010 con 0x562ee7996b40 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e44a48740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562ee78b6060 con 0x562ee79a0370 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e44a48740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562ee79a3190 con 0x562ee79955d0 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e44a48740 10 client.0 _ll_register_callbacks cb 0x562ee76098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e3b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee799d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e3b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee799d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54456/0 (socket says 172.21.15.62:54456) 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e3b7fe640 1 -- 172.21.15.62:0/3918014350 learned_addr learned my addr 172.21.15.62:0/3918014350 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:35.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.888+0000 7f8e40c0c640 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 0x562ee799d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee79955d0 0x562ee799fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e3b7fe640 1 -- 172.21.15.62:0/3918014350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee79955d0 msgr2=0x562ee799fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee79955d0 0x562ee799fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e3b7fe640 1 -- 172.21.15.62:0/3918014350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 msgr2=0x562ee799d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 0x562ee799d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e3b7fe640 1 -- 172.21.15.62:0/3918014350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ee766c3c0 con 0x562ee7996b40 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee799d040 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8e30097d00 tx=0x7f8e30086130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a00f4264137056d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/3918014350 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30020040 con 0x562ee7996b40 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/3918014350 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8e30009d40 con 0x562ee7996b40 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e44a48740 1 -- 172.21.15.62:0/3918014350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e2c004c40 con 0x562ee7996b40 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/3918014350 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8e300a4460 con 0x562ee7996b40 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e44a48740 10 client.17382 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:35.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e44a48740 1 -- 172.21.15.62:0/3918014350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562ee79aeb40 con 0x562ee7996b40 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.889+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/3918014350 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e2c004c40 con 0x562ee7996b40 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/3918014350 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f8e3009ad80 con 0x562ee7996b40 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e44a48740 10 client.17382 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e44a48740 10 client.17382 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e44a48740 20 client.17382 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e44a48740 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x562ee79b0050 0x562ee79b2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e44a48740 4 client.17382 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e44a48740 1 -- 172.21.15.62:0/3918014350 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ee7908860 con 0x562ee79b0050 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x562ee79b0050 0x562ee79b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x562ee79b0050 0x562ee79b2460 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3a7adf3bc7c0faa1 server_cookie=562894932bc3cc1d in_seq=0 out_seq=0 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.890+0000 7f8e1b7fe640 10 client.17382 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/3918014350 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562ee7908860 con 0x562ee79b0050 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e1b7fe640 10 client.17382 handle_command_reply: tid=0 2024-09-15T23:24:35.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 client.17382 shutdown 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 -- 172.21.15.62:0/3918014350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 msgr2=0x562ee799d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee799d040 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8e30097d00 tx=0x7f8e30086130 comp rx=0 tx=0).stop 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 -- 172.21.15.62:0/3918014350 shutdown_connections 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x562ee79b0050 0x562ee79b2460 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 0x562ee799d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee7996b40 0x562ee799d040 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e1b7fe640 0 client.17382 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 --2- 172.21.15.62:0/3918014350 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee79955d0 0x562ee799fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 -- 172.21.15.62:0/3918014350 >> 172.21.15.62:0/3918014350 conn(0x562ee795e310 msgr2=0x562ee7997960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 -- 172.21.15.62:0/3918014350 shutdown_connections 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.891+0000 7f8e44a48740 1 -- 172.21.15.62:0/3918014350 wait complete. 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.892+0000 7f8e44a48740 20 client.17382 trim_cache size 0 max 16384 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e44a48740 1 Processor -- start 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e44a48740 1 -- start start 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 0x562ee79ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 0x562ee79b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79a0370 0x562ee79b8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e44a48740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c1b3740 con 0x562ee79a0370 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e44a48740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c1b1010 con 0x562ee79955d0 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e44a48740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c1aee60 con 0x562ee7996b40 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e40c0c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79a0370 0x562ee79b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e40c0c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79a0370 0x562ee79b8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54470/0 (socket says 172.21.15.62:54470) 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e40c0c640 1 -- 172.21.15.62:0/2163109666 learned_addr learned my addr 172.21.15.62:0/2163109666 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/2163109666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 0x562ee79ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/2163109666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 0x562ee79b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e40c0c640 1 -- 172.21.15.62:0/2163109666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 msgr2=0x562ee79b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e40c0c640 1 --2- 172.21.15.62:0/2163109666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 0x562ee79b7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e40c0c640 1 -- 172.21.15.62:0/2163109666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 msgr2=0x562ee79ac2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e40c0c640 1 --2- 172.21.15.62:0/2163109666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 0x562ee79ac2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e40c0c640 1 -- 172.21.15.62:0/2163109666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ee7998120 con 0x562ee79a0370 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.905+0000 7f8e40c0c640 1 --2- 172.21.15.62:0/2163109666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79a0370 0x562ee79b8270 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400b010 tx=0x7f8e34014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f44a4d08a69a24 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.906+0000 7f8e21ffb640 1 -- 172.21.15.62:0/2163109666 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34023070 con 0x562ee79a0370 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.906+0000 7f8e21ffb640 1 -- 172.21.15.62:0/2163109666 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e34016040 con 0x562ee79a0370 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.906+0000 7f8e21ffb640 1 -- 172.21.15.62:0/2163109666 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34018070 con 0x562ee79a0370 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.906+0000 7f8e44a48740 1 -- 172.21.15.62:0/2163109666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79a0370 msgr2=0x562ee79b8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.906+0000 7f8e44a48740 1 --2- 172.21.15.62:0/2163109666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79a0370 0x562ee79b8270 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400b010 tx=0x7f8e34014000 comp rx=0 tx=0).stop 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.906+0000 7f8e44a48740 1 -- 172.21.15.62:0/2163109666 shutdown_connections 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.906+0000 7f8e44a48740 1 --2- 172.21.15.62:0/2163109666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79a0370 0x562ee79b8270 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.906+0000 7f8e44a48740 1 --2- 172.21.15.62:0/2163109666 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 0x562ee79b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.906+0000 7f8e44a48740 1 --2- 172.21.15.62:0/2163109666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79955d0 0x562ee79ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 -- 172.21.15.62:0/2163109666 >> 172.21.15.62:0/2163109666 conn(0x562ee795e310 msgr2=0x562ee7998310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 -- 172.21.15.62:0/2163109666 shutdown_connections 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 -- 172.21.15.62:0/2163109666 wait complete. 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 -- start start 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 Processor -- start 2024-09-15T23:24:35.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79955d0 0x562ee79a4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 0x562ee79a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 0x562ee7a19d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c1af4d0 con 0x562ee79955d0 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c1b0940 con 0x562ee79a0370 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3c1afba0 con 0x562ee7996b40 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e44a48740 10 client.0 _ll_register_callbacks cb 0x7f8e300057f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.907+0000 7f8e3bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79955d0 0x562ee79a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e3bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79955d0 0x562ee79a4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54476/0 (socket says 172.21.15.62:54476) 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e3bfff640 1 -- 172.21.15.62:0/1213473198 learned_addr learned my addr 172.21.15.62:0/1213473198 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 0x562ee79a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e40c0c640 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 0x562ee7a19d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e3bfff640 1 -- 172.21.15.62:0/1213473198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 msgr2=0x562ee79a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 0x562ee79a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e3bfff640 1 -- 172.21.15.62:0/1213473198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 msgr2=0x562ee7a19d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 0x562ee7a19d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e3bfff640 1 -- 172.21.15.62:0/1213473198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ee7a1a240 con 0x562ee79955d0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e40c0c640 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 0x562ee7a19d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e3bfff640 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79955d0 0x562ee79a4810 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0037e0 tx=0x7f8e2c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c548c887496562 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/1213473198 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c035040 con 0x562ee79955d0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e44a48740 1 -- 172.21.15.62:0/1213473198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e34016040 con 0x562ee79955d0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/1213473198 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8e2c0050c0 con 0x562ee79955d0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/1213473198 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8e2c062050 con 0x562ee79955d0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e44a48740 10 client.17388 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.908+0000 7f8e44a48740 1 -- 172.21.15.62:0/1213473198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x562ee7a1b440 con 0x562ee79955d0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/1213473198 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e34016040 con 0x562ee79955d0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/1213473198 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f8e2c057020 con 0x562ee79955d0 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e44a48740 10 client.17388 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:35.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e44a48740 10 client.17388 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e44a48740 20 client.17388 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e44a48740 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x562ee7a1b780 0x562ee7a1db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e44a48740 4 client.17388 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e44a48740 1 -- 172.21.15.62:0/1213473198 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562ee7902d70 con 0x562ee7a1b780 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x562ee7a1b780 0x562ee7a1db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e3b7fe640 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x562ee7a1b780 0x562ee7a1db70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e93d90f06cfab867 server_cookie=9432f6826ae26cc8 in_seq=0 out_seq=0 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.909+0000 7f8e1b7fe640 10 client.17388 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e1b7fe640 1 -- 172.21.15.62:0/1213473198 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562ee7902d70 con 0x562ee7a1b780 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e1b7fe640 10 client.17388 handle_command_reply: tid=0 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 client.17388 shutdown 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 -- 172.21.15.62:0/1213473198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79955d0 msgr2=0x562ee79a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79955d0 0x562ee79a4810 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c0037e0 tx=0x7f8e2c009260 comp rx=0 tx=0).stop 2024-09-15T23:24:35.942 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:24:35.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 -- 172.21.15.62:0/1213473198 shutdown_connections 2024-09-15T23:24:35.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x562ee7a1b780 0x562ee7a1db70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562ee79a0370 0x562ee7a19d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562ee7996b40 0x562ee79a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 --2- 172.21.15.62:0/1213473198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562ee79955d0 0x562ee79a4810 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e1b7fe640 0 client.17388 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:24:35.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 -- 172.21.15.62:0/1213473198 >> 172.21.15.62:0/1213473198 conn(0x562ee795e310 msgr2=0x562ee799e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:35.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 -- 172.21.15.62:0/1213473198 shutdown_connections 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 1 -- 172.21.15.62:0/1213473198 wait complete. 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.910+0000 7f8e44a48740 20 client.17388 trim_cache size 0 max 16384 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 -- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e240a3240 msgr2=0x7f8e240a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e240a3240 0x7f8e240a5700 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400ad70 tx=0x7f8e3400f790 comp rx=0 tx=0).stop 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 -- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e3c103740 msgr2=0x7f8e3c100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e3c103740 0x7f8e3c100f30 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000d6a0 tx=0x7f8e3000db60 comp rx=0 tx=0).stop 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 -- 172.21.15.62:0/4206558674 shutdown_connections 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8e240a3240 0x7f8e240a5700 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8e3c108290 0x7f8e3c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e3c103740 0x7f8e3c100f30 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 --2- 172.21.15.62:0/4206558674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e3c102d90 0x7f8e3c1009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 -- 172.21.15.62:0/4206558674 >> 172.21.15.62:0/4206558674 conn(0x7f8e3c0fc7d0 msgr2=0x7f8e3c106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 -- 172.21.15.62:0/4206558674 shutdown_connections 2024-09-15T23:24:35.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:35.911+0000 7f8e4266f640 1 -- 172.21.15.62:0/4206558674 wait complete. 2024-09-15T23:24:36.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:35 smithi067 ceph-mon[35665]: pgmap v619: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-09-15T23:24:36.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:35 smithi148 ceph-mon[35610]: pgmap v619: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.1 MiB/s wr, 526 op/s 2024-09-15T23:24:36.882 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:24:37.141 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:37.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.614+0000 7f0491912640 1 -- 172.21.15.62:0/2288642309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c102790 msgr2=0x7f048c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.614+0000 7f0491912640 1 --2- 172.21.15.62:0/2288642309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c102790 0x7f048c1067e0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0480009900 tx=0x7f048002a410 comp rx=0 tx=0).stop 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.614+0000 7f0491912640 1 -- 172.21.15.62:0/2288642309 shutdown_connections 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.614+0000 7f0491912640 1 --2- 172.21.15.62:0/2288642309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048c106f10 0x7f048c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.614+0000 7f0491912640 1 --2- 172.21.15.62:0/2288642309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c102790 0x7f048c1067e0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.614+0000 7f0491912640 1 --2- 172.21.15.62:0/2288642309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f048c101de0 0x7f048c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.614+0000 7f0491912640 1 -- 172.21.15.62:0/2288642309 >> 172.21.15.62:0/2288642309 conn(0x7f048c0fd8c0 msgr2=0x7f048c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.614+0000 7f0491912640 1 -- 172.21.15.62:0/2288642309 shutdown_connections 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.614+0000 7f0491912640 1 -- 172.21.15.62:0/2288642309 wait complete. 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f0491912640 1 Processor -- start 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f0491912640 1 -- start start 2024-09-15T23:24:37.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f0491912640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c101de0 0x7f048c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f0491912640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f048c102790 0x7f048c074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f0491912640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048c106f10 0x7f048c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f0491912640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f048c111f90 con 0x7f048c101de0 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f0491912640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f048c111e10 con 0x7f048c102790 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f0491912640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f048c112110 con 0x7f048c106f10 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f048affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c101de0 0x7f048c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f048affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c101de0 0x7f048c0744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54494/0 (socket says 172.21.15.62:54494) 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.615+0000 7f048affd640 1 -- 172.21.15.62:0/4277556656 learned_addr learned my addr 172.21.15.62:0/4277556656 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f048a7fc640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f048c102790 0x7f048c074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f048b7fe640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048c106f10 0x7f048c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f048affd640 1 -- 172.21.15.62:0/4277556656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048c106f10 msgr2=0x7f048c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f048affd640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048c106f10 0x7f048c074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f048affd640 1 -- 172.21.15.62:0/4277556656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f048c102790 msgr2=0x7f048c074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f048affd640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f048c102790 0x7f048c074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f048affd640 1 -- 172.21.15.62:0/4277556656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048c1a6930 con 0x7f048c101de0 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f048affd640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c101de0 0x7f048c0744f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f047c00c890 tx=0x7f047c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f0490910640 1 -- 172.21.15.62:0/4277556656 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c004220 con 0x7f048c101de0 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f0490910640 1 -- 172.21.15.62:0/4277556656 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f047c0043c0 con 0x7f048c101de0 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f0491912640 1 -- 172.21.15.62:0/4277556656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f048c1a6c20 con 0x7f048c101de0 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f0490910640 1 -- 172.21.15.62:0/4277556656 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c0049e0 con 0x7f048c101de0 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.616+0000 7f0491912640 1 -- 172.21.15.62:0/4277556656 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f048c1a7160 con 0x7f048c101de0 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.618+0000 7f0490910640 1 -- 172.21.15.62:0/4277556656 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f047c0079b0 con 0x7f048c101de0 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0493ceb740 1 Processor -- start 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0493ceb740 1 -- start start 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b40b40 0x564bb6b3f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 0x564bb6b3fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b4c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0493ceb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564bb6aa52f0 con 0x564bb6b40b40 2024-09-15T23:24:37.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0493ceb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564bb6a4ccf0 con 0x564bb6b4a370 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0493ceb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564bb6aa12b0 con 0x564bb6b3f5d0 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b40b40 0x564bb6b3f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b40b40 0x564bb6b3f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54502/0 (socket says 172.21.15.62:54502) 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048b7fe640 1 -- 172.21.15.62:0/1685007120 learned_addr learned my addr 172.21.15.62:0/1685007120 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048a7fc640 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b4c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0490910640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f04740a3240 0x7f04740a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048affd640 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 0x564bb6b3fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f0490910640 1 -- 172.21.15.62:0/4277556656 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f047c0cfe00 con 0x7f048c101de0 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048a7fc640 1 -- 172.21.15.62:0/1685007120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 msgr2=0x564bb6b3fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048a7fc640 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 0x564bb6b3fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048a7fc640 1 -- 172.21.15.62:0/1685007120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b40b40 msgr2=0x564bb6b3f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:37.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048a7fc640 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b40b40 0x564bb6b3f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048a7fc640 1 -- 172.21.15.62:0/1685007120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bb6a313a0 con 0x564bb6b4a370 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.619+0000 7f048b7fe640 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b40b40 0x564bb6b3f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.620+0000 7f048b7fe640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f04740a3240 0x7f04740a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.620+0000 7f048a7fc640 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b4c760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0480009230 tx=0x7f0480031630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28bcb2d4560eda5d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.620+0000 7f0471ffb640 1 -- 172.21.15.62:0/1685007120 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048003f070 con 0x564bb6b4a370 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.620+0000 7f048b7fe640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f04740a3240 0x7f04740a5700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f048400a800 tx=0x7f048400c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.620+0000 7f0471ffb640 1 -- 172.21.15.62:0/1685007120 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0480031a20 con 0x564bb6b4a370 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.620+0000 7f0471ffb640 1 -- 172.21.15.62:0/1685007120 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0480035070 con 0x564bb6b4a370 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 -- 172.21.15.62:0/1685007120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 msgr2=0x564bb6b4c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b4c760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0480009230 tx=0x7f0480031630 comp rx=0 tx=0).stop 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 -- 172.21.15.62:0/1685007120 shutdown_connections 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b4c760 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 0x564bb6b3fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1685007120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b40b40 0x564bb6b3f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 -- 172.21.15.62:0/1685007120 >> 172.21.15.62:0/1685007120 conn(0x564bb6b08310 msgr2=0x564bb6b3bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 -- 172.21.15.62:0/1685007120 shutdown_connections 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 -- 172.21.15.62:0/1685007120 wait complete. 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.621+0000 7f0493ceb740 1 -- start start 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f0493ceb740 1 Processor -- start 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b49d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 0x564bb6b46fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b47560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f0493ceb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564bb6b4d010 con 0x564bb6b3f5d0 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f0493ceb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564bb6a60060 con 0x564bb6b4a370 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f0493ceb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564bb6b4d190 con 0x564bb6b40b40 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f0493ceb740 10 client.0 _ll_register_callbacks cb 0x564bb67b38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:37.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b49d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b49d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54518/0 (socket says 172.21.15.62:54518) 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048affd640 1 -- 172.21.15.62:0/1067012638 learned_addr learned my addr 172.21.15.62:0/1067012638 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048a7fc640 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 0x564bb6b46fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048b7fe640 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b47560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048affd640 1 -- 172.21.15.62:0/1067012638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 msgr2=0x564bb6b46fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048affd640 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 0x564bb6b46fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048affd640 1 -- 172.21.15.62:0/1067012638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 msgr2=0x564bb6b47560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048affd640 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b47560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.622+0000 7f048affd640 1 -- 172.21.15.62:0/1067012638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bb68163c0 con 0x564bb6b3f5d0 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f048affd640 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b49d10 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f047c0cd520 tx=0x7f047c00df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa53bae9b4dd988d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f046affd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f046affd640 1 -- 172.21.15.62:0/1067012638 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c01d040 con 0x564bb6b3f5d0 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f046affd640 1 -- 172.21.15.62:0/1067012638 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f047c0cd630 con 0x564bb6b3f5d0 2024-09-15T23:24:37.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f0493ceb740 1 -- 172.21.15.62:0/1067012638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0480031a20 con 0x564bb6b3f5d0 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f046affd640 1 -- 172.21.15.62:0/1067012638 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f047c09c740 con 0x564bb6b3f5d0 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f0493ceb740 10 client.17400 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f0493ceb740 1 -- 172.21.15.62:0/1067012638 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x564bb6b58a60 con 0x564bb6b3f5d0 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f046affd640 1 -- 172.21.15.62:0/1067012638 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0480031a20 con 0x564bb6b3f5d0 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f046affd640 1 -- 172.21.15.62:0/1067012638 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f047c0a4820 con 0x564bb6b3f5d0 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f0493ceb740 10 client.17400 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.623+0000 7f0493ceb740 10 client.17400 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.624+0000 7f0493ceb740 20 client.17400 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.624+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x564bb6b59f20 0x564bb6b5c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.624+0000 7f0493ceb740 4 client.17400 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.624+0000 7f0493ceb740 1 -- 172.21.15.62:0/1067012638 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bb6ab2860 con 0x564bb6b59f20 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.624+0000 7f048a7fc640 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x564bb6b59f20 0x564bb6b5c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.624+0000 7f048a7fc640 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x564bb6b59f20 0x564bb6b5c310 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fb399423157c74e server_cookie=2bc65f3b2a15758 in_seq=0 out_seq=0 2024-09-15T23:24:37.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.624+0000 7f046affd640 10 client.17400 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f046affd640 1 -- 172.21.15.62:0/1067012638 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564bb6ab2860 con 0x564bb6b59f20 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f046affd640 10 client.17400 handle_command_reply: tid=0 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f0493ceb740 1 client.17400 shutdown 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f0493ceb740 1 -- 172.21.15.62:0/1067012638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 msgr2=0x564bb6b49d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b49d10 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f047c0cd520 tx=0x7f047c00df90 comp rx=0 tx=0).stop 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f0493ceb740 1 -- 172.21.15.62:0/1067012638 shutdown_connections 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x564bb6b59f20 0x564bb6b5c310 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b47560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 0x564bb6b46fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f0493ceb740 1 --2- 172.21.15.62:0/1067012638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b49d10 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f0493ceb740 1 -- 172.21.15.62:0/1067012638 >> 172.21.15.62:0/1067012638 conn(0x564bb6b08310 msgr2=0x564bb6b41960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:37.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.625+0000 7f046affd640 0 client.17400 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:24:37.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.626+0000 7f0493ceb740 1 -- 172.21.15.62:0/1067012638 shutdown_connections 2024-09-15T23:24:37.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.626+0000 7f0493ceb740 1 -- 172.21.15.62:0/1067012638 wait complete. 2024-09-15T23:24:37.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.626+0000 7f0493ceb740 20 client.17400 trim_cache size 0 max 16384 2024-09-15T23:24:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f0493ceb740 1 Processor -- start 2024-09-15T23:24:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f0493ceb740 1 -- start start 2024-09-15T23:24:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b45f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 0x564bb6b3abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b42120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f0493ceb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1b36b0 con 0x564bb6b3f5d0 2024-09-15T23:24:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f0493ceb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1b0f80 con 0x564bb6b4a370 2024-09-15T23:24:37.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f0493ceb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1aedd0 con 0x564bb6b40b40 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b45f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b45f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54530/0 (socket says 172.21.15.62:54530) 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048affd640 1 -- 172.21.15.62:0/3315044404 learned_addr learned my addr 172.21.15.62:0/3315044404 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048a7fc640 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 0x564bb6b3abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048b7fe640 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b42120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048affd640 1 -- 172.21.15.62:0/3315044404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 msgr2=0x564bb6b3abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048affd640 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 0x564bb6b3abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048affd640 1 -- 172.21.15.62:0/3315044404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 msgr2=0x564bb6b42120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048affd640 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b42120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048affd640 1 -- 172.21.15.62:0/3315044404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bb6b3c960 con 0x564bb6b3f5d0 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.639+0000 7f048b7fe640 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b42120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f048affd640 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b45f00 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f047c09a2f0 tx=0x7f047c0bc050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa557002f527fc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f04717fa640 1 -- 172.21.15.62:0/3315044404 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c01d070 con 0x564bb6b3f5d0 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f04717fa640 1 -- 172.21.15.62:0/3315044404 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f047c0a03f0 con 0x564bb6b3f5d0 2024-09-15T23:24:37.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f04717fa640 1 -- 172.21.15.62:0/3315044404 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047c096bb0 con 0x564bb6b3f5d0 2024-09-15T23:24:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f0493ceb740 1 -- 172.21.15.62:0/3315044404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 msgr2=0x564bb6b45f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f0493ceb740 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b45f00 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f047c09a2f0 tx=0x7f047c0bc050 comp rx=0 tx=0).stop 2024-09-15T23:24:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f0493ceb740 1 -- 172.21.15.62:0/3315044404 shutdown_connections 2024-09-15T23:24:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f0493ceb740 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b4a370 0x564bb6b42120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f0493ceb740 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b40b40 0x564bb6b3abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f0493ceb740 1 --2- 172.21.15.62:0/3315044404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b3f5d0 0x564bb6b45f00 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f0493ceb740 1 -- 172.21.15.62:0/3315044404 >> 172.21.15.62:0/3315044404 conn(0x564bb6b08310 msgr2=0x564bb6b3a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.640+0000 7f0493ceb740 1 -- 172.21.15.62:0/3315044404 shutdown_connections 2024-09-15T23:24:37.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 1 -- 172.21.15.62:0/3315044404 wait complete. 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 1 -- start start 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 1 Processor -- start 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 0x564bb6b52b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b40b40 0x564bb6b43cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b4a370 0x564bb6b44250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1af440 con 0x564bb6b4a370 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1b08b0 con 0x564bb6b40b40 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f048c1afb10 con 0x564bb6b3f5d0 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f0493ceb740 10 client.0 _ll_register_callbacks cb 0x7f047c004570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b4a370 0x564bb6b44250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f048b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b4a370 0x564bb6b44250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54536/0 (socket says 172.21.15.62:54536) 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.641+0000 7f048b7fe640 1 -- 172.21.15.62:0/2046209629 learned_addr learned my addr 172.21.15.62:0/2046209629 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:37.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f048affd640 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 0x564bb6b52b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f048a7fc640 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b40b40 0x564bb6b43cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f048b7fe640 1 -- 172.21.15.62:0/2046209629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 msgr2=0x564bb6b52b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f048b7fe640 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 0x564bb6b52b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f048b7fe640 1 -- 172.21.15.62:0/2046209629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b40b40 msgr2=0x564bb6b43cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f048b7fe640 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b40b40 0x564bb6b43cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f048b7fe640 1 -- 172.21.15.62:0/2046209629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bb6b44a80 con 0x564bb6b4a370 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f048b7fe640 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b4a370 0x564bb6b44250 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f048401b480 tx=0x7f0484004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1459da1ddcc39db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f046affd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f046affd640 1 -- 172.21.15.62:0/2046209629 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484018630 con 0x564bb6b4a370 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f0493ceb740 1 -- 172.21.15.62:0/2046209629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f047c0a03f0 con 0x564bb6b4a370 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f046affd640 1 -- 172.21.15.62:0/2046209629 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f04840187d0 con 0x564bb6b4a370 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f046affd640 1 -- 172.21.15.62:0/2046209629 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f04840189d0 con 0x564bb6b4a370 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f0493ceb740 10 client.17406 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.642+0000 7f0493ceb740 1 -- 172.21.15.62:0/2046209629 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x564bb6bc4c00 con 0x564bb6b4a370 2024-09-15T23:24:37.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f046affd640 1 -- 172.21.15.62:0/2046209629 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f047c0a03f0 con 0x564bb6b4a370 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f046affd640 1 -- 172.21.15.62:0/2046209629 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f04840243c0 con 0x564bb6b4a370 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f0493ceb740 10 client.17406 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f0493ceb740 10 client.17406 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f0493ceb740 20 client.17406 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f0493ceb740 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x564bb6bc5380 0x564bb6bc77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f0493ceb740 4 client.17406 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f0493ceb740 1 -- 172.21.15.62:0/2046209629 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564bb6aacd70 con 0x564bb6bc5380 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f048affd640 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x564bb6bc5380 0x564bb6bc77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f046affd640 10 client.17406 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.643+0000 7f048affd640 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x564bb6bc5380 0x564bb6bc77a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=15c22100a104bac server_cookie=e5b65815b5cc347a in_seq=0 out_seq=0 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f046affd640 1 -- 172.21.15.62:0/2046209629 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564bb6aacd70 con 0x564bb6bc5380 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f046affd640 10 client.17406 handle_command_reply: tid=0 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 client.17406 shutdown 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 -- 172.21.15.62:0/2046209629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b4a370 msgr2=0x564bb6b44250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b4a370 0x564bb6b44250 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f048401b480 tx=0x7f0484004b50 comp rx=0 tx=0).stop 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 -- 172.21.15.62:0/2046209629 shutdown_connections 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x564bb6bc5380 0x564bb6bc77a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564bb6b4a370 0x564bb6b44250 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f046affd640 0 client.17406 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564bb6b40b40 0x564bb6b43cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 --2- 172.21.15.62:0/2046209629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564bb6b3f5d0 0x564bb6b52b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 -- 172.21.15.62:0/2046209629 >> 172.21.15.62:0/2046209629 conn(0x564bb6b08310 msgr2=0x564bb6b3ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 -- 172.21.15.62:0/2046209629 shutdown_connections 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 1 -- 172.21.15.62:0/2046209629 wait complete. 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0493ceb740 20 client.17406 trim_cache size 0 max 16384 2024-09-15T23:24:37.650 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:24:37.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0491912640 1 -- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f04740a3240 msgr2=0x7f04740a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0491912640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f04740a3240 0x7f04740a5700 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f048400a800 tx=0x7f048400c040 comp rx=0 tx=0).stop 2024-09-15T23:24:37.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0491912640 1 -- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c101de0 msgr2=0x7f048c0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:37.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.644+0000 7f0491912640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c101de0 0x7f048c0744f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f047c00c890 tx=0x7f047c00cd50 comp rx=0 tx=0).stop 2024-09-15T23:24:37.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.645+0000 7f0491912640 1 -- 172.21.15.62:0/4277556656 shutdown_connections 2024-09-15T23:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.645+0000 7f0491912640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f04740a3240 0x7f04740a5700 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.645+0000 7f0491912640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f048c106f10 0x7f048c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.645+0000 7f0491912640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f048c102790 0x7f048c074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.645+0000 7f0491912640 1 --2- 172.21.15.62:0/4277556656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f048c101de0 0x7f048c0744f0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.645+0000 7f0491912640 1 -- 172.21.15.62:0/4277556656 >> 172.21.15.62:0/4277556656 conn(0x7f048c0fd8c0 msgr2=0x7f048c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.645+0000 7f0491912640 1 -- 172.21.15.62:0/4277556656 shutdown_connections 2024-09-15T23:24:37.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:37.645+0000 7f0491912640 1 -- 172.21.15.62:0/4277556656 wait complete. 2024-09-15T23:24:37.969 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:37 smithi062 ceph-mon[30949]: pgmap v620: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 MiB/s wr, 520 op/s 2024-09-15T23:24:38.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:37 smithi067 ceph-mon[35665]: pgmap v620: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 MiB/s wr, 520 op/s 2024-09-15T23:24:38.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:37 smithi148 ceph-mon[35610]: pgmap v620: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.7 MiB/s wr, 520 op/s 2024-09-15T23:24:39.033 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:24:39.297 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:39.587 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 500.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:24:39.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.784+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1977790582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 msgr2=0x7fcb781029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.784+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1977790582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 0x7fcb781029b0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009ca0 tx=0x7fcb6c02a950 comp rx=0 tx=0).stop 2024-09-15T23:24:39.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.784+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1977790582 shutdown_connections 2024-09-15T23:24:39.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.784+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1977790582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb78102ef0 0x7fcb7810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.784+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1977790582 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 0x7fcb781029b0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.784+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1977790582 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb78100e20 0x7fcb78101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.784+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1977790582 >> 172.21.15.62:0/1977790582 conn(0x7fcb780fc7d0 msgr2=0x7fcb780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:39.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.784+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1977790582 shutdown_connections 2024-09-15T23:24:39.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.784+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1977790582 wait complete. 2024-09-15T23:24:39.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7fcc1640 1 Processor -- start 2024-09-15T23:24:39.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7fcc1640 1 -- start start 2024-09-15T23:24:39.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7fcc1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb78100e20 0x7fcb780fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7fcc1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 0x7fcb780fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7fcc1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb78102ef0 0x7fcb780fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7fcc1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78110f80 con 0x7fcb78102570 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7fcc1640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78110e00 con 0x7fcb78102ef0 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7fcc1640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcb78111100 con 0x7fcb78100e20 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7d25c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 0x7fcb780fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7d25c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 0x7fcb780fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54560/0 (socket says 172.21.15.62:54560) 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7d25c640 1 -- 172.21.15.62:0/1766467812 learned_addr learned my addr 172.21.15.62:0/1766467812 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb78100e20 0x7fcb780fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7d25c640 1 -- 172.21.15.62:0/1766467812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb78100e20 msgr2=0x7fcb780fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb78102ef0 0x7fcb780fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb78100e20 0x7fcb780fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7d25c640 1 -- 172.21.15.62:0/1766467812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb78102ef0 msgr2=0x7fcb780fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb78102ef0 0x7fcb780fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.785+0000 7fcb7d25c640 1 -- 172.21.15.62:0/1766467812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb781a6950 con 0x7fcb78102570 2024-09-15T23:24:39.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.786+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 0x7fcb780fd1c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c005cc0 tx=0x7fcb6c02f650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:39.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.786+0000 7fcb76ffd640 1 -- 172.21.15.62:0/1766467812 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c034040 con 0x7fcb78102570 2024-09-15T23:24:39.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.786+0000 7fcb76ffd640 1 -- 172.21.15.62:0/1766467812 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb6c030920 con 0x7fcb78102570 2024-09-15T23:24:39.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.786+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1766467812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb781a6be0 con 0x7fcb78102570 2024-09-15T23:24:39.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.786+0000 7fcb76ffd640 1 -- 172.21.15.62:0/1766467812 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c004830 con 0x7fcb78102570 2024-09-15T23:24:39.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.786+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1766467812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb781a7110 con 0x7fcb78102570 2024-09-15T23:24:39.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.788+0000 7fcb76ffd640 1 -- 172.21.15.62:0/1766467812 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fcb6c037020 con 0x7fcb78102570 2024-09-15T23:24:39.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.788+0000 7fcb8209a740 1 Processor -- start 2024-09-15T23:24:39.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb8209a740 1 -- start start 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea5b40 0x55ff2cea4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 0x55ff2cea4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 0x55ff2ceb1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb8209a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ff2ce0a2f0 con 0x55ff2cea5b40 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb8209a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ff2cdb1cf0 con 0x55ff2ceaf370 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb8209a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ff2ce062b0 con 0x55ff2cea45d0 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb7e25e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea5b40 0x55ff2cea4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb76ffd640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb600a32c0 0x7fcb600a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb7e25e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea5b40 0x55ff2cea4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54568/0 (socket says 172.21.15.62:54568) 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb7e25e640 1 -- 172.21.15.62:0/865859465 learned_addr learned my addr 172.21.15.62:0/865859465 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 0x55ff2cea4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 0x55ff2ceb1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb600a32c0 0x7fcb600a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.789+0000 7fcb7e25e640 1 -- 172.21.15.62:0/865859465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 msgr2=0x55ff2cea4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 0x55ff2cea4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb7e25e640 1 -- 172.21.15.62:0/865859465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 msgr2=0x55ff2ceb1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 0x55ff2ceb1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb7e25e640 1 -- 172.21.15.62:0/865859465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff2cd963a0 con 0x55ff2cea5b40 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb76ffd640 1 -- 172.21.15.62:0/1766467812 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcb6c0f0c70 con 0x7fcb78102570 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 0x55ff2cea4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 0x55ff2ceb1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea5b40 0x55ff2cea4090 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcb70005940 tx=0x7fcb70014b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5396a2dc3f208f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb5ffff640 1 -- 172.21.15.62:0/865859465 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70021070 con 0x55ff2cea5b40 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb5ffff640 1 -- 172.21.15.62:0/865859465 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb700047c0 con 0x55ff2cea5b40 2024-09-15T23:24:39.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.790+0000 7fcb5ffff640 1 -- 172.21.15.62:0/865859465 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7001c9c0 con 0x55ff2cea5b40 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 -- 172.21.15.62:0/865859465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea5b40 msgr2=0x55ff2cea4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb600a32c0 0x7fcb600a5780 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcb7001e000 tx=0x7fcb7001b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea5b40 0x55ff2cea4090 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcb70005940 tx=0x7fcb70014b00 comp rx=0 tx=0).stop 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 -- 172.21.15.62:0/865859465 shutdown_connections 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 0x55ff2ceb1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 0x55ff2cea4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 --2- 172.21.15.62:0/865859465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea5b40 0x55ff2cea4090 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 -- 172.21.15.62:0/865859465 >> 172.21.15.62:0/865859465 conn(0x55ff2ce6d310 msgr2=0x55ff2cea0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 -- 172.21.15.62:0/865859465 shutdown_connections 2024-09-15T23:24:39.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 -- 172.21.15.62:0/865859465 wait complete. 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 -- start start 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.791+0000 7fcb8209a740 1 Processor -- start 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea45d0 0x55ff2ceaed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2ceac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb8209a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ff2ceb2070 con 0x55ff2ceaf370 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb8209a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ff2cdc5060 con 0x55ff2cea45d0 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb8209a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ff2ceb21f0 con 0x55ff2cea5b40 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb8209a740 10 client.0 _ll_register_callbacks cb 0x55ff2cb188c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7e25e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2ceac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7e25e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2ceac5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54582/0 (socket says 172.21.15.62:54582) 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7e25e640 1 -- 172.21.15.62:0/1059379813 learned_addr learned my addr 172.21.15.62:0/1059379813 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:39.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea45d0 0x55ff2ceaed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7e25e640 1 -- 172.21.15.62:0/1059379813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 msgr2=0x55ff2ceac000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceac000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7e25e640 1 -- 172.21.15.62:0/1059379813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea45d0 msgr2=0x55ff2ceaed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea45d0 0x55ff2ceaed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7e25e640 1 -- 172.21.15.62:0/1059379813 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff2cb7b3c0 con 0x55ff2ceaf370 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.792+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceac000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2ceac5d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcb70017040 tx=0x7fcb70004c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473e0dcf59023a82 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/1059379813 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70032060 con 0x55ff2ceaf370 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/1059379813 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcb70005410 con 0x55ff2ceaf370 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb8209a740 1 -- 172.21.15.62:0/1059379813 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb700047c0 con 0x55ff2ceaf370 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/1059379813 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcb70039050 con 0x55ff2ceaf370 2024-09-15T23:24:39.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb8209a740 10 client.17418 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb8209a740 1 -- 172.21.15.62:0/1059379813 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ff2cebdb40 con 0x55ff2ceaf370 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/1059379813 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb700047c0 con 0x55ff2ceaf370 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/1059379813 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7fcb7001b070 con 0x55ff2ceaf370 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb8209a740 10 client.17418 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.793+0000 7fcb8209a740 10 client.17418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.794+0000 7fcb8209a740 20 client.17418 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.794+0000 7fcb8209a740 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ff2cebefa0 0x55ff2cec1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.794+0000 7fcb8209a740 4 client.17418 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.794+0000 7fcb8209a740 1 -- 172.21.15.62:0/1059379813 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff2ce17860 con 0x55ff2cebefa0 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.794+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ff2cebefa0 0x55ff2cec1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.794+0000 7fcb5d7fa640 10 client.17418 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.794+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ff2cebefa0 0x55ff2cec1390 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbfc690300b6b87a server_cookie=a1b8f8bfe6c498ee in_seq=0 out_seq=0 2024-09-15T23:24:39.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/1059379813 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ff2ce17860 con 0x55ff2cebefa0 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb5d7fa640 10 client.17418 handle_command_reply: tid=0 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 client.17418 shutdown 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 -- 172.21.15.62:0/1059379813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 msgr2=0x55ff2ceac5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2ceac5d0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcb70017040 tx=0x7fcb70004c20 comp rx=0 tx=0).stop 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 -- 172.21.15.62:0/1059379813 shutdown_connections 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ff2cebefa0 0x55ff2cec1390 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2ceac5d0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceac000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 --2- 172.21.15.62:0/1059379813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea45d0 0x55ff2ceaed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb5d7fa640 0 client.17418 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 -- 172.21.15.62:0/1059379813 >> 172.21.15.62:0/1059379813 conn(0x55ff2ce6d310 msgr2=0x55ff2cea6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 -- 172.21.15.62:0/1059379813 shutdown_connections 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 1 -- 172.21.15.62:0/1059379813 wait complete. 2024-09-15T23:24:39.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.795+0000 7fcb8209a740 20 client.17418 trim_cache size 0 max 16384 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.808+0000 7fcb8209a740 1 Processor -- start 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.808+0000 7fcb8209a740 1 -- start start 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 0x55ff2cebb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea5b40 0x55ff2cea7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2cea7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb8209a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781b35a0 con 0x55ff2ceaf370 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb8209a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781b0e70 con 0x55ff2cea5b40 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb8209a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781aecc0 con 0x55ff2cea45d0 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7e25e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2cea7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7e25e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2cea7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54584/0 (socket says 172.21.15.62:54584) 2024-09-15T23:24:39.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7e25e640 1 -- 172.21.15.62:0/3472739843 learned_addr learned my addr 172.21.15.62:0/3472739843 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea5b40 0x55ff2cea7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 0x55ff2cebb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7e25e640 1 -- 172.21.15.62:0/3472739843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 msgr2=0x55ff2cebb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 0x55ff2cebb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7e25e640 1 -- 172.21.15.62:0/3472739843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea5b40 msgr2=0x55ff2cea7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea5b40 0x55ff2cea7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7e25e640 1 -- 172.21.15.62:0/3472739843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff2cea7ba0 con 0x55ff2ceaf370 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea5b40 0x55ff2cea7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.809+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2cea7660 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fcb70025c00 tx=0x7fcb70025e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ebe28f7f1f089f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb5f7fe640 1 -- 172.21.15.62:0/3472739843 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70032030 con 0x55ff2ceaf370 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb5f7fe640 1 -- 172.21.15.62:0/3472739843 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb700047c0 con 0x55ff2ceaf370 2024-09-15T23:24:39.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb5f7fe640 1 -- 172.21.15.62:0/3472739843 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb70021050 con 0x55ff2ceaf370 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb8209a740 1 -- 172.21.15.62:0/3472739843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 msgr2=0x55ff2cea7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb8209a740 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2cea7660 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fcb70025c00 tx=0x7fcb70025e80 comp rx=0 tx=0).stop 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb8209a740 1 -- 172.21.15.62:0/3472739843 shutdown_connections 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb8209a740 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2ceaf370 0x55ff2cea7660 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb8209a740 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2cea5b40 0x55ff2cea7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb8209a740 1 --2- 172.21.15.62:0/3472739843 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea45d0 0x55ff2cebb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb8209a740 1 -- 172.21.15.62:0/3472739843 >> 172.21.15.62:0/3472739843 conn(0x55ff2ce6d310 msgr2=0x55ff2ceab0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb8209a740 1 -- 172.21.15.62:0/3472739843 shutdown_connections 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.810+0000 7fcb8209a740 1 -- 172.21.15.62:0/3472739843 wait complete. 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb8209a740 1 -- start start 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb8209a740 1 Processor -- start 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea45d0 0x55ff2ceb77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceb7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb8209a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 0x55ff2cea8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb8209a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781af330 con 0x55ff2cea45d0 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb8209a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781b07a0 con 0x55ff2ceaf370 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb8209a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcb781afa00 con 0x55ff2cea5b40 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb8209a740 10 client.0 _ll_register_callbacks cb 0x7fcb7001f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb7da5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea45d0 0x55ff2ceb77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb7da5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea45d0 0x55ff2ceb77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54594/0 (socket says 172.21.15.62:54594) 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb7da5d640 1 -- 172.21.15.62:0/984263616 learned_addr learned my addr 172.21.15.62:0/984263616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb7e25e640 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 0x55ff2cea8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.811+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceb7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb7da5d640 1 -- 172.21.15.62:0/984263616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 msgr2=0x55ff2ceb7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceb7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb7da5d640 1 -- 172.21.15.62:0/984263616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 msgr2=0x55ff2cea8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 0x55ff2cea8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb7da5d640 1 -- 172.21.15.62:0/984263616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff2cea9280 con 0x55ff2cea45d0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceb7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb7da5d640 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea45d0 0x55ff2ceb77c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x55ff2cec1af0 tx=0x7fcb6800bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a030125ae7533bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/984263616 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68005220 con 0x55ff2cea45d0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb8209a740 1 -- 172.21.15.62:0/984263616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb700047c0 con 0x55ff2cea45d0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/984263616 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcb680053c0 con 0x55ff2cea45d0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/984263616 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcb6802e050 con 0x55ff2cea45d0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb8209a740 10 client.17424 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb8209a740 1 -- 172.21.15.62:0/984263616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ff2cf29760 con 0x55ff2cea45d0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.812+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/984263616 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb700047c0 con 0x55ff2cea45d0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/984263616 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7fcb6801d6b0 con 0x55ff2cea45d0 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb8209a740 10 client.17424 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:39.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb8209a740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb8209a740 20 client.17424 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb8209a740 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ff2cf29e90 0x55ff2cf2c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb8209a740 4 client.17424 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb8209a740 1 -- 172.21.15.62:0/984263616 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ff2ce11d70 con 0x55ff2cf29e90 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ff2cf29e90 0x55ff2cf2c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb7d25c640 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ff2cf29e90 0x55ff2cf2c2b0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a6b4fdf6ce5c80a server_cookie=fae18319eb4a06fe in_seq=0 out_seq=0 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.813+0000 7fcb5d7fa640 10 client.17424 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb5d7fa640 1 -- 172.21.15.62:0/984263616 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ff2ce11d70 con 0x55ff2cf29e90 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb5d7fa640 10 client.17424 handle_command_reply: tid=0 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 client.17424 shutdown 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 -- 172.21.15.62:0/984263616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea45d0 msgr2=0x55ff2ceb77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea45d0 0x55ff2ceb77c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x55ff2cec1af0 tx=0x7fcb6800bf00 comp rx=0 tx=0).stop 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 -- 172.21.15.62:0/984263616 shutdown_connections 2024-09-15T23:24:39.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ff2cf29e90 0x55ff2cf2c2b0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ff2ceaf370 0x55ff2cea8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb5d7fa640 0 client.17424 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ff2cea5b40 0x55ff2ceb7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 --2- 172.21.15.62:0/984263616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff2cea45d0 0x55ff2ceb77c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 -- 172.21.15.62:0/984263616 >> 172.21.15.62:0/984263616 conn(0x55ff2ce6d310 msgr2=0x55ff2cead860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 -- 172.21.15.62:0/984263616 shutdown_connections 2024-09-15T23:24:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 1 -- 172.21.15.62:0/984263616 wait complete. 2024-09-15T23:24:39.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.814+0000 7fcb8209a740 20 client.17424 trim_cache size 0 max 16384 2024-09-15T23:24:39.820 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:24:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb600a32c0 msgr2=0x7fcb600a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb600a32c0 0x7fcb600a5780 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fcb7001e000 tx=0x7fcb7001b000 comp rx=0 tx=0).stop 2024-09-15T23:24:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 msgr2=0x7fcb780fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 0x7fcb780fd1c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c005cc0 tx=0x7fcb6c02f650 comp rx=0 tx=0).stop 2024-09-15T23:24:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1766467812 shutdown_connections 2024-09-15T23:24:39.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcb600a32c0 0x7fcb600a5780 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb78102ef0 0x7fcb780fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcb78102570 0x7fcb780fd1c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 --2- 172.21.15.62:0/1766467812 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcb78100e20 0x7fcb780fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1766467812 >> 172.21.15.62:0/1766467812 conn(0x7fcb780fc7d0 msgr2=0x7fcb7806a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1766467812 shutdown_connections 2024-09-15T23:24:39.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:39.815+0000 7fcb7fcc1640 1 -- 172.21.15.62:0/1766467812 wait complete. 2024-09-15T23:24:39.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:39 smithi062 ceph-mon[30949]: pgmap v621: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 MiB/s wr, 680 op/s 2024-09-15T23:24:40.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:39 smithi067 ceph-mon[35665]: pgmap v621: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 MiB/s wr, 680 op/s 2024-09-15T23:24:40.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:39 smithi148 ceph-mon[35610]: pgmap v621: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 MiB/s wr, 680 op/s 2024-09-15T23:24:41.442 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:24:41.442 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:24:41.722 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.892+0000 7f7aa01bd640 1 -- 172.21.15.62:0/280944774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98107820 msgr2=0x7f7a98109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.892+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/280944774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98107820 0x7f7a98109c90 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7a90009ca0 tx=0x7f7a9002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.892+0000 7f7aa01bd640 1 -- 172.21.15.62:0/280944774 shutdown_connections 2024-09-15T23:24:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.892+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/280944774 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98107820 0x7f7a98109c90 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.892+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/280944774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a98104ef0 0x7f7a981072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.892+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/280944774 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a981025c0 0x7f7a981049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.892+0000 7f7aa01bd640 1 -- 172.21.15.62:0/280944774 >> 172.21.15.62:0/280944774 conn(0x7f7a980fc1e0 msgr2=0x7f7a980fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.892+0000 7f7aa01bd640 1 -- 172.21.15.62:0/280944774 shutdown_connections 2024-09-15T23:24:41.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.892+0000 7f7aa01bd640 1 -- 172.21.15.62:0/280944774 wait complete. 2024-09-15T23:24:41.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7aa01bd640 1 Processor -- start 2024-09-15T23:24:41.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7aa01bd640 1 -- start start 2024-09-15T23:24:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7aa01bd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a981025c0 0x7f7a98197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7aa01bd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98104ef0 0x7f7a98198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7aa01bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a98107820 0x7f7a9819f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7aa01bd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9810d800 con 0x7f7a98104ef0 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7aa01bd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9810d680 con 0x7f7a98107820 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7aa01bd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9810d980 con 0x7f7a981025c0 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7a9d758640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98104ef0 0x7f7a98198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7a9d758640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98104ef0 0x7f7a98198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54600/0 (socket says 172.21.15.62:54600) 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7a9d758640 1 -- 172.21.15.62:0/1501636928 learned_addr learned my addr 172.21.15.62:0/1501636928 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.893+0000 7f7a9df59640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a981025c0 0x7f7a98197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a9e75a640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a98107820 0x7f7a9819f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a9d758640 1 -- 172.21.15.62:0/1501636928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a981025c0 msgr2=0x7f7a98197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a9d758640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a981025c0 0x7f7a98197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a9d758640 1 -- 172.21.15.62:0/1501636928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a98107820 msgr2=0x7f7a9819f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a9d758640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a98107820 0x7f7a9819f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a9d758640 1 -- 172.21.15.62:0/1501636928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9819fa70 con 0x7f7a98104ef0 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a9d758640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98104ef0 0x7f7a98198390 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7a88005fd0 tx=0x7f7a8800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a8effd640 1 -- 172.21.15.62:0/1501636928 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88017070 con 0x7f7a98104ef0 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a8effd640 1 -- 172.21.15.62:0/1501636928 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a88009c30 con 0x7f7a98104ef0 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7aa01bd640 1 -- 172.21.15.62:0/1501636928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a9819fd60 con 0x7f7a98104ef0 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7a8effd640 1 -- 172.21.15.62:0/1501636928 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88010460 con 0x7f7a98104ef0 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.894+0000 7f7aa01bd640 1 -- 172.21.15.62:0/1501636928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a981a0270 con 0x7f7a98104ef0 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.896+0000 7f7a8effd640 1 -- 172.21.15.62:0/1501636928 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7a88003e70 con 0x7f7a98104ef0 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7aa3034740 1 Processor -- start 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7aa3034740 1 -- start start 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cff9a0 0x5568a0cffda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0d002e0 0x5568a0d04da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d05370 0x5568a0d07830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7aa3034740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5568a0c632f0 con 0x5568a0cff9a0 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7aa3034740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5568a0c0b270 con 0x5568a0d002e0 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7aa3034740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5568a0c5f2b0 con 0x5568a0d05370 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7a9e75a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cff9a0 0x5568a0cffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7a9e75a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cff9a0 0x5568a0cffda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54614/0 (socket says 172.21.15.62:54614) 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7a9e75a640 1 -- 172.21.15.62:0/423487427 learned_addr learned my addr 172.21.15.62:0/423487427 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7a9df59640 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0d002e0 0x5568a0d04da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.897+0000 7f7a8effd640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a840a3370 0x7f7a840a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9d758640 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d05370 0x5568a0d07830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9df59640 1 -- 172.21.15.62:0/423487427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d05370 msgr2=0x5568a0d07830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9df59640 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d05370 0x5568a0d07830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9df59640 1 -- 172.21.15.62:0/423487427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cff9a0 msgr2=0x5568a0cffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9df59640 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cff9a0 0x5568a0cffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9df59640 1 -- 172.21.15.62:0/423487427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568a0c0a340 con 0x5568a0d002e0 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9d758640 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d05370 0x5568a0d07830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9e75a640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a840a3370 0x7f7a840a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a8effd640 1 -- 172.21.15.62:0/1501636928 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7a880cf870 con 0x7f7a98104ef0 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9df59640 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0d002e0 0x5568a0d04da0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7a94004630 tx=0x7f7a9400bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e64688caa88cdc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a83fff640 1 -- 172.21.15.62:0/423487427 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94003bd0 con 0x5568a0d002e0 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a9e75a640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a840a3370 0x7f7a840a5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7a90006010 tx=0x7f7a900095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a83fff640 1 -- 172.21.15.62:0/423487427 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a94015070 con 0x5568a0d002e0 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.898+0000 7f7a83fff640 1 -- 172.21.15.62:0/423487427 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94004d90 con 0x5568a0d002e0 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 -- 172.21.15.62:0/423487427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0d002e0 msgr2=0x5568a0d04da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0d002e0 0x5568a0d04da0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7a94004630 tx=0x7f7a9400bb20 comp rx=0 tx=0).stop 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 -- 172.21.15.62:0/423487427 shutdown_connections 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d05370 0x5568a0d07830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0d002e0 0x5568a0d04da0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 --2- 172.21.15.62:0/423487427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cff9a0 0x5568a0cffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 -- 172.21.15.62:0/423487427 >> 172.21.15.62:0/423487427 conn(0x5568a0cfb010 msgr2=0x5568a0cfb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 -- 172.21.15.62:0/423487427 shutdown_connections 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 -- 172.21.15.62:0/423487427 wait complete. 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.899+0000 7f7aa3034740 1 -- start start 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7aa3034740 1 Processor -- start 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d09c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 0x5568a0d0a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d05370 0x5568a0d111a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7aa3034740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5568a0d08340 con 0x5568a0d05370 2024-09-15T23:24:41.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7aa3034740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5568a0bec290 con 0x5568a0cff9a0 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7aa3034740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5568a0d084c0 con 0x5568a0d002e0 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7aa3034740 10 client.0 _ll_register_callbacks cb 0x5568a0c1d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9df59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d09c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9e75a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d05370 0x5568a0d111a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9df59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d09c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34878/0 (socket says 172.21.15.62:34878) 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9df59640 1 -- 172.21.15.62:0/4082390960 learned_addr learned my addr 172.21.15.62:0/4082390960 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9d758640 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 0x5568a0d0a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9df59640 1 -- 172.21.15.62:0/4082390960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 msgr2=0x5568a0d0a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9df59640 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 0x5568a0d0a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9df59640 1 -- 172.21.15.62:0/4082390960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d05370 msgr2=0x5568a0d111a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9df59640 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d05370 0x5568a0d111a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.900+0000 7f7a9df59640 1 -- 172.21.15.62:0/4082390960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568a0976210 con 0x5568a0cff9a0 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.901+0000 7f7a9df59640 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d09c20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b5f0 tx=0x7f7a9400bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e38e302c09c42c45 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.909 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.901+0000 7f7a817fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.901+0000 7f7a817fa640 1 -- 172.21.15.62:0/4082390960 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94028060 con 0x5568a0cff9a0 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.901+0000 7f7a817fa640 1 -- 172.21.15.62:0/4082390960 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7a94011640 con 0x5568a0cff9a0 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.901+0000 7f7aa3034740 1 -- 172.21.15.62:0/4082390960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a94015070 con 0x5568a0cff9a0 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.901+0000 7f7a817fa640 1 -- 172.21.15.62:0/4082390960 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7a9402f050 con 0x5568a0cff9a0 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.901+0000 7f7aa3034740 10 client.27002 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.901+0000 7f7aa3034740 1 -- 172.21.15.62:0/4082390960 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5568a0a42620 con 0x5568a0cff9a0 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.901+0000 7f7a817fa640 1 -- 172.21.15.62:0/4082390960 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a94015070 con 0x5568a0cff9a0 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7a817fa640 1 -- 172.21.15.62:0/4082390960 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f7a94022070 con 0x5568a0cff9a0 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7aa3034740 10 client.27002 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7aa3034740 10 client.27002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7aa3034740 20 client.27002 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7aa3034740 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5568a0d15520 0x5568a0d17910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.910 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7aa3034740 4 client.27002 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7aa3034740 1 -- 172.21.15.62:0/4082390960 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568a0c70a20 con 0x5568a0d15520 2024-09-15T23:24:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7a9d758640 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5568a0d15520 0x5568a0d17910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7a9d758640 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5568a0d15520 0x5568a0d17910 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=420613cf31e5f128 server_cookie=17f0da292b42a6b5 in_seq=0 out_seq=0 2024-09-15T23:24:41.911 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.902+0000 7f7a817fa640 10 client.27002 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7a817fa640 1 -- 172.21.15.62:0/4082390960 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5568a0c70a20 con 0x5568a0d15520 2024-09-15T23:24:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7a817fa640 10 client.27002 handle_command_reply: tid=0 2024-09-15T23:24:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 client.27002 shutdown 2024-09-15T23:24:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 -- 172.21.15.62:0/4082390960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 msgr2=0x5568a0d09c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d09c20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7a9400b5f0 tx=0x7f7a9400bfa0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 -- 172.21.15.62:0/4082390960 shutdown_connections 2024-09-15T23:24:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5568a0d15520 0x5568a0d17910 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d05370 0x5568a0d111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 0x5568a0d0a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7a817fa640 0 client.27002 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 --2- 172.21.15.62:0/4082390960 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d09c20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 -- 172.21.15.62:0/4082390960 >> 172.21.15.62:0/4082390960 conn(0x5568a0cfb010 msgr2=0x5568a0cfc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 -- 172.21.15.62:0/4082390960 shutdown_connections 2024-09-15T23:24:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 1 -- 172.21.15.62:0/4082390960 wait complete. 2024-09-15T23:24:41.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.903+0000 7f7aa3034740 20 client.27002 trim_cache size 0 max 16384 2024-09-15T23:24:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7aa3034740 1 Processor -- start 2024-09-15T23:24:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7aa3034740 1 -- start start 2024-09-15T23:24:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cfb010 0x5568a0d0a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d11160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 0x5568a0cfc400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7aa3034740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7a981ad020 con 0x5568a0cfb010 2024-09-15T23:24:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7aa3034740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a981aa8f0 con 0x5568a0cff9a0 2024-09-15T23:24:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7aa3034740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7a981a8740 con 0x5568a0d002e0 2024-09-15T23:24:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7a9df59640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cfb010 0x5568a0d0a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7a9df59640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cfb010 0x5568a0d0a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54642/0 (socket says 172.21.15.62:54642) 2024-09-15T23:24:41.926 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7a9df59640 1 -- 172.21.15.62:0/1266214648 learned_addr learned my addr 172.21.15.62:0/1266214648 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7a9d758640 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d11160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.918+0000 7f7a9e75a640 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 0x5568a0cfc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a9df59640 1 -- 172.21.15.62:0/1266214648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 msgr2=0x5568a0cfc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a9df59640 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 0x5568a0cfc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a9df59640 1 -- 172.21.15.62:0/1266214648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 msgr2=0x5568a0d11160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a9df59640 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d11160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a9df59640 1 -- 172.21.15.62:0/1266214648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568a0d17c80 con 0x5568a0cfb010 2024-09-15T23:24:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a9d758640 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d11160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a9df59640 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cfb010 0x5568a0d0a040 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7a94001eb0 tx=0x7f7a9400b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191c17d2224ff5c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a837fe640 1 -- 172.21.15.62:0/1266214648 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94028070 con 0x5568a0cfb010 2024-09-15T23:24:41.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a837fe640 1 -- 172.21.15.62:0/1266214648 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a94015040 con 0x5568a0cfb010 2024-09-15T23:24:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.919+0000 7f7a837fe640 1 -- 172.21.15.62:0/1266214648 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94010d70 con 0x5568a0cfb010 2024-09-15T23:24:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 -- 172.21.15.62:0/1266214648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cfb010 msgr2=0x5568a0d0a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cfb010 0x5568a0d0a040 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7a94001eb0 tx=0x7f7a9400b4c0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 -- 172.21.15.62:0/1266214648 shutdown_connections 2024-09-15T23:24:41.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0d002e0 0x5568a0cfc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cff9a0 0x5568a0d11160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 --2- 172.21.15.62:0/1266214648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0cfb010 0x5568a0d0a040 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 -- 172.21.15.62:0/1266214648 >> 172.21.15.62:0/1266214648 conn(0x5568a0d04ec0 msgr2=0x5568a0cfa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 -- 172.21.15.62:0/1266214648 shutdown_connections 2024-09-15T23:24:41.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 -- 172.21.15.62:0/1266214648 wait complete. 2024-09-15T23:24:41.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 -- start start 2024-09-15T23:24:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.920+0000 7f7aa3034740 1 Processor -- start 2024-09-15T23:24:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cfb010 0x5568a0d0e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0cff9a0 0x5568a0d0ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7aa3034740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d002e0 0x5568a0d0fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7aa3034740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7a981a8db0 con 0x5568a0d002e0 2024-09-15T23:24:41.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7aa3034740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a981aa220 con 0x5568a0cfb010 2024-09-15T23:24:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7aa3034740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7a981a9480 con 0x5568a0cff9a0 2024-09-15T23:24:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7aa3034740 10 client.0 _ll_register_callbacks cb 0x7f7a90009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9e75a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d002e0 0x5568a0d0fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9df59640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cfb010 0x5568a0d0e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9e75a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d002e0 0x5568a0d0fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54652/0 (socket says 172.21.15.62:54652) 2024-09-15T23:24:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9e75a640 1 -- 172.21.15.62:0/2678430506 learned_addr learned my addr 172.21.15.62:0/2678430506 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9d758640 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0cff9a0 0x5568a0d0ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9df59640 1 -- 172.21.15.62:0/2678430506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0cff9a0 msgr2=0x5568a0d0ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9df59640 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0cff9a0 0x5568a0d0ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9df59640 1 -- 172.21.15.62:0/2678430506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d002e0 msgr2=0x5568a0d0fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:41.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9df59640 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d002e0 0x5568a0d0fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9df59640 1 -- 172.21.15.62:0/2678430506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568a0d10310 con 0x5568a0cfb010 2024-09-15T23:24:41.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9e75a640 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d002e0 0x5568a0d0fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:41.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a9df59640 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cfb010 0x5568a0d0e7b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7a94019bd0 tx=0x7f7a940196d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e980dc8cf01fa969 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:41.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.921+0000 7f7a817fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:24:41.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7a817fa640 1 -- 172.21.15.62:0/2678430506 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a94042060 con 0x5568a0cfb010 2024-09-15T23:24:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7aa3034740 1 -- 172.21.15.62:0/2678430506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a94015040 con 0x5568a0cfb010 2024-09-15T23:24:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7a817fa640 1 -- 172.21.15.62:0/2678430506 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7a94010440 con 0x5568a0cfb010 2024-09-15T23:24:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7a817fa640 1 -- 172.21.15.62:0/2678430506 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7a94049050 con 0x5568a0cfb010 2024-09-15T23:24:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7aa3034740 10 client.27008 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7aa3034740 1 -- 172.21.15.62:0/2678430506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5568a0d80140 con 0x5568a0cfb010 2024-09-15T23:24:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7a817fa640 1 -- 172.21.15.62:0/2678430506 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a94015040 con 0x5568a0cfb010 2024-09-15T23:24:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7a817fa640 1 -- 172.21.15.62:0/2678430506 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f7a94022050 con 0x5568a0cfb010 2024-09-15T23:24:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7aa3034740 10 client.27008 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7aa3034740 10 client.27008 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7aa3034740 20 client.27008 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7aa3034740 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5568a0d05370 0x5568a0d82a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:41.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7aa3034740 4 client.27008 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:24:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7aa3034740 1 -- 172.21.15.62:0/2678430506 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5568a0d00fd0 con 0x5568a0d05370 2024-09-15T23:24:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.922+0000 7f7a9d758640 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5568a0d05370 0x5568a0d82a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.923+0000 7f7a9d758640 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5568a0d05370 0x5568a0d82a40 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=577713eb379ea4e4 server_cookie=a3da23e07d6cf5a8 in_seq=0 out_seq=0 2024-09-15T23:24:41.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.923+0000 7f7a817fa640 10 client.27008 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.933+0000 7f7a817fa640 1 -- 172.21.15.62:0/2678430506 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5568a0d00fd0 con 0x5568a0d05370 2024-09-15T23:24:41.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.933+0000 7f7a817fa640 10 client.27008 handle_command_reply: tid=0 2024-09-15T23:24:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.933+0000 7f7aa3034740 1 client.27008 shutdown 2024-09-15T23:24:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 -- 172.21.15.62:0/2678430506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cfb010 msgr2=0x5568a0d0e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cfb010 0x5568a0d0e7b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7a94019bd0 tx=0x7f7a940196d0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 -- 172.21.15.62:0/2678430506 shutdown_connections 2024-09-15T23:24:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5568a0d05370 0x5568a0d82a40 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5568a0d002e0 0x5568a0d0fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5568a0cff9a0 0x5568a0d0ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7a817fa640 0 client.27008 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 --2- 172.21.15.62:0/2678430506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568a0cfb010 0x5568a0d0e7b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 -- 172.21.15.62:0/2678430506 >> 172.21.15.62:0/2678430506 conn(0x5568a0d04ec0 msgr2=0x5568a0cfb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 -- 172.21.15.62:0/2678430506 shutdown_connections 2024-09-15T23:24:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 1 -- 172.21.15.62:0/2678430506 wait complete. 2024-09-15T23:24:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa3034740 20 client.27008 trim_cache size 0 max 16384 2024-09-15T23:24:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:24:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:24:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:24:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:24:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa01bd640 1 -- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a840a3370 msgr2=0x7f7a840a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a840a3370 0x7f7a840a5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7a90006010 tx=0x7f7a900095c0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa01bd640 1 -- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98104ef0 msgr2=0x7f7a98198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.934+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98104ef0 0x7f7a98198390 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7a88005fd0 tx=0x7f7a8800dde0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.935+0000 7f7aa01bd640 1 -- 172.21.15.62:0/1501636928 shutdown_connections 2024-09-15T23:24:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.935+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a840a3370 0x7f7a840a5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.935+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a98107820 0x7f7a9819f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.935+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a98104ef0 0x7f7a98198390 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.935+0000 7f7aa01bd640 1 --2- 172.21.15.62:0/1501636928 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a981025c0 0x7f7a98197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:41.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.935+0000 7f7aa01bd640 1 -- 172.21.15.62:0/1501636928 >> 172.21.15.62:0/1501636928 conn(0x7f7a980fc1e0 msgr2=0x7f7a98102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:41.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.935+0000 7f7aa01bd640 1 -- 172.21.15.62:0/1501636928 shutdown_connections 2024-09-15T23:24:41.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:41.935+0000 7f7aa01bd640 1 -- 172.21.15.62:0/1501636928 wait complete. 2024-09-15T23:24:42.000 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:41 smithi062 ceph-mon[30949]: pgmap v622: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.5 MiB/s wr, 581 op/s 2024-09-15T23:24:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:41 smithi067 ceph-mon[35665]: pgmap v622: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.5 MiB/s wr, 581 op/s 2024-09-15T23:24:42.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:41 smithi148 ceph-mon[35610]: pgmap v622: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.5 MiB/s wr, 581 op/s 2024-09-15T23:24:42.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.221+0000 7f83d51ee640 1 -- 172.21.15.62:0/2873825021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 msgr2=0x7f83d010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.221+0000 7f83d51ee640 1 --2- 172.21.15.62:0/2873825021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 0x7f83d010da40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009ca0 tx=0x7f83c802a950 comp rx=0 tx=0).stop 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.222+0000 7f83d51ee640 1 -- 172.21.15.62:0/2873825021 shutdown_connections 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.222+0000 7f83d51ee640 1 --2- 172.21.15.62:0/2873825021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 0x7f83d010da40 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.222+0000 7f83d51ee640 1 --2- 172.21.15.62:0/2873825021 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83d0102570 0x7f83d01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.222+0000 7f83d51ee640 1 --2- 172.21.15.62:0/2873825021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d0100e20 0x7f83d0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.222+0000 7f83d51ee640 1 -- 172.21.15.62:0/2873825021 >> 172.21.15.62:0/2873825021 conn(0x7f83d00fc7d0 msgr2=0x7f83d00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.222+0000 7f83d51ee640 1 -- 172.21.15.62:0/2873825021 shutdown_connections 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.222+0000 7f83d51ee640 1 -- 172.21.15.62:0/2873825021 wait complete. 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.222+0000 7f83d51ee640 1 Processor -- start 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83d51ee640 1 -- start start 2024-09-15T23:24:42.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83d51ee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83d0100e20 0x7f83d0199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83d51ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d0102570 0x7f83d01997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83d51ee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 0x7f83d01a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83d51ee640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f83d0110f80 con 0x7f83d0102ef0 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83d51ee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83d0110e00 con 0x7f83d0102570 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83d51ee640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f83d0111100 con 0x7f83d0100e20 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83cf59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 0x7f83d01a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83cf59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 0x7f83d01a4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54662/0 (socket says 172.21.15.62:54662) 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83cf59e640 1 -- 172.21.15.62:0/1316825353 learned_addr learned my addr 172.21.15.62:0/1316825353 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83ced9d640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83d0100e20 0x7f83d0199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83ce59c640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d0102570 0x7f83d01997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.223+0000 7f83cf59e640 1 -- 172.21.15.62:0/1316825353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83d0100e20 msgr2=0x7f83d0199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83cf59e640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83d0100e20 0x7f83d0199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83cf59e640 1 -- 172.21.15.62:0/1316825353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d0102570 msgr2=0x7f83d01997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83cf59e640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d0102570 0x7f83d01997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83cf59e640 1 -- 172.21.15.62:0/1316825353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d01a4850 con 0x7f83d0102ef0 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83cf59e640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 0x7f83d01a4310 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f83c802ae70 tx=0x7f83c802f5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83b7fff640 1 -- 172.21.15.62:0/1316825353 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8034040 con 0x7f83d0102ef0 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83d51ee640 1 -- 172.21.15.62:0/1316825353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d01a4a20 con 0x7f83d0102ef0 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83b7fff640 1 -- 172.21.15.62:0/1316825353 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c8030920 con 0x7f83d0102ef0 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83b7fff640 1 -- 172.21.15.62:0/1316825353 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8045480 con 0x7f83d0102ef0 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.224+0000 7f83d51ee640 1 -- 172.21.15.62:0/1316825353 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d01a4e70 con 0x7f83d0102ef0 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.227+0000 7f83d75c7740 1 Processor -- start 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.227+0000 7f83b7fff640 1 -- 172.21.15.62:0/1316825353 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f83c8037020 con 0x7f83d0102ef0 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.227+0000 7f83d75c7740 1 -- start start 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.227+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa70b60 0x563ddfa70f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.227+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa7b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.227+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa7e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.227+0000 7f83d75c7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563ddf9d6280 con 0x563ddfa71480 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.227+0000 7f83d75c7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563ddf97db10 con 0x563ddfa70b60 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.227+0000 7f83d75c7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563ddf9d2240 con 0x563ddfa7bd40 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ced9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa7b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ced9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa7b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54672/0 (socket says 172.21.15.62:54672) 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ced9d640 1 -- 172.21.15.62:0/1181887490 learned_addr learned my addr 172.21.15.62:0/1181887490 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ce59c640 1 --2- 172.21.15.62:0/1181887490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa7e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ce59c640 1 --2- 172.21.15.62:0/1181887490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa70b60 0x563ddfa70f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ced9d640 1 -- 172.21.15.62:0/1181887490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 msgr2=0x563ddfa7e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ced9d640 1 --2- 172.21.15.62:0/1181887490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa7e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83b7fff640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83b80a3240 0x7f83b80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ced9d640 1 -- 172.21.15.62:0/1181887490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa70b60 msgr2=0x563ddfa70f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ced9d640 1 --2- 172.21.15.62:0/1181887490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa70b60 0x563ddfa70f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ced9d640 1 -- 172.21.15.62:0/1181887490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ddf98a1f0 con 0x563ddfa71480 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83cf59e640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83b80a3240 0x7f83b80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83ced9d640 1 --2- 172.21.15.62:0/1181887490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa7b800 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f83c000bcd0 tx=0x7f83c00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59608fd93f8743a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83b7fff640 1 -- 172.21.15.62:0/1316825353 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f83c80f0aa0 con 0x7f83d0102ef0 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83b57fa640 1 -- 172.21.15.62:0/1181887490 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c0005020 con 0x563ddfa71480 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83b57fa640 1 -- 172.21.15.62:0/1181887490 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c00047a0 con 0x563ddfa71480 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.228+0000 7f83b57fa640 1 -- 172.21.15.62:0/1181887490 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c0011730 con 0x563ddfa71480 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83d75c7740 1 -- 172.21.15.62:0/1181887490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 msgr2=0x563ddfa7b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83cf59e640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83b80a3240 0x7f83b80a5700 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f83c80bca40 tx=0x7f83c802a950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83d75c7740 1 --2- 172.21.15.62:0/1181887490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa7b800 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f83c000bcd0 tx=0x7f83c00107a0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83d75c7740 1 -- 172.21.15.62:0/1181887490 shutdown_connections 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83d75c7740 1 --2- 172.21.15.62:0/1181887490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa7e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83d75c7740 1 --2- 172.21.15.62:0/1181887490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa7b800 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83d75c7740 1 --2- 172.21.15.62:0/1181887490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa70b60 0x563ddfa70f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83d75c7740 1 -- 172.21.15.62:0/1181887490 >> 172.21.15.62:0/1181887490 conn(0x563ddfa6d0f0 msgr2=0x563ddfa6d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83d75c7740 1 -- 172.21.15.62:0/1181887490 shutdown_connections 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.229+0000 7f83d75c7740 1 -- 172.21.15.62:0/1181887490 wait complete. 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83d75c7740 1 -- start start 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83d75c7740 1 Processor -- start 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa78c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa79140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa79680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83d75c7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563ddfa7e910 con 0x563ddfa70b60 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83d75c7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563ddf990f10 con 0x563ddfa71480 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83d75c7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563ddfa7ea90 con 0x563ddfa7bd40 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83d75c7740 10 client.0 _ll_register_callbacks cb 0x563ddf959a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:42.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.230+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa78c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa78c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54684/0 (socket says 172.21.15.62:54684) 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ced9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa79140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ce59c640 1 -- 172.21.15.62:0/2364926261 learned_addr learned my addr 172.21.15.62:0/2364926261 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ce59c640 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa79680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ced9d640 1 -- 172.21.15.62:0/2364926261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 msgr2=0x563ddfa79680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ced9d640 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa79680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ced9d640 1 -- 172.21.15.62:0/2364926261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 msgr2=0x563ddfa78c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ced9d640 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa78c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ced9d640 1 -- 172.21.15.62:0/2364926261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ddf7476a0 con 0x563ddfa71480 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ce59c640 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa78c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83ced9d640 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa79140 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f83c0010ca0 tx=0x7f83c0004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2a83738d8e537fd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:24:42.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83b2ff5640 1 -- 172.21.15.62:0/2364926261 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c0004d50 con 0x563ddfa71480 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83d75c7740 1 -- 172.21.15.62:0/2364926261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83c00047a0 con 0x563ddfa71480 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.231+0000 7f83b2ff5640 1 -- 172.21.15.62:0/2364926261 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f83c0004020 con 0x563ddfa71480 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83b2ff5640 1 -- 172.21.15.62:0/2364926261 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f83c0004220 con 0x563ddfa71480 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83d75c7740 10 client.27020 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83d75c7740 1 -- 172.21.15.62:0/2364926261 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563ddfa8a4f0 con 0x563ddfa71480 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83b2ff5640 1 -- 172.21.15.62:0/2364926261 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c00047a0 con 0x563ddfa71480 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83b2ff5640 1 -- 172.21.15.62:0/2364926261 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f83c0003850 con 0x563ddfa71480 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83d75c7740 10 client.27020 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83d75c7740 10 client.27020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83d75c7740 20 client.27020 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:42.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563ddfa8b900 0x563ddfa8dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83d75c7740 4 client.27020 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83d75c7740 1 -- 172.21.15.62:0/2364926261 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ddf9e37f0 con 0x563ddfa8b900 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.232+0000 7f83ce59c640 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563ddfa8b900 0x563ddfa8dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.233+0000 7f83ce59c640 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563ddfa8b900 0x563ddfa8dcf0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd34ad974611d5a0 server_cookie=82c2cbe77aab49a in_seq=0 out_seq=0 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.233+0000 7f83b2ff5640 10 client.27020 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.233+0000 7f83b2ff5640 1 -- 172.21.15.62:0/2364926261 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563ddf9e37f0 con 0x563ddfa8b900 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.233+0000 7f83b2ff5640 10 client.27020 handle_command_reply: tid=0 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.233+0000 7f83d75c7740 1 client.27020 shutdown 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 -- 172.21.15.62:0/2364926261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 msgr2=0x563ddfa79140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa79140 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f83c0010ca0 tx=0x7f83c0004950 comp rx=0 tx=0).stop 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 -- 172.21.15.62:0/2364926261 shutdown_connections 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563ddfa8b900 0x563ddfa8dcf0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa79680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa79140 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83b2ff5640 0 client.27020 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:42.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2364926261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa78c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 -- 172.21.15.62:0/2364926261 >> 172.21.15.62:0/2364926261 conn(0x563ddfa6d0f0 msgr2=0x563ddfa6c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:42.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 -- 172.21.15.62:0/2364926261 shutdown_connections 2024-09-15T23:24:42.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 1 -- 172.21.15.62:0/2364926261 wait complete. 2024-09-15T23:24:42.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.234+0000 7f83d75c7740 20 client.27020 trim_cache size 0 max 16384 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83d75c7740 1 Processor -- start 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83d75c7740 1 -- start start 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa93290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa937d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83d75c7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f83d01b14b0 con 0x563ddfa70b60 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83d75c7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83d01aed80 con 0x563ddfa71480 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83d75c7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f83d01acbd0 con 0x563ddfa7bd40 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa93290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa93290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54694/0 (socket says 172.21.15.62:54694) 2024-09-15T23:24:42.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.252+0000 7f83ce59c640 1 -- 172.21.15.62:0/2311380485 learned_addr learned my addr 172.21.15.62:0/2311380485 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83ce59c640 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83ce59c640 1 -- 172.21.15.62:0/2311380485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 msgr2=0x563ddfa725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83ce59c640 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa725b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83ce59c640 1 -- 172.21.15.62:0/2311380485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 msgr2=0x563ddfa937d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83ced9d640 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa937d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83ce59c640 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa937d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83ce59c640 1 -- 172.21.15.62:0/2311380485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ddfa739d0 con 0x563ddfa70b60 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83ced9d640 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa937d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83ce59c640 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa93290 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f83c4005940 tx=0x7f83c400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43951a9ed5f65ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:42.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83b4ff9640 1 -- 172.21.15.62:0/2311380485 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c40054d0 con 0x563ddfa70b60 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83b4ff9640 1 -- 172.21.15.62:0/2311380485 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c4026040 con 0x563ddfa70b60 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.253+0000 7f83b4ff9640 1 -- 172.21.15.62:0/2311380485 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c40233e0 con 0x563ddfa70b60 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.254+0000 7f83d75c7740 1 -- 172.21.15.62:0/2311380485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 msgr2=0x563ddfa93290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.254+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa93290 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f83c4005940 tx=0x7f83c400b390 comp rx=0 tx=0).stop 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.254+0000 7f83d75c7740 1 -- 172.21.15.62:0/2311380485 shutdown_connections 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.254+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7bd40 0x563ddfa725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.254+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa71480 0x563ddfa937d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.254+0000 7f83d75c7740 1 --2- 172.21.15.62:0/2311380485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa70b60 0x563ddfa93290 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.254+0000 7f83d75c7740 1 -- 172.21.15.62:0/2311380485 >> 172.21.15.62:0/2311380485 conn(0x563ddfa6d0f0 msgr2=0x563ddfa7ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.254+0000 7f83d75c7740 1 -- 172.21.15.62:0/2311380485 shutdown_connections 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.254+0000 7f83d75c7740 1 -- 172.21.15.62:0/2311380485 wait complete. 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83d75c7740 1 -- start start 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83d75c7740 1 Processor -- start 2024-09-15T23:24:42.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa84010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa7bd40 0x563ddfa84550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83d75c7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7f2f0 0x563ddfa7f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83d75c7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f83d01ad240 con 0x563ddfa71480 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83d75c7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83d01ae6b0 con 0x563ddfa7bd40 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83d75c7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f83d01ad910 con 0x563ddfa7f2f0 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83d75c7740 10 client.0 _ll_register_callbacks cb 0x7f83c400bd80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa84010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83ce59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa84010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54710/0 (socket says 172.21.15.62:54710) 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83ce59c640 1 -- 172.21.15.62:0/3350525701 learned_addr learned my addr 172.21.15.62:0/3350525701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:42.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.255+0000 7f83ced9d640 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa7bd40 0x563ddfa84550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83ce59c640 1 -- 172.21.15.62:0/3350525701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7f2f0 msgr2=0x563ddfa7f730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:24:42.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83ce59c640 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7f2f0 0x563ddfa7f730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83ce59c640 1 -- 172.21.15.62:0/3350525701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa7bd40 msgr2=0x563ddfa84550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:42.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83ce59c640 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa7bd40 0x563ddfa84550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83ce59c640 1 -- 172.21.15.62:0/3350525701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ddfa7ff60 con 0x563ddfa71480 2024-09-15T23:24:42.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83ced9d640 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa7bd40 0x563ddfa84550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:42.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83ce59c640 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa84010 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x563ddfa7e150 tx=0x7f83c4003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dede218817696fe server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:42.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83b2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83b2ff5640 1 -- 172.21.15.62:0/3350525701 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c4004c00 con 0x563ddfa71480 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83d75c7740 1 -- 172.21.15.62:0/3350525701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83c4026040 con 0x563ddfa71480 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83b2ff5640 1 -- 172.21.15.62:0/3350525701 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f83c4004da0 con 0x563ddfa71480 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83b2ff5640 1 -- 172.21.15.62:0/3350525701 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f83c4041050 con 0x563ddfa71480 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.256+0000 7f83d75c7740 10 client.17460 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83d75c7740 1 -- 172.21.15.62:0/3350525701 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x563ddfafa5d0 con 0x563ddfa71480 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83b2ff5640 1 -- 172.21.15.62:0/3350525701 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c4026040 con 0x563ddfa71480 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83b2ff5640 1 -- 172.21.15.62:0/3350525701 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f83c4029070 con 0x563ddfa71480 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83d75c7740 10 client.17460 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83d75c7740 10 client.17460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83d75c7740 20 client.17460 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83d75c7740 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563ddfafab70 0x563ddfafcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:42.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83d75c7740 4 client.17460 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:24:42.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83d75c7740 1 -- 172.21.15.62:0/3350525701 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563ddf9ddd00 con 0x563ddfafab70 2024-09-15T23:24:42.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.257+0000 7f83ced9d640 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563ddfafab70 0x563ddfafcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:42.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.258+0000 7f83b2ff5640 10 client.17460 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:42.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.258+0000 7f83ced9d640 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563ddfafab70 0x563ddfafcf90 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9637c845f7d8d56 server_cookie=3e381da5b365471b in_seq=0 out_seq=0 2024-09-15T23:24:42.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83b2ff5640 1 -- 172.21.15.62:0/3350525701 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563ddf9ddd00 con 0x563ddfafab70 2024-09-15T23:24:42.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83b2ff5640 10 client.17460 handle_command_reply: tid=0 2024-09-15T23:24:42.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 client.17460 shutdown 2024-09-15T23:24:42.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 -- 172.21.15.62:0/3350525701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 msgr2=0x563ddfa84010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa84010 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x563ddfa7e150 tx=0x7f83c4003f70 comp rx=0 tx=0).stop 2024-09-15T23:24:42.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 -- 172.21.15.62:0/3350525701 shutdown_connections 2024-09-15T23:24:42.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563ddfafab70 0x563ddfafcf90 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ddfa7f2f0 0x563ddfa7f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ddfa7bd40 0x563ddfa84550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 --2- 172.21.15.62:0/3350525701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ddfa71480 0x563ddfa84010 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83b2ff5640 0 client.17460 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:42.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 -- 172.21.15.62:0/3350525701 >> 172.21.15.62:0/3350525701 conn(0x563ddfa6d0f0 msgr2=0x563ddfa72520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:42.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 -- 172.21.15.62:0/3350525701 shutdown_connections 2024-09-15T23:24:42.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.259+0000 7f83d75c7740 1 -- 172.21.15.62:0/3350525701 wait complete. 2024-09-15T23:24:42.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.260+0000 7f83d75c7740 20 client.17460 trim_cache size 0 max 16384 2024-09-15T23:24:42.267 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:24:42.268 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:24:42.268 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "cacad558-7e8d-47fe-97dd-190395d43f70", 2024-09-15T23:24:42.268 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:24:42.268 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.260+0000 7f83d51ee640 1 -- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83b80a3240 msgr2=0x7f83b80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.260+0000 7f83d51ee640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83b80a3240 0x7f83b80a5700 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f83c80bca40 tx=0x7f83c802a950 comp rx=0 tx=0).stop 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.260+0000 7f83d51ee640 1 -- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 msgr2=0x7f83d01a4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.260+0000 7f83d51ee640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 0x7f83d01a4310 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f83c802ae70 tx=0x7f83c802f5d0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.261+0000 7f83d51ee640 1 -- 172.21.15.62:0/1316825353 shutdown_connections 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.261+0000 7f83d51ee640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f83b80a3240 0x7f83b80a5700 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.261+0000 7f83d51ee640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f83d0102ef0 0x7f83d01a4310 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.261+0000 7f83d51ee640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83d0102570 0x7f83d01997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.261+0000 7f83d51ee640 1 --2- 172.21.15.62:0/1316825353 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f83d0100e20 0x7f83d0199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.261+0000 7f83d51ee640 1 -- 172.21.15.62:0/1316825353 >> 172.21.15.62:0/1316825353 conn(0x7f83d00fc7d0 msgr2=0x7f83d0106820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:42.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.261+0000 7f83d51ee640 1 -- 172.21.15.62:0/1316825353 shutdown_connections 2024-09-15T23:24:42.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:42.261+0000 7f83d51ee640 1 -- 172.21.15.62:0/1316825353 wait complete. 2024-09-15T23:24:42.965 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:42 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:24:42.966 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:42 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:42.966 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:42 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:24:42.966 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:42 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:24:42.966 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:42 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:42.966 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:42 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:24:42.966 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:42 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:42.966 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:42 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:42.995 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:24:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:42 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:24:43.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:42 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:43.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:42 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:24:43.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:42 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:24:43.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:42 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:43.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:42 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:24:43.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:42 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:43.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:42 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:42 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:24:43.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:42 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:43.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:42 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:24:43.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:42 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:24:43.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:42 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:43.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:42 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:24:43.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:42 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:43.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:42 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:43.326 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:43.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.960+0000 7fba155ca640 1 -- 172.21.15.62:0/3501063448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1011b170 msgr2=0x7fba1011d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.960+0000 7fba155ca640 1 --2- 172.21.15.62:0/3501063448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1011b170 0x7fba1011d560 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fba08009860 tx=0x7fba0802a340 comp rx=0 tx=0).stop 2024-09-15T23:24:43.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.961+0000 7fba155ca640 1 -- 172.21.15.62:0/3501063448 shutdown_connections 2024-09-15T23:24:43.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.961+0000 7fba155ca640 1 --2- 172.21.15.62:0/3501063448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1011b170 0x7fba1011d560 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.961+0000 7fba155ca640 1 --2- 172.21.15.62:0/3501063448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba1006e3c0 0x7fba1011ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.961+0000 7fba155ca640 1 --2- 172.21.15.62:0/3501063448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba1006daa0 0x7fba1006de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.961+0000 7fba155ca640 1 -- 172.21.15.62:0/3501063448 >> 172.21.15.62:0/3501063448 conn(0x7fba1006cc20 msgr2=0x7fba1006d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.962+0000 7fba155ca640 1 -- 172.21.15.62:0/3501063448 shutdown_connections 2024-09-15T23:24:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.962+0000 7fba155ca640 1 -- 172.21.15.62:0/3501063448 wait complete. 2024-09-15T23:24:43.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.963+0000 7fba155ca640 1 Processor -- start 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.963+0000 7fba155ca640 1 -- start start 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.963+0000 7fba155ca640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba1006daa0 0x7fba1011cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.963+0000 7fba155ca640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1006e3c0 0x7fba1011d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.963+0000 7fba155ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba10119190 0x7fba10119640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.963+0000 7fba155ca640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fba1010ee00 con 0x7fba1006daa0 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.963+0000 7fba155ca640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba1010ec80 con 0x7fba10119190 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.963+0000 7fba155ca640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fba1010ef80 con 0x7fba1006e3c0 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.964+0000 7fba0e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1006e3c0 0x7fba1011d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.964+0000 7fba0e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1006e3c0 0x7fba1011d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53930/0 (socket says 172.21.15.62:53930) 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.964+0000 7fba0e7fc640 1 -- 172.21.15.62:0/3730055508 learned_addr learned my addr 172.21.15.62:0/3730055508 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.964+0000 7fba0effd640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba1006daa0 0x7fba1011cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.964+0000 7fba0e7fc640 1 -- 172.21.15.62:0/3730055508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba10119190 msgr2=0x7fba10119640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.964+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba10119190 0x7fba10119640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.964+0000 7fba0e7fc640 1 -- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba1006daa0 msgr2=0x7fba1011cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.964+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba1006daa0 0x7fba1011cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.964+0000 7fba0e7fc640 1 -- 172.21.15.62:0/3730055508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba101aaf90 con 0x7fba1006e3c0 2024-09-15T23:24:43.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.965+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1006e3c0 0x7fba1011d4d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fba0000ba50 tx=0x7fba0000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.966+0000 7fba07fff640 1 -- 172.21.15.62:0/3730055508 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00019070 con 0x7fba1006e3c0 2024-09-15T23:24:43.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.966+0000 7fba07fff640 1 -- 172.21.15.62:0/3730055508 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba0000cbe0 con 0x7fba1006e3c0 2024-09-15T23:24:43.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.967+0000 7fba07fff640 1 -- 172.21.15.62:0/3730055508 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00007610 con 0x7fba1006e3c0 2024-09-15T23:24:43.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.967+0000 7fba155ca640 1 -- 172.21.15.62:0/3730055508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba101ab160 con 0x7fba1006e3c0 2024-09-15T23:24:43.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.968+0000 7fba155ca640 1 -- 172.21.15.62:0/3730055508 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba101ab560 con 0x7fba1006e3c0 2024-09-15T23:24:43.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.972+0000 7fba07fff640 1 -- 172.21.15.62:0/3730055508 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fba00014440 con 0x7fba1006e3c0 2024-09-15T23:24:43.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.974+0000 7fba179a3740 1 Processor -- start 2024-09-15T23:24:43.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.974+0000 7fba07fff640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9fc0a32c0 0x7fb9fc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.974+0000 7fba179a3740 1 -- start start 2024-09-15T23:24:43.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.975+0000 7fba07fff640 1 -- 172.21.15.62:0/3730055508 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fba000cf660 con 0x7fba1006e3c0 2024-09-15T23:24:43.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.975+0000 7fba0f7fe640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9fc0a32c0 0x7fb9fc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.975+0000 7fba179a3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3670 0x55566acc3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.975+0000 7fba179a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 0x55566acce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.975+0000 7fba0effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3670 0x55566acc3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.975+0000 7fba0effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3670 0x55566acc3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53936/0 (socket says 172.21.15.62:53936) 2024-09-15T23:24:43.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.976+0000 7fba0f7fe640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9fc0a32c0 0x7fb9fc0a5780 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fba08035040 tx=0x7fba08005760 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.976+0000 7fba179a3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acce850 0x55566acd0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.976+0000 7fba179a3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55566ac292e0 con 0x55566acce850 2024-09-15T23:24:43.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.976+0000 7fba179a3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55566abd0ce0 con 0x55566acc3f90 2024-09-15T23:24:43.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.976+0000 7fba179a3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55566ac252a0 con 0x55566acc3670 2024-09-15T23:24:43.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.976+0000 7fba0effd640 1 -- 172.21.15.62:0/2887982468 learned_addr learned my addr 172.21.15.62:0/2887982468 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:43.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.976+0000 7fba0f7fe640 1 --2- 172.21.15.62:0/2887982468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acce850 0x55566acd0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.977+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/2887982468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 0x55566acce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.978+0000 7fba0effd640 1 -- 172.21.15.62:0/2887982468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 msgr2=0x55566acce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.978+0000 7fba0effd640 1 --2- 172.21.15.62:0/2887982468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 0x55566acce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.978+0000 7fba0effd640 1 -- 172.21.15.62:0/2887982468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acce850 msgr2=0x55566acd0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.978+0000 7fba0effd640 1 --2- 172.21.15.62:0/2887982468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acce850 0x55566acd0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.978+0000 7fba0effd640 1 -- 172.21.15.62:0/2887982468 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55566abb5440 con 0x55566acc3670 2024-09-15T23:24:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.978+0000 7fba0effd640 1 --2- 172.21.15.62:0/2887982468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3670 0x55566acc3a50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb9f801ad30 tx=0x7fb9f80227c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab81b5f3739cf000 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.980+0000 7fba057fa640 1 -- 172.21.15.62:0/2887982468 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f802f070 con 0x55566acc3670 2024-09-15T23:24:43.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.980+0000 7fba057fa640 1 -- 172.21.15.62:0/2887982468 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9f801ae20 con 0x55566acc3670 2024-09-15T23:24:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.980+0000 7fba057fa640 1 -- 172.21.15.62:0/2887982468 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f802a9f0 con 0x55566acc3670 2024-09-15T23:24:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.981+0000 7fba179a3740 1 -- 172.21.15.62:0/2887982468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3670 msgr2=0x55566acc3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.981+0000 7fba179a3740 1 --2- 172.21.15.62:0/2887982468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3670 0x55566acc3a50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb9f801ad30 tx=0x7fb9f80227c0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.981+0000 7fba179a3740 1 -- 172.21.15.62:0/2887982468 shutdown_connections 2024-09-15T23:24:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.981+0000 7fba179a3740 1 --2- 172.21.15.62:0/2887982468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acce850 0x55566acd0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.981+0000 7fba179a3740 1 --2- 172.21.15.62:0/2887982468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 0x55566acce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.981+0000 7fba179a3740 1 --2- 172.21.15.62:0/2887982468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3670 0x55566acc3a50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.981+0000 7fba179a3740 1 -- 172.21.15.62:0/2887982468 >> 172.21.15.62:0/2887982468 conn(0x55566acbfc00 msgr2=0x55566acbffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.981+0000 7fba179a3740 1 -- 172.21.15.62:0/2887982468 shutdown_connections 2024-09-15T23:24:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.981+0000 7fba179a3740 1 -- 172.21.15.62:0/2887982468 wait complete. 2024-09-15T23:24:43.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.982+0000 7fba179a3740 1 -- start start 2024-09-15T23:24:43.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.982+0000 7fba179a3740 1 Processor -- start 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.982+0000 7fba179a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3670 0x55566acce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.982+0000 7fba179a3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566accb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.982+0000 7fba179a3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acce850 0x55566accb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.982+0000 7fba179a3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55566acd1420 con 0x55566acce850 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.982+0000 7fba179a3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55566abe4050 con 0x55566acc3670 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.982+0000 7fba179a3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55566acd15a0 con 0x55566acc3f90 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.982+0000 7fba179a3740 10 client.0 _ll_register_callbacks cb 0x55566ac24160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.983+0000 7fba0e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566accb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.983+0000 7fba0e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566accb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53946/0 (socket says 172.21.15.62:53946) 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.983+0000 7fba0e7fc640 1 -- 172.21.15.62:0/1965138009 learned_addr learned my addr 172.21.15.62:0/1965138009 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.983+0000 7fba0effd640 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3670 0x55566acce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.983+0000 7fba0e7fc640 1 -- 172.21.15.62:0/1965138009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3670 msgr2=0x55566acce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.983+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3670 0x55566acce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.983+0000 7fba0e7fc640 1 -- 172.21.15.62:0/1965138009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acce850 msgr2=0x55566accb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:43.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.983+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acce850 0x55566accb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.983+0000 7fba0e7fc640 1 -- 172.21.15.62:0/1965138009 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55566a99a3c0 con 0x55566acc3f90 2024-09-15T23:24:43.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.984+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566accb410 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fba00098270 tx=0x7fba0009fc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68563be9c9978143 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:43.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.984+0000 7fb9f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:24:43.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.984+0000 7fb9f67fc640 1 -- 172.21.15.62:0/1965138009 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00099c60 con 0x55566acc3f90 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.984+0000 7fba179a3740 1 -- 172.21.15.62:0/1965138009 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9f801ae20 con 0x55566acc3f90 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.984+0000 7fb9f67fc640 1 -- 172.21.15.62:0/1965138009 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fba00014730 con 0x55566acc3f90 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.984+0000 7fb9f67fc640 1 -- 172.21.15.62:0/1965138009 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fba000ba080 con 0x55566acc3f90 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.984+0000 7fba179a3740 10 client.26815 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.984+0000 7fba179a3740 1 -- 172.21.15.62:0/1965138009 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55566acdce00 con 0x55566acc3f90 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.984+0000 7fb9f67fc640 1 -- 172.21.15.62:0/1965138009 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9f801ae20 con 0x55566acc3f90 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.985+0000 7fb9f67fc640 1 -- 172.21.15.62:0/1965138009 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7fba000a4080 con 0x55566acc3f90 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.985+0000 7fba179a3740 10 client.26815 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.985+0000 7fba179a3740 10 client.26815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.985+0000 7fba179a3740 20 client.26815 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.985+0000 7fba179a3740 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55566acde2c0 0x55566ace0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:43.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.985+0000 7fba179a3740 4 client.26815 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:43.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.985+0000 7fba179a3740 1 -- 172.21.15.62:0/1965138009 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55566ac36850 con 0x55566acde2c0 2024-09-15T23:24:43.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.985+0000 7fba0effd640 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55566acde2c0 0x55566ace0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.986+0000 7fba0effd640 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55566acde2c0 0x55566ace0760 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19ebc8c625e5f91f server_cookie=32e1b50a34e36c07 in_seq=0 out_seq=0 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.986+0000 7fb9f67fc640 10 client.26815 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.987+0000 7fb9f67fc640 1 -- 172.21.15.62:0/1965138009 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55566ac36850 con 0x55566acde2c0 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.987+0000 7fb9f67fc640 10 client.26815 handle_command_reply: tid=0 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.987+0000 7fba179a3740 1 client.26815 shutdown 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.987+0000 7fba179a3740 1 -- 172.21.15.62:0/1965138009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 msgr2=0x55566accb410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.987+0000 7fba179a3740 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566accb410 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fba00098270 tx=0x7fba0009fc50 comp rx=0 tx=0).stop 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fba179a3740 1 -- 172.21.15.62:0/1965138009 shutdown_connections 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fba179a3740 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55566acde2c0 0x55566ace0760 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fba179a3740 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acce850 0x55566accb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fba179a3740 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566accb410 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fba179a3740 1 --2- 172.21.15.62:0/1965138009 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3670 0x55566acce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:43.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fba179a3740 1 -- 172.21.15.62:0/1965138009 >> 172.21.15.62:0/1965138009 conn(0x55566acbfc00 msgr2=0x55566acc58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:43.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fb9f67fc640 0 client.26815 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:43.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fba179a3740 1 -- 172.21.15.62:0/1965138009 shutdown_connections 2024-09-15T23:24:43.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fba179a3740 1 -- 172.21.15.62:0/1965138009 wait complete. 2024-09-15T23:24:43.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:43.988+0000 7fba179a3740 20 client.26815 trim_cache size 0 max 16384 2024-09-15T23:24:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:43 smithi067 ceph-mon[35665]: pgmap v623: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.1 MiB/s wr, 529 op/s 2024-09-15T23:24:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:43 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:24:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:43 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:43 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:24:44.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.023+0000 7fba179a3740 1 Processor -- start 2024-09-15T23:24:44.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.023+0000 7fba179a3740 1 -- start start 2024-09-15T23:24:44.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.023+0000 7fba179a3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acc3670 0x55566accd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:44.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.023+0000 7fba179a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 0x55566ace6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:44.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.023+0000 7fba179a3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acce850 0x55566ace6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:44.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba179a3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b7990 con 0x55566acc3670 2024-09-15T23:24:44.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba179a3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b5260 con 0x55566acc3f90 2024-09-15T23:24:44.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba179a3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b30b0 con 0x55566acce850 2024-09-15T23:24:44.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba0f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acce850 0x55566ace6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:44.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba0f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acce850 0x55566ace6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53958/0 (socket says 172.21.15.62:53958) 2024-09-15T23:24:44.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba0f7fe640 1 -- 172.21.15.62:0/1963914975 learned_addr learned my addr 172.21.15.62:0/1963914975 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:44.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/1963914975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 0x55566ace6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:44.027 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:43 smithi062 ceph-mon[30949]: pgmap v623: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.1 MiB/s wr, 529 op/s 2024-09-15T23:24:44.027 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:43 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:24:44.027 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:43 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:44.027 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:43 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:24:44.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba0e7fc640 1 -- 172.21.15.62:0/1963914975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acce850 msgr2=0x55566ace6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:44.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/1963914975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acce850 0x55566ace6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba0e7fc640 1 -- 172.21.15.62:0/1963914975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acc3670 msgr2=0x55566accd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:44.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/1963914975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acc3670 0x55566accd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.024+0000 7fba0e7fc640 1 -- 172.21.15.62:0/1963914975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55566accdd80 con 0x55566acc3f90 2024-09-15T23:24:44.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.025+0000 7fba0e7fc640 1 --2- 172.21.15.62:0/1963914975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 0x55566ace6030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fba00098220 tx=0x7fba00004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63bdc29ef240a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:44.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.025+0000 7fba04ff9640 1 -- 172.21.15.62:0/1963914975 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00097b40 con 0x55566acc3f90 2024-09-15T23:24:44.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.025+0000 7fba04ff9640 1 -- 172.21.15.62:0/1963914975 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba000b4070 con 0x55566acc3f90 2024-09-15T23:24:44.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.025+0000 7fba04ff9640 1 -- 172.21.15.62:0/1963914975 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba000090d0 con 0x55566acc3f90 2024-09-15T23:24:44.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.026+0000 7fba179a3740 1 -- 172.21.15.62:0/1963914975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 msgr2=0x55566ace6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:44.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.026+0000 7fba179a3740 1 --2- 172.21.15.62:0/1963914975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 0x55566ace6030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fba00098220 tx=0x7fba00004b50 comp rx=0 tx=0).stop 2024-09-15T23:24:44.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.026+0000 7fba179a3740 1 -- 172.21.15.62:0/1963914975 shutdown_connections 2024-09-15T23:24:44.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.026+0000 7fba179a3740 1 --2- 172.21.15.62:0/1963914975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acce850 0x55566ace6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.026+0000 7fba179a3740 1 --2- 172.21.15.62:0/1963914975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acc3f90 0x55566ace6030 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.026+0000 7fba179a3740 1 --2- 172.21.15.62:0/1963914975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acc3670 0x55566accd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.026+0000 7fba179a3740 1 -- 172.21.15.62:0/1963914975 >> 172.21.15.62:0/1963914975 conn(0x55566acbfc00 msgr2=0x55566acccc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:44.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.027+0000 7fba179a3740 1 -- 172.21.15.62:0/1963914975 shutdown_connections 2024-09-15T23:24:44.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.027+0000 7fba179a3740 1 -- 172.21.15.62:0/1963914975 wait complete. 2024-09-15T23:24:44.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.027+0000 7fba179a3740 1 -- start start 2024-09-15T23:24:44.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.027+0000 7fba179a3740 1 Processor -- start 2024-09-15T23:24:44.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.028+0000 7fba179a3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acc3670 0x55566acd4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:44.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.028+0000 7fba179a3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566acd4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:44.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.028+0000 7fba179a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acce850 0x55566acd4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:44.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.029+0000 7fba0e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566acd4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:44.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.029+0000 7fba0e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566acd4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53974/0 (socket says 172.21.15.62:53974) 2024-09-15T23:24:44.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.029+0000 7fba0f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acce850 0x55566acd4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:44.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.029+0000 7fba0f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acce850 0x55566acd4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42644/0 (socket says 172.21.15.62:42644) 2024-09-15T23:24:44.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.029+0000 7fba0e7fc640 1 -- 172.21.15.62:0/2489552824 learned_addr learned my addr 172.21.15.62:0/2489552824 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:44.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.029+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b3720 con 0x55566acc3670 2024-09-15T23:24:44.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.029+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b4b90 con 0x55566acce850 2024-09-15T23:24:44.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.029+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fba101b3df0 con 0x55566acc3f90 2024-09-15T23:24:44.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.029+0000 7fba179a3740 10 client.0 _ll_register_callbacks cb 0x7fba0009b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:44.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.030+0000 7fba0effd640 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acc3670 0x55566acd4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:44.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.030+0000 7fba0f7fe640 1 -- 172.21.15.62:0/2489552824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 msgr2=0x55566acd4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:44.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.030+0000 7fba0f7fe640 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566acd4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.030+0000 7fba0f7fe640 1 -- 172.21.15.62:0/2489552824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acc3670 msgr2=0x55566acd4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:44.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.030+0000 7fba0f7fe640 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acc3670 0x55566acd4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.030+0000 7fba0f7fe640 1 -- 172.21.15.62:0/2489552824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55566ad48010 con 0x55566acce850 2024-09-15T23:24:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.031+0000 7fba0f7fe640 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acce850 0x55566acd4fc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fba0802a930 tx=0x7fba08004180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40a460b14cffcf11 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.031+0000 7fb9f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:24:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.031+0000 7fb9f67fc640 1 -- 172.21.15.62:0/2489552824 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba08037040 con 0x55566acce850 2024-09-15T23:24:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.031+0000 7fb9f67fc640 1 -- 172.21.15.62:0/2489552824 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fba08030750 con 0x55566acce850 2024-09-15T23:24:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.031+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba000b4070 con 0x55566acce850 2024-09-15T23:24:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.032+0000 7fb9f67fc640 1 -- 172.21.15.62:0/2489552824 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fba080035f0 con 0x55566acce850 2024-09-15T23:24:44.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.032+0000 7fba179a3740 10 client.27038 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:44.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.032+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55566ad492d0 con 0x55566acce850 2024-09-15T23:24:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.035+0000 7fb9f67fc640 1 -- 172.21.15.62:0/2489552824 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba08003c70 con 0x55566acce850 2024-09-15T23:24:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.035+0000 7fb9f67fc640 1 -- 172.21.15.62:0/2489552824 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7fba0803a030 con 0x55566acce850 2024-09-15T23:24:44.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.035+0000 7fba179a3740 10 client.27038 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.035+0000 7fba179a3740 10 client.27038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.035+0000 7fba179a3740 20 client.27038 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.035+0000 7fba179a3740 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55566ad49a50 0x55566ad4be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.035+0000 7fba179a3740 4 client.27038 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:24:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.035+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fba000b4070 con 0x55566ad49a50 2024-09-15T23:24:44.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.037+0000 7fba0effd640 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55566ad49a50 0x55566ad4be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:43 smithi148 ceph-mon[35610]: pgmap v623: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.1 MiB/s wr, 529 op/s 2024-09-15T23:24:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:43 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:24:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:43 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:24:44.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:43 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.038+0000 7fba0effd640 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55566ad49a50 0x55566ad4be70 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d99ed5952f5d593 server_cookie=a8a0fbe99f740828 in_seq=0 out_seq=0 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.038+0000 7fb9f67fc640 10 client.27038 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.038+0000 7fb9f67fc640 1 -- 172.21.15.62:0/2489552824 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fba000b4070 con 0x55566ad49a50 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.038+0000 7fb9f67fc640 10 client.27038 handle_command_reply: tid=0 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.039+0000 7fba179a3740 1 client.27038 shutdown 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.039+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acce850 msgr2=0x55566acd4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.039+0000 7fba179a3740 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acce850 0x55566acd4fc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fba0802a930 tx=0x7fba08004180 comp rx=0 tx=0).stop 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.039+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 shutdown_connections 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.039+0000 7fba179a3740 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55566ad49a50 0x55566ad4be70 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.039+0000 7fb9f67fc640 0 client.27038 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.039+0000 7fba179a3740 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55566acce850 0x55566acd4fc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.039+0000 7fba179a3740 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55566acc3f90 0x55566acd4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.039+0000 7fba179a3740 1 --2- 172.21.15.62:0/2489552824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55566acc3670 0x55566acd4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.040+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 >> 172.21.15.62:0/2489552824 conn(0x55566acbfc00 msgr2=0x55566acbed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:44.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.040+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 shutdown_connections 2024-09-15T23:24:44.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.040+0000 7fba179a3740 1 -- 172.21.15.62:0/2489552824 wait complete. 2024-09-15T23:24:44.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.040+0000 7fba179a3740 20 client.27038 trim_cache size 0 max 16384 2024-09-15T23:24:44.044 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:24:44.044 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:24:44.044 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:24:44.044 INFO:teuthology.orchestra.run.smithi062.stdout: "cacad558-7e8d-47fe-97dd-190395d43f70": { 2024-09-15T23:24:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:24:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "cacad558-7e8d-47fe-97dd-190395d43f70", 2024-09-15T23:24:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:24:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:44.045 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:44.045 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 -- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9fc0a32c0 msgr2=0x7fb9fc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9fc0a32c0 0x7fb9fc0a5780 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fba08035040 tx=0x7fba08005760 comp rx=0 tx=0).stop 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 -- 172.21.15.62:0/3730055508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1006e3c0 msgr2=0x7fba1011d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1006e3c0 0x7fba1011d4d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fba0000ba50 tx=0x7fba0000bf10 comp rx=0 tx=0).stop 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 -- 172.21.15.62:0/3730055508 shutdown_connections 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9fc0a32c0 0x7fb9fc0a5780 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba10119190 0x7fba10119640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fba1006e3c0 0x7fba1011d4d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 --2- 172.21.15.62:0/3730055508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba1006daa0 0x7fba1011cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.042+0000 7fba155ca640 1 -- 172.21.15.62:0/3730055508 >> 172.21.15.62:0/3730055508 conn(0x7fba1006cc20 msgr2=0x7fba1010b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.043+0000 7fba155ca640 1 -- 172.21.15.62:0/3730055508 shutdown_connections 2024-09-15T23:24:44.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:44.043+0000 7fba155ca640 1 -- 172.21.15.62:0/3730055508 wait complete. 2024-09-15T23:24:44.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 505.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:24:44.743 INFO:tasks.cephfs.filesystem:scrub status for tag:cacad558-7e8d-47fe-97dd-190395d43f70 - {'path': '/', 'tag': 'cacad558-7e8d-47fe-97dd-190395d43f70', 'options': 'recursive,force'} 2024-09-15T23:24:45.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:24:45.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:24:45.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:44 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:24:45.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:24:45.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:24:45.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:44 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:24:45.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:24:45.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:24:45.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:44 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:24:45.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:45.822 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:45.944 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:45 smithi062 ceph-mon[30949]: pgmap v624: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.1 MiB/s wr, 650 op/s 2024-09-15T23:24:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:45 smithi067 ceph-mon[35665]: pgmap v624: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.1 MiB/s wr, 650 op/s 2024-09-15T23:24:46.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:45 smithi148 ceph-mon[35610]: pgmap v624: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 7.1 MiB/s wr, 650 op/s 2024-09-15T23:24:46.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.297+0000 7f7e79830640 1 -- 172.21.15.62:0/3378414490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741036e0 msgr2=0x7f7e74107b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.297+0000 7f7e79830640 1 --2- 172.21.15.62:0/3378414490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741036e0 0x7f7e74107b90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009930 tx=0x7f7e5c02a420 comp rx=0 tx=0).stop 2024-09-15T23:24:46.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.297+0000 7f7e79830640 1 -- 172.21.15.62:0/3378414490 shutdown_connections 2024-09-15T23:24:46.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.297+0000 7f7e79830640 1 --2- 172.21.15.62:0/3378414490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e741082a0 0x7f7e7410ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.297+0000 7f7e79830640 1 --2- 172.21.15.62:0/3378414490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741036e0 0x7f7e74107b90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.297+0000 7f7e79830640 1 --2- 172.21.15.62:0/3378414490 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e74102d30 0x7f7e74103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.298+0000 7f7e79830640 1 -- 172.21.15.62:0/3378414490 >> 172.21.15.62:0/3378414490 conn(0x7f7e740fc7d0 msgr2=0x7f7e740febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:46.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.298+0000 7f7e79830640 1 -- 172.21.15.62:0/3378414490 shutdown_connections 2024-09-15T23:24:46.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.298+0000 7f7e79830640 1 -- 172.21.15.62:0/3378414490 wait complete. 2024-09-15T23:24:46.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.298+0000 7f7e79830640 1 Processor -- start 2024-09-15T23:24:46.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.298+0000 7f7e79830640 1 -- start start 2024-09-15T23:24:46.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e79830640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e74102d30 0x7f7e74100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e79830640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e741036e0 0x7f7e74100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e79830640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741082a0 0x7f7e74101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e79830640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7e741120c0 con 0x7f7e741082a0 2024-09-15T23:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e79830640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74111f40 con 0x7f7e741036e0 2024-09-15T23:24:46.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e79830640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7e74112240 con 0x7f7e74102d30 2024-09-15T23:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e737fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741082a0 0x7f7e74101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e737fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741082a0 0x7f7e74101410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39342/0 (socket says 172.21.15.62:39342) 2024-09-15T23:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e741036e0 0x7f7e74100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e737fe640 1 -- 172.21.15.62:0/837174241 learned_addr learned my addr 172.21.15.62:0/837174241 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e72ffd640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e74102d30 0x7f7e74100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e737fe640 1 -- 172.21.15.62:0/837174241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e74102d30 msgr2=0x7f7e74100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e737fe640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e74102d30 0x7f7e74100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e737fe640 1 -- 172.21.15.62:0/837174241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e741036e0 msgr2=0x7f7e74100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e737fe640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e741036e0 0x7f7e74100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e737fe640 1 -- 172.21.15.62:0/837174241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e741a6ac0 con 0x7f7e741082a0 2024-09-15T23:24:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e727fc640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e741036e0 0x7f7e74100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e72ffd640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e74102d30 0x7f7e74100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e737fe640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741082a0 0x7f7e74101410 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400eaa0 tx=0x7f7e6400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.299+0000 7f7e7882e640 1 -- 172.21.15.62:0/837174241 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6400cdb0 con 0x7f7e741082a0 2024-09-15T23:24:46.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.300+0000 7f7e7882e640 1 -- 172.21.15.62:0/837174241 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e64005020 con 0x7f7e741082a0 2024-09-15T23:24:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.300+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e741a6db0 con 0x7f7e741082a0 2024-09-15T23:24:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.300+0000 7f7e7882e640 1 -- 172.21.15.62:0/837174241 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e640116b0 con 0x7f7e741082a0 2024-09-15T23:24:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.300+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e741a72c0 con 0x7f7e741082a0 2024-09-15T23:24:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.301+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e74103110 con 0x7f7e741082a0 2024-09-15T23:24:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.302+0000 7f7e7882e640 1 -- 172.21.15.62:0/837174241 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7e64005560 con 0x7f7e741082a0 2024-09-15T23:24:46.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.303+0000 7f7e7882e640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7e600a32c0 0x7f7e600a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.303+0000 7f7e72ffd640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7e600a32c0 0x7f7e600a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.303+0000 7f7e7882e640 1 -- 172.21.15.62:0/837174241 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7e640cfe80 con 0x7f7e741082a0 2024-09-15T23:24:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.303+0000 7f7e72ffd640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7e600a32c0 0x7f7e600a5780 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009930 tx=0x7f7e6c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:46.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.307+0000 7f7e7882e640 1 -- 172.21.15.62:0/837174241 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7e64016030 con 0x7f7e741082a0 2024-09-15T23:24:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.516+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e7410a2b0 con 0x7f7e741082a0 2024-09-15T23:24:46.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.517+0000 7f7e7882e640 1 -- 172.21.15.62:0/837174241 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11904 (secure 0 0 0) 0x7f7e6409c820 con 0x7f7e741082a0 2024-09-15T23:24:46.520 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:24:46.520 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":34,"btime":"2024-09-15T23:24:12:668733+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:12.388349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:24:46.520 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 34 2024-09-15T23:24:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7e600a32c0 msgr2=0x7f7e600a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7e600a32c0 0x7f7e600a5780 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009930 tx=0x7f7e6c008040 comp rx=0 tx=0).stop 2024-09-15T23:24:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741082a0 msgr2=0x7f7e74101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:46.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741082a0 0x7f7e74101410 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400eaa0 tx=0x7f7e6400ef60 comp rx=0 tx=0).stop 2024-09-15T23:24:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 shutdown_connections 2024-09-15T23:24:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7e600a32c0 0x7f7e600a5780 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e741082a0 0x7f7e74101410 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e741036e0 0x7f7e74100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 --2- 172.21.15.62:0/837174241 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7e74102d30 0x7f7e74100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 >> 172.21.15.62:0/837174241 conn(0x7f7e740fc7d0 msgr2=0x7f7e74105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 shutdown_connections 2024-09-15T23:24:46.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:46.519+0000 7f7e79830640 1 -- 172.21.15.62:0/837174241 wait complete. 2024-09-15T23:24:47.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:46 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/837174241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:47.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:46 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/837174241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:47.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:46 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/837174241' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:47.255 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 83c3dbcf --release --await' 2024-09-15T23:24:47.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 83c3dbcf --release --await 2024-09-15T23:24:47.514 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:47.903 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:47 smithi062 ceph-mon[30949]: pgmap v625: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 6.5 MiB/s wr, 641 op/s 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 -- 172.21.15.62:0/4000639967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35541036c0 msgr2=0x7f355410bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 --2- 172.21.15.62:0/4000639967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35541036c0 0x7f355410bbb0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3548009900 tx=0x7f354802a480 comp rx=0 tx=0).stop 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 -- 172.21.15.62:0/4000639967 shutdown_connections 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 --2- 172.21.15.62:0/4000639967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f355410c180 0x7f355410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 --2- 172.21.15.62:0/4000639967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35541036c0 0x7f355410bbb0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 --2- 172.21.15.62:0/4000639967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3554102da0 0x7f3554103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 -- 172.21.15.62:0/4000639967 >> 172.21.15.62:0/4000639967 conn(0x7f35540fc7a0 msgr2=0x7f35540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 -- 172.21.15.62:0/4000639967 shutdown_connections 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 -- 172.21.15.62:0/4000639967 wait complete. 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 Processor -- start 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.991+0000 7f35596b9640 1 -- start start 2024-09-15T23:24:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f35596b9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3554102da0 0x7f3554100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f35596b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35541036c0 0x7f3554100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f35596b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355410c180 0x7f3554101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f35596b9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3554111f60 con 0x7f3554102da0 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f35596b9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3554111de0 con 0x7f35541036c0 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f35596b9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f35541120e0 con 0x7f355410c180 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f3552ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3554102da0 0x7f3554100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f3552ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3554102da0 0x7f3554100a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39360/0 (socket says 172.21.15.62:39360) 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f3552ffd640 1 -- 172.21.15.62:0/4165481668 learned_addr learned my addr 172.21.15.62:0/4165481668 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f35527fc640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35541036c0 0x7f3554100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f35537fe640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355410c180 0x7f3554101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f3552ffd640 1 -- 172.21.15.62:0/4165481668 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355410c180 msgr2=0x7f3554101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f3552ffd640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355410c180 0x7f3554101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f3552ffd640 1 -- 172.21.15.62:0/4165481668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35541036c0 msgr2=0x7f3554100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f3552ffd640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35541036c0 0x7f3554100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f3552ffd640 1 -- 172.21.15.62:0/4165481668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35541a68e0 con 0x7f3554102da0 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f3552ffd640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3554102da0 0x7f3554100a00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f354400d770 tx=0x7f354400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f353bfff640 1 -- 172.21.15.62:0/4165481668 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544017070 con 0x7f3554102da0 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f353bfff640 1 -- 172.21.15.62:0/4165481668 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3544005500 con 0x7f3554102da0 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f35596b9640 1 -- 172.21.15.62:0/4165481668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35541a6bd0 con 0x7f3554102da0 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.992+0000 7f353bfff640 1 -- 172.21.15.62:0/4165481668 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544004e00 con 0x7f3554102da0 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.993+0000 7f35596b9640 1 -- 172.21.15.62:0/4165481668 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35541a70c0 con 0x7f3554102da0 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.995+0000 7f353bfff640 1 -- 172.21.15.62:0/4165481668 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3544004fa0 con 0x7f3554102da0 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f355ba92740 1 Processor -- start 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f355ba92740 1 -- start start 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f355ba92740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 0x55aedf964440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f355ba92740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964980 0x55aedf96ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f355ba92740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf96f240 0x55aedf971630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f355ba92740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55aedf8c9620 con 0x55aedf964980 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f355ba92740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55aedf870e50 con 0x55aedf96f240 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f355ba92740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55aedf8c55e0 con 0x55aedf964060 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f3552ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964980 0x55aedf96ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f353bfff640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f353c0a3240 0x7f353c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f3552ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964980 0x55aedf96ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39368/0 (socket says 172.21.15.62:39368) 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f3552ffd640 1 -- 172.21.15.62:0/2768335392 learned_addr learned my addr 172.21.15.62:0/2768335392 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f35527fc640 1 --2- 172.21.15.62:0/2768335392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf96f240 0x55aedf971630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f35537fe640 1 --2- 172.21.15.62:0/2768335392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 0x55aedf964440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f35537fe640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f353c0a3240 0x7f353c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.996+0000 7f353bfff640 1 -- 172.21.15.62:0/4165481668 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f35440cf7c0 con 0x7f3554102da0 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f3552ffd640 1 -- 172.21.15.62:0/2768335392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 msgr2=0x55aedf964440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f3552ffd640 1 --2- 172.21.15.62:0/2768335392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 0x55aedf964440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f3552ffd640 1 -- 172.21.15.62:0/2768335392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf96f240 msgr2=0x55aedf971630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f3552ffd640 1 --2- 172.21.15.62:0/2768335392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf96f240 0x55aedf971630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f3552ffd640 1 -- 172.21.15.62:0/2768335392 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aedf861d40 con 0x55aedf964980 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f3552ffd640 1 --2- 172.21.15.62:0/2768335392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964980 0x55aedf96ed00 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f35440cce80 tx=0x7f3544009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99d0ac4c3eb5ba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f35537fe640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f353c0a3240 0x7f353c0a5700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f354c005c40 tx=0x7f354c005bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f35397fa640 1 -- 172.21.15.62:0/2768335392 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544020070 con 0x55aedf964980 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f35397fa640 1 -- 172.21.15.62:0/2768335392 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f354409c160 con 0x55aedf964980 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.997+0000 7f35397fa640 1 -- 172.21.15.62:0/2768335392 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544009cc0 con 0x55aedf964980 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 -- 172.21.15.62:0/2768335392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964980 msgr2=0x55aedf96ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 --2- 172.21.15.62:0/2768335392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964980 0x55aedf96ed00 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f35440cce80 tx=0x7f3544009c30 comp rx=0 tx=0).stop 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 -- 172.21.15.62:0/2768335392 shutdown_connections 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 --2- 172.21.15.62:0/2768335392 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf96f240 0x55aedf971630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 --2- 172.21.15.62:0/2768335392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964980 0x55aedf96ed00 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 --2- 172.21.15.62:0/2768335392 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 0x55aedf964440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 -- 172.21.15.62:0/2768335392 >> 172.21.15.62:0/2768335392 conn(0x55aedf9605f0 msgr2=0x55aedf9609d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 -- 172.21.15.62:0/2768335392 shutdown_connections 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 -- 172.21.15.62:0/2768335392 wait complete. 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 -- start start 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 Processor -- start 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 0x55aedf96c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 0x55aedf96c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf96cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55aedf971e10 con 0x55aedf96f240 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55aedf851f10 con 0x55aedf964980 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55aedf971f90 con 0x55aedf964060 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.998+0000 7f355ba92740 10 client.0 _ll_register_callbacks cb 0x55aedf875db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35537fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf96cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35537fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf96cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39376/0 (socket says 172.21.15.62:39376) 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35537fe640 1 -- 172.21.15.62:0/3230706753 learned_addr learned my addr 172.21.15.62:0/3230706753 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f3552ffd640 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 0x55aedf96c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35527fc640 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 0x55aedf96c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35537fe640 1 -- 172.21.15.62:0/3230706753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 msgr2=0x55aedf96c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35537fe640 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 0x55aedf96c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35537fe640 1 -- 172.21.15.62:0/3230706753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 msgr2=0x55aedf96c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35537fe640 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 0x55aedf96c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35537fe640 1 -- 172.21.15.62:0/3230706753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aedf5d8580 con 0x55aedf96f240 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f3552ffd640 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 0x55aedf96c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f35537fe640 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf96cbd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f354c00dac0 tx=0x7f354c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3736a6751b13 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f3536ff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f3536ff5640 1 -- 172.21.15.62:0/3230706753 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c010ca0 con 0x55aedf96f240 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f3536ff5640 1 -- 172.21.15.62:0/3230706753 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f354c020440 con 0x55aedf96f240 2024-09-15T23:24:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f355ba92740 1 -- 172.21.15.62:0/3230706753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f354409c160 con 0x55aedf96f240 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f3536ff5640 1 -- 172.21.15.62:0/3230706753 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f354c032050 con 0x55aedf96f240 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f355ba92740 10 client.17487 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:47.999+0000 7f355ba92740 1 -- 172.21.15.62:0/3230706753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55aedf6a2960 con 0x55aedf96f240 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.000+0000 7f3536ff5640 1 -- 172.21.15.62:0/3230706753 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f354409c160 con 0x55aedf96f240 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.000+0000 7f3536ff5640 1 -- 172.21.15.62:0/3230706753 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f354c02a530 con 0x55aedf96f240 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.000+0000 7f355ba92740 10 client.17487 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.000+0000 7f355ba92740 10 client.17487 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.000+0000 7f355ba92740 20 client.17487 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.000+0000 7f355ba92740 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55aedf97ee30 0x55aedf981280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.000+0000 7f355ba92740 4 client.17487 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.000+0000 7f355ba92740 1 -- 172.21.15.62:0/3230706753 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aedf8d6b90 con 0x55aedf97ee30 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.000+0000 7f3552ffd640 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55aedf97ee30 0x55aedf981280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.001+0000 7f3536ff5640 10 client.17487 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.001+0000 7f3552ffd640 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55aedf97ee30 0x55aedf981280 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d431e9ad6104becd server_cookie=de233a4cbd07b59b in_seq=0 out_seq=0 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.001+0000 7f3536ff5640 1 -- 172.21.15.62:0/3230706753 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aedf8d6b90 con 0x55aedf97ee30 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.001+0000 7f3536ff5640 10 client.17487 handle_command_reply: tid=0 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 client.17487 shutdown 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 -- 172.21.15.62:0/3230706753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 msgr2=0x55aedf96cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf96cbd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f354c00dac0 tx=0x7f354c004a40 comp rx=0 tx=0).stop 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 -- 172.21.15.62:0/3230706753 shutdown_connections 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55aedf97ee30 0x55aedf981280 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf96cbd0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 0x55aedf96c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f3536ff5640 0 client.17487 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 --2- 172.21.15.62:0/3230706753 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964060 0x55aedf96c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 -- 172.21.15.62:0/3230706753 >> 172.21.15.62:0/3230706753 conn(0x55aedf9605f0 msgr2=0x55aedf9662d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 -- 172.21.15.62:0/3230706753 shutdown_connections 2024-09-15T23:24:48.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 1 -- 172.21.15.62:0/3230706753 wait complete. 2024-09-15T23:24:48.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.002+0000 7f355ba92740 20 client.17487 trim_cache size 0 max 16384 2024-09-15T23:24:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f355ba92740 1 Processor -- start 2024-09-15T23:24:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f355ba92740 1 -- start start 2024-09-15T23:24:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f355ba92740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964060 0x55aedf966ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f355ba92740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 0x55aedf95f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f355ba92740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf96f240 0x55aedf986b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f355ba92740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f35541b36f0 con 0x55aedf964060 2024-09-15T23:24:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f355ba92740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f35541b0fc0 con 0x55aedf964980 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f355ba92740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f35541aee10 con 0x55aedf96f240 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f3552ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964060 0x55aedf966ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f3552ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964060 0x55aedf966ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39378/0 (socket says 172.21.15.62:39378) 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.019+0000 7f3552ffd640 1 -- 172.21.15.62:0/3477022710 learned_addr learned my addr 172.21.15.62:0/3477022710 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f35527fc640 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 0x55aedf95f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f35537fe640 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf96f240 0x55aedf986b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f3552ffd640 1 -- 172.21.15.62:0/3477022710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf96f240 msgr2=0x55aedf986b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f3552ffd640 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf96f240 0x55aedf986b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f3552ffd640 1 -- 172.21.15.62:0/3477022710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 msgr2=0x55aedf95f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f3552ffd640 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 0x55aedf95f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f3552ffd640 1 -- 172.21.15.62:0/3477022710 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aedf960090 con 0x55aedf964060 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f35527fc640 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 0x55aedf95f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f3552ffd640 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964060 0x55aedf966ac0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f354409b760 tx=0x7f3544086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a5dc992bd50c3a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f3538ff9640 1 -- 172.21.15.62:0/3477022710 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3544020040 con 0x55aedf964060 2024-09-15T23:24:48.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f3538ff9640 1 -- 172.21.15.62:0/3477022710 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f354409c160 con 0x55aedf964060 2024-09-15T23:24:48.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.020+0000 7f3538ff9640 1 -- 172.21.15.62:0/3477022710 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35440980d0 con 0x55aedf964060 2024-09-15T23:24:48.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:47 smithi067 ceph-mon[35665]: pgmap v625: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 6.5 MiB/s wr, 641 op/s 2024-09-15T23:24:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 -- 172.21.15.62:0/3477022710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964060 msgr2=0x55aedf966ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964060 0x55aedf966ac0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f354409b760 tx=0x7f3544086040 comp rx=0 tx=0).stop 2024-09-15T23:24:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 -- 172.21.15.62:0/3477022710 shutdown_connections 2024-09-15T23:24:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf96f240 0x55aedf986b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964980 0x55aedf95f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 --2- 172.21.15.62:0/3477022710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf964060 0x55aedf966ac0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 -- 172.21.15.62:0/3477022710 >> 172.21.15.62:0/3477022710 conn(0x55aedf9605f0 msgr2=0x55aedf95ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 -- 172.21.15.62:0/3477022710 shutdown_connections 2024-09-15T23:24:48.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 -- 172.21.15.62:0/3477022710 wait complete. 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 -- start start 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.021+0000 7f355ba92740 1 Processor -- start 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f355ba92740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964060 0x55aedf973630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f355ba92740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964980 0x55aedf973b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f355ba92740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf9e8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f355ba92740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f35541af480 con 0x55aedf96f240 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f355ba92740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f35541b08f0 con 0x55aedf964060 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f355ba92740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f35541afb50 con 0x55aedf964980 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f355ba92740 10 client.0 _ll_register_callbacks cb 0x7f354c0127e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35537fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf9e8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35537fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf9e8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39394/0 (socket says 172.21.15.62:39394) 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35537fe640 1 -- 172.21.15.62:0/1722671904 learned_addr learned my addr 172.21.15.62:0/1722671904 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:48.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f3552ffd640 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964060 0x55aedf973630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35527fc640 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964980 0x55aedf973b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35537fe640 1 -- 172.21.15.62:0/1722671904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964980 msgr2=0x55aedf973b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35537fe640 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964980 0x55aedf973b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35537fe640 1 -- 172.21.15.62:0/1722671904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964060 msgr2=0x55aedf973630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35537fe640 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964060 0x55aedf973630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35537fe640 1 -- 172.21.15.62:0/1722671904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aedf9e9060 con 0x55aedf96f240 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f3552ffd640 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964060 0x55aedf973630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f35537fe640 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf9e8b20 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f354c02a980 tx=0x7f354c0098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3533747cb5b4c7b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.022+0000 7f3536ff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f3536ff5640 1 -- 172.21.15.62:0/1722671904 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c010820 con 0x55aedf96f240 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f3536ff5640 1 -- 172.21.15.62:0/1722671904 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f354c02ab70 con 0x55aedf96f240 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f355ba92740 1 -- 172.21.15.62:0/1722671904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f354409c160 con 0x55aedf96f240 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f3536ff5640 1 -- 172.21.15.62:0/1722671904 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f354c044050 con 0x55aedf96f240 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f355ba92740 10 client.17493 fetch_fsmap learned FSMap version 34 2024-09-15T23:24:48.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f355ba92740 1 -- 172.21.15.62:0/1722671904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55aedf9ea340 con 0x55aedf96f240 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f3536ff5640 1 -- 172.21.15.62:0/1722671904 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f354c03b070 con 0x55aedf96f240 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f3536ff5640 1 -- 172.21.15.62:0/1722671904 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 34) ==== 8358+0+0 (secure 0 0 0) 0x7f354c005160 con 0x55aedf96f240 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f355ba92740 10 client.17493 fetch_fsmap finished waiting for FSMap version 34 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f355ba92740 10 client.17493 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f355ba92740 20 client.17493 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f355ba92740 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55aedf9ea990 0x55aedf9ecdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f355ba92740 4 client.17493 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "83c3dbcf", "release": true, "await": true}] 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.023+0000 7f355ba92740 1 -- 172.21.15.62:0/1722671904 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "83c3dbcf", "release": true, "await": true}) -- 0x7f354409c160 con 0x55aedf9ea990 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.024+0000 7f3552ffd640 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55aedf9ea990 0x55aedf9ecdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.024+0000 7f3552ffd640 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55aedf9ea990 0x55aedf9ecdb0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e8c1d52e78450c1 server_cookie=4f82b5935eb44f4b in_seq=0 out_seq=0 2024-09-15T23:24:48.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.024+0000 7f3536ff5640 10 client.17493 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:48.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:47 smithi148 ceph-mon[35610]: pgmap v625: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 6.5 MiB/s wr, 641 op/s 2024-09-15T23:24:48.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.060+0000 7f3536ff5640 1 -- 172.21.15.62:0/1722671904 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f354409c160 con 0x55aedf9ea990 2024-09-15T23:24:48.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.060+0000 7f3536ff5640 10 client.17493 handle_command_reply: tid=0 2024-09-15T23:24:48.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.060+0000 7f355ba92740 1 client.17493 shutdown 2024-09-15T23:24:48.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 -- 172.21.15.62:0/1722671904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 msgr2=0x55aedf9e8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf9e8b20 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f354c02a980 tx=0x7f354c0098c0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 -- 172.21.15.62:0/1722671904 shutdown_connections 2024-09-15T23:24:48.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55aedf9ea990 0x55aedf9ecdb0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aedf96f240 0x55aedf9e8b20 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aedf964980 0x55aedf973b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 --2- 172.21.15.62:0/1722671904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aedf964060 0x55aedf973630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 -- 172.21.15.62:0/1722671904 >> 172.21.15.62:0/1722671904 conn(0x55aedf9605f0 msgr2=0x55aedf96c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f3536ff5640 0 client.17493 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:48.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 -- 172.21.15.62:0/1722671904 shutdown_connections 2024-09-15T23:24:48.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.061+0000 7f355ba92740 1 -- 172.21.15.62:0/1722671904 wait complete. 2024-09-15T23:24:48.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.062+0000 7f355ba92740 20 client.17493 trim_cache size 0 max 16384 2024-09-15T23:24:48.065 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:24:48.065 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 34, 2024-09-15T23:24:48.065 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:24:48.065 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 32, 2024-09-15T23:24:48.065 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:24:48.065 INFO:teuthology.orchestra.run.smithi062.stdout: "83c3dbcf": { 2024-09-15T23:24:48.065 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 32, 2024-09-15T23:24:48.065 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 142.3, 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:48.066 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:48.067 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:48.067 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:48.067 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:48.067 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 -- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f353c0a3240 msgr2=0x7f353c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f353c0a3240 0x7f353c0a5700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f354c005c40 tx=0x7f354c005bd0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 -- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3554102da0 msgr2=0x7f3554100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3554102da0 0x7f3554100a00 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f354400d770 tx=0x7f354400dc30 comp rx=0 tx=0).stop 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 -- 172.21.15.62:0/4165481668 shutdown_connections 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f353c0a3240 0x7f353c0a5700 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f355410c180 0x7f3554101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35541036c0 0x7f3554100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 --2- 172.21.15.62:0/4165481668 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3554102da0 0x7f3554100a00 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 -- 172.21.15.62:0/4165481668 >> 172.21.15.62:0/4165481668 conn(0x7f35540fc7a0 msgr2=0x7f35540fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:48.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 -- 172.21.15.62:0/4165481668 shutdown_connections 2024-09-15T23:24:48.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:48.063+0000 7f35596b9640 1 -- 172.21.15.62:0/4165481668 wait complete. 2024-09-15T23:24:48.888 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '83c3dbcf', total seconds elapsed: 25.7 2024-09-15T23:24:48.888 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 22.35 2024-09-15T23:24:48.888 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:24:49.415 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:49.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 510.0 s, 307.1 tps, lat 1455.373 ms stddev 5152.886 2024-09-15T23:24:49.840 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:49 smithi062 ceph-mon[30949]: pgmap v626: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 6.3 MiB/s wr, 687 op/s 2024-09-15T23:24:50.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:49 smithi148 ceph-mon[35610]: pgmap v626: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 6.3 MiB/s wr, 687 op/s 2024-09-15T23:24:50.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.132+0000 7f007546b640 1 -- 172.21.15.62:0/4026815478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00701021b0 msgr2=0x7f00701056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.132+0000 7f007546b640 1 --2- 172.21.15.62:0/4026815478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00701021b0 0x7f00701056f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f0064009930 tx=0x7f006402a420 comp rx=0 tx=0).stop 2024-09-15T23:24:50.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.132+0000 7f007546b640 1 -- 172.21.15.62:0/4026815478 shutdown_connections 2024-09-15T23:24:50.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.132+0000 7f007546b640 1 --2- 172.21.15.62:0/4026815478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0070105e00 0x7f007010d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.132+0000 7f007546b640 1 --2- 172.21.15.62:0/4026815478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00701021b0 0x7f00701056f0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.132+0000 7f007546b640 1 --2- 172.21.15.62:0/4026815478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0070101800 0x7f0070101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.132+0000 7f007546b640 1 -- 172.21.15.62:0/4026815478 >> 172.21.15.62:0/4026815478 conn(0x7f00700fba90 msgr2=0x7f00700fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:50.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.132+0000 7f007546b640 1 -- 172.21.15.62:0/4026815478 shutdown_connections 2024-09-15T23:24:50.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.132+0000 7f007546b640 1 -- 172.21.15.62:0/4026815478 wait complete. 2024-09-15T23:24:50.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.133+0000 7f007546b640 1 Processor -- start 2024-09-15T23:24:50.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.133+0000 7f007546b640 1 -- start start 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f007546b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0070101800 0x7f0070073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f007546b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00701021b0 0x7f007006cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f007546b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0070105e00 0x7f007006d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f007546b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0070110ea0 con 0x7f0070105e00 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f007546b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0070110d20 con 0x7f0070101800 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f006f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0070105e00 0x7f007006d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f006e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00701021b0 0x7f007006cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f006e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00701021b0 0x7f007006cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:54088/0 (socket says 172.21.15.62:54088) 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f006f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0070105e00 0x7f007006d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39400/0 (socket says 172.21.15.62:39400) 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f006e7fc640 1 -- 172.21.15.62:0/1399548262 learned_addr learned my addr 172.21.15.62:0/1399548262 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f006effd640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0070101800 0x7f0070073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.134+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0070111020 con 0x7f00701021b0 2024-09-15T23:24:50.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f006f7fe640 1 -- 172.21.15.62:0/1399548262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00701021b0 msgr2=0x7f007006cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f006f7fe640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00701021b0 0x7f007006cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f006f7fe640 1 -- 172.21.15.62:0/1399548262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0070101800 msgr2=0x7f0070073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f006f7fe640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0070101800 0x7f0070073a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f006f7fe640 1 -- 172.21.15.62:0/1399548262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007006d680 con 0x7f0070105e00 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f006effd640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0070101800 0x7f0070073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f006f7fe640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0070105e00 0x7f007006d040 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0068009fd0 tx=0x7f006800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f0057fff640 1 -- 172.21.15.62:0/1399548262 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006800be40 con 0x7f0070105e00 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f0057fff640 1 -- 172.21.15.62:0/1399548262 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0068017450 con 0x7f0070105e00 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00700715a0 con 0x7f0070105e00 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f0057fff640 1 -- 172.21.15.62:0/1399548262 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0068005280 con 0x7f0070105e00 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.135+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0070071ae0 con 0x7f0070105e00 2024-09-15T23:24:50.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.137+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0070100070 con 0x7f0070105e00 2024-09-15T23:24:50.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.139+0000 7f0057fff640 1 -- 172.21.15.62:0/1399548262 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0068004a40 con 0x7f0070105e00 2024-09-15T23:24:50.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.139+0000 7f0057fff640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00580a32c0 0x7f00580a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:50.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.140+0000 7f0057fff640 1 -- 172.21.15.62:0/1399548262 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f00680d0430 con 0x7f0070105e00 2024-09-15T23:24:50.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.142+0000 7f006effd640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00580a32c0 0x7f00580a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:50.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.143+0000 7f006effd640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00580a32c0 0x7f00580a5780 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0060009fc0 tx=0x7f0060005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:50.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.145+0000 7f0057fff640 1 -- 172.21.15.62:0/1399548262 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0068025070 con 0x7f0070105e00 2024-09-15T23:24:50.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:49 smithi067 ceph-mon[35665]: pgmap v626: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 6.3 MiB/s wr, 687 op/s 2024-09-15T23:24:50.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.537+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f007006e030 con 0x7f0070105e00 2024-09-15T23:24:50.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.539+0000 7f0057fff640 1 -- 172.21.15.62:0/1399548262 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+11904 (secure 0 0 0) 0x7f006809cdd0 con 0x7f0070105e00 2024-09-15T23:24:50.541 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:24:50.541 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":34,"btime":"2024-09-15T23:24:12:668733+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:12.388349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:24:50.542 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 34 2024-09-15T23:24:50.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00580a32c0 msgr2=0x7f00580a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00580a32c0 0x7f00580a5780 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0060009fc0 tx=0x7f0060005f90 comp rx=0 tx=0).stop 2024-09-15T23:24:50.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0070105e00 msgr2=0x7f007006d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:50.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0070105e00 0x7f007006d040 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f0068009fd0 tx=0x7f006800b690 comp rx=0 tx=0).stop 2024-09-15T23:24:50.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 shutdown_connections 2024-09-15T23:24:50.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00580a32c0 0x7f00580a5780 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0070105e00 0x7f007006d040 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00701021b0 0x7f007006cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 --2- 172.21.15.62:0/1399548262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0070101800 0x7f0070073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:50.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 >> 172.21.15.62:0/1399548262 conn(0x7f00700fba90 msgr2=0x7f00700fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:50.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 shutdown_connections 2024-09-15T23:24:50.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:50.542+0000 7f007546b640 1 -- 172.21.15.62:0/1399548262 wait complete. 2024-09-15T23:24:50.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:50 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1399548262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:51.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:50 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1399548262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:51.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:50 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1399548262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:24:51.489 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0' 2024-09-15T23:24:51.490 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 142.35 --await-for 45.0 2024-09-15T23:24:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:24:52.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:51 smithi148 ceph-mon[35610]: pgmap v627: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 4.2 MiB/s wr, 509 op/s 2024-09-15T23:24:52.067 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:51 smithi062 ceph-mon[30949]: pgmap v627: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 4.2 MiB/s wr, 509 op/s 2024-09-15T23:24:52.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:51 smithi067 ceph-mon[35665]: pgmap v627: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 4.2 MiB/s wr, 509 op/s 2024-09-15T23:24:53.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:52 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:24:53.073 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:52 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:24:53.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.123+0000 7f25e4736640 1 -- 172.21.15.62:0/1926484235 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25dc100a80 msgr2=0x7f25dc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:53.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.123+0000 7f25e4736640 1 --2- 172.21.15.62:0/1926484235 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25dc100a80 0x7f25dc100ee0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009930 tx=0x7f25cc02a480 comp rx=0 tx=0).stop 2024-09-15T23:24:53.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.126+0000 7f25e4736640 1 -- 172.21.15.62:0/1926484235 shutdown_connections 2024-09-15T23:24:53.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.127+0000 7f25e4736640 1 --2- 172.21.15.62:0/1926484235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc101420 0x7f25dc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.127+0000 7f25e4736640 1 --2- 172.21.15.62:0/1926484235 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25dc100a80 0x7f25dc100ee0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.127+0000 7f25e4736640 1 --2- 172.21.15.62:0/1926484235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc106a80 0x7f25dc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.127+0000 7f25e4736640 1 -- 172.21.15.62:0/1926484235 >> 172.21.15.62:0/1926484235 conn(0x7f25dc0fc7a0 msgr2=0x7f25dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:53.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.131+0000 7f25e4736640 1 -- 172.21.15.62:0/1926484235 shutdown_connections 2024-09-15T23:24:53.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.132+0000 7f25e4736640 1 -- 172.21.15.62:0/1926484235 wait complete. 2024-09-15T23:24:53.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.133+0000 7f25e4736640 1 Processor -- start 2024-09-15T23:24:53.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.137+0000 7f25e4736640 1 -- start start 2024-09-15T23:24:53.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.137+0000 7f25e4736640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc100a80 0x7f25dc116510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.137+0000 7f25e4736640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25dc101420 0x7f25dc110590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.137+0000 7f25e4736640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc106a80 0x7f25dc110ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.137+0000 7f25e4736640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc10ef90 con 0x7f25dc106a80 2024-09-15T23:24:53.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.137+0000 7f25e4736640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc10ee10 con 0x7f25dc100a80 2024-09-15T23:24:53.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.137+0000 7f25e4736640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc10f110 con 0x7f25dc101420 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.140+0000 7f25e24d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc100a80 0x7f25dc116510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.140+0000 7f25e24d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc100a80 0x7f25dc116510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:43428/0 (socket says 172.21.15.62:43428) 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.140+0000 7f25e24d2640 1 -- 172.21.15.62:0/1100268825 learned_addr learned my addr 172.21.15.62:0/1100268825 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.141+0000 7f25e2cd3640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc106a80 0x7f25dc110ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.141+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25dc101420 0x7f25dc110590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.143+0000 7f25e2cd3640 1 -- 172.21.15.62:0/1100268825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25dc101420 msgr2=0x7f25dc110590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.143+0000 7f25e2cd3640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25dc101420 0x7f25dc110590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.143+0000 7f25e2cd3640 1 -- 172.21.15.62:0/1100268825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc100a80 msgr2=0x7f25dc116510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.143+0000 7f25e2cd3640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc100a80 0x7f25dc116510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.143+0000 7f25e2cd3640 1 -- 172.21.15.62:0/1100268825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25dc111270 con 0x7f25dc106a80 2024-09-15T23:24:53.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.146+0000 7f25e2cd3640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc106a80 0x7f25dc110ad0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f25d400ca30 tx=0x7f25d400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:53.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.149+0000 7f25d37fe640 1 -- 172.21.15.62:0/1100268825 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4007d90 con 0x7f25dc106a80 2024-09-15T23:24:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.149+0000 7f25d37fe640 1 -- 172.21.15.62:0/1100268825 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25d4005020 con 0x7f25dc106a80 2024-09-15T23:24:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.149+0000 7f25d37fe640 1 -- 172.21.15.62:0/1100268825 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d400f740 con 0x7f25dc106a80 2024-09-15T23:24:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.151+0000 7f25e4736640 1 -- 172.21.15.62:0/1100268825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25dc06bc10 con 0x7f25dc106a80 2024-09-15T23:24:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.151+0000 7f25e4736640 1 -- 172.21.15.62:0/1100268825 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25dc06c0d0 con 0x7f25dc106a80 2024-09-15T23:24:53.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.162+0000 7f25e6b0f740 1 Processor -- start 2024-09-15T23:24:53.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.167+0000 7f25e6b0f740 1 -- start start 2024-09-15T23:24:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.167+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d57c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.167+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d58170 0x55ee55d624f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.168+0000 7f25d37fe640 1 -- 172.21.15.62:0/1100268825 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f25d40051d0 con 0x7f25dc106a80 2024-09-15T23:24:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.168+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d64e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.168+0000 7f25e6b0f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ee55cbc160 con 0x55ee55d62a30 2024-09-15T23:24:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.168+0000 7f25e6b0f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ee55c63720 con 0x55ee55d58170 2024-09-15T23:24:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.168+0000 7f25e6b0f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ee55cb8120 con 0x55ee55d57850 2024-09-15T23:24:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.170+0000 7f25e24d2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d58170 0x55ee55d624f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.170+0000 7f25e1cd1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d57c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.171+0000 7f25e1cd1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d57c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49056/0 (socket says 172.21.15.62:49056) 2024-09-15T23:24:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.171+0000 7f25e1cd1640 1 -- 172.21.15.62:0/3473680449 learned_addr learned my addr 172.21.15.62:0/3473680449 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:53.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.182+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/3473680449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d64e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.183+0000 7f25d37fe640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f25c80a3450 0x7f25c80a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.183+0000 7f25e2cd3640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f25c80a3450 0x7f25c80a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.183+0000 7f25d37fe640 1 -- 172.21.15.62:0/1100268825 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f25d40cfde0 con 0x7f25dc106a80 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.183+0000 7f25e2cd3640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f25c80a3450 0x7f25c80a5910 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f25d40a4760 tx=0x7f25d406d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.185+0000 7f25e1cd1640 1 -- 172.21.15.62:0/3473680449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d58170 msgr2=0x55ee55d624f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.185+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/3473680449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d58170 0x55ee55d624f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.185+0000 7f25e1cd1640 1 -- 172.21.15.62:0/3473680449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 msgr2=0x55ee55d64e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.185+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/3473680449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d64e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.185+0000 7f25e1cd1640 1 -- 172.21.15.62:0/3473680449 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee55a2dba0 con 0x55ee55d57850 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.185+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/3473680449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d57c30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f25cc004fa0 tx=0x7f25cc009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6368b4fb5114f49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.186+0000 7f25d0ff9640 1 -- 172.21.15.62:0/3473680449 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc005200 con 0x55ee55d57850 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.186+0000 7f25d0ff9640 1 -- 172.21.15.62:0/3473680449 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25cc0053a0 con 0x55ee55d57850 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.187+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3473680449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 msgr2=0x55ee55d57c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.187+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3473680449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d57c30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f25cc004fa0 tx=0x7f25cc009560 comp rx=0 tx=0).stop 2024-09-15T23:24:53.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.190+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3473680449 shutdown_connections 2024-09-15T23:24:53.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.190+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3473680449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d64e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.190+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3473680449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d58170 0x55ee55d624f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.190+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3473680449 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d57c30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.190+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3473680449 >> 172.21.15.62:0/3473680449 conn(0x55ee55d53de0 msgr2=0x55ee55d541c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:53.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.193+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3473680449 shutdown_connections 2024-09-15T23:24:53.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.194+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3473680449 wait complete. 2024-09-15T23:24:53.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.194+0000 7f25e6b0f740 1 -- start start 2024-09-15T23:24:53.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.194+0000 7f25e6b0f740 1 Processor -- start 2024-09-15T23:24:53.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d60320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e6b0f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ee55d65650 con 0x55ee55d62a30 2024-09-15T23:24:53.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e6b0f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ee55c44900 con 0x55ee55d57850 2024-09-15T23:24:53.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e6b0f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ee55d657d0 con 0x55ee55d58170 2024-09-15T23:24:53.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e6b0f740 10 client.0 _ll_register_callbacks cb 0x55ee55c42780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:53.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e1cd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e1cd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:43444/0 (socket says 172.21.15.62:43444) 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.196+0000 7f25e1cd1640 1 -- 172.21.15.62:0/768846483 learned_addr learned my addr 172.21.15.62:0/768846483 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.197+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d60320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.197+0000 7f25e24d2640 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.197+0000 7f25e1cd1640 1 -- 172.21.15.62:0/768846483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 msgr2=0x55ee55d5fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.197+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.197+0000 7f25e1cd1640 1 -- 172.21.15.62:0/768846483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 msgr2=0x55ee55d60320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.197+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d60320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.197+0000 7f25e1cd1640 1 -- 172.21.15.62:0/768846483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee55c84c50 con 0x55ee55d57850 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.197+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5f8a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009850 tx=0x7f25cc03fb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eb5b4f33c21052b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.198+0000 7f25c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.198+0000 7f25c5ffb640 1 -- 172.21.15.62:0/768846483 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc005550 con 0x55ee55d57850 2024-09-15T23:24:53.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.199+0000 7f25e6b0f740 1 -- 172.21.15.62:0/768846483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25cc0053a0 con 0x55ee55d57850 2024-09-15T23:24:53.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.199+0000 7f25c5ffb640 1 -- 172.21.15.62:0/768846483 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f25cc03f440 con 0x55ee55d57850 2024-09-15T23:24:53.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.199+0000 7f25c5ffb640 1 -- 172.21.15.62:0/768846483 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f25cc081050 con 0x55ee55d57850 2024-09-15T23:24:53.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.199+0000 7f25e6b0f740 10 client.27077 fetch_fsmap learned FSMap version 35 2024-09-15T23:24:53.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.199+0000 7f25e6b0f740 1 -- 172.21.15.62:0/768846483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ee55d712d0 con 0x55ee55d57850 2024-09-15T23:24:53.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.203+0000 7f25c5ffb640 1 -- 172.21.15.62:0/768846483 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25cc0053a0 con 0x55ee55d57850 2024-09-15T23:24:53.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.203+0000 7f25c5ffb640 1 -- 172.21.15.62:0/768846483 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f25cc07b020 con 0x55ee55d57850 2024-09-15T23:24:53.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.204+0000 7f25e6b0f740 10 client.27077 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:24:53.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.204+0000 7f25e6b0f740 10 client.27077 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:24:53.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.204+0000 7f25e6b0f740 20 client.27077 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:53.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.204+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ee55d72620 0x55ee55d74a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.204+0000 7f25e6b0f740 4 client.27077 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:53.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.204+0000 7f25e6b0f740 1 -- 172.21.15.62:0/768846483 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee55cc9890 con 0x55ee55d72620 2024-09-15T23:24:53.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.205+0000 7f25e24d2640 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ee55d72620 0x55ee55d74a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.209+0000 7f25e24d2640 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ee55d72620 0x55ee55d74a10 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc9f885ec8e92932 server_cookie=91befe235d4401ce in_seq=0 out_seq=0 2024-09-15T23:24:53.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.209+0000 7f25c5ffb640 10 client.27077 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.209+0000 7f25c5ffb640 1 -- 172.21.15.62:0/768846483 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ee55cc9890 con 0x55ee55d72620 2024-09-15T23:24:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.209+0000 7f25c5ffb640 10 client.27077 handle_command_reply: tid=0 2024-09-15T23:24:53.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.210+0000 7f25e6b0f740 1 client.27077 shutdown 2024-09-15T23:24:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.214+0000 7f25e6b0f740 1 -- 172.21.15.62:0/768846483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 msgr2=0x55ee55d5f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:53.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.214+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5f8a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009850 tx=0x7f25cc03fb40 comp rx=0 tx=0).stop 2024-09-15T23:24:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.215+0000 7f25e6b0f740 1 -- 172.21.15.62:0/768846483 shutdown_connections 2024-09-15T23:24:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.215+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ee55d72620 0x55ee55d74a10 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.215+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d60320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.215+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.215+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/768846483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5f8a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.215+0000 7f25e6b0f740 1 -- 172.21.15.62:0/768846483 >> 172.21.15.62:0/768846483 conn(0x55ee55d53de0 msgr2=0x55ee55d59b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.215+0000 7f25c5ffb640 0 client.27077 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.216+0000 7f25e6b0f740 1 -- 172.21.15.62:0/768846483 shutdown_connections 2024-09-15T23:24:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.216+0000 7f25e6b0f740 1 -- 172.21.15.62:0/768846483 wait complete. 2024-09-15T23:24:53.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.216+0000 7f25e6b0f740 20 client.27077 trim_cache size 0 max 16384 2024-09-15T23:24:53.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.254+0000 7f25e6b0f740 1 Processor -- start 2024-09-15T23:24:53.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.256+0000 7f25e6b0f740 1 -- start start 2024-09-15T23:24:53.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.257+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.257+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.257+0000 7f25e1cd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.257+0000 7f25e1cd1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:43446/0 (socket says 172.21.15.62:43446) 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.260+0000 7f25e24d2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.260+0000 7f25e24d2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49072/0 (socket says 172.21.15.62:49072) 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.257+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d77650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.261+0000 7f25e1cd1640 1 -- 172.21.15.62:0/513070726 learned_addr learned my addr 172.21.15.62:0/513070726 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.261+0000 7f25e6b0f740 1 -- 172.21.15.62:0/513070726 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc11bf30 con 0x55ee55d62a30 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.261+0000 7f25e6b0f740 1 -- 172.21.15.62:0/513070726 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc119800 con 0x55ee55d57850 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.261+0000 7f25e6b0f740 1 -- 172.21.15.62:0/513070726 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc073e90 con 0x55ee55d58170 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.261+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/513070726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d77650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25e1cd1640 1 -- 172.21.15.62:0/513070726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 msgr2=0x55ee55d5a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:53.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/513070726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25e1cd1640 1 -- 172.21.15.62:0/513070726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 msgr2=0x55ee55d77650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/513070726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d77650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25e1cd1640 1 -- 172.21.15.62:0/513070726 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee55d654e0 con 0x55ee55d57850 2024-09-15T23:24:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25e24d2640 1 --2- 172.21.15.62:0/513070726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-15T23:24:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/513070726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5e7e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f25cc004a70 tx=0x7f25cc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d15ad9a601a8d3c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25c7fff640 1 -- 172.21.15.62:0/513070726 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc000e60 con 0x55ee55d57850 2024-09-15T23:24:53.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25c7fff640 1 -- 172.21.15.62:0/513070726 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25cc009520 con 0x55ee55d57850 2024-09-15T23:24:53.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.262+0000 7f25c7fff640 1 -- 172.21.15.62:0/513070726 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc03f440 con 0x55ee55d57850 2024-09-15T23:24:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.263+0000 7f25e6b0f740 1 -- 172.21.15.62:0/513070726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 msgr2=0x55ee55d5e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:53.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.263+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/513070726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5e7e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f25cc004a70 tx=0x7f25cc005fd0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.267+0000 7f25e6b0f740 1 -- 172.21.15.62:0/513070726 shutdown_connections 2024-09-15T23:24:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.267+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/513070726 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d62a30 0x55ee55d77650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.267+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/513070726 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d58170 0x55ee55d5a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.267+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/513070726 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d57850 0x55ee55d5e7e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.267+0000 7f25e6b0f740 1 -- 172.21.15.62:0/513070726 >> 172.21.15.62:0/513070726 conn(0x55ee55d53de0 msgr2=0x55ee55d6e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:53.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.268+0000 7f25e6b0f740 1 -- 172.21.15.62:0/513070726 shutdown_connections 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.268+0000 7f25e6b0f740 1 -- 172.21.15.62:0/513070726 wait complete. 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.269+0000 7f25e6b0f740 1 -- start start 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.269+0000 7f25e6b0f740 1 Processor -- start 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.269+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d74e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d58170 0x55ee55d753b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e1cd1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d74e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e6b0f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d62a30 0x55ee55d77460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e6b0f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc117cc0 con 0x55ee55d58170 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e6b0f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc119130 con 0x55ee55d62a30 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e6b0f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc118390 con 0x55ee55d57850 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e6b0f740 10 client.0 _ll_register_callbacks cb 0x7f25cc004de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e24d2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d58170 0x55ee55d753b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e24d2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d58170 0x55ee55d753b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34116/0 (socket says 172.21.15.62:34116) 2024-09-15T23:24:53.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e24d2640 1 -- 172.21.15.62:0/3590627857 learned_addr learned my addr 172.21.15.62:0/3590627857 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:53.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.270+0000 7f25e1cd1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d74e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49078/0 (socket says 172.21.15.62:49078) 2024-09-15T23:24:53.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:52 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:24:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.273+0000 7f25e1cd1640 1 -- 172.21.15.62:0/3590627857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d62a30 msgr2=0x55ee55d77460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.273+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d62a30 0x55ee55d77460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.273+0000 7f25e1cd1640 1 -- 172.21.15.62:0/3590627857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d58170 msgr2=0x55ee55d753b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.273+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d58170 0x55ee55d753b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.273+0000 7f25e1cd1640 1 -- 172.21.15.62:0/3590627857 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee55d4f290 con 0x55ee55d57850 2024-09-15T23:24:53.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.273+0000 7f25e1cd1640 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d74e70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f25cc004a70 tx=0x7f25cc02acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7745106fd161d2f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:53.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.273+0000 7f25c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:24:53.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.274+0000 7f25c5ffb640 1 -- 172.21.15.62:0/3590627857 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25cc0035f0 con 0x55ee55d57850 2024-09-15T23:24:53.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.274+0000 7f25c5ffb640 1 -- 172.21.15.62:0/3590627857 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f25cc005680 con 0x55ee55d57850 2024-09-15T23:24:53.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.275+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3590627857 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25cc009520 con 0x55ee55d57850 2024-09-15T23:24:53.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.275+0000 7f25c5ffb640 1 -- 172.21.15.62:0/3590627857 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f25cc093050 con 0x55ee55d57850 2024-09-15T23:24:53.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.275+0000 7f25e6b0f740 10 client.26860 fetch_fsmap learned FSMap version 35 2024-09-15T23:24:53.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.275+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3590627857 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ee55ddc850 con 0x55ee55d57850 2024-09-15T23:24:53.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.276+0000 7f25c5ffb640 1 -- 172.21.15.62:0/3590627857 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25cc089070 con 0x55ee55d57850 2024-09-15T23:24:53.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.276+0000 7f25c5ffb640 1 -- 172.21.15.62:0/3590627857 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f25cc07b040 con 0x55ee55d57850 2024-09-15T23:24:53.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.276+0000 7f25e6b0f740 10 client.26860 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:24:53.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.276+0000 7f25e6b0f740 10 client.26860 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:24:53.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.276+0000 7f25e6b0f740 20 client.26860 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:53.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.276+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ee55ddcfe0 0x55ee55ddf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:53.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.276+0000 7f25e6b0f740 4 client.26860 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}] 2024-09-15T23:24:53.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.276+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3590627857 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 142.35, "await_for": 45.0}) -- 0x7f25cc009520 con 0x55ee55ddcfe0 2024-09-15T23:24:53.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.276+0000 7f25e24d2640 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ee55ddcfe0 0x55ee55ddf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:53.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.279+0000 7f25e24d2640 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ee55ddcfe0 0x55ee55ddf400 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7ea23c479ffb4bc server_cookie=d9a9058568b258d8 in_seq=0 out_seq=0 2024-09-15T23:24:53.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.280+0000 7f25c5ffb640 10 client.26860 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:53.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25c5ffb640 1 -- 172.21.15.62:0/3590627857 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f25cc009520 con 0x55ee55ddcfe0 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25c5ffb640 10 client.26860 handle_command_reply: tid=0 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 client.26860 shutdown 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3590627857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 msgr2=0x55ee55d74e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d74e70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f25cc004a70 tx=0x7f25cc02acf0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3590627857 shutdown_connections 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ee55ddcfe0 0x55ee55ddf400 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ee55d62a30 0x55ee55d77460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ee55d58170 0x55ee55d753b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 --2- 172.21.15.62:0/3590627857 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ee55d57850 0x55ee55d74e70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25c5ffb640 0 client.26860 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3590627857 >> 172.21.15.62:0/3590627857 conn(0x55ee55d53de0 msgr2=0x55ee55d6da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:53.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3590627857 shutdown_connections 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 1 -- 172.21.15.62:0/3590627857 wait complete. 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.565+0000 7f25e6b0f740 20 client.26860 trim_cache size 0 max 16384 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 35, 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 34, 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stdout: "590f31d6": { 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 34, 2024-09-15T23:24:53.569 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 142.3, 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:53.570 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:53.571 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:53.571 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:53.571 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:24:53.571 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:24:53.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 -- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f25c80a3450 msgr2=0x7f25c80a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:53.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f25c80a3450 0x7f25c80a5910 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f25d40a4760 tx=0x7f25d406d040 comp rx=0 tx=0).stop 2024-09-15T23:24:53.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 -- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc106a80 msgr2=0x7f25dc110ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:53.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc106a80 0x7f25dc110ad0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f25d400ca30 tx=0x7f25d400cef0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 -- 172.21.15.62:0/1100268825 shutdown_connections 2024-09-15T23:24:53.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f25c80a3450 0x7f25c80a5910 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25dc106a80 0x7f25dc110ad0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f25dc101420 0x7f25dc110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 --2- 172.21.15.62:0/1100268825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25dc100a80 0x7f25dc116510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:53.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 -- 172.21.15.62:0/1100268825 >> 172.21.15.62:0/1100268825 conn(0x7f25dc0fc7a0 msgr2=0x7f25dc0fce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:53.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 -- 172.21.15.62:0/1100268825 shutdown_connections 2024-09-15T23:24:53.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:24:53.566+0000 7f25e4736640 1 -- 172.21.15.62:0/1100268825 wait complete. 2024-09-15T23:24:53.859 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:53 smithi062 ceph-mon[30949]: pgmap v628: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.5 MiB/s wr, 551 op/s 2024-09-15T23:24:53.860 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:53 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:24:53.860 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:53 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:24:53.860 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:53 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:24:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:53 smithi148 ceph-mon[35610]: pgmap v628: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.5 MiB/s wr, 551 op/s 2024-09-15T23:24:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:53 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:24:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:53 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:24:54.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:53 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:24:54.247 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '590f31d6', quiesce took 5.4 seconds. Will release after: 16.950000000000003 2024-09-15T23:24:54.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:53 smithi067 ceph-mon[35665]: pgmap v628: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.5 MiB/s wr, 551 op/s 2024-09-15T23:24:54.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:53 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:24:54.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:53 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:24:54.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:53 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:24:54.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 515.0 s, 557.2 tps, lat 141.961 ms stddev 157.627 2024-09-15T23:24:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.030+0000 7fcedcbb7640 1 -- 172.21.15.62:0/1973378981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced8107820 msgr2=0x7fced8109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.030+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/1973378981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced8107820 0x7fced8109c90 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fced0009ca0 tx=0x7fced002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.030+0000 7fcedcbb7640 1 -- 172.21.15.62:0/1973378981 shutdown_connections 2024-09-15T23:24:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.030+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/1973378981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced8107820 0x7fced8109c90 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.030+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/1973378981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8104ef0 0x7fced81072e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.030+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/1973378981 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fced81025c0 0x7fced81049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.030+0000 7fcedcbb7640 1 -- 172.21.15.62:0/1973378981 >> 172.21.15.62:0/1973378981 conn(0x7fced80fc1e0 msgr2=0x7fced80fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:55.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.030+0000 7fcedcbb7640 1 -- 172.21.15.62:0/1973378981 shutdown_connections 2024-09-15T23:24:55.034 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.030+0000 7fcedcbb7640 1 -- 172.21.15.62:0/1973378981 wait complete. 2024-09-15T23:24:55.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.031+0000 7fcedcbb7640 1 Processor -- start 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.031+0000 7fcedcbb7640 1 -- start start 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.031+0000 7fcedcbb7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced81025c0 0x7fced8197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.031+0000 7fcedcbb7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fced8104ef0 0x7fced8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.031+0000 7fcedcbb7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8107820 0x7fced819f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.031+0000 7fcedcbb7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fced810d800 con 0x7fced81025c0 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.031+0000 7fcedcbb7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fced810d680 con 0x7fced8107820 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.031+0000 7fcedcbb7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fced810d980 con 0x7fced8104ef0 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced81025c0 0x7fced8197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced81025c0 0x7fced8197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34142/0 (socket says 172.21.15.62:34142) 2024-09-15T23:24:55.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced659c640 1 -- 172.21.15.62:0/371379048 learned_addr learned my addr 172.21.15.62:0/371379048 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced5d9b640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fced8104ef0 0x7fced8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced6d9d640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8107820 0x7fced819f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced659c640 1 -- 172.21.15.62:0/371379048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fced8104ef0 msgr2=0x7fced8198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced659c640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fced8104ef0 0x7fced8198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced659c640 1 -- 172.21.15.62:0/371379048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8107820 msgr2=0x7fced819f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced659c640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8107820 0x7fced819f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced659c640 1 -- 172.21.15.62:0/371379048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced819fa70 con 0x7fced81025c0 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fced659c640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced81025c0 0x7fced8197e50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fcec800d770 tx=0x7fcec800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fcec77fe640 1 -- 172.21.15.62:0/371379048 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec8017070 con 0x7fced81025c0 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fcedcbb7640 1 -- 172.21.15.62:0/371379048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced819fd60 con 0x7fced81025c0 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fcec77fe640 1 -- 172.21.15.62:0/371379048 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcec8005500 con 0x7fced81025c0 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fcec77fe640 1 -- 172.21.15.62:0/371379048 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec8004e00 con 0x7fced81025c0 2024-09-15T23:24:55.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.032+0000 7fcedcbb7640 1 -- 172.21.15.62:0/371379048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced81a01f0 con 0x7fced81025c0 2024-09-15T23:24:55.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.035+0000 7fcec77fe640 1 -- 172.21.15.62:0/371379048 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fcec8004fa0 con 0x7fced81025c0 2024-09-15T23:24:55.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.035+0000 7fcedfa2b740 1 Processor -- start 2024-09-15T23:24:55.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.035+0000 7fcedfa2b740 1 -- start start 2024-09-15T23:24:55.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bb21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bb2780 0x55c325bb4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c325b0c2f0 con 0x55c325bacdb0 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c325ab4270 con 0x55c325bad6f0 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c325b082b0 con 0x55c325bb2780 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bad1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34150/0 (socket says 172.21.15.62:34150) 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced6d9d640 1 -- 172.21.15.62:0/4084160011 learned_addr learned my addr 172.21.15.62:0/4084160011 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced659c640 1 --2- 172.21.15.62:0/4084160011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bb21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced5d9b640 1 --2- 172.21.15.62:0/4084160011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bb2780 0x55c325bb4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced6d9d640 1 -- 172.21.15.62:0/4084160011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bb2780 msgr2=0x55c325bb4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced6d9d640 1 --2- 172.21.15.62:0/4084160011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bb2780 0x55c325bb4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced6d9d640 1 -- 172.21.15.62:0/4084160011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 msgr2=0x55c325bb21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:55.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced6d9d640 1 --2- 172.21.15.62:0/4084160011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bb21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced6d9d640 1 -- 172.21.15.62:0/4084160011 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c325ab3340 con 0x55c325bacdb0 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fcec77fe640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcec00a32f0 0x7fcec00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced659c640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcec00a32f0 0x7fcec00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.036+0000 7fced6d9d640 1 --2- 172.21.15.62:0/4084160011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bad1b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fced0005e30 tx=0x7fced0005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a568c1e46b826ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcec77fe640 1 -- 172.21.15.62:0/371379048 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcec80cf620 con 0x7fced81025c0 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcec4ff9640 1 -- 172.21.15.62:0/4084160011 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0031cb0 con 0x55c325bacdb0 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcec4ff9640 1 -- 172.21.15.62:0/4084160011 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced0036040 con 0x55c325bacdb0 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcec4ff9640 1 -- 172.21.15.62:0/4084160011 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0004320 con 0x55c325bacdb0 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcedfa2b740 1 -- 172.21.15.62:0/4084160011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 msgr2=0x55c325bad1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/4084160011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bad1b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fced0005e30 tx=0x7fced0005ef0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fced659c640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcec00a32f0 0x7fcec00a57b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcec80cd110 tx=0x7fcec80d5550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcedfa2b740 1 -- 172.21.15.62:0/4084160011 shutdown_connections 2024-09-15T23:24:55.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/4084160011 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bb2780 0x55c325bb4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/4084160011 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bb21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/4084160011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bad1b0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.037+0000 7fcedfa2b740 1 -- 172.21.15.62:0/4084160011 >> 172.21.15.62:0/4084160011 conn(0x55c325ba4010 msgr2=0x55c325ba4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 -- 172.21.15.62:0/4084160011 shutdown_connections 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 -- 172.21.15.62:0/4084160011 wait complete. 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 -- start start 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 Processor -- start 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bacdb0 0x55c325bb6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bb72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bb2780 0x55c325bbe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c325bb5510 con 0x55c325bb2780 2024-09-15T23:24:55.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c325a95290 con 0x55c325bad6f0 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c325bb5690 con 0x55c325bacdb0 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fcedfa2b740 10 client.0 _ll_register_callbacks cb 0x55c325ac6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fced6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bb2780 0x55c325bbe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fced6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bb2780 0x55c325bbe2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34166/0 (socket says 172.21.15.62:34166) 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.038+0000 7fced6d9d640 1 -- 172.21.15.62:0/3594126603 learned_addr learned my addr 172.21.15.62:0/3594126603 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fced5d9b640 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bb72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fced6d9d640 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bacdb0 0x55c325bb6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fced6d9d640 1 -- 172.21.15.62:0/3594126603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bacdb0 msgr2=0x55c325bb6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fced6d9d640 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bacdb0 0x55c325bb6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fced6d9d640 1 -- 172.21.15.62:0/3594126603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 msgr2=0x55c325bb72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fced6d9d640 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bb72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fced6d9d640 1 -- 172.21.15.62:0/3594126603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c32581f210 con 0x55c325bb2780 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fced6d9d640 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bb2780 0x55c325bbe2e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fced0009a60 tx=0x7fced0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e9ed31be0097a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fcebdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fcebdffb640 1 -- 172.21.15.62:0/3594126603 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0003b80 con 0x55c325bb2780 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fcebdffb640 1 -- 172.21.15.62:0/3594126603 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fced0041040 con 0x55c325bb2780 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fcedfa2b740 1 -- 172.21.15.62:0/3594126603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fced0036040 con 0x55c325bb2780 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fcebdffb640 1 -- 172.21.15.62:0/3594126603 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fced0004420 con 0x55c325bb2780 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fcedfa2b740 10 client.17526 fetch_fsmap learned FSMap version 35 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fcedfa2b740 1 -- 172.21.15.62:0/3594126603 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c3258eb620 con 0x55c325bb2780 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.039+0000 7fcebdffb640 1 -- 172.21.15.62:0/3594126603 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced0036040 con 0x55c325bb2780 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fcebdffb640 1 -- 172.21.15.62:0/3594126603 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fced002f8e0 con 0x55c325bb2780 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fcedfa2b740 10 client.17526 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:24:55.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fcedfa2b740 10 client.17526 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fcedfa2b740 20 client.17526 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c325bc2740 0x55c325bc4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fcedfa2b740 4 client.17526 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fcedfa2b740 1 -- 172.21.15.62:0/3594126603 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c325b19a20 con 0x55c325bc2740 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fced5d9b640 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c325bc2740 0x55c325bc4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fced5d9b640 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c325bc2740 0x55c325bc4b30 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f457d28b9229afe server_cookie=9a775a1cb99f4d5f in_seq=0 out_seq=0 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.040+0000 7fcebdffb640 10 client.17526 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcebdffb640 1 -- 172.21.15.62:0/3594126603 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c325b19a20 con 0x55c325bc2740 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcebdffb640 10 client.17526 handle_command_reply: tid=0 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 client.17526 shutdown 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 -- 172.21.15.62:0/3594126603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bb2780 msgr2=0x55c325bbe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bb2780 0x55c325bbe2e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fced0009a60 tx=0x7fced0009580 comp rx=0 tx=0).stop 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 -- 172.21.15.62:0/3594126603 shutdown_connections 2024-09-15T23:24:55.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c325bc2740 0x55c325bc4b30 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bb2780 0x55c325bbe2e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bb72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcebdffb640 0 client.17526 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/3594126603 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bacdb0 0x55c325bb6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 -- 172.21.15.62:0/3594126603 >> 172.21.15.62:0/3594126603 conn(0x55c325ba4010 msgr2=0x55c325ba5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 -- 172.21.15.62:0/3594126603 shutdown_connections 2024-09-15T23:24:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.041+0000 7fcedfa2b740 1 -- 172.21.15.62:0/3594126603 wait complete. 2024-09-15T23:24:55.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.042+0000 7fcedfa2b740 20 client.17526 trim_cache size 0 max 16384 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fcedfa2b740 1 Processor -- start 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fcedfa2b740 1 -- start start 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325ba4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bbe270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bb2780 0x55c325ba5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fced81aace0 con 0x55c325bacdb0 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fced81a85b0 con 0x55c325bad6f0 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fced81a6400 con 0x55c325bb2780 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fced6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325ba4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fced6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325ba4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34180/0 (socket says 172.21.15.62:34180) 2024-09-15T23:24:55.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.056+0000 7fced6d9d640 1 -- 172.21.15.62:0/1198074118 learned_addr learned my addr 172.21.15.62:0/1198074118 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fced5d9b640 1 --2- 172.21.15.62:0/1198074118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bbe270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fced6d9d640 1 -- 172.21.15.62:0/1198074118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bb2780 msgr2=0x55c325ba5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fced6d9d640 1 --2- 172.21.15.62:0/1198074118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bb2780 0x55c325ba5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fced6d9d640 1 -- 172.21.15.62:0/1198074118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 msgr2=0x55c325bbe270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fced6d9d640 1 --2- 172.21.15.62:0/1198074118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bbe270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fced6d9d640 1 -- 172.21.15.62:0/1198074118 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c325bc51b0 con 0x55c325bacdb0 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fced6d9d640 1 --2- 172.21.15.62:0/1198074118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325ba4ec0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fced0052040 tx=0x7fced0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b8550fb8d397e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fcebffff640 1 -- 172.21.15.62:0/1198074118 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced003f030 con 0x55c325bacdb0 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fcebffff640 1 -- 172.21.15.62:0/1198074118 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced0036070 con 0x55c325bacdb0 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fcebffff640 1 -- 172.21.15.62:0/1198074118 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced00047e0 con 0x55c325bacdb0 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fcedfa2b740 1 -- 172.21.15.62:0/1198074118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 msgr2=0x55c325ba4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:55.064 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/1198074118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325ba4ec0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fced0052040 tx=0x7fced0004a40 comp rx=0 tx=0).stop 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fcedfa2b740 1 -- 172.21.15.62:0/1198074118 shutdown_connections 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.057+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/1198074118 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bb2780 0x55c325ba5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/1198074118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bad6f0 0x55c325bbe270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/1198074118 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325ba4ec0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 -- 172.21.15.62:0/1198074118 >> 172.21.15.62:0/1198074118 conn(0x55c325ba4010 msgr2=0x55c325bb6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 -- 172.21.15.62:0/1198074118 shutdown_connections 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 -- 172.21.15.62:0/1198074118 wait complete. 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 -- start start 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 Processor -- start 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bbb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bad6f0 0x55c325bbbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bb2780 0x55c325bbcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fced81a6a70 con 0x55c325bacdb0 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fced81a7ee0 con 0x55c325bb2780 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fced81a7140 con 0x55c325bad6f0 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fcedfa2b740 10 client.0 _ll_register_callbacks cb 0x7fcec80d5420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fced6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bbb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fced6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bbb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:34192/0 (socket says 172.21.15.62:34192) 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fced6d9d640 1 -- 172.21.15.62:0/573326828 learned_addr learned my addr 172.21.15.62:0/573326828 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.058+0000 7fced5d9b640 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bad6f0 0x55c325bbbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fced6d9d640 1 -- 172.21.15.62:0/573326828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bad6f0 msgr2=0x55c325bbbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fced6d9d640 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bad6f0 0x55c325bbbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fced6d9d640 1 -- 172.21.15.62:0/573326828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bb2780 msgr2=0x55c325bbcac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fced6d9d640 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bb2780 0x55c325bbcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fced6d9d640 1 -- 172.21.15.62:0/573326828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c325bbd0d0 con 0x55c325bacdb0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fced6d9d640 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bbb5c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fced0000f80 tx=0x7fced0005cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576d6c28c3fb9f21 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcebdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcebdffb640 1 -- 172.21.15.62:0/573326828 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced003f040 con 0x55c325bacdb0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcebdffb640 1 -- 172.21.15.62:0/573326828 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fced0031450 con 0x55c325bacdb0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcedfa2b740 1 -- 172.21.15.62:0/573326828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fced0036070 con 0x55c325bacdb0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcebdffb640 1 -- 172.21.15.62:0/573326828 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fced0031650 con 0x55c325bacdb0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcedfa2b740 10 client.17532 fetch_fsmap learned FSMap version 35 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcedfa2b740 1 -- 172.21.15.62:0/573326828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c325bbe3e0 con 0x55c325bacdb0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcebdffb640 1 -- 172.21.15.62:0/573326828 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced0036070 con 0x55c325bacdb0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcebdffb640 1 -- 172.21.15.62:0/573326828 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fced00305c0 con 0x55c325bacdb0 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcedfa2b740 10 client.17532 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:24:55.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcedfa2b740 10 client.17532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:24:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcedfa2b740 20 client.17532 populate_metadata read hostname 'smithi062' 2024-09-15T23:24:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.059+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c325c2d9c0 0x55c325c2fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:24:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.060+0000 7fcedfa2b740 4 client.17532 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:24:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.060+0000 7fcedfa2b740 1 -- 172.21.15.62:0/573326828 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c325b13d70 con 0x55c325c2d9c0 2024-09-15T23:24:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.060+0000 7fced5d9b640 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c325c2d9c0 0x55c325c2fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:24:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.060+0000 7fced5d9b640 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c325c2d9c0 0x55c325c2fde0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33207875e507ebda server_cookie=f1abe657529325d3 in_seq=0 out_seq=0 2024-09-15T23:24:55.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.060+0000 7fcebdffb640 10 client.17532 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:55.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcebdffb640 1 -- 172.21.15.62:0/573326828 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c325b13d70 con 0x55c325c2d9c0 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcebdffb640 10 client.17532 handle_command_reply: tid=0 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 client.17532 shutdown 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 -- 172.21.15.62:0/573326828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 msgr2=0x55c325bbb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bbb5c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fced0000f80 tx=0x7fced0005cd0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 -- 172.21.15.62:0/573326828 shutdown_connections 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c325c2d9c0 0x55c325c2fde0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c325bb2780 0x55c325bbcac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c325bad6f0 0x55c325bbbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 --2- 172.21.15.62:0/573326828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c325bacdb0 0x55c325bbb5c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcebdffb640 0 client.17532 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 -- 172.21.15.62:0/573326828 >> 172.21.15.62:0/573326828 conn(0x55c325ba4010 msgr2=0x55c325ba29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 -- 172.21.15.62:0/573326828 shutdown_connections 2024-09-15T23:24:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 1 -- 172.21.15.62:0/573326828 wait complete. 2024-09-15T23:24:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedfa2b740 20 client.17532 trim_cache size 0 max 16384 2024-09-15T23:24:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:24:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:24:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:24:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:24:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.072+0000 7fcedcbb7640 1 -- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcec00a32f0 msgr2=0x7fcec00a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcec00a32f0 0x7fcec00a57b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcec80cd110 tx=0x7fcec80d5550 comp rx=0 tx=0).stop 2024-09-15T23:24:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 -- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced81025c0 msgr2=0x7fced8197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:24:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced81025c0 0x7fced8197e50 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fcec800d770 tx=0x7fcec800dc30 comp rx=0 tx=0).stop 2024-09-15T23:24:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 -- 172.21.15.62:0/371379048 shutdown_connections 2024-09-15T23:24:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcec00a32f0 0x7fcec00a57b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.077 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fced8107820 0x7fced819f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fced8104ef0 0x7fced8198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 --2- 172.21.15.62:0/371379048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fced81025c0 0x7fced8197e50 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:24:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 -- 172.21.15.62:0/371379048 >> 172.21.15.62:0/371379048 conn(0x7fced80fc1e0 msgr2=0x7fced8102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:24:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 -- 172.21.15.62:0/371379048 shutdown_connections 2024-09-15T23:24:55.078 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:24:55.073+0000 7fcedcbb7640 1 -- 172.21.15.62:0/371379048 wait complete. 2024-09-15T23:24:55.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:54 smithi062 ceph-mon[30949]: pgmap v629: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 5.5 MiB/s wr, 638 op/s 2024-09-15T23:24:55.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:54 smithi067 ceph-mon[35665]: pgmap v629: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 5.5 MiB/s wr, 638 op/s 2024-09-15T23:24:55.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:54 smithi148 ceph-mon[35610]: pgmap v629: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 5.5 MiB/s wr, 638 op/s 2024-09-15T23:24:57.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:57 smithi067 ceph-mon[35665]: pgmap v630: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 4.7 MiB/s wr, 549 op/s 2024-09-15T23:24:57.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:57 smithi148 ceph-mon[35610]: pgmap v630: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 4.7 MiB/s wr, 549 op/s 2024-09-15T23:24:57.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:56 smithi062 ceph-mon[30949]: pgmap v630: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 4.7 MiB/s wr, 549 op/s 2024-09-15T23:24:59.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:24:59 smithi067 ceph-mon[35665]: pgmap v631: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.5 MiB/s wr, 395 op/s 2024-09-15T23:24:59.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:24:59 smithi148 ceph-mon[35610]: pgmap v631: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.5 MiB/s wr, 395 op/s 2024-09-15T23:24:59.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:24:59 smithi062 ceph-mon[30949]: pgmap v631: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 3.5 MiB/s wr, 395 op/s 2024-09-15T23:24:59.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 520.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:25:01.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:01 smithi067 ceph-mon[35665]: pgmap v632: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.4 MiB/s wr, 264 op/s 2024-09-15T23:25:01.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:01 smithi148 ceph-mon[35610]: pgmap v632: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.4 MiB/s wr, 264 op/s 2024-09-15T23:25:01.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:01 smithi062 ceph-mon[30949]: pgmap v632: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.4 MiB/s wr, 264 op/s 2024-09-15T23:25:03.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:03 smithi067 ceph-mon[35665]: pgmap v633: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-09-15T23:25:03.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:03 smithi148 ceph-mon[35610]: pgmap v633: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-09-15T23:25:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:03 smithi062 ceph-mon[30949]: pgmap v633: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-09-15T23:25:04.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 525.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:25:05.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:05 smithi067 ceph-mon[35665]: pgmap v634: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 MiB/s wr, 182 op/s 2024-09-15T23:25:05.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:05 smithi148 ceph-mon[35610]: pgmap v634: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 MiB/s wr, 182 op/s 2024-09-15T23:25:05.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:05 smithi062 ceph-mon[30949]: pgmap v634: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 MiB/s wr, 182 op/s 2024-09-15T23:25:07.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:07 smithi067 ceph-mon[35665]: pgmap v635: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 783 KiB/s wr, 93 op/s 2024-09-15T23:25:07.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:07 smithi148 ceph-mon[35610]: pgmap v635: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 783 KiB/s wr, 93 op/s 2024-09-15T23:25:07.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:07 smithi062 ceph-mon[30949]: pgmap v635: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 783 KiB/s wr, 93 op/s 2024-09-15T23:25:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.174+0000 7ffb24dcf640 1 -- 172.21.15.62:0/3984248615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 msgr2=0x7ffb20107300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.174+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/3984248615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 0x7ffb20107300 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7ffb14009900 tx=0x7ffb1402a410 comp rx=0 tx=0).stop 2024-09-15T23:25:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.174+0000 7ffb24dcf640 1 -- 172.21.15.62:0/3984248615 shutdown_connections 2024-09-15T23:25:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.174+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/3984248615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb20107840 0x7ffb20109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.174+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/3984248615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 0x7ffb20107300 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.174+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/3984248615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb201025e0 0x7ffb201049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.174+0000 7ffb24dcf640 1 -- 172.21.15.62:0/3984248615 >> 172.21.15.62:0/3984248615 conn(0x7ffb200fc1e0 msgr2=0x7ffb200fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.174+0000 7ffb24dcf640 1 -- 172.21.15.62:0/3984248615 shutdown_connections 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.174+0000 7ffb24dcf640 1 -- 172.21.15.62:0/3984248615 wait complete. 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb24dcf640 1 Processor -- start 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb24dcf640 1 -- start start 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb24dcf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb201025e0 0x7ffb20197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb24dcf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 0x7ffb201983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb24dcf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb20107840 0x7ffb2019f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb24dcf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2010d7f0 con 0x7ffb20104f10 2024-09-15T23:25:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb24dcf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2010d670 con 0x7ffb20107840 2024-09-15T23:25:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb24dcf640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2010d970 con 0x7ffb201025e0 2024-09-15T23:25:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1dd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 0x7ffb201983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1dd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 0x7ffb201983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41688/0 (socket says 172.21.15.62:41688) 2024-09-15T23:25:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1dd9b640 1 -- 172.21.15.62:0/2952631169 learned_addr learned my addr 172.21.15.62:0/2952631169 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb20107840 0x7ffb2019f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb201025e0 0x7ffb20197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1dd9b640 1 -- 172.21.15.62:0/2952631169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb201025e0 msgr2=0x7ffb20197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:08.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1dd9b640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb201025e0 0x7ffb20197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1dd9b640 1 -- 172.21.15.62:0/2952631169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb20107840 msgr2=0x7ffb2019f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1dd9b640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb20107840 0x7ffb2019f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1dd9b640 1 -- 172.21.15.62:0/2952631169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2019fa90 con 0x7ffb20104f10 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb201025e0 0x7ffb20197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.175+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb20107840 0x7ffb2019f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.176+0000 7ffb1dd9b640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 0x7ffb201983b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ffb14031610 tx=0x7ffb14004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.176+0000 7ffb0f7fe640 1 -- 172.21.15.62:0/2952631169 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1403f030 con 0x7ffb20104f10 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.176+0000 7ffb0f7fe640 1 -- 172.21.15.62:0/2952631169 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb14003ad0 con 0x7ffb20104f10 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.176+0000 7ffb24dcf640 1 -- 172.21.15.62:0/2952631169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb2019fd20 con 0x7ffb20104f10 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.176+0000 7ffb0f7fe640 1 -- 172.21.15.62:0/2952631169 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb14035070 con 0x7ffb20104f10 2024-09-15T23:25:08.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.176+0000 7ffb24dcf640 1 -- 172.21.15.62:0/2952631169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb201a01c0 con 0x7ffb20104f10 2024-09-15T23:25:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.178+0000 7ffb0f7fe640 1 -- 172.21.15.62:0/2952631169 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ffb14042030 con 0x7ffb20104f10 2024-09-15T23:25:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb27c47740 1 Processor -- start 2024-09-15T23:25:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb27c47740 1 -- start start 2024-09-15T23:25:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d734ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d734fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d7350370 0x5652d7352830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb27c47740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5652d72ae2f0 con 0x5652d734a9a0 2024-09-15T23:25:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb27c47740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5652d7256270 con 0x5652d734b2e0 2024-09-15T23:25:08.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb27c47740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5652d72aa2b0 con 0x5652d7350370 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb1ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d734ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb1ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d734ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41698/0 (socket says 172.21.15.62:41698) 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb1ed9d640 1 -- 172.21.15.62:0/3876116014 learned_addr learned my addr 172.21.15.62:0/3876116014 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.179+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/3876116014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d734fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb0f7fe640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffb080a32f0 0x7ffb080a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb1ed9d640 1 -- 172.21.15.62:0/3876116014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d7350370 msgr2=0x5652d7352830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/3876116014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d7350370 0x5652d7352830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb1ed9d640 1 -- 172.21.15.62:0/3876116014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 msgr2=0x5652d734fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/3876116014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d734fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb1ed9d640 1 -- 172.21.15.62:0/3876116014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652d7255340 con 0x5652d734a9a0 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb0f7fe640 1 -- 172.21.15.62:0/2952631169 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ffb140f04c0 con 0x7ffb20104f10 2024-09-15T23:25:08.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffb080a32f0 0x7ffb080a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/3876116014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d734ada0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ffb1800f500 tx=0x7ffb1800f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9981a9efb9ae11e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb0cff9640 1 -- 172.21.15.62:0/3876116014 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1801c400 con 0x5652d734a9a0 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb0cff9640 1 -- 172.21.15.62:0/3876116014 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb180124b0 con 0x5652d734a9a0 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.180+0000 7ffb0cff9640 1 -- 172.21.15.62:0/3876116014 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18010890 con 0x5652d734a9a0 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb27c47740 1 -- 172.21.15.62:0/3876116014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 msgr2=0x5652d734ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3876116014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d734ada0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ffb1800f500 tx=0x7ffb1800f9c0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffb080a32f0 0x7ffb080a57b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffb18020000 tx=0x7ffb1801d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb27c47740 1 -- 172.21.15.62:0/3876116014 shutdown_connections 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3876116014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d7350370 0x5652d7352830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3876116014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d734fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3876116014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d734ada0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb27c47740 1 -- 172.21.15.62:0/3876116014 >> 172.21.15.62:0/3876116014 conn(0x5652d7346010 msgr2=0x5652d7346410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb27c47740 1 -- 172.21.15.62:0/3876116014 shutdown_connections 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.181+0000 7ffb27c47740 1 -- 172.21.15.62:0/3876116014 wait complete. 2024-09-15T23:25:08.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb27c47740 1 -- start start 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb27c47740 1 Processor -- start 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734a9a0 0x5652d7354c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d7355190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d7350370 0x5652d735c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb27c47740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5652d7353340 con 0x5652d7350370 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb27c47740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5652d7237290 con 0x5652d734b2e0 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb27c47740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5652d73534c0 con 0x5652d734a9a0 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb27c47740 10 client.0 _ll_register_callbacks cb 0x5652d7268140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d7350370 0x5652d735c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d7350370 0x5652d735c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41714/0 (socket says 172.21.15.62:41714) 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1ed9d640 1 -- 172.21.15.62:0/3063825381 learned_addr learned my addr 172.21.15.62:0/3063825381 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1dd9b640 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d7355190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734a9a0 0x5652d7354c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1ed9d640 1 -- 172.21.15.62:0/3063825381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734a9a0 msgr2=0x5652d7354c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734a9a0 0x5652d7354c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1ed9d640 1 -- 172.21.15.62:0/3063825381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 msgr2=0x5652d7355190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d7355190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.182+0000 7ffb1ed9d640 1 -- 172.21.15.62:0/3063825381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652d6fc1210 con 0x5652d7350370 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d7350370 0x5652d735c1d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ffb180109c0 tx=0x7ffb18009ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a068915de0a3eb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3063825381 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb18009cb0 con 0x5652d7350370 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3063825381 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ffb18010dc0 con 0x5652d7350370 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb27c47740 1 -- 172.21.15.62:0/3063825381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb180124b0 con 0x5652d7350370 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3063825381 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ffb1803c050 con 0x5652d7350370 2024-09-15T23:25:08.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb27c47740 10 client.17544 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb27c47740 1 -- 172.21.15.62:0/3063825381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5652d708d620 con 0x5652d7350370 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3063825381 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb180124b0 con 0x5652d7350370 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.183+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3063825381 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7ffb1802d070 con 0x5652d7350370 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.184+0000 7ffb27c47740 10 client.17544 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.184+0000 7ffb27c47740 10 client.17544 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.184+0000 7ffb27c47740 20 client.17544 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.184+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5652d7360760 0x5652d7362b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.184+0000 7ffb27c47740 4 client.17544 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.184+0000 7ffb27c47740 1 -- 172.21.15.62:0/3063825381 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652d72bba20 con 0x5652d7360760 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.184+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5652d7360760 0x5652d7362b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.184+0000 7ffb05ffb640 10 client.17544 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.184+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5652d7360760 0x5652d7362b50 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f173aa247b83a5db server_cookie=6785a6e0d108d0d3 in_seq=0 out_seq=0 2024-09-15T23:25:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3063825381 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5652d72bba20 con 0x5652d7360760 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb05ffb640 10 client.17544 handle_command_reply: tid=0 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 client.17544 shutdown 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 -- 172.21.15.62:0/3063825381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d7350370 msgr2=0x5652d735c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d7350370 0x5652d735c1d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7ffb180109c0 tx=0x7ffb18009ae0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 -- 172.21.15.62:0/3063825381 shutdown_connections 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5652d7360760 0x5652d7362b50 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d7350370 0x5652d735c1d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d7355190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb05ffb640 0 client.17544 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3063825381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734a9a0 0x5652d7354c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 -- 172.21.15.62:0/3063825381 >> 172.21.15.62:0/3063825381 conn(0x5652d7346010 msgr2=0x5652d7347820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 -- 172.21.15.62:0/3063825381 shutdown_connections 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 1 -- 172.21.15.62:0/3063825381 wait complete. 2024-09-15T23:25:08.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.185+0000 7ffb27c47740 20 client.17544 trim_cache size 0 max 16384 2024-09-15T23:25:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb27c47740 1 Processor -- start 2024-09-15T23:25:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb27c47740 1 -- start start 2024-09-15T23:25:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7346ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d735c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d7350370 0x5652d7347400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb27c47740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffb201acfa0 con 0x5652d734a9a0 2024-09-15T23:25:08.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb27c47740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffb201aa870 con 0x5652d734b2e0 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb27c47740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffb201a86c0 con 0x5652d7350370 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb1e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7346ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb1e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7346ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41718/0 (socket says 172.21.15.62:41718) 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.200+0000 7ffb1e59c640 1 -- 172.21.15.62:0/2394407453 learned_addr learned my addr 172.21.15.62:0/2394407453 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb1dd9b640 1 --2- 172.21.15.62:0/2394407453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d735c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb1e59c640 1 -- 172.21.15.62:0/2394407453 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d7350370 msgr2=0x5652d7347400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/2394407453 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d7350370 0x5652d7347400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb1e59c640 1 -- 172.21.15.62:0/2394407453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 msgr2=0x5652d735c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/2394407453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d735c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb1e59c640 1 -- 172.21.15.62:0/2394407453 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652d7345e30 con 0x5652d734a9a0 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/2394407453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7346ec0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x5652d734bd10 tx=0x7ffb10005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1ef026cbd5ce28 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb07fff640 1 -- 172.21.15.62:0/2394407453 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb10003d50 con 0x5652d734a9a0 2024-09-15T23:25:08.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb07fff640 1 -- 172.21.15.62:0/2394407453 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb1001e900 con 0x5652d734a9a0 2024-09-15T23:25:08.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb07fff640 1 -- 172.21.15.62:0/2394407453 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb10012ae0 con 0x5652d734a9a0 2024-09-15T23:25:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb27c47740 1 -- 172.21.15.62:0/2394407453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 msgr2=0x5652d7346ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.201+0000 7ffb27c47740 1 --2- 172.21.15.62:0/2394407453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7346ec0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x5652d734bd10 tx=0x7ffb10005dc0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 -- 172.21.15.62:0/2394407453 shutdown_connections 2024-09-15T23:25:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 --2- 172.21.15.62:0/2394407453 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d7350370 0x5652d7347400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 --2- 172.21.15.62:0/2394407453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d734b2e0 0x5652d735c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 --2- 172.21.15.62:0/2394407453 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7346ec0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 -- 172.21.15.62:0/2394407453 >> 172.21.15.62:0/2394407453 conn(0x5652d7346010 msgr2=0x5652d7354c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 -- 172.21.15.62:0/2394407453 shutdown_connections 2024-09-15T23:25:08.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 -- 172.21.15.62:0/2394407453 wait complete. 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 -- start start 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 Processor -- start 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7359030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734b2e0 0x5652d7359570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d7350370 0x5652d73c9390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffb201a8d30 con 0x5652d734a9a0 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffb201aa1a0 con 0x5652d7350370 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ffb201a9400 con 0x5652d734b2e0 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb27c47740 10 client.0 _ll_register_callbacks cb 0x7ffb1801f020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:08.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb1e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7359030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb1e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7359030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41728/0 (socket says 172.21.15.62:41728) 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb1e59c640 1 -- 172.21.15.62:0/3701054745 learned_addr learned my addr 172.21.15.62:0/3701054745 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d7350370 0x5652d73c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.202+0000 7ffb1dd9b640 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734b2e0 0x5652d7359570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb1e59c640 1 -- 172.21.15.62:0/3701054745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734b2e0 msgr2=0x5652d7359570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734b2e0 0x5652d7359570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb1e59c640 1 -- 172.21.15.62:0/3701054745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d7350370 msgr2=0x5652d73c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d7350370 0x5652d73c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb1e59c640 1 -- 172.21.15.62:0/3701054745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652d73c98d0 con 0x5652d734a9a0 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb1ed9d640 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d7350370 0x5652d73c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb1e59c640 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7359030 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009460 tx=0x7ffb100125b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450b5589c20d63aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3701054745 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb10003d50 con 0x5652d734a9a0 2024-09-15T23:25:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3701054745 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ffb1001e420 con 0x5652d734a9a0 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb27c47740 1 -- 172.21.15.62:0/3701054745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb1001e900 con 0x5652d734a9a0 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3701054745 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ffb1003b050 con 0x5652d734a9a0 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb27c47740 10 client.17550 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb27c47740 1 -- 172.21.15.62:0/3701054745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5652d73cab40 con 0x5652d734a9a0 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.203+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3701054745 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb1001e900 con 0x5652d734a9a0 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3701054745 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7ffb1000f040 con 0x5652d734a9a0 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb27c47740 10 client.17550 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb27c47740 10 client.17550 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb27c47740 20 client.17550 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5652d73cb4f0 0x5652d73cd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb27c47740 4 client.17550 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb27c47740 1 -- 172.21.15.62:0/3701054745 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5652d72b5d70 con 0x5652d73cb4f0 2024-09-15T23:25:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb1dd9b640 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5652d73cb4f0 0x5652d73cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb05ffb640 10 client.17550 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:08.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.204+0000 7ffb1dd9b640 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5652d73cb4f0 0x5652d73cd910 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3f39029a730f69 server_cookie=90c53a7e45f4041c in_seq=0 out_seq=0 2024-09-15T23:25:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb05ffb640 1 -- 172.21.15.62:0/3701054745 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5652d72b5d70 con 0x5652d73cb4f0 2024-09-15T23:25:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb05ffb640 10 client.17550 handle_command_reply: tid=0 2024-09-15T23:25:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 client.17550 shutdown 2024-09-15T23:25:08.214 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 -- 172.21.15.62:0/3701054745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 msgr2=0x5652d7359030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7359030 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009460 tx=0x7ffb100125b0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 -- 172.21.15.62:0/3701054745 shutdown_connections 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5652d73cb4f0 0x5652d73cd910 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5652d7350370 0x5652d73c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5652d734b2e0 0x5652d7359570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb05ffb640 0 client.17550 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 --2- 172.21.15.62:0/3701054745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5652d734a9a0 0x5652d7359030 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 -- 172.21.15.62:0/3701054745 >> 172.21.15.62:0/3701054745 conn(0x5652d7346010 msgr2=0x5652d73449a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 -- 172.21.15.62:0/3701054745 shutdown_connections 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 1 -- 172.21.15.62:0/3701054745 wait complete. 2024-09-15T23:25:08.215 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.213+0000 7ffb27c47740 20 client.17550 trim_cache size 0 max 16384 2024-09-15T23:25:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:25:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:25:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:25:08.216 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 -- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffb080a32f0 msgr2=0x7ffb080a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffb080a32f0 0x7ffb080a57b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffb18020000 tx=0x7ffb1801d000 comp rx=0 tx=0).stop 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 -- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 msgr2=0x7ffb201983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 0x7ffb201983b0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ffb14031610 tx=0x7ffb14004270 comp rx=0 tx=0).stop 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 -- 172.21.15.62:0/2952631169 shutdown_connections 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ffb080a32f0 0x7ffb080a57b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb20107840 0x7ffb2019f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb20104f10 0x7ffb201983b0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 --2- 172.21.15.62:0/2952631169 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ffb201025e0 0x7ffb20197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 -- 172.21.15.62:0/2952631169 >> 172.21.15.62:0/2952631169 conn(0x7ffb200fc1e0 msgr2=0x7ffb20102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:08.217 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 -- 172.21.15.62:0/2952631169 shutdown_connections 2024-09-15T23:25:08.218 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:08.214+0000 7ffb24dcf640 1 -- 172.21.15.62:0/2952631169 wait complete. 2024-09-15T23:25:09.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:09 smithi067 ceph-mon[35665]: pgmap v636: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 286 KiB/s wr, 34 op/s 2024-09-15T23:25:09.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:09 smithi148 ceph-mon[35610]: pgmap v636: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 286 KiB/s wr, 34 op/s 2024-09-15T23:25:09.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:09 smithi062 ceph-mon[30949]: pgmap v636: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 286 KiB/s wr, 34 op/s 2024-09-15T23:25:09.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 530.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:25:11.199 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:25:11.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:11 smithi067 ceph-mon[35665]: pgmap v637: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-09-15T23:25:11.282 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:11 smithi062 ceph-mon[30949]: pgmap v637: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-09-15T23:25:11.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:11 smithi148 ceph-mon[35610]: pgmap v637: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 2.5 KiB/s wr, 7 op/s 2024-09-15T23:25:11.472 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:11.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.914+0000 7fbe968a2640 1 -- 172.21.15.62:0/1985223548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90102d90 msgr2=0x7fbe90103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:11.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.914+0000 7fbe968a2640 1 --2- 172.21.15.62:0/1985223548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90102d90 0x7fbe90103170 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009930 tx=0x7fbe8002a420 comp rx=0 tx=0).stop 2024-09-15T23:25:11.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.914+0000 7fbe968a2640 1 -- 172.21.15.62:0/1985223548 shutdown_connections 2024-09-15T23:25:11.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.914+0000 7fbe968a2640 1 --2- 172.21.15.62:0/1985223548 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe90108290 0x7fbe9010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.914+0000 7fbe968a2640 1 --2- 172.21.15.62:0/1985223548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe90103740 0x7fbe90107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.914+0000 7fbe968a2640 1 --2- 172.21.15.62:0/1985223548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90102d90 0x7fbe90103170 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.914+0000 7fbe968a2640 1 -- 172.21.15.62:0/1985223548 >> 172.21.15.62:0/1985223548 conn(0x7fbe900fc7d0 msgr2=0x7fbe900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.914+0000 7fbe968a2640 1 -- 172.21.15.62:0/1985223548 shutdown_connections 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.914+0000 7fbe968a2640 1 -- 172.21.15.62:0/1985223548 wait complete. 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.915+0000 7fbe968a2640 1 Processor -- start 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.915+0000 7fbe968a2640 1 -- start start 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.915+0000 7fbe968a2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe90102d90 0x7fbe901985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.915+0000 7fbe968a2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe90103740 0x7fbe90198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.915+0000 7fbe968a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90108290 0x7fbe9019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.915+0000 7fbe968a2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbe901120d0 con 0x7fbe90108290 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.915+0000 7fbe968a2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe90111f50 con 0x7fbe90103740 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.915+0000 7fbe968a2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbe90112250 con 0x7fbe90102d90 2024-09-15T23:25:11.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe94e3f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90108290 0x7fbe9019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe8ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe90102d90 0x7fbe901985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe94e3f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90108290 0x7fbe9019cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41740/0 (socket says 172.21.15.62:41740) 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe94e3f640 1 -- 172.21.15.62:0/2275589194 learned_addr learned my addr 172.21.15.62:0/2275589194 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe8f7fe640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe90103740 0x7fbe90198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe94e3f640 1 -- 172.21.15.62:0/2275589194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe90102d90 msgr2=0x7fbe901985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe94e3f640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe90102d90 0x7fbe901985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe94e3f640 1 -- 172.21.15.62:0/2275589194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe90103740 msgr2=0x7fbe90198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe94e3f640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe90103740 0x7fbe90198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe94e3f640 1 -- 172.21.15.62:0/2275589194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9019d580 con 0x7fbe90108290 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe94e3f640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90108290 0x7fbe9019cea0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800b970 tx=0x7fbe8800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.916+0000 7fbe8d7fa640 1 -- 172.21.15.62:0/2275589194 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88019070 con 0x7fbe90108290 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.917+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe9019d870 con 0x7fbe90108290 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.917+0000 7fbe8d7fa640 1 -- 172.21.15.62:0/2275589194 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe8800cb40 con 0x7fbe90108290 2024-09-15T23:25:11.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.917+0000 7fbe8d7fa640 1 -- 172.21.15.62:0/2275589194 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88007650 con 0x7fbe90108290 2024-09-15T23:25:11.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.917+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe90100920 con 0x7fbe90108290 2024-09-15T23:25:11.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.918+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9006a7f0 con 0x7fbe90108290 2024-09-15T23:25:11.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.919+0000 7fbe8d7fa640 1 -- 172.21.15.62:0/2275589194 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fbe8801c070 con 0x7fbe90108290 2024-09-15T23:25:11.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.920+0000 7fbe8d7fa640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbe780a3240 0x7fbe780a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:11.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.920+0000 7fbe8ffff640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbe780a3240 0x7fbe780a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:11.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.920+0000 7fbe8d7fa640 1 -- 172.21.15.62:0/2275589194 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbe880cf730 con 0x7fbe90108290 2024-09-15T23:25:11.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.920+0000 7fbe8ffff640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbe780a3240 0x7fbe780a5700 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009900 tx=0x7fbe800058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:11.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:11.924+0000 7fbe8d7fa640 1 -- 172.21.15.62:0/2275589194 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbe8809ba80 con 0x7fbe90108290 2024-09-15T23:25:12.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.142+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe9010a2a0 con 0x7fbe90108290 2024-09-15T23:25:12.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.143+0000 7fbe8d7fa640 1 -- 172.21.15.62:0/2275589194 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7fbe8809fd80 con 0x7fbe90108290 2024-09-15T23:25:12.144 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:25:12.144 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:25:12.145 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.145+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbe780a3240 msgr2=0x7fbe780a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.145+0000 7fbe968a2640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbe780a3240 0x7fbe780a5700 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009900 tx=0x7fbe800058d0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.145+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90108290 msgr2=0x7fbe9019cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.145+0000 7fbe968a2640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90108290 0x7fbe9019cea0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800b970 tx=0x7fbe8800be30 comp rx=0 tx=0).stop 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.146+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 shutdown_connections 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.146+0000 7fbe968a2640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbe780a3240 0x7fbe780a5700 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.146+0000 7fbe968a2640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe90108290 0x7fbe9019cea0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.146+0000 7fbe968a2640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe90103740 0x7fbe90198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.146+0000 7fbe968a2640 1 --2- 172.21.15.62:0/2275589194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbe90102d90 0x7fbe901985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:12.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.146+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 >> 172.21.15.62:0/2275589194 conn(0x7fbe900fc7d0 msgr2=0x7fbe90105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:12.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.146+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 shutdown_connections 2024-09-15T23:25:12.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:12.146+0000 7fbe968a2640 1 -- 172.21.15.62:0/2275589194 wait complete. 2024-09-15T23:25:12.949 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 590f31d6 --release --await' 2024-09-15T23:25:12.949 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 590f31d6 --release --await 2024-09-15T23:25:13.210 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:13 smithi062 ceph-mon[30949]: pgmap v638: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.7 KiB/s wr, 10 op/s 2024-09-15T23:25:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:13 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2275589194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:13.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:13 smithi148 ceph-mon[35610]: pgmap v638: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.7 KiB/s wr, 10 op/s 2024-09-15T23:25:13.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:13 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2275589194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:13 smithi067 ceph-mon[35665]: pgmap v638: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.7 KiB/s wr, 10 op/s 2024-09-15T23:25:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:13 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2275589194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:13.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.687+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/2475813901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 msgr2=0x7fa19c101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.687+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/2475813901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 0x7fa19c101200 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fa18c009900 tx=0x7fa18c02a480 comp rx=0 tx=0).stop 2024-09-15T23:25:13.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.687+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/2475813901 shutdown_connections 2024-09-15T23:25:13.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.687+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/2475813901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa19c102ef0 0x7fa19c10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.688+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/2475813901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19c102570 0x7fa19c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.688+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/2475813901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 0x7fa19c101200 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.688+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/2475813901 >> 172.21.15.62:0/2475813901 conn(0x7fa19c0fc7d0 msgr2=0x7fa19c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.688+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/2475813901 shutdown_connections 2024-09-15T23:25:13.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.688+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/2475813901 wait complete. 2024-09-15T23:25:13.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.688+0000 7fa1a2a3b640 1 Processor -- start 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.688+0000 7fa1a2a3b640 1 -- start start 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa1a2a3b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 0x7fa19c199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa1a2a3b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19c102570 0x7fa19c199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa1a2a3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa19c102ef0 0x7fa19c1a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa1a2a3b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c110f80 con 0x7fa19c100e20 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa1a2a3b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c110e00 con 0x7fa19c102570 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa1a2a3b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c111100 con 0x7fa19c102ef0 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa19bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 0x7fa19c199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa19bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 0x7fa19c199310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43494/0 (socket says 172.21.15.62:43494) 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa19bfff640 1 -- 172.21.15.62:0/55764277 learned_addr learned my addr 172.21.15.62:0/55764277 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:13.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa193fff640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19c102570 0x7fa19c199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa19c102ef0 0x7fa19c1a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa19bfff640 1 -- 172.21.15.62:0/55764277 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa19c102ef0 msgr2=0x7fa19c1a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa19bfff640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa19c102ef0 0x7fa19c1a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa19bfff640 1 -- 172.21.15.62:0/55764277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19c102570 msgr2=0x7fa19c199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa19bfff640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19c102570 0x7fa19c199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa19bfff640 1 -- 172.21.15.62:0/55764277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19c1a48d0 con 0x7fa19c100e20 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.689+0000 7fa193fff640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19c102570 0x7fa19c199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.690+0000 7fa19bfff640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 0x7fa19c199310 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fa18c006010 tx=0x7fa18c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.690+0000 7fa199ffb640 1 -- 172.21.15.62:0/55764277 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c03f070 con 0x7fa19c100e20 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.690+0000 7fa199ffb640 1 -- 172.21.15.62:0/55764277 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa18c031c30 con 0x7fa19c100e20 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.690+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/55764277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa19c1a4aa0 con 0x7fa19c100e20 2024-09-15T23:25:13.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.690+0000 7fa199ffb640 1 -- 172.21.15.62:0/55764277 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c035030 con 0x7fa19c100e20 2024-09-15T23:25:13.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.690+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/55764277 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa19c1a4ef0 con 0x7fa19c100e20 2024-09-15T23:25:13.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.692+0000 7fa199ffb640 1 -- 172.21.15.62:0/55764277 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa18c042070 con 0x7fa19c100e20 2024-09-15T23:25:13.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a4e14740 1 Processor -- start 2024-09-15T23:25:13.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a4e14740 1 -- start start 2024-09-15T23:25:13.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f78300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f87110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f89a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d723edd620 con 0x55d723f77f20 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d723e84e50 con 0x55d723f78840 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d723ed95e0 con 0x55d723f87650 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a0fd8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f78300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a0fd8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f78300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43500/0 (socket says 172.21.15.62:43500) 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a0fd8640 1 -- 172.21.15.62:0/3719215674 learned_addr learned my addr 172.21.15.62:0/3719215674 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa19bfff640 1 --2- 172.21.15.62:0/3719215674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f87110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa193fff640 1 --2- 172.21.15.62:0/3719215674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f89a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa199ffb640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1880a3240 0x7fa1880a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a0fd8640 1 -- 172.21.15.62:0/3719215674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 msgr2=0x55d723f89a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.693+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/3719215674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f89a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.694+0000 7fa1a0fd8640 1 -- 172.21.15.62:0/3719215674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 msgr2=0x55d723f87110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.694+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/3719215674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f87110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.694+0000 7fa1a0fd8640 1 -- 172.21.15.62:0/3719215674 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d723e75d40 con 0x55d723f77f20 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.694+0000 7fa199ffb640 1 -- 172.21.15.62:0/55764277 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa18c0f04b0 con 0x7fa19c100e20 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.694+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1880a3240 0x7fa1880a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.694+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/3719215674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f78300 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fa194009fd0 tx=0x7fa19400f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cce566dc9e05988 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.694+0000 7fa1927fc640 1 -- 172.21.15.62:0/3719215674 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19401c910 con 0x55d723f77f20 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.694+0000 7fa1927fc640 1 -- 172.21.15.62:0/3719215674 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19401b440 con 0x55d723f77f20 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.694+0000 7fa1927fc640 1 -- 172.21.15.62:0/3719215674 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194010630 con 0x55d723f77f20 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3719215674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 msgr2=0x55d723f78300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1880a3240 0x7fa1880a5700 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa194020000 tx=0x7fa19401d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3719215674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f78300 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fa194009fd0 tx=0x7fa19400f870 comp rx=0 tx=0).stop 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3719215674 shutdown_connections 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3719215674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f89a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3719215674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f87110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3719215674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f78300 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3719215674 >> 172.21.15.62:0/3719215674 conn(0x55d723f745f0 msgr2=0x55d723f749d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3719215674 shutdown_connections 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3719215674 wait complete. 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.695+0000 7fa1a4e14740 1 -- start start 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a4e14740 1 Processor -- start 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f845b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f84af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f85030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55d723f8a220 con 0x55d723f77f20 2024-09-15T23:25:13.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55d723e65f10 con 0x55d723f78840 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55d723f8a3a0 con 0x55d723f87650 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a4e14740 10 client.0 _ll_register_callbacks cb 0x55d723e89db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa19bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f845b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa19bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f845b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43506/0 (socket says 172.21.15.62:43506) 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa19bfff640 1 -- 172.21.15.62:0/856355704 learned_addr learned my addr 172.21.15.62:0/856355704 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f85030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa193fff640 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f84af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa19bfff640 1 -- 172.21.15.62:0/856355704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 msgr2=0x55d723f85030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa19bfff640 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f85030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa19bfff640 1 -- 172.21.15.62:0/856355704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 msgr2=0x55d723f84af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa19bfff640 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f84af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa19bfff640 1 -- 172.21.15.62:0/856355704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d723bec580 con 0x55d723f77f20 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.696+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f85030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa19bfff640 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f845b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa18c005ec0 tx=0x7fa18c0b8270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5652310e02ddb730 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa17b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa17b7fe640 1 -- 172.21.15.62:0/856355704 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c049070 con 0x55d723f77f20 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa17b7fe640 1 -- 172.21.15.62:0/856355704 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa18c03b460 con 0x55d723f77f20 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa1a4e14740 1 -- 172.21.15.62:0/856355704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa19401b440 con 0x55d723f77f20 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa17b7fe640 1 -- 172.21.15.62:0/856355704 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa18c0bc230 con 0x55d723f77f20 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa1a4e14740 10 client.17568 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa1a4e14740 1 -- 172.21.15.62:0/856355704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d723cb6960 con 0x55d723f77f20 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa17b7fe640 1 -- 172.21.15.62:0/856355704 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa19401b440 con 0x55d723f77f20 2024-09-15T23:25:13.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa17b7fe640 1 -- 172.21.15.62:0/856355704 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa18c0c4ed0 con 0x55d723f77f20 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa1a4e14740 10 client.17568 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa1a4e14740 10 client.17568 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa1a4e14740 20 client.17568 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d723f97270 0x55d723f996a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa1a4e14740 4 client.17568 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.697+0000 7fa1a4e14740 1 -- 172.21.15.62:0/856355704 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d723eeab90 con 0x55d723f97270 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.698+0000 7fa193fff640 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d723f97270 0x55d723f996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.698+0000 7fa193fff640 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d723f97270 0x55d723f996a0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=908808fe63ef34dd server_cookie=7acf9988bb9542d2 in_seq=0 out_seq=0 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.698+0000 7fa17b7fe640 10 client.17568 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa17b7fe640 1 -- 172.21.15.62:0/856355704 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d723eeab90 con 0x55d723f97270 2024-09-15T23:25:13.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa17b7fe640 10 client.17568 handle_command_reply: tid=0 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 client.17568 shutdown 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 -- 172.21.15.62:0/856355704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 msgr2=0x55d723f845b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f845b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa18c005ec0 tx=0x7fa18c0b8270 comp rx=0 tx=0).stop 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 -- 172.21.15.62:0/856355704 shutdown_connections 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d723f97270 0x55d723f996a0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f85030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f84af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa17b7fe640 0 client.17568 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/856355704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f77f20 0x55d723f845b0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 -- 172.21.15.62:0/856355704 >> 172.21.15.62:0/856355704 conn(0x55d723f745f0 msgr2=0x55d723f7a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 -- 172.21.15.62:0/856355704 shutdown_connections 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 1 -- 172.21.15.62:0/856355704 wait complete. 2024-09-15T23:25:13.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.699+0000 7fa1a4e14740 20 client.17568 trim_cache size 0 max 16384 2024-09-15T23:25:13.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.716+0000 7fa1a4e14740 1 Processor -- start 2024-09-15T23:25:13.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.716+0000 7fa1a4e14740 1 -- start start 2024-09-15T23:25:13.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f77f20 0x55d723f84c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f78840 0x55d723f851c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f9ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c1af260 con 0x55d723f78840 2024-09-15T23:25:13.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c1acb30 con 0x55d723f77f20 2024-09-15T23:25:13.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c1aa980 con 0x55d723f87650 2024-09-15T23:25:13.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa193fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f78840 0x55d723f851c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa193fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f78840 0x55d723f851c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43514/0 (socket says 172.21.15.62:43514) 2024-09-15T23:25:13.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa193fff640 1 -- 172.21.15.62:0/3147416229 learned_addr learned my addr 172.21.15.62:0/3147416229 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:13.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa19bfff640 1 --2- 172.21.15.62:0/3147416229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f77f20 0x55d723f84c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/3147416229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f9ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa193fff640 1 -- 172.21.15.62:0/3147416229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 msgr2=0x55d723f9ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa193fff640 1 --2- 172.21.15.62:0/3147416229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f9ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa193fff640 1 -- 172.21.15.62:0/3147416229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f77f20 msgr2=0x55d723f84c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa193fff640 1 --2- 172.21.15.62:0/3147416229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f77f20 0x55d723f84c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa193fff640 1 -- 172.21.15.62:0/3147416229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d723f99cd0 con 0x55d723f78840 2024-09-15T23:25:13.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa193fff640 1 --2- 172.21.15.62:0/3147416229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f78840 0x55d723f851c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa19c06aac0 tx=0x7fa184005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa43cec10a9c5b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa191ffb640 1 -- 172.21.15.62:0/3147416229 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184004310 con 0x55d723f78840 2024-09-15T23:25:13.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa191ffb640 1 -- 172.21.15.62:0/3147416229 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1840044b0 con 0x55d723f78840 2024-09-15T23:25:13.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.717+0000 7fa191ffb640 1 -- 172.21.15.62:0/3147416229 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa184011b30 con 0x55d723f78840 2024-09-15T23:25:13.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.718+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3147416229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f78840 msgr2=0x55d723f851c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.718+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3147416229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f78840 0x55d723f851c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa19c06aac0 tx=0x7fa184005f90 comp rx=0 tx=0).stop 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.718+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3147416229 shutdown_connections 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.718+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3147416229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f87650 0x55d723f9ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.718+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3147416229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f78840 0x55d723f851c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.718+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3147416229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f77f20 0x55d723f84c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.718+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3147416229 >> 172.21.15.62:0/3147416229 conn(0x55d723f745f0 msgr2=0x55d723f84570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.718+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3147416229 shutdown_connections 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.718+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3147416229 wait complete. 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a4e14740 1 -- start start 2024-09-15T23:25:13.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a4e14740 1 Processor -- start 2024-09-15T23:25:13.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f77f20 0x55d723f8fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f8c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a4e14740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f87650 0x55d723f8cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c1aaff0 con 0x55d723f87650 2024-09-15T23:25:13.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c1ac460 con 0x55d723f78840 2024-09-15T23:25:13.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a4e14740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa19c1ab6c0 con 0x55d723f77f20 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a0fd8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f87650 0x55d723f8cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a4e14740 10 client.0 _ll_register_callbacks cb 0x7fa18c0f3be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a0fd8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f87650 0x55d723f8cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43528/0 (socket says 172.21.15.62:43528) 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa1a0fd8640 1 -- 172.21.15.62:0/3138207298 learned_addr learned my addr 172.21.15.62:0/3138207298 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.719+0000 7fa193fff640 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f8c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa1a0fd8640 1 -- 172.21.15.62:0/3138207298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f77f20 msgr2=0x55d723f8fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa19bfff640 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f77f20 0x55d723f8fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f77f20 0x55d723f8fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa1a0fd8640 1 -- 172.21.15.62:0/3138207298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 msgr2=0x55d723f8c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f8c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa1a0fd8640 1 -- 172.21.15.62:0/3138207298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d723f8d230 con 0x55d723f87650 2024-09-15T23:25:13.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa193fff640 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f8c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:13.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa19bfff640 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f77f20 0x55d723f8fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:13.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa1a0fd8640 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f87650 0x55d723f8cb40 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fa194010680 tx=0x7fa1940106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403cb9f5a4b7a9f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:13.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa17b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:13.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa17b7fe640 1 -- 172.21.15.62:0/3138207298 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa194037060 con 0x55d723f87650 2024-09-15T23:25:13.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa17b7fe640 1 -- 172.21.15.62:0/3138207298 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa194010ae0 con 0x55d723f87650 2024-09-15T23:25:13.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3138207298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1840044b0 con 0x55d723f87650 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa17b7fe640 1 -- 172.21.15.62:0/3138207298 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa19403e050 con 0x55d723f87650 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa1a4e14740 10 client.17574 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3138207298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d724002e60 con 0x55d723f87650 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.720+0000 7fa17b7fe640 1 -- 172.21.15.62:0/3138207298 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1840044b0 con 0x55d723f87650 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa17b7fe640 1 -- 172.21.15.62:0/3138207298 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa194026480 con 0x55d723f87650 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa1a4e14740 10 client.17574 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa1a4e14740 10 client.17574 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa1a4e14740 20 client.17574 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d7240033c0 0x55d7240057e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa1a4e14740 4 client.17574 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "590f31d6", "release": true, "await": true}] 2024-09-15T23:25:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3138207298 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "590f31d6", "release": true, "await": true}) -- 0x55d723ee50a0 con 0x55d7240033c0 2024-09-15T23:25:13.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa19bfff640 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d7240033c0 0x55d7240057e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:13.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa17b7fe640 10 client.17574 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:13.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.721+0000 7fa19bfff640 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d7240033c0 0x55d7240057e0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7448e996ecd843b server_cookie=4a74f0cf5d0c72f in_seq=0 out_seq=0 2024-09-15T23:25:13.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa17b7fe640 1 -- 172.21.15.62:0/3138207298 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55d723ee50a0 con 0x55d7240033c0 2024-09-15T23:25:13.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa17b7fe640 10 client.17574 handle_command_reply: tid=0 2024-09-15T23:25:13.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 client.17574 shutdown 2024-09-15T23:25:13.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3138207298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f87650 msgr2=0x55d723f8cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f87650 0x55d723f8cb40 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fa194010680 tx=0x7fa1940106b0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3138207298 shutdown_connections 2024-09-15T23:25:13.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55d7240033c0 0x55d7240057e0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d723f87650 0x55d723f8cb40 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55d723f78840 0x55d723f8c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 --2- 172.21.15.62:0/3138207298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55d723f77f20 0x55d723f8fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3138207298 >> 172.21.15.62:0/3138207298 conn(0x55d723f745f0 msgr2=0x55d723f84d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa17b7fe640 0 client.17574 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3138207298 shutdown_connections 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 1 -- 172.21.15.62:0/3138207298 wait complete. 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.767+0000 7fa1a4e14740 20 client.17574 trim_cache size 0 max 16384 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:25:13.771 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 35, 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 36, 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "590f31d6": { 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 36, 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 142.3, 2024-09-15T23:25:13.772 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:25:13.773 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1880a3240 msgr2=0x7fa1880a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1880a3240 0x7fa1880a5700 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa194020000 tx=0x7fa19401d000 comp rx=0 tx=0).stop 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 msgr2=0x7fa19c199310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 0x7fa19c199310 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fa18c006010 tx=0x7fa18c003c10 comp rx=0 tx=0).stop 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/55764277 shutdown_connections 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1880a3240 0x7fa1880a5700 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa19c102ef0 0x7fa19c1a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa19c102570 0x7fa19c199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 --2- 172.21.15.62:0/55764277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa19c100e20 0x7fa19c199310 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/55764277 >> 172.21.15.62:0/55764277 conn(0x7fa19c0fc7d0 msgr2=0x7fa19c0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:13.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/55764277 shutdown_connections 2024-09-15T23:25:13.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:13.768+0000 7fa1a2a3b640 1 -- 172.21.15.62:0/55764277 wait complete. 2024-09-15T23:25:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 535.0 s, 136.9 tps, lat 3084.522 ms stddev 7238.916 2024-09-15T23:25:14.669 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '590f31d6', total seconds elapsed: 25.8 2024-09-15T23:25:14.744 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:25:15.184 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:15.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:15 smithi148 ceph-mon[35610]: pgmap v639: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-09-15T23:25:15.318 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:15 smithi062 ceph-mon[30949]: pgmap v639: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-09-15T23:25:15.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:15 smithi067 ceph-mon[35665]: pgmap v639: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-09-15T23:25:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.063+0000 7f2462f25640 1 -- 172.21.15.62:0/1483050387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c109450 msgr2=0x7f245c110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.063+0000 7f2462f25640 1 --2- 172.21.15.62:0/1483050387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c109450 0x7f245c110250 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f2454009ca0 tx=0x7f245402a8d0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.064+0000 7f2462f25640 1 -- 172.21.15.62:0/1483050387 shutdown_connections 2024-09-15T23:25:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.064+0000 7f2462f25640 1 --2- 172.21.15.62:0/1483050387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f245c11b140 0x7f245c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.064+0000 7f2462f25640 1 --2- 172.21.15.62:0/1483050387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f245c109830 0x7f245c109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.064+0000 7f2462f25640 1 --2- 172.21.15.62:0/1483050387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c109450 0x7f245c110250 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.064+0000 7f2462f25640 1 -- 172.21.15.62:0/1483050387 >> 172.21.15.62:0/1483050387 conn(0x7f245c06e140 msgr2=0x7f245c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:16.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.065+0000 7f2462f25640 1 -- 172.21.15.62:0/1483050387 shutdown_connections 2024-09-15T23:25:16.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.065+0000 7f2462f25640 1 -- 172.21.15.62:0/1483050387 wait complete. 2024-09-15T23:25:16.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.066+0000 7f2462f25640 1 Processor -- start 2024-09-15T23:25:16.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.066+0000 7f2462f25640 1 -- start start 2024-09-15T23:25:16.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.066+0000 7f2462f25640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f245c109830 0x7f245c1a7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.066+0000 7f2462f25640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f245c11b140 0x7f245c1a79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.066+0000 7f2462f25640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c1abd30 0x7f245c1ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.066+0000 7f2462f25640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f245c120df0 con 0x7f245c1abd30 2024-09-15T23:25:16.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.066+0000 7f2462f25640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f245c120c70 con 0x7f245c11b140 2024-09-15T23:25:16.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.066+0000 7f2462f25640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f245c120f70 con 0x7f245c109830 2024-09-15T23:25:16.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.067+0000 7f24614c2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c1abd30 0x7f245c1ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.067+0000 7f24614c2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c1abd30 0x7f245c1ac1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43544/0 (socket says 172.21.15.62:43544) 2024-09-15T23:25:16.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.067+0000 7f24614c2640 1 -- 172.21.15.62:0/3067540846 learned_addr learned my addr 172.21.15.62:0/3067540846 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:16.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.067+0000 7f245bfff640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f245c11b140 0x7f245c1a79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.068+0000 7f24614c2640 1 -- 172.21.15.62:0/3067540846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f245c109830 msgr2=0x7f245c1a7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:16.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.068+0000 7f24614c2640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f245c109830 0x7f245c1a7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.068+0000 7f24614c2640 1 -- 172.21.15.62:0/3067540846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f245c11b140 msgr2=0x7f245c1a79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:16.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.068+0000 7f24614c2640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f245c11b140 0x7f245c1a79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.068+0000 7f24614c2640 1 -- 172.21.15.62:0/3067540846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245c1ac7f0 con 0x7f245c1abd30 2024-09-15T23:25:16.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.069+0000 7f24614c2640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c1abd30 0x7f245c1ac1e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f245000ae50 tx=0x7f245000cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:16.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.069+0000 7f2459ffb640 1 -- 172.21.15.62:0/3067540846 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2450007bb0 con 0x7f245c1abd30 2024-09-15T23:25:16.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.069+0000 7f2459ffb640 1 -- 172.21.15.62:0/3067540846 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2450007d50 con 0x7f245c1abd30 2024-09-15T23:25:16.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.070+0000 7f2459ffb640 1 -- 172.21.15.62:0/3067540846 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2450031460 con 0x7f245c1abd30 2024-09-15T23:25:16.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.071+0000 7f2462f25640 1 -- 172.21.15.62:0/3067540846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f245c1acae0 con 0x7f245c1abd30 2024-09-15T23:25:16.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.071+0000 7f2462f25640 1 -- 172.21.15.62:0/3067540846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f245c110bb0 con 0x7f245c1abd30 2024-09-15T23:25:16.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.076+0000 7f2459ffb640 1 -- 172.21.15.62:0/3067540846 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2450004eb0 con 0x7f245c1abd30 2024-09-15T23:25:16.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.078+0000 7f2459ffb640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24440a3280 0x7f24440a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.078+0000 7f2459ffb640 1 -- 172.21.15.62:0/3067540846 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f24500f1b60 con 0x7f245c1abd30 2024-09-15T23:25:16.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.079+0000 7f24652fe740 1 Processor -- start 2024-09-15T23:25:16.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.080+0000 7f24652fe740 1 -- start start 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.080+0000 7f24652fe740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f80c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.080+0000 7f24652fe740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f817140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.080+0000 7f24652fe740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 0x55c42f819a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.080+0000 7f24652fe740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c42f7722e0 con 0x55c42f80c4a0 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.080+0000 7f24652fe740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c42f719ce0 con 0x55c42f817680 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.080+0000 7f24652fe740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c42f76e2a0 con 0x55c42f80cdc0 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.081+0000 7f24614c2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 0x55c42f819a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.084+0000 7f2460cc1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f80c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.084+0000 7f2460cc1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f80c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43554/0 (socket says 172.21.15.62:43554) 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.084+0000 7f2460cc1640 1 -- 172.21.15.62:0/2537287179 learned_addr learned my addr 172.21.15.62:0/2537287179 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.084+0000 7f2460cc1640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24440a3280 0x7f24440a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.086+0000 7f245bfff640 1 --2- 172.21.15.62:0/2537287179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f817140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.088+0000 7f2460cc1640 1 -- 172.21.15.62:0/2537287179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 msgr2=0x55c42f817140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.088+0000 7f2460cc1640 1 --2- 172.21.15.62:0/2537287179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f817140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.088+0000 7f2460cc1640 1 -- 172.21.15.62:0/2537287179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 msgr2=0x55c42f819a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.088+0000 7f2460cc1640 1 --2- 172.21.15.62:0/2537287179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 0x55c42f819a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.088+0000 7f2460cc1640 1 -- 172.21.15.62:0/2537287179 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c42f6fe440 con 0x55c42f80c4a0 2024-09-15T23:25:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.089+0000 7f2460cc1640 1 --2- 172.21.15.62:0/2537287179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f80c880 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f245402ae40 tx=0x7f2454031590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771ad28da7814977 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.089+0000 7f2442ffd640 1 -- 172.21.15.62:0/2537287179 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245404e070 con 0x55c42f80c4a0 2024-09-15T23:25:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.089+0000 7f2442ffd640 1 -- 172.21.15.62:0/2537287179 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2454036040 con 0x55c42f80c4a0 2024-09-15T23:25:16.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.089+0000 7f2442ffd640 1 -- 172.21.15.62:0/2537287179 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24540305d0 con 0x55c42f80c4a0 2024-09-15T23:25:16.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.092+0000 7f24652fe740 1 -- 172.21.15.62:0/2537287179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 msgr2=0x55c42f80c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:16.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.092+0000 7f24652fe740 1 --2- 172.21.15.62:0/2537287179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f80c880 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f245402ae40 tx=0x7f2454031590 comp rx=0 tx=0).stop 2024-09-15T23:25:16.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.093+0000 7f2460cc1640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24440a3280 0x7f24440a5740 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f245404b000 tx=0x7f2454030bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:16.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.095+0000 7f24652fe740 1 -- 172.21.15.62:0/2537287179 shutdown_connections 2024-09-15T23:25:16.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.095+0000 7f24652fe740 1 --2- 172.21.15.62:0/2537287179 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 0x55c42f819a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.095+0000 7f24652fe740 1 --2- 172.21.15.62:0/2537287179 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f817140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.095+0000 7f24652fe740 1 --2- 172.21.15.62:0/2537287179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f80c880 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.095+0000 7f24652fe740 1 -- 172.21.15.62:0/2537287179 >> 172.21.15.62:0/2537287179 conn(0x55c42f808a30 msgr2=0x55c42f808e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:16.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.096+0000 7f24652fe740 1 -- 172.21.15.62:0/2537287179 shutdown_connections 2024-09-15T23:25:16.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.097+0000 7f24652fe740 1 -- 172.21.15.62:0/2537287179 wait complete. 2024-09-15T23:25:16.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.097+0000 7f24652fe740 1 -- start start 2024-09-15T23:25:16.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.097+0000 7f24652fe740 1 Processor -- start 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.098+0000 7f24652fe740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f816f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.099+0000 7f24652fe740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f8141f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.099+0000 7f24652fe740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 0x55c42f814730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.099+0000 7f24652fe740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c42f81a250 con 0x55c42f80c4a0 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.099+0000 7f24652fe740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c42f72d050 con 0x55c42f817680 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.099+0000 7f24652fe740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c42f81a3d0 con 0x55c42f80cdc0 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.099+0000 7f24652fe740 10 client.0 _ll_register_callbacks cb 0x55c42f76d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.099+0000 7f2460cc1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f816f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.099+0000 7f2460cc1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f816f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43566/0 (socket says 172.21.15.62:43566) 2024-09-15T23:25:16.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.099+0000 7f2460cc1640 1 -- 172.21.15.62:0/898517433 learned_addr learned my addr 172.21.15.62:0/898517433 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:16.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.101+0000 7f245bfff640 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f8141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.101+0000 7f24614c2640 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 0x55c42f814730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.103+0000 7f2460cc1640 1 -- 172.21.15.62:0/898517433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 msgr2=0x55c42f8141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:16.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.103+0000 7f2460cc1640 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f8141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.103+0000 7f2460cc1640 1 -- 172.21.15.62:0/898517433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 msgr2=0x55c42f814730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:16.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.103+0000 7f2460cc1640 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 0x55c42f814730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.103+0000 7f2460cc1640 1 -- 172.21.15.62:0/898517433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c42f4e33c0 con 0x55c42f80c4a0 2024-09-15T23:25:16.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.107+0000 7f2460cc1640 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f816f40 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2454000f80 tx=0x7f24540095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbaccc5daca4a65 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:16.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.107+0000 7f243bfff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:16.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.107+0000 7f243bfff640 1 -- 172.21.15.62:0/898517433 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245404e040 con 0x55c42f80c4a0 2024-09-15T23:25:16.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.108+0000 7f243bfff640 1 -- 172.21.15.62:0/898517433 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2454004b40 con 0x55c42f80c4a0 2024-09-15T23:25:16.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.108+0000 7f24652fe740 1 -- 172.21.15.62:0/898517433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2454036040 con 0x55c42f80c4a0 2024-09-15T23:25:16.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.108+0000 7f243bfff640 1 -- 172.21.15.62:0/898517433 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2454066050 con 0x55c42f80c4a0 2024-09-15T23:25:16.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.112+0000 7f24652fe740 10 client.17586 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:16.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.112+0000 7f24652fe740 1 -- 172.21.15.62:0/898517433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c42f825ba0 con 0x55c42f80c4a0 2024-09-15T23:25:16.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.113+0000 7f243bfff640 1 -- 172.21.15.62:0/898517433 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f245403fd20 con 0x55c42f80c4a0 2024-09-15T23:25:16.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.114+0000 7f243bfff640 1 -- 172.21.15.62:0/898517433 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f245405c030 con 0x55c42f80c4a0 2024-09-15T23:25:16.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.118+0000 7f24652fe740 10 client.17586 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:16.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.118+0000 7f24652fe740 10 client.17586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:16.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.118+0000 7f24652fe740 20 client.17586 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:16.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.118+0000 7f24652fe740 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c42f826ef0 0x55c42f8292e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.118+0000 7f24652fe740 4 client.17586 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.118+0000 7f24652fe740 1 -- 172.21.15.62:0/898517433 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2454036040 con 0x55c42f826ef0 2024-09-15T23:25:16.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.119+0000 7f245bfff640 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c42f826ef0 0x55c42f8292e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.122+0000 7f243bfff640 10 client.17586 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:16.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.122+0000 7f245bfff640 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c42f826ef0 0x55c42f8292e0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7c0c1154089a0fc server_cookie=7088fc0f20964503 in_seq=0 out_seq=0 2024-09-15T23:25:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.123+0000 7f243bfff640 1 -- 172.21.15.62:0/898517433 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f2454036040 con 0x55c42f826ef0 2024-09-15T23:25:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.123+0000 7f243bfff640 10 client.17586 handle_command_reply: tid=0 2024-09-15T23:25:16.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.124+0000 7f24652fe740 1 client.17586 shutdown 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.126+0000 7f24652fe740 1 -- 172.21.15.62:0/898517433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 msgr2=0x55c42f816f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.126+0000 7f24652fe740 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f816f40 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2454000f80 tx=0x7f24540095c0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.127+0000 7f24652fe740 1 -- 172.21.15.62:0/898517433 shutdown_connections 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.127+0000 7f24652fe740 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c42f826ef0 0x55c42f8292e0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.127+0000 7f24652fe740 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f817680 0x55c42f814730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.127+0000 7f24652fe740 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f8141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.127+0000 7f24652fe740 1 --2- 172.21.15.62:0/898517433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f816f40 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.127+0000 7f24652fe740 1 -- 172.21.15.62:0/898517433 >> 172.21.15.62:0/898517433 conn(0x55c42f808a30 msgr2=0x55c42f807b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.127+0000 7f243bfff640 0 client.17586 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.127+0000 7f24652fe740 1 -- 172.21.15.62:0/898517433 shutdown_connections 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.128+0000 7f24652fe740 1 -- 172.21.15.62:0/898517433 wait complete. 2024-09-15T23:25:16.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.128+0000 7f24652fe740 20 client.17586 trim_cache size 0 max 16384 2024-09-15T23:25:16.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.162+0000 7f24652fe740 1 Processor -- start 2024-09-15T23:25:16.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f24652fe740 1 -- start start 2024-09-15T23:25:16.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f24652fe740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f808a30 0x55c42f812ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f24652fe740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f82bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f24652fe740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f80cdc0 0x55c42f82c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f24652fe740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f245c1b7f40 con 0x55c42f80c4a0 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f24652fe740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f245c1b5810 con 0x55c42f80cdc0 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f24652fe740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f245c1187a0 con 0x55c42f808a30 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f2460cc1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f808a30 0x55c42f812ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f2460cc1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f808a30 0x55c42f812ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49406/0 (socket says 172.21.15.62:49406) 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.166+0000 7f2460cc1640 1 -- 172.21.15.62:0/1892673277 learned_addr learned my addr 172.21.15.62:0/1892673277 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.167+0000 7f2460cc1640 1 -- 172.21.15.62:0/1892673277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f80cdc0 msgr2=0x55c42f82c470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.167+0000 7f2460cc1640 1 --2- 172.21.15.62:0/1892673277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f80cdc0 0x55c42f82c470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.167+0000 7f2460cc1640 1 -- 172.21.15.62:0/1892673277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 msgr2=0x55c42f82bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:16.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.167+0000 7f2460cc1640 1 --2- 172.21.15.62:0/1892673277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f82bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.167+0000 7f2460cc1640 1 -- 172.21.15.62:0/1892673277 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c42f8131a0 con 0x55c42f808a30 2024-09-15T23:25:16.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.168+0000 7f2460cc1640 1 --2- 172.21.15.62:0/1892673277 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f808a30 0x55c42f812ac0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f245404c040 tx=0x7f2454005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=521d36aaaf6c679c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:16.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.171+0000 7f24427fc640 1 -- 172.21.15.62:0/1892673277 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2454003970 con 0x55c42f808a30 2024-09-15T23:25:16.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.171+0000 7f24427fc640 1 -- 172.21.15.62:0/1892673277 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2454036040 con 0x55c42f808a30 2024-09-15T23:25:16.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.171+0000 7f24427fc640 1 -- 172.21.15.62:0/1892673277 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f245403f810 con 0x55c42f808a30 2024-09-15T23:25:16.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.172+0000 7f24652fe740 1 -- 172.21.15.62:0/1892673277 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f808a30 msgr2=0x55c42f812ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:16.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.172+0000 7f24652fe740 1 --2- 172.21.15.62:0/1892673277 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f808a30 0x55c42f812ac0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f245404c040 tx=0x7f2454005d80 comp rx=0 tx=0).stop 2024-09-15T23:25:16.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.176+0000 7f24652fe740 1 -- 172.21.15.62:0/1892673277 shutdown_connections 2024-09-15T23:25:16.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.176+0000 7f24652fe740 1 --2- 172.21.15.62:0/1892673277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f80cdc0 0x55c42f82c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.176+0000 7f24652fe740 1 --2- 172.21.15.62:0/1892673277 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f82bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.176+0000 7f24652fe740 1 --2- 172.21.15.62:0/1892673277 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f808a30 0x55c42f812ac0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.176+0000 7f24652fe740 1 -- 172.21.15.62:0/1892673277 >> 172.21.15.62:0/1892673277 conn(0x55c42f814890 msgr2=0x55c42f807b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:16.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.177+0000 7f24652fe740 1 -- 172.21.15.62:0/1892673277 shutdown_connections 2024-09-15T23:25:16.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.178+0000 7f24652fe740 1 -- 172.21.15.62:0/1892673277 wait complete. 2024-09-15T23:25:16.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.179+0000 7f24652fe740 1 -- start start 2024-09-15T23:25:16.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.179+0000 7f24652fe740 1 Processor -- start 2024-09-15T23:25:16.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.180+0000 7f24652fe740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f808a30 0x55c42f8859a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.180+0000 7f24652fe740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f885ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.180+0000 7f24652fe740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f890840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.180+0000 7f24652fe740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f245c118e10 con 0x55c42f80c4a0 2024-09-15T23:25:16.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.180+0000 7f24652fe740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f245c1b5140 con 0x55c42f808a30 2024-09-15T23:25:16.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.180+0000 7f24652fe740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f245c1b43a0 con 0x55c42f80cdc0 2024-09-15T23:25:16.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.180+0000 7f24652fe740 10 client.0 _ll_register_callbacks cb 0x7f245404c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:16.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.181+0000 7f24614c2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f890840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.181+0000 7f24614c2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f890840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49408/0 (socket says 172.21.15.62:49408) 2024-09-15T23:25:16.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.181+0000 7f24614c2640 1 -- 172.21.15.62:0/2074159604 learned_addr learned my addr 172.21.15.62:0/2074159604 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:16.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.184+0000 7f2460cc1640 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f808a30 0x55c42f8859a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.184+0000 7f245bfff640 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f885ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.186+0000 7f24614c2640 1 -- 172.21.15.62:0/2074159604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f808a30 msgr2=0x55c42f8859a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.186+0000 7f24614c2640 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f808a30 0x55c42f8859a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.186+0000 7f24614c2640 1 -- 172.21.15.62:0/2074159604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 msgr2=0x55c42f885ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.186+0000 7f24614c2640 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f885ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.186+0000 7f24614c2640 1 -- 172.21.15.62:0/2074159604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c42f890d80 con 0x55c42f80cdc0 2024-09-15T23:25:16.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.186+0000 7f24614c2640 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f890840 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2450004980 tx=0x7f24500bdda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63dfbfa65dc106ac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:16.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.187+0000 7f243bfff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:25:16.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.188+0000 7f243bfff640 1 -- 172.21.15.62:0/2074159604 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24500da020 con 0x55c42f80cdc0 2024-09-15T23:25:16.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.188+0000 7f24652fe740 1 -- 172.21.15.62:0/2074159604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2454036040 con 0x55c42f80cdc0 2024-09-15T23:25:16.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.188+0000 7f243bfff640 1 -- 172.21.15.62:0/2074159604 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f24500bd1f0 con 0x55c42f80cdc0 2024-09-15T23:25:16.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.189+0000 7f243bfff640 1 -- 172.21.15.62:0/2074159604 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f24500cf050 con 0x55c42f80cdc0 2024-09-15T23:25:16.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.189+0000 7f24652fe740 10 client.26911 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:16.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.189+0000 7f24652fe740 1 -- 172.21.15.62:0/2074159604 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c42f8920c0 con 0x55c42f80cdc0 2024-09-15T23:25:16.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.190+0000 7f243bfff640 1 -- 172.21.15.62:0/2074159604 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2454036040 con 0x55c42f80cdc0 2024-09-15T23:25:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.191+0000 7f243bfff640 1 -- 172.21.15.62:0/2074159604 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f24500b9af0 con 0x55c42f80cdc0 2024-09-15T23:25:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.191+0000 7f24652fe740 10 client.26911 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.191+0000 7f24652fe740 10 client.26911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.191+0000 7f24652fe740 20 client.26911 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.191+0000 7f24652fe740 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c42f817680 0x55c42f894820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.191+0000 7f24652fe740 4 client.26911 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:25:16.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.191+0000 7f24652fe740 1 -- 172.21.15.62:0/2074159604 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c42f77f850 con 0x55c42f817680 2024-09-15T23:25:16.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.194+0000 7f2460cc1640 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c42f817680 0x55c42f894820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:16.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.195+0000 7f2460cc1640 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c42f817680 0x55c42f894820 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feda7bd9788bf0ec server_cookie=430425affe7d4037 in_seq=0 out_seq=0 2024-09-15T23:25:16.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.195+0000 7f243bfff640 10 client.26911 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:16.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.195+0000 7f243bfff640 1 -- 172.21.15.62:0/2074159604 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c42f77f850 con 0x55c42f817680 2024-09-15T23:25:16.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.195+0000 7f243bfff640 10 client.26911 handle_command_reply: tid=0 2024-09-15T23:25:16.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.195+0000 7f24652fe740 1 client.26911 shutdown 2024-09-15T23:25:16.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f24652fe740 1 -- 172.21.15.62:0/2074159604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 msgr2=0x55c42f890840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:16.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f24652fe740 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f890840 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2450004980 tx=0x7f24500bdda0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f24652fe740 1 -- 172.21.15.62:0/2074159604 shutdown_connections 2024-09-15T23:25:16.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f24652fe740 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c42f817680 0x55c42f894820 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f24652fe740 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c42f80cdc0 0x55c42f890840 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f24652fe740 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c42f80c4a0 0x55c42f885ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f24652fe740 1 --2- 172.21.15.62:0/2074159604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c42f808a30 0x55c42f8859a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f24652fe740 1 -- 172.21.15.62:0/2074159604 >> 172.21.15.62:0/2074159604 conn(0x55c42f814890 msgr2=0x55c42f807b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f243bfff640 0 client.26911 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.196+0000 7f24652fe740 1 -- 172.21.15.62:0/2074159604 shutdown_connections 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.197+0000 7f24652fe740 1 -- 172.21.15.62:0/2074159604 wait complete. 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.197+0000 7f24652fe740 20 client.26911 trim_cache size 0 max 16384 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:25:16.199 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:25:16.200 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:25:16.200 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 -- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24440a3280 msgr2=0x7f24440a5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24440a3280 0x7f24440a5740 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f245404b000 tx=0x7f2454030bc0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 -- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c1abd30 msgr2=0x7f245c1ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c1abd30 0x7f245c1ac1e0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f245000ae50 tx=0x7f245000cd40 comp rx=0 tx=0).stop 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 -- 172.21.15.62:0/3067540846 shutdown_connections 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24440a3280 0x7f24440a5740 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f245c1abd30 0x7f245c1ac1e0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f245c11b140 0x7f245c1a79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 --2- 172.21.15.62:0/3067540846 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f245c109830 0x7f245c1a7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.198+0000 7f2462f25640 1 -- 172.21.15.62:0/3067540846 >> 172.21.15.62:0/3067540846 conn(0x7f245c06e140 msgr2=0x7f245c11c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.199+0000 7f2462f25640 1 -- 172.21.15.62:0/3067540846 shutdown_connections 2024-09-15T23:25:16.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:16.199+0000 7f2462f25640 1 -- 172.21.15.62:0/3067540846 wait complete. 2024-09-15T23:25:16.908 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:25:16.908 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:25:17.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:17 smithi062 ceph-mon[30949]: pgmap v640: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 202 KiB/s wr, 34 op/s 2024-09-15T23:25:17.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:17 smithi148 ceph-mon[35610]: pgmap v640: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 202 KiB/s wr, 34 op/s 2024-09-15T23:25:17.413 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:17.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:17 smithi067 ceph-mon[35665]: pgmap v640: 129 pgs: 129 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 202 KiB/s wr, 34 op/s 2024-09-15T23:25:17.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.970+0000 7f2b9dff6640 1 -- 172.21.15.62:0/40828616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b98109830 msgr2=0x7f2b98109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.970+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/40828616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b98109830 0x7f2b98109c90 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009ca0 tx=0x7f2b9002a8f0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.971+0000 7f2b9dff6640 1 -- 172.21.15.62:0/40828616 shutdown_connections 2024-09-15T23:25:17.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.971+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/40828616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b9811b140 0x7f2b9811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.971+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/40828616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b98109830 0x7f2b98109c90 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.971+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/40828616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b98109450 0x7f2b98110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.971+0000 7f2b9dff6640 1 -- 172.21.15.62:0/40828616 >> 172.21.15.62:0/40828616 conn(0x7f2b9806e140 msgr2=0x7f2b9806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.973+0000 7f2b9dff6640 1 -- 172.21.15.62:0/40828616 shutdown_connections 2024-09-15T23:25:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.975+0000 7f2b9dff6640 1 -- 172.21.15.62:0/40828616 wait complete. 2024-09-15T23:25:17.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.976+0000 7f2b9dff6640 1 Processor -- start 2024-09-15T23:25:17.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.976+0000 7f2b9dff6640 1 -- start start 2024-09-15T23:25:17.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.976+0000 7f2b9dff6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b98109450 0x7f2b98118630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.976+0000 7f2b9dff6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b98109830 0x7f2b98118b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.976+0000 7f2b9dff6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b9811b140 0x7f2b981118a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.976+0000 7f2b9dff6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2b98120ed0 con 0x7f2b9811b140 2024-09-15T23:25:17.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.976+0000 7f2b9dff6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b98120d50 con 0x7f2b98109830 2024-09-15T23:25:17.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.976+0000 7f2b9dff6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2b98121050 con 0x7f2b98109450 2024-09-15T23:25:17.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.978+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b98109450 0x7f2b98118630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.978+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b98109450 0x7f2b98118630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49412/0 (socket says 172.21.15.62:49412) 2024-09-15T23:25:17.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.978+0000 7f2b977fe640 1 -- 172.21.15.62:0/4054766574 learned_addr learned my addr 172.21.15.62:0/4054766574 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:17.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.982+0000 7f2b97fff640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b9811b140 0x7f2b981118a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.982+0000 7f2b96ffd640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b98109830 0x7f2b98118b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:17.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.982+0000 7f2b96ffd640 1 -- 172.21.15.62:0/4054766574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b98109450 msgr2=0x7f2b98118630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.982+0000 7f2b96ffd640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b98109450 0x7f2b98118630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.982+0000 7f2b96ffd640 1 -- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b9811b140 msgr2=0x7f2b981118a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:17.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.982+0000 7f2b96ffd640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b9811b140 0x7f2b981118a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:17.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.982+0000 7f2b96ffd640 1 -- 172.21.15.62:0/4054766574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b98112160 con 0x7f2b98109830 2024-09-15T23:25:17.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.982+0000 7f2b96ffd640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b98109830 0x7f2b98118b70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2b90000f80 tx=0x7f2b9002fba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:17.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.983+0000 7f2b94ff9640 1 -- 172.21.15.62:0/4054766574 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b900307f0 con 0x7f2b98109830 2024-09-15T23:25:17.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.983+0000 7f2b94ff9640 1 -- 172.21.15.62:0/4054766574 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b9002d070 con 0x7f2b98109830 2024-09-15T23:25:17.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.983+0000 7f2b94ff9640 1 -- 172.21.15.62:0/4054766574 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b90046cb0 con 0x7f2b98109830 2024-09-15T23:25:17.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.983+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b981123f0 con 0x7f2b98109830 2024-09-15T23:25:17.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.983+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b98116380 con 0x7f2b98109830 2024-09-15T23:25:17.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.989+0000 7f2b94ff9640 1 -- 172.21.15.62:0/4054766574 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2b90003970 con 0x7f2b98109830 2024-09-15T23:25:17.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.992+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b84005290 con 0x7f2b98109830 2024-09-15T23:25:17.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.995+0000 7f2b94ff9640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b800a3450 0x7f2b800a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:17.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:17.996+0000 7f2b94ff9640 1 -- 172.21.15.62:0/4054766574 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2b900f1ca0 con 0x7f2b98109830 2024-09-15T23:25:18.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.000+0000 7f2b977fe640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b800a3450 0x7f2b800a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:18.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.000+0000 7f2b94ff9640 1 -- 172.21.15.62:0/4054766574 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2b900be5e0 con 0x7f2b98109830 2024-09-15T23:25:18.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.004+0000 7f2b977fe640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b800a3450 0x7f2b800a5910 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009a30 tx=0x7f2b88009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:18.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.375+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b84005580 con 0x7f2b98109830 2024-09-15T23:25:18.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.378+0000 7f2b94ff9640 1 -- 172.21.15.62:0/4054766574 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7f2b900bdee0 con 0x7f2b98109830 2024-09-15T23:25:18.380 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:25:18.380 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:25:18.380 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:25:18.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.386+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b800a3450 msgr2=0x7f2b800a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:18.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.386+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b800a3450 0x7f2b800a5910 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009a30 tx=0x7f2b88009190 comp rx=0 tx=0).stop 2024-09-15T23:25:18.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.386+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b98109830 msgr2=0x7f2b98118b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:18.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.386+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b98109830 0x7f2b98118b70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2b90000f80 tx=0x7f2b9002fba0 comp rx=0 tx=0).stop 2024-09-15T23:25:18.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.387+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 shutdown_connections 2024-09-15T23:25:18.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.387+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b800a3450 0x7f2b800a5910 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:18.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.387+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b9811b140 0x7f2b981118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:18.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.387+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b98109830 0x7f2b98118b70 secure :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2b90000f80 tx=0x7f2b9002fba0 comp rx=0 tx=0).stop 2024-09-15T23:25:18.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.387+0000 7f2b9dff6640 1 --2- 172.21.15.62:0/4054766574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b98109450 0x7f2b98118630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:18.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.387+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 >> 172.21.15.62:0/4054766574 conn(0x7f2b9806e140 msgr2=0x7f2b9811bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:18.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.388+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 shutdown_connections 2024-09-15T23:25:18.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:18.388+0000 7f2b9dff6640 1 -- 172.21.15.62:0/4054766574 wait complete. 2024-09-15T23:25:19.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:19 smithi148 ceph-mon[35610]: pgmap v641: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 530 KiB/s wr, 71 op/s 2024-09-15T23:25:19.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:19 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4054766574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:19.326 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:19 smithi067 ceph-mon[35665]: pgmap v641: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 530 KiB/s wr, 71 op/s 2024-09-15T23:25:19.326 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:19 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4054766574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:19.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:19 smithi062 ceph-mon[30949]: pgmap v641: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 530 KiB/s wr, 71 op/s 2024-09-15T23:25:19.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:19 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4054766574' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:25:19.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 540.0 s, 524.8 tps, lat 188.944 ms stddev 227.896 2024-09-15T23:25:19.861 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:25:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.916+0000 7f830607b640 1 -- 172.21.15.62:0/953948106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 msgr2=0x7f830006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.916+0000 7f830607b640 1 --2- 172.21.15.62:0/953948106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 0x7f830006e4f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f82fc01a910 tx=0x7f82fc03b470 comp rx=0 tx=0).stop 2024-09-15T23:25:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.916+0000 7f830607b640 1 -- 172.21.15.62:0/953948106 shutdown_connections 2024-09-15T23:25:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.916+0000 7f830607b640 1 --2- 172.21.15.62:0/953948106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f830011b140 0x7f830011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.916+0000 7f830607b640 1 --2- 172.21.15.62:0/953948106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f830006ea30 0x7f830006ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.916+0000 7f830607b640 1 --2- 172.21.15.62:0/953948106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 0x7f830006e4f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.916+0000 7f830607b640 1 -- 172.21.15.62:0/953948106 >> 172.21.15.62:0/953948106 conn(0x7f8300073180 msgr2=0x7f8300073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.917+0000 7f830607b640 1 -- 172.21.15.62:0/953948106 shutdown_connections 2024-09-15T23:25:20.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.917+0000 7f830607b640 1 -- 172.21.15.62:0/953948106 wait complete. 2024-09-15T23:25:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f830607b640 1 Processor -- start 2024-09-15T23:25:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f830607b640 1 -- start start 2024-09-15T23:25:20.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f830607b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f830006ea30 0x7f830011a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f830607b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 0x7f83001168d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f830607b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f830011b140 0x7f8300116e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f830607b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8300113300 con 0x7f8300110190 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f830607b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8300113180 con 0x7f830006ea30 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f830607b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8300113480 con 0x7f830011b140 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f8304878640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 0x7f83001168d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f8305079640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f830006ea30 0x7f830011a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f8304878640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 0x7f83001168d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43622/0 (socket says 172.21.15.62:43622) 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f8305079640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f830006ea30 0x7f830011a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39114/0 (socket says 172.21.15.62:39114) 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f8304878640 1 -- 172.21.15.62:0/410220518 learned_addr learned my addr 172.21.15.62:0/410220518 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.918+0000 7f8305079640 1 -- 172.21.15.62:0/410220518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f830011b140 msgr2=0x7f8300116e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f8305079640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f830011b140 0x7f8300116e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f8305079640 1 -- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 msgr2=0x7f83001168d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f8305079640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 0x7f83001168d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f8305079640 1 -- 172.21.15.62:0/410220518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8300117540 con 0x7f830006ea30 2024-09-15T23:25:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f8304878640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 0x7f83001168d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f8305079640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f830006ea30 0x7f830011a830 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f82fc03ba60 tx=0x7f82fc0050e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f82f67fc640 1 -- 172.21.15.62:0/410220518 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc051070 con 0x7f830006ea30 2024-09-15T23:25:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f830607b640 1 -- 172.21.15.62:0/410220518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83001aaf60 con 0x7f830006ea30 2024-09-15T23:25:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f830607b640 1 -- 172.21.15.62:0/410220518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83001ab300 con 0x7f830006ea30 2024-09-15T23:25:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f82f67fc640 1 -- 172.21.15.62:0/410220518 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82fc0409f0 con 0x7f830006ea30 2024-09-15T23:25:20.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.919+0000 7f82f67fc640 1 -- 172.21.15.62:0/410220518 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc042870 con 0x7f830006ea30 2024-09-15T23:25:20.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.921+0000 7f82f67fc640 1 -- 172.21.15.62:0/410220518 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f82fc054070 con 0x7f830006ea30 2024-09-15T23:25:20.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.922+0000 7f83096b6740 1 Processor -- start 2024-09-15T23:25:20.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.922+0000 7f83096b6740 1 -- start start 2024-09-15T23:25:20.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f83096b6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a87d00 0x565081a86250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f83096b6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a86bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f83096b6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 0x565081a93920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f83096b6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5650819ec2f0 con 0x565081a87d00 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f83096b6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x565081993cf0 con 0x565081a86790 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f83096b6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5650819e82b0 con 0x565081a91530 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f830587a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a87d00 0x565081a86250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f830587a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a87d00 0x565081a86250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43628/0 (socket says 172.21.15.62:43628) 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f830587a640 1 -- 172.21.15.62:0/3782723336 learned_addr learned my addr 172.21.15.62:0/3782723336 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f8305079640 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a86bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f8304878640 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 0x565081a93920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f830587a640 1 -- 172.21.15.62:0/3782723336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 msgr2=0x565081a93920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f830587a640 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 0x565081a93920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f830587a640 1 -- 172.21.15.62:0/3782723336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 msgr2=0x565081a86bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f830587a640 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a86bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f830587a640 1 -- 172.21.15.62:0/3782723336 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650819783a0 con 0x565081a87d00 2024-09-15T23:25:20.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.923+0000 7f8304878640 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 0x565081a93920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:20.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.924+0000 7f830587a640 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a87d00 0x565081a86250 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009c40 tx=0x7f82f800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988be74a2fce630e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.924+0000 7f82eb7fe640 1 -- 172.21.15.62:0/3782723336 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8016070 con 0x565081a87d00 2024-09-15T23:25:20.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.924+0000 7f82eb7fe640 1 -- 172.21.15.62:0/3782723336 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f8005020 con 0x565081a87d00 2024-09-15T23:25:20.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.924+0000 7f82eb7fe640 1 -- 172.21.15.62:0/3782723336 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f80116d0 con 0x565081a87d00 2024-09-15T23:25:20.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.925+0000 7f83096b6740 1 -- 172.21.15.62:0/3782723336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a87d00 msgr2=0x565081a86250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.925+0000 7f83096b6740 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a87d00 0x565081a86250 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009c40 tx=0x7f82f800e6c0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.925+0000 7f83096b6740 1 -- 172.21.15.62:0/3782723336 shutdown_connections 2024-09-15T23:25:20.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.925+0000 7f83096b6740 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 0x565081a93920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.925+0000 7f83096b6740 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a86bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.925+0000 7f83096b6740 1 --2- 172.21.15.62:0/3782723336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a87d00 0x565081a86250 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.925+0000 7f82f67fc640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82ec0a32c0 0x7f82ec0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.926+0000 7f83096b6740 1 -- 172.21.15.62:0/3782723336 >> 172.21.15.62:0/3782723336 conn(0x565081a4f310 msgr2=0x565081a82e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.926+0000 7f830587a640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82ec0a32c0 0x7f82ec0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.926+0000 7f82f67fc640 1 -- 172.21.15.62:0/410220518 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f82fc101370 con 0x7f830006ea30 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.926+0000 7f83096b6740 1 -- 172.21.15.62:0/3782723336 shutdown_connections 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.926+0000 7f83096b6740 1 -- 172.21.15.62:0/3782723336 wait complete. 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.926+0000 7f830587a640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82ec0a32c0 0x7f82ec0a5780 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f82f80099d0 tx=0x7f82f8000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.926+0000 7f83096b6740 1 -- start start 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f83096b6740 1 Processor -- start 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f83096b6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a90ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f83096b6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a8e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f83096b6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 0x565081a8e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f83096b6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x565081a94230 con 0x565081a91530 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f83096b6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5650819a7060 con 0x565081a86790 2024-09-15T23:25:20.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f83096b6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x565081a943b0 con 0x565081a87d00 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f83096b6740 10 client.0 _ll_register_callbacks cb 0x5650816fa8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f8304878640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a8e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f8304878640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a8e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49444/0 (socket says 172.21.15.62:49444) 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.927+0000 7f8304878640 1 -- 172.21.15.62:0/108511588 learned_addr learned my addr 172.21.15.62:0/108511588 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f8305079640 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a90ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f830587a640 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 0x565081a8e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f8304878640 1 -- 172.21.15.62:0/108511588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 msgr2=0x565081a90ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f8304878640 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a90ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f8304878640 1 -- 172.21.15.62:0/108511588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 msgr2=0x565081a8e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f8304878640 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 0x565081a8e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f8304878640 1 -- 172.21.15.62:0/108511588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56508175d3c0 con 0x565081a87d00 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f8305079640 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a90ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f8304878640 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a8e180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f82f00117a0 tx=0x7f82f0011c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0879ecaad78ec6c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f82e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f82e8ff9640 1 -- 172.21.15.62:0/108511588 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0018070 con 0x565081a87d00 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.928+0000 7f83096b6740 1 -- 172.21.15.62:0/108511588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f8005020 con 0x565081a87d00 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f82e8ff9640 1 -- 172.21.15.62:0/108511588 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f82f000fd50 con 0x565081a87d00 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f82e8ff9640 1 -- 172.21.15.62:0/108511588 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f82f0027050 con 0x565081a87d00 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f83096b6740 10 client.26920 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f83096b6740 1 -- 172.21.15.62:0/108511588 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x565081a9fcd0 con 0x565081a87d00 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f82e8ff9640 1 -- 172.21.15.62:0/108511588 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f8005020 con 0x565081a87d00 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f82e8ff9640 1 -- 172.21.15.62:0/108511588 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f82f001c400 con 0x565081a87d00 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f83096b6740 10 client.26920 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f83096b6740 10 client.26920 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f83096b6740 20 client.26920 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f83096b6740 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x565081aa1180 0x565081aa35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f83096b6740 4 client.26920 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:20.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f83096b6740 1 -- 172.21.15.62:0/108511588 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5650819f9860 con 0x565081aa1180 2024-09-15T23:25:20.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.929+0000 7f8305079640 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x565081aa1180 0x565081aa35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.930+0000 7f8305079640 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x565081aa1180 0x565081aa35e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a246f78e3933ecb1 server_cookie=4699e17526db59d6 in_seq=0 out_seq=0 2024-09-15T23:25:20.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.930+0000 7f82e8ff9640 10 client.26920 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:25:20.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.930+0000 7f82e8ff9640 1 -- 172.21.15.62:0/108511588 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5650819f9860 con 0x565081aa1180 2024-09-15T23:25:20.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.930+0000 7f82e8ff9640 10 client.26920 handle_command_reply: tid=0 2024-09-15T23:25:20.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.930+0000 7f83096b6740 1 client.26920 shutdown 2024-09-15T23:25:20.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 -- 172.21.15.62:0/108511588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 msgr2=0x565081a8e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a8e180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f82f00117a0 tx=0x7f82f0011c60 comp rx=0 tx=0).stop 2024-09-15T23:25:20.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 -- 172.21.15.62:0/108511588 shutdown_connections 2024-09-15T23:25:20.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x565081aa1180 0x565081aa35e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f82e8ff9640 0 client.26920 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:25:20.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 0x565081a8e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a8e180 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 --2- 172.21.15.62:0/108511588 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a90ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 -- 172.21.15.62:0/108511588 >> 172.21.15.62:0/108511588 conn(0x565081a4f310 msgr2=0x565081a88af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 -- 172.21.15.62:0/108511588 shutdown_connections 2024-09-15T23:25:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 1 -- 172.21.15.62:0/108511588 wait complete. 2024-09-15T23:25:20.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.931+0000 7f83096b6740 20 client.26920 trim_cache size 0 max 16384 2024-09-15T23:25:20.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.953+0000 7f83096b6740 1 Processor -- start 2024-09-15T23:25:20.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.953+0000 7f83096b6740 1 -- start start 2024-09-15T23:25:20.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.953+0000 7f83096b6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a9d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f83096b6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a892e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f83096b6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 0x565081a89820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f83096b6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f83001b79c0 con 0x565081a91530 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f83096b6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83001b5290 con 0x565081a86790 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f8305079640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a9d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f8305079640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a9d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39144/0 (socket says 172.21.15.62:39144) 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f8305079640 1 -- 172.21.15.62:0/1887644073 learned_addr learned my addr 172.21.15.62:0/1887644073 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f8304878640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a892e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f830587a640 1 --2- 172.21.15.62:0/1887644073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 0x565081a89820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.954+0000 7f83096b6740 1 -- 172.21.15.62:0/1887644073 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f83001b30e0 con 0x565081a87d00 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f8305079640 1 -- 172.21.15.62:0/1887644073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 msgr2=0x565081a892e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f8305079640 1 --2- 172.21.15.62:0/1887644073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a892e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f8305079640 1 -- 172.21.15.62:0/1887644073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 msgr2=0x565081a89820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f8305079640 1 --2- 172.21.15.62:0/1887644073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 0x565081a89820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f8305079640 1 -- 172.21.15.62:0/1887644073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565081a91240 con 0x565081a86790 2024-09-15T23:25:20.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f830587a640 1 --2- 172.21.15.62:0/1887644073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 0x565081a89820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f8305079640 1 --2- 172.21.15.62:0/1887644073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a9d500 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0d5cf0 tx=0x7f82fc01a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f37f4e52880a9b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f82eaffd640 1 -- 172.21.15.62:0/1887644073 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc05b040 con 0x565081a86790 2024-09-15T23:25:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f82eaffd640 1 -- 172.21.15.62:0/1887644073 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82fc101150 con 0x565081a86790 2024-09-15T23:25:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.955+0000 7f82eaffd640 1 -- 172.21.15.62:0/1887644073 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc05b040 con 0x565081a86790 2024-09-15T23:25:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.956+0000 7f83096b6740 1 -- 172.21.15.62:0/1887644073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 msgr2=0x565081a9d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.956+0000 7f83096b6740 1 --2- 172.21.15.62:0/1887644073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a9d500 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0d5cf0 tx=0x7f82fc01a270 comp rx=0 tx=0).stop 2024-09-15T23:25:20.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.959+0000 7f83096b6740 1 -- 172.21.15.62:0/1887644073 shutdown_connections 2024-09-15T23:25:20.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.959+0000 7f83096b6740 1 --2- 172.21.15.62:0/1887644073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a91530 0x565081a89820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.959+0000 7f83096b6740 1 --2- 172.21.15.62:0/1887644073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a87d00 0x565081a892e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.959+0000 7f83096b6740 1 --2- 172.21.15.62:0/1887644073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a86790 0x565081a9d500 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.959+0000 7f83096b6740 1 -- 172.21.15.62:0/1887644073 >> 172.21.15.62:0/1887644073 conn(0x565081a4f310 msgr2=0x565081a7db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.959+0000 7f83096b6740 1 -- 172.21.15.62:0/1887644073 shutdown_connections 2024-09-15T23:25:20.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.959+0000 7f83096b6740 1 -- 172.21.15.62:0/1887644073 wait complete. 2024-09-15T23:25:20.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.960+0000 7f83096b6740 1 -- start start 2024-09-15T23:25:20.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.960+0000 7f83096b6740 1 Processor -- start 2024-09-15T23:25:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f83096b6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a86790 0x565081a99e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f83096b6740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a87d00 0x565081a8afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f83096b6740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 0x565081a8b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f83096b6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f83001b3750 con 0x565081a86790 2024-09-15T23:25:20.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f83096b6740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f83001b4bc0 con 0x565081a87d00 2024-09-15T23:25:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f83096b6740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f83001b3e20 con 0x565081a91530 2024-09-15T23:25:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f83096b6740 10 client.0 _ll_register_callbacks cb 0x7f82f000bc70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8305079640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a86790 0x565081a99e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8305079640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a86790 0x565081a99e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43658/0 (socket says 172.21.15.62:43658) 2024-09-15T23:25:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8305079640 1 -- 172.21.15.62:0/3654789234 learned_addr learned my addr 172.21.15.62:0/3654789234 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:20.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f830587a640 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 0x565081a8b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8304878640 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a87d00 0x565081a8afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8305079640 1 -- 172.21.15.62:0/3654789234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 msgr2=0x565081a8b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8305079640 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 0x565081a8b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8305079640 1 -- 172.21.15.62:0/3654789234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a87d00 msgr2=0x565081a8afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8305079640 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a87d00 0x565081a8afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8305079640 1 -- 172.21.15.62:0/3654789234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565081a8bd50 con 0x565081a86790 2024-09-15T23:25:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f8305079640 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a86790 0x565081a99e70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f82fc01a590 tx=0x7f82fc0fea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a4af3ddb4b3214 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.961+0000 7f82e8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f82e8ff9640 1 -- 172.21.15.62:0/3654789234 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82fc05b040 con 0x565081a86790 2024-09-15T23:25:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f83096b6740 1 -- 172.21.15.62:0/3654789234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5650819f3d70 con 0x565081a86790 2024-09-15T23:25:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f82e8ff9640 1 -- 172.21.15.62:0/3654789234 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f82fc0cb650 con 0x565081a86790 2024-09-15T23:25:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f82e8ff9640 1 -- 172.21.15.62:0/3654789234 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f82fc05b040 con 0x565081a86790 2024-09-15T23:25:20.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f83096b6740 10 client.17613 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f83096b6740 1 -- 172.21.15.62:0/3654789234 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x565081b0bf70 con 0x565081a86790 2024-09-15T23:25:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f82e8ff9640 1 -- 172.21.15.62:0/3654789234 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5650819f3d70 con 0x565081a86790 2024-09-15T23:25:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f82e8ff9640 1 -- 172.21.15.62:0/3654789234 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f82fc0d5d90 con 0x565081a86790 2024-09-15T23:25:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f83096b6740 10 client.17613 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f83096b6740 10 client.17613 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:25:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f83096b6740 20 client.17613 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.962+0000 7f83096b6740 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x565081aa1180 0x565081b0e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:20.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.963+0000 7f83096b6740 4 client.17613 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:20.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.963+0000 7f83096b6740 1 -- 172.21.15.62:0/3654789234 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565081970170 con 0x565081aa1180 2024-09-15T23:25:20.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.963+0000 7f8304878640 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x565081aa1180 0x565081b0e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:20.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.963+0000 7f8304878640 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x565081aa1180 0x565081b0e690 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41f39e7aba8794c server_cookie=3d55ca4da15326c3 in_seq=0 out_seq=0 2024-09-15T23:25:20.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.963+0000 7f82e8ff9640 10 client.17613 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:25:20.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.963+0000 7f82e8ff9640 1 -- 172.21.15.62:0/3654789234 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565081970170 con 0x565081aa1180 2024-09-15T23:25:20.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.963+0000 7f82e8ff9640 10 client.17613 handle_command_reply: tid=0 2024-09-15T23:25:20.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.963+0000 7f83096b6740 1 client.17613 shutdown 2024-09-15T23:25:20.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 -- 172.21.15.62:0/3654789234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a86790 msgr2=0x565081a99e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a86790 0x565081a99e70 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f82fc01a590 tx=0x7f82fc0fea50 comp rx=0 tx=0).stop 2024-09-15T23:25:20.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 -- 172.21.15.62:0/3654789234 shutdown_connections 2024-09-15T23:25:20.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x565081aa1180 0x565081b0e690 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f82e8ff9640 0 client.17613 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:25:20.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x565081a91530 0x565081a8b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x565081a87d00 0x565081a8afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 --2- 172.21.15.62:0/3654789234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565081a86790 0x565081a99e70 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 -- 172.21.15.62:0/3654789234 >> 172.21.15.62:0/3654789234 conn(0x565081a4f310 msgr2=0x565081a81780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 -- 172.21.15.62:0/3654789234 shutdown_connections 2024-09-15T23:25:20.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 1 -- 172.21.15.62:0/3654789234 wait complete. 2024-09-15T23:25:20.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f83096b6740 20 client.17613 trim_cache size 0 max 16384 2024-09-15T23:25:20.978 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:25:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.964+0000 7f830607b640 1 -- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82ec0a32c0 msgr2=0x7f82ec0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82ec0a32c0 0x7f82ec0a5780 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f82f80099d0 tx=0x7f82f8000f40 comp rx=0 tx=0).stop 2024-09-15T23:25:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 -- 172.21.15.62:0/410220518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f830006ea30 msgr2=0x7f830011a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f830006ea30 0x7f830011a830 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f82fc03ba60 tx=0x7f82fc0050e0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 -- 172.21.15.62:0/410220518 shutdown_connections 2024-09-15T23:25:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f82ec0a32c0 0x7f82ec0a5780 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f830011b140 0x7f8300116e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8300110190 0x7f83001168d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 --2- 172.21.15.62:0/410220518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f830006ea30 0x7f830011a830 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:20.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 -- 172.21.15.62:0/410220518 >> 172.21.15.62:0/410220518 conn(0x7f8300073180 msgr2=0x7f830011b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:20.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 -- 172.21.15.62:0/410220518 shutdown_connections 2024-09-15T23:25:20.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:20.965+0000 7f830607b640 1 -- 172.21.15.62:0/410220518 wait complete. 2024-09-15T23:25:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.346+0000 7f7a71c65640 1 -- 172.21.15.62:0/3070906905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1056c0 msgr2=0x7f7a6c105b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.346+0000 7f7a71c65640 1 --2- 172.21.15.62:0/3070906905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1056c0 0x7f7a6c105b70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009860 tx=0x7f7a6402a360 comp rx=0 tx=0).stop 2024-09-15T23:25:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.346+0000 7f7a71c65640 1 -- 172.21.15.62:0/3070906905 shutdown_connections 2024-09-15T23:25:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.346+0000 7f7a71c65640 1 --2- 172.21.15.62:0/3070906905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1056c0 0x7f7a6c105b70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.346+0000 7f7a71c65640 1 --2- 172.21.15.62:0/3070906905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c102d10 0x7f7a6c105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.346+0000 7f7a71c65640 1 --2- 172.21.15.62:0/3070906905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a6c1003c0 0x7f7a6c1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.346+0000 7f7a71c65640 1 -- 172.21.15.62:0/3070906905 >> 172.21.15.62:0/3070906905 conn(0x7f7a6c0fc1b0 msgr2=0x7f7a6c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.346+0000 7f7a71c65640 1 -- 172.21.15.62:0/3070906905 shutdown_connections 2024-09-15T23:25:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.347+0000 7f7a71c65640 1 -- 172.21.15.62:0/3070906905 wait complete. 2024-09-15T23:25:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.348+0000 7f7a71c65640 1 Processor -- start 2024-09-15T23:25:21.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.348+0000 7f7a71c65640 1 -- start start 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.348+0000 7f7a71c65640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1003c0 0x7f7a6c19e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a71c65640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c102d10 0x7f7a6c19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a71c65640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a6c1056c0 0x7f7a6c1a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1003c0 0x7f7a6c19e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1003c0 0x7f7a6c19e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49478/0 (socket says 172.21.15.62:49478) 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6b7fe640 1 -- 172.21.15.62:0/2874696175 learned_addr learned my addr 172.21.15.62:0/2874696175 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a6c1056c0 0x7f7a6c1a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6affd640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c102d10 0x7f7a6c19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6c113e00 con 0x7f7a6c102d10 2024-09-15T23:25:21.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6c113c80 con 0x7f7a6c1056c0 2024-09-15T23:25:21.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6c113f80 con 0x7f7a6c1003c0 2024-09-15T23:25:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6b7fe640 1 -- 172.21.15.62:0/2874696175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a6c1056c0 msgr2=0x7f7a6c1a5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a6c1056c0 0x7f7a6c1a5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6b7fe640 1 -- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c102d10 msgr2=0x7f7a6c19e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c102d10 0x7f7a6c19e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.349+0000 7f7a6b7fe640 1 -- 172.21.15.62:0/2874696175 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a6c1a5fe0 con 0x7f7a6c1003c0 2024-09-15T23:25:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.350+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1003c0 0x7f7a6c19e3c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400d950 tx=0x7f7a5400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.350+0000 7f7a68ff9640 1 -- 172.21.15.62:0/2874696175 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54017070 con 0x7f7a6c1003c0 2024-09-15T23:25:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.350+0000 7f7a68ff9640 1 -- 172.21.15.62:0/2874696175 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a54009c30 con 0x7f7a6c1003c0 2024-09-15T23:25:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.350+0000 7f7a68ff9640 1 -- 172.21.15.62:0/2874696175 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5400f6b0 con 0x7f7a6c1003c0 2024-09-15T23:25:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.350+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a6c1a62d0 con 0x7f7a6c1003c0 2024-09-15T23:25:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.350+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a6c1a6710 con 0x7f7a6c1003c0 2024-09-15T23:25:21.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.353+0000 7f7a68ff9640 1 -- 172.21.15.62:0/2874696175 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7a540109d0 con 0x7f7a6c1003c0 2024-09-15T23:25:21.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.354+0000 7f7a74a02740 1 Processor -- start 2024-09-15T23:25:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.354+0000 7f7a74a02740 1 -- start start 2024-09-15T23:25:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.354+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082144da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.354+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 0x556082149da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.354+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608214a370 0x55608214c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.354+0000 7f7a74a02740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5560820a82f0 con 0x5560821452e0 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.354+0000 7f7a74a02740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556082050270 con 0x5560821449a0 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.354+0000 7f7a74a02740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5560820a42b0 con 0x55608214a370 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082144da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39178/0 (socket says 172.21.15.62:39178) 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6bfff640 1 -- 172.21.15.62:0/2450219541 learned_addr learned my addr 172.21.15.62:0/2450219541 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6affd640 1 --2- 172.21.15.62:0/2450219541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608214a370 0x55608214c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/2450219541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 0x556082149da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6bfff640 1 -- 172.21.15.62:0/2450219541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608214a370 msgr2=0x55608214c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/2450219541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608214a370 0x55608214c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a68ff9640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a580a33c0 0x7f7a580a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6bfff640 1 -- 172.21.15.62:0/2450219541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 msgr2=0x556082149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/2450219541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 0x556082149da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a6bfff640 1 -- 172.21.15.62:0/2450219541 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55608204f340 con 0x5560821449a0 2024-09-15T23:25:21.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.355+0000 7f7a68ff9640 1 -- 172.21.15.62:0/2874696175 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7a540cf880 con 0x7f7a6c1003c0 2024-09-15T23:25:21.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.356+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/2450219541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082144da0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009860 tx=0x7f7a64009430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb9da8dd08ff483b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.356+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a580a33c0 0x7f7a580a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.356+0000 7f7a61ffb640 1 -- 172.21.15.62:0/2450219541 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a64037040 con 0x5560821449a0 2024-09-15T23:25:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.356+0000 7f7a61ffb640 1 -- 172.21.15.62:0/2450219541 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a64041050 con 0x5560821449a0 2024-09-15T23:25:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.356+0000 7f7a61ffb640 1 -- 172.21.15.62:0/2450219541 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6404f070 con 0x5560821449a0 2024-09-15T23:25:21.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.357+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a580a33c0 0x7f7a580a5880 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7a64048000 tx=0x7f7a64049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.357+0000 7f7a74a02740 1 -- 172.21.15.62:0/2450219541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 msgr2=0x556082144da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.357+0000 7f7a74a02740 1 --2- 172.21.15.62:0/2450219541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082144da0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009860 tx=0x7f7a64009430 comp rx=0 tx=0).stop 2024-09-15T23:25:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.357+0000 7f7a74a02740 1 -- 172.21.15.62:0/2450219541 shutdown_connections 2024-09-15T23:25:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.357+0000 7f7a74a02740 1 --2- 172.21.15.62:0/2450219541 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55608214a370 0x55608214c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.362 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.357+0000 7f7a74a02740 1 --2- 172.21.15.62:0/2450219541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 0x556082149da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.357+0000 7f7a74a02740 1 --2- 172.21.15.62:0/2450219541 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082144da0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.357+0000 7f7a74a02740 1 -- 172.21.15.62:0/2450219541 >> 172.21.15.62:0/2450219541 conn(0x556082140010 msgr2=0x556082140410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.363 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.357+0000 7f7a74a02740 1 -- 172.21.15.62:0/2450219541 shutdown_connections 2024-09-15T23:25:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.359+0000 7f7a74a02740 1 -- 172.21.15.62:0/2450219541 wait complete. 2024-09-15T23:25:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.361+0000 7f7a74a02740 1 -- start start 2024-09-15T23:25:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.361+0000 7f7a74a02740 1 Processor -- start 2024-09-15T23:25:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821449a0 0x55608214ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x55608214f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x556082156140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a74a02740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55608214d340 con 0x5560821449a0 2024-09-15T23:25:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a74a02740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556082031290 con 0x55608214a370 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a74a02740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55608214d4c0 con 0x5560821452e0 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a74a02740 10 client.0 _ll_register_callbacks cb 0x556082062140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x556082156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821449a0 0x55608214ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x556082156140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39194/0 (socket says 172.21.15.62:39194) 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6bfff640 1 -- 172.21.15.62:0/1428745904 learned_addr learned my addr 172.21.15.62:0/1428745904 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6affd640 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x55608214f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6bfff640 1 -- 172.21.15.62:0/1428745904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 msgr2=0x55608214f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x55608214f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6bfff640 1 -- 172.21.15.62:0/1428745904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821449a0 msgr2=0x55608214ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821449a0 0x55608214ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6bfff640 1 -- 172.21.15.62:0/1428745904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556081dbb210 con 0x55608214a370 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6affd640 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x55608214f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.363+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821449a0 0x55608214ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.364+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x556082156140 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009780 tx=0x7f7a64049c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeddc8e709ed0855 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.364+0000 7f7a4affd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.364+0000 7f7a4affd640 1 -- 172.21.15.62:0/1428745904 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a64057020 con 0x55608214a370 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.364+0000 7f7a74a02740 1 -- 172.21.15.62:0/1428745904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a64041050 con 0x55608214a370 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.364+0000 7f7a4affd640 1 -- 172.21.15.62:0/1428745904 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7a640035f0 con 0x55608214a370 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.364+0000 7f7a4affd640 1 -- 172.21.15.62:0/1428745904 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7a64062050 con 0x55608214a370 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.364+0000 7f7a74a02740 10 client.27182 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.364+0000 7f7a74a02740 1 -- 172.21.15.62:0/1428745904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556081e87620 con 0x55608214a370 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.364+0000 7f7a4affd640 1 -- 172.21.15.62:0/1428745904 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a64041050 con 0x55608214a370 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a4affd640 1 -- 172.21.15.62:0/1428745904 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f7a64053660 con 0x55608214a370 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a74a02740 10 client.27182 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a74a02740 10 client.27182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a74a02740 20 client.27182 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55608215a440 0x55608215c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a74a02740 4 client.27182 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a74a02740 1 -- 172.21.15.62:0/1428745904 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560820b5a20 con 0x55608215a440 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55608215a440 0x55608215c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a4affd640 10 client.27182 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.365+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55608215a440 0x55608215c830 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15fbed4e45429fc server_cookie=fcb8d28d212f1ac5 in_seq=0 out_seq=0 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.366+0000 7f7a4affd640 1 -- 172.21.15.62:0/1428745904 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5560820b5a20 con 0x55608215a440 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.366+0000 7f7a4affd640 10 client.27182 handle_command_reply: tid=0 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.366+0000 7f7a74a02740 1 client.27182 shutdown 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.366+0000 7f7a74a02740 1 -- 172.21.15.62:0/1428745904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 msgr2=0x556082156140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.366+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x556082156140 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7a64009780 tx=0x7f7a64049c70 comp rx=0 tx=0).stop 2024-09-15T23:25:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.366+0000 7f7a74a02740 1 -- 172.21.15.62:0/1428745904 shutdown_connections 2024-09-15T23:25:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.367+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55608215a440 0x55608215c830 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.367+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x556082156140 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.367+0000 7f7a4affd640 0 client.27182 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.367+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x55608214f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.367+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1428745904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821449a0 0x55608214ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.367+0000 7f7a74a02740 1 -- 172.21.15.62:0/1428745904 >> 172.21.15.62:0/1428745904 conn(0x556082140010 msgr2=0x556082141820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.367+0000 7f7a74a02740 1 -- 172.21.15.62:0/1428745904 shutdown_connections 2024-09-15T23:25:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.367+0000 7f7a74a02740 1 -- 172.21.15.62:0/1428745904 wait complete. 2024-09-15T23:25:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.367+0000 7f7a74a02740 20 client.27182 trim_cache size 0 max 16384 2024-09-15T23:25:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.393+0000 7f7a74a02740 1 Processor -- start 2024-09-15T23:25:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.394+0000 7f7a74a02740 1 -- start start 2024-09-15T23:25:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.394+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821449a0 0x556082144d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.394+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 0x556082156100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.394+0000 7f7a6b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821449a0 0x556082144d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.396 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.394+0000 7f7a6b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821449a0 0x556082144d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49488/0 (socket says 172.21.15.62:49488) 2024-09-15T23:25:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.394+0000 7f7a6b7fe640 1 -- 172.21.15.62:0/1373811057 learned_addr learned my addr 172.21.15.62:0/1373811057 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.394+0000 7f7a6affd640 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 0x556082156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.394+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x55608213f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.394+0000 7f7a74a02740 1 -- 172.21.15.62:0/1373811057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6c1b35a0 con 0x5560821452e0 2024-09-15T23:25:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a74a02740 1 -- 172.21.15.62:0/1373811057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6c1b0e70 con 0x55608214a370 2024-09-15T23:25:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a74a02740 1 -- 172.21.15.62:0/1373811057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6c1aecc0 con 0x5560821449a0 2024-09-15T23:25:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x55608213f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a6b7fe640 1 -- 172.21.15.62:0/1373811057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 msgr2=0x55608213f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x55608213f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a6b7fe640 1 -- 172.21.15.62:0/1373811057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 msgr2=0x556082156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 0x556082156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a6b7fe640 1 -- 172.21.15.62:0/1373811057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55608213fe30 con 0x5560821449a0 2024-09-15T23:25:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x55608213f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821449a0 0x556082144d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7a5409b820 tx=0x7f7a54000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad84802df5e75679 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.395+0000 7f7a617fa640 1 -- 172.21.15.62:0/1373811057 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54020040 con 0x5560821449a0 2024-09-15T23:25:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a617fa640 1 -- 172.21.15.62:0/1373811057 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a5409ce50 con 0x5560821449a0 2024-09-15T23:25:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a617fa640 1 -- 172.21.15.62:0/1373811057 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54098190 con 0x5560821449a0 2024-09-15T23:25:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a74a02740 1 -- 172.21.15.62:0/1373811057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821449a0 msgr2=0x556082144d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821449a0 0x556082144d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7a5409b820 tx=0x7f7a54000bc0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a74a02740 1 -- 172.21.15.62:0/1373811057 shutdown_connections 2024-09-15T23:25:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55608214a370 0x55608213f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560821452e0 0x556082156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1373811057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821449a0 0x556082144d80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.402 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a74a02740 1 -- 172.21.15.62:0/1373811057 >> 172.21.15.62:0/1373811057 conn(0x556082140010 msgr2=0x55608213e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a74a02740 1 -- 172.21.15.62:0/1373811057 shutdown_connections 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.396+0000 7f7a74a02740 1 -- 172.21.15.62:0/1373811057 wait complete. 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a74a02740 1 -- start start 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a74a02740 1 Processor -- start 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082153080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x5560821535c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a74a02740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608214a370 0x5560821c3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a74a02740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6c1af330 con 0x55608214a370 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a74a02740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6c1b07a0 con 0x5560821449a0 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a74a02740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6c1afa00 con 0x5560821452e0 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a74a02740 10 client.0 _ll_register_callbacks cb 0x7f7a5c0098a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a6affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x5560821535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a6affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x5560821535c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:49502/0 (socket says 172.21.15.62:49502) 2024-09-15T23:25:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.397+0000 7f7a6affd640 1 -- 172.21.15.62:0/1200735986 learned_addr learned my addr 172.21.15.62:0/1200735986 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608214a370 0x5560821c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6affd640 1 -- 172.21.15.62:0/1200735986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 msgr2=0x556082153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6affd640 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082153080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.404 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6affd640 1 -- 172.21.15.62:0/1200735986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608214a370 msgr2=0x5560821c3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6affd640 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608214a370 0x5560821c3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6affd640 1 -- 172.21.15.62:0/1200735986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560821c3590 con 0x5560821452e0 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6bfff640 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608214a370 0x5560821c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a6affd640 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x5560821535c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c19f8d0 tx=0x7f7a5c00d7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e68d00894c0c0139 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a4affd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.398+0000 7f7a4affd640 1 -- 172.21.15.62:0/1200735986 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c01f020 con 0x5560821452e0 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a4affd640 1 -- 172.21.15.62:0/1200735986 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7a5c01b600 con 0x5560821452e0 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a74a02740 1 -- 172.21.15.62:0/1200735986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a5409ce50 con 0x5560821452e0 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a4affd640 1 -- 172.21.15.62:0/1200735986 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7a5c02a050 con 0x5560821452e0 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a74a02740 10 client.26941 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:21.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a74a02740 1 -- 172.21.15.62:0/1200735986 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5560821c48d0 con 0x5560821452e0 2024-09-15T23:25:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a4affd640 1 -- 172.21.15.62:0/1200735986 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a5c01bce0 con 0x5560821452e0 2024-09-15T23:25:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a4affd640 1 -- 172.21.15.62:0/1200735986 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f7a5c00b040 con 0x5560821452e0 2024-09-15T23:25:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a74a02740 10 client.26941 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a74a02740 10 client.26941 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a74a02740 20 client.26941 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55608215a440 0x5560821c7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a74a02740 4 client.26941 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:25:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.399+0000 7f7a74a02740 1 -- 172.21.15.62:0/1200735986 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7a5409ce50 con 0x55608215a440 2024-09-15T23:25:21.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.400+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55608215a440 0x5560821c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.401+0000 7f7a6b7fe640 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55608215a440 0x5560821c7370 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c312ddcc0af6d98e server_cookie=8c2f8607d75531a7 in_seq=0 out_seq=0 2024-09-15T23:25:21.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.401+0000 7f7a4affd640 10 client.26941 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a4affd640 1 -- 172.21.15.62:0/1200735986 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7a5409ce50 con 0x55608215a440 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a4affd640 10 client.26941 handle_command_reply: tid=0 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 client.26941 shutdown 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 -- 172.21.15.62:0/1200735986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 msgr2=0x5560821535c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x5560821535c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c19f8d0 tx=0x7f7a5c00d7b0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 -- 172.21.15.62:0/1200735986 shutdown_connections 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55608215a440 0x5560821c7370 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55608214a370 0x5560821c3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a4affd640 0 client.26941 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560821452e0 0x5560821535c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.414 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 --2- 172.21.15.62:0/1200735986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560821449a0 0x556082153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 -- 172.21.15.62:0/1200735986 >> 172.21.15.62:0/1200735986 conn(0x556082140010 msgr2=0x55608213f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 -- 172.21.15.62:0/1200735986 shutdown_connections 2024-09-15T23:25:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 1 -- 172.21.15.62:0/1200735986 wait complete. 2024-09-15T23:25:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.412+0000 7f7a74a02740 20 client.26941 trim_cache size 0 max 16384 2024-09-15T23:25:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:25:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:25:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:25:21.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:25:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a580a33c0 msgr2=0x7f7a580a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a580a33c0 0x7f7a580a5880 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7a64048000 tx=0x7f7a64049000 comp rx=0 tx=0).stop 2024-09-15T23:25:21.416 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1003c0 msgr2=0x7f7a6c19e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1003c0 0x7f7a6c19e3c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400d950 tx=0x7f7a5400de10 comp rx=0 tx=0).stop 2024-09-15T23:25:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 shutdown_connections 2024-09-15T23:25:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7a580a33c0 0x7f7a580a5880 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a6c1056c0 0x7f7a6c1a5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c102d10 0x7f7a6c19e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 --2- 172.21.15.62:0/2874696175 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7a6c1003c0 0x7f7a6c19e3c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.413+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 >> 172.21.15.62:0/2874696175 conn(0x7f7a6c0fc1b0 msgr2=0x7f7a6c103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.414+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 shutdown_connections 2024-09-15T23:25:21.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:21.414+0000 7f7a71c65640 1 -- 172.21.15.62:0/2874696175 wait complete. 2024-09-15T23:25:21.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:21 smithi062 ceph-mon[30949]: pgmap v642: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-09-15T23:25:21.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:21 smithi067 ceph-mon[35665]: pgmap v642: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-09-15T23:25:21.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:21 smithi148 ceph-mon[35610]: pgmap v642: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 1.4 MiB/s wr, 164 op/s 2024-09-15T23:25:22.295 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:25:22.787 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:23.219 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:23 smithi062 ceph-mon[30949]: pgmap v643: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 2.5 MiB/s wr, 293 op/s 2024-09-15T23:25:23.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.347+0000 7fa00d59a640 1 -- 172.21.15.62:0/359650075 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0000319b0 con 0x7fa01011b140 2024-09-15T23:25:23.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.347+0000 7fa014e87640 1 -- 172.21.15.62:0/359650075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa01011b140 msgr2=0x7fa01011d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.347+0000 7fa014e87640 1 --2- 172.21.15.62:0/359650075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa01011b140 0x7fa01011d530 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa000009900 tx=0x7fa00002a440 comp rx=0 tx=0).stop 2024-09-15T23:25:23.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.348+0000 7fa014e87640 1 -- 172.21.15.62:0/359650075 shutdown_connections 2024-09-15T23:25:23.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.348+0000 7fa014e87640 1 --2- 172.21.15.62:0/359650075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa01011b140 0x7fa01011d530 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.348+0000 7fa014e87640 1 --2- 172.21.15.62:0/359650075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa010109830 0x7fa010109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.348+0000 7fa014e87640 1 --2- 172.21.15.62:0/359650075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa010109450 0x7fa010110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.348+0000 7fa014e87640 1 -- 172.21.15.62:0/359650075 >> 172.21.15.62:0/359650075 conn(0x7fa01006e140 msgr2=0x7fa01006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.349+0000 7fa014e87640 1 -- 172.21.15.62:0/359650075 shutdown_connections 2024-09-15T23:25:23.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.349+0000 7fa014e87640 1 -- 172.21.15.62:0/359650075 wait complete. 2024-09-15T23:25:23.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.350+0000 7fa014e87640 1 Processor -- start 2024-09-15T23:25:23.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.350+0000 7fa014e87640 1 -- start start 2024-09-15T23:25:23.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.350+0000 7fa014e87640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa010109450 0x7fa010118a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.350+0000 7fa014e87640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa010109830 0x7fa0101109f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.350+0000 7fa014e87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa01011b140 0x7fa010110f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.350+0000 7fa014e87640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa010120e90 con 0x7fa010109450 2024-09-15T23:25:23.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.350+0000 7fa014e87640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa010120d10 con 0x7fa01011b140 2024-09-15T23:25:23.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.350+0000 7fa014e87640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa010121010 con 0x7fa010109830 2024-09-15T23:25:23.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.351+0000 7fa00e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa010109450 0x7fa010118a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.351+0000 7fa00e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa010109450 0x7fa010118a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38628/0 (socket says 172.21.15.62:38628) 2024-09-15T23:25:23.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.351+0000 7fa00e59c640 1 -- 172.21.15.62:0/2666722965 learned_addr learned my addr 172.21.15.62:0/2666722965 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:23.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.351+0000 7fa00ed9d640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa01011b140 0x7fa010110f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.351+0000 7fa00dd9b640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa010109830 0x7fa0101109f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.352+0000 7fa00e59c640 1 -- 172.21.15.62:0/2666722965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa010109830 msgr2=0x7fa0101109f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.352+0000 7fa00e59c640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa010109830 0x7fa0101109f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.352+0000 7fa00e59c640 1 -- 172.21.15.62:0/2666722965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa01011b140 msgr2=0x7fa010110f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.352+0000 7fa00e59c640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa01011b140 0x7fa010110f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.352+0000 7fa00e59c640 1 -- 172.21.15.62:0/2666722965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa010111750 con 0x7fa010109450 2024-09-15T23:25:23.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.358+0000 7fa00e59c640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa010109450 0x7fa010118a20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa00800ba50 tx=0x7fa00800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.362+0000 7f9fff7fe640 1 -- 172.21.15.62:0/2666722965 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008019070 con 0x7fa010109450 2024-09-15T23:25:23.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.362+0000 7f9fff7fe640 1 -- 172.21.15.62:0/2666722965 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa00800cbe0 con 0x7fa010109450 2024-09-15T23:25:23.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.362+0000 7f9fff7fe640 1 -- 172.21.15.62:0/2666722965 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008007660 con 0x7fa010109450 2024-09-15T23:25:23.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.363+0000 7fa014e87640 1 -- 172.21.15.62:0/2666722965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0101bc140 con 0x7fa010109450 2024-09-15T23:25:23.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.363+0000 7fa014e87640 1 -- 172.21.15.62:0/2666722965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0101bc630 con 0x7fa010109450 2024-09-15T23:25:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.370+0000 7f9fff7fe640 1 -- 172.21.15.62:0/2666722965 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa00801c070 con 0x7fa010109450 2024-09-15T23:25:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.371+0000 7f9fff7fe640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ff80a3410 0x7f9ff80a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.371+0000 7f9fff7fe640 1 -- 172.21.15.62:0/2666722965 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa0080d0210 con 0x7fa010109450 2024-09-15T23:25:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.372+0000 7fa00dd9b640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ff80a3410 0x7f9ff80a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.372+0000 7fa017260740 1 Processor -- start 2024-09-15T23:25:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.377+0000 7fa017260740 1 -- start start 2024-09-15T23:25:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.377+0000 7fa017260740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b36090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.377+0000 7fa017260740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b36a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.377+0000 7fa017260740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b43760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.377+0000 7fa017260740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a4a5a9c2f0 con 0x55a4a5b365d0 2024-09-15T23:25:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.377+0000 7fa017260740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a4a5a43cf0 con 0x55a4a5b37b40 2024-09-15T23:25:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.377+0000 7fa017260740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a4a5a982b0 con 0x55a4a5b41370 2024-09-15T23:25:23.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.380+0000 7fa00dd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b36a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.380+0000 7fa00dd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b36a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38638/0 (socket says 172.21.15.62:38638) 2024-09-15T23:25:23.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.380+0000 7fa00dd9b640 1 -- 172.21.15.62:0/3614124554 learned_addr learned my addr 172.21.15.62:0/3614124554 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:23.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.381+0000 7fa00e59c640 1 --2- 172.21.15.62:0/3614124554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b36090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.381+0000 7fa00dd9b640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ff80a3410 0x7f9ff80a58d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa0101121b0 tx=0x7fa004008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.381+0000 7fa00dd9b640 1 -- 172.21.15.62:0/3614124554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 msgr2=0x55a4a5b43760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:25:23.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.381+0000 7fa00dd9b640 1 --2- 172.21.15.62:0/3614124554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b43760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.381+0000 7fa00dd9b640 1 -- 172.21.15.62:0/3614124554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 msgr2=0x55a4a5b36090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.381+0000 7fa00dd9b640 1 --2- 172.21.15.62:0/3614124554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b36090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.381+0000 7fa00dd9b640 1 -- 172.21.15.62:0/3614124554 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4a5a283a0 con 0x55a4a5b365d0 2024-09-15T23:25:23.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.384+0000 7fa00dd9b640 1 --2- 172.21.15.62:0/3614124554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b36a10 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa00400d890 tx=0x7fa004018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c47651e1562e983 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.385+0000 7f9ffcff9640 1 -- 172.21.15.62:0/3614124554 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00401a040 con 0x55a4a5b365d0 2024-09-15T23:25:23.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.385+0000 7f9ffcff9640 1 -- 172.21.15.62:0/3614124554 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa004023070 con 0x55a4a5b365d0 2024-09-15T23:25:23.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.386+0000 7fa017260740 1 -- 172.21.15.62:0/3614124554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 msgr2=0x55a4a5b36a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.386+0000 7fa017260740 1 --2- 172.21.15.62:0/3614124554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b36a10 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fa00400d890 tx=0x7fa004018000 comp rx=0 tx=0).stop 2024-09-15T23:25:23.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.396+0000 7fa017260740 1 -- 172.21.15.62:0/3614124554 shutdown_connections 2024-09-15T23:25:23.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.396+0000 7fa017260740 1 --2- 172.21.15.62:0/3614124554 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b43760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.396+0000 7fa017260740 1 --2- 172.21.15.62:0/3614124554 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b36a10 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.396+0000 7fa017260740 1 --2- 172.21.15.62:0/3614124554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b36090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.396+0000 7fa017260740 1 -- 172.21.15.62:0/3614124554 >> 172.21.15.62:0/3614124554 conn(0x55a4a5aff310 msgr2=0x55a4a5b32c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.398+0000 7fa017260740 1 -- 172.21.15.62:0/3614124554 shutdown_connections 2024-09-15T23:25:23.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.401+0000 7fa017260740 1 -- 172.21.15.62:0/3614124554 wait complete. 2024-09-15T23:25:23.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.401+0000 7fa017260740 1 -- start start 2024-09-15T23:25:23.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.401+0000 7fa017260740 1 Processor -- start 2024-09-15T23:25:23.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.402+0000 7fa017260740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b40c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.402+0000 7fa017260740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b3def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.402+0000 7fa017260740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b3e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.402+0000 7fa017260740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a4a5b44060 con 0x55a4a5b365d0 2024-09-15T23:25:23.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.402+0000 7fa017260740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55a4a5a57060 con 0x55a4a5b37b40 2024-09-15T23:25:23.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.402+0000 7fa017260740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55a4a5b441e0 con 0x55a4a5b41370 2024-09-15T23:25:23.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.402+0000 7fa017260740 10 client.0 _ll_register_callbacks cb 0x55a4a57aa8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:23.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.404+0000 7fa00dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b3def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.404+0000 7fa00e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b40c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.404+0000 7fa00e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b40c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38650/0 (socket says 172.21.15.62:38650) 2024-09-15T23:25:23.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.404+0000 7fa00e59c640 1 -- 172.21.15.62:0/3745363809 learned_addr learned my addr 172.21.15.62:0/3745363809 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:23.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.405+0000 7fa00ed9d640 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b3e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.407+0000 7fa00e59c640 1 -- 172.21.15.62:0/3745363809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 msgr2=0x55a4a5b3e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.407+0000 7fa00e59c640 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b3e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.407+0000 7fa00e59c640 1 -- 172.21.15.62:0/3745363809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 msgr2=0x55a4a5b3def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.407+0000 7fa00e59c640 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b3def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.407+0000 7fa00e59c640 1 -- 172.21.15.62:0/3745363809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4a580d3c0 con 0x55a4a5b365d0 2024-09-15T23:25:23.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.409+0000 7fa00e59c640 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b40c40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa0080cd930 tx=0x7fa00809c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d858ea7a89b3e551 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.410+0000 7f9ff5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:23.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.411+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/3745363809 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0080cda40 con 0x55a4a5b365d0 2024-09-15T23:25:23.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.411+0000 7fa017260740 1 -- 172.21.15.62:0/3745363809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa004023070 con 0x55a4a5b365d0 2024-09-15T23:25:23.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.411+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/3745363809 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa0080a4c30 con 0x55a4a5b365d0 2024-09-15T23:25:23.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.412+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/3745363809 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa0080a4e30 con 0x55a4a5b365d0 2024-09-15T23:25:23.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.412+0000 7fa017260740 10 client.17637 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:23.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.412+0000 7fa017260740 1 -- 172.21.15.62:0/3745363809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a4a5b4fad0 con 0x55a4a5b365d0 2024-09-15T23:25:23.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.417+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/3745363809 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa004023070 con 0x55a4a5b365d0 2024-09-15T23:25:23.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.417+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/3745363809 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa00809b560 con 0x55a4a5b365d0 2024-09-15T23:25:23.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.417+0000 7fa017260740 10 client.17637 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:23.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.417+0000 7fa017260740 10 client.17637 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:25:23.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.417+0000 7fa017260740 20 client.17637 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:23.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.417+0000 7fa017260740 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4a5b50f90 0x55a4a5b53380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.417+0000 7fa017260740 4 client.17637 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:23.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.417+0000 7fa017260740 1 -- 172.21.15.62:0/3745363809 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4a5aa9860 con 0x55a4a5b50f90 2024-09-15T23:25:23.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.418+0000 7fa00dd9b640 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4a5b50f90 0x55a4a5b53380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.431+0000 7fa00dd9b640 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4a5b50f90 0x55a4a5b53380 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1357f4d819b7a3ee server_cookie=3a235dc8e73ac9e in_seq=0 out_seq=0 2024-09-15T23:25:23.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.432+0000 7f9ff5ffb640 10 client.17637 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:25:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.434+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/3745363809 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a4a5aa9860 con 0x55a4a5b50f90 2024-09-15T23:25:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.434+0000 7f9ff5ffb640 10 client.17637 handle_command_reply: tid=0 2024-09-15T23:25:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.435+0000 7fa017260740 1 client.17637 shutdown 2024-09-15T23:25:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.438+0000 7fa017260740 1 -- 172.21.15.62:0/3745363809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 msgr2=0x55a4a5b40c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.438+0000 7fa017260740 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b40c40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa0080cd930 tx=0x7fa00809c580 comp rx=0 tx=0).stop 2024-09-15T23:25:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.438+0000 7fa017260740 1 -- 172.21.15.62:0/3745363809 shutdown_connections 2024-09-15T23:25:23.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.438+0000 7fa017260740 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4a5b50f90 0x55a4a5b53380 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.438+0000 7fa017260740 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b3e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.438+0000 7fa017260740 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b3def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.438+0000 7fa017260740 1 --2- 172.21.15.62:0/3745363809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b365d0 0x55a4a5b40c40 secure :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa0080cd930 tx=0x7fa00809c580 comp rx=0 tx=0).stop 2024-09-15T23:25:23.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.438+0000 7fa017260740 1 -- 172.21.15.62:0/3745363809 >> 172.21.15.62:0/3745363809 conn(0x55a4a5aff310 msgr2=0x55a4a5b38930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.438+0000 7f9ff5ffb640 0 client.17637 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:25:23.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.443+0000 7fa017260740 1 -- 172.21.15.62:0/3745363809 shutdown_connections 2024-09-15T23:25:23.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.443+0000 7fa017260740 1 -- 172.21.15.62:0/3745363809 wait complete. 2024-09-15T23:25:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.444+0000 7fa017260740 20 client.17637 trim_cache size 0 max 16384 2024-09-15T23:25:23.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.472+0000 7fa017260740 1 Processor -- start 2024-09-15T23:25:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.472+0000 7fa017260740 1 -- start start 2024-09-15T23:25:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.472+0000 7fa017260740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b4d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.472+0000 7fa017260740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b58be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.472+0000 7fa017260740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b59120 0x55a4a5b3b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.472+0000 7fa017260740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff0012540 con 0x55a4a5b59120 2024-09-15T23:25:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.472+0000 7fa017260740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff000fe10 con 0x55a4a5b37b40 2024-09-15T23:25:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.472+0000 7fa017260740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff000dc60 con 0x55a4a5b41370 2024-09-15T23:25:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.473+0000 7fa00e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b4d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.473+0000 7fa00dd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b58be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.473+0000 7fa00e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b4d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50538/0 (socket says 172.21.15.62:50538) 2024-09-15T23:25:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.473+0000 7fa00e59c640 1 -- 172.21.15.62:0/2894431370 learned_addr learned my addr 172.21.15.62:0/2894431370 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.474+0000 7fa00e59c640 1 -- 172.21.15.62:0/2894431370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 msgr2=0x55a4a5b58be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.474+0000 7fa00e59c640 1 --2- 172.21.15.62:0/2894431370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b58be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.474+0000 7fa00e59c640 1 -- 172.21.15.62:0/2894431370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b59120 msgr2=0x55a4a5b3b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.474+0000 7fa00e59c640 1 --2- 172.21.15.62:0/2894431370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b59120 0x55a4a5b3b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.474+0000 7fa00e59c640 1 -- 172.21.15.62:0/2894431370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4a5b40fb0 con 0x55a4a5b37b40 2024-09-15T23:25:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.474+0000 7fa00e59c640 1 --2- 172.21.15.62:0/2894431370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b4d3e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa008098960 tx=0x7fa0080987a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d83cfd7fb78098d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.475+0000 7f9ff7fff640 1 -- 172.21.15.62:0/2894431370 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0080a1d20 con 0x55a4a5b37b40 2024-09-15T23:25:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.475+0000 7f9ff7fff640 1 -- 172.21.15.62:0/2894431370 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa00809a800 con 0x55a4a5b37b40 2024-09-15T23:25:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.475+0000 7f9ff7fff640 1 -- 172.21.15.62:0/2894431370 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0080b5160 con 0x55a4a5b37b40 2024-09-15T23:25:23.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.476+0000 7fa017260740 1 -- 172.21.15.62:0/2894431370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 msgr2=0x55a4a5b4d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.476+0000 7fa017260740 1 --2- 172.21.15.62:0/2894431370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b4d3e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa008098960 tx=0x7fa0080987a0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.476+0000 7fa017260740 1 -- 172.21.15.62:0/2894431370 shutdown_connections 2024-09-15T23:25:23.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.476+0000 7fa017260740 1 --2- 172.21.15.62:0/2894431370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b59120 0x55a4a5b3b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.476+0000 7fa017260740 1 --2- 172.21.15.62:0/2894431370 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b58be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.476+0000 7fa017260740 1 --2- 172.21.15.62:0/2894431370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b37b40 0x55a4a5b4d3e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.476+0000 7fa017260740 1 -- 172.21.15.62:0/2894431370 >> 172.21.15.62:0/2894431370 conn(0x55a4a5aff310 msgr2=0x55a4a5b31d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.476+0000 7fa017260740 1 -- 172.21.15.62:0/2894431370 shutdown_connections 2024-09-15T23:25:23.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.476+0000 7fa017260740 1 -- 172.21.15.62:0/2894431370 wait complete. 2024-09-15T23:25:23.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.477+0000 7fa017260740 1 -- start start 2024-09-15T23:25:23.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.477+0000 7fa017260740 1 Processor -- start 2024-09-15T23:25:23.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.477+0000 7fa017260740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b37b40 0x55a4a5b45a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.477+0000 7fa017260740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b45fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.477+0000 7fa017260740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b59120 0x55a4a5bbabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.477+0000 7fa017260740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff000e2d0 con 0x55a4a5b37b40 2024-09-15T23:25:23.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.477+0000 7fa017260740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff000f740 con 0x55a4a5b59120 2024-09-15T23:25:23.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.477+0000 7fa017260740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff000e9a0 con 0x55a4a5b41370 2024-09-15T23:25:23.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa017260740 10 client.0 _ll_register_callbacks cb 0x7fa0080980e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:23.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b59120 0x55a4a5bbabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b59120 0x55a4a5bbabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50546/0 (socket says 172.21.15.62:50546) 2024-09-15T23:25:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00ed9d640 1 -- 172.21.15.62:0/1264015080 learned_addr learned my addr 172.21.15.62:0/1264015080 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00e59c640 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b37b40 0x55a4a5b45a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00dd9b640 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b45fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00ed9d640 1 -- 172.21.15.62:0/1264015080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 msgr2=0x55a4a5b45fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00ed9d640 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b45fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00ed9d640 1 -- 172.21.15.62:0/1264015080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b37b40 msgr2=0x55a4a5b45a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00ed9d640 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b37b40 0x55a4a5b45a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.478+0000 7fa00ed9d640 1 -- 172.21.15.62:0/1264015080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4a5bbb1a0 con 0x55a4a5b59120 2024-09-15T23:25:23.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.479+0000 7fa00ed9d640 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b59120 0x55a4a5bbabb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa000006010 tx=0x7fa0000044d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c959986b0ce82ef4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:23.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.479+0000 7f9ff5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:25:23.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.479+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/1264015080 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa000035040 con 0x55a4a5b59120 2024-09-15T23:25:23.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.479+0000 7fa017260740 1 -- 172.21.15.62:0/1264015080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa00809a800 con 0x55a4a5b59120 2024-09-15T23:25:23.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.479+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/1264015080 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa0000319c0 con 0x55a4a5b59120 2024-09-15T23:25:23.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.480+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/1264015080 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa000005020 con 0x55a4a5b59120 2024-09-15T23:25:23.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.480+0000 7fa017260740 10 client.27203 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.480+0000 7fa017260740 1 -- 172.21.15.62:0/1264015080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a4a5bbc4b0 con 0x55a4a5b59120 2024-09-15T23:25:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.480+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/1264015080 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa00809a800 con 0x55a4a5b59120 2024-09-15T23:25:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.481+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/1264015080 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa000044070 con 0x55a4a5b59120 2024-09-15T23:25:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.481+0000 7fa017260740 10 client.27203 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.481+0000 7fa017260740 10 client.27203 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:25:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.481+0000 7fa017260740 20 client.27203 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.481+0000 7fa017260740 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4a5bbcb80 0x55a4a5bbefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:23.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.481+0000 7fa017260740 4 client.27203 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.481+0000 7fa017260740 1 -- 172.21.15.62:0/1264015080 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a4a5aa3d70 con 0x55a4a5bbcb80 2024-09-15T23:25:23.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.482+0000 7fa00e59c640 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4a5bbcb80 0x55a4a5bbefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:23.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.484+0000 7f9ff5ffb640 10 client.27203 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:25:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.484+0000 7fa00e59c640 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4a5bbcb80 0x55a4a5bbefa0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b43d546c7bf0b2ef server_cookie=346a038a66759e99 in_seq=0 out_seq=0 2024-09-15T23:25:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.485+0000 7f9ff5ffb640 1 -- 172.21.15.62:0/1264015080 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a4a5aa3d70 con 0x55a4a5bbcb80 2024-09-15T23:25:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.485+0000 7f9ff5ffb640 10 client.27203 handle_command_reply: tid=0 2024-09-15T23:25:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7fa017260740 1 client.27203 shutdown 2024-09-15T23:25:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7fa017260740 1 -- 172.21.15.62:0/1264015080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b59120 msgr2=0x55a4a5bbabb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7fa017260740 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b59120 0x55a4a5bbabb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa000006010 tx=0x7fa0000044d0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7fa017260740 1 -- 172.21.15.62:0/1264015080 shutdown_connections 2024-09-15T23:25:23.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7fa017260740 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55a4a5bbcb80 0x55a4a5bbefa0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7fa017260740 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a4a5b59120 0x55a4a5bbabb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7fa017260740 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55a4a5b41370 0x55a4a5b45fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7fa017260740 1 --2- 172.21.15.62:0/1264015080 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a4a5b37b40 0x55a4a5b45a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7fa017260740 1 -- 172.21.15.62:0/1264015080 >> 172.21.15.62:0/1264015080 conn(0x55a4a5aff310 msgr2=0x55a4a5b38110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.486+0000 7f9ff5ffb640 0 client.27203 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:25:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.487+0000 7fa017260740 1 -- 172.21.15.62:0/1264015080 shutdown_connections 2024-09-15T23:25:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.487+0000 7fa017260740 1 -- 172.21.15.62:0/1264015080 wait complete. 2024-09-15T23:25:23.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.487+0000 7fa017260740 20 client.27203 trim_cache size 0 max 16384 2024-09-15T23:25:23.493 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:25:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.487+0000 7fa014e87640 1 -- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ff80a3410 msgr2=0x7f9ff80a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.487+0000 7fa014e87640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ff80a3410 0x7f9ff80a58d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa0101121b0 tx=0x7fa004008040 comp rx=0 tx=0).stop 2024-09-15T23:25:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 -- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa010109450 msgr2=0x7fa010118a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa010109450 0x7fa010118a20 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa00800ba50 tx=0x7fa00800bf10 comp rx=0 tx=0).stop 2024-09-15T23:25:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 -- 172.21.15.62:0/2666722965 shutdown_connections 2024-09-15T23:25:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9ff80a3410 0x7f9ff80a58d0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa01011b140 0x7fa010110f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa010109830 0x7fa0101109f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 --2- 172.21.15.62:0/2666722965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa010109450 0x7fa010118a20 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:23.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 -- 172.21.15.62:0/2666722965 >> 172.21.15.62:0/2666722965 conn(0x7fa01006e140 msgr2=0x7fa01011bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:23.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 -- 172.21.15.62:0/2666722965 shutdown_connections 2024-09-15T23:25:23.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:23.488+0000 7fa014e87640 1 -- 172.21.15.62:0/2666722965 wait complete. 2024-09-15T23:25:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:23 smithi067 ceph-mon[35665]: pgmap v643: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 2.5 MiB/s wr, 293 op/s 2024-09-15T23:25:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:23 smithi148 ceph-mon[35610]: pgmap v643: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 2.5 MiB/s wr, 293 op/s 2024-09-15T23:25:24.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:25:24.573 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 545.0 s, 537.6 tps, lat 184.356 ms stddev 219.536 2024-09-15T23:25:24.655 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:25 smithi062 ceph-mon[30949]: pgmap v644: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 3.4 MiB/s wr, 389 op/s 2024-09-15T23:25:25.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:25 smithi067 ceph-mon[35665]: pgmap v644: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 3.4 MiB/s wr, 389 op/s 2024-09-15T23:25:25.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:25 smithi148 ceph-mon[35610]: pgmap v644: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 3.4 MiB/s wr, 389 op/s 2024-09-15T23:25:26.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.307+0000 7f00a43da640 1 -- 172.21.15.62:0/1736827318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f009c109830 msgr2=0x7f009c109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.307+0000 7f00a43da640 1 --2- 172.21.15.62:0/1736827318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f009c109830 0x7f009c109c90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f008c01a910 tx=0x7f008c03b470 comp rx=0 tx=0).stop 2024-09-15T23:25:26.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.307+0000 7f00a43da640 1 -- 172.21.15.62:0/1736827318 shutdown_connections 2024-09-15T23:25:26.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.307+0000 7f00a43da640 1 --2- 172.21.15.62:0/1736827318 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f009c11b140 0x7f009c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.307+0000 7f00a43da640 1 --2- 172.21.15.62:0/1736827318 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f009c109830 0x7f009c109c90 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.307+0000 7f00a43da640 1 --2- 172.21.15.62:0/1736827318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009c109450 0x7f009c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.307+0000 7f00a43da640 1 -- 172.21.15.62:0/1736827318 >> 172.21.15.62:0/1736827318 conn(0x7f009c06e140 msgr2=0x7f009c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.309+0000 7f00a43da640 1 -- 172.21.15.62:0/1736827318 shutdown_connections 2024-09-15T23:25:26.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.309+0000 7f00a43da640 1 -- 172.21.15.62:0/1736827318 wait complete. 2024-09-15T23:25:26.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.310+0000 7f00a43da640 1 Processor -- start 2024-09-15T23:25:26.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.310+0000 7f00a43da640 1 -- start start 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.310+0000 7f00a43da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f009c109450 0x7f009c11a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.310+0000 7f00a43da640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f009c109830 0x7f009c1168b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.310+0000 7f00a43da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009c11b140 0x7f009c116df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.310+0000 7f00a43da640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f009c113340 con 0x7f009c109450 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.310+0000 7f00a43da640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1131c0 con 0x7f009c11b140 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.310+0000 7f00a43da640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1134c0 con 0x7f009c109830 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.311+0000 7f00a2977640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009c11b140 0x7f009c116df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.311+0000 7f00a2977640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009c11b140 0x7f009c116df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50562/0 (socket says 172.21.15.62:50562) 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.311+0000 7f00a2977640 1 -- 172.21.15.62:0/1510357013 learned_addr learned my addr 172.21.15.62:0/1510357013 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.311+0000 7f00a1975640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f009c109830 0x7f009c1168b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.311+0000 7f00a2977640 1 -- 172.21.15.62:0/1510357013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f009c109830 msgr2=0x7f009c1168b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.311+0000 7f00a2977640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f009c109830 0x7f009c1168b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.311+0000 7f00a2977640 1 -- 172.21.15.62:0/1510357013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f009c109450 msgr2=0x7f009c11a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.311+0000 7f00a2977640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f009c109450 0x7f009c11a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.311+0000 7f00a2977640 1 -- 172.21.15.62:0/1510357013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c1174f0 con 0x7f009c11b140 2024-09-15T23:25:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.312+0000 7f00a2977640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009c11b140 0x7f009c116df0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f009800d720 tx=0x7f009800dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.312+0000 7f00937fe640 1 -- 172.21.15.62:0/1510357013 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098017030 con 0x7f009c11b140 2024-09-15T23:25:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.312+0000 7f00a43da640 1 -- 172.21.15.62:0/1510357013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c1ab170 con 0x7f009c11b140 2024-09-15T23:25:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.312+0000 7f00a43da640 1 -- 172.21.15.62:0/1510357013 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c1ab510 con 0x7f009c11b140 2024-09-15T23:25:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.312+0000 7f00937fe640 1 -- 172.21.15.62:0/1510357013 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f009800de30 con 0x7f009c11b140 2024-09-15T23:25:26.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.312+0000 7f00937fe640 1 -- 172.21.15.62:0/1510357013 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098004ba0 con 0x7f009c11b140 2024-09-15T23:25:26.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.316+0000 7f00937fe640 1 -- 172.21.15.62:0/1510357013 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f009801a070 con 0x7f009c11b140 2024-09-15T23:25:26.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.317+0000 7f00a67b3740 1 Processor -- start 2024-09-15T23:25:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.318+0000 7f00a67b3740 1 -- start start 2024-09-15T23:25:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.318+0000 7f00937fe640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00880a3380 0x7f00880a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.318+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb41ab40 0x557beb419090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.318+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb419a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.318+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 0x557beb426760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.318+0000 7f00a67b3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557beb37f2f0 con 0x557beb424370 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.318+0000 7f00a67b3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557beb326cf0 con 0x557beb4195d0 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.318+0000 7f00a67b3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557beb37b2b0 con 0x557beb41ab40 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.318+0000 7f00937fe640 1 -- 172.21.15.62:0/1510357013 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f00980cf670 con 0x7f009c11b140 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a2977640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 0x557beb426760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a1975640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb419a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a1975640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb419a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50566/0 (socket says 172.21.15.62:50566) 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a1975640 1 -- 172.21.15.62:0/1519094997 learned_addr learned my addr 172.21.15.62:0/1519094997 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a2176640 1 --2- 172.21.15.62:0/1519094997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb41ab40 0x557beb419090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a1975640 1 -- 172.21.15.62:0/1519094997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb41ab40 msgr2=0x557beb419090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a1975640 1 --2- 172.21.15.62:0/1519094997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb41ab40 0x557beb419090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a1975640 1 -- 172.21.15.62:0/1519094997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 msgr2=0x557beb426760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a1975640 1 --2- 172.21.15.62:0/1519094997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 0x557beb426760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.319+0000 7f00a1975640 1 -- 172.21.15.62:0/1519094997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557beb30b3a0 con 0x557beb4195d0 2024-09-15T23:25:26.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.320+0000 7f00a1975640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00880a3380 0x7f00880a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.320+0000 7f00a1975640 1 --2- 172.21.15.62:0/1519094997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb419a10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f008c01a910 tx=0x7f008c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9b7085bd8324a48 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.320+0000 7f0090dea640 1 -- 172.21.15.62:0/1519094997 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c040730 con 0x557beb4195d0 2024-09-15T23:25:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.321+0000 7f0090dea640 1 -- 172.21.15.62:0/1519094997 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008c040e30 con 0x557beb4195d0 2024-09-15T23:25:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.321+0000 7f0090dea640 1 -- 172.21.15.62:0/1519094997 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c050440 con 0x557beb4195d0 2024-09-15T23:25:26.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a67b3740 1 -- 172.21.15.62:0/1519094997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 msgr2=0x557beb419a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a67b3740 1 --2- 172.21.15.62:0/1519094997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb419a10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f008c01a910 tx=0x7f008c0047c0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a1975640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00880a3380 0x7f00880a5840 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f008c05b000 tx=0x7f008c050720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a67b3740 1 -- 172.21.15.62:0/1519094997 shutdown_connections 2024-09-15T23:25:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a67b3740 1 --2- 172.21.15.62:0/1519094997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 0x557beb426760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a67b3740 1 --2- 172.21.15.62:0/1519094997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb419a10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a67b3740 1 --2- 172.21.15.62:0/1519094997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb41ab40 0x557beb419090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a67b3740 1 -- 172.21.15.62:0/1519094997 >> 172.21.15.62:0/1519094997 conn(0x557beb3e2310 msgr2=0x557beb415c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a67b3740 1 -- 172.21.15.62:0/1519094997 shutdown_connections 2024-09-15T23:25:26.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.322+0000 7f00a67b3740 1 -- 172.21.15.62:0/1519094997 wait complete. 2024-09-15T23:25:26.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.323+0000 7f00a67b3740 1 -- start start 2024-09-15T23:25:26.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.323+0000 7f00a67b3740 1 Processor -- start 2024-09-15T23:25:26.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.324+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb423c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.324+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb420ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.324+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb424370 0x557beb421420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.324+0000 7f00a67b3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557beb4270c0 con 0x557beb41ab40 2024-09-15T23:25:26.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.324+0000 7f00a67b3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557beb33a060 con 0x557beb4195d0 2024-09-15T23:25:26.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.324+0000 7f00a67b3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557beb427240 con 0x557beb424370 2024-09-15T23:25:26.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.324+0000 7f00a67b3740 10 client.0 _ll_register_callbacks cb 0x557beb08d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:26.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.325+0000 7f00a1975640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb420ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.325+0000 7f00a1975640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb420ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38714/0 (socket says 172.21.15.62:38714) 2024-09-15T23:25:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.325+0000 7f00a1975640 1 -- 172.21.15.62:0/733128348 learned_addr learned my addr 172.21.15.62:0/733128348 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.325+0000 7f00a2176640 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb423c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.325+0000 7f00a1975640 1 -- 172.21.15.62:0/733128348 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb424370 msgr2=0x557beb421420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:25:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.325+0000 7f00a1975640 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb424370 0x557beb421420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.325+0000 7f00a1975640 1 -- 172.21.15.62:0/733128348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 msgr2=0x557beb423c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.325+0000 7f00a1975640 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb423c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.325+0000 7f00a1975640 1 -- 172.21.15.62:0/733128348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557beb0f03c0 con 0x557beb41ab40 2024-09-15T23:25:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.326+0000 7f00a1975640 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb420ee0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f008c044040 tx=0x7f008c052000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0a46d31b38d042 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.326+0000 7f0085ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.326+0000 7f0085ffb640 1 -- 172.21.15.62:0/733128348 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c06c070 con 0x557beb41ab40 2024-09-15T23:25:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.326+0000 7f00a67b3740 1 -- 172.21.15.62:0/733128348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f008c040e30 con 0x557beb41ab40 2024-09-15T23:25:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.327+0000 7f0085ffb640 1 -- 172.21.15.62:0/733128348 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f008c040630 con 0x557beb41ab40 2024-09-15T23:25:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.327+0000 7f0085ffb640 1 -- 172.21.15.62:0/733128348 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f008c079050 con 0x557beb41ab40 2024-09-15T23:25:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.327+0000 7f00a67b3740 10 client.17646 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.327+0000 7f00a67b3740 1 -- 172.21.15.62:0/733128348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557beb432ae0 con 0x557beb41ab40 2024-09-15T23:25:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.328+0000 7f0085ffb640 1 -- 172.21.15.62:0/733128348 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008c054050 con 0x557beb41ab40 2024-09-15T23:25:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.328+0000 7f0085ffb640 1 -- 172.21.15.62:0/733128348 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f008c042540 con 0x557beb41ab40 2024-09-15T23:25:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.328+0000 7f00a67b3740 10 client.17646 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.328+0000 7f00a67b3740 10 client.17646 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.328+0000 7f00a67b3740 20 client.17646 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.329+0000 7f00a67b3740 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557beb434190 0x557beb436650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.329+0000 7f00a67b3740 4 client.17646 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.329+0000 7f00a67b3740 1 -- 172.21.15.62:0/733128348 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f008c040e30 con 0x557beb434190 2024-09-15T23:25:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.329+0000 7f00a2176640 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557beb434190 0x557beb436650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.329+0000 7f00a2176640 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557beb434190 0x557beb436650 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2803c1ec7d52c035 server_cookie=4e7da7857d0d1433 in_seq=0 out_seq=0 2024-09-15T23:25:26.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.329+0000 7f0085ffb640 10 client.17646 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:26.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.330+0000 7f0085ffb640 1 -- 172.21.15.62:0/733128348 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f008c040e30 con 0x557beb434190 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.330+0000 7f0085ffb640 10 client.17646 handle_command_reply: tid=0 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.330+0000 7f00a67b3740 1 client.17646 shutdown 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.330+0000 7f00a67b3740 1 -- 172.21.15.62:0/733128348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 msgr2=0x557beb420ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.330+0000 7f00a67b3740 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb420ee0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f008c044040 tx=0x7f008c052000 comp rx=0 tx=0).stop 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f00a67b3740 1 -- 172.21.15.62:0/733128348 shutdown_connections 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f00a67b3740 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557beb434190 0x557beb436650 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f00a67b3740 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb424370 0x557beb421420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f00a67b3740 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb420ee0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f00a67b3740 1 --2- 172.21.15.62:0/733128348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb4195d0 0x557beb423c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f00a67b3740 1 -- 172.21.15.62:0/733128348 >> 172.21.15.62:0/733128348 conn(0x557beb3e2310 msgr2=0x557beb41b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f0085ffb640 0 client.17646 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f00a67b3740 1 -- 172.21.15.62:0/733128348 shutdown_connections 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f00a67b3740 1 -- 172.21.15.62:0/733128348 wait complete. 2024-09-15T23:25:26.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.331+0000 7f00a67b3740 20 client.17646 trim_cache size 0 max 16384 2024-09-15T23:25:26.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.347+0000 7f00a67b3740 1 Processor -- start 2024-09-15T23:25:26.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.347+0000 7f00a67b3740 1 -- start start 2024-09-15T23:25:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.348+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb41c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.348+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb41c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.348+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb424370 0x557beb41e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.348+0000 7f00a67b3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f00800124e0 con 0x557beb41ab40 2024-09-15T23:25:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.348+0000 7f00a67b3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f008000fdb0 con 0x557beb424370 2024-09-15T23:25:26.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.348+0000 7f00a67b3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f008000dc00 con 0x557beb4195d0 2024-09-15T23:25:26.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.348+0000 7f00a2176640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb41c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.348+0000 7f00a2176640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb41c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56866/0 (socket says 172.21.15.62:56866) 2024-09-15T23:25:26.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.348+0000 7f00a2176640 1 -- 172.21.15.62:0/1217565134 learned_addr learned my addr 172.21.15.62:0/1217565134 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.349+0000 7f00a1975640 1 --2- 172.21.15.62:0/1217565134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb41c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.349+0000 7f00a2176640 1 -- 172.21.15.62:0/1217565134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb424370 msgr2=0x557beb41e140 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.349+0000 7f00a2176640 1 --2- 172.21.15.62:0/1217565134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb424370 0x557beb41e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.349+0000 7f00a2176640 1 -- 172.21.15.62:0/1217565134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 msgr2=0x557beb41c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.349+0000 7f00a2176640 1 --2- 172.21.15.62:0/1217565134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb41c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.349+0000 7f00a2176640 1 -- 172.21.15.62:0/1217565134 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557beb41f430 con 0x557beb4195d0 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.349+0000 7f00a2176640 1 --2- 172.21.15.62:0/1217565134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb41c120 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00940093c0 tx=0x7f0094009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a035cede586c17b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.349+0000 7f0087fff640 1 -- 172.21.15.62:0/1217565134 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094017630 con 0x557beb4195d0 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.350+0000 7f0087fff640 1 -- 172.21.15.62:0/1217565134 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0094017cb0 con 0x557beb4195d0 2024-09-15T23:25:26.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.350+0000 7f0087fff640 1 -- 172.21.15.62:0/1217565134 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009401fd80 con 0x557beb4195d0 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.351+0000 7f00a67b3740 1 -- 172.21.15.62:0/1217565134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 msgr2=0x557beb41c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.351+0000 7f00a67b3740 1 --2- 172.21.15.62:0/1217565134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb41c120 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00940093c0 tx=0x7f0094009880 comp rx=0 tx=0).stop 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.351+0000 7f00a67b3740 1 -- 172.21.15.62:0/1217565134 shutdown_connections 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.351+0000 7f00a67b3740 1 --2- 172.21.15.62:0/1217565134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb424370 0x557beb41e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.351+0000 7f00a67b3740 1 --2- 172.21.15.62:0/1217565134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb41ab40 0x557beb41c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.351+0000 7f00a67b3740 1 --2- 172.21.15.62:0/1217565134 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb41c120 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.351+0000 7f00a67b3740 1 -- 172.21.15.62:0/1217565134 >> 172.21.15.62:0/1217565134 conn(0x557beb3e2310 msgr2=0x557beb41ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.351+0000 7f00a67b3740 1 -- 172.21.15.62:0/1217565134 shutdown_connections 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.351+0000 7f00a67b3740 1 -- 172.21.15.62:0/1217565134 wait complete. 2024-09-15T23:25:26.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a67b3740 1 -- start start 2024-09-15T23:25:26.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a67b3740 1 Processor -- start 2024-09-15T23:25:26.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb439230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb41ab40 0x557beb439770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a67b3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 0x557beb49cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a67b3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f008000e270 con 0x557beb424370 2024-09-15T23:25:26.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a67b3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f008000f6e0 con 0x557beb41ab40 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a67b3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f008000e940 con 0x557beb4195d0 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a67b3740 10 client.0 _ll_register_callbacks cb 0x7f008c05cc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a1975640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb41ab40 0x557beb439770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a1975640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb41ab40 0x557beb439770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50608/0 (socket says 172.21.15.62:50608) 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a1975640 1 -- 172.21.15.62:0/2223566789 learned_addr learned my addr 172.21.15.62:0/2223566789 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.352+0000 7f00a2977640 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 0x557beb49cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f00a2176640 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb439230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f00a1975640 1 -- 172.21.15.62:0/2223566789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 msgr2=0x557beb439230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f00a1975640 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb439230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f00a1975640 1 -- 172.21.15.62:0/2223566789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 msgr2=0x557beb49cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f00a1975640 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 0x557beb49cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f00a1975640 1 -- 172.21.15.62:0/2223566789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557beb49d310 con 0x557beb41ab40 2024-09-15T23:25:26.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f00a2176640 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb439230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f00a1975640 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb41ab40 0x557beb439770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f008c005c40 tx=0x7f008c050cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eb2e907acf83e85 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f0085ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f0085ffb640 1 -- 172.21.15.62:0/2223566789 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c081070 con 0x557beb41ab40 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f00a67b3740 1 -- 172.21.15.62:0/2223566789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0094017cb0 con 0x557beb41ab40 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f0085ffb640 1 -- 172.21.15.62:0/2223566789 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f008c042440 con 0x557beb41ab40 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.353+0000 7f0085ffb640 1 -- 172.21.15.62:0/2223566789 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f008c08d050 con 0x557beb41ab40 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f00a67b3740 10 client.27212 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f00a67b3740 1 -- 172.21.15.62:0/2223566789 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557beb49e5b0 con 0x557beb41ab40 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f0085ffb640 1 -- 172.21.15.62:0/2223566789 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0094017cb0 con 0x557beb41ab40 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f0085ffb640 1 -- 172.21.15.62:0/2223566789 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f008c0406c0 con 0x557beb41ab40 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f00a67b3740 10 client.27212 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f00a67b3740 10 client.27212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f00a67b3740 20 client.27212 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:26.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f00a67b3740 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557beb49ebe0 0x557beb4a1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f00a67b3740 4 client.27212 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f00a67b3740 1 -- 172.21.15.62:0/2223566789 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557beb4217b0 con 0x557beb49ebe0 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.354+0000 7f00a2176640 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557beb49ebe0 0x557beb4a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f0085ffb640 10 client.27212 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a2176640 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557beb49ebe0 0x557beb4a1000 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d13875fb5eb5344c server_cookie=c23d6b1e9394ea0c in_seq=0 out_seq=0 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f0085ffb640 1 -- 172.21.15.62:0/2223566789 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557beb4217b0 con 0x557beb49ebe0 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f0085ffb640 10 client.27212 handle_command_reply: tid=0 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a67b3740 1 client.27212 shutdown 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a67b3740 1 -- 172.21.15.62:0/2223566789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb41ab40 msgr2=0x557beb439770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a67b3740 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb41ab40 0x557beb439770 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f008c005c40 tx=0x7f008c050cc0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a67b3740 1 -- 172.21.15.62:0/2223566789 shutdown_connections 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a67b3740 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557beb49ebe0 0x557beb4a1000 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a67b3740 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557beb424370 0x557beb49cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f0085ffb640 0 client.27212 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a67b3740 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557beb41ab40 0x557beb439770 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a67b3740 1 --2- 172.21.15.62:0/2223566789 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557beb4195d0 0x557beb439230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.355+0000 7f00a67b3740 1 -- 172.21.15.62:0/2223566789 >> 172.21.15.62:0/2223566789 conn(0x557beb3e2310 msgr2=0x557beb422740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.356+0000 7f00a67b3740 1 -- 172.21.15.62:0/2223566789 shutdown_connections 2024-09-15T23:25:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.356+0000 7f00a67b3740 1 -- 172.21.15.62:0/2223566789 wait complete. 2024-09-15T23:25:26.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.356+0000 7f00a67b3740 20 client.27212 trim_cache size 0 max 16384 2024-09-15T23:25:26.361 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:25:26.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 -- 172.21.15.62:0/1510357013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00880a3380 msgr2=0x7f00880a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00880a3380 0x7f00880a5840 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f008c05b000 tx=0x7f008c050720 comp rx=0 tx=0).stop 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 -- 172.21.15.62:0/1510357013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009c11b140 msgr2=0x7f009c116df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009c11b140 0x7f009c116df0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f009800d720 tx=0x7f009800dbe0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 -- 172.21.15.62:0/1510357013 shutdown_connections 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00880a3380 0x7f00880a5840 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009c11b140 0x7f009c116df0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f009c109830 0x7f009c1168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 --2- 172.21.15.62:0/1510357013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f009c109450 0x7f009c11a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 -- 172.21.15.62:0/1510357013 >> 172.21.15.62:0/1510357013 conn(0x7f009c06e140 msgr2=0x7f009c0722b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 -- 172.21.15.62:0/1510357013 shutdown_connections 2024-09-15T23:25:26.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:26.357+0000 7f00a43da640 1 -- 172.21.15.62:0/1510357013 wait complete. 2024-09-15T23:25:27.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:27 smithi062 ceph-mon[30949]: pgmap v645: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 4.4 MiB/s wr, 498 op/s 2024-09-15T23:25:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:27 smithi067 ceph-mon[35665]: pgmap v645: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 4.4 MiB/s wr, 498 op/s 2024-09-15T23:25:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:27 smithi148 ceph-mon[35610]: pgmap v645: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 4.4 MiB/s wr, 498 op/s 2024-09-15T23:25:28.067 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:25:28.067 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:25:28.569 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:29.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.205+0000 7feeb825e640 1 -- 172.21.15.62:0/2855364461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb0109b10 msgr2=0x7feeb0114570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:29.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.205+0000 7feeb825e640 1 --2- 172.21.15.62:0/2855364461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb0109b10 0x7feeb0114570 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7feea8009ca0 tx=0x7feea802a970 comp rx=0 tx=0).stop 2024-09-15T23:25:29.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.206+0000 7feeb825e640 1 -- 172.21.15.62:0/2855364461 shutdown_connections 2024-09-15T23:25:29.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.206+0000 7feeb825e640 1 --2- 172.21.15.62:0/2855364461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeb0114ab0 0x7feeb0116ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.206+0000 7feeb825e640 1 --2- 172.21.15.62:0/2855364461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb0109b10 0x7feeb0114570 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.206+0000 7feeb825e640 1 --2- 172.21.15.62:0/2855364461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feeb01091f0 0x7feeb01095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.206+0000 7feeb825e640 1 -- 172.21.15.62:0/2855364461 >> 172.21.15.62:0/2855364461 conn(0x7feeb006e140 msgr2=0x7feeb006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:29.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.206+0000 7feeb825e640 1 -- 172.21.15.62:0/2855364461 shutdown_connections 2024-09-15T23:25:29.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.206+0000 7feeb825e640 1 -- 172.21.15.62:0/2855364461 wait complete. 2024-09-15T23:25:29.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.207+0000 7feeb825e640 1 Processor -- start 2024-09-15T23:25:29.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.208+0000 7feeb825e640 1 -- start start 2024-09-15T23:25:29.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.208+0000 7feeb825e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeb01091f0 0x7feeb01a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.208+0000 7feeb825e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feeb0109b10 0x7feeb01a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.209+0000 7feeb825e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb0114ab0 0x7feeb01a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.209+0000 7feeb825e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feeb011a840 con 0x7feeb01091f0 2024-09-15T23:25:29.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.209+0000 7feeb825e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feeb011a6c0 con 0x7feeb0114ab0 2024-09-15T23:25:29.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.209+0000 7feeb825e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7feeb011a9c0 con 0x7feeb0109b10 2024-09-15T23:25:29.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.210+0000 7feeb57f9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feeb0109b10 0x7feeb01a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.210+0000 7feeb57f9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feeb0109b10 0x7feeb01a1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56882/0 (socket says 172.21.15.62:56882) 2024-09-15T23:25:29.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.210+0000 7feeb57f9640 1 -- 172.21.15.62:0/438242484 learned_addr learned my addr 172.21.15.62:0/438242484 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:29.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.210+0000 7feeb67fb640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb0114ab0 0x7feeb01a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.211+0000 7feeb5ffa640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeb01091f0 0x7feeb01a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.211+0000 7feeb57f9640 1 -- 172.21.15.62:0/438242484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb0114ab0 msgr2=0x7feeb01a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:29.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.211+0000 7feeb57f9640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb0114ab0 0x7feeb01a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.211+0000 7feeb57f9640 1 -- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeb01091f0 msgr2=0x7feeb01a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:29.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.212+0000 7feeb57f9640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeb01091f0 0x7feeb01a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.212+0000 7feeb57f9640 1 -- 172.21.15.62:0/438242484 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb01a5cd0 con 0x7feeb0109b10 2024-09-15T23:25:29.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.212+0000 7feeb5ffa640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeb01091f0 0x7feeb01a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:29.214 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.213+0000 7feeb57f9640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feeb0109b10 0x7feeb01a1260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feea802aee0 tx=0x7feea8003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:29.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.213+0000 7feea6ffd640 1 -- 172.21.15.62:0/438242484 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8034050 con 0x7feeb0109b10 2024-09-15T23:25:29.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.213+0000 7feea6ffd640 1 -- 172.21.15.62:0/438242484 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea8030b80 con 0x7feeb0109b10 2024-09-15T23:25:29.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.213+0000 7feea6ffd640 1 -- 172.21.15.62:0/438242484 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8039420 con 0x7feeb0109b10 2024-09-15T23:25:29.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.214+0000 7feeb825e640 1 -- 172.21.15.62:0/438242484 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb01a5f60 con 0x7feeb0109b10 2024-09-15T23:25:29.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.215+0000 7feeb825e640 1 -- 172.21.15.62:0/438242484 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb01a63c0 con 0x7feeb0109b10 2024-09-15T23:25:29.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.222+0000 7feeba637740 1 Processor -- start 2024-09-15T23:25:29.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.222+0000 7feea6ffd640 1 -- 172.21.15.62:0/438242484 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7feea8037020 con 0x7feeb0109b10 2024-09-15T23:25:29.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.223+0000 7feeba637740 1 -- start start 2024-09-15T23:25:29.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.224+0000 7feea6ffd640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fee9c0a77e0 0x7fee9c0a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.224+0000 7feea6ffd640 1 -- 172.21.15.62:0/438242484 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7feea80f10f0 con 0x7feeb0109b10 2024-09-15T23:25:29.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.224+0000 7feeb67fb640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fee9c0a77e0 0x7fee9c0a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.224+0000 7feeba637740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 0x5617c2751f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.224+0000 7feeba637740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c275c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.224+0000 7feeba637740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.224+0000 7feeba637740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5617c26b7280 con 0x5617c2752480 2024-09-15T23:25:29.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.224+0000 7feeba637740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5617c265eb10 con 0x5617c275cd40 2024-09-15T23:25:29.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.224+0000 7feeba637740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5617c26b3240 con 0x5617c2751b60 2024-09-15T23:25:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb5ffa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 0x5617c2751f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb57f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c275c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb67fb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb57f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c275c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38758/0 (socket says 172.21.15.62:38758) 2024-09-15T23:25:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb57f9640 1 -- 172.21.15.62:0/3326403505 learned_addr learned my addr 172.21.15.62:0/3326403505 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb67fb640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fee9c0a77e0 0x7fee9c0a9ca0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7feeb006b430 tx=0x7feea000be20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:29.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb67fb640 1 -- 172.21.15.62:0/3326403505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 msgr2=0x5617c2751f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb67fb640 1 --2- 172.21.15.62:0/3326403505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 0x5617c2751f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb67fb640 1 -- 172.21.15.62:0/3326403505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 msgr2=0x5617c275c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb67fb640 1 --2- 172.21.15.62:0/3326403505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c275c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb67fb640 1 -- 172.21.15.62:0/3326403505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617c266b1f0 con 0x5617c275cd40 2024-09-15T23:25:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.225+0000 7feeb5ffa640 1 --2- 172.21.15.62:0/3326403505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 0x5617c2751f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.226+0000 7feeb67fb640 1 --2- 172.21.15.62:0/3326403505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275f130 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7feeb00743f0 tx=0x7feea0019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e40b03f38e387af0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.226+0000 7fee9bfff640 1 -- 172.21.15.62:0/3326403505 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0017b60 con 0x5617c275cd40 2024-09-15T23:25:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.226+0000 7fee9bfff640 1 -- 172.21.15.62:0/3326403505 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea0024070 con 0x5617c275cd40 2024-09-15T23:25:29.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.226+0000 7fee9bfff640 1 -- 172.21.15.62:0/3326403505 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0015630 con 0x5617c275cd40 2024-09-15T23:25:29.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.227+0000 7feeba637740 1 -- 172.21.15.62:0/3326403505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 msgr2=0x5617c275f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.227+0000 7feeba637740 1 --2- 172.21.15.62:0/3326403505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275f130 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7feeb00743f0 tx=0x7feea0019000 comp rx=0 tx=0).stop 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.227+0000 7feeba637740 1 -- 172.21.15.62:0/3326403505 shutdown_connections 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.227+0000 7feeba637740 1 --2- 172.21.15.62:0/3326403505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275f130 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.227+0000 7feeba637740 1 --2- 172.21.15.62:0/3326403505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c275c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.227+0000 7feeba637740 1 --2- 172.21.15.62:0/3326403505 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 0x5617c2751f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.227+0000 7feeba637740 1 -- 172.21.15.62:0/3326403505 >> 172.21.15.62:0/3326403505 conn(0x5617c274e0f0 msgr2=0x5617c274e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.227+0000 7feeba637740 1 -- 172.21.15.62:0/3326403505 shutdown_connections 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.227+0000 7feeba637740 1 -- 172.21.15.62:0/3326403505 wait complete. 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeba637740 1 -- start start 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeba637740 1 Processor -- start 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeba637740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 0x5617c2759c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeba637740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c275a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeba637740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeba637740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5617c275f910 con 0x5617c2752480 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeb67fb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeb67fb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50648/0 (socket says 172.21.15.62:50648) 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeb67fb640 1 -- 172.21.15.62:0/1267358101 learned_addr learned my addr 172.21.15.62:0/1267358101 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeb5ffa640 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 0x5617c2759c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.228+0000 7feeba637740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5617c2671f10 con 0x5617c275cd40 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeba637740 1 -- 172.21.15.62:0/1267358101 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5617c275fa90 con 0x5617c2751b60 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeb57f9640 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c275a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeba637740 10 client.0 _ll_register_callbacks cb 0x5617c263aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeb67fb640 1 -- 172.21.15.62:0/1267358101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 msgr2=0x5617c2759c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeb67fb640 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 0x5617c2759c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeb67fb640 1 -- 172.21.15.62:0/1267358101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 msgr2=0x5617c275a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:29.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeb67fb640 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c275a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeb67fb640 1 -- 172.21.15.62:0/1267358101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617c24286a0 con 0x5617c275cd40 2024-09-15T23:25:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeba637740 1 -- 172.21.15.62:0/1267358101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feea0024070 con 0x5617c275cd40 2024-09-15T23:25:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7feeb67fb640 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275a6a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7feea0017e10 tx=0x7feea001fb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fc1c3afb526b7a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7fee997fa640 10 client.27224 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:25:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7fee997fa640 1 -- 172.21.15.62:0/1267358101 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea002d060 con 0x5617c275cd40 2024-09-15T23:25:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.229+0000 7fee997fa640 1 -- 172.21.15.62:0/1267358101 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7feea001f680 con 0x5617c275cd40 2024-09-15T23:25:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7fee997fa640 1 -- 172.21.15.62:0/1267358101 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7feea001f880 con 0x5617c275cd40 2024-09-15T23:25:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7feeba637740 10 client.27224 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:29.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7feeba637740 1 -- 172.21.15.62:0/1267358101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=35}) -- 0x5617c276b480 con 0x5617c275cd40 2024-09-15T23:25:29.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7fee997fa640 1 -- 172.21.15.62:0/1267358101 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7feea0030050 con 0x5617c275cd40 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7feeba637740 10 client.27224 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7feeba637740 10 client.27224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7feeba637740 20 client.27224 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7feeba637740 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617c276c7d0 0x5617c276ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7feeba637740 4 client.27224 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7feeba637740 1 -- 172.21.15.62:0/1267358101 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617c26c47f0 con 0x5617c276c7d0 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.230+0000 7feeb5ffa640 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617c276c7d0 0x5617c276ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.231+0000 7fee997fa640 10 client.27224 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.231+0000 7feeb5ffa640 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617c276c7d0 0x5617c276ebc0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=488e262bb6a09c18 server_cookie=e330f58739cec933 in_seq=0 out_seq=0 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7fee997fa640 1 -- 172.21.15.62:0/1267358101 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5617c26c47f0 con 0x5617c276c7d0 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7fee997fa640 10 client.27224 handle_command_reply: tid=0 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 client.27224 shutdown 2024-09-15T23:25:29.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 -- 172.21.15.62:0/1267358101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 msgr2=0x5617c275a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275a6a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7feea0017e10 tx=0x7feea001fb80 comp rx=0 tx=0).stop 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 -- 172.21.15.62:0/1267358101 shutdown_connections 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617c276c7d0 0x5617c276ebc0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c275cd40 0x5617c275a6a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7fee997fa640 0 client.27224 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c275a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 --2- 172.21.15.62:0/1267358101 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c2751b60 0x5617c2759c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 -- 172.21.15.62:0/1267358101 >> 172.21.15.62:0/1267358101 conn(0x5617c274e0f0 msgr2=0x5617c2753e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.232+0000 7feeba637740 1 -- 172.21.15.62:0/1267358101 shutdown_connections 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.233+0000 7feeba637740 1 -- 172.21.15.62:0/1267358101 wait complete. 2024-09-15T23:25:29.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.233+0000 7feeba637740 20 client.27224 trim_cache size 0 max 16384 2024-09-15T23:25:29.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.255+0000 7feeba637740 1 Processor -- start 2024-09-15T23:25:29.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.255+0000 7feeba637740 1 -- start start 2024-09-15T23:25:29.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.255+0000 7feeba637740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c275a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.255+0000 7feeba637740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c2758ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeba637740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 0x5617c2771be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeba637740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feeb01b7bc0 con 0x5617c2752480 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb5ffa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c275a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb5ffa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c275a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50650/0 (socket says 172.21.15.62:50650) 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb5ffa640 1 -- 172.21.15.62:0/1426178161 learned_addr learned my addr 172.21.15.62:0/1426178161 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeba637740 1 -- 172.21.15.62:0/1426178161 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feeb01b5490 con 0x5617c2751b60 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeba637740 1 -- 172.21.15.62:0/1426178161 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7feeb01b32e0 con 0x5617c275cd40 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb67fb640 1 --2- 172.21.15.62:0/1426178161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 0x5617c2771be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb57f9640 1 --2- 172.21.15.62:0/1426178161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c2758ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb67fb640 1 -- 172.21.15.62:0/1426178161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 msgr2=0x5617c275a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb67fb640 1 --2- 172.21.15.62:0/1426178161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c275a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb67fb640 1 -- 172.21.15.62:0/1426178161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 msgr2=0x5617c2758ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb67fb640 1 --2- 172.21.15.62:0/1426178161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c2758ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.256+0000 7feeb67fb640 1 -- 172.21.15.62:0/1426178161 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617c274df20 con 0x5617c275cd40 2024-09-15T23:25:29.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.257+0000 7feeb67fb640 1 --2- 172.21.15.62:0/1426178161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 0x5617c2771be0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feea0017e40 tx=0x7feea001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=170dcdf9ba64d191 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:29.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.257+0000 7fee9b7fe640 1 -- 172.21.15.62:0/1426178161 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea001fac0 con 0x5617c275cd40 2024-09-15T23:25:29.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.257+0000 7fee9b7fe640 1 -- 172.21.15.62:0/1426178161 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea0024070 con 0x5617c275cd40 2024-09-15T23:25:29.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.257+0000 7fee9b7fe640 1 -- 172.21.15.62:0/1426178161 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0006750 con 0x5617c275cd40 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 -- 172.21.15.62:0/1426178161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 msgr2=0x5617c2771be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 --2- 172.21.15.62:0/1426178161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 0x5617c2771be0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feea0017e40 tx=0x7feea001a000 comp rx=0 tx=0).stop 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 -- 172.21.15.62:0/1426178161 shutdown_connections 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 --2- 172.21.15.62:0/1426178161 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 0x5617c2771be0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 --2- 172.21.15.62:0/1426178161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c2758ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 --2- 172.21.15.62:0/1426178161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c275a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 -- 172.21.15.62:0/1426178161 >> 172.21.15.62:0/1426178161 conn(0x5617c274e0f0 msgr2=0x5617c27583c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 -- 172.21.15.62:0/1426178161 shutdown_connections 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 -- 172.21.15.62:0/1426178161 wait complete. 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.258+0000 7feeba637740 1 -- start start 2024-09-15T23:25:29.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.259+0000 7feeba637740 1 Processor -- start 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.259+0000 7feeba637740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c276fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.259+0000 7feeba637740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c2770040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.259+0000 7feeba637740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 0x5617c2770580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.259+0000 7feeba637740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feeb01b3950 con 0x5617c2752480 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.259+0000 7feeba637740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feeb01b4dc0 con 0x5617c2751b60 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.259+0000 7feeba637740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7feeb01b4020 con 0x5617c275cd40 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb5ffa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c276fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeba637740 10 client.0 _ll_register_callbacks cb 0x7feea80b8f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb5ffa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c276fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50658/0 (socket says 172.21.15.62:50658) 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb5ffa640 1 -- 172.21.15.62:0/3255252653 learned_addr learned my addr 172.21.15.62:0/3255252653 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb57f9640 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c2770040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb67fb640 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 0x5617c2770580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb5ffa640 1 -- 172.21.15.62:0/3255252653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 msgr2=0x5617c2770580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:29.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb5ffa640 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 0x5617c2770580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb5ffa640 1 -- 172.21.15.62:0/3255252653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 msgr2=0x5617c2770040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb5ffa640 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c2770040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb5ffa640 1 -- 172.21.15.62:0/3255252653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617c2749830 con 0x5617c2751b60 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeb5ffa640 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c276fb00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7feeac009b50 tx=0x7feeac009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a848d921ec3511d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7fee997fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7fee997fa640 1 -- 172.21.15.62:0/3255252653 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeac01e420 con 0x5617c2751b60 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7fee997fa640 1 -- 172.21.15.62:0/3255252653 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7feeac01e5c0 con 0x5617c2751b60 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.260+0000 7feeba637740 1 -- 172.21.15.62:0/3255252653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feea0024070 con 0x5617c2751b60 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.261+0000 7fee997fa640 1 -- 172.21.15.62:0/3255252653 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7feeac02e050 con 0x5617c2751b60 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.261+0000 7feeba637740 10 client.27230 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.261+0000 7feeba637740 1 -- 172.21.15.62:0/3255252653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5617c27d7460 con 0x5617c2751b60 2024-09-15T23:25:29.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.261+0000 7fee997fa640 1 -- 172.21.15.62:0/3255252653 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feea0024070 con 0x5617c2751b60 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.261+0000 7fee997fa640 1 -- 172.21.15.62:0/3255252653 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7feeac00f040 con 0x5617c2751b60 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.261+0000 7feeba637740 10 client.27230 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.261+0000 7feeba637740 10 client.27230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.261+0000 7feeba637740 20 client.27230 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.262+0000 7feeba637740 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617c27d7b60 0x5617c27d9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.262+0000 7feeba637740 4 client.27230 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.262+0000 7feeba637740 1 -- 172.21.15.62:0/3255252653 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5617c26bed00 con 0x5617c27d7b60 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.262+0000 7feeb57f9640 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617c27d7b60 0x5617c27d9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.262+0000 7fee997fa640 10 client.27230 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:29.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.262+0000 7feeb57f9640 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617c27d7b60 0x5617c27d9f80 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=982c4138a22d6d1d server_cookie=9c94c50863d6ba1d in_seq=0 out_seq=0 2024-09-15T23:25:29.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.263+0000 7fee997fa640 1 -- 172.21.15.62:0/3255252653 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5617c26bed00 con 0x5617c27d7b60 2024-09-15T23:25:29.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.263+0000 7fee997fa640 10 client.27230 handle_command_reply: tid=0 2024-09-15T23:25:29.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.263+0000 7feeba637740 1 client.27230 shutdown 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.263+0000 7feeba637740 1 -- 172.21.15.62:0/3255252653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 msgr2=0x5617c276fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c276fb00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7feeac009b50 tx=0x7feeac009b80 comp rx=0 tx=0).stop 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 1 -- 172.21.15.62:0/3255252653 shutdown_connections 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617c27d7b60 0x5617c27d9f80 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7fee997fa640 0 client.27230 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617c275cd40 0x5617c2770580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617c2752480 0x5617c2770040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 1 --2- 172.21.15.62:0/3255252653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617c2751b60 0x5617c276fb00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 1 -- 172.21.15.62:0/3255252653 >> 172.21.15.62:0/3255252653 conn(0x5617c274e0f0 msgr2=0x5617c274d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 1 -- 172.21.15.62:0/3255252653 shutdown_connections 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 1 -- 172.21.15.62:0/3255252653 wait complete. 2024-09-15T23:25:29.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.264+0000 7feeba637740 20 client.27230 trim_cache size 0 max 16384 2024-09-15T23:25:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:25:29.270 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:25:29.270 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "120fbc1e-bf86-4d79-8d58-5676d41052c7", 2024-09-15T23:25:29.270 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:25:29.270 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:25:29.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 -- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fee9c0a77e0 msgr2=0x7fee9c0a9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:29.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fee9c0a77e0 0x7fee9c0a9ca0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7feeb006b430 tx=0x7feea000be20 comp rx=0 tx=0).stop 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 -- 172.21.15.62:0/438242484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feeb0109b10 msgr2=0x7feeb01a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feeb0109b10 0x7feeb01a1260 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7feea802aee0 tx=0x7feea8003f10 comp rx=0 tx=0).stop 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 -- 172.21.15.62:0/438242484 shutdown_connections 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fee9c0a77e0 0x7fee9c0a9ca0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb0114ab0 0x7feeb01a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feeb0109b10 0x7feeb01a1260 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 --2- 172.21.15.62:0/438242484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feeb01091f0 0x7feeb01a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 -- 172.21.15.62:0/438242484 >> 172.21.15.62:0/438242484 conn(0x7feeb006e140 msgr2=0x7feeb0070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 -- 172.21.15.62:0/438242484 shutdown_connections 2024-09-15T23:25:29.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:29.265+0000 7feeb825e640 1 -- 172.21.15.62:0/438242484 wait complete. 2024-09-15T23:25:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:29 smithi062 ceph-mon[30949]: pgmap v646: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 4.5 MiB/s wr, 520 op/s 2024-09-15T23:25:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:29 smithi067 ceph-mon[35665]: pgmap v646: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 4.5 MiB/s wr, 520 op/s 2024-09-15T23:25:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:29 smithi148 ceph-mon[35610]: pgmap v646: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 4.5 MiB/s wr, 520 op/s 2024-09-15T23:25:29.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 550.0 s, 625.0 tps, lat 162.223 ms stddev 186.061 2024-09-15T23:25:30.322 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:25:30.367 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:30 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:25:30.367 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:30 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.367 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:30 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:25:30.367 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:30 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:25:30.368 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:30 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.368 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:30 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:25:30.368 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:30 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:25:30.368 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:30 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.368 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:30 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:30 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:25:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:30 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:30 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:25:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:30 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:25:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:30 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:30 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:25:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:30 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:25:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:30 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:30 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:30 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:25:30.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:30 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:30 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:25:30.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:30 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:25:30.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:30 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:30 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:25:30.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:30 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:25:30.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:30 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:30 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:30.792 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:25:31.213 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:31 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:31.213 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:31 smithi062 ceph-mon[30949]: pgmap v647: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 4.8 MiB/s wr, 562 op/s 2024-09-15T23:25:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:31 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:31 smithi067 ceph-mon[35665]: pgmap v647: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 4.8 MiB/s wr, 562 op/s 2024-09-15T23:25:31.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.537+0000 7f8915254640 1 -- 172.21.15.62:0/566115784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8910100a80 msgr2=0x7f8910100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.537+0000 7f8915254640 1 --2- 172.21.15.62:0/566115784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8910100a80 0x7f8910100ee0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009930 tx=0x7f88f802a420 comp rx=0 tx=0).stop 2024-09-15T23:25:31.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.538+0000 7f8915254640 1 -- 172.21.15.62:0/566115784 shutdown_connections 2024-09-15T23:25:31.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.538+0000 7f8915254640 1 --2- 172.21.15.62:0/566115784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910101420 0x7f891010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.538+0000 7f8915254640 1 --2- 172.21.15.62:0/566115784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8910100a80 0x7f8910100ee0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.538+0000 7f8915254640 1 --2- 172.21.15.62:0/566115784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8910106a80 0x7f8910106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.538+0000 7f8915254640 1 -- 172.21.15.62:0/566115784 >> 172.21.15.62:0/566115784 conn(0x7f89100fc7a0 msgr2=0x7f89100febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.538+0000 7f8915254640 1 -- 172.21.15.62:0/566115784 shutdown_connections 2024-09-15T23:25:31.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.538+0000 7f8915254640 1 -- 172.21.15.62:0/566115784 wait complete. 2024-09-15T23:25:31.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:31 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:25:31.541 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:31 smithi148 ceph-mon[35610]: pgmap v647: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 4.8 MiB/s wr, 562 op/s 2024-09-15T23:25:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.539+0000 7f8915254640 1 Processor -- start 2024-09-15T23:25:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.539+0000 7f8915254640 1 -- start start 2024-09-15T23:25:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.540+0000 7f8915254640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8910100a80 0x7f89101971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.540+0000 7f8915254640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910101420 0x7f8910197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.540+0000 7f8915254640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8910106a80 0x7f891019ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.540+0000 7f8907fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910101420 0x7f8910197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.540+0000 7f8907fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910101420 0x7f8910197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50676/0 (socket says 172.21.15.62:50676) 2024-09-15T23:25:31.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.540+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8910100a80 0x7f89101971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.540+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8910100a80 0x7f89101971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38802/0 (socket says 172.21.15.62:38802) 2024-09-15T23:25:31.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f8907fff640 1 -- 172.21.15.62:0/3341527181 learned_addr learned my addr 172.21.15.62:0/3341527181 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:31.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8910110e40 con 0x7f8910100a80 2024-09-15T23:25:31.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8910110cc0 con 0x7f8910101420 2024-09-15T23:25:31.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8910110fc0 con 0x7f8910106a80 2024-09-15T23:25:31.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f890f59e640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8910106a80 0x7f891019ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f8907fff640 1 -- 172.21.15.62:0/3341527181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8910106a80 msgr2=0x7f891019ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f8907fff640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8910106a80 0x7f891019ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f8907fff640 1 -- 172.21.15.62:0/3341527181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8910100a80 msgr2=0x7f89101971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f8907fff640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8910100a80 0x7f89101971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f8907fff640 1 -- 172.21.15.62:0/3341527181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891019c170 con 0x7f8910101420 2024-09-15T23:25:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.541+0000 7f890f59e640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8910106a80 0x7f891019ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.542+0000 7f8907fff640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910101420 0x7f8910197700 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009900 tx=0x7f88f8030650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.542+0000 7f890cd99640 1 -- 172.21.15.62:0/3341527181 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8043070 con 0x7f8910101420 2024-09-15T23:25:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.542+0000 7f890cd99640 1 -- 172.21.15.62:0/3341527181 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88f803fe50 con 0x7f8910101420 2024-09-15T23:25:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.542+0000 7f890cd99640 1 -- 172.21.15.62:0/3341527181 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8035030 con 0x7f8910101420 2024-09-15T23:25:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.542+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f891019c400 con 0x7f8910101420 2024-09-15T23:25:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.542+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f891006a790 con 0x7f8910101420 2024-09-15T23:25:31.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.545+0000 7f890cd99640 1 -- 172.21.15.62:0/3341527181 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f88f8046070 con 0x7f8910101420 2024-09-15T23:25:31.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f890cd99640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88fc0a32c0 0x7f88fc0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f891762d740 1 Processor -- start 2024-09-15T23:25:31.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f891762d740 1 -- start start 2024-09-15T23:25:31.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f890ed9d640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88fc0a32c0 0x7f88fc0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f890cd99640 1 -- 172.21.15.62:0/3341527181 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f88f80f4ac0 con 0x7f8910101420 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f891762d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 0x5607beb207a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f891762d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb33950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f891762d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 0x5607beb36280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f891762d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5607bea862e0 con 0x5607beb33e90 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f891762d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5607bea2dce0 con 0x5607beb203c0 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.546+0000 7f891762d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5607bea822a0 con 0x5607beb20ce0 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f8907fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb33950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 0x5607beb207a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f8907fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb33950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:56946/0 (socket says 172.21.15.62:56946) 2024-09-15T23:25:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f8907fff640 1 -- 172.21.15.62:0/1233790031 learned_addr learned my addr 172.21.15.62:0/1233790031 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890f59e640 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 0x5607beb36280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890f59e640 1 -- 172.21.15.62:0/1233790031 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 msgr2=0x5607beb33950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890f59e640 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb33950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890f59e640 1 -- 172.21.15.62:0/1233790031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 msgr2=0x5607beb207a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890f59e640 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 0x5607beb207a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890f59e640 1 -- 172.21.15.62:0/1233790031 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607bea12440 con 0x5607beb33e90 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f8907fff640 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb33950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890ed9d640 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 0x5607beb207a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890ed9d640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88fc0a32c0 0x7f88fc0a5780 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8900005fd0 tx=0x7f8900013b80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f890f59e640 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 0x5607beb36280 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8908009ad0 tx=0x7f8908009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b5bf544572fdee server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.547+0000 7f89057fa640 1 -- 172.21.15.62:0/1233790031 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908010d30 con 0x5607beb33e90 2024-09-15T23:25:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.548+0000 7f89057fa640 1 -- 172.21.15.62:0/1233790031 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8908005500 con 0x5607beb33e90 2024-09-15T23:25:31.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.548+0000 7f89057fa640 1 -- 172.21.15.62:0/1233790031 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8908022650 con 0x5607beb33e90 2024-09-15T23:25:31.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 -- 172.21.15.62:0/1233790031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 msgr2=0x5607beb36280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 0x5607beb36280 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8908009ad0 tx=0x7f8908009f90 comp rx=0 tx=0).stop 2024-09-15T23:25:31.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 -- 172.21.15.62:0/1233790031 shutdown_connections 2024-09-15T23:25:31.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 0x5607beb36280 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb33950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 --2- 172.21.15.62:0/1233790031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 0x5607beb207a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 -- 172.21.15.62:0/1233790031 >> 172.21.15.62:0/1233790031 conn(0x5607beb1ca30 msgr2=0x5607beb1ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 -- 172.21.15.62:0/1233790031 shutdown_connections 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 -- 172.21.15.62:0/1233790031 wait complete. 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 -- start start 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.549+0000 7f891762d740 1 Processor -- start 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f891762d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb203c0 0x5607beb2ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f891762d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb2b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f891762d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb33e90 0x5607beb2b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f891762d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5607beb23320 con 0x5607beb203c0 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f891762d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5607bea41050 con 0x5607beb33e90 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f891762d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5607beb234a0 con 0x5607beb20ce0 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f891762d740 10 client.0 _ll_register_callbacks cb 0x5607bea81160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb203c0 0x5607beb2ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb33e90 0x5607beb2b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb203c0 0x5607beb2ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38814/0 (socket says 172.21.15.62:38814) 2024-09-15T23:25:31.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890ed9d640 1 -- 172.21.15.62:0/823698268 learned_addr learned my addr 172.21.15.62:0/823698268 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f8907fff640 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb2b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890f59e640 1 -- 172.21.15.62:0/823698268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 msgr2=0x5607beb2b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890f59e640 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb2b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890f59e640 1 -- 172.21.15.62:0/823698268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb203c0 msgr2=0x5607beb2ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890f59e640 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb203c0 0x5607beb2ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890f59e640 1 -- 172.21.15.62:0/823698268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607be7f73c0 con 0x5607beb33e90 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890ed9d640 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb203c0 0x5607beb2ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f890f59e640 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb33e90 0x5607beb2b910 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8908009ad0 tx=0x7f8908010de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbe181e674abf229 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.550+0000 7f88f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f88f67fc640 1 -- 172.21.15.62:0/823698268 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89080050d0 con 0x5607beb33e90 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f891762d740 1 -- 172.21.15.62:0/823698268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8908005500 con 0x5607beb33e90 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f88f67fc640 1 -- 172.21.15.62:0/823698268 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8908005270 con 0x5607beb33e90 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f88f67fc640 1 -- 172.21.15.62:0/823698268 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8908040050 con 0x5607beb33e90 2024-09-15T23:25:31.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f891762d740 10 client.27242 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f891762d740 1 -- 172.21.15.62:0/823698268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5607beb39c10 con 0x5607beb33e90 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f88f67fc640 1 -- 172.21.15.62:0/823698268 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8908005500 con 0x5607beb33e90 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f88f67fc640 1 -- 172.21.15.62:0/823698268 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f890800f040 con 0x5607beb33e90 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f891762d740 10 client.27242 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f891762d740 10 client.27242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f891762d740 20 client.27242 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f891762d740 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607beb3b3f0 0x5607beb3d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f891762d740 4 client.27242 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.551+0000 7f891762d740 1 -- 172.21.15.62:0/823698268 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607bea93850 con 0x5607beb3b3f0 2024-09-15T23:25:31.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.552+0000 7f890ed9d640 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607beb3b3f0 0x5607beb3d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.552+0000 7f890ed9d640 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607beb3b3f0 0x5607beb3d820 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcae664dc8c64b47 server_cookie=c0635b12f45620ad in_seq=0 out_seq=0 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.552+0000 7f88f67fc640 10 client.27242 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f88f67fc640 1 -- 172.21.15.62:0/823698268 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5607bea93850 con 0x5607beb3b3f0 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f88f67fc640 10 client.27242 handle_command_reply: tid=0 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 client.27242 shutdown 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 -- 172.21.15.62:0/823698268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb33e90 msgr2=0x5607beb2b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb33e90 0x5607beb2b910 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8908009ad0 tx=0x7f8908010de0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 -- 172.21.15.62:0/823698268 shutdown_connections 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607beb3b3f0 0x5607beb3d820 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb33e90 0x5607beb2b910 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb20ce0 0x5607beb2b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 --2- 172.21.15.62:0/823698268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb203c0 0x5607beb2ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 -- 172.21.15.62:0/823698268 >> 172.21.15.62:0/823698268 conn(0x5607beb1ca30 msgr2=0x5607beb1bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f88f67fc640 0 client.27242 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:31.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 -- 172.21.15.62:0/823698268 shutdown_connections 2024-09-15T23:25:31.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 1 -- 172.21.15.62:0/823698268 wait complete. 2024-09-15T23:25:31.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.553+0000 7f891762d740 20 client.27242 trim_cache size 0 max 16384 2024-09-15T23:25:31.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.574+0000 7f891762d740 1 Processor -- start 2024-09-15T23:25:31.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.574+0000 7f891762d740 1 -- start start 2024-09-15T23:25:31.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f891762d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 0x5607beb3d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f891762d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb20ce0 0x5607beb2b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f891762d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb33e90 0x5607beb2ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f891762d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f89101a6c10 con 0x5607beb20ce0 2024-09-15T23:25:31.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f891762d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89101a44e0 con 0x5607beb203c0 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f891762d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8910072a10 con 0x5607beb33e90 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f8907fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb20ce0 0x5607beb2b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f8907fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb20ce0 0x5607beb2b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38828/0 (socket says 172.21.15.62:38828) 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f8907fff640 1 -- 172.21.15.62:0/1824396878 learned_addr learned my addr 172.21.15.62:0/1824396878 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f890ed9d640 1 --2- 172.21.15.62:0/1824396878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 0x5607beb3d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f890f59e640 1 --2- 172.21.15.62:0/1824396878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb33e90 0x5607beb2ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f8907fff640 1 -- 172.21.15.62:0/1824396878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb33e90 msgr2=0x5607beb2ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f8907fff640 1 --2- 172.21.15.62:0/1824396878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb33e90 0x5607beb2ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f8907fff640 1 -- 172.21.15.62:0/1824396878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 msgr2=0x5607beb3d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f8907fff640 1 --2- 172.21.15.62:0/1824396878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 0x5607beb3d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.575+0000 7f8907fff640 1 -- 172.21.15.62:0/1824396878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607beb1c860 con 0x5607beb20ce0 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.576+0000 7f8907fff640 1 --2- 172.21.15.62:0/1824396878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb20ce0 0x5607beb2b760 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f88f8005ec0 tx=0x7f88f8030c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc686d9026440fd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.576+0000 7f8904ff9640 1 -- 172.21.15.62:0/1824396878 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f800b0b0 con 0x5607beb20ce0 2024-09-15T23:25:31.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.576+0000 7f8904ff9640 1 -- 172.21.15.62:0/1824396878 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88f8004460 con 0x5607beb20ce0 2024-09-15T23:25:31.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.576+0000 7f8904ff9640 1 -- 172.21.15.62:0/1824396878 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f80bd370 con 0x5607beb20ce0 2024-09-15T23:25:31.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.577+0000 7f891762d740 1 -- 172.21.15.62:0/1824396878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb20ce0 msgr2=0x5607beb2b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.577+0000 7f891762d740 1 --2- 172.21.15.62:0/1824396878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb20ce0 0x5607beb2b760 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f88f8005ec0 tx=0x7f88f8030c10 comp rx=0 tx=0).stop 2024-09-15T23:25:31.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.579+0000 7f891762d740 1 -- 172.21.15.62:0/1824396878 shutdown_connections 2024-09-15T23:25:31.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.579+0000 7f891762d740 1 --2- 172.21.15.62:0/1824396878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb33e90 0x5607beb2ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.579+0000 7f891762d740 1 --2- 172.21.15.62:0/1824396878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb20ce0 0x5607beb2b760 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.579+0000 7f891762d740 1 --2- 172.21.15.62:0/1824396878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb203c0 0x5607beb3d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.579+0000 7f891762d740 1 -- 172.21.15.62:0/1824396878 >> 172.21.15.62:0/1824396878 conn(0x5607beb1ca30 msgr2=0x5607beb2e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.579+0000 7f891762d740 1 -- 172.21.15.62:0/1824396878 shutdown_connections 2024-09-15T23:25:31.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.579+0000 7f891762d740 1 -- 172.21.15.62:0/1824396878 wait complete. 2024-09-15T23:25:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.580+0000 7f891762d740 1 -- start start 2024-09-15T23:25:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.580+0000 7f891762d740 1 Processor -- start 2024-09-15T23:25:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.580+0000 7f891762d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb203c0 0x5607beb49de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.580+0000 7f891762d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb20ce0 0x5607beb4a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.580+0000 7f891762d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 0x5607beb4a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.580+0000 7f891762d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8910073080 con 0x5607beb33e90 2024-09-15T23:25:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.580+0000 7f891762d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89101a3e10 con 0x5607beb20ce0 2024-09-15T23:25:31.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.580+0000 7f891762d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8910073750 con 0x5607beb203c0 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f891762d740 10 client.0 _ll_register_callbacks cb 0x7f8900013dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f8907fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb20ce0 0x5607beb4a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f890ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb203c0 0x5607beb49de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f8907fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb20ce0 0x5607beb4a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50700/0 (socket says 172.21.15.62:50700) 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f8907fff640 1 -- 172.21.15.62:0/551365365 learned_addr learned my addr 172.21.15.62:0/551365365 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f890f59e640 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 0x5607beb4a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f8907fff640 1 -- 172.21.15.62:0/551365365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb203c0 msgr2=0x5607beb49de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f8907fff640 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb203c0 0x5607beb49de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f8907fff640 1 -- 172.21.15.62:0/551365365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 msgr2=0x5607beb4a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f8907fff640 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 0x5607beb4a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f8907fff640 1 -- 172.21.15.62:0/551365365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607beba3f60 con 0x5607beb20ce0 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f890ed9d640 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb203c0 0x5607beb49de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f8907fff640 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb20ce0 0x5607beb4a320 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f88f80c9440 tx=0x7f88f8030a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70d3097c1adcedd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:31.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f88f67fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:25:31.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f88f67fc640 1 -- 172.21.15.62:0/551365365 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f80c96e0 con 0x5607beb20ce0 2024-09-15T23:25:31.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f88f67fc640 1 -- 172.21.15.62:0/551365365 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f88f8031b70 con 0x5607beb20ce0 2024-09-15T23:25:31.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.581+0000 7f891762d740 1 -- 172.21.15.62:0/551365365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5607bea8dd60 con 0x5607beb20ce0 2024-09-15T23:25:31.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.582+0000 7f88f67fc640 1 -- 172.21.15.62:0/551365365 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f88f800b0b0 con 0x5607beb20ce0 2024-09-15T23:25:31.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.582+0000 7f891762d740 10 client.27248 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:31.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.582+0000 7f891762d740 1 -- 172.21.15.62:0/551365365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5607beba5250 con 0x5607beb20ce0 2024-09-15T23:25:31.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.582+0000 7f88f67fc640 1 -- 172.21.15.62:0/551365365 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5607bea8dd60 con 0x5607beb20ce0 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.582+0000 7f88f67fc640 1 -- 172.21.15.62:0/551365365 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f88f80c0150 con 0x5607beb20ce0 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 10 client.27248 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 10 client.27248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 20 client.27248 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607beba57a0 0x5607beba7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 4 client.27248 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 -- 172.21.15.62:0/551365365 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5607bea0a210 con 0x5607beba57a0 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f890ed9d640 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607beba57a0 0x5607beba7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f890ed9d640 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607beba57a0 0x5607beba7bc0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59b582c1d06274b2 server_cookie=54971713447d0381 in_seq=0 out_seq=0 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f88f67fc640 10 client.27248 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f88f67fc640 1 -- 172.21.15.62:0/551365365 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5607bea0a210 con 0x5607beba57a0 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f88f67fc640 10 client.27248 handle_command_reply: tid=0 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 client.27248 shutdown 2024-09-15T23:25:31.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 -- 172.21.15.62:0/551365365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb20ce0 msgr2=0x5607beb4a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb20ce0 0x5607beb4a320 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f88f80c9440 tx=0x7f88f8030a70 comp rx=0 tx=0).stop 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 -- 172.21.15.62:0/551365365 shutdown_connections 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607beba57a0 0x5607beba7bc0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607beb33e90 0x5607beb4a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607beb20ce0 0x5607beb4a320 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f88f67fc640 0 client.27248 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 --2- 172.21.15.62:0/551365365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607beb203c0 0x5607beb49de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.583+0000 7f891762d740 1 -- 172.21.15.62:0/551365365 >> 172.21.15.62:0/551365365 conn(0x5607beb1ca30 msgr2=0x5607beb2ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.584+0000 7f891762d740 1 -- 172.21.15.62:0/551365365 shutdown_connections 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.584+0000 7f891762d740 1 -- 172.21.15.62:0/551365365 wait complete. 2024-09-15T23:25:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.584+0000 7f891762d740 20 client.27248 trim_cache size 0 max 16384 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout: "120fbc1e-bf86-4d79-8d58-5676d41052c7": { 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "120fbc1e-bf86-4d79-8d58-5676d41052c7", 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:25:31.589 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:25:31.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.584+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88fc0a32c0 msgr2=0x7f88fc0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.584+0000 7f8915254640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88fc0a32c0 0x7f88fc0a5780 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8900005fd0 tx=0x7f8900013b80 comp rx=0 tx=0).stop 2024-09-15T23:25:31.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.584+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910101420 msgr2=0x7f8910197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:31.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.584+0000 7f8915254640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910101420 0x7f8910197700 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009900 tx=0x7f88f8030650 comp rx=0 tx=0).stop 2024-09-15T23:25:31.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.585+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 shutdown_connections 2024-09-15T23:25:31.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.585+0000 7f8915254640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88fc0a32c0 0x7f88fc0a5780 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.585+0000 7f8915254640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8910106a80 0x7f891019ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.585+0000 7f8915254640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8910101420 0x7f8910197700 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.585+0000 7f8915254640 1 --2- 172.21.15.62:0/3341527181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8910100a80 0x7f89101971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.585+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 >> 172.21.15.62:0/3341527181 conn(0x7f89100fc7a0 msgr2=0x7f89100fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.585+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 shutdown_connections 2024-09-15T23:25:31.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:25:31.585+0000 7f8915254640 1 -- 172.21.15.62:0/3341527181 wait complete. 2024-09-15T23:25:32.639 INFO:tasks.cephfs.filesystem:scrub status for tag:120fbc1e-bf86-4d79-8d58-5676d41052c7 - {'path': '/', 'tag': '120fbc1e-bf86-4d79-8d58-5676d41052c7', 'options': 'recursive,force'} 2024-09-15T23:25:33.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:33 smithi062 ceph-mon[30949]: pgmap v648: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 5.2 MiB/s wr, 640 op/s 2024-09-15T23:25:33.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:33 smithi067 ceph-mon[35665]: pgmap v648: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 5.2 MiB/s wr, 640 op/s 2024-09-15T23:25:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:33 smithi148 ceph-mon[35610]: pgmap v648: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 5.2 MiB/s wr, 640 op/s 2024-09-15T23:25:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.529+0000 7f78edc9b640 1 -- 172.21.15.62:0/3419131323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e8107830 msgr2=0x7f78e8109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.529+0000 7f78edc9b640 1 --2- 172.21.15.62:0/3419131323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e8107830 0x7f78e8109ca0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f78e0009ca0 tx=0x7f78e002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.529+0000 7f78edc9b640 1 -- 172.21.15.62:0/3419131323 shutdown_connections 2024-09-15T23:25:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.529+0000 7f78edc9b640 1 --2- 172.21.15.62:0/3419131323 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e8107830 0x7f78e8109ca0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.529+0000 7f78edc9b640 1 --2- 172.21.15.62:0/3419131323 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8104f00 0x7f78e81072f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.529+0000 7f78edc9b640 1 --2- 172.21.15.62:0/3419131323 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e81025d0 0x7f78e81049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.529+0000 7f78edc9b640 1 -- 172.21.15.62:0/3419131323 >> 172.21.15.62:0/3419131323 conn(0x7f78e80fc1b0 msgr2=0x7f78e80fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.529+0000 7f78edc9b640 1 -- 172.21.15.62:0/3419131323 shutdown_connections 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.529+0000 7f78edc9b640 1 -- 172.21.15.62:0/3419131323 wait complete. 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78edc9b640 1 Processor -- start 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78edc9b640 1 -- start start 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78edc9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e81025d0 0x7f78e8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78edc9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e8104f00 0x7f78e81983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78edc9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8107830 0x7f78e819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78edc9b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78e810d810 con 0x7f78e81025d0 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78edc9b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78e810d690 con 0x7f78e8104f00 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78edc9b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78e810d990 con 0x7f78e8107830 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78e77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e81025d0 0x7f78e8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78e77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e81025d0 0x7f78e8197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38114/0 (socket says 172.21.15.62:38114) 2024-09-15T23:25:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78e77fe640 1 -- 172.21.15.62:0/4075285402 learned_addr learned my addr 172.21.15.62:0/4075285402 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78e7fff640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8107830 0x7f78e819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.530+0000 7f78e6ffd640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e8104f00 0x7f78e81983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78e77fe640 1 -- 172.21.15.62:0/4075285402 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8107830 msgr2=0x7f78e819f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78e77fe640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8107830 0x7f78e819f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78e77fe640 1 -- 172.21.15.62:0/4075285402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e8104f00 msgr2=0x7f78e81983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78e77fe640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e8104f00 0x7f78e81983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78e77fe640 1 -- 172.21.15.62:0/4075285402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e819fa80 con 0x7f78e81025d0 2024-09-15T23:25:34.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78e77fe640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e81025d0 0x7f78e8197e60 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f78d800d950 tx=0x7f78d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78e4ff9640 1 -- 172.21.15.62:0/4075285402 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d8017070 con 0x7f78e81025d0 2024-09-15T23:25:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78e4ff9640 1 -- 172.21.15.62:0/4075285402 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78d8009c30 con 0x7f78e81025d0 2024-09-15T23:25:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78edc9b640 1 -- 172.21.15.62:0/4075285402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e819fd70 con 0x7f78e81025d0 2024-09-15T23:25:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78e4ff9640 1 -- 172.21.15.62:0/4075285402 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d800f6b0 con 0x7f78e81025d0 2024-09-15T23:25:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.531+0000 7f78edc9b640 1 -- 172.21.15.62:0/4075285402 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e81a0200 con 0x7f78e81025d0 2024-09-15T23:25:34.543 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.534+0000 7f78f0a32740 1 Processor -- start 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78f0a32740 1 -- start start 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214201b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 0x559e214251b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e21425780 0x559e21427c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78f0a32740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559e2137f2f0 con 0x559e2141fdb0 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78f0a32740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559e21327270 con 0x559e21425780 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78f0a32740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559e2137b2b0 con 0x559e214206f0 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e4ff9640 1 -- 172.21.15.62:0/4075285402 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f78d80109d0 con 0x7f78e81025d0 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214201b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214201b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38126/0 (socket says 172.21.15.62:38126) 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e7fff640 1 -- 172.21.15.62:0/162539778 learned_addr learned my addr 172.21.15.62:0/162539778 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e6ffd640 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e21425780 0x559e21427c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e77fe640 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 0x559e214251b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e7fff640 1 -- 172.21.15.62:0/162539778 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 msgr2=0x559e214251b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e7fff640 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 0x559e214251b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e7fff640 1 -- 172.21.15.62:0/162539778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e21425780 msgr2=0x559e21427c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e7fff640 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e21425780 0x559e21427c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e7fff640 1 -- 172.21.15.62:0/162539778 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e21326340 con 0x559e2141fdb0 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.535+0000 7f78e6ffd640 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e21425780 0x559e21427c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.536+0000 7f78e7fff640 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214201b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f78e806b1a0 tx=0x7f78e0009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a934870eca9ade server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.536+0000 7f78e4ff9640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78d00a32f0 0x7f78d00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.536+0000 7f78cdffb640 1 -- 172.21.15.62:0/162539778 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e00098c0 con 0x559e2141fdb0 2024-09-15T23:25:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.536+0000 7f78e77fe640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78d00a32f0 0x7f78d00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.536+0000 7f78cdffb640 1 -- 172.21.15.62:0/162539778 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e0036040 con 0x559e2141fdb0 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.536+0000 7f78cdffb640 1 -- 172.21.15.62:0/162539778 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0004290 con 0x559e2141fdb0 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.536+0000 7f78e4ff9640 1 -- 172.21.15.62:0/4075285402 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78d80cf830 con 0x7f78e81025d0 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 -- 172.21.15.62:0/162539778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 msgr2=0x559e214201b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78e77fe640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78d00a32f0 0x7f78d00a57b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f78d80cd320 tx=0x7f78d80d5550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214201b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f78e806b1a0 tx=0x7f78e0009360 comp rx=0 tx=0).stop 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 -- 172.21.15.62:0/162539778 shutdown_connections 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e21425780 0x559e21427c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 0x559e214251b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 --2- 172.21.15.62:0/162539778 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214201b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 -- 172.21.15.62:0/162539778 >> 172.21.15.62:0/162539778 conn(0x559e21417010 msgr2=0x559e21417410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 -- 172.21.15.62:0/162539778 shutdown_connections 2024-09-15T23:25:34.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 -- 172.21.15.62:0/162539778 wait complete. 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.537+0000 7f78f0a32740 1 -- start start 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78f0a32740 1 Processor -- start 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e21429dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 0x559e2142a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e21425780 0x559e21431350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78f0a32740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559e21428510 con 0x559e21425780 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78f0a32740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559e21308290 con 0x559e2141fdb0 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78f0a32740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559e21428690 con 0x559e214206f0 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78f0a32740 10 client.0 _ll_register_callbacks cb 0x559e21339140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:34.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e21425780 0x559e21431350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e21425780 0x559e21431350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38132/0 (socket says 172.21.15.62:38132) 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e7fff640 1 -- 172.21.15.62:0/1175044137 learned_addr learned my addr 172.21.15.62:0/1175044137 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e6ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 0x559e2142a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e7fff640 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e21429dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e7fff640 1 -- 172.21.15.62:0/1175044137 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 msgr2=0x559e2142a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e7fff640 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 0x559e2142a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e7fff640 1 -- 172.21.15.62:0/1175044137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 msgr2=0x559e21429dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e7fff640 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e21429dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.538+0000 7f78e7fff640 1 -- 172.21.15.62:0/1175044137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e21092210 con 0x559e21425780 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78e7fff640 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e21425780 0x559e21431350 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f78e0031ae0 tx=0x7f78e0005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f8b984a8d7089 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78c6ffd640 1 -- 172.21.15.62:0/1175044137 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0056060 con 0x559e21425780 2024-09-15T23:25:34.548 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78f0a32740 1 -- 172.21.15.62:0/1175044137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78e0036040 con 0x559e21425780 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78c6ffd640 1 -- 172.21.15.62:0/1175044137 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78e0041040 con 0x559e21425780 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78c6ffd640 1 -- 172.21.15.62:0/1175044137 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f78e005d050 con 0x559e21425780 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78f0a32740 10 client.17691 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78f0a32740 1 -- 172.21.15.62:0/1175044137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559e2115e620 con 0x559e21425780 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78c6ffd640 1 -- 172.21.15.62:0/1175044137 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e0036040 con 0x559e21425780 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78c6ffd640 1 -- 172.21.15.62:0/1175044137 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f78e002f610 con 0x559e21425780 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78f0a32740 10 client.17691 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.539+0000 7f78f0a32740 10 client.17691 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.540+0000 7f78f0a32740 20 client.17691 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.540+0000 7f78f0a32740 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559e21435740 0x559e21437b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.540+0000 7f78f0a32740 4 client.17691 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.540+0000 7f78f0a32740 1 -- 172.21.15.62:0/1175044137 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e2138ca20 con 0x559e21435740 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.540+0000 7f78e6ffd640 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559e21435740 0x559e21437b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.540+0000 7f78e6ffd640 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559e21435740 0x559e21437b30 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21ca820865145445 server_cookie=7de36c6fc43f43f in_seq=0 out_seq=0 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.540+0000 7f78c6ffd640 10 client.17691 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78c6ffd640 1 -- 172.21.15.62:0/1175044137 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559e2138ca20 con 0x559e21435740 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78c6ffd640 10 client.17691 handle_command_reply: tid=0 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 client.17691 shutdown 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 -- 172.21.15.62:0/1175044137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e21425780 msgr2=0x559e21431350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e21425780 0x559e21431350 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f78e0031ae0 tx=0x7f78e0005d80 comp rx=0 tx=0).stop 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 -- 172.21.15.62:0/1175044137 shutdown_connections 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559e21435740 0x559e21437b30 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e21425780 0x559e21431350 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e214206f0 0x559e2142a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78c6ffd640 0 client.17691 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 --2- 172.21.15.62:0/1175044137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e21429dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 -- 172.21.15.62:0/1175044137 >> 172.21.15.62:0/1175044137 conn(0x559e21417010 msgr2=0x559e21418820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 -- 172.21.15.62:0/1175044137 shutdown_connections 2024-09-15T23:25:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 1 -- 172.21.15.62:0/1175044137 wait complete. 2024-09-15T23:25:34.551 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.541+0000 7f78f0a32740 20 client.17691 trim_cache size 0 max 16384 2024-09-15T23:25:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.566+0000 7f78f0a32740 1 Processor -- start 2024-09-15T23:25:34.571 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 555.0 s, 613.4 tps, lat 161.777 ms stddev 180.113 2024-09-15T23:25:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.570+0000 7f78f0a32740 1 -- start start 2024-09-15T23:25:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.570+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214160b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.570+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e214206f0 0x559e2143c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.570+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e2143c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.570+0000 7f78f0a32740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78e81aadc0 con 0x559e2141fdb0 2024-09-15T23:25:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.570+0000 7f78f0a32740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78e81a8690 con 0x559e214206f0 2024-09-15T23:25:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.570+0000 7f78f0a32740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78e81a64e0 con 0x559e21425780 2024-09-15T23:25:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.572+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e2143c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.573+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e2143c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:45460/0 (socket says 172.21.15.62:45460) 2024-09-15T23:25:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.573+0000 7f78e7fff640 1 -- 172.21.15.62:0/3096741233 learned_addr learned my addr 172.21.15.62:0/3096741233 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.573+0000 7f78e7fff640 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214160b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.576+0000 7f78e6ffd640 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e214206f0 0x559e2143c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.576+0000 7f78e7fff640 1 -- 172.21.15.62:0/3096741233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e214206f0 msgr2=0x559e2143c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.576+0000 7f78e7fff640 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e214206f0 0x559e2143c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.576+0000 7f78e7fff640 1 -- 172.21.15.62:0/3096741233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 msgr2=0x559e214160b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.576+0000 7f78e7fff640 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214160b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.576+0000 7f78e7fff640 1 -- 172.21.15.62:0/3096741233 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e214254f0 con 0x559e21425780 2024-09-15T23:25:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.580+0000 7f78e6ffd640 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e214206f0 0x559e2143c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.580+0000 7f78e7fff640 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e2143c900 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x559e21412a30 tx=0x7f78e002a8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f4ddbfbc415e97 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.589+0000 7f78cd7fa640 1 -- 172.21.15.62:0/3096741233 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0056030 con 0x559e21425780 2024-09-15T23:25:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.589+0000 7f78cd7fa640 1 -- 172.21.15.62:0/3096741233 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78e0036070 con 0x559e21425780 2024-09-15T23:25:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.589+0000 7f78cd7fa640 1 -- 172.21.15.62:0/3096741233 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e003f040 con 0x559e21425780 2024-09-15T23:25:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.591+0000 7f78f0a32740 1 -- 172.21.15.62:0/3096741233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 msgr2=0x559e2143c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.591+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e2143c900 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x559e21412a30 tx=0x7f78e002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.599+0000 7f78f0a32740 1 -- 172.21.15.62:0/3096741233 shutdown_connections 2024-09-15T23:25:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.599+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e2143c900 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.599+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e214206f0 0x559e2143c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.599+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3096741233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e2141fdb0 0x559e214160b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.599+0000 7f78f0a32740 1 -- 172.21.15.62:0/3096741233 >> 172.21.15.62:0/3096741233 conn(0x559e21417010 msgr2=0x559e214159a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.610+0000 7f78f0a32740 1 -- 172.21.15.62:0/3096741233 shutdown_connections 2024-09-15T23:25:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.612+0000 7f78f0a32740 1 -- 172.21.15.62:0/3096741233 wait complete. 2024-09-15T23:25:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.613+0000 7f78f0a32740 1 -- start start 2024-09-15T23:25:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.613+0000 7f78f0a32740 1 Processor -- start 2024-09-15T23:25:34.614 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.614+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e2142b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.614+0000 7f78f0a32740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e214206f0 0x559e2142b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.614+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e2142b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.614+0000 7f78e7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e2142b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:47180/0 (socket says 172.21.15.62:47180) 2024-09-15T23:25:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.614+0000 7f78e7fff640 1 -- 172.21.15.62:0/3445420229 learned_addr learned my addr 172.21.15.62:0/3445420229 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.614+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e21430410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.614+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78e81a6b50 con 0x559e214206f0 2024-09-15T23:25:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.614+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78e81a7fc0 con 0x559e2141fdb0 2024-09-15T23:25:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.614+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78e81a7220 con 0x559e21425780 2024-09-15T23:25:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.615+0000 7f78f0a32740 10 client.0 _ll_register_callbacks cb 0x7f78dc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.615+0000 7f78e7fff640 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e21430410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.615+0000 7f78e7fff640 1 -- 172.21.15.62:0/3445420229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 msgr2=0x559e21430410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.615+0000 7f78e7fff640 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e21430410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.615+0000 7f78e7fff640 1 -- 172.21.15.62:0/3445420229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e214206f0 msgr2=0x559e2142b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.615+0000 7f78e7fff640 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e214206f0 0x559e2142b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.615+0000 7f78e7fff640 1 -- 172.21.15.62:0/3445420229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e21430a20 con 0x559e2141fdb0 2024-09-15T23:25:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.615+0000 7f78e7fff640 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e2142b490 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f78e0006010 tx=0x7f78e0005b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17b892744e4c4d29 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.615+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78e0036070 con 0x559e2141fdb0 2024-09-15T23:25:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.616+0000 7f78c6ffd640 10 client.27263 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:25:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.616+0000 7f78c6ffd640 1 -- 172.21.15.62:0/3445420229 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e0069060 con 0x559e2141fdb0 2024-09-15T23:25:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.616+0000 7f78c6ffd640 1 -- 172.21.15.62:0/3445420229 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78e0004740 con 0x559e2141fdb0 2024-09-15T23:25:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.616+0000 7f78c6ffd640 1 -- 172.21.15.62:0/3445420229 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f78e003f070 con 0x559e2141fdb0 2024-09-15T23:25:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.620+0000 7f78f0a32740 10 client.27263 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:34.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.620+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=35}) -- 0x559e214b5e40 con 0x559e2141fdb0 2024-09-15T23:25:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.624+0000 7f78c6ffd640 1 -- 172.21.15.62:0/3445420229 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f78e002f610 con 0x559e2141fdb0 2024-09-15T23:25:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.624+0000 7f78f0a32740 10 client.27263 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.624+0000 7f78f0a32740 10 client.27263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.624+0000 7f78f0a32740 20 client.27263 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.624+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559e214b6730 0x559e214b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.624+0000 7f78f0a32740 4 client.27263 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:25:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.624+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f78e0036070 con 0x559e214b6730 2024-09-15T23:25:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.630+0000 7f78e6ffd640 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559e214b6730 0x559e214b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.644+0000 7f78e6ffd640 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559e214b6730 0x559e214b8b50 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45842c9c3b0a06ff server_cookie=36b8ec81f27b9f02 in_seq=0 out_seq=0 2024-09-15T23:25:34.646 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.645+0000 7f78c6ffd640 10 client.27263 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.702+0000 7f78c6ffd640 1 -- 172.21.15.62:0/3445420229 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f78e0036070 con 0x559e214b6730 2024-09-15T23:25:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.702+0000 7f78c6ffd640 10 client.27263 handle_command_reply: tid=0 2024-09-15T23:25:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.702+0000 7f78f0a32740 1 client.27263 shutdown 2024-09-15T23:25:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.702+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 msgr2=0x559e2142b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.702+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e2142b490 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f78e0006010 tx=0x7f78e0005b70 comp rx=0 tx=0).stop 2024-09-15T23:25:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.703+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 shutdown_connections 2024-09-15T23:25:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.703+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559e214b6730 0x559e214b8b50 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.704 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.703+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559e21425780 0x559e21430410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.703+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559e214206f0 0x559e2142b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.703+0000 7f78f0a32740 1 --2- 172.21.15.62:0/3445420229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559e2141fdb0 0x559e2142b490 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.703+0000 7f78c6ffd640 0 client.27263 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.703+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 >> 172.21.15.62:0/3445420229 conn(0x559e21417010 msgr2=0x559e214161c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.703+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 shutdown_connections 2024-09-15T23:25:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.704+0000 7f78f0a32740 1 -- 172.21.15.62:0/3445420229 wait complete. 2024-09-15T23:25:34.705 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.704+0000 7f78f0a32740 20 client.27263 trim_cache size 0 max 16384 2024-09-15T23:25:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:25:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:25:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:25:34.706 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:25:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.705+0000 7f78edc9b640 1 -- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78d00a32f0 msgr2=0x7f78d00a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78d00a32f0 0x7f78d00a57b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f78d80cd320 tx=0x7f78d80d5550 comp rx=0 tx=0).stop 2024-09-15T23:25:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 -- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e81025d0 msgr2=0x7f78e8197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e81025d0 0x7f78e8197e60 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f78d800d950 tx=0x7f78d800de10 comp rx=0 tx=0).stop 2024-09-15T23:25:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 -- 172.21.15.62:0/4075285402 shutdown_connections 2024-09-15T23:25:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78d00a32f0 0x7f78d00a57b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78e8107830 0x7f78e819f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78e8104f00 0x7f78e81983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 --2- 172.21.15.62:0/4075285402 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78e81025d0 0x7f78e8197e60 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 -- 172.21.15.62:0/4075285402 >> 172.21.15.62:0/4075285402 conn(0x7f78e80fc1b0 msgr2=0x7f78e8102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 -- 172.21.15.62:0/4075285402 shutdown_connections 2024-09-15T23:25:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:34.706+0000 7f78edc9b640 1 -- 172.21.15.62:0/4075285402 wait complete. 2024-09-15T23:25:35.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:35 smithi062 ceph-mon[30949]: pgmap v649: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 4.7 MiB/s wr, 620 op/s 2024-09-15T23:25:35.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:35 smithi067 ceph-mon[35665]: pgmap v649: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 4.7 MiB/s wr, 620 op/s 2024-09-15T23:25:35.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:35 smithi148 ceph-mon[35610]: pgmap v649: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 4.7 MiB/s wr, 620 op/s 2024-09-15T23:25:37.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:37 smithi062 ceph-mon[30949]: pgmap v650: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 4.9 MiB/s wr, 641 op/s 2024-09-15T23:25:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:37 smithi067 ceph-mon[35665]: pgmap v650: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 4.9 MiB/s wr, 641 op/s 2024-09-15T23:25:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:37 smithi148 ceph-mon[35610]: pgmap v650: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 4.9 MiB/s wr, 641 op/s 2024-09-15T23:25:39.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:39 smithi062 ceph-mon[30949]: pgmap v651: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.3 MiB/s wr, 580 op/s 2024-09-15T23:25:39.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:39 smithi067 ceph-mon[35665]: pgmap v651: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.3 MiB/s wr, 580 op/s 2024-09-15T23:25:39.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:39 smithi148 ceph-mon[35610]: pgmap v651: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.3 MiB/s wr, 580 op/s 2024-09-15T23:25:39.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 560.0 s, 816.2 tps, lat 123.112 ms stddev 132.325 2024-09-15T23:25:41.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:41 smithi062 ceph-mon[30949]: pgmap v652: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 4.8 MiB/s wr, 626 op/s 2024-09-15T23:25:41.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:41 smithi067 ceph-mon[35665]: pgmap v652: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 4.8 MiB/s wr, 626 op/s 2024-09-15T23:25:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:41 smithi148 ceph-mon[35610]: pgmap v652: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 4.8 MiB/s wr, 626 op/s 2024-09-15T23:25:43.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:43 smithi062 ceph-mon[30949]: pgmap v653: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.4 MiB/s wr, 695 op/s 2024-09-15T23:25:43.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:43 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:25:43.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:43 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:25:43.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:43 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:25:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:43 smithi067 ceph-mon[35665]: pgmap v653: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.4 MiB/s wr, 695 op/s 2024-09-15T23:25:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:43 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:25:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:43 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:25:43.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:43 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:25:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:43 smithi148 ceph-mon[35610]: pgmap v653: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 5.4 MiB/s wr, 695 op/s 2024-09-15T23:25:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:43 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:25:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:43 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:25:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:43 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:25:44.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 565.0 s, 646.1 tps, lat 142.867 ms stddev 192.550 2024-09-15T23:25:45.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:45 smithi148 ceph-mon[35610]: pgmap v654: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.6 MiB/s wr, 703 op/s 2024-09-15T23:25:45.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:45 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:25:45.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:45 smithi062 ceph-mon[30949]: pgmap v654: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.6 MiB/s wr, 703 op/s 2024-09-15T23:25:45.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:45 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:25:45.528 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:45 smithi067 ceph-mon[35665]: pgmap v654: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.6 MiB/s wr, 703 op/s 2024-09-15T23:25:45.529 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:45 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:25:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:47 smithi062 ceph-mon[30949]: pgmap v655: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 815 op/s 2024-09-15T23:25:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:47 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:47 smithi067 ceph-mon[35665]: pgmap v655: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 815 op/s 2024-09-15T23:25:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:47 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:47 smithi148 ceph-mon[35610]: pgmap v655: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 815 op/s 2024-09-15T23:25:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:47 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.879+0000 7f8864bb7640 1 -- 172.21.15.62:0/4077042794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8860072d20 msgr2=0x7f8860073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.879+0000 7f8864bb7640 1 --2- 172.21.15.62:0/4077042794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8860072d20 0x7f8860073180 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8858009ca0 tx=0x7f885802a8f0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.885+0000 7f8864bb7640 1 -- 172.21.15.62:0/4077042794 shutdown_connections 2024-09-15T23:25:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.885+0000 7f8864bb7640 1 --2- 172.21.15.62:0/4077042794 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f88600736c0 0x7f8860119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.885+0000 7f8864bb7640 1 --2- 172.21.15.62:0/4077042794 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8860072d20 0x7f8860073180 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.885+0000 7f8864bb7640 1 --2- 172.21.15.62:0/4077042794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8860109700 0x7f8860109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.885+0000 7f8864bb7640 1 -- 172.21.15.62:0/4077042794 >> 172.21.15.62:0/4077042794 conn(0x7f886006ece0 msgr2=0x7f8860071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:47.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.893+0000 7f8864bb7640 1 -- 172.21.15.62:0/4077042794 shutdown_connections 2024-09-15T23:25:47.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.896+0000 7f8864bb7640 1 -- 172.21.15.62:0/4077042794 wait complete. 2024-09-15T23:25:47.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.897+0000 7f8864bb7640 1 Processor -- start 2024-09-15T23:25:47.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.899+0000 7f8864bb7640 1 -- start start 2024-09-15T23:25:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.899+0000 7f8864bb7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8860072d20 0x7f8860116ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.899+0000 7f8864bb7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88600736c0 0x7f88601171e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.899+0000 7f8864bb7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8860109700 0x7f8860118220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.899+0000 7f8864bb7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8860110f30 con 0x7f8860109700 2024-09-15T23:25:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.899+0000 7f8864bb7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8860110db0 con 0x7f88600736c0 2024-09-15T23:25:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.899+0000 7f8864bb7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f88601110b0 con 0x7f8860072d20 2024-09-15T23:25:47.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.899+0000 7f885ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8860109700 0x7f8860118220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.903+0000 7f885dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88600736c0 0x7f88601171e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.903+0000 7f885dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88600736c0 0x7f88601171e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:37090/0 (socket says 172.21.15.62:37090) 2024-09-15T23:25:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.903+0000 7f885dd9b640 1 -- 172.21.15.62:0/3595040598 learned_addr learned my addr 172.21.15.62:0/3595040598 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.903+0000 7f885e59c640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8860072d20 0x7f8860116ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.904+0000 7f885e59c640 1 -- 172.21.15.62:0/3595040598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88600736c0 msgr2=0x7f88601171e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.904+0000 7f885e59c640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88600736c0 0x7f88601171e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.904+0000 7f885e59c640 1 -- 172.21.15.62:0/3595040598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8860109700 msgr2=0x7f8860118220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.904+0000 7f885e59c640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8860109700 0x7f8860118220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.904+0000 7f885e59c640 1 -- 172.21.15.62:0/3595040598 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8860118890 con 0x7f8860072d20 2024-09-15T23:25:47.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.904+0000 7f885e59c640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8860072d20 0x7f8860116ca0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f884800c970 tx=0x7f884800ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.912+0000 7f88577fe640 1 -- 172.21.15.62:0/3595040598 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8848007bb0 con 0x7f8860072d20 2024-09-15T23:25:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.912+0000 7f88577fe640 1 -- 172.21.15.62:0/3595040598 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8848007d50 con 0x7f8860072d20 2024-09-15T23:25:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.912+0000 7f8864bb7640 1 -- 172.21.15.62:0/3595040598 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8860118b80 con 0x7f8860072d20 2024-09-15T23:25:47.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.912+0000 7f8864bb7640 1 -- 172.21.15.62:0/3595040598 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88601190d0 con 0x7f8860072d20 2024-09-15T23:25:47.918 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.916+0000 7f88577fe640 1 -- 172.21.15.62:0/3595040598 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884800f450 con 0x7f8860072d20 2024-09-15T23:25:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.918+0000 7f88577fe640 1 -- 172.21.15.62:0/3595040598 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f884800f6b0 con 0x7f8860072d20 2024-09-15T23:25:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.919+0000 7f88577fe640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f884c0a3520 0x7f884c0a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.919+0000 7f88577fe640 1 -- 172.21.15.62:0/3595040598 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f88480d08b0 con 0x7f8860072d20 2024-09-15T23:25:47.922 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.919+0000 7f8867a29740 1 Processor -- start 2024-09-15T23:25:47.924 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.923+0000 7f885ed9d640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f884c0a3520 0x7f884c0a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.924+0000 7f8867a29740 1 -- start start 2024-09-15T23:25:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.924+0000 7f8867a29740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 0x564ab2970da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.924+0000 7f8867a29740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2975da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.924+0000 7f8867a29740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2978830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.924+0000 7f8867a29740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564ab28d42f0 con 0x564ab29709a0 2024-09-15T23:25:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.924+0000 7f8867a29740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564ab287c270 con 0x564ab29712e0 2024-09-15T23:25:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.924+0000 7f8867a29740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564ab28d02b0 con 0x564ab2976370 2024-09-15T23:25:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.925+0000 7f885ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2978830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.925+0000 7f885ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2978830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:58198/0 (socket says 172.21.15.62:58198) 2024-09-15T23:25:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.925+0000 7f885ed9d640 1 -- 172.21.15.62:0/761378693 learned_addr learned my addr 172.21.15.62:0/761378693 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:47.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.925+0000 7f885dd9b640 1 --2- 172.21.15.62:0/761378693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2975da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.925+0000 7f885ed9d640 1 -- 172.21.15.62:0/761378693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 msgr2=0x564ab2975da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.925+0000 7f885ed9d640 1 --2- 172.21.15.62:0/761378693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2975da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.925+0000 7f885ed9d640 1 -- 172.21.15.62:0/761378693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 msgr2=0x564ab2970da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.925+0000 7f885ed9d640 1 --2- 172.21.15.62:0/761378693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 0x564ab2970da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.925+0000 7f885ed9d640 1 -- 172.21.15.62:0/761378693 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ab287b340 con 0x564ab2976370 2024-09-15T23:25:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.926+0000 7f885ed9d640 1 --2- 172.21.15.62:0/761378693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2978830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f885000b520 tx=0x7f8850017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e08a3afb24273603 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.926+0000 7f885ed9d640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f884c0a3520 0x7f884c0a59e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f885000bd90 tx=0x7f8850014660 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.928 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.926+0000 7f8854ff9640 1 -- 172.21.15.62:0/761378693 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8850022400 con 0x564ab2976370 2024-09-15T23:25:47.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.926+0000 7f8854ff9640 1 -- 172.21.15.62:0/761378693 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8850011890 con 0x564ab2976370 2024-09-15T23:25:47.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.928+0000 7f8867a29740 1 -- 172.21.15.62:0/761378693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 msgr2=0x564ab2978830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.929 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.928+0000 7f8867a29740 1 --2- 172.21.15.62:0/761378693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2978830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f885000b520 tx=0x7f8850017000 comp rx=0 tx=0).stop 2024-09-15T23:25:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.931+0000 7f8867a29740 1 -- 172.21.15.62:0/761378693 shutdown_connections 2024-09-15T23:25:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.931+0000 7f8867a29740 1 --2- 172.21.15.62:0/761378693 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2978830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.931+0000 7f8867a29740 1 --2- 172.21.15.62:0/761378693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2975da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.931+0000 7f8867a29740 1 --2- 172.21.15.62:0/761378693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 0x564ab2970da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.931+0000 7f8867a29740 1 -- 172.21.15.62:0/761378693 >> 172.21.15.62:0/761378693 conn(0x564ab296c010 msgr2=0x564ab296c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.931+0000 7f8867a29740 1 -- 172.21.15.62:0/761378693 shutdown_connections 2024-09-15T23:25:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.931+0000 7f8867a29740 1 -- 172.21.15.62:0/761378693 wait complete. 2024-09-15T23:25:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.932+0000 7f8867a29740 1 -- start start 2024-09-15T23:25:47.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.932+0000 7f8867a29740 1 Processor -- start 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.932+0000 7f8867a29740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29709a0 0x564ab297ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.932+0000 7f8867a29740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab297b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.932+0000 7f8867a29740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 0x564ab2982180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.932+0000 7f8867a29740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564ab2979330 con 0x564ab2976370 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.932+0000 7f8867a29740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564ab285d290 con 0x564ab29712e0 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.932+0000 7f8867a29740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564ab29794b0 con 0x564ab29709a0 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.932+0000 7f8867a29740 10 client.0 _ll_register_callbacks cb 0x564ab288e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.933+0000 7f885dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab297b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.933+0000 7f885dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab297b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:37116/0 (socket says 172.21.15.62:37116) 2024-09-15T23:25:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.933+0000 7f885dd9b640 1 -- 172.21.15.62:0/125034615 learned_addr learned my addr 172.21.15.62:0/125034615 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.933+0000 7f885e59c640 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29709a0 0x564ab297ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.933+0000 7f885dd9b640 1 -- 172.21.15.62:0/125034615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29709a0 msgr2=0x564ab297ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.933+0000 7f885dd9b640 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29709a0 0x564ab297ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.933+0000 7f885dd9b640 1 -- 172.21.15.62:0/125034615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 msgr2=0x564ab2982180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:25:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.933+0000 7f885dd9b640 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 0x564ab2982180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.935 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.933+0000 7f885dd9b640 1 -- 172.21.15.62:0/125034615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ab25e7210 con 0x564ab29712e0 2024-09-15T23:25:47.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.935+0000 7f885dd9b640 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab297b140 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8858006010 tx=0x7f8858031e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=310bb5f42b7f2a23 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:47.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.935+0000 7f8845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:25:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.936+0000 7f8845ffb640 1 -- 172.21.15.62:0/125034615 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885802f850 con 0x564ab29712e0 2024-09-15T23:25:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.936+0000 7f8845ffb640 1 -- 172.21.15.62:0/125034615 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f885802f9f0 con 0x564ab29712e0 2024-09-15T23:25:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.936+0000 7f8867a29740 1 -- 172.21.15.62:0/125034615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8850011890 con 0x564ab29712e0 2024-09-15T23:25:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.936+0000 7f8845ffb640 1 -- 172.21.15.62:0/125034615 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8858051050 con 0x564ab29712e0 2024-09-15T23:25:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.938+0000 7f8867a29740 10 client.27269 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:47.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.938+0000 7f8867a29740 1 -- 172.21.15.62:0/125034615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564ab26b3620 con 0x564ab29712e0 2024-09-15T23:25:47.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.945+0000 7f8845ffb640 1 -- 172.21.15.62:0/125034615 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8850011890 con 0x564ab29712e0 2024-09-15T23:25:47.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.945+0000 7f8845ffb640 1 -- 172.21.15.62:0/125034615 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f885804c030 con 0x564ab29712e0 2024-09-15T23:25:47.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.949+0000 7f8867a29740 10 client.27269 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:47.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.949+0000 7f8867a29740 10 client.27269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:47.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.949+0000 7f8867a29740 20 client.27269 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:47.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.949+0000 7f8867a29740 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ab2986890 0x564ab2988c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:47.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.949+0000 7f8867a29740 4 client.27269 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:25:47.950 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.949+0000 7f8867a29740 1 -- 172.21.15.62:0/125034615 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ab28e1a20 con 0x564ab2986890 2024-09-15T23:25:47.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.952+0000 7f885e59c640 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ab2986890 0x564ab2988c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.959+0000 7f885e59c640 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ab2986890 0x564ab2988c80 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5b63211c24d7d61 server_cookie=7ca6b6b157af6bea in_seq=0 out_seq=0 2024-09-15T23:25:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.959+0000 7f8845ffb640 10 client.27269 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.961+0000 7f8845ffb640 1 -- 172.21.15.62:0/125034615 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564ab28e1a20 con 0x564ab2986890 2024-09-15T23:25:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.961+0000 7f8845ffb640 10 client.27269 handle_command_reply: tid=0 2024-09-15T23:25:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.961+0000 7f8867a29740 1 client.27269 shutdown 2024-09-15T23:25:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.962+0000 7f8867a29740 1 -- 172.21.15.62:0/125034615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 msgr2=0x564ab297b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.962+0000 7f8867a29740 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab297b140 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8858006010 tx=0x7f8858031e20 comp rx=0 tx=0).stop 2024-09-15T23:25:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.962+0000 7f8867a29740 1 -- 172.21.15.62:0/125034615 shutdown_connections 2024-09-15T23:25:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.962+0000 7f8867a29740 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ab2986890 0x564ab2988c80 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.962+0000 7f8867a29740 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 0x564ab2982180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.962+0000 7f8867a29740 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab297b140 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.962+0000 7f8867a29740 1 --2- 172.21.15.62:0/125034615 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29709a0 0x564ab297ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.962+0000 7f8867a29740 1 -- 172.21.15.62:0/125034615 >> 172.21.15.62:0/125034615 conn(0x564ab296c010 msgr2=0x564ab296d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.965+0000 7f8845ffb640 0 client.27269 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:47.966 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.965+0000 7f8867a29740 1 -- 172.21.15.62:0/125034615 shutdown_connections 2024-09-15T23:25:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.966+0000 7f8867a29740 1 -- 172.21.15.62:0/125034615 wait complete. 2024-09-15T23:25:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:47.969+0000 7f8867a29740 20 client.27269 trim_cache size 0 max 16384 2024-09-15T23:25:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.003+0000 7f8867a29740 1 Processor -- start 2024-09-15T23:25:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.003+0000 7f8867a29740 1 -- start start 2024-09-15T23:25:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.003+0000 7f8867a29740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 0x564ab2988670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f8867a29740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2988bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f8867a29740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2982140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f885dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2988bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f885dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2988bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:37128/0 (socket says 172.21.15.62:37128) 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f885dd9b640 1 -- 172.21.15.62:0/88897870 learned_addr learned my addr 172.21.15.62:0/88897870 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f8867a29740 1 -- 172.21.15.62:0/88897870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8840012560 con 0x564ab29709a0 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f8867a29740 1 -- 172.21.15.62:0/88897870 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f884000fe30 con 0x564ab29712e0 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f8867a29740 1 -- 172.21.15.62:0/88897870 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f884000dc80 con 0x564ab2976370 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f885e59c640 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 0x564ab2988670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.004+0000 7f885ed9d640 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2982140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.005+0000 7f885e59c640 1 -- 172.21.15.62:0/88897870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 msgr2=0x564ab2982140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.005+0000 7f885e59c640 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2982140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.005+0000 7f885e59c640 1 -- 172.21.15.62:0/88897870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 msgr2=0x564ab2988bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.005+0000 7f885e59c640 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2988bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.007 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.005+0000 7f885e59c640 1 -- 172.21.15.62:0/88897870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ab2975720 con 0x564ab29709a0 2024-09-15T23:25:48.008 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.005+0000 7f885e59c640 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 0x564ab2988670 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f884809c7f0 tx=0x7f8848098df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915928d1a3ab61b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.005+0000 7f8847fff640 1 -- 172.21.15.62:0/88897870 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88480ba070 con 0x564ab29709a0 2024-09-15T23:25:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.005+0000 7f8847fff640 1 -- 172.21.15.62:0/88897870 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88480a5760 con 0x564ab29709a0 2024-09-15T23:25:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.005+0000 7f8847fff640 1 -- 172.21.15.62:0/88897870 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88480b5070 con 0x564ab29709a0 2024-09-15T23:25:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.006+0000 7f885dd9b640 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2988bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:25:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.006+0000 7f8867a29740 1 -- 172.21.15.62:0/88897870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 msgr2=0x564ab2988670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.009 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.007+0000 7f8867a29740 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 0x564ab2988670 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f884809c7f0 tx=0x7f8848098df0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.008+0000 7f8867a29740 1 -- 172.21.15.62:0/88897870 shutdown_connections 2024-09-15T23:25:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.008+0000 7f8867a29740 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab2976370 0x564ab2982140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.008+0000 7f8867a29740 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29712e0 0x564ab2988bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.008+0000 7f8867a29740 1 --2- 172.21.15.62:0/88897870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab29709a0 0x564ab2988670 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.008+0000 7f8867a29740 1 -- 172.21.15.62:0/88897870 >> 172.21.15.62:0/88897870 conn(0x564ab296c010 msgr2=0x564ab297ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:48.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.017+0000 7f8867a29740 1 -- 172.21.15.62:0/88897870 shutdown_connections 2024-09-15T23:25:48.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.018+0000 7f8867a29740 1 -- 172.21.15.62:0/88897870 wait complete. 2024-09-15T23:25:48.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.019+0000 7f8867a29740 1 -- start start 2024-09-15T23:25:48.020 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.019+0000 7f8867a29740 1 Processor -- start 2024-09-15T23:25:48.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.019+0000 7f8867a29740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29709a0 0x564ab297f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.020+0000 7f8867a29740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29712e0 0x564ab297f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.021 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.020+0000 7f8867a29740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 0x564ab29809a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.020+0000 7f8867a29740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f884000e2f0 con 0x564ab2976370 2024-09-15T23:25:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.020+0000 7f8867a29740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f884000f760 con 0x564ab29709a0 2024-09-15T23:25:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.020+0000 7f8867a29740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f884000e9c0 con 0x564ab29712e0 2024-09-15T23:25:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.020+0000 7f8867a29740 10 client.0 _ll_register_callbacks cb 0x7f8858031cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:25:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.021+0000 7f885ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 0x564ab29809a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.021+0000 7f885ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 0x564ab29809a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42954/0 (socket says 172.21.15.62:42954) 2024-09-15T23:25:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.021+0000 7f885ed9d640 1 -- 172.21.15.62:0/328532439 learned_addr learned my addr 172.21.15.62:0/328532439 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:25:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.024+0000 7f885dd9b640 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29712e0 0x564ab297f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.025 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.024+0000 7f885e59c640 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29709a0 0x564ab297f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.027+0000 7f885ed9d640 1 -- 172.21.15.62:0/328532439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29712e0 msgr2=0x564ab297f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.027+0000 7f885ed9d640 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29712e0 0x564ab297f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.027+0000 7f885ed9d640 1 -- 172.21.15.62:0/328532439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29709a0 msgr2=0x564ab297f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:25:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.027+0000 7f885ed9d640 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29709a0 0x564ab297f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.027+0000 7f885ed9d640 1 -- 172.21.15.62:0/328532439 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ab2980fb0 con 0x564ab2976370 2024-09-15T23:25:48.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.029+0000 7f885ed9d640 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 0x564ab29809a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8850014b60 tx=0x7f8850021f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701ab776139c2558 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:25:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.030+0000 7f8845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:25:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.034+0000 7f8845ffb640 1 -- 172.21.15.62:0/328532439 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885002f060 con 0x564ab2976370 2024-09-15T23:25:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.034+0000 7f8845ffb640 1 -- 172.21.15.62:0/328532439 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8850027040 con 0x564ab2976370 2024-09-15T23:25:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.034+0000 7f8867a29740 1 -- 172.21.15.62:0/328532439 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88480a5760 con 0x564ab2976370 2024-09-15T23:25:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.040+0000 7f8845ffb640 1 -- 172.21.15.62:0/328532439 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8850005c00 con 0x564ab2976370 2024-09-15T23:25:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.041+0000 7f8867a29740 10 client.17706 fetch_fsmap learned FSMap version 35 2024-09-15T23:25:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.041+0000 7f8867a29740 1 -- 172.21.15.62:0/328532439 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564ab29822a0 con 0x564ab2976370 2024-09-15T23:25:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.044+0000 7f8845ffb640 1 -- 172.21.15.62:0/328532439 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8850005da0 con 0x564ab2976370 2024-09-15T23:25:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.045+0000 7f8845ffb640 1 -- 172.21.15.62:0/328532439 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f88500347a0 con 0x564ab2976370 2024-09-15T23:25:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.046+0000 7f8867a29740 10 client.17706 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:25:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.046+0000 7f8867a29740 10 client.17706 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:25:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.046+0000 7f8867a29740 20 client.17706 populate_metadata read hostname 'smithi062' 2024-09-15T23:25:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.046+0000 7f8867a29740 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ab29f1850 0x564ab29f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:25:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.046+0000 7f8867a29740 4 client.17706 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:25:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.046+0000 7f8867a29740 1 -- 172.21.15.62:0/328532439 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f88480a5760 con 0x564ab29f1850 2024-09-15T23:25:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.048+0000 7f885e59c640 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ab29f1850 0x564ab29f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:25:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.058+0000 7f885e59c640 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ab29f1850 0x564ab29f3c40 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a1928e96b36b303 server_cookie=e4edb64150fdaebf in_seq=0 out_seq=0 2024-09-15T23:25:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.058+0000 7f8845ffb640 10 client.17706 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:48.121 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.120+0000 7f8845ffb640 1 -- 172.21.15.62:0/328532439 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f88480a5760 con 0x564ab29f1850 2024-09-15T23:25:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.120+0000 7f8845ffb640 10 client.17706 handle_command_reply: tid=0 2024-09-15T23:25:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.120+0000 7f8867a29740 1 client.17706 shutdown 2024-09-15T23:25:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 -- 172.21.15.62:0/328532439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 msgr2=0x564ab29809a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 0x564ab29809a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8850014b60 tx=0x7f8850021f70 comp rx=0 tx=0).stop 2024-09-15T23:25:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 -- 172.21.15.62:0/328532439 shutdown_connections 2024-09-15T23:25:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564ab29f1850 0x564ab29f3c40 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ab2976370 0x564ab29809a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564ab29712e0 0x564ab297f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 --2- 172.21.15.62:0/328532439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564ab29709a0 0x564ab297f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 -- 172.21.15.62:0/328532439 >> 172.21.15.62:0/328532439 conn(0x564ab296c010 msgr2=0x564ab297ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8845ffb640 0 client.17706 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:25:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 -- 172.21.15.62:0/328532439 shutdown_connections 2024-09-15T23:25:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.123+0000 7f8867a29740 1 -- 172.21.15.62:0/328532439 wait complete. 2024-09-15T23:25:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.124+0000 7f8867a29740 20 client.17706 trim_cache size 0 max 16384 2024-09-15T23:25:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:25:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:25:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:25:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:25:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 -- 172.21.15.62:0/3595040598 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f884c0a3520 msgr2=0x7f884c0a59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f884c0a3520 0x7f884c0a59e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f885000bd90 tx=0x7f8850014660 comp rx=0 tx=0).stop 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 -- 172.21.15.62:0/3595040598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8860072d20 msgr2=0x7f8860116ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8860072d20 0x7f8860116ca0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f884800c970 tx=0x7f884800ce30 comp rx=0 tx=0).stop 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 -- 172.21.15.62:0/3595040598 shutdown_connections 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f884c0a3520 0x7f884c0a59e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8860109700 0x7f8860118220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88600736c0 0x7f88601171e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 --2- 172.21.15.62:0/3595040598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8860072d20 0x7f8860116ca0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 -- 172.21.15.62:0/3595040598 >> 172.21.15.62:0/3595040598 conn(0x7f886006ece0 msgr2=0x7f886010c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 -- 172.21.15.62:0/3595040598 shutdown_connections 2024-09-15T23:25:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:25:48.125+0000 7f8864bb7640 1 -- 172.21.15.62:0/3595040598 wait complete. 2024-09-15T23:25:48.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:48.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:48.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:25:48.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:25:48.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:48 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:48.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:48.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:48.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:25:48.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:25:48.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:48 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:25:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:25:48.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:48 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:25:49.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:49 smithi067 ceph-mon[35665]: pgmap v656: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 756 op/s 2024-09-15T23:25:49.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:49 smithi148 ceph-mon[35610]: pgmap v656: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 756 op/s 2024-09-15T23:25:49.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 570.0 s, 717.1 tps, lat 149.453 ms stddev 204.934 2024-09-15T23:25:49.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:49 smithi062 ceph-mon[30949]: pgmap v656: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 756 op/s 2024-09-15T23:25:51.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:51 smithi067 ceph-mon[35665]: pgmap v657: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.2 MiB/s wr, 821 op/s 2024-09-15T23:25:51.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:51 smithi062 ceph-mon[30949]: pgmap v657: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.2 MiB/s wr, 821 op/s 2024-09-15T23:25:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:51 smithi148 ceph-mon[35610]: pgmap v657: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.2 MiB/s wr, 821 op/s 2024-09-15T23:25:53.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:53 smithi062 ceph-mon[30949]: pgmap v658: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 871 op/s 2024-09-15T23:25:53.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:53 smithi067 ceph-mon[35665]: pgmap v658: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 871 op/s 2024-09-15T23:25:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:53 smithi148 ceph-mon[35610]: pgmap v658: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 MiB/s wr, 871 op/s 2024-09-15T23:25:54.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 575.0 s, 808.6 tps, lat 122.684 ms stddev 162.442 2024-09-15T23:25:55.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:55 smithi062 ceph-mon[30949]: pgmap v659: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.3 MiB/s wr, 841 op/s 2024-09-15T23:25:56.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:55 smithi067 ceph-mon[35665]: pgmap v659: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.3 MiB/s wr, 841 op/s 2024-09-15T23:25:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:55 smithi148 ceph-mon[35610]: pgmap v659: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.3 MiB/s wr, 841 op/s 2024-09-15T23:25:57.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:57 smithi062 ceph-mon[30949]: pgmap v660: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 808 op/s 2024-09-15T23:25:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:57 smithi067 ceph-mon[35665]: pgmap v660: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 808 op/s 2024-09-15T23:25:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:57 smithi148 ceph-mon[35610]: pgmap v660: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 808 op/s 2024-09-15T23:25:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 580.0 s, 804.2 tps, lat 125.927 ms stddev 148.252 2024-09-15T23:25:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:25:59 smithi062 ceph-mon[30949]: pgmap v661: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.7 MiB/s wr, 657 op/s 2024-09-15T23:26:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:25:59 smithi067 ceph-mon[35665]: pgmap v661: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.7 MiB/s wr, 657 op/s 2024-09-15T23:26:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:25:59 smithi148 ceph-mon[35610]: pgmap v661: 129 pgs: 129 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 5.7 MiB/s wr, 657 op/s 2024-09-15T23:26:01.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:01 smithi067 ceph-mon[35665]: pgmap v662: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 MiB/s wr, 704 op/s 2024-09-15T23:26:01.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:01 smithi062 ceph-mon[30949]: pgmap v662: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 MiB/s wr, 704 op/s 2024-09-15T23:26:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:01 smithi148 ceph-mon[35610]: pgmap v662: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 MiB/s wr, 704 op/s 2024-09-15T23:26:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.048+0000 7f52122a6640 1 -- 172.21.15.62:0/1873063109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 msgr2=0x7f520c100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.048+0000 7f52122a6640 1 --2- 172.21.15.62:0/1873063109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 0x7f520c100e50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009900 tx=0x7f51fc02a400 comp rx=0 tx=0).stop 2024-09-15T23:26:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.048+0000 7f52122a6640 1 -- 172.21.15.62:0/1873063109 shutdown_connections 2024-09-15T23:26:02.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.048+0000 7f52122a6640 1 --2- 172.21.15.62:0/1873063109 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f520c102600 0x7f520c108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.048+0000 7f52122a6640 1 --2- 172.21.15.62:0/1873063109 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f520c101c40 0x7f520c1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.048+0000 7f52122a6640 1 --2- 172.21.15.62:0/1873063109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 0x7f520c100e50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.048+0000 7f52122a6640 1 -- 172.21.15.62:0/1873063109 >> 172.21.15.62:0/1873063109 conn(0x7f520c0fc1b0 msgr2=0x7f520c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.049+0000 7f52122a6640 1 -- 172.21.15.62:0/1873063109 shutdown_connections 2024-09-15T23:26:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.049+0000 7f52122a6640 1 -- 172.21.15.62:0/1873063109 wait complete. 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.049+0000 7f52122a6640 1 Processor -- start 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.049+0000 7f52122a6640 1 -- start start 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f52122a6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 0x7f520c10f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f52122a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f520c101c40 0x7f520c10fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f52122a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f520c102600 0x7f520c10c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f52122a6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f520c10b3f0 con 0x7f520c102600 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f52122a6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f520c10b270 con 0x7f520c100a50 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f52122a6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f520c10b570 con 0x7f520c101c40 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f520bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 0x7f520c10f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f520bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 0x7f520c10f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:38984/0 (socket says 172.21.15.62:38984) 2024-09-15T23:26:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f520bfff640 1 -- 172.21.15.62:0/3079370667 learned_addr learned my addr 172.21.15.62:0/3079370667 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f520b7fe640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f520c101c40 0x7f520c10fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f5210843640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f520c102600 0x7f520c10c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f520bfff640 1 -- 172.21.15.62:0/3079370667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f520c101c40 msgr2=0x7f520c10fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f520bfff640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f520c101c40 0x7f520c10fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f520bfff640 1 -- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f520c102600 msgr2=0x7f520c10c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f520bfff640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f520c102600 0x7f520c10c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.050+0000 7f520bfff640 1 -- 172.21.15.62:0/3079370667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c10ca80 con 0x7f520c100a50 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.051+0000 7f520bfff640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 0x7f520c10f650 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f51fc031610 tx=0x7f51fc032c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.051+0000 7f52097fa640 1 -- 172.21.15.62:0/3079370667 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc03f030 con 0x7f520c100a50 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.051+0000 7f52097fa640 1 -- 172.21.15.62:0/3079370667 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51fc003ad0 con 0x7f520c100a50 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.051+0000 7f52122a6640 1 -- 172.21.15.62:0/3079370667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520c1a6490 con 0x7f520c100a50 2024-09-15T23:26:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.051+0000 7f52097fa640 1 -- 172.21.15.62:0/3079370667 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc035070 con 0x7f520c100a50 2024-09-15T23:26:02.057 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.052+0000 7f52122a6640 1 -- 172.21.15.62:0/3079370667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520c1a6840 con 0x7f520c100a50 2024-09-15T23:26:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.055+0000 7f5215038740 1 Processor -- start 2024-09-15T23:26:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.055+0000 7f5215038740 1 -- start start 2024-09-15T23:26:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.055+0000 7f5215038740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d39a0 0x561f259d3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.055+0000 7f5215038740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d42e0 0x561f259d8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.055+0000 7f5215038740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 0x561f259db830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.055+0000 7f5215038740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561f259372f0 con 0x561f259d9370 2024-09-15T23:26:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.055+0000 7f5215038740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561f258df270 con 0x561f259d42e0 2024-09-15T23:26:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.055+0000 7f5215038740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561f259332b0 con 0x561f259d39a0 2024-09-15T23:26:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f52097fa640 1 -- 172.21.15.62:0/3079370667 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f51fc042030 con 0x7f520c100a50 2024-09-15T23:26:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f5210843640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d39a0 0x561f259d3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f5210843640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d39a0 0x561f259d3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:51072/0 (socket says 172.21.15.62:51072) 2024-09-15T23:26:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f5210843640 1 -- 172.21.15.62:0/3203070944 learned_addr learned my addr 172.21.15.62:0/3203070944 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f520b7fe640 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 0x561f259db830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f520bfff640 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d42e0 0x561f259d8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f5210843640 1 -- 172.21.15.62:0/3203070944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d42e0 msgr2=0x561f259d8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:02.060 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f5210843640 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d42e0 0x561f259d8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f5210843640 1 -- 172.21.15.62:0/3203070944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 msgr2=0x561f259db830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f5210843640 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 0x561f259db830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f5210843640 1 -- 172.21.15.62:0/3203070944 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f258de340 con 0x561f259d39a0 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.056+0000 7f520b7fe640 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 0x561f259db830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.057+0000 7f5210843640 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d39a0 0x561f259d3da0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5204009fa0 tx=0x7f520400f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7121e6070cb967e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.057+0000 7f52097fa640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51f40a3340 0x7f51f40a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.057+0000 7f52097fa640 1 -- 172.21.15.62:0/3079370667 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f51fc0f0250 con 0x7f520c100a50 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.058+0000 7f520bfff640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51f40a3340 0x7f51f40a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.058+0000 7f51f27fc640 1 -- 172.21.15.62:0/3203070944 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204015070 con 0x561f259d39a0 2024-09-15T23:26:02.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.058+0000 7f51f27fc640 1 -- 172.21.15.62:0/3203070944 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5204011be0 con 0x561f259d39a0 2024-09-15T23:26:02.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.058+0000 7f51f27fc640 1 -- 172.21.15.62:0/3203070944 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204010d90 con 0x561f259d39a0 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.059+0000 7f5215038740 1 -- 172.21.15.62:0/3203070944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d39a0 msgr2=0x561f259d3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.059+0000 7f5215038740 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d39a0 0x561f259d3da0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5204009fa0 tx=0x7f520400f7f0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.059+0000 7f520bfff640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51f40a3340 0x7f51f40a5800 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f51fc02a9f0 tx=0x7f51fc0b8040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.060+0000 7f5215038740 1 -- 172.21.15.62:0/3203070944 shutdown_connections 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.060+0000 7f5215038740 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 0x561f259db830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.060+0000 7f5215038740 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d42e0 0x561f259d8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.060+0000 7f5215038740 1 --2- 172.21.15.62:0/3203070944 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d39a0 0x561f259d3da0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.060+0000 7f5215038740 1 -- 172.21.15.62:0/3203070944 >> 172.21.15.62:0/3203070944 conn(0x561f259cf010 msgr2=0x561f259cf410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.060+0000 7f5215038740 1 -- 172.21.15.62:0/3203070944 shutdown_connections 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.060+0000 7f5215038740 1 -- 172.21.15.62:0/3203070944 wait complete. 2024-09-15T23:26:02.063 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f5215038740 1 -- start start 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f5215038740 1 Processor -- start 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f5215038740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259ddc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f5215038740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d42e0 0x561f259de1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f5215038740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d9370 0x561f259e5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f5215038740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561f259dc390 con 0x561f259d42e0 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f5215038740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561f258c0290 con 0x561f259d39a0 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f5215038740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561f259dc510 con 0x561f259d9370 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f5215038740 10 client.0 _ll_register_callbacks cb 0x561f258f1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.061+0000 7f520b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d42e0 0x561f259de1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f5210843640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259ddc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f520b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d42e0 0x561f259de1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35892/0 (socket says 172.21.15.62:35892) 2024-09-15T23:26:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f520b7fe640 1 -- 172.21.15.62:0/2980392202 learned_addr learned my addr 172.21.15.62:0/2980392202 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f5210843640 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d9370 0x561f259e5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f5210843640 1 -- 172.21.15.62:0/2980392202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d9370 msgr2=0x561f259e5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f5210843640 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d9370 0x561f259e5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f5210843640 1 -- 172.21.15.62:0/2980392202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d42e0 msgr2=0x561f259de1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f5210843640 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d42e0 0x561f259de1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f5210843640 1 -- 172.21.15.62:0/2980392202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f2564a210 con 0x561f259d39a0 2024-09-15T23:26:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.062+0000 7f520b7fe640 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d42e0 0x561f259de1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.064+0000 7f5210843640 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259ddc90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5204009fa0 tx=0x7f5204009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2c3df47c7e2d550 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.064+0000 7f51eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:26:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.064+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2980392202 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204015040 con 0x561f259d39a0 2024-09-15T23:26:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.064+0000 7f5215038740 1 -- 172.21.15.62:0/2980392202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5204011be0 con 0x561f259d39a0 2024-09-15T23:26:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.064+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2980392202 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5204010b10 con 0x561f259d39a0 2024-09-15T23:26:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.065+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2980392202 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5204030050 con 0x561f259d39a0 2024-09-15T23:26:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.065+0000 7f5215038740 10 client.27284 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.065+0000 7f5215038740 1 -- 172.21.15.62:0/2980392202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561f25716620 con 0x561f259d39a0 2024-09-15T23:26:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.066+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2980392202 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5204011be0 con 0x561f259d39a0 2024-09-15T23:26:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.066+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2980392202 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f5204028020 con 0x561f259d39a0 2024-09-15T23:26:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.068+0000 7f5215038740 10 client.27284 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.068+0000 7f5215038740 10 client.27284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.068+0000 7f5215038740 20 client.27284 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.068+0000 7f5215038740 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561f259e96a0 0x561f259eba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.068+0000 7f5215038740 4 client.27284 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.068+0000 7f5215038740 1 -- 172.21.15.62:0/2980392202 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f25944a20 con 0x561f259e96a0 2024-09-15T23:26:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.068+0000 7f520b7fe640 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561f259e96a0 0x561f259eba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.069+0000 7f520b7fe640 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561f259e96a0 0x561f259eba90 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=443d12cefa48a96e server_cookie=2d572975b569ce45 in_seq=0 out_seq=0 2024-09-15T23:26:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.069+0000 7f51eb7fe640 10 client.27284 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:02.071 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.069+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2980392202 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561f25944a20 con 0x561f259e96a0 2024-09-15T23:26:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.069+0000 7f51eb7fe640 10 client.27284 handle_command_reply: tid=0 2024-09-15T23:26:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 client.27284 shutdown 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 -- 172.21.15.62:0/2980392202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 msgr2=0x561f259ddc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259ddc90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5204009fa0 tx=0x7f5204009680 comp rx=0 tx=0).stop 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 -- 172.21.15.62:0/2980392202 shutdown_connections 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561f259e96a0 0x561f259eba90 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d9370 0x561f259e5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d42e0 0x561f259de1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 --2- 172.21.15.62:0/2980392202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259ddc90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f51eb7fe640 0 client.27284 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 -- 172.21.15.62:0/2980392202 >> 172.21.15.62:0/2980392202 conn(0x561f259cf010 msgr2=0x561f259d0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 -- 172.21.15.62:0/2980392202 shutdown_connections 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 1 -- 172.21.15.62:0/2980392202 wait complete. 2024-09-15T23:26:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.070+0000 7f5215038740 20 client.27284 trim_cache size 0 max 16384 2024-09-15T23:26:02.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.097+0000 7f5215038740 1 Processor -- start 2024-09-15T23:26:02.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.097+0000 7f5215038740 1 -- start start 2024-09-15T23:26:02.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.097+0000 7f5215038740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259d0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.097+0000 7f5215038740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d42e0 0x561f259e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5215038740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 0x561f259d0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5210843640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5210843640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259d0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39008/0 (socket says 172.21.15.62:39008) 2024-09-15T23:26:02.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5210843640 1 -- 172.21.15.62:0/4278884121 learned_addr learned my addr 172.21.15.62:0/4278884121 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:02.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5215038740 1 -- 172.21.15.62:0/4278884121 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1b3630 con 0x561f259d9370 2024-09-15T23:26:02.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5215038740 1 -- 172.21.15.62:0/4278884121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1b0f00 con 0x561f259d39a0 2024-09-15T23:26:02.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5215038740 1 -- 172.21.15.62:0/4278884121 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1aed50 con 0x561f259d42e0 2024-09-15T23:26:02.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5210843640 1 --2- 172.21.15.62:0/4278884121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 0x561f259d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5210843640 1 -- 172.21.15.62:0/4278884121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d42e0 msgr2=0x561f259e51d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:02.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5210843640 1 --2- 172.21.15.62:0/4278884121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d42e0 0x561f259e51d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5210843640 1 -- 172.21.15.62:0/4278884121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 msgr2=0x561f259d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:02.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5210843640 1 --2- 172.21.15.62:0/4278884121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 0x561f259d0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.098+0000 7f5210843640 1 -- 172.21.15.62:0/4278884121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f259ebe00 con 0x561f259d39a0 2024-09-15T23:26:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.099+0000 7f5210843640 1 --2- 172.21.15.62:0/4278884121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259d0400 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5204004fd0 tx=0x7f520400f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf7da33644d97694 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.099+0000 7f51f1ffb640 1 -- 172.21.15.62:0/4278884121 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204015040 con 0x561f259d39a0 2024-09-15T23:26:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.099+0000 7f51f1ffb640 1 -- 172.21.15.62:0/4278884121 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5204005b30 con 0x561f259d39a0 2024-09-15T23:26:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.099+0000 7f51f1ffb640 1 -- 172.21.15.62:0/4278884121 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204011a50 con 0x561f259d39a0 2024-09-15T23:26:02.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.101+0000 7f5215038740 1 -- 172.21.15.62:0/4278884121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 msgr2=0x561f259d0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.101+0000 7f5215038740 1 --2- 172.21.15.62:0/4278884121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259d0400 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5204004fd0 tx=0x7f520400f7f0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.102+0000 7f5215038740 1 -- 172.21.15.62:0/4278884121 shutdown_connections 2024-09-15T23:26:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.102+0000 7f5215038740 1 --2- 172.21.15.62:0/4278884121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f259d9370 0x561f259d0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.102+0000 7f5215038740 1 --2- 172.21.15.62:0/4278884121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d42e0 0x561f259e51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.102+0000 7f5215038740 1 --2- 172.21.15.62:0/4278884121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259d0400 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.102+0000 7f5215038740 1 -- 172.21.15.62:0/4278884121 >> 172.21.15.62:0/4278884121 conn(0x561f259cf010 msgr2=0x561f259ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:02.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.107+0000 7f5215038740 1 -- 172.21.15.62:0/4278884121 shutdown_connections 2024-09-15T23:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.110+0000 7f5215038740 1 -- 172.21.15.62:0/4278884121 wait complete. 2024-09-15T23:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.111+0000 7f5215038740 1 -- start start 2024-09-15T23:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.111+0000 7f5215038740 1 Processor -- start 2024-09-15T23:26:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.115+0000 7f5215038740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259e1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.115+0000 7f5215038740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d9370 0x561f259e24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.115+0000 7f5215038740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f25a612f0 0x561f25a636e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.115+0000 7f5215038740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1af3c0 con 0x561f25a612f0 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.115+0000 7f5215038740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1b0830 con 0x561f259d39a0 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.115+0000 7f5215038740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1afa90 con 0x561f259d9370 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.115+0000 7f5215038740 10 client.0 _ll_register_callbacks cb 0x7f51fc0f3b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.116+0000 7f5210843640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f25a612f0 0x561f25a636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.116+0000 7f5210843640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f25a612f0 0x561f25a636e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35910/0 (socket says 172.21.15.62:35910) 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.116+0000 7f5210843640 1 -- 172.21.15.62:0/2621860781 learned_addr learned my addr 172.21.15.62:0/2621860781 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.116+0000 7f5210843640 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259e1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.117+0000 7f5210843640 1 -- 172.21.15.62:0/2621860781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d9370 msgr2=0x561f259e24a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:02.119 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.117+0000 7f5210843640 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d9370 0x561f259e24a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.117+0000 7f5210843640 1 -- 172.21.15.62:0/2621860781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 msgr2=0x561f259e1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.117+0000 7f5210843640 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259e1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.117+0000 7f5210843640 1 -- 172.21.15.62:0/2621860781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f25a74c30 con 0x561f25a612f0 2024-09-15T23:26:02.122 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.120+0000 7f5210843640 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f25a612f0 0x561f25a636e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x561f259e1490 tx=0x7f5204013fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fbc9b45b031bc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:02.123 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.120+0000 7f51eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:02.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.121+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2621860781 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52040117e0 con 0x561f25a612f0 2024-09-15T23:26:02.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.121+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2621860781 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5204025040 con 0x561f25a612f0 2024-09-15T23:26:02.126 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.123+0000 7f5215038740 1 -- 172.21.15.62:0/2621860781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5204005b30 con 0x561f25a612f0 2024-09-15T23:26:02.129 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.128+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2621860781 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5204011a50 con 0x561f25a612f0 2024-09-15T23:26:02.131 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.129+0000 7f5215038740 10 client.17724 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:02.131 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.129+0000 7f5215038740 1 -- 172.21.15.62:0/2621860781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561f25a75dc0 con 0x561f25a612f0 2024-09-15T23:26:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.132+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2621860781 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5204005b30 con 0x561f25a612f0 2024-09-15T23:26:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.132+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2621860781 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f5204028040 con 0x561f25a612f0 2024-09-15T23:26:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.134+0000 7f5215038740 10 client.17724 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.134+0000 7f5215038740 10 client.17724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.134+0000 7f5215038740 20 client.17724 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.134+0000 7f5215038740 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561f25a76720 0x561f25a78b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.134+0000 7f5215038740 4 client.17724 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:02.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.134+0000 7f5215038740 1 -- 172.21.15.62:0/2621860781 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f2593ed70 con 0x561f25a76720 2024-09-15T23:26:02.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.139+0000 7f520b7fe640 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561f25a76720 0x561f25a78b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:02.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.155+0000 7f520b7fe640 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561f25a76720 0x561f25a78b40 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e76018b58e6fc46f server_cookie=34b5b26136ce2e0d in_seq=0 out_seq=0 2024-09-15T23:26:02.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.156+0000 7f51eb7fe640 10 client.17724 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f51eb7fe640 1 -- 172.21.15.62:0/2621860781 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f2593ed70 con 0x561f25a76720 2024-09-15T23:26:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f51eb7fe640 10 client.17724 handle_command_reply: tid=0 2024-09-15T23:26:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 client.17724 shutdown 2024-09-15T23:26:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 -- 172.21.15.62:0/2621860781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f25a612f0 msgr2=0x561f25a636e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f25a612f0 0x561f25a636e0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x561f259e1490 tx=0x7f5204013fc0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 -- 172.21.15.62:0/2621860781 shutdown_connections 2024-09-15T23:26:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561f25a76720 0x561f25a78b40 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561f25a612f0 0x561f25a636e0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561f259d9370 0x561f259e24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 --2- 172.21.15.62:0/2621860781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561f259d39a0 0x561f259e1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 -- 172.21.15.62:0/2621860781 >> 172.21.15.62:0/2621860781 conn(0x561f259cf010 msgr2=0x561f259cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f51eb7fe640 0 client.17724 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 -- 172.21.15.62:0/2621860781 shutdown_connections 2024-09-15T23:26:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.620+0000 7f5215038740 1 -- 172.21.15.62:0/2621860781 wait complete. 2024-09-15T23:26:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.621+0000 7f5215038740 20 client.17724 trim_cache size 0 max 16384 2024-09-15T23:26:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:26:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:26:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:26:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:26:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.621+0000 7f52122a6640 1 -- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51f40a3340 msgr2=0x7f51f40a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.621+0000 7f52122a6640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51f40a3340 0x7f51f40a5800 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f51fc02a9f0 tx=0x7f51fc0b8040 comp rx=0 tx=0).stop 2024-09-15T23:26:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.621+0000 7f52122a6640 1 -- 172.21.15.62:0/3079370667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 msgr2=0x7f520c10f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.621+0000 7f52122a6640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 0x7f520c10f650 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f51fc031610 tx=0x7f51fc032c80 comp rx=0 tx=0).stop 2024-09-15T23:26:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.622+0000 7f52122a6640 1 -- 172.21.15.62:0/3079370667 shutdown_connections 2024-09-15T23:26:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.622+0000 7f52122a6640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51f40a3340 0x7f51f40a5800 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.622+0000 7f52122a6640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f520c102600 0x7f520c10c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.622+0000 7f52122a6640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f520c101c40 0x7f520c10fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.622+0000 7f52122a6640 1 --2- 172.21.15.62:0/3079370667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f520c100a50 0x7f520c10f650 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.622+0000 7f52122a6640 1 -- 172.21.15.62:0/3079370667 >> 172.21.15.62:0/3079370667 conn(0x7f520c0fc1b0 msgr2=0x7f520c104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.622+0000 7f52122a6640 1 -- 172.21.15.62:0/3079370667 shutdown_connections 2024-09-15T23:26:02.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:02.622+0000 7f52122a6640 1 -- 172.21.15.62:0/3079370667 wait complete. 2024-09-15T23:26:02.640 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:26:03.076 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:03.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.696+0000 7fddeaff0640 1 -- 172.21.15.62:0/492782094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde4109830 msgr2=0x7fdde4109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.696+0000 7fddeaff0640 1 --2- 172.21.15.62:0/492782094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde4109830 0x7fdde4109c90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fddd4009900 tx=0x7fddd402a440 comp rx=0 tx=0).stop 2024-09-15T23:26:03.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.697+0000 7fddeaff0640 1 -- 172.21.15.62:0/492782094 shutdown_connections 2024-09-15T23:26:03.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.697+0000 7fddeaff0640 1 --2- 172.21.15.62:0/492782094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdde411b140 0x7fdde411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.697+0000 7fddeaff0640 1 --2- 172.21.15.62:0/492782094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde4109830 0x7fdde4109c90 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.697+0000 7fddeaff0640 1 --2- 172.21.15.62:0/492782094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde4109450 0x7fdde4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.698+0000 7fddeaff0640 1 -- 172.21.15.62:0/492782094 >> 172.21.15.62:0/492782094 conn(0x7fdde406e140 msgr2=0x7fdde406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.700+0000 7fddeaff0640 1 -- 172.21.15.62:0/492782094 shutdown_connections 2024-09-15T23:26:03.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.704+0000 7fddeaff0640 1 -- 172.21.15.62:0/492782094 wait complete. 2024-09-15T23:26:03.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.704+0000 7fddeaff0640 1 Processor -- start 2024-09-15T23:26:03.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.705+0000 7fddeaff0640 1 -- start start 2024-09-15T23:26:03.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.705+0000 7fddeaff0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde4109450 0x7fdde411a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.705+0000 7fddeaff0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdde4109830 0x7fdde411ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.706+0000 7fddeaff0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde411b140 0x7fdde41138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.706+0000 7fddeaff0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdde4120e30 con 0x7fdde4109450 2024-09-15T23:26:03.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.706+0000 7fddeaff0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdde4120cb0 con 0x7fdde411b140 2024-09-15T23:26:03.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.706+0000 7fddeaff0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdde4120fb0 con 0x7fdde4109830 2024-09-15T23:26:03.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.706+0000 7fdde3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdde4109830 0x7fdde411ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.706+0000 7fdde3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdde4109830 0x7fdde411ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:46798/0 (socket says 172.21.15.62:46798) 2024-09-15T23:26:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.706+0000 7fdde3fff640 1 -- 172.21.15.62:0/2582838088 learned_addr learned my addr 172.21.15.62:0/2582838088 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.707+0000 7fdde3fff640 1 -- 172.21.15.62:0/2582838088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde411b140 msgr2=0x7fdde41138b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:26:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.707+0000 7fdde3fff640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde411b140 0x7fdde41138b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.707+0000 7fdde3fff640 1 -- 172.21.15.62:0/2582838088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde4109450 msgr2=0x7fdde411a640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:26:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.707+0000 7fdde3fff640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde4109450 0x7fdde411a640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.707+0000 7fdde3fff640 1 -- 172.21.15.62:0/2582838088 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde4114170 con 0x7fdde4109830 2024-09-15T23:26:03.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.707+0000 7fdde3fff640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdde4109830 0x7fdde411ab80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fddd4006010 tx=0x7fddd4005000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.707+0000 7fdde1ffb640 1 -- 172.21.15.62:0/2582838088 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd4040030 con 0x7fdde4109830 2024-09-15T23:26:03.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.708+0000 7fddeaff0640 1 -- 172.21.15.62:0/2582838088 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde4114400 con 0x7fdde4109830 2024-09-15T23:26:03.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.708+0000 7fddeaff0640 1 -- 172.21.15.62:0/2582838088 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde4118390 con 0x7fdde4109830 2024-09-15T23:26:03.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.708+0000 7fdde1ffb640 1 -- 172.21.15.62:0/2582838088 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddd4005400 con 0x7fdde4109830 2024-09-15T23:26:03.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.708+0000 7fdde1ffb640 1 -- 172.21.15.62:0/2582838088 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd403c8e0 con 0x7fdde4109830 2024-09-15T23:26:03.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.713+0000 7fdded3c9740 1 Processor -- start 2024-09-15T23:26:03.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.713+0000 7fdded3c9740 1 -- start start 2024-09-15T23:26:03.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.714+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.714+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841aede0 0x5617841b9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.714+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841b96a0 0x5617841bba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.714+0000 7fdded3c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5617841142e0 con 0x5617841ae4c0 2024-09-15T23:26:03.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.714+0000 7fdded3c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5617840bbce0 con 0x5617841b96a0 2024-09-15T23:26:03.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.714+0000 7fdded3c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5617841102a0 con 0x5617841aede0 2024-09-15T23:26:03.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.715+0000 7fdde958d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.715+0000 7fdde958d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57882/0 (socket says 172.21.15.62:57882) 2024-09-15T23:26:03.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.715+0000 7fdde958d640 1 -- 172.21.15.62:0/1452897376 learned_addr learned my addr 172.21.15.62:0/1452897376 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:03.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.721+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/1452897376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841aede0 0x5617841b9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.722+0000 7fdde958d640 1 -- 172.21.15.62:0/1452897376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841aede0 msgr2=0x5617841b9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.722+0000 7fdde958d640 1 --2- 172.21.15.62:0/1452897376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841aede0 0x5617841b9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.722+0000 7fdde958d640 1 -- 172.21.15.62:0/1452897376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841b96a0 msgr2=0x5617841bba90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:03.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.722+0000 7fdde958d640 1 --2- 172.21.15.62:0/1452897376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841b96a0 0x5617841bba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.722+0000 7fdde958d640 1 -- 172.21.15.62:0/1452897376 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617840a0440 con 0x5617841ae4c0 2024-09-15T23:26:03.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.723+0000 7fdde1ffb640 1 -- 172.21.15.62:0/2582838088 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fddd4043030 con 0x7fdde4109830 2024-09-15T23:26:03.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.723+0000 7fdde958d640 1 --2- 172.21.15.62:0/1452897376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841ae8a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009e20 tx=0x7fdddc00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd1e962891d8d0c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.724+0000 7fdde1ffb640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fddcc0a3380 0x7fddcc0a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.724+0000 7fdde1ffb640 1 -- 172.21.15.62:0/2582838088 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fddd40f04e0 con 0x7fdde4109830 2024-09-15T23:26:03.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.728+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fddcc0a3380 0x7fddcc0a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.728+0000 7fddcaffd640 1 -- 172.21.15.62:0/1452897376 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc004d20 con 0x5617841ae4c0 2024-09-15T23:26:03.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.729+0000 7fddcaffd640 1 -- 172.21.15.62:0/1452897376 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdddc00ee70 con 0x5617841ae4c0 2024-09-15T23:26:03.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.729+0000 7fddcaffd640 1 -- 172.21.15.62:0/1452897376 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc010890 con 0x5617841ae4c0 2024-09-15T23:26:03.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.729+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fddcc0a3380 0x7fddcc0a5840 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fddd8006fd0 tx=0x7fddd8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.730+0000 7fdded3c9740 1 -- 172.21.15.62:0/1452897376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 msgr2=0x5617841ae8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.730+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1452897376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841ae8a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fdddc009e20 tx=0x7fdddc00e720 comp rx=0 tx=0).stop 2024-09-15T23:26:03.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:03 smithi062 ceph-mon[30949]: pgmap v663: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.5 MiB/s wr, 739 op/s 2024-09-15T23:26:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.731+0000 7fdded3c9740 1 -- 172.21.15.62:0/1452897376 shutdown_connections 2024-09-15T23:26:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.731+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1452897376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841b96a0 0x5617841bba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.731+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1452897376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841aede0 0x5617841b9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.731+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1452897376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841ae8a0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.731+0000 7fdded3c9740 1 -- 172.21.15.62:0/1452897376 >> 172.21.15.62:0/1452897376 conn(0x5617841aaa50 msgr2=0x5617841aae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.739+0000 7fdded3c9740 1 -- 172.21.15.62:0/1452897376 shutdown_connections 2024-09-15T23:26:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.740+0000 7fdded3c9740 1 -- 172.21.15.62:0/1452897376 wait complete. 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.740+0000 7fdded3c9740 1 -- start start 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.740+0000 7fdded3c9740 1 Processor -- start 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841aede0 0x5617841b6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841b96a0 0x5617841b6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdded3c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5617841bc270 con 0x5617841ae4c0 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdded3c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5617840cf050 con 0x5617841aede0 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdded3c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5617841bc3f0 con 0x5617841b96a0 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdded3c9740 10 client.0 _ll_register_callbacks cb 0x56178410f160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdde3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841aede0 0x5617841b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdde3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841aede0 0x5617841b6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49192/0 (socket says 172.21.15.62:49192) 2024-09-15T23:26:03.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.741+0000 7fdde3fff640 1 -- 172.21.15.62:0/1282347939 learned_addr learned my addr 172.21.15.62:0/1282347939 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:03.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.742+0000 7fdde3fff640 1 -- 172.21.15.62:0/1282347939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841b96a0 msgr2=0x5617841b6900 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:26:03.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.742+0000 7fdde3fff640 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841b96a0 0x5617841b6900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.742+0000 7fdde3fff640 1 -- 172.21.15.62:0/1282347939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 msgr2=0x5617841b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.742+0000 7fdde3fff640 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.742+0000 7fdde3fff640 1 -- 172.21.15.62:0/1282347939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561783e853c0 con 0x5617841aede0 2024-09-15T23:26:03.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.748+0000 7fdde3fff640 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841aede0 0x5617841b6310 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fddd40bc420 tx=0x7fddd40bc320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=216f90c24153c3ac server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.748+0000 7fddc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:26:03.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.754+0000 7fddc3fff640 1 -- 172.21.15.62:0/1282347939 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd404a070 con 0x5617841aede0 2024-09-15T23:26:03.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.754+0000 7fddc3fff640 1 -- 172.21.15.62:0/1282347939 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fddd4003fa0 con 0x5617841aede0 2024-09-15T23:26:03.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.755+0000 7fdded3c9740 1 -- 172.21.15.62:0/1282347939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdddc00ee70 con 0x5617841aede0 2024-09-15T23:26:03.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.759+0000 7fddc3fff640 1 -- 172.21.15.62:0/1282347939 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fddd40bc720 con 0x5617841aede0 2024-09-15T23:26:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.759+0000 7fdded3c9740 10 client.27293 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.759+0000 7fdded3c9740 1 -- 172.21.15.62:0/1282347939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5617841c7c90 con 0x5617841aede0 2024-09-15T23:26:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.763+0000 7fddc3fff640 1 -- 172.21.15.62:0/1282347939 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdddc00ee70 con 0x5617841aede0 2024-09-15T23:26:03.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.763+0000 7fddc3fff640 1 -- 172.21.15.62:0/1282347939 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fddd40bc9c0 con 0x5617841aede0 2024-09-15T23:26:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.763+0000 7fdded3c9740 10 client.27293 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.763+0000 7fdded3c9740 10 client.27293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.763+0000 7fdded3c9740 20 client.27293 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.763+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617841c9140 0x5617841cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.763+0000 7fdded3c9740 4 client.27293 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:03.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.763+0000 7fdded3c9740 1 -- 172.21.15.62:0/1282347939 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561784121850 con 0x5617841c9140 2024-09-15T23:26:03.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.768+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617841c9140 0x5617841cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.782+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617841c9140 0x5617841cb5a0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf2c36efe2e782a9 server_cookie=5e1bcc386d8b58f6 in_seq=0 out_seq=0 2024-09-15T23:26:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.783+0000 7fddc3fff640 10 client.27293 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.783+0000 7fddc3fff640 1 -- 172.21.15.62:0/1282347939 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561784121850 con 0x5617841c9140 2024-09-15T23:26:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.783+0000 7fddc3fff640 10 client.27293 handle_command_reply: tid=0 2024-09-15T23:26:03.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.784+0000 7fdded3c9740 1 client.27293 shutdown 2024-09-15T23:26:03.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.790+0000 7fdded3c9740 1 -- 172.21.15.62:0/1282347939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841aede0 msgr2=0x5617841b6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.790+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841aede0 0x5617841b6310 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fddd40bc420 tx=0x7fddd40bc320 comp rx=0 tx=0).stop 2024-09-15T23:26:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.791+0000 7fdded3c9740 1 -- 172.21.15.62:0/1282347939 shutdown_connections 2024-09-15T23:26:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.791+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5617841c9140 0x5617841cb5a0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.791+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841b96a0 0x5617841b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.791+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841aede0 0x5617841b6310 secure :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fddd40bc420 tx=0x7fddd40bc320 comp rx=0 tx=0).stop 2024-09-15T23:26:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.791+0000 7fdded3c9740 1 --2- 172.21.15.62:0/1282347939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.791+0000 7fddc3fff640 0 client.27293 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:03.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.791+0000 7fdded3c9740 1 -- 172.21.15.62:0/1282347939 >> 172.21.15.62:0/1282347939 conn(0x5617841aaa50 msgr2=0x5617841a9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.792+0000 7fdded3c9740 1 -- 172.21.15.62:0/1282347939 shutdown_connections 2024-09-15T23:26:03.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.796+0000 7fdded3c9740 1 -- 172.21.15.62:0/1282347939 wait complete. 2024-09-15T23:26:03.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.796+0000 7fdded3c9740 20 client.27293 trim_cache size 0 max 16384 2024-09-15T23:26:03.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.834+0000 7fdded3c9740 1 Processor -- start 2024-09-15T23:26:03.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.834+0000 7fdded3c9740 1 -- start start 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.835+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.835+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841b96a0 0x5617841b1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.835+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841d1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.835+0000 7fdded3c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdde407fdd0 con 0x5617841ae4c0 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.835+0000 7fdded3c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdde407d6a0 con 0x5617841d0e70 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.835+0000 7fdded3c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdde4109cd0 con 0x5617841b96a0 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.835+0000 7fdde958d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841d1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.835+0000 7fdde958d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841d1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49200/0 (socket says 172.21.15.62:49200) 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.835+0000 7fdde958d640 1 -- 172.21.15.62:0/2854975701 learned_addr learned my addr 172.21.15.62:0/2854975701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:03.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.836+0000 7fdde958d640 1 -- 172.21.15.62:0/2854975701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841b96a0 msgr2=0x5617841b1090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:03.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.837+0000 7fdde958d640 1 --2- 172.21.15.62:0/2854975701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841b96a0 0x5617841b1090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.837+0000 7fdde958d640 1 -- 172.21.15.62:0/2854975701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 msgr2=0x5617841c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.837+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/2854975701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.841+0000 7fdde958d640 1 --2- 172.21.15.62:0/2854975701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841c54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.841+0000 7fdde958d640 1 -- 172.21.15.62:0/2854975701 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617841abbc0 con 0x5617841d0e70 2024-09-15T23:26:03.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.845+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/2854975701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:03.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.846+0000 7fdde958d640 1 --2- 172.21.15.62:0/2854975701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841d1300 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdddc006010 tx=0x7fdddc009ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a4e1cdf935c1a55 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.846+0000 7fddca7fc640 1 -- 172.21.15.62:0/2854975701 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc025070 con 0x5617841d0e70 2024-09-15T23:26:03.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.846+0000 7fddca7fc640 1 -- 172.21.15.62:0/2854975701 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdddc003e50 con 0x5617841d0e70 2024-09-15T23:26:03.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.846+0000 7fddca7fc640 1 -- 172.21.15.62:0/2854975701 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc010690 con 0x5617841d0e70 2024-09-15T23:26:03.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.848+0000 7fdded3c9740 1 -- 172.21.15.62:0/2854975701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 msgr2=0x5617841d1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.848+0000 7fdded3c9740 1 --2- 172.21.15.62:0/2854975701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841d1300 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdddc006010 tx=0x7fdddc009ac0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.848+0000 7fdded3c9740 1 -- 172.21.15.62:0/2854975701 shutdown_connections 2024-09-15T23:26:03.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.848+0000 7fdded3c9740 1 --2- 172.21.15.62:0/2854975701 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841d1300 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.848+0000 7fdded3c9740 1 --2- 172.21.15.62:0/2854975701 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841b96a0 0x5617841b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.848+0000 7fdded3c9740 1 --2- 172.21.15.62:0/2854975701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841ae4c0 0x5617841c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.848+0000 7fdded3c9740 1 -- 172.21.15.62:0/2854975701 >> 172.21.15.62:0/2854975701 conn(0x5617841aaa50 msgr2=0x5617841a9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.850+0000 7fdded3c9740 1 -- 172.21.15.62:0/2854975701 shutdown_connections 2024-09-15T23:26:03.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.854+0000 7fdded3c9740 1 -- 172.21.15.62:0/2854975701 wait complete. 2024-09-15T23:26:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.854+0000 7fdded3c9740 1 -- start start 2024-09-15T23:26:03.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.854+0000 7fdded3c9740 1 Processor -- start 2024-09-15T23:26:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.858+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841ae4c0 0x5617841bd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.858+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841b96a0 0x5617841bdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.858+0000 7fdded3c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841be0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.858+0000 7fdded3c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdde407bb90 con 0x5617841b96a0 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.858+0000 7fdded3c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdde407cfd0 con 0x5617841d0e70 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.858+0000 7fdded3c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdde407c230 con 0x5617841ae4c0 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.858+0000 7fdded3c9740 10 client.0 _ll_register_callbacks cb 0x7fddd40bc4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.859+0000 7fdde958d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841be0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.859+0000 7fdde3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841b96a0 0x5617841bdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.859+0000 7fdde3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841b96a0 0x5617841bdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57914/0 (socket says 172.21.15.62:57914) 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.859+0000 7fdde3fff640 1 -- 172.21.15.62:0/3987968675 learned_addr learned my addr 172.21.15.62:0/3987968675 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.859+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841ae4c0 0x5617841bd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.860+0000 7fdde8d8c640 1 -- 172.21.15.62:0/3987968675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 msgr2=0x5617841be0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.860+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841be0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.860+0000 7fdde8d8c640 1 -- 172.21.15.62:0/3987968675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841b96a0 msgr2=0x5617841bdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.860+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841b96a0 0x5617841bdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.860+0000 7fdde8d8c640 1 -- 172.21.15.62:0/3987968675 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617842330e0 con 0x5617841ae4c0 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.861+0000 7fdde8d8c640 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841ae4c0 0x5617841bd640 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5617841c9520 tx=0x7fddd800cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4ea9f733b6dad30 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.861+0000 7fddc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.861+0000 7fddc3fff640 1 -- 172.21.15.62:0/3987968675 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddd8019040 con 0x5617841ae4c0 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.861+0000 7fddc3fff640 1 -- 172.21.15.62:0/3987968675 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fddd8013ad0 con 0x5617841ae4c0 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.861+0000 7fdded3c9740 1 -- 172.21.15.62:0/3987968675 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdddc003e50 con 0x5617841ae4c0 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.864+0000 7fddc3fff640 1 -- 172.21.15.62:0/3987968675 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fddd8035050 con 0x5617841ae4c0 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.864+0000 7fdded3c9740 10 client.27055 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.864+0000 7fdded3c9740 1 -- 172.21.15.62:0/3987968675 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5617842343a0 con 0x5617841ae4c0 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.864+0000 7fddc3fff640 1 -- 172.21.15.62:0/3987968675 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdddc003e50 con 0x5617841ae4c0 2024-09-15T23:26:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.864+0000 7fddc3fff640 1 -- 172.21.15.62:0/3987968675 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fddd80273f0 con 0x5617841ae4c0 2024-09-15T23:26:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.864+0000 7fdded3c9740 10 client.27055 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.865+0000 7fdded3c9740 10 client.27055 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.865+0000 7fdded3c9740 20 client.27055 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.865+0000 7fdded3c9740 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561784234b10 0x561784236f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.865+0000 7fdded3c9740 4 client.27055 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:26:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.865+0000 7fdded3c9740 1 -- 172.21.15.62:0/3987968675 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56178411bd60 con 0x561784234b10 2024-09-15T23:26:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.865+0000 7fdde3fff640 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561784234b10 0x561784236f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.866+0000 7fddc3fff640 10 client.27055 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.866+0000 7fdde3fff640 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561784234b10 0x561784236f00 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5cabf32cc762c00 server_cookie=67b4cf92807daa78 in_seq=0 out_seq=0 2024-09-15T23:26:03.879 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:26:03.879 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:26:03.879 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:26:03.879 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:26:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.870+0000 7fddc3fff640 1 -- 172.21.15.62:0/3987968675 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56178411bd60 con 0x561784234b10 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.870+0000 7fddc3fff640 10 client.27055 handle_command_reply: tid=0 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.870+0000 7fdded3c9740 1 client.27055 shutdown 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.870+0000 7fdded3c9740 1 -- 172.21.15.62:0/3987968675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841ae4c0 msgr2=0x5617841bd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.870+0000 7fdded3c9740 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841ae4c0 0x5617841bd640 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5617841c9520 tx=0x7fddd800cd10 comp rx=0 tx=0).stop 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.871+0000 7fdded3c9740 1 -- 172.21.15.62:0/3987968675 shutdown_connections 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.871+0000 7fdded3c9740 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561784234b10 0x561784236f00 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.871+0000 7fdded3c9740 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5617841d0e70 0x5617841be0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.871+0000 7fdded3c9740 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617841b96a0 0x5617841bdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.871+0000 7fddc3fff640 0 client.27055 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.871+0000 7fdded3c9740 1 --2- 172.21.15.62:0/3987968675 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5617841ae4c0 0x5617841bd640 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.871+0000 7fdded3c9740 1 -- 172.21.15.62:0/3987968675 >> 172.21.15.62:0/3987968675 conn(0x5617841aaa50 msgr2=0x5617841d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.871+0000 7fdded3c9740 1 -- 172.21.15.62:0/3987968675 shutdown_connections 2024-09-15T23:26:03.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.871+0000 7fdded3c9740 1 -- 172.21.15.62:0/3987968675 wait complete. 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.872+0000 7fdded3c9740 20 client.27055 trim_cache size 0 max 16384 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 -- 172.21.15.62:0/2582838088 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fddcc0a3380 msgr2=0x7fddcc0a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fddcc0a3380 0x7fddcc0a5840 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fddd8006fd0 tx=0x7fddd8008040 comp rx=0 tx=0).stop 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 -- 172.21.15.62:0/2582838088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdde4109830 msgr2=0x7fdde411ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdde4109830 0x7fdde411ab80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fddd4006010 tx=0x7fddd4005000 comp rx=0 tx=0).stop 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 -- 172.21.15.62:0/2582838088 shutdown_connections 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fddcc0a3380 0x7fddcc0a5840 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdde411b140 0x7fdde41138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdde4109830 0x7fdde411ab80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 --2- 172.21.15.62:0/2582838088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdde4109450 0x7fdde411a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 -- 172.21.15.62:0/2582838088 >> 172.21.15.62:0/2582838088 conn(0x7fdde406e140 msgr2=0x7fdde40722b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 -- 172.21.15.62:0/2582838088 shutdown_connections 2024-09-15T23:26:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:03.873+0000 7fddeaff0640 1 -- 172.21.15.62:0/2582838088 wait complete. 2024-09-15T23:26:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:03 smithi067 ceph-mon[35665]: pgmap v663: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.5 MiB/s wr, 739 op/s 2024-09-15T23:26:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:03 smithi148 ceph-mon[35610]: pgmap v663: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.5 MiB/s wr, 739 op/s 2024-09-15T23:26:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 585.0 s, 790.6 tps, lat 126.189 ms stddev 136.180 2024-09-15T23:26:05.535 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:26:05.536 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:26:05.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:05 smithi062 ceph-mon[30949]: pgmap v664: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.0 MiB/s wr, 681 op/s 2024-09-15T23:26:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:05 smithi067 ceph-mon[35665]: pgmap v664: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.0 MiB/s wr, 681 op/s 2024-09-15T23:26:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:05 smithi148 ceph-mon[35610]: pgmap v664: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.0 MiB/s wr, 681 op/s 2024-09-15T23:26:06.063 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:07.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.570+0000 7fc326b0a640 1 -- 172.21.15.62:0/76796368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc320101450 msgr2=0x7fc320105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.570+0000 7fc326b0a640 1 --2- 172.21.15.62:0/76796368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc320101450 0x7fc320105920 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc31801acb0 tx=0x7fc31803b9b0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.575+0000 7fc326b0a640 1 -- 172.21.15.62:0/76796368 shutdown_connections 2024-09-15T23:26:07.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.575+0000 7fc326b0a640 1 --2- 172.21.15.62:0/76796368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320105f40 0x7fc3201063d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.575+0000 7fc326b0a640 1 --2- 172.21.15.62:0/76796368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc320101450 0x7fc320105920 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.575+0000 7fc326b0a640 1 --2- 172.21.15.62:0/76796368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc320100b30 0x7fc320100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.575+0000 7fc326b0a640 1 -- 172.21.15.62:0/76796368 >> 172.21.15.62:0/76796368 conn(0x7fc3200fc7a0 msgr2=0x7fc3200febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:07.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.576+0000 7fc326b0a640 1 -- 172.21.15.62:0/76796368 shutdown_connections 2024-09-15T23:26:07.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.576+0000 7fc326b0a640 1 -- 172.21.15.62:0/76796368 wait complete. 2024-09-15T23:26:07.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.576+0000 7fc326b0a640 1 Processor -- start 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc326b0a640 1 -- start start 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc326b0a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc320100b30 0x7fc3200738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc326b0a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320101450 0x7fc320073e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc326b0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc320105f40 0x7fc3201151e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc326b0a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc320109910 con 0x7fc320105f40 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc326b0a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc320109790 con 0x7fc320101450 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc326b0a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc320109a90 con 0x7fc320100b30 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc31ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320101450 0x7fc320073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc31ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320101450 0x7fc320073e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49228/0 (socket says 172.21.15.62:49228) 2024-09-15T23:26:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc31ffff640 1 -- 172.21.15.62:0/4026183299 learned_addr learned my addr 172.21.15.62:0/4026183299 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:07.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.577+0000 7fc3248a6640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc320100b30 0x7fc3200738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.580+0000 7fc3248a6640 1 -- 172.21.15.62:0/4026183299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320101450 msgr2=0x7fc320073e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.580+0000 7fc3248a6640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320101450 0x7fc320073e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.580+0000 7fc3248a6640 1 -- 172.21.15.62:0/4026183299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc320105f40 msgr2=0x7fc3201151e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:07.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.580+0000 7fc3248a6640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc320105f40 0x7fc3201151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:07.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.580+0000 7fc3248a6640 1 -- 172.21.15.62:0/4026183299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc320071eb0 con 0x7fc320100b30 2024-09-15T23:26:07.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.584+0000 7fc3248a6640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc320100b30 0x7fc3200738e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc31000b3f0 tx=0x7fc31000b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.588+0000 7fc31dffb640 1 -- 172.21.15.62:0/4026183299 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310015020 con 0x7fc320100b30 2024-09-15T23:26:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.588+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3200721a0 con 0x7fc320100b30 2024-09-15T23:26:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.588+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3200726e0 con 0x7fc320100b30 2024-09-15T23:26:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.589+0000 7fc31dffb640 1 -- 172.21.15.62:0/4026183299 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc310011d10 con 0x7fc320100b30 2024-09-15T23:26:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.589+0000 7fc31dffb640 1 -- 172.21.15.62:0/4026183299 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31000f2e0 con 0x7fc320100b30 2024-09-15T23:26:07.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.591+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc320105920 con 0x7fc320100b30 2024-09-15T23:26:07.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.596+0000 7fc31dffb640 1 -- 172.21.15.62:0/4026183299 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc310018020 con 0x7fc320100b30 2024-09-15T23:26:07.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.597+0000 7fc31dffb640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc3080a3450 0x7fc3080a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:07.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.597+0000 7fc31dffb640 1 -- 172.21.15.62:0/4026183299 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc3100cf780 con 0x7fc320100b30 2024-09-15T23:26:07.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.598+0000 7fc31ffff640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc3080a3450 0x7fc3080a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:07.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.599+0000 7fc31ffff640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc3080a3450 0x7fc3080a5910 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc320073480 tx=0x7fc31804b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:07.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:07.602+0000 7fc31dffb640 1 -- 172.21.15.62:0/4026183299 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc31001f020 con 0x7fc320100b30 2024-09-15T23:26:07.735 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:07 smithi062 ceph-mon[30949]: pgmap v665: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.3 MiB/s wr, 728 op/s 2024-09-15T23:26:08.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.017+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc32019a9a0 con 0x7fc320100b30 2024-09-15T23:26:08.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.022+0000 7fc31dffb640 1 -- 172.21.15.62:0/4026183299 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7fc31009c140 con 0x7fc320100b30 2024-09-15T23:26:08.023 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:26:08.024 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:26:08.024 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:26:08.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:07 smithi067 ceph-mon[35665]: pgmap v665: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.3 MiB/s wr, 728 op/s 2024-09-15T23:26:08.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.029+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc3080a3450 msgr2=0x7fc3080a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:08.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.029+0000 7fc326b0a640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc3080a3450 0x7fc3080a5910 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc320073480 tx=0x7fc31804b040 comp rx=0 tx=0).stop 2024-09-15T23:26:08.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.029+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc320100b30 msgr2=0x7fc3200738e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:08.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.029+0000 7fc326b0a640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc320100b30 0x7fc3200738e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc31000b3f0 tx=0x7fc31000b8b0 comp rx=0 tx=0).stop 2024-09-15T23:26:08.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.030+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 shutdown_connections 2024-09-15T23:26:08.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.030+0000 7fc326b0a640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc3080a3450 0x7fc3080a5910 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:08.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.030+0000 7fc326b0a640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc320105f40 0x7fc3201151e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:08.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.030+0000 7fc326b0a640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320101450 0x7fc320073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:08.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.030+0000 7fc326b0a640 1 --2- 172.21.15.62:0/4026183299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc320100b30 0x7fc3200738e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:08.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.030+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 >> 172.21.15.62:0/4026183299 conn(0x7fc3200fc7a0 msgr2=0x7fc32010a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:08.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.032+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 shutdown_connections 2024-09-15T23:26:08.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:08.033+0000 7fc326b0a640 1 -- 172.21.15.62:0/4026183299 wait complete. 2024-09-15T23:26:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:07 smithi148 ceph-mon[35610]: pgmap v665: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.3 MiB/s wr, 728 op/s 2024-09-15T23:26:08.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:08 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4026183299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:09.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:08 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4026183299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:09.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:08 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4026183299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:09.436 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:26:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 590.0 s, 814.2 tps, lat 120.021 ms stddev 138.336 2024-09-15T23:26:09.846 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:09.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:09 smithi062 ceph-mon[30949]: pgmap v666: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.0 MiB/s wr, 715 op/s 2024-09-15T23:26:10.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:09 smithi067 ceph-mon[35665]: pgmap v666: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.0 MiB/s wr, 715 op/s 2024-09-15T23:26:10.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:09 smithi148 ceph-mon[35610]: pgmap v666: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.0 MiB/s wr, 715 op/s 2024-09-15T23:26:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.466+0000 7fd3d82d7640 1 -- 172.21.15.62:0/2543225509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3d0102e00 msgr2=0x7fd3d01031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.466+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/2543225509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3d0102e00 0x7fd3d01031e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc009900 tx=0x7fd3cc02a480 comp rx=0 tx=0).stop 2024-09-15T23:26:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.466+0000 7fd3d82d7640 1 -- 172.21.15.62:0/2543225509 shutdown_connections 2024-09-15T23:26:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.466+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/2543225509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3d010c1a0 0x7fd3d010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.466+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/2543225509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3d0103720 0x7fd3d010bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.466+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/2543225509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3d0102e00 0x7fd3d01031e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.466+0000 7fd3d82d7640 1 -- 172.21.15.62:0/2543225509 >> 172.21.15.62:0/2543225509 conn(0x7fd3d00fc7a0 msgr2=0x7fd3d00febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.466+0000 7fd3d82d7640 1 -- 172.21.15.62:0/2543225509 shutdown_connections 2024-09-15T23:26:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.466+0000 7fd3d82d7640 1 -- 172.21.15.62:0/2543225509 wait complete. 2024-09-15T23:26:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d82d7640 1 Processor -- start 2024-09-15T23:26:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d82d7640 1 -- start start 2024-09-15T23:26:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d82d7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3d0102e00 0x7fd3d019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d82d7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3d0103720 0x7fd3d019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d82d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3d010c1a0 0x7fd3d019ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d82d7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d0111f70 con 0x7fd3d0102e00 2024-09-15T23:26:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d82d7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d0111df0 con 0x7fd3d010c1a0 2024-09-15T23:26:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d82d7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d01120f0 con 0x7fd3d0103720 2024-09-15T23:26:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d6073640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3d0102e00 0x7fd3d019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d6073640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3d0102e00 0x7fd3d019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57946/0 (socket says 172.21.15.62:57946) 2024-09-15T23:26:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d6073640 1 -- 172.21.15.62:0/4028156228 learned_addr learned my addr 172.21.15.62:0/4028156228 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d6874640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3d010c1a0 0x7fd3d019ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.467+0000 7fd3d5872640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3d0103720 0x7fd3d019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3d6073640 1 -- 172.21.15.62:0/4028156228 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3d0103720 msgr2=0x7fd3d019abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3d6073640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3d0103720 0x7fd3d019abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3d6073640 1 -- 172.21.15.62:0/4028156228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3d010c1a0 msgr2=0x7fd3d019ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3d6073640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3d010c1a0 0x7fd3d019ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3d6073640 1 -- 172.21.15.62:0/4028156228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d019f650 con 0x7fd3d0102e00 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3d6073640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3d0102e00 0x7fd3d019a6a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc006010 tx=0x7fd3cc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3c77fe640 1 -- 172.21.15.62:0/4028156228 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc03f030 con 0x7fd3d0102e00 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3c77fe640 1 -- 172.21.15.62:0/4028156228 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3cc004440 con 0x7fd3d0102e00 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3d82d7640 1 -- 172.21.15.62:0/4028156228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d019f8e0 con 0x7fd3d0102e00 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3c77fe640 1 -- 172.21.15.62:0/4028156228 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc035070 con 0x7fd3d0102e00 2024-09-15T23:26:10.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.468+0000 7fd3d82d7640 1 -- 172.21.15.62:0/4028156228 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d01a7220 con 0x7fd3d0102e00 2024-09-15T23:26:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.472+0000 7fd3da6b0740 1 Processor -- start 2024-09-15T23:26:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.473+0000 7fd3c77fe640 1 -- 172.21.15.62:0/4028156228 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd3cc042030 con 0x7fd3d0102e00 2024-09-15T23:26:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.473+0000 7fd3da6b0740 1 -- start start 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.474+0000 7fd3c77fe640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3bc0a3340 0x7fd3bc0a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.474+0000 7fd3c77fe640 1 -- 172.21.15.62:0/4028156228 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd3cc0effc0 con 0x7fd3d0102e00 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.475+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 0x5628f815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.475+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f815f5d0 0x5628f815fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.475+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f816c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.475+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5628f80c52f0 con 0x5628f816a370 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.475+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5628f806ccf0 con 0x5628f815f5d0 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.475+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5628f80c12b0 con 0x5628f8160b40 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.475+0000 7fd3d6874640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f816c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.475+0000 7fd3d6874640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f816c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57956/0 (socket says 172.21.15.62:57956) 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.475+0000 7fd3d6874640 1 -- 172.21.15.62:0/3315343212 learned_addr learned my addr 172.21.15.62:0/3315343212 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:10.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.476+0000 7fd3d6874640 1 -- 172.21.15.62:0/3315343212 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 msgr2=0x5628f815f090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.476+0000 7fd3d6874640 1 --2- 172.21.15.62:0/3315343212 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 0x5628f815f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.476+0000 7fd3d6874640 1 -- 172.21.15.62:0/3315343212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f815f5d0 msgr2=0x5628f815fa10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.476+0000 7fd3d6874640 1 --2- 172.21.15.62:0/3315343212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f815f5d0 0x5628f815fa10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.476+0000 7fd3d6874640 1 -- 172.21.15.62:0/3315343212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628f80513a0 con 0x5628f816a370 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.476+0000 7fd3d6874640 1 --2- 172.21.15.62:0/3315343212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f816c760 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009fd0 tx=0x7fd3c80119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41764a13f41aade1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.476+0000 7fd3c4dea640 1 -- 172.21.15.62:0/3315343212 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8003b70 con 0x5628f816a370 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.476+0000 7fd3c4dea640 1 -- 172.21.15.62:0/3315343212 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c8003d10 con 0x5628f816a370 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.476+0000 7fd3c4dea640 1 -- 172.21.15.62:0/3315343212 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c800fb60 con 0x5628f816a370 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.477+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3315343212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 msgr2=0x5628f816c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.477+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3315343212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f816c760 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009fd0 tx=0x7fd3c80119f0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.477+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3315343212 shutdown_connections 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.477+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3315343212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f816c760 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.477+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3315343212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f815f5d0 0x5628f815fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.477+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3315343212 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 0x5628f815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.478+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3315343212 >> 172.21.15.62:0/3315343212 conn(0x5628f8128310 msgr2=0x5628f815bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:10.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.478+0000 7fd3d6874640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3bc0a3340 0x7fd3bc0a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.478+0000 7fd3d6874640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3bc0a3340 0x7fd3bc0a5800 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800f640 tx=0x7fd3c8000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:10.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.484+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3315343212 shutdown_connections 2024-09-15T23:26:10.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.488+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3315343212 wait complete. 2024-09-15T23:26:10.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.488+0000 7fd3da6b0740 1 -- start start 2024-09-15T23:26:10.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.488+0000 7fd3da6b0740 1 Processor -- start 2024-09-15T23:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.492+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f815f5d0 0x5628f8169de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.492+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 0x5628f8167090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.492+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f81677d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.493+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5628f816d070 con 0x5628f816a370 2024-09-15T23:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.493+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5628f8080060 con 0x5628f815f5d0 2024-09-15T23:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.493+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5628f816d1f0 con 0x5628f8160b40 2024-09-15T23:26:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.493+0000 7fd3da6b0740 10 client.0 _ll_register_callbacks cb 0x5628f7dd38c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.493+0000 7fd3d6874640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f81677d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.493+0000 7fd3d6874640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f81677d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57966/0 (socket says 172.21.15.62:57966) 2024-09-15T23:26:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.493+0000 7fd3d6874640 1 -- 172.21.15.62:0/1767620611 learned_addr learned my addr 172.21.15.62:0/1767620611 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:10.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.496+0000 7fd3d6073640 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f815f5d0 0x5628f8169de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.498+0000 7fd3d5872640 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 0x5628f8167090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.501+0000 7fd3d5872640 1 -- 172.21.15.62:0/1767620611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f815f5d0 msgr2=0x5628f8169de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.501+0000 7fd3d5872640 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f815f5d0 0x5628f8169de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.501+0000 7fd3d5872640 1 -- 172.21.15.62:0/1767620611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 msgr2=0x5628f81677d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.501+0000 7fd3d5872640 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f81677d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.501+0000 7fd3d5872640 1 -- 172.21.15.62:0/1767620611 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628f7e363c0 con 0x5628f8160b40 2024-09-15T23:26:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.505+0000 7fd3d5872640 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 0x5628f8167090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00fec00 tx=0x7fd3c0011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13612f58d5529042 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.507+0000 7fd3b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:26:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.507+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/1767620611 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c001e060 con 0x5628f8160b40 2024-09-15T23:26:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.507+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/1767620611 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd3c000fd90 con 0x5628f8160b40 2024-09-15T23:26:10.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.510+0000 7fd3da6b0740 1 -- 172.21.15.62:0/1767620611 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3c8003d10 con 0x5628f8160b40 2024-09-15T23:26:10.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.511+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/1767620611 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd3c0019430 con 0x5628f8160b40 2024-09-15T23:26:10.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.511+0000 7fd3da6b0740 10 client.27067 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:10.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.511+0000 7fd3da6b0740 1 -- 172.21.15.62:0/1767620611 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628f819ac10 con 0x5628f8160b40 2024-09-15T23:26:10.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.515+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/1767620611 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c8003d10 con 0x5628f8160b40 2024-09-15T23:26:10.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.515+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/1767620611 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fd3c0022520 con 0x5628f8160b40 2024-09-15T23:26:10.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.515+0000 7fd3da6b0740 10 client.27067 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:10.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.515+0000 7fd3da6b0740 10 client.27067 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:26:10.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.515+0000 7fd3da6b0740 20 client.27067 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:10.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.515+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5628f819c020 0x5628f819e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.515+0000 7fd3da6b0740 4 client.27067 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:10.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.515+0000 7fd3da6b0740 1 -- 172.21.15.62:0/1767620611 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628f80d2860 con 0x5628f819c020 2024-09-15T23:26:10.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.515+0000 7fd3d6073640 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5628f819c020 0x5628f819e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.520+0000 7fd3d6073640 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5628f819c020 0x5628f819e440 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=149c3479e404c24 server_cookie=e76ec2c893b5ada in_seq=0 out_seq=0 2024-09-15T23:26:10.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.520+0000 7fd3b9ffb640 10 client.27067 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:26:10.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.521+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/1767620611 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5628f80d2860 con 0x5628f819c020 2024-09-15T23:26:10.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.521+0000 7fd3b9ffb640 10 client.27067 handle_command_reply: tid=0 2024-09-15T23:26:10.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.521+0000 7fd3da6b0740 1 client.27067 shutdown 2024-09-15T23:26:10.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.527+0000 7fd3da6b0740 1 -- 172.21.15.62:0/1767620611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 msgr2=0x5628f8167090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:10.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.527+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 0x5628f8167090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd3d00fec00 tx=0x7fd3c0011b50 comp rx=0 tx=0).stop 2024-09-15T23:26:10.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.531+0000 7fd3da6b0740 1 -- 172.21.15.62:0/1767620611 shutdown_connections 2024-09-15T23:26:10.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.531+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5628f819c020 0x5628f819e440 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.531+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f816a370 0x5628f81677d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.531+0000 7fd3b9ffb640 0 client.27067 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:26:10.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.531+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f8160b40 0x5628f8167090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.531+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/1767620611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f815f5d0 0x5628f8169de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.531+0000 7fd3da6b0740 1 -- 172.21.15.62:0/1767620611 >> 172.21.15.62:0/1767620611 conn(0x5628f8128310 msgr2=0x5628f8161960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:10.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.534+0000 7fd3da6b0740 1 -- 172.21.15.62:0/1767620611 shutdown_connections 2024-09-15T23:26:10.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.534+0000 7fd3da6b0740 1 -- 172.21.15.62:0/1767620611 wait complete. 2024-09-15T23:26:10.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.534+0000 7fd3da6b0740 20 client.27067 trim_cache size 0 max 16384 2024-09-15T23:26:10.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.562+0000 7fd3da6b0740 1 Processor -- start 2024-09-15T23:26:10.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.566+0000 7fd3da6b0740 1 -- start start 2024-09-15T23:26:10.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.566+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 0x5628f81983a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.566+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f8169000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.566+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 0x5628f81a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.566+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d01aaff0 con 0x5628f8160b40 2024-09-15T23:26:10.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.566+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d01a88c0 con 0x5628f816a370 2024-09-15T23:26:10.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.566+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d0103220 con 0x5628f815f5d0 2024-09-15T23:26:10.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.566+0000 7fd3d6874640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 0x5628f81a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.567+0000 7fd3d5872640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f8169000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.567+0000 7fd3d5872640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f8169000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57980/0 (socket says 172.21.15.62:57980) 2024-09-15T23:26:10.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.567+0000 7fd3d5872640 1 -- 172.21.15.62:0/3842506521 learned_addr learned my addr 172.21.15.62:0/3842506521 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:10.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.570+0000 7fd3d6073640 1 --2- 172.21.15.62:0/3842506521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 0x5628f81983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.570+0000 7fd3d6874640 1 -- 172.21.15.62:0/3842506521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 msgr2=0x5628f81983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:10.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.570+0000 7fd3d6874640 1 --2- 172.21.15.62:0/3842506521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 0x5628f81983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.570+0000 7fd3d6874640 1 -- 172.21.15.62:0/3842506521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 msgr2=0x5628f8169000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:10.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.570+0000 7fd3d6874640 1 --2- 172.21.15.62:0/3842506521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f8169000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.570+0000 7fd3d6874640 1 -- 172.21.15.62:0/3842506521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628f815b4f0 con 0x5628f816a370 2024-09-15T23:26:10.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.571+0000 7fd3d6874640 1 --2- 172.21.15.62:0/3842506521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 0x5628f81a3cd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8011a30 tx=0x7fd3c800f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=251b158915d9fdf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:10.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.571+0000 7fd3bbfff640 1 -- 172.21.15.62:0/3842506521 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c800fe00 con 0x5628f816a370 2024-09-15T23:26:10.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.571+0000 7fd3bbfff640 1 -- 172.21.15.62:0/3842506521 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c801d410 con 0x5628f816a370 2024-09-15T23:26:10.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.571+0000 7fd3bbfff640 1 -- 172.21.15.62:0/3842506521 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c802c640 con 0x5628f816a370 2024-09-15T23:26:10.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.572+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3842506521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 msgr2=0x5628f81a3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:10.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.572+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3842506521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 0x5628f81a3cd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8011a30 tx=0x7fd3c800f740 comp rx=0 tx=0).stop 2024-09-15T23:26:10.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.572+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3842506521 shutdown_connections 2024-09-15T23:26:10.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.572+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3842506521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 0x5628f81a3cd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.572+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3842506521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f8169000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.572+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3842506521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 0x5628f81983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.573+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3842506521 >> 172.21.15.62:0/3842506521 conn(0x5628f8128310 msgr2=0x5628f81688f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:10.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.574+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3842506521 shutdown_connections 2024-09-15T23:26:10.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.574+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3842506521 wait complete. 2024-09-15T23:26:10.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.575+0000 7fd3da6b0740 1 -- start start 2024-09-15T23:26:10.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.575+0000 7fd3da6b0740 1 Processor -- start 2024-09-15T23:26:10.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.575+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 0x5628f81949d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.575+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f818fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.575+0000 7fd3da6b0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 0x5628f8190240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.575+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d0073db0 con 0x5628f8160b40 2024-09-15T23:26:10.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.575+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d01a81f0 con 0x5628f816a370 2024-09-15T23:26:10.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.575+0000 7fd3da6b0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d01a7470 con 0x5628f815f5d0 2024-09-15T23:26:10.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.575+0000 7fd3da6b0740 10 client.0 _ll_register_callbacks cb 0x7fd3c0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:10.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.576+0000 7fd3d6073640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 0x5628f81949d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.576+0000 7fd3d6073640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 0x5628f81949d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:46898/0 (socket says 172.21.15.62:46898) 2024-09-15T23:26:10.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.576+0000 7fd3d6073640 1 -- 172.21.15.62:0/3848030066 learned_addr learned my addr 172.21.15.62:0/3848030066 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:10.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.576+0000 7fd3d6874640 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 0x5628f8190240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.576+0000 7fd3d5872640 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f818fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.577+0000 7fd3d5872640 1 -- 172.21.15.62:0/3848030066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 msgr2=0x5628f81949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:10.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.577+0000 7fd3d5872640 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 0x5628f81949d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.577+0000 7fd3d5872640 1 -- 172.21.15.62:0/3848030066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 msgr2=0x5628f8190240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:10.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.577+0000 7fd3d5872640 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 0x5628f8190240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.577+0000 7fd3d5872640 1 -- 172.21.15.62:0/3848030066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628f81907b0 con 0x5628f8160b40 2024-09-15T23:26:10.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.578+0000 7fd3d5872640 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f818fd00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x5628f8157260 tx=0x7fd3c0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09680eecf7ef889 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:10.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.578+0000 7fd3b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:10.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.578+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/3848030066 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c0031060 con 0x5628f8160b40 2024-09-15T23:26:10.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.578+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3848030066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3c801d410 con 0x5628f8160b40 2024-09-15T23:26:10.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.579+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/3848030066 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd3c00223a0 con 0x5628f8160b40 2024-09-15T23:26:10.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.579+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/3848030066 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd3c001e020 con 0x5628f8160b40 2024-09-15T23:26:10.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.579+0000 7fd3da6b0740 10 client.17754 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:10.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.579+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3848030066 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628f8206a50 con 0x5628f8160b40 2024-09-15T23:26:10.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.580+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/3848030066 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c801d410 con 0x5628f8160b40 2024-09-15T23:26:10.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.580+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/3848030066 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fd3c00035f0 con 0x5628f8160b40 2024-09-15T23:26:10.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.580+0000 7fd3da6b0740 10 client.17754 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:10.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.580+0000 7fd3da6b0740 10 client.17754 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:26:10.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.580+0000 7fd3da6b0740 20 client.17754 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:10.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.580+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5628f819c020 0x5628f8209200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:10.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.580+0000 7fd3da6b0740 4 client.17754 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:10.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.580+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3848030066 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628f80ccd70 con 0x5628f819c020 2024-09-15T23:26:10.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.586+0000 7fd3d6073640 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5628f819c020 0x5628f8209200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:10.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.595+0000 7fd3d6073640 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5628f819c020 0x5628f8209200 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85e4149bea7c1dfa server_cookie=689560d1abbae494 in_seq=0 out_seq=0 2024-09-15T23:26:10.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.596+0000 7fd3b9ffb640 10 client.17754 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:26:10.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.596+0000 7fd3b9ffb640 1 -- 172.21.15.62:0/3848030066 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628f80ccd70 con 0x5628f819c020 2024-09-15T23:26:10.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.596+0000 7fd3b9ffb640 10 client.17754 handle_command_reply: tid=0 2024-09-15T23:26:10.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.596+0000 7fd3da6b0740 1 client.17754 shutdown 2024-09-15T23:26:10.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.597+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3848030066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 msgr2=0x5628f818fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:10.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.597+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f818fd00 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x5628f8157260 tx=0x7fd3c0005d30 comp rx=0 tx=0).stop 2024-09-15T23:26:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.597+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3848030066 shutdown_connections 2024-09-15T23:26:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.597+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5628f819c020 0x5628f8209200 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.597+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628f816a370 0x5628f8190240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.597+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628f8160b40 0x5628f818fd00 secure :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0x5628f8157260 tx=0x7fd3c0005d30 comp rx=0 tx=0).stop 2024-09-15T23:26:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.597+0000 7fd3da6b0740 1 --2- 172.21.15.62:0/3848030066 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628f815f5d0 0x5628f81949d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.597+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3848030066 >> 172.21.15.62:0/3848030066 conn(0x5628f8128310 msgr2=0x5628f815ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.597+0000 7fd3b9ffb640 0 client.17754 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:26:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.599+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3848030066 shutdown_connections 2024-09-15T23:26:10.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.599+0000 7fd3da6b0740 1 -- 172.21.15.62:0/3848030066 wait complete. 2024-09-15T23:26:10.603 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:26:10.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.599+0000 7fd3da6b0740 20 client.17754 trim_cache size 0 max 16384 2024-09-15T23:26:10.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 -- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3bc0a3340 msgr2=0x7fd3bc0a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:10.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3bc0a3340 0x7fd3bc0a5800 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800f640 tx=0x7fd3c8000f40 comp rx=0 tx=0).stop 2024-09-15T23:26:10.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 -- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3d0102e00 msgr2=0x7fd3d019a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:10.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3d0102e00 0x7fd3d019a6a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc006010 tx=0x7fd3cc004040 comp rx=0 tx=0).stop 2024-09-15T23:26:10.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 -- 172.21.15.62:0/4028156228 shutdown_connections 2024-09-15T23:26:10.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3bc0a3340 0x7fd3bc0a5800 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3d010c1a0 0x7fd3d019ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3d0103720 0x7fd3d019abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 --2- 172.21.15.62:0/4028156228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3d0102e00 0x7fd3d019a6a0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:10.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.602+0000 7fd3d82d7640 1 -- 172.21.15.62:0/4028156228 >> 172.21.15.62:0/4028156228 conn(0x7fd3d00fc7a0 msgr2=0x7fd3d010d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:10.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.603+0000 7fd3d82d7640 1 -- 172.21.15.62:0/4028156228 shutdown_connections 2024-09-15T23:26:10.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:10.603+0000 7fd3d82d7640 1 -- 172.21.15.62:0/4028156228 wait complete. 2024-09-15T23:26:11.423 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:26:11.905 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:11.982 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:11 smithi062 ceph-mon[30949]: pgmap v667: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 795 op/s 2024-09-15T23:26:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:11 smithi067 ceph-mon[35665]: pgmap v667: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 795 op/s 2024-09-15T23:26:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:11 smithi148 ceph-mon[35610]: pgmap v667: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 795 op/s 2024-09-15T23:26:12.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.501+0000 7f2ade2ca640 1 -- 172.21.15.62:0/506522287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ad8109450 msgr2=0x7f2ad8110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:12.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.501+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/506522287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ad8109450 0x7f2ad8110250 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0009860 tx=0x7f2ac002a340 comp rx=0 tx=0).stop 2024-09-15T23:26:12.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.501+0000 7f2ade2ca640 1 -- 172.21.15.62:0/506522287 shutdown_connections 2024-09-15T23:26:12.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.501+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/506522287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad811b140 0x7f2ad811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.501+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/506522287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad8109830 0x7f2ad8109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.501+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/506522287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ad8109450 0x7f2ad8110250 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.501+0000 7f2ade2ca640 1 -- 172.21.15.62:0/506522287 >> 172.21.15.62:0/506522287 conn(0x7f2ad806e140 msgr2=0x7f2ad806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:12.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.501+0000 7f2ade2ca640 1 -- 172.21.15.62:0/506522287 shutdown_connections 2024-09-15T23:26:12.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.502+0000 7f2ade2ca640 1 -- 172.21.15.62:0/506522287 wait complete. 2024-09-15T23:26:12.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.502+0000 7f2ade2ca640 1 Processor -- start 2024-09-15T23:26:12.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2ade2ca640 1 -- start start 2024-09-15T23:26:12.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2ade2ca640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ad8109450 0x7f2ad81a7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2ade2ca640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad8109830 0x7f2ad81a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2acffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad8109830 0x7f2ad81a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2acffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad8109830 0x7f2ad81a77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42022/0 (socket says 172.21.15.62:42022) 2024-09-15T23:26:12.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ad8109450 0x7f2ad81a7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ad8109450 0x7f2ad81a7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47664/0 (socket says 172.21.15.62:47664) 2024-09-15T23:26:12.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2ade2ca640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad811b140 0x7f2ad81abb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2acffff640 1 -- 172.21.15.62:0/3437657735 learned_addr learned my addr 172.21.15.62:0/3437657735 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:12.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad8120da0 con 0x7f2ad811b140 2024-09-15T23:26:12.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad8120c20 con 0x7f2ad8109830 2024-09-15T23:26:12.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.503+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad8120f20 con 0x7f2ad8109450 2024-09-15T23:26:12.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.504+0000 7f2adc867640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad811b140 0x7f2ad81abb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.504+0000 7f2acffff640 1 -- 172.21.15.62:0/3437657735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ad8109450 msgr2=0x7f2ad81a7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.504+0000 7f2acffff640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ad8109450 0x7f2ad81a7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.504+0000 7f2acffff640 1 -- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad811b140 msgr2=0x7f2ad81abb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.504+0000 7f2acffff640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad811b140 0x7f2ad81abb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.504+0000 7f2acffff640 1 -- 172.21.15.62:0/3437657735 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad81ac230 con 0x7f2ad8109830 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.504+0000 7f2acffff640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad8109830 0x7f2ad81a77c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800e9c0 tx=0x7f2ac800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.505+0000 7f2ad5ffb640 1 -- 172.21.15.62:0/3437657735 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac800ccc0 con 0x7f2ad8109830 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.505+0000 7f2ad5ffb640 1 -- 172.21.15.62:0/3437657735 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac8005500 con 0x7f2ad8109830 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.505+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad81ac520 con 0x7f2ad8109830 2024-09-15T23:26:12.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.505+0000 7f2ad5ffb640 1 -- 172.21.15.62:0/3437657735 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8010430 con 0x7f2ad8109830 2024-09-15T23:26:12.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.505+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad81b3e00 con 0x7f2ad8109830 2024-09-15T23:26:12.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.511+0000 7f2ae06a3740 1 Processor -- start 2024-09-15T23:26:12.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.512+0000 7f2ae06a3740 1 -- start start 2024-09-15T23:26:12.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.512+0000 7f2ad5ffb640 1 -- 172.21.15.62:0/3437657735 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2ac80046c0 con 0x7f2ad8109830 2024-09-15T23:26:12.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.513+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f6a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.513+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f6aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.513+0000 7f2adc867640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f6a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.513+0000 7f2ad5ffb640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ac40a3410 0x7f2ac40a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.513+0000 7f2adc867640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f6a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36040/0 (socket says 172.21.15.62:36040) 2024-09-15T23:26:12.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.513+0000 7f2ad5ffb640 1 -- 172.21.15.62:0/3437657735 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2ac80cfec0 con 0x7f2ad8109830 2024-09-15T23:26:12.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2adc867640 1 -- 172.21.15.62:0/2139314057 learned_addr learned my addr 172.21.15.62:0/2139314057 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:12.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 0x55b846f77760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2139314057 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b846ed02f0 con 0x55b846f6bb40 2024-09-15T23:26:12.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2139314057 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b846e77cf0 con 0x55b846f75370 2024-09-15T23:26:12.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2139314057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b846ecc2b0 con 0x55b846f6a5d0 2024-09-15T23:26:12.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2acffff640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ac40a3410 0x7f2ac40a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f6aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2adc867640 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 0x55b846f77760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2139314057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 msgr2=0x55b846f77760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 0x55b846f77760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2139314057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 msgr2=0x55b846f6a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f6a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2139314057 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b846e5c3a0 con 0x55b846f6a5d0 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2acffff640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ac40a3410 0x7f2ac40a58d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8098560 tx=0x7f2ac80bc040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.514+0000 7f2adc867640 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 0x55b846f77760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.515+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f6aa10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0030750 tx=0x7f2ac0009490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a78491f1c3037d54 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.515+0000 7f2ace7fc640 1 -- 172.21.15.62:0/2139314057 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0041030 con 0x55b846f6a5d0 2024-09-15T23:26:12.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.515+0000 7f2ace7fc640 1 -- 172.21.15.62:0/2139314057 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac00304c0 con 0x55b846f6a5d0 2024-09-15T23:26:12.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.516+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2139314057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 msgr2=0x55b846f6aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:12.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.516+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f6aa10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0030750 tx=0x7f2ac0009490 comp rx=0 tx=0).stop 2024-09-15T23:26:12.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.516+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2139314057 shutdown_connections 2024-09-15T23:26:12.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.516+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 0x55b846f77760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.516+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f6aa10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.516+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2139314057 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f6a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.516+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2139314057 >> 172.21.15.62:0/2139314057 conn(0x55b846f33310 msgr2=0x55b846f66c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.516+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2139314057 shutdown_connections 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.516+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2139314057 wait complete. 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.517+0000 7f2ae06a3740 1 -- start start 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.517+0000 7f2ae06a3740 1 Processor -- start 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f74c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6bb40 0x55b846f71ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 0x55b846f72400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ae06a3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b846f780b0 con 0x55b846f75370 2024-09-15T23:26:12.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6bb40 0x55b846f71ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2adc867640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f74c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6bb40 0x55b846f71ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42032/0 (socket says 172.21.15.62:42032) 2024-09-15T23:26:12.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2697947099 learned_addr learned my addr 172.21.15.62:0/2697947099 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:12.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b846e8b060 con 0x55b846f6bb40 2024-09-15T23:26:12.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b846f78230 con 0x55b846f6a5d0 2024-09-15T23:26:12.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2adc867640 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 0x55b846f72400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ae06a3740 10 client.0 _ll_register_callbacks cb 0x55b846bde8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2697947099 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 msgr2=0x55b846f74c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f74c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2697947099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 msgr2=0x55b846f72400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.518+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 0x55b846f72400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.519+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2697947099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b846c413c0 con 0x55b846f6bb40 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.519+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac00304c0 con 0x55b846f6bb40 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.519+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6bb40 0x55b846f71ec0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2ac002a380 tx=0x7f2ac0032670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e4c0f40cdcc652c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.519+0000 7f2ab77fe640 10 client.27326 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.519+0000 7f2ab77fe640 1 -- 172.21.15.62:0/2697947099 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0054050 con 0x55b846f6bb40 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.519+0000 7f2ab77fe640 1 -- 172.21.15.62:0/2697947099 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2ac0032bf0 con 0x55b846f6bb40 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.519+0000 7f2ab77fe640 1 -- 172.21.15.62:0/2697947099 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2ac004a070 con 0x55b846f6bb40 2024-09-15T23:26:12.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.519+0000 7f2ae06a3740 10 client.27326 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.519+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55b846f780b0 con 0x55b846f6bb40 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.520+0000 7f2ab77fe640 1 -- 172.21.15.62:0/2697947099 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f2ac003a050 con 0x55b846f6bb40 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.520+0000 7f2ae06a3740 10 client.27326 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.520+0000 7f2ae06a3740 10 client.27326 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.520+0000 7f2ae06a3740 20 client.27326 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.520+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55b846f95ab0 0x55b846f97ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.520+0000 7f2ae06a3740 4 client.27326 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.520+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b846edd860 con 0x55b846f95ab0 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.520+0000 7f2adc867640 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55b846f95ab0 0x55b846f97ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.521+0000 7f2ab77fe640 10 client.27326 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.521+0000 7f2adc867640 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55b846f95ab0 0x55b846f97ea0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ddefa4a34888b5e7 server_cookie=6891f43d6a2dc5c9 in_seq=0 out_seq=0 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.521+0000 7f2ab77fe640 1 -- 172.21.15.62:0/2697947099 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b846edd860 con 0x55b846f95ab0 2024-09-15T23:26:12.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.521+0000 7f2ab77fe640 10 client.27326 handle_command_reply: tid=0 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ae06a3740 1 client.27326 shutdown 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6bb40 msgr2=0x55b846f71ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6bb40 0x55b846f71ec0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2ac002a380 tx=0x7f2ac0032670 comp rx=0 tx=0).stop 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 shutdown_connections 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55b846f95ab0 0x55b846f97ea0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ab77fe640 0 client.27326 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 0x55b846f72400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6bb40 0x55b846f71ec0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2697947099 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f74c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.522+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 >> 172.21.15.62:0/2697947099 conn(0x55b846f33310 msgr2=0x55b846f6c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:12.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.524+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 shutdown_connections 2024-09-15T23:26:12.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.524+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2697947099 wait complete. 2024-09-15T23:26:12.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.525+0000 7f2ae06a3740 20 client.27326 trim_cache size 0 max 16384 2024-09-15T23:26:12.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.546+0000 7f2ae06a3740 1 Processor -- start 2024-09-15T23:26:12.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.546+0000 7f2ae06a3740 1 -- start start 2024-09-15T23:26:12.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f65c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f76760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 0x55b846f6d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ae06a3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad8077fc0 con 0x55b846f6bb40 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ae06a3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad8075890 con 0x55b846f75370 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ae06a3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad81182c0 con 0x55b846f6a5d0 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f76760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f76760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36062/0 (socket says 172.21.15.62:36062) 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2110179545 learned_addr learned my addr 172.21.15.62:0/2110179545 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2adc867640 1 --2- 172.21.15.62:0/2110179545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 0x55b846f6d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2adc867640 1 --2- 172.21.15.62:0/2110179545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f65c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2110179545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 msgr2=0x55b846f65c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2110179545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f65c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2110179545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 msgr2=0x55b846f6d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2110179545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 0x55b846f6d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ad7fff640 1 -- 172.21.15.62:0/2110179545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b846f67960 con 0x55b846f6bb40 2024-09-15T23:26:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/2110179545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f76760 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2ac002ab30 tx=0x7f2ac003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb427b62a65f277c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.547+0000 7f2acdffb640 1 -- 172.21.15.62:0/2110179545 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0054070 con 0x55b846f6bb40 2024-09-15T23:26:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.548+0000 7f2acdffb640 1 -- 172.21.15.62:0/2110179545 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac00304c0 con 0x55b846f6bb40 2024-09-15T23:26:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.548+0000 7f2acdffb640 1 -- 172.21.15.62:0/2110179545 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0041040 con 0x55b846f6bb40 2024-09-15T23:26:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.548+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2110179545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 msgr2=0x55b846f76760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.548+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2110179545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f76760 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f2ac002ab30 tx=0x7f2ac003e000 comp rx=0 tx=0).stop 2024-09-15T23:26:12.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.548+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2110179545 shutdown_connections 2024-09-15T23:26:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.548+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2110179545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f75370 0x55b846f6d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.548+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2110179545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f6bb40 0x55b846f76760 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.548+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/2110179545 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6a5d0 0x55b846f65c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.548+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2110179545 >> 172.21.15.62:0/2110179545 conn(0x55b846f33310 msgr2=0x55b846f65500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.549+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2110179545 shutdown_connections 2024-09-15T23:26:12.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.549+0000 7f2ae06a3740 1 -- 172.21.15.62:0/2110179545 wait complete. 2024-09-15T23:26:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.549+0000 7f2ae06a3740 1 -- start start 2024-09-15T23:26:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.549+0000 7f2ae06a3740 1 Processor -- start 2024-09-15T23:26:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6a5d0 0x55b846f8ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6bb40 0x55b846f6f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ae06a3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 0x55b846f6f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ae06a3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad8118930 con 0x55b846f75370 2024-09-15T23:26:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6bb40 0x55b846f6f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2adc867640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6a5d0 0x55b846f8ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ad7fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6bb40 0x55b846f6f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47716/0 (socket says 172.21.15.62:47716) 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ad7fff640 1 -- 172.21.15.62:0/3602264398 learned_addr learned my addr 172.21.15.62:0/3602264398 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad80751c0 con 0x55b846f6a5d0 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad8074420 con 0x55b846f6bb40 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2adc867640 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 0x55b846f6f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ae06a3740 10 client.0 _ll_register_callbacks cb 0x7f2ad0009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ad7fff640 1 -- 172.21.15.62:0/3602264398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6a5d0 msgr2=0x55b846f8ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6a5d0 0x55b846f8ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ad7fff640 1 -- 172.21.15.62:0/3602264398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 msgr2=0x55b846f6f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.550+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 0x55b846f6f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ad7fff640 1 -- 172.21.15.62:0/3602264398 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b846f6fc70 con 0x55b846f6bb40 2024-09-15T23:26:12.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ad7fff640 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6bb40 0x55b846f6f140 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0005d30 tx=0x7f2ac0030690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=327e663546d7a30f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:12.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ab77fe640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:26:12.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ab77fe640 1 -- 172.21.15.62:0/3602264398 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac0054030 con 0x55b846f6bb40 2024-09-15T23:26:12.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac00304c0 con 0x55b846f6bb40 2024-09-15T23:26:12.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ab77fe640 1 -- 172.21.15.62:0/3602264398 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2ac0004250 con 0x55b846f6bb40 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ab77fe640 1 -- 172.21.15.62:0/3602264398 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2ac0069050 con 0x55b846f6bb40 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ae06a3740 10 client.27088 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b846f9d950 con 0x55b846f6bb40 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.551+0000 7f2ab77fe640 1 -- 172.21.15.62:0/3602264398 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac00304c0 con 0x55b846f6bb40 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.552+0000 7f2ab77fe640 1 -- 172.21.15.62:0/3602264398 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f2ac003a040 con 0x55b846f6bb40 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.552+0000 7f2ae06a3740 10 client.27088 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.552+0000 7f2ae06a3740 10 client.27088 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.552+0000 7f2ae06a3740 20 client.27088 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.552+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55b846f95ab0 0x55b8470028a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.552+0000 7f2ae06a3740 4 client.27088 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.552+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b846ed7d70 con 0x55b846f95ab0 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.552+0000 7f2adc867640 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55b846f95ab0 0x55b8470028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:12.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.553+0000 7f2adc867640 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55b846f95ab0 0x55b8470028a0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=decf9f432687090d server_cookie=a7adf0077a1df1db in_seq=0 out_seq=0 2024-09-15T23:26:12.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.553+0000 7f2ab77fe640 10 client.27088 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:26:12.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.561+0000 7f2ab77fe640 1 -- 172.21.15.62:0/3602264398 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b846ed7d70 con 0x55b846f95ab0 2024-09-15T23:26:12.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.561+0000 7f2ab77fe640 10 client.27088 handle_command_reply: tid=0 2024-09-15T23:26:12.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.561+0000 7f2ae06a3740 1 client.27088 shutdown 2024-09-15T23:26:12.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.562+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6bb40 msgr2=0x55b846f6f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:12.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.562+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6bb40 0x55b846f6f140 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0005d30 tx=0x7f2ac0030690 comp rx=0 tx=0).stop 2024-09-15T23:26:12.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.563+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 shutdown_connections 2024-09-15T23:26:12.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.563+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55b846f95ab0 0x55b8470028a0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.563+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b846f75370 0x55b846f6f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.563+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b846f6bb40 0x55b846f6f140 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.563+0000 7f2ae06a3740 1 --2- 172.21.15.62:0/3602264398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b846f6a5d0 0x55b846f8ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.563+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 >> 172.21.15.62:0/3602264398 conn(0x55b846f33310 msgr2=0x55b846f65500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:12.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.566+0000 7f2ab77fe640 0 client.27088 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:26:12.569 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:26:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.566+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 shutdown_connections 2024-09-15T23:26:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.566+0000 7f2ae06a3740 1 -- 172.21.15.62:0/3602264398 wait complete. 2024-09-15T23:26:12.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.566+0000 7f2ae06a3740 20 client.27088 trim_cache size 0 max 16384 2024-09-15T23:26:12.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ac40a3410 msgr2=0x7f2ac40a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:12.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ac40a3410 0x7f2ac40a58d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8098560 tx=0x7f2ac80bc040 comp rx=0 tx=0).stop 2024-09-15T23:26:12.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad8109830 msgr2=0x7f2ad81a77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:12.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad8109830 0x7f2ad81a77c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800e9c0 tx=0x7f2ac800ee80 comp rx=0 tx=0).stop 2024-09-15T23:26:12.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 shutdown_connections 2024-09-15T23:26:12.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ac40a3410 0x7f2ac40a58d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ad811b140 0x7f2ad81abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ad8109830 0x7f2ad81a77c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 --2- 172.21.15.62:0/3437657735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2ad8109450 0x7f2ad81a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:12.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.574+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 >> 172.21.15.62:0/3437657735 conn(0x7f2ad806e140 msgr2=0x7f2ad811be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:12.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.577+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 shutdown_connections 2024-09-15T23:26:12.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:12.577+0000 7f2ade2ca640 1 -- 172.21.15.62:0/3437657735 wait complete. 2024-09-15T23:26:13.478 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:26:13.937 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:13.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:13 smithi062 ceph-mon[30949]: pgmap v668: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.0 MiB/s wr, 810 op/s 2024-09-15T23:26:14.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:13 smithi067 ceph-mon[35665]: pgmap v668: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.0 MiB/s wr, 810 op/s 2024-09-15T23:26:14.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:13 smithi148 ceph-mon[35610]: pgmap v668: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 7.0 MiB/s wr, 810 op/s 2024-09-15T23:26:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 595.0 s, 622.8 tps, lat 146.829 ms stddev 156.446 2024-09-15T23:26:15.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:15 smithi062 ceph-mon[30949]: pgmap v669: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.3 MiB/s wr, 742 op/s 2024-09-15T23:26:16.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:15 smithi067 ceph-mon[35665]: pgmap v669: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.3 MiB/s wr, 742 op/s 2024-09-15T23:26:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:15 smithi148 ceph-mon[35610]: pgmap v669: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.3 MiB/s wr, 742 op/s 2024-09-15T23:26:16.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.217+0000 7f6570f91640 1 -- 172.21.15.62:0/3171564991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c102d50 msgr2=0x7f656c1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:16.220 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.217+0000 7f6570f91640 1 --2- 172.21.15.62:0/3171564991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c102d50 0x7f656c1031d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f6554009900 tx=0x7f655402a410 comp rx=0 tx=0).stop 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.217+0000 7f6570f91640 1 -- 172.21.15.62:0/3171564991 shutdown_connections 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.217+0000 7f6570f91640 1 --2- 172.21.15.62:0/3171564991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c103710 0x7f656c109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.217+0000 7f6570f91640 1 --2- 172.21.15.62:0/3171564991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c102d50 0x7f656c1031d0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.217+0000 7f6570f91640 1 --2- 172.21.15.62:0/3171564991 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c101b60 0x7f656c101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.217+0000 7f6570f91640 1 -- 172.21.15.62:0/3171564991 >> 172.21.15.62:0/3171564991 conn(0x7f656c0fd2e0 msgr2=0x7f656c0ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.217+0000 7f6570f91640 1 -- 172.21.15.62:0/3171564991 shutdown_connections 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.217+0000 7f6570f91640 1 -- 172.21.15.62:0/3171564991 wait complete. 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f6570f91640 1 Processor -- start 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f6570f91640 1 -- start start 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f6570f91640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c101b60 0x7f656c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f6570f91640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c102d50 0x7f656c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f6570f91640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c103710 0x7f656c19f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f6570f91640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10d7e0 con 0x7f656c103710 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f6570f91640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10d660 con 0x7f656c102d50 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f6570f91640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f656c10d960 con 0x7f656c101b60 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f656a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c101b60 0x7f656c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f656ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c103710 0x7f656c19f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f656a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c101b60 0x7f656c197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47738/0 (socket says 172.21.15.62:47738) 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f6569d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c102d50 0x7f656c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.218+0000 7f656a59c640 1 -- 172.21.15.62:0/2858278063 learned_addr learned my addr 172.21.15.62:0/2858278063 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f656ad9d640 1 -- 172.21.15.62:0/2858278063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c101b60 msgr2=0x7f656c197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f656ad9d640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c101b60 0x7f656c197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f656ad9d640 1 -- 172.21.15.62:0/2858278063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c102d50 msgr2=0x7f656c198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f656ad9d640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c102d50 0x7f656c198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f656ad9d640 1 -- 172.21.15.62:0/2858278063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c19fa20 con 0x7f656c103710 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f656a59c640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c101b60 0x7f656c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:16.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f6569d9b640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c102d50 0x7f656c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:16.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f656ad9d640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c103710 0x7f656c19f3b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f655c00e970 tx=0x7f655c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:16.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f65637fe640 1 -- 172.21.15.62:0/2858278063 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c00ccc0 con 0x7f656c103710 2024-09-15T23:26:16.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f65637fe640 1 -- 172.21.15.62:0/2858278063 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f655c005020 con 0x7f656c103710 2024-09-15T23:26:16.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f65637fe640 1 -- 172.21.15.62:0/2858278063 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c004a00 con 0x7f656c103710 2024-09-15T23:26:16.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f6570f91640 1 -- 172.21.15.62:0/2858278063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656c19fd10 con 0x7f656c103710 2024-09-15T23:26:16.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.219+0000 7f6570f91640 1 -- 172.21.15.62:0/2858278063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f656c1a0180 con 0x7f656c103710 2024-09-15T23:26:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.221+0000 7f65637fe640 1 -- 172.21.15.62:0/2858278063 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f655c0051d0 con 0x7f656c103710 2024-09-15T23:26:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.222+0000 7f65637fe640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3370 0x7f65580a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.222+0000 7f65637fe640 1 -- 172.21.15.62:0/2858278063 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f655c0cfba0 con 0x7f656c103710 2024-09-15T23:26:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.222+0000 7f656a59c640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3370 0x7f65580a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.222+0000 7f656a59c640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3370 0x7f65580a5830 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6564005f40 tx=0x7f6564009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.439+0000 7f6573e09740 1 Processor -- start 2024-09-15T23:26:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.439+0000 7f6573e09740 1 -- start start 2024-09-15T23:26:16.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.439+0000 7f6573e09740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283449a0 0x561228344da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.439+0000 7f6573e09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 0x561228349da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f6573e09740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56122834a370 0x56122834c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f6573e09740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5612282a82f0 con 0x56122834a370 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f6573e09740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561228250270 con 0x5612283449a0 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f6573e09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5612282a42b0 con 0x5612283452e0 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283449a0 0x561228344da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656a59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283449a0 0x561228344da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42076/0 (socket says 172.21.15.62:42076) 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656a59c640 1 -- 172.21.15.62:0/1165539723 learned_addr learned my addr 172.21.15.62:0/1165539723 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656ad9d640 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56122834a370 0x56122834c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f6569d9b640 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 0x561228349da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656a59c640 1 -- 172.21.15.62:0/1165539723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 msgr2=0x561228349da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656a59c640 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 0x561228349da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656a59c640 1 -- 172.21.15.62:0/1165539723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56122834a370 msgr2=0x56122834c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:16.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656a59c640 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56122834a370 0x56122834c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656a59c640 1 -- 172.21.15.62:0/1165539723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56122824f340 con 0x5612283449a0 2024-09-15T23:26:16.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.440+0000 7f656ad9d640 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56122834a370 0x56122834c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:16.443 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.441+0000 7f656a59c640 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283449a0 0x561228344da0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6564011b10 tx=0x7f6564014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19f3b86fb7d4fca6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:16.450 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.441+0000 7f6560ff9640 1 -- 172.21.15.62:0/1165539723 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656400e9c0 con 0x5612283449a0 2024-09-15T23:26:16.450 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.441+0000 7f6560ff9640 1 -- 172.21.15.62:0/1165539723 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656400eb60 con 0x5612283449a0 2024-09-15T23:26:16.450 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.441+0000 7f6560ff9640 1 -- 172.21.15.62:0/1165539723 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564026660 con 0x5612283449a0 2024-09-15T23:26:16.450 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 -- 172.21.15.62:0/1165539723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283449a0 msgr2=0x561228344da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:16.450 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283449a0 0x561228344da0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6564011b10 tx=0x7f6564014000 comp rx=0 tx=0).stop 2024-09-15T23:26:16.450 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 -- 172.21.15.62:0/1165539723 shutdown_connections 2024-09-15T23:26:16.450 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56122834a370 0x56122834c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.450 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 0x561228349da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 --2- 172.21.15.62:0/1165539723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283449a0 0x561228344da0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 -- 172.21.15.62:0/1165539723 >> 172.21.15.62:0/1165539723 conn(0x561228340010 msgr2=0x561228340410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 -- 172.21.15.62:0/1165539723 shutdown_connections 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 -- 172.21.15.62:0/1165539723 wait complete. 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 -- start start 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.442+0000 7f6573e09740 1 Processor -- start 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f6573e09740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f6573e09740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283452e0 0x56122834f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f6573e09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56122834a370 0x561228356140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f6573e09740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56122834d340 con 0x5612283449a0 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f6573e09740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561228231290 con 0x5612283452e0 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f6573e09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56122834d4c0 con 0x56122834a370 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f6573e09740 10 client.0 _ll_register_callbacks cb 0x561228262140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:16.451 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36098/0 (socket says 172.21.15.62:36098) 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f6569d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283452e0 0x56122834f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656a59c640 1 -- 172.21.15.62:0/1372859513 learned_addr learned my addr 172.21.15.62:0/1372859513 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656ad9d640 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56122834a370 0x561228356140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656a59c640 1 -- 172.21.15.62:0/1372859513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56122834a370 msgr2=0x561228356140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656a59c640 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56122834a370 0x561228356140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656a59c640 1 -- 172.21.15.62:0/1372859513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283452e0 msgr2=0x56122834f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656a59c640 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283452e0 0x56122834f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656a59c640 1 -- 172.21.15.62:0/1372859513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561227fbb210 con 0x5612283449a0 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f656a59c640 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834ebc0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f656400ef40 tx=0x7f656401eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c998ff4993c211 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.443+0000 7f6551ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6551ffb640 1 -- 172.21.15.62:0/1372859513 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656401e630 con 0x5612283449a0 2024-09-15T23:26:16.452 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6551ffb640 1 -- 172.21.15.62:0/1372859513 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f656401e7d0 con 0x5612283449a0 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6573e09740 1 -- 172.21.15.62:0/1372859513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f656400eb60 con 0x5612283449a0 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6551ffb640 1 -- 172.21.15.62:0/1372859513 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6564004fb0 con 0x5612283449a0 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6573e09740 10 client.17784 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6573e09740 1 -- 172.21.15.62:0/1372859513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561228087620 con 0x5612283449a0 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6551ffb640 1 -- 172.21.15.62:0/1372859513 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656400eb60 con 0x5612283449a0 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6551ffb640 1 -- 172.21.15.62:0/1372859513 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f656402e070 con 0x5612283449a0 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6573e09740 10 client.17784 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6573e09740 10 client.17784 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6573e09740 20 client.17784 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:16.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6573e09740 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56122835a440 0x56122835c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6573e09740 4 client.17784 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.444+0000 7f6573e09740 1 -- 172.21.15.62:0/1372859513 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612282b5a20 con 0x56122835a440 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.445+0000 7f6569d9b640 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56122835a440 0x56122835c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.445+0000 7f6551ffb640 10 client.17784 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.445+0000 7f6569d9b640 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56122835a440 0x56122835c870 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14bb76cf6bfcdd66 server_cookie=35503202f2c6b7da in_seq=0 out_seq=0 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6551ffb640 1 -- 172.21.15.62:0/1372859513 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5612282b5a20 con 0x56122835a440 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6551ffb640 10 client.17784 handle_command_reply: tid=0 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 client.17784 shutdown 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 -- 172.21.15.62:0/1372859513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 msgr2=0x56122834ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834ebc0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f656400ef40 tx=0x7f656401eb60 comp rx=0 tx=0).stop 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 -- 172.21.15.62:0/1372859513 shutdown_connections 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56122835a440 0x56122835c870 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56122834a370 0x561228356140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5612283452e0 0x56122834f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6551ffb640 0 client.17784 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 --2- 172.21.15.62:0/1372859513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834ebc0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 -- 172.21.15.62:0/1372859513 >> 172.21.15.62:0/1372859513 conn(0x561228340010 msgr2=0x561228341820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 -- 172.21.15.62:0/1372859513 shutdown_connections 2024-09-15T23:26:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 1 -- 172.21.15.62:0/1372859513 wait complete. 2024-09-15T23:26:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.446+0000 7f6573e09740 20 client.17784 trim_cache size 0 max 16384 2024-09-15T23:26:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f6573e09740 1 Processor -- start 2024-09-15T23:26:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f6573e09740 1 -- start start 2024-09-15T23:26:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f6573e09740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f6573e09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 0x561228356060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f6573e09740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 0x56122833f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f6573e09740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1acf10 con 0x5612283449a0 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f6573e09740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1aa7e0 con 0x56122834a370 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f6573e09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1a8630 con 0x5612283452e0 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f656a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f656a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36110/0 (socket says 172.21.15.62:36110) 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f656a59c640 1 -- 172.21.15.62:0/3969480741 learned_addr learned my addr 172.21.15.62:0/3969480741 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f656ad9d640 1 --2- 172.21.15.62:0/3969480741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 0x56122833f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f6569d9b640 1 --2- 172.21.15.62:0/3969480741 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 0x561228356060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f656a59c640 1 -- 172.21.15.62:0/3969480741 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 msgr2=0x561228356060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f656a59c640 1 --2- 172.21.15.62:0/3969480741 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 0x561228356060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f656a59c640 1 -- 172.21.15.62:0/3969480741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 msgr2=0x56122833f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f656a59c640 1 --2- 172.21.15.62:0/3969480741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 0x56122833f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.466 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.461+0000 7f656a59c640 1 -- 172.21.15.62:0/3969480741 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612283457c0 con 0x5612283449a0 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.462+0000 7f656a59c640 1 --2- 172.21.15.62:0/3969480741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834f190 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f6564016040 tx=0x7f6564004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221f76c9f6e2190b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.462+0000 7f6553fff640 1 -- 172.21.15.62:0/3969480741 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564030e60 con 0x5612283449a0 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.462+0000 7f6553fff640 1 -- 172.21.15.62:0/3969480741 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656401ebd0 con 0x5612283449a0 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.462+0000 7f6553fff640 1 -- 172.21.15.62:0/3969480741 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656401e890 con 0x5612283449a0 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 -- 172.21.15.62:0/3969480741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 msgr2=0x56122834f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 --2- 172.21.15.62:0/3969480741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834f190 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f6564016040 tx=0x7f6564004f00 comp rx=0 tx=0).stop 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 -- 172.21.15.62:0/3969480741 shutdown_connections 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 --2- 172.21.15.62:0/3969480741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 0x56122833f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 --2- 172.21.15.62:0/3969480741 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283452e0 0x561228356060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 --2- 172.21.15.62:0/3969480741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283449a0 0x56122834f190 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 -- 172.21.15.62:0/3969480741 >> 172.21.15.62:0/3969480741 conn(0x561228340010 msgr2=0x56122833e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 -- 172.21.15.62:0/3969480741 shutdown_connections 2024-09-15T23:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 -- 172.21.15.62:0/3969480741 wait complete. 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 -- start start 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 Processor -- start 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283449a0 0x561228353970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283452e0 0x561228353eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 0x561228354e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1a8ca0 con 0x5612283452e0 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1aa110 con 0x56122834a370 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f656c1a9370 con 0x5612283449a0 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6573e09740 10 client.0 _ll_register_callbacks cb 0x7f656400ff50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6569d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283452e0 0x561228353eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6569d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283452e0 0x561228353eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36120/0 (socket says 172.21.15.62:36120) 2024-09-15T23:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f6569d9b640 1 -- 172.21.15.62:0/2015683342 learned_addr learned my addr 172.21.15.62:0/2015683342 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f656a59c640 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283449a0 0x561228353970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.463+0000 7f656ad9d640 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 0x561228354e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.464+0000 7f6569d9b640 1 -- 172.21.15.62:0/2015683342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283449a0 msgr2=0x561228353970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.464+0000 7f6569d9b640 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283449a0 0x561228353970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.464+0000 7f6569d9b640 1 -- 172.21.15.62:0/2015683342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 msgr2=0x561228354e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.464+0000 7f6569d9b640 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 0x561228354e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.464+0000 7f6569d9b640 1 -- 172.21.15.62:0/2015683342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561228355480 con 0x5612283452e0 2024-09-15T23:26:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.464+0000 7f6569d9b640 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283452e0 0x561228353eb0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6554031610 tx=0x7f6554004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630c10b3ce0a5c3a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:16.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.464+0000 7f6551ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.465+0000 7f6551ffb640 1 -- 172.21.15.62:0/2015683342 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554049020 con 0x5612283452e0 2024-09-15T23:26:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.465+0000 7f6551ffb640 1 -- 172.21.15.62:0/2015683342 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f65540318b0 con 0x5612283452e0 2024-09-15T23:26:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.467+0000 7f6573e09740 1 -- 172.21.15.62:0/2015683342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f656401ebd0 con 0x5612283452e0 2024-09-15T23:26:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.467+0000 7f6551ffb640 1 -- 172.21.15.62:0/2015683342 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6554035040 con 0x5612283452e0 2024-09-15T23:26:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.467+0000 7f6573e09740 10 client.17790 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.467+0000 7f6573e09740 1 -- 172.21.15.62:0/2015683342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5612283c5280 con 0x5612283452e0 2024-09-15T23:26:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.468+0000 7f6551ffb640 1 -- 172.21.15.62:0/2015683342 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656401ebd0 con 0x5612283452e0 2024-09-15T23:26:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.468+0000 7f6551ffb640 1 -- 172.21.15.62:0/2015683342 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f6554004760 con 0x5612283452e0 2024-09-15T23:26:16.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.468+0000 7f6573e09740 10 client.17790 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.468+0000 7f6573e09740 10 client.17790 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.468+0000 7f6573e09740 20 client.17790 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.468+0000 7f6573e09740 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5612283c5bc0 0x5612283c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.468+0000 7f6573e09740 4 client.17790 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.468+0000 7f6573e09740 1 -- 172.21.15.62:0/2015683342 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612282afd70 con 0x5612283c5bc0 2024-09-15T23:26:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.468+0000 7f656a59c640 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5612283c5bc0 0x5612283c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.469+0000 7f656a59c640 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5612283c5bc0 0x5612283c7fe0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57b3ae2ccc12c091 server_cookie=fbb9bb854ea9596f in_seq=0 out_seq=0 2024-09-15T23:26:16.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.469+0000 7f6551ffb640 10 client.17790 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:16.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.512+0000 7f6551ffb640 1 -- 172.21.15.62:0/2015683342 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612282afd70 con 0x5612283c5bc0 2024-09-15T23:26:16.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.512+0000 7f6551ffb640 10 client.17790 handle_command_reply: tid=0 2024-09-15T23:26:16.514 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.513+0000 7f6573e09740 1 client.17790 shutdown 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.513+0000 7f6573e09740 1 -- 172.21.15.62:0/2015683342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283452e0 msgr2=0x561228353eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.513+0000 7f6573e09740 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283452e0 0x561228353eb0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6554031610 tx=0x7f6554004040 comp rx=0 tx=0).stop 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.513+0000 7f6573e09740 1 -- 172.21.15.62:0/2015683342 shutdown_connections 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.513+0000 7f6573e09740 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5612283c5bc0 0x5612283c7fe0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.514+0000 7f6573e09740 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56122834a370 0x561228354e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.514+0000 7f6573e09740 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612283452e0 0x561228353eb0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.514+0000 7f6573e09740 1 --2- 172.21.15.62:0/2015683342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5612283449a0 0x561228353970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.514+0000 7f6573e09740 1 -- 172.21.15.62:0/2015683342 >> 172.21.15.62:0/2015683342 conn(0x561228340010 msgr2=0x56122833e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.514+0000 7f6551ffb640 0 client.17790 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.514+0000 7f6573e09740 1 -- 172.21.15.62:0/2015683342 shutdown_connections 2024-09-15T23:26:16.515 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.514+0000 7f6573e09740 1 -- 172.21.15.62:0/2015683342 wait complete. 2024-09-15T23:26:16.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.515+0000 7f6573e09740 20 client.17790 trim_cache size 0 max 16384 2024-09-15T23:26:16.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:26:16.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:26:16.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:26:16.516 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:26:16.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.515+0000 7f6570f91640 1 -- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3370 msgr2=0x7f65580a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:16.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.515+0000 7f6570f91640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3370 0x7f65580a5830 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6564005f40 tx=0x7f6564009500 comp rx=0 tx=0).stop 2024-09-15T23:26:16.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.515+0000 7f6570f91640 1 -- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c103710 msgr2=0x7f656c19f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:16.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.515+0000 7f6570f91640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c103710 0x7f656c19f3b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f655c00e970 tx=0x7f655c00ee30 comp rx=0 tx=0).stop 2024-09-15T23:26:16.517 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.516+0000 7f6570f91640 1 -- 172.21.15.62:0/2858278063 shutdown_connections 2024-09-15T23:26:16.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.516+0000 7f6570f91640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65580a3370 0x7f65580a5830 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.516+0000 7f6570f91640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f656c103710 0x7f656c19f3b0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.516+0000 7f6570f91640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f656c102d50 0x7f656c198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.516+0000 7f6570f91640 1 --2- 172.21.15.62:0/2858278063 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f656c101b60 0x7f656c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:16.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.516+0000 7f6570f91640 1 -- 172.21.15.62:0/2858278063 >> 172.21.15.62:0/2858278063 conn(0x7f656c0fd2e0 msgr2=0x7f656c1060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:16.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.516+0000 7f6570f91640 1 -- 172.21.15.62:0/2858278063 shutdown_connections 2024-09-15T23:26:16.518 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:16.516+0000 7f6570f91640 1 -- 172.21.15.62:0/2858278063 wait complete. 2024-09-15T23:26:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.449+0000 7f178c83a640 1 -- 172.21.15.62:0/401571916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788109450 msgr2=0x7f1788110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.449+0000 7f178c83a640 1 --2- 172.21.15.62:0/401571916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788109450 0x7f1788110250 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f177c01a910 tx=0x7f177c03b470 comp rx=0 tx=0).stop 2024-09-15T23:26:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.451+0000 7f178c83a640 1 -- 172.21.15.62:0/401571916 shutdown_connections 2024-09-15T23:26:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.451+0000 7f178c83a640 1 --2- 172.21.15.62:0/401571916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f178811b140 0x7f178811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.451+0000 7f178c83a640 1 --2- 172.21.15.62:0/401571916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1788109830 0x7f1788109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.451+0000 7f178c83a640 1 --2- 172.21.15.62:0/401571916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788109450 0x7f1788110250 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.451+0000 7f178c83a640 1 -- 172.21.15.62:0/401571916 >> 172.21.15.62:0/401571916 conn(0x7f178806e140 msgr2=0x7f178806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:17.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.461+0000 7f178c83a640 1 -- 172.21.15.62:0/401571916 shutdown_connections 2024-09-15T23:26:17.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.462+0000 7f178c83a640 1 -- 172.21.15.62:0/401571916 wait complete. 2024-09-15T23:26:17.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.463+0000 7f178c83a640 1 Processor -- start 2024-09-15T23:26:17.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.463+0000 7f178c83a640 1 -- start start 2024-09-15T23:26:17.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.464+0000 7f178c83a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1788109450 0x7f178811a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.464+0000 7f178c83a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788109830 0x7f1788116a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.464+0000 7f178c83a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f178811b140 0x7f1788116f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.464+0000 7f178c83a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f17881133e0 con 0x7f1788109830 2024-09-15T23:26:17.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.464+0000 7f178c83a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1788113260 con 0x7f178811b140 2024-09-15T23:26:17.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.465+0000 7f178c83a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1788113560 con 0x7f1788109450 2024-09-15T23:26:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.469+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f178811b140 0x7f1788116f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.469+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f178811b140 0x7f1788116f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42104/0 (socket says 172.21.15.62:42104) 2024-09-15T23:26:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.469+0000 7f1786d9d640 1 -- 172.21.15.62:0/2671678397 learned_addr learned my addr 172.21.15.62:0/2671678397 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.469+0000 7f178659c640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1788109450 0x7f178811a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.472+0000 7f1785d9b640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788109830 0x7f1788116a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.474+0000 7f1786d9d640 1 -- 172.21.15.62:0/2671678397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1788109450 msgr2=0x7f178811a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.474+0000 7f1786d9d640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1788109450 0x7f178811a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.474+0000 7f1786d9d640 1 -- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788109830 msgr2=0x7f1788116a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.474+0000 7f1786d9d640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788109830 0x7f1788116a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.474+0000 7f1786d9d640 1 -- 172.21.15.62:0/2671678397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1788117810 con 0x7f178811b140 2024-09-15T23:26:17.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.480+0000 7f1786d9d640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f178811b140 0x7f1788116f80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f178000b9e0 tx=0x7f178000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:17.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.486+0000 7f17777fe640 1 -- 172.21.15.62:0/2671678397 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1780019070 con 0x7f178811b140 2024-09-15T23:26:17.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.486+0000 7f17777fe640 1 -- 172.21.15.62:0/2671678397 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1780014440 con 0x7f178811b140 2024-09-15T23:26:17.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.487+0000 7f17777fe640 1 -- 172.21.15.62:0/2671678397 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800075b0 con 0x7f178811b140 2024-09-15T23:26:17.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.487+0000 7f178c83a640 1 -- 172.21.15.62:0/2671678397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17881cd180 con 0x7f178811b140 2024-09-15T23:26:17.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.487+0000 7f178c83a640 1 -- 172.21.15.62:0/2671678397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17881cd640 con 0x7f178811b140 2024-09-15T23:26:17.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.490+0000 7f178ec13740 1 Processor -- start 2024-09-15T23:26:17.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.492+0000 7f178ec13740 1 -- start start 2024-09-15T23:26:17.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.492+0000 7f178ec13740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640e9b40 0x5560640e8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.492+0000 7f178ec13740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e85d0 0x5560640e8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.492+0000 7f178ec13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640f3370 0x5560640f5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.492+0000 7f178ec13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55606404e2f0 con 0x5560640f3370 2024-09-15T23:26:17.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.492+0000 7f178ec13740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556063ff5cf0 con 0x5560640e9b40 2024-09-15T23:26:17.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.492+0000 7f178ec13740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55606404a2b0 con 0x5560640e85d0 2024-09-15T23:26:17.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.493+0000 7f1785d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640e9b40 0x5560640e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.493+0000 7f1785d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640e9b40 0x5560640e8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42110/0 (socket says 172.21.15.62:42110) 2024-09-15T23:26:17.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.493+0000 7f1785d9b640 1 -- 172.21.15.62:0/4047012821 learned_addr learned my addr 172.21.15.62:0/4047012821 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:17.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.493+0000 7f1785d9b640 1 --2- 172.21.15.62:0/4047012821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640f3370 0x5560640f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.494+0000 7f1785d9b640 1 -- 172.21.15.62:0/4047012821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e85d0 msgr2=0x5560640e8a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.494+0000 7f1785d9b640 1 --2- 172.21.15.62:0/4047012821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e85d0 0x5560640e8a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.494+0000 7f1785d9b640 1 -- 172.21.15.62:0/4047012821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640f3370 msgr2=0x5560640f5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.494+0000 7f1785d9b640 1 --2- 172.21.15.62:0/4047012821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640f3370 0x5560640f5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.494+0000 7f1785d9b640 1 -- 172.21.15.62:0/4047012821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556063fda3a0 con 0x5560640e9b40 2024-09-15T23:26:17.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.494+0000 7f1785d9b640 1 --2- 172.21.15.62:0/4047012821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640e9b40 0x5560640e8090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1778009300 tx=0x7f17780094c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1da0a0c942b5f6ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:17.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.503+0000 7f1774ff9640 1 -- 172.21.15.62:0/4047012821 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778015e70 con 0x5560640e9b40 2024-09-15T23:26:17.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.503+0000 7f1774ff9640 1 -- 172.21.15.62:0/4047012821 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f177801c070 con 0x5560640e9b40 2024-09-15T23:26:17.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.504+0000 7f178ec13740 1 -- 172.21.15.62:0/4047012821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640e9b40 msgr2=0x5560640e8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:17.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.504+0000 7f178ec13740 1 --2- 172.21.15.62:0/4047012821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640e9b40 0x5560640e8090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1778009300 tx=0x7f17780094c0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.506+0000 7f17777fe640 1 -- 172.21.15.62:0/2671678397 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f178000caa0 con 0x7f178811b140 2024-09-15T23:26:17.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.507+0000 7f17777fe640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f17700a32c0 0x7f17700a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.508+0000 7f17777fe640 1 -- 172.21.15.62:0/2671678397 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f17800cf0a0 con 0x7f178811b140 2024-09-15T23:26:17.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.511+0000 7f178ec13740 1 -- 172.21.15.62:0/4047012821 shutdown_connections 2024-09-15T23:26:17.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.511+0000 7f178ec13740 1 --2- 172.21.15.62:0/4047012821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640f3370 0x5560640f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.511+0000 7f178ec13740 1 --2- 172.21.15.62:0/4047012821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e85d0 0x5560640e8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.511+0000 7f178ec13740 1 --2- 172.21.15.62:0/4047012821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640e9b40 0x5560640e8090 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.511+0000 7f178ec13740 1 -- 172.21.15.62:0/4047012821 >> 172.21.15.62:0/4047012821 conn(0x5560640b1310 msgr2=0x5560640e4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:17.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.512+0000 7f1785d9b640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f17700a32c0 0x7f17700a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.516+0000 7f1785d9b640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f17700a32c0 0x7f17700a5780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1778009490 tx=0x7f1778008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:17.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.519+0000 7f178ec13740 1 -- 172.21.15.62:0/4047012821 shutdown_connections 2024-09-15T23:26:17.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.521+0000 7f178ec13740 1 -- 172.21.15.62:0/4047012821 wait complete. 2024-09-15T23:26:17.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.521+0000 7f178ec13740 1 -- start start 2024-09-15T23:26:17.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.521+0000 7f178ec13740 1 Processor -- start 2024-09-15T23:26:17.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.522+0000 7f178ec13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e85d0 0x5560640f2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.522+0000 7f178ec13740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 0x5560640eff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.522+0000 7f178ec13740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640f0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.522+0000 7f178ec13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5560640f60c0 con 0x5560640e85d0 2024-09-15T23:26:17.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.522+0000 7f178ec13740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556064009060 con 0x5560640f3370 2024-09-15T23:26:17.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.522+0000 7f178ec13740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5560640f6240 con 0x5560640e9b40 2024-09-15T23:26:17.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.522+0000 7f178ec13740 10 client.0 _ll_register_callbacks cb 0x556063d5c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:17.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.522+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.523+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640f0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42126/0 (socket says 172.21.15.62:42126) 2024-09-15T23:26:17.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.523+0000 7f1786d9d640 1 -- 172.21.15.62:0/3160819916 learned_addr learned my addr 172.21.15.62:0/3160819916 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:17.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.526+0000 7f1786d9d640 1 -- 172.21.15.62:0/3160819916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 msgr2=0x5560640eff30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:26:17.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.526+0000 7f1786d9d640 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 0x5560640eff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.526+0000 7f1786d9d640 1 -- 172.21.15.62:0/3160819916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e85d0 msgr2=0x5560640f2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:17.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.526+0000 7f1786d9d640 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e85d0 0x5560640f2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.526+0000 7f1786d9d640 1 -- 172.21.15.62:0/3160819916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556063dbf3c0 con 0x5560640f3370 2024-09-15T23:26:17.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.531+0000 7f176dffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:26:17.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.532+0000 7f1786d9d640 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640f0470 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f17800a3a20 tx=0x7f178009f7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7094924d91d1aea0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.534+0000 7f176dffb640 1 -- 172.21.15.62:0/3160819916 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800cc990 con 0x5560640f3370 2024-09-15T23:26:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.534+0000 7f178ec13740 1 -- 172.21.15.62:0/3160819916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f177801c070 con 0x5560640f3370 2024-09-15T23:26:17.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.534+0000 7f176dffb640 1 -- 172.21.15.62:0/3160819916 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f17800145f0 con 0x5560640f3370 2024-09-15T23:26:17.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.535+0000 7f176dffb640 1 -- 172.21.15.62:0/3160819916 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f17800cc990 con 0x5560640f3370 2024-09-15T23:26:17.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.535+0000 7f178ec13740 10 client.27359 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:17.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.535+0000 7f178ec13740 1 -- 172.21.15.62:0/3160819916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556064101b50 con 0x5560640f3370 2024-09-15T23:26:17.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.535+0000 7f176dffb640 1 -- 172.21.15.62:0/3160819916 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f177801c070 con 0x5560640f3370 2024-09-15T23:26:17.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.537+0000 7f176dffb640 1 -- 172.21.15.62:0/3160819916 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f178009f160 con 0x5560640f3370 2024-09-15T23:26:17.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.537+0000 7f178ec13740 10 client.27359 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:17.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.537+0000 7f178ec13740 10 client.27359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:17.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.537+0000 7f178ec13740 20 client.27359 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:17.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.537+0000 7f178ec13740 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556064103500 0x5560641059c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.537+0000 7f178ec13740 4 client.27359 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:17.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.537+0000 7f178ec13740 1 -- 172.21.15.62:0/3160819916 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55606405b860 con 0x556064103500 2024-09-15T23:26:17.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.548+0000 7f178659c640 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556064103500 0x5560641059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.568+0000 7f176dffb640 10 client.27359 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:17.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.568+0000 7f178659c640 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556064103500 0x5560641059c0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eacb33cbda467c0 server_cookie=9902e0ee33f178f0 in_seq=0 out_seq=0 2024-09-15T23:26:17.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.572+0000 7f176dffb640 1 -- 172.21.15.62:0/3160819916 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55606405b860 con 0x556064103500 2024-09-15T23:26:17.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.572+0000 7f176dffb640 10 client.27359 handle_command_reply: tid=0 2024-09-15T23:26:17.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.572+0000 7f178ec13740 1 client.27359 shutdown 2024-09-15T23:26:17.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.577+0000 7f178ec13740 1 -- 172.21.15.62:0/3160819916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 msgr2=0x5560640f0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:17.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.577+0000 7f178ec13740 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640f0470 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f17800a3a20 tx=0x7f178009f7d0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.577+0000 7f178ec13740 1 -- 172.21.15.62:0/3160819916 shutdown_connections 2024-09-15T23:26:17.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.577+0000 7f178ec13740 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556064103500 0x5560641059c0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.577+0000 7f178ec13740 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640f0470 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.577+0000 7f178ec13740 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 0x5560640eff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.577+0000 7f178ec13740 1 --2- 172.21.15.62:0/3160819916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e85d0 0x5560640f2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.577+0000 7f178ec13740 1 -- 172.21.15.62:0/3160819916 >> 172.21.15.62:0/3160819916 conn(0x5560640b1310 msgr2=0x5560640ea960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:17.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.577+0000 7f176dffb640 0 client.27359 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:17.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.578+0000 7f178ec13740 1 -- 172.21.15.62:0/3160819916 shutdown_connections 2024-09-15T23:26:17.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.578+0000 7f178ec13740 1 -- 172.21.15.62:0/3160819916 wait complete. 2024-09-15T23:26:17.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.578+0000 7f178ec13740 20 client.27359 trim_cache size 0 max 16384 2024-09-15T23:26:17.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.605+0000 7f178ec13740 1 Processor -- start 2024-09-15T23:26:17.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.606+0000 7f178ec13740 1 -- start start 2024-09-15T23:26:17.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.606+0000 7f178ec13740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e85d0 0x5560640dfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.606+0000 7f178ec13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e9b40 0x5560640ff3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.606+0000 7f178ec13740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x55606410b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.606+0000 7f178ec13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f17881d9b60 con 0x5560640e9b40 2024-09-15T23:26:17.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.607+0000 7f178ec13740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17881d7430 con 0x5560640f3370 2024-09-15T23:26:17.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.607+0000 7f178ec13740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f17881d5220 con 0x5560640e85d0 2024-09-15T23:26:17.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.607+0000 7f1785d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e9b40 0x5560640ff3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.607+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x55606410b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.607+0000 7f1786d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x55606410b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42142/0 (socket says 172.21.15.62:42142) 2024-09-15T23:26:17.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.607+0000 7f1786d9d640 1 -- 172.21.15.62:0/3689375446 learned_addr learned my addr 172.21.15.62:0/3689375446 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:17.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.607+0000 7f178659c640 1 --2- 172.21.15.62:0/3689375446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e85d0 0x5560640dfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.611+0000 7f1786d9d640 1 -- 172.21.15.62:0/3689375446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e85d0 msgr2=0x5560640dfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:17.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.611+0000 7f1786d9d640 1 --2- 172.21.15.62:0/3689375446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e85d0 0x5560640dfc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.611+0000 7f1786d9d640 1 -- 172.21.15.62:0/3689375446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e9b40 msgr2=0x5560640ff3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:17.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.611+0000 7f1786d9d640 1 --2- 172.21.15.62:0/3689375446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e9b40 0x5560640ff3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.611+0000 7f1786d9d640 1 -- 172.21.15.62:0/3689375446 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560640e5960 con 0x5560640f3370 2024-09-15T23:26:17.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.611+0000 7f1786d9d640 1 --2- 172.21.15.62:0/3689375446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x55606410b2d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1780009400 tx=0x7f17800a3c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf30f5dfd1fb840e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:17.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.612+0000 7f176ffff640 1 -- 172.21.15.62:0/3689375446 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800cc990 con 0x5560640f3370 2024-09-15T23:26:17.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.613+0000 7f176ffff640 1 -- 172.21.15.62:0/3689375446 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f177801c070 con 0x5560640f3370 2024-09-15T23:26:17.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.614+0000 7f178ec13740 1 -- 172.21.15.62:0/3689375446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 msgr2=0x55606410b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:17.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.614+0000 7f178ec13740 1 --2- 172.21.15.62:0/3689375446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x55606410b2d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1780009400 tx=0x7f17800a3c80 comp rx=0 tx=0).stop 2024-09-15T23:26:17.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.614+0000 7f178ec13740 1 -- 172.21.15.62:0/3689375446 shutdown_connections 2024-09-15T23:26:17.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.614+0000 7f178ec13740 1 --2- 172.21.15.62:0/3689375446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x55606410b2d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.614+0000 7f178ec13740 1 --2- 172.21.15.62:0/3689375446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e9b40 0x5560640ff3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.614+0000 7f178ec13740 1 --2- 172.21.15.62:0/3689375446 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e85d0 0x5560640dfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.614+0000 7f178ec13740 1 -- 172.21.15.62:0/3689375446 >> 172.21.15.62:0/3689375446 conn(0x5560640b1310 msgr2=0x5560640efef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:17.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.621+0000 7f178ec13740 1 -- 172.21.15.62:0/3689375446 shutdown_connections 2024-09-15T23:26:17.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.621+0000 7f178ec13740 1 -- 172.21.15.62:0/3689375446 wait complete. 2024-09-15T23:26:17.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.621+0000 7f178ec13740 1 -- start start 2024-09-15T23:26:17.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.621+0000 7f178ec13740 1 Processor -- start 2024-09-15T23:26:17.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.622+0000 7f178ec13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e85d0 0x5560640ed090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.622+0000 7f178ec13740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 0x5560640ed5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.622+0000 7f178ec13740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640edb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.622+0000 7f178ec13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f17881d58f0 con 0x5560640e85d0 2024-09-15T23:26:17.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.622+0000 7f178ec13740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f17881d6d60 con 0x5560640f3370 2024-09-15T23:26:17.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.622+0000 7f178ec13740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f17881d5fc0 con 0x5560640e9b40 2024-09-15T23:26:17.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.622+0000 7f178ec13740 10 client.0 _ll_register_callbacks cb 0x7f17800976b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:17.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.622+0000 7f1785d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 0x5560640ed5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.623+0000 7f1785d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 0x5560640ed5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47816/0 (socket says 172.21.15.62:47816) 2024-09-15T23:26:17.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.623+0000 7f1785d9b640 1 -- 172.21.15.62:0/3282282373 learned_addr learned my addr 172.21.15.62:0/3282282373 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:17.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.623+0000 7f1786d9d640 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640edb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.623+0000 7f1785d9b640 1 -- 172.21.15.62:0/3282282373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 msgr2=0x5560640edb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:17.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.623+0000 7f1785d9b640 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640edb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.623+0000 7f1785d9b640 1 -- 172.21.15.62:0/3282282373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e85d0 msgr2=0x5560640ed090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:17.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.623+0000 7f1785d9b640 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e85d0 0x5560640ed090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.623+0000 7f1785d9b640 1 -- 172.21.15.62:0/3282282373 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55606416d3a0 con 0x5560640e9b40 2024-09-15T23:26:17.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.628+0000 7f1785d9b640 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 0x5560640ed5d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1778009300 tx=0x7f17780094c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91a25b232d93a20b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:17.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.629+0000 7f176dffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:26:17.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.629+0000 7f176dffb640 1 -- 172.21.15.62:0/3282282373 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1778015e70 con 0x5560640e9b40 2024-09-15T23:26:17.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.629+0000 7f176dffb640 1 -- 172.21.15.62:0/3282282373 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f177802c410 con 0x5560640e9b40 2024-09-15T23:26:17.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.630+0000 7f178ec13740 1 -- 172.21.15.62:0/3282282373 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556064055d70 con 0x5560640e9b40 2024-09-15T23:26:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.631+0000 7f176dffb640 1 -- 172.21.15.62:0/3282282373 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f177803c050 con 0x5560640e9b40 2024-09-15T23:26:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.631+0000 7f178ec13740 10 client.27115 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.631+0000 7f178ec13740 1 -- 172.21.15.62:0/3282282373 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55606416e730 con 0x5560640e9b40 2024-09-15T23:26:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.632+0000 7f176dffb640 1 -- 172.21.15.62:0/3282282373 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556064055d70 con 0x5560640e9b40 2024-09-15T23:26:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.632+0000 7f176dffb640 1 -- 172.21.15.62:0/3282282373 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f17780214d0 con 0x5560640e9b40 2024-09-15T23:26:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.633+0000 7f178ec13740 10 client.27115 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.633+0000 7f178ec13740 10 client.27115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.633+0000 7f178ec13740 20 client.27115 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.633+0000 7f178ec13740 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55606416ea70 0x556064170e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.633+0000 7f178ec13740 4 client.27115 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.633+0000 7f178ec13740 1 -- 172.21.15.62:0/3282282373 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556063fd2170 con 0x55606416ea70 2024-09-15T23:26:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.637+0000 7f178659c640 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55606416ea70 0x556064170e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:17.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.641+0000 7f178659c640 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55606416ea70 0x556064170e90 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0f54bdd6cbdad5b server_cookie=e32efe8e37dbfa37 in_seq=0 out_seq=0 2024-09-15T23:26:17.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.642+0000 7f176dffb640 10 client.27115 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.643+0000 7f176dffb640 1 -- 172.21.15.62:0/3282282373 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556063fd2170 con 0x55606416ea70 2024-09-15T23:26:17.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.643+0000 7f176dffb640 10 client.27115 handle_command_reply: tid=0 2024-09-15T23:26:17.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f178ec13740 1 client.27115 shutdown 2024-09-15T23:26:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f178ec13740 1 -- 172.21.15.62:0/3282282373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 msgr2=0x5560640ed5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f178ec13740 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 0x5560640ed5d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1778009300 tx=0x7f17780094c0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f178ec13740 1 -- 172.21.15.62:0/3282282373 shutdown_connections 2024-09-15T23:26:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f178ec13740 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55606416ea70 0x556064170e90 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f178ec13740 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5560640f3370 0x5560640edb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f178ec13740 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5560640e9b40 0x5560640ed5d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f178ec13740 1 --2- 172.21.15.62:0/3282282373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5560640e85d0 0x5560640ed090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f176dffb640 0 client.27115 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.645+0000 7f178ec13740 1 -- 172.21.15.62:0/3282282373 >> 172.21.15.62:0/3282282373 conn(0x5560640b1310 msgr2=0x5560640f1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.646+0000 7f178ec13740 1 -- 172.21.15.62:0/3282282373 shutdown_connections 2024-09-15T23:26:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.646+0000 7f178ec13740 1 -- 172.21.15.62:0/3282282373 wait complete. 2024-09-15T23:26:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.646+0000 7f178ec13740 20 client.27115 trim_cache size 0 max 16384 2024-09-15T23:26:17.649 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 -- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f17700a32c0 msgr2=0x7f17700a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f17700a32c0 0x7f17700a5780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1778009490 tx=0x7f1778008040 comp rx=0 tx=0).stop 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 -- 172.21.15.62:0/2671678397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f178811b140 msgr2=0x7f1788116f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f178811b140 0x7f1788116f80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f178000b9e0 tx=0x7f178000bea0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 -- 172.21.15.62:0/2671678397 shutdown_connections 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f17700a32c0 0x7f17700a5780 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f178811b140 0x7f1788116f80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1788109830 0x7f1788116a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 --2- 172.21.15.62:0/2671678397 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1788109450 0x7f178811a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 -- 172.21.15.62:0/2671678397 >> 172.21.15.62:0/2671678397 conn(0x7f178806e140 msgr2=0x7f1788070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 -- 172.21.15.62:0/2671678397 shutdown_connections 2024-09-15T23:26:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:17.647+0000 7f178c83a640 1 -- 172.21.15.62:0/2671678397 wait complete. 2024-09-15T23:26:17.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:17 smithi062 ceph-mon[30949]: pgmap v670: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.0 MiB/s wr, 793 op/s 2024-09-15T23:26:18.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:17 smithi067 ceph-mon[35665]: pgmap v670: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.0 MiB/s wr, 793 op/s 2024-09-15T23:26:18.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:17 smithi148 ceph-mon[35610]: pgmap v670: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 7.0 MiB/s wr, 793 op/s 2024-09-15T23:26:19.342 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:26:19.342 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:26:19.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 600.0 s, 514.2 tps, lat 213.906 ms stddev 303.402 2024-09-15T23:26:19.918 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:19.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:19 smithi062 ceph-mon[30949]: pgmap v671: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.5 MiB/s wr, 724 op/s 2024-09-15T23:26:20.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:19 smithi067 ceph-mon[35665]: pgmap v671: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.5 MiB/s wr, 724 op/s 2024-09-15T23:26:20.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:19 smithi148 ceph-mon[35610]: pgmap v671: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.5 MiB/s wr, 724 op/s 2024-09-15T23:26:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.606+0000 7f6b25f77640 1 -- 172.21.15.62:0/2145541372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b20109450 msgr2=0x7f6b20110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.606+0000 7f6b25f77640 1 --2- 172.21.15.62:0/2145541372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b20109450 0x7f6b20110250 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009900 tx=0x7f6b1002a440 comp rx=0 tx=0).stop 2024-09-15T23:26:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.607+0000 7f6b25f77640 1 -- 172.21.15.62:0/2145541372 shutdown_connections 2024-09-15T23:26:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.607+0000 7f6b25f77640 1 --2- 172.21.15.62:0/2145541372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b2011b140 0x7f6b2011d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.607+0000 7f6b25f77640 1 --2- 172.21.15.62:0/2145541372 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b20109830 0x7f6b20109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.607+0000 7f6b25f77640 1 --2- 172.21.15.62:0/2145541372 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b20109450 0x7f6b20110250 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.607+0000 7f6b25f77640 1 -- 172.21.15.62:0/2145541372 >> 172.21.15.62:0/2145541372 conn(0x7f6b2006e2d0 msgr2=0x7f6b2006e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.613+0000 7f6b25f77640 1 -- 172.21.15.62:0/2145541372 shutdown_connections 2024-09-15T23:26:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.616+0000 7f6b25f77640 1 -- 172.21.15.62:0/2145541372 wait complete. 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.617+0000 7f6b25f77640 1 Processor -- start 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.617+0000 7f6b25f77640 1 -- start start 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.618+0000 7f6b25f77640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b20109450 0x7f6b20118380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.618+0000 7f6b25f77640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b20109830 0x7f6b201188c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.618+0000 7f6b25f77640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b2011b140 0x7f6b20111410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.618+0000 7f6b25f77640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6b20120f30 con 0x7f6b20109450 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.618+0000 7f6b25f77640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b20120db0 con 0x7f6b20109830 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.618+0000 7f6b25f77640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6b201210b0 con 0x7f6b2011b140 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.619+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b20109450 0x7f6b20118380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.619+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b20109450 0x7f6b20118380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36210/0 (socket says 172.21.15.62:36210) 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.619+0000 7f6b1f7fe640 1 -- 172.21.15.62:0/400039493 learned_addr learned my addr 172.21.15.62:0/400039493 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.619+0000 7f6b1ffff640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b2011b140 0x7f6b20111410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.619+0000 7f6b1effd640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b20109830 0x7f6b201188c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.620+0000 7f6b1effd640 1 -- 172.21.15.62:0/400039493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b2011b140 msgr2=0x7f6b20111410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.620+0000 7f6b1effd640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b2011b140 0x7f6b20111410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.620+0000 7f6b1effd640 1 -- 172.21.15.62:0/400039493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b20109450 msgr2=0x7f6b20118380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.620+0000 7f6b1effd640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b20109450 0x7f6b20118380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.620+0000 7f6b1effd640 1 -- 172.21.15.62:0/400039493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b20111b90 con 0x7f6b20109830 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.623+0000 7f6b1effd640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b20109830 0x7f6b201188c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6b1400d6d0 tx=0x7f6b1400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.624+0000 7f6b1cff9640 1 -- 172.21.15.62:0/400039493 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b14017070 con 0x7f6b20109830 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.624+0000 7f6b25f77640 1 -- 172.21.15.62:0/400039493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b20111e80 con 0x7f6b20109830 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.624+0000 7f6b25f77640 1 -- 172.21.15.62:0/400039493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b20115eb0 con 0x7f6b20109830 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.628+0000 7f6b28350740 1 Processor -- start 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.628+0000 7f6b28350740 1 -- start start 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.629+0000 7f6b28350740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7be90 0x556280c7c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.629+0000 7f6b28350740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c7c7b0 0x556280c86b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.629+0000 7f6b28350740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c87070 0x556280c89460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.629+0000 7f6b28350740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556280be1280 con 0x556280c7be90 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.629+0000 7f6b28350740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556280b88b10 con 0x556280c7c7b0 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.629+0000 7f6b28350740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556280bdd240 con 0x556280c87070 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.629+0000 7f6b1cff9640 1 -- 172.21.15.62:0/400039493 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b14009820 con 0x7f6b20109830 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.629+0000 7f6b1cff9640 1 -- 172.21.15.62:0/400039493 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b14004ce0 con 0x7f6b20109830 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.630+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c7c7b0 0x556280c86b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.630+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c7c7b0 0x556280c86b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:42166/0 (socket says 172.21.15.62:42166) 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.630+0000 7f6b1f7fe640 1 -- 172.21.15.62:0/811135354 learned_addr learned my addr 172.21.15.62:0/811135354 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.630+0000 7f6b1ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7be90 0x556280c7c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.639+0000 7f6b1ffff640 1 -- 172.21.15.62:0/811135354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c87070 msgr2=0x556280c89460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:21.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.639+0000 7f6b1ffff640 1 --2- 172.21.15.62:0/811135354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c87070 0x556280c89460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.640+0000 7f6b1ffff640 1 -- 172.21.15.62:0/811135354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c7c7b0 msgr2=0x556280c86b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.640+0000 7f6b1ffff640 1 --2- 172.21.15.62:0/811135354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c7c7b0 0x556280c86b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.640+0000 7f6b1ffff640 1 -- 172.21.15.62:0/811135354 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556280b951f0 con 0x556280c7be90 2024-09-15T23:26:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.649+0000 7f6b1ffff640 1 --2- 172.21.15.62:0/811135354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7be90 0x556280c7c270 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f6b18005fd0 tx=0x7f6b18010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ebb5512bed2d39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.651+0000 7f6b1cff9640 1 -- 172.21.15.62:0/400039493 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f6b14010460 con 0x7f6b20109830 2024-09-15T23:26:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.651+0000 7f6b057fa640 1 -- 172.21.15.62:0/811135354 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18016070 con 0x556280c7be90 2024-09-15T23:26:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.651+0000 7f6b057fa640 1 -- 172.21.15.62:0/811135354 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b18004d20 con 0x556280c7be90 2024-09-15T23:26:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.651+0000 7f6b057fa640 1 -- 172.21.15.62:0/811135354 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18012810 con 0x556280c7be90 2024-09-15T23:26:21.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.652+0000 7f6b28350740 1 -- 172.21.15.62:0/811135354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7be90 msgr2=0x556280c7c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.652+0000 7f6b28350740 1 --2- 172.21.15.62:0/811135354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7be90 0x556280c7c270 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f6b18005fd0 tx=0x7f6b18010740 comp rx=0 tx=0).stop 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.652+0000 7f6b28350740 1 -- 172.21.15.62:0/811135354 shutdown_connections 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.652+0000 7f6b28350740 1 --2- 172.21.15.62:0/811135354 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c87070 0x556280c89460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.652+0000 7f6b28350740 1 --2- 172.21.15.62:0/811135354 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c7c7b0 0x556280c86b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.652+0000 7f6b28350740 1 --2- 172.21.15.62:0/811135354 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7be90 0x556280c7c270 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.652+0000 7f6b28350740 1 -- 172.21.15.62:0/811135354 >> 172.21.15.62:0/811135354 conn(0x556280c78420 msgr2=0x556280c78800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.653+0000 7f6b1cff9640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b080a3450 0x7f6b080a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.653+0000 7f6b1cff9640 1 -- 172.21.15.62:0/400039493 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6b140cf4c0 con 0x7f6b20109830 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.654+0000 7f6b28350740 1 -- 172.21.15.62:0/811135354 shutdown_connections 2024-09-15T23:26:21.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.654+0000 7f6b28350740 1 -- 172.21.15.62:0/811135354 wait complete. 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.654+0000 7f6b28350740 1 -- start start 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.654+0000 7f6b28350740 1 Processor -- start 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.654+0000 7f6b28350740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c86820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.655+0000 7f6b1f7fe640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b080a3450 0x7f6b080a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.655+0000 7f6b28350740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c87070 0x556280c83ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.655+0000 7f6b28350740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c84490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.655+0000 7f6b28350740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556280c89bb0 con 0x556280c7c7b0 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.655+0000 7f6b28350740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556280b9bf10 con 0x556280c87070 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.655+0000 7f6b28350740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556280c89d30 con 0x556280c84010 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.655+0000 7f6b28350740 10 client.0 _ll_register_callbacks cb 0x556280b64a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.656+0000 7f6b1ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c84490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.656+0000 7f6b1ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c84490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47856/0 (socket says 172.21.15.62:47856) 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.656+0000 7f6b1ffff640 1 -- 172.21.15.62:0/4229083368 learned_addr learned my addr 172.21.15.62:0/4229083368 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:21.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.656+0000 7f6b1effd640 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c87070 0x556280c83ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.671 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:26:21.671 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:26:21.671 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "1f5700ea-cb4f-4c7c-8073-46f70c3f189c", 2024-09-15T23:26:21.671 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:26:21.671 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.657+0000 7f6b1f7fe640 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c86820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.658+0000 7f6b1f7fe640 1 -- 172.21.15.62:0/4229083368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 msgr2=0x556280c84490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.658+0000 7f6b1f7fe640 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c84490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.658+0000 7f6b1f7fe640 1 -- 172.21.15.62:0/4229083368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c87070 msgr2=0x556280c83ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.658+0000 7f6b1f7fe640 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c87070 0x556280c83ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.658+0000 7f6b1f7fe640 1 -- 172.21.15.62:0/4229083368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5562809526a0 con 0x556280c7c7b0 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.658+0000 7f6b1f7fe640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b080a3450 0x7f6b080a5910 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6b100057c0 tx=0x7f6b10005750 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.659+0000 7f6b1f7fe640 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c86820 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x556280c79280 tx=0x7f6b1003e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf72ba150c80896 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.659+0000 7f6b037f6640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.662+0000 7f6b037f6640 1 -- 172.21.15.62:0/4229083368 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1004e070 con 0x556280c7c7b0 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.662+0000 7f6b28350740 1 -- 172.21.15.62:0/4229083368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b18004d20 con 0x556280c7c7b0 2024-09-15T23:26:21.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.662+0000 7f6b037f6640 1 -- 172.21.15.62:0/4229083368 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6b10032240 con 0x556280c7c7b0 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.663+0000 7f6b037f6640 1 -- 172.21.15.62:0/4229083368 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6b1005a050 con 0x556280c7c7b0 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.663+0000 7f6b28350740 10 client.17814 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.664+0000 7f6b28350740 1 -- 172.21.15.62:0/4229083368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556280c95830 con 0x556280c7c7b0 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.674+0000 7f6b037f6640 1 -- 172.21.15.62:0/4229083368 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b18004d20 con 0x556280c7c7b0 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.674+0000 7f6b037f6640 1 -- 172.21.15.62:0/4229083368 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f6b10004990 con 0x556280c7c7b0 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.674+0000 7f6b28350740 10 client.17814 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.674+0000 7f6b28350740 10 client.17814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.674+0000 7f6b28350740 20 client.17814 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.674+0000 7f6b28350740 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556280c970b0 0x556280c99550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.674+0000 7f6b28350740 4 client.17814 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.674+0000 7f6b28350740 1 -- 172.21.15.62:0/4229083368 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556280bee7f0 con 0x556280c970b0 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.675+0000 7f6b1effd640 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556280c970b0 0x556280c99550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.684+0000 7f6b037f6640 10 client.17814 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.684+0000 7f6b1effd640 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556280c970b0 0x556280c99550 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c32d15907d15ce4 server_cookie=dcfaa3e8a2570f83 in_seq=0 out_seq=0 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.690+0000 7f6b037f6640 1 -- 172.21.15.62:0/4229083368 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556280bee7f0 con 0x556280c970b0 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.691+0000 7f6b037f6640 10 client.17814 handle_command_reply: tid=0 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.691+0000 7f6b28350740 1 client.17814 shutdown 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.691+0000 7f6b28350740 1 -- 172.21.15.62:0/4229083368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 msgr2=0x556280c86820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.691+0000 7f6b28350740 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c86820 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x556280c79280 tx=0x7f6b1003e9f0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.694+0000 7f6b28350740 1 -- 172.21.15.62:0/4229083368 shutdown_connections 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.694+0000 7f6b28350740 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556280c970b0 0x556280c99550 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.694+0000 7f6b28350740 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c84490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.694+0000 7f6b28350740 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c87070 0x556280c83ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.694+0000 7f6b28350740 1 --2- 172.21.15.62:0/4229083368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c86820 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.694+0000 7f6b28350740 1 -- 172.21.15.62:0/4229083368 >> 172.21.15.62:0/4229083368 conn(0x556280c78420 msgr2=0x556280c7d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.695+0000 7f6b037f6640 0 client.17814 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.699+0000 7f6b28350740 1 -- 172.21.15.62:0/4229083368 shutdown_connections 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.701+0000 7f6b28350740 1 -- 172.21.15.62:0/4229083368 wait complete. 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.702+0000 7f6b28350740 20 client.17814 trim_cache size 0 max 16384 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.746+0000 7f6b28350740 1 Processor -- start 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.747+0000 7f6b28350740 1 -- start start 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.747+0000 7f6b28350740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c78420 0x556280c7d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.747+0000 7f6b28350740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c7de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.747+0000 7f6b28350740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c88460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.748+0000 7f6b28350740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c0124f0 con 0x556280c7c7b0 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.748+0000 7f6b28350740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c00fdc0 con 0x556280c78420 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.748+0000 7f6b1ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c88460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.748+0000 7f6b1ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c88460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47862/0 (socket says 172.21.15.62:47862) 2024-09-15T23:26:21.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.748+0000 7f6b1ffff640 1 -- 172.21.15.62:0/1718056513 learned_addr learned my addr 172.21.15.62:0/1718056513 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:21.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.748+0000 7f6b28350740 1 -- 172.21.15.62:0/1718056513 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c00dc10 con 0x556280c84010 2024-09-15T23:26:21.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.748+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c78420 0x556280c7d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.749+0000 7f6b1effd640 1 --2- 172.21.15.62:0/1718056513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c7de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.749+0000 7f6b1ffff640 1 -- 172.21.15.62:0/1718056513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c78420 msgr2=0x556280c7d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:21.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.749+0000 7f6b1ffff640 1 --2- 172.21.15.62:0/1718056513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c78420 0x556280c7d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.749+0000 7f6b1ffff640 1 -- 172.21.15.62:0/1718056513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 msgr2=0x556280c7de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.749+0000 7f6b1ffff640 1 --2- 172.21.15.62:0/1718056513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c7de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.749+0000 7f6b1ffff640 1 -- 172.21.15.62:0/1718056513 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556280c4b4f0 con 0x556280c84010 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.751+0000 7f6b1ffff640 1 --2- 172.21.15.62:0/1718056513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c88460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6b18004980 tx=0x7f6b1801c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a171abe0d5d2432f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.751+0000 7f6b04ff9640 1 -- 172.21.15.62:0/1718056513 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18016040 con 0x556280c84010 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.752+0000 7f6b04ff9640 1 -- 172.21.15.62:0/1718056513 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b18005020 con 0x556280c84010 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.752+0000 7f6b04ff9640 1 -- 172.21.15.62:0/1718056513 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18012740 con 0x556280c84010 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.752+0000 7f6b28350740 1 -- 172.21.15.62:0/1718056513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 msgr2=0x556280c88460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.752+0000 7f6b28350740 1 --2- 172.21.15.62:0/1718056513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c88460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6b18004980 tx=0x7f6b1801c040 comp rx=0 tx=0).stop 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.753+0000 7f6b28350740 1 -- 172.21.15.62:0/1718056513 shutdown_connections 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.753+0000 7f6b28350740 1 --2- 172.21.15.62:0/1718056513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c84010 0x556280c88460 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.753+0000 7f6b28350740 1 --2- 172.21.15.62:0/1718056513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c7de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.753+0000 7f6b28350740 1 --2- 172.21.15.62:0/1718056513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c78420 0x556280c7d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.753+0000 7f6b28350740 1 -- 172.21.15.62:0/1718056513 >> 172.21.15.62:0/1718056513 conn(0x556280c895c0 msgr2=0x556280c823a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:21.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.753+0000 7f6b28350740 1 -- 172.21.15.62:0/1718056513 shutdown_connections 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.753+0000 7f6b28350740 1 -- 172.21.15.62:0/1718056513 wait complete. 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.753+0000 7f6b28350740 1 -- start start 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.753+0000 7f6b28350740 1 Processor -- start 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b28350740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c78420 0x556280c8f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b28350740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c89ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c78420 0x556280c8f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b1f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c78420 0x556280c8f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47866/0 (socket says 172.21.15.62:47866) 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b1f7fe640 1 -- 172.21.15.62:0/1167006284 learned_addr learned my addr 172.21.15.62:0/1167006284 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b28350740 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c84010 0x556280c8a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c00e280 con 0x556280c7c7b0 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c00f6f0 con 0x556280c84010 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c00e950 con 0x556280c78420 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b1ffff640 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c84010 0x556280c8a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b28350740 10 client.0 _ll_register_callbacks cb 0x7f6b1000b190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b1f7fe640 1 -- 172.21.15.62:0/1167006284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c84010 msgr2=0x556280c8a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b1f7fe640 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c84010 0x556280c8a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b1f7fe640 1 -- 172.21.15.62:0/1167006284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 msgr2=0x556280c89ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b1f7fe640 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c89ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.754+0000 7f6b1f7fe640 1 -- 172.21.15.62:0/1167006284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556280c8aa70 con 0x556280c78420 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.755+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b18005020 con 0x556280c78420 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.755+0000 7f6b1f7fe640 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c78420 0x556280c8f440 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000b010 tx=0x7f6b10005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=304dca57c5083a49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.755+0000 7f6b02ff5640 10 client.27130 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.756+0000 7f6b02ff5640 1 -- 172.21.15.62:0/1167006284 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1004e040 con 0x556280c78420 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.756+0000 7f6b02ff5640 1 -- 172.21.15.62:0/1167006284 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6b10032240 con 0x556280c78420 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.756+0000 7f6b02ff5640 1 -- 172.21.15.62:0/1167006284 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6b10035040 con 0x556280c78420 2024-09-15T23:26:21.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.756+0000 7f6b28350740 10 client.27130 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.756+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({fsmap=35}) -- 0x556280d029e0 con 0x556280c78420 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.757+0000 7f6b02ff5640 1 -- 172.21.15.62:0/1167006284 <== mon.1 v2:172.21.15.148:3300/0 4 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f6b10032440 con 0x556280c78420 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.757+0000 7f6b28350740 10 client.27130 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.757+0000 7f6b28350740 10 client.27130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.757+0000 7f6b28350740 20 client.27130 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.757+0000 7f6b28350740 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556280c87070 0x556280d04fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.757+0000 7f6b28350740 4 client.27130 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.757+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556280be8d00 con 0x556280c87070 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.757+0000 7f6b1effd640 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556280c87070 0x556280d04fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.759+0000 7f6b1effd640 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556280c87070 0x556280d04fe0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82be89eecbab3c18 server_cookie=4cb3f06d400c9bc8 in_seq=0 out_seq=0 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.759+0000 7f6b02ff5640 10 client.27130 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b02ff5640 1 -- 172.21.15.62:0/1167006284 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556280be8d00 con 0x556280c87070 2024-09-15T23:26:21.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b02ff5640 10 client.27130 handle_command_reply: tid=0 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b28350740 1 client.27130 shutdown 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c78420 msgr2=0x556280c8f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b28350740 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c78420 0x556280c8f440 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000b010 tx=0x7f6b10005fd0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 shutdown_connections 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b28350740 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556280c87070 0x556280d04fe0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b28350740 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556280c84010 0x556280c8a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b28350740 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556280c7c7b0 0x556280c89ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b28350740 1 --2- 172.21.15.62:0/1167006284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556280c78420 0x556280c8f440 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b02ff5640 0 client.27130 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.760+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 >> 172.21.15.62:0/1167006284 conn(0x556280c895c0 msgr2=0x556280c82290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.761+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 shutdown_connections 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.761+0000 7f6b28350740 1 -- 172.21.15.62:0/1167006284 wait complete. 2024-09-15T23:26:21.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.761+0000 7f6b28350740 20 client.27130 trim_cache size 0 max 16384 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 -- 172.21.15.62:0/400039493 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b080a3450 msgr2=0x7f6b080a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b080a3450 0x7f6b080a5910 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6b100057c0 tx=0x7f6b10005750 comp rx=0 tx=0).stop 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 -- 172.21.15.62:0/400039493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b20109830 msgr2=0x7f6b201188c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b20109830 0x7f6b201188c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f6b1400d6d0 tx=0x7f6b1400db90 comp rx=0 tx=0).stop 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 -- 172.21.15.62:0/400039493 shutdown_connections 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6b080a3450 0x7f6b080a5910 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6b2011b140 0x7f6b20111410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b20109830 0x7f6b201188c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 --2- 172.21.15.62:0/400039493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b20109450 0x7f6b20118380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 -- 172.21.15.62:0/400039493 >> 172.21.15.62:0/400039493 conn(0x7f6b2006e2d0 msgr2=0x7f6b20072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 -- 172.21.15.62:0/400039493 shutdown_connections 2024-09-15T23:26:21.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:20.762+0000 7f6b067fc640 1 -- 172.21.15.62:0/400039493 wait complete. 2024-09-15T23:26:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: pgmap v672: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.4 MiB/s wr, 707 op/s 2024-09-15T23:26:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:26:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:26:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:26:21.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:21.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:26:21.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:26:21.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:21.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:21 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: pgmap v672: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.4 MiB/s wr, 707 op/s 2024-09-15T23:26:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:26:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:26:22.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:26:22.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:22.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:26:22.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:26:22.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:22.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:21 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: pgmap v672: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.4 MiB/s wr, 707 op/s 2024-09-15T23:26:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:26:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:26:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:26:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:22.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:26:22.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:26:22.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:22.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:21 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:22.576 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:26:22.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:22 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:23.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:22 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:23.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:22 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:26:23.077 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:23.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.712+0000 7f0d284df640 1 -- 172.21.15.62:0/1140347322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20102570 msgr2=0x7f0d201029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:23.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.712+0000 7f0d284df640 1 --2- 172.21.15.62:0/1140347322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20102570 0x7f0d201029b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009890 tx=0x7f0d1002a410 comp rx=0 tx=0).stop 2024-09-15T23:26:23.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.712+0000 7f0d284df640 1 -- 172.21.15.62:0/1140347322 shutdown_connections 2024-09-15T23:26:23.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.712+0000 7f0d284df640 1 --2- 172.21.15.62:0/1140347322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d20102ef0 0x7f0d2010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.712+0000 7f0d284df640 1 --2- 172.21.15.62:0/1140347322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20102570 0x7f0d201029b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.712+0000 7f0d284df640 1 --2- 172.21.15.62:0/1140347322 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d20100e20 0x7f0d20101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.712+0000 7f0d284df640 1 -- 172.21.15.62:0/1140347322 >> 172.21.15.62:0/1140347322 conn(0x7f0d200fc7d0 msgr2=0x7f0d200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:23.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.712+0000 7f0d284df640 1 -- 172.21.15.62:0/1140347322 shutdown_connections 2024-09-15T23:26:23.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.712+0000 7f0d284df640 1 -- 172.21.15.62:0/1140347322 wait complete. 2024-09-15T23:26:23.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.713+0000 7f0d284df640 1 Processor -- start 2024-09-15T23:26:23.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.713+0000 7f0d284df640 1 -- start start 2024-09-15T23:26:23.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.713+0000 7f0d284df640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20100e20 0x7f0d200fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.713+0000 7f0d284df640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d20102570 0x7f0d200fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d2627b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20100e20 0x7f0d200fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d284df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d20102ef0 0x7f0d200fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d2627b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20100e20 0x7f0d200fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51374/0 (socket says 172.21.15.62:51374) 2024-09-15T23:26:23.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d284df640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20111010 con 0x7f0d20102ef0 2024-09-15T23:26:23.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d2627b640 1 -- 172.21.15.62:0/4076886008 learned_addr learned my addr 172.21.15.62:0/4076886008 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:23.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20110e90 con 0x7f0d20100e20 2024-09-15T23:26:23.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20111190 con 0x7f0d20102570 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d20102570 0x7f0d200fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d26a7c640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d20102ef0 0x7f0d200fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d2627b640 1 -- 172.21.15.62:0/4076886008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d20102570 msgr2=0x7f0d200fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d2627b640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d20102570 0x7f0d200fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d2627b640 1 -- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d20102ef0 msgr2=0x7f0d200fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d2627b640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d20102ef0 0x7f0d200fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d2627b640 1 -- 172.21.15.62:0/4076886008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d201a6af0 con 0x7f0d20100e20 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.714+0000 7f0d26a7c640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d20102ef0 0x7f0d200fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.715+0000 7f0d2627b640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20100e20 0x7f0d200fcc80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00e9f0 tx=0x7f0d1c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.715+0000 7f0d177fe640 1 -- 172.21.15.62:0/4076886008 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c00ccc0 con 0x7f0d20100e20 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.715+0000 7f0d177fe640 1 -- 172.21.15.62:0/4076886008 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d1c005500 con 0x7f0d20100e20 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.715+0000 7f0d177fe640 1 -- 172.21.15.62:0/4076886008 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c010690 con 0x7f0d20100e20 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.715+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d201a6de0 con 0x7f0d20100e20 2024-09-15T23:26:23.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.715+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d201a7240 con 0x7f0d20100e20 2024-09-15T23:26:23.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.718+0000 7f0d177fe640 1 -- 172.21.15.62:0/4076886008 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0d1c0046c0 con 0x7f0d20100e20 2024-09-15T23:26:23.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.719+0000 7f0d2a8b8740 1 Processor -- start 2024-09-15T23:26:23.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d2a8b8740 1 -- start start 2024-09-15T23:26:23.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d177fe640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d0c0a3300 0x7f0d0c0a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893c0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893cb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555e893262e0 con 0x555e893c04a0 2024-09-15T23:26:23.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555e892cdce0 con 0x555e893cb680 2024-09-15T23:26:23.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555e893222a0 con 0x555e893c0dc0 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d177fe640 1 -- 172.21.15.62:0/4076886008 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0d1c0cff60 con 0x7f0d20100e20 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d2627b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893c0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d2627b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893c0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50470/0 (socket says 172.21.15.62:50470) 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d2627b640 1 -- 172.21.15.62:0/859341468 learned_addr learned my addr 172.21.15.62:0/859341468 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.720+0000 7f0d26a7c640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d0c0a3300 0x7f0d0c0a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d26a7c640 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d25a7a640 1 -- 172.21.15.62:0/859341468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 msgr2=0x555e893cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d25a7a640 1 -- 172.21.15.62:0/859341468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 msgr2=0x555e893c0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:23.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893c0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d25a7a640 1 -- 172.21.15.62:0/859341468 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e892b2440 con 0x555e893c0dc0 2024-09-15T23:26:23.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d2627b640 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893c0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:23.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d26a7c640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d0c0a3300 0x7f0d0c0a57c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800eba0 tx=0x7f0d18009d70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:23.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893cb140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0d10000c00 tx=0x7f0d10030d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84aaca98bf0ced35 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:23.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.721+0000 7f0d14ff9640 1 -- 172.21.15.62:0/859341468 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d100305a0 con 0x555e893c0dc0 2024-09-15T23:26:23.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.722+0000 7f0d14ff9640 1 -- 172.21.15.62:0/859341468 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d10041070 con 0x555e893c0dc0 2024-09-15T23:26:23.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.722+0000 7f0d14ff9640 1 -- 172.21.15.62:0/859341468 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d100049c0 con 0x555e893c0dc0 2024-09-15T23:26:23.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.722+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/859341468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 msgr2=0x555e893cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:23.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.722+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893cb140 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0d10000c00 tx=0x7f0d10030d80 comp rx=0 tx=0).stop 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.723+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/859341468 shutdown_connections 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.723+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.723+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893cb140 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.723+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/859341468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893c0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.723+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/859341468 >> 172.21.15.62:0/859341468 conn(0x555e893bca30 msgr2=0x555e893bce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.723+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/859341468 shutdown_connections 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.723+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/859341468 wait complete. 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.723+0000 7f0d2a8b8740 1 -- start start 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.723+0000 7f0d2a8b8740 1 Processor -- start 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893c8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893c8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555e893ce250 con 0x555e893c04a0 2024-09-15T23:26:23.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555e892e1050 con 0x555e893cb680 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555e893ce3d0 con 0x555e893c0dc0 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d2a8b8740 10 client.0 _ll_register_callbacks cb 0x555e89321160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d25a7a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d25a7a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893c8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47364/0 (socket says 172.21.15.62:47364) 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d25a7a640 1 -- 172.21.15.62:0/3155969901 learned_addr learned my addr 172.21.15.62:0/3155969901 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d2627b640 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d26a7c640 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d25a7a640 1 -- 172.21.15.62:0/3155969901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 msgr2=0x555e893c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.724+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d25a7a640 1 -- 172.21.15.62:0/3155969901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 msgr2=0x555e893cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d25a7a640 1 -- 172.21.15.62:0/3155969901 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e890973c0 con 0x555e893c0dc0 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d2627b640 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:23.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893c8250 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0d10030990 tx=0x7f0d100486c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6230a6d0476e5516 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:23.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:26:23.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3155969901 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d100482e0 con 0x555e893c0dc0 2024-09-15T23:26:23.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3155969901 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d10041070 con 0x555e893c0dc0 2024-09-15T23:26:23.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3155969901 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0d10048830 con 0x555e893c0dc0 2024-09-15T23:26:23.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3155969901 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0d10048a30 con 0x555e893c0dc0 2024-09-15T23:26:23.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d2a8b8740 10 client.27142 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:23.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.725+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3155969901 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555e893ce250 con 0x555e893c0dc0 2024-09-15T23:26:23.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.726+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3155969901 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d10041070 con 0x555e893c0dc0 2024-09-15T23:26:23.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.726+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3155969901 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f0d10046950 con 0x555e893c0dc0 2024-09-15T23:26:23.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.726+0000 7f0d2a8b8740 10 client.27142 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:23.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.726+0000 7f0d2a8b8740 10 client.27142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:23.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.726+0000 7f0d2a8b8740 20 client.27142 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:23.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.726+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555e893dad80 0x555e893dd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.726+0000 7f0d2a8b8740 4 client.27142 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.726+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3155969901 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e89333850 con 0x555e893dad80 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.726+0000 7f0d2627b640 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555e893dad80 0x555e893dd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.727+0000 7f0d2627b640 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555e893dad80 0x555e893dd170 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9003cd57936975a server_cookie=e35f67390b69bcd7 in_seq=0 out_seq=0 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.727+0000 7f0d09ffb640 10 client.27142 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.727+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3155969901 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555e89333850 con 0x555e893dad80 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.727+0000 7f0d09ffb640 10 client.27142 handle_command_reply: tid=0 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.727+0000 7f0d2a8b8740 1 client.27142 shutdown 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.727+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3155969901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 msgr2=0x555e893c8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.727+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893c8250 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0d10030990 tx=0x7f0d100486c0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3155969901 shutdown_connections 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555e893dad80 0x555e893dd170 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893c8250 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3155969901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:23.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3155969901 >> 172.21.15.62:0/3155969901 conn(0x555e893bca30 msgr2=0x555e893c2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:23.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d09ffb640 0 client.27142 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:23.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3155969901 shutdown_connections 2024-09-15T23:26:23.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3155969901 wait complete. 2024-09-15T23:26:23.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.728+0000 7f0d2a8b8740 20 client.27142 trim_cache size 0 max 16384 2024-09-15T23:26:23.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.753+0000 7f0d2a8b8740 1 Processor -- start 2024-09-15T23:26:24.045 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:23 smithi148 ceph-mon[35610]: pgmap v673: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.9 MiB/s wr, 676 op/s 2024-09-15T23:26:24.045 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:23 smithi067 ceph-mon[35665]: pgmap v673: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.9 MiB/s wr, 676 op/s 2024-09-15T23:26:24.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.754+0000 7f0d2a8b8740 1 -- start start 2024-09-15T23:26:24.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.755+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893c04a0 0x555e893d74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:24.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.755+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c0dc0 0x555e893c8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:24.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.755+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893cb680 0x555e893e2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.755+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b3870 con 0x555e893c0dc0 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.755+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b1140 con 0x555e893c04a0 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.755+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201aef90 con 0x555e893cb680 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.757+0000 7f0d25a7a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c0dc0 0x555e893c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.758+0000 7f0d25a7a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c0dc0 0x555e893c8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50488/0 (socket says 172.21.15.62:50488) 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.758+0000 7f0d25a7a640 1 -- 172.21.15.62:0/701967091 learned_addr learned my addr 172.21.15.62:0/701967091 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.758+0000 7f0d26a7c640 1 --2- 172.21.15.62:0/701967091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893cb680 0x555e893e2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.758+0000 7f0d25a7a640 1 -- 172.21.15.62:0/701967091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893cb680 msgr2=0x555e893e2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.758+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/701967091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893cb680 0x555e893e2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.758+0000 7f0d25a7a640 1 -- 172.21.15.62:0/701967091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893c04a0 msgr2=0x555e893d74e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.758+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/701967091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893c04a0 0x555e893d74e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.758+0000 7f0d25a7a640 1 -- 172.21.15.62:0/701967091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e893bc860 con 0x555e893c0dc0 2024-09-15T23:26:24.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.761+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/701967091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c0dc0 0x555e893c8440 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0d10030920 tx=0x7f0d10000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8807c06f49f2fa6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.773+0000 7f0d0bfff640 1 -- 172.21.15.62:0/701967091 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10005c90 con 0x555e893c0dc0 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.773+0000 7f0d0bfff640 1 -- 172.21.15.62:0/701967091 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d10041040 con 0x555e893c0dc0 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.773+0000 7f0d0bfff640 1 -- 172.21.15.62:0/701967091 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1002aa50 con 0x555e893c0dc0 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.775+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/701967091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c0dc0 msgr2=0x555e893c8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.775+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/701967091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c0dc0 0x555e893c8440 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0d10030920 tx=0x7f0d10000bc0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.778+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/701967091 shutdown_connections 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.778+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/701967091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893cb680 0x555e893e2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.778+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/701967091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c0dc0 0x555e893c8440 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.778+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/701967091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893c04a0 0x555e893d74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.778+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/701967091 >> 172.21.15.62:0/701967091 conn(0x555e893bca30 msgr2=0x555e893c3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.781+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/701967091 shutdown_connections 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.781+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/701967091 wait complete. 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.782+0000 7f0d2a8b8740 1 -- start start 2024-09-15T23:26:24.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.782+0000 7f0d2a8b8740 1 Processor -- start 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.783+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893d3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.783+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.783+0000 7f0d2a8b8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.783+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201af600 con 0x555e893c04a0 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.783+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201b0a70 con 0x555e893cb680 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.783+0000 7f0d2a8b8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0d201afcd0 con 0x555e893c0dc0 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.783+0000 7f0d2a8b8740 10 client.0 _ll_register_callbacks cb 0x7f0d1800e6b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.784+0000 7f0d26a7c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.784+0000 7f0d2627b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893d3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.784+0000 7f0d2627b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893d3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50490/0 (socket says 172.21.15.62:50490) 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.784+0000 7f0d2627b640 1 -- 172.21.15.62:0/3026474461 learned_addr learned my addr 172.21.15.62:0/3026474461 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.785+0000 7f0d2627b640 1 -- 172.21.15.62:0/3026474461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 msgr2=0x555e893d3e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.785+0000 7f0d2627b640 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893d3e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.785+0000 7f0d2627b640 1 -- 172.21.15.62:0/3026474461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 msgr2=0x555e893cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.785+0000 7f0d2627b640 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.785+0000 7f0d2627b640 1 -- 172.21.15.62:0/3026474461 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e893cf4e0 con 0x555e893c04a0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.792+0000 7f0d2627b640 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893d3920 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c098b80 tx=0x7f0d1c0056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102890aaedc1a4a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.793+0000 7f0d09ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.795+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3026474461 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c0af070 con 0x555e893c04a0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.795+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3026474461 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0d1c0b0270 con 0x555e893c04a0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.798+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3026474461 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d10041040 con 0x555e893c04a0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.799+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3026474461 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0d1c09c900 con 0x555e893c04a0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.800+0000 7f0d2a8b8740 10 client.17829 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.800+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3026474461 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555e894453e0 con 0x555e893c04a0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.801+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3026474461 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d10041040 con 0x555e893c04a0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.801+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3026474461 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f0d1c09c460 con 0x555e893c04a0 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.802+0000 7f0d2a8b8740 10 client.17829 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:24.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.802+0000 7f0d2a8b8740 10 client.17829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.802+0000 7f0d2a8b8740 20 client.17829 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.802+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555e89445ac0 0x555e89447ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.802+0000 7f0d2a8b8740 4 client.17829 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.802+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3026474461 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555e893c8b20 con 0x555e89445ac0 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.805+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555e89445ac0 0x555e89447ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.812+0000 7f0d25a7a640 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555e89445ac0 0x555e89447ee0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74df0b47eb40daf0 server_cookie=7f599a922a0170b9 in_seq=0 out_seq=0 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.813+0000 7f0d09ffb640 10 client.17829 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.814+0000 7f0d09ffb640 1 -- 172.21.15.62:0/3026474461 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555e893c8b20 con 0x555e89445ac0 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.814+0000 7f0d09ffb640 10 client.17829 handle_command_reply: tid=0 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.815+0000 7f0d2a8b8740 1 client.17829 shutdown 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.815+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3026474461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 msgr2=0x555e893d3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.815+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893d3920 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c098b80 tx=0x7f0d1c0056b0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.818+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3026474461 shutdown_connections 2024-09-15T23:26:24.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.818+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x555e89445ac0 0x555e89447ee0 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.818+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555e893cb680 0x555e893cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.818+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555e893c0dc0 0x555e893d3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.818+0000 7f0d2a8b8740 1 --2- 172.21.15.62:0/3026474461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555e893c04a0 0x555e893d3920 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.818+0000 7f0d09ffb640 0 client.17829 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.818+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3026474461 >> 172.21.15.62:0/3026474461 conn(0x555e893bca30 msgr2=0x555e893bb350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.818+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3026474461 shutdown_connections 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.822+0000 7f0d2a8b8740 1 -- 172.21.15.62:0/3026474461 wait complete. 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.822+0000 7f0d2a8b8740 20 client.17829 trim_cache size 0 max 16384 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:26:24.058 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:26:24.059 INFO:teuthology.orchestra.run.smithi062.stdout: "1f5700ea-cb4f-4c7c-8073-46f70c3f189c": { 2024-09-15T23:26:24.059 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:26:24.059 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "1f5700ea-cb4f-4c7c-8073-46f70c3f189c", 2024-09-15T23:26:24.059 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:26:24.059 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:26:24.059 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:26:24.059 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:26:24.059 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:23 smithi062 ceph-mon[30949]: pgmap v673: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 7.9 MiB/s wr, 676 op/s 2024-09-15T23:26:24.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d0c0a3300 msgr2=0x7f0d0c0a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d0c0a3300 0x7f0d0c0a57c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800eba0 tx=0x7f0d18009d70 comp rx=0 tx=0).stop 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20100e20 msgr2=0x7f0d200fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20100e20 0x7f0d200fcc80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00e9f0 tx=0x7f0d1c00eeb0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 shutdown_connections 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d0c0a3300 0x7f0d0c0a57c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d20102ef0 0x7f0d200fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0d20102570 0x7f0d200fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 --2- 172.21.15.62:0/4076886008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0d20100e20 0x7f0d200fcc80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 >> 172.21.15.62:0/4076886008 conn(0x7f0d200fc7d0 msgr2=0x7f0d2006a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.825+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 shutdown_connections 2024-09-15T23:26:24.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:23.827+0000 7f0d284df640 1 -- 172.21.15.62:0/4076886008 wait complete. 2024-09-15T23:26:24.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 605.0 s, 529.4 tps, lat 186.808 ms stddev 235.033 2024-09-15T23:26:25.398 INFO:tasks.cephfs.filesystem:scrub status for tag:1f5700ea-cb4f-4c7c-8073-46f70c3f189c - {'path': '/', 'tag': '1f5700ea-cb4f-4c7c-8073-46f70c3f189c', 'options': 'recursive,force'} 2024-09-15T23:26:26.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:25 smithi067 ceph-mon[35665]: pgmap v674: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 8.4 MiB/s wr, 659 op/s 2024-09-15T23:26:26.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:25 smithi148 ceph-mon[35610]: pgmap v674: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 8.4 MiB/s wr, 659 op/s 2024-09-15T23:26:26.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:25 smithi062 ceph-mon[30949]: pgmap v674: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 8.4 MiB/s wr, 659 op/s 2024-09-15T23:26:28.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:27 smithi067 ceph-mon[35665]: pgmap v675: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 9.1 MiB/s wr, 731 op/s 2024-09-15T23:26:28.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:27 smithi148 ceph-mon[35610]: pgmap v675: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 9.1 MiB/s wr, 731 op/s 2024-09-15T23:26:28.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:27 smithi062 ceph-mon[30949]: pgmap v675: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 9.1 MiB/s wr, 731 op/s 2024-09-15T23:26:29.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 610.0 s, 746.4 tps, lat 136.985 ms stddev 168.698 2024-09-15T23:26:29.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.660+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2632654328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fe00736c0 msgr2=0x7f4fe0119130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:29.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.660+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2632654328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fe00736c0 0x7f4fe0119130 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8009860 tx=0x7f4fc802a360 comp rx=0 tx=0).stop 2024-09-15T23:26:29.664 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.661+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2632654328 shutdown_connections 2024-09-15T23:26:29.665 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.661+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2632654328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fe00736c0 0x7f4fe0119130 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.665 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.661+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2632654328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe0072d20 0x7f4fe0073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.665 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.661+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2632654328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe0109700 0x7f4fe0109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.665 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.661+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2632654328 >> 172.21.15.62:0/2632654328 conn(0x7f4fe006ece0 msgr2=0x7f4fe0071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.667+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2632654328 shutdown_connections 2024-09-15T23:26:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.669+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2632654328 wait complete. 2024-09-15T23:26:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.670+0000 7f4fe6aa4640 1 Processor -- start 2024-09-15T23:26:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.673+0000 7f4fe6aa4640 1 -- start start 2024-09-15T23:26:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.673+0000 7f4fe6aa4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fe0072d20 0x7f4fe019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.673+0000 7f4fe6aa4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe00736c0 0x7f4fe019ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.673+0000 7f4fe6aa4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe0109700 0x7f4fe01a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.673+0000 7f4fe6aa4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe01135e0 con 0x7f4fe0109700 2024-09-15T23:26:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.673+0000 7f4fe6aa4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe0113460 con 0x7f4fe00736c0 2024-09-15T23:26:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.673+0000 7f4fe6aa4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe0113760 con 0x7f4fe0072d20 2024-09-15T23:26:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.674+0000 7f4fe4840640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fe0072d20 0x7f4fe019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.674+0000 7f4fe4840640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fe0072d20 0x7f4fe019e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:47388/0 (socket says 172.21.15.62:47388) 2024-09-15T23:26:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.674+0000 7f4fe4840640 1 -- 172.21.15.62:0/2631479312 learned_addr learned my addr 172.21.15.62:0/2631479312 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.675+0000 7f4fdffff640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe00736c0 0x7f4fe019ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.675+0000 7f4fdffff640 1 -- 172.21.15.62:0/2631479312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fe0072d20 msgr2=0x7f4fe019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.675+0000 7f4fdffff640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fe0072d20 0x7f4fe019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.675+0000 7f4fdffff640 1 -- 172.21.15.62:0/2631479312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe0109700 msgr2=0x7f4fe01a5910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:26:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.675+0000 7f4fdffff640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe0109700 0x7f4fe01a5910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.675+0000 7f4fdffff640 1 -- 172.21.15.62:0/2631479312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe01a5f80 con 0x7f4fe00736c0 2024-09-15T23:26:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.676+0000 7f4fdffff640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe00736c0 0x7f4fe019ee30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800bd50 tx=0x7f4fd800be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.676+0000 7f4fddffb640 1 -- 172.21.15.62:0/2631479312 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8019070 con 0x7f4fe00736c0 2024-09-15T23:26:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.676+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2631479312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fe01a6270 con 0x7f4fe00736c0 2024-09-15T23:26:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.676+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2631479312 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fe01a67c0 con 0x7f4fe00736c0 2024-09-15T23:26:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.677+0000 7f4fddffb640 1 -- 172.21.15.62:0/2631479312 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fd8014920 con 0x7f4fe00736c0 2024-09-15T23:26:29.680 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.677+0000 7f4fddffb640 1 -- 172.21.15.62:0/2631479312 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd80129e0 con 0x7f4fe00736c0 2024-09-15T23:26:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.681+0000 7f4fe9836740 1 Processor -- start 2024-09-15T23:26:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.686+0000 7f4fddffb640 1 -- 172.21.15.62:0/2631479312 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4fd801c070 con 0x7f4fe00736c0 2024-09-15T23:26:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.688+0000 7f4fddffb640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4fcc0a3520 0x7f4fcc0a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.688+0000 7f4fddffb640 1 -- 172.21.15.62:0/2631479312 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4fd80cf430 con 0x7f4fe00736c0 2024-09-15T23:26:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.688+0000 7f4fe9836740 1 -- start start 2024-09-15T23:26:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 0x5628743b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.691 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b12e0 0x5628743b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743b8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fe9836740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5628743142f0 con 0x5628743b09a0 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fe9836740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5628742bc270 con 0x5628743b12e0 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fe9836740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5628743102b0 con 0x5628743b6370 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fe5041640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4fcc0a3520 0x7f4fcc0a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fdffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b12e0 0x5628743b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fdffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b12e0 0x5628743b5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51412/0 (socket says 172.21.15.62:51412) 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fdffff640 1 -- 172.21.15.62:0/31729374 learned_addr learned my addr 172.21.15.62:0/31729374 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.689+0000 7f4fe5041640 1 --2- 172.21.15.62:0/31729374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743b8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.690+0000 7f4fdffff640 1 -- 172.21.15.62:0/31729374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 msgr2=0x5628743b8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.690+0000 7f4fdffff640 1 --2- 172.21.15.62:0/31729374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743b8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.690+0000 7f4fdffff640 1 -- 172.21.15.62:0/31729374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 msgr2=0x5628743b0da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:29.692 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.690+0000 7f4fdffff640 1 --2- 172.21.15.62:0/31729374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 0x5628743b0da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.690+0000 7f4fdffff640 1 -- 172.21.15.62:0/31729374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628742bb340 con 0x5628743b12e0 2024-09-15T23:26:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.690+0000 7f4fe5041640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4fcc0a3520 0x7f4fcc0a59e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4fc80095f0 tx=0x7f4fc802ae70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:29.693 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.690+0000 7f4fdffff640 1 --2- 172.21.15.62:0/31729374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b12e0 0x5628743b5da0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800bfd0 tx=0x7f4fd8097300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8809954a3a59c04 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:29.695 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.690+0000 7f4fd6ffd640 1 -- 172.21.15.62:0/31729374 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd809fa20 con 0x5628743b12e0 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.690+0000 7f4fd6ffd640 1 -- 172.21.15.62:0/31729374 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fd80a42e0 con 0x5628743b12e0 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.691+0000 7f4fe9836740 1 -- 172.21.15.62:0/31729374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b12e0 msgr2=0x5628743b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.691+0000 7f4fe9836740 1 --2- 172.21.15.62:0/31729374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b12e0 0x5628743b5da0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800bfd0 tx=0x7f4fd8097300 comp rx=0 tx=0).stop 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.691+0000 7f4fe9836740 1 -- 172.21.15.62:0/31729374 shutdown_connections 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.691+0000 7f4fe9836740 1 --2- 172.21.15.62:0/31729374 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743b8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.691+0000 7f4fe9836740 1 --2- 172.21.15.62:0/31729374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b12e0 0x5628743b5da0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.692+0000 7f4fe9836740 1 --2- 172.21.15.62:0/31729374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 0x5628743b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.692+0000 7f4fe9836740 1 -- 172.21.15.62:0/31729374 >> 172.21.15.62:0/31729374 conn(0x5628743ac010 msgr2=0x5628743ac410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.692+0000 7f4fe9836740 1 -- 172.21.15.62:0/31729374 shutdown_connections 2024-09-15T23:26:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.692+0000 7f4fe9836740 1 -- 172.21.15.62:0/31729374 wait complete. 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.692+0000 7f4fe9836740 1 -- start start 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.692+0000 7f4fe9836740 1 Processor -- start 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.692+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.692+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743bb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743c20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fe9836740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5628743b9390 con 0x5628743b12e0 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fe9836740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56287429d290 con 0x5628743b09a0 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fe9836740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5628743b9510 con 0x5628743b6370 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fe9836740 10 client.0 _ll_register_callbacks cb 0x5628742ce140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fdffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743bb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fdffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743bb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50514/0 (socket says 172.21.15.62:50514) 2024-09-15T23:26:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fdffff640 1 -- 172.21.15.62:0/2284640880 learned_addr learned my addr 172.21.15.62:0/2284640880 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fe4840640 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fe5041640 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743c20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fdffff640 1 -- 172.21.15.62:0/2284640880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 msgr2=0x5628743c20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fdffff640 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743c20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fdffff640 1 -- 172.21.15.62:0/2284640880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 msgr2=0x5628743bab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fdffff640 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fdffff640 1 -- 172.21.15.62:0/2284640880 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562874027210 con 0x5628743b12e0 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fdffff640 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743bb090 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4fd80980f0 tx=0x7f4fd80ccac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d9d2cb1dd785f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fbffff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fbffff640 1 -- 172.21.15.62:0/2284640880 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd80b8070 con 0x5628743b12e0 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.693+0000 7f4fbffff640 1 -- 172.21.15.62:0/2284640880 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4fd80afb90 con 0x5628743b12e0 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe9836740 1 -- 172.21.15.62:0/2284640880 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fd80a42e0 con 0x5628743b12e0 2024-09-15T23:26:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fbffff640 1 -- 172.21.15.62:0/2284640880 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4fd80afdd0 con 0x5628743b12e0 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe9836740 10 client.17835 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe9836740 1 -- 172.21.15.62:0/2284640880 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628740f3620 con 0x5628743b12e0 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fbffff640 1 -- 172.21.15.62:0/2284640880 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fd80a42e0 con 0x5628743b12e0 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fbffff640 1 -- 172.21.15.62:0/2284640880 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f4fd80ac030 con 0x5628743b12e0 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe9836740 10 client.17835 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe9836740 10 client.17835 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe9836740 20 client.17835 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe9836740 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5628743c63d0 0x5628743c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe9836740 4 client.17835 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe9836740 1 -- 172.21.15.62:0/2284640880 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562874321a20 con 0x5628743c63d0 2024-09-15T23:26:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.694+0000 7f4fe4840640 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5628743c63d0 0x5628743c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.695+0000 7f4fbffff640 10 client.17835 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.695+0000 7f4fe4840640 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5628743c63d0 0x5628743c8810 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54e062d55b712fe0 server_cookie=278d164fa133cd59 in_seq=0 out_seq=0 2024-09-15T23:26:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.696+0000 7f4fbffff640 1 -- 172.21.15.62:0/2284640880 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562874321a20 con 0x5628743c63d0 2024-09-15T23:26:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.696+0000 7f4fbffff640 10 client.17835 handle_command_reply: tid=0 2024-09-15T23:26:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.696+0000 7f4fe9836740 1 client.17835 shutdown 2024-09-15T23:26:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 -- 172.21.15.62:0/2284640880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 msgr2=0x5628743bb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743bb090 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f4fd80980f0 tx=0x7f4fd80ccac0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 -- 172.21.15.62:0/2284640880 shutdown_connections 2024-09-15T23:26:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5628743c63d0 0x5628743c8810 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743c20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743bb090 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.701 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 --2- 172.21.15.62:0/2284640880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.702 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 -- 172.21.15.62:0/2284640880 >> 172.21.15.62:0/2284640880 conn(0x5628743ac010 msgr2=0x5628743ad820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:29.702 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fbffff640 0 client.17835 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:29.702 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 -- 172.21.15.62:0/2284640880 shutdown_connections 2024-09-15T23:26:29.702 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 1 -- 172.21.15.62:0/2284640880 wait complete. 2024-09-15T23:26:29.702 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.697+0000 7f4fe9836740 20 client.17835 trim_cache size 0 max 16384 2024-09-15T23:26:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.719+0000 7f4fe9836740 1 Processor -- start 2024-09-15T23:26:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.719+0000 7f4fe9836740 1 -- start start 2024-09-15T23:26:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.719+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 0x5628743ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.719+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b12e0 0x5628743c2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.719+0000 7f4fe4840640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 0x5628743ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe4840640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 0x5628743ab0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50526/0 (socket says 172.21.15.62:50526) 2024-09-15T23:26:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe4840640 1 -- 172.21.15.62:0/4172790494 learned_addr learned my addr 172.21.15.62:0/4172790494 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe9836740 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b6370 0x5628743cd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe9836740 1 -- 172.21.15.62:0/4172790494 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc00123b0 con 0x5628743b09a0 2024-09-15T23:26:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe9836740 1 -- 172.21.15.62:0/4172790494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc000fc80 con 0x5628743b6370 2024-09-15T23:26:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe5041640 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b6370 0x5628743cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fdffff640 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b12e0 0x5628743c2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe9836740 1 -- 172.21.15.62:0/4172790494 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc000dad0 con 0x5628743b12e0 2024-09-15T23:26:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe4840640 1 -- 172.21.15.62:0/4172790494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b12e0 msgr2=0x5628743c2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe4840640 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b12e0 0x5628743c2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe4840640 1 -- 172.21.15.62:0/4172790494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b6370 msgr2=0x5628743cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe4840640 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b6370 0x5628743cd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.720+0000 7f4fe4840640 1 -- 172.21.15.62:0/4172790494 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628743b5610 con 0x5628743b09a0 2024-09-15T23:26:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.721+0000 7f4fe4840640 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 0x5628743ab0b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x5628743c90d0 tx=0x7f4fd0005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591b5c8fc247d3a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.721+0000 7f4fd67fc640 1 -- 172.21.15.62:0/4172790494 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0004050 con 0x5628743b09a0 2024-09-15T23:26:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.721+0000 7f4fd67fc640 1 -- 172.21.15.62:0/4172790494 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fd0005500 con 0x5628743b09a0 2024-09-15T23:26:29.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.721+0000 7f4fd67fc640 1 -- 172.21.15.62:0/4172790494 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0004ba0 con 0x5628743b09a0 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 -- 172.21.15.62:0/4172790494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 msgr2=0x5628743ab0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 0x5628743ab0b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x5628743c90d0 tx=0x7f4fd0005be0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 -- 172.21.15.62:0/4172790494 shutdown_connections 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b6370 0x5628743cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b12e0 0x5628743c2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 --2- 172.21.15.62:0/4172790494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b09a0 0x5628743ab0b0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 -- 172.21.15.62:0/4172790494 >> 172.21.15.62:0/4172790494 conn(0x5628743ac010 msgr2=0x5628743aa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 -- 172.21.15.62:0/4172790494 shutdown_connections 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 -- 172.21.15.62:0/4172790494 wait complete. 2024-09-15T23:26:29.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.722+0000 7f4fe9836740 1 -- start start 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe9836740 1 Processor -- start 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743be480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe9836740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743c14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe9836740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc000e140 con 0x5628743b12e0 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe9836740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc000f5b0 con 0x5628743b09a0 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe9836740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc000e810 con 0x5628743b6370 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe4840640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fdffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743be480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe4840640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51438/0 (socket says 172.21.15.62:51438) 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe4840640 1 -- 172.21.15.62:0/3444516532 learned_addr learned my addr 172.21.15.62:0/3444516532 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.723+0000 7f4fe5041640 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743c14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.729 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fe9836740 10 client.0 _ll_register_callbacks cb 0x7f4fd8097ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fe4840640 1 -- 172.21.15.62:0/3444516532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 msgr2=0x5628743c14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fe4840640 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743c14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fe4840640 1 -- 172.21.15.62:0/3444516532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 msgr2=0x5628743be480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fe4840640 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743be480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fe4840640 1 -- 172.21.15.62:0/3444516532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628743c1ab0 con 0x5628743b09a0 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fe5041640 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743c14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fe4840640 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bdf40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0000c00 tx=0x7f4fd0005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b67ecef1a4902eb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fbffff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fbffff640 1 -- 172.21.15.62:0/3444516532 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd00056b0 con 0x5628743b09a0 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fbffff640 1 -- 172.21.15.62:0/3444516532 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4fd0019040 con 0x5628743b09a0 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.724+0000 7f4fe9836740 1 -- 172.21.15.62:0/3444516532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fd0005500 con 0x5628743b09a0 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fbffff640 1 -- 172.21.15.62:0/3444516532 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4fd0047050 con 0x5628743b09a0 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fe9836740 10 client.27410 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fe9836740 1 -- 172.21.15.62:0/3444516532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628744318d0 con 0x5628743b09a0 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fbffff640 1 -- 172.21.15.62:0/3444516532 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fd0005500 con 0x5628743b09a0 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fbffff640 1 -- 172.21.15.62:0/3444516532 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f4fd0003ae0 con 0x5628743b09a0 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fe9836740 10 client.27410 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fe9836740 10 client.27410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fe9836740 20 client.27410 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fe9836740 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562874432280 0x5628744346a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fe9836740 4 client.27410 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fe9836740 1 -- 172.21.15.62:0/3444516532 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56287431bd70 con 0x562874432280 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.725+0000 7f4fdffff640 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562874432280 0x5628744346a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.726+0000 7f4fdffff640 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562874432280 0x5628744346a0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=372eb933dc47e0ef server_cookie=d3041211146f54da in_seq=0 out_seq=0 2024-09-15T23:26:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.726+0000 7f4fbffff640 10 client.27410 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.740+0000 7f4fbffff640 1 -- 172.21.15.62:0/3444516532 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56287431bd70 con 0x562874432280 2024-09-15T23:26:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.740+0000 7f4fbffff640 10 client.27410 handle_command_reply: tid=0 2024-09-15T23:26:29.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.740+0000 7f4fe9836740 1 client.27410 shutdown 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.740+0000 7f4fe9836740 1 -- 172.21.15.62:0/3444516532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 msgr2=0x5628743bdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.740+0000 7f4fe9836740 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bdf40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0000c00 tx=0x7f4fd0005250 comp rx=0 tx=0).stop 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.740+0000 7f4fe9836740 1 -- 172.21.15.62:0/3444516532 shutdown_connections 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.740+0000 7f4fe9836740 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x562874432280 0x5628744346a0 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.740+0000 7f4fe9836740 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5628743b6370 0x5628743c14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.740+0000 7f4fe9836740 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5628743b12e0 0x5628743be480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.741+0000 7f4fe9836740 1 --2- 172.21.15.62:0/3444516532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5628743b09a0 0x5628743bdf40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.741+0000 7f4fbffff640 0 client.27410 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.741+0000 7f4fe9836740 1 -- 172.21.15.62:0/3444516532 >> 172.21.15.62:0/3444516532 conn(0x5628743ac010 msgr2=0x5628743ab1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.741+0000 7f4fe9836740 1 -- 172.21.15.62:0/3444516532 shutdown_connections 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.741+0000 7f4fe9836740 1 -- 172.21.15.62:0/3444516532 wait complete. 2024-09-15T23:26:29.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.741+0000 7f4fe9836740 20 client.27410 trim_cache size 0 max 16384 2024-09-15T23:26:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:26:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:26:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:26:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:26:29.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2631479312 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4fcc0a3520 msgr2=0x7f4fcc0a59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4fcc0a3520 0x7f4fcc0a59e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4fc80095f0 tx=0x7f4fc802ae70 comp rx=0 tx=0).stop 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2631479312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe00736c0 msgr2=0x7f4fe019ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe00736c0 0x7f4fe019ee30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4fd800bd50 tx=0x7f4fd800be40 comp rx=0 tx=0).stop 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2631479312 shutdown_connections 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4fcc0a3520 0x7f4fcc0a59e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fe0109700 0x7f4fe01a5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4fe00736c0 0x7f4fe019ee30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 --2- 172.21.15.62:0/2631479312 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4fe0072d20 0x7f4fe019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2631479312 >> 172.21.15.62:0/2631479312 conn(0x7f4fe006ece0 msgr2=0x7f4fe0116030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2631479312 shutdown_connections 2024-09-15T23:26:29.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:29.742+0000 7f4fe6aa4640 1 -- 172.21.15.62:0/2631479312 wait complete. 2024-09-15T23:26:30.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:29 smithi067 ceph-mon[35665]: pgmap v676: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 9.0 MiB/s wr, 704 op/s 2024-09-15T23:26:30.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:29 smithi148 ceph-mon[35610]: pgmap v676: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 9.0 MiB/s wr, 704 op/s 2024-09-15T23:26:30.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:29 smithi062 ceph-mon[30949]: pgmap v676: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 9.0 MiB/s wr, 704 op/s 2024-09-15T23:26:32.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:31 smithi067 ceph-mon[35665]: pgmap v677: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 10 MiB/s wr, 710 op/s 2024-09-15T23:26:32.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:31 smithi148 ceph-mon[35610]: pgmap v677: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 10 MiB/s wr, 710 op/s 2024-09-15T23:26:32.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:31 smithi062 ceph-mon[30949]: pgmap v677: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 10 MiB/s wr, 710 op/s 2024-09-15T23:26:33.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:32 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:26:33.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:32 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:26:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:32 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:26:33.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:32 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:26:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:32 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:26:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:32 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:26:34.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:33 smithi062 ceph-mon[30949]: pgmap v678: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 10 MiB/s wr, 705 op/s 2024-09-15T23:26:34.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:33 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:26:34.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:33 smithi067 ceph-mon[35665]: pgmap v678: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 10 MiB/s wr, 705 op/s 2024-09-15T23:26:34.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:33 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:26:34.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:33 smithi148 ceph-mon[35610]: pgmap v678: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 10 MiB/s wr, 705 op/s 2024-09-15T23:26:34.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:33 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:26:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 615.0 s, 585.6 tps, lat 164.981 ms stddev 190.049 2024-09-15T23:26:36.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:35 smithi062 ceph-mon[30949]: pgmap v679: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.8 MiB/s wr, 640 op/s 2024-09-15T23:26:36.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:35 smithi067 ceph-mon[35665]: pgmap v679: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.8 MiB/s wr, 640 op/s 2024-09-15T23:26:36.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:35 smithi148 ceph-mon[35610]: pgmap v679: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 8.8 MiB/s wr, 640 op/s 2024-09-15T23:26:37.684 INFO:tasks.quiescer.fs.[cephfs]:Total duration (43.7) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:26:37.684 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.85 2024-09-15T23:26:37.685 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:26:37.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:37 smithi062 ceph-mon[30949]: pgmap v680: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.1 MiB/s wr, 654 op/s 2024-09-15T23:26:38.174 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:38.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:37 smithi067 ceph-mon[35665]: pgmap v680: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.1 MiB/s wr, 654 op/s 2024-09-15T23:26:38.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:37 smithi148 ceph-mon[35610]: pgmap v680: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 9.1 MiB/s wr, 654 op/s 2024-09-15T23:26:38.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.807+0000 7f9c36e3c640 1 -- 172.21.15.62:0/789765622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3011b140 msgr2=0x7f9c3011d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:38.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.807+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/789765622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3011b140 0x7f9c3011d530 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009ca0 tx=0x7f9c2802a8f0 comp rx=0 tx=0).stop 2024-09-15T23:26:38.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.807+0000 7f9c36e3c640 1 -- 172.21.15.62:0/789765622 shutdown_connections 2024-09-15T23:26:38.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.807+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/789765622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3011b140 0x7f9c3011d530 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:38.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.807+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/789765622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30109830 0x7f9c30109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.816+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/789765622 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c30109450 0x7f9c30110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:38.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.816+0000 7f9c36e3c640 1 -- 172.21.15.62:0/789765622 >> 172.21.15.62:0/789765622 conn(0x7f9c3006e140 msgr2=0x7f9c3006e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:38.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.822+0000 7f9c36e3c640 1 -- 172.21.15.62:0/789765622 shutdown_connections 2024-09-15T23:26:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.822+0000 7f9c36e3c640 1 -- 172.21.15.62:0/789765622 wait complete. 2024-09-15T23:26:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.823+0000 7f9c36e3c640 1 Processor -- start 2024-09-15T23:26:38.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.824+0000 7f9c36e3c640 1 -- start start 2024-09-15T23:26:38.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.824+0000 7f9c36e3c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c30109450 0x7f9c301a7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:38.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.826+0000 7f9c36e3c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30109830 0x7f9c301a7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:38.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.826+0000 7f9c36e3c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3011b140 0x7f9c301ab9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:38.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.826+0000 7f9c2ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30109830 0x7f9c301a7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:38.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.826+0000 7f9c2ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30109830 0x7f9c301a7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:48428/0 (socket says 172.21.15.62:48428) 2024-09-15T23:26:38.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.827+0000 7f9c2ffff640 1 -- 172.21.15.62:0/3776693976 learned_addr learned my addr 172.21.15.62:0/3776693976 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:38.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.827+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9c30120de0 con 0x7f9c3011b140 2024-09-15T23:26:38.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.827+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9c30120c60 con 0x7f9c30109830 2024-09-15T23:26:38.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.827+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9c30120f60 con 0x7f9c30109450 2024-09-15T23:26:38.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.830+0000 7f9c34bd8640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c30109450 0x7f9c301a7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:38.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.830+0000 7f9c353d9640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3011b140 0x7f9c301ab9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:38.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.830+0000 7f9c2ffff640 1 -- 172.21.15.62:0/3776693976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c30109450 msgr2=0x7f9c301a7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:38.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.830+0000 7f9c2ffff640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c30109450 0x7f9c301a7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:38.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.830+0000 7f9c2ffff640 1 -- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3011b140 msgr2=0x7f9c301ab9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:38.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.830+0000 7f9c2ffff640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3011b140 0x7f9c301ab9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:38.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.830+0000 7f9c2ffff640 1 -- 172.21.15.62:0/3776693976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c301ac0b0 con 0x7f9c30109830 2024-09-15T23:26:38.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.835+0000 7f9c2ffff640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30109830 0x7f9c301a7640 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400d950 tx=0x7f9c2400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:38.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.835+0000 7f9c2dffb640 1 -- 172.21.15.62:0/3776693976 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c24017070 con 0x7f9c30109830 2024-09-15T23:26:38.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.836+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c301ac3a0 con 0x7f9c30109830 2024-09-15T23:26:38.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.836+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c301b3c80 con 0x7f9c30109830 2024-09-15T23:26:38.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.836+0000 7f9c2dffb640 1 -- 172.21.15.62:0/3776693976 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c24009c30 con 0x7f9c30109830 2024-09-15T23:26:38.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.836+0000 7f9c2dffb640 1 -- 172.21.15.62:0/3776693976 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2400f6b0 con 0x7f9c30109830 2024-09-15T23:26:38.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.839+0000 7f9c2dffb640 1 -- 172.21.15.62:0/3776693976 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9c2400f850 con 0x7f9c30109830 2024-09-15T23:26:38.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.839+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c14005290 con 0x7f9c30109830 2024-09-15T23:26:38.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.840+0000 7f9c2dffb640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c200a3450 0x7f9c200a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:38.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.841+0000 7f9c2dffb640 1 -- 172.21.15.62:0/3776693976 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9c240cf710 con 0x7f9c30109830 2024-09-15T23:26:38.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.842+0000 7f9c34bd8640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c200a3450 0x7f9c200a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:38.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.847+0000 7f9c34bd8640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c200a3450 0x7f9c200a5910 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c005fd0 tx=0x7f9c1c009470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:38.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:38.853+0000 7f9c2dffb640 1 -- 172.21.15.62:0/3776693976 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f9c2409c050 con 0x7f9c30109830 2024-09-15T23:26:38.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:38 smithi062 ceph-mon[30949]: pgmap v681: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.5 MiB/s wr, 607 op/s 2024-09-15T23:26:39.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.236+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c14005580 con 0x7f9c30109830 2024-09-15T23:26:39.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.238+0000 7f9c2dffb640 1 -- 172.21.15.62:0/3776693976 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7f9c24009de0 con 0x7f9c30109830 2024-09-15T23:26:39.240 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:26:39.240 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:26:39.240 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:26:39.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.246+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c200a3450 msgr2=0x7f9c200a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.246+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c200a3450 0x7f9c200a5910 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c005fd0 tx=0x7f9c1c009470 comp rx=0 tx=0).stop 2024-09-15T23:26:39.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.246+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30109830 msgr2=0x7f9c301a7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:39.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.246+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30109830 0x7f9c301a7640 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9c2400d950 tx=0x7f9c2400de10 comp rx=0 tx=0).stop 2024-09-15T23:26:39.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.248+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 shutdown_connections 2024-09-15T23:26:39.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.248+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9c200a3450 0x7f9c200a5910 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.248+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c3011b140 0x7f9c301ab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.248+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c30109830 0x7f9c301a7640 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.248+0000 7f9c36e3c640 1 --2- 172.21.15.62:0/3776693976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9c30109450 0x7f9c301a7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:39.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.248+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 >> 172.21.15.62:0/3776693976 conn(0x7f9c3006e140 msgr2=0x7f9c3011bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:39.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.250+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 shutdown_connections 2024-09-15T23:26:39.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:39.251+0000 7f9c36e3c640 1 -- 172.21.15.62:0/3776693976 wait complete. 2024-09-15T23:26:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:38 smithi067 ceph-mon[35665]: pgmap v681: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.5 MiB/s wr, 607 op/s 2024-09-15T23:26:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:38 smithi148 ceph-mon[35610]: pgmap v681: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.5 MiB/s wr, 607 op/s 2024-09-15T23:26:39.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 620.0 s, 819.6 tps, lat 124.701 ms stddev 144.741 2024-09-15T23:26:39.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:39 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3776693976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:40.162 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0' 2024-09-15T23:26:40.162 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0 2024-09-15T23:26:40.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:39 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3776693976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:39 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3776693976' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:40.723 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:41.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:40 smithi062 ceph-mon[30949]: pgmap v682: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 593 op/s 2024-09-15T23:26:41.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:40 smithi067 ceph-mon[35665]: pgmap v682: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 593 op/s 2024-09-15T23:26:41.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:40 smithi148 ceph-mon[35610]: pgmap v682: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 593 op/s 2024-09-15T23:26:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 -- 172.21.15.62:0/710886062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac102620 msgr2=0x7fd9ac104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/710886062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac102620 0x7fd9ac104a10 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd99c009900 tx=0x7fd99c02a410 comp rx=0 tx=0).stop 2024-09-15T23:26:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 -- 172.21.15.62:0/710886062 shutdown_connections 2024-09-15T23:26:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/710886062 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9ac107880 0x7fd9ac109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/710886062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ac104f50 0x7fd9ac107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/710886062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac102620 0x7fd9ac104a10 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 -- 172.21.15.62:0/710886062 >> 172.21.15.62:0/710886062 conn(0x7fd9ac0fc1e0 msgr2=0x7fd9ac0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 -- 172.21.15.62:0/710886062 shutdown_connections 2024-09-15T23:26:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 -- 172.21.15.62:0/710886062 wait complete. 2024-09-15T23:26:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.863+0000 7fd9b28e4640 1 Processor -- start 2024-09-15T23:26:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b28e4640 1 -- start start 2024-09-15T23:26:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b28e4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9ac102620 0x7fd9ac197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b28e4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ac104f50 0x7fd9ac1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b28e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac107880 0x7fd9ac19f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b28e4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac10d810 con 0x7fd9ac107880 2024-09-15T23:26:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b28e4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac10d690 con 0x7fd9ac104f50 2024-09-15T23:26:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b28e4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac10d990 con 0x7fd9ac102620 2024-09-15T23:26:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b0e81640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac107880 0x7fd9ac19f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b0e81640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac107880 0x7fd9ac19f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59708/0 (socket says 172.21.15.62:59708) 2024-09-15T23:26:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b0e81640 1 -- 172.21.15.62:0/2932206699 learned_addr learned my addr 172.21.15.62:0/2932206699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9abfff640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9ac102620 0x7fd9ac197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ac104f50 0x7fd9ac1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b0e81640 1 -- 172.21.15.62:0/2932206699 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9ac102620 msgr2=0x7fd9ac197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b0e81640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9ac102620 0x7fd9ac197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b0e81640 1 -- 172.21.15.62:0/2932206699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ac104f50 msgr2=0x7fd9ac1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b0e81640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ac104f50 0x7fd9ac1983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.864+0000 7fd9b0e81640 1 -- 172.21.15.62:0/2932206699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9ac19fa90 con 0x7fd9ac107880 2024-09-15T23:26:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.865+0000 7fd9b0e81640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac107880 0x7fd9ac19f420 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400eb40 tx=0x7fd9a400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.865+0000 7fd9a97fa640 1 -- 172.21.15.62:0/2932206699 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4004b20 con 0x7fd9ac107880 2024-09-15T23:26:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.865+0000 7fd9b28e4640 1 -- 172.21.15.62:0/2932206699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9ac19fd80 con 0x7fd9ac107880 2024-09-15T23:26:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.865+0000 7fd9a97fa640 1 -- 172.21.15.62:0/2932206699 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9a4004cc0 con 0x7fd9ac107880 2024-09-15T23:26:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.865+0000 7fd9a97fa640 1 -- 172.21.15.62:0/2932206699 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4010850 con 0x7fd9ac107880 2024-09-15T23:26:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.865+0000 7fd9b28e4640 1 -- 172.21.15.62:0/2932206699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9ac1a0220 con 0x7fd9ac107880 2024-09-15T23:26:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9a97fa640 1 -- 172.21.15.62:0/2932206699 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd9a4005020 con 0x7fd9ac107880 2024-09-15T23:26:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9b5684740 1 Processor -- start 2024-09-15T23:26:42.879 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9b5684740 1 -- start start 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9329a0 0x5638ad932da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad937da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad93a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9b5684740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5638ad8962f0 con 0x5638ad9329a0 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9b5684740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5638ad83e270 con 0x5638ad9332e0 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9b5684740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5638ad8922b0 con 0x5638ad938370 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9ab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9329a0 0x5638ad932da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9ab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9329a0 0x5638ad932da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59718/0 (socket says 172.21.15.62:59718) 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9ab7fe640 1 -- 172.21.15.62:0/56519602 learned_addr learned my addr 172.21.15.62:0/56519602 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.868+0000 7fd9abfff640 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad937da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9a97fa640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd9940a32f0 0x7fd9940a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad93a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9abfff640 1 -- 172.21.15.62:0/56519602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 msgr2=0x5638ad93a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9abfff640 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad93a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9abfff640 1 -- 172.21.15.62:0/56519602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9329a0 msgr2=0x5638ad932da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9abfff640 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9329a0 0x5638ad932da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9abfff640 1 -- 172.21.15.62:0/56519602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638ad83d340 con 0x5638ad9332e0 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9b0e81640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd9940a32f0 0x7fd9940a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad93a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9a97fa640 1 -- 172.21.15.62:0/2932206699 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd9a40d04f0 con 0x7fd9ac107880 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9abfff640 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad937da0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd99c031440 tx=0x7fd99c005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=206f8c7d752337b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9927fc640 1 -- 172.21.15.62:0/56519602 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c040070 con 0x5638ad9332e0 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9b0e81640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd9940a32f0 0x7fd9940a57b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4098bf0 tx=0x7fd9a4087040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9927fc640 1 -- 172.21.15.62:0/56519602 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99c004030 con 0x5638ad9332e0 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.869+0000 7fd9927fc640 1 -- 172.21.15.62:0/56519602 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c035040 con 0x5638ad9332e0 2024-09-15T23:26:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 -- 172.21.15.62:0/56519602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 msgr2=0x5638ad937da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad937da0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd99c031440 tx=0x7fd99c005e00 comp rx=0 tx=0).stop 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 -- 172.21.15.62:0/56519602 shutdown_connections 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad93a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad937da0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 --2- 172.21.15.62:0/56519602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9329a0 0x5638ad932da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 -- 172.21.15.62:0/56519602 >> 172.21.15.62:0/56519602 conn(0x5638ad92e010 msgr2=0x5638ad92e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 -- 172.21.15.62:0/56519602 shutdown_connections 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 -- 172.21.15.62:0/56519602 wait complete. 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.870+0000 7fd9b5684740 1 -- start start 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9b5684740 1 Processor -- start 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 0x5638ad93cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9332e0 0x5638ad93d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad9440f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9b5684740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5638ad93b2e0 con 0x5638ad9332e0 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9b5684740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5638ad81f290 con 0x5638ad9329a0 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9b5684740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5638ad93b460 con 0x5638ad938370 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9b5684740 10 client.0 _ll_register_callbacks cb 0x5638ad850140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9332e0 0x5638ad93d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9332e0 0x5638ad93d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59728/0 (socket says 172.21.15.62:59728) 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9abfff640 1 -- 172.21.15.62:0/3335770388 learned_addr learned my addr 172.21.15.62:0/3335770388 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 0x5638ad93cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad9440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9abfff640 1 -- 172.21.15.62:0/3335770388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 msgr2=0x5638ad9440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9abfff640 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad9440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9abfff640 1 -- 172.21.15.62:0/3335770388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 msgr2=0x5638ad93cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9abfff640 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 0x5638ad93cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9abfff640 1 -- 172.21.15.62:0/3335770388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638ad5a9210 con 0x5638ad9332e0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.871+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad9440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9abfff640 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9332e0 0x5638ad93d0b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd99c02a930 tx=0x7fd99c0095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed731d1a8b55b97 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd98b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd98b7fe640 1 -- 172.21.15.62:0/3335770388 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c040070 con 0x5638ad9332e0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd98b7fe640 1 -- 172.21.15.62:0/3335770388 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd99c0043e0 con 0x5638ad9332e0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9b5684740 1 -- 172.21.15.62:0/3335770388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd99c004030 con 0x5638ad9332e0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd98b7fe640 1 -- 172.21.15.62:0/3335770388 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd99c035040 con 0x5638ad9332e0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9b5684740 10 client.17859 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9b5684740 1 -- 172.21.15.62:0/3335770388 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5638ad675620 con 0x5638ad9332e0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd98b7fe640 1 -- 172.21.15.62:0/3335770388 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99c004030 con 0x5638ad9332e0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd98b7fe640 1 -- 172.21.15.62:0/3335770388 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fd99c0035f0 con 0x5638ad9332e0 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9b5684740 10 client.17859 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9b5684740 10 client.17859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:42.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9b5684740 20 client.17859 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9b5684740 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638ad948360 0x5638ad94a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9b5684740 4 client.17859 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.872+0000 7fd9b5684740 1 -- 172.21.15.62:0/3335770388 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638ad8a3a20 con 0x5638ad948360 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.873+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638ad948360 0x5638ad94a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.873+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638ad948360 0x5638ad94a750 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dae1689f9db58f42 server_cookie=e7c5f6f5cc9cc357 in_seq=0 out_seq=0 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.873+0000 7fd98b7fe640 10 client.17859 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd98b7fe640 1 -- 172.21.15.62:0/3335770388 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5638ad8a3a20 con 0x5638ad948360 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd98b7fe640 10 client.17859 handle_command_reply: tid=0 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 client.17859 shutdown 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 -- 172.21.15.62:0/3335770388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9332e0 msgr2=0x5638ad93d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9332e0 0x5638ad93d0b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fd99c02a930 tx=0x7fd99c0095f0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 -- 172.21.15.62:0/3335770388 shutdown_connections 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638ad948360 0x5638ad94a750 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd98b7fe640 0 client.17859 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad938370 0x5638ad9440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad9332e0 0x5638ad93d0b0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 --2- 172.21.15.62:0/3335770388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 0x5638ad93cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 -- 172.21.15.62:0/3335770388 >> 172.21.15.62:0/3335770388 conn(0x5638ad92e010 msgr2=0x5638ad92f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 -- 172.21.15.62:0/3335770388 shutdown_connections 2024-09-15T23:26:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 1 -- 172.21.15.62:0/3335770388 wait complete. 2024-09-15T23:26:42.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.874+0000 7fd9b5684740 20 client.17859 trim_cache size 0 max 16384 2024-09-15T23:26:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.888+0000 7fd9b5684740 1 Processor -- start 2024-09-15T23:26:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.888+0000 7fd9b5684740 1 -- start start 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9329a0 0x5638ad92f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad9440b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad92f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9b5684740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1acf30 con 0x5638ad938370 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9b5684740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1aa800 con 0x5638ad9332e0 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9b5684740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1a8650 con 0x5638ad9329a0 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9ab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad92f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9ab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad92f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59734/0 (socket says 172.21.15.62:59734) 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9ab7fe640 1 -- 172.21.15.62:0/2609013304 learned_addr learned my addr 172.21.15.62:0/2609013304 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9abfff640 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad9440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9329a0 0x5638ad92f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9abfff640 1 -- 172.21.15.62:0/2609013304 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9329a0 msgr2=0x5638ad92f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9abfff640 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9329a0 0x5638ad92f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9abfff640 1 -- 172.21.15.62:0/2609013304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 msgr2=0x5638ad92f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9abfff640 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad92f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9abfff640 1 -- 172.21.15.62:0/2609013304 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638ad94aac0 con 0x5638ad9332e0 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad92f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd9abfff640 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad9440b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0314e0 tx=0x7fd99c005a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8df0e371cc077218 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd991ffb640 1 -- 172.21.15.62:0/2609013304 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c040030 con 0x5638ad9332e0 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd991ffb640 1 -- 172.21.15.62:0/2609013304 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99c031d90 con 0x5638ad9332e0 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.889+0000 7fd991ffb640 1 -- 172.21.15.62:0/2609013304 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c044400 con 0x5638ad9332e0 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 -- 172.21.15.62:0/2609013304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 msgr2=0x5638ad9440b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad9440b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0314e0 tx=0x7fd99c005a40 comp rx=0 tx=0).stop 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 -- 172.21.15.62:0/2609013304 shutdown_connections 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad92f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9332e0 0x5638ad9440b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 --2- 172.21.15.62:0/2609013304 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9329a0 0x5638ad92f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 -- 172.21.15.62:0/2609013304 >> 172.21.15.62:0/2609013304 conn(0x5638ad92e010 msgr2=0x5638ad92d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 -- 172.21.15.62:0/2609013304 shutdown_connections 2024-09-15T23:26:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 -- 172.21.15.62:0/2609013304 wait complete. 2024-09-15T23:26:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.890+0000 7fd9b5684740 1 -- start start 2024-09-15T23:26:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9b5684740 1 Processor -- start 2024-09-15T23:26:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 0x5638ad9416d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9332e0 0x5638ad941c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9b5684740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad942bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9b5684740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1a8cc0 con 0x5638ad938370 2024-09-15T23:26:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9b5684740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1aa130 con 0x5638ad9329a0 2024-09-15T23:26:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9b5684740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd9ac1a9390 con 0x5638ad9332e0 2024-09-15T23:26:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9b5684740 10 client.0 _ll_register_callbacks cb 0x7fd9a0009660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9ab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad942bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9ab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad942bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59738/0 (socket says 172.21.15.62:59738) 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9ab7fe640 1 -- 172.21.15.62:0/1864852540 learned_addr learned my addr 172.21.15.62:0/1864852540 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 0x5638ad9416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9abfff640 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9332e0 0x5638ad941c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9ab7fe640 1 -- 172.21.15.62:0/1864852540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9332e0 msgr2=0x5638ad941c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9332e0 0x5638ad941c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9ab7fe640 1 -- 172.21.15.62:0/1864852540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 msgr2=0x5638ad9416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 0x5638ad9416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.891+0000 7fd9ab7fe640 1 -- 172.21.15.62:0/1864852540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638ad9431e0 con 0x5638ad938370 2024-09-15T23:26:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9ab7fe640 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad942bd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0000c00 tx=0x7fd9a0013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfbed7c615ab007 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd98b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd98b7fe640 1 -- 172.21.15.62:0/1864852540 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0026040 con 0x5638ad938370 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd98b7fe640 1 -- 172.21.15.62:0/1864852540 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd9a0010cf0 con 0x5638ad938370 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9b5684740 1 -- 172.21.15.62:0/1864852540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd99c031d90 con 0x5638ad938370 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd98b7fe640 1 -- 172.21.15.62:0/1864852540 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd9a0004020 con 0x5638ad938370 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9b5684740 10 client.17865 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9b5684740 1 -- 172.21.15.62:0/1864852540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5638ad9b2f90 con 0x5638ad938370 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd98b7fe640 1 -- 172.21.15.62:0/1864852540 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd99c031d90 con 0x5638ad938370 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd98b7fe640 1 -- 172.21.15.62:0/1864852540 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fd9a002c040 con 0x5638ad938370 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9b5684740 10 client.17865 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9b5684740 10 client.17865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9b5684740 20 client.17865 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9b5684740 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638ad9b38d0 0x5638ad9b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9b5684740 4 client.17865 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9b5684740 1 -- 172.21.15.62:0/1864852540 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5638ad89dd70 con 0x5638ad9b38d0 2024-09-15T23:26:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.892+0000 7fd9abfff640 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638ad9b38d0 0x5638ad9b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.893+0000 7fd98b7fe640 10 client.17865 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:42.893+0000 7fd9abfff640 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638ad9b38d0 0x5638ad9b5cf0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53b90e71e5e3df6e server_cookie=554d0e0016716a18 in_seq=0 out_seq=0 2024-09-15T23:26:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.000+0000 7fd98b7fe640 1 -- 172.21.15.62:0/1864852540 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5638ad89dd70 con 0x5638ad9b38d0 2024-09-15T23:26:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.000+0000 7fd98b7fe640 10 client.17865 handle_command_reply: tid=0 2024-09-15T23:26:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.000+0000 7fd9b5684740 1 client.17865 shutdown 2024-09-15T23:26:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.000+0000 7fd9b5684740 1 -- 172.21.15.62:0/1864852540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 msgr2=0x5638ad942bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.000+0000 7fd9b5684740 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad942bd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0000c00 tx=0x7fd9a0013040 comp rx=0 tx=0).stop 2024-09-15T23:26:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd9b5684740 1 -- 172.21.15.62:0/1864852540 shutdown_connections 2024-09-15T23:26:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd9b5684740 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5638ad9b38d0 0x5638ad9b5cf0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd9b5684740 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5638ad938370 0x5638ad942bd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd9b5684740 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5638ad9332e0 0x5638ad941c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd9b5684740 1 --2- 172.21.15.62:0/1864852540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5638ad9329a0 0x5638ad9416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd9b5684740 1 -- 172.21.15.62:0/1864852540 >> 172.21.15.62:0/1864852540 conn(0x5638ad92e010 msgr2=0x5638ad92d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd98b7fe640 0 client.17865 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:26:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:26:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:26:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:26:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd9b5684740 1 -- 172.21.15.62:0/1864852540 shutdown_connections 2024-09-15T23:26:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd9b5684740 1 -- 172.21.15.62:0/1864852540 wait complete. 2024-09-15T23:26:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.001+0000 7fd9b5684740 20 client.17865 trim_cache size 0 max 16384 2024-09-15T23:26:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 -- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd9940a32f0 msgr2=0x7fd9940a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd9940a32f0 0x7fd9940a57b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4098bf0 tx=0x7fd9a4087040 comp rx=0 tx=0).stop 2024-09-15T23:26:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 -- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac107880 msgr2=0x7fd9ac19f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac107880 0x7fd9ac19f420 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400eb40 tx=0x7fd9a400c2d0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 -- 172.21.15.62:0/2932206699 shutdown_connections 2024-09-15T23:26:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd9940a32f0 0x7fd9940a57b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9ac107880 0x7fd9ac19f420 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ac104f50 0x7fd9ac1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 --2- 172.21.15.62:0/2932206699 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd9ac102620 0x7fd9ac197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.002+0000 7fd9b28e4640 1 -- 172.21.15.62:0/2932206699 >> 172.21.15.62:0/2932206699 conn(0x7fd9ac0fc1e0 msgr2=0x7fd9ac103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.003+0000 7fd9b28e4640 1 -- 172.21.15.62:0/2932206699 shutdown_connections 2024-09-15T23:26:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:43.003+0000 7fd9b28e4640 1 -- 172.21.15.62:0/2932206699 wait complete. 2024-09-15T23:26:43.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:43 smithi148 ceph-mon[35610]: pgmap v683: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 610 op/s 2024-09-15T23:26:43.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:43 smithi062 ceph-mon[30949]: pgmap v683: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 610 op/s 2024-09-15T23:26:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:43 smithi067 ceph-mon[35665]: pgmap v683: 129 pgs: 129 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 610 op/s 2024-09-15T23:26:44.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.367+0000 7f27eb4ec640 1 -- 172.21.15.62:0/4223345089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e4114ab0 msgr2=0x7f27e4116ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:44.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.367+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/4223345089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e4114ab0 0x7f27e4116ea0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009ca0 tx=0x7f27dc02a980 comp rx=0 tx=0).stop 2024-09-15T23:26:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.368+0000 7f27eb4ec640 1 -- 172.21.15.62:0/4223345089 shutdown_connections 2024-09-15T23:26:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.368+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/4223345089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e4114ab0 0x7f27e4116ea0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.368+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/4223345089 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27e4109b10 0x7f27e4114570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.368+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/4223345089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27e41091f0 0x7f27e41095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.368+0000 7f27eb4ec640 1 -- 172.21.15.62:0/4223345089 >> 172.21.15.62:0/4223345089 conn(0x7f27e406e140 msgr2=0x7f27e406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:44.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.374+0000 7f27eb4ec640 1 -- 172.21.15.62:0/4223345089 shutdown_connections 2024-09-15T23:26:44.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.377+0000 7f27eb4ec640 1 -- 172.21.15.62:0/4223345089 wait complete. 2024-09-15T23:26:44.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.378+0000 7f27eb4ec640 1 Processor -- start 2024-09-15T23:26:44.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.378+0000 7f27eb4ec640 1 -- start start 2024-09-15T23:26:44.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.379+0000 7f27eb4ec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e41091f0 0x7f27e410a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.379+0000 7f27eb4ec640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27e4109b10 0x7f27e410a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.379+0000 7f27eb4ec640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27e4114ab0 0x7f27e410ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.379+0000 7f27eb4ec640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f27e411a8c0 con 0x7f27e4114ab0 2024-09-15T23:26:44.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.379+0000 7f27eb4ec640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27e411a740 con 0x7f27e41091f0 2024-09-15T23:26:44.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.379+0000 7f27eb4ec640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f27e411aa40 con 0x7f27e4109b10 2024-09-15T23:26:44.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.379+0000 7f27e8a87640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27e4109b10 0x7f27e410a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.379+0000 7f27e8a87640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27e4109b10 0x7f27e410a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:51338/0 (socket says 172.21.15.62:51338) 2024-09-15T23:26:44.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.379+0000 7f27e8a87640 1 -- 172.21.15.62:0/1390264629 learned_addr learned my addr 172.21.15.62:0/1390264629 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:44.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.380+0000 7f27e9288640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e41091f0 0x7f27e410a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.380+0000 7f27e9a89640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27e4114ab0 0x7f27e410ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.381+0000 7f27e9288640 1 -- 172.21.15.62:0/1390264629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27e4109b10 msgr2=0x7f27e410a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:44.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.381+0000 7f27e9288640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27e4109b10 0x7f27e410a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.381+0000 7f27e9288640 1 -- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27e4114ab0 msgr2=0x7f27e410ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:44.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.381+0000 7f27e9288640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27e4114ab0 0x7f27e410ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.381+0000 7f27e9288640 1 -- 172.21.15.62:0/1390264629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e41b3750 con 0x7f27e41091f0 2024-09-15T23:26:44.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.381+0000 7f27e9288640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e41091f0 0x7f27e410a3d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f27e000b3f0 tx=0x7f27e000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.383+0000 7f27da7fc640 1 -- 172.21.15.62:0/1390264629 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0015020 con 0x7f27e41091f0 2024-09-15T23:26:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.383+0000 7f27da7fc640 1 -- 172.21.15.62:0/1390264629 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e0011d10 con 0x7f27e41091f0 2024-09-15T23:26:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.383+0000 7f27da7fc640 1 -- 172.21.15.62:0/1390264629 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e000f2e0 con 0x7f27e41091f0 2024-09-15T23:26:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.383+0000 7f27eb4ec640 1 -- 172.21.15.62:0/1390264629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e41b3980 con 0x7f27e41091f0 2024-09-15T23:26:44.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.383+0000 7f27eb4ec640 1 -- 172.21.15.62:0/1390264629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e41b3e70 con 0x7f27e41091f0 2024-09-15T23:26:44.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.389+0000 7f27ed8c5740 1 Processor -- start 2024-09-15T23:26:44.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.390+0000 7f27ed8c5740 1 -- start start 2024-09-15T23:26:44.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.391+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b5219dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.391+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219e170 0x561b521a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.391+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521aae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.391+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561b52102160 con 0x561b5219d850 2024-09-15T23:26:44.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.391+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561b520a9720 con 0x561b521a8a30 2024-09-15T23:26:44.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.391+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561b520fe120 con 0x561b5219e170 2024-09-15T23:26:44.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.392+0000 7f27e8a87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.392+0000 7f27e8a87640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521aae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51214/0 (socket says 172.21.15.62:51214) 2024-09-15T23:26:44.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.392+0000 7f27e8a87640 1 -- 172.21.15.62:0/475713042 learned_addr learned my addr 172.21.15.62:0/475713042 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:44.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.393+0000 7f27e9a89640 1 --2- 172.21.15.62:0/475713042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b5219dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.397+0000 7f27e9a89640 1 -- 172.21.15.62:0/475713042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219e170 msgr2=0x561b521a84f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:26:44.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.397+0000 7f27e9a89640 1 --2- 172.21.15.62:0/475713042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219e170 0x561b521a84f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.397+0000 7f27e9a89640 1 -- 172.21.15.62:0/475713042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 msgr2=0x561b521aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:44.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.397+0000 7f27e9a89640 1 --2- 172.21.15.62:0/475713042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521aae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.397+0000 7f27e9a89640 1 -- 172.21.15.62:0/475713042 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b51e73ba0 con 0x561b5219d850 2024-09-15T23:26:44.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.398+0000 7f27e9a89640 1 --2- 172.21.15.62:0/475713042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b5219dc30 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f27dc02af70 tx=0x7f27dc02fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9055b8139638981 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:44.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.398+0000 7f27cf7fe640 1 -- 172.21.15.62:0/475713042 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc034040 con 0x561b5219d850 2024-09-15T23:26:44.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.398+0000 7f27cf7fe640 1 -- 172.21.15.62:0/475713042 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27dc02feb0 con 0x561b5219d850 2024-09-15T23:26:44.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.399+0000 7f27cf7fe640 1 -- 172.21.15.62:0/475713042 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc039660 con 0x561b5219d850 2024-09-15T23:26:44.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.399+0000 7f27ed8c5740 1 -- 172.21.15.62:0/475713042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 msgr2=0x561b5219dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:44.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.400+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/475713042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b5219dc30 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f27dc02af70 tx=0x7f27dc02fc80 comp rx=0 tx=0).stop 2024-09-15T23:26:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.400+0000 7f27ed8c5740 1 -- 172.21.15.62:0/475713042 shutdown_connections 2024-09-15T23:26:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.400+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/475713042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.400+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/475713042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219e170 0x561b521a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.400+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/475713042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b5219dc30 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.400+0000 7f27ed8c5740 1 -- 172.21.15.62:0/475713042 >> 172.21.15.62:0/475713042 conn(0x561b52199de0 msgr2=0x561b5219a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:44.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.402+0000 7f27da7fc640 1 -- 172.21.15.62:0/1390264629 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f27e0018020 con 0x7f27e41091f0 2024-09-15T23:26:44.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.403+0000 7f27da7fc640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d00a3390 0x7f27d00a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.403+0000 7f27ed8c5740 1 -- 172.21.15.62:0/475713042 shutdown_connections 2024-09-15T23:26:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.403+0000 7f27ed8c5740 1 -- 172.21.15.62:0/475713042 wait complete. 2024-09-15T23:26:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.403+0000 7f27da7fc640 1 -- 172.21.15.62:0/1390264629 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f27e00cf320 con 0x7f27e41091f0 2024-09-15T23:26:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.404+0000 7f27e8a87640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d00a3390 0x7f27d00a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.404+0000 7f27ed8c5740 1 -- start start 2024-09-15T23:26:44.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.404+0000 7f27ed8c5740 1 Processor -- start 2024-09-15T23:26:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a8a30 0x561b521a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a6510 0x561b521b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561b521ab6a0 con 0x561b521a8a30 2024-09-15T23:26:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561b5208a900 con 0x561b521a6510 2024-09-15T23:26:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27e8a87640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a8a30 0x561b521a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27e8a87640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a8a30 0x561b521a5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59760/0 (socket says 172.21.15.62:59760) 2024-09-15T23:26:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27e8a87640 1 -- 172.21.15.62:0/83128108 learned_addr learned my addr 172.21.15.62:0/83128108 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27ed8c5740 1 -- 172.21.15.62:0/83128108 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561b521ab820 con 0x561b5219d850 2024-09-15T23:26:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.405+0000 7f27ed8c5740 10 client.0 _ll_register_callbacks cb 0x561b52088780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.406+0000 7f27e9288640 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.407+0000 7f27e9a89640 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a6510 0x561b521b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.408+0000 7f27e9a89640 1 -- 172.21.15.62:0/83128108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 msgr2=0x561b521a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.408+0000 7f27e9a89640 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.408+0000 7f27e9a89640 1 -- 172.21.15.62:0/83128108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a8a30 msgr2=0x561b521a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.408+0000 7f27e9a89640 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a8a30 0x561b521a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.408+0000 7f27e9a89640 1 -- 172.21.15.62:0/83128108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b520cac50 con 0x561b521a6510 2024-09-15T23:26:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.408+0000 7f27e9288640 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.409+0000 7f27e8a87640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d00a3390 0x7f27d00a5850 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f27d4017760 tx=0x7f27d40079e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.410+0000 7f27e9a89640 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a6510 0x561b521b4b10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009bf0 tx=0x7f27dc02f600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6658bf74ffa1e80c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.410+0000 7f27ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:26:44.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.411+0000 7f27ccff9640 1 -- 172.21.15.62:0/83128108 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc051070 con 0x561b521a6510 2024-09-15T23:26:44.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.411+0000 7f27ccff9640 1 -- 172.21.15.62:0/83128108 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f27dc030680 con 0x561b521a6510 2024-09-15T23:26:44.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.411+0000 7f27ed8c5740 1 -- 172.21.15.62:0/83128108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561b5210f890 con 0x561b521a6510 2024-09-15T23:26:44.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.411+0000 7f27ccff9640 1 -- 172.21.15.62:0/83128108 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f27dc05d050 con 0x561b521a6510 2024-09-15T23:26:44.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.411+0000 7f27ed8c5740 10 client.27446 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:44.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.411+0000 7f27ed8c5740 1 -- 172.21.15.62:0/83128108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561b521d98b0 con 0x561b521a6510 2024-09-15T23:26:44.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.412+0000 7f27ccff9640 1 -- 172.21.15.62:0/83128108 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561b5210f890 con 0x561b521a6510 2024-09-15T23:26:44.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.412+0000 7f27ccff9640 1 -- 172.21.15.62:0/83128108 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f27dc043070 con 0x561b521a6510 2024-09-15T23:26:44.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.412+0000 7f27ed8c5740 10 client.27446 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:44.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.412+0000 7f27ed8c5740 10 client.27446 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:26:44.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.412+0000 7f27ed8c5740 20 client.27446 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:44.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.412+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561b521dac00 0x561b521dcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.412+0000 7f27ed8c5740 4 client.27446 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:44.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.412+0000 7f27ed8c5740 1 -- 172.21.15.62:0/83128108 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b52109be0 con 0x561b521dac00 2024-09-15T23:26:44.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.413+0000 7f27e9288640 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561b521dac00 0x561b521dcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.416+0000 7f27e9288640 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561b521dac00 0x561b521dcff0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=690c73c7bcda422a server_cookie=979c69895d3a454e in_seq=0 out_seq=0 2024-09-15T23:26:44.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.416+0000 7f27ccff9640 10 client.27446 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:44.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ccff9640 1 -- 172.21.15.62:0/83128108 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561b52109be0 con 0x561b521dac00 2024-09-15T23:26:44.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ccff9640 10 client.27446 handle_command_reply: tid=0 2024-09-15T23:26:44.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ed8c5740 1 client.27446 shutdown 2024-09-15T23:26:44.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ed8c5740 1 -- 172.21.15.62:0/83128108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a6510 msgr2=0x561b521b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:44.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a6510 0x561b521b4b10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009bf0 tx=0x7f27dc02f600 comp rx=0 tx=0).stop 2024-09-15T23:26:44.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ed8c5740 1 -- 172.21.15.62:0/83128108 shutdown_connections 2024-09-15T23:26:44.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561b521dac00 0x561b521dcff0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a6510 0x561b521b4b10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a8a30 0x561b521a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/83128108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ed8c5740 1 -- 172.21.15.62:0/83128108 >> 172.21.15.62:0/83128108 conn(0x561b52199de0 msgr2=0x561b5219fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.418+0000 7f27ccff9640 0 client.27446 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.419+0000 7f27ed8c5740 1 -- 172.21.15.62:0/83128108 shutdown_connections 2024-09-15T23:26:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.419+0000 7f27ed8c5740 1 -- 172.21.15.62:0/83128108 wait complete. 2024-09-15T23:26:44.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.419+0000 7f27ed8c5740 20 client.27446 trim_cache size 0 max 16384 2024-09-15T23:26:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27ed8c5740 1 Processor -- start 2024-09-15T23:26:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27ed8c5740 1 -- start start 2024-09-15T23:26:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a6510 0x561b521a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f27e40807b0 con 0x561b521a6510 2024-09-15T23:26:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27e407e0c0 con 0x561b521a8a30 2024-09-15T23:26:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f27e41095d0 con 0x561b5219d850 2024-09-15T23:26:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27e8a87640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a6510 0x561b521a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27e9288640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27e9288640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:51354/0 (socket says 172.21.15.62:51354) 2024-09-15T23:26:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27e8a87640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a6510 0x561b521a4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59768/0 (socket says 172.21.15.62:59768) 2024-09-15T23:26:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.446+0000 7f27e9288640 1 -- 172.21.15.62:0/682025519 learned_addr learned my addr 172.21.15.62:0/682025519 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27e9a89640 1 --2- 172.21.15.62:0/682025519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27e9288640 1 -- 172.21.15.62:0/682025519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 msgr2=0x561b521b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27e9288640 1 --2- 172.21.15.62:0/682025519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27e9288640 1 -- 172.21.15.62:0/682025519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a6510 msgr2=0x561b521a4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27e9288640 1 --2- 172.21.15.62:0/682025519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a6510 0x561b521a4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27e9288640 1 -- 172.21.15.62:0/682025519 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b521953c0 con 0x561b5219d850 2024-09-15T23:26:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27e9288640 1 --2- 172.21.15.62:0/682025519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a0fe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f27e000bdb0 tx=0x7f27e00047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1210d34654fa7a10 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27ceffd640 1 -- 172.21.15.62:0/682025519 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e00a4230 con 0x561b5219d850 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27ceffd640 1 -- 172.21.15.62:0/682025519 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e0010380 con 0x561b5219d850 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.447+0000 7f27ceffd640 1 -- 172.21.15.62:0/682025519 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0097c30 con 0x561b5219d850 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.448+0000 7f27ed8c5740 1 -- 172.21.15.62:0/682025519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 msgr2=0x561b521a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.448+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/682025519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a0fe0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f27e000bdb0 tx=0x7f27e00047d0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.448+0000 7f27ed8c5740 1 -- 172.21.15.62:0/682025519 shutdown_connections 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.448+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/682025519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.448+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/682025519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b521a6510 0x561b521a4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.448+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/682025519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b5219d850 0x561b521a0fe0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.448+0000 7f27ed8c5740 1 -- 172.21.15.62:0/682025519 >> 172.21.15.62:0/682025519 conn(0x561b52199de0 msgr2=0x561b5219fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.448+0000 7f27ed8c5740 1 -- 172.21.15.62:0/682025519 shutdown_connections 2024-09-15T23:26:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.448+0000 7f27ed8c5740 1 -- 172.21.15.62:0/682025519 wait complete. 2024-09-15T23:26:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.449+0000 7f27ed8c5740 1 -- start start 2024-09-15T23:26:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.449+0000 7f27ed8c5740 1 Processor -- start 2024-09-15T23:26:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.449+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b521e0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.449+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b521a6510 0x561b521dd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.449+0000 7f27ed8c5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521ddc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.449+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f27e407c580 con 0x561b5219d850 2024-09-15T23:26:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27e407d9f0 con 0x561b521a8a30 2024-09-15T23:26:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27ed8c5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f27e407cc50 con 0x561b521a6510 2024-09-15T23:26:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27e9288640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b521e0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27e9288640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b521e0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59772/0 (socket says 172.21.15.62:59772) 2024-09-15T23:26:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27e9288640 1 -- 172.21.15.62:0/3115473181 learned_addr learned my addr 172.21.15.62:0/3115473181 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27ed8c5740 10 client.0 _ll_register_callbacks cb 0x7f27d40071e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27e9288640 1 -- 172.21.15.62:0/3115473181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b521a6510 msgr2=0x561b521dd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27e8a87640 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b521a6510 0x561b521dd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27e9288640 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b521a6510 0x561b521dd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27e9288640 1 -- 172.21.15.62:0/3115473181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 msgr2=0x561b521ddc10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27e9288640 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521ddc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.450+0000 7f27e9288640 1 -- 172.21.15.62:0/3115473181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b521de440 con 0x561b5219d850 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.451+0000 7f27e9288640 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b521e0380 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f27e009b2c0 tx=0x7f27e009fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cae2b590a87c3db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.451+0000 7f27ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.451+0000 7f27ccff9640 1 -- 172.21.15.62:0/3115473181 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e00a8070 con 0x561b5219d850 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.451+0000 7f27ccff9640 1 -- 172.21.15.62:0/3115473181 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f27e00af800 con 0x561b5219d850 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ed8c5740 1 -- 172.21.15.62:0/3115473181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27e0010380 con 0x561b5219d850 2024-09-15T23:26:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ccff9640 1 -- 172.21.15.62:0/3115473181 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f27e00cf160 con 0x561b5219d850 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ed8c5740 10 client.17880 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ed8c5740 1 -- 172.21.15.62:0/3115473181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561b52245510 con 0x561b5219d850 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ccff9640 1 -- 172.21.15.62:0/3115473181 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27e00a4230 con 0x561b5219d850 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ccff9640 1 -- 172.21.15.62:0/3115473181 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f27e0022060 con 0x561b5219d850 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ed8c5740 10 client.17880 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ed8c5740 10 client.17880 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ed8c5740 20 client.17880 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561b52245bf0 0x561b52247fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ed8c5740 4 client.17880 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}] 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.452+0000 7f27ed8c5740 1 -- 172.21.15.62:0/3115473181 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}) -- 0x7f27e0010380 con 0x561b52245bf0 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.453+0000 7f27e8a87640 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561b52245bf0 0x561b52247fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.453+0000 7f27e8a87640 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561b52245bf0 0x561b52247fe0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cc6345a7e1739e4 server_cookie=20da1208964434f4 in_seq=0 out_seq=0 2024-09-15T23:26:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.453+0000 7f27ccff9640 10 client.17880 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:44.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 625.0 s, 493.0 tps, lat 204.604 ms stddev 285.560 2024-09-15T23:26:44.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.709+0000 7f27ccff9640 1 -- 172.21.15.62:0/3115473181 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f27e0010380 con 0x561b52245bf0 2024-09-15T23:26:44.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.709+0000 7f27ccff9640 10 client.17880 handle_command_reply: tid=0 2024-09-15T23:26:44.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.709+0000 7f27ed8c5740 1 client.17880 shutdown 2024-09-15T23:26:44.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 -- 172.21.15.62:0/3115473181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 msgr2=0x561b521e0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:44.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b521e0380 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f27e009b2c0 tx=0x7f27e009fab0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 -- 172.21.15.62:0/3115473181 shutdown_connections 2024-09-15T23:26:44.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561b52245bf0 0x561b52247fe0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561b521a8a30 0x561b521ddc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561b521a6510 0x561b521dd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 --2- 172.21.15.62:0/3115473181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b5219d850 0x561b521e0380 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ccff9640 0 client.17880 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:44.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 -- 172.21.15.62:0/3115473181 >> 172.21.15.62:0/3115473181 conn(0x561b52199de0 msgr2=0x561b521986a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:44.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 -- 172.21.15.62:0/3115473181 shutdown_connections 2024-09-15T23:26:44.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 1 -- 172.21.15.62:0/3115473181 wait complete. 2024-09-15T23:26:44.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.710+0000 7f27ed8c5740 20 client.17880 trim_cache size 0 max 16384 2024-09-15T23:26:44.713 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 35, 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 38, 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "724f1b2d": { 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 38, 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:26:44.714 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 141.8, 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:26:44.715 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:26:44.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 -- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d00a3390 msgr2=0x7f27d00a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:44.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d00a3390 0x7f27d00a5850 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f27d4017760 tx=0x7f27d40079e0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 -- 172.21.15.62:0/1390264629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e41091f0 msgr2=0x7f27e410a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:44.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e41091f0 0x7f27e410a3d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f27e000b3f0 tx=0x7f27e000b8b0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 -- 172.21.15.62:0/1390264629 shutdown_connections 2024-09-15T23:26:44.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27d00a3390 0x7f27d00a5850 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27e4114ab0 0x7f27e410ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27e4109b10 0x7f27e410a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 --2- 172.21.15.62:0/1390264629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e41091f0 0x7f27e410a3d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:44.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 -- 172.21.15.62:0/1390264629 >> 172.21.15.62:0/1390264629 conn(0x7f27e406e140 msgr2=0x7f27e40708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:44.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 -- 172.21.15.62:0/1390264629 shutdown_connections 2024-09-15T23:26:44.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:44.711+0000 7f27eb4ec640 1 -- 172.21.15.62:0/1390264629 wait complete. 2024-09-15T23:26:45.325 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:45 smithi062 ceph-mon[30949]: pgmap v684: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.3 MiB/s wr, 573 op/s 2024-09-15T23:26:45.355 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '724f1b2d', quiesce took 7.7 seconds. Will release after: 14.150000000000002 2024-09-15T23:26:45.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:45 smithi067 ceph-mon[35665]: pgmap v684: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.3 MiB/s wr, 573 op/s 2024-09-15T23:26:45.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:45 smithi148 ceph-mon[35610]: pgmap v684: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.3 MiB/s wr, 573 op/s 2024-09-15T23:26:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:47 smithi062 ceph-mon[30949]: pgmap v685: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.7 MiB/s wr, 649 op/s 2024-09-15T23:26:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:47 smithi067 ceph-mon[35665]: pgmap v685: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.7 MiB/s wr, 649 op/s 2024-09-15T23:26:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:47 smithi148 ceph-mon[35610]: pgmap v685: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.7 MiB/s wr, 649 op/s 2024-09-15T23:26:49.326 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:49 smithi067 ceph-mon[35665]: pgmap v686: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.5 MiB/s wr, 583 op/s 2024-09-15T23:26:49.327 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:49 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:26:49.327 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:49 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:26:49.327 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:49 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:26:49.327 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:49 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:26:49.327 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:49 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:26:49.327 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:49 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:26:49.342 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:49 smithi148 ceph-mon[35610]: pgmap v686: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.5 MiB/s wr, 583 op/s 2024-09-15T23:26:49.342 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:49 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:26:49.342 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:49 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:26:49.342 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:49 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:26:49.342 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:49 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:26:49.342 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:49 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:26:49.342 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:49 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:26:49.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:49 smithi062 ceph-mon[30949]: pgmap v686: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.5 MiB/s wr, 583 op/s 2024-09-15T23:26:49.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:26:49.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:26:49.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:26:49.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:26:49.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:26:49.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:26:49.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 630.0 s, 26.8 tps, lat 124.625 ms stddev 172.724 2024-09-15T23:26:51.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:51 smithi062 ceph-mon[30949]: pgmap v687: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.7 MiB/s wr, 482 op/s 2024-09-15T23:26:51.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:51 smithi067 ceph-mon[35665]: pgmap v687: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.7 MiB/s wr, 482 op/s 2024-09-15T23:26:51.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:51 smithi148 ceph-mon[35610]: pgmap v687: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.7 MiB/s wr, 482 op/s 2024-09-15T23:26:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:53 smithi062 ceph-mon[30949]: pgmap v688: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.2 MiB/s wr, 390 op/s 2024-09-15T23:26:53.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:53 smithi067 ceph-mon[35665]: pgmap v688: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.2 MiB/s wr, 390 op/s 2024-09-15T23:26:53.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:53 smithi148 ceph-mon[35610]: pgmap v688: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 5.2 MiB/s wr, 390 op/s 2024-09-15T23:26:54.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 635.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:26:55.399 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:26:55.428 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:55 smithi062 ceph-mon[30949]: pgmap v689: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-09-15T23:26:55.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:55 smithi067 ceph-mon[35665]: pgmap v689: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-09-15T23:26:55.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:55 smithi148 ceph-mon[35610]: pgmap v689: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 3.9 MiB/s wr, 270 op/s 2024-09-15T23:26:55.672 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:56.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.098+0000 7f300f96d640 1 -- 172.21.15.62:0/3486724170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30081025f0 msgr2=0x7f3008108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.098+0000 7f300f96d640 1 --2- 172.21.15.62:0/3486724170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30081025f0 0x7f3008108e80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f3000009ca0 tx=0x7f300002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.098+0000 7f300f96d640 1 -- 172.21.15.62:0/3486724170 shutdown_connections 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.098+0000 7f300f96d640 1 --2- 172.21.15.62:0/3486724170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30081025f0 0x7f3008108e80 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.098+0000 7f300f96d640 1 --2- 172.21.15.62:0/3486724170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3008101c30 0x7f30081020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.098+0000 7f300f96d640 1 --2- 172.21.15.62:0/3486724170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3008100a40 0x7f3008100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.098+0000 7f300f96d640 1 -- 172.21.15.62:0/3486724170 >> 172.21.15.62:0/3486724170 conn(0x7f30080fc1e0 msgr2=0x7f30080fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.098+0000 7f300f96d640 1 -- 172.21.15.62:0/3486724170 shutdown_connections 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.098+0000 7f300f96d640 1 -- 172.21.15.62:0/3486724170 wait complete. 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300f96d640 1 Processor -- start 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300f96d640 1 -- start start 2024-09-15T23:26:56.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300f96d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3008100a40 0x7f3008197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300f96d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3008101c30 0x7f30081982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300f96d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30081025f0 0x7f300819f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300f96d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f300810c6c0 con 0x7f3008101c30 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300f96d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f300810c540 con 0x7f30081025f0 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300f96d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f300810c840 con 0x7f3008100a40 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300cf08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3008101c30 0x7f30081982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300cf08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3008101c30 0x7f30081982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47648/0 (socket says 172.21.15.62:47648) 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300cf08640 1 -- 172.21.15.62:0/3460119811 learned_addr learned my addr 172.21.15.62:0/3460119811 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300d709640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3008100a40 0x7f3008197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300df0a640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30081025f0 0x7f300819f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300cf08640 1 -- 172.21.15.62:0/3460119811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3008100a40 msgr2=0x7f3008197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300cf08640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3008100a40 0x7f3008197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.103 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300cf08640 1 -- 172.21.15.62:0/3460119811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30081025f0 msgr2=0x7f300819f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300cf08640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30081025f0 0x7f300819f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300cf08640 1 -- 172.21.15.62:0/3460119811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300819f9d0 con 0x7f3008101c30 2024-09-15T23:26:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.099+0000 7f300df0a640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30081025f0 0x7f300819f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:26:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.100+0000 7f300cf08640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3008101c30 0x7f30081982f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800e9f0 tx=0x7f2ff800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.100+0000 7f2ffe7fc640 1 -- 172.21.15.62:0/3460119811 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff800cd80 con 0x7f3008101c30 2024-09-15T23:26:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.100+0000 7f2ffe7fc640 1 -- 172.21.15.62:0/3460119811 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ff8005500 con 0x7f3008101c30 2024-09-15T23:26:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.100+0000 7f300f96d640 1 -- 172.21.15.62:0/3460119811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f300819fcc0 con 0x7f3008101c30 2024-09-15T23:26:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.100+0000 7f2ffe7fc640 1 -- 172.21.15.62:0/3460119811 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8010690 con 0x7f3008101c30 2024-09-15T23:26:56.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.100+0000 7f300f96d640 1 -- 172.21.15.62:0/3460119811 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30081a01c0 con 0x7f3008101c30 2024-09-15T23:26:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.102+0000 7f30127ea740 1 Processor -- start 2024-09-15T23:26:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f2ffe7fc640 1 -- 172.21.15.62:0/3460119811 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2ff80046c0 con 0x7f3008101c30 2024-09-15T23:26:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f30127ea740 1 -- start start 2024-09-15T23:26:56.109 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f30127ea740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 0x55bba6f3ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f30127ea740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3b2e0 0x55bba6f3fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f30127ea740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f40370 0x55bba6f42830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f30127ea740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bba6e9e2f0 con 0x55bba6f3b2e0 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f30127ea740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55bba6e46270 con 0x55bba6f3a9a0 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f30127ea740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bba6e9a2b0 con 0x55bba6f40370 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300d709640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3b2e0 0x55bba6f3fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300d709640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3b2e0 0x55bba6f3fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47654/0 (socket says 172.21.15.62:47654) 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300d709640 1 -- 172.21.15.62:0/2282940058 learned_addr learned my addr 172.21.15.62:0/2282940058 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300df0a640 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 0x55bba6f3ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300cf08640 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f40370 0x55bba6f42830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300d709640 1 -- 172.21.15.62:0/2282940058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f40370 msgr2=0x55bba6f42830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300d709640 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f40370 0x55bba6f42830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300d709640 1 -- 172.21.15.62:0/2282940058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 msgr2=0x55bba6f3ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300d709640 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 0x55bba6f3ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300d709640 1 -- 172.21.15.62:0/2282940058 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bba6e45340 con 0x55bba6f3b2e0 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f300cf08640 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f40370 0x55bba6f42830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.103+0000 7f2ffe7fc640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ff40a3370 0x7f2ff40a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.104+0000 7f300df0a640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ff40a3370 0x7f2ff40a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.104+0000 7f300d709640 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3b2e0 0x55bba6f3fda0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f3004009ca0 tx=0x7f3004011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22b64ff7d39f6af2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.104+0000 7f2ffe7fc640 1 -- 172.21.15.62:0/3460119811 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2ff80d00c0 con 0x7f3008101c30 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.104+0000 7f2ff37fe640 1 -- 172.21.15.62:0/2282940058 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004004220 con 0x55bba6f3b2e0 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.104+0000 7f2ff37fe640 1 -- 172.21.15.62:0/2282940058 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30040043c0 con 0x55bba6f3b2e0 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.104+0000 7f2ff37fe640 1 -- 172.21.15.62:0/2282940058 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004019890 con 0x55bba6f3b2e0 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 -- 172.21.15.62:0/2282940058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3b2e0 msgr2=0x55bba6f3fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3b2e0 0x55bba6f3fda0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f3004009ca0 tx=0x7f3004011810 comp rx=0 tx=0).stop 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f300df0a640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ff40a3370 0x7f2ff40a5830 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3000006010 tx=0x7f3000005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 -- 172.21.15.62:0/2282940058 shutdown_connections 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f40370 0x55bba6f42830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3b2e0 0x55bba6f3fda0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 --2- 172.21.15.62:0/2282940058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 0x55bba6f3ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 -- 172.21.15.62:0/2282940058 >> 172.21.15.62:0/2282940058 conn(0x55bba6f36010 msgr2=0x55bba6f36410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 -- 172.21.15.62:0/2282940058 shutdown_connections 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 -- 172.21.15.62:0/2282940058 wait complete. 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 -- start start 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.105+0000 7f30127ea740 1 Processor -- start 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f30127ea740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f48fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f30127ea740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f49520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f30127ea740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 0x55bba6f50560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f30127ea740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bba6f43340 con 0x55bba6f3a9a0 2024-09-15T23:26:56.112 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f30127ea740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55bba6e27290 con 0x55bba6f40370 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f30127ea740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bba6f434c0 con 0x55bba6f3b2e0 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f30127ea740 10 client.0 _ll_register_callbacks cb 0x55bba6e58140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300d709640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f48fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300d709640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f48fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47658/0 (socket says 172.21.15.62:47658) 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300d709640 1 -- 172.21.15.62:0/1643639538 learned_addr learned my addr 172.21.15.62:0/1643639538 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300df0a640 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 0x55bba6f50560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300cf08640 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f49520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300d709640 1 -- 172.21.15.62:0/1643639538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 msgr2=0x55bba6f49520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300d709640 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f49520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300d709640 1 -- 172.21.15.62:0/1643639538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 msgr2=0x55bba6f50560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300d709640 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 0x55bba6f50560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300d709640 1 -- 172.21.15.62:0/1643639538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bba6bb1210 con 0x55bba6f3a9a0 2024-09-15T23:26:56.113 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f300d709640 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f48fe0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3004009f50 tx=0x7f3004003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f4d47f4d846a8b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.106+0000 7f2ff0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/1643639538 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004028070 con 0x55bba6f3a9a0 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/1643639538 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3004003800 con 0x55bba6f3a9a0 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f30127ea740 1 -- 172.21.15.62:0/1643639538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30040043c0 con 0x55bba6f3a9a0 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/1643639538 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3004034050 con 0x55bba6f3a9a0 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f30127ea740 10 client.17892 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f30127ea740 1 -- 172.21.15.62:0/1643639538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bba6c7d620 con 0x55bba6f3a9a0 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/1643639538 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30040043c0 con 0x55bba6f3a9a0 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/1643639538 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f3004004a80 con 0x55bba6f3a9a0 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f30127ea740 10 client.17892 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f30127ea740 10 client.17892 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f30127ea740 20 client.17892 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f30127ea740 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bba6f54a70 0x55bba6f56ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.114 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f30127ea740 4 client.17892 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.107+0000 7f30127ea740 1 -- 172.21.15.62:0/1643639538 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bba6eaba20 con 0x55bba6f54a70 2024-09-15T23:26:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.108+0000 7f300cf08640 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bba6f54a70 0x55bba6f56ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.108+0000 7f2ff0ff9640 10 client.17892 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:56.115 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.108+0000 7f300cf08640 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bba6f54a70 0x55bba6f56ea0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14945288591c0bb8 server_cookie=2a94ddb5fc7ce796 in_seq=0 out_seq=0 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/1643639538 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bba6eaba20 con 0x55bba6f54a70 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f2ff0ff9640 10 client.17892 handle_command_reply: tid=0 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 client.17892 shutdown 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 -- 172.21.15.62:0/1643639538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 msgr2=0x55bba6f48fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f48fe0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3004009f50 tx=0x7f3004003dd0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 -- 172.21.15.62:0/1643639538 shutdown_connections 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bba6f54a70 0x55bba6f56ea0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 0x55bba6f50560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f49520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 --2- 172.21.15.62:0/1643639538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f48fe0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.116 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 -- 172.21.15.62:0/1643639538 >> 172.21.15.62:0/1643639538 conn(0x55bba6f36010 msgr2=0x55bba6f37820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.117 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f2ff0ff9640 0 client.17892 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:56.117 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 -- 172.21.15.62:0/1643639538 shutdown_connections 2024-09-15T23:26:56.117 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.109+0000 7f30127ea740 1 -- 172.21.15.62:0/1643639538 wait complete. 2024-09-15T23:26:56.117 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.110+0000 7f30127ea740 20 client.17892 trim_cache size 0 max 16384 2024-09-15T23:26:56.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.109+0000 7fe53bbf4640 1 -- 172.21.15.62:0/4176029130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534106a80 msgr2=0x7fe534106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.109+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/4176029130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534106a80 0x7fe534106e60 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fe530009930 tx=0x7fe53002a420 comp rx=0 tx=0).stop 2024-09-15T23:26:56.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.109+0000 7fe53bbf4640 1 -- 172.21.15.62:0/4176029130 shutdown_connections 2024-09-15T23:26:56.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.109+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/4176029130 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe534101420 0x7fe53410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.109+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/4176029130 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe534100a80 0x7fe534100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.109+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/4176029130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534106a80 0x7fe534106e60 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.109+0000 7fe53bbf4640 1 -- 172.21.15.62:0/4176029130 >> 172.21.15.62:0/4176029130 conn(0x7fe5340fc780 msgr2=0x7fe5340feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.109+0000 7fe53bbf4640 1 -- 172.21.15.62:0/4176029130 shutdown_connections 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.109+0000 7fe53bbf4640 1 -- 172.21.15.62:0/4176029130 wait complete. 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.110+0000 7fe53bbf4640 1 Processor -- start 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.110+0000 7fe53bbf4640 1 -- start start 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.110+0000 7fe53bbf4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe534100a80 0x7fe5341a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.110+0000 7fe53bbf4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534101420 0x7fe5341a4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.110+0000 7fe53bbf4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe534106a80 0x7fe53419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.110+0000 7fe53bbf4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe534110e90 con 0x7fe534101420 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.110+0000 7fe53bbf4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe534110d10 con 0x7fe534100a80 2024-09-15T23:26:56.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53bbf4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe534111010 con 0x7fe534106a80 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53918f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534101420 0x7fe5341a4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53918f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534101420 0x7fe5341a4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47670/0 (socket says 172.21.15.62:47670) 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53918f640 1 -- 172.21.15.62:0/2008787269 learned_addr learned my addr 172.21.15.62:0/2008787269 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe539990640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe534100a80 0x7fe5341a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53a191640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe534106a80 0x7fe53419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53918f640 1 -- 172.21.15.62:0/2008787269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe534106a80 msgr2=0x7fe53419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53918f640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe534106a80 0x7fe53419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53918f640 1 -- 172.21.15.62:0/2008787269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe534100a80 msgr2=0x7fe5341a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53918f640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe534100a80 0x7fe5341a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53918f640 1 -- 172.21.15.62:0/2008787269 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe53419f2d0 con 0x7fe534101420 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.111+0000 7fe53918f640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534101420 0x7fe5341a4b30 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe52400e9f0 tx=0x7fe52400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.112+0000 7fe52affd640 1 -- 172.21.15.62:0/2008787269 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52400cd80 con 0x7fe534101420 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.112+0000 7fe53bbf4640 1 -- 172.21.15.62:0/2008787269 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe53419f5c0 con 0x7fe534101420 2024-09-15T23:26:56.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.112+0000 7fe53bbf4640 1 -- 172.21.15.62:0/2008787269 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5341ab3a0 con 0x7fe534101420 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.112+0000 7fe52affd640 1 -- 172.21.15.62:0/2008787269 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe524010910 con 0x7fe534101420 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.112+0000 7fe52affd640 1 -- 172.21.15.62:0/2008787269 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5240188f0 con 0x7fe534101420 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.115+0000 7fe53dfcd740 1 Processor -- start 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.115+0000 7fe52affd640 1 -- 172.21.15.62:0/2008787269 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe524018bd0 con 0x7fe534101420 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.115+0000 7fe53dfcd740 1 -- start start 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.115+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbab91880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.115+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 0x558dbab9c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.115+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbab9ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.115+0000 7fe53dfcd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558dbaaf72e0 con 0x558dbab9c680 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe53dfcd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558dbaa9ece0 con 0x558dbab914a0 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe53dfcd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558dbaaf32a0 con 0x558dbab91dc0 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe53918f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbab9ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe53a191640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbab91880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe53918f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbab9ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47684/0 (socket says 172.21.15.62:47684) 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe53918f640 1 -- 172.21.15.62:0/2400706510 learned_addr learned my addr 172.21.15.62:0/2400706510 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe539990640 1 --2- 172.21.15.62:0/2400706510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 0x558dbab9c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe539990640 1 -- 172.21.15.62:0/2400706510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 msgr2=0x558dbab91880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe539990640 1 --2- 172.21.15.62:0/2400706510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbab91880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe539990640 1 -- 172.21.15.62:0/2400706510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 msgr2=0x558dbab9ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe52affd640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe5200a32c0 0x7fe5200a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe539990640 1 --2- 172.21.15.62:0/2400706510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbab9ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe539990640 1 -- 172.21.15.62:0/2400706510 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dbaa83440 con 0x558dbab91dc0 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.116+0000 7fe53918f640 1 --2- 172.21.15.62:0/2400706510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbab9ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.117+0000 7fe53a191640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe5200a32c0 0x7fe5200a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.117+0000 7fe52affd640 1 -- 172.21.15.62:0/2008787269 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe5240cfef0 con 0x7fe534101420 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.117+0000 7fe539990640 1 --2- 172.21.15.62:0/2400706510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 0x558dbab9c140 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe530009850 tx=0x7fe530000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2652fe8a8c3f1e25 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.118+0000 7fe53a191640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe5200a32c0 0x7fe5200a5780 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe52c005fd0 tx=0x7fe52c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.118+0000 7fe51ffff640 1 -- 172.21.15.62:0/2400706510 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530040070 con 0x558dbab91dc0 2024-09-15T23:26:56.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.118+0000 7fe51ffff640 1 -- 172.21.15.62:0/2400706510 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe530004440 con 0x558dbab91dc0 2024-09-15T23:26:56.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.118+0000 7fe51ffff640 1 -- 172.21.15.62:0/2400706510 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530035040 con 0x558dbab91dc0 2024-09-15T23:26:56.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 -- 172.21.15.62:0/2400706510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 msgr2=0x558dbab9c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/2400706510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 0x558dbab9c140 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe530009850 tx=0x7fe530000bc0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 -- 172.21.15.62:0/2400706510 shutdown_connections 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/2400706510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbab9ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/2400706510 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 0x558dbab9c140 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/2400706510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbab91880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 -- 172.21.15.62:0/2400706510 >> 172.21.15.62:0/2400706510 conn(0x558dbab8da30 msgr2=0x558dbab8de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 -- 172.21.15.62:0/2400706510 shutdown_connections 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 -- 172.21.15.62:0/2400706510 wait complete. 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.119+0000 7fe53dfcd740 1 -- start start 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.120+0000 7fe53dfcd740 1 Processor -- start 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.120+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbab9bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.120+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbab99180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab9c680 0x558dbab996c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53918f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbab99180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53918f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbab99180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47696/0 (socket says 172.21.15.62:47696) 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe539990640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbab9bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53918f640 1 -- 172.21.15.62:0/4157790854 learned_addr learned my addr 172.21.15.62:0/4157790854 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558dbab9f2a0 con 0x558dbab91dc0 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558dbaab2050 con 0x558dbab914a0 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558dbab9f420 con 0x558dbab9c680 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53a191640 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab9c680 0x558dbab996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe539990640 1 -- 172.21.15.62:0/4157790854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab9c680 msgr2=0x558dbab996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe539990640 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab9c680 0x558dbab996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe539990640 1 -- 172.21.15.62:0/4157790854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 msgr2=0x558dbab99180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe539990640 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbab99180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe539990640 1 -- 172.21.15.62:0/4157790854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dba8683c0 con 0x558dbab914a0 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53a191640 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab9c680 0x558dbab996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53918f640 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbab99180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:56.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53dfcd740 10 client.0 _ll_register_callbacks cb 0x558dbaaf2160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe530004440 con 0x558dbab914a0 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe51d7fa640 10 client.27470 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.121+0000 7fe539990640 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbab9bed0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe530009900 tx=0x7fe53000b170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f42dd0ac072f516 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe51d7fa640 1 -- 172.21.15.62:0/4157790854 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530040070 con 0x558dbab914a0 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe51d7fa640 1 -- 172.21.15.62:0/4157790854 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe530004240 con 0x558dbab914a0 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe51d7fa640 1 -- 172.21.15.62:0/4157790854 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe530040070 con 0x558dbab914a0 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe53dfcd740 10 client.27470 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=35}) -- 0x558dbabaa7f0 con 0x558dbab914a0 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe51d7fa640 1 -- 172.21.15.62:0/4157790854 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fe530005050 con 0x558dbab914a0 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe53dfcd740 10 client.27470 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe53dfcd740 10 client.27470 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe53dfcd740 20 client.27470 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558dbababb40 0x558dbabadf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe53dfcd740 4 client.27470 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.122+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558dbab04850 con 0x558dbababb40 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.123+0000 7fe53918f640 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558dbababb40 0x558dbabadf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.123+0000 7fe53918f640 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558dbababb40 0x558dbabadf30 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4771ea43f1cfd38a server_cookie=9b24a4b898e4b329 in_seq=0 out_seq=0 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.123+0000 7fe51d7fa640 10 client.27470 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe51d7fa640 1 -- 172.21.15.62:0/4157790854 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558dbab04850 con 0x558dbababb40 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe51d7fa640 10 client.27470 handle_command_reply: tid=0 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 client.27470 shutdown 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 msgr2=0x558dbab9bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbab9bed0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe530009900 tx=0x7fe53000b170 comp rx=0 tx=0).stop 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 shutdown_connections 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558dbababb40 0x558dbabadf30 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe51d7fa640 0 client.27470 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab9c680 0x558dbab996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbab99180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/4157790854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbab9bed0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 >> 172.21.15.62:0/4157790854 conn(0x558dbab8da30 msgr2=0x558dbab93710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 shutdown_connections 2024-09-15T23:26:56.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 1 -- 172.21.15.62:0/4157790854 wait complete. 2024-09-15T23:26:56.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.124+0000 7fe53dfcd740 20 client.27470 trim_cache size 0 max 16384 2024-09-15T23:26:56.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f30127ea740 1 Processor -- start 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f30127ea740 1 -- start start 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f30127ea740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 0x55bba6f495b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f30127ea740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f50480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f30127ea740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f40370 0x55bba6f350b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f30127ea740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f30081aad40 con 0x55bba6f40370 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f30127ea740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f30081a8610 con 0x55bba6f3a9a0 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f30127ea740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f30081a6460 con 0x55bba6f3b2e0 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f300d709640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 0x55bba6f495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f300df0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f40370 0x55bba6f350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f300df0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f40370 0x55bba6f350b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47708/0 (socket says 172.21.15.62:47708) 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f300df0a640 1 -- 172.21.15.62:0/2420068768 learned_addr learned my addr 172.21.15.62:0/2420068768 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.128+0000 7f300cf08640 1 --2- 172.21.15.62:0/2420068768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f50480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f300d709640 1 -- 172.21.15.62:0/2420068768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 msgr2=0x55bba6f50480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f300d709640 1 --2- 172.21.15.62:0/2420068768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f50480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f300d709640 1 -- 172.21.15.62:0/2420068768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f40370 msgr2=0x55bba6f350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f300d709640 1 --2- 172.21.15.62:0/2420068768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f40370 0x55bba6f350b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f300d709640 1 -- 172.21.15.62:0/2420068768 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bba6f3b7c0 con 0x55bba6f3a9a0 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f300df0a640 1 --2- 172.21.15.62:0/2420068768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f40370 0x55bba6f350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f300d709640 1 --2- 172.21.15.62:0/2420068768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 0x55bba6f495b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3004009250 tx=0x7f3004001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb6566d00eeef589 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f2ff2ffd640 1 -- 172.21.15.62:0/2420068768 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004028070 con 0x55bba6f3a9a0 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f2ff2ffd640 1 -- 172.21.15.62:0/2420068768 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3004019430 con 0x55bba6f3a9a0 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.129+0000 7f2ff2ffd640 1 -- 172.21.15.62:0/2420068768 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30040054f0 con 0x55bba6f3a9a0 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 -- 172.21.15.62:0/2420068768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 msgr2=0x55bba6f495b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 --2- 172.21.15.62:0/2420068768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 0x55bba6f495b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3004009250 tx=0x7f3004001e50 comp rx=0 tx=0).stop 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 -- 172.21.15.62:0/2420068768 shutdown_connections 2024-09-15T23:26:56.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 --2- 172.21.15.62:0/2420068768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f40370 0x55bba6f350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 --2- 172.21.15.62:0/2420068768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f50480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 --2- 172.21.15.62:0/2420068768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f3a9a0 0x55bba6f495b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 -- 172.21.15.62:0/2420068768 >> 172.21.15.62:0/2420068768 conn(0x55bba6f36010 msgr2=0x55bba6f349a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 -- 172.21.15.62:0/2420068768 shutdown_connections 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 -- 172.21.15.62:0/2420068768 wait complete. 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 -- start start 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.130+0000 7f30127ea740 1 Processor -- start 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f30127ea740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f4db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f30127ea740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f4e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f30127ea740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 0x55bba6f4f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f30127ea740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f30081a6ad0 con 0x55bba6f3a9a0 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f30127ea740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f30081a7f40 con 0x55bba6f40370 2024-09-15T23:26:56.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f30127ea740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f30081a71a0 con 0x55bba6f3b2e0 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300d709640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f4db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f30127ea740 10 client.0 _ll_register_callbacks cb 0x7f3004005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300d709640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f4db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47718/0 (socket says 172.21.15.62:47718) 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300d709640 1 -- 172.21.15.62:0/3581071736 learned_addr learned my addr 172.21.15.62:0/3581071736 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300df0a640 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 0x55bba6f4f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300cf08640 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f4e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300d709640 1 -- 172.21.15.62:0/3581071736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 msgr2=0x55bba6f4e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300d709640 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f4e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300d709640 1 -- 172.21.15.62:0/3581071736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 msgr2=0x55bba6f4f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300d709640 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 0x55bba6f4f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300d709640 1 -- 172.21.15.62:0/3581071736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bba6f4f6e0 con 0x55bba6f3a9a0 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f300d709640 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f4db80 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3004000c00 tx=0x7f30040096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9152cb3f628df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.140 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.131+0000 7f2ff0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/3581071736 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004028040 con 0x55bba6f3a9a0 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f30127ea740 1 -- 172.21.15.62:0/3581071736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3004019430 con 0x55bba6f3a9a0 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/3581071736 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3004003800 con 0x55bba6f3a9a0 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/3581071736 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3004003a00 con 0x55bba6f3a9a0 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f30127ea740 10 client.17910 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f30127ea740 1 -- 172.21.15.62:0/3581071736 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bba6fbf780 con 0x55bba6f3a9a0 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/3581071736 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3004019430 con 0x55bba6f3a9a0 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/3581071736 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f3004004af0 con 0x55bba6f3a9a0 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f30127ea740 10 client.17910 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f30127ea740 10 client.17910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f30127ea740 20 client.17910 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f30127ea740 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bba6fc0060 0x55bba6fc2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f30127ea740 4 client.17910 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:26:56.141 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.132+0000 7f30127ea740 1 -- 172.21.15.62:0/3581071736 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bba6ea5d70 con 0x55bba6fc0060 2024-09-15T23:26:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.133+0000 7f300cf08640 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bba6fc0060 0x55bba6fc2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.133+0000 7f300cf08640 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bba6fc0060 0x55bba6fc2480 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a9c7df625b6fa5 server_cookie=c8b7e5341f753b68 in_seq=0 out_seq=0 2024-09-15T23:26:56.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.133+0000 7f2ff0ff9640 10 client.17910 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53dfcd740 1 Processor -- start 2024-09-15T23:26:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53dfcd740 1 -- start start 2024-09-15T23:26:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab914a0 0x558dbabb3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbabb3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53918f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbabb3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53918f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbabb3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47724/0 (socket says 172.21.15.62:47724) 2024-09-15T23:26:56.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53918f640 1 -- 172.21.15.62:0/1694296931 learned_addr learned my addr 172.21.15.62:0/1694296931 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab9c680 0x558dbab963e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe539990640 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab914a0 0x558dbabb3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53dfcd740 1 -- 172.21.15.62:0/1694296931 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe518012600 con 0x558dbab91dc0 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53dfcd740 1 -- 172.21.15.62:0/1694296931 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe51800fed0 con 0x558dbab9c680 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.143+0000 7fe53dfcd740 1 -- 172.21.15.62:0/1694296931 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe51800dcc0 con 0x558dbab914a0 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe53a191640 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab9c680 0x558dbab963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe53918f640 1 -- 172.21.15.62:0/1694296931 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab914a0 msgr2=0x558dbabb3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe53918f640 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab914a0 0x558dbabb3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe53918f640 1 -- 172.21.15.62:0/1694296931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab9c680 msgr2=0x558dbab963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe53918f640 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab9c680 0x558dbab963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe53918f640 1 -- 172.21.15.62:0/1694296931 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558dbab8d860 con 0x558dbab91dc0 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe53918f640 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbabb3bb0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fe5240cd9e0 tx=0x7fe5240bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c76163c6785ae4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe51f7fe640 1 -- 172.21.15.62:0/1694296931 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52409c010 con 0x558dbab91dc0 2024-09-15T23:26:56.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe51f7fe640 1 -- 172.21.15.62:0/1694296931 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe52409c240 con 0x558dbab91dc0 2024-09-15T23:26:56.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.144+0000 7fe51f7fe640 1 -- 172.21.15.62:0/1694296931 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52409bba0 con 0x558dbab91dc0 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 -- 172.21.15.62:0/1694296931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 msgr2=0x558dbabb3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbabb3bb0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fe5240cd9e0 tx=0x7fe5240bc040 comp rx=0 tx=0).stop 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 -- 172.21.15.62:0/1694296931 shutdown_connections 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab9c680 0x558dbab963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab91dc0 0x558dbabb3bb0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/1694296931 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab914a0 0x558dbabb3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 -- 172.21.15.62:0/1694296931 >> 172.21.15.62:0/1694296931 conn(0x558dbab8da30 msgr2=0x558dbab8cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 -- 172.21.15.62:0/1694296931 shutdown_connections 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 -- 172.21.15.62:0/1694296931 wait complete. 2024-09-15T23:26:56.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 -- start start 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.145+0000 7fe53dfcd740 1 Processor -- start 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbaba27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 0x558dbaba2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53dfcd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbac15650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53dfcd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe51800e390 con 0x558dbab9c680 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53dfcd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe51800f800 con 0x558dbab914a0 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53dfcd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe51800ea60 con 0x558dbab91dc0 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53a191640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbac15650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53a191640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbac15650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47740/0 (socket says 172.21.15.62:47740) 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53a191640 1 -- 172.21.15.62:0/3762433140 learned_addr learned my addr 172.21.15.62:0/3762433140 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53dfcd740 10 client.0 _ll_register_callbacks cb 0x7fe5240d38a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe53918f640 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 0x558dbaba2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.146+0000 7fe539990640 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbaba27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe53a191640 1 -- 172.21.15.62:0/3762433140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 msgr2=0x558dbaba2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe53a191640 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 0x558dbaba2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe53a191640 1 -- 172.21.15.62:0/3762433140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 msgr2=0x558dbaba27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe53a191640 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbaba27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe53a191640 1 -- 172.21.15.62:0/3762433140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558dbaba3630 con 0x558dbab9c680 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe53a191640 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbac15650 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe52c003fd0 tx=0x7fe52c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=348ef0a7d67841c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe51d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe51d7fa640 1 -- 172.21.15.62:0/3762433140 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c02e060 con 0x558dbab9c680 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe51d7fa640 1 -- 172.21.15.62:0/3762433140 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe52c00f480 con 0x558dbab9c680 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe53dfcd740 1 -- 172.21.15.62:0/3762433140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe52409c240 con 0x558dbab9c680 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe51d7fa640 1 -- 172.21.15.62:0/3762433140 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe52c035050 con 0x558dbab9c680 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe53dfcd740 10 client.17916 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe53dfcd740 1 -- 172.21.15.62:0/3762433140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558dbac16d10 con 0x558dbab9c680 2024-09-15T23:26:56.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.147+0000 7fe51d7fa640 1 -- 172.21.15.62:0/3762433140 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe52c00fb60 con 0x558dbab9c680 2024-09-15T23:26:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe51d7fa640 1 -- 172.21.15.62:0/3762433140 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fe52c028070 con 0x558dbab9c680 2024-09-15T23:26:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe53dfcd740 10 client.17916 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe53dfcd740 10 client.17916 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:26:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe53dfcd740 20 client.17916 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558dbac17490 0x558dbac198b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe53dfcd740 4 client.17916 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:26:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe53dfcd740 1 -- 172.21.15.62:0/3762433140 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe52409c240 con 0x558dbac17490 2024-09-15T23:26:56.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe539990640 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558dbac17490 0x558dbac198b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe51d7fa640 10 client.17916 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.148+0000 7fe539990640 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558dbac17490 0x558dbac198b0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5088488f04bd59c server_cookie=92e50964aa0586c5 in_seq=0 out_seq=0 2024-09-15T23:26:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe51d7fa640 1 -- 172.21.15.62:0/3762433140 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fe52409c240 con 0x558dbac17490 2024-09-15T23:26:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe51d7fa640 10 client.17916 handle_command_reply: tid=0 2024-09-15T23:26:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 client.17916 shutdown 2024-09-15T23:26:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 -- 172.21.15.62:0/3762433140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 msgr2=0x558dbac15650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbac15650 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fe52c003fd0 tx=0x7fe52c004ab0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f2ff0ff9640 1 -- 172.21.15.62:0/3581071736 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bba6ea5d70 con 0x55bba6fc0060 2024-09-15T23:26:56.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f2ff0ff9640 10 client.17910 handle_command_reply: tid=0 2024-09-15T23:26:56.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f30127ea740 1 client.17910 shutdown 2024-09-15T23:26:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 -- 172.21.15.62:0/3762433140 shutdown_connections 2024-09-15T23:26:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558dbac17490 0x558dbac198b0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558dbab9c680 0x558dbac15650 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558dbab91dc0 0x558dbaba2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 --2- 172.21.15.62:0/3762433140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558dbab914a0 0x558dbaba27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 -- 172.21.15.62:0/3762433140 >> 172.21.15.62:0/3762433140 conn(0x558dbab8da30 msgr2=0x558dbab9a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe51d7fa640 0 client.17916 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 -- 172.21.15.62:0/3762433140 shutdown_connections 2024-09-15T23:26:56.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 1 -- 172.21.15.62:0/3762433140 wait complete. 2024-09-15T23:26:56.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f30127ea740 1 -- 172.21.15.62:0/3581071736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 msgr2=0x55bba6f4db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f30127ea740 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f4db80 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3004000c00 tx=0x7f30040096d0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.149+0000 7fe53dfcd740 20 client.17916 trim_cache size 0 max 16384 2024-09-15T23:26:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f30127ea740 1 -- 172.21.15.62:0/3581071736 shutdown_connections 2024-09-15T23:26:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f30127ea740 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55bba6fc0060 0x55bba6fc2480 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f2ff0ff9640 0 client.17910 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:26:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f30127ea740 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bba6f40370 0x55bba6f4f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f30127ea740 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bba6f3b2e0 0x55bba6f4e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.149+0000 7f30127ea740 1 --2- 172.21.15.62:0/3581071736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bba6f3a9a0 0x55bba6f4db80 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.150+0000 7f30127ea740 1 -- 172.21.15.62:0/3581071736 >> 172.21.15.62:0/3581071736 conn(0x55bba6f36010 msgr2=0x55bba6f351c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.150+0000 7f30127ea740 1 -- 172.21.15.62:0/3581071736 shutdown_connections 2024-09-15T23:26:56.157 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:26:56.157 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:26:56.157 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:26:56.157 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:26:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.150+0000 7f30127ea740 1 -- 172.21.15.62:0/3581071736 wait complete. 2024-09-15T23:26:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.150+0000 7f30127ea740 20 client.17910 trim_cache size 0 max 16384 2024-09-15T23:26:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:26:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:26:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:26:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:26:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 -- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe5200a32c0 msgr2=0x7fe5200a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe5200a32c0 0x7fe5200a5780 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe52c005fd0 tx=0x7fe52c009500 comp rx=0 tx=0).stop 2024-09-15T23:26:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 -- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534101420 msgr2=0x7fe5341a4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534101420 0x7fe5341a4b30 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe52400e9f0 tx=0x7fe52400eeb0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 -- 172.21.15.62:0/2008787269 shutdown_connections 2024-09-15T23:26:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe5200a32c0 0x7fe5200a5780 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe534106a80 0x7fe53419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe534101420 0x7fe5341a4b30 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 --2- 172.21.15.62:0/2008787269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe534100a80 0x7fe5341a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 -- 172.21.15.62:0/2008787269 >> 172.21.15.62:0/2008787269 conn(0x7fe5340fc780 msgr2=0x7fe53406a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 -- 172.21.15.62:0/2008787269 shutdown_connections 2024-09-15T23:26:56.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:56.150+0000 7fe53bbf4640 1 -- 172.21.15.62:0/2008787269 wait complete. 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.150+0000 7f300f96d640 1 -- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ff40a3370 msgr2=0x7f2ff40a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.150+0000 7f300f96d640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ff40a3370 0x7f2ff40a5830 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3000006010 tx=0x7f3000005c50 comp rx=0 tx=0).stop 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.150+0000 7f300f96d640 1 -- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3008101c30 msgr2=0x7f30081982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.150+0000 7f300f96d640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3008101c30 0x7f30081982f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800e9f0 tx=0x7f2ff800edf0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.151+0000 7f300f96d640 1 -- 172.21.15.62:0/3460119811 shutdown_connections 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.151+0000 7f300f96d640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2ff40a3370 0x7f2ff40a5830 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.151+0000 7f300f96d640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30081025f0 0x7f300819f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.151+0000 7f300f96d640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3008101c30 0x7f30081982f0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.151+0000 7f300f96d640 1 --2- 172.21.15.62:0/3460119811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3008100a40 0x7f3008197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.151+0000 7f300f96d640 1 -- 172.21.15.62:0/3460119811 >> 172.21.15.62:0/3460119811 conn(0x7f30080fc1e0 msgr2=0x7f30080fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.151+0000 7f300f96d640 1 -- 172.21.15.62:0/3460119811 shutdown_connections 2024-09-15T23:26:56.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:26:56.151+0000 7f300f96d640 1 -- 172.21.15.62:0/3460119811 wait complete. 2024-09-15T23:26:56.846 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:26:56.847 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:26:57.147 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:57.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:57 smithi062 ceph-mon[30949]: pgmap v690: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.6 MiB/s wr, 194 op/s 2024-09-15T23:26:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:57 smithi067 ceph-mon[35665]: pgmap v690: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.6 MiB/s wr, 194 op/s 2024-09-15T23:26:57.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:57 smithi148 ceph-mon[35610]: pgmap v690: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.6 MiB/s wr, 194 op/s 2024-09-15T23:26:57.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.601+0000 7f4514fed640 1 -- 172.21.15.62:0/3988634151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510100e20 msgr2=0x7f4510101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:57.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.601+0000 7f4514fed640 1 --2- 172.21.15.62:0/3988634151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510100e20 0x7f4510101200 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4500009900 tx=0x7f450002a480 comp rx=0 tx=0).stop 2024-09-15T23:26:57.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.601+0000 7f4514fed640 1 -- 172.21.15.62:0/3988634151 shutdown_connections 2024-09-15T23:26:57.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.601+0000 7f4514fed640 1 --2- 172.21.15.62:0/3988634151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4510102ef0 0x7f451010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.601+0000 7f4514fed640 1 --2- 172.21.15.62:0/3988634151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4510102570 0x7f45101029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.601+0000 7f4514fed640 1 --2- 172.21.15.62:0/3988634151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510100e20 0x7f4510101200 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.601+0000 7f4514fed640 1 -- 172.21.15.62:0/3988634151 >> 172.21.15.62:0/3988634151 conn(0x7f45100fc7d0 msgr2=0x7f45100febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.601+0000 7f4514fed640 1 -- 172.21.15.62:0/3988634151 shutdown_connections 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.601+0000 7f4514fed640 1 -- 172.21.15.62:0/3988634151 wait complete. 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f4514fed640 1 Processor -- start 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f4514fed640 1 -- start start 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f4514fed640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4510100e20 0x7f4510198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f4514fed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510102570 0x7f4510198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f4514fed640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4510102ef0 0x7f451019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f4514fed640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f45101120a0 con 0x7f4510102570 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f4514fed640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4510111f20 con 0x7f4510102ef0 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f4514fed640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4510112220 con 0x7f4510100e20 2024-09-15T23:26:57.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450dd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510102570 0x7f4510198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450dd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510102570 0x7f4510198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47760/0 (socket says 172.21.15.62:47760) 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450dd9b640 1 -- 172.21.15.62:0/3259132524 learned_addr learned my addr 172.21.15.62:0/3259132524 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450e59c640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4510100e20 0x7f4510198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450ed9d640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4510102ef0 0x7f451019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450dd9b640 1 -- 172.21.15.62:0/3259132524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4510100e20 msgr2=0x7f4510198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450dd9b640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4510100e20 0x7f4510198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450dd9b640 1 -- 172.21.15.62:0/3259132524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4510102ef0 msgr2=0x7f451019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450dd9b640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4510102ef0 0x7f451019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.602+0000 7f450dd9b640 1 -- 172.21.15.62:0/3259132524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f451019d400 con 0x7f4510102570 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.603+0000 7f450dd9b640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510102570 0x7f4510198990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e9f0 tx=0x7f44f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.603+0000 7f45077fe640 1 -- 172.21.15.62:0/3259132524 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f800cd80 con 0x7f4510102570 2024-09-15T23:26:57.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.603+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f451019d6f0 con 0x7f4510102570 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.603+0000 7f45077fe640 1 -- 172.21.15.62:0/3259132524 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44f8010910 con 0x7f4510102570 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.603+0000 7f45077fe640 1 -- 172.21.15.62:0/3259132524 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f80188f0 con 0x7f4510102570 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.603+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45101a4fd0 con 0x7f4510102570 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.604+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4510101200 con 0x7f4510102570 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.605+0000 7f45077fe640 1 -- 172.21.15.62:0/3259132524 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f44f8010ac0 con 0x7f4510102570 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.606+0000 7f45077fe640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44fc0a3240 0x7f44fc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.606+0000 7f450e59c640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44fc0a3240 0x7f44fc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.606+0000 7f45077fe640 1 -- 172.21.15.62:0/3259132524 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f44f80d0420 con 0x7f4510102570 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.606+0000 7f450e59c640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44fc0a3240 0x7f44fc0a5700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4500009230 tx=0x7f45000058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:57.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.609+0000 7f45077fe640 1 -- 172.21.15.62:0/3259132524 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f44f809c770 con 0x7f4510102570 2024-09-15T23:26:57.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.819+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45101a5200 con 0x7f4510102570 2024-09-15T23:26:57.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.820+0000 7f45077fe640 1 -- 172.21.15.62:0/3259132524 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7f44f80a0a70 con 0x7f4510102570 2024-09-15T23:26:57.824 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:26:57.824 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:26:57.825 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:26:57.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44fc0a3240 msgr2=0x7f44fc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44fc0a3240 0x7f44fc0a5700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4500009230 tx=0x7f45000058d0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510102570 msgr2=0x7f4510198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510102570 0x7f4510198990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f44f800e9f0 tx=0x7f44f800eeb0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 shutdown_connections 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f44fc0a3240 0x7f44fc0a5700 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4510102ef0 0x7f451019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4510102570 0x7f4510198990 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 --2- 172.21.15.62:0/3259132524 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4510100e20 0x7f4510198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 >> 172.21.15.62:0/3259132524 conn(0x7f45100fc7d0 msgr2=0x7f45100fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 shutdown_connections 2024-09-15T23:26:57.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:57.822+0000 7f4514fed640 1 -- 172.21.15.62:0/3259132524 wait complete. 2024-09-15T23:26:58.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:58 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3259132524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:58.498 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:26:58.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:58 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3259132524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:58.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:58 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3259132524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:26:58.758 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:26:59.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.204+0000 7f7929fec640 1 -- 172.21.15.62:0/558582922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 msgr2=0x7f79241021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.204+0000 7f7929fec640 1 --2- 172.21.15.62:0/558582922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 0x7f79241021c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f7914009900 tx=0x7f791402a410 comp rx=0 tx=0).stop 2024-09-15T23:26:59.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:26:59 smithi062 ceph-mon[30949]: pgmap v691: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 728 KiB/s wr, 68 op/s 2024-09-15T23:26:59.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.204+0000 7f7929fec640 1 -- 172.21.15.62:0/558582922 shutdown_connections 2024-09-15T23:26:59.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.204+0000 7f7929fec640 1 --2- 172.21.15.62:0/558582922 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7924106f10 0x7f792410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.204+0000 7f7929fec640 1 --2- 172.21.15.62:0/558582922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924102790 0x7f79241067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.204+0000 7f7929fec640 1 --2- 172.21.15.62:0/558582922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 0x7f79241021c0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.204+0000 7f7929fec640 1 -- 172.21.15.62:0/558582922 >> 172.21.15.62:0/558582922 conn(0x7f79240fd8c0 msgr2=0x7f79240ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:59.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.204+0000 7f7929fec640 1 -- 172.21.15.62:0/558582922 shutdown_connections 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.204+0000 7f7929fec640 1 -- 172.21.15.62:0/558582922 wait complete. 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.205+0000 7f7929fec640 1 Processor -- start 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.205+0000 7f7929fec640 1 -- start start 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.205+0000 7f7929fec640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 0x7f7924198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.205+0000 7f7929fec640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7924102790 0x7f7924198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.205+0000 7f7929fec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924106f10 0x7f792419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.205+0000 7f7929fec640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7924111f70 con 0x7f7924101de0 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.205+0000 7f7929fec640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7924111df0 con 0x7f7924106f10 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.205+0000 7f7929fec640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f79241120f0 con 0x7f7924102790 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f79237fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 0x7f7924198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f79237fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 0x7f7924198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47780/0 (socket says 172.21.15.62:47780) 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f79237fe640 1 -- 172.21.15.62:0/513769294 learned_addr learned my addr 172.21.15.62:0/513769294 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:59.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f7922ffd640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7924102790 0x7f7924198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f7923fff640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924106f10 0x7f792419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f79237fe640 1 -- 172.21.15.62:0/513769294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7924102790 msgr2=0x7f7924198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f79237fe640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7924102790 0x7f7924198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f79237fe640 1 -- 172.21.15.62:0/513769294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924106f10 msgr2=0x7f792419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f79237fe640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924106f10 0x7f792419cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f79237fe640 1 -- 172.21.15.62:0/513769294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792419d3c0 con 0x7f7924101de0 2024-09-15T23:26:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f79237fe640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 0x7f7924198410 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f7914006010 tx=0x7f79140042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.206+0000 7f7920ff9640 1 -- 172.21.15.62:0/513769294 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791403f030 con 0x7f7924101de0 2024-09-15T23:26:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.207+0000 7f7920ff9640 1 -- 172.21.15.62:0/513769294 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7914003ce0 con 0x7f7924101de0 2024-09-15T23:26:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.207+0000 7f7929fec640 1 -- 172.21.15.62:0/513769294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792419d650 con 0x7f7924101de0 2024-09-15T23:26:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.207+0000 7f7929fec640 1 -- 172.21.15.62:0/513769294 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79241a4f90 con 0x7f7924101de0 2024-09-15T23:26:59.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.207+0000 7f7920ff9640 1 -- 172.21.15.62:0/513769294 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7914035070 con 0x7f7924101de0 2024-09-15T23:26:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.209+0000 7f7920ff9640 1 -- 172.21.15.62:0/513769294 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7914042030 con 0x7f7924101de0 2024-09-15T23:26:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f792c3c5740 1 Processor -- start 2024-09-15T23:26:59.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f792c3c5740 1 -- start start 2024-09-15T23:26:59.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x558646189090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586461895d0 0x558646189a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558646194370 0x558646196760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f792c3c5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5586460ef2f0 con 0x55864618ab40 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f792c3c5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558646096cf0 con 0x5586461895d0 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f792c3c5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5586460eb2b0 con 0x558646194370 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7923fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x558646189090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7923fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x558646189090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47792/0 (socket says 172.21.15.62:47792) 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7923fff640 1 -- 172.21.15.62:0/13820538 learned_addr learned my addr 172.21.15.62:0/13820538 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7922ffd640 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558646194370 0x558646196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f79237fe640 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586461895d0 0x558646189a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7923fff640 1 -- 172.21.15.62:0/13820538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558646194370 msgr2=0x558646196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:59.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7923fff640 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558646194370 0x558646196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7923fff640 1 -- 172.21.15.62:0/13820538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586461895d0 msgr2=0x558646189a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7923fff640 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586461895d0 0x558646189a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7923fff640 1 -- 172.21.15.62:0/13820538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55864607b3a0 con 0x55864618ab40 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.210+0000 7f7922ffd640 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558646194370 0x558646196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.211+0000 7f7920ff9640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f790c0a3240 0x7f790c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.211+0000 7f79237fe640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f790c0a3240 0x7f790c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.211+0000 7f7923fff640 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x558646189090 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f791c009f70 tx=0x7f791c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=882b3cb9fc9512fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.211+0000 7f7920ff9640 1 -- 172.21.15.62:0/513769294 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f79140f0210 con 0x7f7924101de0 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.211+0000 7f7909ffb640 1 -- 172.21.15.62:0/13820538 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c016070 con 0x55864618ab40 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.211+0000 7f7909ffb640 1 -- 172.21.15.62:0/13820538 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f791c004d20 con 0x55864618ab40 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.211+0000 7f7909ffb640 1 -- 172.21.15.62:0/13820538 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c012950 con 0x55864618ab40 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 -- 172.21.15.62:0/13820538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 msgr2=0x558646189090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f79237fe640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f790c0a3240 0x7f790c0a5700 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f79140bc5b0 tx=0x7f79140f5040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:59.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x558646189090 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f791c009f70 tx=0x7f791c010800 comp rx=0 tx=0).stop 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 -- 172.21.15.62:0/13820538 shutdown_connections 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558646194370 0x558646196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5586461895d0 0x558646189a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 --2- 172.21.15.62:0/13820538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x558646189090 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 -- 172.21.15.62:0/13820538 >> 172.21.15.62:0/13820538 conn(0x558646152310 msgr2=0x558646185c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 -- 172.21.15.62:0/13820538 shutdown_connections 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 -- 172.21.15.62:0/13820538 wait complete. 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 -- start start 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.212+0000 7f792c3c5740 1 Processor -- start 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x558646193e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x5586461910c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 0x558646191850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f792c3c5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558646197070 con 0x55864618ab40 2024-09-15T23:26:59.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f792c3c5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5586460aa060 con 0x558646194370 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f792c3c5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5586461971f0 con 0x5586461895d0 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f792c3c5740 10 client.0 _ll_register_callbacks cb 0x558645dfd8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7922ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x5586461910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7922ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x5586461910c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47802/0 (socket says 172.21.15.62:47802) 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7922ffd640 1 -- 172.21.15.62:0/2722903613 learned_addr learned my addr 172.21.15.62:0/2722903613 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7923fff640 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 0x558646191850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7923fff640 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x558646193e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7922ffd640 1 -- 172.21.15.62:0/2722903613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 msgr2=0x558646193e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7922ffd640 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x558646193e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7922ffd640 1 -- 172.21.15.62:0/2722903613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 msgr2=0x558646191850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7922ffd640 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 0x558646191850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7922ffd640 1 -- 172.21.15.62:0/2722903613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558645e603c0 con 0x55864618ab40 2024-09-15T23:26:59.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7923fff640 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 0x558646191850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7922ffd640 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x5586461910c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f791800ba70 tx=0x7f791800bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab249dd5356c419 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.213+0000 7f7902ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f7902ffd640 1 -- 172.21.15.62:0/2722903613 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791801b640 con 0x55864618ab40 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f7902ffd640 1 -- 172.21.15.62:0/2722903613 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7918011440 con 0x55864618ab40 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f792c3c5740 1 -- 172.21.15.62:0/2722903613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f791c004d20 con 0x55864618ab40 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f7902ffd640 1 -- 172.21.15.62:0/2722903613 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7918003e40 con 0x55864618ab40 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f792c3c5740 10 client.17934 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f792c3c5740 1 -- 172.21.15.62:0/2722903613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5586461a2c20 con 0x55864618ab40 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f7902ffd640 1 -- 172.21.15.62:0/2722903613 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f791c004d20 con 0x55864618ab40 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f7902ffd640 1 -- 172.21.15.62:0/2722903613 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f7918019020 con 0x55864618ab40 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f792c3c5740 10 client.17934 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f792c3c5740 10 client.17934 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f792c3c5740 20 client.17934 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:59.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f792c3c5740 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5586461a4130 0x5586461a6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f792c3c5740 4 client.17934 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:26:59.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.214+0000 7f792c3c5740 1 -- 172.21.15.62:0/2722903613 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586460fc860 con 0x5586461a4130 2024-09-15T23:26:59.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.215+0000 7f7923fff640 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5586461a4130 0x5586461a6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.215+0000 7f7902ffd640 10 client.17934 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:26:59.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.215+0000 7f7923fff640 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5586461a4130 0x5586461a6550 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dbd021db8b808de9 server_cookie=a3f9f1d84d06331 in_seq=0 out_seq=0 2024-09-15T23:26:59.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f7902ffd640 1 -- 172.21.15.62:0/2722903613 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5586460fc860 con 0x5586461a4130 2024-09-15T23:26:59.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f7902ffd640 10 client.17934 handle_command_reply: tid=0 2024-09-15T23:26:59.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 client.17934 shutdown 2024-09-15T23:26:59.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 -- 172.21.15.62:0/2722903613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 msgr2=0x5586461910c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x5586461910c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f791800ba70 tx=0x7f791800bf30 comp rx=0 tx=0).stop 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 -- 172.21.15.62:0/2722903613 shutdown_connections 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5586461a4130 0x5586461a6550 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 0x558646191850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x5586461910c0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 --2- 172.21.15.62:0/2722903613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x558646193e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 -- 172.21.15.62:0/2722903613 >> 172.21.15.62:0/2722903613 conn(0x558646152310 msgr2=0x55864618b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f7902ffd640 0 client.17934 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 -- 172.21.15.62:0/2722903613 shutdown_connections 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 1 -- 172.21.15.62:0/2722903613 wait complete. 2024-09-15T23:26:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.216+0000 7f792c3c5740 20 client.17934 trim_cache size 0 max 16384 2024-09-15T23:26:59.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.229+0000 7f792c3c5740 1 Processor -- start 2024-09-15T23:26:59.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.229+0000 7f792c3c5740 1 -- start start 2024-09-15T23:26:59.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.229+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x55864618ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.229+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x55864619c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.229+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 0x55864619cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.229+0000 7f792c3c5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f79241b1540 con 0x55864618ab40 2024-09-15T23:26:59.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.229+0000 7f792c3c5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79241aee10 con 0x558646194370 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.229+0000 7f792c3c5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f79241acc60 con 0x5586461895d0 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7922ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x55864619c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7922ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x55864619c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47818/0 (socket says 172.21.15.62:47818) 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7922ffd640 1 -- 172.21.15.62:0/3832106788 learned_addr learned my addr 172.21.15.62:0/3832106788 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7923fff640 1 --2- 172.21.15.62:0/3832106788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 0x55864619cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7922ffd640 1 -- 172.21.15.62:0/3832106788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 msgr2=0x55864618ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7922ffd640 1 --2- 172.21.15.62:0/3832106788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x55864618ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7922ffd640 1 -- 172.21.15.62:0/3832106788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 msgr2=0x55864619cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7922ffd640 1 --2- 172.21.15.62:0/3832106788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 0x55864619cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7922ffd640 1 -- 172.21.15.62:0/3832106788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55864618f610 con 0x55864618ab40 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f7922ffd640 1 --2- 172.21.15.62:0/3832106788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x55864619c860 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f7918011c30 tx=0x7f7918005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc69c61bd72ce4e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:59.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f79097fa640 1 -- 172.21.15.62:0/3832106788 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791802c020 con 0x55864618ab40 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f79097fa640 1 -- 172.21.15.62:0/3832106788 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7918005850 con 0x55864618ab40 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.230+0000 7f79097fa640 1 -- 172.21.15.62:0/3832106788 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79180048e0 con 0x55864618ab40 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 -- 172.21.15.62:0/3832106788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 msgr2=0x55864619c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3832106788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x55864619c860 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f7918011c30 tx=0x7f7918005b90 comp rx=0 tx=0).stop 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 -- 172.21.15.62:0/3832106788 shutdown_connections 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3832106788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558646194370 0x55864619cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3832106788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55864618ab40 0x55864619c860 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3832106788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x55864618ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 -- 172.21.15.62:0/3832106788 >> 172.21.15.62:0/3832106788 conn(0x558646152310 msgr2=0x55864618f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 -- 172.21.15.62:0/3832106788 shutdown_connections 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 -- 172.21.15.62:0/3832106788 wait complete. 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 -- start start 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.231+0000 7f792c3c5740 1 Processor -- start 2024-09-15T23:26:59.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x5586461ab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55864618ab40 0x5586461abe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f792c3c5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558646194370 0x55864620a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f792c3c5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f79241ad2d0 con 0x558646194370 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f792c3c5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79241ae740 con 0x55864618ab40 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f792c3c5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f79241ad9a0 con 0x5586461895d0 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f792c3c5740 10 client.0 _ll_register_callbacks cb 0x7f7918004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558646194370 0x55864620a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7922ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55864618ab40 0x5586461abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558646194370 0x55864620a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47832/0 (socket says 172.21.15.62:47832) 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 -- 172.21.15.62:0/3600171875 learned_addr learned my addr 172.21.15.62:0/3600171875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x5586461ab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 -- 172.21.15.62:0/3600171875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 msgr2=0x5586461ab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:59.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x5586461ab8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 -- 172.21.15.62:0/3600171875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55864618ab40 msgr2=0x5586461abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55864618ab40 0x5586461abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 -- 172.21.15.62:0/3600171875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55864620a6f0 con 0x558646194370 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7923fff640 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558646194370 0x55864620a050 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f791c009ec0 tx=0x7f791c012450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e1806645e6f7e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.232+0000 7f7902ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f7902ffd640 1 -- 172.21.15.62:0/3600171875 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c016040 con 0x558646194370 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f7902ffd640 1 -- 172.21.15.62:0/3600171875 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f791c003a30 con 0x558646194370 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f792c3c5740 1 -- 172.21.15.62:0/3600171875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7918005850 con 0x558646194370 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f7902ffd640 1 -- 172.21.15.62:0/3600171875 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f791c023040 con 0x558646194370 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f792c3c5740 10 client.17940 fetch_fsmap learned FSMap version 35 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f792c3c5740 1 -- 172.21.15.62:0/3600171875 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558646212c60 con 0x558646194370 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f7902ffd640 1 -- 172.21.15.62:0/3600171875 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7918005850 con 0x558646194370 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f7902ffd640 1 -- 172.21.15.62:0/3600171875 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f791c029040 con 0x558646194370 2024-09-15T23:26:59.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f792c3c5740 10 client.17940 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f792c3c5740 10 client.17940 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f792c3c5740 20 client.17940 populate_metadata read hostname 'smithi062' 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x558646213240 0x558646215630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f792c3c5740 4 client.17940 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.233+0000 7f792c3c5740 1 -- 172.21.15.62:0/3600171875 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5586460f6d70 con 0x558646213240 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f7922ffd640 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x558646213240 0x558646215630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f7922ffd640 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x558646213240 0x558646215630 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8353390a8c412ec9 server_cookie=414d5e50f7a1be59 in_seq=0 out_seq=0 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f7902ffd640 10 client.17940 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f7902ffd640 1 -- 172.21.15.62:0/3600171875 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5586460f6d70 con 0x558646213240 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f7902ffd640 10 client.17940 handle_command_reply: tid=0 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f792c3c5740 1 client.17940 shutdown 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f792c3c5740 1 -- 172.21.15.62:0/3600171875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558646194370 msgr2=0x55864620a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558646194370 0x55864620a050 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f791c009ec0 tx=0x7f791c012450 comp rx=0 tx=0).stop 2024-09-15T23:26:59.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f792c3c5740 1 -- 172.21.15.62:0/3600171875 shutdown_connections 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x558646213240 0x558646215630 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558646194370 0x55864620a050 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55864618ab40 0x5586461abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f792c3c5740 1 --2- 172.21.15.62:0/3600171875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5586461895d0 0x5586461ab8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f792c3c5740 1 -- 172.21.15.62:0/3600171875 >> 172.21.15.62:0/3600171875 conn(0x558646152310 msgr2=0x55864618b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.234+0000 7f7902ffd640 0 client.17940 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f792c3c5740 1 -- 172.21.15.62:0/3600171875 shutdown_connections 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f792c3c5740 1 -- 172.21.15.62:0/3600171875 wait complete. 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f792c3c5740 20 client.17940 trim_cache size 0 max 16384 2024-09-15T23:26:59.244 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:26:59.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 -- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f790c0a3240 msgr2=0x7f790c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f790c0a3240 0x7f790c0a5700 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f79140bc5b0 tx=0x7f79140f5040 comp rx=0 tx=0).stop 2024-09-15T23:26:59.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 -- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 msgr2=0x7f7924198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:26:59.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 0x7f7924198410 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f7914006010 tx=0x7f79140042f0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 -- 172.21.15.62:0/513769294 shutdown_connections 2024-09-15T23:26:59.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f790c0a3240 0x7f790c0a5700 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7924106f10 0x7f792419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7924102790 0x7f7924198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 --2- 172.21.15.62:0/513769294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7924101de0 0x7f7924198410 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:26:59.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 -- 172.21.15.62:0/513769294 >> 172.21.15.62:0/513769294 conn(0x7f79240fd8c0 msgr2=0x7f79240fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:26:59.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 -- 172.21.15.62:0/513769294 shutdown_connections 2024-09-15T23:26:59.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:26:59.235+0000 7f7929fec640 1 -- 172.21.15.62:0/513769294 wait complete. 2024-09-15T23:26:59.507 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:26:59.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:26:59 smithi067 ceph-mon[35665]: pgmap v691: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 728 KiB/s wr, 68 op/s 2024-09-15T23:26:59.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:26:59 smithi148 ceph-mon[35610]: pgmap v691: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 728 KiB/s wr, 68 op/s 2024-09-15T23:26:59.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 640.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:26:59.890 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:26:59.948 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:00.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 -- 172.21.15.62:0/973922235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b641037a0 msgr2=0x7f8b6410bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 --2- 172.21.15.62:0/973922235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b641037a0 0x7f8b6410bc00 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009900 tx=0x7f8b4c02a410 comp rx=0 tx=0).stop 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 -- 172.21.15.62:0/973922235 shutdown_connections 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 --2- 172.21.15.62:0/973922235 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8b6410c2f0 0x7f8b6410e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 --2- 172.21.15.62:0/973922235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b641037a0 0x7f8b6410bc00 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 --2- 172.21.15.62:0/973922235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b64102df0 0x7f8b641031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 -- 172.21.15.62:0/973922235 >> 172.21.15.62:0/973922235 conn(0x7f8b640fc7d0 msgr2=0x7f8b640febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 -- 172.21.15.62:0/973922235 shutdown_connections 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 -- 172.21.15.62:0/973922235 wait complete. 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.409+0000 7f8b6962c640 1 Processor -- start 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b6962c640 1 -- start start 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b6962c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b64102df0 0x7f8b64100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b6962c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8b641037a0 0x7f8b64100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:00.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b6962c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b6410c2f0 0x7f8b641014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b6962c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8b641120d0 con 0x7f8b6410c2f0 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b6962c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64111f50 con 0x7f8b64102df0 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b6962c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64112250 con 0x7f8b641037a0 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b637fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b6410c2f0 0x7f8b641014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b637fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b6410c2f0 0x7f8b641014d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47846/0 (socket says 172.21.15.62:47846) 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b637fe640 1 -- 172.21.15.62:0/4087232404 learned_addr learned my addr 172.21.15.62:0/4087232404 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b627fc640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8b641037a0 0x7f8b64100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b62ffd640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b64102df0 0x7f8b64100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b637fe640 1 -- 172.21.15.62:0/4087232404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8b641037a0 msgr2=0x7f8b64100f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b637fe640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8b641037a0 0x7f8b64100f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b637fe640 1 -- 172.21.15.62:0/4087232404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b64102df0 msgr2=0x7f8b64100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b637fe640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b64102df0 0x7f8b64100a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.410+0000 7f8b637fe640 1 -- 172.21.15.62:0/4087232404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b641a6a30 con 0x7f8b6410c2f0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.411+0000 7f8b637fe640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b6410c2f0 0x7f8b641014d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400d750 tx=0x7f8b5400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.411+0000 7f8b5bfff640 1 -- 172.21.15.62:0/4087232404 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b54017070 con 0x7f8b6410c2f0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.411+0000 7f8b5bfff640 1 -- 172.21.15.62:0/4087232404 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b54005500 con 0x7f8b6410c2f0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.411+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b641a6d20 con 0x7f8b6410c2f0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.411+0000 7f8b5bfff640 1 -- 172.21.15.62:0/4087232404 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b54004d90 con 0x7f8b6410c2f0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.411+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b641a7230 con 0x7f8b6410c2f0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.412+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b641031d0 con 0x7f8b6410c2f0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.414+0000 7f8b5bfff640 1 -- 172.21.15.62:0/4087232404 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8b54004f30 con 0x7f8b6410c2f0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.415+0000 7f8b5bfff640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8b500a3240 0x7f8b500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.415+0000 7f8b62ffd640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8b500a3240 0x7f8b500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.415+0000 7f8b5bfff640 1 -- 172.21.15.62:0/4087232404 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8b540cf920 con 0x7f8b6410c2f0 2024-09-15T23:27:00.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.415+0000 7f8b62ffd640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8b500a3240 0x7f8b500a5700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007ec0 tx=0x7f8b5c0075b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:00.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.418+0000 7f8b5bfff640 1 -- 172.21.15.62:0/4087232404 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8b5409bc70 con 0x7f8b6410c2f0 2024-09-15T23:27:00.468 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:00.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.644+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b641072c0 con 0x7f8b6410c2f0 2024-09-15T23:27:00.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.645+0000 7f8b5bfff640 1 -- 172.21.15.62:0/4087232404 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7f8b5409ff70 con 0x7f8b6410c2f0 2024-09-15T23:27:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:27:00.648 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:27:00.649 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8b500a3240 msgr2=0x7f8b500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8b500a3240 0x7f8b500a5700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007ec0 tx=0x7f8b5c0075b0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b6410c2f0 msgr2=0x7f8b641014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b6410c2f0 0x7f8b641014d0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400d750 tx=0x7f8b5400dc10 comp rx=0 tx=0).stop 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 shutdown_connections 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8b500a3240 0x7f8b500a5700 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b6410c2f0 0x7f8b641014d0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8b641037a0 0x7f8b64100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 --2- 172.21.15.62:0/4087232404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b64102df0 0x7f8b64100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.647+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 >> 172.21.15.62:0/4087232404 conn(0x7f8b640fc7d0 msgr2=0x7f8b640fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.648+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 shutdown_connections 2024-09-15T23:27:00.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:00.648+0000 7f8b6962c640 1 -- 172.21.15.62:0/4087232404 wait complete. 2024-09-15T23:27:01.436 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:01 smithi062 ceph-mon[30949]: pgmap v692: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 50 KiB/s wr, 15 op/s 2024-09-15T23:27:01.436 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:01 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4087232404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:01.464 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 724f1b2d --release --await' 2024-09-15T23:27:01.465 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 724f1b2d --release --await 2024-09-15T23:27:01.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:01 smithi067 ceph-mon[35665]: pgmap v692: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 50 KiB/s wr, 15 op/s 2024-09-15T23:27:01.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:01 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4087232404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:01.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:01 smithi148 ceph-mon[35610]: pgmap v692: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 50 KiB/s wr, 15 op/s 2024-09-15T23:27:01.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:01 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4087232404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.679+0000 7f529b368640 1 -- 172.21.15.62:0/1795276060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 msgr2=0x7f529410ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.679+0000 7f529b368640 1 --2- 172.21.15.62:0/1795276060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 0x7f529410ec50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f528c009cd0 tx=0x7f528c02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.679+0000 7f529b368640 1 -- 172.21.15.62:0/1795276060 shutdown_connections 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.679+0000 7f529b368640 1 --2- 172.21.15.62:0/1795276060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 0x7f529410ec50 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.679+0000 7f529b368640 1 --2- 172.21.15.62:0/1795276060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5294103760 0x7f5294107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.679+0000 7f529b368640 1 --2- 172.21.15.62:0/1795276060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294102db0 0x7f5294103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.679+0000 7f529b368640 1 -- 172.21.15.62:0/1795276060 >> 172.21.15.62:0/1795276060 conn(0x7f52940fc7d0 msgr2=0x7f52940febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.679+0000 7f529b368640 1 -- 172.21.15.62:0/1795276060 shutdown_connections 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.679+0000 7f529b368640 1 -- 172.21.15.62:0/1795276060 wait complete. 2024-09-15T23:27:01.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f529b368640 1 Processor -- start 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f529b368640 1 -- start start 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f529b368640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5294102db0 0x7f52941985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f529b368640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294103760 0x7f5294198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f529b368640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 0x7f529419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f529b368640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f52941120b0 con 0x7f52941082e0 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f529b368640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5294111f30 con 0x7f5294103760 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f529b368640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5294112230 con 0x7f5294102db0 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5299905640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 0x7f529419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5299905640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 0x7f529419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47862/0 (socket says 172.21.15.62:47862) 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5299905640 1 -- 172.21.15.62:0/2650250716 learned_addr learned my addr 172.21.15.62:0/2650250716 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5299104640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5294102db0 0x7f52941985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5298903640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294103760 0x7f5294198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5299905640 1 -- 172.21.15.62:0/2650250716 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5294102db0 msgr2=0x7f52941985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5299905640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5294102db0 0x7f52941985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5299905640 1 -- 172.21.15.62:0/2650250716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294103760 msgr2=0x7f5294198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5299905640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294103760 0x7f5294198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.680+0000 7f5299905640 1 -- 172.21.15.62:0/2650250716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529419d560 con 0x7f52941082e0 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.681+0000 7f5299905640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 0x7f529419ce80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f528c005e30 tx=0x7f528c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.681+0000 7f52867fc640 1 -- 172.21.15.62:0/2650250716 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c004040 con 0x7f52941082e0 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.681+0000 7f52867fc640 1 -- 172.21.15.62:0/2650250716 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f528c0046c0 con 0x7f52941082e0 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.681+0000 7f529b368640 1 -- 172.21.15.62:0/2650250716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529419d7f0 con 0x7f52941082e0 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.681+0000 7f52867fc640 1 -- 172.21.15.62:0/2650250716 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528c03a400 con 0x7f52941082e0 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.681+0000 7f529b368640 1 -- 172.21.15.62:0/2650250716 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52941a5130 con 0x7f52941082e0 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.683+0000 7f52867fc640 1 -- 172.21.15.62:0/2650250716 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f528c030630 con 0x7f52941082e0 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.683+0000 7f529d741740 1 Processor -- start 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f529d741740 1 -- start start 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f529d741740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b14074b40 0x562b14073090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f529d741740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b14073a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f529d741740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b1407e370 0x562b14080760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f529d741740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562b13fd92f0 con 0x562b140735d0 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f529d741740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562b13f80cf0 con 0x562b14074b40 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f529d741740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562b13fd52b0 con 0x562b1407e370 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f5298903640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b14074b40 0x562b14073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f5299104640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b14073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f5298903640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b14074b40 0x562b14073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:57268/0 (socket says 172.21.15.62:57268) 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f52867fc640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f52800a3240 0x7f52800a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f5298903640 1 -- 172.21.15.62:0/4055218103 learned_addr learned my addr 172.21.15.62:0/4055218103 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f5298903640 1 --2- 172.21.15.62:0/4055218103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b1407e370 0x562b14080760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.684+0000 7f52867fc640 1 -- 172.21.15.62:0/2650250716 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f528c02e030 con 0x7f52941082e0 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f5298903640 1 -- 172.21.15.62:0/4055218103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b1407e370 msgr2=0x562b14080760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f5298903640 1 --2- 172.21.15.62:0/4055218103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b1407e370 0x562b14080760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f5298903640 1 -- 172.21.15.62:0/4055218103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 msgr2=0x562b14073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f5298903640 1 --2- 172.21.15.62:0/4055218103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b14073a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f5298903640 1 -- 172.21.15.62:0/4055218103 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b13f653a0 con 0x562b14074b40 2024-09-15T23:27:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f5299104640 1 --2- 172.21.15.62:0/4055218103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b14073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f5299905640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f52800a3240 0x7f52800a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f5298903640 1 --2- 172.21.15.62:0/4055218103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b14074b40 0x562b14073090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f528800bfd0 tx=0x7f528800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ada2f283933623cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f5299905640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f52800a3240 0x7f52800a5700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f528c0c56e0 tx=0x7f528c0309d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f527f7fe640 1 -- 172.21.15.62:0/4055218103 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288004250 con 0x562b14074b40 2024-09-15T23:27:01.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f527f7fe640 1 -- 172.21.15.62:0/4055218103 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52880043f0 con 0x562b14074b40 2024-09-15T23:27:01.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.685+0000 7f527f7fe640 1 -- 172.21.15.62:0/4055218103 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52880035f0 con 0x562b14074b40 2024-09-15T23:27:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.686+0000 7f529d741740 1 -- 172.21.15.62:0/4055218103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b14074b40 msgr2=0x562b14073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.686+0000 7f529d741740 1 --2- 172.21.15.62:0/4055218103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b14074b40 0x562b14073090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f528800bfd0 tx=0x7f528800b790 comp rx=0 tx=0).stop 2024-09-15T23:27:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.686+0000 7f529d741740 1 -- 172.21.15.62:0/4055218103 shutdown_connections 2024-09-15T23:27:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.686+0000 7f529d741740 1 --2- 172.21.15.62:0/4055218103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b1407e370 0x562b14080760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.686+0000 7f529d741740 1 --2- 172.21.15.62:0/4055218103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b14073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.686+0000 7f529d741740 1 --2- 172.21.15.62:0/4055218103 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b14074b40 0x562b14073090 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.686+0000 7f529d741740 1 -- 172.21.15.62:0/4055218103 >> 172.21.15.62:0/4055218103 conn(0x562b1403c310 msgr2=0x562b1406fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.686+0000 7f529d741740 1 -- 172.21.15.62:0/4055218103 shutdown_connections 2024-09-15T23:27:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.686+0000 7f529d741740 1 -- 172.21.15.62:0/4055218103 wait complete. 2024-09-15T23:27:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f529d741740 1 -- start start 2024-09-15T23:27:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f529d741740 1 Processor -- start 2024-09-15T23:27:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f529d741740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 0x562b1407dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f529d741740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b1407b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f529d741740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1407b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f529d741740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x562b14081070 con 0x562b1407e370 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f529d741740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x562b13f94060 con 0x562b140735d0 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f529d741740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x562b140811f0 con 0x562b14074b40 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f529d741740 10 client.0 _ll_register_callbacks cb 0x562b13ce78c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f5298903640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1407b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f5298903640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1407b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47878/0 (socket says 172.21.15.62:47878) 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.687+0000 7f5298903640 1 -- 172.21.15.62:0/1520544151 learned_addr learned my addr 172.21.15.62:0/1520544151 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f5298903640 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 0x562b1407dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f5298903640 1 -- 172.21.15.62:0/1520544151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 msgr2=0x562b1407b010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f5298903640 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b1407b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f5298903640 1 -- 172.21.15.62:0/1520544151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 msgr2=0x562b1407dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f5298903640 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 0x562b1407dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f5298903640 1 -- 172.21.15.62:0/1520544151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b13d4a3c0 con 0x562b1407e370 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f527cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f5298903640 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1407b600 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f528800b3f0 tx=0x7f5288004c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdbe3858cfd53842 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f527cff9640 1 -- 172.21.15.62:0/1520544151 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288004f00 con 0x562b1407e370 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f529d741740 1 -- 172.21.15.62:0/1520544151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52880043f0 con 0x562b1407e370 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f527cff9640 1 -- 172.21.15.62:0/1520544151 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5288029040 con 0x562b1407e370 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.688+0000 7f527cff9640 1 -- 172.21.15.62:0/1520544151 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5288004f00 con 0x562b1407e370 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f529d741740 10 client.17958 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f529d741740 1 -- 172.21.15.62:0/1520544151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562b1408cb10 con 0x562b1407e370 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f527cff9640 1 -- 172.21.15.62:0/1520544151 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52880043f0 con 0x562b1407e370 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f527cff9640 1 -- 172.21.15.62:0/1520544151 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f528801c020 con 0x562b1407e370 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f529d741740 10 client.17958 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f529d741740 10 client.17958 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f529d741740 20 client.17958 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:01.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f529d741740 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x562b1408e1d0 0x562b14090650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f529d741740 4 client.17958 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f529d741740 1 -- 172.21.15.62:0/1520544151 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b13fe6860 con 0x562b1408e1d0 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.689+0000 7f5299104640 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x562b1408e1d0 0x562b14090650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.690+0000 7f5299104640 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x562b1408e1d0 0x562b14090650 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=576ebe00f13ecf92 server_cookie=167526d7f0be1b20 in_seq=0 out_seq=0 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.690+0000 7f527cff9640 10 client.17958 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.690+0000 7f527cff9640 1 -- 172.21.15.62:0/1520544151 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562b13fe6860 con 0x562b1408e1d0 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.690+0000 7f527cff9640 10 client.17958 handle_command_reply: tid=0 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 client.17958 shutdown 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 -- 172.21.15.62:0/1520544151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 msgr2=0x562b1407b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1407b600 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f528800b3f0 tx=0x7f5288004c10 comp rx=0 tx=0).stop 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 -- 172.21.15.62:0/1520544151 shutdown_connections 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x562b1408e1d0 0x562b14090650 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1407b600 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b1407b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 --2- 172.21.15.62:0/1520544151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 0x562b1407dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f527cff9640 0 client.17958 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:27:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 -- 172.21.15.62:0/1520544151 >> 172.21.15.62:0/1520544151 conn(0x562b1403c310 msgr2=0x562b14075930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 -- 172.21.15.62:0/1520544151 shutdown_connections 2024-09-15T23:27:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 1 -- 172.21.15.62:0/1520544151 wait complete. 2024-09-15T23:27:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.691+0000 7f529d741740 20 client.17958 trim_cache size 0 max 16384 2024-09-15T23:27:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.705+0000 7f529d741740 1 Processor -- start 2024-09-15T23:27:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.705+0000 7f529d741740 1 -- start start 2024-09-15T23:27:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f529d741740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 0x562b1408a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f529d741740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b1406f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f529d741740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1406fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f529d741740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f52941b14e0 con 0x562b1407e370 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f529d741740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f52941aedb0 con 0x562b140735d0 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f529d741740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f52941acc00 con 0x562b14074b40 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5298903640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1406fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5298903640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1406fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47894/0 (socket says 172.21.15.62:47894) 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5298903640 1 -- 172.21.15.62:0/3472304137 learned_addr learned my addr 172.21.15.62:0/3472304137 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5299104640 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b1406f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5298903640 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 0x562b1408a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5298903640 1 -- 172.21.15.62:0/3472304137 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 msgr2=0x562b1406f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5298903640 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b1406f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5298903640 1 -- 172.21.15.62:0/3472304137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 msgr2=0x562b1408a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5298903640 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 0x562b1408a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5298903640 1 -- 172.21.15.62:0/3472304137 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b14079560 con 0x562b1407e370 2024-09-15T23:27:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.706+0000 7f5299104640 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b1406f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.707+0000 7f5298903640 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1406fa50 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f528802c720 tx=0x7f5288004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61eb38bbac30069 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.707+0000 7f527effd640 1 -- 172.21.15.62:0/3472304137 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288004920 con 0x562b1407e370 2024-09-15T23:27:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.707+0000 7f527effd640 1 -- 172.21.15.62:0/3472304137 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f528801d3e0 con 0x562b1407e370 2024-09-15T23:27:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.707+0000 7f527effd640 1 -- 172.21.15.62:0/3472304137 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528802d380 con 0x562b1407e370 2024-09-15T23:27:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 -- 172.21.15.62:0/3472304137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 msgr2=0x562b1406fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1406fa50 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f528802c720 tx=0x7f5288004c80 comp rx=0 tx=0).stop 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 -- 172.21.15.62:0/3472304137 shutdown_connections 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1407e370 0x562b1406fa50 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b1406f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 --2- 172.21.15.62:0/3472304137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b140735d0 0x562b1408a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 -- 172.21.15.62:0/3472304137 >> 172.21.15.62:0/3472304137 conn(0x562b1403c310 msgr2=0x562b1407afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 -- 172.21.15.62:0/3472304137 shutdown_connections 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 -- 172.21.15.62:0/3472304137 wait complete. 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 -- start start 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.708+0000 7f529d741740 1 Processor -- start 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f529d741740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b140810e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f529d741740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b14081620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f529d741740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b1407e370 0x562b14081b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f529d741740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f52941ad270 con 0x562b140735d0 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f529d741740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f52941ae6e0 con 0x562b1407e370 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f529d741740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f52941ad940 con 0x562b14074b40 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f529d741740 10 client.0 _ll_register_callbacks cb 0x7f529000b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f5298903640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b140810e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f5298903640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b140810e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47908/0 (socket says 172.21.15.62:47908) 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f5298903640 1 -- 172.21.15.62:0/1376491015 learned_addr learned my addr 172.21.15.62:0/1376491015 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f5299104640 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b14081620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.709+0000 7f5298903640 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b1407e370 0x562b14081b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f5298903640 1 -- 172.21.15.62:0/1376491015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 msgr2=0x562b14081620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f5298903640 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b14081620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f5298903640 1 -- 172.21.15.62:0/1376491015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b1407e370 msgr2=0x562b14081b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f5298903640 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b1407e370 0x562b14081b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f5298903640 1 -- 172.21.15.62:0/1376491015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b140f6f20 con 0x562b140735d0 2024-09-15T23:27:01.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f5299104640 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b14081620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f5298903640 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b140810e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f528800dc40 tx=0x7f528800ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7906f5798bd52fc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f527cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f527cff9640 1 -- 172.21.15.62:0/1376491015 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288004920 con 0x562b140735d0 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f527cff9640 1 -- 172.21.15.62:0/1376491015 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f528802d590 con 0x562b140735d0 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.710+0000 7f529d741740 1 -- 172.21.15.62:0/1376491015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f528801d3e0 con 0x562b140735d0 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.711+0000 7f527cff9640 1 -- 172.21.15.62:0/1376491015 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f528800d4f0 con 0x562b140735d0 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.711+0000 7f529d741740 10 client.17964 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.711+0000 7f529d741740 1 -- 172.21.15.62:0/1376491015 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562b140f8100 con 0x562b140735d0 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.711+0000 7f527cff9640 1 -- 172.21.15.62:0/1376491015 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f528801d3e0 con 0x562b140735d0 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.711+0000 7f527cff9640 1 -- 172.21.15.62:0/1376491015 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f528801c020 con 0x562b140735d0 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.711+0000 7f529d741740 10 client.17964 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.711+0000 7f529d741740 10 client.17964 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:27:01.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.711+0000 7f529d741740 20 client.17964 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.712+0000 7f529d741740 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x562b140f8540 0x562b140fa930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.712+0000 7f529d741740 4 client.17964 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.712+0000 7f529d741740 1 -- 172.21.15.62:0/1376491015 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b13fe0d70 con 0x562b140f8540 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.712+0000 7f5299104640 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x562b140f8540 0x562b140fa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.712+0000 7f527cff9640 10 client.17964 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.712+0000 7f5299104640 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x562b140f8540 0x562b140fa930 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=efa7e723b1d3c2b9 server_cookie=c522fa71e247d402 in_seq=0 out_seq=0 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f527cff9640 1 -- 172.21.15.62:0/1376491015 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b13fe0d70 con 0x562b140f8540 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f527cff9640 10 client.17964 handle_command_reply: tid=0 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 client.17964 shutdown 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 -- 172.21.15.62:0/1376491015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 msgr2=0x562b140810e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b140810e0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f528800dc40 tx=0x7f528800ba70 comp rx=0 tx=0).stop 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 -- 172.21.15.62:0/1376491015 shutdown_connections 2024-09-15T23:27:01.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x562b140f8540 0x562b140fa930 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562b1407e370 0x562b14081b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x562b14074b40 0x562b14081620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 --2- 172.21.15.62:0/1376491015 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b140735d0 0x562b140810e0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f527cff9640 0 client.17964 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:27:01.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 -- 172.21.15.62:0/1376491015 >> 172.21.15.62:0/1376491015 conn(0x562b1403c310 msgr2=0x562b14079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 -- 172.21.15.62:0/1376491015 shutdown_connections 2024-09-15T23:27:01.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 1 -- 172.21.15.62:0/1376491015 wait complete. 2024-09-15T23:27:01.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.713+0000 7f529d741740 20 client.17964 trim_cache size 0 max 16384 2024-09-15T23:27:01.719 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 -- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f52800a3240 msgr2=0x7f52800a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f52800a3240 0x7f52800a5700 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f528c0c56e0 tx=0x7f528c0309d0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 -- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 msgr2=0x7f529419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 0x7f529419ce80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f528c005e30 tx=0x7f528c003970 comp rx=0 tx=0).stop 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 -- 172.21.15.62:0/2650250716 shutdown_connections 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f52800a3240 0x7f52800a5700 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52941082e0 0x7f529419ce80 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5294103760 0x7f5294198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 --2- 172.21.15.62:0/2650250716 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5294102db0 0x7f52941985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:01.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 -- 172.21.15.62:0/2650250716 >> 172.21.15.62:0/2650250716 conn(0x7f52940fc7d0 msgr2=0x7f5294105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:01.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 -- 172.21.15.62:0/2650250716 shutdown_connections 2024-09-15T23:27:01.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:01.714+0000 7f529b368640 1 -- 172.21.15.62:0/2650250716 wait complete. 2024-09-15T23:27:02.332 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:02.362 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:27:02.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 -- 172.21.15.62:0/3810530764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01082a0 msgr2=0x7f5bd010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 --2- 172.21.15.62:0/3810530764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01082a0 0x7f5bd010ec40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009cd0 tx=0x7f5bc802a910 comp rx=0 tx=0).stop 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 -- 172.21.15.62:0/3810530764 shutdown_connections 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 --2- 172.21.15.62:0/3810530764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01082a0 0x7f5bd010ec40 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 --2- 172.21.15.62:0/3810530764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bd01036e0 0x7f5bd0107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 --2- 172.21.15.62:0/3810530764 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bd0102d30 0x7f5bd0103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 -- 172.21.15.62:0/3810530764 >> 172.21.15.62:0/3810530764 conn(0x7f5bd00fc7d0 msgr2=0x7f5bd00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 -- 172.21.15.62:0/3810530764 shutdown_connections 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 -- 172.21.15.62:0/3810530764 wait complete. 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 Processor -- start 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.831+0000 7f5bd8794640 1 -- start start 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd8794640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bd0102d30 0x7f5bd0100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd8794640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01036e0 0x7f5bd0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd8794640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bd01082a0 0x7f5bd0101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd8794640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01121b0 con 0x7f5bd01036e0 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd8794640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd0112030 con 0x7f5bd0102d30 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd8794640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd0112330 con 0x7f5bd01082a0 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd5d2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01036e0 0x7f5bd0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd5d2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01036e0 0x7f5bd0100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52730/0 (socket says 172.21.15.62:52730) 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/1382732149 learned_addr learned my addr 172.21.15.62:0/1382732149 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd6530640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bd0102d30 0x7f5bd0100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd6d31640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bd01082a0 0x7f5bd0101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/1382732149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bd01082a0 msgr2=0x7f5bd0101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bd01082a0 0x7f5bd0101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/1382732149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bd0102d30 msgr2=0x7f5bd0100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bd0102d30 0x7f5bd0100990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/1382732149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd01a6b70 con 0x7f5bd01036e0 2024-09-15T23:27:02.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01036e0 0x7f5bd0100ed0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0006fd0 tx=0x7f5bc000b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:02.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bc77fe640 1 -- 172.21.15.62:0/1382732149 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0015020 con 0x7f5bd01036e0 2024-09-15T23:27:02.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bc77fe640 1 -- 172.21.15.62:0/1382732149 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc0011d10 con 0x7f5bd01036e0 2024-09-15T23:27:02.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd8794640 1 -- 172.21.15.62:0/1382732149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd01a6e60 con 0x7f5bd01036e0 2024-09-15T23:27:02.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bc77fe640 1 -- 172.21.15.62:0/1382732149 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc000f2e0 con 0x7f5bd01036e0 2024-09-15T23:27:02.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.832+0000 7f5bd8794640 1 -- 172.21.15.62:0/1382732149 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd01a7310 con 0x7f5bd01036e0 2024-09-15T23:27:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.836+0000 7f5bdab6d740 1 Processor -- start 2024-09-15T23:27:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.836+0000 7f5bc77fe640 1 -- 172.21.15.62:0/1382732149 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f5bc0018020 con 0x7f5bd01036e0 2024-09-15T23:27:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.836+0000 7f5bdab6d740 1 -- start start 2024-09-15T23:27:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.836+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf8e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.836+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf98d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.836+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 0x55b94cf9b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.836+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b94cef3620 con 0x55b94cf8e980 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.836+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b94ce9ae50 con 0x55b94cf8e060 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.836+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b94ceef5e0 con 0x55b94cf99240 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6530640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf98d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6530640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf98d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52740/0 (socket says 172.21.15.62:52740) 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6d31640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf8e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6530640 1 -- 172.21.15.62:0/3519285373 learned_addr learned my addr 172.21.15.62:0/3519285373 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/3519285373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 0x55b94cf9b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bc77fe640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5bbc0a3240 0x7f5bbc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6d31640 1 -- 172.21.15.62:0/3519285373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 msgr2=0x55b94cf9b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6d31640 1 --2- 172.21.15.62:0/3519285373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 0x55b94cf9b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6d31640 1 -- 172.21.15.62:0/3519285373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 msgr2=0x55b94cf98d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6d31640 1 --2- 172.21.15.62:0/3519285373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf98d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6d31640 1 -- 172.21.15.62:0/3519285373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b94ce8bd40 con 0x55b94cf8e060 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bc77fe640 1 -- 172.21.15.62:0/1382732149 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5bc00cf510 con 0x7f5bd01036e0 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.837+0000 7f5bd6d31640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5bbc0a3240 0x7f5bbc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.838+0000 7f5bd6d31640 1 --2- 172.21.15.62:0/3519285373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf8e440 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5bc802ae30 tx=0x7f5bc8009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be3506d31dd503ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.838+0000 7f5bc4ff9640 1 -- 172.21.15.62:0/3519285373 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc804d070 con 0x55b94cf8e060 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.838+0000 7f5bc4ff9640 1 -- 172.21.15.62:0/3519285373 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc8036040 con 0x55b94cf8e060 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.838+0000 7f5bc4ff9640 1 -- 172.21.15.62:0/3519285373 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80046a0 con 0x55b94cf8e060 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 -- 172.21.15.62:0/3519285373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 msgr2=0x55b94cf8e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bd6d31640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5bbc0a3240 0x7f5bbc0a5700 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8046000 tx=0x7f5bc804b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:02.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/3519285373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf8e440 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5bc802ae30 tx=0x7f5bc8009800 comp rx=0 tx=0).stop 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 -- 172.21.15.62:0/3519285373 shutdown_connections 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/3519285373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 0x55b94cf9b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/3519285373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf98d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/3519285373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf8e440 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 -- 172.21.15.62:0/3519285373 >> 172.21.15.62:0/3519285373 conn(0x55b94cf8a5f0 msgr2=0x55b94cf8a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 -- 172.21.15.62:0/3519285373 shutdown_connections 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 -- 172.21.15.62:0/3519285373 wait complete. 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 -- start start 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.839+0000 7f5bdab6d740 1 Processor -- start 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e060 0x55b94cf960f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e980 0x55b94cf96630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 0x55b94cf96b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b94cf9be10 con 0x55b94cf8e060 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b94ce7bf10 con 0x55b94cf99240 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b94cf9bf90 con 0x55b94cf8e980 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bdab6d740 10 client.0 _ll_register_callbacks cb 0x55b94ce9fdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd6530640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e060 0x55b94cf960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd6530640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e060 0x55b94cf960f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52754/0 (socket says 172.21.15.62:52754) 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd6530640 1 -- 172.21.15.62:0/2302634060 learned_addr learned my addr 172.21.15.62:0/2302634060 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e980 0x55b94cf96630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd6d31640 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 0x55b94cf96b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd6530640 1 -- 172.21.15.62:0/2302634060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e980 msgr2=0x55b94cf96630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd6530640 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e980 0x55b94cf96630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd6530640 1 -- 172.21.15.62:0/2302634060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 msgr2=0x55b94cf96b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd6530640 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 0x55b94cf96b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.840+0000 7f5bd6530640 1 -- 172.21.15.62:0/2302634060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b94cc02580 con 0x55b94cf8e060 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.841+0000 7f5bd6530640 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e060 0x55b94cf960f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc011570 tx=0x7f5bcc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd8f151c832c7f5e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.841+0000 7f5bb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.841+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/2302634060 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc00f6d0 con 0x55b94cf8e060 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.841+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/2302634060 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5bcc00f870 con 0x55b94cf8e060 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.841+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2302634060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bc8036040 con 0x55b94cf8e060 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.841+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/2302634060 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5bcc026050 con 0x55b94cf8e060 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.841+0000 7f5bdab6d740 10 client.17976 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.841+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2302634060 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b94cccc960 con 0x55b94cf8e060 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.841+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/2302634060 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc8036040 con 0x55b94cf8e060 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/2302634060 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f5bcc01d400 con 0x55b94cf8e060 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bdab6d740 10 client.17976 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bdab6d740 10 client.17976 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bdab6d740 20 client.17976 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:02.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b94cfa8e10 0x55b94cfab280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bdab6d740 4 client.17976 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:02.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2302634060 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b94cf00b90 con 0x55b94cfa8e10 2024-09-15T23:27:02.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b94cfa8e10 0x55b94cfab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b94cfa8e10 0x55b94cfab280 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ebc0c0e7c883f0d server_cookie=c54c8b2bb35ad7f1 in_seq=0 out_seq=0 2024-09-15T23:27:02.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.842+0000 7f5bb9ffb640 10 client.17976 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:02.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/2302634060 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b94cf00b90 con 0x55b94cfa8e10 2024-09-15T23:27:02.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bb9ffb640 10 client.17976 handle_command_reply: tid=0 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 client.17976 shutdown 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2302634060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e060 msgr2=0x55b94cf960f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e060 0x55b94cf960f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc011570 tx=0x7f5bcc011a30 comp rx=0 tx=0).stop 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2302634060 shutdown_connections 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b94cfa8e10 0x55b94cfab280 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 0x55b94cf96b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e980 0x55b94cf96630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2302634060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e060 0x55b94cf960f0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2302634060 >> 172.21.15.62:0/2302634060 conn(0x55b94cf8a5f0 msgr2=0x55b94cf90320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bb9ffb640 0 client.17976 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2302634060 shutdown_connections 2024-09-15T23:27:02.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.843+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2302634060 wait complete. 2024-09-15T23:27:02.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.844+0000 7f5bdab6d740 20 client.17976 trim_cache size 0 max 16384 2024-09-15T23:27:02.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bdab6d740 1 Processor -- start 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bdab6d740 1 -- start start 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf90210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cfb0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 0x55b94cfb1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01b3850 con 0x55b94cf8e980 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01b1120 con 0x55b94cf8e060 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01aef70 con 0x55b94cf99240 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd5d2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cfb0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd5d2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cfb0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52764/0 (socket says 172.21.15.62:52764) 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd6530640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf90210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/2630845631 learned_addr learned my addr 172.21.15.62:0/2630845631 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:02.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd6d31640 1 --2- 172.21.15.62:0/2630845631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 0x55b94cfb1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/2630845631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 msgr2=0x55b94cfb1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/2630845631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 0x55b94cfb1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/2630845631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 msgr2=0x55b94cf90210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/2630845631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf90210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.862+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/2630845631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b94cf8a420 con 0x55b94cf8e980 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.863+0000 7f5bd6530640 1 --2- 172.21.15.62:0/2630845631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf90210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.863+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/2630845631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cfb0b50 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00ccc90 tx=0x7f5bc00ccb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfd3eb0d84d1b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.863+0000 7f5bbbfff640 1 -- 172.21.15.62:0/2630845631 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc009b580 con 0x55b94cf8e980 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.863+0000 7f5bbbfff640 1 -- 172.21.15.62:0/2630845631 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc009beb0 con 0x55b94cf8e980 2024-09-15T23:27:02.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.863+0000 7f5bbbfff640 1 -- 172.21.15.62:0/2630845631 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc009b110 con 0x55b94cf8e980 2024-09-15T23:27:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2630845631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 msgr2=0x55b94cfb0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2630845631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cfb0b50 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00ccc90 tx=0x7f5bc00ccb60 comp rx=0 tx=0).stop 2024-09-15T23:27:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2630845631 shutdown_connections 2024-09-15T23:27:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2630845631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf99240 0x55b94cfb1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2630845631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cfb0b50 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/2630845631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf8e060 0x55b94cf90210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2630845631 >> 172.21.15.62:0/2630845631 conn(0x55b94cf8a5f0 msgr2=0x55b94cf8fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2630845631 shutdown_connections 2024-09-15T23:27:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 -- 172.21.15.62:0/2630845631 wait complete. 2024-09-15T23:27:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.864+0000 7f5bdab6d740 1 -- start start 2024-09-15T23:27:02.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bdab6d740 1 Processor -- start 2024-09-15T23:27:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e060 0x55b94cf9d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf9d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bdab6d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 0x55b94cf9dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01af5e0 con 0x55b94cf8e980 2024-09-15T23:27:02.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01b0a50 con 0x55b94cf99240 2024-09-15T23:27:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bdab6d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01afcb0 con 0x55b94cf8e060 2024-09-15T23:27:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bdab6d740 10 client.0 _ll_register_callbacks cb 0x7f5bcc009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bd5d2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf9d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bd5d2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf9d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52774/0 (socket says 172.21.15.62:52774) 2024-09-15T23:27:02.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/265254107 learned_addr learned my addr 172.21.15.62:0/265254107 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:02.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bd6530640 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e060 0x55b94cf9d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.865+0000 7f5bd6d31640 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 0x55b94cf9dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/265254107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e060 msgr2=0x55b94cf9d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:02.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e060 0x55b94cf9d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/265254107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 msgr2=0x55b94cf9dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 0x55b94cf9dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bd5d2f640 1 -- 172.21.15.62:0/265254107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b94d012e10 con 0x55b94cf8e980 2024-09-15T23:27:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bd5d2f640 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf9d7b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000bd20 tx=0x7f5bc0099950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c039a5ab107f02a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:02.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/265254107 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00b3070 con 0x55b94cf8e980 2024-09-15T23:27:02.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/265254107 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5bc00cf230 con 0x55b94cf8e980 2024-09-15T23:27:02.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bdab6d740 1 -- 172.21.15.62:0/265254107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bc009beb0 con 0x55b94cf8e980 2024-09-15T23:27:02.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/265254107 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5bc009b2f0 con 0x55b94cf8e980 2024-09-15T23:27:02.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bdab6d740 10 client.17982 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:02.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bdab6d740 1 -- 172.21.15.62:0/265254107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b94d014150 con 0x55b94cf8e980 2024-09-15T23:27:02.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.866+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/265254107 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc009beb0 con 0x55b94cf8e980 2024-09-15T23:27:02.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/265254107 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f5bc0097e20 con 0x55b94cf8e980 2024-09-15T23:27:02.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bdab6d740 10 client.17982 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:02.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bdab6d740 10 client.17982 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:27:02.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bdab6d740 20 client.17982 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:02.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b94d014810 0x55b94d016c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:02.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bdab6d740 4 client.17982 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "724f1b2d", "release": true, "await": true}] 2024-09-15T23:27:02.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bdab6d740 1 -- 172.21.15.62:0/265254107 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "724f1b2d", "release": true, "await": true}) -- 0x55b94cefb0a0 con 0x55b94d014810 2024-09-15T23:27:02.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bd6530640 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b94d014810 0x55b94d016c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:02.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bb9ffb640 10 client.17982 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:02.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.867+0000 7f5bd6530640 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b94d014810 0x55b94d016c30 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daf144cd3ff67cf9 server_cookie=7026985b2b205696 in_seq=0 out_seq=0 2024-09-15T23:27:02.881 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:02.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.907+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/265254107 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55b94cefb0a0 con 0x55b94d014810 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.907+0000 7f5bb9ffb640 10 client.17982 handle_command_reply: tid=0 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bdab6d740 1 client.17982 shutdown 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bdab6d740 1 -- 172.21.15.62:0/265254107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 msgr2=0x55b94cf9d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf9d7b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000bd20 tx=0x7f5bc0099950 comp rx=0 tx=0).stop 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bdab6d740 1 -- 172.21.15.62:0/265254107 shutdown_connections 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b94d014810 0x55b94d016c30 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b94cf99240 0x55b94cf9dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bb9ffb640 0 client.17982 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b94cf8e980 0x55b94cf9d7b0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bdab6d740 1 --2- 172.21.15.62:0/265254107 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b94cf8e060 0x55b94cf9d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.908+0000 7f5bdab6d740 1 -- 172.21.15.62:0/265254107 >> 172.21.15.62:0/265254107 conn(0x55b94cf8a5f0 msgr2=0x55b94cf89750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.909+0000 7f5bdab6d740 1 -- 172.21.15.62:0/265254107 shutdown_connections 2024-09-15T23:27:02.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.909+0000 7f5bdab6d740 1 -- 172.21.15.62:0/265254107 wait complete. 2024-09-15T23:27:02.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.909+0000 7f5bdab6d740 20 client.17982 trim_cache size 0 max 16384 2024-09-15T23:27:02.913 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 35, 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 40, 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "724f1b2d": { 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 40, 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 141.8, 2024-09-15T23:27:02.914 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:02.915 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:27:02.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 -- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5bbc0a3240 msgr2=0x7f5bbc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5bbc0a3240 0x7f5bbc0a5700 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8046000 tx=0x7f5bc804b000 comp rx=0 tx=0).stop 2024-09-15T23:27:02.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 -- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01036e0 msgr2=0x7f5bd0100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:02.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01036e0 0x7f5bd0100ed0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0006fd0 tx=0x7f5bc000b820 comp rx=0 tx=0).stop 2024-09-15T23:27:02.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 -- 172.21.15.62:0/1382732149 shutdown_connections 2024-09-15T23:27:02.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5bbc0a3240 0x7f5bbc0a5700 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5bd01082a0 0x7f5bd0101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bd01036e0 0x7f5bd0100ed0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 --2- 172.21.15.62:0/1382732149 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5bd0102d30 0x7f5bd0100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:02.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.912+0000 7f5bd8794640 1 -- 172.21.15.62:0/1382732149 >> 172.21.15.62:0/1382732149 conn(0x7f5bd00fc7d0 msgr2=0x7f5bd00fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:02.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.913+0000 7f5bd8794640 1 -- 172.21.15.62:0/1382732149 shutdown_connections 2024-09-15T23:27:02.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:02.913+0000 7f5bd8794640 1 -- 172.21.15.62:0/1382732149 wait complete. 2024-09-15T23:27:03.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:03 smithi062 ceph-mon[30949]: pgmap v693: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 7.8 KiB/s wr, 12 op/s 2024-09-15T23:27:03.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:03 smithi067 ceph-mon[35665]: pgmap v693: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 7.8 KiB/s wr, 12 op/s 2024-09-15T23:27:03.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:03 smithi148 ceph-mon[35610]: pgmap v693: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 7.8 KiB/s wr, 12 op/s 2024-09-15T23:27:03.886 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '724f1b2d', total seconds elapsed: 26.2 2024-09-15T23:27:03.886 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 21.85 2024-09-15T23:27:03.886 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:27:04.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.141+0000 7ff804118640 1 -- 172.21.15.62:0/4245482634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc1082e0 msgr2=0x7ff7fc10ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.141+0000 7ff804118640 1 --2- 172.21.15.62:0/4245482634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc1082e0 0x7ff7fc10ec50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009cd0 tx=0x7ff7f402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.141+0000 7ff804118640 1 -- 172.21.15.62:0/4245482634 shutdown_connections 2024-09-15T23:27:04.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.141+0000 7ff804118640 1 --2- 172.21.15.62:0/4245482634 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc1082e0 0x7ff7fc10ec50 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.141+0000 7ff804118640 1 --2- 172.21.15.62:0/4245482634 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7fc103760 0x7ff7fc107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.141+0000 7ff804118640 1 --2- 172.21.15.62:0/4245482634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc102db0 0x7ff7fc103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.141+0000 7ff804118640 1 -- 172.21.15.62:0/4245482634 >> 172.21.15.62:0/4245482634 conn(0x7ff7fc0fc7d0 msgr2=0x7ff7fc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.141+0000 7ff804118640 1 -- 172.21.15.62:0/4245482634 shutdown_connections 2024-09-15T23:27:04.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.141+0000 7ff804118640 1 -- 172.21.15.62:0/4245482634 wait complete. 2024-09-15T23:27:04.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff804118640 1 Processor -- start 2024-09-15T23:27:04.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff804118640 1 -- start start 2024-09-15T23:27:04.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff804118640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7fc102db0 0x7ff7fc100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff804118640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc103760 0x7ff7fc100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff804118640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc1082e0 0x7ff7fc101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff804118640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc112100 con 0x7ff7fc103760 2024-09-15T23:27:04.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff804118640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc111f80 con 0x7ff7fc1082e0 2024-09-15T23:27:04.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff804118640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc112280 con 0x7ff7fc102db0 2024-09-15T23:27:04.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff801eb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7fc102db0 0x7ff7fc100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff801eb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7fc102db0 0x7ff7fc100a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:41184/0 (socket says 172.21.15.62:41184) 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff801eb4640 1 -- 172.21.15.62:0/3204358611 learned_addr learned my addr 172.21.15.62:0/3204358611 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff8026b5640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc1082e0 0x7ff7fc101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff801eb4640 1 -- 172.21.15.62:0/3204358611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc1082e0 msgr2=0x7ff7fc101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc1082e0 0x7ff7fc101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff801eb4640 1 -- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc103760 msgr2=0x7ff7fc100f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.142+0000 7ff8016b3640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc103760 0x7ff7fc100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc103760 0x7ff7fc100f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff801eb4640 1 -- 172.21.15.62:0/3204358611 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7fc1a6a40 con 0x7ff7fc102db0 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff8016b3640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc103760 0x7ff7fc100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7fc102db0 0x7ff7fc100a10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800e9f0 tx=0x7ff7f800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff7f2ffd640 1 -- 172.21.15.62:0/3204358611 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f800cd80 con 0x7ff7fc102db0 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff7f2ffd640 1 -- 172.21.15.62:0/3204358611 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f8010910 con 0x7ff7fc102db0 2024-09-15T23:27:04.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff7f2ffd640 1 -- 172.21.15.62:0/3204358611 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8018990 con 0x7ff7fc102db0 2024-09-15T23:27:04.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.143+0000 7ff804118640 1 -- 172.21.15.62:0/3204358611 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7fc1a6d30 con 0x7ff7fc102db0 2024-09-15T23:27:04.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.144+0000 7ff804118640 1 -- 172.21.15.62:0/3204358611 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7fc1a71c0 con 0x7ff7fc102db0 2024-09-15T23:27:04.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.147+0000 7ff8064f1740 1 Processor -- start 2024-09-15T23:27:04.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.156+0000 7ff7f2ffd640 1 -- 172.21.15.62:0/3204358611 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff7f8010ac0 con 0x7ff7fc102db0 2024-09-15T23:27:04.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.157+0000 7ff7f2ffd640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7e80a32c0 0x7ff7e80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.157+0000 7ff7f2ffd640 1 -- 172.21.15.62:0/3204358611 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff7f80d0090 con 0x7ff7fc102db0 2024-09-15T23:27:04.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.157+0000 7ff8064f1740 1 -- start start 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.158+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a2350 0x55829a4a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.158+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4a1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.158+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4bcb90 0x55829a4bef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.158+0000 7ff8064f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55829a3fe2f0 con 0x55829a4a2350 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.158+0000 7ff8064f1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55829a3a5cf0 con 0x55829a4a0de0 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.158+0000 7ff8064f1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55829a3fa2b0 con 0x55829a4bcb90 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.159+0000 7ff8016b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.159+0000 7ff8016b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4a1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52594/0 (socket says 172.21.15.62:52594) 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.159+0000 7ff8016b3640 1 -- 172.21.15.62:0/1615908882 learned_addr learned my addr 172.21.15.62:0/1615908882 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:04.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.159+0000 7ff801eb4640 1 --2- 172.21.15.62:0/1615908882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a2350 0x55829a4a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.163+0000 7ff8016b3640 1 -- 172.21.15.62:0/1615908882 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4bcb90 msgr2=0x55829a4bef80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:27:04.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.163+0000 7ff8016b3640 1 --2- 172.21.15.62:0/1615908882 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4bcb90 0x55829a4bef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.163+0000 7ff8016b3640 1 -- 172.21.15.62:0/1615908882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a2350 msgr2=0x55829a4a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.163+0000 7ff8016b3640 1 --2- 172.21.15.62:0/1615908882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a2350 0x55829a4a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.163+0000 7ff8016b3640 1 -- 172.21.15.62:0/1615908882 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55829a38a3a0 con 0x55829a4a0de0 2024-09-15T23:27:04.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.164+0000 7ff8026b5640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7e80a32c0 0x7ff7e80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.164+0000 7ff8016b3640 1 --2- 172.21.15.62:0/1615908882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4a1220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009b30 tx=0x7ff7ec00e2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e15e636b722c373c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.170+0000 7ff7e7fff640 1 -- 172.21.15.62:0/1615908882 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec016070 con 0x55829a4a0de0 2024-09-15T23:27:04.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.170+0000 7ff7e7fff640 1 -- 172.21.15.62:0/1615908882 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7ec005500 con 0x55829a4a0de0 2024-09-15T23:27:04.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.170+0000 7ff7e7fff640 1 -- 172.21.15.62:0/1615908882 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec011470 con 0x55829a4a0de0 2024-09-15T23:27:04.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.171+0000 7ff8064f1740 1 -- 172.21.15.62:0/1615908882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 msgr2=0x55829a4a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.171+0000 7ff8064f1740 1 --2- 172.21.15.62:0/1615908882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4a1220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009b30 tx=0x7ff7ec00e2f0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.174+0000 7ff8064f1740 1 -- 172.21.15.62:0/1615908882 shutdown_connections 2024-09-15T23:27:04.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.174+0000 7ff8064f1740 1 --2- 172.21.15.62:0/1615908882 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4bcb90 0x55829a4bef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.174+0000 7ff8064f1740 1 --2- 172.21.15.62:0/1615908882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4a1220 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.174+0000 7ff8064f1740 1 --2- 172.21.15.62:0/1615908882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a2350 0x55829a4a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.174+0000 7ff8064f1740 1 -- 172.21.15.62:0/1615908882 >> 172.21.15.62:0/1615908882 conn(0x55829a461310 msgr2=0x55829a49d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.179+0000 7ff8026b5640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7e80a32c0 0x7ff7e80a5780 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4004af0 tx=0x7ff7f4005d80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.179+0000 7ff8064f1740 1 -- 172.21.15.62:0/1615908882 shutdown_connections 2024-09-15T23:27:04.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.181+0000 7ff8064f1740 1 -- 172.21.15.62:0/1615908882 wait complete. 2024-09-15T23:27:04.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.181+0000 7ff8064f1740 1 -- start start 2024-09-15T23:27:04.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.182+0000 7ff8064f1740 1 Processor -- start 2024-09-15T23:27:04.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.184+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.184+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a2350 0x55829a4a8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.184+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 0x55829a4a9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.184+0000 7ff8064f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55829a4bf890 con 0x55829a4bcb90 2024-09-15T23:27:04.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.184+0000 7ff8064f1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55829a3b9060 con 0x55829a4a0de0 2024-09-15T23:27:04.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.184+0000 7ff8064f1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55829a4bfa10 con 0x55829a4a2350 2024-09-15T23:27:04.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.184+0000 7ff8064f1740 10 client.0 _ll_register_callbacks cb 0x55829a10c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:04.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.185+0000 7ff8016b3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a2350 0x55829a4a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.185+0000 7ff8016b3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a2350 0x55829a4a8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:41190/0 (socket says 172.21.15.62:41190) 2024-09-15T23:27:04.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.185+0000 7ff8016b3640 1 -- 172.21.15.62:0/3481023823 learned_addr learned my addr 172.21.15.62:0/3481023823 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:04.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.185+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.185+0000 7ff801eb4640 1 -- 172.21.15.62:0/3481023823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a2350 msgr2=0x55829a4a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.185+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a2350 0x55829a4a8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.185+0000 7ff801eb4640 1 -- 172.21.15.62:0/3481023823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 msgr2=0x55829a4a9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.185+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 0x55829a4a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.185+0000 7ff801eb4640 1 -- 172.21.15.62:0/3481023823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55829a16f3c0 con 0x55829a4a0de0 2024-09-15T23:27:04.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.187+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4ab5c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800e9c0 tx=0x7ff7f809cb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59331d2bf0b554d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.187+0000 7ff7e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:27:04.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.187+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3481023823 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80b5060 con 0x55829a4a0de0 2024-09-15T23:27:04.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.188+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3481023823 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff7f80a0170 con 0x55829a4a0de0 2024-09-15T23:27:04.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.191+0000 7ff8064f1740 1 -- 172.21.15.62:0/3481023823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7ec005500 con 0x55829a4a0de0 2024-09-15T23:27:04.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.194+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3481023823 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff7f80ba050 con 0x55829a4a0de0 2024-09-15T23:27:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.196+0000 7ff8064f1740 10 client.27554 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.196+0000 7ff8064f1740 1 -- 172.21.15.62:0/3481023823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55829a49cbb0 con 0x55829a4a0de0 2024-09-15T23:27:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.199+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3481023823 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7ec005500 con 0x55829a4a0de0 2024-09-15T23:27:04.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.200+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3481023823 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7ff7f809b720 con 0x55829a4a0de0 2024-09-15T23:27:04.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.204+0000 7ff8064f1740 10 client.27554 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:04.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.204+0000 7ff8064f1740 10 client.27554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:04.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.204+0000 7ff8064f1740 20 client.27554 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:04.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.204+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55829a4c2090 0x55829a4c4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.204+0000 7ff8064f1740 4 client.27554 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:04.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.204+0000 7ff8064f1740 1 -- 172.21.15.62:0/3481023823 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55829a40b860 con 0x55829a4c2090 2024-09-15T23:27:04.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.205+0000 7ff8016b3640 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55829a4c2090 0x55829a4c4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.206+0000 7ff8016b3640 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55829a4c2090 0x55829a4c4550 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4667344429953678 server_cookie=4eaa5a25696096a1 in_seq=0 out_seq=0 2024-09-15T23:27:04.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.206+0000 7ff7e57fa640 10 client.27554 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:04.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.207+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3481023823 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55829a40b860 con 0x55829a4c2090 2024-09-15T23:27:04.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.207+0000 7ff7e57fa640 10 client.27554 handle_command_reply: tid=0 2024-09-15T23:27:04.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.207+0000 7ff8064f1740 1 client.27554 shutdown 2024-09-15T23:27:04.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 -- 172.21.15.62:0/3481023823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 msgr2=0x55829a4ab5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4ab5c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800e9c0 tx=0x7ff7f809cb10 comp rx=0 tx=0).stop 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 -- 172.21.15.62:0/3481023823 shutdown_connections 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55829a4c2090 0x55829a4c4550 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 0x55829a4a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a2350 0x55829a4a8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3481023823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a0de0 0x55829a4ab5c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 -- 172.21.15.62:0/3481023823 >> 172.21.15.62:0/3481023823 conn(0x55829a461310 msgr2=0x55829a4a3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff7e57fa640 0 client.27554 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 -- 172.21.15.62:0/3481023823 shutdown_connections 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.208+0000 7ff8064f1740 1 -- 172.21.15.62:0/3481023823 wait complete. 2024-09-15T23:27:04.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.209+0000 7ff8064f1740 20 client.27554 trim_cache size 0 max 16384 2024-09-15T23:27:04.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff8064f1740 1 Processor -- start 2024-09-15T23:27:04.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff8064f1740 1 -- start start 2024-09-15T23:27:04.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a0de0 0x55829a49d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a2350 0x55829a4a7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 0x55829a4c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff801eb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a0de0 0x55829a49d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff801eb4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a0de0 0x55829a49d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:41196/0 (socket says 172.21.15.62:41196) 2024-09-15T23:27:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff801eb4640 1 -- 172.21.15.62:0/26674934 learned_addr learned my addr 172.21.15.62:0/26674934 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff8064f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc073b90 con 0x55829a4bcb90 2024-09-15T23:27:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff8064f1740 1 -- 172.21.15.62:0/26674934 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc071460 con 0x55829a4a2350 2024-09-15T23:27:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff8064f1740 1 -- 172.21.15.62:0/26674934 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc06f360 con 0x55829a4a0de0 2024-09-15T23:27:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.241+0000 7ff8026b5640 1 --2- 172.21.15.62:0/26674934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 0x55829a4c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.242+0000 7ff8026b5640 1 -- 172.21.15.62:0/26674934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a0de0 msgr2=0x55829a49d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.242+0000 7ff8026b5640 1 --2- 172.21.15.62:0/26674934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a0de0 0x55829a49d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.242+0000 7ff8026b5640 1 -- 172.21.15.62:0/26674934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a2350 msgr2=0x55829a4a7850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:27:04.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.242+0000 7ff8026b5640 1 --2- 172.21.15.62:0/26674934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a2350 0x55829a4a7850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.242+0000 7ff8026b5640 1 -- 172.21.15.62:0/26674934 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55829a4d2b20 con 0x55829a4bcb90 2024-09-15T23:27:04.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.242+0000 7ff8026b5640 1 --2- 172.21.15.62:0/26674934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 0x55829a4c9e20 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009ca0 tx=0x7ff7f403d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2e8250bc217e00 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.245+0000 7ff7e77fe640 1 -- 172.21.15.62:0/26674934 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4031910 con 0x55829a4bcb90 2024-09-15T23:27:04.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.245+0000 7ff7e77fe640 1 -- 172.21.15.62:0/26674934 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f4003f80 con 0x55829a4bcb90 2024-09-15T23:27:04.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.249+0000 7ff7e77fe640 1 -- 172.21.15.62:0/26674934 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4030800 con 0x55829a4bcb90 2024-09-15T23:27:04.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.250+0000 7ff8064f1740 1 -- 172.21.15.62:0/26674934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 msgr2=0x55829a4c9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.250+0000 7ff8064f1740 1 --2- 172.21.15.62:0/26674934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 0x55829a4c9e20 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009ca0 tx=0x7ff7f403d040 comp rx=0 tx=0).stop 2024-09-15T23:27:04.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.251+0000 7ff8064f1740 1 -- 172.21.15.62:0/26674934 shutdown_connections 2024-09-15T23:27:04.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.251+0000 7ff8064f1740 1 --2- 172.21.15.62:0/26674934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4bcb90 0x55829a4c9e20 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.251+0000 7ff8064f1740 1 --2- 172.21.15.62:0/26674934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a2350 0x55829a4a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.251+0000 7ff8064f1740 1 --2- 172.21.15.62:0/26674934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4a0de0 0x55829a49d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.251+0000 7ff8064f1740 1 -- 172.21.15.62:0/26674934 >> 172.21.15.62:0/26674934 conn(0x55829a461310 msgr2=0x55829a4a7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.253+0000 7ff8064f1740 1 -- 172.21.15.62:0/26674934 shutdown_connections 2024-09-15T23:27:04.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.257+0000 7ff8064f1740 1 -- 172.21.15.62:0/26674934 wait complete. 2024-09-15T23:27:04.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.257+0000 7ff8064f1740 1 -- start start 2024-09-15T23:27:04.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.257+0000 7ff8064f1740 1 Processor -- start 2024-09-15T23:27:04.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.258+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a0de0 0x55829a495e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.258+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a2350 0x55829a496350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.258+0000 7ff8064f1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4bcb90 0x55829a49a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.258+0000 7ff8064f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc06f920 con 0x55829a4a0de0 2024-09-15T23:27:04.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.258+0000 7ff8064f1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc070d90 con 0x55829a4a2350 2024-09-15T23:27:04.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.258+0000 7ff8064f1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff7fc06fff0 con 0x55829a4bcb90 2024-09-15T23:27:04.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.258+0000 7ff8064f1740 10 client.0 _ll_register_callbacks cb 0x7ff7f809c9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:04.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.259+0000 7ff8026b5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4bcb90 0x55829a49a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.259+0000 7ff801eb4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a0de0 0x55829a495e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.259+0000 7ff801eb4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a0de0 0x55829a495e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52822/0 (socket says 172.21.15.62:52822) 2024-09-15T23:27:04.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.259+0000 7ff801eb4640 1 -- 172.21.15.62:0/3929460151 learned_addr learned my addr 172.21.15.62:0/3929460151 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:04.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.261+0000 7ff801eb4640 1 -- 172.21.15.62:0/3929460151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4bcb90 msgr2=0x55829a49a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.261+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4bcb90 0x55829a49a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.261+0000 7ff801eb4640 1 -- 172.21.15.62:0/3929460151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a2350 msgr2=0x55829a496350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:04.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.261+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a2350 0x55829a496350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.261+0000 7ff801eb4640 1 -- 172.21.15.62:0/3929460151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55829a49aac0 con 0x55829a4a0de0 2024-09-15T23:27:04.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.262+0000 7ff801eb4640 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a0de0 0x55829a495e10 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff7f809c260 tx=0x7ff7f80bc460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfde49b17610e76 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:04.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.263+0000 7ff7e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:04.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.264+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3929460151 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80b5040 con 0x55829a4a0de0 2024-09-15T23:27:04.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.264+0000 7ff8064f1740 1 -- 172.21.15.62:0/3929460151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7f4003f80 con 0x55829a4a0de0 2024-09-15T23:27:04.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.264+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3929460151 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff7f8018b30 con 0x55829a4a0de0 2024-09-15T23:27:04.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.264+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3929460151 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff7f809c0a0 con 0x55829a4a0de0 2024-09-15T23:27:04.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.265+0000 7ff8064f1740 10 client.17994 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:04.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.265+0000 7ff8064f1740 1 -- 172.21.15.62:0/3929460151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55829a550a40 con 0x55829a4a0de0 2024-09-15T23:27:04.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.267+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3929460151 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f4003f80 con 0x55829a4a0de0 2024-09-15T23:27:04.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.268+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3929460151 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7ff7f8098020 con 0x55829a4a0de0 2024-09-15T23:27:04.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.268+0000 7ff8064f1740 10 client.17994 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:04.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.268+0000 7ff8064f1740 10 client.17994 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:04.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.268+0000 7ff8064f1740 20 client.17994 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:04.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.268+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55829a551140 0x55829a553560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:04.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.268+0000 7ff8064f1740 4 client.17994 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:04.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.268+0000 7ff8064f1740 1 -- 172.21.15.62:0/3929460151 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55829a405d70 con 0x55829a551140 2024-09-15T23:27:04.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.275+0000 7ff8016b3640 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55829a551140 0x55829a553560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:04.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.284+0000 7ff7e57fa640 10 client.17994 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:04.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.284+0000 7ff8016b3640 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55829a551140 0x55829a553560 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17d6c528caf952f5 server_cookie=20a61adfefc4c7e4 in_seq=0 out_seq=0 2024-09-15T23:27:04.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.285+0000 7ff7e57fa640 1 -- 172.21.15.62:0/3929460151 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55829a405d70 con 0x55829a551140 2024-09-15T23:27:04.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.285+0000 7ff7e57fa640 10 client.17994 handle_command_reply: tid=0 2024-09-15T23:27:04.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.285+0000 7ff8064f1740 1 client.17994 shutdown 2024-09-15T23:27:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.286+0000 7ff8064f1740 1 -- 172.21.15.62:0/3929460151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a0de0 msgr2=0x55829a495e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.286+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a0de0 0x55829a495e10 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff7f809c260 tx=0x7ff7f80bc460 comp rx=0 tx=0).stop 2024-09-15T23:27:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.286+0000 7ff8064f1740 1 -- 172.21.15.62:0/3929460151 shutdown_connections 2024-09-15T23:27:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.286+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55829a551140 0x55829a553560 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.286+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55829a4bcb90 0x55829a49a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.286+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55829a4a2350 0x55829a496350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.286+0000 7ff8064f1740 1 --2- 172.21.15.62:0/3929460151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829a4a0de0 0x55829a495e10 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.286+0000 7ff8064f1740 1 -- 172.21.15.62:0/3929460151 >> 172.21.15.62:0/3929460151 conn(0x55829a461310 msgr2=0x55829a4a7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.287+0000 7ff7e57fa640 0 client.17994 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:04.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.289+0000 7ff8064f1740 1 -- 172.21.15.62:0/3929460151 shutdown_connections 2024-09-15T23:27:04.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.289+0000 7ff8064f1740 1 -- 172.21.15.62:0/3929460151 wait complete. 2024-09-15T23:27:04.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.293+0000 7ff8064f1740 20 client.17994 trim_cache size 0 max 16384 2024-09-15T23:27:04.298 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.293+0000 7ff804118640 1 -- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7e80a32c0 msgr2=0x7ff7e80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.293+0000 7ff804118640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7e80a32c0 0x7ff7e80a5780 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4004af0 tx=0x7ff7f4005d80 comp rx=0 tx=0).stop 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 -- 172.21.15.62:0/3204358611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7fc102db0 msgr2=0x7ff7fc100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7fc102db0 0x7ff7fc100a10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800e9f0 tx=0x7ff7f800eeb0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 -- 172.21.15.62:0/3204358611 shutdown_connections 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff7e80a32c0 0x7ff7e80a5780 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7fc1082e0 0x7ff7fc101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff7fc103760 0x7ff7fc100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 --2- 172.21.15.62:0/3204358611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff7fc102db0 0x7ff7fc100a10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 -- 172.21.15.62:0/3204358611 >> 172.21.15.62:0/3204358611 conn(0x7ff7fc0fc7d0 msgr2=0x7ff7fc106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:04.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 -- 172.21.15.62:0/3204358611 shutdown_connections 2024-09-15T23:27:04.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:04.294+0000 7ff804118640 1 -- 172.21.15.62:0/3204358611 wait complete. 2024-09-15T23:27:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 645.0 s, 190.8 tps, lat 2064.570 ms stddev 5614.342 2024-09-15T23:27:05.195 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:05.370 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:05 smithi062 ceph-mon[30949]: pgmap v694: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 24 KiB/s wr, 13 op/s 2024-09-15T23:27:05.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:05 smithi067 ceph-mon[35665]: pgmap v694: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 24 KiB/s wr, 13 op/s 2024-09-15T23:27:05.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:05 smithi148 ceph-mon[35610]: pgmap v694: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 24 KiB/s wr, 13 op/s 2024-09-15T23:27:05.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.813+0000 7fb96a884640 1 -- 172.21.15.62:0/370317743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb964106b80 msgr2=0x7fb964106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.813+0000 7fb96a884640 1 --2- 172.21.15.62:0/370317743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb964106b80 0x7fb964106f60 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fb954009be0 tx=0x7fb95402a6d0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.813+0000 7fb96a884640 1 -- 172.21.15.62:0/370317743 shutdown_connections 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.813+0000 7fb96a884640 1 --2- 172.21.15.62:0/370317743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9641005a0 0x7fb96410ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.813+0000 7fb96a884640 1 --2- 172.21.15.62:0/370317743 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9640ffc00 0x7fb964100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.813+0000 7fb96a884640 1 --2- 172.21.15.62:0/370317743 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb964106b80 0x7fb964106f60 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.813+0000 7fb96a884640 1 -- 172.21.15.62:0/370317743 >> 172.21.15.62:0/370317743 conn(0x7fb9640fba90 msgr2=0x7fb9640fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.814+0000 7fb96a884640 1 -- 172.21.15.62:0/370317743 shutdown_connections 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.814+0000 7fb96a884640 1 -- 172.21.15.62:0/370317743 wait complete. 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.814+0000 7fb96a884640 1 Processor -- start 2024-09-15T23:27:05.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb96a884640 1 -- start start 2024-09-15T23:27:05.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb96a884640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9640ffc00 0x7fb9641983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb96a884640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9641005a0 0x7fb964198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb96a884640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb964106b80 0x7fb96419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb96a884640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb964111f60 con 0x7fb964106b80 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb96a884640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb964111de0 con 0x7fb9640ffc00 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb96a884640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb9641120e0 con 0x7fb9641005a0 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb9637fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9641005a0 0x7fb964198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb9637fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9641005a0 0x7fb964198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:41204/0 (socket says 172.21.15.62:41204) 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb9637fe640 1 -- 172.21.15.62:0/1580603585 learned_addr learned my addr 172.21.15.62:0/1580603585 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb963fff640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9640ffc00 0x7fb9641983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.815+0000 7fb968e21640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb964106b80 0x7fb96419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.816+0000 7fb963fff640 1 -- 172.21.15.62:0/1580603585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9641005a0 msgr2=0x7fb964198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.816+0000 7fb963fff640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9641005a0 0x7fb964198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.816+0000 7fb963fff640 1 -- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb964106b80 msgr2=0x7fb96419ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.816+0000 7fb963fff640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb964106b80 0x7fb96419ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:05.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.816+0000 7fb963fff640 1 -- 172.21.15.62:0/1580603585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96419d390 con 0x7fb9640ffc00 2024-09-15T23:27:05.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.816+0000 7fb963fff640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9640ffc00 0x7fb9641983e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb95402abf0 tx=0x7fb954030c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:05.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.816+0000 7fb9617fa640 1 -- 172.21.15.62:0/1580603585 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954041020 con 0x7fb9640ffc00 2024-09-15T23:27:05.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.819+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96419d620 con 0x7fb9640ffc00 2024-09-15T23:27:05.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.819+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9641a4f60 con 0x7fb9640ffc00 2024-09-15T23:27:05.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.821+0000 7fb9617fa640 1 -- 172.21.15.62:0/1580603585 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb954003e50 con 0x7fb9640ffc00 2024-09-15T23:27:05.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.821+0000 7fb9617fa640 1 -- 172.21.15.62:0/1580603585 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954036040 con 0x7fb9640ffc00 2024-09-15T23:27:05.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.825+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb944005290 con 0x7fb9640ffc00 2024-09-15T23:27:05.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.827+0000 7fb9617fa640 1 -- 172.21.15.62:0/1580603585 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb954044020 con 0x7fb9640ffc00 2024-09-15T23:27:05.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.828+0000 7fb9617fa640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9500a3340 0x7fb9500a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:05.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.829+0000 7fb9617fa640 1 -- 172.21.15.62:0/1580603585 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb9540f0170 con 0x7fb9640ffc00 2024-09-15T23:27:05.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.829+0000 7fb9637fe640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9500a3340 0x7fb9500a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:05.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.830+0000 7fb9637fe640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9500a3340 0x7fb9500a5800 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb9640fe060 tx=0x7fb94c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:05.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:05.840+0000 7fb9617fa640 1 -- 172.21.15.62:0/1580603585 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb95404b070 con 0x7fb9640ffc00 2024-09-15T23:27:06.196 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:27:06.196 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:27:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.240+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb944005580 con 0x7fb9640ffc00 2024-09-15T23:27:06.247 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:27:06.247 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:27:06.248 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:27:06.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.245+0000 7fb9617fa640 1 -- 172.21.15.62:0/1580603585 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7fb9540bcab0 con 0x7fb9640ffc00 2024-09-15T23:27:06.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9500a3340 msgr2=0x7fb9500a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9500a3340 0x7fb9500a5800 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb9640fe060 tx=0x7fb94c005f90 comp rx=0 tx=0).stop 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9640ffc00 msgr2=0x7fb9641983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9640ffc00 0x7fb9641983e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb95402abf0 tx=0x7fb954030c50 comp rx=0 tx=0).stop 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 shutdown_connections 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb9500a3340 0x7fb9500a5800 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb964106b80 0x7fb96419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb9641005a0 0x7fb964198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 --2- 172.21.15.62:0/1580603585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9640ffc00 0x7fb9641983e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 >> 172.21.15.62:0/1580603585 conn(0x7fb9640fba90 msgr2=0x7fb9640fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 shutdown_connections 2024-09-15T23:27:06.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:06.252+0000 7fb96a884640 1 -- 172.21.15.62:0/1580603585 wait complete. 2024-09-15T23:27:07.090 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:07 smithi062 ceph-mon[30949]: pgmap v695: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 726 KiB/s wr, 57 op/s 2024-09-15T23:27:07.097 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:07 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1580603585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:07.116 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0' 2024-09-15T23:27:07.117 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 141.85 --await-for 45.0 2024-09-15T23:27:07.246 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:07.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:07 smithi067 ceph-mon[35665]: pgmap v695: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 726 KiB/s wr, 57 op/s 2024-09-15T23:27:07.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:07 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1580603585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:07.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:07 smithi148 ceph-mon[35610]: pgmap v695: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 726 KiB/s wr, 57 op/s 2024-09-15T23:27:07.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:07 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1580603585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:07.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.870+0000 7fa96d0be640 1 -- 172.21.15.62:0/1136391482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968101df0 msgr2=0x7fa9681021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:07.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.870+0000 7fa96d0be640 1 --2- 172.21.15.62:0/1136391482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968101df0 0x7fa9681021d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa958009900 tx=0x7fa95802a410 comp rx=0 tx=0).stop 2024-09-15T23:27:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.871+0000 7fa96d0be640 1 -- 172.21.15.62:0/1136391482 shutdown_connections 2024-09-15T23:27:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.871+0000 7fa96d0be640 1 --2- 172.21.15.62:0/1136391482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa968106f20 0x7fa96810eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.871+0000 7fa96d0be640 1 --2- 172.21.15.62:0/1136391482 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9681027a0 0x7fa9681067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.871+0000 7fa96d0be640 1 --2- 172.21.15.62:0/1136391482 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa968101df0 0x7fa9681021d0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.871+0000 7fa96d0be640 1 -- 172.21.15.62:0/1136391482 >> 172.21.15.62:0/1136391482 conn(0x7fa9680fd8f0 msgr2=0x7fa9680ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.871+0000 7fa96d0be640 1 -- 172.21.15.62:0/1136391482 shutdown_connections 2024-09-15T23:27:07.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.871+0000 7fa96d0be640 1 -- 172.21.15.62:0/1136391482 wait complete. 2024-09-15T23:27:07.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.872+0000 7fa96d0be640 1 Processor -- start 2024-09-15T23:27:07.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.872+0000 7fa96d0be640 1 -- start start 2024-09-15T23:27:07.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96d0be640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa968101df0 0x7fa9681984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96d0be640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9681027a0 0x7fa9681989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96d0be640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa968106f20 0x7fa96819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96d0be640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa968111fd0 con 0x7fa9681027a0 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96d0be640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa968111e50 con 0x7fa968106f20 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9681027a0 0x7fa9681989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9681027a0 0x7fa9681989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52850/0 (socket says 172.21.15.62:52850) 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96659c640 1 -- 172.21.15.62:0/3402994412 learned_addr learned my addr 172.21.15.62:0/3402994412 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96759e640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa968106f20 0x7fa96819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa966d9d640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa968101df0 0x7fa9681984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96d0be640 1 -- 172.21.15.62:0/3402994412 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa968112150 con 0x7fa968101df0 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96759e640 1 -- 172.21.15.62:0/3402994412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa968101df0 msgr2=0x7fa9681984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96759e640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa968101df0 0x7fa9681984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96759e640 1 -- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9681027a0 msgr2=0x7fa9681989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:07.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96759e640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9681027a0 0x7fa9681989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96759e640 1 -- 172.21.15.62:0/3402994412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96819d460 con 0x7fa968106f20 2024-09-15T23:27:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.873+0000 7fa96659c640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9681027a0 0x7fa9681989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.874+0000 7fa96759e640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa968106f20 0x7fa96819cd80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa96000ec50 tx=0x7fa96000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.874+0000 7fa94ffff640 1 -- 172.21.15.62:0/3402994412 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960003b70 con 0x7fa968106f20 2024-09-15T23:27:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.874+0000 7fa94ffff640 1 -- 172.21.15.62:0/3402994412 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa960003d10 con 0x7fa968106f20 2024-09-15T23:27:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.874+0000 7fa94ffff640 1 -- 172.21.15.62:0/3402994412 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960010890 con 0x7fa968106f20 2024-09-15T23:27:07.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.874+0000 7fa96d0be640 1 -- 172.21.15.62:0/3402994412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96819d750 con 0x7fa968106f20 2024-09-15T23:27:07.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.874+0000 7fa96d0be640 1 -- 172.21.15.62:0/3402994412 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9681a5030 con 0x7fa968106f20 2024-09-15T23:27:07.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.878+0000 7fa96f497740 1 Processor -- start 2024-09-15T23:27:07.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.879+0000 7fa94ffff640 1 -- 172.21.15.62:0/3402994412 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa960003fc0 con 0x7fa968106f20 2024-09-15T23:27:07.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.879+0000 7fa96f497740 1 -- start start 2024-09-15T23:27:07.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.879+0000 7fa96f497740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 0x560ee401b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.879+0000 7fa96f497740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401b6c0 0x560ee40156f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.879+0000 7fa96f497740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40161e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.879+0000 7fa96f497740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560ee3f71280 con 0x560ee401ada0 2024-09-15T23:27:07.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.879+0000 7fa96f497740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560ee3f18b10 con 0x560ee4015d60 2024-09-15T23:27:07.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.879+0000 7fa96f497740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560ee3f6d240 con 0x560ee401b6c0 2024-09-15T23:27:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa94ffff640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9500a3340 0x7fa9500a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa96659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 0x560ee401b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa96659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 0x560ee401b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52862/0 (socket says 172.21.15.62:52862) 2024-09-15T23:27:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa96659c640 1 -- 172.21.15.62:0/3265574784 learned_addr learned my addr 172.21.15.62:0/3265574784 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa96659c640 1 --2- 172.21.15.62:0/3265574784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa966d9d640 1 --2- 172.21.15.62:0/3265574784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401b6c0 0x560ee40156f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa96659c640 1 -- 172.21.15.62:0/3265574784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401b6c0 msgr2=0x560ee40156f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa96659c640 1 --2- 172.21.15.62:0/3265574784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401b6c0 0x560ee40156f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa96659c640 1 -- 172.21.15.62:0/3265574784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 msgr2=0x560ee401b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:07.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa96659c640 1 --2- 172.21.15.62:0/3265574784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 0x560ee401b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa94ffff640 1 -- 172.21.15.62:0/3402994412 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa9600d01e0 con 0x7fa968106f20 2024-09-15T23:27:07.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.880+0000 7fa96659c640 1 -- 172.21.15.62:0/3265574784 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ee3f251f0 con 0x560ee4015d60 2024-09-15T23:27:07.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.881+0000 7fa96659c640 1 --2- 172.21.15.62:0/3265574784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40161e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa95c011cf0 tx=0x7fa95c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=947ed170d1d7c187 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:07.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.881+0000 7fa96759e640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9500a3340 0x7fa9500a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.881+0000 7fa94d7fa640 1 -- 172.21.15.62:0/3265574784 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00fc50 con 0x560ee4015d60 2024-09-15T23:27:07.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.881+0000 7fa94d7fa640 1 -- 172.21.15.62:0/3265574784 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa95c00fdf0 con 0x560ee4015d60 2024-09-15T23:27:07.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.881+0000 7fa94d7fa640 1 -- 172.21.15.62:0/3265574784 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00e5e0 con 0x560ee4015d60 2024-09-15T23:27:07.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.882+0000 7fa96759e640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9500a3340 0x7fa9500a5800 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa960098880 tx=0x7fa96000ce70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:07.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.882+0000 7fa96f497740 1 -- 172.21.15.62:0/3265574784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 msgr2=0x560ee40161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.882+0000 7fa96f497740 1 --2- 172.21.15.62:0/3265574784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40161e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa95c011cf0 tx=0x7fa95c005f90 comp rx=0 tx=0).stop 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.882+0000 7fa96f497740 1 -- 172.21.15.62:0/3265574784 shutdown_connections 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.882+0000 7fa96f497740 1 --2- 172.21.15.62:0/3265574784 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40161e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.882+0000 7fa96f497740 1 --2- 172.21.15.62:0/3265574784 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401b6c0 0x560ee40156f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.882+0000 7fa96f497740 1 --2- 172.21.15.62:0/3265574784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 0x560ee401b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.882+0000 7fa96f497740 1 -- 172.21.15.62:0/3265574784 >> 172.21.15.62:0/3265574784 conn(0x560ee4007ba0 msgr2=0x560ee4007f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.883+0000 7fa96f497740 1 -- 172.21.15.62:0/3265574784 shutdown_connections 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.884+0000 7fa96f497740 1 -- 172.21.15.62:0/3265574784 wait complete. 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.884+0000 7fa96f497740 1 -- start start 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.884+0000 7fa96f497740 1 Processor -- start 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.884+0000 7fa96f497740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee4015d60 0x560ee401ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96f497740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96f497740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee401b6c0 0x560ee401f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96f497740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560ee40220e0 con 0x560ee4015d60 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96f497740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560ee3f2bf10 con 0x560ee401b6c0 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96f497740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560ee4022260 con 0x560ee401ada0 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96f497740 10 client.0 _ll_register_callbacks cb 0x560ee3ef4a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa966d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee4015d60 0x560ee401ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa966d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:41232/0 (socket says 172.21.15.62:41232) 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee4015d60 0x560ee401ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52864/0 (socket says 172.21.15.62:52864) 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa966d9d640 1 -- 172.21.15.62:0/1366271001 learned_addr learned my addr 172.21.15.62:0/1366271001 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96659c640 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee401b6c0 0x560ee401f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96659c640 1 -- 172.21.15.62:0/1366271001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 msgr2=0x560ee401f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96659c640 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96659c640 1 -- 172.21.15.62:0/1366271001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee401b6c0 msgr2=0x560ee401f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:07.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96659c640 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee401b6c0 0x560ee401f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa96659c640 1 -- 172.21.15.62:0/1366271001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ee3ce26a0 con 0x560ee4015d60 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.885+0000 7fa966d9d640 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.886+0000 7fa96659c640 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee4015d60 0x560ee401ec60 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009810 tx=0x7fa95c01ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571b227190796941 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.886+0000 7fa94aff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.886+0000 7fa94aff5640 1 -- 172.21.15.62:0/1366271001 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c00fc20 con 0x560ee4015d60 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.886+0000 7fa96f497740 1 -- 172.21.15.62:0/1366271001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa95c00fdf0 con 0x560ee4015d60 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.886+0000 7fa94aff5640 1 -- 172.21.15.62:0/1366271001 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa95c01e5b0 con 0x560ee4015d60 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.886+0000 7fa94aff5640 1 -- 172.21.15.62:0/1366271001 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa95c01e7b0 con 0x560ee4015d60 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.886+0000 7fa96f497740 10 client.18012 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.886+0000 7fa96f497740 1 -- 172.21.15.62:0/1366271001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560ee402de50 con 0x560ee4015d60 2024-09-15T23:27:07.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.887+0000 7fa94aff5640 1 -- 172.21.15.62:0/1366271001 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa95c01e950 con 0x560ee4015d60 2024-09-15T23:27:07.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.887+0000 7fa94aff5640 1 -- 172.21.15.62:0/1366271001 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa95c02b3d0 con 0x560ee4015d60 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.887+0000 7fa96f497740 10 client.18012 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.887+0000 7fa96f497740 10 client.18012 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.887+0000 7fa96f497740 20 client.18012 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.887+0000 7fa96f497740 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560ee402f540 0x560ee40319c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.887+0000 7fa96f497740 4 client.18012 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.887+0000 7fa96f497740 1 -- 172.21.15.62:0/1366271001 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa95c00fdf0 con 0x560ee402f540 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.887+0000 7fa966d9d640 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560ee402f540 0x560ee40319c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.888+0000 7fa94aff5640 10 client.18012 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.888+0000 7fa966d9d640 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560ee402f540 0x560ee40319c0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b55733828a93726c server_cookie=86028c018a6391f0 in_seq=0 out_seq=0 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.888+0000 7fa94aff5640 1 -- 172.21.15.62:0/1366271001 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fa95c00fdf0 con 0x560ee402f540 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.888+0000 7fa94aff5640 10 client.18012 handle_command_reply: tid=0 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.888+0000 7fa96f497740 1 client.18012 shutdown 2024-09-15T23:27:07.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 -- 172.21.15.62:0/1366271001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee4015d60 msgr2=0x560ee401ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee4015d60 0x560ee401ec60 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009810 tx=0x7fa95c01ed70 comp rx=0 tx=0).stop 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 -- 172.21.15.62:0/1366271001 shutdown_connections 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560ee402f540 0x560ee40319c0 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee401b6c0 0x560ee401f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 --2- 172.21.15.62:0/1366271001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee4015d60 0x560ee401ec60 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa94aff5640 0 client.18012 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 -- 172.21.15.62:0/1366271001 >> 172.21.15.62:0/1366271001 conn(0x560ee4007ba0 msgr2=0x560ee4007270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 -- 172.21.15.62:0/1366271001 shutdown_connections 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 1 -- 172.21.15.62:0/1366271001 wait complete. 2024-09-15T23:27:07.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.889+0000 7fa96f497740 20 client.18012 trim_cache size 0 max 16384 2024-09-15T23:27:07.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.923+0000 7fa96f497740 1 Processor -- start 2024-09-15T23:27:07.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.924+0000 7fa96f497740 1 -- start start 2024-09-15T23:27:07.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.924+0000 7fa96f497740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40156f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.924+0000 7fa96f497740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.925+0000 7fa96659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40156f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.925+0000 7fa96659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40156f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52696/0 (socket says 172.21.15.62:52696) 2024-09-15T23:27:07.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.925+0000 7fa96659c640 1 -- 172.21.15.62:0/2396350527 learned_addr learned my addr 172.21.15.62:0/2396350527 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:07.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.925+0000 7fa966d9d640 1 --2- 172.21.15.62:0/2396350527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.925+0000 7fa96f497740 1 --2- 172.21.15.62:0/2396350527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401b6c0 0x560ee4020ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.925+0000 7fa96f497740 1 -- 172.21.15.62:0/2396350527 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa9681af2f0 con 0x560ee401b6c0 2024-09-15T23:27:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.925+0000 7fa96f497740 1 -- 172.21.15.62:0/2396350527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9681acbc0 con 0x560ee4015d60 2024-09-15T23:27:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.925+0000 7fa96f497740 1 -- 172.21.15.62:0/2396350527 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa9681aaa10 con 0x560ee401ada0 2024-09-15T23:27:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.928+0000 7fa966d9d640 1 -- 172.21.15.62:0/2396350527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 msgr2=0x560ee40156f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.928+0000 7fa966d9d640 1 --2- 172.21.15.62:0/2396350527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40156f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.928+0000 7fa966d9d640 1 -- 172.21.15.62:0/2396350527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401b6c0 msgr2=0x560ee4020ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.928+0000 7fa966d9d640 1 --2- 172.21.15.62:0/2396350527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401b6c0 0x560ee4020ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.928+0000 7fa966d9d640 1 -- 172.21.15.62:0/2396350527 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ee4021530 con 0x560ee401ada0 2024-09-15T23:27:07.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.929+0000 7fa966d9d640 1 --2- 172.21.15.62:0/2396350527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401a630 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa958005c40 tx=0x7fa958005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80dc935f9f2ec7d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:07.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.933+0000 7fa94cff9640 1 -- 172.21.15.62:0/2396350527 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa958035040 con 0x560ee401ada0 2024-09-15T23:27:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.934+0000 7fa94cff9640 1 -- 172.21.15.62:0/2396350527 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa95804b070 con 0x560ee401ada0 2024-09-15T23:27:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.934+0000 7fa94cff9640 1 -- 172.21.15.62:0/2396350527 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9580315b0 con 0x560ee401ada0 2024-09-15T23:27:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.935+0000 7fa96f497740 1 -- 172.21.15.62:0/2396350527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 msgr2=0x560ee401a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:07.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.935+0000 7fa96f497740 1 --2- 172.21.15.62:0/2396350527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401a630 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa958005c40 tx=0x7fa958005b50 comp rx=0 tx=0).stop 2024-09-15T23:27:07.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.936+0000 7fa96f497740 1 -- 172.21.15.62:0/2396350527 shutdown_connections 2024-09-15T23:27:07.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.936+0000 7fa96f497740 1 --2- 172.21.15.62:0/2396350527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401b6c0 0x560ee4020ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.936+0000 7fa96f497740 1 --2- 172.21.15.62:0/2396350527 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401ada0 0x560ee401a630 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.936+0000 7fa96f497740 1 --2- 172.21.15.62:0/2396350527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40156f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.936+0000 7fa96f497740 1 -- 172.21.15.62:0/2396350527 >> 172.21.15.62:0/2396350527 conn(0x560ee4007ba0 msgr2=0x560ee40203b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:07.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.950+0000 7fa96f497740 1 -- 172.21.15.62:0/2396350527 shutdown_connections 2024-09-15T23:27:07.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.950+0000 7fa96f497740 1 -- 172.21.15.62:0/2396350527 wait complete. 2024-09-15T23:27:07.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.951+0000 7fa96f497740 1 -- start start 2024-09-15T23:27:07.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.951+0000 7fa96f497740 1 Processor -- start 2024-09-15T23:27:07.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.952+0000 7fa96f497740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40274f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.952+0000 7fa96f497740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 0x560ee4027a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.952+0000 7fa96659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40274f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.952+0000 7fa96659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40274f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52708/0 (socket says 172.21.15.62:52708) 2024-09-15T23:27:07.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.952+0000 7fa96659c640 1 -- 172.21.15.62:0/1528992979 learned_addr learned my addr 172.21.15.62:0/1528992979 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.953+0000 7fa96f497740 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401b6c0 0x560ee40244c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.953+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa9681ab080 con 0x560ee401ada0 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.953+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9681ac4f0 con 0x560ee4015d60 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.953+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa9681ab750 con 0x560ee401b6c0 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.953+0000 7fa96659c640 1 -- 172.21.15.62:0/1528992979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401b6c0 msgr2=0x560ee40244c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.953+0000 7fa96659c640 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401b6c0 0x560ee40244c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.953+0000 7fa96659c640 1 -- 172.21.15.62:0/1528992979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 msgr2=0x560ee4027a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.953+0000 7fa96659c640 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 0x560ee4027a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.953+0000 7fa96659c640 1 -- 172.21.15.62:0/1528992979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ee4028120 con 0x560ee4015d60 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.954+0000 7fa96f497740 10 client.0 _ll_register_callbacks cb 0x7fa95c011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:07.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.954+0000 7fa96659c640 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40274f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa95c013010 tx=0x7fa95c011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea9a450141eb3811 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:07.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.954+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa95804b070 con 0x560ee4015d60 2024-09-15T23:27:07.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.959+0000 7fa94aff5640 10 client.27578 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:27:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.960+0000 7fa94aff5640 1 -- 172.21.15.62:0/1528992979 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c01e5b0 con 0x560ee4015d60 2024-09-15T23:27:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.960+0000 7fa94aff5640 1 -- 172.21.15.62:0/1528992979 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa95c02ba40 con 0x560ee4015d60 2024-09-15T23:27:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.960+0000 7fa94aff5640 1 -- 172.21.15.62:0/1528992979 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa95c01e950 con 0x560ee4015d60 2024-09-15T23:27:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.960+0000 7fa96f497740 10 client.27578 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.960+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({fsmap=35}) -- 0x560ee409a650 con 0x560ee4015d60 2024-09-15T23:27:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.961+0000 7fa94aff5640 1 -- 172.21.15.62:0/1528992979 <== mon.2 v2:172.21.15.67:3300/0 4 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa95c02b3d0 con 0x560ee4015d60 2024-09-15T23:27:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.961+0000 7fa96f497740 10 client.27578 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:07.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.961+0000 7fa96f497740 10 client.27578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.961+0000 7fa96f497740 20 client.27578 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.961+0000 7fa96f497740 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560ee409ac70 0x560ee409d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.961+0000 7fa96f497740 4 client.27578 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:27:07.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.961+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fa95804b070 con 0x560ee409ac70 2024-09-15T23:27:07.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.964+0000 7fa966d9d640 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560ee409ac70 0x560ee409d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:07.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.969+0000 7fa966d9d640 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560ee409ac70 0x560ee409d110 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63f8d8bebcd9d15b server_cookie=eb1141d547b0c9d3 in_seq=0 out_seq=0 2024-09-15T23:27:07.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.969+0000 7fa94aff5640 10 client.27578 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:07.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.971+0000 7fa94aff5640 1 -- 172.21.15.62:0/1528992979 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fa95804b070 con 0x560ee409ac70 2024-09-15T23:27:07.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.971+0000 7fa94aff5640 10 client.27578 handle_command_reply: tid=0 2024-09-15T23:27:07.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.971+0000 7fa96f497740 1 client.27578 shutdown 2024-09-15T23:27:07.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.974+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 msgr2=0x560ee40274f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:07.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.974+0000 7fa96f497740 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40274f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa95c013010 tx=0x7fa95c011c50 comp rx=0 tx=0).stop 2024-09-15T23:27:07.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.974+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 shutdown_connections 2024-09-15T23:27:07.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.974+0000 7fa96f497740 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560ee409ac70 0x560ee409d110 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.975+0000 7fa96f497740 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560ee401b6c0 0x560ee40244c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.975+0000 7fa94aff5640 0 client.27578 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:07.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.975+0000 7fa96f497740 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560ee401ada0 0x560ee4027a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.975+0000 7fa96f497740 1 --2- 172.21.15.62:0/1528992979 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560ee4015d60 0x560ee40274f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.975+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 >> 172.21.15.62:0/1528992979 conn(0x560ee4007ba0 msgr2=0x560ee4007270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:07.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.979+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 shutdown_connections 2024-09-15T23:27:07.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.979+0000 7fa96f497740 1 -- 172.21.15.62:0/1528992979 wait complete. 2024-09-15T23:27:07.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.979+0000 7fa96f497740 20 client.27578 trim_cache size 0 max 16384 2024-09-15T23:27:07.985 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:27:07.985 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:27:07.985 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "6983d6fe-08b9-4992-845c-f133ac2460b2", 2024-09-15T23:27:07.985 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:27:07.985 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:27:07.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.988+0000 7fa96d0be640 1 -- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9500a3340 msgr2=0x7fa9500a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:07.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.988+0000 7fa96d0be640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9500a3340 0x7fa9500a5800 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fa960098880 tx=0x7fa96000ce70 comp rx=0 tx=0).stop 2024-09-15T23:27:07.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.988+0000 7fa96d0be640 1 -- 172.21.15.62:0/3402994412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa968106f20 msgr2=0x7fa96819cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:07.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.988+0000 7fa96d0be640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa968106f20 0x7fa96819cd80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa96000ec50 tx=0x7fa96000c6a0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.993 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:07.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.993+0000 7fa96d0be640 1 -- 172.21.15.62:0/3402994412 shutdown_connections 2024-09-15T23:27:07.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.993+0000 7fa96d0be640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9500a3340 0x7fa9500a5800 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.993+0000 7fa96d0be640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa968106f20 0x7fa96819cd80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.993+0000 7fa96d0be640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9681027a0 0x7fa9681989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.993+0000 7fa96d0be640 1 --2- 172.21.15.62:0/3402994412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa968101df0 0x7fa9681984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:07.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.993+0000 7fa96d0be640 1 -- 172.21.15.62:0/3402994412 >> 172.21.15.62:0/3402994412 conn(0x7fa9680fd8f0 msgr2=0x7fa9680fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:07.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.993+0000 7fa96d0be640 1 -- 172.21.15.62:0/3402994412 shutdown_connections 2024-09-15T23:27:07.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:07.993+0000 7fa96d0be640 1 -- 172.21.15.62:0/3402994412 wait complete. 2024-09-15T23:27:08.276 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:27:08 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: 2024-09-15T23:27:08.009+0000 7ff293927640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000f [...2,head] /volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans/ auth{2=1} v3362 ap=2 f(v9 m2024-09-15T23:26:13.788638+0000 3=3+0) n(v30 rc2024-09-15T23:26:13.793638+0000 b737280 4=3+1) (inest sync->mix g=2) | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55598f2f4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v282>,<0x10000000005/data v121>,<0x100000001fb/postgres v25>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v43>,<0x10000000001/sv_1 v57>,<0x10000000000/qa v44>,<0x1/volumes v43>]//[]","memoryvalue":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v3362>,<0x10000000005/data v806>,<0x100000001fb/postgres v558>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v595>,<0x10000000001/sv_1 v584>,<0x10000000000/qa v589>,<0x1/volumes v545>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-09-15T23:24:19.529348+0000 2=2+0)","ondisk_value.rstat":"n(v0 rc2024-09-15T23:26:13.793638+0000 b737280 4=3+1)","memory_value.dirstat":"f(v9 m2024-09-15T23:26:13.788638+0000 3=3+0)","memory_value.rstat":"n(v30 rc2024-09-15T23:26:13.793638+0000 b737280 4=3+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-09-15T23:27:08.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:08 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:27:08.385 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:08 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:27:08.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:08 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:27:08.829 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:27:09.098 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:27:09.098 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.098 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:27:09.098 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:27:09.098 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.098 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:27:09.098 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: pgmap v696: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-09-15T23:27:09.098 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:27:09.099 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.099 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.099 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:09 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.325+0000 7f7b6c042640 1 -- 172.21.15.62:0/2599117802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b640737d0 msgr2=0x7f7b64112540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.325+0000 7f7b6c042640 1 --2- 172.21.15.62:0/2599117802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b640737d0 0x7f7b64112540 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009ca0 tx=0x7f7b5c02a8d0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.325+0000 7f7b6c042640 1 -- 172.21.15.62:0/2599117802 shutdown_connections 2024-09-15T23:27:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.325+0000 7f7b6c042640 1 --2- 172.21.15.62:0/2599117802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b640737d0 0x7f7b64112540 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.325+0000 7f7b6c042640 1 --2- 172.21.15.62:0/2599117802 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64072d20 0x7f7b64073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.325+0000 7f7b6c042640 1 --2- 172.21.15.62:0/2599117802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b64109930 0x7f7b64109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.325+0000 7f7b6c042640 1 -- 172.21.15.62:0/2599117802 >> 172.21.15.62:0/2599117802 conn(0x7f7b6406ece0 msgr2=0x7f7b64071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.326+0000 7f7b6c042640 1 -- 172.21.15.62:0/2599117802 shutdown_connections 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.326+0000 7f7b6c042640 1 -- 172.21.15.62:0/2599117802 wait complete. 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.327+0000 7f7b6c042640 1 Processor -- start 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.327+0000 7f7b6c042640 1 -- start start 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b6c042640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b64072d20 0x7f7b64119620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b6c042640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b640737d0 0x7f7b64119b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b6c042640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64109930 0x7f7b64120bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b6c042640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6410b600 con 0x7f7b640737d0 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b6c042640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6410b480 con 0x7f7b64072d20 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b6c042640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6410b780 con 0x7f7b64109930 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b6a5df640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64109930 0x7f7b64120bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b69dde640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b64072d20 0x7f7b64119620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b69dde640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b64072d20 0x7f7b64119620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52714/0 (socket says 172.21.15.62:52714) 2024-09-15T23:27:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.328+0000 7f7b69dde640 1 -- 172.21.15.62:0/1914262713 learned_addr learned my addr 172.21.15.62:0/1914262713 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.329+0000 7f7b69dde640 1 -- 172.21.15.62:0/1914262713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64109930 msgr2=0x7f7b64120bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.329+0000 7f7b69dde640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64109930 0x7f7b64120bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.329+0000 7f7b69dde640 1 -- 172.21.15.62:0/1914262713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b640737d0 msgr2=0x7f7b64119b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.329+0000 7f7b69dde640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b640737d0 0x7f7b64119b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.329+0000 7f7b69dde640 1 -- 172.21.15.62:0/1914262713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b64121240 con 0x7f7b64072d20 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.329+0000 7f7b69dde640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b64072d20 0x7f7b64119620 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000d950 tx=0x7f7b6000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.331+0000 7f7b5affd640 1 -- 172.21.15.62:0/1914262713 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60017070 con 0x7f7b64072d20 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.331+0000 7f7b5affd640 1 -- 172.21.15.62:0/1914262713 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b60009c30 con 0x7f7b64072d20 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.331+0000 7f7b5affd640 1 -- 172.21.15.62:0/1914262713 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6000f620 con 0x7f7b64072d20 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.332+0000 7f7b6c042640 1 -- 172.21.15.62:0/1914262713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b64121530 con 0x7f7b64072d20 2024-09-15T23:27:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.332+0000 7f7b6c042640 1 -- 172.21.15.62:0/1914262713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b641a67e0 con 0x7f7b64072d20 2024-09-15T23:27:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.335+0000 7f7b6edd3740 1 Processor -- start 2024-09-15T23:27:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.335+0000 7f7b6edd3740 1 -- start start 2024-09-15T23:27:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.335+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5c9a0 0x55921db5cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.335+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db61da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.335+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db62370 0x55921db64830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.335+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55921dac02f0 con 0x55921db5c9a0 2024-09-15T23:27:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.335+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55921da68270 con 0x55921db5d2e0 2024-09-15T23:27:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.335+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55921dabc2b0 con 0x55921db62370 2024-09-15T23:27:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b6a5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5c9a0 0x55921db5cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b6a5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5c9a0 0x55921db5cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52908/0 (socket says 172.21.15.62:52908) 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b6a5df640 1 -- 172.21.15.62:0/1751325711 learned_addr learned my addr 172.21.15.62:0/1751325711 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b695dd640 1 --2- 172.21.15.62:0/1751325711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db62370 0x55921db64830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b5affd640 1 -- 172.21.15.62:0/1914262713 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7b60010460 con 0x7f7b64072d20 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b6a5df640 1 -- 172.21.15.62:0/1751325711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db62370 msgr2=0x55921db64830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/1751325711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db62370 0x55921db64830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b6a5df640 1 -- 172.21.15.62:0/1751325711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 msgr2=0x55921db61da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/1751325711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db61da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.336+0000 7f7b6a5df640 1 -- 172.21.15.62:0/1751325711 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55921da67340 con 0x55921db5c9a0 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.337+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/1751325711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5c9a0 0x55921db5cda0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009ca0 tx=0x7f7b5c031df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615a25fc45809af2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.337+0000 7f7b4ffff640 1 -- 172.21.15.62:0/1751325711 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c031660 con 0x55921db5c9a0 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.337+0000 7f7b4ffff640 1 -- 172.21.15.62:0/1751325711 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b5c036040 con 0x55921db5c9a0 2024-09-15T23:27:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.337+0000 7f7b4ffff640 1 -- 172.21.15.62:0/1751325711 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c0046b0 con 0x55921db5c9a0 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.337+0000 7f7b5affd640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b500a34b0 0x7f7b500a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.338+0000 7f7b69dde640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b500a34b0 0x7f7b500a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.338+0000 7f7b5affd640 1 -- 172.21.15.62:0/1914262713 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7b600cf2f0 con 0x7f7b64072d20 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.338+0000 7f7b6edd3740 1 -- 172.21.15.62:0/1751325711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5c9a0 msgr2=0x55921db5cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.338+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/1751325711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5c9a0 0x55921db5cda0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009ca0 tx=0x7f7b5c031df0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.338+0000 7f7b69dde640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b500a34b0 0x7f7b500a5970 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7b600cca70 tx=0x7f7b6007a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.338+0000 7f7b6edd3740 1 -- 172.21.15.62:0/1751325711 shutdown_connections 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.338+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/1751325711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db62370 0x55921db64830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.338+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/1751325711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db61da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.339+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/1751325711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5c9a0 0x55921db5cda0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.339+0000 7f7b6edd3740 1 -- 172.21.15.62:0/1751325711 >> 172.21.15.62:0/1751325711 conn(0x55921db58010 msgr2=0x55921db58410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.339+0000 7f7b6edd3740 1 -- 172.21.15.62:0/1751325711 shutdown_connections 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.339+0000 7f7b6edd3740 1 -- 172.21.15.62:0/1751325711 wait complete. 2024-09-15T23:27:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.339+0000 7f7b6edd3740 1 -- start start 2024-09-15T23:27:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.339+0000 7f7b6edd3740 1 Processor -- start 2024-09-15T23:27:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db66b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5d2e0 0x55921db670d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db62370 0x55921db6e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55921db652e0 con 0x55921db5d2e0 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55921da49290 con 0x55921db62370 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55921db65460 con 0x55921db5c9a0 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6edd3740 10 client.0 _ll_register_callbacks cb 0x55921da7a140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6a5df640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db62370 0x55921db6e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6a5df640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db62370 0x55921db6e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52728/0 (socket says 172.21.15.62:52728) 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6a5df640 1 -- 172.21.15.62:0/3005686278 learned_addr learned my addr 172.21.15.62:0/3005686278 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db66b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b695dd640 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5d2e0 0x55921db670d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6a5df640 1 -- 172.21.15.62:0/3005686278 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 msgr2=0x55921db66b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db66b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6a5df640 1 -- 172.21.15.62:0/3005686278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5d2e0 msgr2=0x55921db670d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5d2e0 0x55921db670d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.340+0000 7f7b6a5df640 1 -- 172.21.15.62:0/3005686278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55921d7d3210 con 0x55921db62370 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.343+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db62370 0x55921db6e110 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55921db660c0 tx=0x7f7b5c031920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3ca5d2d863103db server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.343+0000 7f7b4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.343+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/3005686278 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c04b040 con 0x55921db62370 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.343+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3005686278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b5c036040 con 0x55921db62370 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.343+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/3005686278 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7b5c0046e0 con 0x55921db62370 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.343+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/3005686278 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7b5c004b40 con 0x55921db62370 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.343+0000 7f7b6edd3740 10 client.27587 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.343+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3005686278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55921d89f620 con 0x55921db62370 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.343+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/3005686278 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b5c036040 con 0x55921db62370 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/3005686278 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f7b5c051040 con 0x55921db62370 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b6edd3740 10 client.27587 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b6edd3740 10 client.27587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b6edd3740 20 client.27587 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55921db72410 0x55921db74800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b6edd3740 4 client.27587 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3005686278 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55921dacda20 con 0x55921db72410 2024-09-15T23:27:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b695dd640 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55921db72410 0x55921db74800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b4d7fa640 10 client.27587 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.344+0000 7f7b695dd640 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55921db72410 0x55921db74800 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bd55ebe5b298ed4 server_cookie=eaf61b4fdb22e09 in_seq=0 out_seq=0 2024-09-15T23:27:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.345+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/3005686278 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55921dacda20 con 0x55921db72410 2024-09-15T23:27:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.345+0000 7f7b4d7fa640 10 client.27587 handle_command_reply: tid=0 2024-09-15T23:27:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.346+0000 7f7b6edd3740 1 client.27587 shutdown 2024-09-15T23:27:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3005686278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db62370 msgr2=0x55921db6e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db62370 0x55921db6e110 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55921db660c0 tx=0x7f7b5c031920 comp rx=0 tx=0).stop 2024-09-15T23:27:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3005686278 shutdown_connections 2024-09-15T23:27:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55921db72410 0x55921db74800 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b4d7fa640 0 client.27587 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db62370 0x55921db6e110 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db5d2e0 0x55921db670d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.350 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3005686278 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db66b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3005686278 >> 172.21.15.62:0/3005686278 conn(0x55921db58010 msgr2=0x55921db59820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3005686278 shutdown_connections 2024-09-15T23:27:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.347+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3005686278 wait complete. 2024-09-15T23:27:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.350+0000 7f7b6edd3740 20 client.27587 trim_cache size 0 max 16384 2024-09-15T23:27:09.389 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.387+0000 7f7b6edd3740 1 Processor -- start 2024-09-15T23:27:09.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.387+0000 7f7b6edd3740 1 -- start start 2024-09-15T23:27:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.388+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 0x55921db58ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.388+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db6e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.388+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db59400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.388+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641b3560 con 0x55921db58010 2024-09-15T23:27:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.389+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641b0e30 con 0x55921db5d2e0 2024-09-15T23:27:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.389+0000 7f7b6a5df640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db59400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.389+0000 7f7b6a5df640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db59400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52734/0 (socket says 172.21.15.62:52734) 2024-09-15T23:27:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.389+0000 7f7b6a5df640 1 -- 172.21.15.62:0/3088331335 learned_addr learned my addr 172.21.15.62:0/3088331335 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.391 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.389+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3088331335 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641aec80 con 0x55921db5c9a0 2024-09-15T23:27:09.392 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.390+0000 7f7b695dd640 1 --2- 172.21.15.62:0/3088331335 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db6e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.396 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.393+0000 7f7b6a5df640 1 -- 172.21.15.62:0/3088331335 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 msgr2=0x55921db6e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.393+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/3088331335 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db6e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.393+0000 7f7b6a5df640 1 -- 172.21.15.62:0/3088331335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 msgr2=0x55921db58ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.393+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/3088331335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 0x55921db58ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.393+0000 7f7b6a5df640 1 -- 172.21.15.62:0/3088331335 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55921db57e30 con 0x55921db5d2e0 2024-09-15T23:27:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.396+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/3088331335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db59400 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009620 tx=0x7f7b5c031df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82099064fba9bd20 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.396+0000 7f7b4f7fe640 1 -- 172.21.15.62:0/3088331335 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c05b070 con 0x55921db5d2e0 2024-09-15T23:27:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.396+0000 7f7b4f7fe640 1 -- 172.21.15.62:0/3088331335 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b5c036030 con 0x55921db5d2e0 2024-09-15T23:27:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.397+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3088331335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 msgr2=0x55921db59400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.397+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3088331335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db59400 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c009620 tx=0x7f7b5c031df0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.400 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.398+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3088331335 shutdown_connections 2024-09-15T23:27:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.398+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3088331335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db59400 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.398+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3088331335 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db6e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.398+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/3088331335 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 0x55921db58ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.398+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3088331335 >> 172.21.15.62:0/3088331335 conn(0x55921db61e90 msgr2=0x55921db66b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.398+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3088331335 shutdown_connections 2024-09-15T23:27:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.403+0000 7f7b6edd3740 1 -- 172.21.15.62:0/3088331335 wait complete. 2024-09-15T23:27:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.403+0000 7f7b6edd3740 1 -- start start 2024-09-15T23:27:09.405 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.403+0000 7f7b6edd3740 1 Processor -- start 2024-09-15T23:27:09.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.404+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 0x55921db6b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db6b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6edd3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db6c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641af2f0 con 0x55921db58010 2024-09-15T23:27:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641b0760 con 0x55921db5d2e0 2024-09-15T23:27:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6edd3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641af9c0 con 0x55921db5c9a0 2024-09-15T23:27:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6edd3740 10 client.0 _ll_register_callbacks cb 0x7f7b54005e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:09.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6a5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 0x55921db6b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6a5df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 0x55921db6b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52928/0 (socket says 172.21.15.62:52928) 2024-09-15T23:27:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6a5df640 1 -- 172.21.15.62:0/2068208877 learned_addr learned my addr 172.21.15.62:0/2068208877 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.405+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db6c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.406+0000 7f7b6a5df640 1 -- 172.21.15.62:0/2068208877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 msgr2=0x55921db6b8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:27:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.406+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db6b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.406+0000 7f7b6a5df640 1 -- 172.21.15.62:0/2068208877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 msgr2=0x55921db6c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.406+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db6c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.406+0000 7f7b6a5df640 1 -- 172.21.15.62:0/2068208877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55921db6ce50 con 0x55921db58010 2024-09-15T23:27:09.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.406+0000 7f7b6a5df640 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 0x55921db6b390 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c000f80 tx=0x7f7b5c02b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec567fb29478b7b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.407+0000 7f7b4d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:09.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.407+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/2068208877 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c05b030 con 0x55921db58010 2024-09-15T23:27:09.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.407+0000 7f7b6edd3740 1 -- 172.21.15.62:0/2068208877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b5c036030 con 0x55921db58010 2024-09-15T23:27:09.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.407+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/2068208877 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7b5c005af0 con 0x55921db58010 2024-09-15T23:27:09.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.410+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/2068208877 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7b5c06d050 con 0x55921db58010 2024-09-15T23:27:09.413 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.411+0000 7f7b6edd3740 10 client.18024 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:09.413 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.411+0000 7f7b6edd3740 1 -- 172.21.15.62:0/2068208877 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55921db6e160 con 0x55921db58010 2024-09-15T23:27:09.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.413+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/2068208877 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b5c036030 con 0x55921db58010 2024-09-15T23:27:09.415 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.413+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/2068208877 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f7b5c051070 con 0x55921db58010 2024-09-15T23:27:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.415+0000 7f7b6edd3740 10 client.18024 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.415+0000 7f7b6edd3740 10 client.18024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.415+0000 7f7b6edd3740 20 client.18024 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.415+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55921db62370 0x55921dbdf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.415+0000 7f7b6edd3740 4 client.18024 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:27:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.415+0000 7f7b6edd3740 1 -- 172.21.15.62:0/2068208877 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7b5c004e70 con 0x55921db62370 2024-09-15T23:27:09.418 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.415+0000 7f7b695dd640 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55921db62370 0x55921dbdf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.418 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.416+0000 7f7b695dd640 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55921db62370 0x55921dbdf2f0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5771e3d48eaab8c server_cookie=7c26e0b048390c1d in_seq=0 out_seq=0 2024-09-15T23:27:09.418 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.416+0000 7f7b4d7fa640 10 client.18024 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.451+0000 7f7b4d7fa640 1 -- 172.21.15.62:0/2068208877 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7b5c004e70 con 0x55921db62370 2024-09-15T23:27:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.451+0000 7f7b4d7fa640 10 client.18024 handle_command_reply: tid=0 2024-09-15T23:27:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.451+0000 7f7b6edd3740 1 client.18024 shutdown 2024-09-15T23:27:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.451+0000 7f7b6edd3740 1 -- 172.21.15.62:0/2068208877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 msgr2=0x55921db6b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.451+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 0x55921db6b390 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c000f80 tx=0x7f7b5c02b050 comp rx=0 tx=0).stop 2024-09-15T23:27:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.451+0000 7f7b6edd3740 1 -- 172.21.15.62:0/2068208877 shutdown_connections 2024-09-15T23:27:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.451+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55921db62370 0x55921dbdf2f0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.452+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55921db5d2e0 0x55921db6c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.452+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55921db5c9a0 0x55921db6b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.452+0000 7f7b6edd3740 1 --2- 172.21.15.62:0/2068208877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55921db58010 0x55921db6b390 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.452+0000 7f7b6edd3740 1 -- 172.21.15.62:0/2068208877 >> 172.21.15.62:0/2068208877 conn(0x55921db61e90 msgr2=0x55921db66b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.452+0000 7f7b4d7fa640 0 client.18024 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.452+0000 7f7b6edd3740 1 -- 172.21.15.62:0/2068208877 shutdown_connections 2024-09-15T23:27:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.452+0000 7f7b6edd3740 1 -- 172.21.15.62:0/2068208877 wait complete. 2024-09-15T23:27:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.452+0000 7f7b6edd3740 20 client.18024 trim_cache size 0 max 16384 2024-09-15T23:27:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:27:09.455 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:27:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:27:09.456 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:27:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.453+0000 7f7b6c042640 1 -- 172.21.15.62:0/1914262713 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b500a34b0 msgr2=0x7f7b500a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.457 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.453+0000 7f7b6c042640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b500a34b0 0x7f7b500a5970 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f7b600cca70 tx=0x7f7b6007a040 comp rx=0 tx=0).stop 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.453+0000 7f7b6c042640 1 -- 172.21.15.62:0/1914262713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b64072d20 msgr2=0x7f7b64119620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.453+0000 7f7b6c042640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b64072d20 0x7f7b64119620 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000d950 tx=0x7f7b6000de10 comp rx=0 tx=0).stop 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.454+0000 7f7b6c042640 1 -- 172.21.15.62:0/1914262713 shutdown_connections 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.454+0000 7f7b6c042640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7b500a34b0 0x7f7b500a5970 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.454+0000 7f7b6c042640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b64109930 0x7f7b64120bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.454+0000 7f7b6c042640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7b640737d0 0x7f7b64119b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.454+0000 7f7b6c042640 1 --2- 172.21.15.62:0/1914262713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b64072d20 0x7f7b64119620 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.454+0000 7f7b6c042640 1 -- 172.21.15.62:0/1914262713 >> 172.21.15.62:0/1914262713 conn(0x7f7b6406ece0 msgr2=0x7f7b6410dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.455+0000 7f7b6c042640 1 -- 172.21.15.62:0/1914262713 shutdown_connections 2024-09-15T23:27:09.458 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:09.455+0000 7f7b6c042640 1 -- 172.21.15.62:0/1914262713 wait complete. 2024-09-15T23:27:09.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.501+0000 7fb7809d1640 1 -- 172.21.15.62:0/2655384211 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb77c101df0 msgr2=0x7fb77c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.501+0000 7fb7809d1640 1 --2- 172.21.15.62:0/2655384211 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb77c101df0 0x7fb77c1021d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009900 tx=0x7fb76c02a410 comp rx=0 tx=0).stop 2024-09-15T23:27:09.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.501+0000 7fb7809d1640 1 -- 172.21.15.62:0/2655384211 shutdown_connections 2024-09-15T23:27:09.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.501+0000 7fb7809d1640 1 --2- 172.21.15.62:0/2655384211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb77c106f20 0x7fb77c10eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.501+0000 7fb7809d1640 1 --2- 172.21.15.62:0/2655384211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c1027a0 0x7fb77c1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.501+0000 7fb7809d1640 1 --2- 172.21.15.62:0/2655384211 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb77c101df0 0x7fb77c1021d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.501+0000 7fb7809d1640 1 -- 172.21.15.62:0/2655384211 >> 172.21.15.62:0/2655384211 conn(0x7fb77c0fd8b0 msgr2=0x7fb77c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.501+0000 7fb7809d1640 1 -- 172.21.15.62:0/2655384211 shutdown_connections 2024-09-15T23:27:09.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.501+0000 7fb7809d1640 1 -- 172.21.15.62:0/2655384211 wait complete. 2024-09-15T23:27:09.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.502+0000 7fb7809d1640 1 Processor -- start 2024-09-15T23:27:09.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb7809d1640 1 -- start start 2024-09-15T23:27:09.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb7809d1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb77c101df0 0x7fb77c1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb7809d1640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb77c1027a0 0x7fb77c198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb7809d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c106f20 0x7fb77c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb77c101df0 0x7fb77c1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb77c101df0 0x7fb77c1982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52942/0 (socket says 172.21.15.62:52942) 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb77a59c640 1 -- 172.21.15.62:0/3850542235 learned_addr learned my addr 172.21.15.62:0/3850542235 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c111ff0 con 0x7fb77c101df0 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c111e70 con 0x7fb77c106f20 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c112170 con 0x7fb77c1027a0 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb77ad9d640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c106f20 0x7fb77c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.503+0000 7fb779d9b640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb77c1027a0 0x7fb77c198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb77ad9d640 1 -- 172.21.15.62:0/3850542235 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb77c1027a0 msgr2=0x7fb77c198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb77ad9d640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb77c1027a0 0x7fb77c198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb77ad9d640 1 -- 172.21.15.62:0/3850542235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb77c101df0 msgr2=0x7fb77c1982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb77ad9d640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb77c101df0 0x7fb77c1982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb77ad9d640 1 -- 172.21.15.62:0/3850542235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c19d280 con 0x7fb77c106f20 2024-09-15T23:27:09.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb77ad9d640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c106f20 0x7fb77c19cba0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb77400be20 tx=0x7fb77400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb76b7fe640 1 -- 172.21.15.62:0/3850542235 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774019070 con 0x7fb77c106f20 2024-09-15T23:27:09.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb76b7fe640 1 -- 172.21.15.62:0/3850542235 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb774014920 con 0x7fb77c106f20 2024-09-15T23:27:09.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77c19d570 con 0x7fb77c106f20 2024-09-15T23:27:09.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb76b7fe640 1 -- 172.21.15.62:0/3850542235 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb774012920 con 0x7fb77c106f20 2024-09-15T23:27:09.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.504+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77c1a4e50 con 0x7fb77c106f20 2024-09-15T23:27:09.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.508+0000 7fb76b7fe640 1 -- 172.21.15.62:0/3850542235 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb77401c070 con 0x7fb77c106f20 2024-09-15T23:27:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.509+0000 7fb76b7fe640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb7640a3300 0x7fb7640a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.509+0000 7fb77a59c640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb7640a3300 0x7fb7640a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.509+0000 7fb76b7fe640 1 -- 172.21.15.62:0/3850542235 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb7740cf500 con 0x7fb77c106f20 2024-09-15T23:27:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb782daa740 1 Processor -- start 2024-09-15T23:27:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb77a59c640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb7640a3300 0x7fb7640a57c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb76c005ab0 tx=0x7fb76c00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb782daa740 1 -- start start 2024-09-15T23:27:09.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb782daa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 0x556068002c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb782daa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068003170 0x55606800d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb782daa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x55606800fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb782daa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556067f67160 con 0x55606800da30 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb782daa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556067f0e720 con 0x556068003170 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb782daa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556067f63120 con 0x556068002850 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb77ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x55606800fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 0x556068002c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb77ad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x55606800fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52956/0 (socket says 172.21.15.62:52956) 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb77ad9d640 1 -- 172.21.15.62:0/849668544 learned_addr learned my addr 172.21.15.62:0/849668544 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.510+0000 7fb779d9b640 1 --2- 172.21.15.62:0/849668544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068003170 0x55606800d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb779d9b640 1 -- 172.21.15.62:0/849668544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 msgr2=0x556068002c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb779d9b640 1 --2- 172.21.15.62:0/849668544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 0x556068002c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb779d9b640 1 -- 172.21.15.62:0/849668544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 msgr2=0x55606800fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb779d9b640 1 --2- 172.21.15.62:0/849668544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x55606800fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb779d9b640 1 -- 172.21.15.62:0/849668544 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556067cd8ba0 con 0x556068003170 2024-09-15T23:27:09.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb77ad9d640 1 --2- 172.21.15.62:0/849668544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x55606800fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:09.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb779d9b640 1 --2- 172.21.15.62:0/849668544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068003170 0x55606800d4f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb77000bbb0 tx=0x7fb77000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d884dc309317e61a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb768ff9640 1 -- 172.21.15.62:0/849668544 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770015070 con 0x556068003170 2024-09-15T23:27:09.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb768ff9640 1 -- 172.21.15.62:0/849668544 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7700035f0 con 0x556068003170 2024-09-15T23:27:09.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.511+0000 7fb768ff9640 1 -- 172.21.15.62:0/849668544 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7700116a0 con 0x556068003170 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.512+0000 7fb782daa740 1 -- 172.21.15.62:0/849668544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068003170 msgr2=0x55606800d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.512+0000 7fb782daa740 1 --2- 172.21.15.62:0/849668544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068003170 0x55606800d4f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fb77000bbb0 tx=0x7fb77000f730 comp rx=0 tx=0).stop 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.512+0000 7fb782daa740 1 -- 172.21.15.62:0/849668544 shutdown_connections 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.512+0000 7fb782daa740 1 --2- 172.21.15.62:0/849668544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x55606800fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.512+0000 7fb782daa740 1 --2- 172.21.15.62:0/849668544 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068003170 0x55606800d4f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.512+0000 7fb782daa740 1 --2- 172.21.15.62:0/849668544 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 0x556068002c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.512+0000 7fb782daa740 1 -- 172.21.15.62:0/849668544 >> 172.21.15.62:0/849668544 conn(0x556067ffede0 msgr2=0x556067fff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 -- 172.21.15.62:0/849668544 shutdown_connections 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 -- 172.21.15.62:0/849668544 wait complete. 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 -- start start 2024-09-15T23:27:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 Processor -- start 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068002850 0x55606800a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x55606800ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 0x55606800b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556068014a10 con 0x556068002850 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556067eef900 con 0x55606800da30 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556068014b90 con 0x556068003170 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.513+0000 7fb782daa740 10 client.0 _ll_register_callbacks cb 0x556067eed780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb779d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x55606800ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb779d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x55606800ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:41334/0 (socket says 172.21.15.62:41334) 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb779d9b640 1 -- 172.21.15.62:0/2445124188 learned_addr learned my addr 172.21.15.62:0/2445124188 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb77a59c640 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068002850 0x55606800a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb77ad9d640 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 0x55606800b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb779d9b640 1 -- 172.21.15.62:0/2445124188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 msgr2=0x55606800b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb779d9b640 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 0x55606800b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb779d9b640 1 -- 172.21.15.62:0/2445124188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068002850 msgr2=0x55606800a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb779d9b640 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068002850 0x55606800a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb779d9b640 1 -- 172.21.15.62:0/2445124188 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556067f2fc50 con 0x556068003170 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb77a59c640 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068002850 0x55606800a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb779d9b640 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x55606800ae40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb770003d90 tx=0x7fb770005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed1a500c00950cd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.514+0000 7fb761ffb640 1 -- 172.21.15.62:0/2445124188 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770015070 con 0x556068003170 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb782daa740 1 -- 172.21.15.62:0/2445124188 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7700035f0 con 0x556068003170 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb761ffb640 1 -- 172.21.15.62:0/2445124188 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb7700037a0 con 0x556068003170 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb761ffb640 1 -- 172.21.15.62:0/2445124188 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb77002d050 con 0x556068003170 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb782daa740 10 client.27343 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb782daa740 1 -- 172.21.15.62:0/2445124188 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556068010360 con 0x556068003170 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb761ffb640 1 -- 172.21.15.62:0/2445124188 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7700035f0 con 0x556068003170 2024-09-15T23:27:09.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb761ffb640 1 -- 172.21.15.62:0/2445124188 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fb770011930 con 0x556068003170 2024-09-15T23:27:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb782daa740 10 client.27343 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb782daa740 10 client.27343 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:27:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb782daa740 20 client.27343 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.515+0000 7fb782daa740 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556068011bb0 0x556068014010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.516+0000 7fb782daa740 4 client.27343 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.516+0000 7fb782daa740 1 -- 172.21.15.62:0/2445124188 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556067f74890 con 0x556068011bb0 2024-09-15T23:27:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.516+0000 7fb77a59c640 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556068011bb0 0x556068014010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.517+0000 7fb77a59c640 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556068011bb0 0x556068014010 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27797791114c6eb4 server_cookie=39b10b7c460ec5f7 in_seq=0 out_seq=0 2024-09-15T23:27:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.517+0000 7fb761ffb640 10 client.27343 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.517+0000 7fb761ffb640 1 -- 172.21.15.62:0/2445124188 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556067f74890 con 0x556068011bb0 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.517+0000 7fb761ffb640 10 client.27343 handle_command_reply: tid=0 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.517+0000 7fb782daa740 1 client.27343 shutdown 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 -- 172.21.15.62:0/2445124188 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 msgr2=0x55606800ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x55606800ae40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb770003d90 tx=0x7fb770005740 comp rx=0 tx=0).stop 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 -- 172.21.15.62:0/2445124188 shutdown_connections 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556068011bb0 0x556068014010 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 0x55606800b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x55606800ae40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 --2- 172.21.15.62:0/2445124188 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068002850 0x55606800a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 -- 172.21.15.62:0/2445124188 >> 172.21.15.62:0/2445124188 conn(0x556067ffede0 msgr2=0x556068004ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb761ffb640 0 client.27343 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 -- 172.21.15.62:0/2445124188 shutdown_connections 2024-09-15T23:27:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 1 -- 172.21.15.62:0/2445124188 wait complete. 2024-09-15T23:27:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.518+0000 7fb782daa740 20 client.27343 trim_cache size 0 max 16384 2024-09-15T23:27:09.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:27:09.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:27:09.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:27:09.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:27:09.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: pgmap v696: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-09-15T23:27:09.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:27:09.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.526 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:09 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:27:09.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:27:09.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:27:09.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:27:09.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: pgmap v696: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 1.9 MiB/s wr, 140 op/s 2024-09-15T23:27:09.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:27:09.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:09 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:09.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb782daa740 1 Processor -- start 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb782daa740 1 -- start start 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb782daa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 0x556068009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb782daa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068003170 0x5560680052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb782daa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 0x556068005e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb782daa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c1af3e0 con 0x556068003170 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb782daa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c1accb0 con 0x55606800da30 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb782daa740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c1aab00 con 0x556068002850 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb779d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068003170 0x5560680052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb779d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068003170 0x5560680052b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52970/0 (socket says 172.21.15.62:52970) 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb779d9b640 1 -- 172.21.15.62:0/1221811801 learned_addr learned my addr 172.21.15.62:0/1221811801 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb77a59c640 1 --2- 172.21.15.62:0/1221811801 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 0x556068009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.539+0000 7fb77ad9d640 1 --2- 172.21.15.62:0/1221811801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 0x556068005e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.540+0000 7fb77a59c640 1 -- 172.21.15.62:0/1221811801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 msgr2=0x556068005e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.540+0000 7fb77a59c640 1 --2- 172.21.15.62:0/1221811801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 0x556068005e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.540+0000 7fb77a59c640 1 -- 172.21.15.62:0/1221811801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068003170 msgr2=0x5560680052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.540+0000 7fb77a59c640 1 --2- 172.21.15.62:0/1221811801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068003170 0x5560680052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.540+0000 7fb77a59c640 1 -- 172.21.15.62:0/1221811801 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556067ffa3f0 con 0x556068002850 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.540+0000 7fb77a59c640 1 --2- 172.21.15.62:0/1221811801 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 0x556068009840 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x556068014610 tx=0x7fb76c03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94c86d5d91e8d9af server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.540+0000 7fb763fff640 1 -- 172.21.15.62:0/1221811801 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c058070 con 0x556068002850 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.540+0000 7fb763fff640 1 -- 172.21.15.62:0/1221811801 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb76c032df0 con 0x556068002850 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.540+0000 7fb763fff640 1 -- 172.21.15.62:0/1221811801 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c035040 con 0x556068002850 2024-09-15T23:27:09.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.541+0000 7fb782daa740 1 -- 172.21.15.62:0/1221811801 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 msgr2=0x556068009840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.541+0000 7fb782daa740 1 --2- 172.21.15.62:0/1221811801 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 0x556068009840 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x556068014610 tx=0x7fb76c03d000 comp rx=0 tx=0).stop 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.541+0000 7fb782daa740 1 -- 172.21.15.62:0/1221811801 shutdown_connections 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.541+0000 7fb782daa740 1 --2- 172.21.15.62:0/1221811801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55606800da30 0x556068005e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.541+0000 7fb782daa740 1 --2- 172.21.15.62:0/1221811801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556068003170 0x5560680052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.541+0000 7fb782daa740 1 --2- 172.21.15.62:0/1221811801 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068002850 0x556068009840 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.541+0000 7fb782daa740 1 -- 172.21.15.62:0/1221811801 >> 172.21.15.62:0/1221811801 conn(0x556067ffede0 msgr2=0x55606800c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.541+0000 7fb782daa740 1 -- 172.21.15.62:0/1221811801 shutdown_connections 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.541+0000 7fb782daa740 1 -- 172.21.15.62:0/1221811801 wait complete. 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.542+0000 7fb782daa740 1 -- start start 2024-09-15T23:27:09.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.542+0000 7fb782daa740 1 Processor -- start 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.542+0000 7fb782daa740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068002850 0x556068014da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb782daa740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x5560680152e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb782daa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x556068015820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb782daa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c1ab170 con 0x55606800da30 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb782daa740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c1ac5e0 con 0x556068002850 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb779d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x5560680152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb779d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x5560680152e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:41340/0 (socket says 172.21.15.62:41340) 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb779d9b640 1 -- 172.21.15.62:0/573510355 learned_addr learned my addr 172.21.15.62:0/573510355 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb782daa740 1 -- 172.21.15.62:0/573510355 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c1ab840 con 0x556068003170 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb77ad9d640 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x556068015820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb782daa740 10 client.0 _ll_register_callbacks cb 0x7fb7740973f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:09.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.543+0000 7fb77a59c640 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068002850 0x556068014da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.544+0000 7fb779d9b640 1 -- 172.21.15.62:0/573510355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068002850 msgr2=0x556068014da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.544+0000 7fb779d9b640 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068002850 0x556068014da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.544+0000 7fb779d9b640 1 -- 172.21.15.62:0/573510355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 msgr2=0x556068015820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.544+0000 7fb779d9b640 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x556068015820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.544+0000 7fb779d9b640 1 -- 172.21.15.62:0/573510355 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560680869f0 con 0x556068003170 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.544+0000 7fb779d9b640 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x5560680152e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb770000c00 tx=0x7fb770024470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6542d05df5f20549 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.544+0000 7fb761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.545+0000 7fb761ffb640 1 -- 172.21.15.62:0/573510355 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770037060 con 0x556068003170 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.545+0000 7fb761ffb640 1 -- 172.21.15.62:0/573510355 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb770004030 con 0x556068003170 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.545+0000 7fb782daa740 1 -- 172.21.15.62:0/573510355 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb76c032df0 con 0x556068003170 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.545+0000 7fb761ffb640 1 -- 172.21.15.62:0/573510355 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb770015070 con 0x556068003170 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.545+0000 7fb782daa740 10 client.27349 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:09.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.545+0000 7fb782daa740 1 -- 172.21.15.62:0/573510355 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556068087c70 con 0x556068003170 2024-09-15T23:27:09.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.545+0000 7fb761ffb640 1 -- 172.21.15.62:0/573510355 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb76c032df0 con 0x556068003170 2024-09-15T23:27:09.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.545+0000 7fb761ffb640 1 -- 172.21.15.62:0/573510355 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fb770011770 con 0x556068003170 2024-09-15T23:27:09.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.546+0000 7fb782daa740 10 client.27349 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:09.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.546+0000 7fb782daa740 10 client.27349 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:27:09.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.546+0000 7fb782daa740 20 client.27349 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:09.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.546+0000 7fb782daa740 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5560680883d0 0x55606808a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:09.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.546+0000 7fb782daa740 4 client.27349 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}] 2024-09-15T23:27:09.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.546+0000 7fb782daa740 1 -- 172.21.15.62:0/573510355 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 141.85, "await_for": 45.0}) -- 0x556067f6ebe0 con 0x5560680883d0 2024-09-15T23:27:09.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.546+0000 7fb77a59c640 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5560680883d0 0x55606808a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:09.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.547+0000 7fb77a59c640 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5560680883d0 0x55606808a7f0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=824b4b82316af31 server_cookie=cd1920ffa529ab39 in_seq=0 out_seq=0 2024-09-15T23:27:09.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.547+0000 7fb761ffb640 10 client.27349 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 650.0 s, 539.6 tps, lat 188.893 ms stddev 242.585 2024-09-15T23:27:09.637 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb761ffb640 1 -- 172.21.15.62:0/573510355 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x556067f6ebe0 con 0x5560680883d0 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb761ffb640 10 client.27349 handle_command_reply: tid=0 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 client.27349 shutdown 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 -- 172.21.15.62:0/573510355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 msgr2=0x5560680152e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x5560680152e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb770000c00 tx=0x7fb770024470 comp rx=0 tx=0).stop 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 -- 172.21.15.62:0/573510355 shutdown_connections 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5560680883d0 0x55606808a7f0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606800da30 0x556068015820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556068003170 0x5560680152e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb761ffb640 0 client.27349 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 --2- 172.21.15.62:0/573510355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556068002850 0x556068014da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 -- 172.21.15.62:0/573510355 >> 172.21.15.62:0/573510355 conn(0x556067ffede0 msgr2=0x556067ffd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 -- 172.21.15.62:0/573510355 shutdown_connections 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 1 -- 172.21.15.62:0/573510355 wait complete. 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb782daa740 20 client.27349 trim_cache size 0 max 16384 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 35, 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 42, 2024-09-15T23:27:09.976 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "de217ad0": { 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 42, 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 141.8, 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:27:09.977 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:27:09.978 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:27:09.978 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:09.978 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:09.978 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:09.978 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:09.978 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:09.978 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:27:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb7640a3300 msgr2=0x7fb7640a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb7640a3300 0x7fb7640a57c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb76c005ab0 tx=0x7fb76c00b040 comp rx=0 tx=0).stop 2024-09-15T23:27:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c106f20 msgr2=0x7fb77c19cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c106f20 0x7fb77c19cba0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb77400be20 tx=0x7fb77400bf10 comp rx=0 tx=0).stop 2024-09-15T23:27:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 shutdown_connections 2024-09-15T23:27:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb7640a3300 0x7fb7640a57c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb77c106f20 0x7fb77c19cba0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb77c1027a0 0x7fb77c198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 --2- 172.21.15.62:0/3850542235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb77c101df0 0x7fb77c1982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:09.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 >> 172.21.15.62:0/3850542235 conn(0x7fb77c0fd8b0 msgr2=0x7fb77c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:09.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 shutdown_connections 2024-09-15T23:27:09.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:09.973+0000 7fb7809d1640 1 -- 172.21.15.62:0/3850542235 wait complete. 2024-09-15T23:27:10.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 -- 172.21.15.62:0/685260050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 msgr2=0x7fab74103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:10.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 --2- 172.21.15.62:0/685260050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 0x7fab74103180 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fab70009900 tx=0x7fab7002a480 comp rx=0 tx=0).stop 2024-09-15T23:27:10.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 -- 172.21.15.62:0/685260050 shutdown_connections 2024-09-15T23:27:10.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 --2- 172.21.15.62:0/685260050 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab7410c180 0x7fab7410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 --2- 172.21.15.62:0/685260050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab741036c0 0x7fab7410bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 --2- 172.21.15.62:0/685260050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 0x7fab74103180 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 -- 172.21.15.62:0/685260050 >> 172.21.15.62:0/685260050 conn(0x7fab740fc7a0 msgr2=0x7fab740febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:10.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 -- 172.21.15.62:0/685260050 shutdown_connections 2024-09-15T23:27:10.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 -- 172.21.15.62:0/685260050 wait complete. 2024-09-15T23:27:10.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 Processor -- start 2024-09-15T23:27:10.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.461+0000 7fab7b35a640 1 -- start start 2024-09-15T23:27:10.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab7b35a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 0x7fab7419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab7b35a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab741036c0 0x7fab7419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab7b35a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab7410c180 0x7fab7419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab7b35a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fab74111f60 con 0x7fab74102da0 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab7b35a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab74111de0 con 0x7fab741036c0 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab7b35a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fab741120e0 con 0x7fab7410c180 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab790f6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 0x7fab7419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab790f6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 0x7fab7419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52992/0 (socket says 172.21.15.62:52992) 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab790f6640 1 -- 172.21.15.62:0/460973935 learned_addr learned my addr 172.21.15.62:0/460973935 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab788f5640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab741036c0 0x7fab7419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab798f7640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab7410c180 0x7fab7419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab790f6640 1 -- 172.21.15.62:0/460973935 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab7410c180 msgr2=0x7fab7419ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab790f6640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab7410c180 0x7fab7419ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab790f6640 1 -- 172.21.15.62:0/460973935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab741036c0 msgr2=0x7fab7419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab790f6640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab741036c0 0x7fab7419ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab790f6640 1 -- 172.21.15.62:0/460973935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7419f600 con 0x7fab74102da0 2024-09-15T23:27:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab788f5640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab741036c0 0x7fab7419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.462+0000 7fab790f6640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 0x7fab7419a650 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fab70006010 tx=0x7fab700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.463+0000 7fab6a7fc640 1 -- 172.21.15.62:0/460973935 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7003f030 con 0x7fab74102da0 2024-09-15T23:27:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.463+0000 7fab7b35a640 1 -- 172.21.15.62:0/460973935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7419f890 con 0x7fab74102da0 2024-09-15T23:27:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.463+0000 7fab6a7fc640 1 -- 172.21.15.62:0/460973935 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab70003ce0 con 0x7fab74102da0 2024-09-15T23:27:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.463+0000 7fab6a7fc640 1 -- 172.21.15.62:0/460973935 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70035070 con 0x7fab74102da0 2024-09-15T23:27:10.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.463+0000 7fab7b35a640 1 -- 172.21.15.62:0/460973935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab741a71d0 con 0x7fab74102da0 2024-09-15T23:27:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.465+0000 7fab6a7fc640 1 -- 172.21.15.62:0/460973935 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fab70042030 con 0x7fab74102da0 2024-09-15T23:27:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.465+0000 7fab7d733740 1 Processor -- start 2024-09-15T23:27:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.465+0000 7fab7d733740 1 -- start start 2024-09-15T23:27:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab7d733740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60c4a0 0x559b2e60c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab7d733740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e60cdc0 0x559b2e617140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab7d733740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e617680 0x559b2e619a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab7d733740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559b2e5722e0 con 0x559b2e60cdc0 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab7d733740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559b2e519ce0 con 0x559b2e617680 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab7d733740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559b2e56e2a0 con 0x559b2e60c4a0 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab790f6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e60cdc0 0x559b2e617140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab790f6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e60cdc0 0x559b2e617140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52998/0 (socket says 172.21.15.62:52998) 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab790f6640 1 -- 172.21.15.62:0/3896066550 learned_addr learned my addr 172.21.15.62:0/3896066550 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab798f7640 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60c4a0 0x559b2e60c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab788f5640 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e617680 0x559b2e619a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab6a7fc640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fab600a3240 0x7fab600a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab790f6640 1 -- 172.21.15.62:0/3896066550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60c4a0 msgr2=0x559b2e60c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab790f6640 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60c4a0 0x559b2e60c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab790f6640 1 -- 172.21.15.62:0/3896066550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e617680 msgr2=0x559b2e619a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:10.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab790f6640 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e617680 0x559b2e619a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab790f6640 1 -- 172.21.15.62:0/3896066550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b2e4fe440 con 0x559b2e60cdc0 2024-09-15T23:27:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab798f7640 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60c4a0 0x559b2e60c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.466+0000 7fab798f7640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fab600a3240 0x7fab600a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.467+0000 7fab6a7fc640 1 -- 172.21.15.62:0/460973935 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fab700f02c0 con 0x7fab74102da0 2024-09-15T23:27:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.467+0000 7fab790f6640 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e60cdc0 0x559b2e617140 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fab700c78e0 tx=0x7fab700b8940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7c9ed5214fa6b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.467+0000 7fab5f7fe640 1 -- 172.21.15.62:0/3896066550 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70049070 con 0x559b2e60cdc0 2024-09-15T23:27:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.467+0000 7fab5f7fe640 1 -- 172.21.15.62:0/3896066550 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab700c7910 con 0x559b2e60cdc0 2024-09-15T23:27:10.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.467+0000 7fab5f7fe640 1 -- 172.21.15.62:0/3896066550 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab700bc3f0 con 0x559b2e60cdc0 2024-09-15T23:27:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 -- 172.21.15.62:0/3896066550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e60cdc0 msgr2=0x559b2e617140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab798f7640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fab600a3240 0x7fab600a5700 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fab6c005fd0 tx=0x7fab6c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e60cdc0 0x559b2e617140 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fab700c78e0 tx=0x7fab700b8940 comp rx=0 tx=0).stop 2024-09-15T23:27:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 -- 172.21.15.62:0/3896066550 shutdown_connections 2024-09-15T23:27:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e617680 0x559b2e619a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e60cdc0 0x559b2e617140 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 --2- 172.21.15.62:0/3896066550 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60c4a0 0x559b2e60c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 -- 172.21.15.62:0/3896066550 >> 172.21.15.62:0/3896066550 conn(0x559b2e608a30 msgr2=0x559b2e608e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:10.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 -- 172.21.15.62:0/3896066550 shutdown_connections 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 -- 172.21.15.62:0/3896066550 wait complete. 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 -- start start 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.468+0000 7fab7d733740 1 Processor -- start 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab7d733740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e616fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab7d733740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e614260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab7d733740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e6147a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab7d733740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559b2e61a010 con 0x559b2e617680 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab7d733740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559b2e52d050 con 0x559b2e60c4a0 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab7d733740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559b2e61a190 con 0x559b2e60cdc0 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab7d733740 10 client.0 _ll_register_callbacks cb 0x559b2e56d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab798f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e6147a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab798f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e6147a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53008/0 (socket says 172.21.15.62:53008) 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab798f7640 1 -- 172.21.15.62:0/3161013682 learned_addr learned my addr 172.21.15.62:0/3161013682 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:10.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab788f5640 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e614260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab790f6640 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e616fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab798f7640 1 -- 172.21.15.62:0/3161013682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 msgr2=0x559b2e614260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab798f7640 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e614260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab798f7640 1 -- 172.21.15.62:0/3161013682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 msgr2=0x559b2e616fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab798f7640 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e616fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab798f7640 1 -- 172.21.15.62:0/3161013682 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b2e2e33c0 con 0x559b2e617680 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab790f6640 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e616fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab798f7640 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e6147a0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009fd0 tx=0x7fab6c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6a859f592fc84 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab5cff9640 1 -- 172.21.15.62:0/3161013682 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c01db30 con 0x559b2e617680 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab5cff9640 1 -- 172.21.15.62:0/3161013682 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fab6c01dcd0 con 0x559b2e617680 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.469+0000 7fab7d733740 1 -- 172.21.15.62:0/3161013682 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559b2e57f850 con 0x559b2e617680 2024-09-15T23:27:10.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab5cff9640 1 -- 172.21.15.62:0/3161013682 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fab6c027470 con 0x559b2e617680 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab7d733740 10 client.18048 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab7d733740 1 -- 172.21.15.62:0/3161013682 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559b2e625990 con 0x559b2e617680 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab5cff9640 1 -- 172.21.15.62:0/3161013682 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559b2e57f850 con 0x559b2e617680 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab5cff9640 1 -- 172.21.15.62:0/3161013682 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fab6c00f040 con 0x559b2e617680 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab7d733740 10 client.18048 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab7d733740 10 client.18048 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab7d733740 20 client.18048 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab7d733740 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b2e626e50 0x559b2e6292a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab7d733740 4 client.18048 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab7d733740 1 -- 172.21.15.62:0/3161013682 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b2e579d60 con 0x559b2e626e50 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.470+0000 7fab790f6640 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b2e626e50 0x559b2e6292a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.471+0000 7fab790f6640 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b2e626e50 0x559b2e6292a0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fd06c28d999a7bb server_cookie=21acaa7697248f98 in_seq=0 out_seq=0 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.471+0000 7fab5cff9640 10 client.18048 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:10.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.471+0000 7fab5cff9640 1 -- 172.21.15.62:0/3161013682 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559b2e579d60 con 0x559b2e626e50 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.471+0000 7fab5cff9640 10 client.18048 handle_command_reply: tid=0 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 client.18048 shutdown 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 -- 172.21.15.62:0/3161013682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 msgr2=0x559b2e6147a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e6147a0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009fd0 tx=0x7fab6c004ab0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 -- 172.21.15.62:0/3161013682 shutdown_connections 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b2e626e50 0x559b2e6292a0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e6147a0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e614260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 --2- 172.21.15.62:0/3161013682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e616fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 -- 172.21.15.62:0/3161013682 >> 172.21.15.62:0/3161013682 conn(0x559b2e608a30 msgr2=0x559b2e60e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab5cff9640 0 client.18048 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 -- 172.21.15.62:0/3161013682 shutdown_connections 2024-09-15T23:27:10.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 1 -- 172.21.15.62:0/3161013682 wait complete. 2024-09-15T23:27:10.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.472+0000 7fab7d733740 20 client.18048 trim_cache size 0 max 16384 2024-09-15T23:27:10.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.489+0000 7fab7d733740 1 Processor -- start 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.489+0000 7fab7d733740 1 -- start start 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab7d733740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e623190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab7d733740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e628e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab7d733740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e60f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab7d733740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fab741b37f0 con 0x559b2e617680 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab7d733740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab741b10c0 con 0x559b2e60c4a0 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab7d733740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fab741aef10 con 0x559b2e60cdc0 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab790f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e623190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab790f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e623190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52844/0 (socket says 172.21.15.62:52844) 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab798f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e60f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab790f6640 1 -- 172.21.15.62:0/392790796 learned_addr learned my addr 172.21.15.62:0/392790796 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab790f6640 1 -- 172.21.15.62:0/392790796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 msgr2=0x559b2e628e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:10.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab788f5640 1 --2- 172.21.15.62:0/392790796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e628e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab790f6640 1 --2- 172.21.15.62:0/392790796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e628e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab790f6640 1 -- 172.21.15.62:0/392790796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 msgr2=0x559b2e60f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab790f6640 1 --2- 172.21.15.62:0/392790796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e60f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab790f6640 1 -- 172.21.15.62:0/392790796 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b2e613210 con 0x559b2e60c4a0 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab798f7640 1 --2- 172.21.15.62:0/392790796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e60f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab790f6640 1 --2- 172.21.15.62:0/392790796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e623190 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fab70004650 tx=0x7fab7002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a1b50b106314930 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.490+0000 7fab5effd640 1 -- 172.21.15.62:0/392790796 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70049040 con 0x559b2e60c4a0 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab5effd640 1 -- 172.21.15.62:0/392790796 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab7003b580 con 0x559b2e60c4a0 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab5effd640 1 -- 172.21.15.62:0/392790796 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7003bcb0 con 0x559b2e60c4a0 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab7d733740 1 -- 172.21.15.62:0/392790796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 msgr2=0x559b2e623190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab7d733740 1 --2- 172.21.15.62:0/392790796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e623190 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fab70004650 tx=0x7fab7002a480 comp rx=0 tx=0).stop 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab7d733740 1 -- 172.21.15.62:0/392790796 shutdown_connections 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab7d733740 1 --2- 172.21.15.62:0/392790796 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e60f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab7d733740 1 --2- 172.21.15.62:0/392790796 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e628e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab7d733740 1 --2- 172.21.15.62:0/392790796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e623190 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab7d733740 1 -- 172.21.15.62:0/392790796 >> 172.21.15.62:0/392790796 conn(0x559b2e608a30 msgr2=0x559b2e60f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab7d733740 1 -- 172.21.15.62:0/392790796 shutdown_connections 2024-09-15T23:27:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.491+0000 7fab7d733740 1 -- 172.21.15.62:0/392790796 wait complete. 2024-09-15T23:27:10.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab7d733740 1 -- start start 2024-09-15T23:27:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab7d733740 1 Processor -- start 2024-09-15T23:27:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab7d733740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e61f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab7d733740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e61fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab7d733740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e61aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab7d733740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fab741af580 con 0x559b2e617680 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab7d733740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab741b09f0 con 0x559b2e60c4a0 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab7d733740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fab741afc50 con 0x559b2e60cdc0 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab7d733740 10 client.0 _ll_register_callbacks cb 0x7fab6c009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab790f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e61f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab798f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e61aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab790f6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e61f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52852/0 (socket says 172.21.15.62:52852) 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.492+0000 7fab790f6640 1 -- 172.21.15.62:0/453587744 learned_addr learned my addr 172.21.15.62:0/453587744 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab788f5640 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e61fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab790f6640 1 -- 172.21.15.62:0/453587744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 msgr2=0x559b2e61fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab790f6640 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e61fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab790f6640 1 -- 172.21.15.62:0/453587744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 msgr2=0x559b2e61aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab790f6640 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e61aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab790f6640 1 -- 172.21.15.62:0/453587744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b2e60cbf0 con 0x559b2e60c4a0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab798f7640 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e61aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab790f6640 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e61f620 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fab700bc5b0 tx=0x7fab700f3970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=435eece20ca96b08 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab5cff9640 1 -- 172.21.15.62:0/453587744 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab700c7cf0 con 0x559b2e60c4a0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab5cff9640 1 -- 172.21.15.62:0/453587744 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fab700ba6c0 con 0x559b2e60c4a0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab7d733740 1 -- 172.21.15.62:0/453587744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559b2e629630 con 0x559b2e60c4a0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab5cff9640 1 -- 172.21.15.62:0/453587744 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fab700bbf80 con 0x559b2e60c4a0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab7d733740 10 client.27629 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab7d733740 1 -- 172.21.15.62:0/453587744 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559b2e60cbf0 con 0x559b2e60c4a0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.493+0000 7fab5cff9640 1 -- 172.21.15.62:0/453587744 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559b2e629630 con 0x559b2e60c4a0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab5cff9640 1 -- 172.21.15.62:0/453587744 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fab700b8c90 con 0x559b2e60c4a0 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab7d733740 10 client.27629 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab7d733740 10 client.27629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:10.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab7d733740 20 client.27629 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab7d733740 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b2e691710 0x559b2e693b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab7d733740 4 client.27629 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab7d733740 1 -- 172.21.15.62:0/453587744 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559b2e4f6210 con 0x559b2e691710 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab788f5640 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b2e691710 0x559b2e693b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab5cff9640 10 client.27629 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab788f5640 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b2e691710 0x559b2e693b00 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0b4e3c2e4da05f0 server_cookie=52c31b7743757289 in_seq=0 out_seq=0 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab5cff9640 1 -- 172.21.15.62:0/453587744 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559b2e4f6210 con 0x559b2e691710 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab5cff9640 10 client.27629 handle_command_reply: tid=0 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.494+0000 7fab7d733740 1 client.27629 shutdown 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 -- 172.21.15.62:0/453587744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 msgr2=0x559b2e61f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e61f620 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fab700bc5b0 tx=0x7fab700f3970 comp rx=0 tx=0).stop 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 -- 172.21.15.62:0/453587744 shutdown_connections 2024-09-15T23:27:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b2e691710 0x559b2e693b00 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b2e617680 0x559b2e61aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b2e60cdc0 0x559b2e61fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab5cff9640 0 client.27629 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 --2- 172.21.15.62:0/453587744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b2e60c4a0 0x559b2e61f620 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 -- 172.21.15.62:0/453587744 >> 172.21.15.62:0/453587744 conn(0x559b2e608a30 msgr2=0x559b2e607b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 -- 172.21.15.62:0/453587744 shutdown_connections 2024-09-15T23:27:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 1 -- 172.21.15.62:0/453587744 wait complete. 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7d733740 20 client.27629 trim_cache size 0 max 16384 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stdout: "6983d6fe-08b9-4992-845c-f133ac2460b2": { 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "6983d6fe-08b9-4992-845c-f133ac2460b2", 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:10.503 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:10.504 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:27:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 -- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fab600a3240 msgr2=0x7fab600a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fab600a3240 0x7fab600a5700 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fab6c005fd0 tx=0x7fab6c004a40 comp rx=0 tx=0).stop 2024-09-15T23:27:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 -- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 msgr2=0x7fab7419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 0x7fab7419a650 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fab70006010 tx=0x7fab700042f0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 -- 172.21.15.62:0/460973935 shutdown_connections 2024-09-15T23:27:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fab600a3240 0x7fab600a5700 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab7410c180 0x7fab7419ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab741036c0 0x7fab7419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 --2- 172.21.15.62:0/460973935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74102da0 0x7fab7419a650 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.495+0000 7fab7b35a640 1 -- 172.21.15.62:0/460973935 >> 172.21.15.62:0/460973935 conn(0x7fab740fc7a0 msgr2=0x7fab741019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.496+0000 7fab7b35a640 1 -- 172.21.15.62:0/460973935 shutdown_connections 2024-09-15T23:27:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:10.496+0000 7fab7b35a640 1 -- 172.21.15.62:0/460973935 wait complete. 2024-09-15T23:27:11.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:11 smithi062 ceph-mon[30949]: pgmap v697: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 2.9 MiB/s wr, 247 op/s 2024-09-15T23:27:11.312 INFO:tasks.cephfs.filesystem:scrub status for tag:6983d6fe-08b9-4992-845c-f133ac2460b2 - {'path': '/', 'tag': '6983d6fe-08b9-4992-845c-f133ac2460b2', 'options': 'recursive,force'} 2024-09-15T23:27:11.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:11 smithi067 ceph-mon[35665]: pgmap v697: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 2.9 MiB/s wr, 247 op/s 2024-09-15T23:27:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:11 smithi148 ceph-mon[35610]: pgmap v697: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 2.9 MiB/s wr, 247 op/s 2024-09-15T23:27:12.019 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set 'de217ad0', quiesce took 8.1 seconds. Will release after: 13.750000000000002 2024-09-15T23:27:13.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:13 smithi062 ceph-mon[30949]: pgmap v698: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-09-15T23:27:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:13 smithi067 ceph-mon[35665]: pgmap v698: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-09-15T23:27:13.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:13 smithi148 ceph-mon[35610]: pgmap v698: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 3.2 MiB/s wr, 316 op/s 2024-09-15T23:27:14.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 655.0 s, 19.0 tps, lat 311.632 ms stddev 210.292 2024-09-15T23:27:15.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:15 smithi062 ceph-mon[30949]: pgmap v699: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.1 MiB/s wr, 392 op/s 2024-09-15T23:27:15.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:15 smithi067 ceph-mon[35665]: pgmap v699: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.1 MiB/s wr, 392 op/s 2024-09-15T23:27:15.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:15 smithi148 ceph-mon[35610]: pgmap v699: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.1 MiB/s wr, 392 op/s 2024-09-15T23:27:17.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:17 smithi062 ceph-mon[30949]: pgmap v700: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.3 MiB/s wr, 409 op/s 2024-09-15T23:27:17.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:17 smithi067 ceph-mon[35665]: pgmap v700: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.3 MiB/s wr, 409 op/s 2024-09-15T23:27:17.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:17 smithi148 ceph-mon[35610]: pgmap v700: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.3 MiB/s wr, 409 op/s 2024-09-15T23:27:19.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:19 smithi062 ceph-mon[30949]: pgmap v701: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 3.6 MiB/s wr, 372 op/s 2024-09-15T23:27:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:19 smithi067 ceph-mon[35665]: pgmap v701: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 3.6 MiB/s wr, 372 op/s 2024-09-15T23:27:19.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:19 smithi148 ceph-mon[35610]: pgmap v701: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 3.6 MiB/s wr, 372 op/s 2024-09-15T23:27:19.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 660.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:27:21.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:21 smithi062 ceph-mon[30949]: pgmap v702: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.4 MiB/s wr, 289 op/s 2024-09-15T23:27:21.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:21 smithi067 ceph-mon[35665]: pgmap v702: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.4 MiB/s wr, 289 op/s 2024-09-15T23:27:21.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:21 smithi148 ceph-mon[35610]: pgmap v702: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.4 MiB/s wr, 289 op/s 2024-09-15T23:27:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.559+0000 7f1ca471a640 1 -- 172.21.15.62:0/3798294579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c101c50 msgr2=0x7f1c9c1020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.559+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3798294579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c101c50 0x7f1c9c1020d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c009900 tx=0x7f1c8c02a410 comp rx=0 tx=0).stop 2024-09-15T23:27:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.559+0000 7f1ca471a640 1 -- 172.21.15.62:0/3798294579 shutdown_connections 2024-09-15T23:27:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.559+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3798294579 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c102610 0x7f1c9c108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.562 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.559+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3798294579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c101c50 0x7f1c9c1020d0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.559+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3798294579 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c100a60 0x7f1c9c100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.559+0000 7f1ca471a640 1 -- 172.21.15.62:0/3798294579 >> 172.21.15.62:0/3798294579 conn(0x7f1c9c0fc1e0 msgr2=0x7f1c9c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.559+0000 7f1ca471a640 1 -- 172.21.15.62:0/3798294579 shutdown_connections 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.559+0000 7f1ca471a640 1 -- 172.21.15.62:0/3798294579 wait complete. 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca471a640 1 Processor -- start 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca471a640 1 -- start start 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca471a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c100a60 0x7f1c9c10b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca471a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c101c50 0x7f1c9c109a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca471a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c102610 0x7f1c9c109f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca471a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10d890 con 0x7f1c9c102610 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca471a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10d710 con 0x7f1c9c101c50 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca471a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10da10 con 0x7f1c9c100a60 2024-09-15T23:27:22.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca2cb7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c102610 0x7f1c9c109f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca2cb7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c102610 0x7f1c9c109f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44206/0 (socket says 172.21.15.62:44206) 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca2cb7640 1 -- 172.21.15.62:0/3437884823 learned_addr learned my addr 172.21.15.62:0/3437884823 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c100a60 0x7f1c9c10b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c101c50 0x7f1c9c109a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca2cb7640 1 -- 172.21.15.62:0/3437884823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c100a60 msgr2=0x7f1c9c10b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca2cb7640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c100a60 0x7f1c9c10b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca2cb7640 1 -- 172.21.15.62:0/3437884823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c101c50 msgr2=0x7f1c9c109a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca2cb7640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c101c50 0x7f1c9c109a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.560+0000 7f1ca2cb7640 1 -- 172.21.15.62:0/3437884823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c10a7f0 con 0x7f1c9c102610 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.561+0000 7f1ca2cb7640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c102610 0x7f1c9c109f60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400eb10 tx=0x7f1c9400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.561+0000 7f1c937fe640 1 -- 172.21.15.62:0/3437884823 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94004b20 con 0x7f1c9c102610 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.561+0000 7f1c937fe640 1 -- 172.21.15.62:0/3437884823 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c94004cc0 con 0x7f1c9c102610 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.561+0000 7f1ca471a640 1 -- 172.21.15.62:0/3437884823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c1a1ee0 con 0x7f1c9c102610 2024-09-15T23:27:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.561+0000 7f1c937fe640 1 -- 172.21.15.62:0/3437884823 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94010760 con 0x7f1c9c102610 2024-09-15T23:27:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.561+0000 7f1ca471a640 1 -- 172.21.15.62:0/3437884823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c1a2340 con 0x7f1c9c102610 2024-09-15T23:27:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1c937fe640 1 -- 172.21.15.62:0/3437884823 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f1c94005020 con 0x7f1c9c102610 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca74ae740 1 Processor -- start 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca74ae740 1 -- start start 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1c945b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 0x561dc1c995b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c99b80 0x561dc1c9c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561dc1bef2f0 con 0x561dc1c99b80 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561dc1b97270 con 0x561dc1c941b0 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561dc1beb2b0 con 0x561dc1c94af0 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca1cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c99b80 0x561dc1c9c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca1cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c99b80 0x561dc1c9c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44220/0 (socket says 172.21.15.62:44220) 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.564+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/2027615747 learned_addr learned my addr 172.21.15.62:0/2027615747 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/2027615747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1c945b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/2027615747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 0x561dc1c995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1c937fe640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1c880a32f0 0x7f1c880a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/2027615747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 msgr2=0x561dc1c995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/2027615747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 0x561dc1c995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/2027615747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 msgr2=0x561dc1c945b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/2027615747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1c945b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/2027615747 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dc1b96340 con 0x561dc1c99b80 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1c937fe640 1 -- 172.21.15.62:0/3437884823 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1c940d0480 con 0x7f1c9c102610 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1ca2cb7640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1c880a32f0 0x7f1c880a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/2027615747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c99b80 0x561dc1c9c040 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c005c40 tx=0x7f1c8c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f07c7bbd37d763 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1c90ff9640 1 -- 172.21.15.62:0/2027615747 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c035040 con 0x561dc1c99b80 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1c90ff9640 1 -- 172.21.15.62:0/2027615747 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c03d920 con 0x561dc1c99b80 2024-09-15T23:27:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.565+0000 7f1c90ff9640 1 -- 172.21.15.62:0/2027615747 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c003b50 con 0x561dc1c99b80 2024-09-15T23:27:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca74ae740 1 -- 172.21.15.62:0/2027615747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c99b80 msgr2=0x561dc1c9c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca2cb7640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1c880a32f0 0x7f1c880a57b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1c94005ba0 tx=0x7f1c94087040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:22.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/2027615747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c99b80 0x561dc1c9c040 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c005c40 tx=0x7f1c8c000bc0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca74ae740 1 -- 172.21.15.62:0/2027615747 shutdown_connections 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/2027615747 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c99b80 0x561dc1c9c040 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/2027615747 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 0x561dc1c995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/2027615747 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1c945b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca74ae740 1 -- 172.21.15.62:0/2027615747 >> 172.21.15.62:0/2027615747 conn(0x561dc1c87010 msgr2=0x561dc1c87410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca74ae740 1 -- 172.21.15.62:0/2027615747 shutdown_connections 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.566+0000 7f1ca74ae740 1 -- 172.21.15.62:0/2027615747 wait complete. 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca74ae740 1 -- start start 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca74ae740 1 Processor -- start 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c941b0 0x561dc1c9e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c94af0 0x561dc1c9e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c99b80 0x561dc1ca5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561dc1c9caf0 con 0x561dc1c941b0 2024-09-15T23:27:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561dc1b78290 con 0x561dc1c94af0 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561dc1c9cc70 con 0x561dc1c99b80 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca74ae740 10 client.0 _ll_register_callbacks cb 0x561dc1ba9140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca1cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c941b0 0x561dc1c9e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca1cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c941b0 0x561dc1c9e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44234/0 (socket says 172.21.15.62:44234) 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/571286583 learned_addr learned my addr 172.21.15.62:0/571286583 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c94af0 0x561dc1c9e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c99b80 0x561dc1ca5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/571286583 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c99b80 msgr2=0x561dc1ca5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c99b80 0x561dc1ca5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/571286583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c94af0 msgr2=0x561dc1c9e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c94af0 0x561dc1c9e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/571286583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dc1902210 con 0x561dc1c941b0 2024-09-15T23:27:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.567+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c94af0 0x561dc1c9e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c941b0 0x561dc1c9e3d0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c001eb0 tx=0x7f1c8c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e4dbe2faaf8169 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1c85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1c85ffb640 1 -- 172.21.15.62:0/571286583 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c035070 con 0x561dc1c941b0 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1c85ffb640 1 -- 172.21.15.62:0/571286583 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1c8c0038d0 con 0x561dc1c941b0 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1ca74ae740 1 -- 172.21.15.62:0/571286583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c8c03d920 con 0x561dc1c941b0 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1c85ffb640 1 -- 172.21.15.62:0/571286583 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1c8c057050 con 0x561dc1c941b0 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1ca74ae740 10 client.18066 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1ca74ae740 1 -- 172.21.15.62:0/571286583 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561dc19ce620 con 0x561dc1c941b0 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1c85ffb640 1 -- 172.21.15.62:0/571286583 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c03d920 con 0x561dc1c941b0 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1c85ffb640 1 -- 172.21.15.62:0/571286583 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f1c8c03a040 con 0x561dc1c941b0 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1ca74ae740 10 client.18066 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.568+0000 7f1ca74ae740 10 client.18066 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.569+0000 7f1ca74ae740 20 client.18066 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.569+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561dc1ca9c50 0x561dc1cac040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.569+0000 7f1ca74ae740 4 client.18066 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.569+0000 7f1ca74ae740 1 -- 172.21.15.62:0/571286583 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dc1bfca20 con 0x561dc1ca9c50 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.569+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561dc1ca9c50 0x561dc1cac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.569+0000 7f1c85ffb640 10 client.18066 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.569+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561dc1ca9c50 0x561dc1cac040 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c12642cc694bced server_cookie=74111a1fcd51ac8d in_seq=0 out_seq=0 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1c85ffb640 1 -- 172.21.15.62:0/571286583 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561dc1bfca20 con 0x561dc1ca9c50 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1c85ffb640 10 client.18066 handle_command_reply: tid=0 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 client.18066 shutdown 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 -- 172.21.15.62:0/571286583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c941b0 msgr2=0x561dc1c9e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c941b0 0x561dc1c9e3d0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c001eb0 tx=0x7f1c8c0035f0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 -- 172.21.15.62:0/571286583 shutdown_connections 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561dc1ca9c50 0x561dc1cac040 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c99b80 0x561dc1ca5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c94af0 0x561dc1c9e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/571286583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c941b0 0x561dc1c9e3d0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1c85ffb640 0 client.18066 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 -- 172.21.15.62:0/571286583 >> 172.21.15.62:0/571286583 conn(0x561dc1c87010 msgr2=0x561dc1c88820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 -- 172.21.15.62:0/571286583 shutdown_connections 2024-09-15T23:27:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 1 -- 172.21.15.62:0/571286583 wait complete. 2024-09-15T23:27:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.570+0000 7f1ca74ae740 20 client.18066 trim_cache size 0 max 16384 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca74ae740 1 Processor -- start 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca74ae740 1 -- start start 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c87010 0x561dc1c9e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c94af0 0x561dc1c869b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1aad60 con 0x561dc1c94af0 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1a8630 con 0x561dc1c941b0 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1a6480 con 0x561dc1c87010 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca1cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c94af0 0x561dc1c869b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca1cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c94af0 0x561dc1c869b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44240/0 (socket says 172.21.15.62:44240) 2024-09-15T23:27:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/1793982042 learned_addr learned my addr 172.21.15.62:0/1793982042 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.585+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c87010 0x561dc1c9e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/1793982042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c87010 msgr2=0x561dc1c9e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c87010 0x561dc1c9e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/1793982042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 msgr2=0x561dc1ca58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/1793982042 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dc1c822d0 con 0x561dc1c94af0 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c94af0 0x561dc1c869b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c031610 tx=0x7f1c8c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59845de59c5138e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1c87fff640 1 -- 172.21.15.62:0/1793982042 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c035030 con 0x561dc1c94af0 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1c87fff640 1 -- 172.21.15.62:0/1793982042 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c041020 con 0x561dc1c94af0 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.586+0000 7f1c87fff640 1 -- 172.21.15.62:0/1793982042 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c049a10 con 0x561dc1c94af0 2024-09-15T23:27:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1793982042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c94af0 msgr2=0x561dc1c869b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c94af0 0x561dc1c869b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c031610 tx=0x7f1c8c02a410 comp rx=0 tx=0).stop 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1793982042 shutdown_connections 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c94af0 0x561dc1c869b0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1793982042 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c87010 0x561dc1c9e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1793982042 >> 172.21.15.62:0/1793982042 conn(0x561dc1c996d0 msgr2=0x561dc1c87ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1793982042 shutdown_connections 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1793982042 wait complete. 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 -- start start 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 Processor -- start 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c87010 0x561dc1ca32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 0x561dc1ca4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1a6af0 con 0x561dc1c87010 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1a7f60 con 0x561dc1c941b0 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c1a71c0 con 0x561dc1c94af0 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca74ae740 10 client.0 _ll_register_callbacks cb 0x7f1c940093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca1cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c87010 0x561dc1ca32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca1cb5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c87010 0x561dc1ca32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44252/0 (socket says 172.21.15.62:44252) 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.587+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/1623232536 learned_addr learned my addr 172.21.15.62:0/1623232536 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 0x561dc1ca4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/1623232536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 msgr2=0x561dc1ca4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 0x561dc1ca4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/1623232536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 msgr2=0x561dc1ca3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:22.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca1cb5640 1 -- 172.21.15.62:0/1623232536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dc1ca4e40 con 0x561dc1c87010 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca1cb5640 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c87010 0x561dc1ca32e0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c03d940 tx=0x7f1c8c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6aacfc415d5278 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1c85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1c85ffb640 1 -- 172.21.15.62:0/1623232536 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8c035040 con 0x561dc1c87010 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1c85ffb640 1 -- 172.21.15.62:0/1623232536 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f1c8c049790 con 0x561dc1c87010 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1623232536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c8c041020 con 0x561dc1c87010 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1c85ffb640 1 -- 172.21.15.62:0/1623232536 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1c8c070050 con 0x561dc1c87010 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca74ae740 10 client.18072 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.588+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1623232536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561dc1d14a80 con 0x561dc1c87010 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1c85ffb640 1 -- 172.21.15.62:0/1623232536 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c8c041020 con 0x561dc1c87010 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1c85ffb640 1 -- 172.21.15.62:0/1623232536 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f1c8c05b950 con 0x561dc1c87010 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1ca74ae740 10 client.18072 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:22.595 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1ca74ae740 10 client.18072 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1ca74ae740 20 client.18072 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561dc1c99b80 0x561dc1d173b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1ca74ae740 4 client.18072 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:27:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1623232536 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561dc1bf6d70 con 0x561dc1c99b80 2024-09-15T23:27:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561dc1c99b80 0x561dc1d173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1c85ffb640 10 client.18072 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:22.596 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.589+0000 7f1ca24b6640 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561dc1c99b80 0x561dc1d173b0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5e8ad5dc5fc5105 server_cookie=52ce99d7a8731a54 in_seq=0 out_seq=0 2024-09-15T23:27:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1c85ffb640 1 -- 172.21.15.62:0/1623232536 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561dc1bf6d70 con 0x561dc1c99b80 2024-09-15T23:27:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1c85ffb640 10 client.18072 handle_command_reply: tid=0 2024-09-15T23:27:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 client.18072 shutdown 2024-09-15T23:27:22.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1623232536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c87010 msgr2=0x561dc1ca32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c87010 0x561dc1ca32e0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c03d940 tx=0x7f1c8c005fd0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1623232536 shutdown_connections 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561dc1c99b80 0x561dc1d173b0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561dc1c94af0 0x561dc1ca4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561dc1c941b0 0x561dc1ca3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 --2- 172.21.15.62:0/1623232536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561dc1c87010 0x561dc1ca32e0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1c85ffb640 0 client.18072 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1623232536 >> 172.21.15.62:0/1623232536 conn(0x561dc1c996d0 msgr2=0x561dc1c886e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1623232536 shutdown_connections 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 1 -- 172.21.15.62:0/1623232536 wait complete. 2024-09-15T23:27:22.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.602+0000 7f1ca74ae740 20 client.18072 trim_cache size 0 max 16384 2024-09-15T23:27:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:27:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:27:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:27:22.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 -- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1c880a32f0 msgr2=0x7f1c880a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1c880a32f0 0x7f1c880a57b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1c94005ba0 tx=0x7f1c94087040 comp rx=0 tx=0).stop 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 -- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c102610 msgr2=0x7f1c9c109f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c102610 0x7f1c9c109f60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400eb10 tx=0x7f1c9400c2d0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 -- 172.21.15.62:0/3437884823 shutdown_connections 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f1c880a32f0 0x7f1c880a57b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c9c102610 0x7f1c9c109f60 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c9c101c50 0x7f1c9c109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 --2- 172.21.15.62:0/3437884823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1c9c100a60 0x7f1c9c10b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 -- 172.21.15.62:0/3437884823 >> 172.21.15.62:0/3437884823 conn(0x7f1c9c0fc1e0 msgr2=0x7f1c9c104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 -- 172.21.15.62:0/3437884823 shutdown_connections 2024-09-15T23:27:22.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:22.603+0000 7f1ca471a640 1 -- 172.21.15.62:0/3437884823 wait complete. 2024-09-15T23:27:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:23 smithi062 ceph-mon[30949]: pgmap v703: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 1.4 MiB/s wr, 179 op/s 2024-09-15T23:27:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:23 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:27:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:23 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:27:23.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:23 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:27:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:23 smithi067 ceph-mon[35665]: pgmap v703: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 1.4 MiB/s wr, 179 op/s 2024-09-15T23:27:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:23 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:27:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:23 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:27:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:23 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:27:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:23 smithi148 ceph-mon[35610]: pgmap v703: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 1.4 MiB/s wr, 179 op/s 2024-09-15T23:27:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:23 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:27:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:23 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:27:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:23 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:27:24.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 665.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:27:25.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:25 smithi062 ceph-mon[30949]: pgmap v704: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.1 MiB/s wr, 111 op/s 2024-09-15T23:27:25.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:25 smithi067 ceph-mon[35665]: pgmap v704: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.1 MiB/s wr, 111 op/s 2024-09-15T23:27:25.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:25 smithi148 ceph-mon[35610]: pgmap v704: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 1.1 MiB/s wr, 111 op/s 2024-09-15T23:27:25.770 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:27:26.042 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:26.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.505+0000 7fda4701b640 1 -- 172.21.15.62:0/1819313564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 msgr2=0x7fda40100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.505+0000 7fda4701b640 1 --2- 172.21.15.62:0/1819313564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 0x7fda40100ee0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fda34009930 tx=0x7fda3402a420 comp rx=0 tx=0).stop 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.505+0000 7fda4701b640 1 -- 172.21.15.62:0/1819313564 shutdown_connections 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.505+0000 7fda4701b640 1 --2- 172.21.15.62:0/1819313564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda40101420 0x7fda4010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.505+0000 7fda4701b640 1 --2- 172.21.15.62:0/1819313564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 0x7fda40100ee0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.505+0000 7fda4701b640 1 --2- 172.21.15.62:0/1819313564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda40106a80 0x7fda40106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.505+0000 7fda4701b640 1 -- 172.21.15.62:0/1819313564 >> 172.21.15.62:0/1819313564 conn(0x7fda400fc7a0 msgr2=0x7fda400febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.505+0000 7fda4701b640 1 -- 172.21.15.62:0/1819313564 shutdown_connections 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.505+0000 7fda4701b640 1 -- 172.21.15.62:0/1819313564 wait complete. 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda4701b640 1 Processor -- start 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda4701b640 1 -- start start 2024-09-15T23:27:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda4701b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 0x7fda4019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda4701b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda40101420 0x7fda40196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda4701b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda40106a80 0x7fda40196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda4701b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fda40110e80 con 0x7fda40100a80 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda4701b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fda40110d00 con 0x7fda40101420 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda4701b640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fda40111000 con 0x7fda40106a80 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda44db7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 0x7fda4019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda44db7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 0x7fda4019e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44258/0 (socket says 172.21.15.62:44258) 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.506+0000 7fda44db7640 1 -- 172.21.15.62:0/2456637515 learned_addr learned my addr 172.21.15.62:0/2456637515 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda3ffff640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda40101420 0x7fda40196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda455b8640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda40106a80 0x7fda40196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda44db7640 1 -- 172.21.15.62:0/2456637515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda40106a80 msgr2=0x7fda40196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda44db7640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda40106a80 0x7fda40196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda44db7640 1 -- 172.21.15.62:0/2456637515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda40101420 msgr2=0x7fda40196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda44db7640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda40101420 0x7fda40196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda44db7640 1 -- 172.21.15.62:0/2456637515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda40196d90 con 0x7fda40100a80 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda455b8640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda40106a80 0x7fda40196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda44db7640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 0x7fda4019e0c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fda3000d950 tx=0x7fda3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda3dffb640 1 -- 172.21.15.62:0/2456637515 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda30017070 con 0x7fda40100a80 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda3dffb640 1 -- 172.21.15.62:0/2456637515 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda30009c30 con 0x7fda40100a80 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda3dffb640 1 -- 172.21.15.62:0/2456637515 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3000f6b0 con 0x7fda40100a80 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.507+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda40197080 con 0x7fda40100a80 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.508+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda401b3d70 con 0x7fda40100a80 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.509+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda401021c0 con 0x7fda40100a80 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.510+0000 7fda3dffb640 1 -- 172.21.15.62:0/2456637515 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fda300109d0 con 0x7fda40100a80 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.510+0000 7fda3dffb640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda280a3240 0x7fda280a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.511+0000 7fda3ffff640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda280a3240 0x7fda280a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.511+0000 7fda3dffb640 1 -- 172.21.15.62:0/2456637515 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fda300cf880 con 0x7fda40100a80 2024-09-15T23:27:26.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.511+0000 7fda3ffff640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda280a3240 0x7fda280a5700 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fda34009900 tx=0x7fda340058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:26.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.515+0000 7fda3dffb640 1 -- 172.21.15.62:0/2456637515 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fda30009de0 con 0x7fda40100a80 2024-09-15T23:27:26.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.728+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda401b3fa0 con 0x7fda40100a80 2024-09-15T23:27:26.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.729+0000 7fda3dffb640 1 -- 172.21.15.62:0/2456637515 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7fda3009c1c0 con 0x7fda40100a80 2024-09-15T23:27:26.730 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:27:26.730 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:27:26.731 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:27:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda280a3240 msgr2=0x7fda280a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda280a3240 0x7fda280a5700 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fda34009900 tx=0x7fda340058d0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 msgr2=0x7fda4019e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 0x7fda4019e0c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fda3000d950 tx=0x7fda3000de10 comp rx=0 tx=0).stop 2024-09-15T23:27:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 shutdown_connections 2024-09-15T23:27:26.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fda280a3240 0x7fda280a5700 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fda40106a80 0x7fda40196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda40101420 0x7fda40196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 --2- 172.21.15.62:0/2456637515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda40100a80 0x7fda4019e0c0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:26.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 >> 172.21.15.62:0/2456637515 conn(0x7fda400fc7a0 msgr2=0x7fda4006a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:26.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 shutdown_connections 2024-09-15T23:27:26.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:26.732+0000 7fda4701b640 1 -- 172.21.15.62:0/2456637515 wait complete. 2024-09-15T23:27:27.377 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:27 smithi062 ceph-mon[30949]: pgmap v705: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 207 KiB/s wr, 35 op/s 2024-09-15T23:27:27.377 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:27 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2456637515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:27.412 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id de217ad0 --release --await' 2024-09-15T23:27:27.412 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id de217ad0 --release --await 2024-09-15T23:27:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:27 smithi067 ceph-mon[35665]: pgmap v705: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 207 KiB/s wr, 35 op/s 2024-09-15T23:27:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:27 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2456637515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:27 smithi148 ceph-mon[35610]: pgmap v705: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 207 KiB/s wr, 35 op/s 2024-09-15T23:27:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:27 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2456637515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:28.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.153+0000 7f55ea14a640 1 -- 172.21.15.62:0/2900303758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 msgr2=0x7f55e41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:28.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.153+0000 7f55ea14a640 1 --2- 172.21.15.62:0/2900303758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 0x7f55e41067e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f55cc009900 tx=0x7f55cc02a410 comp rx=0 tx=0).stop 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.153+0000 7f55ea14a640 1 -- 172.21.15.62:0/2900303758 shutdown_connections 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.153+0000 7f55ea14a640 1 --2- 172.21.15.62:0/2900303758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e4106f10 0x7f55e410ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.153+0000 7f55ea14a640 1 --2- 172.21.15.62:0/2900303758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 0x7f55e41067e0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.153+0000 7f55ea14a640 1 --2- 172.21.15.62:0/2900303758 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55e4101de0 0x7f55e41021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.153+0000 7f55ea14a640 1 -- 172.21.15.62:0/2900303758 >> 172.21.15.62:0/2900303758 conn(0x7f55e40fd8c0 msgr2=0x7f55e40ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.153+0000 7f55ea14a640 1 -- 172.21.15.62:0/2900303758 shutdown_connections 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.153+0000 7f55ea14a640 1 -- 172.21.15.62:0/2900303758 wait complete. 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55ea14a640 1 Processor -- start 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55ea14a640 1 -- start start 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55ea14a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e4101de0 0x7f55e40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55ea14a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 0x7f55e4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55ea14a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55e4106f10 0x7f55e4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55ea14a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55e4111f30 con 0x7f55e4102790 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55ea14a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55e4111db0 con 0x7f55e4101de0 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55ea14a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41120b0 con 0x7f55e4106f10 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 0x7f55e4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 0x7f55e4074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44286/0 (socket says 172.21.15.62:44286) 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55dbfff640 1 -- 172.21.15.62:0/414412477 learned_addr learned my addr 172.21.15.62:0/414412477 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.154+0000 7f55e37fe640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e4101de0 0x7f55e40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.155+0000 7f55e3fff640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55e4106f10 0x7f55e4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.155+0000 7f55dbfff640 1 -- 172.21.15.62:0/414412477 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55e4106f10 msgr2=0x7f55e4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.155+0000 7f55dbfff640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55e4106f10 0x7f55e4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.155+0000 7f55dbfff640 1 -- 172.21.15.62:0/414412477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e4101de0 msgr2=0x7f55e40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.155+0000 7f55dbfff640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e4101de0 0x7f55e40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.155+0000 7f55dbfff640 1 -- 172.21.15.62:0/414412477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e41a6850 con 0x7f55e4102790 2024-09-15T23:27:28.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.155+0000 7f55e3fff640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55e4106f10 0x7f55e4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.155+0000 7f55dbfff640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 0x7f55e4074a30 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f55cc006010 tx=0x7f55cc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.155+0000 7f55e17fa640 1 -- 172.21.15.62:0/414412477 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc03f030 con 0x7f55e4102790 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.156+0000 7f55e17fa640 1 -- 172.21.15.62:0/414412477 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55cc003ce0 con 0x7f55e4102790 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.156+0000 7f55ea14a640 1 -- 172.21.15.62:0/414412477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e41a6ae0 con 0x7f55e4102790 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.156+0000 7f55e17fa640 1 -- 172.21.15.62:0/414412477 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc035070 con 0x7f55e4102790 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.156+0000 7f55ea14a640 1 -- 172.21.15.62:0/414412477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e41a6fc0 con 0x7f55e4102790 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.158+0000 7f55e17fa640 1 -- 172.21.15.62:0/414412477 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f55cc042030 con 0x7f55e4102790 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.158+0000 7f55ec523740 1 Processor -- start 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.158+0000 7f55ec523740 1 -- start start 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.158+0000 7f55ec523740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b1c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.158+0000 7f55ec523740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b26d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.158+0000 7f55ec523740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b29630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55ec523740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563a40a81620 con 0x563a40b27240 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55ec523740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563a40a28e50 con 0x563a40b1c980 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55ec523740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563a40a7d5e0 con 0x563a40b1c060 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55e17fa640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55d00a3390 0x7f55d00a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b29630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55e37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b26d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55e37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b26d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:39144/0 (socket says 172.21.15.62:39144) 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b29630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44288/0 (socket says 172.21.15.62:44288) 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55e37fe640 1 -- 172.21.15.62:0/1878412525 learned_addr learned my addr 172.21.15.62:0/1878412525 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55e3fff640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55d00a3390 0x7f55d00a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55e3fff640 1 --2- 172.21.15.62:0/1878412525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b1c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55e17fa640 1 -- 172.21.15.62:0/414412477 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f55cc0f02e0 con 0x7f55e4102790 2024-09-15T23:27:28.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55dbfff640 1 -- 172.21.15.62:0/1878412525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 msgr2=0x563a40b1c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55dbfff640 1 --2- 172.21.15.62:0/1878412525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b1c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55dbfff640 1 -- 172.21.15.62:0/1878412525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 msgr2=0x563a40b26d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55dbfff640 1 --2- 172.21.15.62:0/1878412525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b26d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55dbfff640 1 -- 172.21.15.62:0/1878412525 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a40a19d40 con 0x563a40b27240 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55e37fe640 1 --2- 172.21.15.62:0/1878412525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b26d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55e3fff640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55d00a3390 0x7f55d00a5850 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009270 tx=0x7f55dc009de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.159+0000 7f55dbfff640 1 --2- 172.21.15.62:0/1878412525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b29630 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0edab0 tx=0x7f55cc005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d286fac06d586595 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55d9ffb640 1 -- 172.21.15.62:0/1878412525 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc049070 con 0x563a40b27240 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55d9ffb640 1 -- 172.21.15.62:0/1878412525 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55cc0bcc80 con 0x563a40b27240 2024-09-15T23:27:28.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55d9ffb640 1 -- 172.21.15.62:0/1878412525 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0ba910 con 0x563a40b27240 2024-09-15T23:27:28.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55ec523740 1 -- 172.21.15.62:0/1878412525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 msgr2=0x563a40b29630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:28.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55ec523740 1 --2- 172.21.15.62:0/1878412525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b29630 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0edab0 tx=0x7f55cc005e50 comp rx=0 tx=0).stop 2024-09-15T23:27:28.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55ec523740 1 -- 172.21.15.62:0/1878412525 shutdown_connections 2024-09-15T23:27:28.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55ec523740 1 --2- 172.21.15.62:0/1878412525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b29630 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55ec523740 1 --2- 172.21.15.62:0/1878412525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b26d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55ec523740 1 --2- 172.21.15.62:0/1878412525 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b1c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55ec523740 1 -- 172.21.15.62:0/1878412525 >> 172.21.15.62:0/1878412525 conn(0x563a40b185f0 msgr2=0x563a40b189d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:28.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.160+0000 7f55ec523740 1 -- 172.21.15.62:0/1878412525 shutdown_connections 2024-09-15T23:27:28.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 1 -- 172.21.15.62:0/1878412525 wait complete. 2024-09-15T23:27:28.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 1 -- start start 2024-09-15T23:27:28.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 1 Processor -- start 2024-09-15T23:27:28.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b24200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b24740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b24c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563a40b29bd0 con 0x563a40b1c980 2024-09-15T23:27:28.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563a40a09f10 con 0x563a40b27240 2024-09-15T23:27:28.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563a40b29d50 con 0x563a40b1c060 2024-09-15T23:27:28.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.161+0000 7f55ec523740 10 client.0 _ll_register_callbacks cb 0x563a40a2ddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:28.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b24740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b24740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44294/0 (socket says 172.21.15.62:44294) 2024-09-15T23:27:28.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55dbfff640 1 -- 172.21.15.62:0/1410038329 learned_addr learned my addr 172.21.15.62:0/1410038329 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55e37fe640 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b24200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55e3fff640 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b24c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55dbfff640 1 -- 172.21.15.62:0/1410038329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 msgr2=0x563a40b24200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55dbfff640 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b24200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55dbfff640 1 -- 172.21.15.62:0/1410038329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 msgr2=0x563a40b24c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55dbfff640 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b24c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55dbfff640 1 -- 172.21.15.62:0/1410038329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a40790580 con 0x563a40b1c980 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55e37fe640 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b24200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55e3fff640 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b24c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55dbfff640 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b24740 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f55cc005ab0 tx=0x7f55cc0bbf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b9e65154989df2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1410038329 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0d5060 con 0x563a40b1c980 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1410038329 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f55cc0b8bf0 con 0x563a40b1c980 2024-09-15T23:27:28.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55ec523740 1 -- 172.21.15.62:0/1410038329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55cc0bcc80 con 0x563a40b1c980 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1410038329 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f55cc049070 con 0x563a40b1c980 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55ec523740 10 client.18090 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.162+0000 7f55ec523740 1 -- 172.21.15.62:0/1410038329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563a4085a960 con 0x563a40b1c980 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1410038329 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55cc0bcc80 con 0x563a40b1c980 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1410038329 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f55cc0c4d60 con 0x563a40b1c980 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55ec523740 10 client.18090 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55ec523740 10 client.18090 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55ec523740 20 client.18090 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55ec523740 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563a40b369b0 0x563a40b38da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55ec523740 4 client.18090 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55ec523740 1 -- 172.21.15.62:0/1410038329 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a40a8eb90 con 0x563a40b369b0 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55e37fe640 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563a40b369b0 0x563a40b38da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55e37fe640 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563a40b369b0 0x563a40b38da0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37db2f10ce720a82 server_cookie=98ad1f2cb901309 in_seq=0 out_seq=0 2024-09-15T23:27:28.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.163+0000 7f55c2ffd640 10 client.18090 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1410038329 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563a40a8eb90 con 0x563a40b369b0 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55c2ffd640 10 client.18090 handle_command_reply: tid=0 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55ec523740 1 client.18090 shutdown 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55ec523740 1 -- 172.21.15.62:0/1410038329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 msgr2=0x563a40b24740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55ec523740 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b24740 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f55cc005ab0 tx=0x7f55cc0bbf50 comp rx=0 tx=0).stop 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55ec523740 1 -- 172.21.15.62:0/1410038329 shutdown_connections 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55ec523740 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563a40b369b0 0x563a40b38da0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55ec523740 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b24c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55ec523740 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b24740 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55ec523740 1 --2- 172.21.15.62:0/1410038329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b24200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55ec523740 1 -- 172.21.15.62:0/1410038329 >> 172.21.15.62:0/1410038329 conn(0x563a40b185f0 msgr2=0x563a40b1e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.164+0000 7f55c2ffd640 0 client.18090 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:28.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.165+0000 7f55ec523740 1 -- 172.21.15.62:0/1410038329 shutdown_connections 2024-09-15T23:27:28.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.165+0000 7f55ec523740 1 -- 172.21.15.62:0/1410038329 wait complete. 2024-09-15T23:27:28.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.165+0000 7f55ec523740 20 client.18090 trim_cache size 0 max 16384 2024-09-15T23:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55ec523740 1 Processor -- start 2024-09-15T23:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55ec523740 1 -- start start 2024-09-15T23:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55ec523740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b24730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55ec523740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b24c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55ec523740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b3e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55ec523740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41b3640 con 0x563a40b27240 2024-09-15T23:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55ec523740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41b0f10 con 0x563a40b1c980 2024-09-15T23:27:28.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55ec523740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41aed60 con 0x563a40b1c060 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b3e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b3e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44308/0 (socket says 172.21.15.62:44308) 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55e3fff640 1 -- 172.21.15.62:0/3488748888 learned_addr learned my addr 172.21.15.62:0/3488748888 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55dbfff640 1 --2- 172.21.15.62:0/3488748888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b24c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55e37fe640 1 --2- 172.21.15.62:0/3488748888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b24730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55e3fff640 1 -- 172.21.15.62:0/3488748888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 msgr2=0x563a40b24730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55e3fff640 1 --2- 172.21.15.62:0/3488748888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b24730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55e3fff640 1 -- 172.21.15.62:0/3488748888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 msgr2=0x563a40b24c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55e3fff640 1 --2- 172.21.15.62:0/3488748888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b24c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.182+0000 7f55e3fff640 1 -- 172.21.15.62:0/3488748888 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a40b19280 con 0x563a40b27240 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55e3fff640 1 --2- 172.21.15.62:0/3488748888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b3e510 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009750 tx=0x7f55dc01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82de0e2f7574ba03 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55d97fa640 1 -- 172.21.15.62:0/3488748888 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc003e10 con 0x563a40b27240 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55d97fa640 1 -- 172.21.15.62:0/3488748888 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55dc004510 con 0x563a40b27240 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55d97fa640 1 -- 172.21.15.62:0/3488748888 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc012bd0 con 0x563a40b27240 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55ec523740 1 -- 172.21.15.62:0/3488748888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 msgr2=0x563a40b3e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55ec523740 1 --2- 172.21.15.62:0/3488748888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b3e510 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009750 tx=0x7f55dc01c000 comp rx=0 tx=0).stop 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55ec523740 1 -- 172.21.15.62:0/3488748888 shutdown_connections 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55ec523740 1 --2- 172.21.15.62:0/3488748888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b27240 0x563a40b3e510 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55ec523740 1 --2- 172.21.15.62:0/3488748888 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b1c980 0x563a40b24c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55ec523740 1 --2- 172.21.15.62:0/3488748888 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b24730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.183+0000 7f55ec523740 1 -- 172.21.15.62:0/3488748888 >> 172.21.15.62:0/3488748888 conn(0x563a40b185f0 msgr2=0x563a40b24020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 -- 172.21.15.62:0/3488748888 shutdown_connections 2024-09-15T23:27:28.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 -- 172.21.15.62:0/3488748888 wait complete. 2024-09-15T23:27:28.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 -- start start 2024-09-15T23:27:28.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 Processor -- start 2024-09-15T23:27:28.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b2ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b2f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b2be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41af3d0 con 0x563a40b1c980 2024-09-15T23:27:28.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41b0840 con 0x563a40b27240 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f55e41afaa0 con 0x563a40b1c060 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.184+0000 7f55ec523740 10 client.0 _ll_register_callbacks cb 0x7f55cc0314b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b2ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b2f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e37fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b2ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:54984/0 (socket says 172.21.15.62:54984) 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e37fe640 1 -- 172.21.15.62:0/1186232617 learned_addr learned my addr 172.21.15.62:0/1186232617 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e3fff640 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b2be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e37fe640 1 -- 172.21.15.62:0/1186232617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 msgr2=0x563a40b2be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e37fe640 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b2be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e37fe640 1 -- 172.21.15.62:0/1186232617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 msgr2=0x563a40b2f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e37fe640 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b2f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e37fe640 1 -- 172.21.15.62:0/1186232617 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a40b2c3a0 con 0x563a40b1c060 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55dbfff640 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b2f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:28.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e3fff640 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b2be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55e37fe640 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b2ee90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f55d40106b0 tx=0x7f55d4010b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d258bf7c37bd43e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1186232617 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d402a060 con 0x563a40b1c060 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55ec523740 1 -- 172.21.15.62:0/1186232617 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55dc004510 con 0x563a40b1c060 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.185+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1186232617 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f55d400fd90 con 0x563a40b1c060 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1186232617 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f55d4031050 con 0x563a40b1c060 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55ec523740 10 client.27397 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55ec523740 1 -- 172.21.15.62:0/1186232617 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563a40ba1cd0 con 0x563a40b1c060 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1186232617 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55dc004510 con 0x563a40b1c060 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1186232617 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f55d401c400 con 0x563a40b1c060 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55ec523740 10 client.27397 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55ec523740 10 client.27397 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:27:28.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55ec523740 20 client.27397 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:28.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55ec523740 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563a40ba2450 0x563a40ba4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:28.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55ec523740 4 client.27397 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "de217ad0", "release": true, "await": true}] 2024-09-15T23:27:28.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55ec523740 1 -- 172.21.15.62:0/1186232617 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "de217ad0", "release": true, "await": true}) -- 0x563a40a890a0 con 0x563a40ba2450 2024-09-15T23:27:28.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.186+0000 7f55dbfff640 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563a40ba2450 0x563a40ba4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:28.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.187+0000 7f55dbfff640 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563a40ba2450 0x563a40ba4870 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=599b7cdd77d8d30a server_cookie=97a609c74d025144 in_seq=0 out_seq=0 2024-09-15T23:27:28.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.187+0000 7f55c2ffd640 10 client.27397 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55c2ffd640 1 -- 172.21.15.62:0/1186232617 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x563a40a890a0 con 0x563a40ba2450 2024-09-15T23:27:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55c2ffd640 10 client.27397 handle_command_reply: tid=0 2024-09-15T23:27:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 client.27397 shutdown 2024-09-15T23:27:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 -- 172.21.15.62:0/1186232617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 msgr2=0x563a40b2ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b2ee90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f55d40106b0 tx=0x7f55d4010b70 comp rx=0 tx=0).stop 2024-09-15T23:27:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 -- 172.21.15.62:0/1186232617 shutdown_connections 2024-09-15T23:27:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563a40ba2450 0x563a40ba4870 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563a40b27240 0x563a40b2be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563a40b1c980 0x563a40b2f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55c2ffd640 0 client.27397 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:28.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 --2- 172.21.15.62:0/1186232617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563a40b1c060 0x563a40b2ee90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 -- 172.21.15.62:0/1186232617 >> 172.21.15.62:0/1186232617 conn(0x563a40b185f0 msgr2=0x563a40b26280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:28.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.218+0000 7f55ec523740 1 -- 172.21.15.62:0/1186232617 shutdown_connections 2024-09-15T23:27:28.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.221+0000 7f55ec523740 1 -- 172.21.15.62:0/1186232617 wait complete. 2024-09-15T23:27:28.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.223+0000 7f55ec523740 20 client.27397 trim_cache size 0 max 16384 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 35, 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 44, 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "de217ad0": { 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 44, 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:27:28.225 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 141.8, 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:28.226 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:27:28.227 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:27:28.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 -- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55d00a3390 msgr2=0x7f55d00a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:28.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55d00a3390 0x7f55d00a5850 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009270 tx=0x7f55dc009de0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 -- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 msgr2=0x7f55e4074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:28.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 0x7f55e4074a30 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f55cc006010 tx=0x7f55cc0042f0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 -- 172.21.15.62:0/414412477 shutdown_connections 2024-09-15T23:27:28.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55d00a3390 0x7f55d00a5850 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55e4106f10 0x7f55e4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55e4102790 0x7f55e4074a30 secure :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f55cc006010 tx=0x7f55cc0042f0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 --2- 172.21.15.62:0/414412477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55e4101de0 0x7f55e40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:28.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.224+0000 7f55ea14a640 1 -- 172.21.15.62:0/414412477 >> 172.21.15.62:0/414412477 conn(0x7f55e40fd8c0 msgr2=0x7f55e40fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:28.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.227+0000 7f55ea14a640 1 -- 172.21.15.62:0/414412477 shutdown_connections 2024-09-15T23:27:28.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:28.227+0000 7f55ea14a640 1 -- 172.21.15.62:0/414412477 wait complete. 2024-09-15T23:27:28.924 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set 'de217ad0', total seconds elapsed: 25.0 2024-09-15T23:27:29.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:29 smithi062 ceph-mon[30949]: pgmap v706: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 61 KiB/s wr, 20 op/s 2024-09-15T23:27:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:29 smithi067 ceph-mon[35665]: pgmap v706: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 61 KiB/s wr, 20 op/s 2024-09-15T23:27:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:29 smithi148 ceph-mon[35610]: pgmap v706: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 61 KiB/s wr, 20 op/s 2024-09-15T23:27:29.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 670.0 s, 189.8 tps, lat 2069.961 ms stddev 5701.228 2024-09-15T23:27:31.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:31 smithi062 ceph-mon[30949]: pgmap v707: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 119 KiB/s wr, 26 op/s 2024-09-15T23:27:31.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:31 smithi067 ceph-mon[35665]: pgmap v707: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 119 KiB/s wr, 26 op/s 2024-09-15T23:27:31.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:31 smithi148 ceph-mon[35610]: pgmap v707: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 119 KiB/s wr, 26 op/s 2024-09-15T23:27:33.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:33 smithi062 ceph-mon[30949]: pgmap v708: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 347 KiB/s wr, 57 op/s 2024-09-15T23:27:33.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:33 smithi067 ceph-mon[35665]: pgmap v708: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 347 KiB/s wr, 57 op/s 2024-09-15T23:27:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:33 smithi148 ceph-mon[35610]: pgmap v708: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 347 KiB/s wr, 57 op/s 2024-09-15T23:27:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 675.0 s, 607.0 tps, lat 166.554 ms stddev 188.289 2024-09-15T23:27:35.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:35 smithi062 ceph-mon[30949]: pgmap v709: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 2.3 MiB/s wr, 153 op/s 2024-09-15T23:27:35.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:35 smithi067 ceph-mon[35665]: pgmap v709: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 2.3 MiB/s wr, 153 op/s 2024-09-15T23:27:35.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:35 smithi148 ceph-mon[35610]: pgmap v709: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 2.3 MiB/s wr, 153 op/s 2024-09-15T23:27:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.754+0000 7f3630503640 1 -- 172.21.15.62:0/2131077509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f36280736c0 msgr2=0x7f3628119130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.754+0000 7f3630503640 1 --2- 172.21.15.62:0/2131077509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f36280736c0 0x7f3628119130 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3620009ca0 tx=0x7f362002a8f0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.757+0000 7f3630503640 1 -- 172.21.15.62:0/2131077509 shutdown_connections 2024-09-15T23:27:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.757+0000 7f3630503640 1 --2- 172.21.15.62:0/2131077509 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f36280736c0 0x7f3628119130 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.757+0000 7f3630503640 1 --2- 172.21.15.62:0/2131077509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3628072d20 0x7f3628073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.757+0000 7f3630503640 1 --2- 172.21.15.62:0/2131077509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3628109700 0x7f3628109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.757+0000 7f3630503640 1 -- 172.21.15.62:0/2131077509 >> 172.21.15.62:0/2131077509 conn(0x7f362806ece0 msgr2=0x7f3628071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.760+0000 7f3630503640 1 -- 172.21.15.62:0/2131077509 shutdown_connections 2024-09-15T23:27:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.760+0000 7f3630503640 1 -- 172.21.15.62:0/2131077509 wait complete. 2024-09-15T23:27:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.761+0000 7f3630503640 1 Processor -- start 2024-09-15T23:27:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.761+0000 7f3630503640 1 -- start start 2024-09-15T23:27:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.761+0000 7f3630503640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3628072d20 0x7f36281188b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.766 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.763+0000 7f3630503640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f36280736c0 0x7f3628118df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.767+0000 7f362da9e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f36280736c0 0x7f3628118df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.769+0000 7f3630503640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3628109700 0x7f3628116ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.769+0000 7f3630503640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3628111370 con 0x7f3628109700 2024-09-15T23:27:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.769+0000 7f3630503640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f36281111f0 con 0x7f3628072d20 2024-09-15T23:27:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.769+0000 7f3630503640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f36281114f0 con 0x7f36280736c0 2024-09-15T23:27:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.769+0000 7f362e29f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3628072d20 0x7f36281188b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.769+0000 7f362eaa0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3628109700 0x7f3628116ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.769+0000 7f362eaa0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3628109700 0x7f3628116ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50842/0 (socket says 172.21.15.62:50842) 2024-09-15T23:27:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.769+0000 7f362eaa0640 1 -- 172.21.15.62:0/3315108866 learned_addr learned my addr 172.21.15.62:0/3315108866 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.776+0000 7f362e29f640 1 -- 172.21.15.62:0/3315108866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f36280736c0 msgr2=0x7f3628118df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.776+0000 7f362e29f640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f36280736c0 0x7f3628118df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.776+0000 7f362e29f640 1 -- 172.21.15.62:0/3315108866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3628109700 msgr2=0x7f3628116ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.776+0000 7f362e29f640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3628109700 0x7f3628116ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.776+0000 7f362e29f640 1 -- 172.21.15.62:0/3315108866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3628117590 con 0x7f3628072d20 2024-09-15T23:27:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.777+0000 7f362e29f640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3628072d20 0x7f36281188b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3618009fd0 tx=0x7f361800b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.780+0000 7f361f7fe640 1 -- 172.21.15.62:0/3315108866 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618015020 con 0x7f3628072d20 2024-09-15T23:27:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.781+0000 7f361f7fe640 1 -- 172.21.15.62:0/3315108866 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3618011d10 con 0x7f3628072d20 2024-09-15T23:27:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.781+0000 7f361f7fe640 1 -- 172.21.15.62:0/3315108866 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361800f2e0 con 0x7f3628072d20 2024-09-15T23:27:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.781+0000 7f3630503640 1 -- 172.21.15.62:0/3315108866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3628117880 con 0x7f3628072d20 2024-09-15T23:27:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.781+0000 7f3630503640 1 -- 172.21.15.62:0/3315108866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36281b7930 con 0x7f3628072d20 2024-09-15T23:27:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.787+0000 7f363329c740 1 Processor -- start 2024-09-15T23:27:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.789+0000 7f361f7fe640 1 -- 172.21.15.62:0/3315108866 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3618018020 con 0x7f3628072d20 2024-09-15T23:27:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.790+0000 7f361f7fe640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36140a3530 0x7f36140a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.790+0000 7f361f7fe640 1 -- 172.21.15.62:0/3315108866 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f36180d03d0 con 0x7f3628072d20 2024-09-15T23:27:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.792+0000 7f362eaa0640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36140a3530 0x7f36140a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f363329c740 1 -- start start 2024-09-15T23:27:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f363329c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f06da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f363329c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f0bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f363329c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f0e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f363329c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fe05e6a2f0 con 0x55fe05f072e0 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f363329c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe05e12270 con 0x55fe05f0c370 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f363329c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55fe05e662b0 con 0x55fe05f069a0 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f362da9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f0bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f362da9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f0bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50850/0 (socket says 172.21.15.62:50850) 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f362e29f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f06da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.793+0000 7f362da9e640 1 -- 172.21.15.62:0/367179003 learned_addr learned my addr 172.21.15.62:0/367179003 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.794+0000 7f362eaa0640 1 --2- 172.21.15.62:0/367179003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f0e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.794+0000 7f362e29f640 1 -- 172.21.15.62:0/367179003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 msgr2=0x55fe05f0e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.794+0000 7f362e29f640 1 --2- 172.21.15.62:0/367179003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f0e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.794+0000 7f362e29f640 1 -- 172.21.15.62:0/367179003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 msgr2=0x55fe05f0bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.794+0000 7f362e29f640 1 --2- 172.21.15.62:0/367179003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f0bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.794+0000 7f362e29f640 1 -- 172.21.15.62:0/367179003 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe05e11340 con 0x55fe05f069a0 2024-09-15T23:27:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.794+0000 7f362e29f640 1 --2- 172.21.15.62:0/367179003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f06da0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f36180cdaf0 tx=0x7f361809c150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6b01cdc4dafa3d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.794+0000 7f361cbfe640 1 -- 172.21.15.62:0/367179003 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36180b5070 con 0x55fe05f069a0 2024-09-15T23:27:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.795+0000 7f361cbfe640 1 -- 172.21.15.62:0/367179003 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3618098040 con 0x55fe05f069a0 2024-09-15T23:27:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.795+0000 7f361cbfe640 1 -- 172.21.15.62:0/367179003 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3618098c80 con 0x55fe05f069a0 2024-09-15T23:27:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.796+0000 7f363329c740 1 -- 172.21.15.62:0/367179003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 msgr2=0x55fe05f06da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.796+0000 7f363329c740 1 --2- 172.21.15.62:0/367179003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f06da0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f36180cdaf0 tx=0x7f361809c150 comp rx=0 tx=0).stop 2024-09-15T23:27:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.796+0000 7f363329c740 1 -- 172.21.15.62:0/367179003 shutdown_connections 2024-09-15T23:27:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.796+0000 7f363329c740 1 --2- 172.21.15.62:0/367179003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f0e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.796+0000 7f363329c740 1 --2- 172.21.15.62:0/367179003 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f0bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.796+0000 7f363329c740 1 --2- 172.21.15.62:0/367179003 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f06da0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.796+0000 7f363329c740 1 -- 172.21.15.62:0/367179003 >> 172.21.15.62:0/367179003 conn(0x55fe05f02010 msgr2=0x55fe05f02410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.796+0000 7f362eaa0640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36140a3530 0x7f36140a59f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f362002f570 tx=0x7f3620009880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.797+0000 7f363329c740 1 -- 172.21.15.62:0/367179003 shutdown_connections 2024-09-15T23:27:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.797+0000 7f363329c740 1 -- 172.21.15.62:0/367179003 wait complete. 2024-09-15T23:27:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.798+0000 7f363329c740 1 -- start start 2024-09-15T23:27:35.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.798+0000 7f363329c740 1 Processor -- start 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.798+0000 7f363329c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f10bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.798+0000 7f363329c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f11130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.798+0000 7f363329c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f18100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.798+0000 7f363329c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fe05f0f340 con 0x55fe05f072e0 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.798+0000 7f363329c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe05df3290 con 0x55fe05f0c370 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.798+0000 7f363329c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55fe05f0f4c0 con 0x55fe05f069a0 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.798+0000 7f363329c740 10 client.0 _ll_register_callbacks cb 0x55fe05e24140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362e29f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f10bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362e29f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f10bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:48522/0 (socket says 172.21.15.62:48522) 2024-09-15T23:27:35.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362e29f640 1 -- 172.21.15.62:0/3288436045 learned_addr learned my addr 172.21.15.62:0/3288436045 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362eaa0640 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f18100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362e29f640 1 -- 172.21.15.62:0/3288436045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 msgr2=0x55fe05f18100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362e29f640 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f18100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362e29f640 1 -- 172.21.15.62:0/3288436045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 msgr2=0x55fe05f11130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362e29f640 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f11130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362e29f640 1 -- 172.21.15.62:0/3288436045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe05b7d210 con 0x55fe05f069a0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f362e29f640 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f10bf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f361809c5a0 tx=0x7f361809c610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=983ecc22a49f52d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f3611ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.799+0000 7f3611ffb640 1 -- 172.21.15.62:0/3288436045 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36180aa060 con 0x55fe05f069a0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f3611ffb640 1 -- 172.21.15.62:0/3288436045 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3618010510 con 0x55fe05f069a0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f363329c740 1 -- 172.21.15.62:0/3288436045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3618098040 con 0x55fe05f069a0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f3611ffb640 1 -- 172.21.15.62:0/3288436045 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f36180b9030 con 0x55fe05f069a0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f363329c740 10 client.27409 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f363329c740 1 -- 172.21.15.62:0/3288436045 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fe05c49620 con 0x55fe05f069a0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f3611ffb640 1 -- 172.21.15.62:0/3288436045 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3618098040 con 0x55fe05f069a0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f3611ffb640 1 -- 172.21.15.62:0/3288436045 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f361809ba60 con 0x55fe05f069a0 2024-09-15T23:27:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f363329c740 10 client.27409 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f363329c740 10 client.27409 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f363329c740 20 client.27409 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f363329c740 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe05f1c400 0x55fe05f1e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f363329c740 4 client.27409 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.800+0000 7f363329c740 1 -- 172.21.15.62:0/3288436045 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe05e77a20 con 0x55fe05f1c400 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.801+0000 7f362da9e640 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe05f1c400 0x55fe05f1e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.801+0000 7f362da9e640 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe05f1c400 0x55fe05f1e7f0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b9143e86dbe4ab2 server_cookie=6d05d492ff786391 in_seq=0 out_seq=0 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.801+0000 7f3611ffb640 10 client.27409 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f3611ffb640 1 -- 172.21.15.62:0/3288436045 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fe05e77a20 con 0x55fe05f1c400 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f3611ffb640 10 client.27409 handle_command_reply: tid=0 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 client.27409 shutdown 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 -- 172.21.15.62:0/3288436045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 msgr2=0x55fe05f10bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f10bf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f361809c5a0 tx=0x7f361809c610 comp rx=0 tx=0).stop 2024-09-15T23:27:35.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 -- 172.21.15.62:0/3288436045 shutdown_connections 2024-09-15T23:27:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe05f1c400 0x55fe05f1e7f0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f18100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f3611ffb640 0 client.27409 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f11130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 --2- 172.21.15.62:0/3288436045 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f10bf0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 -- 172.21.15.62:0/3288436045 >> 172.21.15.62:0/3288436045 conn(0x55fe05f02010 msgr2=0x55fe05f03820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 -- 172.21.15.62:0/3288436045 shutdown_connections 2024-09-15T23:27:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 1 -- 172.21.15.62:0/3288436045 wait complete. 2024-09-15T23:27:35.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.802+0000 7f363329c740 20 client.27409 trim_cache size 0 max 16384 2024-09-15T23:27:35.825 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.819+0000 7f363329c740 1 Processor -- start 2024-09-15T23:27:35.825 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.819+0000 7f363329c740 1 -- start start 2024-09-15T23:27:35.825 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.819+0000 7f363329c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f069a0 0x55fe05f110b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f363329c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f072e0 0x55fe05f180c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f363329c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f0c370 0x55fe05f02ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f363329c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f36281c4570 con 0x55fe05f069a0 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f363329c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f36281c1e40 con 0x55fe05f072e0 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362e29f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f069a0 0x55fe05f110b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362e29f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f069a0 0x55fe05f110b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50860/0 (socket says 172.21.15.62:50860) 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362e29f640 1 -- 172.21.15.62:0/2647250163 learned_addr learned my addr 172.21.15.62:0/2647250163 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362da9e640 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f072e0 0x55fe05f180c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f363329c740 1 -- 172.21.15.62:0/2647250163 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f36281bfc90 con 0x55fe05f0c370 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362eaa0640 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f0c370 0x55fe05f02ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362e29f640 1 -- 172.21.15.62:0/2647250163 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f0c370 msgr2=0x55fe05f02ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.826 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362e29f640 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f0c370 0x55fe05f02ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362e29f640 1 -- 172.21.15.62:0/2647250163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f072e0 msgr2=0x55fe05f180c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362e29f640 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f072e0 0x55fe05f180c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362e29f640 1 -- 172.21.15.62:0/2647250163 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe05f077c0 con 0x55fe05f069a0 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362da9e640 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f072e0 0x55fe05f180c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f362e29f640 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f069a0 0x55fe05f110b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3618098a70 tx=0x7f361809c150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b252ae4bdb2a73a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.820+0000 7f3613fff640 1 -- 172.21.15.62:0/2647250163 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36180aa040 con 0x55fe05f069a0 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f3613fff640 1 -- 172.21.15.62:0/2647250163 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3618098040 con 0x55fe05f069a0 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f3613fff640 1 -- 172.21.15.62:0/2647250163 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36180b5040 con 0x55fe05f069a0 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f363329c740 1 -- 172.21.15.62:0/2647250163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f069a0 msgr2=0x55fe05f110b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f363329c740 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f069a0 0x55fe05f110b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3618098a70 tx=0x7f361809c150 comp rx=0 tx=0).stop 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f363329c740 1 -- 172.21.15.62:0/2647250163 shutdown_connections 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f363329c740 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f0c370 0x55fe05f02ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f363329c740 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f072e0 0x55fe05f180c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f363329c740 1 --2- 172.21.15.62:0/2647250163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f069a0 0x55fe05f110b0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f363329c740 1 -- 172.21.15.62:0/2647250163 >> 172.21.15.62:0/2647250163 conn(0x55fe05f02010 msgr2=0x55fe05f109a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f363329c740 1 -- 172.21.15.62:0/2647250163 shutdown_connections 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.821+0000 7f363329c740 1 -- 172.21.15.62:0/2647250163 wait complete. 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f363329c740 1 -- start start 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f363329c740 1 Processor -- start 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f363329c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f159d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f363329c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f15f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f363329c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f16f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f363329c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f36281c0300 con 0x55fe05f072e0 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f363329c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f36281c1770 con 0x55fe05f0c370 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f363329c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f36281c09d0 con 0x55fe05f069a0 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f363329c740 10 client.0 _ll_register_callbacks cb 0x7f36180b3070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:35.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362da9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f15f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362e29f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f159d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362da9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f15f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50862/0 (socket says 172.21.15.62:50862) 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362da9e640 1 -- 172.21.15.62:0/166759115 learned_addr learned my addr 172.21.15.62:0/166759115 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362eaa0640 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f16f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362da9e640 1 -- 172.21.15.62:0/166759115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 msgr2=0x55fe05f159d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362da9e640 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f159d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362da9e640 1 -- 172.21.15.62:0/166759115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 msgr2=0x55fe05f16f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362da9e640 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f16f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362da9e640 1 -- 172.21.15.62:0/166759115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe05f17530 con 0x55fe05f072e0 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.822+0000 7f362e29f640 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f362da9e640 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f15f10 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3624000c00 tx=0x7f3624010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b98415ca39d789 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f3611ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f3611ffb640 1 -- 172.21.15.62:0/166759115 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3624005460 con 0x55fe05f072e0 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f3611ffb640 1 -- 172.21.15.62:0/166759115 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f36240046c0 con 0x55fe05f072e0 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f363329c740 1 -- 172.21.15.62:0/166759115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3618098040 con 0x55fe05f072e0 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f3611ffb640 1 -- 172.21.15.62:0/166759115 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f362402c050 con 0x55fe05f072e0 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f363329c740 10 client.18108 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f363329c740 1 -- 172.21.15.62:0/166759115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fe05f872d0 con 0x55fe05f072e0 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f3611ffb640 1 -- 172.21.15.62:0/166759115 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3618098040 con 0x55fe05f072e0 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f3611ffb640 1 -- 172.21.15.62:0/166759115 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f362401c990 con 0x55fe05f072e0 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f363329c740 10 client.18108 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f363329c740 10 client.18108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f363329c740 20 client.18108 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f363329c740 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe05f87c60 0x55fe05f8a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f363329c740 4 client.18108 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.823+0000 7f363329c740 1 -- 172.21.15.62:0/166759115 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fe05f0c0f0 con 0x55fe05f87c60 2024-09-15T23:27:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.824+0000 7f362e29f640 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe05f87c60 0x55fe05f8a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.824+0000 7f362e29f640 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe05f87c60 0x55fe05f8a080 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f969a61b2892862 server_cookie=6e04f5b5572ca010 in_seq=0 out_seq=0 2024-09-15T23:27:35.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.824+0000 7f3611ffb640 10 client.18108 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:35.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f3611ffb640 1 -- 172.21.15.62:0/166759115 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fe05f0c0f0 con 0x55fe05f87c60 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f3611ffb640 10 client.18108 handle_command_reply: tid=0 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 client.18108 shutdown 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 -- 172.21.15.62:0/166759115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 msgr2=0x55fe05f15f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f15f10 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3624000c00 tx=0x7f3624010800 comp rx=0 tx=0).stop 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 -- 172.21.15.62:0/166759115 shutdown_connections 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fe05f87c60 0x55fe05f8a080 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe05f0c370 0x55fe05f16f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f3611ffb640 0 client.18108 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fe05f072e0 0x55fe05f15f10 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 --2- 172.21.15.62:0/166759115 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fe05f069a0 0x55fe05f159d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 -- 172.21.15.62:0/166759115 >> 172.21.15.62:0/166759115 conn(0x55fe05f02010 msgr2=0x55fe05f009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 -- 172.21.15.62:0/166759115 shutdown_connections 2024-09-15T23:27:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 1 -- 172.21.15.62:0/166759115 wait complete. 2024-09-15T23:27:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.839+0000 7f363329c740 20 client.18108 trim_cache size 0 max 16384 2024-09-15T23:27:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:27:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:27:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:27:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:27:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 -- 172.21.15.62:0/3315108866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36140a3530 msgr2=0x7f36140a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36140a3530 0x7f36140a59f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f362002f570 tx=0x7f3620009880 comp rx=0 tx=0).stop 2024-09-15T23:27:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 -- 172.21.15.62:0/3315108866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3628072d20 msgr2=0x7f36281188b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3628072d20 0x7f36281188b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3618009fd0 tx=0x7f361800b870 comp rx=0 tx=0).stop 2024-09-15T23:27:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 -- 172.21.15.62:0/3315108866 shutdown_connections 2024-09-15T23:27:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f36140a3530 0x7f36140a59f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3628109700 0x7f3628116ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.843 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f36280736c0 0x7f3628118df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 --2- 172.21.15.62:0/3315108866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3628072d20 0x7f36281188b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 -- 172.21.15.62:0/3315108866 >> 172.21.15.62:0/3315108866 conn(0x7f362806ece0 msgr2=0x7f362810c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 -- 172.21.15.62:0/3315108866 shutdown_connections 2024-09-15T23:27:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:35.840+0000 7f3630503640 1 -- 172.21.15.62:0/3315108866 wait complete. 2024-09-15T23:27:37.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:37 smithi062 ceph-mon[30949]: pgmap v710: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 3.4 MiB/s wr, 265 op/s 2024-09-15T23:27:37.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:37 smithi067 ceph-mon[35665]: pgmap v710: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 3.4 MiB/s wr, 265 op/s 2024-09-15T23:27:37.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:37 smithi148 ceph-mon[35610]: pgmap v710: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 3.4 MiB/s wr, 265 op/s 2024-09-15T23:27:39.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:39 smithi062 ceph-mon[30949]: pgmap v711: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 4.6 MiB/s wr, 368 op/s 2024-09-15T23:27:39.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:39 smithi067 ceph-mon[35665]: pgmap v711: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 4.6 MiB/s wr, 368 op/s 2024-09-15T23:27:39.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:39 smithi148 ceph-mon[35610]: pgmap v711: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 4.6 MiB/s wr, 368 op/s 2024-09-15T23:27:39.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 680.0 s, 610.8 tps, lat 158.788 ms stddev 176.892 2024-09-15T23:27:41.313 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:27:41.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:41 smithi062 ceph-mon[30949]: pgmap v712: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 6.2 MiB/s wr, 473 op/s 2024-09-15T23:27:41.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:41 smithi067 ceph-mon[35665]: pgmap v712: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 6.2 MiB/s wr, 473 op/s 2024-09-15T23:27:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:41 smithi148 ceph-mon[35610]: pgmap v712: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 6.2 MiB/s wr, 473 op/s 2024-09-15T23:27:41.838 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:42.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.412+0000 7f24912ea640 1 -- 172.21.15.62:0/347088684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c109450 msgr2=0x7f248c109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:42.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.412+0000 7f24912ea640 1 --2- 172.21.15.62:0/347088684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c109450 0x7f248c109830 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f247c005c50 tx=0x7f247c02b590 comp rx=0 tx=0).stop 2024-09-15T23:27:42.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.416+0000 7f24912ea640 1 -- 172.21.15.62:0/347088684 shutdown_connections 2024-09-15T23:27:42.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.416+0000 7f24912ea640 1 --2- 172.21.15.62:0/347088684 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c074990 0x7f248c07b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.416+0000 7f24912ea640 1 --2- 172.21.15.62:0/347088684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c109e00 0x7f248c0742d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.416+0000 7f24912ea640 1 --2- 172.21.15.62:0/347088684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c109450 0x7f248c109830 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.416+0000 7f24912ea640 1 -- 172.21.15.62:0/347088684 >> 172.21.15.62:0/347088684 conn(0x7f248c06e140 msgr2=0x7f248c06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:42.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.424+0000 7f24912ea640 1 -- 172.21.15.62:0/347088684 shutdown_connections 2024-09-15T23:27:42.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.437+0000 7f24912ea640 1 -- 172.21.15.62:0/347088684 wait complete. 2024-09-15T23:27:42.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.438+0000 7f24912ea640 1 Processor -- start 2024-09-15T23:27:42.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.438+0000 7f24912ea640 1 -- start start 2024-09-15T23:27:42.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.438+0000 7f24912ea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c074990 0x7f248c07b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f24912ea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c109450 0x7f248c0762d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f24912ea640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c109e00 0x7f248c076810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f24912ea640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f248c084ea0 con 0x7f248c109e00 2024-09-15T23:27:42.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f24912ea640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f248c084d20 con 0x7f248c109450 2024-09-15T23:27:42.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f24912ea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f248c085020 con 0x7f248c074990 2024-09-15T23:27:42.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f248a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c109450 0x7f248c0762d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f248a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c109450 0x7f248c0762d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34446/0 (socket says 172.21.15.62:34446) 2024-09-15T23:27:42.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f248a7fc640 1 -- 172.21.15.62:0/2418159925 learned_addr learned my addr 172.21.15.62:0/2418159925 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:42.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f248b7fe640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c109e00 0x7f248c076810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f248a7fc640 1 -- 172.21.15.62:0/2418159925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c074990 msgr2=0x7f248c07b240 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f248a7fc640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c074990 0x7f248c07b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f248a7fc640 1 -- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c109e00 msgr2=0x7f248c076810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f248a7fc640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c109e00 0x7f248c076810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.439+0000 7f248a7fc640 1 -- 172.21.15.62:0/2418159925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f248c0770a0 con 0x7f248c109450 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.440+0000 7f248a7fc640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c109450 0x7f248c0762d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f248000b450 tx=0x7f248000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.440+0000 7f2473fff640 1 -- 172.21.15.62:0/2418159925 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480015020 con 0x7f248c109450 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.440+0000 7f2473fff640 1 -- 172.21.15.62:0/2418159925 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2480011d70 con 0x7f248c109450 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.440+0000 7f2473fff640 1 -- 172.21.15.62:0/2418159925 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248000f340 con 0x7f248c109450 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.440+0000 7f24912ea640 1 -- 172.21.15.62:0/2418159925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f248c137350 con 0x7f248c109450 2024-09-15T23:27:42.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.440+0000 7f24912ea640 1 -- 172.21.15.62:0/2418159925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f248c1378c0 con 0x7f248c109450 2024-09-15T23:27:42.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.444+0000 7f24936c3740 1 Processor -- start 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.447+0000 7f2473fff640 1 -- 172.21.15.62:0/2418159925 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2480018020 con 0x7f248c109450 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.448+0000 7f2473fff640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24740a3380 0x7f24740a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.448+0000 7f2473fff640 1 -- 172.21.15.62:0/2418159925 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f24800cf830 con 0x7f248c109450 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.449+0000 7f248b7fe640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24740a3380 0x7f24740a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.449+0000 7f24936c3740 1 -- start start 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.449+0000 7f24936c3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 0x557a53742590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.449+0000 7f24936c3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53742ad0 0x557a53742f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.450+0000 7f24936c3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.450+0000 7f24936c3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557a536a82e0 con 0x557a53743dc0 2024-09-15T23:27:42.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.450+0000 7f24936c3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557a5364fce0 con 0x557a5374d430 2024-09-15T23:27:42.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.450+0000 7f24936c3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557a536a42a0 con 0x557a53742ad0 2024-09-15T23:27:42.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.450+0000 7f248b7fe640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24740a3380 0x7f24740a5840 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2484009ec0 tx=0x7f2484009430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:42.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.450+0000 7f248b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.450+0000 7f248b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34448/0 (socket says 172.21.15.62:34448) 2024-09-15T23:27:42.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.450+0000 7f248b7fe640 1 -- 172.21.15.62:0/3403900766 learned_addr learned my addr 172.21.15.62:0/3403900766 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:42.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.451+0000 7f248a7fc640 1 --2- 172.21.15.62:0/3403900766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53742ad0 0x557a53742f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.454+0000 7f248affd640 1 --2- 172.21.15.62:0/3403900766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 0x557a53742590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.455+0000 7f248b7fe640 1 -- 172.21.15.62:0/3403900766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53742ad0 msgr2=0x557a53742f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:42.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.455+0000 7f248b7fe640 1 --2- 172.21.15.62:0/3403900766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53742ad0 0x557a53742f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.455+0000 7f248b7fe640 1 -- 172.21.15.62:0/3403900766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 msgr2=0x557a53742590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:42.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.455+0000 7f248b7fe640 1 --2- 172.21.15.62:0/3403900766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 0x557a53742590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.455+0000 7f248b7fe640 1 -- 172.21.15.62:0/3403900766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a53634440 con 0x557a5374d430 2024-09-15T23:27:42.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.460+0000 7f248b7fe640 1 --2- 172.21.15.62:0/3403900766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374f820 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f24840093a0 tx=0x7f2484011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5619ea7e26a9bce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:42.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.461+0000 7f24912ea640 1 -- 172.21.15.62:0/3403900766 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2484027070 con 0x557a5374d430 2024-09-15T23:27:42.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.461+0000 7f24912ea640 1 -- 172.21.15.62:0/3403900766 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2484017dd0 con 0x557a5374d430 2024-09-15T23:27:42.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.461+0000 7f24912ea640 1 -- 172.21.15.62:0/3403900766 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2484016600 con 0x557a5374d430 2024-09-15T23:27:42.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.466+0000 7f24936c3740 1 -- 172.21.15.62:0/3403900766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 msgr2=0x557a5374f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:42.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.466+0000 7f24936c3740 1 --2- 172.21.15.62:0/3403900766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374f820 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f24840093a0 tx=0x7f2484011000 comp rx=0 tx=0).stop 2024-09-15T23:27:42.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.472+0000 7f24936c3740 1 -- 172.21.15.62:0/3403900766 shutdown_connections 2024-09-15T23:27:42.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.472+0000 7f24936c3740 1 --2- 172.21.15.62:0/3403900766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374f820 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.472+0000 7f24936c3740 1 --2- 172.21.15.62:0/3403900766 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53742ad0 0x557a53742f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.472+0000 7f24936c3740 1 --2- 172.21.15.62:0/3403900766 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 0x557a53742590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.472+0000 7f24936c3740 1 -- 172.21.15.62:0/3403900766 >> 172.21.15.62:0/3403900766 conn(0x557a5373ed20 msgr2=0x557a5373f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:42.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.474+0000 7f24936c3740 1 -- 172.21.15.62:0/3403900766 shutdown_connections 2024-09-15T23:27:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.475+0000 7f24936c3740 1 -- 172.21.15.62:0/3403900766 wait complete. 2024-09-15T23:27:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.476+0000 7f24936c3740 1 -- start start 2024-09-15T23:27:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.476+0000 7f24936c3740 1 Processor -- start 2024-09-15T23:27:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.476+0000 7f24936c3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 0x557a5374cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.476+0000 7f24936c3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53743dc0 0x557a53749fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.477+0000 7f24936c3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.477+0000 7f24936c3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557a537502a0 con 0x557a53742ad0 2024-09-15T23:27:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.477+0000 7f24936c3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557a53663050 con 0x557a5374d430 2024-09-15T23:27:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.477+0000 7f24936c3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557a53750420 con 0x557a53743dc0 2024-09-15T23:27:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.477+0000 7f24936c3740 10 client.0 _ll_register_callbacks cb 0x557a536a3160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:42.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.478+0000 7f248a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53743dc0 0x557a53749fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.478+0000 7f248b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.478+0000 7f248a7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53743dc0 0x557a53749fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:34990/0 (socket says 172.21.15.62:34990) 2024-09-15T23:27:42.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.478+0000 7f248a7fc640 1 -- 172.21.15.62:0/3702504806 learned_addr learned my addr 172.21.15.62:0/3702504806 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:42.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.478+0000 7f248affd640 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 0x557a5374cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.479+0000 7f248affd640 1 -- 172.21.15.62:0/3702504806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53743dc0 msgr2=0x557a53749fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.479+0000 7f248affd640 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53743dc0 0x557a53749fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.479+0000 7f248affd640 1 -- 172.21.15.62:0/3702504806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 msgr2=0x557a5374a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.479+0000 7f248affd640 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.479+0000 7f248affd640 1 -- 172.21.15.62:0/3702504806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a534193c0 con 0x557a53742ad0 2024-09-15T23:27:42.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.480+0000 7f248affd640 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 0x557a5374cd30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f247c009d20 tx=0x7f247c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0de8e56e347bf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:42.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.481+0000 7f246eff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:42.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.481+0000 7f246eff5640 1 -- 172.21.15.62:0/3702504806 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247c040070 con 0x557a53742ad0 2024-09-15T23:27:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.481+0000 7f246eff5640 1 -- 172.21.15.62:0/3702504806 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f247c005440 con 0x557a53742ad0 2024-09-15T23:27:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.481+0000 7f24936c3740 1 -- 172.21.15.62:0/3702504806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2484017dd0 con 0x557a53742ad0 2024-09-15T23:27:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.482+0000 7f246eff5640 1 -- 172.21.15.62:0/3702504806 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f247c04a050 con 0x557a53742ad0 2024-09-15T23:27:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.482+0000 7f24936c3740 10 client.18120 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.482+0000 7f24936c3740 1 -- 172.21.15.62:0/3702504806 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557a5375bc50 con 0x557a53742ad0 2024-09-15T23:27:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.483+0000 7f246eff5640 1 -- 172.21.15.62:0/3702504806 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f247c0044a0 con 0x557a53742ad0 2024-09-15T23:27:42.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.484+0000 7f246eff5640 1 -- 172.21.15.62:0/3702504806 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f247c0046c0 con 0x557a53742ad0 2024-09-15T23:27:42.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.484+0000 7f24936c3740 10 client.18120 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:42.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.484+0000 7f24936c3740 10 client.18120 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:42.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.484+0000 7f24936c3740 20 client.18120 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:42.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.485+0000 7f24936c3740 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557a5375d110 0x557a5375f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.485+0000 7f24936c3740 4 client.18120 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:42.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.485+0000 7f24936c3740 1 -- 172.21.15.62:0/3702504806 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f2484017dd0 con 0x557a5375d110 2024-09-15T23:27:42.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.487+0000 7f248a7fc640 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557a5375d110 0x557a5375f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.490+0000 7f248a7fc640 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557a5375d110 0x557a5375f500 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ab1c0c20d06514b server_cookie=d0f580e487ef9661 in_seq=0 out_seq=0 2024-09-15T23:27:42.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.490+0000 7f246eff5640 10 client.18120 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:42.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.492+0000 7f246eff5640 1 -- 172.21.15.62:0/3702504806 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f2484017dd0 con 0x557a5375d110 2024-09-15T23:27:42.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.493+0000 7f246eff5640 10 client.18120 handle_command_reply: tid=0 2024-09-15T23:27:42.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.493+0000 7f24936c3740 1 client.18120 shutdown 2024-09-15T23:27:42.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.499+0000 7f24936c3740 1 -- 172.21.15.62:0/3702504806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 msgr2=0x557a5374cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:42.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.499+0000 7f24936c3740 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 0x557a5374cd30 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f247c009d20 tx=0x7f247c004270 comp rx=0 tx=0).stop 2024-09-15T23:27:42.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.503+0000 7f24936c3740 1 -- 172.21.15.62:0/3702504806 shutdown_connections 2024-09-15T23:27:42.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.503+0000 7f24936c3740 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557a5375d110 0x557a5375f500 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.503+0000 7f24936c3740 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a5374a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.503+0000 7f24936c3740 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53743dc0 0x557a53749fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.503+0000 7f24936c3740 1 --2- 172.21.15.62:0/3702504806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 0x557a5374cd30 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.503+0000 7f24936c3740 1 -- 172.21.15.62:0/3702504806 >> 172.21.15.62:0/3702504806 conn(0x557a5373ed20 msgr2=0x557a53744a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:42.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.503+0000 7f246eff5640 0 client.18120 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:42.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.503+0000 7f24936c3740 1 -- 172.21.15.62:0/3702504806 shutdown_connections 2024-09-15T23:27:42.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.503+0000 7f24936c3740 1 -- 172.21.15.62:0/3702504806 wait complete. 2024-09-15T23:27:42.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.505+0000 7f24936c3740 20 client.18120 trim_cache size 0 max 16384 2024-09-15T23:27:42.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.542+0000 7f24936c3740 1 Processor -- start 2024-09-15T23:27:42.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.542+0000 7f24936c3740 1 -- start start 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.543+0000 7f24936c3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a53742ad0 0x557a53759540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.543+0000 7f24936c3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 0x557a537488b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.543+0000 7f24936c3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a5374d430 0x557a5373ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.543+0000 7f24936c3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1d9c70 con 0x557a53743dc0 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.543+0000 7f24936c3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1d7540 con 0x557a53742ad0 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.543+0000 7f24936c3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1d5390 con 0x557a5374d430 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.543+0000 7f248affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a53742ad0 0x557a53759540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.543+0000 7f248affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a53742ad0 0x557a53759540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34468/0 (socket says 172.21.15.62:34468) 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.543+0000 7f248affd640 1 -- 172.21.15.62:0/1142910474 learned_addr learned my addr 172.21.15.62:0/1142910474 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:42.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.544+0000 7f248a7fc640 1 --2- 172.21.15.62:0/1142910474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 0x557a537488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.545+0000 7f248a7fc640 1 -- 172.21.15.62:0/1142910474 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a5374d430 msgr2=0x557a5373ddd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:27:42.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.545+0000 7f248a7fc640 1 --2- 172.21.15.62:0/1142910474 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a5374d430 0x557a5373ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.545+0000 7f248a7fc640 1 -- 172.21.15.62:0/1142910474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a53742ad0 msgr2=0x557a53759540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:42.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.545+0000 7f248a7fc640 1 --2- 172.21.15.62:0/1142910474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a53742ad0 0x557a53759540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.545+0000 7f248a7fc640 1 -- 172.21.15.62:0/1142910474 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a53748f90 con 0x557a53743dc0 2024-09-15T23:27:42.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.547+0000 7f248a7fc640 1 --2- 172.21.15.62:0/1142910474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 0x557a537488b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f248000be10 tx=0x7f24800cce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf86346049c5449 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:42.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.547+0000 7f2470ff9640 1 -- 172.21.15.62:0/1142910474 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480099d70 con 0x557a53743dc0 2024-09-15T23:27:42.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.548+0000 7f2470ff9640 1 -- 172.21.15.62:0/1142910474 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24800cd060 con 0x557a53743dc0 2024-09-15T23:27:42.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.548+0000 7f24936c3740 1 -- 172.21.15.62:0/1142910474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 msgr2=0x557a537488b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:42.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.548+0000 7f24936c3740 1 --2- 172.21.15.62:0/1142910474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 0x557a537488b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f248000be10 tx=0x7f24800cce20 comp rx=0 tx=0).stop 2024-09-15T23:27:42.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.550+0000 7f24936c3740 1 -- 172.21.15.62:0/1142910474 shutdown_connections 2024-09-15T23:27:42.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.550+0000 7f24936c3740 1 --2- 172.21.15.62:0/1142910474 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a5374d430 0x557a5373ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.550+0000 7f24936c3740 1 --2- 172.21.15.62:0/1142910474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53743dc0 0x557a537488b0 secure :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f248000be10 tx=0x7f24800cce20 comp rx=0 tx=0).stop 2024-09-15T23:27:42.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.550+0000 7f24936c3740 1 --2- 172.21.15.62:0/1142910474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a53742ad0 0x557a53759540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.550+0000 7f24936c3740 1 -- 172.21.15.62:0/1142910474 >> 172.21.15.62:0/1142910474 conn(0x557a5373ed20 msgr2=0x557a5373d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:42.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.553+0000 7f24936c3740 1 -- 172.21.15.62:0/1142910474 shutdown_connections 2024-09-15T23:27:42.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.553+0000 7f24936c3740 1 -- 172.21.15.62:0/1142910474 wait complete. 2024-09-15T23:27:42.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.554+0000 7f24936c3740 1 -- start start 2024-09-15T23:27:42.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.554+0000 7f24936c3740 1 Processor -- start 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.554+0000 7f24936c3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 0x557a537556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.554+0000 7f24936c3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a53755bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.554+0000 7f24936c3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53752680 0x557a53752b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.554+0000 7f24936c3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1d5a00 con 0x557a53742ad0 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.554+0000 7f24936c3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1d6e70 con 0x557a5374d430 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.554+0000 7f24936c3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f248c1d60d0 con 0x557a53752680 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.555+0000 7f24936c3740 10 client.0 _ll_register_callbacks cb 0x7f247c009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.555+0000 7f248a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a53755bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.555+0000 7f248b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53752680 0x557a53752b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.555+0000 7f248a7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a53755bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34478/0 (socket says 172.21.15.62:34478) 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.555+0000 7f248a7fc640 1 -- 172.21.15.62:0/3034858717 learned_addr learned my addr 172.21.15.62:0/3034858717 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.556+0000 7f248a7fc640 1 -- 172.21.15.62:0/3034858717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53752680 msgr2=0x557a53752b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.556+0000 7f248a7fc640 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53752680 0x557a53752b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.556+0000 7f248a7fc640 1 -- 172.21.15.62:0/3034858717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 msgr2=0x557a537556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.556+0000 7f248a7fc640 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 0x557a537556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.556+0000 7f248a7fc640 1 -- 172.21.15.62:0/3034858717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a53753040 con 0x557a5374d430 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.557+0000 7f248a7fc640 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a53755bf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f24800ccef0 tx=0x7f2480010950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43ef17de7c550e19 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.557+0000 7f246eff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.557+0000 7f246eff5640 1 -- 172.21.15.62:0/3034858717 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24800ba2b0 con 0x557a5374d430 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.558+0000 7f24936c3740 1 -- 172.21.15.62:0/3034858717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24800cd060 con 0x557a5374d430 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.558+0000 7f246eff5640 1 -- 172.21.15.62:0/3034858717 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2480017b50 con 0x557a5374d430 2024-09-15T23:27:42.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.558+0000 7f246eff5640 1 -- 172.21.15.62:0/3034858717 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f24800ba450 con 0x557a5374d430 2024-09-15T23:27:42.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.560+0000 7f24936c3740 10 client.27701 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:42.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.560+0000 7f24936c3740 1 -- 172.21.15.62:0/3034858717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557a537ea930 con 0x557a5374d430 2024-09-15T23:27:42.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.567+0000 7f246eff5640 1 -- 172.21.15.62:0/3034858717 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24800cd060 con 0x557a5374d430 2024-09-15T23:27:42.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.567+0000 7f246eff5640 1 -- 172.21.15.62:0/3034858717 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f24800977c0 con 0x557a5374d430 2024-09-15T23:27:42.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.567+0000 7f24936c3740 10 client.27701 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:42.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.567+0000 7f24936c3740 10 client.27701 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:42.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.567+0000 7f24936c3740 20 client.27701 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:42.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.567+0000 7f24936c3740 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557a537eae20 0x557a537ed240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:42.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.567+0000 7f24936c3740 4 client.27701 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:27:42.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.567+0000 7f24936c3740 1 -- 172.21.15.62:0/3034858717 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f2480098270 con 0x557a537eae20 2024-09-15T23:27:42.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.567+0000 7f248affd640 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557a537eae20 0x557a537ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:42.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.578+0000 7f248affd640 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557a537eae20 0x557a537ed240 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3346a7d57a4f7c1c server_cookie=6785a1c54e6a4460 in_seq=0 out_seq=0 2024-09-15T23:27:42.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.579+0000 7f246eff5640 10 client.27701 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:42.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.583+0000 7f246eff5640 1 -- 172.21.15.62:0/3034858717 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f2480098270 con 0x557a537eae20 2024-09-15T23:27:42.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.583+0000 7f246eff5640 10 client.27701 handle_command_reply: tid=0 2024-09-15T23:27:42.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.583+0000 7f24936c3740 1 client.27701 shutdown 2024-09-15T23:27:42.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.584+0000 7f24936c3740 1 -- 172.21.15.62:0/3034858717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 msgr2=0x557a53755bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:42.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.584+0000 7f24936c3740 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a53755bf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f24800ccef0 tx=0x7f2480010950 comp rx=0 tx=0).stop 2024-09-15T23:27:42.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.584+0000 7f24936c3740 1 -- 172.21.15.62:0/3034858717 shutdown_connections 2024-09-15T23:27:42.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.584+0000 7f24936c3740 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557a537eae20 0x557a537ed240 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.584+0000 7f24936c3740 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557a53752680 0x557a53752b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.584+0000 7f24936c3740 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557a5374d430 0x557a53755bf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.584+0000 7f24936c3740 1 --2- 172.21.15.62:0/3034858717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a53742ad0 0x557a537556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.584+0000 7f24936c3740 1 -- 172.21.15.62:0/3034858717 >> 172.21.15.62:0/3034858717 conn(0x557a5373ed20 msgr2=0x557a5373d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:42.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.584+0000 7f246eff5640 0 client.27701 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:42.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.590+0000 7f24936c3740 1 -- 172.21.15.62:0/3034858717 shutdown_connections 2024-09-15T23:27:42.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.590+0000 7f24936c3740 1 -- 172.21.15.62:0/3034858717 wait complete. 2024-09-15T23:27:42.591 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:27:42.591 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:27:42.592 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:27:42.593 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.590+0000 7f24936c3740 20 client.27701 trim_cache size 0 max 16384 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 -- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24740a3380 msgr2=0x7f24740a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24740a3380 0x7f24740a5840 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2484009ec0 tx=0x7f2484009430 comp rx=0 tx=0).stop 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 -- 172.21.15.62:0/2418159925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c109450 msgr2=0x7f248c0762d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c109450 0x7f248c0762d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f248000b450 tx=0x7f248000b910 comp rx=0 tx=0).stop 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 -- 172.21.15.62:0/2418159925 shutdown_connections 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f24740a3380 0x7f24740a5840 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f248c109e00 0x7f248c076810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f248c109450 0x7f248c0762d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 --2- 172.21.15.62:0/2418159925 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f248c074990 0x7f248c07b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:42.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.591+0000 7f2471ffb640 1 -- 172.21.15.62:0/2418159925 >> 172.21.15.62:0/2418159925 conn(0x7f248c06e140 msgr2=0x7f248c071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:42.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.592+0000 7f2471ffb640 1 -- 172.21.15.62:0/2418159925 shutdown_connections 2024-09-15T23:27:42.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:42.592+0000 7f2471ffb640 1 -- 172.21.15.62:0/2418159925 wait complete. 2024-09-15T23:27:43.303 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:43 smithi062 ceph-mon[30949]: pgmap v713: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.9 MiB/s wr, 524 op/s 2024-09-15T23:27:43.344 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:27:43.344 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:27:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:43 smithi067 ceph-mon[35665]: pgmap v713: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.9 MiB/s wr, 524 op/s 2024-09-15T23:27:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:43 smithi148 ceph-mon[35610]: pgmap v713: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 6.9 MiB/s wr, 524 op/s 2024-09-15T23:27:43.792 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.401+0000 7fd3027ee640 1 -- 172.21.15.62:0/3806799067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 msgr2=0x7fd2fc109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.401+0000 7fd3027ee640 1 --2- 172.21.15.62:0/3806799067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 0x7fd2fc109c80 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0074e0 tx=0x7fd2ec02b130 comp rx=0 tx=0).stop 2024-09-15T23:27:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.401+0000 7fd3027ee640 1 -- 172.21.15.62:0/3806799067 shutdown_connections 2024-09-15T23:27:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.401+0000 7fd3027ee640 1 --2- 172.21.15.62:0/3806799067 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2fc112d20 0x7fd2fc115110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.401+0000 7fd3027ee640 1 --2- 172.21.15.62:0/3806799067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc10a1c0 0x7fd2fc1126b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.401+0000 7fd3027ee640 1 --2- 172.21.15.62:0/3806799067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 0x7fd2fc109c80 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.401+0000 7fd3027ee640 1 -- 172.21.15.62:0/3806799067 >> 172.21.15.62:0/3806799067 conn(0x7fd2fc06e140 msgr2=0x7fd2fc06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:44.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.405+0000 7fd3027ee640 1 -- 172.21.15.62:0/3806799067 shutdown_connections 2024-09-15T23:27:44.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.405+0000 7fd3027ee640 1 -- 172.21.15.62:0/3806799067 wait complete. 2024-09-15T23:27:44.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.405+0000 7fd3027ee640 1 Processor -- start 2024-09-15T23:27:44.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd3027ee640 1 -- start start 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd3027ee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 0x7fd2fc1a7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd3027ee640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc10a1c0 0x7fd2fc1a7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd3027ee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2fc112d20 0x7fd2fc1abb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd3027ee640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc118670 con 0x7fd2fc1098a0 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd3027ee640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc1184f0 con 0x7fd2fc10a1c0 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd3027ee640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd2fc1187f0 con 0x7fd2fc112d20 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 0x7fd2fc1a7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 0x7fd2fc1a7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45600/0 (socket says 172.21.15.62:45600) 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd2fb7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc10a1c0 0x7fd2fc1a7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.406+0000 7fd2fbfff640 1 -- 172.21.15.62:0/516745429 learned_addr learned my addr 172.21.15.62:0/516745429 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.407+0000 7fd300d8b640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2fc112d20 0x7fd2fc1abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:44.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.407+0000 7fd2fbfff640 1 -- 172.21.15.62:0/516745429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2fc112d20 msgr2=0x7fd2fc1abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.407+0000 7fd2fbfff640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2fc112d20 0x7fd2fc1abb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.407+0000 7fd2fbfff640 1 -- 172.21.15.62:0/516745429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc10a1c0 msgr2=0x7fd2fc1a7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.407+0000 7fd2fbfff640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc10a1c0 0x7fd2fc1a7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.407+0000 7fd2fbfff640 1 -- 172.21.15.62:0/516745429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2fc1ac210 con 0x7fd2fc1098a0 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.407+0000 7fd300d8b640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2fc112d20 0x7fd2fc1abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.407+0000 7fd2fbfff640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 0x7fd2fc1a7240 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec000c00 tx=0x7fd2ec0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.408+0000 7fd2f97fa640 1 -- 172.21.15.62:0/516745429 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec007ce0 con 0x7fd2fc1098a0 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.408+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2fc1ac4a0 con 0x7fd2fc1098a0 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.408+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2fc19db40 con 0x7fd2fc1098a0 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.408+0000 7fd2f97fa640 1 -- 172.21.15.62:0/516745429 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2ec004e10 con 0x7fd2fc1098a0 2024-09-15T23:27:44.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.408+0000 7fd2f97fa640 1 -- 172.21.15.62:0/516745429 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec00a550 con 0x7fd2fc1098a0 2024-09-15T23:27:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.411+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e8005290 con 0x7fd2fc1098a0 2024-09-15T23:27:44.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.411+0000 7fd2f97fa640 1 -- 172.21.15.62:0/516745429 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd2ec043020 con 0x7fd2fc1098a0 2024-09-15T23:27:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.412+0000 7fd2f97fa640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.412+0000 7fd2f97fa640 1 -- 172.21.15.62:0/516745429 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd2ec0ef500 con 0x7fd2fc1098a0 2024-09-15T23:27:44.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.412+0000 7fd2fb7fe640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:44.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.413+0000 7fd2fb7fe640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc073200 tx=0x7fd2f0009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:44.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.419+0000 7fd2f97fa640 1 -- 172.21.15.62:0/516745429 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd2ec0bbea0 con 0x7fd2fc1098a0 2024-09-15T23:27:44.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 685.0 s, 685.0 tps, lat 153.990 ms stddev 206.915 2024-09-15T23:27:44.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.793+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2e8005580 con 0x7fd2fc1098a0 2024-09-15T23:27:44.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.799+0000 7fd2f97fa640 1 -- 172.21.15.62:0/516745429 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7fd2ec0bb7a0 con 0x7fd2fc1098a0 2024-09-15T23:27:44.801 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:27:44.801 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:27:44.801 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:27:44.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.805+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 msgr2=0x7fd2e40a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:44.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.805+0000 7fd3027ee640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc073200 tx=0x7fd2f0009110 comp rx=0 tx=0).stop 2024-09-15T23:27:44.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.805+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 msgr2=0x7fd2fc1a7240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:44.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.805+0000 7fd3027ee640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 0x7fd2fc1a7240 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec000c00 tx=0x7fd2ec0048f0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.807+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 shutdown_connections 2024-09-15T23:27:44.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.807+0000 7fd3027ee640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd2e40a3300 0x7fd2e40a57c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.807+0000 7fd3027ee640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd2fc112d20 0x7fd2fc1abb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.807+0000 7fd3027ee640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2fc10a1c0 0x7fd2fc1a7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.807+0000 7fd3027ee640 1 --2- 172.21.15.62:0/516745429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd2fc1098a0 0x7fd2fc1a7240 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:44.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.807+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 >> 172.21.15.62:0/516745429 conn(0x7fd2fc06e140 msgr2=0x7fd2fc070800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:44.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.808+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 shutdown_connections 2024-09-15T23:27:44.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:44.812+0000 7fd3027ee640 1 -- 172.21.15.62:0/516745429 wait complete. 2024-09-15T23:27:45.453 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:45 smithi062 ceph-mon[30949]: pgmap v714: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 8.2 MiB/s wr, 602 op/s 2024-09-15T23:27:45.454 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:45 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/516745429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:45.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:27:45.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:45 smithi067 ceph-mon[35665]: pgmap v714: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 8.2 MiB/s wr, 602 op/s 2024-09-15T23:27:45.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:45 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/516745429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:45.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:45 smithi148 ceph-mon[35610]: pgmap v714: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 8.2 MiB/s wr, 602 op/s 2024-09-15T23:27:45.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:45 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/516745429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:27:46.017 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:46.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.609+0000 7f6f594c8640 1 -- 172.21.15.62:0/987850309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f4c09a4c0 msgr2=0x7f6f4c09a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:46.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.609+0000 7f6f594c8640 1 --2- 172.21.15.62:0/987850309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f4c09a4c0 0x7f6f4c09a8a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009900 tx=0x7f6f4402a440 comp rx=0 tx=0).stop 2024-09-15T23:27:46.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.610+0000 7f6f594c8640 1 -- 172.21.15.62:0/987850309 shutdown_connections 2024-09-15T23:27:46.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.610+0000 7f6f594c8640 1 --2- 172.21.15.62:0/987850309 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f4c0051d0 0x7f6f4c097fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.610+0000 7f6f594c8640 1 --2- 172.21.15.62:0/987850309 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4c004740 0x7f6f4c004b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.610+0000 7f6f594c8640 1 --2- 172.21.15.62:0/987850309 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f4c09a4c0 0x7f6f4c09a8a0 secure :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009900 tx=0x7f6f4402a440 comp rx=0 tx=0).stop 2024-09-15T23:27:46.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.610+0000 7f6f594c8640 1 -- 172.21.15.62:0/987850309 >> 172.21.15.62:0/987850309 conn(0x7f6f4c00a1b0 msgr2=0x7f6f4c00a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:46.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.612+0000 7f6f594c8640 1 -- 172.21.15.62:0/987850309 shutdown_connections 2024-09-15T23:27:46.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.615+0000 7f6f594c8640 1 -- 172.21.15.62:0/987850309 wait complete. 2024-09-15T23:27:46.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.616+0000 7f6f594c8640 1 Processor -- start 2024-09-15T23:27:46.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.616+0000 7f6f594c8640 1 -- start start 2024-09-15T23:27:46.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.617+0000 7f6f594c8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4c004740 0x7f6f4c1383e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.617+0000 7f6f594c8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f4c0051d0 0x7f6f4c138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.617+0000 7f6f594c8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f4c1325a0 0x7f6f4c132a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.617+0000 7f6f594c8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c0ac010 con 0x7f6f4c1325a0 2024-09-15T23:27:46.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.617+0000 7f6f594c8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c0abe90 con 0x7f6f4c004740 2024-09-15T23:27:46.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.617+0000 7f6f594c8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c0ac190 con 0x7f6f4c0051d0 2024-09-15T23:27:46.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.618+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4c004740 0x7f6f4c1383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.618+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4c004740 0x7f6f4c1383e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34496/0 (socket says 172.21.15.62:34496) 2024-09-15T23:27:46.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.618+0000 7f6f52ffd640 1 -- 172.21.15.62:0/2227269647 learned_addr learned my addr 172.21.15.62:0/2227269647 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:46.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.619+0000 7f6f52ffd640 1 -- 172.21.15.62:0/2227269647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f4c0051d0 msgr2=0x7f6f4c138920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:27:46.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.620+0000 7f6f52ffd640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f4c0051d0 0x7f6f4c138920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.620+0000 7f6f52ffd640 1 -- 172.21.15.62:0/2227269647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f4c1325a0 msgr2=0x7f6f4c132a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:46.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.620+0000 7f6f52ffd640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f4c1325a0 0x7f6f4c132a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.620+0000 7f6f52ffd640 1 -- 172.21.15.62:0/2227269647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4c1330c0 con 0x7f6f4c004740 2024-09-15T23:27:46.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.624+0000 7f6f52ffd640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4c004740 0x7f6f4c1383e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6f44033040 tx=0x7f6f440056e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:46.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.624+0000 7f6f3bfff640 1 -- 172.21.15.62:0/2227269647 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44040030 con 0x7f6f4c004740 2024-09-15T23:27:46.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.625+0000 7f6f3bfff640 1 -- 172.21.15.62:0/2227269647 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f44031bb0 con 0x7f6f4c004740 2024-09-15T23:27:46.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.626+0000 7f6f3bfff640 1 -- 172.21.15.62:0/2227269647 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44040030 con 0x7f6f4c004740 2024-09-15T23:27:46.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.626+0000 7f6f594c8640 1 -- 172.21.15.62:0/2227269647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f4c133350 con 0x7f6f4c004740 2024-09-15T23:27:46.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.626+0000 7f6f594c8640 1 -- 172.21.15.62:0/2227269647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f4c13f4e0 con 0x7f6f4c004740 2024-09-15T23:27:46.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.630+0000 7f6f5b8a1740 1 Processor -- start 2024-09-15T23:27:46.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.632+0000 7f6f3bfff640 1 -- 172.21.15.62:0/2227269647 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f6f44043030 con 0x7f6f4c004740 2024-09-15T23:27:46.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.633+0000 7f6f3bfff640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6f3c0a3680 0x7f6f3c0a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.633+0000 7f6f3bfff640 1 -- 172.21.15.62:0/2227269647 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6f44041070 con 0x7f6f4c004740 2024-09-15T23:27:46.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.636+0000 7f6f5b8a1740 1 -- start start 2024-09-15T23:27:46.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.636+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd2f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.636+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd2da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.636+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd3a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.636+0000 7f6f5b8a1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bb5dc932f0 con 0x55bb5dd2ed20 2024-09-15T23:27:46.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.636+0000 7f6f5b8a1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55bb5dc3acf0 con 0x55bb5dd2e040 2024-09-15T23:27:46.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.636+0000 7f6f5b8a1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bb5dc8f2b0 con 0x55bb5dd2d5c0 2024-09-15T23:27:46.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.637+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd3a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.637+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd3a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34498/0 (socket says 172.21.15.62:34498) 2024-09-15T23:27:46.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.637+0000 7f6f537fe640 1 -- 172.21.15.62:0/3143578502 learned_addr learned my addr 172.21.15.62:0/3143578502 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:46.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.637+0000 7f6f527fc640 1 --2- 172.21.15.62:0/3143578502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd2da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.637+0000 7f6f537fe640 1 -- 172.21.15.62:0/3143578502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2d5c0 msgr2=0x55bb5dd2da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:46.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.637+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3143578502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd2da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.637+0000 7f6f537fe640 1 -- 172.21.15.62:0/3143578502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 msgr2=0x55bb5dd2f100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:27:46.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.637+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3143578502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd2f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.637+0000 7f6f537fe640 1 -- 172.21.15.62:0/3143578502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb5dc1f3a0 con 0x55bb5dd2e040 2024-09-15T23:27:46.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.638+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3143578502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd3a820 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6f54065a40 tx=0x7f6f540691c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3205c54d69a26783 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:46.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.638+0000 7f6f537fe640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6f3c0a3680 0x7f6f3c0a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.639+0000 7f6f594c8640 1 -- 172.21.15.62:0/3143578502 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f54070c30 con 0x55bb5dd2e040 2024-09-15T23:27:46.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.639+0000 7f6f594c8640 1 -- 172.21.15.62:0/3143578502 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f54070dd0 con 0x55bb5dd2e040 2024-09-15T23:27:46.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.640+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3143578502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 msgr2=0x55bb5dd3a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:46.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.640+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3143578502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd3a820 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6f54065a40 tx=0x7f6f540691c0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.641+0000 7f6f537fe640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6f3c0a3680 0x7f6f3c0a5b40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55bb5dd4beb0 tx=0x7f6f54071ba0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:46.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.642+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3143578502 shutdown_connections 2024-09-15T23:27:46.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.642+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3143578502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd3a820 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.642+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3143578502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd2da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.642+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3143578502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd2f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.642+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3143578502 >> 172.21.15.62:0/3143578502 conn(0x55bb5dcf6310 msgr2=0x55bb5dd2a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:46.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.644+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3143578502 shutdown_connections 2024-09-15T23:27:46.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.645+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3143578502 wait complete. 2024-09-15T23:27:46.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.645+0000 7f6f5b8a1740 1 -- start start 2024-09-15T23:27:46.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.645+0000 7f6f5b8a1740 1 Processor -- start 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.646+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd3a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.646+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2e040 0x55bb5dd37860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.646+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd37da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.646+0000 7f6f5b8a1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55bb5dd4c260 con 0x55bb5dd2e040 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.646+0000 7f6f5b8a1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55bb5dc4e060 con 0x55bb5dd2d5c0 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.646+0000 7f6f5b8a1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55bb5dd4c3e0 con 0x55bb5dd2ed20 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.647+0000 7f6f5b8a1740 10 client.0 _ll_register_callbacks cb 0x55bb5d9a18c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.647+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd37da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.647+0000 7f6f537fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd37da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35048/0 (socket says 172.21.15.62:35048) 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.647+0000 7f6f537fe640 1 -- 172.21.15.62:0/3397907412 learned_addr learned my addr 172.21.15.62:0/3397907412 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:46.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.647+0000 7f6f537fe640 1 -- 172.21.15.62:0/3397907412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 msgr2=0x55bb5dd3a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:46.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.647+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd3a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.647+0000 7f6f537fe640 1 -- 172.21.15.62:0/3397907412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2e040 msgr2=0x55bb5dd37860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:46.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.647+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2e040 0x55bb5dd37860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.647+0000 7f6f537fe640 1 -- 172.21.15.62:0/3397907412 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb5da043c0 con 0x55bb5dd2ed20 2024-09-15T23:27:46.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.648+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd37da0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6f54065360 tx=0x7f6f54071a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95d7692a0b174962 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:46.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.648+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3397907412 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f54070dd0 con 0x55bb5dd2ed20 2024-09-15T23:27:46.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.649+0000 7f6f36ff5640 10 client.27439 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:27:46.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.649+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3397907412 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f54063440 con 0x55bb5dd2ed20 2024-09-15T23:27:46.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.649+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3397907412 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6f54062630 con 0x55bb5dd2ed20 2024-09-15T23:27:46.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.650+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3397907412 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6f5409c050 con 0x55bb5dd2ed20 2024-09-15T23:27:46.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.650+0000 7f6f5b8a1740 10 client.27439 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:46.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.650+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3397907412 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55bb5dd57be0 con 0x55bb5dd2ed20 2024-09-15T23:27:46.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.651+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3397907412 <== mon.1 v2:172.21.15.148:3300/0 4 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f6f5407b040 con 0x55bb5dd2ed20 2024-09-15T23:27:46.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.651+0000 7f6f5b8a1740 10 client.27439 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:46.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.651+0000 7f6f5b8a1740 10 client.27439 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:27:46.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.651+0000 7f6f5b8a1740 20 client.27439 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:46.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.651+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55bb5dd58f30 0x55bb5dd5b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.651+0000 7f6f5b8a1740 4 client.27439 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:46.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.651+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3397907412 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb5dca0860 con 0x55bb5dd58f30 2024-09-15T23:27:46.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.651+0000 7f6f52ffd640 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55bb5dd58f30 0x55bb5dd5b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.652+0000 7f6f52ffd640 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55bb5dd58f30 0x55bb5dd5b320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3fed5c00f0b1e029 server_cookie=24459d771bb411e7 in_seq=0 out_seq=0 2024-09-15T23:27:46.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.652+0000 7f6f36ff5640 10 client.27439 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:27:46.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.654+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3397907412 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bb5dca0860 con 0x55bb5dd58f30 2024-09-15T23:27:46.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.654+0000 7f6f36ff5640 10 client.27439 handle_command_reply: tid=0 2024-09-15T23:27:46.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.654+0000 7f6f5b8a1740 1 client.27439 shutdown 2024-09-15T23:27:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.654+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3397907412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 msgr2=0x55bb5dd37da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.654+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd37da0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6f54065360 tx=0x7f6f54071a90 comp rx=0 tx=0).stop 2024-09-15T23:27:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.654+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3397907412 shutdown_connections 2024-09-15T23:27:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.654+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55bb5dd58f30 0x55bb5dd5b320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.655+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd37da0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.655+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2e040 0x55bb5dd37860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.655+0000 7f6f36ff5640 0 client.27439 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:27:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.655+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3397907412 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd3a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.655+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3397907412 >> 172.21.15.62:0/3397907412 conn(0x55bb5dcf6310 msgr2=0x55bb5dd29300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:46.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.658+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3397907412 shutdown_connections 2024-09-15T23:27:46.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.658+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3397907412 wait complete. 2024-09-15T23:27:46.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.658+0000 7f6f5b8a1740 20 client.27439 trim_cache size 0 max 16384 2024-09-15T23:27:46.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f5b8a1740 1 Processor -- start 2024-09-15T23:27:46.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f5b8a1740 1 -- start start 2024-09-15T23:27:46.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd397d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd60bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f5b8a1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c14bbc0 con 0x55bb5dd2d5c0 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f5b8a1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c149490 con 0x55bb5dd2e040 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f5b8a1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c1472e0 con 0x55bb5dd2ed20 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f527fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.681+0000 7f6f527fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd397d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34518/0 (socket says 172.21.15.62:34518) 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.682+0000 7f6f527fc640 1 -- 172.21.15.62:0/2790697060 learned_addr learned my addr 172.21.15.62:0/2790697060 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.682+0000 7f6f527fc640 1 -- 172.21.15.62:0/2790697060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 msgr2=0x55bb5dd60bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.682+0000 7f6f537fe640 1 --2- 172.21.15.62:0/2790697060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd60bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.682+0000 7f6f527fc640 1 --2- 172.21.15.62:0/2790697060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd60bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.682+0000 7f6f527fc640 1 -- 172.21.15.62:0/2790697060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2d5c0 msgr2=0x55bb5dd555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:46.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.682+0000 7f6f527fc640 1 --2- 172.21.15.62:0/2790697060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.682+0000 7f6f527fc640 1 -- 172.21.15.62:0/2790697060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb5dd35dd0 con 0x55bb5dd2e040 2024-09-15T23:27:46.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.682+0000 7f6f537fe640 1 --2- 172.21.15.62:0/2790697060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd60bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:46.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.682+0000 7f6f527fc640 1 --2- 172.21.15.62:0/2790697060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd397d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c1336e0 tx=0x7f6f4800d900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77118291e0e23695 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.683+0000 7f6f38ff9640 1 -- 172.21.15.62:0/2790697060 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f480125f0 con 0x55bb5dd2e040 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.683+0000 7f6f38ff9640 1 -- 172.21.15.62:0/2790697060 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f48018070 con 0x55bb5dd2e040 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.683+0000 7f6f38ff9640 1 -- 172.21.15.62:0/2790697060 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48013d60 con 0x55bb5dd2e040 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.684+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/2790697060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 msgr2=0x55bb5dd397d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.684+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/2790697060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd397d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c1336e0 tx=0x7f6f4800d900 comp rx=0 tx=0).stop 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.684+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/2790697060 shutdown_connections 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.684+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/2790697060 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd60bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.684+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/2790697060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2e040 0x55bb5dd397d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.684+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/2790697060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.684+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/2790697060 >> 172.21.15.62:0/2790697060 conn(0x55bb5dcf6310 msgr2=0x55bb5dd390c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:46.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.685+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/2790697060 shutdown_connections 2024-09-15T23:27:46.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.685+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/2790697060 wait complete. 2024-09-15T23:27:46.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.685+0000 7f6f5b8a1740 1 -- start start 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.686+0000 7f6f5b8a1740 1 Processor -- start 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.686+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd51ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.686+0000 7f6f5b8a1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2e040 0x55bb5dd345b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.686+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd51ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.686+0000 7f6f52ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd51ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34534/0 (socket says 172.21.15.62:34534) 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.687+0000 7f6f52ffd640 1 -- 172.21.15.62:0/3120133805 learned_addr learned my addr 172.21.15.62:0/3120133805 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.687+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd34af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.687+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c147950 con 0x55bb5dd2ed20 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.687+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd34af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.687+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c148dc0 con 0x55bb5dd2d5c0 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.687+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4c148020 con 0x55bb5dd2e040 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.687+0000 7f6f5b8a1740 10 client.0 _ll_register_callbacks cb 0x7f6f54068650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:46.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.687+0000 7f6f527fc640 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2e040 0x55bb5dd345b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.688+0000 7f6f537fe640 1 -- 172.21.15.62:0/3120133805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2e040 msgr2=0x55bb5dd345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:46.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.688+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2e040 0x55bb5dd345b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.688+0000 7f6f537fe640 1 -- 172.21.15.62:0/3120133805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 msgr2=0x55bb5dd51ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:46.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.688+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd51ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.688+0000 7f6f537fe640 1 -- 172.21.15.62:0/3120133805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb5dd352d0 con 0x55bb5dd2ed20 2024-09-15T23:27:46.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.688+0000 7f6f537fe640 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd34af0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f6f54068370 tx=0x7f6f54051030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d94e6285ee5b24 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:46.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.688+0000 7f6f36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:46.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.689+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3120133805 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f54071430 con 0x55bb5dd2ed20 2024-09-15T23:27:46.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.689+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3120133805 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6f54062c70 con 0x55bb5dd2ed20 2024-09-15T23:27:46.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.689+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f48018070 con 0x55bb5dd2ed20 2024-09-15T23:27:46.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.689+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3120133805 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6f5406e420 con 0x55bb5dd2ed20 2024-09-15T23:27:46.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.689+0000 7f6f5b8a1740 10 client.18138 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:46.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.689+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bb5ddc3b10 con 0x55bb5dd2ed20 2024-09-15T23:27:46.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.690+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3120133805 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f54070dd0 con 0x55bb5dd2ed20 2024-09-15T23:27:46.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.690+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3120133805 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f6f5409f040 con 0x55bb5dd2ed20 2024-09-15T23:27:46.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.690+0000 7f6f5b8a1740 10 client.18138 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.690+0000 7f6f5b8a1740 10 client.18138 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.690+0000 7f6f5b8a1740 20 client.18138 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.690+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55bb5dd58f30 0x55bb5ddc6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.690+0000 7f6f5b8a1740 4 client.18138 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.690+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f6f48018070 con 0x55bb5dd58f30 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.691+0000 7f6f52ffd640 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55bb5dd58f30 0x55bb5ddc6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.691+0000 7f6f52ffd640 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55bb5dd58f30 0x55bb5ddc6230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc566087246112a4 server_cookie=70ff5d50d6932891 in_seq=0 out_seq=0 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.692+0000 7f6f36ff5640 10 client.18138 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.692+0000 7f6f36ff5640 1 -- 172.21.15.62:0/3120133805 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f6f48018070 con 0x55bb5dd58f30 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.692+0000 7f6f36ff5640 10 client.18138 handle_command_reply: tid=0 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.692+0000 7f6f5b8a1740 1 client.18138 shutdown 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 msgr2=0x55bb5dd34af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd34af0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f6f54068370 tx=0x7f6f54051030 comp rx=0 tx=0).stop 2024-09-15T23:27:46.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 shutdown_connections 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x55bb5dd58f30 0x55bb5ddc6230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb5dd2ed20 0x55bb5dd34af0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55bb5dd2e040 0x55bb5dd345b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 --2- 172.21.15.62:0/3120133805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55bb5dd2d5c0 0x55bb5dd51ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f36ff5640 0 client.18138 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 >> 172.21.15.62:0/3120133805 conn(0x55bb5dcf6310 msgr2=0x55bb5dd398e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 shutdown_connections 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 1 -- 172.21.15.62:0/3120133805 wait complete. 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.693+0000 7f6f5b8a1740 20 client.18138 trim_cache size 0 max 16384 2024-09-15T23:27:46.696 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:27:46.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 -- 172.21.15.62:0/2227269647 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6f3c0a3680 msgr2=0x7f6f3c0a5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:46.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6f3c0a3680 0x7f6f3c0a5b40 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55bb5dd4beb0 tx=0x7f6f54071ba0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 -- 172.21.15.62:0/2227269647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4c004740 msgr2=0x7f6f4c1383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4c004740 0x7f6f4c1383e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6f44033040 tx=0x7f6f440056e0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 -- 172.21.15.62:0/2227269647 shutdown_connections 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f6f3c0a3680 0x7f6f3c0a5b40 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f4c1325a0 0x7f6f4c132a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6f4c0051d0 0x7f6f4c138920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 --2- 172.21.15.62:0/2227269647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f4c004740 0x7f6f4c1383e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.694+0000 7f6f39ffb640 1 -- 172.21.15.62:0/2227269647 >> 172.21.15.62:0/2227269647 conn(0x7f6f4c00a1b0 msgr2=0x7f6f4c092580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.695+0000 7f6f39ffb640 1 -- 172.21.15.62:0/2227269647 shutdown_connections 2024-09-15T23:27:46.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:46.695+0000 7f6f39ffb640 1 -- 172.21.15.62:0/2227269647 wait complete. 2024-09-15T23:27:47.460 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:47 smithi062 ceph-mon[30949]: pgmap v715: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.7 MiB/s wr, 613 op/s 2024-09-15T23:27:47.510 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:27:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:47 smithi067 ceph-mon[35665]: pgmap v715: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.7 MiB/s wr, 613 op/s 2024-09-15T23:27:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:47 smithi148 ceph-mon[35610]: pgmap v715: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 7.7 MiB/s wr, 613 op/s 2024-09-15T23:27:47.985 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.596+0000 7f0597f5c640 1 -- 172.21.15.62:0/1131132250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0590102d30 msgr2=0x7f0590103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.596+0000 7f0597f5c640 1 --2- 172.21.15.62:0/1131132250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0590102d30 0x7f0590103110 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f0580009930 tx=0x7f058002a440 comp rx=0 tx=0).stop 2024-09-15T23:27:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.597+0000 7f0597f5c640 1 -- 172.21.15.62:0/1131132250 shutdown_connections 2024-09-15T23:27:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.597+0000 7f0597f5c640 1 --2- 172.21.15.62:0/1131132250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05901082a0 0x7f059010ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.597+0000 7f0597f5c640 1 --2- 172.21.15.62:0/1131132250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f05901036e0 0x7f0590107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.597+0000 7f0597f5c640 1 --2- 172.21.15.62:0/1131132250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0590102d30 0x7f0590103110 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.597+0000 7f0597f5c640 1 -- 172.21.15.62:0/1131132250 >> 172.21.15.62:0/1131132250 conn(0x7f05900fc7d0 msgr2=0x7f05900febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.597+0000 7f0597f5c640 1 -- 172.21.15.62:0/1131132250 shutdown_connections 2024-09-15T23:27:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.597+0000 7f0597f5c640 1 -- 172.21.15.62:0/1131132250 wait complete. 2024-09-15T23:27:48.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.598+0000 7f0597f5c640 1 Processor -- start 2024-09-15T23:27:48.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.598+0000 7f0597f5c640 1 -- start start 2024-09-15T23:27:48.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f0597f5c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0590102d30 0x7f0590100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f0597f5c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05901036e0 0x7f0590100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f0597f5c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05901082a0 0x7f0590101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f0597f5c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f05901120e0 con 0x7f05901036e0 2024-09-15T23:27:48.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f0597f5c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0590111f60 con 0x7f05901082a0 2024-09-15T23:27:48.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f0597f5c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0590112260 con 0x7f0590102d30 2024-09-15T23:27:48.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05954f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05901036e0 0x7f0590100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05954f7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05901036e0 0x7f0590100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45640/0 (socket says 172.21.15.62:45640) 2024-09-15T23:27:48.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05954f7640 1 -- 172.21.15.62:0/3019351215 learned_addr learned my addr 172.21.15.62:0/3019351215 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:48.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05964f9640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05901082a0 0x7f0590101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05954f7640 1 -- 172.21.15.62:0/3019351215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0590102d30 msgr2=0x7f0590100990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05954f7640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0590102d30 0x7f0590100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05954f7640 1 -- 172.21.15.62:0/3019351215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05901082a0 msgr2=0x7f0590101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05954f7640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05901082a0 0x7f0590101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05954f7640 1 -- 172.21.15.62:0/3019351215 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05901a6a60 con 0x7f05901036e0 2024-09-15T23:27:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05964f9640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05901082a0 0x7f0590101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.599+0000 7f05954f7640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05901036e0 0x7f0590100ed0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f058c00e970 tx=0x7f058c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.600+0000 7f0586ffd640 1 -- 172.21.15.62:0/3019351215 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c00cd80 con 0x7f05901036e0 2024-09-15T23:27:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.600+0000 7f0597f5c640 1 -- 172.21.15.62:0/3019351215 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05901a6d50 con 0x7f05901036e0 2024-09-15T23:27:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.600+0000 7f0597f5c640 1 -- 172.21.15.62:0/3019351215 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05901a7250 con 0x7f05901036e0 2024-09-15T23:27:48.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.600+0000 7f0586ffd640 1 -- 172.21.15.62:0/3019351215 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f058c0093f0 con 0x7f05901036e0 2024-09-15T23:27:48.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.601+0000 7f0586ffd640 1 -- 172.21.15.62:0/3019351215 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c021430 con 0x7f05901036e0 2024-09-15T23:27:48.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.604+0000 7f0586ffd640 1 -- 172.21.15.62:0/3019351215 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f058c0035f0 con 0x7f05901036e0 2024-09-15T23:27:48.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.605+0000 7f059a335740 1 Processor -- start 2024-09-15T23:27:48.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.605+0000 7f059a335740 1 -- start start 2024-09-15T23:27:48.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f059a335740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70271090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f059a335740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70271a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f059a335740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa7027c370 0x55aa7027e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f059a335740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55aa701d72f0 con 0x55aa702715d0 2024-09-15T23:27:48.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f059a335740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55aa7017ecf0 con 0x55aa7027c370 2024-09-15T23:27:48.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f059a335740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55aa701d32b0 con 0x55aa70272b40 2024-09-15T23:27:48.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f05964f9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70271090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f0595cf8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70271a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f0595cf8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70271a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45654/0 (socket says 172.21.15.62:45654) 2024-09-15T23:27:48.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f0595cf8640 1 -- 172.21.15.62:0/3612834513 learned_addr learned my addr 172.21.15.62:0/3612834513 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:48.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f05954f7640 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa7027c370 0x55aa7027e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f0595cf8640 1 -- 172.21.15.62:0/3612834513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 msgr2=0x55aa70271090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:48.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f0595cf8640 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70271090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f0595cf8640 1 -- 172.21.15.62:0/3612834513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa7027c370 msgr2=0x55aa7027e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:48.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f0595cf8640 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa7027c370 0x55aa7027e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f0595cf8640 1 -- 172.21.15.62:0/3612834513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa701633a0 con 0x55aa702715d0 2024-09-15T23:27:48.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.606+0000 7f05964f9640 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70271090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:48.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.607+0000 7f05954f7640 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa7027c370 0x55aa7027e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:48.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.607+0000 7f0586ffd640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f057c0a32c0 0x7f057c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.607+0000 7f0586ffd640 1 -- 172.21.15.62:0/3019351215 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f058c0cfff0 con 0x7f05901036e0 2024-09-15T23:27:48.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.607+0000 7f0595cf8640 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70271a10 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f05800314d0 tx=0x7f0580031500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe25b9d2c5086cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:48.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.607+0000 7f05964f9640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f057c0a32c0 0x7f057c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.608+0000 7f057bfff640 1 -- 172.21.15.62:0/3612834513 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058003e030 con 0x55aa702715d0 2024-09-15T23:27:48.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.608+0000 7f057bfff640 1 -- 172.21.15.62:0/3612834513 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05800035f0 con 0x55aa702715d0 2024-09-15T23:27:48.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.608+0000 7f057bfff640 1 -- 172.21.15.62:0/3612834513 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580035040 con 0x55aa702715d0 2024-09-15T23:27:48.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.608+0000 7f059a335740 1 -- 172.21.15.62:0/3612834513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 msgr2=0x55aa70271a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.608+0000 7f05964f9640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f057c0a32c0 0x7f057c0a5780 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0588010600 tx=0x7f058800a4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:48.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.608+0000 7f059a335740 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70271a10 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f05800314d0 tx=0x7f0580031500 comp rx=0 tx=0).stop 2024-09-15T23:27:48.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.609+0000 7f059a335740 1 -- 172.21.15.62:0/3612834513 shutdown_connections 2024-09-15T23:27:48.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.609+0000 7f059a335740 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa7027c370 0x55aa7027e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.609+0000 7f059a335740 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70271a10 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.609+0000 7f059a335740 1 --2- 172.21.15.62:0/3612834513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70271090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.609+0000 7f059a335740 1 -- 172.21.15.62:0/3612834513 >> 172.21.15.62:0/3612834513 conn(0x55aa7023a310 msgr2=0x55aa7026dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:48.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.609+0000 7f059a335740 1 -- 172.21.15.62:0/3612834513 shutdown_connections 2024-09-15T23:27:48.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.609+0000 7f059a335740 1 -- 172.21.15.62:0/3612834513 wait complete. 2024-09-15T23:27:48.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.609+0000 7f059a335740 1 -- start start 2024-09-15T23:27:48.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.609+0000 7f059a335740 1 Processor -- start 2024-09-15T23:27:48.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f059a335740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa702715d0 0x55aa7027bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f059a335740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70278ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f059a335740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa7027c370 0x55aa70279430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f05964f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa7027c370 0x55aa70279430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f05964f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa7027c370 0x55aa70279430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45658/0 (socket says 172.21.15.62:45658) 2024-09-15T23:27:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f05964f9640 1 -- 172.21.15.62:0/3825146359 learned_addr learned my addr 172.21.15.62:0/3825146359 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f059a335740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55aa7027f010 con 0x55aa7027c370 2024-09-15T23:27:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55aa70192060 con 0x55aa702715d0 2024-09-15T23:27:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f05954f7640 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70278ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55aa7027f190 con 0x55aa70272b40 2024-09-15T23:27:48.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f059a335740 10 client.0 _ll_register_callbacks cb 0x55aa6fee58c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:48.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f0595cf8640 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa702715d0 0x55aa7027bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f05954f7640 1 -- 172.21.15.62:0/3825146359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa702715d0 msgr2=0x55aa7027bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:48.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f05954f7640 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa702715d0 0x55aa7027bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f05954f7640 1 -- 172.21.15.62:0/3825146359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa7027c370 msgr2=0x55aa70279430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:48.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f05954f7640 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa7027c370 0x55aa70279430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.610+0000 7f05954f7640 1 -- 172.21.15.62:0/3825146359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa6ff483c0 con 0x55aa70272b40 2024-09-15T23:27:48.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.611+0000 7f05954f7640 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70278ef0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55aa70272070 tx=0x7f058c09be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80155d3af4e43155 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:48.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.611+0000 7f05797fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:27:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.611+0000 7f05797fa640 1 -- 172.21.15.62:0/3825146359 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c0a0370 con 0x55aa70272b40 2024-09-15T23:27:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.611+0000 7f05797fa640 1 -- 172.21.15.62:0/3825146359 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f058c0bc0c0 con 0x55aa70272b40 2024-09-15T23:27:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.611+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05800035f0 con 0x55aa70272b40 2024-09-15T23:27:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.611+0000 7f05797fa640 1 -- 172.21.15.62:0/3825146359 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f058c0b5200 con 0x55aa70272b40 2024-09-15T23:27:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.611+0000 7f059a335740 10 client.27448 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.611+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aa7028aa00 con 0x55aa70272b40 2024-09-15T23:27:48.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.611+0000 7f05797fa640 1 -- 172.21.15.62:0/3825146359 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05800035f0 con 0x55aa70272b40 2024-09-15T23:27:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.612+0000 7f05797fa640 1 -- 172.21.15.62:0/3825146359 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f058c0d3020 con 0x55aa70272b40 2024-09-15T23:27:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.612+0000 7f059a335740 10 client.27448 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.612+0000 7f059a335740 10 client.27448 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:27:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.612+0000 7f059a335740 20 client.27448 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.612+0000 7f059a335740 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55aa7028beb0 0x55aa7028e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.612+0000 7f059a335740 4 client.27448 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.612+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa701e4860 con 0x55aa7028beb0 2024-09-15T23:27:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.612+0000 7f0595cf8640 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55aa7028beb0 0x55aa7028e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.613+0000 7f05797fa640 10 client.27448 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:27:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.613+0000 7f0595cf8640 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55aa7028beb0 0x55aa7028e2f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4e8e4bea85fc4876 server_cookie=aa988a7e5ef9803f in_seq=0 out_seq=0 2024-09-15T23:27:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.613+0000 7f05797fa640 1 -- 172.21.15.62:0/3825146359 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55aa701e4860 con 0x55aa7028beb0 2024-09-15T23:27:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.613+0000 7f05797fa640 10 client.27448 handle_command_reply: tid=0 2024-09-15T23:27:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.613+0000 7f059a335740 1 client.27448 shutdown 2024-09-15T23:27:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.613+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 msgr2=0x55aa70278ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.613+0000 7f059a335740 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70278ef0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55aa70272070 tx=0x7f058c09be60 comp rx=0 tx=0).stop 2024-09-15T23:27:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.613+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 shutdown_connections 2024-09-15T23:27:48.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.613+0000 7f059a335740 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55aa7028beb0 0x55aa7028e2f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.614+0000 7f059a335740 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa7027c370 0x55aa70279430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.614+0000 7f059a335740 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa70272b40 0x55aa70278ef0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.614+0000 7f059a335740 1 --2- 172.21.15.62:0/3825146359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa702715d0 0x55aa7027bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.614+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 >> 172.21.15.62:0/3825146359 conn(0x55aa7023a310 msgr2=0x55aa70273930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.614+0000 7f05797fa640 0 client.27448 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:27:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.614+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 shutdown_connections 2024-09-15T23:27:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.614+0000 7f059a335740 1 -- 172.21.15.62:0/3825146359 wait complete. 2024-09-15T23:27:48.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.614+0000 7f059a335740 20 client.27448 trim_cache size 0 max 16384 2024-09-15T23:27:48.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f059a335740 1 Processor -- start 2024-09-15T23:27:48.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f059a335740 1 -- start start 2024-09-15T23:27:48.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f059a335740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70288340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f059a335740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa70272b40 0x55aa7027d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f059a335740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 0x55aa7027dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f059a335740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f05740123d0 con 0x55aa702715d0 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f059a335740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f057400fca0 con 0x55aa70272b40 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f059a335740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f057400daf0 con 0x55aa7027c370 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f0595cf8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70288340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f0595cf8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70288340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45670/0 (socket says 172.21.15.62:45670) 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f0595cf8640 1 -- 172.21.15.62:0/4270144171 learned_addr learned my addr 172.21.15.62:0/4270144171 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.632+0000 7f05954f7640 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa70272b40 0x55aa7027d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f05964f9640 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 0x55aa7027dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f0595cf8640 1 -- 172.21.15.62:0/4270144171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 msgr2=0x55aa7027dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f0595cf8640 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 0x55aa7027dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f0595cf8640 1 -- 172.21.15.62:0/4270144171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa70272b40 msgr2=0x55aa7027d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f0595cf8640 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa70272b40 0x55aa7027d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f0595cf8640 1 -- 172.21.15.62:0/4270144171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa702795c0 con 0x55aa702715d0 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f05964f9640 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 0x55aa7027dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:48.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f0595cf8640 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70288340 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f05800314d0 tx=0x7f0580003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66b7ba63b5bc088 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f057b7fe640 1 -- 172.21.15.62:0/4270144171 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580050070 con 0x55aa702715d0 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f057b7fe640 1 -- 172.21.15.62:0/4270144171 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0580003ad0 con 0x55aa702715d0 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.633+0000 7f057b7fe640 1 -- 172.21.15.62:0/4270144171 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580035040 con 0x55aa702715d0 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.634+0000 7f059a335740 1 -- 172.21.15.62:0/4270144171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 msgr2=0x55aa70288340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.634+0000 7f059a335740 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70288340 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f05800314d0 tx=0x7f0580003ee0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.634+0000 7f059a335740 1 -- 172.21.15.62:0/4270144171 shutdown_connections 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.634+0000 7f059a335740 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 0x55aa7027dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.634+0000 7f059a335740 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa70272b40 0x55aa7027d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.634+0000 7f059a335740 1 --2- 172.21.15.62:0/4270144171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa70288340 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.634+0000 7f059a335740 1 -- 172.21.15.62:0/4270144171 >> 172.21.15.62:0/4270144171 conn(0x55aa7023a310 msgr2=0x55aa70278eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.635+0000 7f059a335740 1 -- 172.21.15.62:0/4270144171 shutdown_connections 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.635+0000 7f059a335740 1 -- 172.21.15.62:0/4270144171 wait complete. 2024-09-15T23:27:48.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.635+0000 7f059a335740 1 -- start start 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.635+0000 7f059a335740 1 Processor -- start 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.636+0000 7f059a335740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa702849b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.636+0000 7f059a335740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa70272b40 0x55aa70293250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.636+0000 7f059a335740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 0x55aa70293790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.636+0000 7f059a335740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f057400e160 con 0x55aa702715d0 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.636+0000 7f059a335740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f057400f5d0 con 0x55aa70272b40 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.636+0000 7f059a335740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f057400e830 con 0x55aa7027c370 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.636+0000 7f059a335740 10 client.0 _ll_register_callbacks cb 0x7f058c09c090 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.637+0000 7f0595cf8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa702849b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.637+0000 7f0595cf8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa702849b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45682/0 (socket says 172.21.15.62:45682) 2024-09-15T23:27:48.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.637+0000 7f0595cf8640 1 -- 172.21.15.62:0/3942864851 learned_addr learned my addr 172.21.15.62:0/3942864851 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:48.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.637+0000 7f0595cf8640 1 -- 172.21.15.62:0/3942864851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 msgr2=0x55aa70293790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:48.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.637+0000 7f0595cf8640 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 0x55aa70293790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.637+0000 7f0595cf8640 1 -- 172.21.15.62:0/3942864851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa70272b40 msgr2=0x55aa70293250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:27:48.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.637+0000 7f0595cf8640 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa70272b40 0x55aa70293250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.637+0000 7f0595cf8640 1 -- 172.21.15.62:0/3942864851 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa70293d50 con 0x55aa702715d0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.638+0000 7f0595cf8640 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa702849b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f058002a960 tx=0x7f0580005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7745cba43163cda4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.638+0000 7f05797fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.638+0000 7f05797fa640 1 -- 172.21.15.62:0/3942864851 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580059060 con 0x55aa702715d0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.639+0000 7f05797fa640 1 -- 172.21.15.62:0/3942864851 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0580003c80 con 0x55aa702715d0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.639+0000 7f059a335740 1 -- 172.21.15.62:0/3942864851 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0580003ad0 con 0x55aa702715d0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.639+0000 7f05797fa640 1 -- 172.21.15.62:0/3942864851 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0580035070 con 0x55aa702715d0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.639+0000 7f059a335740 10 client.18156 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.639+0000 7f059a335740 1 -- 172.21.15.62:0/3942864851 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aa70283180 con 0x55aa702715d0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.640+0000 7f05797fa640 1 -- 172.21.15.62:0/3942864851 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05800315b0 con 0x55aa702715d0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.640+0000 7f05797fa640 1 -- 172.21.15.62:0/3942864851 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f058004b550 con 0x55aa702715d0 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.641+0000 7f059a335740 10 client.18156 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.641+0000 7f059a335740 10 client.18156 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:27:48.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.641+0000 7f059a335740 20 client.18156 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:48.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.641+0000 7f059a335740 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55aa702f6b90 0x55aa702f8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.641+0000 7f059a335740 4 client.18156 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:48.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.641+0000 7f059a335740 1 -- 172.21.15.62:0/3942864851 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f0580003ad0 con 0x55aa702f6b90 2024-09-15T23:27:48.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.648+0000 7f05954f7640 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55aa702f6b90 0x55aa702f8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.670+0000 7f05954f7640 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55aa702f6b90 0x55aa702f8f80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=95972cbe199a8de8 server_cookie=cc8dcaef2fd85865 in_seq=0 out_seq=0 2024-09-15T23:27:48.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.670+0000 7f05797fa640 10 client.18156 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:27:48.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.671+0000 7f05797fa640 1 -- 172.21.15.62:0/3942864851 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f0580003ad0 con 0x55aa702f6b90 2024-09-15T23:27:48.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.671+0000 7f05797fa640 10 client.18156 handle_command_reply: tid=0 2024-09-15T23:27:48.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.671+0000 7f059a335740 1 client.18156 shutdown 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f059a335740 1 -- 172.21.15.62:0/3942864851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 msgr2=0x55aa702849b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f059a335740 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa702849b0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f058002a960 tx=0x7f0580005b50 comp rx=0 tx=0).stop 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f059a335740 1 -- 172.21.15.62:0/3942864851 shutdown_connections 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f059a335740 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55aa702f6b90 0x55aa702f8f80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f059a335740 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55aa7027c370 0x55aa70293790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f059a335740 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55aa70272b40 0x55aa70293250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f059a335740 1 --2- 172.21.15.62:0/3942864851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aa702715d0 0x55aa702849b0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f059a335740 1 -- 172.21.15.62:0/3942864851 >> 172.21.15.62:0/3942864851 conn(0x55aa7023a310 msgr2=0x55aa7026d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f05797fa640 0 client.18156 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.677+0000 7f059a335740 1 -- 172.21.15.62:0/3942864851 shutdown_connections 2024-09-15T23:27:48.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.678+0000 7f059a335740 1 -- 172.21.15.62:0/3942864851 wait complete. 2024-09-15T23:27:48.682 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:27:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.678+0000 7f059a335740 20 client.18156 trim_cache size 0 max 16384 2024-09-15T23:27:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.681+0000 7f0597f5c640 1 -- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f057c0a32c0 msgr2=0x7f057c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.681+0000 7f0597f5c640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f057c0a32c0 0x7f057c0a5780 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0588010600 tx=0x7f058800a4d0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.681+0000 7f0597f5c640 1 -- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05901036e0 msgr2=0x7f0590100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.681+0000 7f0597f5c640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05901036e0 0x7f0590100ed0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f058c00e970 tx=0x7f058c00ee30 comp rx=0 tx=0).stop 2024-09-15T23:27:48.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.681+0000 7f0597f5c640 1 -- 172.21.15.62:0/3019351215 shutdown_connections 2024-09-15T23:27:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.681+0000 7f0597f5c640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f057c0a32c0 0x7f057c0a5780 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.681+0000 7f0597f5c640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05901082a0 0x7f0590101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.681+0000 7f0597f5c640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05901036e0 0x7f0590100ed0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.682+0000 7f0597f5c640 1 --2- 172.21.15.62:0/3019351215 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0590102d30 0x7f0590100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.682+0000 7f0597f5c640 1 -- 172.21.15.62:0/3019351215 >> 172.21.15.62:0/3019351215 conn(0x7f05900fc7d0 msgr2=0x7f0590106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.682+0000 7f0597f5c640 1 -- 172.21.15.62:0/3019351215 shutdown_connections 2024-09-15T23:27:48.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:48.682+0000 7f0597f5c640 1 -- 172.21.15.62:0/3019351215 wait complete. 2024-09-15T23:27:48.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.986+0000 7fdbf4d11640 1 -- 172.21.15.62:0/873092319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 msgr2=0x7fdbf01105d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.986+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/873092319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 0x7fdbf01105d0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009ca0 tx=0x7fdbe802a8b0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.987+0000 7fdbf4d11640 1 -- 172.21.15.62:0/873092319 shutdown_connections 2024-09-15T23:27:48.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.987+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/873092319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 0x7fdbf01105d0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.988 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.987+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/873092319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf0109380 0x7fdbf0109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.989 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.987+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/873092319 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf0108470 0x7fdbf0108870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.989 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.987+0000 7fdbf4d11640 1 -- 172.21.15.62:0/873092319 >> 172.21.15.62:0/873092319 conn(0x7fdbf0077490 msgr2=0x7fdbf00798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:48.989 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.987+0000 7fdbf4d11640 1 -- 172.21.15.62:0/873092319 shutdown_connections 2024-09-15T23:27:48.989 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.987+0000 7fdbf4d11640 1 -- 172.21.15.62:0/873092319 wait complete. 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.987+0000 7fdbf4d11640 1 Processor -- start 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.987+0000 7fdbf4d11640 1 -- start start 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbf4d11640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf0108470 0x7fdbf019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbf4d11640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf0109380 0x7fdbf019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbf4d11640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 0x7fdbf01a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbf4d11640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf0113e70 con 0x7fdbf0109d40 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbf4d11640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf0113cf0 con 0x7fdbf0108470 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbf4d11640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf0113ff0 con 0x7fdbf0109380 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbeed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 0x7fdbf01a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbeed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 0x7fdbf01a5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45696/0 (socket says 172.21.15.62:45696) 2024-09-15T23:27:48.991 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbeed9d640 1 -- 172.21.15.62:0/2879073483 learned_addr learned my addr 172.21.15.62:0/2879073483 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf0109380 0x7fdbf019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbee59c640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf0108470 0x7fdbf019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbeed9d640 1 -- 172.21.15.62:0/2879073483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf0109380 msgr2=0x7fdbf019e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbeed9d640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf0109380 0x7fdbf019e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbeed9d640 1 -- 172.21.15.62:0/2879073483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf0108470 msgr2=0x7fdbf019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbeed9d640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf0108470 0x7fdbf019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.988+0000 7fdbeed9d640 1 -- 172.21.15.62:0/2879073483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf01a6000 con 0x7fdbf0109d40 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.989+0000 7fdbee59c640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf0108470 0x7fdbf019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.989+0000 7fdbeed9d640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 0x7fdbf01a5990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009dc0 tx=0x7fdbe8003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.989+0000 7fdbdf7fe640 1 -- 172.21.15.62:0/2879073483 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8004450 con 0x7fdbf0109d40 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.989+0000 7fdbf4d11640 1 -- 172.21.15.62:0/2879073483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf01a6290 con 0x7fdbf0109d40 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.989+0000 7fdbdf7fe640 1 -- 172.21.15.62:0/2879073483 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe8040070 con 0x7fdbf0109d40 2024-09-15T23:27:48.992 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.989+0000 7fdbdf7fe640 1 -- 172.21.15.62:0/2879073483 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe803b430 con 0x7fdbf0109d40 2024-09-15T23:27:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.989+0000 7fdbf4d11640 1 -- 172.21.15.62:0/2879073483 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf01a6860 con 0x7fdbf0109d40 2024-09-15T23:27:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.992+0000 7fdbdf7fe640 1 -- 172.21.15.62:0/2879073483 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fdbe8031440 con 0x7fdbf0109d40 2024-09-15T23:27:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.992+0000 7fdbf7aa7740 1 Processor -- start 2024-09-15T23:27:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.992+0000 7fdbf7aa7740 1 -- start start 2024-09-15T23:27:48.993 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.992+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5ea9a0 0x55babc5eada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5efda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5f0370 0x55babc5f2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbee59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbee59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5efda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34636/0 (socket says 172.21.15.62:34636) 2024-09-15T23:27:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbf7aa7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55babc54e2f0 con 0x55babc5ea9a0 2024-09-15T23:27:48.994 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbf7aa7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55babc4f6270 con 0x55babc5eb2e0 2024-09-15T23:27:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbee59c640 1 -- 172.21.15.62:0/1894452457 learned_addr learned my addr 172.21.15.62:0/1894452457 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1894452457 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55babc54a2b0 con 0x55babc5f0370 2024-09-15T23:27:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbdf7fe640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdbd80a32f0 0x7fdbd80a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/1894452457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5f0370 0x55babc5f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.993+0000 7fdbdf7fe640 1 -- 172.21.15.62:0/2879073483 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdbe80f13a0 con 0x7fdbf0109d40 2024-09-15T23:27:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.994+0000 7fdbedd9b640 1 -- 172.21.15.62:0/1894452457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 msgr2=0x55babc5efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.994+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/1894452457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.995 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.994+0000 7fdbedd9b640 1 -- 172.21.15.62:0/1894452457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5ea9a0 msgr2=0x55babc5eada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.994+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/1894452457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5ea9a0 0x55babc5eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.994+0000 7fdbedd9b640 1 -- 172.21.15.62:0/1894452457 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55babc4f5340 con 0x55babc5f0370 2024-09-15T23:27:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.994+0000 7fdbeed9d640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdbd80a32f0 0x7fdbd80a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.994+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/1894452457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5f0370 0x55babc5f2830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdbf007a9f0 tx=0x7fdbe4009e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0a2bbcfb5282a71 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.995+0000 7fdbdcff9640 1 -- 172.21.15.62:0/1894452457 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4018020 con 0x55babc5f0370 2024-09-15T23:27:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.995+0000 7fdbdcff9640 1 -- 172.21.15.62:0/1894452457 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbe4010b60 con 0x55babc5f0370 2024-09-15T23:27:48.996 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.995+0000 7fdbdcff9640 1 -- 172.21.15.62:0/1894452457 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4013610 con 0x55babc5f0370 2024-09-15T23:27:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbeed9d640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdbd80a32f0 0x7fdbd80a57b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80c5d80 tx=0x7fdbe8004950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1894452457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5f0370 msgr2=0x55babc5f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1894452457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5f0370 0x55babc5f2830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdbf007a9f0 tx=0x7fdbe4009e90 comp rx=0 tx=0).stop 2024-09-15T23:27:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1894452457 shutdown_connections 2024-09-15T23:27:48.997 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1894452457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5f0370 0x55babc5f2830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1894452457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1894452457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5ea9a0 0x55babc5eada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1894452457 >> 172.21.15.62:0/1894452457 conn(0x55babc5e6010 msgr2=0x55babc5e6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1894452457 shutdown_connections 2024-09-15T23:27:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.996+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1894452457 wait complete. 2024-09-15T23:27:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.997+0000 7fdbf7aa7740 1 -- start start 2024-09-15T23:27:48.998 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.997+0000 7fdbf7aa7740 1 Processor -- start 2024-09-15T23:27:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.998+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:48.999 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.998+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5eb2e0 0x55babc5f5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.998+0000 7fdbedd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.998+0000 7fdbedd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35164/0 (socket says 172.21.15.62:35164) 2024-09-15T23:27:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.998+0000 7fdbedd9b640 1 -- 172.21.15.62:0/1285492540 learned_addr learned my addr 172.21.15.62:0/1285492540 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.998+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5f0370 0x55babc5fc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbee59c640 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5eb2e0 0x55babc5f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55babc5f3340 con 0x55babc5eb2e0 2024-09-15T23:27:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55babc4d7290 con 0x55babc5f0370 2024-09-15T23:27:49.000 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55babc5f34c0 con 0x55babc5ea9a0 2024-09-15T23:27:49.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5f0370 0x55babc5fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbedd9b640 1 -- 172.21.15.62:0/1285492540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5f0370 msgr2=0x55babc5fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:49.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5f0370 0x55babc5fc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbedd9b640 1 -- 172.21.15.62:0/1285492540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5eb2e0 msgr2=0x55babc5f5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:49.001 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5eb2e0 0x55babc5f5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:48.999+0000 7fdbedd9b640 1 -- 172.21.15.62:0/1285492540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55babc261210 con 0x55babc5ea9a0 2024-09-15T23:27:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.000+0000 7fdbf7aa7740 10 client.0 _ll_register_callbacks cb 0x55babc508140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.000+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbe4010b60 con 0x55babc5ea9a0 2024-09-15T23:27:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.000+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f4cf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009420 tx=0x7fdbe40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ef4034a7039cd17 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.000+0000 7fdbd5ffb640 10 client.27457 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:27:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.000+0000 7fdbee59c640 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5eb2e0 0x55babc5f5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.000+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/1285492540 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe4018070 con 0x55babc5ea9a0 2024-09-15T23:27:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.000+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/1285492540 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdbe401f1d0 con 0x55babc5ea9a0 2024-09-15T23:27:49.002 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.000+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/1285492540 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fdbe401d420 con 0x55babc5ea9a0 2024-09-15T23:27:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.001+0000 7fdbf7aa7740 10 client.27457 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.001+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55babc32d620 con 0x55babc5ea9a0 2024-09-15T23:27:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.001+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/1285492540 <== mon.1 v2:172.21.15.148:3300/0 4 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fdbe401d5c0 con 0x55babc5ea9a0 2024-09-15T23:27:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.001+0000 7fdbf7aa7740 10 client.27457 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.001+0000 7fdbf7aa7740 10 client.27457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:49.003 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.001+0000 7fdbf7aa7740 20 client.27457 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.001+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55babc600440 0x55babc602830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.001+0000 7fdbf7aa7740 4 client.27457 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.001+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55babc55ba20 con 0x55babc600440 2024-09-15T23:27:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.002+0000 7fdbee59c640 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55babc600440 0x55babc602830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.002+0000 7fdbd5ffb640 10 client.27457 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.002+0000 7fdbee59c640 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55babc600440 0x55babc602830 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9af6b3510947bb0 server_cookie=4d877532ba23d9d in_seq=0 out_seq=0 2024-09-15T23:27:49.004 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.003+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/1285492540 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55babc55ba20 con 0x55babc600440 2024-09-15T23:27:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.003+0000 7fdbd5ffb640 10 client.27457 handle_command_reply: tid=0 2024-09-15T23:27:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.003+0000 7fdbf7aa7740 1 client.27457 shutdown 2024-09-15T23:27:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.003+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 msgr2=0x55babc5f4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:49.005 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.003+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f4cf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009420 tx=0x7fdbe40046c0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.004+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 shutdown_connections 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.004+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55babc600440 0x55babc602830 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.004+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5f0370 0x55babc5fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.004+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5eb2e0 0x55babc5f5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.004+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/1285492540 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f4cf0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.004+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 >> 172.21.15.62:0/1285492540 conn(0x55babc5e6010 msgr2=0x55babc5e7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.004+0000 7fdbd5ffb640 0 client.27457 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.005+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 shutdown_connections 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.005+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/1285492540 wait complete. 2024-09-15T23:27:49.006 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.005+0000 7fdbf7aa7740 20 client.27457 trim_cache size 0 max 16384 2024-09-15T23:27:49.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.027+0000 7fdbf7aa7740 1 Processor -- start 2024-09-15T23:27:49.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.027+0000 7fdbf7aa7740 1 -- start start 2024-09-15T23:27:49.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 0x55babc602450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbf7aa7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf01b3480 con 0x55babc5f0370 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbf7aa7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf01b0d50 con 0x55babc5eb2e0 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbf7aa7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf01aeba0 con 0x55babc5ea9a0 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbedd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbee59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbedd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35178/0 (socket says 172.21.15.62:35178) 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbee59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5fc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34642/0 (socket says 172.21.15.62:34642) 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbedd9b640 1 -- 172.21.15.62:0/2861263633 learned_addr learned my addr 172.21.15.62:0/2861263633 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/2861263633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 0x55babc602450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbee59c640 1 -- 172.21.15.62:0/2861263633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 msgr2=0x55babc5f52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbee59c640 1 --2- 172.21.15.62:0/2861263633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbee59c640 1 -- 172.21.15.62:0/2861263633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 msgr2=0x55babc602450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbee59c640 1 --2- 172.21.15.62:0/2861263633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 0x55babc602450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbee59c640 1 -- 172.21.15.62:0/2861263633 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55babc5fcfe0 con 0x55babc5eb2e0 2024-09-15T23:27:49.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.028+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/2861263633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.029+0000 7fdbee59c640 1 --2- 172.21.15.62:0/2861263633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5fc1c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x55babc602ff0 tx=0x7fdbe000a2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1badaadfdf017a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.029+0000 7fdbd7fff640 1 -- 172.21.15.62:0/2861263633 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0023030 con 0x55babc5eb2e0 2024-09-15T23:27:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.029+0000 7fdbd7fff640 1 -- 172.21.15.62:0/2861263633 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55babc55ba20 con 0x55babc5eb2e0 2024-09-15T23:27:49.032 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.029+0000 7fdbd7fff640 1 -- 172.21.15.62:0/2861263633 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0004a00 con 0x55babc5eb2e0 2024-09-15T23:27:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.030+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/2861263633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 msgr2=0x55babc5fc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.030+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/2861263633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5fc1c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x55babc602ff0 tx=0x7fdbe000a2d0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.030+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/2861263633 shutdown_connections 2024-09-15T23:27:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.030+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/2861263633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 0x55babc602450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.030+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/2861263633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5fc1c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.030+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/2861263633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.030+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/2861263633 >> 172.21.15.62:0/2861263633 conn(0x55babc5e6010 msgr2=0x55babc5e6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.030+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/2861263633 shutdown_connections 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.030+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/2861263633 wait complete. 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbf7aa7740 1 -- start start 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbf7aa7740 1 Processor -- start 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5f9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbf7aa7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 0x55babc5faf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbf7aa7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf01af210 con 0x55babc5f0370 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbf7aa7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf01b0680 con 0x55babc5eb2e0 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbf7aa7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf01af8e0 con 0x55babc5ea9a0 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbf7aa7740 10 client.0 _ll_register_callbacks cb 0x7fdbe000aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.031+0000 7fdbedd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbedd9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35184/0 (socket says 172.21.15.62:35184) 2024-09-15T23:27:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbedd9b640 1 -- 172.21.15.62:0/675969479 learned_addr learned my addr 172.21.15.62:0/675969479 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 0x55babc5faf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbee59c640 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5f9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbedd9b640 1 -- 172.21.15.62:0/675969479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 msgr2=0x55babc5f9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5f9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbedd9b640 1 -- 172.21.15.62:0/675969479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 msgr2=0x55babc5faf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 0x55babc5faf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbedd9b640 1 -- 172.21.15.62:0/675969479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55babc5fb570 con 0x55babc5ea9a0 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbee59c640 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbedd9b640 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f9a10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009840 tx=0x7fdbe4008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40f399396dd3beb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/675969479 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe403f060 con 0x55babc5ea9a0 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.032+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/675969479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55babc55ba20 con 0x55babc5ea9a0 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/675969479 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fdbe401dac0 con 0x55babc5ea9a0 2024-09-15T23:27:49.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/675969479 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fdbe4018040 con 0x55babc5ea9a0 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbf7aa7740 10 client.27463 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/675969479 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55babc66b170 con 0x55babc5ea9a0 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/675969479 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55babc55ba20 con 0x55babc5ea9a0 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/675969479 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fdbe4039720 con 0x55babc5ea9a0 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbf7aa7740 10 client.27463 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbf7aa7740 10 client.27463 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbf7aa7740 20 client.27463 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55babc66ba00 0x55babc66ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbf7aa7740 4 client.27463 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/675969479 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55babc555d70 con 0x55babc66ba00 2024-09-15T23:27:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.033+0000 7fdbee59c640 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55babc66ba00 0x55babc66ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.034+0000 7fdbee59c640 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55babc66ba00 0x55babc66ddf0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8122a32e0d86054d server_cookie=6c030d48c8afa77e in_seq=0 out_seq=0 2024-09-15T23:27:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.034+0000 7fdbd5ffb640 10 client.27463 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbd5ffb640 1 -- 172.21.15.62:0/675969479 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55babc555d70 con 0x55babc66ba00 2024-09-15T23:27:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbd5ffb640 10 client.27463 handle_command_reply: tid=0 2024-09-15T23:27:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 client.27463 shutdown 2024-09-15T23:27:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/675969479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 msgr2=0x55babc5f9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f9a10 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4009840 tx=0x7fdbe4008040 comp rx=0 tx=0).stop 2024-09-15T23:27:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/675969479 shutdown_connections 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55babc66ba00 0x55babc66ddf0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55babc5f0370 0x55babc5faf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55babc5eb2e0 0x55babc5f9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 --2- 172.21.15.62:0/675969479 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55babc5ea9a0 0x55babc5f9a10 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/675969479 >> 172.21.15.62:0/675969479 conn(0x55babc5e6010 msgr2=0x55babc5e51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbd5ffb640 0 client.27463 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/675969479 shutdown_connections 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 1 -- 172.21.15.62:0/675969479 wait complete. 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.046+0000 7fdbf7aa7740 20 client.27463 trim_cache size 0 max 16384 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:27:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:27:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:27:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:27:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.047+0000 7fdbf4d11640 1 -- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdbd80a32f0 msgr2=0x7fdbd80a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:49.050 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.047+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdbd80a32f0 0x7fdbd80a57b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80c5d80 tx=0x7fdbe8004950 comp rx=0 tx=0).stop 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 -- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 msgr2=0x7fdbf01a5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 0x7fdbf01a5990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009dc0 tx=0x7fdbe8003d60 comp rx=0 tx=0).stop 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 -- 172.21.15.62:0/2879073483 shutdown_connections 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fdbd80a32f0 0x7fdbd80a57b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbf0109d40 0x7fdbf01a5990 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fdbf0109380 0x7fdbf019e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 --2- 172.21.15.62:0/2879073483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf0108470 0x7fdbf019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 -- 172.21.15.62:0/2879073483 >> 172.21.15.62:0/2879073483 conn(0x7fdbf0077490 msgr2=0x7fdbf010c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 -- 172.21.15.62:0/2879073483 shutdown_connections 2024-09-15T23:27:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:27:49.048+0000 7fdbf4d11640 1 -- 172.21.15.62:0/2879073483 wait complete. 2024-09-15T23:27:49.399 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:27:49.469 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:49 smithi062 ceph-mon[30949]: pgmap v716: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 8.3 MiB/s wr, 601 op/s 2024-09-15T23:27:49.470 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:49 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:27:49.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:49 smithi067 ceph-mon[35665]: pgmap v716: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 8.3 MiB/s wr, 601 op/s 2024-09-15T23:27:49.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:49 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:27:49.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:49 smithi148 ceph-mon[35610]: pgmap v716: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 8.3 MiB/s wr, 601 op/s 2024-09-15T23:27:49.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:49 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:27:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 690.0 s, 577.7 tps, lat 165.395 ms stddev 212.021 2024-09-15T23:27:49.933 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:50.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.547+0000 7f881acbe640 1 -- 172.21.15.62:0/3840562472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8814108280 msgr2=0x7f8814108660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:50.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.547+0000 7f881acbe640 1 --2- 172.21.15.62:0/3840562472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8814108280 0x7f8814108660 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f880c009ca0 tx=0x7f880c02a8f0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.548+0000 7f881acbe640 1 -- 172.21.15.62:0/3840562472 shutdown_connections 2024-09-15T23:27:50.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.548+0000 7f881acbe640 1 --2- 172.21.15.62:0/3840562472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881410d360 0x7f8814114ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.548+0000 7f881acbe640 1 --2- 172.21.15.62:0/3840562472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8814108c30 0x7f881410cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.548+0000 7f881acbe640 1 --2- 172.21.15.62:0/3840562472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8814108280 0x7f8814108660 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.548+0000 7f881acbe640 1 -- 172.21.15.62:0/3840562472 >> 172.21.15.62:0/3840562472 conn(0x7f88140779f0 msgr2=0x7f8814079e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:50.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.554+0000 7f881acbe640 1 -- 172.21.15.62:0/3840562472 shutdown_connections 2024-09-15T23:27:50.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.554+0000 7f881acbe640 1 -- 172.21.15.62:0/3840562472 wait complete. 2024-09-15T23:27:50.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.556+0000 7f881acbe640 1 Processor -- start 2024-09-15T23:27:50.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.556+0000 7f881acbe640 1 -- start start 2024-09-15T23:27:50.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.556+0000 7f881acbe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8814108280 0x7f88140827e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.556+0000 7f881acbe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8814108c30 0x7f881407b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.556+0000 7f881acbe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f881410d360 0x7f881407be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.556+0000 7f881acbe640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8814118450 con 0x7f881410d360 2024-09-15T23:27:50.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.556+0000 7f881acbe640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88141182d0 con 0x7f8814108280 2024-09-15T23:27:50.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.556+0000 7f881acbe640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f88141185d0 con 0x7f8814108c30 2024-09-15T23:27:50.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.558+0000 7f88194bb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8814108c30 0x7f881407b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.558+0000 7f8819cbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8814108280 0x7f88140827e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.559+0000 7f88194bb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8814108c30 0x7f881407b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35202/0 (socket says 172.21.15.62:35202) 2024-09-15T23:27:50.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.559+0000 7f88194bb640 1 -- 172.21.15.62:0/4243488085 learned_addr learned my addr 172.21.15.62:0/4243488085 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:50.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.560+0000 7f88194bb640 1 -- 172.21.15.62:0/4243488085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8814108280 msgr2=0x7f88140827e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:50.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.560+0000 7f88194bb640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8814108280 0x7f88140827e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.560+0000 7f88194bb640 1 -- 172.21.15.62:0/4243488085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f881410d360 msgr2=0x7f881407be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:50.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.560+0000 7f88194bb640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f881410d360 0x7f881407be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.560+0000 7f88194bb640 1 -- 172.21.15.62:0/4243488085 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881407c3c0 con 0x7f8814108c30 2024-09-15T23:27:50.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.561+0000 7f88194bb640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8814108c30 0x7f881407b910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f881000c890 tx=0x7f881000cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:50.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.572+0000 7f880affd640 1 -- 172.21.15.62:0/4243488085 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8810004220 con 0x7f8814108c30 2024-09-15T23:27:50.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.572+0000 7f880affd640 1 -- 172.21.15.62:0/4243488085 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88100043c0 con 0x7f8814108c30 2024-09-15T23:27:50.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.572+0000 7f880affd640 1 -- 172.21.15.62:0/4243488085 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88100049e0 con 0x7f8814108c30 2024-09-15T23:27:50.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.573+0000 7f881acbe640 1 -- 172.21.15.62:0/4243488085 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881407c6b0 con 0x7f8814108c30 2024-09-15T23:27:50.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.573+0000 7f881acbe640 1 -- 172.21.15.62:0/4243488085 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8814080540 con 0x7f8814108c30 2024-09-15T23:27:50.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.577+0000 7f881e2f9740 1 Processor -- start 2024-09-15T23:27:50.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.579+0000 7f881e2f9740 1 -- start start 2024-09-15T23:27:50.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.579+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d8891cf0 0x5636d8890240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.579+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d8890780 0x5636d8890bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.579+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d889b520 0x5636d889d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.580+0000 7f881e2f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5636d87f62f0 con 0x5636d889b520 2024-09-15T23:27:50.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.580+0000 7f881e2f9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5636d879dcf0 con 0x5636d8890780 2024-09-15T23:27:50.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.580+0000 7f881e2f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5636d87f22b0 con 0x5636d8891cf0 2024-09-15T23:27:50.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.580+0000 7f880affd640 1 -- 172.21.15.62:0/4243488085 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8810004c40 con 0x7f8814108c30 2024-09-15T23:27:50.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.581+0000 7f8819cbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d8890780 0x5636d8890bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.581+0000 7f881a4bd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d8891cf0 0x5636d8890240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.581+0000 7f880affd640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88000a3450 0x7f88000a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.581+0000 7f8819cbc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d8890780 0x5636d8890bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34680/0 (socket says 172.21.15.62:34680) 2024-09-15T23:27:50.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.581+0000 7f8819cbc640 1 -- 172.21.15.62:0/3234765834 learned_addr learned my addr 172.21.15.62:0/3234765834 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:50.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.582+0000 7f880affd640 1 -- 172.21.15.62:0/4243488085 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f88100d0ca0 con 0x7f8814108c30 2024-09-15T23:27:50.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.583+0000 7f8819cbc640 1 -- 172.21.15.62:0/3234765834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d8891cf0 msgr2=0x5636d8890240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:50.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.583+0000 7f8819cbc640 1 --2- 172.21.15.62:0/3234765834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d8891cf0 0x5636d8890240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.583+0000 7f8819cbc640 1 -- 172.21.15.62:0/3234765834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d889b520 msgr2=0x5636d889d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:50.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.583+0000 7f8819cbc640 1 --2- 172.21.15.62:0/3234765834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d889b520 0x5636d889d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.583+0000 7f8819cbc640 1 -- 172.21.15.62:0/3234765834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636d87823a0 con 0x5636d8890780 2024-09-15T23:27:50.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.586+0000 7f881a4bd640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88000a3450 0x7f88000a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.592+0000 7f8819cbc640 1 --2- 172.21.15.62:0/3234765834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d8890780 0x5636d8890bc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f880c035000 tx=0x7f880c0099c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=328e87d20cb46fdc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:50.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.594+0000 7f881a4bd640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88000a3450 0x7f88000a5910 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f88040098a0 tx=0x7f8804008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:50.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.596+0000 7f87fffff640 1 -- 172.21.15.62:0/3234765834 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c003970 con 0x5636d8890780 2024-09-15T23:27:50.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.596+0000 7f87fffff640 1 -- 172.21.15.62:0/3234765834 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f880c036030 con 0x5636d8890780 2024-09-15T23:27:50.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.597+0000 7f881e2f9740 1 -- 172.21.15.62:0/3234765834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d8890780 msgr2=0x5636d8890bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:50.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.597+0000 7f881e2f9740 1 --2- 172.21.15.62:0/3234765834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d8890780 0x5636d8890bc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f880c035000 tx=0x7f880c0099c0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.597+0000 7f881e2f9740 1 -- 172.21.15.62:0/3234765834 shutdown_connections 2024-09-15T23:27:50.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.597+0000 7f881e2f9740 1 --2- 172.21.15.62:0/3234765834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d889b520 0x5636d889d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.597+0000 7f881e2f9740 1 --2- 172.21.15.62:0/3234765834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d8890780 0x5636d8890bc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.597+0000 7f881e2f9740 1 --2- 172.21.15.62:0/3234765834 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d8891cf0 0x5636d8890240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.597+0000 7f881e2f9740 1 -- 172.21.15.62:0/3234765834 >> 172.21.15.62:0/3234765834 conn(0x5636d8859310 msgr2=0x5636d888ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:50.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.598+0000 7f881e2f9740 1 -- 172.21.15.62:0/3234765834 shutdown_connections 2024-09-15T23:27:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.599+0000 7f881e2f9740 1 -- 172.21.15.62:0/3234765834 wait complete. 2024-09-15T23:27:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.599+0000 7f881e2f9740 1 -- start start 2024-09-15T23:27:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.600+0000 7f881e2f9740 1 Processor -- start 2024-09-15T23:27:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.600+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d889acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.600+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 0x5636d8897f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.600+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 0x5636d8898940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.600+0000 7f881e2f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5636d889e210 con 0x5636d8891cf0 2024-09-15T23:27:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.600+0000 7f881e2f9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5636d87b1060 con 0x5636d88984c0 2024-09-15T23:27:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.600+0000 7f881e2f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5636d889e390 con 0x5636d889b520 2024-09-15T23:27:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.600+0000 7f881e2f9740 10 client.0 _ll_register_callbacks cb 0x5636d85048c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.601+0000 7f8819cbc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d889acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.601+0000 7f88194bb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 0x5636d8897f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.602+0000 7f881a4bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 0x5636d8898940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.602+0000 7f881a4bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 0x5636d8898940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34686/0 (socket says 172.21.15.62:34686) 2024-09-15T23:27:50.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.602+0000 7f881a4bd640 1 -- 172.21.15.62:0/1961504757 learned_addr learned my addr 172.21.15.62:0/1961504757 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.606+0000 7f881a4bd640 1 -- 172.21.15.62:0/1961504757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 msgr2=0x5636d8897f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.606+0000 7f881a4bd640 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 0x5636d8897f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.606+0000 7f881a4bd640 1 -- 172.21.15.62:0/1961504757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 msgr2=0x5636d889acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.606+0000 7f881a4bd640 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d889acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.606+0000 7f881a4bd640 1 -- 172.21.15.62:0/1961504757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636d85673c0 con 0x5636d88984c0 2024-09-15T23:27:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.606+0000 7f881a4bd640 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 0x5636d8898940 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f88040156e0 tx=0x7f8804004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86f2064a5b30055a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.606+0000 7f87fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:27:50.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.607+0000 7f87fd7fa640 1 -- 172.21.15.62:0/1961504757 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804025060 con 0x5636d88984c0 2024-09-15T23:27:50.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.607+0000 7f881e2f9740 1 -- 172.21.15.62:0/1961504757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5636d8803860 con 0x5636d88984c0 2024-09-15T23:27:50.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.607+0000 7f87fd7fa640 1 -- 172.21.15.62:0/1961504757 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8804013790 con 0x5636d88984c0 2024-09-15T23:27:50.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.607+0000 7f87fd7fa640 1 -- 172.21.15.62:0/1961504757 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f880402c050 con 0x5636d88984c0 2024-09-15T23:27:50.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.608+0000 7f881e2f9740 10 client.27746 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:50.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.608+0000 7f881e2f9740 1 -- 172.21.15.62:0/1961504757 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5636d88a9fd0 con 0x5636d88984c0 2024-09-15T23:27:50.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.612+0000 7f87fd7fa640 1 -- 172.21.15.62:0/1961504757 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5636d8803860 con 0x5636d88984c0 2024-09-15T23:27:50.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.612+0000 7f87fd7fa640 1 -- 172.21.15.62:0/1961504757 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f880401fc30 con 0x5636d88984c0 2024-09-15T23:27:50.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.613+0000 7f881e2f9740 10 client.27746 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:50.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.613+0000 7f881e2f9740 10 client.27746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:50.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.613+0000 7f881e2f9740 20 client.27746 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:50.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.613+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5636d88ab870 0x5636d88adc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.613+0000 7f881e2f9740 4 client.27746 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.613+0000 7f881e2f9740 1 -- 172.21.15.62:0/1961504757 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636d87fdd70 con 0x5636d88ab870 2024-09-15T23:27:50.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.614+0000 7f8819cbc640 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5636d88ab870 0x5636d88adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.618+0000 7f87fd7fa640 10 client.27746 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:50.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.618+0000 7f8819cbc640 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5636d88ab870 0x5636d88adc80 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc1b57bf5dd129ff server_cookie=f8f3f023ca69a7e8 in_seq=0 out_seq=0 2024-09-15T23:27:50.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.621+0000 7f87fd7fa640 1 -- 172.21.15.62:0/1961504757 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5636d87fdd70 con 0x5636d88ab870 2024-09-15T23:27:50.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.621+0000 7f87fd7fa640 10 client.27746 handle_command_reply: tid=0 2024-09-15T23:27:50.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.621+0000 7f881e2f9740 1 client.27746 shutdown 2024-09-15T23:27:50.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 -- 172.21.15.62:0/1961504757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 msgr2=0x5636d8898940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 0x5636d8898940 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f88040156e0 tx=0x7f8804004770 comp rx=0 tx=0).stop 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 -- 172.21.15.62:0/1961504757 shutdown_connections 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5636d88ab870 0x5636d88adc80 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 0x5636d8898940 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 0x5636d8897f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1961504757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d889acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 -- 172.21.15.62:0/1961504757 >> 172.21.15.62:0/1961504757 conn(0x5636d8859310 msgr2=0x5636d8892b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f87fd7fa640 0 client.27746 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 -- 172.21.15.62:0/1961504757 shutdown_connections 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 1 -- 172.21.15.62:0/1961504757 wait complete. 2024-09-15T23:27:50.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.622+0000 7f881e2f9740 20 client.27746 trim_cache size 0 max 16384 2024-09-15T23:27:50.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.647+0000 7f881e2f9740 1 Processor -- start 2024-09-15T23:27:50.650 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:50 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.650 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:50 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.650 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:50 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.650 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:50 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.648+0000 7f881e2f9740 1 -- start start 2024-09-15T23:27:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.649+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d888cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.649+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 0x5636d8897840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.649+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 0x5636d8897d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.649+0000 7f881e2f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f88141c8a20 con 0x5636d8891cf0 2024-09-15T23:27:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.649+0000 7f881e2f9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88141c62f0 con 0x5636d88984c0 2024-09-15T23:27:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.649+0000 7f881e2f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f88141c4160 con 0x5636d889b520 2024-09-15T23:27:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.650+0000 7f8819cbc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d888cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.650+0000 7f8819cbc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d888cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45774/0 (socket says 172.21.15.62:45774) 2024-09-15T23:27:50.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.650+0000 7f8819cbc640 1 -- 172.21.15.62:0/1215466139 learned_addr learned my addr 172.21.15.62:0/1215466139 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:50.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.651+0000 7f881a4bd640 1 --2- 172.21.15.62:0/1215466139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 0x5636d8897d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.652+0000 7f8819cbc640 1 -- 172.21.15.62:0/1215466139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 msgr2=0x5636d8897d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.652+0000 7f8819cbc640 1 --2- 172.21.15.62:0/1215466139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 0x5636d8897d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.652+0000 7f8819cbc640 1 -- 172.21.15.62:0/1215466139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 msgr2=0x5636d8897840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:27:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.652+0000 7f8819cbc640 1 --2- 172.21.15.62:0/1215466139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 0x5636d8897840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.652+0000 7f8819cbc640 1 -- 172.21.15.62:0/1215466139 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636d888df80 con 0x5636d8891cf0 2024-09-15T23:27:50.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.653+0000 7f8819cbc640 1 --2- 172.21.15.62:0/1215466139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d888cfc0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f880c045000 tx=0x7f880c0095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90ca24ad4b6ca38 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.654+0000 7f87ff7fe640 1 -- 172.21.15.62:0/1215466139 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c051020 con 0x5636d8891cf0 2024-09-15T23:27:50.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.654+0000 7f87ff7fe640 1 -- 172.21.15.62:0/1215466139 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f880c036040 con 0x5636d8891cf0 2024-09-15T23:27:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.655+0000 7f881e2f9740 1 -- 172.21.15.62:0/1215466139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 msgr2=0x5636d888cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.655+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1215466139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d888cfc0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f880c045000 tx=0x7f880c0095e0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.656+0000 7f881e2f9740 1 -- 172.21.15.62:0/1215466139 shutdown_connections 2024-09-15T23:27:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.656+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1215466139 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d889b520 0x5636d8897d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.656+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1215466139 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d88984c0 0x5636d8897840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.656+0000 7f881e2f9740 1 --2- 172.21.15.62:0/1215466139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d8891cf0 0x5636d888cfc0 secure :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f880c045000 tx=0x7f880c0095e0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.656+0000 7f881e2f9740 1 -- 172.21.15.62:0/1215466139 >> 172.21.15.62:0/1215466139 conn(0x5636d8859310 msgr2=0x5636d888c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:50.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.659+0000 7f881e2f9740 1 -- 172.21.15.62:0/1215466139 shutdown_connections 2024-09-15T23:27:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.659+0000 7f881e2f9740 1 -- 172.21.15.62:0/1215466139 wait complete. 2024-09-15T23:27:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.660+0000 7f881e2f9740 1 -- start start 2024-09-15T23:27:50.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.660+0000 7f881e2f9740 1 Processor -- start 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.660+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d88984c0 0x5636d88a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.660+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d889b520 0x5636d889e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.660+0000 7f881e2f9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d889ec10 0x5636d889f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.660+0000 7f881e2f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f88141c47b0 con 0x5636d889b520 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.661+0000 7f881e2f9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f88141c5c20 con 0x5636d889ec10 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.661+0000 7f881e2f9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f88141c4e80 con 0x5636d88984c0 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.661+0000 7f881e2f9740 10 client.0 _ll_register_callbacks cb 0x7f88040095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.662+0000 7f881a4bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d889ec10 0x5636d889f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.662+0000 7f881a4bd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d889ec10 0x5636d889f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:34694/0 (socket says 172.21.15.62:34694) 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.662+0000 7f881a4bd640 1 -- 172.21.15.62:0/616350972 learned_addr learned my addr 172.21.15.62:0/616350972 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:50.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.662+0000 7f88194bb640 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d889b520 0x5636d889e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.663+0000 7f881a4bd640 1 -- 172.21.15.62:0/616350972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d88984c0 msgr2=0x5636d88a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:50.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.663+0000 7f881a4bd640 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d88984c0 0x5636d88a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.663+0000 7f881a4bd640 1 -- 172.21.15.62:0/616350972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d889b520 msgr2=0x5636d889e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:50.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.663+0000 7f881a4bd640 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d889b520 0x5636d889e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.663+0000 7f881a4bd640 1 -- 172.21.15.62:0/616350972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5636d89147c0 con 0x5636d889ec10 2024-09-15T23:27:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.667+0000 7f881a4bd640 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d889ec10 0x5636d889f070 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8804006e40 tx=0x7f88040127f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83d016e815e1a9d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.676+0000 7f87fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:27:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.676+0000 7f87fd7fa640 1 -- 172.21.15.62:0/616350972 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804009110 con 0x5636d889ec10 2024-09-15T23:27:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.677+0000 7f87fd7fa640 1 -- 172.21.15.62:0/616350972 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8804012bd0 con 0x5636d889ec10 2024-09-15T23:27:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.677+0000 7f881e2f9740 1 -- 172.21.15.62:0/616350972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f880c036040 con 0x5636d889ec10 2024-09-15T23:27:50.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.688+0000 7f87fd7fa640 1 -- 172.21.15.62:0/616350972 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8804025030 con 0x5636d889ec10 2024-09-15T23:27:50.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.689+0000 7f881e2f9740 10 client.27749 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:50.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.689+0000 7f881e2f9740 1 -- 172.21.15.62:0/616350972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5636d8915ac0 con 0x5636d889ec10 2024-09-15T23:27:50.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.698+0000 7f87fd7fa640 1 -- 172.21.15.62:0/616350972 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f880c036040 con 0x5636d889ec10 2024-09-15T23:27:50.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.698+0000 7f87fd7fa640 1 -- 172.21.15.62:0/616350972 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f8804032440 con 0x5636d889ec10 2024-09-15T23:27:50.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.698+0000 7f881e2f9740 10 client.27749 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:50.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.698+0000 7f881e2f9740 10 client.27749 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.698+0000 7f881e2f9740 20 client.27749 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.698+0000 7f881e2f9740 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5636d8916240 0x5636d8918660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.698+0000 7f881e2f9740 4 client.27749 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:27:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.698+0000 7f881e2f9740 1 -- 172.21.15.62:0/616350972 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5636d877a170 con 0x5636d8916240 2024-09-15T23:27:50.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.698+0000 7f8819cbc640 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5636d8916240 0x5636d8918660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:50.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.699+0000 7f87fd7fa640 10 client.27749 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:50.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.700+0000 7f8819cbc640 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5636d8916240 0x5636d8918660 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=400b3a4f42169020 server_cookie=a5e9263843f66b8f in_seq=0 out_seq=0 2024-09-15T23:27:50.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.701+0000 7f87fd7fa640 1 -- 172.21.15.62:0/616350972 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5636d877a170 con 0x5636d8916240 2024-09-15T23:27:50.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.701+0000 7f87fd7fa640 10 client.27749 handle_command_reply: tid=0 2024-09-15T23:27:50.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.701+0000 7f881e2f9740 1 client.27749 shutdown 2024-09-15T23:27:50.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.704+0000 7f881e2f9740 1 -- 172.21.15.62:0/616350972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d889ec10 msgr2=0x5636d889f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:50.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.704+0000 7f881e2f9740 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d889ec10 0x5636d889f070 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8804006e40 tx=0x7f88040127f0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f881e2f9740 1 -- 172.21.15.62:0/616350972 shutdown_connections 2024-09-15T23:27:50.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f881e2f9740 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5636d8916240 0x5636d8918660 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f881e2f9740 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636d889ec10 0x5636d889f070 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f881e2f9740 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5636d889b520 0x5636d889e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f881e2f9740 1 --2- 172.21.15.62:0/616350972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5636d88984c0 0x5636d88a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f87fd7fa640 0 client.27749 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f881e2f9740 1 -- 172.21.15.62:0/616350972 >> 172.21.15.62:0/616350972 conn(0x5636d8859310 msgr2=0x5636d88976f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f881e2f9740 1 -- 172.21.15.62:0/616350972 shutdown_connections 2024-09-15T23:27:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f881e2f9740 1 -- 172.21.15.62:0/616350972 wait complete. 2024-09-15T23:27:50.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.705+0000 7f881e2f9740 20 client.27749 trim_cache size 0 max 16384 2024-09-15T23:27:50.710 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:27:50.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 -- 172.21.15.62:0/4243488085 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88000a3450 msgr2=0x7f88000a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:50.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88000a3450 0x7f88000a5910 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f88040098a0 tx=0x7f8804008040 comp rx=0 tx=0).stop 2024-09-15T23:27:50.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 -- 172.21.15.62:0/4243488085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8814108c30 msgr2=0x7f881407b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:50.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8814108c30 0x7f881407b910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f881000c890 tx=0x7f881000cd50 comp rx=0 tx=0).stop 2024-09-15T23:27:50.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 -- 172.21.15.62:0/4243488085 shutdown_connections 2024-09-15T23:27:50.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f88000a3450 0x7f88000a5910 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f881410d360 0x7f881407be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8814108c30 0x7f881407b910 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 --2- 172.21.15.62:0/4243488085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8814108280 0x7f88140827e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:50.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 -- 172.21.15.62:0/4243488085 >> 172.21.15.62:0/4243488085 conn(0x7f88140779f0 msgr2=0x7f8814078d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:50.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 -- 172.21.15.62:0/4243488085 shutdown_connections 2024-09-15T23:27:50.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:50.710+0000 7f881acbe640 1 -- 172.21.15.62:0/4243488085 wait complete. 2024-09-15T23:27:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:50 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:50 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:50 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:50 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:50 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:50 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:50 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:50.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:50 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:51 smithi067 ceph-mon[35665]: pgmap v717: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.1 MiB/s wr, 575 op/s 2024-09-15T23:27:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:27:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:27:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:51 smithi148 ceph-mon[35610]: pgmap v717: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.1 MiB/s wr, 575 op/s 2024-09-15T23:27:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:27:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:27:51.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:51 smithi062 ceph-mon[30949]: pgmap v717: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.1 MiB/s wr, 575 op/s 2024-09-15T23:27:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:27:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:27:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:27:53.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:53 smithi067 ceph-mon[35665]: pgmap v718: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.0 MiB/s wr, 531 op/s 2024-09-15T23:27:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:53 smithi148 ceph-mon[35610]: pgmap v718: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.0 MiB/s wr, 531 op/s 2024-09-15T23:27:53.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:53 smithi062 ceph-mon[30949]: pgmap v718: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.0 MiB/s wr, 531 op/s 2024-09-15T23:27:54.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 695.0 s, 437.7 tps, lat 228.393 ms stddev 308.893 2024-09-15T23:27:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:55 smithi062 ceph-mon[30949]: pgmap v719: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.0 MiB/s wr, 586 op/s 2024-09-15T23:27:56.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:55 smithi067 ceph-mon[35665]: pgmap v719: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.0 MiB/s wr, 586 op/s 2024-09-15T23:27:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:55 smithi148 ceph-mon[35610]: pgmap v719: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.0 MiB/s wr, 586 op/s 2024-09-15T23:27:57.104 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:27:57.104 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:27:57.590 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:57.961 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:57 smithi062 ceph-mon[30949]: pgmap v720: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.6 MiB/s wr, 596 op/s 2024-09-15T23:27:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:57 smithi067 ceph-mon[35665]: pgmap v720: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.6 MiB/s wr, 596 op/s 2024-09-15T23:27:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:57 smithi148 ceph-mon[35610]: pgmap v720: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.6 MiB/s wr, 596 op/s 2024-09-15T23:27:58.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.297+0000 7fbd78e73640 1 -- 172.21.15.62:0/2118222076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd74100a70 msgr2=0x7fbd74100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.297+0000 7fbd78e73640 1 --2- 172.21.15.62:0/2118222076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd74100a70 0x7fbd74100ed0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009890 tx=0x7fbd5c02a390 comp rx=0 tx=0).stop 2024-09-15T23:27:58.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.299+0000 7fbd78e73640 1 -- 172.21.15.62:0/2118222076 shutdown_connections 2024-09-15T23:27:58.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.299+0000 7fbd78e73640 1 --2- 172.21.15.62:0/2118222076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd74101410 0x7fbd7410d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.299+0000 7fbd78e73640 1 --2- 172.21.15.62:0/2118222076 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd74100a70 0x7fbd74100ed0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.299+0000 7fbd78e73640 1 --2- 172.21.15.62:0/2118222076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74106a70 0x7fbd74106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.299+0000 7fbd78e73640 1 -- 172.21.15.62:0/2118222076 >> 172.21.15.62:0/2118222076 conn(0x7fbd740fc790 msgr2=0x7fbd740febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.301+0000 7fbd78e73640 1 -- 172.21.15.62:0/2118222076 shutdown_connections 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.301+0000 7fbd78e73640 1 -- 172.21.15.62:0/2118222076 wait complete. 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.302+0000 7fbd78e73640 1 Processor -- start 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.302+0000 7fbd78e73640 1 -- start start 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.302+0000 7fbd78e73640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74100a70 0x7fbd74073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.302+0000 7fbd78e73640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd74101410 0x7fbd7406cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.302+0000 7fbd78e73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd74106a70 0x7fbd7406d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.302+0000 7fbd78e73640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7410fdb0 con 0x7fbd74100a70 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.302+0000 7fbd78e73640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7410fc30 con 0x7fbd74106a70 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.302+0000 7fbd78e73640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7410ff30 con 0x7fbd74101410 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.303+0000 7fbd7259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74100a70 0x7fbd74073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.303+0000 7fbd7259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74100a70 0x7fbd74073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55516/0 (socket says 172.21.15.62:55516) 2024-09-15T23:27:58.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.303+0000 7fbd7259c640 1 -- 172.21.15.62:0/3517873816 learned_addr learned my addr 172.21.15.62:0/3517873816 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:58.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.304+0000 7fbd7259c640 1 -- 172.21.15.62:0/3517873816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd74101410 msgr2=0x7fbd7406cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.304+0000 7fbd7259c640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd74101410 0x7fbd7406cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.304+0000 7fbd7259c640 1 -- 172.21.15.62:0/3517873816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd74106a70 msgr2=0x7fbd7406d060 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:27:58.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.305+0000 7fbd7259c640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd74106a70 0x7fbd7406d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.305+0000 7fbd7259c640 1 -- 172.21.15.62:0/3517873816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd7406d6a0 con 0x7fbd74100a70 2024-09-15T23:27:58.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.305+0000 7fbd7259c640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74100a70 0x7fbd74073a40 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400eab0 tx=0x7fbd6400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.306+0000 7fbd6b7fe640 1 -- 172.21.15.62:0/3517873816 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400ce40 con 0x7fbd74100a70 2024-09-15T23:27:58.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.306+0000 7fbd78e73640 1 -- 172.21.15.62:0/3517873816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd74071570 con 0x7fbd74100a70 2024-09-15T23:27:58.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.306+0000 7fbd78e73640 1 -- 172.21.15.62:0/3517873816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd74071b00 con 0x7fbd74100a70 2024-09-15T23:27:58.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.307+0000 7fbd6b7fe640 1 -- 172.21.15.62:0/3517873816 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd64011950 con 0x7fbd74100a70 2024-09-15T23:27:58.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.307+0000 7fbd6b7fe640 1 -- 172.21.15.62:0/3517873816 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400ce40 con 0x7fbd74100a70 2024-09-15T23:27:58.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.310+0000 7fbd7b24c740 1 Processor -- start 2024-09-15T23:27:58.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.312+0000 7fbd6b7fe640 1 -- 172.21.15.62:0/3517873816 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fbd64011470 con 0x7fbd74100a70 2024-09-15T23:27:58.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.313+0000 7fbd6b7fe640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd540a3680 0x7fbd540a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.314+0000 7fbd6b7fe640 1 -- 172.21.15.62:0/3517873816 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbd6400cc40 con 0x7fbd74100a70 2024-09-15T23:27:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.316+0000 7fbd7b24c740 1 -- start start 2024-09-15T23:27:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.316+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b368460f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.316+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 0x55b36846b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.316+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 0x55b36846e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.316+0000 7fbd7b24c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b3683c6280 con 0x55b36846bd40 2024-09-15T23:27:58.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.317+0000 7fbd7b24c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b36836db10 con 0x55b368460b60 2024-09-15T23:27:58.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.317+0000 7fbd7b24c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b3683c2240 con 0x55b368461480 2024-09-15T23:27:58.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.318+0000 7fbd7259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b368460f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.320+0000 7fbd7259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b368460f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54182/0 (socket says 172.21.15.62:54182) 2024-09-15T23:27:58.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.320+0000 7fbd7259c640 1 -- 172.21.15.62:0/2703583829 learned_addr learned my addr 172.21.15.62:0/2703583829 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:58.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.321+0000 7fbd7259c640 1 -- 172.21.15.62:0/2703583829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 msgr2=0x55b36846b800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:27:58.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.321+0000 7fbd7259c640 1 --2- 172.21.15.62:0/2703583829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 0x55b36846b800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.321+0000 7fbd7259c640 1 -- 172.21.15.62:0/2703583829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 msgr2=0x55b36846e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.321+0000 7fbd7259c640 1 --2- 172.21.15.62:0/2703583829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 0x55b36846e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.321+0000 7fbd7259c640 1 -- 172.21.15.62:0/2703583829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b36837a1f0 con 0x55b368460b60 2024-09-15T23:27:58.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.325+0000 7fbd72d9d640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd540a3680 0x7fbd540a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.325+0000 7fbd7259c640 1 --2- 172.21.15.62:0/2703583829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b368460f40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbd6409b8f0 tx=0x7fbd6400cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5968756ccc4a1ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.325+0000 7fbd72d9d640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd540a3680 0x7fbd540a5b40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c005fd0 tx=0x7fbd6c005a00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.326+0000 7fbd68ff9640 1 -- 172.21.15.62:0/2703583829 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd640b2070 con 0x55b368460b60 2024-09-15T23:27:58.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.326+0000 7fbd68ff9640 1 -- 172.21.15.62:0/2703583829 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd6409ffa0 con 0x55b368460b60 2024-09-15T23:27:58.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.327+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2703583829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 msgr2=0x55b368460f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.327+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2703583829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b368460f40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbd6409b8f0 tx=0x7fbd6400cf00 comp rx=0 tx=0).stop 2024-09-15T23:27:58.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.333+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2703583829 shutdown_connections 2024-09-15T23:27:58.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.333+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2703583829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 0x55b36846e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.333+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2703583829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 0x55b36846b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.333+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2703583829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b368460f40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.333+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2703583829 >> 172.21.15.62:0/2703583829 conn(0x55b36845d0f0 msgr2=0x55b36845d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.341+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2703583829 shutdown_connections 2024-09-15T23:27:58.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.345+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2703583829 wait complete. 2024-09-15T23:27:58.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.345+0000 7fbd7b24c740 1 -- start start 2024-09-15T23:27:58.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.345+0000 7fbd7b24c740 1 Processor -- start 2024-09-15T23:27:58.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.347+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b368460b60 0x55b36846b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.347+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368461480 0x55b368468b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.347+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b36846bd40 0x55b3684692c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.347+0000 7fbd7b24c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b36846e960 con 0x55b368460b60 2024-09-15T23:27:58.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.347+0000 7fbd7b24c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b368380f10 con 0x55b368461480 2024-09-15T23:27:58.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.347+0000 7fbd7b24c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b36846eae0 con 0x55b36846bd40 2024-09-15T23:27:58.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.347+0000 7fbd7b24c740 10 client.0 _ll_register_callbacks cb 0x55b368349a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:58.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.348+0000 7fbd72d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b36846bd40 0x55b3684692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.348+0000 7fbd72d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b36846bd40 0x55b3684692c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:50820/0 (socket says 172.21.15.62:50820) 2024-09-15T23:27:58.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.348+0000 7fbd72d9d640 1 -- 172.21.15.62:0/1367460468 learned_addr learned my addr 172.21.15.62:0/1367460468 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:58.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.348+0000 7fbd71d9b640 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368461480 0x55b368468b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.352+0000 7fbd71d9b640 1 -- 172.21.15.62:0/1367460468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b36846bd40 msgr2=0x55b3684692c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.352+0000 7fbd71d9b640 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b36846bd40 0x55b3684692c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.352+0000 7fbd71d9b640 1 -- 172.21.15.62:0/1367460468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b368460b60 msgr2=0x55b36846b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.352+0000 7fbd71d9b640 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b368460b60 0x55b36846b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.352+0000 7fbd71d9b640 1 -- 172.21.15.62:0/1367460468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3681376a0 con 0x55b368461480 2024-09-15T23:27:58.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.353+0000 7fbd71d9b640 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368461480 0x55b368468b30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55b36845df50 tx=0x7fbd5c004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0cfeba5c1b376aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.354+0000 7fbd61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:27:58.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.357+0000 7fbd61ffb640 1 -- 172.21.15.62:0/1367460468 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c037040 con 0x55b368461480 2024-09-15T23:27:58.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.357+0000 7fbd61ffb640 1 -- 172.21.15.62:0/1367460468 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbd5c004470 con 0x55b368461480 2024-09-15T23:27:58.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.358+0000 7fbd7b24c740 1 -- 172.21.15.62:0/1367460468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd6409ffa0 con 0x55b368461480 2024-09-15T23:27:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.358+0000 7fbd61ffb640 1 -- 172.21.15.62:0/1367460468 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbd5c0305f0 con 0x55b368461480 2024-09-15T23:27:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.359+0000 7fbd7b24c740 10 client.27758 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:58.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.359+0000 7fbd7b24c740 1 -- 172.21.15.62:0/1367460468 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b36847a550 con 0x55b368461480 2024-09-15T23:27:58.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.361+0000 7fbd61ffb640 1 -- 172.21.15.62:0/1367460468 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd6409ffa0 con 0x55b368461480 2024-09-15T23:27:58.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.361+0000 7fbd61ffb640 1 -- 172.21.15.62:0/1367460468 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fbd5c043030 con 0x55b368461480 2024-09-15T23:27:58.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.361+0000 7fbd7b24c740 10 client.27758 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:58.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.361+0000 7fbd7b24c740 10 client.27758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:58.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.361+0000 7fbd7b24c740 20 client.27758 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:58.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.361+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b36847bf00 0x55b36847e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.361+0000 7fbd7b24c740 4 client.27758 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:27:58.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.361+0000 7fbd7b24c740 1 -- 172.21.15.62:0/1367460468 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3683d37f0 con 0x55b36847bf00 2024-09-15T23:27:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.365+0000 7fbd7259c640 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b36847bf00 0x55b36847e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.367+0000 7fbd61ffb640 10 client.27758 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.368+0000 7fbd7259c640 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b36847bf00 0x55b36847e3a0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8df33e4a817009d server_cookie=5a92a8e63ad429a2 in_seq=0 out_seq=0 2024-09-15T23:27:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.369+0000 7fbd61ffb640 1 -- 172.21.15.62:0/1367460468 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b3683d37f0 con 0x55b36847bf00 2024-09-15T23:27:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.369+0000 7fbd61ffb640 10 client.27758 handle_command_reply: tid=0 2024-09-15T23:27:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.369+0000 7fbd7b24c740 1 client.27758 shutdown 2024-09-15T23:27:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.371+0000 7fbd7b24c740 1 -- 172.21.15.62:0/1367460468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368461480 msgr2=0x55b368468b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.371+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368461480 0x55b368468b30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55b36845df50 tx=0x7fbd5c004040 comp rx=0 tx=0).stop 2024-09-15T23:27:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.373+0000 7fbd7b24c740 1 -- 172.21.15.62:0/1367460468 shutdown_connections 2024-09-15T23:27:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.373+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b36847bf00 0x55b36847e3a0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.373+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b36846bd40 0x55b3684692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.373+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368461480 0x55b368468b30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.373+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/1367460468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b368460b60 0x55b36846b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.373+0000 7fbd7b24c740 1 -- 172.21.15.62:0/1367460468 >> 172.21.15.62:0/1367460468 conn(0x55b36845d0f0 msgr2=0x55b36845c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.373+0000 7fbd61ffb640 0 client.27758 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:58.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.378+0000 7fbd7b24c740 1 -- 172.21.15.62:0/1367460468 shutdown_connections 2024-09-15T23:27:58.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.383+0000 7fbd7b24c740 1 -- 172.21.15.62:0/1367460468 wait complete. 2024-09-15T23:27:58.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.384+0000 7fbd7b24c740 20 client.27758 trim_cache size 0 max 16384 2024-09-15T23:27:58.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd7b24c740 1 Processor -- start 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd7b24c740 1 -- start start 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b3684635c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 0x55b368467b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 0x55b368483cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd7259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b3684635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd7259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b3684635c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54194/0 (socket says 172.21.15.62:54194) 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd71d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 0x55b368467b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd71d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 0x55b368467b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:50824/0 (socket says 172.21.15.62:50824) 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.417+0000 7fbd7259c640 1 -- 172.21.15.62:0/382318863 learned_addr learned my addr 172.21.15.62:0/382318863 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:58.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd7b24c740 1 -- 172.21.15.62:0/382318863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbd58012490 con 0x55b36846bd40 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd7b24c740 1 -- 172.21.15.62:0/382318863 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5800fd60 con 0x55b368460b60 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd72d9d640 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 0x55b368483cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd7b24c740 1 -- 172.21.15.62:0/382318863 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5800dbb0 con 0x55b368461480 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd72d9d640 1 -- 172.21.15.62:0/382318863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 msgr2=0x55b368467b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd72d9d640 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 0x55b368467b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd72d9d640 1 -- 172.21.15.62:0/382318863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 msgr2=0x55b3684635c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd72d9d640 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b3684635c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd72d9d640 1 -- 172.21.15.62:0/382318863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b368469630 con 0x55b36846bd40 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd7259c640 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b3684635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.418+0000 7fbd71d9b640 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 0x55b368467b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.419+0000 7fbd72d9d640 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 0x55b368483cb0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c005c80 tx=0x7fbd6c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda54bbc078c7bc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.419+0000 7fbd63fff640 1 -- 172.21.15.62:0/382318863 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c01f450 con 0x55b36846bd40 2024-09-15T23:27:58.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.419+0000 7fbd63fff640 1 -- 172.21.15.62:0/382318863 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd6c024070 con 0x55b36846bd40 2024-09-15T23:27:58.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.419+0000 7fbd63fff640 1 -- 172.21.15.62:0/382318863 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c00f7c0 con 0x55b36846bd40 2024-09-15T23:27:58.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.420+0000 7fbd7b24c740 1 -- 172.21.15.62:0/382318863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 msgr2=0x55b368483cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.420+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 0x55b368483cb0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c005c80 tx=0x7fbd6c010000 comp rx=0 tx=0).stop 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.420+0000 7fbd7b24c740 1 -- 172.21.15.62:0/382318863 shutdown_connections 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.420+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b36846bd40 0x55b368483cb0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.420+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368461480 0x55b368467b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.420+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/382318863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b368460b60 0x55b3684635c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.420+0000 7fbd7b24c740 1 -- 172.21.15.62:0/382318863 >> 172.21.15.62:0/382318863 conn(0x55b36845d0f0 msgr2=0x55b368467400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.420+0000 7fbd7b24c740 1 -- 172.21.15.62:0/382318863 shutdown_connections 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.421+0000 7fbd7b24c740 1 -- 172.21.15.62:0/382318863 wait complete. 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.421+0000 7fbd7b24c740 1 -- start start 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.421+0000 7fbd7b24c740 1 Processor -- start 2024-09-15T23:27:58.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368460b60 0x55b368465ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7b24c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b368461480 0x55b368466220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368460b60 0x55b368465ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7259c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368460b60 0x55b368465ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:50830/0 (socket says 172.21.15.62:50830) 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7259c640 1 -- 172.21.15.62:0/2693949898 learned_addr learned my addr 172.21.15.62:0/2693949898 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b36846bd40 0x55b368466760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5800e220 con 0x55b368461480 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd71d9b640 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b368461480 0x55b368466220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5800f690 con 0x55b36846bd40 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5800e8f0 con 0x55b368460b60 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.422+0000 7fbd7b24c740 10 client.0 _ll_register_callbacks cb 0x7fbd5c035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd72d9d640 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b36846bd40 0x55b368466760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd7259c640 1 -- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b36846bd40 msgr2=0x55b368466760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd7259c640 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b36846bd40 0x55b368466760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd7259c640 1 -- 172.21.15.62:0/2693949898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b368461480 msgr2=0x55b368466220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd7259c640 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b368461480 0x55b368466220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd7259c640 1 -- 172.21.15.62:0/2693949898 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3684e5e80 con 0x55b368460b60 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd72d9d640 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b36846bd40 0x55b368466760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd7259c640 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368460b60 0x55b368465ce0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd64097f00 tx=0x7fbd6409ffc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d498ee5c3631a6cd server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd61ffb640 1 -- 172.21.15.62:0/2693949898 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd640b2040 con 0x55b368460b60 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd61ffb640 1 -- 172.21.15.62:0/2693949898 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbd64020db0 con 0x55b368460b60 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd6c024070 con 0x55b368460b60 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd61ffb640 1 -- 172.21.15.62:0/2693949898 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbd64011760 con 0x55b368460b60 2024-09-15T23:27:58.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd7b24c740 10 client.27484 fetch_fsmap learned FSMap version 35 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.423+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b3684e7140 con 0x55b368460b60 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd61ffb640 1 -- 172.21.15.62:0/2693949898 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd6c024070 con 0x55b368460b60 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd61ffb640 1 -- 172.21.15.62:0/2693949898 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fbd64021020 con 0x55b368460b60 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd7b24c740 10 client.27484 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd7b24c740 10 client.27484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd7b24c740 20 client.27484 populate_metadata read hostname 'smithi062' 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b3684e7430 0x55b3684e9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd7b24c740 4 client.27484 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b3683cdd00 con 0x55b3684e7430 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd71d9b640 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b3684e7430 0x55b3684e9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:27:58.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd71d9b640 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b3684e7430 0x55b3684e9820 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a523bd450647c95 server_cookie=270d9bf740bf25d in_seq=0 out_seq=0 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.424+0000 7fbd61ffb640 10 client.27484 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.425+0000 7fbd61ffb640 1 -- 172.21.15.62:0/2693949898 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b3683cdd00 con 0x55b3684e7430 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.425+0000 7fbd61ffb640 10 client.27484 handle_command_reply: tid=0 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.425+0000 7fbd7b24c740 1 client.27484 shutdown 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.425+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368460b60 msgr2=0x55b368465ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.425+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368460b60 0x55b368465ce0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd64097f00 tx=0x7fbd6409ffc0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 shutdown_connections 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b3684e7430 0x55b3684e9820 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b36846bd40 0x55b368466760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b368461480 0x55b368466220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd7b24c740 1 --2- 172.21.15.62:0/2693949898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b368460b60 0x55b368465ce0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd61ffb640 0 client.27484 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:27:58.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 >> 172.21.15.62:0/2693949898 conn(0x55b36845d0f0 msgr2=0x55b368467c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 shutdown_connections 2024-09-15T23:27:58.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd7b24c740 1 -- 172.21.15.62:0/2693949898 wait complete. 2024-09-15T23:27:58.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.426+0000 7fbd7b24c740 20 client.27484 trim_cache size 0 max 16384 2024-09-15T23:27:58.433 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:27:58.433 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:27:58.433 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "031d0c8d-e514-441a-bc06-2297e09d88d9", 2024-09-15T23:27:58.434 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:27:58.434 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:27:58.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 -- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd540a3680 msgr2=0x7fbd540a5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd540a3680 0x7fbd540a5b40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c005fd0 tx=0x7fbd6c005a00 comp rx=0 tx=0).stop 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 -- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74100a70 msgr2=0x7fbd74073a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74100a70 0x7fbd74073a40 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400eab0 tx=0x7fbd6400ef70 comp rx=0 tx=0).stop 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 -- 172.21.15.62:0/3517873816 shutdown_connections 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbd540a3680 0x7fbd540a5b40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd74106a70 0x7fbd7406d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbd74101410 0x7fbd7406cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 --2- 172.21.15.62:0/3517873816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd74100a70 0x7fbd74073a40 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 -- 172.21.15.62:0/3517873816 >> 172.21.15.62:0/3517873816 conn(0x7fbd740fc790 msgr2=0x7fbd740fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 -- 172.21.15.62:0/3517873816 shutdown_connections 2024-09-15T23:27:58.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:27:58.427+0000 7fbd78e73640 1 -- 172.21.15.62:0/3517873816 wait complete. 2024-09-15T23:27:58.713 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:27:58 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: 2024-09-15T23:27:58.446+0000 7ff293927640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000f [...2,head] /volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans/ auth{2=1} v3362 ap=2 f(v9 m2024-09-15T23:26:13.788638+0000 3=3+0) n(v30 rc2024-09-15T23:26:13.793638+0000 b737280 4=3+1) (inest sync->mix g=2) | dirtyscattered=0 request=0 lock=0 dirfrag=1 caps=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x55598f2f4000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v282>,<0x10000000005/data v121>,<0x100000001fb/postgres v25>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v43>,<0x10000000001/sv_1 v57>,<0x10000000000/qa v44>,<0x1/volumes v43>]//[]","memoryvalue":"(2)0x1000000000f:[<0x10000000006/pg_subtrans v3362>,<0x10000000005/data v855>,<0x100000001fb/postgres v594>,<0x100000001fa/597b9bb6-f4c4-4e49-b195-c049d8e7226e v628>,<0x10000000001/sv_1 v621>,<0x10000000000/qa v629>,<0x1/volumes v594>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-09-15T23:24:19.529348+0000 2=2+0)","ondisk_value.rstat":"n(v0 rc2024-09-15T23:26:13.793638+0000 b737280 4=3+1)","memory_value.dirstat":"f(v9 m2024-09-15T23:26:13.788638+0000 3=3+0)","memory_value.rstat":"n(v30 rc2024-09-15T23:26:13.793638+0000 b737280 4=3+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-09-15T23:27:58.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:58 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:27:58.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:58 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:58.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:58 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:27:58.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:58 smithi062 ceph-mon[30949]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:27:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:58 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:27:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:58 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:59.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:58 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:27:59.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:58 smithi067 ceph-mon[35665]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:27:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:58 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:27:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:58 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:58 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:27:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:58 smithi148 ceph-mon[35610]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:27:59.316 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:27:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 700.0 s, 740.8 tps, lat 138.555 ms stddev 175.503 2024-09-15T23:27:59.800 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:27:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:59 smithi062 ceph-mon[30949]: pgmap v721: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 562 op/s 2024-09-15T23:27:59.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:59 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:27:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:59 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:59 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:27:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:59 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:27:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:59 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:59 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:27:59.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:27:59 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:59 smithi067 ceph-mon[35665]: pgmap v721: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 562 op/s 2024-09-15T23:28:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:59 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:28:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:59 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:59 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:28:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:59 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:28:00.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:59 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:00.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:59 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:00.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:27:59 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:59 smithi148 ceph-mon[35610]: pgmap v721: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 562 op/s 2024-09-15T23:28:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:59 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:28:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:59 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:00.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:59 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:28:00.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:59 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:28:00.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:59 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:00.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:59 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:00.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:27:59 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:00.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.430+0000 7fb4127d8640 1 -- 172.21.15.62:0/859953292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb40c110190 msgr2=0x7fb40c06e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.430+0000 7fb4127d8640 1 --2- 172.21.15.62:0/859953292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb40c110190 0x7fb40c06e4f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb408009860 tx=0x7fb40802a380 comp rx=0 tx=0).stop 2024-09-15T23:28:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.430+0000 7fb4127d8640 1 -- 172.21.15.62:0/859953292 shutdown_connections 2024-09-15T23:28:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.430+0000 7fb4127d8640 1 --2- 172.21.15.62:0/859953292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb40c11b140 0x7fb40c11d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.430+0000 7fb4127d8640 1 --2- 172.21.15.62:0/859953292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c06ea30 0x7fb40c06ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.430+0000 7fb4127d8640 1 --2- 172.21.15.62:0/859953292 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb40c110190 0x7fb40c06e4f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.430+0000 7fb4127d8640 1 -- 172.21.15.62:0/859953292 >> 172.21.15.62:0/859953292 conn(0x7fb40c073180 msgr2=0x7fb40c073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:00.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.430+0000 7fb4127d8640 1 -- 172.21.15.62:0/859953292 shutdown_connections 2024-09-15T23:28:00.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.431+0000 7fb4127d8640 1 -- 172.21.15.62:0/859953292 wait complete. 2024-09-15T23:28:00.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.432+0000 7fb4127d8640 1 Processor -- start 2024-09-15T23:28:00.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.435+0000 7fb4127d8640 1 -- start start 2024-09-15T23:28:00.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.435+0000 7fb4127d8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c06ea30 0x7fb40c11a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.435+0000 7fb4127d8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb40c110190 0x7fb40c116a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.435+0000 7fb4127d8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb40c11b140 0x7fb40c116f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.435+0000 7fb4127d8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c113300 con 0x7fb40c11b140 2024-09-15T23:28:00.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.435+0000 7fb4127d8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c113180 con 0x7fb40c06ea30 2024-09-15T23:28:00.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.435+0000 7fb4127d8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c113480 con 0x7fb40c110190 2024-09-15T23:28:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.439+0000 7fb411fd7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb40c11b140 0x7fb40c116f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.439+0000 7fb411fd7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb40c11b140 0x7fb40c116f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55588/0 (socket says 172.21.15.62:55588) 2024-09-15T23:28:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.439+0000 7fb411fd7640 1 -- 172.21.15.62:0/3786290375 learned_addr learned my addr 172.21.15.62:0/3786290375 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.439+0000 7fb410fd5640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb40c110190 0x7fb40c116a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.439+0000 7fb4117d6640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c06ea30 0x7fb40c11a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.440+0000 7fb4117d6640 1 -- 172.21.15.62:0/3786290375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb40c110190 msgr2=0x7fb40c116a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:00.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.440+0000 7fb4117d6640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb40c110190 0x7fb40c116a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.440+0000 7fb4117d6640 1 -- 172.21.15.62:0/3786290375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb40c11b140 msgr2=0x7fb40c116f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:00.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.440+0000 7fb4117d6640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb40c11b140 0x7fb40c116f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.440+0000 7fb4117d6640 1 -- 172.21.15.62:0/3786290375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb40c117830 con 0x7fb40c06ea30 2024-09-15T23:28:00.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.440+0000 7fb4117d6640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c06ea30 0x7fb40c11a990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb408035040 tx=0x7fb408033d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:00.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.442+0000 7fb4027fc640 1 -- 172.21.15.62:0/3786290375 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb408037040 con 0x7fb40c06ea30 2024-09-15T23:28:00.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.442+0000 7fb4027fc640 1 -- 172.21.15.62:0/3786290375 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4080303b0 con 0x7fb40c06ea30 2024-09-15T23:28:00.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.442+0000 7fb4027fc640 1 -- 172.21.15.62:0/3786290375 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb408005380 con 0x7fb40c06ea30 2024-09-15T23:28:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.444+0000 7fb4127d8640 1 -- 172.21.15.62:0/3786290375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb40c1aaf60 con 0x7fb40c06ea30 2024-09-15T23:28:00.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.444+0000 7fb4127d8640 1 -- 172.21.15.62:0/3786290375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40c1ab300 con 0x7fb40c06ea30 2024-09-15T23:28:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.451+0000 7fb415e13740 1 Processor -- start 2024-09-15T23:28:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.453+0000 7fb415e13740 1 -- start start 2024-09-15T23:28:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.453+0000 7fb4027fc640 1 -- 172.21.15.62:0/3786290375 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb408042070 con 0x7fb40c06ea30 2024-09-15T23:28:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.453+0000 7fb415e13740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.453+0000 7fb415e13740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 0x5626af068310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.453+0000 7fb415e13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af068850 0x5626af06ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.453+0000 7fb415e13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5626aefc32e0 con 0x5626af068850 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.453+0000 7fb415e13740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5626aef6ace0 con 0x5626af05df90 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.453+0000 7fb415e13740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5626aefbf2a0 con 0x5626af05d670 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.454+0000 7fb411fd7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.454+0000 7fb411fd7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:50852/0 (socket says 172.21.15.62:50852) 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.454+0000 7fb411fd7640 1 -- 172.21.15.62:0/3612870861 learned_addr learned my addr 172.21.15.62:0/3612870861 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.456+0000 7fb4117d6640 1 --2- 172.21.15.62:0/3612870861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 0x5626af068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.457+0000 7fb4027fc640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3f80a3450 0x7fb3f80a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.458+0000 7fb410fd5640 1 --2- 172.21.15.62:0/3612870861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af068850 0x5626af06ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.458+0000 7fb4027fc640 1 -- 172.21.15.62:0/3786290375 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb4080eee40 con 0x7fb40c06ea30 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.458+0000 7fb411fd7640 1 -- 172.21.15.62:0/3612870861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 msgr2=0x5626af068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.458+0000 7fb411fd7640 1 --2- 172.21.15.62:0/3612870861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 0x5626af068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.458+0000 7fb411fd7640 1 -- 172.21.15.62:0/3612870861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af068850 msgr2=0x5626af06ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.458+0000 7fb411fd7640 1 --2- 172.21.15.62:0/3612870861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af068850 0x5626af06ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.458+0000 7fb411fd7640 1 -- 172.21.15.62:0/3612870861 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626aef4f440 con 0x5626af05d670 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.458+0000 7fb411fd7640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3f80a3450 0x7fb3f80a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.458+0000 7fb411fd7640 1 --2- 172.21.15.62:0/3612870861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05da50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb4040205e0 tx=0x7fb404020aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7093f3f840d85fa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.459+0000 7fb411fd7640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3f80a3450 0x7fb3f80a5910 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb404033000 tx=0x7fb40402c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:00.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.459+0000 7fb3f77fe640 1 -- 172.21.15.62:0/3612870861 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40402c440 con 0x5626af05d670 2024-09-15T23:28:00.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.461+0000 7fb3f77fe640 1 -- 172.21.15.62:0/3612870861 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb40402cb80 con 0x5626af05d670 2024-09-15T23:28:00.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.462+0000 7fb415e13740 1 -- 172.21.15.62:0/3612870861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 msgr2=0x5626af05da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:00.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.462+0000 7fb415e13740 1 --2- 172.21.15.62:0/3612870861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05da50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb4040205e0 tx=0x7fb404020aa0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.462+0000 7fb415e13740 1 -- 172.21.15.62:0/3612870861 shutdown_connections 2024-09-15T23:28:00.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.462+0000 7fb415e13740 1 --2- 172.21.15.62:0/3612870861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af068850 0x5626af06ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.462+0000 7fb415e13740 1 --2- 172.21.15.62:0/3612870861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 0x5626af068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.462+0000 7fb415e13740 1 --2- 172.21.15.62:0/3612870861 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05da50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.462+0000 7fb415e13740 1 -- 172.21.15.62:0/3612870861 >> 172.21.15.62:0/3612870861 conn(0x5626af059c00 msgr2=0x5626af059fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.465+0000 7fb415e13740 1 -- 172.21.15.62:0/3612870861 shutdown_connections 2024-09-15T23:28:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.465+0000 7fb415e13740 1 -- 172.21.15.62:0/3612870861 wait complete. 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.465+0000 7fb415e13740 1 -- start start 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.465+0000 7fb415e13740 1 Processor -- start 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb415e13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05d670 0x5626af0680b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb415e13740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05df90 0x5626af065360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb415e13740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af068850 0x5626af0658a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb415e13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5626af06b4c0 con 0x5626af05d670 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb415e13740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5626aef7e050 con 0x5626af068850 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb415e13740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5626af06b640 con 0x5626af05df90 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb415e13740 10 client.0 _ll_register_callbacks cb 0x5626aefbe160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb410fd5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05df90 0x5626af065360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb410fd5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05df90 0x5626af065360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:50862/0 (socket says 172.21.15.62:50862) 2024-09-15T23:28:00.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.466+0000 7fb410fd5640 1 -- 172.21.15.62:0/1629795644 learned_addr learned my addr 172.21.15.62:0/1629795644 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:00.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.467+0000 7fb411fd7640 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af068850 0x5626af0658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.467+0000 7fb410fd5640 1 -- 172.21.15.62:0/1629795644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af068850 msgr2=0x5626af0658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:00.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.467+0000 7fb410fd5640 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af068850 0x5626af0658a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.467+0000 7fb410fd5640 1 -- 172.21.15.62:0/1629795644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05d670 msgr2=0x5626af0680b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:00.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.467+0000 7fb410fd5640 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05d670 0x5626af0680b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.467+0000 7fb410fd5640 1 -- 172.21.15.62:0/1629795644 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626aed343c0 con 0x5626af05df90 2024-09-15T23:28:00.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.472+0000 7fb410fd5640 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05df90 0x5626af065360 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc012700 tx=0x7fb3fc012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b899ec87208a8037 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:00.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.476+0000 7fb3f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:00.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.476+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/1629795644 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc004050 con 0x5626af05df90 2024-09-15T23:28:00.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.476+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/1629795644 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb3fc0041f0 con 0x5626af05df90 2024-09-15T23:28:00.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.476+0000 7fb415e13740 1 -- 172.21.15.62:0/1629795644 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb40402cb80 con 0x5626af05df90 2024-09-15T23:28:00.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.476+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/1629795644 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb3fc0043f0 con 0x5626af05df90 2024-09-15T23:28:00.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.478+0000 7fb415e13740 10 client.27496 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:00.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.478+0000 7fb415e13740 1 -- 172.21.15.62:0/1629795644 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5626af076ee0 con 0x5626af05df90 2024-09-15T23:28:00.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.480+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/1629795644 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb40402cb80 con 0x5626af05df90 2024-09-15T23:28:00.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.480+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/1629795644 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fb3fc003c80 con 0x5626af05df90 2024-09-15T23:28:00.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.481+0000 7fb415e13740 10 client.27496 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:00.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.481+0000 7fb415e13740 10 client.27496 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:00.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.481+0000 7fb415e13740 20 client.27496 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:00.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.481+0000 7fb415e13740 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5626af0783a0 0x5626af07a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.481+0000 7fb415e13740 4 client.27496 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:00.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.481+0000 7fb415e13740 1 -- 172.21.15.62:0/1629795644 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626aefd0850 con 0x5626af0783a0 2024-09-15T23:28:00.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.483+0000 7fb4117d6640 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5626af0783a0 0x5626af07a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.489+0000 7fb4117d6640 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5626af0783a0 0x5626af07a7d0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65602f3b85fac9ee server_cookie=9db1e1411bb20ebc in_seq=0 out_seq=0 2024-09-15T23:28:00.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.489+0000 7fb3f4ff9640 10 client.27496 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:00.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.492+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/1629795644 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5626aefd0850 con 0x5626af0783a0 2024-09-15T23:28:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.492+0000 7fb3f4ff9640 10 client.27496 handle_command_reply: tid=0 2024-09-15T23:28:00.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.492+0000 7fb415e13740 1 client.27496 shutdown 2024-09-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.497+0000 7fb415e13740 1 -- 172.21.15.62:0/1629795644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05df90 msgr2=0x5626af065360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:00.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.497+0000 7fb415e13740 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05df90 0x5626af065360 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc012700 tx=0x7fb3fc012bc0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.497+0000 7fb415e13740 1 -- 172.21.15.62:0/1629795644 shutdown_connections 2024-09-15T23:28:00.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.497+0000 7fb415e13740 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5626af0783a0 0x5626af07a7d0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.497+0000 7fb415e13740 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af068850 0x5626af0658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.497+0000 7fb415e13740 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05df90 0x5626af065360 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.497+0000 7fb415e13740 1 --2- 172.21.15.62:0/1629795644 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05d670 0x5626af0680b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.497+0000 7fb415e13740 1 -- 172.21.15.62:0/1629795644 >> 172.21.15.62:0/1629795644 conn(0x5626af059c00 msgr2=0x5626af058d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:00.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.497+0000 7fb3f4ff9640 0 client.27496 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:00.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.503+0000 7fb415e13740 1 -- 172.21.15.62:0/1629795644 shutdown_connections 2024-09-15T23:28:00.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.504+0000 7fb415e13740 1 -- 172.21.15.62:0/1629795644 wait complete. 2024-09-15T23:28:00.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.504+0000 7fb415e13740 20 client.27496 trim_cache size 0 max 16384 2024-09-15T23:28:00.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.540+0000 7fb415e13740 1 Processor -- start 2024-09-15T23:28:00.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.541+0000 7fb415e13740 1 -- start start 2024-09-15T23:28:00.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.541+0000 7fb415e13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af059c00 0x5626af063c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.541+0000 7fb415e13740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.541+0000 7fb415e13740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 0x5626af0800e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.541+0000 7fb415e13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c1b7b60 con 0x5626af059c00 2024-09-15T23:28:00.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.541+0000 7fb415e13740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c1b5430 con 0x5626af05df90 2024-09-15T23:28:00.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.541+0000 7fb415e13740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c1b3280 con 0x5626af05d670 2024-09-15T23:28:00.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.542+0000 7fb410fd5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.542+0000 7fb411fd7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 0x5626af0800e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.543+0000 7fb411fd7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 0x5626af0800e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54230/0 (socket says 172.21.15.62:54230) 2024-09-15T23:28:00.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.543+0000 7fb411fd7640 1 -- 172.21.15.62:0/2545779652 learned_addr learned my addr 172.21.15.62:0/2545779652 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:00.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.543+0000 7fb4117d6640 1 --2- 172.21.15.62:0/2545779652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af059c00 0x5626af063c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.543+0000 7fb410fd5640 1 -- 172.21.15.62:0/2545779652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 msgr2=0x5626af0800e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.543+0000 7fb410fd5640 1 --2- 172.21.15.62:0/2545779652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 0x5626af0800e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.543+0000 7fb410fd5640 1 -- 172.21.15.62:0/2545779652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af059c00 msgr2=0x5626af063c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.543+0000 7fb410fd5640 1 --2- 172.21.15.62:0/2545779652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af059c00 0x5626af063c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.543+0000 7fb410fd5640 1 -- 172.21.15.62:0/2545779652 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626af064310 con 0x5626af05d670 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.544+0000 7fb410fd5640 1 --2- 172.21.15.62:0/2545779652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05f7d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x5626af066910 tx=0x7fb3fc009d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4524d69fd67af620 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.544+0000 7fb3f6ffd640 1 -- 172.21.15.62:0/2545779652 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc0106f0 con 0x5626af05d670 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.545+0000 7fb3f6ffd640 1 -- 172.21.15.62:0/2545779652 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3fc010890 con 0x5626af05d670 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.545+0000 7fb3f6ffd640 1 -- 172.21.15.62:0/2545779652 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc02e800 con 0x5626af05d670 2024-09-15T23:28:00.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.546+0000 7fb415e13740 1 -- 172.21.15.62:0/2545779652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 msgr2=0x5626af05f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:00.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.546+0000 7fb415e13740 1 --2- 172.21.15.62:0/2545779652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05f7d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x5626af066910 tx=0x7fb3fc009d50 comp rx=0 tx=0).stop 2024-09-15T23:28:00.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.547+0000 7fb415e13740 1 -- 172.21.15.62:0/2545779652 shutdown_connections 2024-09-15T23:28:00.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.547+0000 7fb415e13740 1 --2- 172.21.15.62:0/2545779652 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af05df90 0x5626af0800e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.547+0000 7fb415e13740 1 --2- 172.21.15.62:0/2545779652 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af05f7d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.547+0000 7fb415e13740 1 --2- 172.21.15.62:0/2545779652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af059c00 0x5626af063c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.547+0000 7fb415e13740 1 -- 172.21.15.62:0/2545779652 >> 172.21.15.62:0/2545779652 conn(0x5626af065a00 msgr2=0x5626af05f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:00.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.548+0000 7fb415e13740 1 -- 172.21.15.62:0/2545779652 shutdown_connections 2024-09-15T23:28:00.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.548+0000 7fb415e13740 1 -- 172.21.15.62:0/2545779652 wait complete. 2024-09-15T23:28:00.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.548+0000 7fb415e13740 1 -- start start 2024-09-15T23:28:00.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.548+0000 7fb415e13740 1 Processor -- start 2024-09-15T23:28:00.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb415e13740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af059c00 0x5626af070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb415e13740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af0710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb415e13740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05df90 0x5626af06b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb415e13740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c1b38f0 con 0x5626af05df90 2024-09-15T23:28:00.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb415e13740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c1b4d60 con 0x5626af059c00 2024-09-15T23:28:00.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb415e13740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb40c1b3fc0 con 0x5626af05d670 2024-09-15T23:28:00.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb415e13740 10 client.0 _ll_register_callbacks cb 0x7fb3fc0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:00.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb4117d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af059c00 0x5626af070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb4117d6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af059c00 0x5626af070bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54234/0 (socket says 172.21.15.62:54234) 2024-09-15T23:28:00.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.549+0000 7fb4117d6640 1 -- 172.21.15.62:0/127882902 learned_addr learned my addr 172.21.15.62:0/127882902 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:00.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.550+0000 7fb411fd7640 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05df90 0x5626af06b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.550+0000 7fb410fd5640 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af0710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.550+0000 7fb411fd7640 1 -- 172.21.15.62:0/127882902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 msgr2=0x5626af0710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:00.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.550+0000 7fb411fd7640 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af0710f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.550+0000 7fb411fd7640 1 -- 172.21.15.62:0/127882902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af059c00 msgr2=0x5626af070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:00.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.550+0000 7fb411fd7640 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af059c00 0x5626af070bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.550+0000 7fb411fd7640 1 -- 172.21.15.62:0/127882902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626af068420 con 0x5626af05df90 2024-09-15T23:28:00.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.552+0000 7fb411fd7640 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05df90 0x5626af06b8f0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb40401a550 tx=0x7fb40401ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994e950e86ffea7d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:00.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.552+0000 7fb3f4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:00.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.554+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/127882902 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb404044060 con 0x5626af05df90 2024-09-15T23:28:00.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.554+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/127882902 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb404004c10 con 0x5626af05df90 2024-09-15T23:28:00.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.554+0000 7fb415e13740 1 -- 172.21.15.62:0/127882902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3fc010890 con 0x5626af05df90 2024-09-15T23:28:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.559+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/127882902 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb40404b050 con 0x5626af05df90 2024-09-15T23:28:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.560+0000 7fb415e13740 10 client.18207 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:00.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.560+0000 7fb415e13740 1 -- 172.21.15.62:0/127882902 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5626af105a50 con 0x5626af05df90 2024-09-15T23:28:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.564+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/127882902 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3fc010890 con 0x5626af05df90 2024-09-15T23:28:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.564+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/127882902 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fb40402cce0 con 0x5626af05df90 2024-09-15T23:28:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.565+0000 7fb415e13740 10 client.18207 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.565+0000 7fb415e13740 10 client.18207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.565+0000 7fb415e13740 20 client.18207 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.565+0000 7fb415e13740 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5626af068850 0x5626af107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.565+0000 7fb415e13740 4 client.18207 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:28:00.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.565+0000 7fb415e13740 1 -- 172.21.15.62:0/127882902 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5626aefcad60 con 0x5626af068850 2024-09-15T23:28:00.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.570+0000 7fb4117d6640 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5626af068850 0x5626af107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:00.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.581+0000 7fb4117d6640 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5626af068850 0x5626af107ee0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1260133ff99fa500 server_cookie=829577dfbb8e5698 in_seq=0 out_seq=0 2024-09-15T23:28:00.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.583+0000 7fb3f4ff9640 10 client.18207 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:00.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.583+0000 7fb3f4ff9640 1 -- 172.21.15.62:0/127882902 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5626aefcad60 con 0x5626af068850 2024-09-15T23:28:00.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.583+0000 7fb3f4ff9640 10 client.18207 handle_command_reply: tid=0 2024-09-15T23:28:00.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.584+0000 7fb415e13740 1 client.18207 shutdown 2024-09-15T23:28:00.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.586+0000 7fb415e13740 1 -- 172.21.15.62:0/127882902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05df90 msgr2=0x5626af06b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:00.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.586+0000 7fb415e13740 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05df90 0x5626af06b8f0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fb40401a550 tx=0x7fb40401ed20 comp rx=0 tx=0).stop 2024-09-15T23:28:00.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.586+0000 7fb415e13740 1 -- 172.21.15.62:0/127882902 shutdown_connections 2024-09-15T23:28:00.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.586+0000 7fb415e13740 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5626af068850 0x5626af107ee0 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.586+0000 7fb415e13740 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5626af05df90 0x5626af06b8f0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.586+0000 7fb415e13740 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5626af05d670 0x5626af0710f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.586+0000 7fb415e13740 1 --2- 172.21.15.62:0/127882902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5626af059c00 0x5626af070bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.586+0000 7fb415e13740 1 -- 172.21.15.62:0/127882902 >> 172.21.15.62:0/127882902 conn(0x5626af065a00 msgr2=0x5626af066bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:00.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.587+0000 7fb3f4ff9640 0 client.18207 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:00.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.587+0000 7fb415e13740 1 -- 172.21.15.62:0/127882902 shutdown_connections 2024-09-15T23:28:00.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.591+0000 7fb415e13740 1 -- 172.21.15.62:0/127882902 wait complete. 2024-09-15T23:28:00.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.591+0000 7fb415e13740 20 client.18207 trim_cache size 0 max 16384 2024-09-15T23:28:00.593 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:28:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:28:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:28:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "031d0c8d-e514-441a-bc06-2297e09d88d9": { 2024-09-15T23:28:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:28:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "031d0c8d-e514-441a-bc06-2297e09d88d9", 2024-09-15T23:28:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:28:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:28:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:28:00.594 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:28:00.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.595+0000 7fb4127d8640 1 -- 172.21.15.62:0/3786290375 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3f80a3450 msgr2=0x7fb3f80a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:00.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.595+0000 7fb4127d8640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3f80a3450 0x7fb3f80a5910 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb404033000 tx=0x7fb40402c000 comp rx=0 tx=0).stop 2024-09-15T23:28:00.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.595+0000 7fb4127d8640 1 -- 172.21.15.62:0/3786290375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c06ea30 msgr2=0x7fb40c11a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:00.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.595+0000 7fb4127d8640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c06ea30 0x7fb40c11a990 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb408035040 tx=0x7fb408033d10 comp rx=0 tx=0).stop 2024-09-15T23:28:00.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.596+0000 7fb4127d8640 1 -- 172.21.15.62:0/3786290375 shutdown_connections 2024-09-15T23:28:00.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.596+0000 7fb4127d8640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3f80a3450 0x7fb3f80a5910 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.596+0000 7fb4127d8640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb40c11b140 0x7fb40c116f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.596+0000 7fb4127d8640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb40c110190 0x7fb40c116a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:00.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.596+0000 7fb4127d8640 1 --2- 172.21.15.62:0/3786290375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40c06ea30 0x7fb40c11a990 secure :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb408035040 tx=0x7fb408033d10 comp rx=0 tx=0).stop 2024-09-15T23:28:00.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.596+0000 7fb4127d8640 1 -- 172.21.15.62:0/3786290375 >> 172.21.15.62:0/3786290375 conn(0x7fb40c073180 msgr2=0x7fb40c11b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:00.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.599+0000 7fb4127d8640 1 -- 172.21.15.62:0/3786290375 shutdown_connections 2024-09-15T23:28:00.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:00.600+0000 7fb4127d8640 1 -- 172.21.15.62:0/3786290375 wait complete. 2024-09-15T23:28:01.456 INFO:tasks.cephfs.filesystem:scrub status for tag:031d0c8d-e514-441a-bc06-2297e09d88d9 - {'path': '/', 'tag': '031d0c8d-e514-441a-bc06-2297e09d88d9', 'options': 'recursive,force'} 2024-09-15T23:28:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:01 smithi067 ceph-mon[35665]: pgmap v722: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.2 MiB/s wr, 579 op/s 2024-09-15T23:28:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:01 smithi148 ceph-mon[35610]: pgmap v722: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.2 MiB/s wr, 579 op/s 2024-09-15T23:28:02.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.182+0000 7fb1fb282640 1 -- 172.21.15.62:0/1924705667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f41003b0 msgr2=0x7fb1f41027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.182+0000 7fb1fb282640 1 --2- 172.21.15.62:0/1924705667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f41003b0 0x7fb1f41027c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009900 tx=0x7fb1f002a440 comp rx=0 tx=0).stop 2024-09-15T23:28:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.182+0000 7fb1fb282640 1 -- 172.21.15.62:0/1924705667 shutdown_connections 2024-09-15T23:28:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.182+0000 7fb1fb282640 1 --2- 172.21.15.62:0/1924705667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f41056b0 0x7fb1f4105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.182+0000 7fb1fb282640 1 --2- 172.21.15.62:0/1924705667 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1f4102d00 0x7fb1f4105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.182+0000 7fb1fb282640 1 --2- 172.21.15.62:0/1924705667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f41003b0 0x7fb1f41027c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.182+0000 7fb1fb282640 1 -- 172.21.15.62:0/1924705667 >> 172.21.15.62:0/1924705667 conn(0x7fb1f40fc1e0 msgr2=0x7fb1f40fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.183+0000 7fb1fb282640 1 -- 172.21.15.62:0/1924705667 shutdown_connections 2024-09-15T23:28:02.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.183+0000 7fb1fb282640 1 -- 172.21.15.62:0/1924705667 wait complete. 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.183+0000 7fb1fb282640 1 Processor -- start 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.183+0000 7fb1fb282640 1 -- start start 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1fb282640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f41003b0 0x7fb1f419e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1fb282640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1f4102d00 0x7fb1f419eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1fb282640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f41056b0 0x7fb1f41a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1fb282640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f4113f20 con 0x7fb1f41003b0 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1fb282640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f4113da0 con 0x7fb1f41056b0 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1fb282640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41140a0 con 0x7fb1f4102d00 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f901e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f41003b0 0x7fb1f419e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f901e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f41003b0 0x7fb1f419e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55618/0 (socket says 172.21.15.62:55618) 2024-09-15T23:28:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f901e640 1 -- 172.21.15.62:0/762150710 learned_addr learned my addr 172.21.15.62:0/762150710 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f881d640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1f4102d00 0x7fb1f419eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f981f640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f41056b0 0x7fb1f41a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f881d640 1 -- 172.21.15.62:0/762150710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f41056b0 msgr2=0x7fb1f41a5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f881d640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f41056b0 0x7fb1f41a5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f881d640 1 -- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f41003b0 msgr2=0x7fb1f419e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f881d640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f41003b0 0x7fb1f419e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f881d640 1 -- 172.21.15.62:0/762150710 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f41a6190 con 0x7fb1f4102d00 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.184+0000 7fb1f901e640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f41003b0 0x7fb1f419e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.185+0000 7fb1f881d640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1f4102d00 0x7fb1f419eab0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009fd0 tx=0x7fb1e400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.185+0000 7fb1ea7fc640 1 -- 172.21.15.62:0/762150710 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4015020 con 0x7fb1f4102d00 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.185+0000 7fb1ea7fc640 1 -- 172.21.15.62:0/762150710 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1e4011d10 con 0x7fb1f4102d00 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.185+0000 7fb1fb282640 1 -- 172.21.15.62:0/762150710 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f41a6480 con 0x7fb1f4102d00 2024-09-15T23:28:02.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.185+0000 7fb1ea7fc640 1 -- 172.21.15.62:0/762150710 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e400f2e0 con 0x7fb1f4102d00 2024-09-15T23:28:02.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.185+0000 7fb1fb282640 1 -- 172.21.15.62:0/762150710 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f41a69a0 con 0x7fb1f4102d00 2024-09-15T23:28:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.188+0000 7fb1ea7fc640 1 -- 172.21.15.62:0/762150710 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb1e4018020 con 0x7fb1f4102d00 2024-09-15T23:28:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.189+0000 7fb1ea7fc640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1e00a3370 0x7fb1e00a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.189+0000 7fb1fe020740 1 Processor -- start 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.189+0000 7fb1f901e640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1e00a3370 0x7fb1e00a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.189+0000 7fb1ea7fc640 1 -- 172.21.15.62:0/762150710 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb1e40cf530 con 0x7fb1f4102d00 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.189+0000 7fb1fe020740 1 -- start start 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.189+0000 7fb1f901e640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1e00a3370 0x7fb1e00a5830 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00096c0 tx=0x7fb1f00315b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.189+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec443da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec448da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec44b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1f901e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec443da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1f901e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec443da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55628/0 (socket says 172.21.15.62:55628) 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1f901e640 1 -- 172.21.15.62:0/1070898070 learned_addr learned my addr 172.21.15.62:0/1070898070 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1f881d640 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec448da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1f981f640 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec44b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1fe020740 1 -- 172.21.15.62:0/1070898070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564cec3a72f0 con 0x564cec4439a0 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1fe020740 1 -- 172.21.15.62:0/1070898070 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564cec34f270 con 0x564cec449370 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.190+0000 7fb1fe020740 1 -- 172.21.15.62:0/1070898070 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564cec3a32b0 con 0x564cec4442e0 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1f901e640 1 -- 172.21.15.62:0/1070898070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 msgr2=0x564cec448da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1f901e640 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec448da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1f901e640 1 -- 172.21.15.62:0/1070898070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 msgr2=0x564cec44b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1f901e640 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec44b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1f901e640 1 -- 172.21.15.62:0/1070898070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cec34e340 con 0x564cec4439a0 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1f881d640 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec448da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1f981f640 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec44b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1f901e640 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec443da0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0006010 tx=0x7fb1f003d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22cb59ea1dcb62a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1df7fe640 1 -- 172.21.15.62:0/1070898070 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0035040 con 0x564cec4439a0 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1df7fe640 1 -- 172.21.15.62:0/1070898070 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1f004e070 con 0x564cec4439a0 2024-09-15T23:28:02.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.191+0000 7fb1df7fe640 1 -- 172.21.15.62:0/1070898070 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f003c440 con 0x564cec4439a0 2024-09-15T23:28:02.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.192+0000 7fb1fe020740 1 -- 172.21.15.62:0/1070898070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 msgr2=0x564cec443da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:02.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.192+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec443da0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0006010 tx=0x7fb1f003d000 comp rx=0 tx=0).stop 2024-09-15T23:28:02.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.192+0000 7fb1fe020740 1 -- 172.21.15.62:0/1070898070 shutdown_connections 2024-09-15T23:28:02.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.192+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec44b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.192+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec448da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.192+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1070898070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec443da0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.192+0000 7fb1fe020740 1 -- 172.21.15.62:0/1070898070 >> 172.21.15.62:0/1070898070 conn(0x564cec43f010 msgr2=0x564cec43f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:02.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.192+0000 7fb1fe020740 1 -- 172.21.15.62:0/1070898070 shutdown_connections 2024-09-15T23:28:02.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.192+0000 7fb1fe020740 1 -- 172.21.15.62:0/1070898070 wait complete. 2024-09-15T23:28:02.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.193+0000 7fb1fe020740 1 -- start start 2024-09-15T23:28:02.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.193+0000 7fb1fe020740 1 Processor -- start 2024-09-15T23:28:02.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec44db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec44e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec4550b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1fe020740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564cec44c2e0 con 0x564cec4439a0 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1f881d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec44e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1fe020740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564cec330290 con 0x564cec449370 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1fe020740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564cec44c460 con 0x564cec4442e0 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1f881d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec44e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:50924/0 (socket says 172.21.15.62:50924) 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1f881d640 1 -- 172.21.15.62:0/3670433108 learned_addr learned my addr 172.21.15.62:0/3670433108 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.196+0000 7fb1fe020740 10 client.0 _ll_register_callbacks cb 0x564cec361140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1f981f640 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec4550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.198 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1f901e640 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec44db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1f881d640 1 -- 172.21.15.62:0/3670433108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 msgr2=0x564cec4550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:02.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1f881d640 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec4550b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1f881d640 1 -- 172.21.15.62:0/3670433108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 msgr2=0x564cec44db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1f881d640 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec44db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1f881d640 1 -- 172.21.15.62:0/3670433108 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cec0ba210 con 0x564cec4442e0 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1f881d640 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec44e070 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400bd70 tx=0x7fb1e409b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=613afdbf4ae7c230 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.197+0000 7fb1dcff9640 1 -- 172.21.15.62:0/3670433108 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e40b3030 con 0x564cec4442e0 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.198+0000 7fb1dcff9640 1 -- 172.21.15.62:0/3670433108 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb1e40cf1f0 con 0x564cec4442e0 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.198+0000 7fb1fe020740 1 -- 172.21.15.62:0/3670433108 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1f004e070 con 0x564cec4442e0 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.198+0000 7fb1dcff9640 1 -- 172.21.15.62:0/3670433108 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb1e409b770 con 0x564cec4442e0 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.198+0000 7fb1fe020740 10 client.27511 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.198+0000 7fb1fe020740 1 -- 172.21.15.62:0/3670433108 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564cec186620 con 0x564cec4442e0 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.198+0000 7fb1dcff9640 1 -- 172.21.15.62:0/3670433108 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1f004e070 con 0x564cec4442e0 2024-09-15T23:28:02.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.199+0000 7fb1dcff9640 1 -- 172.21.15.62:0/3670433108 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fb1e4011830 con 0x564cec4442e0 2024-09-15T23:28:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.199+0000 7fb1fe020740 10 client.27511 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.199+0000 7fb1fe020740 10 client.27511 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.199+0000 7fb1fe020740 20 client.27511 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.199+0000 7fb1fe020740 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cec459750 0x564cec45bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.199+0000 7fb1fe020740 4 client.27511 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.199+0000 7fb1fe020740 1 -- 172.21.15.62:0/3670433108 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cec3b4a20 con 0x564cec459750 2024-09-15T23:28:02.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.199+0000 7fb1f901e640 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cec459750 0x564cec45bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.200+0000 7fb1dcff9640 10 client.27511 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.200+0000 7fb1f901e640 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cec459750 0x564cec45bb60 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4c25d1315d35d47 server_cookie=132f9f5cea50fd37 in_seq=0 out_seq=0 2024-09-15T23:28:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.200+0000 7fb1dcff9640 1 -- 172.21.15.62:0/3670433108 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564cec3b4a20 con 0x564cec459750 2024-09-15T23:28:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.200+0000 7fb1dcff9640 10 client.27511 handle_command_reply: tid=0 2024-09-15T23:28:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 client.27511 shutdown 2024-09-15T23:28:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 -- 172.21.15.62:0/3670433108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 msgr2=0x564cec44e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec44e070 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb1e400bd70 tx=0x7fb1e409b600 comp rx=0 tx=0).stop 2024-09-15T23:28:02.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 -- 172.21.15.62:0/3670433108 shutdown_connections 2024-09-15T23:28:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cec459750 0x564cec45bb60 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec4550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec44e070 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1dcff9640 0 client.27511 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 --2- 172.21.15.62:0/3670433108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec44db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 -- 172.21.15.62:0/3670433108 >> 172.21.15.62:0/3670433108 conn(0x564cec43f010 msgr2=0x564cec440820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 -- 172.21.15.62:0/3670433108 shutdown_connections 2024-09-15T23:28:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 1 -- 172.21.15.62:0/3670433108 wait complete. 2024-09-15T23:28:02.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.201+0000 7fb1fe020740 20 client.27511 trim_cache size 0 max 16384 2024-09-15T23:28:02.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.222+0000 7fb1fe020740 1 Processor -- start 2024-09-15T23:28:02.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.223+0000 7fb1fe020740 1 -- start start 2024-09-15T23:28:02.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.223+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec43e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.223+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec455070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.223+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec460450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.223+0000 7fb1fe020740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41b3510 con 0x564cec4439a0 2024-09-15T23:28:02.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.223+0000 7fb1fe020740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41b0de0 con 0x564cec449370 2024-09-15T23:28:02.225 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.223+0000 7fb1fe020740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41aec30 con 0x564cec4442e0 2024-09-15T23:28:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.224+0000 7fb1f981f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec460450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.224+0000 7fb1f981f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec460450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54286/0 (socket says 172.21.15.62:54286) 2024-09-15T23:28:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.224+0000 7fb1f981f640 1 -- 172.21.15.62:0/1309114567 learned_addr learned my addr 172.21.15.62:0/1309114567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.224+0000 7fb1f901e640 1 --2- 172.21.15.62:0/1309114567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec43e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.225+0000 7fb1f981f640 1 -- 172.21.15.62:0/1309114567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 msgr2=0x564cec455070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:28:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.225+0000 7fb1f981f640 1 --2- 172.21.15.62:0/1309114567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec455070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.225+0000 7fb1f981f640 1 -- 172.21.15.62:0/1309114567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 msgr2=0x564cec43e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.225+0000 7fb1f981f640 1 --2- 172.21.15.62:0/1309114567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec43e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.225+0000 7fb1f981f640 1 -- 172.21.15.62:0/1309114567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cec44d670 con 0x564cec449370 2024-09-15T23:28:02.227 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.225+0000 7fb1f981f640 1 --2- 172.21.15.62:0/1309114567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec460450 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00b4a0 tx=0x7fb1ec009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8a2dc6f95e4877e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:02.230 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.229+0000 7fb1deffd640 1 -- 172.21.15.62:0/1309114567 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec00a8b0 con 0x564cec449370 2024-09-15T23:28:02.230 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.229+0000 7fb1deffd640 1 -- 172.21.15.62:0/1309114567 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1ec017420 con 0x564cec449370 2024-09-15T23:28:02.230 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.229+0000 7fb1deffd640 1 -- 172.21.15.62:0/1309114567 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec01f660 con 0x564cec449370 2024-09-15T23:28:02.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:01 smithi062 ceph-mon[30949]: pgmap v722: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 6.2 MiB/s wr, 579 op/s 2024-09-15T23:28:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.231+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309114567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 msgr2=0x564cec460450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:02.232 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.231+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309114567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec460450 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00b4a0 tx=0x7fb1ec009630 comp rx=0 tx=0).stop 2024-09-15T23:28:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.235+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309114567 shutdown_connections 2024-09-15T23:28:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.235+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309114567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec460450 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.235+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309114567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4442e0 0x564cec455070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.235+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309114567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4439a0 0x564cec43e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.241 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.235+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309114567 >> 172.21.15.62:0/1309114567 conn(0x564cec43f010 msgr2=0x564cec43d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:02.244 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.242+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309114567 shutdown_connections 2024-09-15T23:28:02.247 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.246+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309114567 wait complete. 2024-09-15T23:28:02.247 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.246+0000 7fb1fe020740 1 -- start start 2024-09-15T23:28:02.247 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.246+0000 7fb1fe020740 1 Processor -- start 2024-09-15T23:28:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.248+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4439a0 0x564cec452e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.248+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4442e0 0x564cec4533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.248+0000 7fb1fe020740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec4543d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.248+0000 7fb1fe020740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41af2a0 con 0x564cec4442e0 2024-09-15T23:28:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.248+0000 7fb1fe020740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41b0710 con 0x564cec449370 2024-09-15T23:28:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.248+0000 7fb1fe020740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41af970 con 0x564cec4439a0 2024-09-15T23:28:02.249 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.248+0000 7fb1fe020740 10 client.0 _ll_register_callbacks cb 0x7fb1e4010630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.250+0000 7fb1f881d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4442e0 0x564cec4533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.250+0000 7fb1f881d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4442e0 0x564cec4533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55660/0 (socket says 172.21.15.62:55660) 2024-09-15T23:28:02.251 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.250+0000 7fb1f881d640 1 -- 172.21.15.62:0/1309475191 learned_addr learned my addr 172.21.15.62:0/1309475191 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:02.254 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.250+0000 7fb1f901e640 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4439a0 0x564cec452e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.255 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.250+0000 7fb1f981f640 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec4543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.254+0000 7fb1f901e640 1 -- 172.21.15.62:0/1309475191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 msgr2=0x564cec4543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:02.256 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.254+0000 7fb1f901e640 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec4543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.254+0000 7fb1f901e640 1 -- 172.21.15.62:0/1309475191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4442e0 msgr2=0x564cec4533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.254+0000 7fb1f901e640 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4442e0 0x564cec4533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.254+0000 7fb1f901e640 1 -- 172.21.15.62:0/1309475191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cec4549e0 con 0x564cec4439a0 2024-09-15T23:28:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.259+0000 7fb1f901e640 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4439a0 0x564cec452e80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0005ec0 tx=0x7fb1f0009880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9c95c97de99eb69 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:02.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.260+0000 7fb1dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.264+0000 7fb1dcff9640 1 -- 172.21.15.62:0/1309475191 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0035040 con 0x564cec4439a0 2024-09-15T23:28:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.264+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309475191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1ec017420 con 0x564cec4439a0 2024-09-15T23:28:02.265 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.264+0000 7fb1dcff9640 1 -- 172.21.15.62:0/1309475191 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb1f00495d0 con 0x564cec4439a0 2024-09-15T23:28:02.268 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.268+0000 7fb1dcff9640 1 -- 172.21.15.62:0/1309475191 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb1f003c440 con 0x564cec4439a0 2024-09-15T23:28:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.271+0000 7fb1fe020740 10 client.27514 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:02.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.271+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309475191 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564cec4d5b70 con 0x564cec4439a0 2024-09-15T23:28:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.277+0000 7fb1dcff9640 1 -- 172.21.15.62:0/1309475191 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1ec017420 con 0x564cec4439a0 2024-09-15T23:28:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.277+0000 7fb1dcff9640 1 -- 172.21.15.62:0/1309475191 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fb1f003c700 con 0x564cec4439a0 2024-09-15T23:28:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.277+0000 7fb1fe020740 10 client.27514 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.277+0000 7fb1fe020740 10 client.27514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.277+0000 7fb1fe020740 20 client.27514 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.277+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cec4d64d0 0x564cec4d88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.277+0000 7fb1fe020740 4 client.27514 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:28:02.278 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.277+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309475191 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564cec3aed70 con 0x564cec4d64d0 2024-09-15T23:28:02.281 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.280+0000 7fb1f881d640 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cec4d64d0 0x564cec4d88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:02.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.290+0000 7fb1f881d640 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cec4d64d0 0x564cec4d88f0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8cb11c1f4cd49e server_cookie=ccc6a7bf5946de65 in_seq=0 out_seq=0 2024-09-15T23:28:02.296 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.291+0000 7fb1dcff9640 10 client.27514 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1dcff9640 1 -- 172.21.15.62:0/1309475191 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564cec3aed70 con 0x564cec4d64d0 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1dcff9640 10 client.27514 handle_command_reply: tid=0 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 client.27514 shutdown 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309475191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4439a0 msgr2=0x564cec452e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4439a0 0x564cec452e80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0005ec0 tx=0x7fb1f0009880 comp rx=0 tx=0).stop 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309475191 shutdown_connections 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cec4d64d0 0x564cec4d88f0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cec449370 0x564cec4543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cec4442e0 0x564cec4533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 --2- 172.21.15.62:0/1309475191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cec4439a0 0x564cec452e80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1dcff9640 0 client.27514 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309475191 >> 172.21.15.62:0/1309475191 conn(0x564cec43f010 msgr2=0x564cec43e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309475191 shutdown_connections 2024-09-15T23:28:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 1 -- 172.21.15.62:0/1309475191 wait complete. 2024-09-15T23:28:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.405+0000 7fb1fe020740 20 client.27514 trim_cache size 0 max 16384 2024-09-15T23:28:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:28:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:28:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:28:02.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 -- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1e00a3370 msgr2=0x7fb1e00a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1e00a3370 0x7fb1e00a5830 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb1f00096c0 tx=0x7fb1f00315b0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 -- 172.21.15.62:0/762150710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1f4102d00 msgr2=0x7fb1f419eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1f4102d00 0x7fb1f419eab0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009fd0 tx=0x7fb1e400b870 comp rx=0 tx=0).stop 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 -- 172.21.15.62:0/762150710 shutdown_connections 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb1e00a3370 0x7fb1e00a5830 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1f41056b0 0x7fb1f41a5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb1f4102d00 0x7fb1f419eab0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 --2- 172.21.15.62:0/762150710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb1f41003b0 0x7fb1f419e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 -- 172.21.15.62:0/762150710 >> 172.21.15.62:0/762150710 conn(0x7fb1f40fc1e0 msgr2=0x7fb1f4103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.408+0000 7fb1fb282640 1 -- 172.21.15.62:0/762150710 shutdown_connections 2024-09-15T23:28:02.410 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:02.409+0000 7fb1fb282640 1 -- 172.21.15.62:0/762150710 wait complete. 2024-09-15T23:28:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:03 smithi067 ceph-mon[35665]: pgmap v723: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 6.4 MiB/s wr, 586 op/s 2024-09-15T23:28:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:03 smithi148 ceph-mon[35610]: pgmap v723: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 6.4 MiB/s wr, 586 op/s 2024-09-15T23:28:04.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:03 smithi062 ceph-mon[30949]: pgmap v723: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 6.4 MiB/s wr, 586 op/s 2024-09-15T23:28:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 705.0 s, 723.6 tps, lat 138.641 ms stddev 153.498 2024-09-15T23:28:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:05 smithi067 ceph-mon[35665]: pgmap v724: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.0 MiB/s wr, 649 op/s 2024-09-15T23:28:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:05 smithi148 ceph-mon[35610]: pgmap v724: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.0 MiB/s wr, 649 op/s 2024-09-15T23:28:06.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:05 smithi062 ceph-mon[30949]: pgmap v724: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.0 MiB/s wr, 649 op/s 2024-09-15T23:28:08.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:07 smithi067 ceph-mon[35665]: pgmap v725: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 7.0 MiB/s wr, 645 op/s 2024-09-15T23:28:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:07 smithi148 ceph-mon[35610]: pgmap v725: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 7.0 MiB/s wr, 645 op/s 2024-09-15T23:28:08.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:07 smithi062 ceph-mon[30949]: pgmap v725: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 7.0 MiB/s wr, 645 op/s 2024-09-15T23:28:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 710.0 s, 671.4 tps, lat 148.441 ms stddev 164.016 2024-09-15T23:28:10.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:09 smithi067 ceph-mon[35665]: pgmap v726: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 6.4 MiB/s wr, 584 op/s 2024-09-15T23:28:10.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:09 smithi148 ceph-mon[35610]: pgmap v726: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 6.4 MiB/s wr, 584 op/s 2024-09-15T23:28:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:09 smithi062 ceph-mon[30949]: pgmap v726: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 6.4 MiB/s wr, 584 op/s 2024-09-15T23:28:12.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:11 smithi067 ceph-mon[35665]: pgmap v727: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.1 MiB/s wr, 630 op/s 2024-09-15T23:28:12.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:11 smithi148 ceph-mon[35610]: pgmap v727: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.1 MiB/s wr, 630 op/s 2024-09-15T23:28:12.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:11 smithi062 ceph-mon[30949]: pgmap v727: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.1 MiB/s wr, 630 op/s 2024-09-15T23:28:13.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:12 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:28:13.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:12 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:28:13.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:12 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:28:13.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:12 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:28:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:12 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:28:13.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:12 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:28:14.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:13 smithi067 ceph-mon[35665]: pgmap v728: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 6.3 MiB/s wr, 574 op/s 2024-09-15T23:28:14.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:13 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:28:14.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:13 smithi148 ceph-mon[35610]: pgmap v728: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 6.3 MiB/s wr, 574 op/s 2024-09-15T23:28:14.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:13 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:28:14.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:13 smithi062 ceph-mon[30949]: pgmap v728: 129 pgs: 129 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 6.3 MiB/s wr, 574 op/s 2024-09-15T23:28:14.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:13 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:28:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 715.0 s, 647.4 tps, lat 152.531 ms stddev 196.847 2024-09-15T23:28:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.547+0000 7f72bdd9b640 1 -- 172.21.15.62:0/528379515 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b40319b0 con 0x7f72c0101c30 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.548+0000 7f72c50e6640 1 -- 172.21.15.62:0/528379515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0101c30 msgr2=0x7f72c01020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.548+0000 7f72c50e6640 1 --2- 172.21.15.62:0/528379515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0101c30 0x7f72c01020b0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f72b4009900 tx=0x7f72b402a410 comp rx=0 tx=0).stop 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.548+0000 7f72c50e6640 1 -- 172.21.15.62:0/528379515 shutdown_connections 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.548+0000 7f72c50e6640 1 --2- 172.21.15.62:0/528379515 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72c01025f0 0x7f72c0108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.548+0000 7f72c50e6640 1 --2- 172.21.15.62:0/528379515 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0101c30 0x7f72c01020b0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.548+0000 7f72c50e6640 1 --2- 172.21.15.62:0/528379515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c0100a40 0x7f72c0100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.548+0000 7f72c50e6640 1 -- 172.21.15.62:0/528379515 >> 172.21.15.62:0/528379515 conn(0x7f72c00fc1e0 msgr2=0x7f72c00fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.548+0000 7f72c50e6640 1 -- 172.21.15.62:0/528379515 shutdown_connections 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.548+0000 7f72c50e6640 1 -- 172.21.15.62:0/528379515 wait complete. 2024-09-15T23:28:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72c50e6640 1 Processor -- start 2024-09-15T23:28:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72c50e6640 1 -- start start 2024-09-15T23:28:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72c50e6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0100a40 0x7f72c0197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72c50e6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72c0101c30 0x7f72c01982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72c50e6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c01025f0 0x7f72c019f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.555 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72c50e6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f72c010c6c0 con 0x7f72c0100a40 2024-09-15T23:28:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72c50e6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f72c010c540 con 0x7f72c01025f0 2024-09-15T23:28:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72c50e6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f72c010c840 con 0x7f72c0101c30 2024-09-15T23:28:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72bed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0100a40 0x7f72c0197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.549+0000 7f72bed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0100a40 0x7f72c0197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48708/0 (socket says 172.21.15.62:48708) 2024-09-15T23:28:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72bed9d640 1 -- 172.21.15.62:0/4117209489 learned_addr learned my addr 172.21.15.62:0/4117209489 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72be59c640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72c0101c30 0x7f72c01982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72bf59e640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c01025f0 0x7f72c019f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72be59c640 1 -- 172.21.15.62:0/4117209489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c01025f0 msgr2=0x7f72c019f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72be59c640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c01025f0 0x7f72c019f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72be59c640 1 -- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0100a40 msgr2=0x7f72c0197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72be59c640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0100a40 0x7f72c0197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72be59c640 1 -- 172.21.15.62:0/4117209489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c019f9d0 con 0x7f72c0101c30 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72bed9d640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0100a40 0x7f72c0197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72be59c640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72c0101c30 0x7f72c01982f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f72b4031440 tx=0x7f72b4031b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72a7fff640 1 -- 172.21.15.62:0/4117209489 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4035070 con 0x7f72c0101c30 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72c50e6640 1 -- 172.21.15.62:0/4117209489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c0102e50 con 0x7f72c0101c30 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.550+0000 7f72c50e6640 1 -- 172.21.15.62:0/4117209489 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c0103390 con 0x7f72c0101c30 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.551+0000 7f72a7fff640 1 -- 172.21.15.62:0/4117209489 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b403dc40 con 0x7f72c0101c30 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.551+0000 7f72a7fff640 1 -- 172.21.15.62:0/4117209489 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4003cb0 con 0x7f72c0101c30 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.554+0000 7f72c7e76740 1 Processor -- start 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.554+0000 7f72c7e76740 1 -- start start 2024-09-15T23:28:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.554+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e025fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.554+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e02602e0 0x5614e0264da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.554+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 0x5614e0267830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.554+0000 7f72c7e76740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5614e01c32f0 con 0x5614e02602e0 2024-09-15T23:28:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.554+0000 7f72c7e76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5614e016b270 con 0x5614e025f9a0 2024-09-15T23:28:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.554+0000 7f72c7e76740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5614e01bf2b0 con 0x5614e0265370 2024-09-15T23:28:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e02602e0 0x5614e0264da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e02602e0 0x5614e0264da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48718/0 (socket says 172.21.15.62:48718) 2024-09-15T23:28:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bed9d640 1 -- 172.21.15.62:0/476378347 learned_addr learned my addr 172.21.15.62:0/476378347 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72a7fff640 1 -- 172.21.15.62:0/4117209489 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f72b403d760 con 0x7f72c0101c30 2024-09-15T23:28:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bed9d640 1 -- 172.21.15.62:0/476378347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 msgr2=0x5614e0267830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:28:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bed9d640 1 --2- 172.21.15.62:0/476378347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 0x5614e0267830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bed9d640 1 -- 172.21.15.62:0/476378347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 msgr2=0x5614e025fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bf59e640 1 --2- 172.21.15.62:0/476378347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e025fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bed9d640 1 --2- 172.21.15.62:0/476378347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e025fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bed9d640 1 -- 172.21.15.62:0/476378347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614e016a340 con 0x5614e02602e0 2024-09-15T23:28:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.555+0000 7f72bed9d640 1 --2- 172.21.15.62:0/476378347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e02602e0 0x5614e0264da0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f72b0010770 tx=0x7f72b0010c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa320f1ad05c61e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.556+0000 7f72a57fa640 1 -- 172.21.15.62:0/476378347 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b000e930 con 0x5614e02602e0 2024-09-15T23:28:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.556+0000 7f72a57fa640 1 -- 172.21.15.62:0/476378347 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b0011440 con 0x5614e02602e0 2024-09-15T23:28:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.556+0000 7f72a57fa640 1 -- 172.21.15.62:0/476378347 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b0019630 con 0x5614e02602e0 2024-09-15T23:28:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.556+0000 7f72a7fff640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72a80a32f0 0x7f72a80a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.556+0000 7f72a7fff640 1 -- 172.21.15.62:0/4117209489 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f72b40f4680 con 0x7f72c0101c30 2024-09-15T23:28:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.557+0000 7f72c7e76740 1 -- 172.21.15.62:0/476378347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e02602e0 msgr2=0x5614e0264da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.557+0000 7f72c7e76740 1 --2- 172.21.15.62:0/476378347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e02602e0 0x5614e0264da0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f72b0010770 tx=0x7f72b0010c30 comp rx=0 tx=0).stop 2024-09-15T23:28:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.557+0000 7f72bf59e640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72a80a32f0 0x7f72a80a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.557+0000 7f72c7e76740 1 -- 172.21.15.62:0/476378347 shutdown_connections 2024-09-15T23:28:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.557+0000 7f72c7e76740 1 --2- 172.21.15.62:0/476378347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 0x5614e0267830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.557+0000 7f72c7e76740 1 --2- 172.21.15.62:0/476378347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e02602e0 0x5614e0264da0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.557+0000 7f72c7e76740 1 --2- 172.21.15.62:0/476378347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e025fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.557+0000 7f72c7e76740 1 -- 172.21.15.62:0/476378347 >> 172.21.15.62:0/476378347 conn(0x5614e025b010 msgr2=0x5614e025b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.558+0000 7f72bf59e640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72a80a32f0 0x7f72a80a57b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f72b8005fd0 tx=0x7f72b8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.563+0000 7f72c7e76740 1 -- 172.21.15.62:0/476378347 shutdown_connections 2024-09-15T23:28:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.565+0000 7f72c7e76740 1 -- 172.21.15.62:0/476378347 wait complete. 2024-09-15T23:28:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.565+0000 7f72c7e76740 1 -- start start 2024-09-15T23:28:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.565+0000 7f72c7e76740 1 Processor -- start 2024-09-15T23:28:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.569+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e0272670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.569+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 0x5614e0272bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.569+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e0279fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.569+0000 7f72c7e76740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5614e0268340 con 0x5614e0277be0 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.569+0000 7f72c7e76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5614e014c290 con 0x5614e025f9a0 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.569+0000 7f72c7e76740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5614e02684c0 con 0x5614e0265370 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.569+0000 7f72c7e76740 10 client.0 _ll_register_callbacks cb 0x5614e017d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.570+0000 7f72bf59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e0279fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.570+0000 7f72bf59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e0279fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48730/0 (socket says 172.21.15.62:48730) 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.570+0000 7f72bf59e640 1 -- 172.21.15.62:0/1751528725 learned_addr learned my addr 172.21.15.62:0/1751528725 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.571+0000 7f72bf59e640 1 -- 172.21.15.62:0/1751528725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 msgr2=0x5614e0272bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.571+0000 7f72bf59e640 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 0x5614e0272bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.571+0000 7f72bf59e640 1 -- 172.21.15.62:0/1751528725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 msgr2=0x5614e0272670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.571+0000 7f72bf59e640 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e0272670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.571+0000 7f72bf59e640 1 -- 172.21.15.62:0/1751528725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614dfed6210 con 0x5614e0277be0 2024-09-15T23:28:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.571+0000 7f72bf59e640 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e0279fd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f72b80042d0 tx=0x7f72b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f114c1b14f2d671 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.572+0000 7f72a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.572+0000 7f72a2ff5640 1 -- 172.21.15.62:0/1751528725 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b802a060 con 0x5614e0277be0 2024-09-15T23:28:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.572+0000 7f72a2ff5640 1 -- 172.21.15.62:0/1751528725 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f72b8011e00 con 0x5614e0277be0 2024-09-15T23:28:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.574+0000 7f72c7e76740 1 -- 172.21.15.62:0/1751528725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b0011440 con 0x5614e0277be0 2024-09-15T23:28:15.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.579+0000 7f72a2ff5640 1 -- 172.21.15.62:0/1751528725 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f72b8004850 con 0x5614e0277be0 2024-09-15T23:28:15.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.580+0000 7f72c7e76740 10 client.18234 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:15.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.580+0000 7f72c7e76740 1 -- 172.21.15.62:0/1751528725 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5614dffa2620 con 0x5614e0277be0 2024-09-15T23:28:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.585+0000 7f72a2ff5640 1 -- 172.21.15.62:0/1751528725 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b0011440 con 0x5614e0277be0 2024-09-15T23:28:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.585+0000 7f72a2ff5640 1 -- 172.21.15.62:0/1751528725 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f72b8024030 con 0x5614e0277be0 2024-09-15T23:28:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.585+0000 7f72c7e76740 10 client.18234 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.586+0000 7f72c7e76740 10 client.18234 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.586+0000 7f72c7e76740 20 client.18234 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.586+0000 7f72c7e76740 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614e028f750 0x5614e0291c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.586+0000 7f72c7e76740 4 client.18234 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.586+0000 7f72c7e76740 1 -- 172.21.15.62:0/1751528725 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614e01d0a20 con 0x5614e028f750 2024-09-15T23:28:15.589 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.587+0000 7f72bed9d640 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614e028f750 0x5614e0291c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.590+0000 7f72bed9d640 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614e028f750 0x5614e0291c10 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c2a0dce49399b8 server_cookie=c54e91e8bd638195 in_seq=0 out_seq=0 2024-09-15T23:28:15.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.590+0000 7f72a2ff5640 10 client.18234 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.594+0000 7f72a2ff5640 1 -- 172.21.15.62:0/1751528725 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5614e01d0a20 con 0x5614e028f750 2024-09-15T23:28:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.594+0000 7f72a2ff5640 10 client.18234 handle_command_reply: tid=0 2024-09-15T23:28:15.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.594+0000 7f72c7e76740 1 client.18234 shutdown 2024-09-15T23:28:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.598+0000 7f72c7e76740 1 -- 172.21.15.62:0/1751528725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 msgr2=0x5614e0279fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.600 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.598+0000 7f72c7e76740 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e0279fd0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f72b80042d0 tx=0x7f72b8004a40 comp rx=0 tx=0).stop 2024-09-15T23:28:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.603+0000 7f72c7e76740 1 -- 172.21.15.62:0/1751528725 shutdown_connections 2024-09-15T23:28:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.603+0000 7f72c7e76740 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614e028f750 0x5614e0291c10 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.603+0000 7f72c7e76740 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e0279fd0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.603+0000 7f72c7e76740 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 0x5614e0272bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.603+0000 7f72c7e76740 1 --2- 172.21.15.62:0/1751528725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e0272670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.603+0000 7f72c7e76740 1 -- 172.21.15.62:0/1751528725 >> 172.21.15.62:0/1751528725 conn(0x5614e025b010 msgr2=0x5614e025c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:15.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.603+0000 7f72a2ff5640 0 client.18234 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:15.608 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.605+0000 7f72c7e76740 1 -- 172.21.15.62:0/1751528725 shutdown_connections 2024-09-15T23:28:15.611 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.609+0000 7f72c7e76740 1 -- 172.21.15.62:0/1751528725 wait complete. 2024-09-15T23:28:15.612 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.610+0000 7f72c7e76740 20 client.18234 trim_cache size 0 max 16384 2024-09-15T23:28:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.641+0000 7f72c7e76740 1 Processor -- start 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.641+0000 7f72c7e76740 1 -- start start 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.641+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e025b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.641+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 0x5614e025b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.641+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e025be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.641+0000 7f72c7e76740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac012520 con 0x5614e0277be0 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.641+0000 7f72c7e76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac00fdf0 con 0x5614e025f9a0 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.641+0000 7f72c7e76740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac00dc40 con 0x5614e0265370 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.642+0000 7f72bf59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e025be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.642+0000 7f72bf59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e025be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48746/0 (socket says 172.21.15.62:48746) 2024-09-15T23:28:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.642+0000 7f72bf59e640 1 -- 172.21.15.62:0/115639468 learned_addr learned my addr 172.21.15.62:0/115639468 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.642+0000 7f72bf59e640 1 -- 172.21.15.62:0/115639468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 msgr2=0x5614e025b930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:28:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.642+0000 7f72bf59e640 1 --2- 172.21.15.62:0/115639468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 0x5614e025b930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.642+0000 7f72bf59e640 1 -- 172.21.15.62:0/115639468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 msgr2=0x5614e025b3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:28:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.642+0000 7f72bf59e640 1 --2- 172.21.15.62:0/115639468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e025b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.642+0000 7f72bf59e640 1 -- 172.21.15.62:0/115639468 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614e027a340 con 0x5614e0277be0 2024-09-15T23:28:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.643+0000 7f72bf59e640 1 --2- 172.21.15.62:0/115639468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e025be70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f72b8013d60 tx=0x7f72b8009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa193b95b4ccce09 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.643+0000 7f72a4ff9640 1 -- 172.21.15.62:0/115639468 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8034080 con 0x5614e0277be0 2024-09-15T23:28:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.643+0000 7f72a4ff9640 1 -- 172.21.15.62:0/115639468 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b8004850 con 0x5614e0277be0 2024-09-15T23:28:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.644+0000 7f72c7e76740 1 -- 172.21.15.62:0/115639468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 msgr2=0x5614e025be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.644+0000 7f72c7e76740 1 --2- 172.21.15.62:0/115639468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e025be70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f72b8013d60 tx=0x7f72b8009570 comp rx=0 tx=0).stop 2024-09-15T23:28:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.646+0000 7f72c7e76740 1 -- 172.21.15.62:0/115639468 shutdown_connections 2024-09-15T23:28:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.646+0000 7f72c7e76740 1 --2- 172.21.15.62:0/115639468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e025be70 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.646+0000 7f72c7e76740 1 --2- 172.21.15.62:0/115639468 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e0265370 0x5614e025b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.646+0000 7f72c7e76740 1 --2- 172.21.15.62:0/115639468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e025f9a0 0x5614e025b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.646+0000 7f72c7e76740 1 -- 172.21.15.62:0/115639468 >> 172.21.15.62:0/115639468 conn(0x5614e025b010 msgr2=0x5614e0271850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.648+0000 7f72c7e76740 1 -- 172.21.15.62:0/115639468 shutdown_connections 2024-09-15T23:28:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.648+0000 7f72c7e76740 1 -- 172.21.15.62:0/115639468 wait complete. 2024-09-15T23:28:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.649+0000 7f72c7e76740 1 -- start start 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.649+0000 7f72c7e76740 1 Processor -- start 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.649+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e025f9a0 0x5614e02a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.649+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e0265370 0x5614e02a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.649+0000 7f72c7e76740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e02a24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.649+0000 7f72c7e76740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac00e2b0 con 0x5614e0277be0 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.649+0000 7f72c7e76740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac00f720 con 0x5614e0265370 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.649+0000 7f72c7e76740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac00e980 con 0x5614e025f9a0 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.649+0000 7f72c7e76740 10 client.0 _ll_register_callbacks cb 0x7f72b8003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.650+0000 7f72bf59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e02a24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.650+0000 7f72bf59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e02a24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48756/0 (socket says 172.21.15.62:48756) 2024-09-15T23:28:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.650+0000 7f72bf59e640 1 -- 172.21.15.62:0/2453548178 learned_addr learned my addr 172.21.15.62:0/2453548178 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.651+0000 7f72bf59e640 1 -- 172.21.15.62:0/2453548178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e025f9a0 msgr2=0x5614e02a0f80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:28:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.651+0000 7f72bf59e640 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e025f9a0 0x5614e02a0f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.651+0000 7f72bf59e640 1 -- 172.21.15.62:0/2453548178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e0265370 msgr2=0x5614e02a14c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:28:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.651+0000 7f72bf59e640 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e0265370 0x5614e02a14c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.651+0000 7f72bf59e640 1 -- 172.21.15.62:0/2453548178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614e02a2ae0 con 0x5614e0277be0 2024-09-15T23:28:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.652+0000 7f72bf59e640 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e02a24d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f72b8018040 tx=0x7f72b8011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee0e62d42cc9573 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.652+0000 7f72a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.652+0000 7f72a2ff5640 1 -- 172.21.15.62:0/2453548178 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8004c10 con 0x5614e0277be0 2024-09-15T23:28:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.652+0000 7f72c7e76740 1 -- 172.21.15.62:0/2453548178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b8004850 con 0x5614e0277be0 2024-09-15T23:28:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.652+0000 7f72a2ff5640 1 -- 172.21.15.62:0/2453548178 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f72b8004db0 con 0x5614e0277be0 2024-09-15T23:28:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.653+0000 7f72a2ff5640 1 -- 172.21.15.62:0/2453548178 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f72b8005260 con 0x5614e0277be0 2024-09-15T23:28:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.654+0000 7f72c7e76740 10 client.18240 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:15.656 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.654+0000 7f72c7e76740 1 -- 172.21.15.62:0/2453548178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5614e02a3df0 con 0x5614e0277be0 2024-09-15T23:28:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.655+0000 7f72a2ff5640 1 -- 172.21.15.62:0/2453548178 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b8004850 con 0x5614e0277be0 2024-09-15T23:28:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.655+0000 7f72a2ff5640 1 -- 172.21.15.62:0/2453548178 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f72b8024040 con 0x5614e0277be0 2024-09-15T23:28:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.656+0000 7f72c7e76740 10 client.18240 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.656+0000 7f72c7e76740 10 client.18240 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.656+0000 7f72c7e76740 20 client.18240 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.656+0000 7f72c7e76740 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614e0280b80 0x5614e0282fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.656+0000 7f72c7e76740 4 client.18240 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:28:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.656+0000 7f72c7e76740 1 -- 172.21.15.62:0/2453548178 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614e01cad70 con 0x5614e0280b80 2024-09-15T23:28:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.656+0000 7f72bed9d640 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614e0280b80 0x5614e0282fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.657+0000 7f72bed9d640 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614e0280b80 0x5614e0282fa0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=879923218f40a2d1 server_cookie=89cac19015aafc82 in_seq=0 out_seq=0 2024-09-15T23:28:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.657+0000 7f72a2ff5640 10 client.18240 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72a2ff5640 1 -- 172.21.15.62:0/2453548178 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614e01cad70 con 0x5614e0280b80 2024-09-15T23:28:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72a2ff5640 10 client.18240 handle_command_reply: tid=0 2024-09-15T23:28:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 client.18240 shutdown 2024-09-15T23:28:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 -- 172.21.15.62:0/2453548178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 msgr2=0x5614e02a24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e02a24d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f72b8018040 tx=0x7f72b8011e30 comp rx=0 tx=0).stop 2024-09-15T23:28:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 -- 172.21.15.62:0/2453548178 shutdown_connections 2024-09-15T23:28:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614e0280b80 0x5614e0282fa0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614e0277be0 0x5614e02a24d0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.671 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614e0265370 0x5614e02a14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 --2- 172.21.15.62:0/2453548178 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614e025f9a0 0x5614e02a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 -- 172.21.15.62:0/2453548178 >> 172.21.15.62:0/2453548178 conn(0x5614e025b010 msgr2=0x5614e02599a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72a2ff5640 0 client.18240 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 -- 172.21.15.62:0/2453548178 shutdown_connections 2024-09-15T23:28:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.668+0000 7f72c7e76740 1 -- 172.21.15.62:0/2453548178 wait complete. 2024-09-15T23:28:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c7e76740 20 client.18240 trim_cache size 0 max 16384 2024-09-15T23:28:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:28:15.672 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:28:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:28:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:28:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 -- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72a80a32f0 msgr2=0x7f72a80a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72a80a32f0 0x7f72a80a57b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f72b8005fd0 tx=0x7f72b8009500 comp rx=0 tx=0).stop 2024-09-15T23:28:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 -- 172.21.15.62:0/4117209489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72c0101c30 msgr2=0x7f72c01982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72c0101c30 0x7f72c01982f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f72b4031440 tx=0x7f72b4031b60 comp rx=0 tx=0).stop 2024-09-15T23:28:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 -- 172.21.15.62:0/4117209489 shutdown_connections 2024-09-15T23:28:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f72a80a32f0 0x7f72a80a57b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c01025f0 0x7f72c019f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f72c0101c30 0x7f72c01982f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 --2- 172.21.15.62:0/4117209489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f72c0100a40 0x7f72c0197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 -- 172.21.15.62:0/4117209489 >> 172.21.15.62:0/4117209489 conn(0x7f72c00fc1e0 msgr2=0x7f72c00fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 -- 172.21.15.62:0/4117209489 shutdown_connections 2024-09-15T23:28:15.674 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:15.669+0000 7f72c50e6640 1 -- 172.21.15.62:0/4117209489 wait complete. 2024-09-15T23:28:16.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:15 smithi148 ceph-mon[35610]: pgmap v729: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.0 MiB/s wr, 672 op/s 2024-09-15T23:28:16.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:15 smithi062 ceph-mon[30949]: pgmap v729: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.0 MiB/s wr, 672 op/s 2024-09-15T23:28:16.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:15 smithi067 ceph-mon[35665]: pgmap v729: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.0 MiB/s wr, 672 op/s 2024-09-15T23:28:18.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:17 smithi148 ceph-mon[35610]: pgmap v730: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.3 MiB/s wr, 724 op/s 2024-09-15T23:28:18.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:17 smithi062 ceph-mon[30949]: pgmap v730: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.3 MiB/s wr, 724 op/s 2024-09-15T23:28:18.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:17 smithi067 ceph-mon[35665]: pgmap v730: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.3 MiB/s wr, 724 op/s 2024-09-15T23:28:19.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 720.0 s, 562.6 tps, lat 179.728 ms stddev 215.903 2024-09-15T23:28:20.076 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:19 smithi067 ceph-mon[35665]: pgmap v731: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.3 MiB/s wr, 672 op/s 2024-09-15T23:28:20.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:19 smithi062 ceph-mon[30949]: pgmap v731: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.3 MiB/s wr, 672 op/s 2024-09-15T23:28:20.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:19 smithi148 ceph-mon[35610]: pgmap v731: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.3 MiB/s wr, 672 op/s 2024-09-15T23:28:22.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:21 smithi062 ceph-mon[30949]: pgmap v732: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.3 MiB/s wr, 760 op/s 2024-09-15T23:28:22.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:21 smithi067 ceph-mon[35665]: pgmap v732: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.3 MiB/s wr, 760 op/s 2024-09-15T23:28:22.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:21 smithi148 ceph-mon[35610]: pgmap v732: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.3 MiB/s wr, 760 op/s 2024-09-15T23:28:24.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:23 smithi062 ceph-mon[30949]: pgmap v733: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 728 op/s 2024-09-15T23:28:24.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:23 smithi067 ceph-mon[35665]: pgmap v733: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 728 op/s 2024-09-15T23:28:24.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:23 smithi148 ceph-mon[35610]: pgmap v733: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 728 op/s 2024-09-15T23:28:24.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 725.0 s, 871.8 tps, lat 113.561 ms stddev 125.567 2024-09-15T23:28:26.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:25 smithi062 ceph-mon[30949]: pgmap v734: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.5 MiB/s wr, 747 op/s 2024-09-15T23:28:26.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:25 smithi067 ceph-mon[35665]: pgmap v734: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.5 MiB/s wr, 747 op/s 2024-09-15T23:28:26.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:25 smithi148 ceph-mon[35610]: pgmap v734: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.5 MiB/s wr, 747 op/s 2024-09-15T23:28:28.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:27 smithi062 ceph-mon[30949]: pgmap v735: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.2 MiB/s wr, 700 op/s 2024-09-15T23:28:28.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:27 smithi067 ceph-mon[35665]: pgmap v735: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.2 MiB/s wr, 700 op/s 2024-09-15T23:28:28.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:27 smithi148 ceph-mon[35610]: pgmap v735: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.2 MiB/s wr, 700 op/s 2024-09-15T23:28:28.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.826+0000 7effd619f640 1 -- 172.21.15.62:0/1046806914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd0072d20 msgr2=0x7effd0073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:28.828 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.826+0000 7effd619f640 1 --2- 172.21.15.62:0/1046806914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd0072d20 0x7effd0073180 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7effc4009900 tx=0x7effc402a400 comp rx=0 tx=0).stop 2024-09-15T23:28:28.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.828+0000 7effd619f640 1 -- 172.21.15.62:0/1046806914 shutdown_connections 2024-09-15T23:28:28.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.828+0000 7effd619f640 1 --2- 172.21.15.62:0/1046806914 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effd00736c0 0x7effd0119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.828+0000 7effd619f640 1 --2- 172.21.15.62:0/1046806914 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd0072d20 0x7effd0073180 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.828+0000 7effd619f640 1 --2- 172.21.15.62:0/1046806914 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0109700 0x7effd0109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.828+0000 7effd619f640 1 -- 172.21.15.62:0/1046806914 >> 172.21.15.62:0/1046806914 conn(0x7effd006ece0 msgr2=0x7effd0071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.836+0000 7effd619f640 1 -- 172.21.15.62:0/1046806914 shutdown_connections 2024-09-15T23:28:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.839+0000 7effd619f640 1 -- 172.21.15.62:0/1046806914 wait complete. 2024-09-15T23:28:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.840+0000 7effd619f640 1 Processor -- start 2024-09-15T23:28:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.842+0000 7effd619f640 1 -- start start 2024-09-15T23:28:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.842+0000 7effd619f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0072d20 0x7effd0115800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.842+0000 7effd619f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd00736c0 0x7effd0115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.842+0000 7effd619f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effd0109700 0x7effd01b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.842+0000 7effd619f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7effd011c660 con 0x7effd0109700 2024-09-15T23:28:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.842+0000 7effd619f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effd011c4e0 con 0x7effd0072d20 2024-09-15T23:28:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.842+0000 7effd619f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7effd011c7e0 con 0x7effd00736c0 2024-09-15T23:28:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.843+0000 7effceffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd00736c0 0x7effd0115d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.843+0000 7effceffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd00736c0 0x7effd0115d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:44678/0 (socket says 172.21.15.62:44678) 2024-09-15T23:28:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.843+0000 7effceffd640 1 -- 172.21.15.62:0/3663798771 learned_addr learned my addr 172.21.15.62:0/3663798771 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.843+0000 7effcf7fe640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0072d20 0x7effd0115800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.844+0000 7effceffd640 1 -- 172.21.15.62:0/3663798771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0072d20 msgr2=0x7effd0115800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.844+0000 7effceffd640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0072d20 0x7effd0115800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.844+0000 7effceffd640 1 -- 172.21.15.62:0/3663798771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effd0109700 msgr2=0x7effd01b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.844+0000 7effceffd640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effd0109700 0x7effd01b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.844+0000 7effceffd640 1 -- 172.21.15.62:0/3663798771 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd0118ec0 con 0x7effd00736c0 2024-09-15T23:28:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.844+0000 7effceffd640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd00736c0 0x7effd0115d40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7effc4031440 tx=0x7effc4032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.846+0000 7effccff9640 1 -- 172.21.15.62:0/3663798771 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc403f030 con 0x7effd00736c0 2024-09-15T23:28:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.846+0000 7effccff9640 1 -- 172.21.15.62:0/3663798771 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effc4003ce0 con 0x7effd00736c0 2024-09-15T23:28:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.846+0000 7effccff9640 1 -- 172.21.15.62:0/3663798771 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc4035070 con 0x7effd00736c0 2024-09-15T23:28:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.848+0000 7effd619f640 1 -- 172.21.15.62:0/3663798771 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd01b7310 con 0x7effd00736c0 2024-09-15T23:28:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.849+0000 7effd619f640 1 -- 172.21.15.62:0/3663798771 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd01b7800 con 0x7effd00736c0 2024-09-15T23:28:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.857+0000 7effd901a740 1 Processor -- start 2024-09-15T23:28:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.857+0000 7effd901a740 1 -- start start 2024-09-15T23:28:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.858+0000 7effd901a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77eada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.858+0000 7effd901a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77efda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.858+0000 7effd901a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77f2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.858+0000 7effd901a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5609f774e2f0 con 0x5609f77ea9a0 2024-09-15T23:28:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.858+0000 7effd901a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5609f76f6270 con 0x5609f77f0370 2024-09-15T23:28:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.858+0000 7effd901a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5609f774a2b0 con 0x5609f77eb2e0 2024-09-15T23:28:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.859+0000 7effccff9640 1 -- 172.21.15.62:0/3663798771 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7effc4042030 con 0x7effd00736c0 2024-09-15T23:28:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.859+0000 7effcf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.859+0000 7effcf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77efda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:44694/0 (socket says 172.21.15.62:44694) 2024-09-15T23:28:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.859+0000 7effcf7fe640 1 -- 172.21.15.62:0/1627661425 learned_addr learned my addr 172.21.15.62:0/1627661425 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.859+0000 7effceffd640 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.859+0000 7effcffff640 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77eada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.861+0000 7effcffff640 1 -- 172.21.15.62:0/1627661425 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 msgr2=0x5609f77efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.861+0000 7effcffff640 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.861+0000 7effcffff640 1 -- 172.21.15.62:0/1627661425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 msgr2=0x5609f77f2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.862+0000 7effcffff640 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77f2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.862+0000 7effcffff640 1 -- 172.21.15.62:0/1627661425 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609f76f5340 con 0x5609f77ea9a0 2024-09-15T23:28:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.862+0000 7effceffd640 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.862+0000 7effccff9640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7effb80a3530 0x7effb80a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.863+0000 7effccff9640 1 -- 172.21.15.62:0/3663798771 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7effc40f1410 con 0x7effd00736c0 2024-09-15T23:28:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.870+0000 7effcffff640 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77eada0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7effc80096d0 tx=0x7effc8009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bca4a79ac9a7e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.870+0000 7effcf7fe640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7effb80a3530 0x7effb80a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.871+0000 7effb5ffb640 1 -- 172.21.15.62:0/1627661425 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8015020 con 0x5609f77ea9a0 2024-09-15T23:28:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.871+0000 7effb5ffb640 1 -- 172.21.15.62:0/1627661425 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effc800bda0 con 0x5609f77ea9a0 2024-09-15T23:28:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.871+0000 7effb5ffb640 1 -- 172.21.15.62:0/1627661425 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8010380 con 0x5609f77ea9a0 2024-09-15T23:28:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.873+0000 7effd901a740 1 -- 172.21.15.62:0/1627661425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 msgr2=0x5609f77eada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.873+0000 7effd901a740 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77eada0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7effc80096d0 tx=0x7effc8009b90 comp rx=0 tx=0).stop 2024-09-15T23:28:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.874+0000 7effcf7fe640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7effb80a3530 0x7effb80a59f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x5609f77ebd10 tx=0x7effc000a590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.874+0000 7effd901a740 1 -- 172.21.15.62:0/1627661425 shutdown_connections 2024-09-15T23:28:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.874+0000 7effd901a740 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77f2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.874+0000 7effd901a740 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.874+0000 7effd901a740 1 --2- 172.21.15.62:0/1627661425 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77eada0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.874+0000 7effd901a740 1 -- 172.21.15.62:0/1627661425 >> 172.21.15.62:0/1627661425 conn(0x5609f77e6010 msgr2=0x5609f77e6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:28.877 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.875+0000 7effd901a740 1 -- 172.21.15.62:0/1627661425 shutdown_connections 2024-09-15T23:28:28.880 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.879+0000 7effd901a740 1 -- 172.21.15.62:0/1627661425 wait complete. 2024-09-15T23:28:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.879+0000 7effd901a740 1 -- start start 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.879+0000 7effd901a740 1 Processor -- start 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.880+0000 7effd901a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77f4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.880+0000 7effd901a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77f5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.880+0000 7effd901a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.880+0000 7effd901a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5609f77f3340 con 0x5609f77f0370 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.880+0000 7effd901a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5609f76d7290 con 0x5609f77ea9a0 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.880+0000 7effd901a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5609f77f34c0 con 0x5609f77eb2e0 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.880+0000 7effd901a740 10 client.0 _ll_register_callbacks cb 0x5609f7708140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.881+0000 7effcf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77f4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.881+0000 7effcf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77f4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36496/0 (socket says 172.21.15.62:36496) 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.881+0000 7effcf7fe640 1 -- 172.21.15.62:0/4188325986 learned_addr learned my addr 172.21.15.62:0/4188325986 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.882+0000 7effcffff640 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.883+0000 7effceffd640 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.884+0000 7effcffff640 1 -- 172.21.15.62:0/4188325986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 msgr2=0x5609f77f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.884+0000 7effcffff640 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77f5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.884+0000 7effcffff640 1 -- 172.21.15.62:0/4188325986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 msgr2=0x5609f77f4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.884+0000 7effcffff640 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77f4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.884+0000 7effcffff640 1 -- 172.21.15.62:0/4188325986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609f7461210 con 0x5609f77f0370 2024-09-15T23:28:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.884+0000 7effcf7fe640 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.884+0000 7effcffff640 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77fc1b0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7effc8000c00 tx=0x7effc800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68822ef653ab288d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.885+0000 7effaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.885+0000 7effaeffd640 1 -- 172.21.15.62:0/4188325986 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8024670 con 0x5609f77f0370 2024-09-15T23:28:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.888+0000 7effd901a740 1 -- 172.21.15.62:0/4188325986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effc800bda0 con 0x5609f77f0370 2024-09-15T23:28:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.893+0000 7effaeffd640 1 -- 172.21.15.62:0/4188325986 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7effc8024810 con 0x5609f77f0370 2024-09-15T23:28:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.893+0000 7effaeffd640 1 -- 172.21.15.62:0/4188325986 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7effc802d050 con 0x5609f77f0370 2024-09-15T23:28:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.893+0000 7effd901a740 10 client.18246 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.893+0000 7effd901a740 1 -- 172.21.15.62:0/4188325986 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5609f752d620 con 0x5609f77f0370 2024-09-15T23:28:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.897+0000 7effaeffd640 1 -- 172.21.15.62:0/4188325986 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effc800bda0 con 0x5609f77f0370 2024-09-15T23:28:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.897+0000 7effaeffd640 1 -- 172.21.15.62:0/4188325986 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7effc801b9b0 con 0x5609f77f0370 2024-09-15T23:28:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.897+0000 7effd901a740 10 client.18246 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.897+0000 7effd901a740 10 client.18246 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.897+0000 7effd901a740 20 client.18246 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.897+0000 7effd901a740 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5609f7800b30 0x5609f7802f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.897+0000 7effd901a740 4 client.18246 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.897+0000 7effd901a740 1 -- 172.21.15.62:0/4188325986 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609f775ba20 con 0x5609f7800b30 2024-09-15T23:28:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.901+0000 7effcf7fe640 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5609f7800b30 0x5609f7802f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.902+0000 7effcf7fe640 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5609f7800b30 0x5609f7802f70 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aca94c110adf6e36 server_cookie=1f7b7b28c9b7f6cc in_seq=0 out_seq=0 2024-09-15T23:28:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.902+0000 7effaeffd640 10 client.18246 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.903+0000 7effaeffd640 1 -- 172.21.15.62:0/4188325986 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5609f775ba20 con 0x5609f7800b30 2024-09-15T23:28:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.903+0000 7effaeffd640 10 client.18246 handle_command_reply: tid=0 2024-09-15T23:28:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.903+0000 7effd901a740 1 client.18246 shutdown 2024-09-15T23:28:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.904+0000 7effd901a740 1 -- 172.21.15.62:0/4188325986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 msgr2=0x5609f77fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.904+0000 7effd901a740 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77fc1b0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7effc8000c00 tx=0x7effc800b8f0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.906+0000 7effd901a740 1 -- 172.21.15.62:0/4188325986 shutdown_connections 2024-09-15T23:28:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.906+0000 7effd901a740 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5609f7800b30 0x5609f7802f70 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.906+0000 7effd901a740 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77fc1b0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.906+0000 7effd901a740 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.906+0000 7effd901a740 1 --2- 172.21.15.62:0/4188325986 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77f4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.906+0000 7effd901a740 1 -- 172.21.15.62:0/4188325986 >> 172.21.15.62:0/4188325986 conn(0x5609f77e6010 msgr2=0x5609f77e7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.906+0000 7effaeffd640 0 client.18246 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.906+0000 7effd901a740 1 -- 172.21.15.62:0/4188325986 shutdown_connections 2024-09-15T23:28:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.906+0000 7effd901a740 1 -- 172.21.15.62:0/4188325986 wait complete. 2024-09-15T23:28:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.907+0000 7effd901a740 20 client.18246 trim_cache size 0 max 16384 2024-09-15T23:28:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.928+0000 7effd901a740 1 Processor -- start 2024-09-15T23:28:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.928+0000 7effd901a740 1 -- start start 2024-09-15T23:28:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effd901a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77ead80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effd901a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77fc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effd901a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77e50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effd901a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7effbc0124d0 con 0x5609f77f0370 2024-09-15T23:28:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effd901a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effbc00fda0 con 0x5609f77ea9a0 2024-09-15T23:28:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effd901a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7effbc00dbf0 con 0x5609f77eb2e0 2024-09-15T23:28:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77ead80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcf7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77ead80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:36498/0 (socket says 172.21.15.62:36498) 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcf7fe640 1 -- 172.21.15.62:0/1713051322 learned_addr learned my addr 172.21.15.62:0/1713051322 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effceffd640 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77fc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcffff640 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77e50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcf7fe640 1 -- 172.21.15.62:0/1713051322 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 msgr2=0x5609f77fc170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcf7fe640 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77fc170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcf7fe640 1 -- 172.21.15.62:0/1713051322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 msgr2=0x5609f77e50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcf7fe640 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77e50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcf7fe640 1 -- 172.21.15.62:0/1713051322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609f77f4770 con 0x5609f77ea9a0 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effceffd640 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcffff640 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77e50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.929+0000 7effcf7fe640 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77ead80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7effc0005fa0 tx=0x7effc0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55a5ed4eaa67644a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.930+0000 7effb57fa640 1 -- 172.21.15.62:0/1713051322 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc001f0d0 con 0x5609f77ea9a0 2024-09-15T23:28:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.930+0000 7effb57fa640 1 -- 172.21.15.62:0/1713051322 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effc0003e50 con 0x5609f77ea9a0 2024-09-15T23:28:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.930+0000 7effb57fa640 1 -- 172.21.15.62:0/1713051322 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0004260 con 0x5609f77ea9a0 2024-09-15T23:28:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.930+0000 7effd901a740 1 -- 172.21.15.62:0/1713051322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 msgr2=0x5609f77ead80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.930+0000 7effd901a740 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77ead80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7effc0005fa0 tx=0x7effc0015000 comp rx=0 tx=0).stop 2024-09-15T23:28:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.931+0000 7effd901a740 1 -- 172.21.15.62:0/1713051322 shutdown_connections 2024-09-15T23:28:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.931+0000 7effd901a740 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77f0370 0x5609f77e50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.931+0000 7effd901a740 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77fc170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.931+0000 7effd901a740 1 --2- 172.21.15.62:0/1713051322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77ea9a0 0x5609f77ead80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.931+0000 7effd901a740 1 -- 172.21.15.62:0/1713051322 >> 172.21.15.62:0/1713051322 conn(0x5609f77e6010 msgr2=0x5609f77e49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.931+0000 7effd901a740 1 -- 172.21.15.62:0/1713051322 shutdown_connections 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.931+0000 7effd901a740 1 -- 172.21.15.62:0/1713051322 wait complete. 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.931+0000 7effd901a740 1 -- start start 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.931+0000 7effd901a740 1 Processor -- start 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effd901a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77f9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effd901a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77f9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effd901a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77fa780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effd901a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7effbc00e260 con 0x5609f77ea9a0 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effcf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effcf7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77f9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:43766/0 (socket says 172.21.15.62:43766) 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effcf7fe640 1 -- 172.21.15.62:0/3356099641 learned_addr learned my addr 172.21.15.62:0/3356099641 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effceffd640 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77f9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7effbc00f6d0 con 0x5609f77f0370 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7effbc00e930 con 0x5609f77eb2e0 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effcffff640 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77fa780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.932+0000 7effd901a740 10 client.0 _ll_register_callbacks cb 0x7effc000a990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effceffd640 1 -- 172.21.15.62:0/3356099641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 msgr2=0x5609f77fa780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effceffd640 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77fa780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effceffd640 1 -- 172.21.15.62:0/3356099641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 msgr2=0x5609f77f9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effceffd640 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77f9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effceffd640 1 -- 172.21.15.62:0/3356099641 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609f78035f0 con 0x5609f77eb2e0 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effcffff640 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77fa780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effceffd640 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77f9770 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7effc403f000 tx=0x7effc4035000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=461b9002deaa36bb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effaeffd640 1 -- 172.21.15.62:0/3356099641 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc40bb790 con 0x5609f77eb2e0 2024-09-15T23:28:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effaeffd640 1 -- 172.21.15.62:0/3356099641 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7effc40b9c30 con 0x5609f77eb2e0 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.933+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effc0003e50 con 0x5609f77eb2e0 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effaeffd640 1 -- 172.21.15.62:0/3356099641 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7effc40bdd50 con 0x5609f77eb2e0 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effd901a740 10 client.27538 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5609f77fbee0 con 0x5609f77eb2e0 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effaeffd640 1 -- 172.21.15.62:0/3356099641 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effc0003e50 con 0x5609f77eb2e0 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effaeffd640 1 -- 172.21.15.62:0/3356099641 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7effc4003920 con 0x5609f77eb2e0 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effd901a740 10 client.27538 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effd901a740 10 client.27538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effd901a740 20 client.27538 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effd901a740 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5609f786b9f0 0x5609f786de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effd901a740 4 client.27538 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5609f7755d70 con 0x5609f786b9f0 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.934+0000 7effcf7fe640 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5609f786b9f0 0x5609f786de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.935+0000 7effaeffd640 10 client.27538 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.935+0000 7effcf7fe640 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5609f786b9f0 0x5609f786de10 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89930c5683d42d2a server_cookie=d8df91939821fe15 in_seq=0 out_seq=0 2024-09-15T23:28:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.944+0000 7effaeffd640 1 -- 172.21.15.62:0/3356099641 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5609f7755d70 con 0x5609f786b9f0 2024-09-15T23:28:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.944+0000 7effaeffd640 10 client.27538 handle_command_reply: tid=0 2024-09-15T23:28:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.944+0000 7effd901a740 1 client.27538 shutdown 2024-09-15T23:28:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.944+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 msgr2=0x5609f77f9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.944+0000 7effd901a740 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77f9770 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7effc403f000 tx=0x7effc4035000 comp rx=0 tx=0).stop 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.944+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 shutdown_connections 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.944+0000 7effd901a740 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5609f786b9f0 0x5609f786de10 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.944+0000 7effd901a740 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609f77f0370 0x5609f77fa780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.945+0000 7effaeffd640 0 client.27538 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.945+0000 7effd901a740 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5609f77eb2e0 0x5609f77f9770 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.945+0000 7effd901a740 1 --2- 172.21.15.62:0/3356099641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609f77ea9a0 0x5609f77f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.945+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 >> 172.21.15.62:0/3356099641 conn(0x5609f77e6010 msgr2=0x5609f77e51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.945+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 shutdown_connections 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.945+0000 7effd901a740 1 -- 172.21.15.62:0/3356099641 wait complete. 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.945+0000 7effd901a740 20 client.27538 trim_cache size 0 max 16384 2024-09-15T23:28:28.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:28:28.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:28:28.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:28:28.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:28:28.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.946+0000 7effd619f640 1 -- 172.21.15.62:0/3663798771 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7effb80a3530 msgr2=0x7effb80a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:28.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.946+0000 7effd619f640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7effb80a3530 0x7effb80a59f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x5609f77ebd10 tx=0x7effc000a590 comp rx=0 tx=0).stop 2024-09-15T23:28:28.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.946+0000 7effd619f640 1 -- 172.21.15.62:0/3663798771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd00736c0 msgr2=0x7effd0115d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:28.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.946+0000 7effd619f640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd00736c0 0x7effd0115d40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7effc4031440 tx=0x7effc4032dc0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.951+0000 7effd619f640 1 -- 172.21.15.62:0/3663798771 shutdown_connections 2024-09-15T23:28:28.954 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.951+0000 7effd619f640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7effb80a3530 0x7effb80a59f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.954 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.951+0000 7effd619f640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7effd0109700 0x7effd01b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.954 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.951+0000 7effd619f640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7effd00736c0 0x7effd0115d40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.954 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.951+0000 7effd619f640 1 --2- 172.21.15.62:0/3663798771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7effd0072d20 0x7effd0115800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:28.954 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.951+0000 7effd619f640 1 -- 172.21.15.62:0/3663798771 >> 172.21.15.62:0/3663798771 conn(0x7effd006ece0 msgr2=0x7effd00716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:28.959 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.957+0000 7effd619f640 1 -- 172.21.15.62:0/3663798771 shutdown_connections 2024-09-15T23:28:28.962 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:28.961+0000 7effd619f640 1 -- 172.21.15.62:0/3663798771 wait complete. 2024-09-15T23:28:29.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 730.0 s, 610.4 tps, lat 160.582 ms stddev 168.576 2024-09-15T23:28:30.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:29 smithi062 ceph-mon[30949]: pgmap v736: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 593 op/s 2024-09-15T23:28:30.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:29 smithi067 ceph-mon[35665]: pgmap v736: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 593 op/s 2024-09-15T23:28:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:29 smithi148 ceph-mon[35610]: pgmap v736: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.1 MiB/s wr, 593 op/s 2024-09-15T23:28:31.457 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:28:32.029 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:32.158 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:31 smithi062 ceph-mon[30949]: pgmap v737: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 630 op/s 2024-09-15T23:28:32.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:31 smithi067 ceph-mon[35665]: pgmap v737: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 630 op/s 2024-09-15T23:28:32.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:31 smithi148 ceph-mon[35610]: pgmap v737: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 630 op/s 2024-09-15T23:28:32.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.626+0000 7f0da9ffb640 1 -- 172.21.15.62:0/1116009995 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c040880 con 0x7f0dac105ec0 2024-09-15T23:28:32.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.627+0000 7f0db17ff640 1 -- 172.21.15.62:0/1116009995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dac105ec0 msgr2=0x7f0dac106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:32.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.627+0000 7f0db17ff640 1 --2- 172.21.15.62:0/1116009995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dac105ec0 0x7f0dac106340 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c01a910 tx=0x7f0d9c03b470 comp rx=0 tx=0).stop 2024-09-15T23:28:32.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.630+0000 7f0db17ff640 1 -- 172.21.15.62:0/1116009995 shutdown_connections 2024-09-15T23:28:32.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.630+0000 7f0db17ff640 1 --2- 172.21.15.62:0/1116009995 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dac105ec0 0x7f0dac106340 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.630+0000 7f0db17ff640 1 --2- 172.21.15.62:0/1116009995 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dac1011e0 0x7f0dac105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.630+0000 7f0db17ff640 1 --2- 172.21.15.62:0/1116009995 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac1008c0 0x7f0dac100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.630+0000 7f0db17ff640 1 -- 172.21.15.62:0/1116009995 >> 172.21.15.62:0/1116009995 conn(0x7f0dac0fc7d0 msgr2=0x7f0dac0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:32.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.630+0000 7f0db17ff640 1 -- 172.21.15.62:0/1116009995 shutdown_connections 2024-09-15T23:28:32.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.631+0000 7f0db17ff640 1 -- 172.21.15.62:0/1116009995 wait complete. 2024-09-15T23:28:32.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.632+0000 7f0db17ff640 1 Processor -- start 2024-09-15T23:28:32.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.635+0000 7f0db17ff640 1 -- start start 2024-09-15T23:28:32.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.635+0000 7f0db17ff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dac1008c0 0x7f0dac114dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.635+0000 7f0db17ff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac1011e0 0x7f0dac110e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.635+0000 7f0db17ff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dac105ec0 0x7f0dac1113b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.635+0000 7f0db17ff640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac1098e0 con 0x7f0dac1008c0 2024-09-15T23:28:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.635+0000 7f0db17ff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac109760 con 0x7f0dac1011e0 2024-09-15T23:28:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.635+0000 7f0db17ff640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac109a60 con 0x7f0dac105ec0 2024-09-15T23:28:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.636+0000 7f0daa7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac1011e0 0x7f0dac110e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.636+0000 7f0daa7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac1011e0 0x7f0dac110e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50956/0 (socket says 172.21.15.62:50956) 2024-09-15T23:28:32.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.636+0000 7f0daa7fc640 1 -- 172.21.15.62:0/3891704987 learned_addr learned my addr 172.21.15.62:0/3891704987 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:32.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.636+0000 7f0dab7fe640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dac105ec0 0x7f0dac1113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.637+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dac1008c0 0x7f0dac114dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.638+0000 7f0daa7fc640 1 -- 172.21.15.62:0/3891704987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dac105ec0 msgr2=0x7f0dac1113b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:32.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.638+0000 7f0daa7fc640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dac105ec0 0x7f0dac1113b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.638+0000 7f0daa7fc640 1 -- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dac1008c0 msgr2=0x7f0dac114dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:32.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.638+0000 7f0daa7fc640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dac1008c0 0x7f0dac114dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.638+0000 7f0daa7fc640 1 -- 172.21.15.62:0/3891704987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dac111a90 con 0x7f0dac1011e0 2024-09-15T23:28:32.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.639+0000 7f0daa7fc640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac1011e0 0x7f0dac110e70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0da400ba50 tx=0x7f0da400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:32.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.639+0000 7f0d9bfff640 1 -- 172.21.15.62:0/3891704987 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da4019070 con 0x7f0dac1011e0 2024-09-15T23:28:32.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.639+0000 7f0d9bfff640 1 -- 172.21.15.62:0/3891704987 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da400cc10 con 0x7f0dac1011e0 2024-09-15T23:28:32.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.639+0000 7f0db17ff640 1 -- 172.21.15.62:0/3891704987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dac1c4a70 con 0x7f0dac1011e0 2024-09-15T23:28:32.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.639+0000 7f0db17ff640 1 -- 172.21.15.62:0/3891704987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dac1c4e60 con 0x7f0dac1011e0 2024-09-15T23:28:32.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.639+0000 7f0d9bfff640 1 -- 172.21.15.62:0/3891704987 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da4007730 con 0x7f0dac1011e0 2024-09-15T23:28:32.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.644+0000 7f0db3bd8740 1 Processor -- start 2024-09-15T23:28:32.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.647+0000 7f0d9bfff640 1 -- 172.21.15.62:0/3891704987 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0da401c070 con 0x7f0dac1011e0 2024-09-15T23:28:32.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.648+0000 7f0d9bfff640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d940a3450 0x7f0d940a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.648+0000 7f0d9bfff640 1 -- 172.21.15.62:0/3891704987 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0da40cf090 con 0x7f0dac1011e0 2024-09-15T23:28:32.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.649+0000 7f0db3bd8740 1 -- start start 2024-09-15T23:28:32.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.650+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce34a0 0x561aa0ce3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.650+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0cee140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.650+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0cee680 0x561aa0cf0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.650+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561aa0c492e0 con 0x561aa0ce34a0 2024-09-15T23:28:32.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.650+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561aa0bf0ce0 con 0x561aa0cee680 2024-09-15T23:28:32.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.650+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561aa0c452a0 con 0x561aa0ce3dc0 2024-09-15T23:28:32.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.651+0000 7f0dab7fe640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d940a3450 0x7f0d940a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.651+0000 7f0daa7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0cee140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.651+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce34a0 0x561aa0ce3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.651+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce34a0 0x561aa0ce3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50602/0 (socket says 172.21.15.62:50602) 2024-09-15T23:28:32.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.651+0000 7f0daaffd640 1 -- 172.21.15.62:0/3023695601 learned_addr learned my addr 172.21.15.62:0/3023695601 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:32.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.652+0000 7f0daaffd640 1 -- 172.21.15.62:0/3023695601 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 msgr2=0x561aa0cee140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:32.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.652+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3023695601 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0cee140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.652+0000 7f0daaffd640 1 -- 172.21.15.62:0/3023695601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0cee680 msgr2=0x561aa0cf0a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:28:32.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.652+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3023695601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0cee680 0x561aa0cf0a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.652+0000 7f0daaffd640 1 -- 172.21.15.62:0/3023695601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aa0bd5440 con 0x561aa0ce34a0 2024-09-15T23:28:32.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.653+0000 7f0daa7fc640 1 --2- 172.21.15.62:0/3023695601 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0cee140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:32.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.653+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3023695601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce34a0 0x561aa0ce3880 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0da000cbf0 tx=0x7f0da000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf1ac2f9efd418 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:32.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.654+0000 7f0d997fa640 1 -- 172.21.15.62:0/3023695601 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0010630 con 0x561aa0ce34a0 2024-09-15T23:28:32.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.654+0000 7f0d997fa640 1 -- 172.21.15.62:0/3023695601 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da0010cb0 con 0x561aa0ce34a0 2024-09-15T23:28:32.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.655+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3023695601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce34a0 msgr2=0x561aa0ce3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:32.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.655+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3023695601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce34a0 0x561aa0ce3880 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f0da000cbf0 tx=0x7f0da000e720 comp rx=0 tx=0).stop 2024-09-15T23:28:32.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.656+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3023695601 shutdown_connections 2024-09-15T23:28:32.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.656+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3023695601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0cee680 0x561aa0cf0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.656+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3023695601 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0cee140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.656+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3023695601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce34a0 0x561aa0ce3880 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.656+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3023695601 >> 172.21.15.62:0/3023695601 conn(0x561aa0cdfa30 msgr2=0x561aa0cdfe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:32.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.656+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3023695601 shutdown_connections 2024-09-15T23:28:32.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.661+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3023695601 wait complete. 2024-09-15T23:28:32.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.661+0000 7f0db3bd8740 1 -- start start 2024-09-15T23:28:32.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.661+0000 7f0db3bd8740 1 Processor -- start 2024-09-15T23:28:32.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.662+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce34a0 0x561aa0ceded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.662+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce3dc0 0x561aa0ceb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.662+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0cee680 0x561aa0ceb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.662+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561aa0d022b0 con 0x561aa0ce3dc0 2024-09-15T23:28:32.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.662+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561aa0c04050 con 0x561aa0cee680 2024-09-15T23:28:32.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.662+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561aa0d02430 con 0x561aa0ce34a0 2024-09-15T23:28:32.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.662+0000 7f0db3bd8740 10 client.0 _ll_register_callbacks cb 0x561aa0c44160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:32.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.664+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce34a0 0x561aa0ceded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.664+0000 7f0daa7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce3dc0 0x561aa0ceb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.664+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce34a0 0x561aa0ceded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53296/0 (socket says 172.21.15.62:53296) 2024-09-15T23:28:32.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.664+0000 7f0daaffd640 1 -- 172.21.15.62:0/3224771172 learned_addr learned my addr 172.21.15.62:0/3224771172 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:32.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.665+0000 7f0daaffd640 1 -- 172.21.15.62:0/3224771172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0cee680 msgr2=0x561aa0ceb6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:28:32.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.665+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0cee680 0x561aa0ceb6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.665+0000 7f0daaffd640 1 -- 172.21.15.62:0/3224771172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce3dc0 msgr2=0x561aa0ceb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:32.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.665+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce3dc0 0x561aa0ceb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.665+0000 7f0daaffd640 1 -- 172.21.15.62:0/3224771172 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aa09ba3c0 con 0x561aa0ce34a0 2024-09-15T23:28:32.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.666+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce34a0 0x561aa0ceded0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0da0005d20 tx=0x7f0da0005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f776e04ddd289085 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:32.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.666+0000 7f0d927fc640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:32.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.666+0000 7f0d927fc640 1 -- 172.21.15.62:0/3224771172 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0010e60 con 0x561aa0ce34a0 2024-09-15T23:28:32.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.666+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3224771172 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0da0010cb0 con 0x561aa0ce34a0 2024-09-15T23:28:32.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.666+0000 7f0d927fc640 1 -- 172.21.15.62:0/3224771172 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0da000c730 con 0x561aa0ce34a0 2024-09-15T23:28:32.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.666+0000 7f0d927fc640 1 -- 172.21.15.62:0/3224771172 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0da0010430 con 0x561aa0ce34a0 2024-09-15T23:28:32.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.667+0000 7f0db3bd8740 10 client.27544 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:32.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.667+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3224771172 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561aa0d0dbd0 con 0x561aa0ce34a0 2024-09-15T23:28:32.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.667+0000 7f0d927fc640 1 -- 172.21.15.62:0/3224771172 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da0010cb0 con 0x561aa0ce34a0 2024-09-15T23:28:32.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.667+0000 7f0d927fc640 1 -- 172.21.15.62:0/3224771172 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f0da0004c10 con 0x561aa0ce34a0 2024-09-15T23:28:32.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.668+0000 7f0db3bd8740 10 client.27544 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:32.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.668+0000 7f0db3bd8740 10 client.27544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:32.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.668+0000 7f0db3bd8740 20 client.27544 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:32.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.668+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561aa0d0f440 0x561aa0d11900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.668+0000 7f0db3bd8740 4 client.27544 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:32.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.668+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3224771172 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561aa0c56850 con 0x561aa0d0f440 2024-09-15T23:28:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.665+0000 7f0dab7fe640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d940a3450 0x7f0d940a5910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x561aa0ce0db0 tx=0x7f0d9c01a600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:32.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.670+0000 7f0daa7fc640 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561aa0d0f440 0x561aa0d11900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.671+0000 7f0daa7fc640 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561aa0d0f440 0x561aa0d11900 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d5a21f01b70199 server_cookie=f33741ef14ba27b3 in_seq=0 out_seq=0 2024-09-15T23:28:32.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.671+0000 7f0d927fc640 10 client.27544 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.672+0000 7f0d927fc640 1 -- 172.21.15.62:0/3224771172 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561aa0c56850 con 0x561aa0d0f440 2024-09-15T23:28:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.672+0000 7f0d927fc640 10 client.27544 handle_command_reply: tid=0 2024-09-15T23:28:32.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.672+0000 7f0db3bd8740 1 client.27544 shutdown 2024-09-15T23:28:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.672+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3224771172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce34a0 msgr2=0x561aa0ceded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.672+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce34a0 0x561aa0ceded0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0da0005d20 tx=0x7f0da0005d50 comp rx=0 tx=0).stop 2024-09-15T23:28:32.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.673+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3224771172 shutdown_connections 2024-09-15T23:28:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.673+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561aa0d0f440 0x561aa0d11900 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.673+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0cee680 0x561aa0ceb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.673+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0ce3dc0 0x561aa0ceb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.673+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3224771172 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce34a0 0x561aa0ceded0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.673+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3224771172 >> 172.21.15.62:0/3224771172 conn(0x561aa0cdfa30 msgr2=0x561aa0ce5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.674+0000 7f0d927fc640 0 client.27544 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:32.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.675+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3224771172 shutdown_connections 2024-09-15T23:28:32.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.678+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3224771172 wait complete. 2024-09-15T23:28:32.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.678+0000 7f0db3bd8740 20 client.27544 trim_cache size 0 max 16384 2024-09-15T23:28:32.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.714+0000 7f0db3bd8740 1 Processor -- start 2024-09-15T23:28:32.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.715+0000 7f0db3bd8740 1 -- start start 2024-09-15T23:28:32.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.715+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d0b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.715+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0ce6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.716+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 0x561aa0cdea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.716+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c012440 con 0x561aa0cee680 2024-09-15T23:28:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.716+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c00fd10 con 0x561aa0ce34a0 2024-09-15T23:28:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.716+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c00db60 con 0x561aa0ce3dc0 2024-09-15T23:28:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.716+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d0b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.716+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d0b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50984/0 (socket says 172.21.15.62:50984) 2024-09-15T23:28:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.716+0000 7f0daaffd640 1 -- 172.21.15.62:0/193778360 learned_addr learned my addr 172.21.15.62:0/193778360 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.716+0000 7f0dab7fe640 1 --2- 172.21.15.62:0/193778360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 0x561aa0cdea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.718+0000 7f0daa7fc640 1 --2- 172.21.15.62:0/193778360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0ce6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.719+0000 7f0dab7fe640 1 -- 172.21.15.62:0/193778360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 msgr2=0x561aa0ce6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:32.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.719+0000 7f0dab7fe640 1 --2- 172.21.15.62:0/193778360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0ce6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.719+0000 7f0dab7fe640 1 -- 172.21.15.62:0/193778360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 msgr2=0x561aa0d0b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:32.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.719+0000 7f0dab7fe640 1 --2- 172.21.15.62:0/193778360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d0b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.719+0000 7f0dab7fe640 1 -- 172.21.15.62:0/193778360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aa0cdf860 con 0x561aa0cee680 2024-09-15T23:28:32.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.720+0000 7f0dab7fe640 1 --2- 172.21.15.62:0/193778360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 0x561aa0cdea80 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c006010 tx=0x7f0d9c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3387affc5447cca5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:32.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.723+0000 7f0d98ff9640 1 -- 172.21.15.62:0/193778360 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c015070 con 0x561aa0cee680 2024-09-15T23:28:32.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.723+0000 7f0d98ff9640 1 -- 172.21.15.62:0/193778360 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d9c04f070 con 0x561aa0cee680 2024-09-15T23:28:32.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.724+0000 7f0db3bd8740 1 -- 172.21.15.62:0/193778360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 msgr2=0x561aa0cdea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:32.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.724+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/193778360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 0x561aa0cdea80 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c006010 tx=0x7f0d9c00f000 comp rx=0 tx=0).stop 2024-09-15T23:28:32.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.729+0000 7f0db3bd8740 1 -- 172.21.15.62:0/193778360 shutdown_connections 2024-09-15T23:28:32.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.729+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/193778360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 0x561aa0cdea80 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.729+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/193778360 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0ce6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.729+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/193778360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d0b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.729+0000 7f0db3bd8740 1 -- 172.21.15.62:0/193778360 >> 172.21.15.62:0/193778360 conn(0x561aa0cdfa30 msgr2=0x561aa0cde370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:32.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.731+0000 7f0db3bd8740 1 -- 172.21.15.62:0/193778360 shutdown_connections 2024-09-15T23:28:32.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.732+0000 7f0db3bd8740 1 -- 172.21.15.62:0/193778360 wait complete. 2024-09-15T23:28:32.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.733+0000 7f0db3bd8740 1 -- start start 2024-09-15T23:28:32.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.733+0000 7f0db3bd8740 1 Processor -- start 2024-09-15T23:28:32.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.736+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d72380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.736+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0d728c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.736+0000 7f0db3bd8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 0x561aa0d073f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.736+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c00e1d0 con 0x561aa0cee680 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.736+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c00f640 con 0x561aa0ce34a0 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.736+0000 7f0db3bd8740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8c00e8a0 con 0x561aa0ce3dc0 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.736+0000 7f0db3bd8740 10 client.0 _ll_register_callbacks cb 0x7f0da0014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.738+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d72380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.738+0000 7f0daa7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0d728c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.738+0000 7f0daaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d72380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50986/0 (socket says 172.21.15.62:50986) 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.738+0000 7f0daaffd640 1 -- 172.21.15.62:0/3502576959 learned_addr learned my addr 172.21.15.62:0/3502576959 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:32.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.738+0000 7f0dab7fe640 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 0x561aa0d073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.740+0000 7f0daaffd640 1 -- 172.21.15.62:0/3502576959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 msgr2=0x561aa0d728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.740+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0d728c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.740+0000 7f0daaffd640 1 -- 172.21.15.62:0/3502576959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 msgr2=0x561aa0d073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.740+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 0x561aa0d073f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.740+0000 7f0daaffd640 1 -- 172.21.15.62:0/3502576959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aa0d07ae0 con 0x561aa0ce34a0 2024-09-15T23:28:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.742+0000 7f0daaffd640 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d72380 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0da0025610 tx=0x7f0da0005cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba8901d807a881d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:32.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.743+0000 7f0d927fc640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:28:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.747+0000 7f0d927fc640 1 -- 172.21.15.62:0/3502576959 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da00195d0 con 0x561aa0ce34a0 2024-09-15T23:28:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.747+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3502576959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d9c04f070 con 0x561aa0ce34a0 2024-09-15T23:28:32.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.747+0000 7f0d927fc640 1 -- 172.21.15.62:0/3502576959 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0da0019770 con 0x561aa0ce34a0 2024-09-15T23:28:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.755+0000 7f0d927fc640 1 -- 172.21.15.62:0/3502576959 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0da0025ae0 con 0x561aa0ce34a0 2024-09-15T23:28:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.756+0000 7f0db3bd8740 10 client.27821 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:32.757 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.756+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3502576959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561aa0d79c40 con 0x561aa0ce34a0 2024-09-15T23:28:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.759+0000 7f0d927fc640 1 -- 172.21.15.62:0/3502576959 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d9c04f070 con 0x561aa0ce34a0 2024-09-15T23:28:32.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.759+0000 7f0d927fc640 1 -- 172.21.15.62:0/3502576959 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f0da00105d0 con 0x561aa0ce34a0 2024-09-15T23:28:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.759+0000 7f0db3bd8740 10 client.27821 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.759+0000 7f0db3bd8740 10 client.27821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.759+0000 7f0db3bd8740 20 client.27821 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.759+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561aa0d7a170 0x561aa0d7c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.759+0000 7f0db3bd8740 4 client.27821 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:28:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.759+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3502576959 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561aa0c50d60 con 0x561aa0d7a170 2024-09-15T23:28:32.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.759+0000 7f0daa7fc640 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561aa0d7a170 0x561aa0d7c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.760+0000 7f0daa7fc640 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561aa0d7a170 0x561aa0d7c560 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc8a555d49af5ab5 server_cookie=17b521278a5325e3 in_seq=0 out_seq=0 2024-09-15T23:28:32.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.760+0000 7f0d927fc640 10 client.27821 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:32.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.763+0000 7f0d927fc640 1 -- 172.21.15.62:0/3502576959 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561aa0c50d60 con 0x561aa0d7a170 2024-09-15T23:28:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.763+0000 7f0d927fc640 10 client.27821 handle_command_reply: tid=0 2024-09-15T23:28:32.764 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.764+0000 7f0db3bd8740 1 client.27821 shutdown 2024-09-15T23:28:32.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.769+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3502576959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 msgr2=0x561aa0d72380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:32.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.769+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d72380 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0da0025610 tx=0x7f0da0005cb0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.773+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3502576959 shutdown_connections 2024-09-15T23:28:32.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.773+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561aa0d7a170 0x561aa0d7c560 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.773+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561aa0cee680 0x561aa0d073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.773+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561aa0ce3dc0 0x561aa0d728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.773+0000 7f0db3bd8740 1 --2- 172.21.15.62:0/3502576959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561aa0ce34a0 0x561aa0d72380 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.773+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3502576959 >> 172.21.15.62:0/3502576959 conn(0x561aa0cdfa30 msgr2=0x561aa0ce5710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:32.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.773+0000 7f0d927fc640 0 client.27821 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:32.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.777+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3502576959 shutdown_connections 2024-09-15T23:28:32.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.777+0000 7f0db3bd8740 1 -- 172.21.15.62:0/3502576959 wait complete. 2024-09-15T23:28:32.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.778+0000 7f0db3bd8740 20 client.27821 trim_cache size 0 max 16384 2024-09-15T23:28:32.779 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:28:32.779 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:28:32.779 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:28:32.779 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 -- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d940a3450 msgr2=0x7f0d940a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d940a3450 0x7f0d940a5910 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x561aa0ce0db0 tx=0x7f0d9c01a600 comp rx=0 tx=0).stop 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 -- 172.21.15.62:0/3891704987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac1011e0 msgr2=0x7f0dac110e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac1011e0 0x7f0dac110e70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0da400ba50 tx=0x7f0da400bf10 comp rx=0 tx=0).stop 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 -- 172.21.15.62:0/3891704987 shutdown_connections 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0d940a3450 0x7f0d940a5910 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0dac105ec0 0x7f0dac1113b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dac1011e0 0x7f0dac110e70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 --2- 172.21.15.62:0/3891704987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0dac1008c0 0x7f0dac114dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:32.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 -- 172.21.15.62:0/3891704987 >> 172.21.15.62:0/3891704987 conn(0x7f0dac0fc7d0 msgr2=0x7f0dac103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:32.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 -- 172.21.15.62:0/3891704987 shutdown_connections 2024-09-15T23:28:32.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:32.779+0000 7f0db17ff640 1 -- 172.21.15.62:0/3891704987 wait complete. 2024-09-15T23:28:33.603 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:28:33.603 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:28:34.097 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:34.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:33 smithi062 ceph-mon[30949]: pgmap v738: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.4 MiB/s wr, 557 op/s 2024-09-15T23:28:34.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:33 smithi067 ceph-mon[35665]: pgmap v738: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.4 MiB/s wr, 557 op/s 2024-09-15T23:28:34.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:33 smithi148 ceph-mon[35610]: pgmap v738: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.4 MiB/s wr, 557 op/s 2024-09-15T23:28:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 735.0 s, 643.2 tps, lat 161.576 ms stddev 190.083 2024-09-15T23:28:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.656+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4059831297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc908100a90 msgr2=0x7fc908100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.656+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4059831297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc908100a90 0x7fc908100ef0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fc900013490 tx=0x7fc900032b30 comp rx=0 tx=0).stop 2024-09-15T23:28:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.657+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4059831297 shutdown_connections 2024-09-15T23:28:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.657+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4059831297 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908101430 0x7fc90806a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:34.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.657+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4059831297 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc908100a90 0x7fc908100ef0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:34.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.657+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4059831297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc908106a90 0x7fc908106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:34.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.657+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4059831297 >> 172.21.15.62:0/4059831297 conn(0x7fc9080fc7d0 msgr2=0x7fc9080febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:34.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.658+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4059831297 shutdown_connections 2024-09-15T23:28:34.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.661+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4059831297 wait complete. 2024-09-15T23:28:34.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.662+0000 7fc90e4f5640 1 Processor -- start 2024-09-15T23:28:34.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.668+0000 7fc90e4f5640 1 -- start start 2024-09-15T23:28:34.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.668+0000 7fc90e4f5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908100a90 0x7fc908073960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:34.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.668+0000 7fc90e4f5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc908101430 0x7fc908073ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:34.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.668+0000 7fc90e4f5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc908106a90 0x7fc90806cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:34.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.668+0000 7fc90e4f5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc90810feb0 con 0x7fc908106a90 2024-09-15T23:28:34.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.668+0000 7fc90e4f5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc90810fd30 con 0x7fc908101430 2024-09-15T23:28:34.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.668+0000 7fc90e4f5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc908110030 con 0x7fc908100a90 2024-09-15T23:28:34.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.674+0000 7fc9077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc908101430 0x7fc908073ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:34.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.674+0000 7fc9077fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc908101430 0x7fc908073ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50990/0 (socket says 172.21.15.62:50990) 2024-09-15T23:28:34.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.674+0000 7fc9077fe640 1 -- 172.21.15.62:0/4209816712 learned_addr learned my addr 172.21.15.62:0/4209816712 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:34.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.678+0000 7fc90ca92640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc908106a90 0x7fc90806cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:34.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.679+0000 7fc907fff640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908100a90 0x7fc908073960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:34.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.682+0000 7fc9077fe640 1 -- 172.21.15.62:0/4209816712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908100a90 msgr2=0x7fc908073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:34.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.682+0000 7fc9077fe640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908100a90 0x7fc908073960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:34.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.682+0000 7fc9077fe640 1 -- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc908106a90 msgr2=0x7fc90806cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:34.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.682+0000 7fc9077fe640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc908106a90 0x7fc90806cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:34.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.682+0000 7fc9077fe640 1 -- 172.21.15.62:0/4209816712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90806d520 con 0x7fc908101430 2024-09-15T23:28:34.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.686+0000 7fc9077fe640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc908101430 0x7fc908073ea0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc900006010 tx=0x7fc900011360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:34.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.686+0000 7fc9057fa640 1 -- 172.21.15.62:0/4209816712 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90003a040 con 0x7fc908101430 2024-09-15T23:28:34.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.686+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc90806d7b0 con 0x7fc908101430 2024-09-15T23:28:34.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.686+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc908071710 con 0x7fc908101430 2024-09-15T23:28:34.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.686+0000 7fc9057fa640 1 -- 172.21.15.62:0/4209816712 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc900011820 con 0x7fc908101430 2024-09-15T23:28:34.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.686+0000 7fc9057fa640 1 -- 172.21.15.62:0/4209816712 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9000042b0 con 0x7fc908101430 2024-09-15T23:28:34.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.692+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8f4005290 con 0x7fc908101430 2024-09-15T23:28:34.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.695+0000 7fc9057fa640 1 -- 172.21.15.62:0/4209816712 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc90003d070 con 0x7fc908101430 2024-09-15T23:28:34.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.696+0000 7fc9057fa640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc8f00a3380 0x7fc8f00a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:34.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.696+0000 7fc9057fa640 1 -- 172.21.15.62:0/4209816712 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc9000efbd0 con 0x7fc908101430 2024-09-15T23:28:34.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.697+0000 7fc907fff640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc8f00a3380 0x7fc8f00a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:34.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.698+0000 7fc907fff640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc8f00a3380 0x7fc8f00a5840 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8006fd0 tx=0x7fc8f80090e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:34.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:34.704+0000 7fc9057fa640 1 -- 172.21.15.62:0/4209816712 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc9000bc510 con 0x7fc908101430 2024-09-15T23:28:35.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.054+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8f4005580 con 0x7fc908101430 2024-09-15T23:28:35.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.056+0000 7fc9057fa640 1 -- 172.21.15.62:0/4209816712 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7fc9000bbe10 con 0x7fc908101430 2024-09-15T23:28:35.058 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:28:35.058 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:28:35.058 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:28:35.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.068+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc8f00a3380 msgr2=0x7fc8f00a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:35.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.068+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc8f00a3380 0x7fc8f00a5840 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8006fd0 tx=0x7fc8f80090e0 comp rx=0 tx=0).stop 2024-09-15T23:28:35.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.068+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc908101430 msgr2=0x7fc908073ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:35.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.068+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc908101430 0x7fc908073ea0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc900006010 tx=0x7fc900011360 comp rx=0 tx=0).stop 2024-09-15T23:28:35.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.070+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 shutdown_connections 2024-09-15T23:28:35.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.070+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc8f00a3380 0x7fc8f00a5840 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.070+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc908106a90 0x7fc90806cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.070+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc908101430 0x7fc908073ea0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.070+0000 7fc90e4f5640 1 --2- 172.21.15.62:0/4209816712 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc908100a90 0x7fc908073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.071+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 >> 172.21.15.62:0/4209816712 conn(0x7fc9080fc7d0 msgr2=0x7fc9080fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.071+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 shutdown_connections 2024-09-15T23:28:35.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:35.071+0000 7fc90e4f5640 1 -- 172.21.15.62:0/4209816712 wait complete. 2024-09-15T23:28:35.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:34 smithi062 ceph-mon[30949]: pgmap v739: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.8 MiB/s wr, 542 op/s 2024-09-15T23:28:35.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:34 smithi067 ceph-mon[35665]: pgmap v739: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.8 MiB/s wr, 542 op/s 2024-09-15T23:28:35.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:34 smithi148 ceph-mon[35610]: pgmap v739: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 5.8 MiB/s wr, 542 op/s 2024-09-15T23:28:35.972 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:28:36.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:35 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4209816712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:36.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:35 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4209816712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:36.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:35 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4209816712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:36.423 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:37.169 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:36 smithi062 ceph-mon[30949]: pgmap v740: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.9 MiB/s wr, 599 op/s 2024-09-15T23:28:37.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18c9ffb640 1 -- 172.21.15.62:0/3674015419 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4030e30 con 0x7f18cc1083f0 2024-09-15T23:28:37.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18d18d7640 1 -- 172.21.15.62:0/3674015419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18cc1083f0 msgr2=0x7f18cc1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18d18d7640 1 --2- 172.21.15.62:0/3674015419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18cc1083f0 0x7f18cc1087d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009be0 tx=0x7f18c402a6c0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18d18d7640 1 -- 172.21.15.62:0/3674015419 shutdown_connections 2024-09-15T23:28:37.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18d18d7640 1 --2- 172.21.15.62:0/3674015419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18cc10d520 0x7f18cc1150a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18d18d7640 1 --2- 172.21.15.62:0/3674015419 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18cc108da0 0x7f18cc10cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18d18d7640 1 --2- 172.21.15.62:0/3674015419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18cc1083f0 0x7f18cc1087d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18d18d7640 1 -- 172.21.15.62:0/3674015419 >> 172.21.15.62:0/3674015419 conn(0x7f18cc077af0 msgr2=0x7f18cc079f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18d18d7640 1 -- 172.21.15.62:0/3674015419 shutdown_connections 2024-09-15T23:28:37.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.234+0000 7f18d18d7640 1 -- 172.21.15.62:0/3674015419 wait complete. 2024-09-15T23:28:37.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.235+0000 7f18d18d7640 1 Processor -- start 2024-09-15T23:28:37.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.235+0000 7f18d18d7640 1 -- start start 2024-09-15T23:28:37.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18d18d7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18cc1083f0 0x7f18cc082870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18d18d7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18cc108da0 0x7f18cc07b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18d18d7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18cc10d520 0x7f18cc07be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18cc108da0 0x7f18cc07b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18cc108da0 0x7f18cc07b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51006/0 (socket says 172.21.15.62:51006) 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18caffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18cc1083f0 0x7f18cc082870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18d18d7640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc118610 con 0x7f18cc1083f0 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18ca7fc640 1 -- 172.21.15.62:0/2434769181 learned_addr learned my addr 172.21.15.62:0/2434769181 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc118490 con 0x7f18cc108da0 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc118790 con 0x7f18cc10d520 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.236+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18cc10d520 0x7f18cc07be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18cb7fe640 1 -- 172.21.15.62:0/2434769181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18cc108da0 msgr2=0x7f18cc07b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18cc108da0 0x7f18cc07b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18cb7fe640 1 -- 172.21.15.62:0/2434769181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18cc1083f0 msgr2=0x7f18cc082870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18cc1083f0 0x7f18cc082870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18cb7fe640 1 -- 172.21.15.62:0/2434769181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18cc07c480 con 0x7f18cc10d520 2024-09-15T23:28:37.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18cc10d520 0x7f18cc07be40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f18b8005fd0 tx=0x7f18b800ed20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18d08d5640 1 -- 172.21.15.62:0/2434769181 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b800ccc0 con 0x7f18cc10d520 2024-09-15T23:28:37.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18cc0803a0 con 0x7f18cc10d520 2024-09-15T23:28:37.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18d08d5640 1 -- 172.21.15.62:0/2434769181 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b80035f0 con 0x7f18cc10d520 2024-09-15T23:28:37.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18d08d5640 1 -- 172.21.15.62:0/2434769181 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b800ce60 con 0x7f18cc10d520 2024-09-15T23:28:37.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.237+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18cc0808e0 con 0x7f18cc10d520 2024-09-15T23:28:37.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.240+0000 7f18d08d5640 1 -- 172.21.15.62:0/2434769181 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f18b80037a0 con 0x7f18cc10d520 2024-09-15T23:28:37.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.241+0000 7f18d08d5640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18b00a3280 0x7f18b00a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.241+0000 7f18d08d5640 1 -- 172.21.15.62:0/2434769181 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f18b80cf8c0 con 0x7f18cc10d520 2024-09-15T23:28:37.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.241+0000 7f18caffd640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18b00a3280 0x7f18b00a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18d3cb0740 1 Processor -- start 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18d3cb0740 1 -- start start 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18caffd640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18b00a3280 0x7f18b00a5740 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f18c4006010 tx=0x7f18c4000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 0x561bc76c0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c05d0 0x561bc76c0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18d3cb0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561bc76262f0 con 0x561bc76cb370 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18d3cb0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561bc75cdcf0 con 0x561bc76c05d0 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18d3cb0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561bc76222b0 con 0x561bc76c1b40 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c05d0 0x561bc76c0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c05d0 0x561bc76c0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51010/0 (socket says 172.21.15.62:51010) 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18ca7fc640 1 -- 172.21.15.62:0/2626758078 learned_addr learned my addr 172.21.15.62:0/2626758078 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:37.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18caffd640 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 0x561bc76c0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18ca7fc640 1 -- 172.21.15.62:0/2626758078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 msgr2=0x561bc76c0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 0x561bc76c0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.242+0000 7f18ca7fc640 1 -- 172.21.15.62:0/2626758078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 msgr2=0x561bc76cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.243+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.243+0000 7f18ca7fc640 1 -- 172.21.15.62:0/2626758078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc75b23a0 con 0x561bc76c05d0 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.243+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.243+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c05d0 0x561bc76c0a10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009a80 tx=0x7f18bc009f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cc5a1bbf2e660c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.243+0000 7f18c1ffb640 1 -- 172.21.15.62:0/2626758078 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc017070 con 0x561bc76c05d0 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.243+0000 7f18c1ffb640 1 -- 172.21.15.62:0/2626758078 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18bc00eba0 con 0x561bc76c05d0 2024-09-15T23:28:37.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.243+0000 7f18c1ffb640 1 -- 172.21.15.62:0/2626758078 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc012630 con 0x561bc76c05d0 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 -- 172.21.15.62:0/2626758078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c05d0 msgr2=0x561bc76c0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c05d0 0x561bc76c0a10 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009a80 tx=0x7f18bc009f40 comp rx=0 tx=0).stop 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 -- 172.21.15.62:0/2626758078 shutdown_connections 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c05d0 0x561bc76c0a10 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/2626758078 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 0x561bc76c0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 -- 172.21.15.62:0/2626758078 >> 172.21.15.62:0/2626758078 conn(0x561bc7689310 msgr2=0x561bc76bcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 -- 172.21.15.62:0/2626758078 shutdown_connections 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 -- 172.21.15.62:0/2626758078 wait complete. 2024-09-15T23:28:37.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.244+0000 7f18d3cb0740 1 -- start start 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18d3cb0740 1 Processor -- start 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 0x561bc76c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18caffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18caffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50696/0 (socket says 172.21.15.62:50696) 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18caffd640 1 -- 172.21.15.62:0/1264395970 learned_addr learned my addr 172.21.15.62:0/1264395970 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76cb370 0x561bc76c8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561bc76ce070 con 0x561bc76c05d0 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561bc75e1060 con 0x561bc76cb370 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561bc76ce1f0 con 0x561bc76c1b40 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 0x561bc76c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18d3cb0740 10 client.0 _ll_register_callbacks cb 0x561bc73348c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:37.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76cb370 0x561bc76c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18caffd640 1 -- 172.21.15.62:0/1264395970 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 msgr2=0x561bc76c7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18caffd640 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 0x561bc76c7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18caffd640 1 -- 172.21.15.62:0/1264395970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76cb370 msgr2=0x561bc76c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18caffd640 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76cb370 0x561bc76c8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18caffd640 1 -- 172.21.15.62:0/1264395970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc73973c0 con 0x561bc76c05d0 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.245+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76cb370 0x561bc76c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.246+0000 7f18caffd640 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76cad00 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f18c4033000 tx=0x7f18c4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3112780ba2c54b75 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.246+0000 7f18aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.246+0000 7f18aeffd640 1 -- 172.21.15.62:0/1264395970 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c404e070 con 0x561bc76c05d0 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.246+0000 7f18aeffd640 1 -- 172.21.15.62:0/1264395970 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f18c4004040 con 0x561bc76c05d0 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.246+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18bc00eba0 con 0x561bc76c05d0 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.247+0000 7f18aeffd640 1 -- 172.21.15.62:0/1264395970 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f18c4058050 con 0x561bc76c05d0 2024-09-15T23:28:37.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.247+0000 7f18d3cb0740 10 client.18285 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.247+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561bc76d9a60 con 0x561bc76c05d0 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.248+0000 7f18aeffd640 1 -- 172.21.15.62:0/1264395970 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18bc00eba0 con 0x561bc76c05d0 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.248+0000 7f18aeffd640 1 -- 172.21.15.62:0/1264395970 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f18c403d8b0 con 0x561bc76c05d0 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.248+0000 7f18d3cb0740 10 client.18285 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.248+0000 7f18d3cb0740 10 client.18285 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.248+0000 7f18d3cb0740 20 client.18285 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.248+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x561bc76daf20 0x561bc76dd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.248+0000 7f18d3cb0740 4 client.18285 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.248+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bc7633860 con 0x561bc76daf20 2024-09-15T23:28:37.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.248+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x561bc76daf20 0x561bc76dd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.249+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x561bc76daf20 0x561bc76dd3e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f6e810589f75d27b server_cookie=373c61735d5301d8 in_seq=0 out_seq=0 2024-09-15T23:28:37.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.249+0000 7f18aeffd640 10 client.18285 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.249+0000 7f18aeffd640 1 -- 172.21.15.62:0/1264395970 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561bc7633860 con 0x561bc76daf20 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.249+0000 7f18aeffd640 10 client.18285 handle_command_reply: tid=0 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.249+0000 7f18d3cb0740 1 client.18285 shutdown 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 msgr2=0x561bc76cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76cad00 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f18c4033000 tx=0x7f18c4009500 comp rx=0 tx=0).stop 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 shutdown_connections 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x561bc76daf20 0x561bc76dd3e0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18aeffd640 0 client.18285 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76cb370 0x561bc76c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c1b40 0x561bc76c7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1264395970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76cad00 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 >> 172.21.15.62:0/1264395970 conn(0x561bc7689310 msgr2=0x561bc76c2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 shutdown_connections 2024-09-15T23:28:37.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1264395970 wait complete. 2024-09-15T23:28:37.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.250+0000 7f18d3cb0740 20 client.18285 trim_cache size 0 max 16384 2024-09-15T23:28:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18d3cb0740 1 Processor -- start 2024-09-15T23:28:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18d3cb0740 1 -- start start 2024-09-15T23:28:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76d7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76cb370 0x561bc76ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18d3cb0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc1c0470 con 0x561bc76c05d0 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18d3cb0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc1bdd40 con 0x561bc76c1b40 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18d3cb0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc1bbb90 con 0x561bc76cb370 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18caffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76d7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51024/0 (socket says 172.21.15.62:51024) 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18caffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76d7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50702/0 (socket says 172.21.15.62:50702) 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18ca7fc640 1 -- 172.21.15.62:0/4008161237 learned_addr learned my addr 172.21.15.62:0/4008161237 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:37.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.266+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/4008161237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76cb370 0x561bc76ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18ca7fc640 1 -- 172.21.15.62:0/4008161237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76cb370 msgr2=0x561bc76ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/4008161237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76cb370 0x561bc76ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18ca7fc640 1 -- 172.21.15.62:0/4008161237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 msgr2=0x561bc76d7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/4008161237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76d7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18ca7fc640 1 -- 172.21.15.62:0/4008161237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc76cb030 con 0x561bc76c1b40 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18caffd640 1 --2- 172.21.15.62:0/4008161237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76d7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/4008161237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76cc760 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f18bc013df0 tx=0x7f18bc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=828744b3f34407d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18c17fa640 1 -- 172.21.15.62:0/4008161237 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc017040 con 0x561bc76c1b40 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18c17fa640 1 -- 172.21.15.62:0/4008161237 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18bc013950 con 0x561bc76c1b40 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.267+0000 7f18c17fa640 1 -- 172.21.15.62:0/4008161237 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc028cb0 con 0x561bc76c1b40 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 -- 172.21.15.62:0/4008161237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 msgr2=0x561bc76cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/4008161237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76cc760 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f18bc013df0 tx=0x7f18bc008040 comp rx=0 tx=0).stop 2024-09-15T23:28:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 -- 172.21.15.62:0/4008161237 shutdown_connections 2024-09-15T23:28:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/4008161237 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76cb370 0x561bc76ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/4008161237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76cc760 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/4008161237 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76c05d0 0x561bc76d7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 -- 172.21.15.62:0/4008161237 >> 172.21.15.62:0/4008161237 conn(0x561bc7689310 msgr2=0x561bc76bc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 -- 172.21.15.62:0/4008161237 shutdown_connections 2024-09-15T23:28:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 -- 172.21.15.62:0/4008161237 wait complete. 2024-09-15T23:28:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.268+0000 7f18d3cb0740 1 -- start start 2024-09-15T23:28:37.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18d3cb0740 1 Processor -- start 2024-09-15T23:28:37.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c05d0 0x561bc76d3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76d39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18d3cb0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18d3cb0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc1bc200 con 0x561bc76cb370 2024-09-15T23:28:37.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18d3cb0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc1bd670 con 0x561bc76c1b40 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18caffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c05d0 0x561bc76d3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76d39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18ca7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76d39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51038/0 (socket says 172.21.15.62:51038) 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18ca7fc640 1 -- 172.21.15.62:0/1773314848 learned_addr learned my addr 172.21.15.62:0/1773314848 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18caffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c05d0 0x561bc76d3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53388/0 (socket says 172.21.15.62:53388) 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18d3cb0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc1bc8d0 con 0x561bc76c05d0 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.269+0000 7f18d3cb0740 10 client.0 _ll_register_callbacks cb 0x7f18c4009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18caffd640 1 -- 172.21.15.62:0/1773314848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 msgr2=0x561bc76d39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18caffd640 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76d39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18caffd640 1 -- 172.21.15.62:0/1773314848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 msgr2=0x561bc76ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18caffd640 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18caffd640 1 -- 172.21.15.62:0/1773314848 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc76d4090 con 0x561bc76c05d0 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76d39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18cb7fe640 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18caffd640 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c05d0 0x561bc76d3490 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f18c403dec0 tx=0x7f18c402ad40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4412da9a53514af0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18aeffd640 1 -- 172.21.15.62:0/1773314848 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4062060 con 0x561bc76c05d0 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18aeffd640 1 -- 172.21.15.62:0/1773314848 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f18c4053850 con 0x561bc76c05d0 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.270+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1773314848 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18bc013950 con 0x561bc76c05d0 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.272+0000 7f18aeffd640 1 -- 172.21.15.62:0/1773314848 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f18c404e070 con 0x561bc76c05d0 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.272+0000 7f18d3cb0740 10 client.27571 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:37.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.272+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1773314848 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561bc7746290 con 0x561bc76c05d0 2024-09-15T23:28:37.276 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:36 smithi067 ceph-mon[35665]: pgmap v740: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.9 MiB/s wr, 599 op/s 2024-09-15T23:28:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.273+0000 7f18aeffd640 1 -- 172.21.15.62:0/1773314848 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c4030cd0 con 0x561bc76c05d0 2024-09-15T23:28:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.273+0000 7f18aeffd640 1 -- 172.21.15.62:0/1773314848 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f18c405d9e0 con 0x561bc76c05d0 2024-09-15T23:28:37.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.273+0000 7f18d3cb0740 10 client.27571 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.273+0000 7f18d3cb0740 10 client.27571 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.273+0000 7f18d3cb0740 20 client.27571 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.273+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x561bc76daf20 0x561bc77489d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.273+0000 7f18d3cb0740 4 client.27571 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.273+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1773314848 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f18bc013950 con 0x561bc76daf20 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x561bc76daf20 0x561bc77489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18aeffd640 10 client.27571 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18ca7fc640 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x561bc76daf20 0x561bc77489d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4df00eec5f08ac68 server_cookie=2d0dccbb95c1d236 in_seq=0 out_seq=0 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18aeffd640 1 -- 172.21.15.62:0/1773314848 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f18bc013950 con 0x561bc76daf20 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18aeffd640 10 client.27571 handle_command_reply: tid=0 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18d3cb0740 1 client.27571 shutdown 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1773314848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c05d0 msgr2=0x561bc76d3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c05d0 0x561bc76d3490 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f18c403dec0 tx=0x7f18c402ad40 comp rx=0 tx=0).stop 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1773314848 shutdown_connections 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x561bc76daf20 0x561bc77489d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bc76cb370 0x561bc76ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561bc76c1b40 0x561bc76d39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18d3cb0740 1 --2- 172.21.15.62:0/1773314848 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561bc76c05d0 0x561bc76d3490 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18aeffd640 0 client.27571 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.274+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1773314848 >> 172.21.15.62:0/1773314848 conn(0x561bc7689310 msgr2=0x561bc76bb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1773314848 shutdown_connections 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d3cb0740 1 -- 172.21.15.62:0/1773314848 wait complete. 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d3cb0740 20 client.27571 trim_cache size 0 max 16384 2024-09-15T23:28:37.279 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:28:37.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18b00a3280 msgr2=0x7f18b00a5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18b00a3280 0x7f18b00a5740 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f18c4006010 tx=0x7f18c4000f40 comp rx=0 tx=0).stop 2024-09-15T23:28:37.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18cc10d520 msgr2=0x7f18cc07be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:37.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18cc10d520 0x7f18cc07be40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f18b8005fd0 tx=0x7f18b800ed20 comp rx=0 tx=0).stop 2024-09-15T23:28:37.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 shutdown_connections 2024-09-15T23:28:37.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f18b00a3280 0x7f18b00a5740 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18cc10d520 0x7f18cc07be40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18cc108da0 0x7f18cc07b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 --2- 172.21.15.62:0/2434769181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18cc1083f0 0x7f18cc082870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:37.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.275+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 >> 172.21.15.62:0/2434769181 conn(0x7f18cc077af0 msgr2=0x7f18cc078f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:37.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.276+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 shutdown_connections 2024-09-15T23:28:37.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:37.276+0000 7f18d18d7640 1 -- 172.21.15.62:0/2434769181 wait complete. 2024-09-15T23:28:37.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:36 smithi148 ceph-mon[35610]: pgmap v740: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.9 MiB/s wr, 599 op/s 2024-09-15T23:28:38.054 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:28:38.564 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:39.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:39 smithi062 ceph-mon[30949]: pgmap v741: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.3 MiB/s wr, 522 op/s 2024-09-15T23:28:39.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:39 smithi067 ceph-mon[35665]: pgmap v741: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.3 MiB/s wr, 522 op/s 2024-09-15T23:28:39.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:39 smithi148 ceph-mon[35610]: pgmap v741: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.3 MiB/s wr, 522 op/s 2024-09-15T23:28:39.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 740.0 s, 664.0 tps, lat 135.552 ms stddev 166.936 2024-09-15T23:28:40.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.190+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1461620061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 msgr2=0x7f2f4810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.190+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1461620061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 0x7f2f4810ec40 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009cd0 tx=0x7f2f4002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.190+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1461620061 shutdown_connections 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.190+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1461620061 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 0x7f2f4810ec40 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.190+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1461620061 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f481036e0 0x7f2f48107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.190+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1461620061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f48102d30 0x7f2f48103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.190+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1461620061 >> 172.21.15.62:0/1461620061 conn(0x7f2f480fc7d0 msgr2=0x7f2f480febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.190+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1461620061 shutdown_connections 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.190+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1461620061 wait complete. 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.191+0000 7f2f4fa55640 1 Processor -- start 2024-09-15T23:28:40.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.191+0000 7f2f4fa55640 1 -- start start 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4fa55640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f48102d30 0x7f2f48198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4fa55640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f481036e0 0x7f2f48198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4fa55640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 0x7f2f4819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4fa55640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481120a0 con 0x7f2f481082a0 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4fa55640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f48111f20 con 0x7f2f48102d30 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4fa55640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2f48112220 con 0x7f2f481036e0 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4dff2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 0x7f2f4819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4dff2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 0x7f2f4819ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50734/0 (socket says 172.21.15.62:50734) 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4dff2640 1 -- 172.21.15.62:0/1159973203 learned_addr learned my addr 172.21.15.62:0/1159973203 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f48102d30 0x7f2f48198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f481036e0 0x7f2f48198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4dff2640 1 -- 172.21.15.62:0/1159973203 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f481036e0 msgr2=0x7f2f48198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:40.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4dff2640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f481036e0 0x7f2f48198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4dff2640 1 -- 172.21.15.62:0/1159973203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f48102d30 msgr2=0x7f2f48198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4dff2640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f48102d30 0x7f2f48198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4dff2640 1 -- 172.21.15.62:0/1159973203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4819d510 con 0x7f2f481082a0 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4dff2640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 0x7f2f4819ce30 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f2f40005cc0 tx=0x7f2f40003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f3e7fc640 1 -- 172.21.15.62:0/1159973203 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40004040 con 0x7f2f481082a0 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f3e7fc640 1 -- 172.21.15.62:0/1159973203 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f400046c0 con 0x7f2f481082a0 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f3e7fc640 1 -- 172.21.15.62:0/1159973203 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40046480 con 0x7f2f481082a0 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.192+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1159973203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4819d7a0 con 0x7f2f481082a0 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.193+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1159973203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f481a50e0 con 0x7f2f481082a0 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.195+0000 7f2f3e7fc640 1 -- 172.21.15.62:0/1159973203 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2f400305a0 con 0x7f2f481082a0 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.195+0000 7f2f51e2e740 1 Processor -- start 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.195+0000 7f2f51e2e740 1 -- start start 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.195+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda740090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.195+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 0x55ffda740a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.195+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda74b370 0x55ffda74d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.195+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ffda6a62f0 con 0x55ffda741b40 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.195+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ffda64dcf0 con 0x55ffda74b370 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.195+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ffda6a22b0 con 0x55ffda7405d0 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda740090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda740090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50748/0 (socket says 172.21.15.62:50748) 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 -- 172.21.15.62:0/419244990 learned_addr learned my addr 172.21.15.62:0/419244990 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f3e7fc640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f340a32c0 0x7f2f340a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/419244990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 0x55ffda740a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/419244990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda74b370 0x55ffda74d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 -- 172.21.15.62:0/419244990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 msgr2=0x55ffda740a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:40.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/419244990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 0x55ffda740a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 -- 172.21.15.62:0/419244990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda74b370 msgr2=0x55ffda74d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/419244990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda74b370 0x55ffda74d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 -- 172.21.15.62:0/419244990 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffda6323a0 con 0x55ffda741b40 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f3e7fc640 1 -- 172.21.15.62:0/1159973203 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2f4002e030 con 0x7f2f481082a0 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4dff2640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f340a32c0 0x7f2f340a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/419244990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda740090 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2f380058c0 tx=0x7f2f3800b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b38971673ade63b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f337fe640 1 -- 172.21.15.62:0/419244990 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38019030 con 0x55ffda741b40 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f337fe640 1 -- 172.21.15.62:0/419244990 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f38005bf0 con 0x55ffda741b40 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.196+0000 7f2f337fe640 1 -- 172.21.15.62:0/419244990 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f380106c0 con 0x55ffda741b40 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f51e2e740 1 -- 172.21.15.62:0/419244990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 msgr2=0x55ffda740090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f4dff2640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f340a32c0 0x7f2f340a5780 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2f40005e30 tx=0x7f2f400095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/419244990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda740090 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f2f380058c0 tx=0x7f2f3800b5c0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f51e2e740 1 -- 172.21.15.62:0/419244990 shutdown_connections 2024-09-15T23:28:40.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/419244990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda74b370 0x55ffda74d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/419244990 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 0x55ffda740a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/419244990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda740090 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f51e2e740 1 -- 172.21.15.62:0/419244990 >> 172.21.15.62:0/419244990 conn(0x55ffda709310 msgr2=0x55ffda73cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f51e2e740 1 -- 172.21.15.62:0/419244990 shutdown_connections 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.197+0000 7f2f51e2e740 1 -- 172.21.15.62:0/419244990 wait complete. 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f51e2e740 1 -- start start 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f51e2e740 1 Processor -- start 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 0x55ffda74ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda741b40 0x55ffda748050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda74b370 0x55ffda7487e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ffda74e070 con 0x55ffda74b370 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ffda661060 con 0x55ffda741b40 2024-09-15T23:28:40.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ffda74e1f0 con 0x55ffda7405d0 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f51e2e740 10 client.0 _ll_register_callbacks cb 0x55ffda3b48c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f4cff0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda74b370 0x55ffda7487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f4cff0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda74b370 0x55ffda7487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50762/0 (socket says 172.21.15.62:50762) 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f4cff0640 1 -- 172.21.15.62:0/2034001142 learned_addr learned my addr 172.21.15.62:0/2034001142 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.198+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda741b40 0x55ffda748050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 0x55ffda74ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f4cff0640 1 -- 172.21.15.62:0/2034001142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 msgr2=0x55ffda74ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 0x55ffda74ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f4cff0640 1 -- 172.21.15.62:0/2034001142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda741b40 msgr2=0x55ffda748050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda741b40 0x55ffda748050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f4cff0640 1 -- 172.21.15.62:0/2034001142 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffda4173c0 con 0x55ffda74b370 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda74b370 0x55ffda7487e0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800be10 tx=0x7f2f38015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f1747a08698221 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:40.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f30ff9640 1 -- 172.21.15.62:0/2034001142 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f380053a0 con 0x55ffda74b370 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f30ff9640 1 -- 172.21.15.62:0/2034001142 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2f38005540 con 0x55ffda74b370 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2034001142 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ffda6b3860 con 0x55ffda74b370 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f30ff9640 1 -- 172.21.15.62:0/2034001142 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2f38010460 con 0x55ffda74b370 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f51e2e740 10 client.18303 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2034001142 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ffda759b50 con 0x55ffda74b370 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.199+0000 7f2f30ff9640 1 -- 172.21.15.62:0/2034001142 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ffda6b3860 con 0x55ffda74b370 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f30ff9640 1 -- 172.21.15.62:0/2034001142 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f2f3801d5c0 con 0x55ffda74b370 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f51e2e740 10 client.18303 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f51e2e740 10 client.18303 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f51e2e740 20 client.18303 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55ffda75af50 0x55ffda75d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f51e2e740 4 client.18303 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2034001142 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffda6add70 con 0x55ffda75af50 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55ffda75af50 0x55ffda75d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55ffda75af50 0x55ffda75d340 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=36e5871ad27c9a6a server_cookie=51c443421127918c in_seq=0 out_seq=0 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.200+0000 7f2f30ff9640 10 client.18303 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f30ff9640 1 -- 172.21.15.62:0/2034001142 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ffda6add70 con 0x55ffda75af50 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f30ff9640 10 client.18303 handle_command_reply: tid=0 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f51e2e740 1 client.18303 shutdown 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2034001142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda74b370 msgr2=0x55ffda7487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda74b370 0x55ffda7487e0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800be10 tx=0x7f2f38015040 comp rx=0 tx=0).stop 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2034001142 shutdown_connections 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55ffda75af50 0x55ffda75d340 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda74b370 0x55ffda7487e0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda741b40 0x55ffda748050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f30ff9640 0 client.18303 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2034001142 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda7405d0 0x55ffda74ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.201+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2034001142 >> 172.21.15.62:0/2034001142 conn(0x55ffda709310 msgr2=0x55ffda742960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.202+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2034001142 shutdown_connections 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.202+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2034001142 wait complete. 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.202+0000 7f2f51e2e740 20 client.18303 trim_cache size 0 max 16384 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f51e2e740 1 Processor -- start 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f51e2e740 1 -- start start 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 0x55ffda741300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda73c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda74b370 0x55ffda73cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481b14e0 con 0x55ffda741b40 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481aedb0 con 0x55ffda7405d0 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481acc00 con 0x55ffda74b370 2024-09-15T23:28:40.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4d7f1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda73c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4d7f1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda73c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50774/0 (socket says 172.21.15.62:50774) 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4d7f1640 1 -- 172.21.15.62:0/2845477593 learned_addr learned my addr 172.21.15.62:0/2845477593 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/2845477593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 0x55ffda741300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4d7f1640 1 -- 172.21.15.62:0/2845477593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda74b370 msgr2=0x55ffda73cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/2845477593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda74b370 0x55ffda73cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4d7f1640 1 -- 172.21.15.62:0/2845477593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 msgr2=0x55ffda741300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/2845477593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 0x55ffda741300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4d7f1640 1 -- 172.21.15.62:0/2845477593 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffda73c4f0 con 0x55ffda741b40 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.215+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/2845477593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda73c700 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x55ffda73dac0 tx=0x7f2f440139a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0c8c0065472855 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f32ffd640 1 -- 172.21.15.62:0/2845477593 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f44004020 con 0x55ffda741b40 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f32ffd640 1 -- 172.21.15.62:0/2845477593 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f440041c0 con 0x55ffda741b40 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f32ffd640 1 -- 172.21.15.62:0/2845477593 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f440107e0 con 0x55ffda741b40 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2845477593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 msgr2=0x55ffda73c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:40.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2845477593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda73c700 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x55ffda73dac0 tx=0x7f2f440139a0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2845477593 shutdown_connections 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2845477593 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda74b370 0x55ffda73cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2845477593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda73c700 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/2845477593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 0x55ffda741300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2845477593 >> 172.21.15.62:0/2845477593 conn(0x55ffda709310 msgr2=0x55ffda737a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2845477593 shutdown_connections 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.216+0000 7f2f51e2e740 1 -- 172.21.15.62:0/2845477593 wait complete. 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f51e2e740 1 -- start start 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f51e2e740 1 Processor -- start 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 0x55ffda7539f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda74ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f51e2e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda74b370 0x55ffda74f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481ad270 con 0x55ffda741b40 2024-09-15T23:28:40.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481ae6e0 con 0x55ffda7405d0 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f51e2e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2f481ad940 con 0x55ffda74b370 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f51e2e740 10 client.0 _ll_register_callbacks cb 0x7f2f3800bf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f4d7f1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda74ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f4cff0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 0x55ffda7539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f4d7f1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda74ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50782/0 (socket says 172.21.15.62:50782) 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f4cff0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 0x55ffda7539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51092/0 (socket says 172.21.15.62:51092) 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.217+0000 7f2f4d7f1640 1 -- 172.21.15.62:0/409517955 learned_addr learned my addr 172.21.15.62:0/409517955 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda74b370 0x55ffda74f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f4d7f1640 1 -- 172.21.15.62:0/409517955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda74b370 msgr2=0x55ffda74f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda74b370 0x55ffda74f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f4d7f1640 1 -- 172.21.15.62:0/409517955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 msgr2=0x55ffda7539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 0x55ffda7539f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f4d7f1640 1 -- 172.21.15.62:0/409517955 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffda74f8a0 con 0x55ffda741b40 2024-09-15T23:28:40.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f4d7f1640 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda74ed20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400bdc0 tx=0x7f2f44004130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf40e2ad7cf7b20 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f30ff9640 1 -- 172.21.15.62:0/409517955 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f440035f0 con 0x55ffda741b40 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f30ff9640 1 -- 172.21.15.62:0/409517955 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2f44003790 con 0x55ffda741b40 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f51e2e740 1 -- 172.21.15.62:0/409517955 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f440041c0 con 0x55ffda741b40 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f30ff9640 1 -- 172.21.15.62:0/409517955 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2f44003990 con 0x55ffda741b40 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f51e2e740 10 client.18309 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.218+0000 7f2f51e2e740 1 -- 172.21.15.62:0/409517955 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ffda7c5950 con 0x55ffda741b40 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f30ff9640 1 -- 172.21.15.62:0/409517955 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f440041c0 con 0x55ffda741b40 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f30ff9640 1 -- 172.21.15.62:0/409517955 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f2f4401c460 con 0x55ffda741b40 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f51e2e740 10 client.18309 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f51e2e740 10 client.18309 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f51e2e740 20 client.18309 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55ffda7c6100 0x55ffda7c8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:40.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f51e2e740 4 client.18309 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:28:40.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f51e2e740 1 -- 172.21.15.62:0/409517955 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ffda62a170 con 0x55ffda7c6100 2024-09-15T23:28:40.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55ffda7c6100 0x55ffda7c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:40.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f30ff9640 10 client.18309 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:28:40.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.219+0000 7f2f4cff0640 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55ffda7c6100 0x55ffda7c8520 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f40b20563cd0a45 server_cookie=4715396ee887b699 in_seq=0 out_seq=0 2024-09-15T23:28:40.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f30ff9640 1 -- 172.21.15.62:0/409517955 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ffda62a170 con 0x55ffda7c6100 2024-09-15T23:28:40.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f30ff9640 10 client.18309 handle_command_reply: tid=0 2024-09-15T23:28:40.434 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:28:40.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 client.18309 shutdown 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 -- 172.21.15.62:0/409517955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 msgr2=0x55ffda74ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda74ed20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400bdc0 tx=0x7f2f44004130 comp rx=0 tx=0).stop 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 -- 172.21.15.62:0/409517955 shutdown_connections 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55ffda7c6100 0x55ffda7c8520 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ffda74b370 0x55ffda74f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ffda741b40 0x55ffda74ed20 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 --2- 172.21.15.62:0/409517955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ffda7405d0 0x55ffda7539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f30ff9640 0 client.18309 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 -- 172.21.15.62:0/409517955 >> 172.21.15.62:0/409517955 conn(0x55ffda709310 msgr2=0x55ffda7464f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 -- 172.21.15.62:0/409517955 shutdown_connections 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 1 -- 172.21.15.62:0/409517955 wait complete. 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.220+0000 7f2f51e2e740 20 client.18309 trim_cache size 0 max 16384 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f340a32c0 msgr2=0x7f2f340a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:40.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f340a32c0 0x7f2f340a5780 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f2f40005e30 tx=0x7f2f400095c0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 msgr2=0x7f2f4819ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 0x7f2f4819ce30 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f2f40005cc0 tx=0x7f2f40003970 comp rx=0 tx=0).stop 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1159973203 shutdown_connections 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2f340a32c0 0x7f2f340a5780 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f481082a0 0x7f2f4819ce30 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2f481036e0 0x7f2f48198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 --2- 172.21.15.62:0/1159973203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f48102d30 0x7f2f48198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1159973203 >> 172.21.15.62:0/1159973203 conn(0x7f2f480fc7d0 msgr2=0x7f2f48105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1159973203 shutdown_connections 2024-09-15T23:28:40.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:39.221+0000 7f2f4fa55640 1 -- 172.21.15.62:0/1159973203 wait complete. 2024-09-15T23:28:41.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:41 smithi062 ceph-mon[30949]: pgmap v742: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 MiB/s wr, 572 op/s 2024-09-15T23:28:41.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:41 smithi067 ceph-mon[35665]: pgmap v742: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 MiB/s wr, 572 op/s 2024-09-15T23:28:41.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:41 smithi148 ceph-mon[35610]: pgmap v742: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 MiB/s wr, 572 op/s 2024-09-15T23:28:42.136 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:28:42.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.134+0000 7f00c553a640 1 -- 172.21.15.62:0/4186327609 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c0109380 msgr2=0x7f00c0109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:42.137 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.134+0000 7f00c553a640 1 --2- 172.21.15.62:0/4186327609 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c0109380 0x7f00c0109800 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f00b8009900 tx=0x7f00b802a440 comp rx=0 tx=0).stop 2024-09-15T23:28:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.140+0000 7f00c553a640 1 -- 172.21.15.62:0/4186327609 shutdown_connections 2024-09-15T23:28:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.140+0000 7f00c553a640 1 --2- 172.21.15.62:0/4186327609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c0109d40 0x7f00c01105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.145 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.140+0000 7f00c553a640 1 --2- 172.21.15.62:0/4186327609 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c0109380 0x7f00c0109800 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.140+0000 7f00c553a640 1 --2- 172.21.15.62:0/4186327609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00c0108470 0x7f00c0108870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.140+0000 7f00c553a640 1 -- 172.21.15.62:0/4186327609 >> 172.21.15.62:0/4186327609 conn(0x7f00c0077490 msgr2=0x7f00c00798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.142+0000 7f00c553a640 1 -- 172.21.15.62:0/4186327609 shutdown_connections 2024-09-15T23:28:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.142+0000 7f00c553a640 1 -- 172.21.15.62:0/4186327609 wait complete. 2024-09-15T23:28:42.146 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.142+0000 7f00c553a640 1 Processor -- start 2024-09-15T23:28:42.149 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00c553a640 1 -- start start 2024-09-15T23:28:42.149 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00c553a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00c0108470 0x7f00c0073750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00c553a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c0109380 0x7f00c0073c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00c553a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c0109d40 0x7f00c0071d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00c553a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f00c006c7a0 con 0x7f00c0108470 2024-09-15T23:28:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00c553a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00c006c620 con 0x7f00c0109380 2024-09-15T23:28:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00c553a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f00c006c920 con 0x7f00c0109d40 2024-09-15T23:28:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00beffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00c0108470 0x7f00c0073750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00beffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00c0108470 0x7f00c0073750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50802/0 (socket says 172.21.15.62:50802) 2024-09-15T23:28:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.146+0000 7f00beffd640 1 -- 172.21.15.62:0/3699519320 learned_addr learned my addr 172.21.15.62:0/3699519320 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:42.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.147+0000 7f00be7fc640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c0109380 0x7f00c0073c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.148+0000 7f00beffd640 1 -- 172.21.15.62:0/3699519320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c0109d40 msgr2=0x7f00c0071d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.148+0000 7f00beffd640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c0109d40 0x7f00c0071d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.148+0000 7f00beffd640 1 -- 172.21.15.62:0/3699519320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c0109380 msgr2=0x7f00c0073c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.148+0000 7f00beffd640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c0109380 0x7f00c0073c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.148+0000 7f00beffd640 1 -- 172.21.15.62:0/3699519320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c0072560 con 0x7f00c0108470 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.149+0000 7f00beffd640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00c0108470 0x7f00c0073750 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f00b400d950 tx=0x7f00b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.150+0000 7f00a7fff640 1 -- 172.21.15.62:0/3699519320 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4017070 con 0x7f00c0108470 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.150+0000 7f00c553a640 1 -- 172.21.15.62:0/3699519320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c0072850 con 0x7f00c0108470 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.150+0000 7f00c553a640 1 -- 172.21.15.62:0/3699519320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c0080a70 con 0x7f00c0108470 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.150+0000 7f00a7fff640 1 -- 172.21.15.62:0/3699519320 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b4009c30 con 0x7f00c0108470 2024-09-15T23:28:42.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.150+0000 7f00a7fff640 1 -- 172.21.15.62:0/3699519320 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b400f690 con 0x7f00c0108470 2024-09-15T23:28:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.157+0000 7f00c82d4740 1 Processor -- start 2024-09-15T23:28:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.158+0000 7f00a7fff640 1 -- 172.21.15.62:0/3699519320 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f00b400f830 con 0x7f00c0108470 2024-09-15T23:28:42.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.158+0000 7f00c82d4740 1 -- start start 2024-09-15T23:28:42.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.164+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 0x5625f94afda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 0x5625f94b2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00c82d4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5625f940e2f0 con 0x5625f94aa9a0 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00c82d4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5625f93b6270 con 0x5625f94ab2e0 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00c82d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5625f940a2b0 con 0x5625f94b0370 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 0x5625f94afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00beffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 0x5625f94afda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51128/0 (socket says 172.21.15.62:51128) 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00beffd640 1 -- 172.21.15.62:0/271660866 learned_addr learned my addr 172.21.15.62:0/271660866 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00beffd640 1 -- 172.21.15.62:0/271660866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 msgr2=0x5625f94b2830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00beffd640 1 --2- 172.21.15.62:0/271660866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 0x5625f94b2830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.168+0000 7f00beffd640 1 -- 172.21.15.62:0/271660866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 msgr2=0x5625f94aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.170+0000 7f00bf7fe640 1 --2- 172.21.15.62:0/271660866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.170+0000 7f00a7fff640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a80a3520 0x7f00a80a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.172 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.170+0000 7f00a7fff640 1 -- 172.21.15.62:0/3699519320 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f00b40cf790 con 0x7f00c0108470 2024-09-15T23:28:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.171+0000 7f00beffd640 1 --2- 172.21.15.62:0/271660866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94aada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.171+0000 7f00beffd640 1 -- 172.21.15.62:0/271660866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625f93b5340 con 0x5625f94ab2e0 2024-09-15T23:28:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.175+0000 7f00be7fc640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a80a3520 0x7f00a80a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.176+0000 7f00beffd640 1 --2- 172.21.15.62:0/271660866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 0x5625f94afda0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f00b4097e30 tx=0x7f00b400bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab3e0468bc317c8c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.176+0000 7f00be7fc640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a80a3520 0x7f00a80a59e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f00b80315a0 tx=0x7f00b8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.177+0000 7f00a57fa640 1 -- 172.21.15.62:0/271660866 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4020030 con 0x5625f94ab2e0 2024-09-15T23:28:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.177+0000 7f00a57fa640 1 -- 172.21.15.62:0/271660866 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b4009de0 con 0x5625f94ab2e0 2024-09-15T23:28:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.178+0000 7f00c82d4740 1 -- 172.21.15.62:0/271660866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 msgr2=0x5625f94afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.178+0000 7f00c82d4740 1 --2- 172.21.15.62:0/271660866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 0x5625f94afda0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f00b4097e30 tx=0x7f00b400bf60 comp rx=0 tx=0).stop 2024-09-15T23:28:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.182+0000 7f00c82d4740 1 -- 172.21.15.62:0/271660866 shutdown_connections 2024-09-15T23:28:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.182+0000 7f00c82d4740 1 --2- 172.21.15.62:0/271660866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 0x5625f94b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.182+0000 7f00c82d4740 1 --2- 172.21.15.62:0/271660866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 0x5625f94afda0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.182+0000 7f00c82d4740 1 --2- 172.21.15.62:0/271660866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.182+0000 7f00c82d4740 1 -- 172.21.15.62:0/271660866 >> 172.21.15.62:0/271660866 conn(0x5625f94a6010 msgr2=0x5625f94a6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.183+0000 7f00c82d4740 1 -- 172.21.15.62:0/271660866 shutdown_connections 2024-09-15T23:28:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.183+0000 7f00c82d4740 1 -- 172.21.15.62:0/271660866 wait complete. 2024-09-15T23:28:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.183+0000 7f00c82d4740 1 -- start start 2024-09-15T23:28:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.184+0000 7f00c82d4740 1 Processor -- start 2024-09-15T23:28:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.184+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.184+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.184+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 0x5625f94bc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.184+0000 7f00c82d4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5625f94b3330 con 0x5625f94aa9a0 2024-09-15T23:28:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.184+0000 7f00c82d4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5625f9397290 con 0x5625f94b0370 2024-09-15T23:28:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.184+0000 7f00c82d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5625f94b34b0 con 0x5625f94ab2e0 2024-09-15T23:28:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.184+0000 7f00c82d4740 10 client.0 _ll_register_callbacks cb 0x5625f93c8140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00beffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00beffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94b4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50818/0 (socket says 172.21.15.62:50818) 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00beffd640 1 -- 172.21.15.62:0/288665463 learned_addr learned my addr 172.21.15.62:0/288665463 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00bf7fe640 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 0x5625f94bc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00be7fc640 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00bf7fe640 1 -- 172.21.15.62:0/288665463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 msgr2=0x5625f94b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00bf7fe640 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00bf7fe640 1 -- 172.21.15.62:0/288665463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 msgr2=0x5625f94b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00bf7fe640 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.185+0000 7f00bf7fe640 1 -- 172.21.15.62:0/288665463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625f9121210 con 0x5625f94b0370 2024-09-15T23:28:42.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.186+0000 7f00bf7fe640 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 0x5625f94bc130 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f00b000c5e0 tx=0x7f00b000caa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8307bbfc55709cca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.189+0000 7f00a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:28:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.189+0000 7f00a2ff5640 1 -- 172.21.15.62:0/288665463 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b0015070 con 0x5625f94b0370 2024-09-15T23:28:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.190+0000 7f00a2ff5640 1 -- 172.21.15.62:0/288665463 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f00b0011610 con 0x5625f94b0370 2024-09-15T23:28:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.191+0000 7f00c82d4740 1 -- 172.21.15.62:0/288665463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00b4009de0 con 0x5625f94b0370 2024-09-15T23:28:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.192+0000 7f00a2ff5640 1 -- 172.21.15.62:0/288665463 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f00b001f050 con 0x5625f94b0370 2024-09-15T23:28:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.199+0000 7f00c82d4740 10 client.27869 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.199+0000 7f00c82d4740 1 -- 172.21.15.62:0/288665463 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5625f91ed620 con 0x5625f94b0370 2024-09-15T23:28:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.200+0000 7f00a2ff5640 1 -- 172.21.15.62:0/288665463 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b4009de0 con 0x5625f94b0370 2024-09-15T23:28:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.200+0000 7f00a2ff5640 1 -- 172.21.15.62:0/288665463 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f00b0010050 con 0x5625f94b0370 2024-09-15T23:28:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.200+0000 7f00c82d4740 10 client.27869 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.200+0000 7f00c82d4740 10 client.27869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.200+0000 7f00c82d4740 20 client.27869 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.200+0000 7f00c82d4740 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5625f94c0cc0 0x5625f94c3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.200+0000 7f00c82d4740 4 client.27869 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.200+0000 7f00c82d4740 1 -- 172.21.15.62:0/288665463 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5625f941ba20 con 0x5625f94c0cc0 2024-09-15T23:28:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.202+0000 7f00beffd640 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5625f94c0cc0 0x5625f94c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.212+0000 7f00beffd640 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5625f94c0cc0 0x5625f94c3140 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76419651bee32b40 server_cookie=5051d8bc999274f9 in_seq=0 out_seq=0 2024-09-15T23:28:42.214 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.213+0000 7f00a2ff5640 10 client.27869 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.217+0000 7f00a2ff5640 1 -- 172.21.15.62:0/288665463 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5625f941ba20 con 0x5625f94c0cc0 2024-09-15T23:28:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.217+0000 7f00a2ff5640 10 client.27869 handle_command_reply: tid=0 2024-09-15T23:28:42.218 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.217+0000 7f00c82d4740 1 client.27869 shutdown 2024-09-15T23:28:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.221+0000 7f00c82d4740 1 -- 172.21.15.62:0/288665463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 msgr2=0x5625f94bc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.221+0000 7f00c82d4740 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 0x5625f94bc130 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f00b000c5e0 tx=0x7f00b000caa0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.223+0000 7f00c82d4740 1 -- 172.21.15.62:0/288665463 shutdown_connections 2024-09-15T23:28:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.223+0000 7f00c82d4740 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5625f94c0cc0 0x5625f94c3140 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.223+0000 7f00c82d4740 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 0x5625f94bc130 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.223+0000 7f00c82d4740 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.223+0000 7f00c82d4740 1 --2- 172.21.15.62:0/288665463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.223+0000 7f00c82d4740 1 -- 172.21.15.62:0/288665463 >> 172.21.15.62:0/288665463 conn(0x5625f94a6010 msgr2=0x5625f94a7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.223+0000 7f00a2ff5640 0 client.27869 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.230+0000 7f00c82d4740 1 -- 172.21.15.62:0/288665463 shutdown_connections 2024-09-15T23:28:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.230+0000 7f00c82d4740 1 -- 172.21.15.62:0/288665463 wait complete. 2024-09-15T23:28:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.231+0000 7f00c82d4740 20 client.27869 trim_cache size 0 max 16384 2024-09-15T23:28:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.270+0000 7f00c82d4740 1 Processor -- start 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00c82d4740 1 -- start start 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94afa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 0x5625f94bc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 0x5625f94b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00c82d4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac012570 con 0x5625f94aa9a0 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00c82d4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac00fe40 con 0x5625f94ab2e0 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00c82d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac00dc90 con 0x5625f94b0370 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00bf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 0x5625f94b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00bf7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 0x5625f94b4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53480/0 (socket says 172.21.15.62:53480) 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.271+0000 7f00bf7fe640 1 -- 172.21.15.62:0/4147096680 learned_addr learned my addr 172.21.15.62:0/4147096680 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.272+0000 7f00beffd640 1 --2- 172.21.15.62:0/4147096680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.272+0000 7f00beffd640 1 -- 172.21.15.62:0/4147096680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 msgr2=0x5625f94b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.272+0000 7f00beffd640 1 --2- 172.21.15.62:0/4147096680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 0x5625f94b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.272+0000 7f00beffd640 1 -- 172.21.15.62:0/4147096680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 msgr2=0x5625f94bc0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:28:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.272+0000 7f00beffd640 1 --2- 172.21.15.62:0/4147096680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 0x5625f94bc0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.272+0000 7f00beffd640 1 -- 172.21.15.62:0/4147096680 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625f94ab720 con 0x5625f94aa9a0 2024-09-15T23:28:42.274 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.273+0000 7f00beffd640 1 --2- 172.21.15.62:0/4147096680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94afa50 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f00b40d5040 tx=0x7f00b4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db211513455c8366 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.274+0000 7f00a4ff9640 1 -- 172.21.15.62:0/4147096680 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4020040 con 0x5625f94aa9a0 2024-09-15T23:28:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.274+0000 7f00a4ff9640 1 -- 172.21.15.62:0/4147096680 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b40d5380 con 0x5625f94aa9a0 2024-09-15T23:28:42.275 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.274+0000 7f00a4ff9640 1 -- 172.21.15.62:0/4147096680 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b40109d0 con 0x5625f94aa9a0 2024-09-15T23:28:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.275+0000 7f00c82d4740 1 -- 172.21.15.62:0/4147096680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 msgr2=0x5625f94afa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.275+0000 7f00c82d4740 1 --2- 172.21.15.62:0/4147096680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94afa50 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f00b40d5040 tx=0x7f00b4000bc0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.279+0000 7f00c82d4740 1 -- 172.21.15.62:0/4147096680 shutdown_connections 2024-09-15T23:28:42.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.279+0000 7f00c82d4740 1 --2- 172.21.15.62:0/4147096680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94b0370 0x5625f94b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.279+0000 7f00c82d4740 1 --2- 172.21.15.62:0/4147096680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94ab2e0 0x5625f94bc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.279+0000 7f00c82d4740 1 --2- 172.21.15.62:0/4147096680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94afa50 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.280 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.279+0000 7f00c82d4740 1 -- 172.21.15.62:0/4147096680 >> 172.21.15.62:0/4147096680 conn(0x5625f94a6010 msgr2=0x5625f94ac3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:42.283 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.282+0000 7f00c82d4740 1 -- 172.21.15.62:0/4147096680 shutdown_connections 2024-09-15T23:28:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.283+0000 7f00c82d4740 1 -- 172.21.15.62:0/4147096680 wait complete. 2024-09-15T23:28:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.283+0000 7f00c82d4740 1 -- start start 2024-09-15T23:28:42.284 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.283+0000 7f00c82d4740 1 Processor -- start 2024-09-15T23:28:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.284+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94bc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.285 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.284+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.284+0000 7f00c82d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 0x5625f94b9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.285+0000 7f00be7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.285+0000 7f00be7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:53488/0 (socket says 172.21.15.62:53488) 2024-09-15T23:28:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.285+0000 7f00be7fc640 1 -- 172.21.15.62:0/478117631 learned_addr learned my addr 172.21.15.62:0/478117631 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.285+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac00e300 con 0x5625f94aa9a0 2024-09-15T23:28:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.285+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac00f770 con 0x5625f94b0370 2024-09-15T23:28:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.285+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac00e9d0 con 0x5625f94ab2e0 2024-09-15T23:28:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.285+0000 7f00c82d4740 10 client.0 _ll_register_callbacks cb 0x7f00b00073e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.289+0000 7f00beffd640 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94bc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.289+0000 7f00bf7fe640 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 0x5625f94b9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.289+0000 7f00beffd640 1 -- 172.21.15.62:0/478117631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 msgr2=0x5625f94b8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.289+0000 7f00beffd640 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.291 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.289+0000 7f00beffd640 1 -- 172.21.15.62:0/478117631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 msgr2=0x5625f94b9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.289+0000 7f00beffd640 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 0x5625f94b9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.289+0000 7f00beffd640 1 -- 172.21.15.62:0/478117631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5625f9529a40 con 0x5625f94aa9a0 2024-09-15T23:28:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.290+0000 7f00beffd640 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94bc0c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x5625f94bd6c0 tx=0x7f00b4009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5d50d19f58a5a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:42.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.290+0000 7f00be7fc640 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.293+0000 7f00a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:42.294 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.293+0000 7f00a2ff5640 1 -- 172.21.15.62:0/478117631 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4020070 con 0x5625f94aa9a0 2024-09-15T23:28:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.293+0000 7f00a2ff5640 1 -- 172.21.15.62:0/478117631 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f00b40ba2b0 con 0x5625f94aa9a0 2024-09-15T23:28:42.295 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.294+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00b40d5380 con 0x5625f94aa9a0 2024-09-15T23:28:42.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.299+0000 7f00a2ff5640 1 -- 172.21.15.62:0/478117631 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f00b40a9050 con 0x5625f94aa9a0 2024-09-15T23:28:42.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.300+0000 7f00c82d4740 10 client.18324 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:42.301 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.300+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5625f952ad50 con 0x5625f94aa9a0 2024-09-15T23:28:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.301+0000 7f00a2ff5640 1 -- 172.21.15.62:0/478117631 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00b40d5380 con 0x5625f94aa9a0 2024-09-15T23:28:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.301+0000 7f00a2ff5640 1 -- 172.21.15.62:0/478117631 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f00b40b4bd0 con 0x5625f94aa9a0 2024-09-15T23:28:42.302 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.301+0000 7f00c82d4740 10 client.18324 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.301+0000 7f00c82d4740 10 client.18324 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.302+0000 7f00c82d4740 20 client.18324 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.302+0000 7f00c82d4740 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5625f952b140 0x5625f952d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.302+0000 7f00c82d4740 4 client.18324 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:28:42.303 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.302+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625f94b0130 con 0x5625f952b140 2024-09-15T23:28:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.304+0000 7f00be7fc640 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5625f952b140 0x5625f952d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.305+0000 7f00be7fc640 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5625f952b140 0x5625f952d560 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f253431e276d8b5f server_cookie=2fdedb8260bfefa0 in_seq=0 out_seq=0 2024-09-15T23:28:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.306+0000 7f00a2ff5640 10 client.18324 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.388+0000 7f00a2ff5640 1 -- 172.21.15.62:0/478117631 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5625f94b0130 con 0x5625f952b140 2024-09-15T23:28:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.388+0000 7f00a2ff5640 10 client.18324 handle_command_reply: tid=0 2024-09-15T23:28:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.388+0000 7f00c82d4740 1 client.18324 shutdown 2024-09-15T23:28:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.395+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 msgr2=0x5625f94bc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.395+0000 7f00c82d4740 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94bc0c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x5625f94bd6c0 tx=0x7f00b4009740 comp rx=0 tx=0).stop 2024-09-15T23:28:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.400+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 shutdown_connections 2024-09-15T23:28:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.400+0000 7f00c82d4740 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5625f952b140 0x5625f952d560 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.400+0000 7f00c82d4740 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5625f94b0370 0x5625f94b9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.400+0000 7f00c82d4740 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5625f94ab2e0 0x5625f94b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.400+0000 7f00c82d4740 1 --2- 172.21.15.62:0/478117631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5625f94aa9a0 0x5625f94bc0c0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.400+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 >> 172.21.15.62:0/478117631 conn(0x5625f94a6010 msgr2=0x5625f94ac3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.401+0000 7f00a2ff5640 0 client.18324 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:42.406 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.406+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 shutdown_connections 2024-09-15T23:28:42.408 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.406+0000 7f00c82d4740 1 -- 172.21.15.62:0/478117631 wait complete. 2024-09-15T23:28:42.409 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.407+0000 7f00c82d4740 20 client.18324 trim_cache size 0 max 16384 2024-09-15T23:28:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:28:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:28:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:28:42.412 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:28:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.416+0000 7f00c553a640 1 -- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a80a3520 msgr2=0x7f00a80a59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.416+0000 7f00c553a640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a80a3520 0x7f00a80a59e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f00b80315a0 tx=0x7f00b8000bc0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.416+0000 7f00c553a640 1 -- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00c0108470 msgr2=0x7f00c0073750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.416+0000 7f00c553a640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00c0108470 0x7f00c0073750 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f00b400d950 tx=0x7f00b400de10 comp rx=0 tx=0).stop 2024-09-15T23:28:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.424+0000 7f00c553a640 1 -- 172.21.15.62:0/3699519320 shutdown_connections 2024-09-15T23:28:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.424+0000 7f00c553a640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00a80a3520 0x7f00a80a59e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.424+0000 7f00c553a640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00c0109d40 0x7f00c0071d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.424+0000 7f00c553a640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00c0109380 0x7f00c0073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.424+0000 7f00c553a640 1 --2- 172.21.15.62:0/3699519320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00c0108470 0x7f00c0073750 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.424+0000 7f00c553a640 1 -- 172.21.15.62:0/3699519320 >> 172.21.15.62:0/3699519320 conn(0x7f00c0077490 msgr2=0x7f00c010c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:42.434 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.433+0000 7f00c553a640 1 -- 172.21.15.62:0/3699519320 shutdown_connections 2024-09-15T23:28:42.434 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:42.433+0000 7f00c553a640 1 -- 172.21.15.62:0/3699519320 wait complete. 2024-09-15T23:28:42.776 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:43.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:43 smithi062 ceph-mon[30949]: pgmap v743: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.7 MiB/s wr, 622 op/s 2024-09-15T23:28:43.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.340+0000 7f9e8147d640 1 -- 172.21.15.62:0/3341340631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c100430 msgr2=0x7f9e7c100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:43.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.340+0000 7f9e8147d640 1 --2- 172.21.15.62:0/3341340631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c100430 0x7f9e7c100810 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009860 tx=0x7f9e6c02a340 comp rx=0 tx=0).stop 2024-09-15T23:28:43.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.341+0000 7f9e8147d640 1 -- 172.21.15.62:0/3341340631 shutdown_connections 2024-09-15T23:28:43.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.341+0000 7f9e8147d640 1 --2- 172.21.15.62:0/3341340631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e7c105560 0x7f9e7c10d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.341+0000 7f9e8147d640 1 --2- 172.21.15.62:0/3341340631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7c100de0 0x7f9e7c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.341+0000 7f9e8147d640 1 --2- 172.21.15.62:0/3341340631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c100430 0x7f9e7c100810 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.341+0000 7f9e8147d640 1 -- 172.21.15.62:0/3341340631 >> 172.21.15.62:0/3341340631 conn(0x7f9e7c077230 msgr2=0x7f9e7c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:43.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.341+0000 7f9e8147d640 1 -- 172.21.15.62:0/3341340631 shutdown_connections 2024-09-15T23:28:43.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.341+0000 7f9e8147d640 1 -- 172.21.15.62:0/3341340631 wait complete. 2024-09-15T23:28:43.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.342+0000 7f9e8147d640 1 Processor -- start 2024-09-15T23:28:43.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.342+0000 7f9e8147d640 1 -- start start 2024-09-15T23:28:43.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e8147d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e7c100430 0x7f9e7c196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e8147d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7c100de0 0x7f9e7c196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e8147d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c105560 0x7f9e7c19aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e8147d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c10fd60 con 0x7f9e7c100de0 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e8147d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c10fbe0 con 0x7f9e7c105560 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e8147d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e7c10fee0 con 0x7f9e7c100430 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e7a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7c100de0 0x7f9e7c196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e7a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7c100de0 0x7f9e7c196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37590/0 (socket says 172.21.15.62:37590) 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e7a7fc640 1 -- 172.21.15.62:0/1585813777 learned_addr learned my addr 172.21.15.62:0/1585813777 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e7affd640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e7c100430 0x7f9e7c196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e7b7fe640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c105560 0x7f9e7c19aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e7a7fc640 1 -- 172.21.15.62:0/1585813777 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e7c100430 msgr2=0x7f9e7c196140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.343+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e7c100430 0x7f9e7c196140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e7a7fc640 1 -- 172.21.15.62:0/1585813777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c105560 msgr2=0x7f9e7c19aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c105560 0x7f9e7c19aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e7a7fc640 1 -- 172.21.15.62:0/1585813777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e7c19b0f0 con 0x7f9e7c100de0 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e7affd640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e7c100430 0x7f9e7c196140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7c100de0 0x7f9e7c196680 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000cdb0 tx=0x7f9e7000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e63fff640 1 -- 172.21.15.62:0/1585813777 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70013e10 con 0x7f9e7c100de0 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e8147d640 1 -- 172.21.15.62:0/1585813777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e7c19b3e0 con 0x7f9e7c100de0 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e8147d640 1 -- 172.21.15.62:0/1585813777 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e7c1a2cc0 con 0x7f9e7c100de0 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e63fff640 1 -- 172.21.15.62:0/1585813777 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e70017070 con 0x7f9e7c100de0 2024-09-15T23:28:43.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.344+0000 7f9e63fff640 1 -- 172.21.15.62:0/1585813777 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e70003d10 con 0x7f9e7c100de0 2024-09-15T23:28:43.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e63fff640 1 -- 172.21.15.62:0/1585813777 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9e70019030 con 0x7f9e7c100de0 2024-09-15T23:28:43.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e83856740 1 Processor -- start 2024-09-15T23:28:43.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e83856740 1 -- start start 2024-09-15T23:28:43.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e83856740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efdff090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e83856740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 0x5588efdffa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e83856740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe0c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e83856740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5588efd652f0 con 0x5588efdff5d0 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e83856740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5588efd0ccf0 con 0x5588efe00b40 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e83856740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5588efd612b0 con 0x5588efe0a370 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e7b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efdff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e7affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 0x5588efdffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e7b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efdff090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49740/0 (socket says 172.21.15.62:49740) 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.347+0000 7f9e7b7fe640 1 -- 172.21.15.62:0/4031953273 learned_addr learned my addr 172.21.15.62:0/4031953273 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.348+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/4031953273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe0c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.348+0000 7f9e7affd640 1 -- 172.21.15.62:0/4031953273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 msgr2=0x5588efe0c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.348+0000 7f9e7affd640 1 --2- 172.21.15.62:0/4031953273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe0c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.348+0000 7f9e7affd640 1 -- 172.21.15.62:0/4031953273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 msgr2=0x5588efdff090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.348+0000 7f9e7affd640 1 --2- 172.21.15.62:0/4031953273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efdff090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.348+0000 7f9e7affd640 1 -- 172.21.15.62:0/4031953273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588efcf13a0 con 0x5588efdff5d0 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.348+0000 7f9e7b7fe640 1 --2- 172.21.15.62:0/4031953273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efdff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.349+0000 7f9e7affd640 1 --2- 172.21.15.62:0/4031953273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 0x5588efdffa10 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009830 tx=0x7f9e6c005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645d04b29c4c624a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.349+0000 7f9e617fa640 1 -- 172.21.15.62:0/4031953273 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c037070 con 0x5588efdff5d0 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.349+0000 7f9e63fff640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e640a3300 0x7f9e640a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.349+0000 7f9e617fa640 1 -- 172.21.15.62:0/4031953273 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e6c038040 con 0x5588efdff5d0 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.349+0000 7f9e617fa640 1 -- 172.21.15.62:0/4031953273 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c003850 con 0x5588efdff5d0 2024-09-15T23:28:43.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.349+0000 7f9e63fff640 1 -- 172.21.15.62:0/1585813777 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e700d43c0 con 0x7f9e7c100de0 2024-09-15T23:28:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.350+0000 7f9e7b7fe640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e640a3300 0x7f9e640a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.350+0000 7f9e83856740 1 -- 172.21.15.62:0/4031953273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 msgr2=0x5588efdffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:43.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.351+0000 7f9e83856740 1 --2- 172.21.15.62:0/4031953273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 0x5588efdffa10 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009830 tx=0x7f9e6c005fc0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.351+0000 7f9e83856740 1 -- 172.21.15.62:0/4031953273 shutdown_connections 2024-09-15T23:28:43.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.351+0000 7f9e83856740 1 --2- 172.21.15.62:0/4031953273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe0c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.351+0000 7f9e83856740 1 --2- 172.21.15.62:0/4031953273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 0x5588efdffa10 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.351+0000 7f9e83856740 1 --2- 172.21.15.62:0/4031953273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efdff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.351+0000 7f9e83856740 1 -- 172.21.15.62:0/4031953273 >> 172.21.15.62:0/4031953273 conn(0x5588efdc8310 msgr2=0x5588efdfbc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:43.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.351+0000 7f9e83856740 1 -- 172.21.15.62:0/4031953273 shutdown_connections 2024-09-15T23:28:43.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.351+0000 7f9e83856740 1 -- 172.21.15.62:0/4031953273 wait complete. 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.351+0000 7f9e7b7fe640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e640a3300 0x7f9e640a57c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009fd0 tx=0x7f9e74009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e83856740 1 -- start start 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e83856740 1 Processor -- start 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e83856740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 0x5588efe09d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e83856740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efe07010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e83856740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe07600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e83856740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5588efe15820 con 0x5588efdff5d0 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e83856740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5588efd20060 con 0x5588efe00b40 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e83856740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5588efe159a0 con 0x5588efe0a370 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e83856740 10 client.0 _ll_register_callbacks cb 0x5588efa738c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.352+0000 7f9e7b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe07600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7b7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe07600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:58370/0 (socket says 172.21.15.62:58370) 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7b7fe640 1 -- 172.21.15.62:0/4076058140 learned_addr learned my addr 172.21.15.62:0/4076058140 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:43.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efe07010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7affd640 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 0x5588efe09d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7b7fe640 1 -- 172.21.15.62:0/4076058140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 msgr2=0x5588efe07010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7b7fe640 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efe07010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7b7fe640 1 -- 172.21.15.62:0/4076058140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 msgr2=0x5588efe09d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7b7fe640 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 0x5588efe09d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7b7fe640 1 -- 172.21.15.62:0/4076058140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588efad63c0 con 0x5588efe0a370 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e7b7fe640 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe07600 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400fb60 tx=0x7f9e74004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73a8890bacda3eb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e5eff5640 1 -- 172.21.15.62:0/4076058140 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7400fd10 con 0x5588efe0a370 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e5eff5640 1 -- 172.21.15.62:0/4076058140 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e74017030 con 0x5588efe0a370 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.353+0000 7f9e83856740 1 -- 172.21.15.62:0/4076058140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e6c038040 con 0x5588efe0a370 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e5eff5640 1 -- 172.21.15.62:0/4076058140 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9e74033050 con 0x5588efe0a370 2024-09-15T23:28:43.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e83856740 10 client.27604 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e83856740 1 -- 172.21.15.62:0/4076058140 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5588efe212f0 con 0x5588efe0a370 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e5eff5640 1 -- 172.21.15.62:0/4076058140 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e6c038040 con 0x5588efe0a370 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e5eff5640 1 -- 172.21.15.62:0/4076058140 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f9e74004030 con 0x5588efe0a370 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e83856740 10 client.27604 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e83856740 10 client.27604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e83856740 20 client.27604 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e83856740 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588efe22800 0x5588efe24ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e83856740 4 client.27604 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e83856740 1 -- 172.21.15.62:0/4076058140 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588efd72860 con 0x5588efe22800 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.354+0000 7f9e7affd640 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588efe22800 0x5588efe24ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.355+0000 7f9e7affd640 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588efe22800 0x5588efe24ca0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b3b6174420b33f3 server_cookie=2e90c25b894db403 in_seq=0 out_seq=0 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.355+0000 7f9e5eff5640 10 client.27604 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:43.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.355+0000 7f9e5eff5640 1 -- 172.21.15.62:0/4076058140 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5588efd72860 con 0x5588efe22800 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.355+0000 7f9e5eff5640 10 client.27604 handle_command_reply: tid=0 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 client.27604 shutdown 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 -- 172.21.15.62:0/4076058140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 msgr2=0x5588efe07600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe07600 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400fb60 tx=0x7f9e74004ab0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 -- 172.21.15.62:0/4076058140 shutdown_connections 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588efe22800 0x5588efe24ca0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe07600 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efe00b40 0x5588efe07010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 --2- 172.21.15.62:0/4076058140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efdff5d0 0x5588efe09d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 -- 172.21.15.62:0/4076058140 >> 172.21.15.62:0/4076058140 conn(0x5588efdc8310 msgr2=0x5588efe01930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e5eff5640 0 client.27604 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 -- 172.21.15.62:0/4076058140 shutdown_connections 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.356+0000 7f9e83856740 1 -- 172.21.15.62:0/4076058140 wait complete. 2024-09-15T23:28:43.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.357+0000 7f9e83856740 20 client.27604 trim_cache size 0 max 16384 2024-09-15T23:28:43.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.374+0000 7f9e83856740 1 Processor -- start 2024-09-15T23:28:43.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.374+0000 7f9e83856740 1 -- start start 2024-09-15T23:28:43.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.374+0000 7f9e83856740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e83856740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e83856740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe0b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37608/0 (socket says 172.21.15.62:37608) 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7a7fc640 1 -- 172.21.15.62:0/2976365173 learned_addr learned my addr 172.21.15.62:0/2976365173 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e83856740 1 -- 172.21.15.62:0/2976365173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e680123d0 con 0x5588efe00b40 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e83856740 1 -- 172.21.15.62:0/2976365173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6800fca0 con 0x5588efdff5d0 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e83856740 1 -- 172.21.15.62:0/2976365173 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6800daf0 con 0x5588efe0a370 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7affd640 1 --2- 172.21.15.62:0/2976365173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7a7fc640 1 -- 172.21.15.62:0/2976365173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 msgr2=0x5588efe0b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/2976365173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe0b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7a7fc640 1 -- 172.21.15.62:0/2976365173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 msgr2=0x5588efe1eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:43.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/2976365173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7a7fc640 1 -- 172.21.15.62:0/2976365173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588efe00760 con 0x5588efe00b40 2024-09-15T23:28:43.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.375+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/2976365173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe068d0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9e7009ca60 tx=0x7f9e7000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79b85de28785758 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:43.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.376+0000 7f9e60ff9640 1 -- 172.21.15.62:0/2976365173 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7009ff60 con 0x5588efe00b40 2024-09-15T23:28:43.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.376+0000 7f9e60ff9640 1 -- 172.21.15.62:0/2976365173 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e700d41b0 con 0x5588efe00b40 2024-09-15T23:28:43.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.376+0000 7f9e60ff9640 1 -- 172.21.15.62:0/2976365173 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e700a9270 con 0x5588efe00b40 2024-09-15T23:28:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.376+0000 7f9e83856740 1 -- 172.21.15.62:0/2976365173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 msgr2=0x5588efe068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 --2- 172.21.15.62:0/2976365173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe068d0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f9e7009ca60 tx=0x7f9e7000cf60 comp rx=0 tx=0).stop 2024-09-15T23:28:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 -- 172.21.15.62:0/2976365173 shutdown_connections 2024-09-15T23:28:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 --2- 172.21.15.62:0/2976365173 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe0b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 --2- 172.21.15.62:0/2976365173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe068d0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 --2- 172.21.15.62:0/2976365173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 -- 172.21.15.62:0/2976365173 >> 172.21.15.62:0/2976365173 conn(0x5588efdc8310 msgr2=0x5588efe058e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 -- 172.21.15.62:0/2976365173 shutdown_connections 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 -- 172.21.15.62:0/2976365173 wait complete. 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 -- start start 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.377+0000 7f9e83856740 1 Processor -- start 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e83856740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e83856740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe29cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e83856740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe2a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e83856740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6800e160 con 0x5588efe00b40 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e83856740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6800f5d0 con 0x5588efdff5d0 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e83856740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6800e830 con 0x5588efe0a370 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e83856740 10 client.0 _ll_register_callbacks cb 0x7f9e74009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe29cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49768/0 (socket says 172.21.15.62:49768) 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7affd640 1 -- 172.21.15.62:0/1467737729 learned_addr learned my addr 172.21.15.62:0/1467737729 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7b7fe640 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe2a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7affd640 1 -- 172.21.15.62:0/1467737729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 msgr2=0x5588efe2a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7affd640 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe2a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7affd640 1 -- 172.21.15.62:0/1467737729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 msgr2=0x5588efe29cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7affd640 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe29cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7affd640 1 -- 172.21.15.62:0/1467737729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588efe2a7f0 con 0x5588efdff5d0 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.378+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe29cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.379+0000 7f9e7affd640 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1b1f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009860 tx=0x7f9e6c004420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a231a329c8da1a0e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:43.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.379+0000 7f9e5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.379+0000 7f9e5eff5640 1 -- 172.21.15.62:0/1467737729 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c037040 con 0x5588efdff5d0 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.379+0000 7f9e5eff5640 1 -- 172.21.15.62:0/1467737729 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9e6c030750 con 0x5588efdff5d0 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.379+0000 7f9e83856740 1 -- 172.21.15.62:0/1467737729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e700d41b0 con 0x5588efdff5d0 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.379+0000 7f9e5eff5640 1 -- 172.21.15.62:0/1467737729 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9e6c05d050 con 0x5588efdff5d0 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.379+0000 7f9e83856740 10 client.27890 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.379+0000 7f9e83856740 1 -- 172.21.15.62:0/1467737729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5588efe19ac0 con 0x5588efdff5d0 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.380+0000 7f9e5eff5640 1 -- 172.21.15.62:0/1467737729 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e6c054070 con 0x5588efdff5d0 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.380+0000 7f9e5eff5640 1 -- 172.21.15.62:0/1467737729 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f9e6c04f030 con 0x5588efdff5d0 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.380+0000 7f9e83856740 10 client.27890 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.380+0000 7f9e83856740 10 client.27890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.380+0000 7f9e83856740 20 client.27890 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.380+0000 7f9e83856740 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588efe8d710 0x5588efe8fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:43.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.380+0000 7f9e83856740 4 client.27890 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:28:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.380+0000 7f9e83856740 1 -- 172.21.15.62:0/1467737729 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f9e700d41b0 con 0x5588efe8d710 2024-09-15T23:28:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.380+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588efe8d710 0x5588efe8fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.381+0000 7f9e5eff5640 10 client.27890 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.381+0000 7f9e7a7fc640 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588efe8d710 0x5588efe8fb00 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=730654587cff6950 server_cookie=fe99c9fdcad534b7 in_seq=0 out_seq=0 2024-09-15T23:28:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.381+0000 7f9e5eff5640 1 -- 172.21.15.62:0/1467737729 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f9e700d41b0 con 0x5588efe8d710 2024-09-15T23:28:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.381+0000 7f9e5eff5640 10 client.27890 handle_command_reply: tid=0 2024-09-15T23:28:43.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.381+0000 7f9e83856740 1 client.27890 shutdown 2024-09-15T23:28:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.381+0000 7f9e83856740 1 -- 172.21.15.62:0/1467737729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 msgr2=0x5588efe1b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.381+0000 7f9e83856740 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1b1f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009860 tx=0x7f9e6c004420 comp rx=0 tx=0).stop 2024-09-15T23:28:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.382+0000 7f9e83856740 1 -- 172.21.15.62:0/1467737729 shutdown_connections 2024-09-15T23:28:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.382+0000 7f9e83856740 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5588efe8d710 0x5588efe8fb00 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.382+0000 7f9e83856740 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5588efe0a370 0x5588efe2a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.382+0000 7f9e83856740 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588efe00b40 0x5588efe29cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.382+0000 7f9e83856740 1 --2- 172.21.15.62:0/1467737729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5588efdff5d0 0x5588efe1b1f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.382+0000 7f9e83856740 1 -- 172.21.15.62:0/1467737729 >> 172.21.15.62:0/1467737729 conn(0x5588efdc8310 msgr2=0x5588efe05860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:43.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.386+0000 7f9e5eff5640 0 client.27890 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:43.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.386+0000 7f9e83856740 1 -- 172.21.15.62:0/1467737729 shutdown_connections 2024-09-15T23:28:43.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.386+0000 7f9e83856740 1 -- 172.21.15.62:0/1467737729 wait complete. 2024-09-15T23:28:43.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.387+0000 7f9e83856740 20 client.27890 trim_cache size 0 max 16384 2024-09-15T23:28:43.393 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:28:43.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.400+0000 7f9e8147d640 1 -- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e640a3300 msgr2=0x7f9e640a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:43.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.400+0000 7f9e8147d640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e640a3300 0x7f9e640a57c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009fd0 tx=0x7f9e74009500 comp rx=0 tx=0).stop 2024-09-15T23:28:43.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.400+0000 7f9e8147d640 1 -- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7c100de0 msgr2=0x7f9e7c196680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:43.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.400+0000 7f9e8147d640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7c100de0 0x7f9e7c196680 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000cdb0 tx=0x7f9e7000b730 comp rx=0 tx=0).stop 2024-09-15T23:28:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.405+0000 7f9e8147d640 1 -- 172.21.15.62:0/1585813777 shutdown_connections 2024-09-15T23:28:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.405+0000 7f9e8147d640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9e640a3300 0x7f9e640a57c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.405+0000 7f9e8147d640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c105560 0x7f9e7c19aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.405+0000 7f9e8147d640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e7c100de0 0x7f9e7c196680 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.405+0000 7f9e8147d640 1 --2- 172.21.15.62:0/1585813777 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9e7c100430 0x7f9e7c196140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:43.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.405+0000 7f9e8147d640 1 -- 172.21.15.62:0/1585813777 >> 172.21.15.62:0/1585813777 conn(0x7f9e7c077230 msgr2=0x7f9e7c103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:43.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.406+0000 7f9e8147d640 1 -- 172.21.15.62:0/1585813777 shutdown_connections 2024-09-15T23:28:43.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:43.406+0000 7f9e8147d640 1 -- 172.21.15.62:0/1585813777 wait complete. 2024-09-15T23:28:43.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:43 smithi067 ceph-mon[35665]: pgmap v743: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.7 MiB/s wr, 622 op/s 2024-09-15T23:28:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:43 smithi148 ceph-mon[35610]: pgmap v743: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 6.7 MiB/s wr, 622 op/s 2024-09-15T23:28:44.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 745.0 s, 490.8 tps, lat 218.963 ms stddev 291.532 2024-09-15T23:28:45.483 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:45 smithi062 ceph-mon[30949]: pgmap v744: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 7.1 MiB/s wr, 687 op/s 2024-09-15T23:28:45.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:45 smithi067 ceph-mon[35665]: pgmap v744: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 7.1 MiB/s wr, 687 op/s 2024-09-15T23:28:45.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:45 smithi148 ceph-mon[35610]: pgmap v744: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 7.1 MiB/s wr, 687 op/s 2024-09-15T23:28:47.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:47 smithi062 ceph-mon[30949]: pgmap v745: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.7 MiB/s wr, 767 op/s 2024-09-15T23:28:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:47 smithi067 ceph-mon[35665]: pgmap v745: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.7 MiB/s wr, 767 op/s 2024-09-15T23:28:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:47 smithi148 ceph-mon[35610]: pgmap v745: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 7.7 MiB/s wr, 767 op/s 2024-09-15T23:28:49.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:49 smithi062 ceph-mon[30949]: pgmap v746: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.0 MiB/s wr, 694 op/s 2024-09-15T23:28:49.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:49 smithi067 ceph-mon[35665]: pgmap v746: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.0 MiB/s wr, 694 op/s 2024-09-15T23:28:49.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:49 smithi148 ceph-mon[35610]: pgmap v746: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 7.0 MiB/s wr, 694 op/s 2024-09-15T23:28:49.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 750.0 s, 619.4 tps, lat 161.924 ms stddev 197.732 2024-09-15T23:28:50.030 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:28:50.031 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:28:50.082 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 11.4 2024-09-15T23:28:50.083 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:28:50.590 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:50.618 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:51.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:51 smithi062 ceph-mon[30949]: pgmap v747: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.4 MiB/s wr, 741 op/s 2024-09-15T23:28:51.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:51 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:28:51.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:51 smithi067 ceph-mon[35665]: pgmap v747: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.4 MiB/s wr, 741 op/s 2024-09-15T23:28:51.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:51 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:28:51.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:51 smithi148 ceph-mon[35610]: pgmap v747: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 7.4 MiB/s wr, 741 op/s 2024-09-15T23:28:51.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:51 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:28:52.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:52 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:28:52.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:52 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:52.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:52 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:28:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:52 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:28:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:52 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:52.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:52 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:52.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:52 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:28:52.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:52 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:52.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:52 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:28:52.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:52 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:28:52.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:52 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:52.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:52 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:52.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:52 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:28:52.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:52 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:52.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:52 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:28:52.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:52 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:28:52.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:52 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:52.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:52 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:52.806 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:28:52.806 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:28:52.806 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "f28c7295-f7ae-47d5-b6db-b9563604d77f", 2024-09-15T23:28:52.807 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:28:52.807 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:28:52.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.347+0000 7f4b659e9640 1 -- 172.21.15.62:0/524563620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60102ef0 msgr2=0x7f4b6010da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.347+0000 7f4b659e9640 1 --2- 172.21.15.62:0/524563620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60102ef0 0x7f4b6010da40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009ca0 tx=0x7f4b5802a950 comp rx=0 tx=0).stop 2024-09-15T23:28:52.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.347+0000 7f4b659e9640 1 -- 172.21.15.62:0/524563620 shutdown_connections 2024-09-15T23:28:52.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.347+0000 7f4b659e9640 1 --2- 172.21.15.62:0/524563620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60102ef0 0x7f4b6010da40 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.347+0000 7f4b659e9640 1 --2- 172.21.15.62:0/524563620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b60102570 0x7f4b601029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.347+0000 7f4b659e9640 1 --2- 172.21.15.62:0/524563620 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b60100e20 0x7f4b60101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.347+0000 7f4b659e9640 1 -- 172.21.15.62:0/524563620 >> 172.21.15.62:0/524563620 conn(0x7f4b600fc7d0 msgr2=0x7f4b600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:52.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.347+0000 7f4b659e9640 1 -- 172.21.15.62:0/524563620 shutdown_connections 2024-09-15T23:28:52.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.347+0000 7f4b659e9640 1 -- 172.21.15.62:0/524563620 wait complete. 2024-09-15T23:28:52.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.348+0000 7f4b659e9640 1 Processor -- start 2024-09-15T23:28:52.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.348+0000 7f4b659e9640 1 -- start start 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.348+0000 7f4b659e9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60100e20 0x7f4b601984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.348+0000 7f4b659e9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b60102570 0x7f4b60198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.348+0000 7f4b659e9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b60102ef0 0x7f4b6019cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.348+0000 7f4b659e9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4b60110f80 con 0x7f4b60100e20 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.348+0000 7f4b659e9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b60110e00 con 0x7f4b60102ef0 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.348+0000 7f4b659e9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4b60111100 con 0x7f4b60102570 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.348+0000 7f4b5effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60100e20 0x7f4b601984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60100e20 0x7f4b601984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37640/0 (socket says 172.21.15.62:37640) 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5effd640 1 -- 172.21.15.62:0/3415964296 learned_addr learned my addr 172.21.15.62:0/3415964296 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:52.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b60102570 0x7f4b60198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b60102ef0 0x7f4b6019cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5effd640 1 -- 172.21.15.62:0/3415964296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b60102570 msgr2=0x7f4b60198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5effd640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b60102570 0x7f4b60198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5effd640 1 -- 172.21.15.62:0/3415964296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b60102ef0 msgr2=0x7f4b6019cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5effd640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b60102ef0 0x7f4b6019cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5effd640 1 -- 172.21.15.62:0/3415964296 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6019d480 con 0x7f4b60100e20 2024-09-15T23:28:52.810 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b5effd640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60100e20 0x7f4b601984d0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000e9f0 tx=0x7f4b5000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.349+0000 7f4b649e7640 1 -- 172.21.15.62:0/3415964296 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5000cd80 con 0x7f4b60100e20 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.350+0000 7f4b649e7640 1 -- 172.21.15.62:0/3415964296 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b50005500 con 0x7f4b60100e20 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.350+0000 7f4b649e7640 1 -- 172.21.15.62:0/3415964296 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b50010690 con 0x7f4b60100e20 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.350+0000 7f4b659e9640 1 -- 172.21.15.62:0/3415964296 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6019d770 con 0x7f4b60100e20 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.350+0000 7f4b659e9640 1 -- 172.21.15.62:0/3415964296 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b601a5050 con 0x7f4b60100e20 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.353+0000 7f4b649e7640 1 -- 172.21.15.62:0/3415964296 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4b500046c0 con 0x7f4b60100e20 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b67dc2740 1 Processor -- start 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b67dc2740 1 -- start start 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef720be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720c360 0x559ef721ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 0x559ef721d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559ef7171280 con 0x559ef720c360 2024-09-15T23:28:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559ef7118b10 con 0x559ef720ba40 2024-09-15T23:28:52.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559ef716d240 con 0x559ef721b150 2024-09-15T23:28:52.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b5effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720c360 0x559ef721ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b5f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef720be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b5f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef720be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49780/0 (socket says 172.21.15.62:49780) 2024-09-15T23:28:52.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.354+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/858569957 learned_addr learned my addr 172.21.15.62:0/858569957 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:52.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/858569957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 0x559ef721d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/858569957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 msgr2=0x559ef721d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:52.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/858569957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 0x559ef721d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/858569957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720c360 msgr2=0x559ef721ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/858569957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720c360 0x559ef721ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b649e7640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4b480a32c0 0x7f4b480a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/858569957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ef71251f0 con 0x559ef720ba40 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b5effd640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4b480a32c0 0x7f4b480a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/858569957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef720be20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4b58035000 tx=0x7f4b580093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d79160afb4966a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b649e7640 1 -- 172.21.15.62:0/3415964296 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4b500d0010 con 0x7f4b60100e20 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b45ffb640 1 -- 172.21.15.62:0/858569957 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58048070 con 0x559ef720ba40 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b45ffb640 1 -- 172.21.15.62:0/858569957 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b5802f930 con 0x559ef720ba40 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.355+0000 7f4b45ffb640 1 -- 172.21.15.62:0/858569957 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58034040 con 0x559ef720ba40 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b67dc2740 1 -- 172.21.15.62:0/858569957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 msgr2=0x559ef720be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/858569957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef720be20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4b58035000 tx=0x7f4b580093e0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b67dc2740 1 -- 172.21.15.62:0/858569957 shutdown_connections 2024-09-15T23:28:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/858569957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 0x559ef721d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/858569957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720c360 0x559ef721ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/858569957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef720be20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b67dc2740 1 -- 172.21.15.62:0/858569957 >> 172.21.15.62:0/858569957 conn(0x559ef72080f0 msgr2=0x559ef72084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b5effd640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4b480a32c0 0x7f4b480a5780 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4b500cdb00 tx=0x7f4b500d3af0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b67dc2740 1 -- 172.21.15.62:0/858569957 shutdown_connections 2024-09-15T23:28:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.356+0000 7f4b67dc2740 1 -- 172.21.15.62:0/858569957 wait complete. 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b67dc2740 1 -- start start 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b67dc2740 1 Processor -- start 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 0x559ef7218120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720c360 0x559ef7218660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 0x559ef7218ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559ef721dd20 con 0x559ef720ba40 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559ef712bf10 con 0x559ef720c360 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559ef721dea0 con 0x559ef721b150 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b67dc2740 10 client.0 _ll_register_callbacks cb 0x559ef70f4a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:52.815 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b5e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720c360 0x559ef7218660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b5f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 0x559ef7218120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b5e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720c360 0x559ef7218660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49788/0 (socket says 172.21.15.62:49788) 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.357+0000 7f4b5e7fc640 1 -- 172.21.15.62:0/3038009939 learned_addr learned my addr 172.21.15.62:0/3038009939 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 0x559ef7218ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b5e7fc640 1 -- 172.21.15.62:0/3038009939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 msgr2=0x559ef7218ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 0x559ef7218ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b5e7fc640 1 -- 172.21.15.62:0/3038009939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 msgr2=0x559ef7218120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 0x559ef7218120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b5e7fc640 1 -- 172.21.15.62:0/3038009939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ef6ee26a0 con 0x559ef720c360 2024-09-15T23:28:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720c360 0x559ef7218660 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4b54012700 tx=0x7f4b54012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce0a0a2fb220a410 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b3effd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:28:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b3effd640 1 -- 172.21.15.62:0/3038009939 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b54004050 con 0x559ef720c360 2024-09-15T23:28:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b67dc2740 1 -- 172.21.15.62:0/3038009939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b5802f930 con 0x559ef720c360 2024-09-15T23:28:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b3effd640 1 -- 172.21.15.62:0/3038009939 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4b540041f0 con 0x559ef720c360 2024-09-15T23:28:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b3effd640 1 -- 172.21.15.62:0/3038009939 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4b540043f0 con 0x559ef720c360 2024-09-15T23:28:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b67dc2740 10 client.27902 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.358+0000 7f4b67dc2740 1 -- 172.21.15.62:0/3038009939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559ef7229af0 con 0x559ef720c360 2024-09-15T23:28:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b3effd640 1 -- 172.21.15.62:0/3038009939 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b5802f930 con 0x559ef720c360 2024-09-15T23:28:52.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b3effd640 1 -- 172.21.15.62:0/3038009939 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f4b54003bf0 con 0x559ef720c360 2024-09-15T23:28:52.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b67dc2740 10 client.27902 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:52.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b67dc2740 10 client.27902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:52.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b67dc2740 20 client.27902 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:52.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559ef722afb0 0x559ef722d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b67dc2740 4 client.27902 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:52.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b67dc2740 1 -- 172.21.15.62:0/3038009939 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ef717e7f0 con 0x559ef722afb0 2024-09-15T23:28:52.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559ef722afb0 0x559ef722d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559ef722afb0 0x559ef722d3e0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9d778d07b1e6996 server_cookie=59d12d7900bd72b4 in_seq=0 out_seq=0 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.359+0000 7f4b3effd640 10 client.27902 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.360+0000 7f4b3effd640 1 -- 172.21.15.62:0/3038009939 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559ef717e7f0 con 0x559ef722afb0 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.360+0000 7f4b3effd640 10 client.27902 handle_command_reply: tid=0 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.360+0000 7f4b67dc2740 1 client.27902 shutdown 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.360+0000 7f4b67dc2740 1 -- 172.21.15.62:0/3038009939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720c360 msgr2=0x559ef7218660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.360+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720c360 0x559ef7218660 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4b54012700 tx=0x7f4b54012bc0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.360+0000 7f4b67dc2740 1 -- 172.21.15.62:0/3038009939 shutdown_connections 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.360+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559ef722afb0 0x559ef722d3e0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.360+0000 7f4b3effd640 0 client.27902 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.360+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef721b150 0x559ef7218ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.361+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720c360 0x559ef7218660 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.361+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/3038009939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 0x559ef7218120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.361+0000 7f4b67dc2740 1 -- 172.21.15.62:0/3038009939 >> 172.21.15.62:0/3038009939 conn(0x559ef72080f0 msgr2=0x559ef720de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:52.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.361+0000 7f4b67dc2740 1 -- 172.21.15.62:0/3038009939 shutdown_connections 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.361+0000 7f4b67dc2740 1 -- 172.21.15.62:0/3038009939 wait complete. 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.361+0000 7f4b67dc2740 20 client.27902 trim_cache size 0 max 16384 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.388+0000 7f4b67dc2740 1 Processor -- start 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.389+0000 7f4b67dc2740 1 -- start start 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.389+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 0x559ef7232cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.389+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 0x559ef72331f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.389+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef721b150 0x559ef7215380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601af1e0 con 0x559ef720ba40 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601acab0 con 0x559ef721b150 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601aa900 con 0x559ef720c360 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef721b150 0x559ef7215380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef721b150 0x559ef7215380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49790/0 (socket says 172.21.15.62:49790) 2024-09-15T23:28:52.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/2725498875 learned_addr learned my addr 172.21.15.62:0/2725498875 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 0x559ef72331f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 0x559ef7232cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/2725498875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 msgr2=0x559ef72331f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 0x559ef72331f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/2725498875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 msgr2=0x559ef7232cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 0x559ef7232cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.390+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/2725498875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ef7209050 con 0x559ef721b150 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.391+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef721b150 0x559ef7215380 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4b580099d0 tx=0x7f4b58003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8e7712cc1d336ce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.391+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 0x559ef72331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.392+0000 7f4b457fa640 1 -- 172.21.15.62:0/2725498875 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58048040 con 0x559ef721b150 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.392+0000 7f4b457fa640 1 -- 172.21.15.62:0/2725498875 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b58030440 con 0x559ef721b150 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.392+0000 7f4b457fa640 1 -- 172.21.15.62:0/2725498875 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58034040 con 0x559ef721b150 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.393+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2725498875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef721b150 msgr2=0x559ef7215380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.821 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.393+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef721b150 0x559ef7215380 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4b580099d0 tx=0x7f4b58003970 comp rx=0 tx=0).stop 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.394+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2725498875 shutdown_connections 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.394+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef721b150 0x559ef7215380 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.394+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 0x559ef72331f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.394+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2725498875 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef720ba40 0x559ef7232cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.394+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2725498875 >> 172.21.15.62:0/2725498875 conn(0x559ef72080f0 msgr2=0x559ef721a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.405+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2725498875 shutdown_connections 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.408+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2725498875 wait complete. 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.408+0000 7f4b67dc2740 1 -- start start 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.408+0000 7f4b67dc2740 1 Processor -- start 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef72123c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 0x559ef7223130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b67dc2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef721b150 0x559ef7223670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601aaf70 con 0x559ef721b150 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601ac3e0 con 0x559ef720ba40 2024-09-15T23:28:52.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b67dc2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4b601ab640 con 0x559ef720c360 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b67dc2740 10 client.0 _ll_register_callbacks cb 0x7f4b540097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b5f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef72123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b5f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef72123c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49806/0 (socket says 172.21.15.62:49806) 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/2213029946 learned_addr learned my addr 172.21.15.62:0/2213029946 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.409+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef721b150 0x559ef7223670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.410+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 0x559ef7223130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.410+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/2213029946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 msgr2=0x559ef7223130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.410+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 0x559ef7223130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.410+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/2213029946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef721b150 msgr2=0x559ef7223670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.410+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef721b150 0x559ef7223670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.410+0000 7f4b5f7fe640 1 -- 172.21.15.62:0/2213029946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ef720f710 con 0x559ef720ba40 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.414+0000 7f4b5f7fe640 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef72123c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009ca0 tx=0x7f4b5802a950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a552672cf4a090c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:52.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.414+0000 7f4b3effd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.414+0000 7f4b3effd640 1 -- 172.21.15.62:0/2213029946 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58034040 con 0x559ef720ba40 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.414+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2213029946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b58030440 con 0x559ef720ba40 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b3effd640 1 -- 172.21.15.62:0/2213029946 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4b580305f0 con 0x559ef720ba40 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b3effd640 1 -- 172.21.15.62:0/2213029946 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4b580307f0 con 0x559ef720ba40 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b67dc2740 10 client.27908 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2213029946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559ef7291e70 con 0x559ef720ba40 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b3effd640 1 -- 172.21.15.62:0/2213029946 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b58030440 con 0x559ef720ba40 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b3effd640 1 -- 172.21.15.62:0/2213029946 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f4b5803a9f0 con 0x559ef720ba40 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b67dc2740 10 client.27908 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b67dc2740 10 client.27908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b67dc2740 20 client.27908 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559ef722afb0 0x559ef7294490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b67dc2740 4 client.27908 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:28:52.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.415+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2213029946 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559ef7178d00 con 0x559ef722afb0 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.416+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559ef722afb0 0x559ef7294490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.418+0000 7f4b5e7fc640 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559ef722afb0 0x559ef7294490 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc949cb68f2ac71f server_cookie=d3a8f5b1a4944368 in_seq=0 out_seq=0 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.418+0000 7f4b3effd640 10 client.27908 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.419+0000 7f4b3effd640 1 -- 172.21.15.62:0/2213029946 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559ef7178d00 con 0x559ef722afb0 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.419+0000 7f4b3effd640 10 client.27908 handle_command_reply: tid=0 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.419+0000 7f4b67dc2740 1 client.27908 shutdown 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.421+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2213029946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 msgr2=0x559ef72123c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.421+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef72123c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f4b58009ca0 tx=0x7f4b5802a950 comp rx=0 tx=0).stop 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.424+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2213029946 shutdown_connections 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.424+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559ef722afb0 0x559ef7294490 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.424+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559ef721b150 0x559ef7223670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.424+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559ef720c360 0x559ef7223130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.424+0000 7f4b67dc2740 1 --2- 172.21.15.62:0/2213029946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559ef720ba40 0x559ef72123c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.424+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2213029946 >> 172.21.15.62:0/2213029946 conn(0x559ef72080f0 msgr2=0x559ef720d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.424+0000 7f4b3effd640 0 client.27908 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.426+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2213029946 shutdown_connections 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.427+0000 7f4b67dc2740 1 -- 172.21.15.62:0/2213029946 wait complete. 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.427+0000 7f4b67dc2740 20 client.27908 trim_cache size 0 max 16384 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 -- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4b480a32c0 msgr2=0x7f4b480a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4b480a32c0 0x7f4b480a5780 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4b500cdb00 tx=0x7f4b500d3af0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 -- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60100e20 msgr2=0x7f4b601984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60100e20 0x7f4b601984d0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4b5000e9f0 tx=0x7f4b5000edf0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 -- 172.21.15.62:0/3415964296 shutdown_connections 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4b480a32c0 0x7f4b480a5780 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b60102ef0 0x7f4b6019cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4b60102570 0x7f4b60198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 --2- 172.21.15.62:0/3415964296 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b60100e20 0x7f4b601984d0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:52.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 -- 172.21.15.62:0/3415964296 >> 172.21.15.62:0/3415964296 conn(0x7f4b600fc7d0 msgr2=0x7f4b6006a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:52.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 -- 172.21.15.62:0/3415964296 shutdown_connections 2024-09-15T23:28:52.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:51.433+0000 7f4b659e9640 1 -- 172.21.15.62:0/3415964296 wait complete. 2024-09-15T23:28:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:53 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:28:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:53 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:53 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:28:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:53 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:53.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:53 smithi062 ceph-mon[30949]: pgmap v748: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-09-15T23:28:53.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:53 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:28:53.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:53 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:53.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:53 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:28:53.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:53 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:53.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:53 smithi067 ceph-mon[35665]: pgmap v748: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-09-15T23:28:53.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:53 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:28:53.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:53 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:53.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:53 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:28:53.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:53 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:28:53.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:53 smithi148 ceph-mon[35610]: pgmap v748: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-09-15T23:28:54.320 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:28:54.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:54 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:28:54.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:54 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:28:54.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:54 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:28:54.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:54 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:28:54.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:54 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:28:54.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:54 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:28:54.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:54 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:28:54.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:54 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:28:54.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:54 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:28:54.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 755.0 s, 496.6 tps, lat 199.499 ms stddev 232.126 2024-09-15T23:28:54.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.654+0000 7f7693daa640 1 -- 172.21.15.62:0/8226183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c10e3b0 msgr2=0x7f768c110800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:54.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.654+0000 7f7693daa640 1 --2- 172.21.15.62:0/8226183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c10e3b0 0x7f768c110800 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f7684009ca0 tx=0x7f768402a940 comp rx=0 tx=0).stop 2024-09-15T23:28:54.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.654+0000 7f7693daa640 1 -- 172.21.15.62:0/8226183 shutdown_connections 2024-09-15T23:28:54.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.654+0000 7f7693daa640 1 --2- 172.21.15.62:0/8226183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c10e3b0 0x7f768c110800 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:54.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.654+0000 7f7693daa640 1 --2- 172.21.15.62:0/8226183 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f768c074f40 0x7f768c10de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:54.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.654+0000 7f7693daa640 1 --2- 172.21.15.62:0/8226183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f768c074620 0x7f768c074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:54.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.654+0000 7f7693daa640 1 -- 172.21.15.62:0/8226183 >> 172.21.15.62:0/8226183 conn(0x7f768c0fd8b0 msgr2=0x7f768c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:54.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.655+0000 7f7693daa640 1 -- 172.21.15.62:0/8226183 shutdown_connections 2024-09-15T23:28:54.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.655+0000 7f7693daa640 1 -- 172.21.15.62:0/8226183 wait complete. 2024-09-15T23:28:54.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7693daa640 1 Processor -- start 2024-09-15T23:28:54.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7693daa640 1 -- start start 2024-09-15T23:28:54.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7693daa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f768c074620 0x7f768c19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:54.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7693daa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c074f40 0x7f768c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:54.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7693daa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f768c10e3b0 0x7f768c19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7691345640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c074f40 0x7f768c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7691345640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c074f40 0x7f768c19ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47940/0 (socket says 172.21.15.62:47940) 2024-09-15T23:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7691345640 1 -- 172.21.15.62:0/995755052 learned_addr learned my addr 172.21.15.62:0/995755052 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f768c114190 con 0x7f768c074f40 2024-09-15T23:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f768c114010 con 0x7f768c10e3b0 2024-09-15T23:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.656+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f768c114310 con 0x7f768c074620 2024-09-15T23:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.657+0000 7f7692347640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f768c10e3b0 0x7f768c19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.657+0000 7f7691b46640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f768c074620 0x7f768c19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:54.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.657+0000 7f7692347640 1 -- 172.21.15.62:0/995755052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f768c074620 msgr2=0x7f768c19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.657+0000 7f7692347640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f768c074620 0x7f768c19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.657+0000 7f7692347640 1 -- 172.21.15.62:0/995755052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c074f40 msgr2=0x7f768c19ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.657+0000 7f7692347640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c074f40 0x7f768c19ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.657+0000 7f7692347640 1 -- 172.21.15.62:0/995755052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768c19f600 con 0x7f768c10e3b0 2024-09-15T23:28:54.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.657+0000 7f7692347640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f768c10e3b0 0x7f768c19ef20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7684005e30 tx=0x7f768402f630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:54.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.658+0000 7f7682ffd640 1 -- 172.21.15.62:0/995755052 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7684034040 con 0x7f768c10e3b0 2024-09-15T23:28:54.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.658+0000 7f7682ffd640 1 -- 172.21.15.62:0/995755052 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f768402fe10 con 0x7f768c10e3b0 2024-09-15T23:28:54.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.658+0000 7f7682ffd640 1 -- 172.21.15.62:0/995755052 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76840045d0 con 0x7f768c10e3b0 2024-09-15T23:28:54.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.658+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f768c19f890 con 0x7f768c10e3b0 2024-09-15T23:28:54.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.658+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f768c19fd20 con 0x7f768c10e3b0 2024-09-15T23:28:54.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.659+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f768c074a00 con 0x7f768c10e3b0 2024-09-15T23:28:54.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.661+0000 7f7682ffd640 1 -- 172.21.15.62:0/995755052 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7684037020 con 0x7f768c10e3b0 2024-09-15T23:28:54.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.662+0000 7f7682ffd640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76780a3390 0x7f76780a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:54.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.662+0000 7f7682ffd640 1 -- 172.21.15.62:0/995755052 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f76840f0b30 con 0x7f768c10e3b0 2024-09-15T23:28:54.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.663+0000 7f7691b46640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76780a3390 0x7f76780a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.665+0000 7f7682ffd640 1 -- 172.21.15.62:0/995755052 <== mon.2 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f76840bd550 con 0x7f768c10e3b0 2024-09-15T23:28:54.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:54.665+0000 7f7691b46640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76780a3390 0x7f76780a5850 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f76880095f0 tx=0x7f7688005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:54.723 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:55.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.009+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f768c19b9b0 con 0x7f768c10e3b0 2024-09-15T23:28:55.012 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:28:55.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.010+0000 7f7682ffd640 1 -- 172.21.15.62:0/995755052 <== mon.2 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7f768c19b9b0 con 0x7f768c10e3b0 2024-09-15T23:28:55.012 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:28:55.013 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:28:55.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76780a3390 msgr2=0x7f76780a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76780a3390 0x7f76780a5850 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f76880095f0 tx=0x7f7688005f90 comp rx=0 tx=0).stop 2024-09-15T23:28:55.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f768c10e3b0 msgr2=0x7f768c19ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f768c10e3b0 0x7f768c19ef20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7684005e30 tx=0x7f768402f630 comp rx=0 tx=0).stop 2024-09-15T23:28:55.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 shutdown_connections 2024-09-15T23:28:55.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f76780a3390 0x7f76780a5850 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f768c10e3b0 0x7f768c19ef20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f768c074f40 0x7f768c19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 --2- 172.21.15.62:0/995755052 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f768c074620 0x7f768c19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.018+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 >> 172.21.15.62:0/995755052 conn(0x7f768c0fd8b0 msgr2=0x7f768c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.019+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 shutdown_connections 2024-09-15T23:28:55.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.019+0000 7f7693daa640 1 -- 172.21.15.62:0/995755052 wait complete. 2024-09-15T23:28:55.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.290+0000 7fea042de640 1 -- 172.21.15.62:0/3427210301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 msgr2=0x7fe9fc11d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.290+0000 7fea042de640 1 --2- 172.21.15.62:0/3427210301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 0x7fe9fc11d530 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009be0 tx=0x7fe9f402a6f0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.290+0000 7fea042de640 1 -- 172.21.15.62:0/3427210301 shutdown_connections 2024-09-15T23:28:55.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.290+0000 7fea042de640 1 --2- 172.21.15.62:0/3427210301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 0x7fe9fc11d530 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.290+0000 7fea042de640 1 --2- 172.21.15.62:0/3427210301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9fc109830 0x7fe9fc109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.290+0000 7fea042de640 1 --2- 172.21.15.62:0/3427210301 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9fc109450 0x7fe9fc110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.290+0000 7fea042de640 1 -- 172.21.15.62:0/3427210301 >> 172.21.15.62:0/3427210301 conn(0x7fe9fc06e140 msgr2=0x7fe9fc06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.290+0000 7fea042de640 1 -- 172.21.15.62:0/3427210301 shutdown_connections 2024-09-15T23:28:55.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.295+0000 7fea042de640 1 -- 172.21.15.62:0/3427210301 wait complete. 2024-09-15T23:28:55.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.295+0000 7fea042de640 1 Processor -- start 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.302+0000 7fea042de640 1 -- start start 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.302+0000 7fea042de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9fc109450 0x7fe9fc11a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.302+0000 7fea042de640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9fc109830 0x7fe9fc11aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.303+0000 7fea042de640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 0x7fe9fc113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.303+0000 7fea042de640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc120da0 con 0x7fe9fc109450 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.303+0000 7fea042de640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc120c20 con 0x7fe9fc11b140 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.303+0000 7fea042de640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc120f20 con 0x7fe9fc109830 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.303+0000 7fea0287b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 0x7fe9fc113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.303+0000 7fea0287b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 0x7fe9fc113560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51374/0 (socket says 172.21.15.62:51374) 2024-09-15T23:28:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.304+0000 7fea0287b640 1 -- 172.21.15.62:0/1443966365 learned_addr learned my addr 172.21.15.62:0/1443966365 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.304+0000 7fea01879640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9fc109830 0x7fe9fc11aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.304+0000 7fea0207a640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9fc109450 0x7fe9fc11a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.305+0000 7fea0287b640 1 -- 172.21.15.62:0/1443966365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9fc109830 msgr2=0x7fe9fc11aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.305+0000 7fea0287b640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9fc109830 0x7fe9fc11aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.305+0000 7fea0287b640 1 -- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9fc109450 msgr2=0x7fe9fc11a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.305+0000 7fea0287b640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9fc109450 0x7fe9fc11a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.305+0000 7fea0287b640 1 -- 172.21.15.62:0/1443966365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc113e20 con 0x7fe9fc11b140 2024-09-15T23:28:55.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.309+0000 7fea0287b640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 0x7fe9fc113560 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4034040 tx=0x7fe9f4030890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.309+0000 7fe9f37fe640 1 -- 172.21.15.62:0/1443966365 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4041020 con 0x7fe9fc11b140 2024-09-15T23:28:55.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.309+0000 7fe9f37fe640 1 -- 172.21.15.62:0/1443966365 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f4030ea0 con 0x7fe9fc11b140 2024-09-15T23:28:55.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.310+0000 7fe9f37fe640 1 -- 172.21.15.62:0/1443966365 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4036040 con 0x7fe9fc11b140 2024-09-15T23:28:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.312+0000 7fea042de640 1 -- 172.21.15.62:0/1443966365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9fc1140b0 con 0x7fe9fc11b140 2024-09-15T23:28:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.312+0000 7fea042de640 1 -- 172.21.15.62:0/1443966365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9fc118000 con 0x7fe9fc11b140 2024-09-15T23:28:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.315+0000 7fea066b7740 1 Processor -- start 2024-09-15T23:28:55.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.317+0000 7fea066b7740 1 -- start start 2024-09-15T23:28:55.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.317+0000 7fea066b7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3b6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.317+0000 7fea066b7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.317+0000 7fea066b7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.317+0000 7fea066b7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b28e31c2e0 con 0x55b28e3b6dc0 2024-09-15T23:28:55.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.317+0000 7fea066b7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b28e2c3ce0 con 0x55b28e3b64a0 2024-09-15T23:28:55.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.317+0000 7fea066b7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b28e3182a0 con 0x55b28e3c1680 2024-09-15T23:28:55.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.321+0000 7fea0207a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.321+0000 7fea0207a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47986/0 (socket says 172.21.15.62:47986) 2024-09-15T23:28:55.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.321+0000 7fea0207a640 1 -- 172.21.15.62:0/52320617 learned_addr learned my addr 172.21.15.62:0/52320617 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.321+0000 7fea01879640 1 --2- 172.21.15.62:0/52320617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3b6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.321+0000 7fea01879640 1 --2- 172.21.15.62:0/52320617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.322+0000 7fe9f37fe640 1 -- 172.21.15.62:0/1443966365 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe9f4044020 con 0x7fe9fc11b140 2024-09-15T23:28:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.324+0000 7fe9f37fe640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9e80a32c0 0x7fe9e80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.324+0000 7fe9f37fe640 1 -- 172.21.15.62:0/1443966365 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe9f40efd40 con 0x7fe9fc11b140 2024-09-15T23:28:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.327+0000 7fea01879640 1 -- 172.21.15.62:0/52320617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 msgr2=0x55b28e3c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.327+0000 7fea01879640 1 --2- 172.21.15.62:0/52320617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.327+0000 7fea01879640 1 -- 172.21.15.62:0/52320617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 msgr2=0x55b28e3c1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.327+0000 7fea01879640 1 --2- 172.21.15.62:0/52320617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.327+0000 7fea01879640 1 -- 172.21.15.62:0/52320617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b28e2a8440 con 0x55b28e3b64a0 2024-09-15T23:28:55.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.330+0000 7fea0287b640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9e80a32c0 0x7fe9e80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.331+0000 7fea01879640 1 --2- 172.21.15.62:0/52320617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3b6880 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009690 tx=0x7fe9ec009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d000895e669a9f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.333+0000 7fe9f0dea640 1 -- 172.21.15.62:0/52320617 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec013cd0 con 0x55b28e3b64a0 2024-09-15T23:28:55.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.333+0000 7fe9f0dea640 1 -- 172.21.15.62:0/52320617 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ec005020 con 0x55b28e3b64a0 2024-09-15T23:28:55.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.333+0000 7fe9f0dea640 1 -- 172.21.15.62:0/52320617 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec004890 con 0x55b28e3b64a0 2024-09-15T23:28:55.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.334+0000 7fea0287b640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9e80a32c0 0x7fe9e80a5780 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40bbfa0 tx=0x7fe9f400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.337+0000 7fea066b7740 1 -- 172.21.15.62:0/52320617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 msgr2=0x55b28e3b6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.337+0000 7fea066b7740 1 --2- 172.21.15.62:0/52320617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3b6880 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009690 tx=0x7fe9ec009620 comp rx=0 tx=0).stop 2024-09-15T23:28:55.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.343+0000 7fea066b7740 1 -- 172.21.15.62:0/52320617 shutdown_connections 2024-09-15T23:28:55.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.343+0000 7fea066b7740 1 --2- 172.21.15.62:0/52320617 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.343+0000 7fea066b7740 1 --2- 172.21.15.62:0/52320617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.343+0000 7fea066b7740 1 --2- 172.21.15.62:0/52320617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3b6880 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.343+0000 7fea066b7740 1 -- 172.21.15.62:0/52320617 >> 172.21.15.62:0/52320617 conn(0x55b28e3b2a30 msgr2=0x55b28e3b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.349+0000 7fea066b7740 1 -- 172.21.15.62:0/52320617 shutdown_connections 2024-09-15T23:28:55.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.353+0000 7fea066b7740 1 -- 172.21.15.62:0/52320617 wait complete. 2024-09-15T23:28:55.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.354+0000 7fea066b7740 1 -- start start 2024-09-15T23:28:55.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.354+0000 7fea066b7740 1 Processor -- start 2024-09-15T23:28:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.355+0000 7fea066b7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b64a0 0x55b28e3c1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.355+0000 7fea066b7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b6dc0 0x55b28e3be310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.355+0000 7fea066b7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3be8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.355+0000 7fea066b7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b28e3c4250 con 0x55b28e3b64a0 2024-09-15T23:28:55.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.355+0000 7fea066b7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b28e2d7050 con 0x55b28e3b6dc0 2024-09-15T23:28:55.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.355+0000 7fea066b7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b28e3c43d0 con 0x55b28e3c1680 2024-09-15T23:28:55.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.355+0000 7fea066b7740 10 client.0 _ll_register_callbacks cb 0x55b28e317160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:55.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.358+0000 7fea01879640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3be8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.358+0000 7fea01879640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b64a0 0x55b28e3c1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.358+0000 7fea01879640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b64a0 0x55b28e3c1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47996/0 (socket says 172.21.15.62:47996) 2024-09-15T23:28:55.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.358+0000 7fea01879640 1 -- 172.21.15.62:0/1858174732 learned_addr learned my addr 172.21.15.62:0/1858174732 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.358+0000 7fea0207a640 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b6dc0 0x55b28e3be310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.359+0000 7fea01879640 1 -- 172.21.15.62:0/1858174732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b6dc0 msgr2=0x55b28e3be310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.359+0000 7fea01879640 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b6dc0 0x55b28e3be310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.359+0000 7fea01879640 1 -- 172.21.15.62:0/1858174732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b64a0 msgr2=0x55b28e3c1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.359+0000 7fea01879640 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b64a0 0x55b28e3c1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.359+0000 7fea01879640 1 -- 172.21.15.62:0/1858174732 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b28e08d3c0 con 0x55b28e3c1680 2024-09-15T23:28:55.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.362+0000 7fea01879640 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3be8e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec005970 tx=0x7fe9ec009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=333a61e4aef8f165 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.364+0000 7fe9e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:55.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.364+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/1858174732 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec02d060 con 0x55b28e3c1680 2024-09-15T23:28:55.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.364+0000 7fea066b7740 1 -- 172.21.15.62:0/1858174732 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9ec005020 con 0x55b28e3c1680 2024-09-15T23:28:55.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.364+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/1858174732 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe9ec0117b0 con 0x55b28e3c1680 2024-09-15T23:28:55.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.368+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/1858174732 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe9ec004e50 con 0x55b28e3c1680 2024-09-15T23:28:55.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.368+0000 7fea066b7740 10 client.27643 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:55.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.368+0000 7fea066b7740 1 -- 172.21.15.62:0/1858174732 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b28e3cfc90 con 0x55b28e3c1680 2024-09-15T23:28:55.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.372+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/1858174732 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ec005020 con 0x55b28e3c1680 2024-09-15T23:28:55.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.372+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/1858174732 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fe9ec014ce0 con 0x55b28e3c1680 2024-09-15T23:28:55.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.372+0000 7fea066b7740 10 client.27643 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:55.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.372+0000 7fea066b7740 10 client.27643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:55.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.372+0000 7fea066b7740 20 client.27643 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:55.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.372+0000 7fea066b7740 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b28e3d1230 0x55b28e3d3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.372+0000 7fea066b7740 4 client.27643 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:55.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.372+0000 7fea066b7740 1 -- 172.21.15.62:0/1858174732 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b28e329850 con 0x55b28e3d1230 2024-09-15T23:28:55.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.373+0000 7fea0207a640 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b28e3d1230 0x55b28e3d3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.374+0000 7fea0207a640 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b28e3d1230 0x55b28e3d3670 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81b37e9b57ae3a63 server_cookie=7624b4d3ef2682b6 in_seq=0 out_seq=0 2024-09-15T23:28:55.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.375+0000 7fe9e5ffb640 10 client.27643 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:55.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.377+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/1858174732 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b28e329850 con 0x55b28e3d1230 2024-09-15T23:28:55.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.377+0000 7fe9e5ffb640 10 client.27643 handle_command_reply: tid=0 2024-09-15T23:28:55.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.378+0000 7fea066b7740 1 client.27643 shutdown 2024-09-15T23:28:55.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.380+0000 7fea066b7740 1 -- 172.21.15.62:0/1858174732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 msgr2=0x55b28e3be8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.380+0000 7fea066b7740 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3be8e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec005970 tx=0x7fe9ec009620 comp rx=0 tx=0).stop 2024-09-15T23:28:55.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.380+0000 7fea066b7740 1 -- 172.21.15.62:0/1858174732 shutdown_connections 2024-09-15T23:28:55.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.380+0000 7fea066b7740 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b28e3d1230 0x55b28e3d3670 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.380+0000 7fea066b7740 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3be8e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.380+0000 7fea066b7740 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b6dc0 0x55b28e3be310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.380+0000 7fea066b7740 1 --2- 172.21.15.62:0/1858174732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b64a0 0x55b28e3c1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.380+0000 7fea066b7740 1 -- 172.21.15.62:0/1858174732 >> 172.21.15.62:0/1858174732 conn(0x55b28e3b2a30 msgr2=0x55b28e3b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.380+0000 7fe9e5ffb640 0 client.27643 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:55.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.381+0000 7fea066b7740 1 -- 172.21.15.62:0/1858174732 shutdown_connections 2024-09-15T23:28:55.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.381+0000 7fea066b7740 1 -- 172.21.15.62:0/1858174732 wait complete. 2024-09-15T23:28:55.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.381+0000 7fea066b7740 20 client.27643 trim_cache size 0 max 16384 2024-09-15T23:28:55.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.416+0000 7fea066b7740 1 Processor -- start 2024-09-15T23:28:55.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.416+0000 7fea066b7740 1 -- start start 2024-09-15T23:28:55.418 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:55 smithi062 ceph-mon[30949]: pgmap v749: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.9 MiB/s wr, 645 op/s 2024-09-15T23:28:55.418 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:55 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/995755052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:55.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.416+0000 7fea066b7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3bcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.416+0000 7fea066b7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea066b7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3d8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea066b7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1c88d0 con 0x55b28e3b6dc0 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea066b7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1c61a0 con 0x55b28e3b64a0 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea066b7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1c3ff0 con 0x55b28e3c1680 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea0207a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea01879640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3bcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea0207a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48008/0 (socket says 172.21.15.62:48008) 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea0207a640 1 -- 172.21.15.62:0/1623220567 learned_addr learned my addr 172.21.15.62:0/1623220567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea01879640 1 --2- 172.21.15.62:0/1623220567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3d8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea01879640 1 -- 172.21.15.62:0/1623220567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 msgr2=0x55b28e3d8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea01879640 1 --2- 172.21.15.62:0/1623220567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3d8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea01879640 1 -- 172.21.15.62:0/1623220567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 msgr2=0x55b28e3c0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea01879640 1 --2- 172.21.15.62:0/1623220567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.417+0000 7fea01879640 1 -- 172.21.15.62:0/1623220567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b28e3bd2c0 con 0x55b28e3b64a0 2024-09-15T23:28:55.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.418+0000 7fea01879640 1 --2- 172.21.15.62:0/1623220567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3bcbe0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec020c30 tx=0x7fe9ec0092b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d926dff1a9f0159 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.418+0000 7fe9e7fff640 1 -- 172.21.15.62:0/1623220567 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec02d070 con 0x55b28e3b64a0 2024-09-15T23:28:55.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.418+0000 7fe9e7fff640 1 -- 172.21.15.62:0/1623220567 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ec009a50 con 0x55b28e3b64a0 2024-09-15T23:28:55.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.418+0000 7fe9e7fff640 1 -- 172.21.15.62:0/1623220567 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec037080 con 0x55b28e3b64a0 2024-09-15T23:28:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.419+0000 7fea066b7740 1 -- 172.21.15.62:0/1623220567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 msgr2=0x55b28e3bcbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.419+0000 7fea066b7740 1 --2- 172.21.15.62:0/1623220567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3bcbe0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec020c30 tx=0x7fe9ec0092b0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.419+0000 7fea066b7740 1 -- 172.21.15.62:0/1623220567 shutdown_connections 2024-09-15T23:28:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.419+0000 7fea066b7740 1 --2- 172.21.15.62:0/1623220567 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3c1680 0x55b28e3d8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.419+0000 7fea066b7740 1 --2- 172.21.15.62:0/1623220567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.419+0000 7fea066b7740 1 --2- 172.21.15.62:0/1623220567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3bcbe0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.420+0000 7fea066b7740 1 -- 172.21.15.62:0/1623220567 >> 172.21.15.62:0/1623220567 conn(0x55b28e3b2a30 msgr2=0x55b28e3bfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.420+0000 7fea066b7740 1 -- 172.21.15.62:0/1623220567 shutdown_connections 2024-09-15T23:28:55.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.420+0000 7fea066b7740 1 -- 172.21.15.62:0/1623220567 wait complete. 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.421+0000 7fea066b7740 1 -- start start 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.421+0000 7fea066b7740 1 Processor -- start 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.421+0000 7fea066b7740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.421+0000 7fea066b7740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.421+0000 7fea066b7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3c1680 0x55b28e439f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.421+0000 7fea066b7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1c4660 con 0x55b28e3c1680 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.421+0000 7fea066b7740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1c5ad0 con 0x55b28e3b64a0 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.421+0000 7fea066b7740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1c4d30 con 0x55b28e3b6dc0 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.421+0000 7fea01879640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea01879640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3c6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51390/0 (socket says 172.21.15.62:51390) 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea01879640 1 -- 172.21.15.62:0/2438530830 learned_addr learned my addr 172.21.15.62:0/2438530830 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea01879640 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3c1680 0x55b28e439f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea066b7740 10 client.0 _ll_register_callbacks cb 0x7fe9f800b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:55.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea0207a640 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea01879640 1 -- 172.21.15.62:0/2438530830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3b6dc0 msgr2=0x55b28e3c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea01879640 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea01879640 1 -- 172.21.15.62:0/2438530830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3c1680 msgr2=0x55b28e439f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea01879640 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3c1680 0x55b28e439f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.422+0000 7fea01879640 1 -- 172.21.15.62:0/2438530830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b28e3c13d0 con 0x55b28e3b64a0 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.423+0000 7fea01879640 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3c6b70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009c60 tx=0x7fe9ec004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fe125684d3ba8d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.424+0000 7fe9e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.424+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/2438530830 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec037030 con 0x55b28e3b64a0 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.424+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/2438530830 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe9ec004040 con 0x55b28e3b64a0 2024-09-15T23:28:55.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.424+0000 7fea066b7740 1 -- 172.21.15.62:0/2438530830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9ec009a50 con 0x55b28e3b64a0 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.425+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/2438530830 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe9ec02d040 con 0x55b28e3b64a0 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.425+0000 7fea066b7740 10 client.27929 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.425+0000 7fea066b7740 1 -- 172.21.15.62:0/2438530830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b28e3b6bf0 con 0x55b28e3b64a0 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.425+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/2438530830 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ec009a50 con 0x55b28e3b64a0 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.425+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/2438530830 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fe9ec003680 con 0x55b28e3b64a0 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.425+0000 7fea066b7740 10 client.27929 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.426+0000 7fea066b7740 10 client.27929 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.426+0000 7fea066b7740 20 client.27929 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.426+0000 7fea066b7740 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b28e43b9b0 0x55b28e43ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.426+0000 7fea066b7740 4 client.27929 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:28:55.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.426+0000 7fea066b7740 1 -- 172.21.15.62:0/2438530830 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b28e323d60 con 0x55b28e43b9b0 2024-09-15T23:28:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.427+0000 7fea0207a640 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b28e43b9b0 0x55b28e43ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.427+0000 7fea0207a640 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b28e43b9b0 0x55b28e43ddd0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adc7c0af735a40ad server_cookie=dd37f77115d00eb7 in_seq=0 out_seq=0 2024-09-15T23:28:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.427+0000 7fe9e5ffb640 10 client.27929 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.428+0000 7fe9e5ffb640 1 -- 172.21.15.62:0/2438530830 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b28e323d60 con 0x55b28e43b9b0 2024-09-15T23:28:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.428+0000 7fe9e5ffb640 10 client.27929 handle_command_reply: tid=0 2024-09-15T23:28:55.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.428+0000 7fea066b7740 1 client.27929 shutdown 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.428+0000 7fea066b7740 1 -- 172.21.15.62:0/2438530830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 msgr2=0x55b28e3c6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.428+0000 7fea066b7740 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3c6b70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009c60 tx=0x7fe9ec004760 comp rx=0 tx=0).stop 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.428+0000 7fea066b7740 1 -- 172.21.15.62:0/2438530830 shutdown_connections 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.428+0000 7fea066b7740 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b28e43b9b0 0x55b28e43ddd0 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.429+0000 7fea066b7740 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b28e3c1680 0x55b28e439f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.429+0000 7fe9e5ffb640 0 client.27929 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.429+0000 7fea066b7740 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b28e3b6dc0 0x55b28e3c70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.429+0000 7fea066b7740 1 --2- 172.21.15.62:0/2438530830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b28e3b64a0 0x55b28e3c6b70 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.429+0000 7fea066b7740 1 -- 172.21.15.62:0/2438530830 >> 172.21.15.62:0/2438530830 conn(0x55b28e3b2a30 msgr2=0x55b28e3bfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.429+0000 7fea066b7740 1 -- 172.21.15.62:0/2438530830 shutdown_connections 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.429+0000 7fea066b7740 1 -- 172.21.15.62:0/2438530830 wait complete. 2024-09-15T23:28:55.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.429+0000 7fea066b7740 20 client.27929 trim_cache size 0 max 16384 2024-09-15T23:28:55.432 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:28:55.432 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:28:55.432 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:28:55.432 INFO:teuthology.orchestra.run.smithi062.stdout: "f28c7295-f7ae-47d5-b6db-b9563604d77f": { 2024-09-15T23:28:55.432 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:28:55.432 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "f28c7295-f7ae-47d5-b6db-b9563604d77f", 2024-09-15T23:28:55.433 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:28:55.433 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:28:55.433 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:28:55.433 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 -- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9e80a32c0 msgr2=0x7fe9e80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9e80a32c0 0x7fe9e80a5780 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe9f40bbfa0 tx=0x7fe9f400b040 comp rx=0 tx=0).stop 2024-09-15T23:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 -- 172.21.15.62:0/1443966365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 msgr2=0x7fe9fc113560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 0x7fe9fc113560 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4034040 tx=0x7fe9f4030890 comp rx=0 tx=0).stop 2024-09-15T23:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 -- 172.21.15.62:0/1443966365 shutdown_connections 2024-09-15T23:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe9e80a32c0 0x7fe9e80a5780 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9fc11b140 0x7fe9fc113560 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe9fc109830 0x7fe9fc11aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 --2- 172.21.15.62:0/1443966365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9fc109450 0x7fe9fc11a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.431+0000 7fea042de640 1 -- 172.21.15.62:0/1443966365 >> 172.21.15.62:0/1443966365 conn(0x7fe9fc06e140 msgr2=0x7fe9fc072450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.432+0000 7fea042de640 1 -- 172.21.15.62:0/1443966365 shutdown_connections 2024-09-15T23:28:55.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:55.432+0000 7fea042de640 1 -- 172.21.15.62:0/1443966365 wait complete. 2024-09-15T23:28:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 -- 172.21.15.62:0/1873517282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884104f40 msgr2=0x7f78841053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 --2- 172.21.15.62:0/1873517282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884104f40 0x7f78841053c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7874009900 tx=0x7f787402a3f0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 -- 172.21.15.62:0/1873517282 shutdown_connections 2024-09-15T23:28:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 --2- 172.21.15.62:0/1873517282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7884105900 0x7f788410c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 --2- 172.21.15.62:0/1873517282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884104f40 0x7f78841053c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 --2- 172.21.15.62:0/1873517282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7884103d50 0x7f7884104150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 -- 172.21.15.62:0/1873517282 >> 172.21.15.62:0/1873517282 conn(0x7f78840ff4b0 msgr2=0x7f78841018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.591 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 -- 172.21.15.62:0/1873517282 shutdown_connections 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 -- 172.21.15.62:0/1873517282 wait complete. 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 Processor -- start 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.588+0000 7f788b352640 1 -- start start 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f788b352640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884103d50 0x7f7884199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f788b352640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7884104f40 0x7f788419a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f788b352640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7884105900 0x7f78841a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f788b352640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f788410f9d0 con 0x7f7884104f40 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f788b352640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f788410f850 con 0x7f7884105900 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f788b352640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f788410fb50 con 0x7f7884103d50 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78890ee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884103d50 0x7f7884199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78888ed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7884104f40 0x7f788419a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78898ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7884105900 0x7f78841a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.592 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78890ee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884103d50 0x7f7884199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33862/0 (socket says 172.21.15.62:33862) 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78890ee640 1 -- 172.21.15.62:0/3541462328 learned_addr learned my addr 172.21.15.62:0/3541462328 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78898ef640 1 -- 172.21.15.62:0/3541462328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884103d50 msgr2=0x7f7884199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78898ef640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884103d50 0x7f7884199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78898ef640 1 -- 172.21.15.62:0/3541462328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7884104f40 msgr2=0x7f788419a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78898ef640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7884104f40 0x7f788419a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78898ef640 1 -- 172.21.15.62:0/3541462328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78841a1bf0 con 0x7f7884105900 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78890ee640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884103d50 0x7f7884199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.589+0000 7f78898ef640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7884105900 0x7f78841a1580 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f787c009fd0 tx=0x7f787c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.590+0000 7f787a7fc640 1 -- 172.21.15.62:0/3541462328 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c00ce20 con 0x7f7884105900 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.590+0000 7f787a7fc640 1 -- 172.21.15.62:0/3541462328 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f787c005020 con 0x7f7884105900 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.590+0000 7f788b352640 1 -- 172.21.15.62:0/3541462328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78841a1ee0 con 0x7f7884105900 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.590+0000 7f787a7fc640 1 -- 172.21.15.62:0/3541462328 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c010720 con 0x7f7884105900 2024-09-15T23:28:55.593 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.590+0000 7f788b352640 1 -- 172.21.15.62:0/3541462328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78841a2350 con 0x7f7884105900 2024-09-15T23:28:55.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.593+0000 7f787a7fc640 1 -- 172.21.15.62:0/3541462328 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f787c005540 con 0x7f7884105900 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.594+0000 7f787a7fc640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78700a32f0 0x7f78700a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.594+0000 7f787a7fc640 1 -- 172.21.15.62:0/3541462328 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f787c0d03a0 con 0x7f7884105900 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.594+0000 7f78890ee640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78700a32f0 0x7f78700a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.594+0000 7f788e1ca740 1 Processor -- start 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.594+0000 7f788e1ca740 1 -- start start 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.594+0000 7f78890ee640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78700a32f0 0x7f78700a57b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f78800058c0 tx=0x7f7880005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.594+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb66da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb672e0 0x55ae1cb6bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb6e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.597 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f788e1ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ae1caca2f0 con 0x55ae1cb672e0 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f788e1ca740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ae1ca72270 con 0x55ae1cb6c370 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f788e1ca740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ae1cac62b0 con 0x55ae1cb669a0 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78888ed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb672e0 0x55ae1cb6bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78890ee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb66da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78898ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb6e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78890ee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb66da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33864/0 (socket says 172.21.15.62:33864) 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78898ef640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb6e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51414/0 (socket says 172.21.15.62:51414) 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78890ee640 1 -- 172.21.15.62:0/3206814016 learned_addr learned my addr 172.21.15.62:0/3206814016 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78890ee640 1 -- 172.21.15.62:0/3206814016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 msgr2=0x55ae1cb6e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78890ee640 1 --2- 172.21.15.62:0/3206814016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb6e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78890ee640 1 -- 172.21.15.62:0/3206814016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb672e0 msgr2=0x55ae1cb6bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78890ee640 1 --2- 172.21.15.62:0/3206814016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb672e0 0x55ae1cb6bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78890ee640 1 -- 172.21.15.62:0/3206814016 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae1ca71340 con 0x55ae1cb669a0 2024-09-15T23:28:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78898ef640 1 --2- 172.21.15.62:0/3206814016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb6e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.595+0000 7f78890ee640 1 --2- 172.21.15.62:0/3206814016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb66da0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f788000f6f0 tx=0x7f7880014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eb8c6fe0f3e32a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.596+0000 7f786f7fe640 1 -- 172.21.15.62:0/3206814016 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788002a070 con 0x55ae1cb669a0 2024-09-15T23:28:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.596+0000 7f786f7fe640 1 -- 172.21.15.62:0/3206814016 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f788001b910 con 0x55ae1cb669a0 2024-09-15T23:28:55.599 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.596+0000 7f786f7fe640 1 -- 172.21.15.62:0/3206814016 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788001cb10 con 0x55ae1cb669a0 2024-09-15T23:28:55.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.596+0000 7f788e1ca740 1 -- 172.21.15.62:0/3206814016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 msgr2=0x55ae1cb66da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.596+0000 7f788e1ca740 1 --2- 172.21.15.62:0/3206814016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb66da0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f788000f6f0 tx=0x7f7880014000 comp rx=0 tx=0).stop 2024-09-15T23:28:55.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.596+0000 7f788e1ca740 1 -- 172.21.15.62:0/3206814016 shutdown_connections 2024-09-15T23:28:55.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.596+0000 7f788e1ca740 1 --2- 172.21.15.62:0/3206814016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb6e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.596+0000 7f788e1ca740 1 --2- 172.21.15.62:0/3206814016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb672e0 0x55ae1cb6bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 --2- 172.21.15.62:0/3206814016 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb66da0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 -- 172.21.15.62:0/3206814016 >> 172.21.15.62:0/3206814016 conn(0x55ae1cb62010 msgr2=0x55ae1cb62410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 -- 172.21.15.62:0/3206814016 shutdown_connections 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 -- 172.21.15.62:0/3206814016 wait complete. 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 -- start start 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 Processor -- start 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb672e0 0x55ae1cb711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 0x55ae1cb78170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ae1cb6f340 con 0x55ae1cb6c370 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ae1ca53290 con 0x55ae1cb672e0 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.597+0000 7f788e1ca740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ae1cb6f4c0 con 0x55ae1cb669a0 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78890ee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78890ee640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33878/0 (socket says 172.21.15.62:33878) 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78890ee640 1 -- 172.21.15.62:0/2578599438 learned_addr learned my addr 172.21.15.62:0/2578599438 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f788e1ca740 10 client.0 _ll_register_callbacks cb 0x55ae1ca84140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78888ed640 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb672e0 0x55ae1cb711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78898ef640 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 0x55ae1cb78170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78890ee640 1 -- 172.21.15.62:0/2578599438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb672e0 msgr2=0x55ae1cb711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78890ee640 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb672e0 0x55ae1cb711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78890ee640 1 -- 172.21.15.62:0/2578599438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 msgr2=0x55ae1cb78170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78890ee640 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 0x55ae1cb78170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78890ee640 1 -- 172.21.15.62:0/2578599438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae1c7dd210 con 0x55ae1cb669a0 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f78890ee640 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70c60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f788000fbb0 tx=0x7f788001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=262727ee10c0ff3b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f786cff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f786cff9640 1 -- 172.21.15.62:0/2578599438 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880033060 con 0x55ae1cb669a0 2024-09-15T23:28:55.605 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f788e1ca740 1 -- 172.21.15.62:0/2578599438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f788001b910 con 0x55ae1cb669a0 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f786cff9640 1 -- 172.21.15.62:0/2578599438 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f788001b630 con 0x55ae1cb669a0 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f786cff9640 1 -- 172.21.15.62:0/2578599438 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f788003a050 con 0x55ae1cb669a0 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f788e1ca740 10 client.27661 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.598+0000 7f788e1ca740 1 -- 172.21.15.62:0/2578599438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ae1c8a9620 con 0x55ae1cb669a0 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f786cff9640 1 -- 172.21.15.62:0/2578599438 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f788001b910 con 0x55ae1cb669a0 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f786cff9640 1 -- 172.21.15.62:0/2578599438 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f7880024030 con 0x55ae1cb669a0 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f788e1ca740 10 client.27661 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f788e1ca740 10 client.27661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f788e1ca740 20 client.27661 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f788e1ca740 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ae1cb7c450 0x55ae1cb7e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f788e1ca740 4 client.27661 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f788e1ca740 1 -- 172.21.15.62:0/2578599438 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae1cad7a20 con 0x55ae1cb7c450 2024-09-15T23:28:55.606 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f78888ed640 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ae1cb7c450 0x55ae1cb7e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f786cff9640 10 client.27661 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.599+0000 7f78888ed640 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ae1cb7c450 0x55ae1cb7e8b0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3592646de5b60c0c server_cookie=5b914bc7d1c8a456 in_seq=0 out_seq=0 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.600+0000 7f786cff9640 1 -- 172.21.15.62:0/2578599438 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ae1cad7a20 con 0x55ae1cb7c450 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.600+0000 7f786cff9640 10 client.27661 handle_command_reply: tid=0 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.600+0000 7f788e1ca740 1 client.27661 shutdown 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.600+0000 7f788e1ca740 1 -- 172.21.15.62:0/2578599438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 msgr2=0x55ae1cb70c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.600+0000 7f788e1ca740 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70c60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f788000fbb0 tx=0x7f788001c4d0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f788e1ca740 1 -- 172.21.15.62:0/2578599438 shutdown_connections 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f788e1ca740 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ae1cb7c450 0x55ae1cb7e8b0 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f788e1ca740 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 0x55ae1cb78170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f788e1ca740 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb672e0 0x55ae1cb711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f788e1ca740 1 --2- 172.21.15.62:0/2578599438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70c60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f786cff9640 0 client.27661 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f788e1ca740 1 -- 172.21.15.62:0/2578599438 >> 172.21.15.62:0/2578599438 conn(0x55ae1cb62010 msgr2=0x55ae1cb63820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f788e1ca740 1 -- 172.21.15.62:0/2578599438 shutdown_connections 2024-09-15T23:28:55.608 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f788e1ca740 1 -- 172.21.15.62:0/2578599438 wait complete. 2024-09-15T23:28:55.608 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.601+0000 7f788e1ca740 20 client.27661 trim_cache size 0 max 16384 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.618+0000 7f788e1ca740 1 Processor -- start 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.618+0000 7f788e1ca740 1 -- start start 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb78090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb70f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f788e1ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78841af080 con 0x55ae1cb669a0 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f788e1ca740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78841ac950 con 0x55ae1cb6c370 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f788e1ca740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78841aa7a0 con 0x55ae1cb672e0 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78890ee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78890ee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48058/0 (socket says 172.21.15.62:48058) 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78890ee640 1 -- 172.21.15.62:0/3400426750 learned_addr learned my addr 172.21.15.62:0/3400426750 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78888ed640 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb78090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78898ef640 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb70f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78890ee640 1 -- 172.21.15.62:0/3400426750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 msgr2=0x55ae1cb78090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78890ee640 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb78090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78890ee640 1 -- 172.21.15.62:0/3400426750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 msgr2=0x55ae1cb70f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78890ee640 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb70f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78890ee640 1 -- 172.21.15.62:0/3400426750 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae1cb6f620 con 0x55ae1cb669a0 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78888ed640 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb78090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f78890ee640 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70a10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f788001cda0 tx=0x7f7880004ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428d547e0f183149 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.619+0000 7f786effd640 1 -- 172.21.15.62:0/3400426750 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880033040 con 0x55ae1cb669a0 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f786effd640 1 -- 172.21.15.62:0/3400426750 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f788001c540 con 0x55ae1cb669a0 2024-09-15T23:28:55.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f786effd640 1 -- 172.21.15.62:0/3400426750 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788002a070 con 0x55ae1cb669a0 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f788e1ca740 1 -- 172.21.15.62:0/3400426750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb669a0 msgr2=0x55ae1cb70a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f788e1ca740 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70a10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f788001cda0 tx=0x7f7880004ca0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f788e1ca740 1 -- 172.21.15.62:0/3400426750 shutdown_connections 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f788e1ca740 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb6c370 0x55ae1cb70f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f788e1ca740 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb78090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f788e1ca740 1 --2- 172.21.15.62:0/3400426750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb669a0 0x55ae1cb70a10 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f788e1ca740 1 -- 172.21.15.62:0/3400426750 >> 172.21.15.62:0/3400426750 conn(0x55ae1cb62010 msgr2=0x55ae1cb68530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f788e1ca740 1 -- 172.21.15.62:0/3400426750 shutdown_connections 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.620+0000 7f788e1ca740 1 -- 172.21.15.62:0/3400426750 wait complete. 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.621+0000 7f788e1ca740 1 -- start start 2024-09-15T23:28:55.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.621+0000 7f788e1ca740 1 Processor -- start 2024-09-15T23:28:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.621+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb669a0 0x55ae1cb74eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.621+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb753f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.621+0000 7f788e1ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 0x55ae1cbe5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.621+0000 7f788e1ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78841aae10 con 0x55ae1cb6c370 2024-09-15T23:28:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78888ed640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb753f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78888ed640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb753f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33902/0 (socket says 172.21.15.62:33902) 2024-09-15T23:28:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78888ed640 1 -- 172.21.15.62:0/1912485225 learned_addr learned my addr 172.21.15.62:0/1912485225 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78898ef640 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 0x55ae1cbe5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78890ee640 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb669a0 0x55ae1cb74eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78841ac280 con 0x55ae1cb669a0 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78841ab4e0 con 0x55ae1cb672e0 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f788e1ca740 10 client.0 _ll_register_callbacks cb 0x7f7880028070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78888ed640 1 -- 172.21.15.62:0/1912485225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb669a0 msgr2=0x55ae1cb74eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78888ed640 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb669a0 0x55ae1cb74eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78888ed640 1 -- 172.21.15.62:0/1912485225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 msgr2=0x55ae1cbe5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78888ed640 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 0x55ae1cbe5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.622+0000 7f78888ed640 1 -- 172.21.15.62:0/1912485225 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae1cb784e0 con 0x55ae1cb672e0 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.623+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f788001c540 con 0x55ae1cb672e0 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.623+0000 7f78898ef640 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 0x55ae1cbe5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.624+0000 7f78888ed640 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb753f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7874000c00 tx=0x7f7874032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce39e117baed7b51 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:55.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.624+0000 7f786cff9640 10 client.27667 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.624+0000 7f786cff9640 1 -- 172.21.15.62:0/1912485225 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874035040 con 0x55ae1cb672e0 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.624+0000 7f786cff9640 1 -- 172.21.15.62:0/1912485225 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7874005040 con 0x55ae1cb672e0 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.624+0000 7f786cff9640 1 -- 172.21.15.62:0/1912485225 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7874005240 con 0x55ae1cb672e0 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.624+0000 7f788e1ca740 10 client.27667 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.624+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55ae1cbe66f0 con 0x55ae1cb672e0 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f786cff9640 1 -- 172.21.15.62:0/1912485225 <== mon.1 v2:172.21.15.148:3300/0 4 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f78740053e0 con 0x55ae1cb672e0 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f788e1ca740 10 client.27667 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f788e1ca740 10 client.27667 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f788e1ca740 20 client.27667 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f788e1ca740 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ae1cbe7040 0x55ae1cbe9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f788e1ca740 4 client.27667 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ae1cad1d70 con 0x55ae1cbe7040 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f78890ee640 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ae1cbe7040 0x55ae1cbe9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:55.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f78890ee640 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ae1cbe7040 0x55ae1cbe9460 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=785c1fca6ee793bd server_cookie=afab546d73bd3189 in_seq=0 out_seq=0 2024-09-15T23:28:55.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.625+0000 7f786cff9640 10 client.27667 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:55.640 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f786cff9640 1 -- 172.21.15.62:0/1912485225 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ae1cad1d70 con 0x55ae1cbe7040 2024-09-15T23:28:55.640 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f786cff9640 10 client.27667 handle_command_reply: tid=0 2024-09-15T23:28:55.640 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 client.27667 shutdown 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 msgr2=0x55ae1cb753f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb753f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7874000c00 tx=0x7f7874032ec0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 shutdown_connections 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ae1cbe7040 0x55ae1cbe9460 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ae1cb6c370 0x55ae1cbe5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ae1cb672e0 0x55ae1cb753f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f786cff9640 0 client.27667 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 --2- 172.21.15.62:0/1912485225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ae1cb669a0 0x55ae1cb74eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 >> 172.21.15.62:0/1912485225 conn(0x55ae1cb62010 msgr2=0x55ae1cb70a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 shutdown_connections 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 1 -- 172.21.15.62:0/1912485225 wait complete. 2024-09-15T23:28:55.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.639+0000 7f788e1ca740 20 client.27667 trim_cache size 0 max 16384 2024-09-15T23:28:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:28:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:28:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:28:55.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:28:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 -- 172.21.15.62:0/3541462328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78700a32f0 msgr2=0x7f78700a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78700a32f0 0x7f78700a57b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f78800058c0 tx=0x7f7880005f90 comp rx=0 tx=0).stop 2024-09-15T23:28:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 -- 172.21.15.62:0/3541462328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7884105900 msgr2=0x7f78841a1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7884105900 0x7f78841a1580 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f787c009fd0 tx=0x7f787c00ef10 comp rx=0 tx=0).stop 2024-09-15T23:28:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 -- 172.21.15.62:0/3541462328 shutdown_connections 2024-09-15T23:28:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78700a32f0 0x7f78700a57b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7884105900 0x7f78841a1580 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7884104f40 0x7f788419a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 --2- 172.21.15.62:0/3541462328 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7884103d50 0x7f7884199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:55.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 -- 172.21.15.62:0/3541462328 >> 172.21.15.62:0/3541462328 conn(0x7f78840ff4b0 msgr2=0x7f7884108290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:55.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 -- 172.21.15.62:0/3541462328 shutdown_connections 2024-09-15T23:28:55.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:28:55.640+0000 7f788b352640 1 -- 172.21.15.62:0/3541462328 wait complete. 2024-09-15T23:28:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:55 smithi067 ceph-mon[35665]: pgmap v749: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.9 MiB/s wr, 645 op/s 2024-09-15T23:28:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:55 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/995755052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:55.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:55 smithi148 ceph-mon[35610]: pgmap v749: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 5.9 MiB/s wr, 645 op/s 2024-09-15T23:28:55.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:55 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/995755052' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:28:56.387 INFO:tasks.cephfs.filesystem:scrub status for tag:f28c7295-f7ae-47d5-b6db-b9563604d77f - {'path': '/', 'tag': 'f28c7295-f7ae-47d5-b6db-b9563604d77f', 'options': 'recursive,force'} 2024-09-15T23:28:56.797 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 131.4 --await-for 45.0' 2024-09-15T23:28:56.798 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 131.4 --await-for 45.0 2024-09-15T23:28:57.325 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:28:57.640 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:57 smithi062 ceph-mon[30949]: pgmap v750: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.4 MiB/s wr, 700 op/s 2024-09-15T23:28:57.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:57 smithi067 ceph-mon[35665]: pgmap v750: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.4 MiB/s wr, 700 op/s 2024-09-15T23:28:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:57 smithi148 ceph-mon[35610]: pgmap v750: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 6.4 MiB/s wr, 700 op/s 2024-09-15T23:28:57.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.913+0000 7f785ded6640 1 -- 172.21.15.62:0/2178368947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7858103710 msgr2=0x7f785810bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:57.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.913+0000 7f785ded6640 1 --2- 172.21.15.62:0/2178368947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7858103710 0x7f785810bc00 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f784c009900 tx=0x7f784c02a480 comp rx=0 tx=0).stop 2024-09-15T23:28:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.914+0000 7f785ded6640 1 -- 172.21.15.62:0/2178368947 shutdown_connections 2024-09-15T23:28:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.914+0000 7f785ded6640 1 --2- 172.21.15.62:0/2178368947 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f785810c190 0x7f785810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.914+0000 7f785ded6640 1 --2- 172.21.15.62:0/2178368947 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7858103710 0x7f785810bc00 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.914+0000 7f785ded6640 1 --2- 172.21.15.62:0/2178368947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7858102df0 0x7f78581031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.914+0000 7f785ded6640 1 -- 172.21.15.62:0/2178368947 >> 172.21.15.62:0/2178368947 conn(0x7f78580fc7d0 msgr2=0x7f78580febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:57.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.914+0000 7f785ded6640 1 -- 172.21.15.62:0/2178368947 shutdown_connections 2024-09-15T23:28:57.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.914+0000 7f785ded6640 1 -- 172.21.15.62:0/2178368947 wait complete. 2024-09-15T23:28:57.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.915+0000 7f785ded6640 1 Processor -- start 2024-09-15T23:28:57.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f785ded6640 1 -- start start 2024-09-15T23:28:57.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f785ded6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7858102df0 0x7f78581984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f785ded6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7858103710 0x7f78581989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f785ded6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785810c190 0x7f785819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f785ded6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7858111fd0 con 0x7f7858103710 2024-09-15T23:28:57.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f785ded6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7858111e50 con 0x7f785810c190 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f785ded6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7858112150 con 0x7f7858102df0 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f78577fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7858102df0 0x7f78581984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f78577fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7858102df0 0x7f78581984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:33928/0 (socket says 172.21.15.62:33928) 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f78577fe640 1 -- 172.21.15.62:0/4063278721 learned_addr learned my addr 172.21.15.62:0/4063278721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f7856ffd640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7858103710 0x7f78581989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.916+0000 7f7857fff640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785810c190 0x7f785819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f78577fe640 1 -- 172.21.15.62:0/4063278721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785810c190 msgr2=0x7f785819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f78577fe640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785810c190 0x7f785819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f78577fe640 1 -- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7858103710 msgr2=0x7f78581989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f78577fe640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7858103710 0x7f78581989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f78577fe640 1 -- 172.21.15.62:0/4063278721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785819d460 con 0x7f7858102df0 2024-09-15T23:28:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f7857fff640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785810c190 0x7f785819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:28:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f78577fe640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7858102df0 0x7f78581984b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f784800e9f0 tx=0x7f784800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f7854ff9640 1 -- 172.21.15.62:0/4063278721 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784800cd80 con 0x7f7858102df0 2024-09-15T23:28:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f7854ff9640 1 -- 172.21.15.62:0/4063278721 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848010910 con 0x7f7858102df0 2024-09-15T23:28:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f785ded6640 1 -- 172.21.15.62:0/4063278721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785819d750 con 0x7f7858102df0 2024-09-15T23:28:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.917+0000 7f785ded6640 1 -- 172.21.15.62:0/4063278721 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78581a5030 con 0x7f7858102df0 2024-09-15T23:28:57.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.918+0000 7f7854ff9640 1 -- 172.21.15.62:0/4063278721 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78480188f0 con 0x7f7858102df0 2024-09-15T23:28:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.920+0000 7f7854ff9640 1 -- 172.21.15.62:0/4063278721 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7848018bb0 con 0x7f7858102df0 2024-09-15T23:28:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f7854ff9640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78400a32c0 0x7f78400a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f78602af740 1 Processor -- start 2024-09-15T23:28:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f78602af740 1 -- start start 2024-09-15T23:28:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f7856ffd640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78400a32c0 0x7f78400a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f78602af740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540ee8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f78602af740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee9080 0x558540ef3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f78602af740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540ef5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f78602af740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558540e4e2a0 con 0x558540ef3940 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f78602af740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558540df5b10 con 0x558540ee8760 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.921+0000 7f78602af740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558540e4a260 con 0x558540ee9080 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f78577fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540ee8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f78577fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540ee8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:51466/0 (socket says 172.21.15.62:51466) 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f78577fe640 1 -- 172.21.15.62:0/1525003329 learned_addr learned my addr 172.21.15.62:0/1525003329 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f7857fff640 1 --2- 172.21.15.62:0/1525003329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540ef5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f7856ffd640 1 --2- 172.21.15.62:0/1525003329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee9080 0x558540ef3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f78577fe640 1 -- 172.21.15.62:0/1525003329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee9080 msgr2=0x558540ef3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f78577fe640 1 --2- 172.21.15.62:0/1525003329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee9080 0x558540ef3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f78577fe640 1 -- 172.21.15.62:0/1525003329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 msgr2=0x558540ef5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f78577fe640 1 --2- 172.21.15.62:0/1525003329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540ef5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f78577fe640 1 -- 172.21.15.62:0/1525003329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558540bbfb90 con 0x558540ee8760 2024-09-15T23:28:57.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f7854ff9640 1 -- 172.21.15.62:0/4063278721 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78480cfed0 con 0x7f7858102df0 2024-09-15T23:28:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f78577fe640 1 --2- 172.21.15.62:0/1525003329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540ee8b40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f784800e9c0 tx=0x7f7848013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce17eeabd210659c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f783dffb640 1 -- 172.21.15.62:0/1525003329 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7848026040 con 0x558540ee8760 2024-09-15T23:28:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f7856ffd640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78400a32c0 0x7f78400a5780 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f784c000c00 tx=0x7f784c004360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.922+0000 7f783dffb640 1 -- 172.21.15.62:0/1525003329 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848010430 con 0x558540ee8760 2024-09-15T23:28:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.923+0000 7f783dffb640 1 -- 172.21.15.62:0/1525003329 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784809d2d0 con 0x558540ee8760 2024-09-15T23:28:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.923+0000 7f78602af740 1 -- 172.21.15.62:0/1525003329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 msgr2=0x558540ee8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:57.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.923+0000 7f78602af740 1 --2- 172.21.15.62:0/1525003329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540ee8b40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f784800e9c0 tx=0x7f7848013000 comp rx=0 tx=0).stop 2024-09-15T23:28:57.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.924+0000 7f78602af740 1 -- 172.21.15.62:0/1525003329 shutdown_connections 2024-09-15T23:28:57.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.924+0000 7f78602af740 1 --2- 172.21.15.62:0/1525003329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540ef5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.924+0000 7f78602af740 1 --2- 172.21.15.62:0/1525003329 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee9080 0x558540ef3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.924+0000 7f78602af740 1 --2- 172.21.15.62:0/1525003329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540ee8b40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.924+0000 7f78602af740 1 -- 172.21.15.62:0/1525003329 >> 172.21.15.62:0/1525003329 conn(0x558540ee4cf0 msgr2=0x558540ee50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.926+0000 7f78602af740 1 -- 172.21.15.62:0/1525003329 shutdown_connections 2024-09-15T23:28:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.926+0000 7f78602af740 1 -- 172.21.15.62:0/1525003329 wait complete. 2024-09-15T23:28:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.927+0000 7f78602af740 1 -- start start 2024-09-15T23:28:57.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.927+0000 7f78602af740 1 Processor -- start 2024-09-15T23:28:57.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.927+0000 7f78602af740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee8760 0x558540ef0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.927+0000 7f78602af740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540ef0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f78602af740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ef3940 0x558540ef1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f78602af740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558540ef6510 con 0x558540ee9080 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f78602af740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558540dd66e0 con 0x558540ef3940 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f78602af740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558540ef6690 con 0x558540ee8760 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f78602af740 10 client.0 _ll_register_callbacks cb 0x558540dfaa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f7856ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540ef0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f7856ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540ef0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48102/0 (socket says 172.21.15.62:48102) 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f7856ffd640 1 -- 172.21.15.62:0/3201312493 learned_addr learned my addr 172.21.15.62:0/3201312493 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f7857fff640 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ef3940 0x558540ef1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f7856ffd640 1 -- 172.21.15.62:0/3201312493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee8760 msgr2=0x558540ef0810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:28:57.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f7856ffd640 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee8760 0x558540ef0810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f7856ffd640 1 -- 172.21.15.62:0/3201312493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ef3940 msgr2=0x558540ef1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:57.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f7856ffd640 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ef3940 0x558540ef1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.928+0000 7f7856ffd640 1 -- 172.21.15.62:0/3201312493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558540effa60 con 0x558540ee9080 2024-09-15T23:28:57.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.930+0000 7f7856ffd640 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540ef0d50 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f784c033000 tx=0x7f784c02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c865a359f0024ef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.935+0000 7f7836ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.935+0000 7f7836ffd640 1 -- 172.21.15.62:0/3201312493 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c053070 con 0x558540ee9080 2024-09-15T23:28:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.935+0000 7f78602af740 1 -- 172.21.15.62:0/3201312493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7848010430 con 0x558540ee9080 2024-09-15T23:28:57.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.935+0000 7f7836ffd640 1 -- 172.21.15.62:0/3201312493 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f784c041cd0 con 0x558540ee9080 2024-09-15T23:28:57.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.938+0000 7f7836ffd640 1 -- 172.21.15.62:0/3201312493 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f784c05d050 con 0x558540ee9080 2024-09-15T23:28:57.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.939+0000 7f78602af740 10 client.18411 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:57.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.939+0000 7f78602af740 1 -- 172.21.15.62:0/3201312493 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558540f02400 con 0x558540ee9080 2024-09-15T23:28:57.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.949+0000 7f7836ffd640 1 -- 172.21.15.62:0/3201312493 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7848010430 con 0x558540ee9080 2024-09-15T23:28:57.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.949+0000 7f7836ffd640 1 -- 172.21.15.62:0/3201312493 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f784c032820 con 0x558540ee9080 2024-09-15T23:28:57.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.949+0000 7f78602af740 10 client.18411 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:57.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.949+0000 7f78602af740 10 client.18411 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:28:57.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.949+0000 7f78602af740 20 client.18411 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:57.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.949+0000 7f78602af740 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558540f038c0 0x558540f05d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:57.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.949+0000 7f78602af740 4 client.18411 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:28:57.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.949+0000 7f78602af740 1 -- 172.21.15.62:0/3201312493 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558540e5b9d0 con 0x558540f038c0 2024-09-15T23:28:57.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.950+0000 7f78577fe640 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558540f038c0 0x558540f05d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:57.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.958+0000 7f78577fe640 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558540f038c0 0x558540f05d60 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2e8ef8ec9eddefc server_cookie=5baec46cef09f326 in_seq=0 out_seq=0 2024-09-15T23:28:57.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.959+0000 7f7836ffd640 10 client.18411 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:57.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.959+0000 7f7836ffd640 1 -- 172.21.15.62:0/3201312493 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558540e5b9d0 con 0x558540f038c0 2024-09-15T23:28:57.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.960+0000 7f7836ffd640 10 client.18411 handle_command_reply: tid=0 2024-09-15T23:28:57.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.961+0000 7f78602af740 1 client.18411 shutdown 2024-09-15T23:28:57.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.961+0000 7f78602af740 1 -- 172.21.15.62:0/3201312493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 msgr2=0x558540ef0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:57.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.961+0000 7f78602af740 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540ef0d50 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f784c033000 tx=0x7f784c02a480 comp rx=0 tx=0).stop 2024-09-15T23:28:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.964+0000 7f78602af740 1 -- 172.21.15.62:0/3201312493 shutdown_connections 2024-09-15T23:28:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.964+0000 7f78602af740 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558540f038c0 0x558540f05d60 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.964+0000 7f78602af740 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ef3940 0x558540ef1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.964+0000 7f78602af740 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540ef0d50 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.964+0000 7f78602af740 1 --2- 172.21.15.62:0/3201312493 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee8760 0x558540ef0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.964+0000 7f78602af740 1 -- 172.21.15.62:0/3201312493 >> 172.21.15.62:0/3201312493 conn(0x558540ee4cf0 msgr2=0x558540eeaa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:57.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.965+0000 7f7836ffd640 0 client.18411 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:57.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.969+0000 7f78602af740 1 -- 172.21.15.62:0/3201312493 shutdown_connections 2024-09-15T23:28:57.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.969+0000 7f78602af740 1 -- 172.21.15.62:0/3201312493 wait complete. 2024-09-15T23:28:57.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:57.969+0000 7f78602af740 20 client.18411 trim_cache size 0 max 16384 2024-09-15T23:28:58.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.002+0000 7f78602af740 1 Processor -- start 2024-09-15T23:28:58.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.006+0000 7f78602af740 1 -- start start 2024-09-15T23:28:58.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.006+0000 7f78602af740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee8760 0x558540f0b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.006+0000 7f78602af740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee9080 0x558540f0bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.006+0000 7f78602af740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540eeda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.006+0000 7f78602af740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7844012500 con 0x558540ef3940 2024-09-15T23:28:58.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.006+0000 7f78602af740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f784400fdd0 con 0x558540ee9080 2024-09-15T23:28:58.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.006+0000 7f78602af740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f784400dc20 con 0x558540ee8760 2024-09-15T23:28:58.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.008+0000 7f7857fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540eeda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.008+0000 7f7857fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540eeda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48110/0 (socket says 172.21.15.62:48110) 2024-09-15T23:28:58.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.008+0000 7f7857fff640 1 -- 172.21.15.62:0/877443584 learned_addr learned my addr 172.21.15.62:0/877443584 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:58.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.009+0000 7f7856ffd640 1 --2- 172.21.15.62:0/877443584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee9080 0x558540f0bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.010+0000 7f78577fe640 1 --2- 172.21.15.62:0/877443584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee8760 0x558540f0b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.013+0000 7f7857fff640 1 -- 172.21.15.62:0/877443584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee8760 msgr2=0x558540f0b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.013+0000 7f7857fff640 1 --2- 172.21.15.62:0/877443584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee8760 0x558540f0b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.013+0000 7f7857fff640 1 -- 172.21.15.62:0/877443584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee9080 msgr2=0x558540f0bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.013+0000 7f7857fff640 1 --2- 172.21.15.62:0/877443584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee9080 0x558540f0bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.013+0000 7f7857fff640 1 -- 172.21.15.62:0/877443584 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558540eeb5d0 con 0x558540ef3940 2024-09-15T23:28:58.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.018+0000 7f7857fff640 1 --2- 172.21.15.62:0/877443584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540eeda70 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f7850010910 tx=0x7f7850010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b12c24a2cda0f86 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:58.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.023+0000 7f783d7fa640 1 -- 172.21.15.62:0/877443584 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7850021070 con 0x558540ef3940 2024-09-15T23:28:58.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.023+0000 7f783d7fa640 1 -- 172.21.15.62:0/877443584 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f785000fe40 con 0x558540ef3940 2024-09-15T23:28:58.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.023+0000 7f783d7fa640 1 -- 172.21.15.62:0/877443584 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7850004800 con 0x558540ef3940 2024-09-15T23:28:58.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.025+0000 7f78602af740 1 -- 172.21.15.62:0/877443584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 msgr2=0x558540eeda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.025+0000 7f78602af740 1 --2- 172.21.15.62:0/877443584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540eeda70 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f7850010910 tx=0x7f7850010dd0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.032+0000 7f78602af740 1 -- 172.21.15.62:0/877443584 shutdown_connections 2024-09-15T23:28:58.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.032+0000 7f78602af740 1 --2- 172.21.15.62:0/877443584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ef3940 0x558540eeda70 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.032+0000 7f78602af740 1 --2- 172.21.15.62:0/877443584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee9080 0x558540f0bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.032+0000 7f78602af740 1 --2- 172.21.15.62:0/877443584 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ee8760 0x558540f0b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.032+0000 7f78602af740 1 -- 172.21.15.62:0/877443584 >> 172.21.15.62:0/877443584 conn(0x558540ee4cf0 msgr2=0x558540ef2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:58.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.037+0000 7f78602af740 1 -- 172.21.15.62:0/877443584 shutdown_connections 2024-09-15T23:28:58.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.038+0000 7f78602af740 1 -- 172.21.15.62:0/877443584 wait complete. 2024-09-15T23:28:58.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.038+0000 7f78602af740 1 -- start start 2024-09-15T23:28:58.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.039+0000 7f78602af740 1 Processor -- start 2024-09-15T23:28:58.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.039+0000 7f78602af740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540f728a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.039+0000 7f78602af740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540efb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.039+0000 7f78602af740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ef3940 0x558540efc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.039+0000 7f78602af740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f784400e290 con 0x558540ee9080 2024-09-15T23:28:58.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.039+0000 7f78602af740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f784400f700 con 0x558540ee8760 2024-09-15T23:28:58.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.039+0000 7f78602af740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f784400e960 con 0x558540ef3940 2024-09-15T23:28:58.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.040+0000 7f78602af740 10 client.0 _ll_register_callbacks cb 0x7f784c004140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:28:58.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.041+0000 7f78577fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540f728a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.041+0000 7f7857fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ef3940 0x558540efc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.044+0000 7f7856ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540efb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.044+0000 7f7856ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540efb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48122/0 (socket says 172.21.15.62:48122) 2024-09-15T23:28:58.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.044+0000 7f7856ffd640 1 -- 172.21.15.62:0/2664338686 learned_addr learned my addr 172.21.15.62:0/2664338686 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:28:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.045+0000 7f7856ffd640 1 -- 172.21.15.62:0/2664338686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ef3940 msgr2=0x558540efc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.045+0000 7f7856ffd640 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ef3940 0x558540efc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.046+0000 7f7856ffd640 1 -- 172.21.15.62:0/2664338686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 msgr2=0x558540f728a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:28:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.046+0000 7f7856ffd640 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540f728a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.046+0000 7f7856ffd640 1 -- 172.21.15.62:0/2664338686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558540f6fcf0 con 0x558540ee9080 2024-09-15T23:28:58.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.046+0000 7f7856ffd640 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540efb940 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f784c0096c0 tx=0x7f784c032b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3598370f622d9341 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:28:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.049+0000 7f7836ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:28:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.049+0000 7f7836ffd640 1 -- 172.21.15.62:0/2664338686 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c040020 con 0x558540ee9080 2024-09-15T23:28:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.049+0000 7f7836ffd640 1 -- 172.21.15.62:0/2664338686 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f784c041420 con 0x558540ee9080 2024-09-15T23:28:58.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.049+0000 7f78602af740 1 -- 172.21.15.62:0/2664338686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f785000fe40 con 0x558540ee9080 2024-09-15T23:28:58.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.053+0000 7f7836ffd640 1 -- 172.21.15.62:0/2664338686 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f784c053030 con 0x558540ee9080 2024-09-15T23:28:58.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.053+0000 7f78602af740 10 client.18417 fetch_fsmap learned FSMap version 35 2024-09-15T23:28:58.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.053+0000 7f78602af740 1 -- 172.21.15.62:0/2664338686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558540f66920 con 0x558540ee9080 2024-09-15T23:28:58.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.060+0000 7f7836ffd640 1 -- 172.21.15.62:0/2664338686 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f784c032e30 con 0x558540ee9080 2024-09-15T23:28:58.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.060+0000 7f7836ffd640 1 -- 172.21.15.62:0/2664338686 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f784c0049c0 con 0x558540ee9080 2024-09-15T23:28:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.060+0000 7f78602af740 10 client.18417 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:28:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.060+0000 7f78602af740 10 client.18417 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:28:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.060+0000 7f78602af740 20 client.18417 populate_metadata read hostname 'smithi062' 2024-09-15T23:28:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.060+0000 7f78602af740 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558540f67140 0x558540f69560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:28:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.060+0000 7f78602af740 4 client.18417 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.4, "await_for": 45.0}] 2024-09-15T23:28:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.060+0000 7f78602af740 1 -- 172.21.15.62:0/2664338686 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 131.4, "await_for": 45.0}) -- 0x7f785000fe40 con 0x558540f67140 2024-09-15T23:28:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.061+0000 7f78577fe640 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558540f67140 0x558540f69560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:28:58.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.063+0000 7f78577fe640 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558540f67140 0x558540f69560 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=853b79dc728f6372 server_cookie=73beb9128dd7b5d1 in_seq=0 out_seq=0 2024-09-15T23:28:58.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.066+0000 7f7836ffd640 10 client.18417 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f7836ffd640 1 -- 172.21.15.62:0/2664338686 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x7f785000fe40 con 0x558540f67140 2024-09-15T23:28:58.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f7836ffd640 10 client.18417 handle_command_reply: tid=0 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 client.18417 shutdown 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 -- 172.21.15.62:0/2664338686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 msgr2=0x558540efb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540efb940 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f784c0096c0 tx=0x7f784c032b60 comp rx=0 tx=0).stop 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 -- 172.21.15.62:0/2664338686 shutdown_connections 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558540f67140 0x558540f69560 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558540ef3940 0x558540efc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f7836ffd640 0 client.18417 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558540ee9080 0x558540efb940 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 --2- 172.21.15.62:0/2664338686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558540ee8760 0x558540f728a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 -- 172.21.15.62:0/2664338686 >> 172.21.15.62:0/2664338686 conn(0x558540ee4cf0 msgr2=0x558540eea1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 -- 172.21.15.62:0/2664338686 shutdown_connections 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 1 -- 172.21.15.62:0/2664338686 wait complete. 2024-09-15T23:28:58.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.413+0000 7f78602af740 20 client.18417 trim_cache size 0 max 16384 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 35, 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 46, 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "78d00243": { 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 46, 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:28:58.416 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 131.4, 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:28:58.417 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:28:58.418 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:28:58.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 -- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78400a32c0 msgr2=0x7f78400a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78400a32c0 0x7f78400a5780 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f784c000c00 tx=0x7f784c004360 comp rx=0 tx=0).stop 2024-09-15T23:28:58.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 -- 172.21.15.62:0/4063278721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7858102df0 msgr2=0x7f78581984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:28:58.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7858102df0 0x7f78581984b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f784800e9f0 tx=0x7f784800eeb0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 -- 172.21.15.62:0/4063278721 shutdown_connections 2024-09-15T23:28:58.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78400a32c0 0x7f78400a5780 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785810c190 0x7f785819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7858103710 0x7f78581989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 --2- 172.21.15.62:0/4063278721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7858102df0 0x7f78581984b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:28:58.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 -- 172.21.15.62:0/4063278721 >> 172.21.15.62:0/4063278721 conn(0x7f78580fc7d0 msgr2=0x7f78580fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:28:58.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 -- 172.21.15.62:0/4063278721 shutdown_connections 2024-09-15T23:28:58.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:28:58.414+0000 7f785ded6640 1 -- 172.21.15.62:0/4063278721 wait complete. 2024-09-15T23:28:59.571 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 760.0 s, 401.5 tps, lat 196.430 ms stddev 223.977 2024-09-15T23:28:59.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:28:59 smithi062 ceph-mon[30949]: pgmap v751: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 5.4 MiB/s wr, 602 op/s 2024-09-15T23:28:59.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:28:59 smithi067 ceph-mon[35665]: pgmap v751: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 5.4 MiB/s wr, 602 op/s 2024-09-15T23:28:59.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:28:59 smithi148 ceph-mon[35610]: pgmap v751: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 5.4 MiB/s wr, 602 op/s 2024-09-15T23:28:59.805 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '78d00243', quiesce took 9.7 seconds. Will release after: 1.700000000000001 2024-09-15T23:29:01.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:29:01.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:01 smithi062 ceph-mon[30949]: pgmap v752: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.2 MiB/s wr, 618 op/s 2024-09-15T23:29:01.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:01 smithi067 ceph-mon[35665]: pgmap v752: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.2 MiB/s wr, 618 op/s 2024-09-15T23:29:03.037 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:01 smithi148 ceph-mon[35610]: pgmap v752: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.2 MiB/s wr, 618 op/s 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.202+0000 7f2977a97640 1 -- 172.21.15.62:0/3958597304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970102790 msgr2=0x7f29701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.202+0000 7f2977a97640 1 --2- 172.21.15.62:0/3958597304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970102790 0x7f29701067e0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f2960009900 tx=0x7f296002a410 comp rx=0 tx=0).stop 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.202+0000 7f2977a97640 1 -- 172.21.15.62:0/3958597304 shutdown_connections 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.202+0000 7f2977a97640 1 --2- 172.21.15.62:0/3958597304 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2970106f10 0x7f297010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.202+0000 7f2977a97640 1 --2- 172.21.15.62:0/3958597304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970102790 0x7f29701067e0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.202+0000 7f2977a97640 1 --2- 172.21.15.62:0/3958597304 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2970101de0 0x7f29701021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.202+0000 7f2977a97640 1 -- 172.21.15.62:0/3958597304 >> 172.21.15.62:0/3958597304 conn(0x7f29700fd8c0 msgr2=0x7f29700ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.202+0000 7f2977a97640 1 -- 172.21.15.62:0/3958597304 shutdown_connections 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.202+0000 7f2977a97640 1 -- 172.21.15.62:0/3958597304 wait complete. 2024-09-15T23:29:03.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2977a97640 1 Processor -- start 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2977a97640 1 -- start start 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2977a97640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970101de0 0x7f29700744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2977a97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2970102790 0x7f2970074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2977a97640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2970106f10 0x7f2970074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2977a97640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2970111f70 con 0x7f2970101de0 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2977a97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2970111df0 con 0x7f2970102790 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2977a97640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f29701120f0 con 0x7f2970106f10 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2975833640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970101de0 0x7f29700744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2975833640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970101de0 0x7f29700744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48132/0 (socket says 172.21.15.62:48132) 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.203+0000 7f2975833640 1 -- 172.21.15.62:0/86055352 learned_addr learned my addr 172.21.15.62:0/86055352 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2976034640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2970106f10 0x7f2970074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2975032640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2970102790 0x7f2970074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2975833640 1 -- 172.21.15.62:0/86055352 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2970106f10 msgr2=0x7f2970074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2975833640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2970106f10 0x7f2970074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2975833640 1 -- 172.21.15.62:0/86055352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2970102790 msgr2=0x7f2970074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2975833640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2970102790 0x7f2970074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2975833640 1 -- 172.21.15.62:0/86055352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29701a6870 con 0x7f2970101de0 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2976034640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2970106f10 0x7f2970074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2975032640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2970102790 0x7f2970074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2975833640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970101de0 0x7f29700744f0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f296c00c890 tx=0x7f296c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2966ffd640 1 -- 172.21.15.62:0/86055352 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c004220 con 0x7f2970101de0 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2966ffd640 1 -- 172.21.15.62:0/86055352 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f296c0043c0 con 0x7f2970101de0 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29701a6b60 con 0x7f2970101de0 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2966ffd640 1 -- 172.21.15.62:0/86055352 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c0049e0 con 0x7f2970101de0 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.204+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29701a70f0 con 0x7f2970101de0 2024-09-15T23:29:03.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.205+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29701021c0 con 0x7f2970101de0 2024-09-15T23:29:03.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.206+0000 7f2966ffd640 1 -- 172.21.15.62:0/86055352 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f296c0079b0 con 0x7f2970101de0 2024-09-15T23:29:03.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.207+0000 7f2966ffd640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f295c0a32c0 0x7f295c0a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.207+0000 7f2975032640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f295c0a32c0 0x7f295c0a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.207+0000 7f2966ffd640 1 -- 172.21.15.62:0/86055352 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f296c0cfd10 con 0x7f2970101de0 2024-09-15T23:29:03.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.208+0000 7f2975032640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f295c0a32c0 0x7f295c0a5780 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2960000c00 tx=0x7f2960032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.212+0000 7f2966ffd640 1 -- 172.21.15.62:0/86055352 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f296c014030 con 0x7f2970101de0 2024-09-15T23:29:03.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.424+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2970075a20 con 0x7f2970101de0 2024-09-15T23:29:03.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.425+0000 7f2966ffd640 1 -- 172.21.15.62:0/86055352 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7f296c09c6b0 con 0x7f2970101de0 2024-09-15T23:29:03.041 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:29:03.042 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:29:03.042 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:29:03.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f295c0a32c0 msgr2=0x7f295c0a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f295c0a32c0 0x7f295c0a5780 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2960000c00 tx=0x7f2960032d50 comp rx=0 tx=0).stop 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970101de0 msgr2=0x7f29700744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970101de0 0x7f29700744f0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f296c00c890 tx=0x7f296c00cd50 comp rx=0 tx=0).stop 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 shutdown_connections 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f295c0a32c0 0x7f295c0a5780 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2970106f10 0x7f2970074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2970102790 0x7f2970074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 --2- 172.21.15.62:0/86055352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2970101de0 0x7f29700744f0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 >> 172.21.15.62:0/86055352 conn(0x7f29700fd8c0 msgr2=0x7f29700fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 shutdown_connections 2024-09-15T23:29:03.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:02.427+0000 7f2977a97640 1 -- 172.21.15.62:0/86055352 wait complete. 2024-09-15T23:29:03.105 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 78d00243 --release --await' 2024-09-15T23:29:03.106 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 78d00243 --release --await 2024-09-15T23:29:03.362 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:03.634 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:03 smithi062 ceph-mon[30949]: pgmap v753: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 4.5 MiB/s wr, 550 op/s 2024-09-15T23:29:03.634 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:03 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:29:03.634 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:03 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:29:03.634 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:03 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:29:03.634 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:03 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/86055352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:03.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:03 smithi067 ceph-mon[35665]: pgmap v753: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 4.5 MiB/s wr, 550 op/s 2024-09-15T23:29:03.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:03 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:29:03.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:03 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:29:03.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:03 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:29:03.776 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:03 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/86055352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:03 smithi148 ceph-mon[35610]: pgmap v753: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 4.5 MiB/s wr, 550 op/s 2024-09-15T23:29:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:03 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:29:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:03 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:29:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:03 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:29:03.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:03 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/86055352' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:03.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.859+0000 7fa9c0207640 1 -- 172.21.15.62:0/3935079535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 msgr2=0x7fa9b810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.859+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3935079535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 0x7fa9b810ea90 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0009ca0 tx=0x7fa9b002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.859+0000 7fa9c0207640 1 -- 172.21.15.62:0/3935079535 shutdown_connections 2024-09-15T23:29:03.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.859+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3935079535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 0x7fa9b810ea90 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.859+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3935079535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9b8102790 0x7fa9b81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.859+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3935079535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b8101de0 0x7fa9b81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.859+0000 7fa9c0207640 1 -- 172.21.15.62:0/3935079535 >> 172.21.15.62:0/3935079535 conn(0x7fa9b80fd8c0 msgr2=0x7fa9b80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.859+0000 7fa9c0207640 1 -- 172.21.15.62:0/3935079535 shutdown_connections 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.859+0000 7fa9c0207640 1 -- 172.21.15.62:0/3935079535 wait complete. 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9c0207640 1 Processor -- start 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9c0207640 1 -- start start 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9c0207640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9b8101de0 0x7fa9b81983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9c0207640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b8102790 0x7fa9b81988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9c0207640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 0x7fa9b819cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9c0207640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b8111f70 con 0x7fa9b8106f10 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9c0207640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b8111df0 con 0x7fa9b8102790 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9c0207640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b81120f0 con 0x7fa9b8101de0 2024-09-15T23:29:03.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9be7a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 0x7fa9b819cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9be7a4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 0x7fa9b819cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44402/0 (socket says 172.21.15.62:44402) 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9be7a4640 1 -- 172.21.15.62:0/3425446856 learned_addr learned my addr 172.21.15.62:0/3425446856 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.860+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b8102790 0x7fa9b81988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9bdfa3640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9b8101de0 0x7fa9b81983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9be7a4640 1 -- 172.21.15.62:0/3425446856 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9b8101de0 msgr2=0x7fa9b81983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9be7a4640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9b8101de0 0x7fa9b81983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9be7a4640 1 -- 172.21.15.62:0/3425446856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b8102790 msgr2=0x7fa9b81988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9be7a4640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b8102790 0x7fa9b81988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9be7a4640 1 -- 172.21.15.62:0/3425446856 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b819d360 con 0x7fa9b8106f10 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9be7a4640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 0x7fa9b819cc80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0031a60 tx=0x7fa9b0031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9aeffd640 1 -- 172.21.15.62:0/3425446856 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0003c30 con 0x7fa9b8106f10 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9aeffd640 1 -- 172.21.15.62:0/3425446856 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9b0003dd0 con 0x7fa9b8106f10 2024-09-15T23:29:03.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9c0207640 1 -- 172.21.15.62:0/3425446856 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b819d5f0 con 0x7fa9b8106f10 2024-09-15T23:29:03.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9aeffd640 1 -- 172.21.15.62:0/3425446856 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b00047e0 con 0x7fa9b8106f10 2024-09-15T23:29:03.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.861+0000 7fa9c0207640 1 -- 172.21.15.62:0/3425446856 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b81a4f30 con 0x7fa9b8106f10 2024-09-15T23:29:03.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9c25e0740 1 Processor -- start 2024-09-15T23:29:03.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9aeffd640 1 -- 172.21.15.62:0/3425446856 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa9b002f790 con 0x7fa9b8106f10 2024-09-15T23:29:03.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9c25e0740 1 -- start start 2024-09-15T23:29:03.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20060 0x55ed3ac20440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 0x55ed3ac2ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac2b240 0x55ed3ac2d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ed3ab85620 con 0x55ed3ac20060 2024-09-15T23:29:03.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ed3ab2ce50 con 0x55ed3ac20980 2024-09-15T23:29:03.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ed3ab815e0 con 0x55ed3ac2b240 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9bd7a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20060 0x55ed3ac20440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9bd7a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20060 0x55ed3ac20440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44410/0 (socket says 172.21.15.62:44410) 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.864+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/3399384595 learned_addr learned my addr 172.21.15.62:0/3399384595 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9bdfa3640 1 --2- 172.21.15.62:0/3399384595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 0x55ed3ac2ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9aeffd640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9a40a32c0 0x7fa9a40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/3399384595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac2b240 0x55ed3ac2d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/3399384595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac2b240 msgr2=0x55ed3ac2d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/3399384595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac2b240 0x55ed3ac2d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/3399384595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 msgr2=0x55ed3ac2ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/3399384595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 0x55ed3ac2ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/3399384595 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed3ab1dd40 con 0x55ed3ac20060 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9be7a4640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9a40a32c0 0x7fa9a40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/3399384595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20060 0x55ed3ac20440 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8005940 tx=0x7fa9a800a700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d16908bbc7d821a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9aeffd640 1 -- 172.21.15.62:0/3425446856 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa9b00f0f90 con 0x7fa9b8106f10 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9a3fff640 1 -- 172.21.15.62:0/3399384595 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8005020 con 0x55ed3ac20060 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9a3fff640 1 -- 172.21.15.62:0/3399384595 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a8017030 con 0x55ed3ac20060 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.865+0000 7fa9a3fff640 1 -- 172.21.15.62:0/3399384595 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8012450 con 0x55ed3ac20060 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9c25e0740 1 -- 172.21.15.62:0/3399384595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20060 msgr2=0x55ed3ac20440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9be7a4640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9a40a32c0 0x7fa9a40a5780 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00bd2e0 tx=0x7fa9b00c1040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/3399384595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20060 0x55ed3ac20440 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8005940 tx=0x7fa9a800a700 comp rx=0 tx=0).stop 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9c25e0740 1 -- 172.21.15.62:0/3399384595 shutdown_connections 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/3399384595 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac2b240 0x55ed3ac2d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/3399384595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 0x55ed3ac2ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/3399384595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20060 0x55ed3ac20440 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9c25e0740 1 -- 172.21.15.62:0/3399384595 >> 172.21.15.62:0/3399384595 conn(0x55ed3ac1c5f0 msgr2=0x55ed3ac1c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9c25e0740 1 -- 172.21.15.62:0/3399384595 shutdown_connections 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.866+0000 7fa9c25e0740 1 -- 172.21.15.62:0/3399384595 wait complete. 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9c25e0740 1 -- start start 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9c25e0740 1 Processor -- start 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 0x55ed3ac280b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 0x55ed3ac285f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac28b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ed3ac2de10 con 0x55ed3ac2b240 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ed3ab0df10 con 0x55ed3ac20980 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ed3ac2df90 con 0x55ed3ac20060 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9c25e0740 10 client.0 _ll_register_callbacks cb 0x55ed3ab31db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9bd7a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac28b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9bdfa3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 0x55ed3ac285f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9bd7a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac28b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44412/0 (socket says 172.21.15.62:44412) 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.867+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/172061520 learned_addr learned my addr 172.21.15.62:0/172061520 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 0x55ed3ac280b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/172061520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 msgr2=0x55ed3ac280b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 0x55ed3ac280b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/172061520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 msgr2=0x55ed3ac285f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 0x55ed3ac285f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/172061520 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed3a894580 con 0x55ed3ac2b240 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9bdfa3640 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 0x55ed3ac285f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac28b30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8004ec0 tx=0x7fa9a8004b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e3618ab6a563ec server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9a17fa640 1 -- 172.21.15.62:0/172061520 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8004fa0 con 0x55ed3ac2b240 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9a17fa640 1 -- 172.21.15.62:0/172061520 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa9a8005140 con 0x55ed3ac2b240 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9c25e0740 1 -- 172.21.15.62:0/172061520 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9a8017030 con 0x55ed3ac2b240 2024-09-15T23:29:03.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9a17fa640 1 -- 172.21.15.62:0/172061520 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa9a8030050 con 0x55ed3ac2b240 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9c25e0740 10 client.18435 fetch_fsmap learned FSMap version 35 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9c25e0740 1 -- 172.21.15.62:0/172061520 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ed3a95e960 con 0x55ed3ac2b240 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.868+0000 7fa9a17fa640 1 -- 172.21.15.62:0/172061520 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a8017030 con 0x55ed3ac2b240 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9a17fa640 1 -- 172.21.15.62:0/172061520 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa9a801cce0 con 0x55ed3ac2b240 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9c25e0740 10 client.18435 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9c25e0740 10 client.18435 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9c25e0740 20 client.18435 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ed3ac3ac20 0x55ed3ac3d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9c25e0740 4 client.18435 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9c25e0740 1 -- 172.21.15.62:0/172061520 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed3ab92b90 con 0x55ed3ac3ac20 2024-09-15T23:29:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9bdfa3640 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ed3ac3ac20 0x55ed3ac3d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9bdfa3640 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ed3ac3ac20 0x55ed3ac3d010 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aff182f7f1d7855a server_cookie=67d2095e7d875ab in_seq=0 out_seq=0 2024-09-15T23:29:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.869+0000 7fa9a17fa640 10 client.18435 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9a17fa640 1 -- 172.21.15.62:0/172061520 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ed3ab92b90 con 0x55ed3ac3ac20 2024-09-15T23:29:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9a17fa640 10 client.18435 handle_command_reply: tid=0 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9c25e0740 1 client.18435 shutdown 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9c25e0740 1 -- 172.21.15.62:0/172061520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 msgr2=0x55ed3ac28b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac28b30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8004ec0 tx=0x7fa9a8004b10 comp rx=0 tx=0).stop 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9c25e0740 1 -- 172.21.15.62:0/172061520 shutdown_connections 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ed3ac3ac20 0x55ed3ac3d010 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac28b30 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20980 0x55ed3ac285f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9a17fa640 0 client.18435 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/172061520 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 0x55ed3ac280b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.870+0000 7fa9c25e0740 1 -- 172.21.15.62:0/172061520 >> 172.21.15.62:0/172061520 conn(0x55ed3ac1c5f0 msgr2=0x55ed3ac22320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.871+0000 7fa9c25e0740 1 -- 172.21.15.62:0/172061520 shutdown_connections 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.871+0000 7fa9c25e0740 1 -- 172.21.15.62:0/172061520 wait complete. 2024-09-15T23:29:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.871+0000 7fa9c25e0740 20 client.18435 trim_cache size 0 max 16384 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9c25e0740 1 Processor -- start 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9c25e0740 1 -- start start 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20060 0x55ed3ac26f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20980 0x55ed3ac427a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac42ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b81b14b0 con 0x55ed3ac2b240 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b81aed80 con 0x55ed3ac20060 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b81acbd0 con 0x55ed3ac20980 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac42ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac42ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44422/0 (socket says 172.21.15.62:44422) 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/1609248273 learned_addr learned my addr 172.21.15.62:0/1609248273 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/1609248273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20060 0x55ed3ac26f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/1609248273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20980 msgr2=0x55ed3ac427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bdfa3640 1 --2- 172.21.15.62:0/1609248273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20980 0x55ed3ac427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/1609248273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20980 0x55ed3ac427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/1609248273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20060 msgr2=0x55ed3ac26f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/1609248273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20060 0x55ed3ac26f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 -- 172.21.15.62:0/1609248273 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed3ac1c420 con 0x55ed3ac2b240 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.888+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/1609248273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac42ce0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800a3c0 tx=0x7fa9a802a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e035fb4e731b8b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9a37fe640 1 -- 172.21.15.62:0/1609248273 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8011130 con 0x55ed3ac2b240 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9a37fe640 1 -- 172.21.15.62:0/1609248273 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a8017040 con 0x55ed3ac2b240 2024-09-15T23:29:03.891 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9a37fe640 1 -- 172.21.15.62:0/1609248273 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8004280 con 0x55ed3ac2b240 2024-09-15T23:29:03.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9c25e0740 1 -- 172.21.15.62:0/1609248273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 msgr2=0x55ed3ac42ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/1609248273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac42ce0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800a3c0 tx=0x7fa9a802a040 comp rx=0 tx=0).stop 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9c25e0740 1 -- 172.21.15.62:0/1609248273 shutdown_connections 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/1609248273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac2b240 0x55ed3ac42ce0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/1609248273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20980 0x55ed3ac427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/1609248273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac20060 0x55ed3ac26f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9c25e0740 1 -- 172.21.15.62:0/1609248273 >> 172.21.15.62:0/1609248273 conn(0x55ed3ac1c5f0 msgr2=0x55ed3ac26850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9c25e0740 1 -- 172.21.15.62:0/1609248273 shutdown_connections 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.889+0000 7fa9c25e0740 1 -- 172.21.15.62:0/1609248273 wait complete. 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9c25e0740 1 -- start start 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9c25e0740 1 Processor -- start 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 0x55ed3ac33a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20980 0x55ed3ac2fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9c25e0740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac2b240 0x55ed3ac30370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b81ad240 con 0x55ed3ac20980 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b81ae6b0 con 0x55ed3ac2b240 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9c25e0740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b81ad910 con 0x55ed3ac20060 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9c25e0740 10 client.0 _ll_register_callbacks cb 0x7fa9a8004cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9bdfa3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20980 0x55ed3ac2fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9bdfa3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20980 0x55ed3ac2fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44438/0 (socket says 172.21.15.62:44438) 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.890+0000 7fa9bdfa3640 1 -- 172.21.15.62:0/2701046929 learned_addr learned my addr 172.21.15.62:0/2701046929 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac2b240 0x55ed3ac30370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 0x55ed3ac33a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9bdfa3640 1 -- 172.21.15.62:0/2701046929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 msgr2=0x55ed3ac33a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9bdfa3640 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 0x55ed3ac33a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9bdfa3640 1 -- 172.21.15.62:0/2701046929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac2b240 msgr2=0x55ed3ac30370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:03.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9bdfa3640 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac2b240 0x55ed3ac30370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9bdfa3640 1 -- 172.21.15.62:0/2701046929 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed3ac22ed0 con 0x55ed3ac20980 2024-09-15T23:29:03.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac2b240 0x55ed3ac30370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:03.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9bdfa3640 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20980 0x55ed3ac2fd80 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x55ed3ac1d450 tx=0x7fa9b4010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a72d8eb8a5ced3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:03.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:03.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9a17fa640 1 -- 172.21.15.62:0/2701046929 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b402a020 con 0x55ed3ac20980 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9a17fa640 1 -- 172.21.15.62:0/2701046929 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa9b4004220 con 0x55ed3ac20980 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9c25e0740 1 -- 172.21.15.62:0/2701046929 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9a8017040 con 0x55ed3ac20980 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9a17fa640 1 -- 172.21.15.62:0/2701046929 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa9b4032050 con 0x55ed3ac20980 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9c25e0740 10 client.18441 fetch_fsmap learned FSMap version 35 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.891+0000 7fa9c25e0740 1 -- 172.21.15.62:0/2701046929 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ed3ac1d280 con 0x55ed3ac20980 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9a17fa640 1 -- 172.21.15.62:0/2701046929 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a8017040 con 0x55ed3ac20980 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9a17fa640 1 -- 172.21.15.62:0/2701046929 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa9b4020420 con 0x55ed3ac20980 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9c25e0740 10 client.18441 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9c25e0740 10 client.18441 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9c25e0740 20 client.18441 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ed3acab150 0x55ed3acad570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9c25e0740 4 client.18441 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "78d00243", "release": true, "await": true}] 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9c25e0740 1 -- 172.21.15.62:0/2701046929 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "78d00243", "release": true, "await": true}) -- 0x55ed3ab8d0a0 con 0x55ed3acab150 2024-09-15T23:29:03.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ed3acab150 0x55ed3acad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:03.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9a17fa640 10 client.18441 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:03.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.892+0000 7fa9bd7a2640 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ed3acab150 0x55ed3acad570 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43d66dec563cd666 server_cookie=927265a65f17f764 in_seq=0 out_seq=0 2024-09-15T23:29:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.915+0000 7fa9a17fa640 1 -- 172.21.15.62:0/2701046929 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55ed3ab8d0a0 con 0x55ed3acab150 2024-09-15T23:29:03.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.915+0000 7fa9a17fa640 10 client.18441 handle_command_reply: tid=0 2024-09-15T23:29:03.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.917+0000 7fa9c25e0740 1 client.18441 shutdown 2024-09-15T23:29:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.925+0000 7fa9c25e0740 1 -- 172.21.15.62:0/2701046929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20980 msgr2=0x55ed3ac2fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.925+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20980 0x55ed3ac2fd80 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x55ed3ac1d450 tx=0x7fa9b4010cb0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.932+0000 7fa9c25e0740 1 -- 172.21.15.62:0/2701046929 shutdown_connections 2024-09-15T23:29:03.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.932+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ed3acab150 0x55ed3acad570 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.932+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed3ac2b240 0x55ed3ac30370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.932+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed3ac20980 0x55ed3ac2fd80 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.932+0000 7fa9c25e0740 1 --2- 172.21.15.62:0/2701046929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ed3ac20060 0x55ed3ac33a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.932+0000 7fa9a17fa640 0 client.18441 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:03.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.932+0000 7fa9c25e0740 1 -- 172.21.15.62:0/2701046929 >> 172.21.15.62:0/2701046929 conn(0x55ed3ac1c5f0 msgr2=0x55ed3ac27070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.937+0000 7fa9c25e0740 1 -- 172.21.15.62:0/2701046929 shutdown_connections 2024-09-15T23:29:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.937+0000 7fa9c25e0740 1 -- 172.21.15.62:0/2701046929 wait complete. 2024-09-15T23:29:03.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.937+0000 7fa9c25e0740 20 client.18441 trim_cache size 0 max 16384 2024-09-15T23:29:03.939 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:29:03.939 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 35, 2024-09-15T23:29:03.940 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:29:03.940 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 48, 2024-09-15T23:29:03.940 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:29:03.940 INFO:teuthology.orchestra.run.smithi062.stdout: "78d00243": { 2024-09-15T23:29:03.940 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 48, 2024-09-15T23:29:03.940 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:29:03.940 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:29:03.940 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:29:03.940 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:29:03.941 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:29:03.941 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:29:03.941 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 131.4, 2024-09-15T23:29:03.941 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:29:03.941 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:29:03.941 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:29:03.941 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:29:03.941 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:29:03.941 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:29:03.942 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:03.942 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:03.942 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:03.942 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:03.942 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:03.942 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:29:03.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.938+0000 7fa9c0207640 1 -- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9a40a32c0 msgr2=0x7fa9a40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.938+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9a40a32c0 0x7fa9a40a5780 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00bd2e0 tx=0x7fa9b00c1040 comp rx=0 tx=0).stop 2024-09-15T23:29:03.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.938+0000 7fa9c0207640 1 -- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 msgr2=0x7fa9b819cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:03.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.938+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 0x7fa9b819cc80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0031a60 tx=0x7fa9b0031a90 comp rx=0 tx=0).stop 2024-09-15T23:29:03.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.939+0000 7fa9c0207640 1 -- 172.21.15.62:0/3425446856 shutdown_connections 2024-09-15T23:29:03.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.939+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa9a40a32c0 0x7fa9a40a5780 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.939+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9b8106f10 0x7fa9b819cc80 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.939+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b8102790 0x7fa9b81988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.939+0000 7fa9c0207640 1 --2- 172.21.15.62:0/3425446856 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa9b8101de0 0x7fa9b81983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:03.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.940+0000 7fa9c0207640 1 -- 172.21.15.62:0/3425446856 >> 172.21.15.62:0/3425446856 conn(0x7fa9b80fd8c0 msgr2=0x7fa9b80feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:03.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.944+0000 7fa9c0207640 1 -- 172.21.15.62:0/3425446856 shutdown_connections 2024-09-15T23:29:03.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:03.947+0000 7fa9c0207640 1 -- 172.21.15.62:0/3425446856 wait complete. 2024-09-15T23:29:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 765.0 s, 55.4 tps, lat 2044.220 ms stddev 2746.211 2024-09-15T23:29:04.736 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '78d00243', total seconds elapsed: 14.7 2024-09-15T23:29:05.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:05 smithi062 ceph-mon[30949]: pgmap v754: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 3.9 MiB/s wr, 487 op/s 2024-09-15T23:29:05.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:05 smithi067 ceph-mon[35665]: pgmap v754: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 3.9 MiB/s wr, 487 op/s 2024-09-15T23:29:05.792 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:05 smithi148 ceph-mon[35610]: pgmap v754: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 3.9 MiB/s wr, 487 op/s 2024-09-15T23:29:07.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:07 smithi062 ceph-mon[30949]: pgmap v755: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 3.6 MiB/s wr, 412 op/s 2024-09-15T23:29:07.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:07 smithi067 ceph-mon[35665]: pgmap v755: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 3.6 MiB/s wr, 412 op/s 2024-09-15T23:29:07.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:07 smithi148 ceph-mon[35610]: pgmap v755: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 3.6 MiB/s wr, 412 op/s 2024-09-15T23:29:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.792+0000 7f4777d9f640 1 -- 172.21.15.62:0/2257349399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4770109cd0 msgr2=0x7f4770110560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.792+0000 7f4777d9f640 1 --2- 172.21.15.62:0/2257349399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4770109cd0 0x7f4770110560 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4768009ca0 tx=0x7f476802a8b0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.792+0000 7f4777d9f640 1 -- 172.21.15.62:0/2257349399 shutdown_connections 2024-09-15T23:29:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.792+0000 7f4777d9f640 1 --2- 172.21.15.62:0/2257349399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4770109cd0 0x7f4770110560 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.792+0000 7f4777d9f640 1 --2- 172.21.15.62:0/2257349399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4770109310 0x7f4770109790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.792+0000 7f4777d9f640 1 --2- 172.21.15.62:0/2257349399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47701084c0 0x7f47701088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.792+0000 7f4777d9f640 1 -- 172.21.15.62:0/2257349399 >> 172.21.15.62:0/2257349399 conn(0x7f477006b6d0 msgr2=0x7f4770079830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.792+0000 7f4777d9f640 1 -- 172.21.15.62:0/2257349399 shutdown_connections 2024-09-15T23:29:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.792+0000 7f4777d9f640 1 -- 172.21.15.62:0/2257349399 wait complete. 2024-09-15T23:29:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.793+0000 7f4777d9f640 1 Processor -- start 2024-09-15T23:29:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.793+0000 7f4777d9f640 1 -- start start 2024-09-15T23:29:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.793+0000 7f4777d9f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47701084c0 0x7f477019e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f4777d9f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4770109310 0x7f477019e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f4777d9f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4770109cd0 0x7f47701a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f4777d9f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4770113e60 con 0x7f4770109cd0 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f477533a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4770109310 0x7f477019e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f477533a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4770109310 0x7f477019e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:32888/0 (socket says 172.21.15.62:32888) 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f477533a640 1 -- 172.21.15.62:0/3589183167 learned_addr learned my addr 172.21.15.62:0/3589183167 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f477633c640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4770109cd0 0x7f47701a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4770113ce0 con 0x7f4770109310 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47701084c0 0x7f477019e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.794+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4770113fe0 con 0x7f47701084c0 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4775b3b640 1 -- 172.21.15.62:0/3589183167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4770109310 msgr2=0x7f477019e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4770109310 0x7f477019e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4775b3b640 1 -- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4770109cd0 msgr2=0x7f47701a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4770109cd0 0x7f47701a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4775b3b640 1 -- 172.21.15.62:0/3589183167 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47701a5fd0 con 0x7f47701084c0 2024-09-15T23:29:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f477633c640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4770109cd0 0x7f47701a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47701084c0 0x7f477019e3b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f476c00e9c0 tx=0x7f476c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4766ffd640 1 -- 172.21.15.62:0/3589183167 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c00ccc0 con 0x7f47701084c0 2024-09-15T23:29:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4766ffd640 1 -- 172.21.15.62:0/3589183167 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f476c0035f0 con 0x7f47701084c0 2024-09-15T23:29:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47701a62c0 con 0x7f47701084c0 2024-09-15T23:29:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4766ffd640 1 -- 172.21.15.62:0/3589183167 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c00ce60 con 0x7f47701084c0 2024-09-15T23:29:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.795+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47701a6810 con 0x7f47701084c0 2024-09-15T23:29:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.798+0000 7f4766ffd640 1 -- 172.21.15.62:0/3589183167 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f476c0037a0 con 0x7f47701084c0 2024-09-15T23:29:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.799+0000 7f4766ffd640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f475c0a32f0 0x7f475c0a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.799+0000 7f477ac1b740 1 Processor -- start 2024-09-15T23:29:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.799+0000 7f477533a640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f475c0a32f0 0x7f475c0a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.799+0000 7f4766ffd640 1 -- 172.21.15.62:0/3589183167 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f476c0cffe0 con 0x7f47701084c0 2024-09-15T23:29:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.799+0000 7f477ac1b740 1 -- start start 2024-09-15T23:29:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c87da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c882e0 0x564667c8cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c8f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477ac1b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564667beb2f0 con 0x564667c879a0 2024-09-15T23:29:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477ac1b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564667b93270 con 0x564667c882e0 2024-09-15T23:29:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477ac1b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564667be72b0 con 0x564667c8d370 2024-09-15T23:29:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f4775b3b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c87da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f4775b3b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c87da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44478/0 (socket says 172.21.15.62:44478) 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f4775b3b640 1 -- 172.21.15.62:0/2319554878 learned_addr learned my addr 172.21.15.62:0/2319554878 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477533a640 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c882e0 0x564667c8cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477633c640 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c8f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477533a640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f475c0a32f0 0x7f475c0a57b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f477019f8c0 tx=0x7f4760011810 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f4775b3b640 1 -- 172.21.15.62:0/2319554878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 msgr2=0x564667c8f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f4775b3b640 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c8f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f4775b3b640 1 -- 172.21.15.62:0/2319554878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c882e0 msgr2=0x564667c8cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f4775b3b640 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c882e0 0x564667c8cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f4775b3b640 1 -- 172.21.15.62:0/2319554878 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564667b92340 con 0x564667c879a0 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.800+0000 7f477533a640 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c882e0 0x564667c8cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.801+0000 7f4775b3b640 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c87da0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f476c0986c0 tx=0x7f476c09bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db90d922971ff1e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.801+0000 7f475bfff640 1 -- 172.21.15.62:0/2319554878 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0a0610 con 0x564667c879a0 2024-09-15T23:29:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.801+0000 7f475bfff640 1 -- 172.21.15.62:0/2319554878 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f476c0a4ed0 con 0x564667c879a0 2024-09-15T23:29:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.801+0000 7f475bfff640 1 -- 172.21.15.62:0/2319554878 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0a0410 con 0x564667c879a0 2024-09-15T23:29:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 -- 172.21.15.62:0/2319554878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 msgr2=0x564667c87da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c87da0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f476c0986c0 tx=0x7f476c09bc40 comp rx=0 tx=0).stop 2024-09-15T23:29:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 -- 172.21.15.62:0/2319554878 shutdown_connections 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c8f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c882e0 0x564667c8cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2319554878 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c87da0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 -- 172.21.15.62:0/2319554878 >> 172.21.15.62:0/2319554878 conn(0x564667c83010 msgr2=0x564667c83410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 -- 172.21.15.62:0/2319554878 shutdown_connections 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 -- 172.21.15.62:0/2319554878 wait complete. 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 -- start start 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.802+0000 7f477ac1b740 1 Processor -- start 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.803+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 0x564667c91c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.803+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 0x564667c92160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.803+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c99130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.803+0000 7f477ac1b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564667c90340 con 0x564667c882e0 2024-09-15T23:29:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.803+0000 7f477ac1b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564667b74290 con 0x564667c879a0 2024-09-15T23:29:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.803+0000 7f477ac1b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564667c904c0 con 0x564667c8d370 2024-09-15T23:29:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.803+0000 7f477ac1b740 10 client.0 _ll_register_callbacks cb 0x564667ba5140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.804+0000 7f477633c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c99130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.804+0000 7f477633c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c99130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:52038/0 (socket says 172.21.15.62:52038) 2024-09-15T23:29:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.804+0000 7f477633c640 1 -- 172.21.15.62:0/2972542084 learned_addr learned my addr 172.21.15.62:0/2972542084 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.805+0000 7f477533a640 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 0x564667c92160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.805+0000 7f477633c640 1 -- 172.21.15.62:0/2972542084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 msgr2=0x564667c91c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:29:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.805+0000 7f477633c640 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 0x564667c91c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.805+0000 7f477633c640 1 -- 172.21.15.62:0/2972542084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 msgr2=0x564667c92160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.805+0000 7f477633c640 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 0x564667c92160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.805+0000 7f477633c640 1 -- 172.21.15.62:0/2972542084 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646678fe210 con 0x564667c8d370 2024-09-15T23:29:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.806+0000 7f477633c640 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c99130 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4768009ca0 tx=0x7f4768005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c88f5c0155a5322 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.807+0000 7f47597fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:29:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.807+0000 7f47597fa640 1 -- 172.21.15.62:0/2972542084 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768003c40 con 0x564667c8d370 2024-09-15T23:29:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.807+0000 7f47597fa640 1 -- 172.21.15.62:0/2972542084 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4768003de0 con 0x564667c8d370 2024-09-15T23:29:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.807+0000 7f477ac1b740 1 -- 172.21.15.62:0/2972542084 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f476c0a4ed0 con 0x564667c8d370 2024-09-15T23:29:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.807+0000 7f47597fa640 1 -- 172.21.15.62:0/2972542084 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4768004b40 con 0x564667c8d370 2024-09-15T23:29:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.807+0000 7f477ac1b740 10 client.27697 fetch_fsmap learned FSMap version 35 2024-09-15T23:29:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.807+0000 7f477ac1b740 1 -- 172.21.15.62:0/2972542084 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5646679ca620 con 0x564667c8d370 2024-09-15T23:29:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.808+0000 7f47597fa640 1 -- 172.21.15.62:0/2972542084 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f476c0a4ed0 con 0x564667c8d370 2024-09-15T23:29:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.808+0000 7f47597fa640 1 -- 172.21.15.62:0/2972542084 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f47680051e0 con 0x564667c8d370 2024-09-15T23:29:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.808+0000 7f477ac1b740 10 client.27697 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:29:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.808+0000 7f477ac1b740 10 client.27697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.808+0000 7f477ac1b740 20 client.27697 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.808+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564667c9d840 0x564667c9fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.808+0000 7f477ac1b740 4 client.27697 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:08.810 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.808+0000 7f477ac1b740 1 -- 172.21.15.62:0/2972542084 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564667bf8a20 con 0x564667c9d840 2024-09-15T23:29:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.808+0000 7f4775b3b640 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564667c9d840 0x564667c9fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.809+0000 7f47597fa640 10 client.27697 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.809+0000 7f4775b3b640 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564667c9d840 0x564667c9fcc0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7f19dd7af9fa040 server_cookie=1053ce159aaee51d in_seq=0 out_seq=0 2024-09-15T23:29:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.809+0000 7f47597fa640 1 -- 172.21.15.62:0/2972542084 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564667bf8a20 con 0x564667c9d840 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.809+0000 7f47597fa640 10 client.27697 handle_command_reply: tid=0 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.809+0000 7f477ac1b740 1 client.27697 shutdown 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 -- 172.21.15.62:0/2972542084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 msgr2=0x564667c99130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c99130 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4768009ca0 tx=0x7f4768005ef0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 -- 172.21.15.62:0/2972542084 shutdown_connections 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564667c9d840 0x564667c9fcc0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c8d370 0x564667c99130 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f47597fa640 0 client.27697 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 0x564667c92160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 --2- 172.21.15.62:0/2972542084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 0x564667c91c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 -- 172.21.15.62:0/2972542084 >> 172.21.15.62:0/2972542084 conn(0x564667c83010 msgr2=0x564667c84820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 -- 172.21.15.62:0/2972542084 shutdown_connections 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 1 -- 172.21.15.62:0/2972542084 wait complete. 2024-09-15T23:29:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.810+0000 7f477ac1b740 20 client.27697 trim_cache size 0 max 16384 2024-09-15T23:29:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.829+0000 7f477ac1b740 1 Processor -- start 2024-09-15T23:29:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.829+0000 7f477ac1b740 1 -- start start 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.829+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c83010 0x564667c87d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.829+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 0x564667c990f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.829+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 0x564667c84400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.829+0000 7f477ac1b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f47701b3670 con 0x564667c882e0 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.829+0000 7f477ac1b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47701b0f40 con 0x564667c879a0 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.829+0000 7f477ac1b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f47701aede0 con 0x564667c83010 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f477633c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 0x564667c84400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f477533a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 0x564667c990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f477533a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 0x564667c990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:32906/0 (socket says 172.21.15.62:32906) 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f4775b3b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c83010 0x564667c87d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f477633c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 0x564667c84400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44500/0 (socket says 172.21.15.62:44500) 2024-09-15T23:29:08.832 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f477533a640 1 -- 172.21.15.62:0/3705162929 learned_addr learned my addr 172.21.15.62:0/3705162929 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f4775b3b640 1 -- 172.21.15.62:0/3705162929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 msgr2=0x564667c990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3705162929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 0x564667c990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f4775b3b640 1 -- 172.21.15.62:0/3705162929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 msgr2=0x564667c84400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3705162929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 0x564667c84400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f4775b3b640 1 -- 172.21.15.62:0/3705162929 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564667c887c0 con 0x564667c83010 2024-09-15T23:29:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3705162929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c83010 0x564667c87d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x564667c91150 tx=0x7f476c0985f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f1b17613acdc35f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f475b7fe640 1 -- 172.21.15.62:0/3705162929 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0ba070 con 0x564667c83010 2024-09-15T23:29:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f475b7fe640 1 -- 172.21.15.62:0/3705162929 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f476c0a4ed0 con 0x564667c83010 2024-09-15T23:29:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.830+0000 7f475b7fe640 1 -- 172.21.15.62:0/3705162929 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0a0410 con 0x564667c83010 2024-09-15T23:29:08.837 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.831+0000 7f477ac1b740 1 -- 172.21.15.62:0/3705162929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c83010 msgr2=0x564667c87d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.831+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3705162929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c83010 0x564667c87d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x564667c91150 tx=0x7f476c0985f0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.831+0000 7f477ac1b740 1 -- 172.21.15.62:0/3705162929 shutdown_connections 2024-09-15T23:29:08.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.831+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3705162929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c882e0 0x564667c84400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.831+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3705162929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c879a0 0x564667c990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.831+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3705162929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c83010 0x564667c87d80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.831+0000 7f477ac1b740 1 -- 172.21.15.62:0/3705162929 >> 172.21.15.62:0/3705162929 conn(0x564667c8cf00 msgr2=0x564667c821c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.832+0000 7f477ac1b740 1 -- 172.21.15.62:0/3705162929 shutdown_connections 2024-09-15T23:29:08.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.832+0000 7f477ac1b740 1 -- 172.21.15.62:0/3705162929 wait complete. 2024-09-15T23:29:08.838 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.832+0000 7f477ac1b740 1 -- start start 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.832+0000 7f477ac1b740 1 Processor -- start 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.832+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c83010 0x564667c96de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.832+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c97320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.832+0000 7f4775b3b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c83010 0x564667c96de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f477533a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c97320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f4775b3b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c83010 0x564667c96de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:32910/0 (socket says 172.21.15.62:32910) 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f477ac1b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c882e0 0x564667c98330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f4775b3b640 1 -- 172.21.15.62:0/3714538401 learned_addr learned my addr 172.21.15.62:0/3714538401 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f477533a640 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c97320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44514/0 (socket says 172.21.15.62:44514) 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f47701af400 con 0x564667c879a0 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47701b0870 con 0x564667c83010 2024-09-15T23:29:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f47701afad0 con 0x564667c882e0 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f477633c640 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c882e0 0x564667c98330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f477ac1b740 10 client.0 _ll_register_callbacks cb 0x7f4768009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f4775b3b640 1 -- 172.21.15.62:0/3714538401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c882e0 msgr2=0x564667c98330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c882e0 0x564667c98330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f4775b3b640 1 -- 172.21.15.62:0/3714538401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 msgr2=0x564667c97320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c97320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f4775b3b640 1 -- 172.21.15.62:0/3714538401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564667c98940 con 0x564667c83010 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f4775b3b640 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c83010 0x564667c96de0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f476c0986c0 tx=0x7f476c09c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c6728095f5254c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f47597fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:29:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f47597fa640 1 -- 172.21.15.62:0/3714538401 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c07f060 con 0x564667c83010 2024-09-15T23:29:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f47597fa640 1 -- 172.21.15.62:0/3714538401 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f476c0a01b0 con 0x564667c83010 2024-09-15T23:29:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.833+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f476c0a4ed0 con 0x564667c83010 2024-09-15T23:29:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f47597fa640 1 -- 172.21.15.62:0/3714538401 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f476c0ba070 con 0x564667c83010 2024-09-15T23:29:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f477ac1b740 10 client.27992 fetch_fsmap learned FSMap version 35 2024-09-15T23:29:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564667d08b30 con 0x564667c83010 2024-09-15T23:29:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f47597fa640 1 -- 172.21.15.62:0/3714538401 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f476c0a4ed0 con 0x564667c83010 2024-09-15T23:29:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f47597fa640 1 -- 172.21.15.62:0/3714538401 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f476c0bc030 con 0x564667c83010 2024-09-15T23:29:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f477ac1b740 10 client.27992 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:29:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f477ac1b740 10 client.27992 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f477ac1b740 20 client.27992 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564667c8d370 0x564667d0b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f477ac1b740 4 client.27992 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:29:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564667bf2d70 con 0x564667c8d370 2024-09-15T23:29:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.834+0000 7f477533a640 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564667c8d370 0x564667d0b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.835+0000 7f477533a640 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564667c8d370 0x564667d0b050 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f07af95471292c9 server_cookie=390c1131aa4cdcd in_seq=0 out_seq=0 2024-09-15T23:29:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.835+0000 7f47597fa640 10 client.27992 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.848+0000 7f47597fa640 1 -- 172.21.15.62:0/3714538401 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564667bf2d70 con 0x564667c8d370 2024-09-15T23:29:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.848+0000 7f47597fa640 10 client.27992 handle_command_reply: tid=0 2024-09-15T23:29:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.848+0000 7f477ac1b740 1 client.27992 shutdown 2024-09-15T23:29:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.848+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c83010 msgr2=0x564667c96de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.848+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c83010 0x564667c96de0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f476c0986c0 tx=0x7f476c09c3e0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.848+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 shutdown_connections 2024-09-15T23:29:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.848+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564667c8d370 0x564667d0b050 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.851 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.848+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564667c882e0 0x564667c98330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.848+0000 7f47597fa640 0 client.27992 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564667c879a0 0x564667c97320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f477ac1b740 1 --2- 172.21.15.62:0/3714538401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564667c83010 0x564667c96de0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 >> 172.21.15.62:0/3714538401 conn(0x564667c8cf00 msgr2=0x564667c821c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 shutdown_connections 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f477ac1b740 1 -- 172.21.15.62:0/3714538401 wait complete. 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f477ac1b740 20 client.27992 trim_cache size 0 max 16384 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:29:08.852 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:29:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f475c0a32f0 msgr2=0x7f475c0a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f475c0a32f0 0x7f475c0a57b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f477019f8c0 tx=0x7f4760011810 comp rx=0 tx=0).stop 2024-09-15T23:29:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47701084c0 msgr2=0x7f477019e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:08.853 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47701084c0 0x7f477019e3b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f476c00e9c0 tx=0x7f476c00ee80 comp rx=0 tx=0).stop 2024-09-15T23:29:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 shutdown_connections 2024-09-15T23:29:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f475c0a32f0 0x7f475c0a57b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4770109cd0 0x7f47701a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4770109310 0x7f477019e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 --2- 172.21.15.62:0/3589183167 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f47701084c0 0x7f477019e3b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.849+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 >> 172.21.15.62:0/3589183167 conn(0x7f477006b6d0 msgr2=0x7f477010c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.850+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 shutdown_connections 2024-09-15T23:29:08.854 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:08.850+0000 7f4777d9f640 1 -- 172.21.15.62:0/3589183167 wait complete. 2024-09-15T23:29:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 770.0 s, 578.4 tps, lat 190.452 ms stddev 410.625 2024-09-15T23:29:09.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:09 smithi062 ceph-mon[30949]: pgmap v756: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-09-15T23:29:09.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:09 smithi067 ceph-mon[35665]: pgmap v756: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-09-15T23:29:09.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:09 smithi148 ceph-mon[35610]: pgmap v756: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 2.5 MiB/s wr, 265 op/s 2024-09-15T23:29:11.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:11 smithi062 ceph-mon[30949]: pgmap v757: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.0 MiB/s wr, 310 op/s 2024-09-15T23:29:11.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:11 smithi067 ceph-mon[35665]: pgmap v757: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.0 MiB/s wr, 310 op/s 2024-09-15T23:29:11.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:11 smithi148 ceph-mon[35610]: pgmap v757: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.0 MiB/s wr, 310 op/s 2024-09-15T23:29:13.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:13 smithi062 ceph-mon[30949]: pgmap v758: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.0 MiB/s wr, 301 op/s 2024-09-15T23:29:13.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:13 smithi067 ceph-mon[35665]: pgmap v758: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.0 MiB/s wr, 301 op/s 2024-09-15T23:29:13.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:13 smithi148 ceph-mon[35610]: pgmap v758: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.0 MiB/s wr, 301 op/s 2024-09-15T23:29:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 775.0 s, 693.0 tps, lat 143.919 ms stddev 175.375 2024-09-15T23:29:15.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:15 smithi062 ceph-mon[30949]: pgmap v759: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.3 MiB/s wr, 341 op/s 2024-09-15T23:29:15.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:15 smithi067 ceph-mon[35665]: pgmap v759: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.3 MiB/s wr, 341 op/s 2024-09-15T23:29:15.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:15 smithi148 ceph-mon[35610]: pgmap v759: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 3.3 MiB/s wr, 341 op/s 2024-09-15T23:29:17.736 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:17 smithi062 ceph-mon[30949]: pgmap v760: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.2 MiB/s wr, 434 op/s 2024-09-15T23:29:17.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:17 smithi067 ceph-mon[35665]: pgmap v760: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.2 MiB/s wr, 434 op/s 2024-09-15T23:29:17.791 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:17 smithi148 ceph-mon[35610]: pgmap v760: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 4.2 MiB/s wr, 434 op/s 2024-09-15T23:29:19.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 780.0 s, 548.0 tps, lat 180.559 ms stddev 199.314 2024-09-15T23:29:19.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:19 smithi062 ceph-mon[30949]: pgmap v761: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 5.6 MiB/s wr, 538 op/s 2024-09-15T23:29:19.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:19 smithi067 ceph-mon[35665]: pgmap v761: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 5.6 MiB/s wr, 538 op/s 2024-09-15T23:29:19.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:19 smithi148 ceph-mon[35610]: pgmap v761: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 5.6 MiB/s wr, 538 op/s 2024-09-15T23:29:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:21 smithi062 ceph-mon[30949]: pgmap v762: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 5.9 MiB/s wr, 594 op/s 2024-09-15T23:29:21.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:21 smithi067 ceph-mon[35665]: pgmap v762: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 5.9 MiB/s wr, 594 op/s 2024-09-15T23:29:21.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:21 smithi148 ceph-mon[35610]: pgmap v762: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 5.9 MiB/s wr, 594 op/s 2024-09-15T23:29:22.014 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.012+0000 7fa33d781640 1 -- 172.21.15.62:0/1502465752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa338101c80 msgr2=0x7fa338102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.012+0000 7fa33d781640 1 --2- 172.21.15.62:0/1502465752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa338101c80 0x7fa338102100 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa328009860 tx=0x7fa32802a410 comp rx=0 tx=0).stop 2024-09-15T23:29:22.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.016+0000 7fa33d781640 1 -- 172.21.15.62:0/1502465752 shutdown_connections 2024-09-15T23:29:22.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.016+0000 7fa33d781640 1 --2- 172.21.15.62:0/1502465752 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa338102640 0x7fa338108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.016+0000 7fa33d781640 1 --2- 172.21.15.62:0/1502465752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa338101c80 0x7fa338102100 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.016+0000 7fa33d781640 1 --2- 172.21.15.62:0/1502465752 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa338100f70 0x7fa338101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.016+0000 7fa33d781640 1 -- 172.21.15.62:0/1502465752 >> 172.21.15.62:0/1502465752 conn(0x7fa3380fb930 msgr2=0x7fa3380fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.018+0000 7fa33d781640 1 -- 172.21.15.62:0/1502465752 shutdown_connections 2024-09-15T23:29:22.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.021+0000 7fa33d781640 1 -- 172.21.15.62:0/1502465752 wait complete. 2024-09-15T23:29:22.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.022+0000 7fa33d781640 1 Processor -- start 2024-09-15T23:29:22.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.022+0000 7fa33d781640 1 -- start start 2024-09-15T23:29:22.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.025+0000 7fa33d781640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa338100f70 0x7fa338195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.025+0000 7fa33d781640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa338101c80 0x7fa3381961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.027 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.025+0000 7fa33d781640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa338102640 0x7fa33819d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.025+0000 7fa33d781640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa33810b6b0 con 0x7fa338101c80 2024-09-15T23:29:22.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.025+0000 7fa33d781640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa33810b530 con 0x7fa338102640 2024-09-15T23:29:22.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.025+0000 7fa33d781640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa33810b830 con 0x7fa338100f70 2024-09-15T23:29:22.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.025+0000 7fa336ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa338100f70 0x7fa338195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.025+0000 7fa336ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa338100f70 0x7fa338195c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:54374/0 (socket says 172.21.15.62:54374) 2024-09-15T23:29:22.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.025+0000 7fa336ffd640 1 -- 172.21.15.62:0/17026975 learned_addr learned my addr 172.21.15.62:0/17026975 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:22.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.027+0000 7fa3367fc640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa338101c80 0x7fa3381961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.028+0000 7fa3367fc640 1 -- 172.21.15.62:0/17026975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa338100f70 msgr2=0x7fa338195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.028+0000 7fa3367fc640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa338100f70 0x7fa338195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.028+0000 7fa3367fc640 1 -- 172.21.15.62:0/17026975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa338102640 msgr2=0x7fa33819d210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:29:22.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.028+0000 7fa3367fc640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa338102640 0x7fa33819d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.028+0000 7fa3367fc640 1 -- 172.21.15.62:0/17026975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3381a1cc0 con 0x7fa338101c80 2024-09-15T23:29:22.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.028+0000 7fa3367fc640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa338101c80 0x7fa3381961b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fa328009780 tx=0x7fa328030940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.029+0000 7fa31ffff640 1 -- 172.21.15.62:0/17026975 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328040070 con 0x7fa338101c80 2024-09-15T23:29:22.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.029+0000 7fa31ffff640 1 -- 172.21.15.62:0/17026975 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa328005610 con 0x7fa338101c80 2024-09-15T23:29:22.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.029+0000 7fa31ffff640 1 -- 172.21.15.62:0/17026975 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328004920 con 0x7fa338101c80 2024-09-15T23:29:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.030+0000 7fa33d781640 1 -- 172.21.15.62:0/17026975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33819d750 con 0x7fa338101c80 2024-09-15T23:29:22.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.030+0000 7fa33d781640 1 -- 172.21.15.62:0/17026975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa33819dc70 con 0x7fa338101c80 2024-09-15T23:29:22.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.034+0000 7fa3405f3740 1 Processor -- start 2024-09-15T23:29:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.035+0000 7fa31ffff640 1 -- 172.21.15.62:0/17026975 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa328039040 con 0x7fa338101c80 2024-09-15T23:29:22.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.036+0000 7fa3405f3740 1 -- start start 2024-09-15T23:29:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.036+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172289a0 0x558517228da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.036+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 0x55851722dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.036+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 0x558517230830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.036+0000 7fa3405f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55851718c2f0 con 0x5585172289a0 2024-09-15T23:29:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.036+0000 7fa3405f3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558517134270 con 0x5585172292e0 2024-09-15T23:29:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.036+0000 7fa3405f3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5585171882b0 con 0x55851722e370 2024-09-15T23:29:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.036+0000 7fa31ffff640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3200a3530 0x7fa3200a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.037+0000 7fa31ffff640 1 -- 172.21.15.62:0/17026975 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa3280f0790 con 0x7fa338101c80 2024-09-15T23:29:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.037+0000 7fa3367fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 0x558517230830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.037+0000 7fa3367fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 0x558517230830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:54384/0 (socket says 172.21.15.62:54384) 2024-09-15T23:29:22.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.037+0000 7fa3367fc640 1 -- 172.21.15.62:0/2680759844 learned_addr learned my addr 172.21.15.62:0/2680759844 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.039+0000 7fa3377fe640 1 --2- 172.21.15.62:0/2680759844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172289a0 0x558517228da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.039+0000 7fa3377fe640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3200a3530 0x7fa3200a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.041+0000 7fa3377fe640 1 -- 172.21.15.62:0/2680759844 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 msgr2=0x558517230830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.041+0000 7fa3377fe640 1 --2- 172.21.15.62:0/2680759844 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 0x558517230830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.041+0000 7fa3377fe640 1 -- 172.21.15.62:0/2680759844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 msgr2=0x55851722dda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:29:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.041+0000 7fa3377fe640 1 --2- 172.21.15.62:0/2680759844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 0x55851722dda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.041+0000 7fa3377fe640 1 -- 172.21.15.62:0/2680759844 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558517133340 con 0x5585172289a0 2024-09-15T23:29:22.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.042+0000 7fa3377fe640 1 --2- 172.21.15.62:0/2680759844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172289a0 0x558517228da0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fa330005fd0 tx=0x7fa33000dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6779187860882d11 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.043+0000 7fa3377fe640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3200a3530 0x7fa3200a59f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa33001a000 tx=0x7fa33001c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.043+0000 7fa31d7fa640 1 -- 172.21.15.62:0/2680759844 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33001d040 con 0x5585172289a0 2024-09-15T23:29:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.043+0000 7fa31d7fa640 1 -- 172.21.15.62:0/2680759844 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa330003e50 con 0x5585172289a0 2024-09-15T23:29:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.044+0000 7fa3405f3740 1 -- 172.21.15.62:0/2680759844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172289a0 msgr2=0x558517228da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.044+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2680759844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172289a0 0x558517228da0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fa330005fd0 tx=0x7fa33000dd80 comp rx=0 tx=0).stop 2024-09-15T23:29:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.044+0000 7fa3405f3740 1 -- 172.21.15.62:0/2680759844 shutdown_connections 2024-09-15T23:29:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.044+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2680759844 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 0x558517230830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.044+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2680759844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 0x55851722dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.044+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2680759844 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172289a0 0x558517228da0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.044+0000 7fa3405f3740 1 -- 172.21.15.62:0/2680759844 >> 172.21.15.62:0/2680759844 conn(0x558517224010 msgr2=0x558517224410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.050+0000 7fa3405f3740 1 -- 172.21.15.62:0/2680759844 shutdown_connections 2024-09-15T23:29:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.051+0000 7fa3405f3740 1 -- 172.21.15.62:0/2680759844 wait complete. 2024-09-15T23:29:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.051+0000 7fa3405f3740 1 -- start start 2024-09-15T23:29:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.051+0000 7fa3405f3740 1 Processor -- start 2024-09-15T23:29:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x558517232c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172292e0 0x558517233140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 0x55851723a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa3405f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558517231330 con 0x5585172292e0 2024-09-15T23:29:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa3405f3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558517115290 con 0x5585172289a0 2024-09-15T23:29:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa3405f3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5585172314b0 con 0x55851722e370 2024-09-15T23:29:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa3405f3740 10 client.0 _ll_register_callbacks cb 0x558517146140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa336ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x558517232c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa336ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x558517232c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54264/0 (socket says 172.21.15.62:54264) 2024-09-15T23:29:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa336ffd640 1 -- 172.21.15.62:0/208464387 learned_addr learned my addr 172.21.15.62:0/208464387 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.052+0000 7fa3377fe640 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 0x55851723a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.053+0000 7fa336ffd640 1 -- 172.21.15.62:0/208464387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 msgr2=0x55851723a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.053+0000 7fa336ffd640 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 0x55851723a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.053+0000 7fa336ffd640 1 -- 172.21.15.62:0/208464387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172292e0 msgr2=0x558517233140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.053+0000 7fa3367fc640 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172292e0 0x558517233140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.053+0000 7fa336ffd640 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172292e0 0x558517233140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.053+0000 7fa336ffd640 1 -- 172.21.15.62:0/208464387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558516e9f210 con 0x5585172289a0 2024-09-15T23:29:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.053+0000 7fa3367fc640 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172292e0 0x558517233140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:22.056 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.053+0000 7fa336ffd640 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x558517232c00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00e480 tx=0x7fa32c00e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e7b5604ccacc38c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.056+0000 7fa31aff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:29:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.056+0000 7fa31aff5640 1 -- 172.21.15.62:0/208464387 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c017070 con 0x5585172289a0 2024-09-15T23:29:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.056+0000 7fa3405f3740 1 -- 172.21.15.62:0/208464387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa330003e50 con 0x5585172289a0 2024-09-15T23:29:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.056+0000 7fa31aff5640 1 -- 172.21.15.62:0/208464387 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa32c013680 con 0x5585172289a0 2024-09-15T23:29:22.058 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.056+0000 7fa31aff5640 1 -- 172.21.15.62:0/208464387 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa32c023050 con 0x5585172289a0 2024-09-15T23:29:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.060+0000 7fa3405f3740 10 client.27995 fetch_fsmap learned FSMap version 35 2024-09-15T23:29:22.062 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.060+0000 7fa3405f3740 1 -- 172.21.15.62:0/208464387 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558516f6b620 con 0x5585172289a0 2024-09-15T23:29:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.063+0000 7fa31aff5640 1 -- 172.21.15.62:0/208464387 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa330003e50 con 0x5585172289a0 2024-09-15T23:29:22.065 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.063+0000 7fa31aff5640 1 -- 172.21.15.62:0/208464387 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa32c012b10 con 0x5585172289a0 2024-09-15T23:29:22.066 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.065+0000 7fa3405f3740 10 client.27995 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:29:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.065+0000 7fa3405f3740 10 client.27995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.065+0000 7fa3405f3740 20 client.27995 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.065+0000 7fa3405f3740 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55851724fd00 0x558517252150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.065+0000 7fa3405f3740 4 client.27995 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:22.067 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.065+0000 7fa3405f3740 1 -- 172.21.15.62:0/208464387 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558517199a20 con 0x55851724fd00 2024-09-15T23:29:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.068+0000 7fa3367fc640 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55851724fd00 0x558517252150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.079+0000 7fa3367fc640 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55851724fd00 0x558517252150 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c84711d742af8b12 server_cookie=1f0ef219a035ab07 in_seq=0 out_seq=0 2024-09-15T23:29:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.079+0000 7fa31aff5640 10 client.27995 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.084+0000 7fa31aff5640 1 -- 172.21.15.62:0/208464387 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558517199a20 con 0x55851724fd00 2024-09-15T23:29:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.084+0000 7fa31aff5640 10 client.27995 handle_command_reply: tid=0 2024-09-15T23:29:22.086 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.084+0000 7fa3405f3740 1 client.27995 shutdown 2024-09-15T23:29:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.090+0000 7fa3405f3740 1 -- 172.21.15.62:0/208464387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 msgr2=0x558517232c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.090+0000 7fa3405f3740 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x558517232c00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00e480 tx=0x7fa32c00e940 comp rx=0 tx=0).stop 2024-09-15T23:29:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.093+0000 7fa3405f3740 1 -- 172.21.15.62:0/208464387 shutdown_connections 2024-09-15T23:29:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.093+0000 7fa3405f3740 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55851724fd00 0x558517252150 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.093+0000 7fa3405f3740 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55851722e370 0x55851723a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.093+0000 7fa3405f3740 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5585172292e0 0x558517233140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.093+0000 7fa3405f3740 1 --2- 172.21.15.62:0/208464387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x558517232c00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.093+0000 7fa3405f3740 1 -- 172.21.15.62:0/208464387 >> 172.21.15.62:0/208464387 conn(0x558517224010 msgr2=0x558517225820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.094+0000 7fa31aff5640 0 client.27995 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:22.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.095+0000 7fa3405f3740 1 -- 172.21.15.62:0/208464387 shutdown_connections 2024-09-15T23:29:22.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.096+0000 7fa3405f3740 1 -- 172.21.15.62:0/208464387 wait complete. 2024-09-15T23:29:22.098 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.096+0000 7fa3405f3740 20 client.27995 trim_cache size 0 max 16384 2024-09-15T23:29:22.131 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.129+0000 7fa3405f3740 1 Processor -- start 2024-09-15T23:29:22.132 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.129+0000 7fa3405f3740 1 -- start start 2024-09-15T23:29:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.130+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585172289a0 0x5585172243f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.130+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 0x55851723a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.130+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 0x558517224930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.130+0000 7fa3405f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381aaf80 con 0x55851722e370 2024-09-15T23:29:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.130+0000 7fa3405f3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa338072510 con 0x5585172292e0 2024-09-15T23:29:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.130+0000 7fa3405f3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa338070360 con 0x5585172289a0 2024-09-15T23:29:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa3377fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 0x558517224930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa3377fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 0x558517224930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51318/0 (socket says 172.21.15.62:51318) 2024-09-15T23:29:22.136 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa3377fe640 1 -- 172.21.15.62:0/2133882681 learned_addr learned my addr 172.21.15.62:0/2133882681 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:22.138 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa336ffd640 1 --2- 172.21.15.62:0/2133882681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585172289a0 0x5585172243f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa3367fc640 1 --2- 172.21.15.62:0/2133882681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 0x55851723a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa3367fc640 1 -- 172.21.15.62:0/2133882681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585172289a0 msgr2=0x5585172243f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa3367fc640 1 --2- 172.21.15.62:0/2133882681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585172289a0 0x5585172243f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa3367fc640 1 -- 172.21.15.62:0/2133882681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 msgr2=0x558517224930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa3367fc640 1 --2- 172.21.15.62:0/2133882681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 0x558517224930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.139 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.134+0000 7fa3367fc640 1 -- 172.21.15.62:0/2133882681 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558517222650 con 0x5585172292e0 2024-09-15T23:29:22.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.139+0000 7fa3367fc640 1 --2- 172.21.15.62:0/2133882681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 0x55851723a140 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa3280edeb0 tx=0x7fa32802a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23bd916e5637fbc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.140+0000 7fa31cff9640 1 -- 172.21.15.62:0/2133882681 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3280da070 con 0x5585172292e0 2024-09-15T23:29:22.142 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.140+0000 7fa31cff9640 1 -- 172.21.15.62:0/2133882681 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3280edfc0 con 0x5585172292e0 2024-09-15T23:29:22.143 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.141+0000 7fa3405f3740 1 -- 172.21.15.62:0/2133882681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 msgr2=0x55851723a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.141+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2133882681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 0x55851723a140 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa3280edeb0 tx=0x7fa32802a410 comp rx=0 tx=0).stop 2024-09-15T23:29:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.142+0000 7fa3405f3740 1 -- 172.21.15.62:0/2133882681 shutdown_connections 2024-09-15T23:29:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.142+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2133882681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 0x558517224930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.142+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2133882681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172292e0 0x55851723a140 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.142+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2133882681 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5585172289a0 0x5585172243f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.142+0000 7fa3405f3740 1 -- 172.21.15.62:0/2133882681 >> 172.21.15.62:0/2133882681 conn(0x558517224010 msgr2=0x5585172329b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.142+0000 7fa3405f3740 1 -- 172.21.15.62:0/2133882681 shutdown_connections 2024-09-15T23:29:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.146+0000 7fa3405f3740 1 -- 172.21.15.62:0/2133882681 wait complete. 2024-09-15T23:29:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.146+0000 7fa3405f3740 1 -- start start 2024-09-15T23:29:22.149 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.146+0000 7fa3405f3740 1 Processor -- start 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.148+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x5585172376f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.148+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 0x558517237c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.148+0000 7fa3405f3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558517238c40 0x5585172390a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.148+0000 7fa3405f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa3380709d0 con 0x55851722e370 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.148+0000 7fa3405f3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa338071e40 con 0x5585172289a0 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.148+0000 7fa3405f3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa3380710a0 con 0x558517238c40 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.148+0000 7fa3405f3740 10 client.0 _ll_register_callbacks cb 0x7fa32c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.149+0000 7fa336ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x5585172376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.149+0000 7fa336ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x5585172376f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:54286/0 (socket says 172.21.15.62:54286) 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.149+0000 7fa336ffd640 1 -- 172.21.15.62:0/2463921992 learned_addr learned my addr 172.21.15.62:0/2463921992 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:22.153 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.150+0000 7fa336ffd640 1 -- 172.21.15.62:0/2463921992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558517238c40 msgr2=0x5585172390a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:29:22.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.150+0000 7fa336ffd640 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558517238c40 0x5585172390a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.150+0000 7fa336ffd640 1 -- 172.21.15.62:0/2463921992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 msgr2=0x558517237c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:22.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.150+0000 7fa336ffd640 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 0x558517237c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:22.154 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.150+0000 7fa336ffd640 1 -- 172.21.15.62:0/2463921992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558517239630 con 0x5585172289a0 2024-09-15T23:29:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.153+0000 7fa336ffd640 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x5585172376f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa32c006fd0 tx=0x7fa32c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f41be0a6d421950 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:22.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.153+0000 7fa31aff5640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:29:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.154+0000 7fa31aff5640 1 -- 172.21.15.62:0/2463921992 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c02c060 con 0x5585172289a0 2024-09-15T23:29:22.156 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.154+0000 7fa31aff5640 1 -- 172.21.15.62:0/2463921992 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa32c01b900 con 0x5585172289a0 2024-09-15T23:29:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.155+0000 7fa3405f3740 1 -- 172.21.15.62:0/2463921992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558517193d70 con 0x5585172289a0 2024-09-15T23:29:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.157+0000 7fa31aff5640 1 -- 172.21.15.62:0/2463921992 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa32c017030 con 0x5585172289a0 2024-09-15T23:29:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.159+0000 7fa3405f3740 10 client.28001 fetch_fsmap learned FSMap version 35 2024-09-15T23:29:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.159+0000 7fa3405f3740 1 -- 172.21.15.62:0/2463921992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5585172badf0 con 0x5585172289a0 2024-09-15T23:29:22.165 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.163+0000 7fa31aff5640 1 -- 172.21.15.62:0/2463921992 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558517193d70 con 0x5585172289a0 2024-09-15T23:29:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.163+0000 7fa31aff5640 1 -- 172.21.15.62:0/2463921992 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7fa32c0037c0 con 0x5585172289a0 2024-09-15T23:29:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.164+0000 7fa3405f3740 10 client.28001 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:29:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.164+0000 7fa3405f3740 10 client.28001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.164+0000 7fa3405f3740 20 client.28001 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.164+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5585172bb780 0x5585172bdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.164+0000 7fa3405f3740 4 client.28001 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:29:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.164+0000 7fa3405f3740 1 -- 172.21.15.62:0/2463921992 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558517110900 con 0x5585172bb780 2024-09-15T23:29:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.165+0000 7fa3367fc640 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5585172bb780 0x5585172bdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:22.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.175+0000 7fa3367fc640 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5585172bb780 0x5585172bdbc0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ac1516ea88fc15f server_cookie=f6be4d51b4971ef0 in_seq=0 out_seq=0 2024-09-15T23:29:22.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:22.177+0000 7fa31aff5640 10 client.28001 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:23.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.069+0000 7fa31aff5640 1 -- 172.21.15.62:0/2463921992 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558517110900 con 0x5585172bb780 2024-09-15T23:29:23.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.069+0000 7fa31aff5640 10 client.28001 handle_command_reply: tid=0 2024-09-15T23:29:23.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.069+0000 7fa3405f3740 1 client.28001 shutdown 2024-09-15T23:29:23.072 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 -- 172.21.15.62:0/2463921992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 msgr2=0x5585172376f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x5585172376f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa32c006fd0 tx=0x7fa32c00eeb0 comp rx=0 tx=0).stop 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 -- 172.21.15.62:0/2463921992 shutdown_connections 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5585172bb780 0x5585172bdbc0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558517238c40 0x5585172390a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55851722e370 0x558517237c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 --2- 172.21.15.62:0/2463921992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5585172289a0 0x5585172376f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 -- 172.21.15.62:0/2463921992 >> 172.21.15.62:0/2463921992 conn(0x558517224010 msgr2=0x5585172229a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa31aff5640 0 client.28001 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 -- 172.21.15.62:0/2463921992 shutdown_connections 2024-09-15T23:29:23.073 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 1 -- 172.21.15.62:0/2463921992 wait complete. 2024-09-15T23:29:23.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.070+0000 7fa3405f3740 20 client.28001 trim_cache size 0 max 16384 2024-09-15T23:29:23.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:29:23.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:29:23.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:29:23.074 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.071+0000 7fa33d781640 1 -- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3200a3530 msgr2=0x7fa3200a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.071+0000 7fa33d781640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3200a3530 0x7fa3200a59f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa33001a000 tx=0x7fa33001c000 comp rx=0 tx=0).stop 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.071+0000 7fa33d781640 1 -- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa338101c80 msgr2=0x7fa3381961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.071+0000 7fa33d781640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa338101c80 0x7fa3381961b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fa328009780 tx=0x7fa328030940 comp rx=0 tx=0).stop 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.072+0000 7fa33d781640 1 -- 172.21.15.62:0/17026975 shutdown_connections 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.072+0000 7fa33d781640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa3200a3530 0x7fa3200a59f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.072+0000 7fa33d781640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa338102640 0x7fa33819d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.072+0000 7fa33d781640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa338101c80 0x7fa3381961b0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.072+0000 7fa33d781640 1 --2- 172.21.15.62:0/17026975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa338100f70 0x7fa338195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.073+0000 7fa33d781640 1 -- 172.21.15.62:0/17026975 >> 172.21.15.62:0/17026975 conn(0x7fa3380fb930 msgr2=0x7fa338104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.073+0000 7fa33d781640 1 -- 172.21.15.62:0/17026975 shutdown_connections 2024-09-15T23:29:23.076 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:23.073+0000 7fa33d781640 1 -- 172.21.15.62:0/17026975 wait complete. 2024-09-15T23:29:23.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:23 smithi062 ceph-mon[30949]: pgmap v763: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.6 MiB/s wr, 686 op/s 2024-09-15T23:29:23.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:23 smithi067 ceph-mon[35665]: pgmap v763: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.6 MiB/s wr, 686 op/s 2024-09-15T23:29:23.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:23 smithi148 ceph-mon[35610]: pgmap v763: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 7.6 MiB/s wr, 686 op/s 2024-09-15T23:29:24.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 785.0 s, 386.6 tps, lat 256.673 ms stddev 430.901 2024-09-15T23:29:25.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:25 smithi062 ceph-mon[30949]: pgmap v764: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 965 op/s 2024-09-15T23:29:25.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:25 smithi067 ceph-mon[35665]: pgmap v764: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 965 op/s 2024-09-15T23:29:25.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:25 smithi148 ceph-mon[35610]: pgmap v764: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 12 MiB/s wr, 965 op/s 2024-09-15T23:29:25.908 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 26.5 2024-09-15T23:29:25.909 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:29:26.319 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:26.388 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:29:27.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:27 smithi062 ceph-mon[30949]: pgmap v765: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 16 MiB/s wr, 1.23k op/s 2024-09-15T23:29:27.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:27 smithi067 ceph-mon[35665]: pgmap v765: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 16 MiB/s wr, 1.23k op/s 2024-09-15T23:29:27.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:27 smithi148 ceph-mon[35610]: pgmap v765: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 16 MiB/s wr, 1.23k op/s 2024-09-15T23:29:29.451 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:29:29.451 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":35,"btime":"2024-09-15T23:24:52:780284+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:24:52.385385+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24466,24470,24497]},"id":1}]} 2024-09-15T23:29:29.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.181+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1801361653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 msgr2=0x7fb8b41021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:29.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.181+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1801361653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 0x7fb8b41021f0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4009900 tx=0x7fb8a402a3f0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.181+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1801361653 shutdown_connections 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.181+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1801361653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4106f40 0x7fb8b410eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.181+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1801361653 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8b41027c0 0x7fb8b4106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.181+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1801361653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 0x7fb8b41021f0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.181+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1801361653 >> 172.21.15.62:0/1801361653 conn(0x7fb8b40fd8d0 msgr2=0x7fb8b40ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.181+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1801361653 shutdown_connections 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.181+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1801361653 wait complete. 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b8e97640 1 Processor -- start 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b8e97640 1 -- start start 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b8e97640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 0x7fb8b40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b8e97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b41027c0 0x7fb8b4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:29.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b8e97640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8b4106f40 0x7fb8b4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b8e97640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b4111f60 con 0x7fb8b4101e10 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b8e97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b4111de0 con 0x7fb8b41027c0 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b8e97640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b41120e0 con 0x7fb8b4106f40 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 0x7fb8b40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 0x7fb8b40744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38318/0 (socket says 172.21.15.62:38318) 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b259c640 1 -- 172.21.15.62:0/1918098260 learned_addr learned my addr 172.21.15.62:0/1918098260 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b1d9b640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b41027c0 0x7fb8b4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.182+0000 7fb8b2d9d640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8b4106f40 0x7fb8b4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8b259c640 1 -- 172.21.15.62:0/1918098260 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8b4106f40 msgr2=0x7fb8b4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8b259c640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8b4106f40 0x7fb8b4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8b259c640 1 -- 172.21.15.62:0/1918098260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b41027c0 msgr2=0x7fb8b4074a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8b259c640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b41027c0 0x7fb8b4074a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8b259c640 1 -- 172.21.15.62:0/1918098260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b41a6890 con 0x7fb8b4101e10 2024-09-15T23:29:29.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8b259c640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 0x7fb8b40744f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4006010 tx=0x7fb8a40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8ab7fe640 1 -- 172.21.15.62:0/1918098260 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a403f030 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8ab7fe640 1 -- 172.21.15.62:0/1918098260 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8a4003ce0 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b41a6b20 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8ab7fe640 1 -- 172.21.15.62:0/1918098260 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4035070 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.183+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b41a7000 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.184+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b41021f0 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.185+0000 7fb8ab7fe640 1 -- 172.21.15.62:0/1918098260 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb8a4042030 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.186+0000 7fb8ab7fe640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8a00a3240 0x7fb8a00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.186+0000 7fb8b1d9b640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8a00a3240 0x7fb8a00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.186+0000 7fb8ab7fe640 1 -- 172.21.15.62:0/1918098260 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb8a40f0510 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.186+0000 7fb8b1d9b640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8a00a3240 0x7fb8a00a5700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4075f80 tx=0x7fb89c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.189+0000 7fb8ab7fe640 1 -- 172.21.15.62:0/1918098260 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb8a40bceb0 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.530+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8b4075a20 con 0x7fb8b4101e10 2024-09-15T23:29:29.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.532+0000 7fb8ab7fe640 1 -- 172.21.15.62:0/1918098260 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11903 (secure 0 0 0) 0x7fb8b4075a20 con 0x7fb8b4101e10 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 35 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8a00a3240 msgr2=0x7fb8a00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8a00a3240 0x7fb8a00a5700 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4075f80 tx=0x7fb89c00a2d0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 msgr2=0x7fb8b40744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 0x7fb8b40744f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb8a4006010 tx=0x7fb8a40042f0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 shutdown_connections 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb8a00a3240 0x7fb8a00a5700 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb8b4106f40 0x7fb8b4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b41027c0 0x7fb8b4074a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 --2- 172.21.15.62:0/1918098260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8b4101e10 0x7fb8b40744f0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 >> 172.21.15.62:0/1918098260 conn(0x7fb8b40fd8d0 msgr2=0x7fb8b40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 shutdown_connections 2024-09-15T23:29:29.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:28.534+0000 7fb8b8e97640 1 -- 172.21.15.62:0/1918098260 wait complete. 2024-09-15T23:29:29.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 790.0 s, 396.4 tps, lat 250.297 ms stddev 313.315 2024-09-15T23:29:29.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:29 smithi062 ceph-mon[30949]: pgmap v766: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 17 MiB/s wr, 1.34k op/s 2024-09-15T23:29:29.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:29 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1918098260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:29 smithi067 ceph-mon[35665]: pgmap v766: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 17 MiB/s wr, 1.34k op/s 2024-09-15T23:29:29.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:29 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1918098260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:29 smithi148 ceph-mon[35610]: pgmap v766: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 17 MiB/s wr, 1.34k op/s 2024-09-15T23:29:29.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:29 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1918098260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:31.355 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 146.5 --await-for 45.0' 2024-09-15T23:29:31.355 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 146.5 --await-for 45.0 2024-09-15T23:29:31.566 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:31.721 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:31 smithi062 ceph-mon[30949]: pgmap v767: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 18 MiB/s wr, 1.43k op/s 2024-09-15T23:29:31.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:31 smithi067 ceph-mon[35665]: pgmap v767: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 18 MiB/s wr, 1.43k op/s 2024-09-15T23:29:31.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:31 smithi148 ceph-mon[35610]: pgmap v767: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 18 MiB/s wr, 1.43k op/s 2024-09-15T23:29:32.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.213+0000 7f65ffe6c640 1 -- 172.21.15.62:0/669316656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f8101dd0 msgr2=0x7f65f81021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:32.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.213+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/669316656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f8101dd0 0x7f65f81021b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009900 tx=0x7f65f402a410 comp rx=0 tx=0).stop 2024-09-15T23:29:32.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.213+0000 7f65ffe6c640 1 -- 172.21.15.62:0/669316656 shutdown_connections 2024-09-15T23:29:32.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.213+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/669316656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8106f00 0x7f65f810ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.213+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/669316656 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65f8102780 0x7f65f81067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.213+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/669316656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f8101dd0 0x7f65f81021b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.213+0000 7f65ffe6c640 1 -- 172.21.15.62:0/669316656 >> 172.21.15.62:0/669316656 conn(0x7f65f80fd8b0 msgr2=0x7f65f80ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:32.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.213+0000 7f65ffe6c640 1 -- 172.21.15.62:0/669316656 shutdown_connections 2024-09-15T23:29:32.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.213+0000 7f65ffe6c640 1 -- 172.21.15.62:0/669316656 wait complete. 2024-09-15T23:29:32.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65ffe6c640 1 Processor -- start 2024-09-15T23:29:32.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65ffe6c640 1 -- start start 2024-09-15T23:29:32.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65ffe6c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f8101dd0 0x7f65f80744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65ffe6c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65f8102780 0x7f65f8074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65ffe6c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8106f00 0x7f65f8074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65ffe6c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8111fb0 con 0x7f65f8106f00 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65ffe6c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8111e30 con 0x7f65f8101dd0 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65ffe6c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8112130 con 0x7f65f8102780 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65fe409640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8106f00 0x7f65f8074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65fd407640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65f8102780 0x7f65f8074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65fd407640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65f8102780 0x7f65f8074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38438/0 (socket says 172.21.15.62:38438) 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.214+0000 7f65fd407640 1 -- 172.21.15.62:0/2139262182 learned_addr learned my addr 172.21.15.62:0/2139262182 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65fdc08640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f8101dd0 0x7f65f80744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65fd407640 1 -- 172.21.15.62:0/2139262182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f8101dd0 msgr2=0x7f65f80744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65fd407640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f8101dd0 0x7f65f80744f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65fd407640 1 -- 172.21.15.62:0/2139262182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8106f00 msgr2=0x7f65f8074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:32.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65fd407640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8106f00 0x7f65f8074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65fd407640 1 -- 172.21.15.62:0/2139262182 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f81a6910 con 0x7f65f8102780 2024-09-15T23:29:32.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65fd407640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65f8102780 0x7f65f8074a30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f65e800e9f0 tx=0x7f65e800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:32.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65eeffd640 1 -- 172.21.15.62:0/2139262182 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e800cd80 con 0x7f65f8102780 2024-09-15T23:29:32.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65ffe6c640 1 -- 172.21.15.62:0/2139262182 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f81a6c00 con 0x7f65f8102780 2024-09-15T23:29:32.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65eeffd640 1 -- 172.21.15.62:0/2139262182 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e8010910 con 0x7f65f8102780 2024-09-15T23:29:32.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.215+0000 7f65ffe6c640 1 -- 172.21.15.62:0/2139262182 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f81a7190 con 0x7f65f8102780 2024-09-15T23:29:32.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.216+0000 7f65eeffd640 1 -- 172.21.15.62:0/2139262182 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e8018990 con 0x7f65f8102780 2024-09-15T23:29:32.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.218+0000 7f65eeffd640 1 -- 172.21.15.62:0/2139262182 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f65e8018b30 con 0x7f65f8102780 2024-09-15T23:29:32.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.219+0000 7f6602245740 1 Processor -- start 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.219+0000 7f6602245740 1 -- start start 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.219+0000 7f6602245740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30bcb0 0x564f5b30c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.219+0000 7f6602245740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30c5d0 0x564f5b316950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.219+0000 7f6602245740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 0x564f5b319280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.219+0000 7f6602245740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564f5b2692e0 con 0x564f5b30c5d0 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.219+0000 7f6602245740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564f5b210ce0 con 0x564f5b30bcb0 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.219+0000 7f6602245740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564f5b2652a0 con 0x564f5b316e90 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.220+0000 7f65fe409640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30bcb0 0x564f5b30c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.220+0000 7f65fe409640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30bcb0 0x564f5b30c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49928/0 (socket says 172.21.15.62:49928) 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.220+0000 7f65fe409640 1 -- 172.21.15.62:0/1450605162 learned_addr learned my addr 172.21.15.62:0/1450605162 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.220+0000 7f65fe409640 1 -- 172.21.15.62:0/1450605162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 msgr2=0x564f5b319280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:29:32.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.220+0000 7f65fe409640 1 --2- 172.21.15.62:0/1450605162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 0x564f5b319280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.220+0000 7f65fe409640 1 -- 172.21.15.62:0/1450605162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30c5d0 msgr2=0x564f5b316950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.220+0000 7f65fe409640 1 --2- 172.21.15.62:0/1450605162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30c5d0 0x564f5b316950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.220+0000 7f65fe409640 1 -- 172.21.15.62:0/1450605162 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f5b1f5440 con 0x564f5b30bcb0 2024-09-15T23:29:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.221+0000 7f65eeffd640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65e40a32c0 0x7f65e40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.221+0000 7f65fe409640 1 --2- 172.21.15.62:0/1450605162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30bcb0 0x564f5b30c090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009f50 tx=0x7f65f00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b49745a99c7a7f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.221+0000 7f65eeffd640 1 -- 172.21.15.62:0/2139262182 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f65e80cfef0 con 0x7f65f8102780 2024-09-15T23:29:32.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.221+0000 7f65e3fff640 1 -- 172.21.15.62:0/1450605162 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0004b20 con 0x564f5b30bcb0 2024-09-15T23:29:32.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.221+0000 7f65e3fff640 1 -- 172.21.15.62:0/1450605162 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f0004cc0 con 0x564f5b30bcb0 2024-09-15T23:29:32.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.221+0000 7f65fdc08640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65e40a32c0 0x7f65e40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.222+0000 7f6602245740 1 -- 172.21.15.62:0/1450605162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30bcb0 msgr2=0x564f5b30c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:32.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.222+0000 7f6602245740 1 --2- 172.21.15.62:0/1450605162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30bcb0 0x564f5b30c090 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009f50 tx=0x7f65f00118d0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.223+0000 7f65fdc08640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65e40a32c0 0x7f65e40a5780 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f65f40096c0 tx=0x7f65f4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:32.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.223+0000 7f6602245740 1 -- 172.21.15.62:0/1450605162 shutdown_connections 2024-09-15T23:29:32.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.223+0000 7f6602245740 1 --2- 172.21.15.62:0/1450605162 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 0x564f5b319280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.223+0000 7f6602245740 1 --2- 172.21.15.62:0/1450605162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30c5d0 0x564f5b316950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.223+0000 7f6602245740 1 --2- 172.21.15.62:0/1450605162 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30bcb0 0x564f5b30c090 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.223+0000 7f6602245740 1 -- 172.21.15.62:0/1450605162 >> 172.21.15.62:0/1450605162 conn(0x564f5b308240 msgr2=0x564f5b308620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:32.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.225+0000 7f6602245740 1 -- 172.21.15.62:0/1450605162 shutdown_connections 2024-09-15T23:29:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.225+0000 7f6602245740 1 -- 172.21.15.62:0/1450605162 wait complete. 2024-09-15T23:29:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.225+0000 7f6602245740 1 -- start start 2024-09-15T23:29:32.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.225+0000 7f6602245740 1 Processor -- start 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f6602245740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b316740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f6602245740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 0x564f5b3139f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f6602245740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b313f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f65fdc08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b316740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f65fdc08640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b316740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38354/0 (socket says 172.21.15.62:38354) 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f65fdc08640 1 -- 172.21.15.62:0/2646373868 learned_addr learned my addr 172.21.15.62:0/2646373868 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f65fd407640 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 0x564f5b3139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564f5b319ab0 con 0x564f5b30bcb0 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564f5b224050 con 0x564f5b316e90 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564f5b319c30 con 0x564f5b30c5d0 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f65fe409640 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b313f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.226+0000 7f6602245740 10 client.0 _ll_register_callbacks cb 0x564f5b264160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:32.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.227+0000 7f65fdc08640 1 -- 172.21.15.62:0/2646373868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 msgr2=0x564f5b3139f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:32.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.227+0000 7f65fdc08640 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 0x564f5b3139f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.227+0000 7f65fdc08640 1 -- 172.21.15.62:0/2646373868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 msgr2=0x564f5b313f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:32.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.227+0000 7f65fdc08640 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b313f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.227+0000 7f65fdc08640 1 -- 172.21.15.62:0/2646373868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f5afda3c0 con 0x564f5b30bcb0 2024-09-15T23:29:32.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.227+0000 7f65fd407640 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 0x564f5b3139f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:32.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.227+0000 7f65fe409640 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b313f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:32.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.227+0000 7f65fdc08640 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b316740 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f65f40092a0 tx=0x7f65f402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e25223ff7447023 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:32.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.227+0000 7f65e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:32.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.228+0000 7f65e17fa640 1 -- 172.21.15.62:0/2646373868 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4041070 con 0x564f5b30bcb0 2024-09-15T23:29:32.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.228+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f0004cc0 con 0x564f5b30bcb0 2024-09-15T23:29:32.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.228+0000 7f65e17fa640 1 -- 172.21.15.62:0/2646373868 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f65f4005590 con 0x564f5b30bcb0 2024-09-15T23:29:32.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.229+0000 7f65e17fa640 1 -- 172.21.15.62:0/2646373868 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f65f4035050 con 0x564f5b30bcb0 2024-09-15T23:29:32.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.229+0000 7f6602245740 10 client.18486 fetch_fsmap learned FSMap version 35 2024-09-15T23:29:32.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.229+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564f5b3254c0 con 0x564f5b30bcb0 2024-09-15T23:29:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.229+0000 7f65e17fa640 1 -- 172.21.15.62:0/2646373868 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f0004cc0 con 0x564f5b30bcb0 2024-09-15T23:29:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.229+0000 7f65e17fa640 1 -- 172.21.15.62:0/2646373868 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f65f4003c00 con 0x564f5b30bcb0 2024-09-15T23:29:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.230+0000 7f6602245740 10 client.18486 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:29:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.230+0000 7f6602245740 10 client.18486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.230+0000 7f6602245740 20 client.18486 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.230+0000 7f6602245740 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f5b3269d0 0x564f5b328df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.230+0000 7f6602245740 4 client.18486 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:32.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.230+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f5b276850 con 0x564f5b3269d0 2024-09-15T23:29:32.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.237+0000 7f65fd407640 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f5b3269d0 0x564f5b328df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.249+0000 7f65fd407640 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f5b3269d0 0x564f5b328df0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea21df0682ee8dad server_cookie=9743a46e56b47a25 in_seq=0 out_seq=0 2024-09-15T23:29:32.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.249+0000 7f65e17fa640 10 client.18486 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:32.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.253+0000 7f65e17fa640 1 -- 172.21.15.62:0/2646373868 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564f5b276850 con 0x564f5b3269d0 2024-09-15T23:29:32.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.253+0000 7f65e17fa640 10 client.18486 handle_command_reply: tid=0 2024-09-15T23:29:32.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.253+0000 7f6602245740 1 client.18486 shutdown 2024-09-15T23:29:32.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.258+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 msgr2=0x564f5b316740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:32.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.258+0000 7f6602245740 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b316740 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f65f40092a0 tx=0x7f65f402a410 comp rx=0 tx=0).stop 2024-09-15T23:29:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.268+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 shutdown_connections 2024-09-15T23:29:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.268+0000 7f6602245740 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f5b3269d0 0x564f5b328df0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.268+0000 7f6602245740 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b313f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.268+0000 7f6602245740 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 0x564f5b3139f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.268+0000 7f6602245740 1 --2- 172.21.15.62:0/2646373868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b316740 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.268+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 >> 172.21.15.62:0/2646373868 conn(0x564f5b308240 msgr2=0x564f5b30df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:32.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.268+0000 7f65e17fa640 0 client.18486 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:32.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.276+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 shutdown_connections 2024-09-15T23:29:32.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.276+0000 7f6602245740 1 -- 172.21.15.62:0/2646373868 wait complete. 2024-09-15T23:29:32.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.276+0000 7f6602245740 20 client.18486 trim_cache size 0 max 16384 2024-09-15T23:29:32.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.313+0000 7f6602245740 1 Processor -- start 2024-09-15T23:29:32.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.313+0000 7f6602245740 1 -- start start 2024-09-15T23:29:32.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.313+0000 7f6602245740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b32e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f6602245740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 0x564f5b32ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f6602245740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b31ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f6602245740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc012490 con 0x564f5b30bcb0 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f6602245740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc00fd60 con 0x564f5b316e90 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f6602245740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc00dbb0 con 0x564f5b30c5d0 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f65fe409640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b31ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f65fe409640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b31ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52734/0 (socket says 172.21.15.62:52734) 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f65fe409640 1 -- 172.21.15.62:0/1910012936 learned_addr learned my addr 172.21.15.62:0/1910012936 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f65fd407640 1 --2- 172.21.15.62:0/1910012936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 0x564f5b32ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.314+0000 7f65fe409640 1 -- 172.21.15.62:0/1910012936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 msgr2=0x564f5b32ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:32.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.315+0000 7f65fe409640 1 --2- 172.21.15.62:0/1910012936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 0x564f5b32ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.315+0000 7f65fe409640 1 -- 172.21.15.62:0/1910012936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 msgr2=0x564f5b32e6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:29:32.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.315+0000 7f65fe409640 1 --2- 172.21.15.62:0/1910012936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b32e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.315+0000 7f65fe409640 1 -- 172.21.15.62:0/1910012936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f5b308070 con 0x564f5b316e90 2024-09-15T23:29:32.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.316+0000 7f65fe409640 1 --2- 172.21.15.62:0/1910012936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b31ef50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f65f0011dd0 tx=0x7f65f000fe30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bbeb6917c8f9572 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:32.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.316+0000 7f65e37fe640 1 -- 172.21.15.62:0/1910012936 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f004a020 con 0x564f5b316e90 2024-09-15T23:29:32.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.317+0000 7f65e37fe640 1 -- 172.21.15.62:0/1910012936 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f0003e50 con 0x564f5b316e90 2024-09-15T23:29:32.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.317+0000 7f65e37fe640 1 -- 172.21.15.62:0/1910012936 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0046470 con 0x564f5b316e90 2024-09-15T23:29:32.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.318+0000 7f6602245740 1 -- 172.21.15.62:0/1910012936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 msgr2=0x564f5b31ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:32.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.318+0000 7f6602245740 1 --2- 172.21.15.62:0/1910012936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b31ef50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f65f0011dd0 tx=0x7f65f000fe30 comp rx=0 tx=0).stop 2024-09-15T23:29:32.320 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:32.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.319+0000 7f6602245740 1 -- 172.21.15.62:0/1910012936 shutdown_connections 2024-09-15T23:29:32.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.319+0000 7f6602245740 1 --2- 172.21.15.62:0/1910012936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b316e90 0x564f5b31ef50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.319+0000 7f6602245740 1 --2- 172.21.15.62:0/1910012936 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b30c5d0 0x564f5b32ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.319+0000 7f6602245740 1 --2- 172.21.15.62:0/1910012936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b32e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.319+0000 7f6602245740 1 -- 172.21.15.62:0/1910012936 >> 172.21.15.62:0/1910012936 conn(0x564f5b308240 msgr2=0x564f5b2feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:32.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.325+0000 7f6602245740 1 -- 172.21.15.62:0/1910012936 shutdown_connections 2024-09-15T23:29:32.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.328+0000 7f6602245740 1 -- 172.21.15.62:0/1910012936 wait complete. 2024-09-15T23:29:32.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.328+0000 7f6602245740 1 -- start start 2024-09-15T23:29:32.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.329+0000 7f6602245740 1 Processor -- start 2024-09-15T23:29:32.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.329+0000 7f6602245740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b31c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.330+0000 7f6602245740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30c5d0 0x564f5b31c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.330+0000 7f6602245740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 0x564f5b38f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.330+0000 7f6602245740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc00e220 con 0x564f5b30bcb0 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.330+0000 7f65fd407640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30c5d0 0x564f5b31c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.330+0000 7f65fd407640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30c5d0 0x564f5b31c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52738/0 (socket says 172.21.15.62:52738) 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.330+0000 7f65fd407640 1 -- 172.21.15.62:0/2844313897 learned_addr learned my addr 172.21.15.62:0/2844313897 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.330+0000 7f65fe409640 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 0x564f5b38f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.330+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc00f690 con 0x564f5b30c5d0 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.330+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc00e8f0 con 0x564f5b316e90 2024-09-15T23:29:32.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.331+0000 7f6602245740 10 client.0 _ll_register_callbacks cb 0x7f65f00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:32.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.335+0000 7f65fe409640 1 -- 172.21.15.62:0/2844313897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30c5d0 msgr2=0x564f5b31c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:32.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.335+0000 7f65fe409640 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30c5d0 0x564f5b31c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.335+0000 7f65fe409640 1 -- 172.21.15.62:0/2844313897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 msgr2=0x564f5b31c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:32.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.335+0000 7f65fe409640 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b31c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.335+0000 7f65fe409640 1 -- 172.21.15.62:0/2844313897 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f5b38fbd0 con 0x564f5b316e90 2024-09-15T23:29:32.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.335+0000 7f65fe409640 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 0x564f5b38f690 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f65f001b040 tx=0x7f65f000f850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8690ba1ff51682d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:32.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.335+0000 7f65e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:29:32.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.335+0000 7f65e17fa640 1 -- 172.21.15.62:0/2844313897 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f0003b80 con 0x564f5b316e90 2024-09-15T23:29:32.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.335+0000 7f65e17fa640 1 -- 172.21.15.62:0/2844313897 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f65f0046470 con 0x564f5b316e90 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.336+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65f0003e50 con 0x564f5b316e90 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.336+0000 7f65e17fa640 1 -- 172.21.15.62:0/2844313897 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f65f0061050 con 0x564f5b316e90 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.336+0000 7f6602245740 10 client.27736 fetch_fsmap learned FSMap version 35 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.336+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564f5b390e90 con 0x564f5b316e90 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.337+0000 7f65e17fa640 1 -- 172.21.15.62:0/2844313897 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f0055070 con 0x564f5b316e90 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.337+0000 7f65e17fa640 1 -- 172.21.15.62:0/2844313897 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 35) ==== 8354+0+0 (secure 0 0 0) 0x7f65f005a740 con 0x564f5b316e90 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.337+0000 7f6602245740 10 client.27736 fetch_fsmap finished waiting for FSMap version 35 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.337+0000 7f6602245740 10 client.27736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.337+0000 7f6602245740 20 client.27736 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.337+0000 7f6602245740 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f5b391210 0x564f5b393600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.338+0000 7f6602245740 4 client.27736 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:29:32.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.338+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f65f0003e50 con 0x564f5b391210 2024-09-15T23:29:32.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.339+0000 7f65fdc08640 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f5b391210 0x564f5b393600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:32.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f65e17fa640 10 client.27736 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:32.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f65fdc08640 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f5b391210 0x564f5b393600 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d59157eb1c0a3e1 server_cookie=aa9cc6eb8b1ebec5 in_seq=0 out_seq=0 2024-09-15T23:29:32.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f65e17fa640 1 -- 172.21.15.62:0/2844313897 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f65f0003e50 con 0x564f5b391210 2024-09-15T23:29:32.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f65e17fa640 10 client.27736 handle_command_reply: tid=0 2024-09-15T23:29:32.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f6602245740 1 client.27736 shutdown 2024-09-15T23:29:32.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 msgr2=0x564f5b38f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:32.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f6602245740 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 0x564f5b38f690 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f65f001b040 tx=0x7f65f000f850 comp rx=0 tx=0).stop 2024-09-15T23:29:32.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 shutdown_connections 2024-09-15T23:29:32.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f6602245740 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564f5b391210 0x564f5b393600 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f6602245740 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564f5b316e90 0x564f5b38f690 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f6602245740 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564f5b30c5d0 0x564f5b31c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f6602245740 1 --2- 172.21.15.62:0/2844313897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f5b30bcb0 0x564f5b31c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 >> 172.21.15.62:0/2844313897 conn(0x564f5b308240 msgr2=0x564f5b315250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:32.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.352+0000 7f65e17fa640 0 client.27736 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:32.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.354+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 shutdown_connections 2024-09-15T23:29:32.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.354+0000 7f6602245740 1 -- 172.21.15.62:0/2844313897 wait complete. 2024-09-15T23:29:32.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.354+0000 7f6602245740 20 client.27736 trim_cache size 0 max 16384 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 -- 172.21.15.62:0/2139262182 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65e40a32c0 msgr2=0x7f65e40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65e40a32c0 0x7f65e40a5780 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f65f40096c0 tx=0x7f65f4000bc0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 -- 172.21.15.62:0/2139262182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65f8102780 msgr2=0x7f65f8074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65f8102780 0x7f65f8074a30 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f65e800e9f0 tx=0x7f65e800eeb0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 -- 172.21.15.62:0/2139262182 shutdown_connections 2024-09-15T23:29:32.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65e40a32c0 0x7f65e40a5780 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f8106f00 0x7f65f8074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f65f8102780 0x7f65f8074a30 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 --2- 172.21.15.62:0/2139262182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65f8101dd0 0x7f65f80744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:32.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 -- 172.21.15.62:0/2139262182 >> 172.21.15.62:0/2139262182 conn(0x7f65f80fd8b0 msgr2=0x7f65f80fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:32.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 -- 172.21.15.62:0/2139262182 shutdown_connections 2024-09-15T23:29:32.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:32.356+0000 7f65ffe6c640 1 -- 172.21.15.62:0/2139262182 wait complete. 2024-09-15T23:29:33.308 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:29:33.309 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:29:33.581 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:33 smithi062 ceph-mon[30949]: pgmap v768: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 MiB/s wr, 1.52k op/s 2024-09-15T23:29:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:33 smithi067 ceph-mon[35665]: pgmap v768: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 MiB/s wr, 1.52k op/s 2024-09-15T23:29:33.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:33 smithi148 ceph-mon[35610]: pgmap v768: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 20 MiB/s wr, 1.52k op/s 2024-09-15T23:29:33.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.917+0000 7f2b8cd40640 1 -- 172.21.15.62:0/3966372680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b8806ea30 msgr2=0x7f2b8806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:33.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.917+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/3966372680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b8806ea30 0x7f2b8806ee70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009860 tx=0x7f2b7802a360 comp rx=0 tx=0).stop 2024-09-15T23:29:33.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.918+0000 7f2b8cd40640 1 -- 172.21.15.62:0/3966372680 shutdown_connections 2024-09-15T23:29:33.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.918+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/3966372680 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b8811b140 0x7f2b8811d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.918+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/3966372680 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b8806ea30 0x7f2b8806ee70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.918+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/3966372680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b88110190 0x7f2b8806e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.918+0000 7f2b8cd40640 1 -- 172.21.15.62:0/3966372680 >> 172.21.15.62:0/3966372680 conn(0x7f2b88073180 msgr2=0x7f2b88073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:33.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.918+0000 7f2b8cd40640 1 -- 172.21.15.62:0/3966372680 shutdown_connections 2024-09-15T23:29:33.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.918+0000 7f2b8cd40640 1 -- 172.21.15.62:0/3966372680 wait complete. 2024-09-15T23:29:33.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.919+0000 7f2b8cd40640 1 Processor -- start 2024-09-15T23:29:33.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.919+0000 7f2b8cd40640 1 -- start start 2024-09-15T23:29:33.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.920+0000 7f2b8cd40640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b8806ea30 0x7f2b881196d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.920+0000 7f2b8cd40640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b88110190 0x7f2b881127b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.920+0000 7f2b8cd40640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811b140 0x7f2b88112cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.920+0000 7f2b8cd40640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88120be0 con 0x7f2b8806ea30 2024-09-15T23:29:33.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.920+0000 7f2b8cd40640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88120a60 con 0x7f2b8811b140 2024-09-15T23:29:33.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.920+0000 7f2b8cd40640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2b88120d60 con 0x7f2b88110190 2024-09-15T23:29:33.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.921+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b8806ea30 0x7f2b881196d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:33.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.921+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b8806ea30 0x7f2b881196d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35856/0 (socket says 172.21.15.62:35856) 2024-09-15T23:29:33.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.921+0000 7f2b877fe640 1 -- 172.21.15.62:0/1658146965 learned_addr learned my addr 172.21.15.62:0/1658146965 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:33.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.922+0000 7f2b877fe640 1 -- 172.21.15.62:0/1658146965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b88110190 msgr2=0x7f2b881127b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:29:33.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.922+0000 7f2b877fe640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b88110190 0x7f2b881127b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.922+0000 7f2b877fe640 1 -- 172.21.15.62:0/1658146965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811b140 msgr2=0x7f2b88112cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:29:33.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.922+0000 7f2b877fe640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811b140 0x7f2b88112cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.922+0000 7f2b877fe640 1 -- 172.21.15.62:0/1658146965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b88113260 con 0x7f2b8806ea30 2024-09-15T23:29:33.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.923+0000 7f2b877fe640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b8806ea30 0x7f2b881196d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000ebc0 tx=0x7f2b8000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:33.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.924+0000 7f2b84ff9640 1 -- 172.21.15.62:0/1658146965 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80010630 con 0x7f2b8806ea30 2024-09-15T23:29:33.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.924+0000 7f2b8cd40640 1 -- 172.21.15.62:0/1658146965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b88113550 con 0x7f2b8806ea30 2024-09-15T23:29:33.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.924+0000 7f2b8cd40640 1 -- 172.21.15.62:0/1658146965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b881174e0 con 0x7f2b8806ea30 2024-09-15T23:29:33.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.924+0000 7f2b84ff9640 1 -- 172.21.15.62:0/1658146965 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b80010cb0 con 0x7f2b8806ea30 2024-09-15T23:29:33.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.924+0000 7f2b84ff9640 1 -- 172.21.15.62:0/1658146965 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80018c00 con 0x7f2b8806ea30 2024-09-15T23:29:33.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.930+0000 7f2b9037b740 1 Processor -- start 2024-09-15T23:29:33.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.933+0000 7f2b84ff9640 1 -- 172.21.15.62:0/1658146965 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2b80018420 con 0x7f2b8806ea30 2024-09-15T23:29:33.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.934+0000 7f2b9037b740 1 -- start start 2024-09-15T23:29:33.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.934+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 0x561c83d27ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.934+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.934+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d32ae0 0x561c83d34ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.934+0000 7f2b9037b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561c83c8d2a0 con 0x561c83d28220 2024-09-15T23:29:33.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.934+0000 7f2b9037b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561c83c34b10 con 0x561c83d27900 2024-09-15T23:29:33.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.934+0000 7f2b9037b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561c83c89260 con 0x561c83d32ae0 2024-09-15T23:29:33.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.934+0000 7f2b84ff9640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b700a3450 0x7f2b700a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.935+0000 7f2b84ff9640 1 -- 172.21.15.62:0/1658146965 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2b800d0310 con 0x7f2b8806ea30 2024-09-15T23:29:33.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.936+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:33.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.936+0000 7f2b877fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d325a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35858/0 (socket says 172.21.15.62:35858) 2024-09-15T23:29:33.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.936+0000 7f2b877fe640 1 -- 172.21.15.62:0/2784022000 learned_addr learned my addr 172.21.15.62:0/2784022000 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:33.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.936+0000 7f2b87fff640 1 --2- 172.21.15.62:0/2784022000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 0x561c83d27ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:33.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.936+0000 7f2b87fff640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b700a3450 0x7f2b700a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:33.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.937+0000 7f2b86ffd640 1 --2- 172.21.15.62:0/2784022000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d32ae0 0x561c83d34ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:33.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.937+0000 7f2b877fe640 1 -- 172.21.15.62:0/2784022000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d32ae0 msgr2=0x561c83d34ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:33.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.937+0000 7f2b877fe640 1 --2- 172.21.15.62:0/2784022000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d32ae0 0x561c83d34ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.937+0000 7f2b877fe640 1 -- 172.21.15.62:0/2784022000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 msgr2=0x561c83d27ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:33.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.937+0000 7f2b877fe640 1 --2- 172.21.15.62:0/2784022000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 0x561c83d27ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.937+0000 7f2b877fe640 1 -- 172.21.15.62:0/2784022000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c839feb90 con 0x561c83d28220 2024-09-15T23:29:33.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.937+0000 7f2b877fe640 1 --2- 172.21.15.62:0/2784022000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d325a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2b80018850 tx=0x7f2b8009beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66275eabacd5d7c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:33.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.938+0000 7f2b6dffb640 1 -- 172.21.15.62:0/2784022000 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80010e60 con 0x561c83d28220 2024-09-15T23:29:33.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.938+0000 7f2b6dffb640 1 -- 172.21.15.62:0/2784022000 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b8009c550 con 0x561c83d28220 2024-09-15T23:29:33.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.938+0000 7f2b6dffb640 1 -- 172.21.15.62:0/2784022000 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80098bc0 con 0x561c83d28220 2024-09-15T23:29:33.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.939+0000 7f2b9037b740 1 -- 172.21.15.62:0/2784022000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 msgr2=0x561c83d325a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:33.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.939+0000 7f2b9037b740 1 --2- 172.21.15.62:0/2784022000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d325a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2b80018850 tx=0x7f2b8009beb0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.939+0000 7f2b9037b740 1 -- 172.21.15.62:0/2784022000 shutdown_connections 2024-09-15T23:29:33.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.939+0000 7f2b9037b740 1 --2- 172.21.15.62:0/2784022000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d32ae0 0x561c83d34ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.939+0000 7f2b9037b740 1 --2- 172.21.15.62:0/2784022000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d325a0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.939+0000 7f2b9037b740 1 --2- 172.21.15.62:0/2784022000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 0x561c83d27ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.939+0000 7f2b9037b740 1 -- 172.21.15.62:0/2784022000 >> 172.21.15.62:0/2784022000 conn(0x561c83d23e90 msgr2=0x561c83d24270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:33.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.939+0000 7f2b87fff640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b700a3450 0x7f2b700a5910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005ab0 tx=0x7f2b7c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:33.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.947+0000 7f2b9037b740 1 -- 172.21.15.62:0/2784022000 shutdown_connections 2024-09-15T23:29:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.948+0000 7f2b9037b740 1 -- 172.21.15.62:0/2784022000 wait complete. 2024-09-15T23:29:33.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.948+0000 7f2b9037b740 1 -- start start 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.949+0000 7f2b9037b740 1 Processor -- start 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.949+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 0x561c83d2f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.949+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d2fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.949+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d32ae0 0x561c83d30340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.949+0000 7f2b9037b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561c83d356b0 con 0x561c83d28220 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.949+0000 7f2b9037b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561c83c156e0 con 0x561c83d32ae0 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.949+0000 7f2b9037b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561c83d35830 con 0x561c83d27900 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.949+0000 7f2b9037b740 10 client.0 _ll_register_callbacks cb 0x561c83c39a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.951+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d2fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.951+0000 7f2b86ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d2fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35872/0 (socket says 172.21.15.62:35872) 2024-09-15T23:29:33.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.951+0000 7f2b86ffd640 1 -- 172.21.15.62:0/3714258273 learned_addr learned my addr 172.21.15.62:0/3714258273 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:33.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.952+0000 7f2b877fe640 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 0x561c83d2f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:33.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.953+0000 7f2b877fe640 1 -- 172.21.15.62:0/3714258273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d32ae0 msgr2=0x561c83d30340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.953+0000 7f2b877fe640 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d32ae0 0x561c83d30340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.953+0000 7f2b877fe640 1 -- 172.21.15.62:0/3714258273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 msgr2=0x561c83d2fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.953+0000 7f2b877fe640 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d2fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.954+0000 7f2b877fe640 1 -- 172.21.15.62:0/3714258273 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c83d3ebb0 con 0x561c83d27900 2024-09-15T23:29:33.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.960+0000 7f2b877fe640 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 0x561c83d2f8c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b800a4c90 tx=0x7f2b800cd9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e3e1ee5e7c6fdef server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:33.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.961+0000 7f2b66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:29:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.965+0000 7f2b66ffd640 1 -- 172.21.15.62:0/3714258273 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b800b4020 con 0x561c83d27900 2024-09-15T23:29:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.965+0000 7f2b66ffd640 1 -- 172.21.15.62:0/3714258273 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2b8007c7c0 con 0x561c83d27900 2024-09-15T23:29:33.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.965+0000 7f2b9037b740 1 -- 172.21.15.62:0/3714258273 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b8009c550 con 0x561c83d27900 2024-09-15T23:29:33.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.969+0000 7f2b66ffd640 1 -- 172.21.15.62:0/3714258273 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2b80010e60 con 0x561c83d27900 2024-09-15T23:29:33.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.971+0000 7f2b9037b740 10 client.27742 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:33.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.971+0000 7f2b9037b740 1 -- 172.21.15.62:0/3714258273 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561c83d356b0 con 0x561c83d27900 2024-09-15T23:29:33.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.973+0000 7f2b66ffd640 1 -- 172.21.15.62:0/3714258273 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b8009c550 con 0x561c83d27900 2024-09-15T23:29:33.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.974+0000 7f2b66ffd640 1 -- 172.21.15.62:0/3714258273 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f2b8009b9a0 con 0x561c83d27900 2024-09-15T23:29:33.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.975+0000 7f2b9037b740 10 client.27742 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:33.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.975+0000 7f2b9037b740 10 client.27742 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:29:33.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.975+0000 7f2b9037b740 20 client.27742 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:33.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.975+0000 7f2b9037b740 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561c83d64650 0x561c83d66a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:33.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.975+0000 7f2b9037b740 4 client.27742 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:33.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.975+0000 7f2b9037b740 1 -- 172.21.15.62:0/3714258273 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c83c9a9d0 con 0x561c83d64650 2024-09-15T23:29:33.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.977+0000 7f2b86ffd640 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561c83d64650 0x561c83d66a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:33.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.985+0000 7f2b86ffd640 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561c83d64650 0x561c83d66a40 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc23c93f3259099 server_cookie=428f486a0f7a7783 in_seq=0 out_seq=0 2024-09-15T23:29:33.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.985+0000 7f2b66ffd640 10 client.27742 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:33.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.985+0000 7f2b66ffd640 1 -- 172.21.15.62:0/3714258273 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561c83c9a9d0 con 0x561c83d64650 2024-09-15T23:29:33.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.985+0000 7f2b66ffd640 10 client.27742 handle_command_reply: tid=0 2024-09-15T23:29:33.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.986+0000 7f2b9037b740 1 client.27742 shutdown 2024-09-15T23:29:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.993+0000 7f2b9037b740 1 -- 172.21.15.62:0/3714258273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 msgr2=0x561c83d2f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:33.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.993+0000 7f2b9037b740 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 0x561c83d2f8c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2b800a4c90 tx=0x7f2b800cd9a0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.997+0000 7f2b9037b740 1 -- 172.21.15.62:0/3714258273 shutdown_connections 2024-09-15T23:29:33.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.997+0000 7f2b9037b740 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561c83d64650 0x561c83d66a40 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.997+0000 7f2b9037b740 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d32ae0 0x561c83d30340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.997+0000 7f2b9037b740 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d28220 0x561c83d2fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.997+0000 7f2b9037b740 1 --2- 172.21.15.62:0/3714258273 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 0x561c83d2f8c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:33.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.997+0000 7f2b9037b740 1 -- 172.21.15.62:0/3714258273 >> 172.21.15.62:0/3714258273 conn(0x561c83d23e90 msgr2=0x561c83d29bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:33.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.997+0000 7f2b66ffd640 0 client.27742 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:34.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.999+0000 7f2b9037b740 1 -- 172.21.15.62:0/3714258273 shutdown_connections 2024-09-15T23:29:34.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:33.999+0000 7f2b9037b740 1 -- 172.21.15.62:0/3714258273 wait complete. 2024-09-15T23:29:34.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.001+0000 7f2b9037b740 20 client.27742 trim_cache size 0 max 16384 2024-09-15T23:29:34.010 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:34.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.036+0000 7f2b9037b740 1 Processor -- start 2024-09-15T23:29:34.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.036+0000 7f2b9037b740 1 -- start start 2024-09-15T23:29:34.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.037+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 0x561c83d2a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:34.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.037+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d28220 0x561c83d2f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.037+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d23000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.038+0000 7f2b9037b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2b74012350 con 0x561c83d32ae0 2024-09-15T23:29:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.038+0000 7f2b87fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d23000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:34.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.038+0000 7f2b87fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d23000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35886/0 (socket says 172.21.15.62:35886) 2024-09-15T23:29:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.038+0000 7f2b87fff640 1 -- 172.21.15.62:0/1662311571 learned_addr learned my addr 172.21.15.62:0/1662311571 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.039+0000 7f2b9037b740 1 -- 172.21.15.62:0/1662311571 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7400fbf0 con 0x561c83d28220 2024-09-15T23:29:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.039+0000 7f2b9037b740 1 -- 172.21.15.62:0/1662311571 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7400daf0 con 0x561c83d27900 2024-09-15T23:29:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.039+0000 7f2b87fff640 1 -- 172.21.15.62:0/1662311571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 msgr2=0x561c83d2a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.039+0000 7f2b87fff640 1 --2- 172.21.15.62:0/1662311571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 0x561c83d2a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.039+0000 7f2b87fff640 1 -- 172.21.15.62:0/1662311571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d28220 msgr2=0x561c83d2f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.039+0000 7f2b87fff640 1 --2- 172.21.15.62:0/1662311571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d28220 0x561c83d2f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.039+0000 7f2b87fff640 1 -- 172.21.15.62:0/1662311571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c83d66f30 con 0x561c83d32ae0 2024-09-15T23:29:34.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.040+0000 7f2b87fff640 1 --2- 172.21.15.62:0/1662311571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d23000 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009c50 tx=0x7f2b7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a320291e9c63332 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:34.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.044+0000 7f2b6d7fa640 1 -- 172.21.15.62:0/1662311571 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c00ee20 con 0x561c83d32ae0 2024-09-15T23:29:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.044+0000 7f2b6d7fa640 1 -- 172.21.15.62:0/1662311571 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b7c01f920 con 0x561c83d32ae0 2024-09-15T23:29:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.045+0000 7f2b6d7fa640 1 -- 172.21.15.62:0/1662311571 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c01e7e0 con 0x561c83d32ae0 2024-09-15T23:29:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.046+0000 7f2b9037b740 1 -- 172.21.15.62:0/1662311571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 msgr2=0x561c83d23000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:34.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.046+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1662311571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d23000 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009c50 tx=0x7f2b7c004730 comp rx=0 tx=0).stop 2024-09-15T23:29:34.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.049+0000 7f2b9037b740 1 -- 172.21.15.62:0/1662311571 shutdown_connections 2024-09-15T23:29:34.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.049+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1662311571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d23000 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.049+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1662311571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d28220 0x561c83d2f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.049+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1662311571 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d27900 0x561c83d2a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.049+0000 7f2b9037b740 1 -- 172.21.15.62:0/1662311571 >> 172.21.15.62:0/1662311571 conn(0x561c83d23e90 msgr2=0x561c83d319c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:34.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.052+0000 7f2b9037b740 1 -- 172.21.15.62:0/1662311571 shutdown_connections 2024-09-15T23:29:34.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.052+0000 7f2b9037b740 1 -- 172.21.15.62:0/1662311571 wait complete. 2024-09-15T23:29:34.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.052+0000 7f2b9037b740 1 -- start start 2024-09-15T23:29:34.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.053+0000 7f2b9037b740 1 Processor -- start 2024-09-15T23:29:34.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.053+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 0x561c83d3aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.054+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d28220 0x561c83d36370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.054+0000 7f2b9037b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d368b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.054+0000 7f2b9037b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7400e0b0 con 0x561c83d32ae0 2024-09-15T23:29:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.054+0000 7f2b9037b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7400f520 con 0x561c83d27900 2024-09-15T23:29:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.054+0000 7f2b9037b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7400e780 con 0x561c83d28220 2024-09-15T23:29:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.054+0000 7f2b9037b740 10 client.0 _ll_register_callbacks cb 0x561c83c39a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.055+0000 7f2b87fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d368b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.055+0000 7f2b87fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d368b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35900/0 (socket says 172.21.15.62:35900) 2024-09-15T23:29:34.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.055+0000 7f2b87fff640 1 -- 172.21.15.62:0/1846636269 learned_addr learned my addr 172.21.15.62:0/1846636269 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:34.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.060+0000 7f2b86ffd640 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d28220 0x561c83d36370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:34.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.060+0000 7f2b877fe640 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 0x561c83d3aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:34.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.062+0000 7f2b86ffd640 1 -- 172.21.15.62:0/1846636269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 msgr2=0x561c83d3aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:34.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.062+0000 7f2b86ffd640 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 0x561c83d3aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.062+0000 7f2b86ffd640 1 -- 172.21.15.62:0/1846636269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 msgr2=0x561c83d368b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:34.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.062+0000 7f2b86ffd640 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d368b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.062+0000 7f2b86ffd640 1 -- 172.21.15.62:0/1846636269 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c83d36e70 con 0x561c83d28220 2024-09-15T23:29:34.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.062+0000 7f2b86ffd640 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d28220 0x561c83d36370 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009780 tx=0x7f2b7802a8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4441e47d7331372 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:34.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.062+0000 7f2b66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:29:34.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.063+0000 7f2b66ffd640 1 -- 172.21.15.62:0/1846636269 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78037040 con 0x561c83d28220 2024-09-15T23:29:34.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.063+0000 7f2b66ffd640 1 -- 172.21.15.62:0/1846636269 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2b780305f0 con 0x561c83d28220 2024-09-15T23:29:34.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.063+0000 7f2b9037b740 1 -- 172.21.15.62:0/1846636269 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b7c01f920 con 0x561c83d28220 2024-09-15T23:29:34.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.063+0000 7f2b66ffd640 1 -- 172.21.15.62:0/1846636269 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2b780307f0 con 0x561c83d28220 2024-09-15T23:29:34.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.063+0000 7f2b9037b740 10 client.27745 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:34.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.063+0000 7f2b9037b740 1 -- 172.21.15.62:0/1846636269 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561c83dcef00 con 0x561c83d28220 2024-09-15T23:29:34.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.067+0000 7f2b66ffd640 1 -- 172.21.15.62:0/1846636269 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b7c01f920 con 0x561c83d28220 2024-09-15T23:29:34.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.067+0000 7f2b66ffd640 1 -- 172.21.15.62:0/1846636269 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f2b780046c0 con 0x561c83d28220 2024-09-15T23:29:34.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.067+0000 7f2b9037b740 10 client.27745 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:34.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.067+0000 7f2b9037b740 10 client.27745 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:29:34.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.067+0000 7f2b9037b740 20 client.27745 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:34.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.067+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561c83dcf6d0 0x561c83dd1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:34.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.067+0000 7f2b9037b740 4 client.27745 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.5, "await_for": 45.0}] 2024-09-15T23:29:34.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.067+0000 7f2b9037b740 1 -- 172.21.15.62:0/1846636269 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 146.5, "await_for": 45.0}) -- 0x561c83c94d20 con 0x561c83dcf6d0 2024-09-15T23:29:34.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.070+0000 7f2b877fe640 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561c83dcf6d0 0x561c83dd1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:34.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.070+0000 7f2b877fe640 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561c83dcf6d0 0x561c83dd1af0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=201e5d33d278efa1 server_cookie=e79bec913d24549f in_seq=0 out_seq=0 2024-09-15T23:29:34.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.070+0000 7f2b66ffd640 10 client.27745 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:34.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.328+0000 7f2b66ffd640 1 -- 172.21.15.62:0/1846636269 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x561c83c94d20 con 0x561c83dcf6d0 2024-09-15T23:29:34.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.328+0000 7f2b66ffd640 10 client.27745 handle_command_reply: tid=0 2024-09-15T23:29:34.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.328+0000 7f2b9037b740 1 client.27745 shutdown 2024-09-15T23:29:34.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 -- 172.21.15.62:0/1846636269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d28220 msgr2=0x561c83d36370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:34.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d28220 0x561c83d36370 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009780 tx=0x7f2b7802a8f0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 -- 172.21.15.62:0/1846636269 shutdown_connections 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561c83dcf6d0 0x561c83dd1af0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561c83d32ae0 0x561c83d368b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561c83d28220 0x561c83d36370 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 --2- 172.21.15.62:0/1846636269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561c83d27900 0x561c83d3aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b66ffd640 0 client.27745 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 -- 172.21.15.62:0/1846636269 >> 172.21.15.62:0/1846636269 conn(0x561c83d23e90 msgr2=0x561c83d300a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 -- 172.21.15.62:0/1846636269 shutdown_connections 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 1 -- 172.21.15.62:0/1846636269 wait complete. 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b9037b740 20 client.27745 trim_cache size 0 max 16384 2024-09-15T23:29:34.331 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 36, 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 50, 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "5d785437": { 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 50, 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 146.5, 2024-09-15T23:29:34.332 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:34.333 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:29:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b8cd40640 1 -- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b700a3450 msgr2=0x7f2b700a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b700a3450 0x7f2b700a5910 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005ab0 tx=0x7f2b7c009260 comp rx=0 tx=0).stop 2024-09-15T23:29:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b8cd40640 1 -- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b8806ea30 msgr2=0x7f2b881196d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.329+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b8806ea30 0x7f2b881196d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000ebc0 tx=0x7f2b8000c6a0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.330+0000 7f2b8cd40640 1 -- 172.21.15.62:0/1658146965 shutdown_connections 2024-09-15T23:29:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.330+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2b700a3450 0x7f2b700a5910 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.330+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2b8811b140 0x7f2b88112cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.330+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b88110190 0x7f2b881127b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.330+0000 7f2b8cd40640 1 --2- 172.21.15.62:0/1658146965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b8806ea30 0x7f2b881196d0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:34.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.330+0000 7f2b8cd40640 1 -- 172.21.15.62:0/1658146965 >> 172.21.15.62:0/1658146965 conn(0x7f2b88073180 msgr2=0x7f2b8806b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:34.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.330+0000 7f2b8cd40640 1 -- 172.21.15.62:0/1658146965 shutdown_connections 2024-09-15T23:29:34.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:34.330+0000 7f2b8cd40640 1 -- 172.21.15.62:0/1658146965 wait complete. 2024-09-15T23:29:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 795.0 s, 472.0 tps, lat 206.887 ms stddev 237.594 2024-09-15T23:29:34.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:34 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:29:34.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:34 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:29:34.960 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:34 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:29:34.988 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '5d785437', quiesce took 9.1 seconds. Will release after: 17.4 2024-09-15T23:29:35.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.226+0000 7fd9352a6640 1 -- 172.21.15.62:0/117947547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 msgr2=0x7fd930106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:35.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.226+0000 7fd9352a6640 1 --2- 172.21.15.62:0/117947547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 0x7fd930106e60 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fd920009930 tx=0x7fd92002a420 comp rx=0 tx=0).stop 2024-09-15T23:29:35.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.226+0000 7fd9352a6640 1 -- 172.21.15.62:0/117947547 shutdown_connections 2024-09-15T23:29:35.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.226+0000 7fd9352a6640 1 --2- 172.21.15.62:0/117947547 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd930101420 0x7fd93010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:35.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.226+0000 7fd9352a6640 1 --2- 172.21.15.62:0/117947547 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd930100a80 0x7fd930100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:35.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.226+0000 7fd9352a6640 1 --2- 172.21.15.62:0/117947547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 0x7fd930106e60 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:35.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.226+0000 7fd9352a6640 1 -- 172.21.15.62:0/117947547 >> 172.21.15.62:0/117947547 conn(0x7fd9300fc7a0 msgr2=0x7fd9300febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:35.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.226+0000 7fd9352a6640 1 -- 172.21.15.62:0/117947547 shutdown_connections 2024-09-15T23:29:35.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.226+0000 7fd9352a6640 1 -- 172.21.15.62:0/117947547 wait complete. 2024-09-15T23:29:35.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd9352a6640 1 Processor -- start 2024-09-15T23:29:35.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd9352a6640 1 -- start start 2024-09-15T23:29:35.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd9352a6640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd930100a80 0x7fd930198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:35.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd9352a6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd930101420 0x7fd9301989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:35.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd9352a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 0x7fd93019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:35.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd9352a6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd930110ea0 con 0x7fd930106a80 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd9352a6640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd930110d20 con 0x7fd930100a80 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd9352a6640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd930111020 con 0x7fd930101420 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd92f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 0x7fd93019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd92f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 0x7fd93019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35910/0 (socket says 172.21.15.62:35910) 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd92f7fe640 1 -- 172.21.15.62:0/1593085363 learned_addr learned my addr 172.21.15.62:0/1593085363 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd92effd640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd930100a80 0x7fd930198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.227+0000 7fd92e7fc640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd930101420 0x7fd9301989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd92f7fe640 1 -- 172.21.15.62:0/1593085363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd930101420 msgr2=0x7fd9301989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd92f7fe640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd930101420 0x7fd9301989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd92f7fe640 1 -- 172.21.15.62:0/1593085363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd930100a80 msgr2=0x7fd930198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd92f7fe640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd930100a80 0x7fd930198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd92f7fe640 1 -- 172.21.15.62:0/1593085363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd93019d440 con 0x7fd930106a80 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd92effd640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd930100a80 0x7fd930198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:35.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd92f7fe640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 0x7fd93019cd60 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd92800b970 tx=0x7fd92800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:35.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd927fff640 1 -- 172.21.15.62:0/1593085363 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928019070 con 0x7fd930106a80 2024-09-15T23:29:35.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd93019d730 con 0x7fd930106a80 2024-09-15T23:29:35.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd927fff640 1 -- 172.21.15.62:0/1593085363 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd92800cb40 con 0x7fd930106a80 2024-09-15T23:29:35.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd927fff640 1 -- 172.21.15.62:0/1593085363 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928007650 con 0x7fd930106a80 2024-09-15T23:29:35.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.228+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9301a5010 con 0x7fd930106a80 2024-09-15T23:29:35.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.229+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd930100ee0 con 0x7fd930106a80 2024-09-15T23:29:35.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.231+0000 7fd927fff640 1 -- 172.21.15.62:0/1593085363 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd92801c070 con 0x7fd930106a80 2024-09-15T23:29:35.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.232+0000 7fd927fff640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd91c0a3240 0x7fd91c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:35.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.232+0000 7fd92effd640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd91c0a3240 0x7fd91c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:35.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.232+0000 7fd927fff640 1 -- 172.21.15.62:0/1593085363 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd9280cf7e0 con 0x7fd930106a80 2024-09-15T23:29:35.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.232+0000 7fd92effd640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd91c0a3240 0x7fd91c0a5700 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd920000c00 tx=0x7fd920005a20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:35.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.235+0000 7fd927fff640 1 -- 172.21.15.62:0/1593085363 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd92809c180 con 0x7fd930106a80 2024-09-15T23:29:35.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.444+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9301a52b0 con 0x7fd930106a80 2024-09-15T23:29:35.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.445+0000 7fd927fff640 1 -- 172.21.15.62:0/1593085363 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7fd92809ba80 con 0x7fd930106a80 2024-09-15T23:29:35.446 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:29:35.447 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:29:35.447 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:29:35.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd91c0a3240 msgr2=0x7fd91c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd91c0a3240 0x7fd91c0a5700 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd920000c00 tx=0x7fd920005a20 comp rx=0 tx=0).stop 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 msgr2=0x7fd93019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 0x7fd93019cd60 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd92800b970 tx=0x7fd92800be30 comp rx=0 tx=0).stop 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 shutdown_connections 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd91c0a3240 0x7fd91c0a5700 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd930106a80 0x7fd93019cd60 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd930101420 0x7fd9301989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 --2- 172.21.15.62:0/1593085363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd930100a80 0x7fd930198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 >> 172.21.15.62:0/1593085363 conn(0x7fd9300fc7a0 msgr2=0x7fd930104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 shutdown_connections 2024-09-15T23:29:35.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:35.447+0000 7fd9352a6640 1 -- 172.21.15.62:0/1593085363 wait complete. 2024-09-15T23:29:35.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:35 smithi062 ceph-mon[30949]: pgmap v769: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 MiB/s wr, 1.40k op/s 2024-09-15T23:29:35.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:35 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1593085363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:35.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:35 smithi067 ceph-mon[35665]: pgmap v769: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 MiB/s wr, 1.40k op/s 2024-09-15T23:29:35.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:35 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1593085363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:35 smithi148 ceph-mon[35610]: pgmap v769: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 MiB/s wr, 1.40k op/s 2024-09-15T23:29:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:35 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1593085363' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:36.173 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.172+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2510333346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 msgr2=0x7fbbbc1020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.172+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2510333346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 0x7fbbbc1020d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009900 tx=0x7fbbb002a410 comp rx=0 tx=0).stop 2024-09-15T23:29:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.172+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2510333346 shutdown_connections 2024-09-15T23:29:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.172+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2510333346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc102610 0x7fbbbc108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.172+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2510333346 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 0x7fbbbc1020d0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.172+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2510333346 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbbc100a60 0x7fbbbc100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.172+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2510333346 >> 172.21.15.62:0/2510333346 conn(0x7fbbbc0fc1e0 msgr2=0x7fbbbc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.172+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2510333346 shutdown_connections 2024-09-15T23:29:36.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.172+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2510333346 wait complete. 2024-09-15T23:29:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbc08d0640 1 Processor -- start 2024-09-15T23:29:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbc08d0640 1 -- start start 2024-09-15T23:29:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbc08d0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbbc100a60 0x7fbbbc104b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbc08d0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 0x7fbbbc1031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbc08d0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc102610 0x7fbbbc103710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbc08d0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc10c6c0 con 0x7fbbbc101c50 2024-09-15T23:29:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbc08d0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc10c540 con 0x7fbbbc102610 2024-09-15T23:29:36.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbc08d0640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc10c840 con 0x7fbbbc100a60 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbb9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 0x7fbbbc1031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbb9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 0x7fbbbc1031d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35930/0 (socket says 172.21.15.62:35930) 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbb9d9b640 1 -- 172.21.15.62:0/2420917521 learned_addr learned my addr 172.21.15.62:0/2420917521 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.173+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc102610 0x7fbbbc103710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbba59c640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbbc100a60 0x7fbbbc104b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbb9d9b640 1 -- 172.21.15.62:0/2420917521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbbc100a60 msgr2=0x7fbbbc104b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbbc100a60 0x7fbbbc104b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbb9d9b640 1 -- 172.21.15.62:0/2420917521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc102610 msgr2=0x7fbbbc103710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc102610 0x7fbbbc103710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbb9d9b640 1 -- 172.21.15.62:0/2420917521 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc103fa0 con 0x7fbbbc101c50 2024-09-15T23:29:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 0x7fbbbc1031d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009230 tx=0x7fbbb0004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbab7fe640 1 -- 172.21.15.62:0/2420917521 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb003f030 con 0x7fbbbc101c50 2024-09-15T23:29:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2420917521 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc1a1eb0 con 0x7fbbbc101c50 2024-09-15T23:29:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbab7fe640 1 -- 172.21.15.62:0/2420917521 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb0003ce0 con 0x7fbbbc101c50 2024-09-15T23:29:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbab7fe640 1 -- 172.21.15.62:0/2420917521 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0035070 con 0x7fbbbc101c50 2024-09-15T23:29:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.174+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2420917521 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbbc1a2320 con 0x7fbbbc101c50 2024-09-15T23:29:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.177+0000 7fbbc366d740 1 Processor -- start 2024-09-15T23:29:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.177+0000 7fbbc366d740 1 -- start start 2024-09-15T23:29:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434819a0 0x55ba43481da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434822e0 0x55ba43486da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba43487370 0x55ba43489830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbc366d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ba433e52f0 con 0x55ba434822e0 2024-09-15T23:29:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbc366d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ba4338d270 con 0x55ba434819a0 2024-09-15T23:29:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbc366d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ba433e12b0 con 0x55ba43487370 2024-09-15T23:29:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434822e0 0x55ba43486da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434822e0 0x55ba43486da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35940/0 (socket says 172.21.15.62:35940) 2024-09-15T23:29:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbba59c640 1 -- 172.21.15.62:0/3121499014 learned_addr learned my addr 172.21.15.62:0/3121499014 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/3121499014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434819a0 0x55ba43481da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.189 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbab7fe640 1 -- 172.21.15.62:0/2420917521 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fbbb0042030 con 0x7fbbbc101c50 2024-09-15T23:29:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbbad9d640 1 -- 172.21.15.62:0/3121499014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba43487370 msgr2=0x55ba43489830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:29:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/3121499014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba43487370 0x55ba43489830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbbad9d640 1 -- 172.21.15.62:0/3121499014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434822e0 msgr2=0x55ba43486da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/3121499014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434822e0 0x55ba43486da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbbad9d640 1 -- 172.21.15.62:0/3121499014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba4338c340 con 0x55ba434819a0 2024-09-15T23:29:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbba59c640 1 --2- 172.21.15.62:0/3121499014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434822e0 0x55ba43486da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:36.190 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.178+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/3121499014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434819a0 0x55ba43481da0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009f50 tx=0x7fbbb4003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a75720ee750c4948 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.179+0000 7fbba8ff9640 1 -- 172.21.15.62:0/3121499014 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4011440 con 0x55ba434819a0 2024-09-15T23:29:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.179+0000 7fbba8ff9640 1 -- 172.21.15.62:0/3121499014 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb4015030 con 0x55ba434819a0 2024-09-15T23:29:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.179+0000 7fbba8ff9640 1 -- 172.21.15.62:0/3121499014 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40108c0 con 0x55ba434819a0 2024-09-15T23:29:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.179+0000 7fbbab7fe640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba40a32f0 0x7fbba40a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.179+0000 7fbbba59c640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba40a32f0 0x7fbba40a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.179+0000 7fbbab7fe640 1 -- 172.21.15.62:0/2420917521 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbbb00f05f0 con 0x7fbbbc101c50 2024-09-15T23:29:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 -- 172.21.15.62:0/3121499014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434819a0 msgr2=0x55ba43481da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbba59c640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba40a32f0 0x7fbba40a57b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x55ba43482d10 tx=0x7fbbac00c4d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.191 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 --2- 172.21.15.62:0/3121499014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434819a0 0x55ba43481da0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009f50 tx=0x7fbbb4003e00 comp rx=0 tx=0).stop 2024-09-15T23:29:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 -- 172.21.15.62:0/3121499014 shutdown_connections 2024-09-15T23:29:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 --2- 172.21.15.62:0/3121499014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba43487370 0x55ba43489830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 --2- 172.21.15.62:0/3121499014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434822e0 0x55ba43486da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 --2- 172.21.15.62:0/3121499014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434819a0 0x55ba43481da0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 -- 172.21.15.62:0/3121499014 >> 172.21.15.62:0/3121499014 conn(0x55ba4347d010 msgr2=0x55ba4347d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 -- 172.21.15.62:0/3121499014 shutdown_connections 2024-09-15T23:29:36.192 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 -- 172.21.15.62:0/3121499014 wait complete. 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 -- start start 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.180+0000 7fbbc366d740 1 Processor -- start 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 0x55ba4348bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba4348c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba43493150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbc366d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55ba4348a340 con 0x55ba43487370 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbc366d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ba4336e290 con 0x55ba434822e0 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbc366d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55ba4348a4c0 con 0x55ba434819a0 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbc366d740 10 client.0 _ll_register_callbacks cb 0x55ba4339f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:36.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbbad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba43493150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbb9d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba4348c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbbad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba43493150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35942/0 (socket says 172.21.15.62:35942) 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbbad9d640 1 -- 172.21.15.62:0/391974289 learned_addr learned my addr 172.21.15.62:0/391974289 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba4348c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:52852/0 (socket says 172.21.15.62:52852) 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbba59c640 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 0x55ba4348bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbb9d9b640 1 -- 172.21.15.62:0/391974289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 msgr2=0x55ba4348bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 0x55ba4348bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbb9d9b640 1 -- 172.21.15.62:0/391974289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 msgr2=0x55ba43493150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba43493150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbb9d9b640 1 -- 172.21.15.62:0/391974289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba430f8210 con 0x55ba434822e0 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba43493150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:36.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba4348c110 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00ee0e0 tx=0x7fbbb00b8bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5405a0052ac1359 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.181+0000 7fbba1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbba1ffb640 1 -- 172.21.15.62:0/391974289 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0049040 con 0x55ba434822e0 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbbc366d740 1 -- 172.21.15.62:0/391974289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbb4015030 con 0x55ba434822e0 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbba1ffb640 1 -- 172.21.15.62:0/391974289 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbbb00f0350 con 0x55ba434822e0 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbba1ffb640 1 -- 172.21.15.62:0/391974289 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbbb00da050 con 0x55ba434822e0 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbbc366d740 10 client.28049 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbbc366d740 1 -- 172.21.15.62:0/391974289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ba431c4620 con 0x55ba434822e0 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbba1ffb640 1 -- 172.21.15.62:0/391974289 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb4015030 con 0x55ba434822e0 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbba1ffb640 1 -- 172.21.15.62:0/391974289 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fbbb00b8f00 con 0x55ba434822e0 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbbc366d740 10 client.28049 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbbc366d740 10 client.28049 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:36.195 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbbc366d740 20 client.28049 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbbc366d740 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ba43497880 0x55ba43499c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbbc366d740 4 client.28049 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.182+0000 7fbbc366d740 1 -- 172.21.15.62:0/391974289 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba433f2a20 con 0x55ba43497880 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.183+0000 7fbbba59c640 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ba43497880 0x55ba43499c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.183+0000 7fbba1ffb640 10 client.28049 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.183+0000 7fbbba59c640 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ba43497880 0x55ba43499c70 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddc980f5b9409d19 server_cookie=d4b774ac5dbf6a0c in_seq=0 out_seq=0 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbba1ffb640 1 -- 172.21.15.62:0/391974289 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ba433f2a20 con 0x55ba43497880 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbba1ffb640 10 client.28049 handle_command_reply: tid=0 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 client.28049 shutdown 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 -- 172.21.15.62:0/391974289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 msgr2=0x55ba4348c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.196 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba4348c110 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00ee0e0 tx=0x7fbbb00b8bf0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 -- 172.21.15.62:0/391974289 shutdown_connections 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ba43497880 0x55ba43499c70 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba43493150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba4348c110 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbba1ffb640 0 client.28049 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 --2- 172.21.15.62:0/391974289 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 0x55ba4348bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 -- 172.21.15.62:0/391974289 >> 172.21.15.62:0/391974289 conn(0x55ba4347d010 msgr2=0x55ba4347e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 -- 172.21.15.62:0/391974289 shutdown_connections 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 1 -- 172.21.15.62:0/391974289 wait complete. 2024-09-15T23:29:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.184+0000 7fbbc366d740 20 client.28049 trim_cache size 0 max 16384 2024-09-15T23:29:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbc366d740 1 Processor -- start 2024-09-15T23:29:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbc366d740 1 -- start start 2024-09-15T23:29:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 0x55ba43481d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba43493110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba4347dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbc366d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc1af110 con 0x55ba43487370 2024-09-15T23:29:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbc366d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc1ac9b0 con 0x55ba434822e0 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbc366d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc1aa800 con 0x55ba434819a0 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbbad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba4347dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbbad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba4347dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35952/0 (socket says 172.21.15.62:35952) 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbbad9d640 1 -- 172.21.15.62:0/2906704068 learned_addr learned my addr 172.21.15.62:0/2906704068 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbba59c640 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 0x55ba43481d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba43493110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.199+0000 7fbbbad9d640 1 -- 172.21.15.62:0/2906704068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 msgr2=0x55ba43481d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.200+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 0x55ba43481d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.200+0000 7fbbbad9d640 1 -- 172.21.15.62:0/2906704068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 msgr2=0x55ba43493110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.200+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba43493110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.200+0000 7fbbbad9d640 1 -- 172.21.15.62:0/2906704068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba4349a2a0 con 0x55ba43487370 2024-09-15T23:29:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.200+0000 7fbbba59c640 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 0x55ba43481d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.200+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba4347dec0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009990 tx=0x7fbbb4009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cf710c3ed2795a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.200+0000 7fbba3fff640 1 -- 172.21.15.62:0/2906704068 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb402b070 con 0x55ba43487370 2024-09-15T23:29:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.200+0000 7fbba3fff640 1 -- 172.21.15.62:0/2906704068 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb4015070 con 0x55ba43487370 2024-09-15T23:29:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.200+0000 7fbba3fff640 1 -- 172.21.15.62:0/2906704068 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4010620 con 0x55ba43487370 2024-09-15T23:29:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 -- 172.21.15.62:0/2906704068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 msgr2=0x55ba4347dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba4347dec0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009990 tx=0x7fbbb4009b90 comp rx=0 tx=0).stop 2024-09-15T23:29:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 -- 172.21.15.62:0/2906704068 shutdown_connections 2024-09-15T23:29:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba43487370 0x55ba4347dec0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba434822e0 0x55ba43493110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 --2- 172.21.15.62:0/2906704068 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434819a0 0x55ba43481d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 -- 172.21.15.62:0/2906704068 >> 172.21.15.62:0/2906704068 conn(0x55ba4347d010 msgr2=0x55ba4348bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 -- 172.21.15.62:0/2906704068 shutdown_connections 2024-09-15T23:29:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 -- 172.21.15.62:0/2906704068 wait complete. 2024-09-15T23:29:36.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 -- start start 2024-09-15T23:29:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 Processor -- start 2024-09-15T23:29:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434819a0 0x55ba43490480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434822e0 0x55ba434909c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba43487370 0x55ba434919d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc1aae70 con 0x55ba434819a0 2024-09-15T23:29:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc1ac2e0 con 0x55ba43487370 2024-09-15T23:29:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc1ab540 con 0x55ba434822e0 2024-09-15T23:29:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.201+0000 7fbbc366d740 10 client.0 _ll_register_callbacks cb 0x7fbbb00095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:36.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434819a0 0x55ba43490480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434819a0 0x55ba43490480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35968/0 (socket says 172.21.15.62:35968) 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbba59c640 1 -- 172.21.15.62:0/677623714 learned_addr learned my addr 172.21.15.62:0/677623714 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbbad9d640 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba43487370 0x55ba434919d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434822e0 0x55ba434909c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbba59c640 1 -- 172.21.15.62:0/677623714 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434822e0 msgr2=0x55ba434909c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbba59c640 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434822e0 0x55ba434909c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbba59c640 1 -- 172.21.15.62:0/677623714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba43487370 msgr2=0x55ba434919d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbba59c640 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba43487370 0x55ba434919d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbba59c640 1 -- 172.21.15.62:0/677623714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba43491fe0 con 0x55ba434819a0 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbba59c640 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434819a0 0x55ba43490480 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x55ba4347de90 tx=0x7fbbac013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8e0a8190710587 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbba1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbba1ffb640 1 -- 172.21.15.62:0/677623714 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac027830 con 0x55ba434819a0 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbba1ffb640 1 -- 172.21.15.62:0/677623714 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbbac0279d0 con 0x55ba434819a0 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.202+0000 7fbbc366d740 1 -- 172.21.15.62:0/677623714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbb4015070 con 0x55ba434819a0 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbba1ffb640 1 -- 172.21.15.62:0/677623714 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbbac036050 con 0x55ba434819a0 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbbc366d740 10 client.18528 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbbc366d740 1 -- 172.21.15.62:0/677623714 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ba434933c0 con 0x55ba434819a0 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbba1ffb640 1 -- 172.21.15.62:0/677623714 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb4015070 con 0x55ba434819a0 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbba1ffb640 1 -- 172.21.15.62:0/677623714 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fbbac02f060 con 0x55ba434819a0 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbbc366d740 10 client.18528 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbbc366d740 10 client.18528 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbbc366d740 20 client.18528 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbbc366d740 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ba43502b40 0x55ba43504f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbbc366d740 4 client.18528 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:29:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbbc366d740 1 -- 172.21.15.62:0/677623714 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ba433ecd70 con 0x55ba43502b40 2024-09-15T23:29:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.203+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ba43502b40 0x55ba43504f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.204+0000 7fbbb9d9b640 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ba43502b40 0x55ba43504f60 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0d7422d2518da2e server_cookie=6d8681807d27e93c in_seq=0 out_seq=0 2024-09-15T23:29:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.204+0000 7fbba1ffb640 10 client.18528 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:36.213 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:29:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.219+0000 7fbba1ffb640 1 -- 172.21.15.62:0/677623714 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ba433ecd70 con 0x55ba43502b40 2024-09-15T23:29:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.219+0000 7fbba1ffb640 10 client.18528 handle_command_reply: tid=0 2024-09-15T23:29:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.219+0000 7fbbc366d740 1 client.18528 shutdown 2024-09-15T23:29:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.219+0000 7fbbc366d740 1 -- 172.21.15.62:0/677623714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434819a0 msgr2=0x55ba43490480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.219+0000 7fbbc366d740 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434819a0 0x55ba43490480 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x55ba4347de90 tx=0x7fbbac013040 comp rx=0 tx=0).stop 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc366d740 1 -- 172.21.15.62:0/677623714 shutdown_connections 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc366d740 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55ba43502b40 0x55ba43504f60 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc366d740 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba43487370 0x55ba434919d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc366d740 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55ba434822e0 0x55ba434909c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbba1ffb640 0 client.18528 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc366d740 1 --2- 172.21.15.62:0/677623714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ba434819a0 0x55ba43490480 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc366d740 1 -- 172.21.15.62:0/677623714 >> 172.21.15.62:0/677623714 conn(0x55ba4347d010 msgr2=0x55ba4347b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc366d740 1 -- 172.21.15.62:0/677623714 shutdown_connections 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc366d740 1 -- 172.21.15.62:0/677623714 wait complete. 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc366d740 20 client.18528 trim_cache size 0 max 16384 2024-09-15T23:29:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:29:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:29:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:29:36.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba40a32f0 msgr2=0x7fbba40a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba40a32f0 0x7fbba40a57b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x55ba43482d10 tx=0x7fbbac00c4d0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 msgr2=0x7fbbbc1031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 0x7fbbbc1031d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009230 tx=0x7fbbb0004360 comp rx=0 tx=0).stop 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2420917521 shutdown_connections 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba40a32f0 0x7fbba40a57b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc102610 0x7fbbbc103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbbc101c50 0x7fbbbc1031d0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 --2- 172.21.15.62:0/2420917521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbbc100a60 0x7fbbbc104b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.220+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2420917521 >> 172.21.15.62:0/2420917521 conn(0x7fbbbc0fc1e0 msgr2=0x7fbbbc06a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.221+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2420917521 shutdown_connections 2024-09-15T23:29:36.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:36.221+0000 7fbbc08d0640 1 -- 172.21.15.62:0/2420917521 wait complete. 2024-09-15T23:29:36.471 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.951+0000 7f097217d640 1 -- 172.21.15.62:0/908339400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c10c180 msgr2=0x7f096c10e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.951+0000 7f097217d640 1 --2- 172.21.15.62:0/908339400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c10c180 0x7f096c10e600 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f0964009ca0 tx=0x7f096402a950 comp rx=0 tx=0).stop 2024-09-15T23:29:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.951+0000 7f097217d640 1 -- 172.21.15.62:0/908339400 shutdown_connections 2024-09-15T23:29:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.951+0000 7f097217d640 1 --2- 172.21.15.62:0/908339400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c10c180 0x7f096c10e600 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.951+0000 7f097217d640 1 --2- 172.21.15.62:0/908339400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f096c1036c0 0x7f096c10bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.951+0000 7f097217d640 1 --2- 172.21.15.62:0/908339400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c102da0 0x7f096c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.951+0000 7f097217d640 1 -- 172.21.15.62:0/908339400 >> 172.21.15.62:0/908339400 conn(0x7f096c0fc7a0 msgr2=0x7f096c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.951+0000 7f097217d640 1 -- 172.21.15.62:0/908339400 shutdown_connections 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.951+0000 7f097217d640 1 -- 172.21.15.62:0/908339400 wait complete. 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f097217d640 1 Processor -- start 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f097217d640 1 -- start start 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f097217d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f096c102da0 0x7f096c100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f097217d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c1036c0 0x7f096c100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f097217d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c10c180 0x7f096c101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f097217d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f096c111f20 con 0x7f096c1036c0 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f097217d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f096c111da0 con 0x7f096c10c180 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f097217d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1120a0 con 0x7f096c102da0 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f096affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c1036c0 0x7f096c100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f096affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c1036c0 0x7f096c100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35990/0 (socket says 172.21.15.62:35990) 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.952+0000 7f096affd640 1 -- 172.21.15.62:0/3098722201 learned_addr learned my addr 172.21.15.62:0/3098722201 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096b7fe640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f096c102da0 0x7f096c100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096bfff640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c10c180 0x7f096c101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096affd640 1 -- 172.21.15.62:0/3098722201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f096c102da0 msgr2=0x7f096c100a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096affd640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f096c102da0 0x7f096c100a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096affd640 1 -- 172.21.15.62:0/3098722201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c10c180 msgr2=0x7f096c101480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096affd640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c10c180 0x7f096c101480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096affd640 1 -- 172.21.15.62:0/3098722201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096c1a6870 con 0x7f096c1036c0 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096b7fe640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f096c102da0 0x7f096c100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096bfff640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c10c180 0x7f096c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f096affd640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c1036c0 0x7f096c100f40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f096000d950 tx=0x7f096000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f0968ff9640 1 -- 172.21.15.62:0/3098722201 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0960017070 con 0x7f096c1036c0 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f0968ff9640 1 -- 172.21.15.62:0/3098722201 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0960009c30 con 0x7f096c1036c0 2024-09-15T23:29:36.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f097217d640 1 -- 172.21.15.62:0/3098722201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096c1a6b60 con 0x7f096c1036c0 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f0968ff9640 1 -- 172.21.15.62:0/3098722201 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096000f6b0 con 0x7f096c1036c0 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.953+0000 7f097217d640 1 -- 172.21.15.62:0/3098722201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096c1a70a0 con 0x7f096c1036c0 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f0968ff9640 1 -- 172.21.15.62:0/3098722201 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f09600109d0 con 0x7f096c1036c0 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f0974556740 1 Processor -- start 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f0974556740 1 -- start start 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f0974556740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8de090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f0974556740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 0x563acf8dea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f0974556740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 0x563acf8eb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f0974556740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563acf8442f0 con 0x563acf8dfb40 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f0974556740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563acf7ebcf0 con 0x563acf8e9370 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f0974556740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563acf8402b0 con 0x563acf8de5d0 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.956+0000 7f096bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8de090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8de090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35998/0 (socket says 172.21.15.62:35998) 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096bfff640 1 -- 172.21.15.62:0/176989280 learned_addr learned my addr 172.21.15.62:0/176989280 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f0968ff9640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09540a3240 0x7f09540a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096b7fe640 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 0x563acf8dea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096affd640 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 0x563acf8eb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096bfff640 1 -- 172.21.15.62:0/176989280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 msgr2=0x563acf8dea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096bfff640 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 0x563acf8dea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096bfff640 1 -- 172.21.15.62:0/176989280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 msgr2=0x563acf8eb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096bfff640 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 0x563acf8eb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096bfff640 1 -- 172.21.15.62:0/176989280 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563acf7d03a0 con 0x563acf8dfb40 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096b7fe640 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 0x563acf8dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f0968ff9640 1 -- 172.21.15.62:0/3098722201 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f09600cf940 con 0x7f096c1036c0 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096bfff640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09540a3240 0x7f09540a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f096bfff640 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8de090 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0964009880 tx=0x7f09640098b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4646b91401280a77 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f0951ffb640 1 -- 172.21.15.62:0/176989280 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0964040040 con 0x563acf8dfb40 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f0951ffb640 1 -- 172.21.15.62:0/176989280 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0964003c30 con 0x563acf8dfb40 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.957+0000 7f0951ffb640 1 -- 172.21.15.62:0/176989280 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0964034070 con 0x563acf8dfb40 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 -- 172.21.15.62:0/176989280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 msgr2=0x563acf8de090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8de090 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f0964009880 tx=0x7f09640098b0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f096bfff640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09540a3240 0x7f09540a5700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f096404b000 tx=0x7f096404c000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 -- 172.21.15.62:0/176989280 shutdown_connections 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 0x563acf8eb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 0x563acf8dea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 --2- 172.21.15.62:0/176989280 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8de090 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 -- 172.21.15.62:0/176989280 >> 172.21.15.62:0/176989280 conn(0x563acf8a7310 msgr2=0x563acf8dac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 -- 172.21.15.62:0/176989280 shutdown_connections 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 -- 172.21.15.62:0/176989280 wait complete. 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.958+0000 7f0974556740 1 -- start start 2024-09-15T23:29:36.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f0974556740 1 Processor -- start 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f0974556740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8de5d0 0x563acf8e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f0974556740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8dfb40 0x563acf8e5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f0974556740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 0x563acf8e6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f0974556740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563acf8ec070 con 0x563acf8de5d0 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f0974556740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563acf7ff060 con 0x563acf8dfb40 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f0974556740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563acf8ec1f0 con 0x563acf8e9370 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f0974556740 10 client.0 _ll_register_callbacks cb 0x563acf5528c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f096b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8de5d0 0x563acf8e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f096b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8de5d0 0x563acf8e8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36008/0 (socket says 172.21.15.62:36008) 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f096b7fe640 1 -- 172.21.15.62:0/298541140 learned_addr learned my addr 172.21.15.62:0/298541140 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f096affd640 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8dfb40 0x563acf8e5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f096bfff640 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 0x563acf8e6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.959+0000 7f096b7fe640 1 -- 172.21.15.62:0/298541140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 msgr2=0x563acf8e6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f096b7fe640 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 0x563acf8e6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f096b7fe640 1 -- 172.21.15.62:0/298541140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8dfb40 msgr2=0x563acf8e5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f096b7fe640 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8dfb40 0x563acf8e5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f096b7fe640 1 -- 172.21.15.62:0/298541140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563acf5b53c0 con 0x563acf8de5d0 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f096bfff640 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 0x563acf8e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f096affd640 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8dfb40 0x563acf8e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f096b7fe640 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8de5d0 0x563acf8e8d10 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f095c0124c0 tx=0x7f095c012980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc651e89b3de4054 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f094affd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f094affd640 1 -- 172.21.15.62:0/298541140 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c004220 con 0x563acf8de5d0 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f094affd640 1 -- 172.21.15.62:0/298541140 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f095c0043c0 con 0x563acf8de5d0 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f0974556740 1 -- 172.21.15.62:0/298541140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0964003c30 con 0x563acf8de5d0 2024-09-15T23:29:36.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f094affd640 1 -- 172.21.15.62:0/298541140 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f095c010780 con 0x563acf8de5d0 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f0974556740 10 client.18540 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f0974556740 1 -- 172.21.15.62:0/298541140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563acf8f7b80 con 0x563acf8de5d0 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f094affd640 1 -- 172.21.15.62:0/298541140 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0964003c30 con 0x563acf8de5d0 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.960+0000 7f094affd640 1 -- 172.21.15.62:0/298541140 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f095c00f040 con 0x563acf8de5d0 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.961+0000 7f0974556740 10 client.18540 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.961+0000 7f0974556740 10 client.18540 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.961+0000 7f0974556740 20 client.18540 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.961+0000 7f0974556740 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x563acf8f9040 0x563acf8fb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.961+0000 7f0974556740 4 client.18540 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.961+0000 7f0974556740 1 -- 172.21.15.62:0/298541140 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563acf851860 con 0x563acf8f9040 2024-09-15T23:29:36.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.961+0000 7f096affd640 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x563acf8f9040 0x563acf8fb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f096affd640 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x563acf8f9040 0x563acf8fb490 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=18933cde72b5cead server_cookie=58720bda6d099dfb in_seq=0 out_seq=0 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f094affd640 10 client.18540 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f094affd640 1 -- 172.21.15.62:0/298541140 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563acf851860 con 0x563acf8f9040 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f094affd640 10 client.18540 handle_command_reply: tid=0 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f0974556740 1 client.18540 shutdown 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f0974556740 1 -- 172.21.15.62:0/298541140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8de5d0 msgr2=0x563acf8e8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f0974556740 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8de5d0 0x563acf8e8d10 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f095c0124c0 tx=0x7f095c012980 comp rx=0 tx=0).stop 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f0974556740 1 -- 172.21.15.62:0/298541140 shutdown_connections 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f0974556740 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x563acf8f9040 0x563acf8fb490 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f0974556740 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 0x563acf8e6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f0974556740 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8dfb40 0x563acf8e5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f0974556740 1 --2- 172.21.15.62:0/298541140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8de5d0 0x563acf8e8d10 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.962+0000 7f0974556740 1 -- 172.21.15.62:0/298541140 >> 172.21.15.62:0/298541140 conn(0x563acf8a7310 msgr2=0x563acf8e0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.963+0000 7f094affd640 0 client.18540 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:29:36.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.963+0000 7f0974556740 1 -- 172.21.15.62:0/298541140 shutdown_connections 2024-09-15T23:29:36.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.963+0000 7f0974556740 1 -- 172.21.15.62:0/298541140 wait complete. 2024-09-15T23:29:36.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.963+0000 7f0974556740 20 client.18540 trim_cache size 0 max 16384 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f0974556740 1 Processor -- start 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f0974556740 1 -- start start 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f0974556740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8de5d0 0x563acf8f5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f0974556740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8e5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f0974556740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 0x563acf8e1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f0974556740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1b36d0 con 0x563acf8dfb40 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f0974556740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1b0f70 con 0x563acf8de5d0 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f0974556740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1aedc0 con 0x563acf8e9370 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8e5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8e5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36020/0 (socket says 172.21.15.62:36020) 2024-09-15T23:29:36.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096affd640 1 -- 172.21.15.62:0/3794521740 learned_addr learned my addr 172.21.15.62:0/3794521740 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096b7fe640 1 --2- 172.21.15.62:0/3794521740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8de5d0 0x563acf8f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096bfff640 1 --2- 172.21.15.62:0/3794521740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 0x563acf8e1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096affd640 1 -- 172.21.15.62:0/3794521740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 msgr2=0x563acf8e1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096affd640 1 --2- 172.21.15.62:0/3794521740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 0x563acf8e1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096affd640 1 -- 172.21.15.62:0/3794521740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8de5d0 msgr2=0x563acf8f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096affd640 1 --2- 172.21.15.62:0/3794521740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8de5d0 0x563acf8f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.976+0000 7f096affd640 1 -- 172.21.15.62:0/3794521740 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563acf8db960 con 0x563acf8dfb40 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f096affd640 1 --2- 172.21.15.62:0/3794521740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8e5880 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f09600cd060 tx=0x7f096006ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb78fa69792022ab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f09517fa640 1 -- 172.21.15.62:0/3794521740 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0960020040 con 0x563acf8dfb40 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f09517fa640 1 -- 172.21.15.62:0/3794521740 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f096009ff30 con 0x563acf8dfb40 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f09517fa640 1 -- 172.21.15.62:0/3794521740 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09600a4360 con 0x563acf8dfb40 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f0974556740 1 -- 172.21.15.62:0/3794521740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 msgr2=0x563acf8e5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f0974556740 1 --2- 172.21.15.62:0/3794521740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8e5880 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f09600cd060 tx=0x7f096006ea00 comp rx=0 tx=0).stop 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f0974556740 1 -- 172.21.15.62:0/3794521740 shutdown_connections 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f0974556740 1 --2- 172.21.15.62:0/3794521740 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8e9370 0x563acf8e1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f0974556740 1 --2- 172.21.15.62:0/3794521740 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8e5880 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f0974556740 1 --2- 172.21.15.62:0/3794521740 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8de5d0 0x563acf8f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f0974556740 1 -- 172.21.15.62:0/3794521740 >> 172.21.15.62:0/3794521740 conn(0x563acf8a7310 msgr2=0x563acf8da700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f0974556740 1 -- 172.21.15.62:0/3794521740 shutdown_connections 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.977+0000 7f0974556740 1 -- 172.21.15.62:0/3794521740 wait complete. 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f0974556740 1 -- start start 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f0974556740 1 Processor -- start 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f0974556740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 0x563acf8f1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f0974556740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f0974556740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 0x563acf8e27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f0974556740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1af430 con 0x563acf8dfb40 2024-09-15T23:29:36.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f0974556740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1b08a0 con 0x563acf8e9370 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f0974556740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f096c1afb00 con 0x563acf8de5d0 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f0974556740 10 client.0 _ll_register_callbacks cb 0x7f095c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f096affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f096affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8f1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36026/0 (socket says 172.21.15.62:36026) 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.978+0000 7f096affd640 1 -- 172.21.15.62:0/1421509918 learned_addr learned my addr 172.21.15.62:0/1421509918 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f096b7fe640 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 0x563acf8f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f096bfff640 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 0x563acf8e27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f096affd640 1 -- 172.21.15.62:0/1421509918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 msgr2=0x563acf8f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f096affd640 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 0x563acf8f1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f096affd640 1 -- 172.21.15.62:0/1421509918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 msgr2=0x563acf8e27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f096affd640 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 0x563acf8e27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f096affd640 1 -- 172.21.15.62:0/1421509918 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563acf8e2fd0 con 0x563acf8dfb40 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f096affd640 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8f1c70 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f0960098550 tx=0x7f096009b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fc023527fb1d03 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:36.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f094affd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f094affd640 1 -- 172.21.15.62:0/1421509918 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0960020070 con 0x563acf8dfb40 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f094affd640 1 -- 172.21.15.62:0/1421509918 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f09600981f0 con 0x563acf8dfb40 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f0974556740 1 -- 172.21.15.62:0/1421509918 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f096009ff30 con 0x563acf8dfb40 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f094affd640 1 -- 172.21.15.62:0/1421509918 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f096009c280 con 0x563acf8dfb40 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f0974556740 10 client.18546 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f0974556740 1 -- 172.21.15.62:0/1421509918 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563acf963810 con 0x563acf8dfb40 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.979+0000 7f094affd640 1 -- 172.21.15.62:0/1421509918 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09600b3080 con 0x563acf8dfb40 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f094affd640 1 -- 172.21.15.62:0/1421509918 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f09600ba2a0 con 0x563acf8dfb40 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f0974556740 10 client.18546 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f0974556740 10 client.18546 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f0974556740 20 client.18546 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f0974556740 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x563acf963e60 0x563acf966280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f0974556740 4 client.18546 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:36.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f0974556740 1 -- 172.21.15.62:0/1421509918 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563acf84bd70 con 0x563acf963e60 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f096b7fe640 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x563acf963e60 0x563acf966280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f094affd640 10 client.18546 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.980+0000 7f096b7fe640 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x563acf963e60 0x563acf966280 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f3990057b56aed72 server_cookie=461827481f16814f in_seq=0 out_seq=0 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f094affd640 1 -- 172.21.15.62:0/1421509918 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563acf84bd70 con 0x563acf963e60 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f094affd640 10 client.18546 handle_command_reply: tid=0 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 client.18546 shutdown 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 -- 172.21.15.62:0/1421509918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 msgr2=0x563acf8f1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8f1c70 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f0960098550 tx=0x7f096009b6c0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 -- 172.21.15.62:0/1421509918 shutdown_connections 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x563acf963e60 0x563acf966280 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563acf8e9370 0x563acf8e27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563acf8dfb40 0x563acf8f1c70 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f094affd640 0 client.18546 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:29:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 --2- 172.21.15.62:0/1421509918 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563acf8de5d0 0x563acf8f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 -- 172.21.15.62:0/1421509918 >> 172.21.15.62:0/1421509918 conn(0x563acf8a7310 msgr2=0x563acf8d9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 -- 172.21.15.62:0/1421509918 shutdown_connections 2024-09-15T23:29:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 1 -- 172.21.15.62:0/1421509918 wait complete. 2024-09-15T23:29:36.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f0974556740 20 client.18546 trim_cache size 0 max 16384 2024-09-15T23:29:36.990 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.981+0000 7f097217d640 1 -- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09540a3240 msgr2=0x7f09540a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09540a3240 0x7f09540a5700 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f096404b000 tx=0x7f096404c000 comp rx=0 tx=0).stop 2024-09-15T23:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 -- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c1036c0 msgr2=0x7f096c100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c1036c0 0x7f096c100f40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f096000d950 tx=0x7f096000de10 comp rx=0 tx=0).stop 2024-09-15T23:29:36.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 -- 172.21.15.62:0/3098722201 shutdown_connections 2024-09-15T23:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f09540a3240 0x7f09540a5700 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f096c10c180 0x7f096c101480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f096c1036c0 0x7f096c100f40 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 --2- 172.21.15.62:0/3098722201 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f096c102da0 0x7f096c100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 -- 172.21.15.62:0/3098722201 >> 172.21.15.62:0/3098722201 conn(0x7f096c0fc7a0 msgr2=0x7f096c0fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 -- 172.21.15.62:0/3098722201 shutdown_connections 2024-09-15T23:29:36.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:36.982+0000 7f097217d640 1 -- 172.21.15.62:0/3098722201 wait complete. 2024-09-15T23:29:37.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:37 smithi148 ceph-mon[35610]: pgmap v770: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 14 MiB/s wr, 1.11k op/s 2024-09-15T23:29:37.833 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:37 smithi062 ceph-mon[30949]: pgmap v770: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 14 MiB/s wr, 1.11k op/s 2024-09-15T23:29:37.862 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:29:38.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:37 smithi067 ceph-mon[35665]: pgmap v770: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 14 MiB/s wr, 1.11k op/s 2024-09-15T23:29:38.162 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:38.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.762+0000 7f3e2c341640 1 -- 172.21.15.62:0/2022383957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e241036a0 msgr2=0x7f3e2410bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.762+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2022383957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e241036a0 0x7f3e2410bb90 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009900 tx=0x7f3e1402a480 comp rx=0 tx=0).stop 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.762+0000 7f3e2c341640 1 -- 172.21.15.62:0/2022383957 shutdown_connections 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.762+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2022383957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2410c160 0x7f3e2410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.762+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2022383957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e241036a0 0x7f3e2410bb90 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.762+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2022383957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e24102d80 0x7f3e24103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.762+0000 7f3e2c341640 1 -- 172.21.15.62:0/2022383957 >> 172.21.15.62:0/2022383957 conn(0x7f3e240fc7a0 msgr2=0x7f3e240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.762+0000 7f3e2c341640 1 -- 172.21.15.62:0/2022383957 shutdown_connections 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.762+0000 7f3e2c341640 1 -- 172.21.15.62:0/2022383957 wait complete. 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2c341640 1 Processor -- start 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2c341640 1 -- start start 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2c341640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e24102d80 0x7f3e241984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2c341640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e241036a0 0x7f3e241989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2c341640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2410c160 0x7f3e2419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2c341640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24111fa0 con 0x7f3e2410c160 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2c341640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24111e20 con 0x7f3e24102d80 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2c341640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24112120 con 0x7f3e241036a0 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2a8de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2410c160 0x7f3e2419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2a8de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2410c160 0x7f3e2419cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36048/0 (socket says 172.21.15.62:36048) 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2a8de640 1 -- 172.21.15.62:0/2150295951 learned_addr learned my addr 172.21.15.62:0/2150295951 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e24102d80 0x7f3e241984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.763+0000 7f3e298dc640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e241036a0 0x7f3e241989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e2a8de640 1 -- 172.21.15.62:0/2150295951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e241036a0 msgr2=0x7f3e241989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e2a8de640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e241036a0 0x7f3e241989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e2a8de640 1 -- 172.21.15.62:0/2150295951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e24102d80 msgr2=0x7f3e241984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e2a8de640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e24102d80 0x7f3e241984b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e2a8de640 1 -- 172.21.15.62:0/2150295951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e2419d460 con 0x7f3e2410c160 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e298dc640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e241036a0 0x7f3e241989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e2a8de640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2410c160 0x7f3e2419cd80 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00eaa0 tx=0x7f3e1c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e1b7fe640 1 -- 172.21.15.62:0/2150295951 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c004d20 con 0x7f3e2410c160 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e1b7fe640 1 -- 172.21.15.62:0/2150295951 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e1c005020 con 0x7f3e2410c160 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e2c341640 1 -- 172.21.15.62:0/2150295951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e2419d750 con 0x7f3e2410c160 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e1b7fe640 1 -- 172.21.15.62:0/2150295951 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0116d0 con 0x7f3e2410c160 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.764+0000 7f3e2c341640 1 -- 172.21.15.62:0/2150295951 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e241a5030 con 0x7f3e2410c160 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.766+0000 7f3e1b7fe640 1 -- 172.21.15.62:0/2150295951 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f3e1c0051d0 con 0x7f3e2410c160 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2e71a740 1 Processor -- start 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2e71a740 1 -- start start 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9bb40 0x55e965d9a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d9aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965da5370 0x55e965da7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e965d002f0 con 0x55e965d9a5d0 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e965ca7cf0 con 0x55e965da5370 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e965cfc2b0 con 0x55e965d9bb40 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2a0dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d9aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2a0dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d9aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36060/0 (socket says 172.21.15.62:36060) 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e2a0dd640 1 -- 172.21.15.62:0/2549109259 learned_addr learned my addr 172.21.15.62:0/2549109259 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.767+0000 7f3e298dc640 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9bb40 0x55e965d9a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e298dc640 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965da5370 0x55e965da7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e1b7fe640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e100a3240 0x7f3e100a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e2a0dd640 1 -- 172.21.15.62:0/2549109259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9bb40 msgr2=0x55e965d9a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9bb40 0x55e965d9a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e2a0dd640 1 -- 172.21.15.62:0/2549109259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965da5370 msgr2=0x55e965da7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965da5370 0x55e965da7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e2a0dd640 1 -- 172.21.15.62:0/2549109259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e965c8c3a0 con 0x55e965d9a5d0 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e298dc640 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9bb40 0x55e965d9a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e2a8de640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e100a3240 0x7f3e100a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e1b7fe640 1 -- 172.21.15.62:0/2150295951 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3e1c0cff20 con 0x7f3e2410c160 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d9aa10 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009a70 tx=0x7f3e20009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc6cc9376274f64 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e18dea640 1 -- 172.21.15.62:0/2549109259 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20017070 con 0x55e965d9a5d0 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e18dea640 1 -- 172.21.15.62:0/2549109259 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e20013470 con 0x55e965d9a5d0 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.768+0000 7f3e18dea640 1 -- 172.21.15.62:0/2549109259 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20012630 con 0x55e965d9a5d0 2024-09-15T23:29:38.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2e71a740 1 -- 172.21.15.62:0/2549109259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 msgr2=0x55e965d9aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2a8de640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e100a3240 0x7f3e100a5700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c098620 tx=0x7f3e1c0bc040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d9aa10 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009a70 tx=0x7f3e20009f30 comp rx=0 tx=0).stop 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2e71a740 1 -- 172.21.15.62:0/2549109259 shutdown_connections 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965da5370 0x55e965da7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d9aa10 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/2549109259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9bb40 0x55e965d9a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2e71a740 1 -- 172.21.15.62:0/2549109259 >> 172.21.15.62:0/2549109259 conn(0x55e965d63310 msgr2=0x55e965d96c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2e71a740 1 -- 172.21.15.62:0/2549109259 shutdown_connections 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.769+0000 7f3e2e71a740 1 -- 172.21.15.62:0/2549109259 wait complete. 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2e71a740 1 -- start start 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2e71a740 1 Processor -- start 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965da4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965da2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 0x55e965da25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55e965da8070 con 0x55e965d9a5d0 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55e965cbb060 con 0x55e965d9bb40 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55e965da81f0 con 0x55e965da5370 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2e71a740 10 client.0 _ll_register_callbacks cb 0x55e965a0e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e298dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965da4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e298dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965da4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36064/0 (socket says 172.21.15.62:36064) 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2a0dd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965da2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e298dc640 1 -- 172.21.15.62:0/1482239255 learned_addr learned my addr 172.21.15.62:0/1482239255 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e298dc640 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 0x55e965da25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e298dc640 1 -- 172.21.15.62:0/1482239255 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 msgr2=0x55e965da25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e298dc640 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 0x55e965da25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e298dc640 1 -- 172.21.15.62:0/1482239255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 msgr2=0x55e965da2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e298dc640 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965da2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e298dc640 1 -- 172.21.15.62:0/1482239255 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e965a713c0 con 0x55e965d9a5d0 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.770+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965da2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e298dc640 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965da4d50 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3e1402a9a0 tx=0x7f3e14009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47740de70aaa89aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e0dffb640 1 -- 172.21.15.62:0/1482239255 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14035040 con 0x55e965d9a5d0 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e0dffb640 1 -- 172.21.15.62:0/1482239255 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3e14003a00 con 0x55e965d9a5d0 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e2e71a740 1 -- 172.21.15.62:0/1482239255 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e20013470 con 0x55e965d9a5d0 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e0dffb640 1 -- 172.21.15.62:0/1482239255 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3e14051050 con 0x55e965d9a5d0 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e2e71a740 10 client.18558 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:38.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e2e71a740 1 -- 172.21.15.62:0/1482239255 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e965db3af0 con 0x55e965d9a5d0 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e0dffb640 1 -- 172.21.15.62:0/1482239255 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e20013470 con 0x55e965d9a5d0 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e0dffb640 1 -- 172.21.15.62:0/1482239255 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f3e1403a040 con 0x55e965d9a5d0 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e2e71a740 10 client.18558 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e2e71a740 10 client.18558 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e2e71a740 20 client.18558 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e965db4f00 0x55e965db72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e2e71a740 4 client.18558 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.771+0000 7f3e2e71a740 1 -- 172.21.15.62:0/1482239255 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e965d0d860 con 0x55e965db4f00 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.772+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e965db4f00 0x55e965db72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.772+0000 7f3e0dffb640 10 client.18558 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.772+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e965db4f00 0x55e965db72f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ab5a2e82ade0f62d server_cookie=691288305b0da1e5 in_seq=0 out_seq=0 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e0dffb640 1 -- 172.21.15.62:0/1482239255 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e965d0d860 con 0x55e965db4f00 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e0dffb640 10 client.18558 handle_command_reply: tid=0 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 client.18558 shutdown 2024-09-15T23:29:38.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 -- 172.21.15.62:0/1482239255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 msgr2=0x55e965da4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965da4d50 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3e1402a9a0 tx=0x7f3e14009680 comp rx=0 tx=0).stop 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 -- 172.21.15.62:0/1482239255 shutdown_connections 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e965db4f00 0x55e965db72f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 0x55e965da25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965da2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e0dffb640 0 client.18558 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/1482239255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965da4d50 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 -- 172.21.15.62:0/1482239255 >> 172.21.15.62:0/1482239255 conn(0x55e965d63310 msgr2=0x55e965d9c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 -- 172.21.15.62:0/1482239255 shutdown_connections 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 1 -- 172.21.15.62:0/1482239255 wait complete. 2024-09-15T23:29:38.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.773+0000 7f3e2e71a740 20 client.18558 trim_cache size 0 max 16384 2024-09-15T23:29:38.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.786+0000 7f3e2e71a740 1 Processor -- start 2024-09-15T23:29:38.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.786+0000 7f3e2e71a740 1 -- start start 2024-09-15T23:29:38.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d95bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965d9d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 0x55e965d9d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241b1520 con 0x55e965d9a5d0 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241aedc0 con 0x55e965d9bb40 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241acc10 con 0x55e965da5370 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d95bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d95bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36074/0 (socket says 172.21.15.62:36074) 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 -- 172.21.15.62:0/731972332 learned_addr learned my addr 172.21.15.62:0/731972332 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/731972332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965d9d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 --2- 172.21.15.62:0/731972332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 0x55e965d9d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 -- 172.21.15.62:0/731972332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 msgr2=0x55e965d9d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 --2- 172.21.15.62:0/731972332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 0x55e965d9d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 -- 172.21.15.62:0/731972332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 msgr2=0x55e965d9d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 --2- 172.21.15.62:0/731972332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965d9d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 -- 172.21.15.62:0/731972332 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e965db7950 con 0x55e965d9a5d0 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.787+0000 7f3e298dc640 1 --2- 172.21.15.62:0/731972332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d95bf0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f3e140059d0 tx=0x7f3e14005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59e7e9eba9f1083 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e0ffff640 1 -- 172.21.15.62:0/731972332 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14035070 con 0x55e965d9a5d0 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e0ffff640 1 -- 172.21.15.62:0/731972332 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e14005550 con 0x55e965d9a5d0 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e0ffff640 1 -- 172.21.15.62:0/731972332 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1403d480 con 0x55e965d9a5d0 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e2e71a740 1 -- 172.21.15.62:0/731972332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 msgr2=0x55e965d95bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:38.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/731972332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d95bf0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f3e140059d0 tx=0x7f3e14005c90 comp rx=0 tx=0).stop 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e2e71a740 1 -- 172.21.15.62:0/731972332 shutdown_connections 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/731972332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965da5370 0x55e965d9d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/731972332 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965d9d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/731972332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965d9a5d0 0x55e965d95bf0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e2e71a740 1 -- 172.21.15.62:0/731972332 >> 172.21.15.62:0/731972332 conn(0x55e965d63310 msgr2=0x55e965d954e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e2e71a740 1 -- 172.21.15.62:0/731972332 shutdown_connections 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.788+0000 7f3e2e71a740 1 -- 172.21.15.62:0/731972332 wait complete. 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.789+0000 7f3e2e71a740 1 -- start start 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.789+0000 7f3e2e71a740 1 Processor -- start 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.789+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9a5d0 0x55e965d9fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.789+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965dbc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.789+0000 7f3e2e71a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965da5370 0x55e965dbcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.789+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241ad280 con 0x55e965da5370 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.789+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241ae6f0 con 0x55e965d9bb40 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.789+0000 7f3e2e71a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241ad950 con 0x55e965d9a5d0 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.789+0000 7f3e2e71a740 10 client.0 _ll_register_callbacks cb 0x7f3e14009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965da5370 0x55e965dbcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965da5370 0x55e965dbcfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36076/0 (socket says 172.21.15.62:36076) 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 -- 172.21.15.62:0/3059369630 learned_addr learned my addr 172.21.15.62:0/3059369630 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9a5d0 0x55e965d9fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965dbc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 -- 172.21.15.62:0/3059369630 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9a5d0 msgr2=0x55e965d9fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9a5d0 0x55e965d9fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 -- 172.21.15.62:0/3059369630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 msgr2=0x55e965dbc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:38.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965dbc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 -- 172.21.15.62:0/3059369630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e965e22ba0 con 0x55e965da5370 2024-09-15T23:29:38.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965dbc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:38.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e298dc640 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965da5370 0x55e965dbcfd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f3e14000c00 tx=0x7f3e1403cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e825e63f7fcacd1e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:38.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e0dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:38.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e0dffb640 1 -- 172.21.15.62:0/3059369630 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e14035070 con 0x55e965da5370 2024-09-15T23:29:38.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e0dffb640 1 -- 172.21.15.62:0/3059369630 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f3e1403c9d0 con 0x55e965da5370 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.790+0000 7f3e2e71a740 1 -- 172.21.15.62:0/3059369630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e14005550 con 0x55e965da5370 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e0dffb640 1 -- 172.21.15.62:0/3059369630 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3e14063050 con 0x55e965da5370 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e2e71a740 10 client.18564 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e2e71a740 1 -- 172.21.15.62:0/3059369630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e965e23da0 con 0x55e965da5370 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e0dffb640 1 -- 172.21.15.62:0/3059369630 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e14005550 con 0x55e965da5370 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e0dffb640 1 -- 172.21.15.62:0/3059369630 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f3e14003c10 con 0x55e965da5370 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e2e71a740 10 client.18564 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e2e71a740 10 client.18564 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e2e71a740 20 client.18564 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:38.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e965e24490 0x55e965e268b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e2e71a740 4 client.18564 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e2e71a740 1 -- 172.21.15.62:0/3059369630 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e965d07d70 con 0x55e965e24490 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.791+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e965e24490 0x55e965e268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2a0dd640 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e965e24490 0x55e965e268b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f0c097e6a56aa537 server_cookie=dfb58d4f91ce3f16 in_seq=0 out_seq=0 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e0dffb640 10 client.18564 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e0dffb640 1 -- 172.21.15.62:0/3059369630 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e965d07d70 con 0x55e965e24490 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e0dffb640 10 client.18564 handle_command_reply: tid=0 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 client.18564 shutdown 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 -- 172.21.15.62:0/3059369630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965da5370 msgr2=0x55e965dbcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965da5370 0x55e965dbcfd0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f3e14000c00 tx=0x7f3e1403cc60 comp rx=0 tx=0).stop 2024-09-15T23:29:38.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 -- 172.21.15.62:0/3059369630 shutdown_connections 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x55e965e24490 0x55e965e268b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e965da5370 0x55e965dbcfd0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e965d9bb40 0x55e965dbc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 --2- 172.21.15.62:0/3059369630 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55e965d9a5d0 0x55e965d9fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 -- 172.21.15.62:0/3059369630 >> 172.21.15.62:0/3059369630 conn(0x55e965d63310 msgr2=0x55e965d95d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e0dffb640 0 client.18564 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 -- 172.21.15.62:0/3059369630 shutdown_connections 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 1 -- 172.21.15.62:0/3059369630 wait complete. 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.792+0000 7f3e2e71a740 20 client.18564 trim_cache size 0 max 16384 2024-09-15T23:29:38.802 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 -- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e100a3240 msgr2=0x7f3e100a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e100a3240 0x7f3e100a5700 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c098620 tx=0x7f3e1c0bc040 comp rx=0 tx=0).stop 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 -- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2410c160 msgr2=0x7f3e2419cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2410c160 0x7f3e2419cd80 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c00eaa0 tx=0x7f3e1c00eea0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 -- 172.21.15.62:0/2150295951 shutdown_connections 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f3e100a3240 0x7f3e100a5700 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3e2410c160 0x7f3e2419cd80 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3e241036a0 0x7f3e241989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 --2- 172.21.15.62:0/2150295951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e24102d80 0x7f3e241984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 -- 172.21.15.62:0/2150295951 >> 172.21.15.62:0/2150295951 conn(0x7f3e240fc7a0 msgr2=0x7f3e240fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 -- 172.21.15.62:0/2150295951 shutdown_connections 2024-09-15T23:29:38.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:38.793+0000 7f3e2c341640 1 -- 172.21.15.62:0/2150295951 wait complete. 2024-09-15T23:29:39.563 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:29:39.576 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 800.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:29:39.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:39 smithi148 ceph-mon[35610]: pgmap v771: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.7 MiB/s wr, 814 op/s 2024-09-15T23:29:39.823 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:39.945 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:39 smithi062 ceph-mon[30949]: pgmap v771: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.7 MiB/s wr, 814 op/s 2024-09-15T23:29:40.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:39 smithi067 ceph-mon[35665]: pgmap v771: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 9.7 MiB/s wr, 814 op/s 2024-09-15T23:29:40.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.311+0000 7f4c8c193640 1 -- 172.21.15.62:0/99290532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 msgr2=0x7f4c8410bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:40.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.311+0000 7f4c8c193640 1 --2- 172.21.15.62:0/99290532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 0x7f4c8410bb50 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009900 tx=0x7f4c7402a480 comp rx=0 tx=0).stop 2024-09-15T23:29:40.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.311+0000 7f4c8c193640 1 -- 172.21.15.62:0/99290532 shutdown_connections 2024-09-15T23:29:40.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.311+0000 7f4c8c193640 1 --2- 172.21.15.62:0/99290532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c8410c170 0x7f4c8410e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.311+0000 7f4c8c193640 1 --2- 172.21.15.62:0/99290532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 0x7f4c8410bb50 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.311+0000 7f4c8c193640 1 --2- 172.21.15.62:0/99290532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c84102d40 0x7f4c84103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.311+0000 7f4c8c193640 1 -- 172.21.15.62:0/99290532 >> 172.21.15.62:0/99290532 conn(0x7f4c840fc7a0 msgr2=0x7f4c840febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:40.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.311+0000 7f4c8c193640 1 -- 172.21.15.62:0/99290532 shutdown_connections 2024-09-15T23:29:40.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.311+0000 7f4c8c193640 1 -- 172.21.15.62:0/99290532 wait complete. 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8c193640 1 Processor -- start 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8c193640 1 -- start start 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8c193640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c84102d40 0x7f4c841009a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8c193640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 0x7f4c84100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8c193640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8410c170 0x7f4c84101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8c193640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4c84111fb0 con 0x7f4c84103660 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8c193640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c84111e30 con 0x7f4c8410c170 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8c193640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4c84112130 con 0x7f4c84102d40 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8972e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 0x7f4c84100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8972e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 0x7f4c84100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36098/0 (socket says 172.21.15.62:36098) 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8972e640 1 -- 172.21.15.62:0/4105055896 learned_addr learned my addr 172.21.15.62:0/4105055896 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:40.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.312+0000 7f4c8a730640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8410c170 0x7f4c84101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c84102d40 0x7f4c841009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c8972e640 1 -- 172.21.15.62:0/4105055896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c84102d40 msgr2=0x7f4c841009a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c8972e640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c84102d40 0x7f4c841009a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c8972e640 1 -- 172.21.15.62:0/4105055896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8410c170 msgr2=0x7f4c84101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c8972e640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8410c170 0x7f4c84101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c8972e640 1 -- 172.21.15.62:0/4105055896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c841a6960 con 0x7f4c84103660 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c84102d40 0x7f4c841009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c8972e640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 0x7f4c84100ee0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4c74006010 tx=0x7f4c74004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c7affd640 1 -- 172.21.15.62:0/4105055896 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7403f030 con 0x7f4c84103660 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c7affd640 1 -- 172.21.15.62:0/4105055896 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c74004440 con 0x7f4c84103660 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c7affd640 1 -- 172.21.15.62:0/4105055896 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74035070 con 0x7f4c84103660 2024-09-15T23:29:40.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c8c193640 1 -- 172.21.15.62:0/4105055896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c841a6bf0 con 0x7f4c84103660 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.313+0000 7f4c8c193640 1 -- 172.21.15.62:0/4105055896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c841a7080 con 0x7f4c84103660 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.315+0000 7f4c7affd640 1 -- 172.21.15.62:0/4105055896 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4c74042030 con 0x7f4c84103660 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8e56c740 1 Processor -- start 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8e56c740 1 -- start start 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 0x5596ce715090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce715a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce722760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5596ce67b2f0 con 0x5596ce7155d0 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5596ce622cf0 con 0x5596ce716b40 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5596ce6772b0 con 0x5596ce720370 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c89f2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce715a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c89f2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce715a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36110/0 (socket says 172.21.15.62:36110) 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c89f2f640 1 -- 172.21.15.62:0/1968625905 learned_addr learned my addr 172.21.15.62:0/1968625905 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:40.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c7affd640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c700a32c0 0x7f4c700a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8a730640 1 --2- 172.21.15.62:0/1968625905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 0x5596ce715090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.316+0000 7f4c8972e640 1 --2- 172.21.15.62:0/1968625905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce722760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c89f2f640 1 -- 172.21.15.62:0/1968625905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 msgr2=0x5596ce722760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/1968625905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce722760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c89f2f640 1 -- 172.21.15.62:0/1968625905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 msgr2=0x5596ce715090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/1968625905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 0x5596ce715090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c8a730640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c700a32c0 0x7f4c700a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c89f2f640 1 -- 172.21.15.62:0/1968625905 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596ce6073a0 con 0x5596ce7155d0 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c7affd640 1 -- 172.21.15.62:0/4105055896 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4c740f0060 con 0x7f4c84103660 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/1968625905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce715a10 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009c30 tx=0x7f4c80010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a405319811e6e73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c6ffff640 1 -- 172.21.15.62:0/1968625905 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80004030 con 0x5596ce7155d0 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c6ffff640 1 -- 172.21.15.62:0/1968625905 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c800041d0 con 0x5596ce7155d0 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.317+0000 7f4c6ffff640 1 -- 172.21.15.62:0/1968625905 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c800126b0 con 0x5596ce7155d0 2024-09-15T23:29:40.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 -- 172.21.15.62:0/1968625905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 msgr2=0x5596ce715a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8a730640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c700a32c0 0x7f4c700a5780 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009fd0 tx=0x7f4c7c004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/1968625905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce715a10 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009c30 tx=0x7f4c80010740 comp rx=0 tx=0).stop 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 -- 172.21.15.62:0/1968625905 shutdown_connections 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/1968625905 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce722760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/1968625905 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce715a10 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/1968625905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 0x5596ce715090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 -- 172.21.15.62:0/1968625905 >> 172.21.15.62:0/1968625905 conn(0x5596ce6de310 msgr2=0x5596ce711c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 -- 172.21.15.62:0/1968625905 shutdown_connections 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 -- 172.21.15.62:0/1968625905 wait complete. 2024-09-15T23:29:40.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 -- start start 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.318+0000 7f4c8e56c740 1 Processor -- start 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce71fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 0x5596ce71cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce71d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5596ce723070 con 0x5596ce7155d0 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5596ce636060 con 0x5596ce716b40 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5596ce7231f0 con 0x5596ce720370 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8e56c740 10 client.0 _ll_register_callbacks cb 0x5596ce3898c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c89f2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce71fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c89f2f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce71fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36120/0 (socket says 172.21.15.62:36120) 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c89f2f640 1 -- 172.21.15.62:0/2018154934 learned_addr learned my addr 172.21.15.62:0/2018154934 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8972e640 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 0x5596ce71cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8a730640 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce71d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c89f2f640 1 -- 172.21.15.62:0/2018154934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 msgr2=0x5596ce71d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce71d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c89f2f640 1 -- 172.21.15.62:0/2018154934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 msgr2=0x5596ce71cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 0x5596ce71cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c89f2f640 1 -- 172.21.15.62:0/2018154934 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596ce3ec3c0 con 0x5596ce7155d0 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c8a730640 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce71d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce71fcf0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4c80003eb0 tx=0x7f4c80003be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9608a593cb739d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.319+0000 7f4c6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2018154934 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c800035f0 con 0x5596ce7155d0 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2018154934 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4c80003790 con 0x5596ce7155d0 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2018154934 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c800041d0 con 0x5596ce7155d0 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2018154934 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4c80003990 con 0x5596ce7155d0 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c8e56c740 10 client.18576 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2018154934 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5596ce72ead0 con 0x5596ce7155d0 2024-09-15T23:29:40.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2018154934 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c800041d0 con 0x5596ce7155d0 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2018154934 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f4c8000f040 con 0x5596ce7155d0 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c8e56c740 10 client.18576 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c8e56c740 10 client.18576 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c8e56c740 20 client.18576 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5596ce72fed0 0x5596ce7322c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c8e56c740 4 client.18576 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.320+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2018154934 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596ce688860 con 0x5596ce72fed0 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.321+0000 7f4c8972e640 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5596ce72fed0 0x5596ce7322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.321+0000 7f4c8972e640 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5596ce72fed0 0x5596ce7322c0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49e017fd070eb882 server_cookie=7d6ed75dfa412414 in_seq=0 out_seq=0 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.321+0000 7f4c6d7fa640 10 client.18576 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2018154934 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5596ce688860 con 0x5596ce72fed0 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c6d7fa640 10 client.18576 handle_command_reply: tid=0 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 client.18576 shutdown 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2018154934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 msgr2=0x5596ce71fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:40.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce71fcf0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4c80003eb0 tx=0x7f4c80003be0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2018154934 shutdown_connections 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5596ce72fed0 0x5596ce7322c0 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce71d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce716b40 0x5596ce71cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c6d7fa640 0 client.18576 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2018154934 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce7155d0 0x5596ce71fcf0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2018154934 >> 172.21.15.62:0/2018154934 conn(0x5596ce6de310 msgr2=0x5596ce717930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2018154934 shutdown_connections 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2018154934 wait complete. 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.322+0000 7f4c8e56c740 20 client.18576 trim_cache size 0 max 16384 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.335+0000 7f4c8e56c740 1 Processor -- start 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.335+0000 7f4c8e56c740 1 -- start start 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.335+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce7155d0 0x5596ce711d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.335+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce722760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.335+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce720370 0x5596ce7352e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.335+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841b3830 con 0x5596ce716b40 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.335+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841b10d0 con 0x5596ce720370 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.335+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841aef20 con 0x5596ce7155d0 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8972e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce722760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8972e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce722760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36124/0 (socket says 172.21.15.62:36124) 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8972e640 1 -- 172.21.15.62:0/3887989478 learned_addr learned my addr 172.21.15.62:0/3887989478 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8a730640 1 --2- 172.21.15.62:0/3887989478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce720370 0x5596ce7352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/3887989478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce7155d0 0x5596ce711d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8972e640 1 -- 172.21.15.62:0/3887989478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce7155d0 msgr2=0x5596ce711d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8972e640 1 --2- 172.21.15.62:0/3887989478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce7155d0 0x5596ce711d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8972e640 1 -- 172.21.15.62:0/3887989478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce720370 msgr2=0x5596ce7352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8972e640 1 --2- 172.21.15.62:0/3887989478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce720370 0x5596ce7352e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8972e640 1 -- 172.21.15.62:0/3887989478 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596ce71d890 con 0x5596ce716b40 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c8972e640 1 --2- 172.21.15.62:0/3887989478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce722760 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f4c740b87c0 tx=0x7f4c7402a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa8df37710d520a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c6f7fe640 1 -- 172.21.15.62:0/3887989478 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74049040 con 0x5596ce716b40 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c6f7fe640 1 -- 172.21.15.62:0/3887989478 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c74003df0 con 0x5596ce716b40 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.336+0000 7f4c6f7fe640 1 -- 172.21.15.62:0/3887989478 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c740bc5c0 con 0x5596ce716b40 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 -- 172.21.15.62:0/3887989478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 msgr2=0x5596ce722760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/3887989478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce722760 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f4c740b87c0 tx=0x7f4c7402a480 comp rx=0 tx=0).stop 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 -- 172.21.15.62:0/3887989478 shutdown_connections 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/3887989478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce720370 0x5596ce7352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/3887989478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce722760 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/3887989478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce7155d0 0x5596ce711d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 -- 172.21.15.62:0/3887989478 >> 172.21.15.62:0/3887989478 conn(0x5596ce6de310 msgr2=0x5596ce71c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:40.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 -- 172.21.15.62:0/3887989478 shutdown_connections 2024-09-15T23:29:40.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 -- 172.21.15.62:0/3887989478 wait complete. 2024-09-15T23:29:40.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 -- start start 2024-09-15T23:29:40.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.337+0000 7f4c8e56c740 1 Processor -- start 2024-09-15T23:29:40.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce7155d0 0x5596ce7358f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce732c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8e56c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce733180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841af590 con 0x5596ce716b40 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841b0a00 con 0x5596ce7155d0 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8e56c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4c841afc60 con 0x5596ce720370 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8e56c740 10 client.0 _ll_register_callbacks cb 0x7f4c7c0099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8972e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce732c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c89f2f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce7155d0 0x5596ce7358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8972e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce732c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:36136/0 (socket says 172.21.15.62:36136) 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c89f2f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce7155d0 0x5596ce7358f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:53036/0 (socket says 172.21.15.62:53036) 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8972e640 1 -- 172.21.15.62:0/2036400949 learned_addr learned my addr 172.21.15.62:0/2036400949 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8a730640 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce733180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c89f2f640 1 -- 172.21.15.62:0/2036400949 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 msgr2=0x5596ce733180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce733180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c89f2f640 1 -- 172.21.15.62:0/2036400949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 msgr2=0x5596ce732c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce732c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c89f2f640 1 -- 172.21.15.62:0/2036400949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596ce733870 con 0x5596ce7155d0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.338+0000 7f4c8972e640 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce732c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c89f2f640 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce7155d0 0x5596ce7358f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009370 tx=0x7f4c80009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd105525224dee87 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2036400949 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c800096e0 con 0x5596ce7155d0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2036400949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c74003df0 con 0x5596ce7155d0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2036400949 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4c80004590 con 0x5596ce7155d0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2036400949 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4c80009880 con 0x5596ce7155d0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c8e56c740 10 client.28106 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2036400949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5596ce79a6b0 con 0x5596ce7155d0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2036400949 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c74003df0 con 0x5596ce7155d0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2036400949 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f4c80004e70 con 0x5596ce7155d0 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c8e56c740 10 client.28106 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c8e56c740 10 client.28106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:40.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.339+0000 7f4c8e56c740 20 client.28106 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5596ce79acc0 0x5596ce79d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8e56c740 4 client.28106 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2036400949 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596ce682d70 con 0x5596ce79acc0 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8972e640 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5596ce79acc0 0x5596ce79d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c6d7fa640 10 client.28106 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8972e640 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5596ce79acc0 0x5596ce79d0b0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0eafbeda2dac1f0 server_cookie=506c062c8663e774 in_seq=0 out_seq=0 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c6d7fa640 1 -- 172.21.15.62:0/2036400949 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596ce682d70 con 0x5596ce79acc0 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c6d7fa640 10 client.28106 handle_command_reply: tid=0 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8e56c740 1 client.28106 shutdown 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2036400949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce7155d0 msgr2=0x5596ce7358f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce7155d0 0x5596ce7358f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009370 tx=0x7f4c80009b30 comp rx=0 tx=0).stop 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2036400949 shutdown_connections 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.340+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5596ce79acc0 0x5596ce79d0b0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5596ce720370 0x5596ce733180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5596ce716b40 0x5596ce732c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8e56c740 1 --2- 172.21.15.62:0/2036400949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5596ce7155d0 0x5596ce7358f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c6d7fa640 0 client.28106 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:40.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2036400949 >> 172.21.15.62:0/2036400949 conn(0x5596ce6de310 msgr2=0x5596ce717110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:40.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2036400949 shutdown_connections 2024-09-15T23:29:40.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8e56c740 1 -- 172.21.15.62:0/2036400949 wait complete. 2024-09-15T23:29:40.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8e56c740 20 client.28106 trim_cache size 0 max 16384 2024-09-15T23:29:40.359 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:29:40.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 -- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c700a32c0 msgr2=0x7f4c700a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:40.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c700a32c0 0x7f4c700a5780 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009fd0 tx=0x7f4c7c004a40 comp rx=0 tx=0).stop 2024-09-15T23:29:40.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 -- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 msgr2=0x7f4c84100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:40.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 0x7f4c84100ee0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4c74006010 tx=0x7f4c74004040 comp rx=0 tx=0).stop 2024-09-15T23:29:40.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 -- 172.21.15.62:0/4105055896 shutdown_connections 2024-09-15T23:29:40.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c700a32c0 0x7f4c700a5780 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8410c170 0x7f4c84101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c84103660 0x7f4c84100ee0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 --2- 172.21.15.62:0/4105055896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c84102d40 0x7f4c841009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:40.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 -- 172.21.15.62:0/4105055896 >> 172.21.15.62:0/4105055896 conn(0x7f4c840fc7a0 msgr2=0x7f4c840fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:40.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.341+0000 7f4c8c193640 1 -- 172.21.15.62:0/4105055896 shutdown_connections 2024-09-15T23:29:40.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:40.342+0000 7f4c8c193640 1 -- 172.21.15.62:0/4105055896 wait complete. 2024-09-15T23:29:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:41 smithi148 ceph-mon[35610]: pgmap v772: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.1 MiB/s wr, 608 op/s 2024-09-15T23:29:41.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:41 smithi062 ceph-mon[30949]: pgmap v772: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.1 MiB/s wr, 608 op/s 2024-09-15T23:29:42.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:41 smithi067 ceph-mon[35665]: pgmap v772: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.1 MiB/s wr, 608 op/s 2024-09-15T23:29:42.045 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:29:42.045 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:29:42.318 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:42.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 -- 172.21.15.62:0/1461291872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 msgr2=0x7f2c3410d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/1461291872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 0x7f2c3410d930 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c009930 tx=0x7f2c2c02a420 comp rx=0 tx=0).stop 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 -- 172.21.15.62:0/1461291872 shutdown_connections 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/1461291872 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 0x7f2c3410d930 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/1461291872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c34100a60 0x7f2c34100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/1461291872 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c34106a60 0x7f2c34106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 -- 172.21.15.62:0/1461291872 >> 172.21.15.62:0/1461291872 conn(0x7f2c340fc780 msgr2=0x7f2c340feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 -- 172.21.15.62:0/1461291872 shutdown_connections 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 -- 172.21.15.62:0/1461291872 wait complete. 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 Processor -- start 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.789+0000 7f2c3bf72640 1 -- start start 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3bf72640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c34100a60 0x7f2c3419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3bf72640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 0x7f2c3419ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3bf72640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c34106a60 0x7f2c3419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3bf72640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2c34110e60 con 0x7f2c34101400 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3bf72640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c34110ce0 con 0x7f2c34100a60 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3bf72640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2c34110fe0 con 0x7f2c34106a60 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3950d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 0x7f2c3419ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3950d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 0x7f2c3419ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37474/0 (socket says 172.21.15.62:37474) 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3950d640 1 -- 172.21.15.62:0/445581457 learned_addr learned my addr 172.21.15.62:0/445581457 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c34100a60 0x7f2c3419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c34106a60 0x7f2c3419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3950d640 1 -- 172.21.15.62:0/445581457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c34106a60 msgr2=0x7f2c3419eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3950d640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c34106a60 0x7f2c3419eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3950d640 1 -- 172.21.15.62:0/445581457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c34100a60 msgr2=0x7f2c3419a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3950d640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c34100a60 0x7f2c3419a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3950d640 1 -- 172.21.15.62:0/445581457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c3419f590 con 0x7f2c34101400 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3950d640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 0x7f2c3419ab20 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000e9f0 tx=0x7f2c3000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c26ffd640 1 -- 172.21.15.62:0/445581457 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3000cd80 con 0x7f2c34101400 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c26ffd640 1 -- 172.21.15.62:0/445581457 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c30010910 con 0x7f2c34101400 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c3bf72640 1 -- 172.21.15.62:0/445581457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c3419f880 con 0x7f2c34101400 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.790+0000 7f2c26ffd640 1 -- 172.21.15.62:0/445581457 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c30018990 con 0x7f2c34101400 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.791+0000 7f2c3bf72640 1 -- 172.21.15.62:0/445581457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c341a7160 con 0x7f2c34101400 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.793+0000 7f2c26ffd640 1 -- 172.21.15.62:0/445581457 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2c30010ac0 con 0x7f2c34101400 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.793+0000 7f2c3e34b740 1 Processor -- start 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.793+0000 7f2c26ffd640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c200a3240 0x7f2c200a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.793+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c200a3240 0x7f2c200a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.793+0000 7f2c26ffd640 1 -- 172.21.15.62:0/445581457 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2c300d0070 con 0x7f2c34101400 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3e34b740 1 -- start start 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 0x56456259cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56456259d480 0x5645625a7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5645625a7d40 0x5645625aa130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564562502280 con 0x56456259d480 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5645624a9b10 con 0x5645625a7d40 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5645624fe240 con 0x56456259cb60 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3950d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56456259d480 0x5645625a7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c200a3240 0x7f2c200a5700 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0096c0 tx=0x7f2c2c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3950d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56456259d480 0x5645625a7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37478/0 (socket says 172.21.15.62:37478) 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3950d640 1 -- 172.21.15.62:0/2787383913 learned_addr learned my addr 172.21.15.62:0/2787383913 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 0x56456259cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5645625a7d40 0x5645625aa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3950d640 1 -- 172.21.15.62:0/2787383913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 msgr2=0x56456259cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3950d640 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 0x56456259cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3950d640 1 -- 172.21.15.62:0/2787383913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5645625a7d40 msgr2=0x5645625aa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3950d640 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5645625a7d40 0x5645625aa130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3950d640 1 -- 172.21.15.62:0/2787383913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645624b61f0 con 0x56456259d480 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5645625a7d40 0x5645625aa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.794+0000 7f2c3950d640 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56456259d480 0x5645625a7800 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f2c3009c2a0 tx=0x7f2c3009ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0c5671e1ce12d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c1ffff640 1 -- 172.21.15.62:0/2787383913 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c300a4f80 con 0x56456259d480 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c1ffff640 1 -- 172.21.15.62:0/2787383913 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c3009a610 con 0x56456259d480 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c1ffff640 1 -- 172.21.15.62:0/2787383913 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3009b960 con 0x56456259d480 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c3e34b740 1 -- 172.21.15.62:0/2787383913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56456259d480 msgr2=0x5645625a7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56456259d480 0x5645625a7800 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f2c3009c2a0 tx=0x7f2c3009ca10 comp rx=0 tx=0).stop 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c3e34b740 1 -- 172.21.15.62:0/2787383913 shutdown_connections 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5645625a7d40 0x5645625aa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56456259d480 0x5645625a7800 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/2787383913 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 0x56456259cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c3e34b740 1 -- 172.21.15.62:0/2787383913 >> 172.21.15.62:0/2787383913 conn(0x5645625990f0 msgr2=0x5645625994d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.795+0000 7f2c3e34b740 1 -- 172.21.15.62:0/2787383913 shutdown_connections 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 1 -- 172.21.15.62:0/2787383913 wait complete. 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 1 -- start start 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 1 Processor -- start 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 0x5645625a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 0x5645625a5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625a5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5645625aa910 con 0x5645625a7d40 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5645624bcf10 con 0x56456259cb60 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5645625aaa90 con 0x56456259d480 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.796+0000 7f2c3e34b740 10 client.0 _ll_register_callbacks cb 0x564562485a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3a50f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c39d0e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 0x5645625a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3a50f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625a5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37492/0 (socket says 172.21.15.62:37492) 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3a50f640 1 -- 172.21.15.62:0/707676683 learned_addr learned my addr 172.21.15.62:0/707676683 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3950d640 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 0x5645625a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3a50f640 1 -- 172.21.15.62:0/707676683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 msgr2=0x5645625a5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 0x5645625a5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3a50f640 1 -- 172.21.15.62:0/707676683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 msgr2=0x5645625a4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 0x5645625a4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3a50f640 1 -- 172.21.15.62:0/707676683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645622736a0 con 0x5645625a7d40 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 0x5645625a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625a5670 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c02a940 tx=0x7f2c2c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af00a542eac3cf11 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/707676683 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c035050 con 0x5645625a7d40 2024-09-15T23:29:42.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/707676683 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2c2c005230 con 0x5645625a7d40 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.797+0000 7f2c3e34b740 1 -- 172.21.15.62:0/707676683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c3009a610 con 0x5645625a7d40 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/707676683 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2c2c057050 con 0x5645625a7d40 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c3e34b740 10 client.18594 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c3e34b740 1 -- 172.21.15.62:0/707676683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5645625b6450 con 0x5645625a7d40 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/707676683 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c3009a610 con 0x5645625a7d40 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/707676683 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f2c2c0036f0 con 0x5645625a7d40 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c3e34b740 10 client.18594 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c3e34b740 10 client.18594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c3e34b740 20 client.18594 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5645625b7910 0x5645625b9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c3e34b740 4 client.18594 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c3e34b740 1 -- 172.21.15.62:0/707676683 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56456250f7f0 con 0x5645625b7910 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.798+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5645625b7910 0x5645625b9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.799+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5645625b7910 0x5645625b9d60 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94c74ce17df601d6 server_cookie=17ba67173ea32d49 in_seq=0 out_seq=0 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.799+0000 7f2c1d7fa640 10 client.18594 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/707676683 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56456250f7f0 con 0x5645625b7910 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c1d7fa640 10 client.18594 handle_command_reply: tid=0 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 client.18594 shutdown 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 -- 172.21.15.62:0/707676683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 msgr2=0x5645625a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625a5670 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c02a940 tx=0x7f2c2c009260 comp rx=0 tx=0).stop 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 -- 172.21.15.62:0/707676683 shutdown_connections 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5645625b7910 0x5645625b9d60 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625a5670 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 0x5645625a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c1d7fa640 0 client.18594 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/707676683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 0x5645625a4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 -- 172.21.15.62:0/707676683 >> 172.21.15.62:0/707676683 conn(0x5645625990f0 msgr2=0x564562598270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:42.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 -- 172.21.15.62:0/707676683 shutdown_connections 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 1 -- 172.21.15.62:0/707676683 wait complete. 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.800+0000 7f2c3e34b740 20 client.18594 trim_cache size 0 max 16384 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.817+0000 7f2c3e34b740 1 Processor -- start 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.817+0000 7f2c3e34b740 1 -- start start 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.817+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 0x5645625a6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.817+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 0x5645625bf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.817+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2c341b3810 con 0x5645625a7d40 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c341b10b0 con 0x56456259cb60 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2c341aef00 con 0x56456259d480 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3a50f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3a50f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625bfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37494/0 (socket says 172.21.15.62:37494) 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3a50f640 1 -- 172.21.15.62:0/1544215194 learned_addr learned my addr 172.21.15.62:0/1544215194 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:42.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/1544215194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 0x5645625a6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3950d640 1 --2- 172.21.15.62:0/1544215194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 0x5645625bf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3a50f640 1 -- 172.21.15.62:0/1544215194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 msgr2=0x5645625bf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/1544215194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 0x5645625bf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3a50f640 1 -- 172.21.15.62:0/1544215194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 msgr2=0x5645625a6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/1544215194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 0x5645625a6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3a50f640 1 -- 172.21.15.62:0/1544215194 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564562599d80 con 0x5645625a7d40 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/1544215194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625bfbc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c003ae0 tx=0x7f2c2c0045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3b18bfa4c28ae9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c1f7fe640 1 -- 172.21.15.62:0/1544215194 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c035070 con 0x5645625a7d40 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c1f7fe640 1 -- 172.21.15.62:0/1544215194 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c041040 con 0x5645625a7d40 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.818+0000 7f2c1f7fe640 1 -- 172.21.15.62:0/1544215194 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c004b30 con 0x5645625a7d40 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 -- 172.21.15.62:0/1544215194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 msgr2=0x5645625bfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/1544215194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625bfbc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c003ae0 tx=0x7f2c2c0045a0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 -- 172.21.15.62:0/1544215194 shutdown_connections 2024-09-15T23:29:42.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/1544215194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625bfbc0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/1544215194 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259d480 0x5645625bf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/1544215194 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259cb60 0x5645625a6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 -- 172.21.15.62:0/1544215194 >> 172.21.15.62:0/1544215194 conn(0x5645625990f0 msgr2=0x5645625a6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 -- 172.21.15.62:0/1544215194 shutdown_connections 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 -- 172.21.15.62:0/1544215194 wait complete. 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.819+0000 7f2c3e34b740 1 -- start start 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3e34b740 1 Processor -- start 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 0x5645625b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259d480 0x5645625ab810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3e34b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625abd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2c341af570 con 0x5645625a7d40 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2c341b09e0 con 0x56456259d480 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3e34b740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2c341afc40 con 0x56456259cb60 2024-09-15T23:29:42.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3e34b740 10 client.0 _ll_register_callbacks cb 0x7f2c2c032e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3a50f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3a50f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37510/0 (socket says 172.21.15.62:37510) 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3a50f640 1 -- 172.21.15.62:0/3532143574 learned_addr learned my addr 172.21.15.62:0/3532143574 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c3950d640 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259d480 0x5645625ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.820+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 0x5645625b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3a50f640 1 -- 172.21.15.62:0/3532143574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 msgr2=0x5645625b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 0x5645625b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3a50f640 1 -- 172.21.15.62:0/3532143574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259d480 msgr2=0x5645625ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259d480 0x5645625ab810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3a50f640 1 -- 172.21.15.62:0/3532143574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645625ac5c0 con 0x5645625a7d40 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 0x5645625b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3950d640 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259d480 0x5645625ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:42.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3a50f640 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625abd90 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c03d040 tx=0x7f2c2c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793db4e51f697f73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c1d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/3532143574 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2c069060 con 0x5645625a7d40 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/3532143574 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2c2c03b870 con 0x5645625a7d40 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3e34b740 1 -- 172.21.15.62:0/3532143574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c2c041040 con 0x5645625a7d40 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/3532143574 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2c2c070050 con 0x5645625a7d40 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3e34b740 10 client.18600 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c3e34b740 1 -- 172.21.15.62:0/3532143574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564562622730 con 0x5645625a7d40 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.821+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/3532143574 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c2c041040 con 0x5645625a7d40 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/3532143574 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f2c2c05c7c0 con 0x5645625a7d40 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c3e34b740 10 client.18600 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c3e34b740 10 client.18600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c3e34b740 20 client.18600 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564562622ce0 0x5645626250d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:42.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c3e34b740 4 client.18600 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:29:42.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c3e34b740 1 -- 172.21.15.62:0/3532143574 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564562509d00 con 0x564562622ce0 2024-09-15T23:29:42.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564562622ce0 0x5645626250d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:42.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c39d0e640 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564562622ce0 0x5645626250d0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad20a558c00ed3db server_cookie=1e5b5d33071aac2e in_seq=0 out_seq=0 2024-09-15T23:29:42.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.822+0000 7f2c1d7fa640 10 client.18600 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.823+0000 7f2c1d7fa640 1 -- 172.21.15.62:0/3532143574 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564562509d00 con 0x564562622ce0 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.823+0000 7f2c1d7fa640 10 client.18600 handle_command_reply: tid=0 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.823+0000 7f2c3e34b740 1 client.18600 shutdown 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 -- 172.21.15.62:0/3532143574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 msgr2=0x5645625abd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625abd90 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c03d040 tx=0x7f2c2c003820 comp rx=0 tx=0).stop 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 -- 172.21.15.62:0/3532143574 shutdown_connections 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564562622ce0 0x5645626250d0 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5645625a7d40 0x5645625abd90 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56456259d480 0x5645625ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c1d7fa640 0 client.18600 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 --2- 172.21.15.62:0/3532143574 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56456259cb60 0x5645625b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 -- 172.21.15.62:0/3532143574 >> 172.21.15.62:0/3532143574 conn(0x5645625990f0 msgr2=0x564562598270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:42.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 -- 172.21.15.62:0/3532143574 shutdown_connections 2024-09-15T23:29:42.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 1 -- 172.21.15.62:0/3532143574 wait complete. 2024-09-15T23:29:42.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3e34b740 20 client.18600 trim_cache size 0 max 16384 2024-09-15T23:29:42.840 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:29:42.840 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:29:42.840 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "d317d89b-86a6-462d-b572-88f09e1a986b", 2024-09-15T23:29:42.840 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:29:42.840 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:29:42.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3bf72640 1 -- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c200a3240 msgr2=0x7f2c200a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:42.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c200a3240 0x7f2c200a5700 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0096c0 tx=0x7f2c2c005fd0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3bf72640 1 -- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 msgr2=0x7f2c3419ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:42.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.824+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 0x7f2c3419ab20 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000e9f0 tx=0x7f2c3000eeb0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.825+0000 7f2c3bf72640 1 -- 172.21.15.62:0/445581457 shutdown_connections 2024-09-15T23:29:42.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.825+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2c200a3240 0x7f2c200a5700 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.825+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2c34106a60 0x7f2c3419eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.825+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c34101400 0x7f2c3419ab20 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.825+0000 7f2c3bf72640 1 --2- 172.21.15.62:0/445581457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c34100a60 0x7f2c3419a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:42.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.825+0000 7f2c3bf72640 1 -- 172.21.15.62:0/445581457 >> 172.21.15.62:0/445581457 conn(0x7f2c340fc780 msgr2=0x7f2c3406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:42.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.825+0000 7f2c3bf72640 1 -- 172.21.15.62:0/445581457 shutdown_connections 2024-09-15T23:29:42.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:42.825+0000 7f2c3bf72640 1 -- 172.21.15.62:0/445581457 wait complete. 2024-09-15T23:29:43.560 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:29:43.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: pgmap v773: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.2 MiB/s wr, 385 op/s 2024-09-15T23:29:43.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:29:43.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:29:43.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:29:43.731 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:29:43.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:29:43.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:43 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: pgmap v773: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.2 MiB/s wr, 385 op/s 2024-09-15T23:29:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:29:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:29:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:29:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:29:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:29:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:43 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:43.823 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: pgmap v773: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.2 MiB/s wr, 385 op/s 2024-09-15T23:29:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:29:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:44.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:29:44.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:29:44.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:44.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:29:44.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:29:44.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:44.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:44.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:43 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.388+0000 7fc12d556640 1 -- 172.21.15.62:0/1548065191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 msgr2=0x7fc1281029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.388+0000 7fc12d556640 1 --2- 172.21.15.62:0/1548065191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 0x7fc1281029b0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009900 tx=0x7fc11c02a480 comp rx=0 tx=0).stop 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.389+0000 7fc12d556640 1 -- 172.21.15.62:0/1548065191 shutdown_connections 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.389+0000 7fc12d556640 1 --2- 172.21.15.62:0/1548065191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128102ef0 0x7fc12810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.389+0000 7fc12d556640 1 --2- 172.21.15.62:0/1548065191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 0x7fc1281029b0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.389+0000 7fc12d556640 1 --2- 172.21.15.62:0/1548065191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc128100e20 0x7fc128101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.389+0000 7fc12d556640 1 -- 172.21.15.62:0/1548065191 >> 172.21.15.62:0/1548065191 conn(0x7fc1280fc7d0 msgr2=0x7fc1280febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.389+0000 7fc12d556640 1 -- 172.21.15.62:0/1548065191 shutdown_connections 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.389+0000 7fc12d556640 1 -- 172.21.15.62:0/1548065191 wait complete. 2024-09-15T23:29:44.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.389+0000 7fc12d556640 1 Processor -- start 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.389+0000 7fc12d556640 1 -- start start 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc12d556640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128100e20 0x7fc1280fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc12d556640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 0x7fc1280fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc12d556640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc128102ef0 0x7fc1280fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc12d556640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc128110f80 con 0x7fc128102570 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc12d556640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc128110e00 con 0x7fc128100e20 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc12d556640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc128111100 con 0x7fc128102ef0 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1267fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 0x7fc1280fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1267fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 0x7fc1280fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37540/0 (socket says 172.21.15.62:37540) 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1267fc640 1 -- 172.21.15.62:0/2389745810 learned_addr learned my addr 172.21.15.62:0/2389745810 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc126ffd640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128100e20 0x7fc1280fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1277fe640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc128102ef0 0x7fc1280fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1267fc640 1 -- 172.21.15.62:0/2389745810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc128102ef0 msgr2=0x7fc1280fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1267fc640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc128102ef0 0x7fc1280fd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1267fc640 1 -- 172.21.15.62:0/2389745810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128100e20 msgr2=0x7fc1280fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1267fc640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128100e20 0x7fc1280fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1267fc640 1 -- 172.21.15.62:0/2389745810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1281a6970 con 0x7fc128102570 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc1267fc640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 0x7fc1280fd1c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fc11c006010 tx=0x7fc11c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.390+0000 7fc10ffff640 1 -- 172.21.15.62:0/2389745810 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c03f070 con 0x7fc128102570 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.391+0000 7fc10ffff640 1 -- 172.21.15.62:0/2389745810 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc11c031c30 con 0x7fc128102570 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.391+0000 7fc12d556640 1 -- 172.21.15.62:0/2389745810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1281a6c00 con 0x7fc128102570 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.391+0000 7fc10ffff640 1 -- 172.21.15.62:0/2389745810 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c035030 con 0x7fc128102570 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.391+0000 7fc12d556640 1 -- 172.21.15.62:0/2389745810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1281a7130 con 0x7fc128102570 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.393+0000 7fc10ffff640 1 -- 172.21.15.62:0/2389745810 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc11c042070 con 0x7fc128102570 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.393+0000 7fc12f92f740 1 Processor -- start 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.393+0000 7fc12f92f740 1 -- start start 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e944a0 0x560d75e94880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75e9f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75ea1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc12f92f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560d75dfa2e0 con 0x560d75e94dc0 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc12f92f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560d75da1ce0 con 0x560d75e9f680 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc12f92f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560d75df62a0 con 0x560d75e944a0 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc126ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75e9f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc126ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75e9f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37548/0 (socket says 172.21.15.62:37548) 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc126ffd640 1 -- 172.21.15.62:0/3164125997 learned_addr learned my addr 172.21.15.62:0/3164125997 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc1267fc640 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75ea1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc1277fe640 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e944a0 0x560d75e94880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc126ffd640 1 -- 172.21.15.62:0/3164125997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e944a0 msgr2=0x560d75e94880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc126ffd640 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e944a0 0x560d75e94880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc126ffd640 1 -- 172.21.15.62:0/3164125997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 msgr2=0x560d75ea1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc126ffd640 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75ea1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc126ffd640 1 -- 172.21.15.62:0/3164125997 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d75d86440 con 0x560d75e94dc0 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc10ffff640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1100a32c0 0x7fc1100a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc1267fc640 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75ea1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc1277fe640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1100a32c0 0x7fc1100a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc10ffff640 1 -- 172.21.15.62:0/2389745810 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc11c0f04d0 con 0x7fc128102570 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.394+0000 7fc126ffd640 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75e9f140 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fc11800ac20 tx=0x7fc11800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1e2ca7e594aa2c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:29:44.450 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:29:44.451 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:29:44.451 INFO:teuthology.orchestra.run.smithi062.stdout: "d317d89b-86a6-462d-b572-88f09e1a986b": { 2024-09-15T23:29:44.451 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:29:44.451 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "d317d89b-86a6-462d-b572-88f09e1a986b", 2024-09-15T23:29:44.451 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:29:44.451 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:44.451 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:44.451 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:29:44.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc10d7fa640 1 -- 172.21.15.62:0/3164125997 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11801d070 con 0x560d75e94dc0 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc10d7fa640 1 -- 172.21.15.62:0/3164125997 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc118018b20 con 0x560d75e94dc0 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc10d7fa640 1 -- 172.21.15.62:0/3164125997 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc118010c00 con 0x560d75e94dc0 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc12f92f740 1 -- 172.21.15.62:0/3164125997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 msgr2=0x560d75e9f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc1277fe640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1100a32c0 0x7fc1100a5780 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc120005fd0 tx=0x7fc1200093c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75e9f140 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fc11800ac20 tx=0x7fc11800f790 comp rx=0 tx=0).stop 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc12f92f740 1 -- 172.21.15.62:0/3164125997 shutdown_connections 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75ea1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75e9f140 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3164125997 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e944a0 0x560d75e94880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc12f92f740 1 -- 172.21.15.62:0/3164125997 >> 172.21.15.62:0/3164125997 conn(0x560d75e90a30 msgr2=0x560d75e90e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc12f92f740 1 -- 172.21.15.62:0/3164125997 shutdown_connections 2024-09-15T23:29:44.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.395+0000 7fc12f92f740 1 -- 172.21.15.62:0/3164125997 wait complete. 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc12f92f740 1 -- start start 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc12f92f740 1 Processor -- start 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75e9ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e94dc0 0x560d75e9c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 0x560d75e9c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc12f92f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x560d75ea2250 con 0x560d75e944a0 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc12f92f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560d75db5050 con 0x560d75e94dc0 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc12f92f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560d75ea23d0 con 0x560d75e9f680 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc12f92f740 10 client.0 _ll_register_callbacks cb 0x560d75df5160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc126ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75e9ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc126ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75e9ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37564/0 (socket says 172.21.15.62:37564) 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc126ffd640 1 -- 172.21.15.62:0/545905535 learned_addr learned my addr 172.21.15.62:0/545905535 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc1267fc640 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e94dc0 0x560d75e9c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc126ffd640 1 -- 172.21.15.62:0/545905535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 msgr2=0x560d75e9c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.396+0000 7fc1277fe640 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 0x560d75e9c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc126ffd640 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 0x560d75e9c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc126ffd640 1 -- 172.21.15.62:0/545905535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e94dc0 msgr2=0x560d75e9c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc126ffd640 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e94dc0 0x560d75e9c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc126ffd640 1 -- 172.21.15.62:0/545905535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d75b6b3c0 con 0x560d75e944a0 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc126ffd640 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75e9ef50 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc1180107a0 tx=0x7fc1180186f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a4c18212a5b9f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc10aff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc10aff5640 1 -- 172.21.15.62:0/545905535 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11801d050 con 0x560d75e944a0 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc10aff5640 1 -- 172.21.15.62:0/545905535 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc118010b00 con 0x560d75e944a0 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc12f92f740 1 -- 172.21.15.62:0/545905535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc118018b20 con 0x560d75e944a0 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc10aff5640 1 -- 172.21.15.62:0/545905535 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc118032050 con 0x560d75e944a0 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc12f92f740 10 client.18612 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc12f92f740 1 -- 172.21.15.62:0/545905535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560d75eadbd0 con 0x560d75e944a0 2024-09-15T23:29:44.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc10aff5640 1 -- 172.21.15.62:0/545905535 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc118018b20 con 0x560d75e944a0 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.397+0000 7fc10aff5640 1 -- 172.21.15.62:0/545905535 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fc118004730 con 0x560d75e944a0 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.398+0000 7fc12f92f740 10 client.18612 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.398+0000 7fc12f92f740 10 client.18612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.398+0000 7fc12f92f740 20 client.18612 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.398+0000 7fc12f92f740 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560d75eaf090 0x560d75eb1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.398+0000 7fc12f92f740 4 client.18612 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.398+0000 7fc12f92f740 1 -- 172.21.15.62:0/545905535 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d75e07850 con 0x560d75eaf090 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.398+0000 7fc1267fc640 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560d75eaf090 0x560d75eb1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.398+0000 7fc1267fc640 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560d75eaf090 0x560d75eb1480 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f0b705bb3cfab38 server_cookie=d63b15a9a92dad61 in_seq=0 out_seq=0 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.398+0000 7fc10aff5640 10 client.18612 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc10aff5640 1 -- 172.21.15.62:0/545905535 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560d75e07850 con 0x560d75eaf090 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc10aff5640 10 client.18612 handle_command_reply: tid=0 2024-09-15T23:29:44.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 client.18612 shutdown 2024-09-15T23:29:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 -- 172.21.15.62:0/545905535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 msgr2=0x560d75e9ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75e9ef50 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc1180107a0 tx=0x7fc1180186f0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 -- 172.21.15.62:0/545905535 shutdown_connections 2024-09-15T23:29:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560d75eaf090 0x560d75eb1480 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 0x560d75e9c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e94dc0 0x560d75e9c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 --2- 172.21.15.62:0/545905535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75e9ef50 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 -- 172.21.15.62:0/545905535 >> 172.21.15.62:0/545905535 conn(0x560d75e90a30 msgr2=0x560d75e96710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc10aff5640 0 client.18612 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 -- 172.21.15.62:0/545905535 shutdown_connections 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 1 -- 172.21.15.62:0/545905535 wait complete. 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.399+0000 7fc12f92f740 20 client.18612 trim_cache size 0 max 16384 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc12f92f740 1 Processor -- start 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc12f92f740 1 -- start start 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e944a0 0x560d75eab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75eb6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 0x560d75eb6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc12f92f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc1281b36d0 con 0x560d75e94dc0 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc12f92f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1281b0f70 con 0x560d75e944a0 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc12f92f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc1281aedc0 con 0x560d75e9f680 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc1267fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75eb6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc1267fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75eb6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37574/0 (socket says 172.21.15.62:37574) 2024-09-15T23:29:44.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.417+0000 7fc1267fc640 1 -- 172.21.15.62:0/1945881465 learned_addr learned my addr 172.21.15.62:0/1945881465 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc126ffd640 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e944a0 0x560d75eab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc1277fe640 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 0x560d75eb6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc1267fc640 1 -- 172.21.15.62:0/1945881465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 msgr2=0x560d75eb6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc1267fc640 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 0x560d75eb6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc1267fc640 1 -- 172.21.15.62:0/1945881465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e944a0 msgr2=0x560d75eab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc1267fc640 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e944a0 0x560d75eab4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc1267fc640 1 -- 172.21.15.62:0/1945881465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d75e94bf0 con 0x560d75e94dc0 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc1277fe640 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 0x560d75eb6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc1267fc640 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75eb6ab0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0bc410 tx=0x7fc11c02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23734ffcc143fc26 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc10cff9640 1 -- 172.21.15.62:0/1945881465 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c049040 con 0x560d75e94dc0 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc10cff9640 1 -- 172.21.15.62:0/1945881465 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc11c0bce10 con 0x560d75e94dc0 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.418+0000 7fc10cff9640 1 -- 172.21.15.62:0/1945881465 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0f01a0 con 0x560d75e94dc0 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 -- 172.21.15.62:0/1945881465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 msgr2=0x560d75eb6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75eb6ab0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0bc410 tx=0x7fc11c02a480 comp rx=0 tx=0).stop 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 -- 172.21.15.62:0/1945881465 shutdown_connections 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e9f680 0x560d75eb6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e94dc0 0x560d75eb6ab0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 --2- 172.21.15.62:0/1945881465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e944a0 0x560d75eab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 -- 172.21.15.62:0/1945881465 >> 172.21.15.62:0/1945881465 conn(0x560d75e90a30 msgr2=0x560d75e8fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 -- 172.21.15.62:0/1945881465 shutdown_connections 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 -- 172.21.15.62:0/1945881465 wait complete. 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 -- start start 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.419+0000 7fc12f92f740 1 Processor -- start 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75ea4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e94dc0 0x560d75ea5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc12f92f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75f17a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc12f92f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc1281af430 con 0x560d75e944a0 2024-09-15T23:29:44.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc12f92f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc1281b08a0 con 0x560d75e9f680 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc12f92f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc1281afb00 con 0x560d75e94dc0 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc12f92f740 10 client.0 _ll_register_callbacks cb 0x7fc11801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc126ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75ea4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc126ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75ea4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37578/0 (socket says 172.21.15.62:37578) 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc126ffd640 1 -- 172.21.15.62:0/3299912841 learned_addr learned my addr 172.21.15.62:0/3299912841 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc1277fe640 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75f17a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc126ffd640 1 -- 172.21.15.62:0/3299912841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e94dc0 msgr2=0x560d75ea5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc1267fc640 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e94dc0 0x560d75ea5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc126ffd640 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e94dc0 0x560d75ea5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc126ffd640 1 -- 172.21.15.62:0/3299912841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 msgr2=0x560d75f17a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc126ffd640 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75f17a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc126ffd640 1 -- 172.21.15.62:0/3299912841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d75f180c0 con 0x560d75e944a0 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc1277fe640 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75f17a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:44.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc126ffd640 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75ea4c20 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fc11800f7d0 tx=0x7fc11800a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa45af5400e4d93 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.420+0000 7fc10aff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc10aff5640 1 -- 172.21.15.62:0/3299912841 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11801d040 con 0x560d75e944a0 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc10aff5640 1 -- 172.21.15.62:0/3299912841 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc118004860 con 0x560d75e944a0 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc12f92f740 1 -- 172.21.15.62:0/3299912841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc11c0bce10 con 0x560d75e944a0 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc10aff5640 1 -- 172.21.15.62:0/3299912841 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc118046050 con 0x560d75e944a0 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc12f92f740 10 client.18618 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc12f92f740 1 -- 172.21.15.62:0/3299912841 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560d75f19320 con 0x560d75e944a0 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc10aff5640 1 -- 172.21.15.62:0/3299912841 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc11c0bce10 con 0x560d75e944a0 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc10aff5640 1 -- 172.21.15.62:0/3299912841 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fc118004ca0 con 0x560d75e944a0 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc12f92f740 10 client.18618 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc12f92f740 10 client.18618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc12f92f740 20 client.18618 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:44.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560d75f199e0 0x560d75f1be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc12f92f740 4 client.18618 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc12f92f740 1 -- 172.21.15.62:0/3299912841 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560d75e01d60 con 0x560d75f199e0 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.421+0000 7fc1267fc640 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560d75f199e0 0x560d75f1be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc1267fc640 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560d75f199e0 0x560d75f1be00 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27255cafef3b8654 server_cookie=1b22b5231ab9e440 in_seq=0 out_seq=0 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc10aff5640 10 client.18618 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc10aff5640 1 -- 172.21.15.62:0/3299912841 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560d75e01d60 con 0x560d75f199e0 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc10aff5640 10 client.18618 handle_command_reply: tid=0 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 client.18618 shutdown 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 -- 172.21.15.62:0/3299912841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 msgr2=0x560d75ea4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75ea4c20 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fc11800f7d0 tx=0x7fc11800a440 comp rx=0 tx=0).stop 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 -- 172.21.15.62:0/3299912841 shutdown_connections 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x560d75f199e0 0x560d75f1be00 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560d75e9f680 0x560d75f17a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.462 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560d75e94dc0 0x560d75ea5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc10aff5640 0 client.18618 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 --2- 172.21.15.62:0/3299912841 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560d75e944a0 0x560d75ea4c20 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 -- 172.21.15.62:0/3299912841 >> 172.21.15.62:0/3299912841 conn(0x560d75e90a30 msgr2=0x560d75e9a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 -- 172.21.15.62:0/3299912841 shutdown_connections 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 1 -- 172.21.15.62:0/3299912841 wait complete. 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.422+0000 7fc12f92f740 20 client.18618 trim_cache size 0 max 16384 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 -- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1100a32c0 msgr2=0x7fc1100a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1100a32c0 0x7fc1100a5780 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc120005fd0 tx=0x7fc1200093c0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 -- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 msgr2=0x7fc1280fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 0x7fc1280fd1c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fc11c006010 tx=0x7fc11c003c10 comp rx=0 tx=0).stop 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 -- 172.21.15.62:0/2389745810 shutdown_connections 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc1100a32c0 0x7fc1100a5780 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc128102ef0 0x7fc1280fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc128102570 0x7fc1280fd1c0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 --2- 172.21.15.62:0/2389745810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc128100e20 0x7fc1280fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 -- 172.21.15.62:0/2389745810 >> 172.21.15.62:0/2389745810 conn(0x7fc1280fc7d0 msgr2=0x7fc12806a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 -- 172.21.15.62:0/2389745810 shutdown_connections 2024-09-15T23:29:44.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:44.423+0000 7fc12d556640 1 -- 172.21.15.62:0/2389745810 wait complete. 2024-09-15T23:29:44.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 805.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:29:45.297 INFO:tasks.cephfs.filesystem:scrub status for tag:d317d89b-86a6-462d-b572-88f09e1a986b - {'path': '/', 'tag': 'd317d89b-86a6-462d-b572-88f09e1a986b', 'options': 'recursive,force'} 2024-09-15T23:29:45.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:45 smithi148 ceph-mon[35610]: pgmap v774: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 1.6 MiB/s wr, 216 op/s 2024-09-15T23:29:45.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:45 smithi062 ceph-mon[30949]: pgmap v774: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 1.6 MiB/s wr, 216 op/s 2024-09-15T23:29:46.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:45 smithi067 ceph-mon[35665]: pgmap v774: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 1.6 MiB/s wr, 216 op/s 2024-09-15T23:29:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:47 smithi148 ceph-mon[35610]: pgmap v775: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 986 KiB/s wr, 174 op/s 2024-09-15T23:29:47.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:47 smithi062 ceph-mon[30949]: pgmap v775: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 986 KiB/s wr, 174 op/s 2024-09-15T23:29:48.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:47 smithi067 ceph-mon[35665]: pgmap v775: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 986 KiB/s wr, 174 op/s 2024-09-15T23:29:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.323+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/1898979674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4102d50 msgr2=0x7fbbb41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.323+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/1898979674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4102d50 0x7fbbb41031d0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009900 tx=0x7fbbb002a410 comp rx=0 tx=0).stop 2024-09-15T23:29:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.323+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/1898979674 shutdown_connections 2024-09-15T23:29:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.323+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/1898979674 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbb4103710 0x7fbbb4109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.323+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/1898979674 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4102d50 0x7fbbb41031d0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.323+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/1898979674 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbb4101b60 0x7fbbb4101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.323+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/1898979674 >> 172.21.15.62:0/1898979674 conn(0x7fbbb40fd2e0 msgr2=0x7fbbb40ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.323+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/1898979674 shutdown_connections 2024-09-15T23:29:49.325 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.323+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/1898979674 wait complete. 2024-09-15T23:29:49.327 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbbc4ba640 1 Processor -- start 2024-09-15T23:29:49.327 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbbc4ba640 1 -- start start 2024-09-15T23:29:49.327 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbbc4ba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4101b60 0x7fbbb4197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbbc4ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbb4102d50 0x7fbbb4198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbbc4ba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbb4103710 0x7fbbb419f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbbc4ba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb410d810 con 0x7fbbb4101b60 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbbc4ba640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb410d690 con 0x7fbbb4102d50 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbbc4ba640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb410d990 con 0x7fbbb4103710 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbba256640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4101b60 0x7fbbb4197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbba256640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4101b60 0x7fbbb4197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37606/0 (socket says 172.21.15.62:37606) 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbba256640 1 -- 172.21.15.62:0/614340262 learned_addr learned my addr 172.21.15.62:0/614340262 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.324+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbb4103710 0x7fbbb419f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbb4102d50 0x7fbbb4198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbba256640 1 -- 172.21.15.62:0/614340262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbb4103710 msgr2=0x7fbbb419f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbba256640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbb4103710 0x7fbbb419f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbba256640 1 -- 172.21.15.62:0/614340262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbb4102d50 msgr2=0x7fbbb4198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbba256640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbb4102d50 0x7fbbb4198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbba256640 1 -- 172.21.15.62:0/614340262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb419f9f0 con 0x7fbbb4101b60 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbb4102d50 0x7fbbb4198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbba256640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4101b60 0x7fbbb4197dd0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fbba400e9c0 tx=0x7fbba400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbab7fe640 1 -- 172.21.15.62:0/614340262 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba400cd80 con 0x7fbbb4101b60 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbab7fe640 1 -- 172.21.15.62:0/614340262 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbba40035f0 con 0x7fbbb4101b60 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/614340262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb419fce0 con 0x7fbbb4101b60 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbab7fe640 1 -- 172.21.15.62:0/614340262 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4010690 con 0x7fbbb4101b60 2024-09-15T23:29:49.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.325+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/614340262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb41a0180 con 0x7fbbb4101b60 2024-09-15T23:29:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.327+0000 7fbbab7fe640 1 -- 172.21.15.62:0/614340262 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fbba40037a0 con 0x7fbbb4101b60 2024-09-15T23:29:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.328+0000 7fbbbf257740 1 Processor -- start 2024-09-15T23:29:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.328+0000 7fbbbf257740 1 -- start start 2024-09-15T23:29:49.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 0x5607816bbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 0x5607816be830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbf257740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56078161a2f0 con 0x5607816b69a0 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbf257740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5607815c2270 con 0x5607816b72e0 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbf257740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5607816162b0 con 0x5607816bc370 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbaa57640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbaa57640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37610/0 (socket says 172.21.15.62:37610) 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbaa57640 1 -- 172.21.15.62:0/151521158 learned_addr learned my addr 172.21.15.62:0/151521158 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbab7fe640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba00a32f0 0x7fbba00a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbba256640 1 --2- 172.21.15.62:0/151521158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 0x5607816bbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/151521158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 0x5607816be830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba00a32f0 0x7fbba00a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbaa57640 1 -- 172.21.15.62:0/151521158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 msgr2=0x5607816be830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/151521158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 0x5607816be830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbaa57640 1 -- 172.21.15.62:0/151521158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 msgr2=0x5607816bbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/151521158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 0x5607816bbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbbaa57640 1 -- 172.21.15.62:0/151521158 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607815c1340 con 0x5607816b69a0 2024-09-15T23:29:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.329+0000 7fbbab7fe640 1 -- 172.21.15.62:0/614340262 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbba40d0180 con 0x7fbbb4101b60 2024-09-15T23:29:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.330+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/151521158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b6da0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fbbac005c40 tx=0x7fbbac00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c35998795446d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.330+0000 7fbba8bfe640 1 -- 172.21.15.62:0/151521158 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac022020 con 0x5607816b69a0 2024-09-15T23:29:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.330+0000 7fbba8bfe640 1 -- 172.21.15.62:0/151521158 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbac013170 con 0x5607816b69a0 2024-09-15T23:29:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.330+0000 7fbba8bfe640 1 -- 172.21.15.62:0/151521158 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac010720 con 0x5607816b69a0 2024-09-15T23:29:49.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 -- 172.21.15.62:0/151521158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 msgr2=0x5607816b6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba00a32f0 0x7fbba00a57b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbbac023000 tx=0x7fbbac021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 --2- 172.21.15.62:0/151521158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b6da0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fbbac005c40 tx=0x7fbbac00f3c0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 -- 172.21.15.62:0/151521158 shutdown_connections 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 --2- 172.21.15.62:0/151521158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 0x5607816be830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 --2- 172.21.15.62:0/151521158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 0x5607816bbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 --2- 172.21.15.62:0/151521158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b6da0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 -- 172.21.15.62:0/151521158 >> 172.21.15.62:0/151521158 conn(0x5607816b2010 msgr2=0x5607816b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 -- 172.21.15.62:0/151521158 shutdown_connections 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 -- 172.21.15.62:0/151521158 wait complete. 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 -- start start 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 Processor -- start 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b69a0 0x5607816c0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 0x5607816c1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.331+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816bc370 0x5607816c8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbf257740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5607816bf340 con 0x5607816bc370 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbf257740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5607815a3290 con 0x5607816b69a0 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbf257740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5607816bf4c0 con 0x5607816b72e0 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbf257740 10 client.0 _ll_register_callbacks cb 0x5607815d4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbba256640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b69a0 0x5607816c0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbaa57640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816bc370 0x5607816c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbba256640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b69a0 0x5607816c0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:58056/0 (socket says 172.21.15.62:58056) 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbba256640 1 -- 172.21.15.62:0/3933073413 learned_addr learned my addr 172.21.15.62:0/3933073413 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 0x5607816c1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbaa57640 1 -- 172.21.15.62:0/3933073413 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 msgr2=0x5607816c1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 0x5607816c1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.339 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbaa57640 1 -- 172.21.15.62:0/3933073413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b69a0 msgr2=0x5607816c0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b69a0 0x5607816c0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbaa57640 1 -- 172.21.15.62:0/3933073413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56078132d210 con 0x5607816bc370 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbba256640 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b69a0 0x5607816c0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 0x5607816c1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816bc370 0x5607816c8140 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fbbac005c40 tx=0x7fbbac009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90c3df6c1431fff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbb9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbb9dffb640 1 -- 172.21.15.62:0/3933073413 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac022070 con 0x5607816bc370 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbb9dffb640 1 -- 172.21.15.62:0/3933073413 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbbac0106a0 con 0x5607816bc370 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.332+0000 7fbbbf257740 1 -- 172.21.15.62:0/3933073413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbac013170 con 0x5607816bc370 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbb9dffb640 1 -- 172.21.15.62:0/3933073413 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbbac010ab0 con 0x5607816bc370 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbbbf257740 10 client.18630 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbbbf257740 1 -- 172.21.15.62:0/3933073413 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5607813f9620 con 0x5607816bc370 2024-09-15T23:29:49.340 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbb9dffb640 1 -- 172.21.15.62:0/3933073413 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbac013170 con 0x5607816bc370 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbb9dffb640 1 -- 172.21.15.62:0/3933073413 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fbbac005050 con 0x5607816bc370 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbbbf257740 10 client.18630 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbbbf257740 10 client.18630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbbbf257740 20 client.18630 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbbbf257740 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607816cc6a0 0x5607816cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbbbf257740 4 client.18630 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbbbf257740 1 -- 172.21.15.62:0/3933073413 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560781627a20 con 0x5607816cc6a0 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.333+0000 7fbbba256640 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607816cc6a0 0x5607816cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.334+0000 7fbbba256640 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607816cc6a0 0x5607816cea90 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8be7d51413366b1 server_cookie=635cb39fa8b838c5 in_seq=0 out_seq=0 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.334+0000 7fbb9dffb640 10 client.18630 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.334+0000 7fbb9dffb640 1 -- 172.21.15.62:0/3933073413 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560781627a20 con 0x5607816cc6a0 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.334+0000 7fbb9dffb640 10 client.18630 handle_command_reply: tid=0 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.334+0000 7fbbbf257740 1 client.18630 shutdown 2024-09-15T23:29:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 -- 172.21.15.62:0/3933073413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816bc370 msgr2=0x5607816c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816bc370 0x5607816c8140 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fbbac005c40 tx=0x7fbbac009200 comp rx=0 tx=0).stop 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 -- 172.21.15.62:0/3933073413 shutdown_connections 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5607816cc6a0 0x5607816cea90 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816bc370 0x5607816c8140 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 0x5607816c1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 --2- 172.21.15.62:0/3933073413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b69a0 0x5607816c0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbb9dffb640 0 client.18630 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 -- 172.21.15.62:0/3933073413 >> 172.21.15.62:0/3933073413 conn(0x5607816b2010 msgr2=0x5607816b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 -- 172.21.15.62:0/3933073413 shutdown_connections 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 1 -- 172.21.15.62:0/3933073413 wait complete. 2024-09-15T23:29:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.335+0000 7fbbbf257740 20 client.18630 trim_cache size 0 max 16384 2024-09-15T23:29:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbbf257740 1 Processor -- start 2024-09-15T23:29:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbbf257740 1 -- start start 2024-09-15T23:29:49.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 0x5607816b3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 0x5607816b3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbbf257740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb41acfb0 con 0x5607816b69a0 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbbf257740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb41aa850 con 0x5607816b72e0 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbbf257740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb41a86a0 con 0x5607816bc370 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbba256640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbba256640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37642/0 (socket says 172.21.15.62:37642) 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbba256640 1 -- 172.21.15.62:0/1119104815 learned_addr learned my addr 172.21.15.62:0/1119104815 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 0x5607816b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 0x5607816b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbba256640 1 -- 172.21.15.62:0/1119104815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 msgr2=0x5607816b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbba256640 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 0x5607816b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbba256640 1 -- 172.21.15.62:0/1119104815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 msgr2=0x5607816b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbba256640 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 0x5607816b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbba256640 1 -- 172.21.15.62:0/1119104815 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607816b1e30 con 0x5607816b69a0 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 0x5607816b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.350+0000 7fbbba256640 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b10b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fbba409c350 tx=0x7fbba40707c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7d012b53bc4a02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbb9ffff640 1 -- 172.21.15.62:0/1119104815 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba409c3c0 con 0x5607816b69a0 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbb9ffff640 1 -- 172.21.15.62:0/1119104815 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbba40a0770 con 0x5607816b69a0 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbb9ffff640 1 -- 172.21.15.62:0/1119104815 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba409b810 con 0x5607816b69a0 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 -- 172.21.15.62:0/1119104815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 msgr2=0x5607816b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b10b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fbba409c350 tx=0x7fbba40707c0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 -- 172.21.15.62:0/1119104815 shutdown_connections 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816bc370 0x5607816b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.357 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816b72e0 0x5607816b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1119104815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x5607816b10b0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 -- 172.21.15.62:0/1119104815 >> 172.21.15.62:0/1119104815 conn(0x5607816b2010 msgr2=0x5607816b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 -- 172.21.15.62:0/1119104815 shutdown_connections 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 -- 172.21.15.62:0/1119104815 wait complete. 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.351+0000 7fbbbf257740 1 -- start start 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbbf257740 1 Processor -- start 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x560781732890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 0x560781732dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbbf257740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816bc370 0x560781739e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbbf257740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb41a8d10 con 0x5607816b69a0 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbbf257740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb41aa180 con 0x5607816bc370 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbbf257740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb41a93e0 con 0x5607816b72e0 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbbf257740 10 client.0 _ll_register_callbacks cb 0x7fbba4098490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:49.358 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbba256640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x560781732890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbba256640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x560781732890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37644/0 (socket says 172.21.15.62:37644) 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbba256640 1 -- 172.21.15.62:0/1862515966 learned_addr learned my addr 172.21.15.62:0/1862515966 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbbaa57640 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816bc370 0x560781739e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 0x560781732dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbba256640 1 -- 172.21.15.62:0/1862515966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 msgr2=0x560781732dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbba256640 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 0x560781732dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbba256640 1 -- 172.21.15.62:0/1862515966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816bc370 msgr2=0x560781739e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbba256640 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816bc370 0x560781739e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbba256640 1 -- 172.21.15.62:0/1862515966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56078173a420 con 0x5607816b69a0 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbbba256640 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x560781732890 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fbba40bc010 tx=0x7fbba4003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e713d286f079af30 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.352+0000 7fbb9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbb9dffb640 1 -- 172.21.15.62:0/1862515966 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba40a5030 con 0x5607816b69a0 2024-09-15T23:29:49.359 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbb9dffb640 1 -- 172.21.15.62:0/1862515966 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbba4098110 con 0x5607816b69a0 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbbf257740 1 -- 172.21.15.62:0/1862515966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbba40a0770 con 0x5607816b69a0 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbb9dffb640 1 -- 172.21.15.62:0/1862515966 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbba40d2110 con 0x5607816b69a0 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbbf257740 10 client.18636 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbbf257740 1 -- 172.21.15.62:0/1862515966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56078173b7a0 con 0x5607816b69a0 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbb9dffb640 1 -- 172.21.15.62:0/1862515966 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbba40a0770 con 0x5607816b69a0 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbb9dffb640 1 -- 172.21.15.62:0/1862515966 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fbba409b810 con 0x5607816b69a0 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbbf257740 10 client.18636 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbbf257740 10 client.18636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbbf257740 20 client.18636 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56078173c0a0 0x56078173e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbbf257740 4 client.18636 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbbf257740 1 -- 172.21.15.62:0/1862515966 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560781621d70 con 0x56078173c0a0 2024-09-15T23:29:49.360 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.353+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56078173c0a0 0x56078173e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.354+0000 7fbbb9a55640 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56078173c0a0 0x56078173e4c0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84892c4c8154471b server_cookie=b62737bc38de5a06 in_seq=0 out_seq=0 2024-09-15T23:29:49.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.354+0000 7fbb9dffb640 10 client.18636 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbb9dffb640 1 -- 172.21.15.62:0/1862515966 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560781621d70 con 0x56078173c0a0 2024-09-15T23:29:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbb9dffb640 10 client.18636 handle_command_reply: tid=0 2024-09-15T23:29:49.366 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 client.18636 shutdown 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 -- 172.21.15.62:0/1862515966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 msgr2=0x560781732890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x560781732890 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fbba40bc010 tx=0x7fbba4003d80 comp rx=0 tx=0).stop 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 -- 172.21.15.62:0/1862515966 shutdown_connections 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56078173c0a0 0x56078173e4c0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607816bc370 0x560781739e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5607816b72e0 0x560781732dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 --2- 172.21.15.62:0/1862515966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5607816b69a0 0x560781732890 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbb9dffb640 0 client.18636 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 -- 172.21.15.62:0/1862515966 >> 172.21.15.62:0/1862515966 conn(0x5607816b2010 msgr2=0x5607816b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 -- 172.21.15.62:0/1862515966 shutdown_connections 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 1 -- 172.21.15.62:0/1862515966 wait complete. 2024-09-15T23:29:49.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.365+0000 7fbbbf257740 20 client.18636 trim_cache size 0 max 16384 2024-09-15T23:29:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:29:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:29:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:29:49.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba00a32f0 msgr2=0x7fbba00a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba00a32f0 0x7fbba00a57b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbbac023000 tx=0x7fbbac021000 comp rx=0 tx=0).stop 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4101b60 msgr2=0x7fbbb4197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4101b60 0x7fbbb4197dd0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fbba400e9c0 tx=0x7fbba400ee80 comp rx=0 tx=0).stop 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/614340262 shutdown_connections 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbba00a32f0 0x7fbba00a57b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbbb4103710 0x7fbbb419f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbb4102d50 0x7fbbb4198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 --2- 172.21.15.62:0/614340262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbb4101b60 0x7fbbb4197dd0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:49.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/614340262 >> 172.21.15.62:0/614340262 conn(0x7fbbb40fd2e0 msgr2=0x7fbbb4106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/614340262 shutdown_connections 2024-09-15T23:29:49.370 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:29:49.366+0000 7fbbbc4ba640 1 -- 172.21.15.62:0/614340262 wait complete. 2024-09-15T23:29:49.571 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 810.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:29:49.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:49 smithi062 ceph-mon[30949]: pgmap v776: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 434 KiB/s wr, 147 op/s 2024-09-15T23:29:50.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:49 smithi067 ceph-mon[35665]: pgmap v776: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 434 KiB/s wr, 147 op/s 2024-09-15T23:29:50.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:49 smithi148 ceph-mon[35610]: pgmap v776: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 434 KiB/s wr, 147 op/s 2024-09-15T23:29:51.826 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:51 smithi067 ceph-mon[35665]: pgmap v777: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 39 KiB/s wr, 102 op/s 2024-09-15T23:29:51.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:51 smithi062 ceph-mon[30949]: pgmap v777: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 39 KiB/s wr, 102 op/s 2024-09-15T23:29:52.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:51 smithi148 ceph-mon[35610]: pgmap v777: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 39 KiB/s wr, 102 op/s 2024-09-15T23:29:52.389 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:29:52.665 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:53.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 -- 172.21.15.62:0/2276034481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9181082e0 msgr2=0x7fc91810ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 --2- 172.21.15.62:0/2276034481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9181082e0 0x7fc91810ec50 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc910009930 tx=0x7fc91002a420 comp rx=0 tx=0).stop 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 -- 172.21.15.62:0/2276034481 shutdown_connections 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 --2- 172.21.15.62:0/2276034481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9181082e0 0x7fc91810ec50 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 --2- 172.21.15.62:0/2276034481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc918103760 0x7fc918107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 --2- 172.21.15.62:0/2276034481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc918102db0 0x7fc918103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 -- 172.21.15.62:0/2276034481 >> 172.21.15.62:0/2276034481 conn(0x7fc9180fc7d0 msgr2=0x7fc9180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 -- 172.21.15.62:0/2276034481 shutdown_connections 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 -- 172.21.15.62:0/2276034481 wait complete. 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.162+0000 7fc91dd70640 1 Processor -- start 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc91dd70640 1 -- start start 2024-09-15T23:29:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc91dd70640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc918102db0 0x7fc918100a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc91dd70640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc918103760 0x7fc918100f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc91dd70640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9181082e0 0x7fc918101490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc91dd70640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc9181120b0 con 0x7fc918102db0 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc91dd70640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc918111f30 con 0x7fc918103760 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc91dd70640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc918112230 con 0x7fc9181082e0 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc9177fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc918102db0 0x7fc918100a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc9177fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc918102db0 0x7fc918100a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57262/0 (socket says 172.21.15.62:57262) 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc9177fe640 1 -- 172.21.15.62:0/3789311613 learned_addr learned my addr 172.21.15.62:0/3789311613 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc916ffd640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc918103760 0x7fc918100f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc917fff640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9181082e0 0x7fc918101490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc9177fe640 1 -- 172.21.15.62:0/3789311613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9181082e0 msgr2=0x7fc918101490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc9177fe640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9181082e0 0x7fc918101490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc9177fe640 1 -- 172.21.15.62:0/3789311613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc918103760 msgr2=0x7fc918100f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc9177fe640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc918103760 0x7fc918100f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc9177fe640 1 -- 172.21.15.62:0/3789311613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9181a6a20 con 0x7fc918102db0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.163+0000 7fc917fff640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9181082e0 0x7fc918101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.164+0000 7fc9177fe640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc918102db0 0x7fc918100a10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc90800eab0 tx=0x7fc90800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.164+0000 7fc914ff9640 1 -- 172.21.15.62:0/3789311613 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90800ce40 con 0x7fc918102db0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.164+0000 7fc914ff9640 1 -- 172.21.15.62:0/3789311613 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc908011950 con 0x7fc918102db0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.164+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9181a6d10 con 0x7fc918102db0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.164+0000 7fc914ff9640 1 -- 172.21.15.62:0/3789311613 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908010ae0 con 0x7fc918102db0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.164+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9181a71f0 con 0x7fc918102db0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.165+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc918103190 con 0x7fc918102db0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.166+0000 7fc914ff9640 1 -- 172.21.15.62:0/3789311613 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc908011b00 con 0x7fc918102db0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.167+0000 7fc914ff9640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9040a3240 0x7fc9040a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.167+0000 7fc916ffd640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9040a3240 0x7fc9040a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:53.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.167+0000 7fc914ff9640 1 -- 172.21.15.62:0/3789311613 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc9080d0360 con 0x7fc918102db0 2024-09-15T23:29:53.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.167+0000 7fc916ffd640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9040a3240 0x7fc9040a5700 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc9181024a0 tx=0x7fc90000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:53.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.171+0000 7fc914ff9640 1 -- 172.21.15.62:0/3789311613 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fc908016030 con 0x7fc918102db0 2024-09-15T23:29:53.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.384+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc918101f40 con 0x7fc918102db0 2024-09-15T23:29:53.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.384+0000 7fc914ff9640 1 -- 172.21.15.62:0/3789311613 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7fc90809cd00 con 0x7fc918102db0 2024-09-15T23:29:53.387 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:29:53.387 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:29:53.388 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:29:53.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9040a3240 msgr2=0x7fc9040a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:53.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9040a3240 0x7fc9040a5700 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc9181024a0 tx=0x7fc90000b390 comp rx=0 tx=0).stop 2024-09-15T23:29:53.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc918102db0 msgr2=0x7fc918100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:53.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc918102db0 0x7fc918100a10 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fc90800eab0 tx=0x7fc90800ef70 comp rx=0 tx=0).stop 2024-09-15T23:29:53.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 shutdown_connections 2024-09-15T23:29:53.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc9040a3240 0x7fc9040a5700 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc9181082e0 0x7fc918101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc918103760 0x7fc918100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 --2- 172.21.15.62:0/3789311613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc918102db0 0x7fc918100a10 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:53.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 >> 172.21.15.62:0/3789311613 conn(0x7fc9180fc7d0 msgr2=0x7fc918105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:53.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 shutdown_connections 2024-09-15T23:29:53.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:53.387+0000 7fc91dd70640 1 -- 172.21.15.62:0/3789311613 wait complete. 2024-09-15T23:29:53.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:53 smithi062 ceph-mon[30949]: pgmap v778: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 3.0 KiB/s wr, 97 op/s 2024-09-15T23:29:53.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:53 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3789311613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:53 smithi067 ceph-mon[35665]: pgmap v778: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 3.0 KiB/s wr, 97 op/s 2024-09-15T23:29:54.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:53 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3789311613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:53 smithi148 ceph-mon[35610]: pgmap v778: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 3.0 KiB/s wr, 97 op/s 2024-09-15T23:29:54.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:53 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3789311613' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:29:54.182 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 5d785437 --release --await' 2024-09-15T23:29:54.182 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 5d785437 --release --await 2024-09-15T23:29:54.447 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:29:54.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 815.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:29:54.821 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:54 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:29:54.821 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:54 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:29:54.821 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:54 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:29:54.821 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:54 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:29:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.909+0000 7f78e246d640 1 -- 172.21.15.62:0/3777325326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc106a80 msgr2=0x7f78dc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.909+0000 7f78e246d640 1 --2- 172.21.15.62:0/3777325326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc106a80 0x7f78dc106e60 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009930 tx=0x7f78cc02a420 comp rx=0 tx=0).stop 2024-09-15T23:29:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.909+0000 7f78e246d640 1 -- 172.21.15.62:0/3777325326 shutdown_connections 2024-09-15T23:29:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.909+0000 7f78e246d640 1 --2- 172.21.15.62:0/3777325326 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78dc101420 0x7f78dc10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.909+0000 7f78e246d640 1 --2- 172.21.15.62:0/3777325326 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78dc100a80 0x7f78dc100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.909+0000 7f78e246d640 1 --2- 172.21.15.62:0/3777325326 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc106a80 0x7f78dc106e60 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 -- 172.21.15.62:0/3777325326 >> 172.21.15.62:0/3777325326 conn(0x7f78dc0fc7a0 msgr2=0x7f78dc0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 -- 172.21.15.62:0/3777325326 shutdown_connections 2024-09-15T23:29:54.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 -- 172.21.15.62:0/3777325326 wait complete. 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 Processor -- start 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 -- start start 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc100a80 0x7f78dc199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78dc101420 0x7f78dc1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78dc106a80 0x7f78dc1a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc110e80 con 0x7f78dc100a80 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc110d00 con 0x7f78dc101420 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78e246d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc111000 con 0x7f78dc106a80 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.910+0000 7f78dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc100a80 0x7f78dc199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc100a80 0x7f78dc199270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57272/0 (socket says 172.21.15.62:57272) 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78dbfff640 1 -- 172.21.15.62:0/1928448403 learned_addr learned my addr 172.21.15.62:0/1928448403 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78d3fff640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78dc101420 0x7f78dc1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78dc106a80 0x7f78dc1a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78dbfff640 1 -- 172.21.15.62:0/1928448403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78dc106a80 msgr2=0x7f78dc1a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78dbfff640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78dc106a80 0x7f78dc1a42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78dbfff640 1 -- 172.21.15.62:0/1928448403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78dc101420 msgr2=0x7f78dc1997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78dbfff640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78dc101420 0x7f78dc1997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78dbfff640 1 -- 172.21.15.62:0/1928448403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78dc1a4830 con 0x7f78dc100a80 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78dbfff640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc100a80 0x7f78dc199270 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009230 tx=0x7f78cc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78d9ffb640 1 -- 172.21.15.62:0/1928448403 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc03f030 con 0x7f78dc100a80 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78d9ffb640 1 -- 172.21.15.62:0/1928448403 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78cc003ad0 con 0x7f78dc100a80 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78d9ffb640 1 -- 172.21.15.62:0/1928448403 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc035070 con 0x7f78dc100a80 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78e246d640 1 -- 172.21.15.62:0/1928448403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78dc1a4a00 con 0x7f78dc100a80 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.911+0000 7f78e246d640 1 -- 172.21.15.62:0/1928448403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78dc1a4eb0 con 0x7f78dc100a80 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78d9ffb640 1 -- 172.21.15.62:0/1928448403 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f78cc042030 con 0x7f78dc100a80 2024-09-15T23:29:54.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e4846740 1 Processor -- start 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e4846740 1 -- start start 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e4846740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2645440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e4846740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 0x5614a264fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e4846740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 0x5614a2652630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e4846740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5614a25aa620 con 0x5614a2645060 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e4846740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5614a2551e50 con 0x5614a2645980 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e4846740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5614a25a65e0 con 0x5614a2650240 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e0a0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2645440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e0a0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2645440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57280/0 (socket says 172.21.15.62:57280) 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78e0a0a640 1 -- 172.21.15.62:0/3242330769 learned_addr learned my addr 172.21.15.62:0/3242330769 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.914+0000 7f78d9ffb640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78c80a32c0 0x7f78c80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78d3fff640 1 --2- 172.21.15.62:0/3242330769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 0x5614a2652630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78c80a32c0 0x7f78c80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78d9ffb640 1 -- 172.21.15.62:0/1928448403 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78cc0efd40 con 0x7f78dc100a80 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78dbfff640 1 --2- 172.21.15.62:0/3242330769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 0x5614a264fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78e0a0a640 1 -- 172.21.15.62:0/3242330769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 msgr2=0x5614a2652630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/3242330769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 0x5614a2652630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78e0a0a640 1 -- 172.21.15.62:0/3242330769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 msgr2=0x5614a264fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/3242330769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 0x5614a264fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78e0a0a640 1 -- 172.21.15.62:0/3242330769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614a2542d40 con 0x5614a2645060 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/3242330769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2645440 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f78d40139d0 tx=0x7f78d4013e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fed80302340f675 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78d27fc640 1 -- 172.21.15.62:0/3242330769 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d4013050 con 0x5614a2645060 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78d27fc640 1 -- 172.21.15.62:0/3242330769 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78d4022070 con 0x5614a2645060 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.915+0000 7f78d27fc640 1 -- 172.21.15.62:0/3242330769 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d4012740 con 0x5614a2645060 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 -- 172.21.15.62:0/3242330769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 msgr2=0x5614a2645440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78c80a32c0 0x7f78c80a5780 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f78d4021000 tx=0x7f78d401f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 --2- 172.21.15.62:0/3242330769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2645440 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f78d40139d0 tx=0x7f78d4013e90 comp rx=0 tx=0).stop 2024-09-15T23:29:54.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 -- 172.21.15.62:0/3242330769 shutdown_connections 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 --2- 172.21.15.62:0/3242330769 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 0x5614a2652630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 --2- 172.21.15.62:0/3242330769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 0x5614a264fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 --2- 172.21.15.62:0/3242330769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2645440 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 -- 172.21.15.62:0/3242330769 >> 172.21.15.62:0/3242330769 conn(0x5614a26415f0 msgr2=0x5614a26419d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 -- 172.21.15.62:0/3242330769 shutdown_connections 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 -- 172.21.15.62:0/3242330769 wait complete. 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.916+0000 7f78e4846740 1 -- start start 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e4846740 1 Processor -- start 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e4846740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645060 0x5614a264d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e4846740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645980 0x5614a264d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e4846740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2650240 0x5614a264dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e4846740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5614a2652e10 con 0x5614a2650240 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e4846740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5614a2532f10 con 0x5614a2645060 2024-09-15T23:29:54.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e4846740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5614a2652f90 con 0x5614a2645980 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e4846740 10 client.0 _ll_register_callbacks cb 0x5614a2556db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e0a0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2650240 0x5614a264dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e0a0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2650240 0x5614a264dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57284/0 (socket says 172.21.15.62:57284) 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e0a0a640 1 -- 172.21.15.62:0/2279460073 learned_addr learned my addr 172.21.15.62:0/2279460073 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78dbfff640 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645060 0x5614a264d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78d3fff640 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645980 0x5614a264d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e0a0a640 1 -- 172.21.15.62:0/2279460073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645980 msgr2=0x5614a264d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645980 0x5614a264d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e0a0a640 1 -- 172.21.15.62:0/2279460073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645060 msgr2=0x5614a264d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645060 0x5614a264d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78e0a0a640 1 -- 172.21.15.62:0/2279460073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614a22b9580 con 0x5614a2650240 2024-09-15T23:29:54.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.917+0000 7f78dbfff640 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645060 0x5614a264d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2650240 0x5614a264dc00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f78d4013480 tx=0x7f78d4007520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b62f6d6ceed25f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78bb7fe640 1 -- 172.21.15.62:0/2279460073 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d4036060 con 0x5614a2650240 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78bb7fe640 1 -- 172.21.15.62:0/2279460073 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78d40129a0 con 0x5614a2650240 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78e4846740 1 -- 172.21.15.62:0/2279460073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78d4022070 con 0x5614a2650240 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78bb7fe640 1 -- 172.21.15.62:0/2279460073 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f78d403d050 con 0x5614a2650240 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78e4846740 10 client.18654 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78e4846740 1 -- 172.21.15.62:0/2279460073 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5614a2383960 con 0x5614a2650240 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78bb7fe640 1 -- 172.21.15.62:0/2279460073 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78d4022070 con 0x5614a2650240 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78bb7fe640 1 -- 172.21.15.62:0/2279460073 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f78d4012c30 con 0x5614a2650240 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78e4846740 10 client.18654 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78e4846740 10 client.18654 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.918+0000 7f78e4846740 20 client.18654 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:54.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.919+0000 7f78e4846740 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614a265fc60 0x5614a2662050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.919+0000 7f78e4846740 4 client.18654 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.919+0000 7f78e4846740 1 -- 172.21.15.62:0/2279460073 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614a25b7b90 con 0x5614a265fc60 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.919+0000 7f78dbfff640 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614a265fc60 0x5614a2662050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.919+0000 7f78bb7fe640 10 client.18654 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.919+0000 7f78dbfff640 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614a265fc60 0x5614a2662050 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3c55b4b54c679db server_cookie=54f1e406559c9b1c in_seq=0 out_seq=0 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78bb7fe640 1 -- 172.21.15.62:0/2279460073 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5614a25b7b90 con 0x5614a265fc60 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78bb7fe640 10 client.18654 handle_command_reply: tid=0 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 client.18654 shutdown 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 -- 172.21.15.62:0/2279460073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2650240 msgr2=0x5614a264dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2650240 0x5614a264dc00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f78d4013480 tx=0x7f78d4007520 comp rx=0 tx=0).stop 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 -- 172.21.15.62:0/2279460073 shutdown_connections 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614a265fc60 0x5614a2662050 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2650240 0x5614a264dc00 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645980 0x5614a264d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 --2- 172.21.15.62:0/2279460073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645060 0x5614a264d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 -- 172.21.15.62:0/2279460073 >> 172.21.15.62:0/2279460073 conn(0x5614a26415f0 msgr2=0x5614a2640750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:54.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78bb7fe640 0 client.18654 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:54.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 -- 172.21.15.62:0/2279460073 shutdown_connections 2024-09-15T23:29:54.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 1 -- 172.21.15.62:0/2279460073 wait complete. 2024-09-15T23:29:54.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.920+0000 7f78e4846740 20 client.18654 trim_cache size 0 max 16384 2024-09-15T23:29:54.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78e4846740 1 Processor -- start 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78e4846740 1 -- start start 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78e4846740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645060 0x5614a264c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78e4846740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645980 0x5614a2667700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78e4846740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2650240 0x5614a2667c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78e4846740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc1b14a0 con 0x5614a2645980 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78e4846740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc1aed40 con 0x5614a2650240 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78e4846740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc1acb90 con 0x5614a2645060 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645980 0x5614a2667700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78d3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645980 0x5614a2667700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57288/0 (socket says 172.21.15.62:57288) 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.937+0000 7f78d3fff640 1 -- 172.21.15.62:0/3389202539 learned_addr learned my addr 172.21.15.62:0/3389202539 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2650240 0x5614a2667c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78d3fff640 1 -- 172.21.15.62:0/3389202539 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645060 msgr2=0x5614a264c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:54.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78dbfff640 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645060 0x5614a264c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78d3fff640 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645060 0x5614a264c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78d3fff640 1 -- 172.21.15.62:0/3389202539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2650240 msgr2=0x5614a2667c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:54.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78d3fff640 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2650240 0x5614a2667c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78d3fff640 1 -- 172.21.15.62:0/3389202539 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614a264faf0 con 0x5614a2645980 2024-09-15T23:29:54.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2650240 0x5614a2667c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:29:54.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78d3fff640 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645980 0x5614a2667700 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f78dc06b2e0 tx=0x7f78c400bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1bc28fd096109d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:54.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78d1ffb640 1 -- 172.21.15.62:0/3389202539 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c400bd30 con 0x5614a2645980 2024-09-15T23:29:54.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78d1ffb640 1 -- 172.21.15.62:0/3389202539 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78c4011440 con 0x5614a2645980 2024-09-15T23:29:54.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.938+0000 7f78d1ffb640 1 -- 172.21.15.62:0/3389202539 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4010690 con 0x5614a2645980 2024-09-15T23:29:54.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 -- 172.21.15.62:0/3389202539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645980 msgr2=0x5614a2667700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645980 0x5614a2667700 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f78dc06b2e0 tx=0x7f78c400bb40 comp rx=0 tx=0).stop 2024-09-15T23:29:54.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 -- 172.21.15.62:0/3389202539 shutdown_connections 2024-09-15T23:29:54.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2650240 0x5614a2667c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645980 0x5614a2667700 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 --2- 172.21.15.62:0/3389202539 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2645060 0x5614a264c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 -- 172.21.15.62:0/3389202539 >> 172.21.15.62:0/3389202539 conn(0x5614a26415f0 msgr2=0x5614a264f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 -- 172.21.15.62:0/3389202539 shutdown_connections 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 -- 172.21.15.62:0/3389202539 wait complete. 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 -- start start 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.939+0000 7f78e4846740 1 Processor -- start 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78e4846740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2653ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78e4846740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 0x5614a2654400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78e4846740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 0x5614a2654940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78e4846740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc1ad200 con 0x5614a2645060 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78e4846740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc1ae670 con 0x5614a2645980 2024-09-15T23:29:54.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78e4846740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f78dc1ad8d0 con 0x5614a2650240 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78e4846740 10 client.0 _ll_register_callbacks cb 0x7f78d401dbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2653ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78dbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2653ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57294/0 (socket says 172.21.15.62:57294) 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78dbfff640 1 -- 172.21.15.62:0/951117033 learned_addr learned my addr 172.21.15.62:0/951117033 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78d3fff640 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 0x5614a2654400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 0x5614a2654940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78dbfff640 1 -- 172.21.15.62:0/951117033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 msgr2=0x5614a2654940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78dbfff640 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 0x5614a2654940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78dbfff640 1 -- 172.21.15.62:0/951117033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 msgr2=0x5614a2654400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78dbfff640 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 0x5614a2654400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78dbfff640 1 -- 172.21.15.62:0/951117033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614a26cdae0 con 0x5614a2645060 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78e0a0a640 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 0x5614a2654940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:29:54.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78dbfff640 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2653ec0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009900 tx=0x7f78cc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54df0497a7d237f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.940+0000 7f78bb7fe640 1 -- 172.21.15.62:0/951117033 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc049040 con 0x5614a2645060 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78bb7fe640 1 -- 172.21.15.62:0/951117033 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f78cc03b460 con 0x5614a2645060 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78e4846740 1 -- 172.21.15.62:0/951117033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78c4011440 con 0x5614a2645060 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78bb7fe640 1 -- 172.21.15.62:0/951117033 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f78cc0c4c50 con 0x5614a2645060 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78e4846740 10 client.18660 fetch_fsmap learned FSMap version 36 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78e4846740 1 -- 172.21.15.62:0/951117033 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5614a26ced50 con 0x5614a2645060 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78bb7fe640 1 -- 172.21.15.62:0/951117033 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78c4011440 con 0x5614a2645060 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78bb7fe640 1 -- 172.21.15.62:0/951117033 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f78cc031c60 con 0x5614a2645060 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78e4846740 10 client.18660 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78e4846740 10 client.18660 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78e4846740 20 client.18660 populate_metadata read hostname 'smithi062' 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78e4846740 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614a26cf4a0 0x5614a26d18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:29:54.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78e4846740 4 client.18660 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "5d785437", "release": true, "await": true}] 2024-09-15T23:29:54.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78e4846740 1 -- 172.21.15.62:0/951117033 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "5d785437", "release": true, "await": true}) -- 0x5614a25b20a0 con 0x5614a26cf4a0 2024-09-15T23:29:54.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.941+0000 7f78d3fff640 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614a26cf4a0 0x5614a26d18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:29:54.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.942+0000 7f78d3fff640 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614a26cf4a0 0x5614a26d18c0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83686a8feab88e50 server_cookie=61b9df545d0436c4 in_seq=0 out_seq=0 2024-09-15T23:29:54.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.942+0000 7f78bb7fe640 10 client.18660 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:54.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.968+0000 7f78bb7fe640 1 -- 172.21.15.62:0/951117033 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5614a25b20a0 con 0x5614a26cf4a0 2024-09-15T23:29:54.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.968+0000 7f78bb7fe640 10 client.18660 handle_command_reply: tid=0 2024-09-15T23:29:54.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.968+0000 7f78e4846740 1 client.18660 shutdown 2024-09-15T23:29:54.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.968+0000 7f78e4846740 1 -- 172.21.15.62:0/951117033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 msgr2=0x5614a2653ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.968+0000 7f78e4846740 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2653ec0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009900 tx=0x7f78cc009260 comp rx=0 tx=0).stop 2024-09-15T23:29:54.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78e4846740 1 -- 172.21.15.62:0/951117033 shutdown_connections 2024-09-15T23:29:54.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78e4846740 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5614a26cf4a0 0x5614a26d18c0 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78e4846740 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5614a2650240 0x5614a2654940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78e4846740 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5614a2645980 0x5614a2654400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78e4846740 1 --2- 172.21.15.62:0/951117033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5614a2645060 0x5614a2653ec0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78e4846740 1 -- 172.21.15.62:0/951117033 >> 172.21.15.62:0/951117033 conn(0x5614a26415f0 msgr2=0x5614a2646ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:54.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78bb7fe640 0 client.18660 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:29:54.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78e4846740 1 -- 172.21.15.62:0/951117033 shutdown_connections 2024-09-15T23:29:54.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78e4846740 1 -- 172.21.15.62:0/951117033 wait complete. 2024-09-15T23:29:54.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.969+0000 7f78e4846740 20 client.18660 trim_cache size 0 max 16384 2024-09-15T23:29:54.972 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:29:54.972 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 36, 2024-09-15T23:29:54.972 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:29:54.972 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 52, 2024-09-15T23:29:54.972 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:29:54.972 INFO:teuthology.orchestra.run.smithi062.stdout: "5d785437": { 2024-09-15T23:29:54.972 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 52, 2024-09-15T23:29:54.972 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 146.5, 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:29:54.973 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:54.974 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:54.974 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:54.974 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:54.974 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:29:54.974 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:29:54.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.980+0000 7f78e246d640 1 -- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78c80a32c0 msgr2=0x7f78c80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.980+0000 7f78e246d640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78c80a32c0 0x7f78c80a5780 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f78d4021000 tx=0x7f78d401f000 comp rx=0 tx=0).stop 2024-09-15T23:29:54.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.980+0000 7f78e246d640 1 -- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc100a80 msgr2=0x7f78dc199270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:29:54.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.980+0000 7f78e246d640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc100a80 0x7f78dc199270 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f78cc009230 tx=0x7f78cc004270 comp rx=0 tx=0).stop 2024-09-15T23:29:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.981+0000 7f78e246d640 1 -- 172.21.15.62:0/1928448403 shutdown_connections 2024-09-15T23:29:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.981+0000 7f78e246d640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78c80a32c0 0x7f78c80a5780 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.981+0000 7f78e246d640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f78dc106a80 0x7f78dc1a42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.981+0000 7f78e246d640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78dc101420 0x7f78dc1997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.981+0000 7f78e246d640 1 --2- 172.21.15.62:0/1928448403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f78dc100a80 0x7f78dc199270 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:29:54.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.981+0000 7f78e246d640 1 -- 172.21.15.62:0/1928448403 >> 172.21.15.62:0/1928448403 conn(0x7f78dc0fc7a0 msgr2=0x7f78dc0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:29:54.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.985+0000 7f78e246d640 1 -- 172.21.15.62:0/1928448403 shutdown_connections 2024-09-15T23:29:54.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:29:54.986+0000 7f78e246d640 1 -- 172.21.15.62:0/1928448403 wait complete. 2024-09-15T23:29:55.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:54 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:29:55.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:54 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:29:55.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:54 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:29:55.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:54 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:29:55.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:54 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:29:55.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:54 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:29:55.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:54 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:29:55.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:54 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:29:55.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:55 smithi062 ceph-mon[30949]: pgmap v779: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.2 KiB/s wr, 98 op/s 2024-09-15T23:29:56.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:55 smithi067 ceph-mon[35665]: pgmap v779: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.2 KiB/s wr, 98 op/s 2024-09-15T23:29:56.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:55 smithi148 ceph-mon[35610]: pgmap v779: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 5.2 KiB/s wr, 98 op/s 2024-09-15T23:29:56.724 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '5d785437', total seconds elapsed: 30.8 2024-09-15T23:29:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:57 smithi067 ceph-mon[35665]: pgmap v780: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 18 KiB/s wr, 79 op/s 2024-09-15T23:29:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:57 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:29:58.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:57 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:29:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:57 smithi148 ceph-mon[35610]: pgmap v780: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 18 KiB/s wr, 79 op/s 2024-09-15T23:29:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:57 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:29:58.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:57 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:29:58.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:57 smithi062 ceph-mon[30949]: pgmap v780: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 18 KiB/s wr, 79 op/s 2024-09-15T23:29:58.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:57 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:29:58.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:57 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:29:59.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:58 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:29:59.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:58 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:29:59.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:58 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:29:59.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 820.0 s, 524.7 tps, lat 964.968 ms stddev 4018.926 2024-09-15T23:30:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:29:59 smithi067 ceph-mon[35665]: pgmap v781: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 459 KiB/s wr, 90 op/s 2024-09-15T23:30:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:29:59 smithi148 ceph-mon[35610]: pgmap v781: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 459 KiB/s wr, 90 op/s 2024-09-15T23:30:00.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:29:59 smithi062 ceph-mon[30949]: pgmap v781: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 459 KiB/s wr, 90 op/s 2024-09-15T23:30:01.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:00 smithi067 ceph-mon[35665]: overall HEALTH_OK 2024-09-15T23:30:01.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:00 smithi148 ceph-mon[35610]: overall HEALTH_OK 2024-09-15T23:30:01.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:00 smithi062 ceph-mon[30949]: overall HEALTH_OK 2024-09-15T23:30:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:01 smithi067 ceph-mon[35665]: pgmap v782: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.0 MiB/s wr, 123 op/s 2024-09-15T23:30:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:01 smithi148 ceph-mon[35610]: pgmap v782: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.0 MiB/s wr, 123 op/s 2024-09-15T23:30:02.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:01 smithi062 ceph-mon[30949]: pgmap v782: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.0 MiB/s wr, 123 op/s 2024-09-15T23:30:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.545+0000 7f71bb331640 1 -- 172.21.15.62:0/1800969994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71b4102d50 msgr2=0x7f71b41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:02.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.545+0000 7f71bb331640 1 --2- 172.21.15.62:0/1800969994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71b4102d50 0x7f71b41031d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009900 tx=0x7f71b002a420 comp rx=0 tx=0).stop 2024-09-15T23:30:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.555+0000 7f71bb331640 1 -- 172.21.15.62:0/1800969994 shutdown_connections 2024-09-15T23:30:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.555+0000 7f71bb331640 1 --2- 172.21.15.62:0/1800969994 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71b4103710 0x7f71b4109fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.555+0000 7f71bb331640 1 --2- 172.21.15.62:0/1800969994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71b4102d50 0x7f71b41031d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.555+0000 7f71bb331640 1 --2- 172.21.15.62:0/1800969994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4101b60 0x7f71b4101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.557 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.555+0000 7f71bb331640 1 -- 172.21.15.62:0/1800969994 >> 172.21.15.62:0/1800969994 conn(0x7f71b40fd2c0 msgr2=0x7f71b40ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:02.561 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.560+0000 7f71bb331640 1 -- 172.21.15.62:0/1800969994 shutdown_connections 2024-09-15T23:30:02.563 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.562+0000 7f71bb331640 1 -- 172.21.15.62:0/1800969994 wait complete. 2024-09-15T23:30:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.563+0000 7f71bb331640 1 Processor -- start 2024-09-15T23:30:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.567+0000 7f71bb331640 1 -- start start 2024-09-15T23:30:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.567+0000 7f71bb331640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71b4101b60 0x7f71b4070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.567+0000 7f71bb331640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71b4102d50 0x7f71b40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.567+0000 7f71bb331640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4103710 0x7f71b410ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.567+0000 7f71bb331640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f71b4113e70 con 0x7f71b4103710 2024-09-15T23:30:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.567+0000 7f71bb331640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71b4113cf0 con 0x7f71b4101b60 2024-09-15T23:30:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.567+0000 7f71bb331640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f71b4113ff0 con 0x7f71b4102d50 2024-09-15T23:30:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.569+0000 7f71b90cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71b4101b60 0x7f71b4070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.569+0000 7f71b90cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71b4101b60 0x7f71b4070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40410/0 (socket says 172.21.15.62:40410) 2024-09-15T23:30:02.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.569+0000 7f71b90cd640 1 -- 172.21.15.62:0/2246966106 learned_addr learned my addr 172.21.15.62:0/2246966106 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:02.572 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.570+0000 7f71b88cc640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71b4102d50 0x7f71b40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.573 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.572+0000 7f71b98ce640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4103710 0x7f71b410ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.575+0000 7f71b88cc640 1 -- 172.21.15.62:0/2246966106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71b4101b60 msgr2=0x7f71b4070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.575+0000 7f71b88cc640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71b4101b60 0x7f71b4070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.575+0000 7f71b88cc640 1 -- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4103710 msgr2=0x7f71b410ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.575+0000 7f71b88cc640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4103710 0x7f71b410ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.576 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.575+0000 7f71b88cc640 1 -- 172.21.15.62:0/2246966106 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b410f150 con 0x7f71b4102d50 2024-09-15T23:30:02.577 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.575+0000 7f71b88cc640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71b4102d50 0x7f71b40710c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009230 tx=0x7f71b0032c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.580+0000 7f71aa7fc640 1 -- 172.21.15.62:0/2246966106 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b003f030 con 0x7f71b4102d50 2024-09-15T23:30:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.580+0000 7f71aa7fc640 1 -- 172.21.15.62:0/2246966106 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b0003ce0 con 0x7f71b4102d50 2024-09-15T23:30:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.580+0000 7f71aa7fc640 1 -- 172.21.15.62:0/2246966106 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0035070 con 0x7f71b4102d50 2024-09-15T23:30:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.581+0000 7f71bb331640 1 -- 172.21.15.62:0/2246966106 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b410f380 con 0x7f71b4102d50 2024-09-15T23:30:02.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.581+0000 7f71bb331640 1 -- 172.21.15.62:0/2246966106 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b410f820 con 0x7f71b4102d50 2024-09-15T23:30:02.587 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.586+0000 7f71aa7fc640 1 -- 172.21.15.62:0/2246966106 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f71b0042030 con 0x7f71b4102d50 2024-09-15T23:30:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.588+0000 7f71aa7fc640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f71a00a3530 0x7f71a00a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.589 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.588+0000 7f71aa7fc640 1 -- 172.21.15.62:0/2246966106 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f71b00eff70 con 0x7f71b4102d50 2024-09-15T23:30:02.594 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.593+0000 7f71be0d2740 1 Processor -- start 2024-09-15T23:30:02.598 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.596+0000 7f71b90cd640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f71a00a3530 0x7f71a00a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.600+0000 7f71be0d2740 1 -- start start 2024-09-15T23:30:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.600+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d489da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.601 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.600+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d48eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.601+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d491830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.601+0000 7f71b88cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d48eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.601+0000 7f71b88cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d48eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40424/0 (socket says 172.21.15.62:40424) 2024-09-15T23:30:02.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.601+0000 7f71b88cc640 1 -- 172.21.15.62:0/689342826 learned_addr learned my addr 172.21.15.62:0/689342826 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:02.602 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.601+0000 7f71b90cd640 1 --2- 172.21.15.62:0/689342826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d489da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.601+0000 7f71be0d2740 1 -- 172.21.15.62:0/689342826 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b04d3ed2f0 con 0x55b04d4899a0 2024-09-15T23:30:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.601+0000 7f71be0d2740 1 -- 172.21.15.62:0/689342826 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b04d395270 con 0x55b04d48a2e0 2024-09-15T23:30:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.601+0000 7f71be0d2740 1 -- 172.21.15.62:0/689342826 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b04d3e92b0 con 0x55b04d48f370 2024-09-15T23:30:02.603 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.601+0000 7f71b98ce640 1 --2- 172.21.15.62:0/689342826 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d491830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.602+0000 7f71b90cd640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f71a00a3530 0x7f71a00a59f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f71ac011440 tx=0x7f71ac009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.602+0000 7f71b90cd640 1 -- 172.21.15.62:0/689342826 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 msgr2=0x55b04d491830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.602+0000 7f71b90cd640 1 --2- 172.21.15.62:0/689342826 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d491830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.602+0000 7f71b90cd640 1 -- 172.21.15.62:0/689342826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 msgr2=0x55b04d48eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.602+0000 7f71b90cd640 1 --2- 172.21.15.62:0/689342826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d48eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.602+0000 7f71b90cd640 1 -- 172.21.15.62:0/689342826 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04d394340 con 0x55b04d4899a0 2024-09-15T23:30:02.604 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.603+0000 7f71b90cd640 1 --2- 172.21.15.62:0/689342826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d489da0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00f680 tx=0x7f71ac01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba159642562442ca server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:02.607 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.607+0000 7f719f7fe640 1 -- 172.21.15.62:0/689342826 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac01c560 con 0x55b04d4899a0 2024-09-15T23:30:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.607+0000 7f719f7fe640 1 -- 172.21.15.62:0/689342826 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac01cbe0 con 0x55b04d4899a0 2024-09-15T23:30:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.608+0000 7f71be0d2740 1 -- 172.21.15.62:0/689342826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 msgr2=0x55b04d489da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:02.609 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.608+0000 7f71be0d2740 1 --2- 172.21.15.62:0/689342826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d489da0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00f680 tx=0x7f71ac01b000 comp rx=0 tx=0).stop 2024-09-15T23:30:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.610+0000 7f71be0d2740 1 -- 172.21.15.62:0/689342826 shutdown_connections 2024-09-15T23:30:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.610+0000 7f71be0d2740 1 --2- 172.21.15.62:0/689342826 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d491830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.610+0000 7f71be0d2740 1 --2- 172.21.15.62:0/689342826 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d48eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.610+0000 7f71be0d2740 1 --2- 172.21.15.62:0/689342826 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d489da0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.612 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.610+0000 7f71be0d2740 1 -- 172.21.15.62:0/689342826 >> 172.21.15.62:0/689342826 conn(0x55b04d485010 msgr2=0x55b04d485410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:02.614 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.613+0000 7f71be0d2740 1 -- 172.21.15.62:0/689342826 shutdown_connections 2024-09-15T23:30:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.613+0000 7f71be0d2740 1 -- 172.21.15.62:0/689342826 wait complete. 2024-09-15T23:30:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.614+0000 7f71be0d2740 1 -- start start 2024-09-15T23:30:02.615 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.614+0000 7f71be0d2740 1 Processor -- start 2024-09-15T23:30:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.615+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d493c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.615+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d494150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.615+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d49b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.615+0000 7f71be0d2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b04d492390 con 0x55b04d4899a0 2024-09-15T23:30:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.615+0000 7f71be0d2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b04d376290 con 0x55b04d48a2e0 2024-09-15T23:30:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.615+0000 7f71be0d2740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b04d492510 con 0x55b04d48f370 2024-09-15T23:30:02.616 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.615+0000 7f71be0d2740 10 client.0 _ll_register_callbacks cb 0x55b04d3a7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.616+0000 7f71b98ce640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d49b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.616+0000 7f71b88cc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d494150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.616+0000 7f71b90cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d493c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.616+0000 7f71b90cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d493c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:47378/0 (socket says 172.21.15.62:47378) 2024-09-15T23:30:02.617 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.616+0000 7f71b90cd640 1 -- 172.21.15.62:0/1211374518 learned_addr learned my addr 172.21.15.62:0/1211374518 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.617+0000 7f71b90cd640 1 -- 172.21.15.62:0/1211374518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 msgr2=0x55b04d49b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.617+0000 7f71b90cd640 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d49b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.617+0000 7f71b90cd640 1 -- 172.21.15.62:0/1211374518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 msgr2=0x55b04d494150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:02.618 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.617+0000 7f71b90cd640 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d494150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.617+0000 7f71b90cd640 1 -- 172.21.15.62:0/1211374518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04d100210 con 0x55b04d4899a0 2024-09-15T23:30:02.619 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.618+0000 7f71b90cd640 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d493c10 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f71ac011cd0 tx=0x7f71ac025480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673b2003187a7c00 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.619+0000 7f719cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:30:02.620 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.619+0000 7f719cff9640 1 -- 172.21.15.62:0/1211374518 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac02c070 con 0x55b04d4899a0 2024-09-15T23:30:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.619+0000 7f71be0d2740 1 -- 172.21.15.62:0/1211374518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71ac01cbe0 con 0x55b04d4899a0 2024-09-15T23:30:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.619+0000 7f719cff9640 1 -- 172.21.15.62:0/1211374518 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f71ac004c00 con 0x55b04d4899a0 2024-09-15T23:30:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.620+0000 7f719cff9640 1 -- 172.21.15.62:0/1211374518 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f71ac037050 con 0x55b04d4899a0 2024-09-15T23:30:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.620+0000 7f71be0d2740 10 client.18672 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:02.621 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.620+0000 7f71be0d2740 1 -- 172.21.15.62:0/1211374518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b04d1cc620 con 0x55b04d4899a0 2024-09-15T23:30:02.622 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.621+0000 7f719cff9640 1 -- 172.21.15.62:0/1211374518 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac01cbe0 con 0x55b04d4899a0 2024-09-15T23:30:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.622+0000 7f719cff9640 1 -- 172.21.15.62:0/1211374518 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f71ac004030 con 0x55b04d4899a0 2024-09-15T23:30:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.622+0000 7f71be0d2740 10 client.18672 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.622+0000 7f71be0d2740 10 client.18672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.622+0000 7f71be0d2740 20 client.18672 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.622+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b04d49fa10 0x55b04d4a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.622+0000 7f71be0d2740 4 client.18672 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:02.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.622+0000 7f71be0d2740 1 -- 172.21.15.62:0/1211374518 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f71ac025e50 con 0x55b04d49fa10 2024-09-15T23:30:02.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.624+0000 7f71b88cc640 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b04d49fa10 0x55b04d4a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.627+0000 7f71b88cc640 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b04d49fa10 0x55b04d4a1e50 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59331f2af1bb6586 server_cookie=19499187f4bd04e7 in_seq=0 out_seq=0 2024-09-15T23:30:02.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.628+0000 7f719cff9640 10 client.18672 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.632+0000 7f719cff9640 1 -- 172.21.15.62:0/1211374518 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f71ac025e50 con 0x55b04d49fa10 2024-09-15T23:30:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.632+0000 7f719cff9640 10 client.18672 handle_command_reply: tid=0 2024-09-15T23:30:02.633 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.632+0000 7f71be0d2740 1 client.18672 shutdown 2024-09-15T23:30:02.640 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.637+0000 7f71be0d2740 1 -- 172.21.15.62:0/1211374518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 msgr2=0x55b04d493c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:02.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.637+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d493c10 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f71ac011cd0 tx=0x7f71ac025480 comp rx=0 tx=0).stop 2024-09-15T23:30:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.648+0000 7f71be0d2740 1 -- 172.21.15.62:0/1211374518 shutdown_connections 2024-09-15T23:30:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.648+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b04d49fa10 0x55b04d4a1e50 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.648+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d49b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.648+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d494150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.648+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1211374518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d4899a0 0x55b04d493c10 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.648+0000 7f71be0d2740 1 -- 172.21.15.62:0/1211374518 >> 172.21.15.62:0/1211374518 conn(0x55b04d485010 msgr2=0x55b04d486820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:02.652 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.651+0000 7f719cff9640 0 client.18672 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:02.659 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.658+0000 7f71be0d2740 1 -- 172.21.15.62:0/1211374518 shutdown_connections 2024-09-15T23:30:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.658+0000 7f71be0d2740 1 -- 172.21.15.62:0/1211374518 wait complete. 2024-09-15T23:30:02.662 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.661+0000 7f71be0d2740 20 client.18672 trim_cache size 0 max 16384 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.710+0000 7f71be0d2740 1 Processor -- start 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.710+0000 7f71be0d2740 1 -- start start 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.710+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d4899a0 0x55b04d4939c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.710+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d48a2e0 0x55b04d49b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.710+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48f370 0x55b04d493f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.711+0000 7f71b98ce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48f370 0x55b04d493f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.711+0000 7f71b98ce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48f370 0x55b04d493f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40442/0 (socket says 172.21.15.62:40442) 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.711+0000 7f71b98ce640 1 -- 172.21.15.62:0/2612709690 learned_addr learned my addr 172.21.15.62:0/2612709690 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.711+0000 7f71be0d2740 1 -- 172.21.15.62:0/2612709690 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41c45a0 con 0x55b04d48a2e0 2024-09-15T23:30:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.711+0000 7f71be0d2740 1 -- 172.21.15.62:0/2612709690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41c1e40 con 0x55b04d48f370 2024-09-15T23:30:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.711+0000 7f71be0d2740 1 -- 172.21.15.62:0/2612709690 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41bfd40 con 0x55b04d4899a0 2024-09-15T23:30:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.711+0000 7f71b90cd640 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d4899a0 0x55b04d4939c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.714+0000 7f71b90cd640 1 -- 172.21.15.62:0/2612709690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48f370 msgr2=0x55b04d493f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.714+0000 7f71b90cd640 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48f370 0x55b04d493f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.714+0000 7f71b88cc640 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d48a2e0 0x55b04d49b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.714+0000 7f71b90cd640 1 -- 172.21.15.62:0/2612709690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d48a2e0 msgr2=0x55b04d49b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.714+0000 7f71b90cd640 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d48a2e0 0x55b04d49b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.714+0000 7f71b90cd640 1 -- 172.21.15.62:0/2612709690 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04d484e30 con 0x55b04d4899a0 2024-09-15T23:30:02.716 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.714+0000 7f71b88cc640 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d48a2e0 0x55b04d49b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:02.721 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.720+0000 7f71b90cd640 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d4899a0 0x55b04d4939c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009dc0 tx=0x7f71ac0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4c9e849343dda72 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.720+0000 7f719effd640 1 -- 172.21.15.62:0/2612709690 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac02c050 con 0x55b04d4899a0 2024-09-15T23:30:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.721+0000 7f719effd640 1 -- 172.21.15.62:0/2612709690 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac01ca40 con 0x55b04d4899a0 2024-09-15T23:30:02.722 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.721+0000 7f719effd640 1 -- 172.21.15.62:0/2612709690 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ac003bd0 con 0x55b04d4899a0 2024-09-15T23:30:02.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.724+0000 7f71be0d2740 1 -- 172.21.15.62:0/2612709690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d4899a0 msgr2=0x55b04d4939c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.724+0000 7f71be0d2740 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d4899a0 0x55b04d4939c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009dc0 tx=0x7f71ac0047b0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.724+0000 7f71be0d2740 1 -- 172.21.15.62:0/2612709690 shutdown_connections 2024-09-15T23:30:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.724+0000 7f71be0d2740 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48f370 0x55b04d493f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.724+0000 7f71be0d2740 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d48a2e0 0x55b04d49b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.724+0000 7f71be0d2740 1 --2- 172.21.15.62:0/2612709690 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d4899a0 0x55b04d4939c0 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f71ac009dc0 tx=0x7f71ac0047b0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.724+0000 7f71be0d2740 1 -- 172.21.15.62:0/2612709690 >> 172.21.15.62:0/2612709690 conn(0x55b04d485010 msgr2=0x55b04d48b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:02.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.729+0000 7f71be0d2740 1 -- 172.21.15.62:0/2612709690 shutdown_connections 2024-09-15T23:30:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.730+0000 7f71be0d2740 1 -- 172.21.15.62:0/2612709690 wait complete. 2024-09-15T23:30:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.730+0000 7f71be0d2740 1 -- start start 2024-09-15T23:30:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.731+0000 7f71be0d2740 1 Processor -- start 2024-09-15T23:30:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.731+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d49ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.731+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d497850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.731+0000 7f71be0d2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d49b340 0x55b04d497d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.731+0000 7f71be0d2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41c0300 con 0x55b04d49b340 2024-09-15T23:30:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.731+0000 7f71be0d2740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41c1770 con 0x55b04d48a2e0 2024-09-15T23:30:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.731+0000 7f71b90cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d49ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.732+0000 7f71b90cd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d49ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40458/0 (socket says 172.21.15.62:40458) 2024-09-15T23:30:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.732+0000 7f71b90cd640 1 -- 172.21.15.62:0/1247480672 learned_addr learned my addr 172.21.15.62:0/1247480672 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.732+0000 7f71be0d2740 1 -- 172.21.15.62:0/1247480672 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f71b41c09d0 con 0x55b04d48f370 2024-09-15T23:30:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.732+0000 7f71be0d2740 10 client.0 _ll_register_callbacks cb 0x7f71b0003e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.732+0000 7f71b88cc640 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d497850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.734+0000 7f71b88cc640 1 -- 172.21.15.62:0/1247480672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 msgr2=0x55b04d49ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.734+0000 7f71b88cc640 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d49ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.734+0000 7f71b88cc640 1 -- 172.21.15.62:0/1247480672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d49b340 msgr2=0x55b04d497d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.734+0000 7f71b88cc640 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d49b340 0x55b04d497d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.734+0000 7f71b88cc640 1 -- 172.21.15.62:0/1247480672 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04d508720 con 0x55b04d48f370 2024-09-15T23:30:02.739 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.738+0000 7f71b88cc640 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d497850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009900 tx=0x7f71b003b5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa07904b050955a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.742+0000 7f719cff9640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.742+0000 7f719cff9640 1 -- 172.21.15.62:0/1247480672 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b00bc1b0 con 0x55b04d48f370 2024-09-15T23:30:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.742+0000 7f719cff9640 1 -- 172.21.15.62:0/1247480672 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f71b00d50d0 con 0x55b04d48f370 2024-09-15T23:30:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.743+0000 7f71be0d2740 1 -- 172.21.15.62:0/1247480672 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71ac01ca40 con 0x55b04d48f370 2024-09-15T23:30:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.744+0000 7f719cff9640 1 -- 172.21.15.62:0/1247480672 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f71b00c0560 con 0x55b04d48f370 2024-09-15T23:30:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.745+0000 7f71be0d2740 10 client.27859 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:02.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.745+0000 7f71be0d2740 1 -- 172.21.15.62:0/1247480672 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b04d5099a0 con 0x55b04d48f370 2024-09-15T23:30:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.747+0000 7f719cff9640 1 -- 172.21.15.62:0/1247480672 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71ac01ca40 con 0x55b04d48f370 2024-09-15T23:30:02.748 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.747+0000 7f719cff9640 1 -- 172.21.15.62:0/1247480672 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f71b00dc0b0 con 0x55b04d48f370 2024-09-15T23:30:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.747+0000 7f71be0d2740 10 client.27859 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.747+0000 7f71be0d2740 10 client.27859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.747+0000 7f71be0d2740 20 client.27859 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.747+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b04d50a470 0x55b04d50c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.747+0000 7f71be0d2740 4 client.27859 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:30:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.747+0000 7f71be0d2740 1 -- 172.21.15.62:0/1247480672 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b04d3faa20 con 0x55b04d50a470 2024-09-15T23:30:02.749 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.748+0000 7f71b90cd640 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b04d50a470 0x55b04d50c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:02.756 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.754+0000 7f71b90cd640 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b04d50a470 0x55b04d50c890 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=927a9d8f16141a21 server_cookie=e8d30927cab39cdf in_seq=0 out_seq=0 2024-09-15T23:30:02.761 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.756+0000 7f719cff9640 10 client.27859 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f719cff9640 1 -- 172.21.15.62:0/1247480672 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b04d3faa20 con 0x55b04d50a470 2024-09-15T23:30:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f719cff9640 10 client.27859 handle_command_reply: tid=0 2024-09-15T23:30:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 client.27859 shutdown 2024-09-15T23:30:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 -- 172.21.15.62:0/1247480672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 msgr2=0x55b04d497850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d497850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009900 tx=0x7f71b003b5d0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 -- 172.21.15.62:0/1247480672 shutdown_connections 2024-09-15T23:30:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b04d50a470 0x55b04d50c890 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b04d49b340 0x55b04d497d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b04d48f370 0x55b04d497850 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 --2- 172.21.15.62:0/1247480672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b04d48a2e0 0x55b04d49ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 -- 172.21.15.62:0/1247480672 >> 172.21.15.62:0/1247480672 conn(0x55b04d485010 msgr2=0x55b04d4839a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f719cff9640 0 client.27859 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 -- 172.21.15.62:0/1247480672 shutdown_connections 2024-09-15T23:30:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 1 -- 172.21.15.62:0/1247480672 wait complete. 2024-09-15T23:30:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.843+0000 7f71be0d2740 20 client.27859 trim_cache size 0 max 16384 2024-09-15T23:30:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:30:02.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:30:02.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:30:02.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:30:02.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 -- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f71a00a3530 msgr2=0x7f71a00a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:02.847 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f71a00a3530 0x7f71a00a59f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f71ac011440 tx=0x7f71ac009500 comp rx=0 tx=0).stop 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 -- 172.21.15.62:0/2246966106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71b4102d50 msgr2=0x7f71b40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71b4102d50 0x7f71b40710c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009230 tx=0x7f71b0032c90 comp rx=0 tx=0).stop 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 -- 172.21.15.62:0/2246966106 shutdown_connections 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f71a00a3530 0x7f71a00a59f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71b4103710 0x7f71b410ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f71b4102d50 0x7f71b40710c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 --2- 172.21.15.62:0/2246966106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71b4101b60 0x7f71b4070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 -- 172.21.15.62:0/2246966106 >> 172.21.15.62:0/2246966106 conn(0x7f71b40fd2c0 msgr2=0x7f71b41060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 -- 172.21.15.62:0/2246966106 shutdown_connections 2024-09-15T23:30:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:02.844+0000 7f71bb331640 1 -- 172.21.15.62:0/2246966106 wait complete. 2024-09-15T23:30:04.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:03 smithi067 ceph-mon[35665]: pgmap v783: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.1 MiB/s wr, 214 op/s 2024-09-15T23:30:04.555 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:03 smithi062 ceph-mon[30949]: pgmap v783: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.1 MiB/s wr, 214 op/s 2024-09-15T23:30:04.555 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:03 smithi148 ceph-mon[35610]: pgmap v783: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 2.1 MiB/s wr, 214 op/s 2024-09-15T23:30:04.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 825.0 s, 698.6 tps, lat 139.516 ms stddev 154.802 2024-09-15T23:30:06.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:05 smithi067 ceph-mon[35665]: pgmap v784: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-09-15T23:30:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:05 smithi148 ceph-mon[35610]: pgmap v784: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-09-15T23:30:06.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:05 smithi062 ceph-mon[30949]: pgmap v784: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-09-15T23:30:08.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:07 smithi148 ceph-mon[35610]: pgmap v785: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4.1 MiB/s wr, 420 op/s 2024-09-15T23:30:08.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:07 smithi062 ceph-mon[30949]: pgmap v785: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4.1 MiB/s wr, 420 op/s 2024-09-15T23:30:08.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:07 smithi067 ceph-mon[35665]: pgmap v785: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4.1 MiB/s wr, 420 op/s 2024-09-15T23:30:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 830.0 s, 642.8 tps, lat 159.983 ms stddev 198.467 2024-09-15T23:30:10.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:09 smithi148 ceph-mon[35610]: pgmap v786: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.9 MiB/s wr, 525 op/s 2024-09-15T23:30:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:09 smithi062 ceph-mon[30949]: pgmap v786: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.9 MiB/s wr, 525 op/s 2024-09-15T23:30:10.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:09 smithi067 ceph-mon[35665]: pgmap v786: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.9 MiB/s wr, 525 op/s 2024-09-15T23:30:12.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:11 smithi062 ceph-mon[30949]: pgmap v787: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 588 op/s 2024-09-15T23:30:12.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:11 smithi067 ceph-mon[35665]: pgmap v787: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 588 op/s 2024-09-15T23:30:12.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:11 smithi148 ceph-mon[35610]: pgmap v787: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 5.6 MiB/s wr, 588 op/s 2024-09-15T23:30:14.049 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:13 smithi062 ceph-mon[30949]: pgmap v788: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.0 MiB/s wr, 605 op/s 2024-09-15T23:30:14.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:13 smithi067 ceph-mon[35665]: pgmap v788: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.0 MiB/s wr, 605 op/s 2024-09-15T23:30:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:13 smithi148 ceph-mon[35610]: pgmap v788: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.0 MiB/s wr, 605 op/s 2024-09-15T23:30:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 835.0 s, 540.0 tps, lat 182.260 ms stddev 214.041 2024-09-15T23:30:15.297 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:30:15.816 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.006+0000 7fc23400e640 1 -- 172.21.15.62:0/3742592523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c1025f0 msgr2=0x7fc22c108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.006+0000 7fc23400e640 1 --2- 172.21.15.62:0/3742592523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c1025f0 0x7fc22c108e80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc224009be0 tx=0x7fc22402a710 comp rx=0 tx=0).stop 2024-09-15T23:30:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.010+0000 7fc23400e640 1 -- 172.21.15.62:0/3742592523 shutdown_connections 2024-09-15T23:30:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.010+0000 7fc23400e640 1 --2- 172.21.15.62:0/3742592523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c1025f0 0x7fc22c108e80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.010+0000 7fc23400e640 1 --2- 172.21.15.62:0/3742592523 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc22c101c30 0x7fc22c1020b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.010+0000 7fc23400e640 1 --2- 172.21.15.62:0/3742592523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc22c100a40 0x7fc22c100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.010+0000 7fc23400e640 1 -- 172.21.15.62:0/3742592523 >> 172.21.15.62:0/3742592523 conn(0x7fc22c0fc1e0 msgr2=0x7fc22c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.012 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.012+0000 7fc23400e640 1 -- 172.21.15.62:0/3742592523 shutdown_connections 2024-09-15T23:30:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.012+0000 7fc23400e640 1 -- 172.21.15.62:0/3742592523 wait complete. 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.013+0000 7fc23400e640 1 Processor -- start 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.013+0000 7fc23400e640 1 -- start start 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.013+0000 7fc23400e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c100a40 0x7fc22c107f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.013+0000 7fc23400e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc22c101c30 0x7fc22c108480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.013+0000 7fc23400e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc22c1025f0 0x7fc22c1b6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.013+0000 7fc23400e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c06d740 con 0x7fc22c1025f0 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.013+0000 7fc23400e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c06d5c0 con 0x7fc22c100a40 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.013+0000 7fc23400e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c06d8c0 con 0x7fc22c101c30 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.014+0000 7fc2325ab640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc22c1025f0 0x7fc22c1b6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.014+0000 7fc2325ab640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc22c1025f0 0x7fc22c1b6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55884/0 (socket says 172.21.15.62:55884) 2024-09-15T23:30:16.017 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.014+0000 7fc2325ab640 1 -- 172.21.15.62:0/3500578921 learned_addr learned my addr 172.21.15.62:0/3500578921 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.015+0000 7fc2325ab640 1 -- 172.21.15.62:0/3500578921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc22c101c30 msgr2=0x7fc22c108480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.015+0000 7fc2325ab640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc22c101c30 0x7fc22c108480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.015+0000 7fc2325ab640 1 -- 172.21.15.62:0/3500578921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c100a40 msgr2=0x7fc22c107f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.015+0000 7fc2325ab640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c100a40 0x7fc22c107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.015+0000 7fc2325ab640 1 -- 172.21.15.62:0/3500578921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c1b7020 con 0x7fc22c1025f0 2024-09-15T23:30:16.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.016+0000 7fc2325ab640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc22c1025f0 0x7fc22c1b6950 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc224009b30 tx=0x7fc224030c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.016+0000 7fc222ffd640 1 -- 172.21.15.62:0/3500578921 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224041020 con 0x7fc22c1025f0 2024-09-15T23:30:16.018 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.016+0000 7fc23400e640 1 -- 172.21.15.62:0/3500578921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc22c1b7250 con 0x7fc22c1025f0 2024-09-15T23:30:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.016+0000 7fc23400e640 1 -- 172.21.15.62:0/3500578921 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22c1b7740 con 0x7fc22c1025f0 2024-09-15T23:30:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.017+0000 7fc222ffd640 1 -- 172.21.15.62:0/3500578921 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc22403c950 con 0x7fc22c1025f0 2024-09-15T23:30:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.017+0000 7fc222ffd640 1 -- 172.21.15.62:0/3500578921 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224036040 con 0x7fc22c1025f0 2024-09-15T23:30:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.021+0000 7fc222ffd640 1 -- 172.21.15.62:0/3500578921 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc224044020 con 0x7fc22c1025f0 2024-09-15T23:30:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.023+0000 7fc222ffd640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2180abd30 0x7fc2180ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.023+0000 7fc222ffd640 1 -- 172.21.15.62:0/3500578921 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc2240f0fd0 con 0x7fc22c1025f0 2024-09-15T23:30:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.023+0000 7fc236e87740 1 Processor -- start 2024-09-15T23:30:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.024+0000 7fc236e87740 1 -- start start 2024-09-15T23:30:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.024+0000 7fc236e87740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24df9a0 0x5649c24dfda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.024+0000 7fc236e87740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24e02e0 0x5649c24e4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.024+0000 7fc236e87740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24e7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.024+0000 7fc236e87740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5649c24432f0 con 0x5649c24df9a0 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.024+0000 7fc236e87740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5649c23eb270 con 0x5649c24e5370 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.024+0000 7fc236e87740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5649c243f2b0 con 0x5649c24e02e0 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.026+0000 7fc2315a9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24e02e0 0x5649c24e4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.026+0000 7fc231daa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24df9a0 0x5649c24dfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.026+0000 7fc231daa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24df9a0 0x5649c24dfda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55886/0 (socket says 172.21.15.62:55886) 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.026+0000 7fc231daa640 1 -- 172.21.15.62:0/1739941689 learned_addr learned my addr 172.21.15.62:0/1739941689 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.026+0000 7fc231daa640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2180abd30 0x7fc2180ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.026+0000 7fc2325ab640 1 --2- 172.21.15.62:0/1739941689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24e7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.027+0000 7fc2315a9640 1 -- 172.21.15.62:0/1739941689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 msgr2=0x5649c24e7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.027+0000 7fc2315a9640 1 --2- 172.21.15.62:0/1739941689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24e7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.027+0000 7fc2315a9640 1 -- 172.21.15.62:0/1739941689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24df9a0 msgr2=0x5649c24dfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.027+0000 7fc2315a9640 1 --2- 172.21.15.62:0/1739941689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24df9a0 0x5649c24dfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.027+0000 7fc2315a9640 1 -- 172.21.15.62:0/1739941689 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c23ea340 con 0x5649c24e02e0 2024-09-15T23:30:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.028+0000 7fc2315a9640 1 --2- 172.21.15.62:0/1739941689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24e02e0 0x5649c24e4da0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009b40 tx=0x7fc21c00e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75cdc52ed963cdab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.028+0000 7fc231daa640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2180abd30 0x7fc2180ae1f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc228005ab0 tx=0x7fc22800c5f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.028+0000 7fc217fff640 1 -- 172.21.15.62:0/1739941689 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c00ed00 con 0x5649c24e02e0 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.028+0000 7fc217fff640 1 -- 172.21.15.62:0/1739941689 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc21c005020 con 0x5649c24e02e0 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.029+0000 7fc236e87740 1 -- 172.21.15.62:0/1739941689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24e02e0 msgr2=0x5649c24e4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.029+0000 7fc236e87740 1 --2- 172.21.15.62:0/1739941689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24e02e0 0x5649c24e4da0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009b40 tx=0x7fc21c00e350 comp rx=0 tx=0).stop 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.030+0000 7fc236e87740 1 -- 172.21.15.62:0/1739941689 shutdown_connections 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.030+0000 7fc236e87740 1 --2- 172.21.15.62:0/1739941689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24e7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.030+0000 7fc236e87740 1 --2- 172.21.15.62:0/1739941689 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24e02e0 0x5649c24e4da0 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009b40 tx=0x7fc21c00e350 comp rx=0 tx=0).stop 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.030+0000 7fc236e87740 1 --2- 172.21.15.62:0/1739941689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24df9a0 0x5649c24dfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.030+0000 7fc236e87740 1 -- 172.21.15.62:0/1739941689 >> 172.21.15.62:0/1739941689 conn(0x5649c24db010 msgr2=0x5649c24db410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.033 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.033+0000 7fc236e87740 1 -- 172.21.15.62:0/1739941689 shutdown_connections 2024-09-15T23:30:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.034+0000 7fc236e87740 1 -- 172.21.15.62:0/1739941689 wait complete. 2024-09-15T23:30:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.034+0000 7fc236e87740 1 -- start start 2024-09-15T23:30:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.034+0000 7fc236e87740 1 Processor -- start 2024-09-15T23:30:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.034+0000 7fc236e87740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24ea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.036 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.034+0000 7fc236e87740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24ea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc236e87740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc236e87740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5649c24e8390 con 0x5649c24ef630 2024-09-15T23:30:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc236e87740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5649c23cc290 con 0x5649c24e5370 2024-09-15T23:30:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc236e87740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5649c24e8510 con 0x5649c24df9a0 2024-09-15T23:30:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc236e87740 10 client.0 _ll_register_callbacks cb 0x5649c23fd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc231daa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24ea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc231daa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24ea0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35948/0 (socket says 172.21.15.62:35948) 2024-09-15T23:30:16.037 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc231daa640 1 -- 172.21.15.62:0/2637492279 learned_addr learned my addr 172.21.15.62:0/2637492279 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc2325ab640 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.035+0000 7fc2315a9640 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24ea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.037+0000 7fc2315a9640 1 -- 172.21.15.62:0/2637492279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 msgr2=0x5649c24ea0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.037+0000 7fc2315a9640 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24ea0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.038 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.037+0000 7fc2315a9640 1 -- 172.21.15.62:0/2637492279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 msgr2=0x5649c24f1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.037+0000 7fc2315a9640 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24f1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.037+0000 7fc2315a9640 1 -- 172.21.15.62:0/2637492279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c2156210 con 0x5649c24e5370 2024-09-15T23:30:16.039 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.037+0000 7fc2325ab640 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.038+0000 7fc2315a9640 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24ea600 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x5649c24e0880 tx=0x7fc21c004f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65a287cb8c7a13d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.038+0000 7fc2157fa640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:30:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.038+0000 7fc2157fa640 1 -- 172.21.15.62:0/2637492279 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c028060 con 0x5649c24e5370 2024-09-15T23:30:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.038+0000 7fc2157fa640 1 -- 172.21.15.62:0/2637492279 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc21c005430 con 0x5649c24e5370 2024-09-15T23:30:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.038+0000 7fc236e87740 1 -- 172.21.15.62:0/2637492279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5649c2450a20 con 0x5649c24e5370 2024-09-15T23:30:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.038+0000 7fc2157fa640 1 -- 172.21.15.62:0/2637492279 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc21c02f050 con 0x5649c24e5370 2024-09-15T23:30:16.040 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.038+0000 7fc236e87740 10 client.28202 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.038+0000 7fc236e87740 1 -- 172.21.15.62:0/2637492279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5649c2222620 con 0x5649c24e5370 2024-09-15T23:30:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.039+0000 7fc2157fa640 1 -- 172.21.15.62:0/2637492279 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5649c2450a20 con 0x5649c24e5370 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.039+0000 7fc2157fa640 1 -- 172.21.15.62:0/2637492279 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fc21c018470 con 0x5649c24e5370 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.039+0000 7fc236e87740 10 client.28202 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.039+0000 7fc236e87740 10 client.28202 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.039+0000 7fc236e87740 20 client.28202 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.039+0000 7fc236e87740 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5649c24f6500 0x5649c24f8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.039+0000 7fc236e87740 4 client.28202 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.039+0000 7fc236e87740 1 -- 172.21.15.62:0/2637492279 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649c244ad70 con 0x5649c24f6500 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.040+0000 7fc231daa640 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5649c24f6500 0x5649c24f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.040+0000 7fc2157fa640 10 client.28202 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:16.042 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.040+0000 7fc231daa640 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5649c24f6500 0x5649c24f8960 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=312c68ea70870906 server_cookie=4424622350bd939c in_seq=0 out_seq=0 2024-09-15T23:30:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.041+0000 7fc2157fa640 1 -- 172.21.15.62:0/2637492279 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5649c244ad70 con 0x5649c24f6500 2024-09-15T23:30:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.041+0000 7fc2157fa640 10 client.28202 handle_command_reply: tid=0 2024-09-15T23:30:16.043 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.041+0000 7fc236e87740 1 client.28202 shutdown 2024-09-15T23:30:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.042+0000 7fc236e87740 1 -- 172.21.15.62:0/2637492279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 msgr2=0x5649c24ea600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.042+0000 7fc236e87740 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24ea600 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x5649c24e0880 tx=0x7fc21c004f90 comp rx=0 tx=0).stop 2024-09-15T23:30:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc236e87740 1 -- 172.21.15.62:0/2637492279 shutdown_connections 2024-09-15T23:30:16.044 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc236e87740 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5649c24f6500 0x5649c24f8960 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc236e87740 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24f1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc236e87740 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24ea600 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc236e87740 1 --2- 172.21.15.62:0/2637492279 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24ea0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc236e87740 1 -- 172.21.15.62:0/2637492279 >> 172.21.15.62:0/2637492279 conn(0x5649c24db010 msgr2=0x5649c24dc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc2157fa640 0 client.28202 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc236e87740 1 -- 172.21.15.62:0/2637492279 shutdown_connections 2024-09-15T23:30:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc236e87740 1 -- 172.21.15.62:0/2637492279 wait complete. 2024-09-15T23:30:16.045 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.043+0000 7fc236e87740 20 client.28202 trim_cache size 0 max 16384 2024-09-15T23:30:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.073+0000 7fc236e87740 1 Processor -- start 2024-09-15T23:30:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.073+0000 7fc236e87740 1 -- start start 2024-09-15T23:30:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.074+0000 7fc236e87740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24f8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.074+0000 7fc236e87740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24ea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.074+0000 7fc236e87740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24ea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.074+0000 7fc236e87740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc210012480 con 0x5649c24ef630 2024-09-15T23:30:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.074+0000 7fc236e87740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000fd20 con 0x5649c24e5370 2024-09-15T23:30:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.074+0000 7fc236e87740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000db70 con 0x5649c24df9a0 2024-09-15T23:30:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.075+0000 7fc231daa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24f8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.076+0000 7fc231daa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24f8510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35958/0 (socket says 172.21.15.62:35958) 2024-09-15T23:30:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.076+0000 7fc231daa640 1 -- 172.21.15.62:0/2038067053 learned_addr learned my addr 172.21.15.62:0/2038067053 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.078+0000 7fc2325ab640 1 --2- 172.21.15.62:0/2038067053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24ea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.078+0000 7fc2325ab640 1 -- 172.21.15.62:0/2038067053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 msgr2=0x5649c24f8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.078+0000 7fc2325ab640 1 --2- 172.21.15.62:0/2038067053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24f8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.078+0000 7fc2325ab640 1 -- 172.21.15.62:0/2038067053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 msgr2=0x5649c24ea0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.078+0000 7fc2325ab640 1 --2- 172.21.15.62:0/2038067053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24ea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.078+0000 7fc2325ab640 1 -- 172.21.15.62:0/2038067053 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c24f1d90 con 0x5649c24ef630 2024-09-15T23:30:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.079+0000 7fc2325ab640 1 --2- 172.21.15.62:0/2038067053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24ea600 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc2240bd1a0 tx=0x7fc22403d420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f79d665d1fa8a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.080+0000 7fc2177fe640 1 -- 172.21.15.62:0/2038067053 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2240bd210 con 0x5649c24ef630 2024-09-15T23:30:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.080+0000 7fc2177fe640 1 -- 172.21.15.62:0/2038067053 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2240dbe50 con 0x5649c24ef630 2024-09-15T23:30:16.083 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.080+0000 7fc2177fe640 1 -- 172.21.15.62:0/2038067053 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2240c1180 con 0x5649c24ef630 2024-09-15T23:30:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.082+0000 7fc236e87740 1 -- 172.21.15.62:0/2038067053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 msgr2=0x5649c24ea600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.082+0000 7fc236e87740 1 --2- 172.21.15.62:0/2038067053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24ea600 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fc2240bd1a0 tx=0x7fc22403d420 comp rx=0 tx=0).stop 2024-09-15T23:30:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.087+0000 7fc236e87740 1 -- 172.21.15.62:0/2038067053 shutdown_connections 2024-09-15T23:30:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.087+0000 7fc236e87740 1 --2- 172.21.15.62:0/2038067053 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24ef630 0x5649c24ea600 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.087+0000 7fc236e87740 1 --2- 172.21.15.62:0/2038067053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24e5370 0x5649c24ea0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.087+0000 7fc236e87740 1 --2- 172.21.15.62:0/2038067053 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c24f8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.088 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.087+0000 7fc236e87740 1 -- 172.21.15.62:0/2038067053 >> 172.21.15.62:0/2038067053 conn(0x5649c24db010 msgr2=0x5649c24e92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.094+0000 7fc236e87740 1 -- 172.21.15.62:0/2038067053 shutdown_connections 2024-09-15T23:30:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.098+0000 7fc236e87740 1 -- 172.21.15.62:0/2038067053 wait complete. 2024-09-15T23:30:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.098+0000 7fc236e87740 1 -- start start 2024-09-15T23:30:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.098+0000 7fc236e87740 1 Processor -- start 2024-09-15T23:30:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc236e87740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c2506910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc236e87740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24e5370 0x5649c2506e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc236e87740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24ef630 0x5649c2509e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc236e87740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000e1e0 con 0x5649c24e5370 2024-09-15T23:30:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc236e87740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000f650 con 0x5649c24ef630 2024-09-15T23:30:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc236e87740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000e8b0 con 0x5649c24df9a0 2024-09-15T23:30:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc236e87740 10 client.0 _ll_register_callbacks cb 0x7fc22800ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc2315a9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24e5370 0x5649c2506e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc2315a9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24e5370 0x5649c2506e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55900/0 (socket says 172.21.15.62:55900) 2024-09-15T23:30:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.099+0000 7fc2315a9640 1 -- 172.21.15.62:0/2035620386 learned_addr learned my addr 172.21.15.62:0/2035620386 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.100+0000 7fc2315a9640 1 -- 172.21.15.62:0/2035620386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 msgr2=0x5649c2506910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.100+0000 7fc2315a9640 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c2506910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.100+0000 7fc2315a9640 1 -- 172.21.15.62:0/2035620386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24ef630 msgr2=0x5649c2509e20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.100+0000 7fc2315a9640 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24ef630 0x5649c2509e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.100+0000 7fc2315a9640 1 -- 172.21.15.62:0/2035620386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c250a430 con 0x5649c24e5370 2024-09-15T23:30:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.101+0000 7fc2315a9640 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24e5370 0x5649c2506e50 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc21c018d40 tx=0x7fc21c018d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24e0c529ae11f25 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.102+0000 7fc2157fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:30:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.102+0000 7fc2157fa640 1 -- 172.21.15.62:0/2035620386 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c028040 con 0x5649c24e5370 2024-09-15T23:30:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.102+0000 7fc236e87740 1 -- 172.21.15.62:0/2035620386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2240dbe50 con 0x5649c24e5370 2024-09-15T23:30:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.102+0000 7fc2157fa640 1 -- 172.21.15.62:0/2035620386 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc21c005430 con 0x5649c24e5370 2024-09-15T23:30:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.106+0000 7fc2157fa640 1 -- 172.21.15.62:0/2035620386 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc21c018680 con 0x5649c24e5370 2024-09-15T23:30:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.106+0000 7fc236e87740 10 client.18690 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:16.107 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.106+0000 7fc236e87740 1 -- 172.21.15.62:0/2035620386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5649c25709f0 con 0x5649c24e5370 2024-09-15T23:30:16.110 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.109+0000 7fc2157fa640 1 -- 172.21.15.62:0/2035620386 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2240dbe50 con 0x5649c24e5370 2024-09-15T23:30:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.110+0000 7fc2157fa640 1 -- 172.21.15.62:0/2035620386 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fc21c0035f0 con 0x5649c24e5370 2024-09-15T23:30:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.110+0000 7fc236e87740 10 client.18690 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.110+0000 7fc236e87740 10 client.18690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.110+0000 7fc236e87740 20 client.18690 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.110+0000 7fc236e87740 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5649c25712f0 0x5649c2573710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.110+0000 7fc236e87740 4 client.18690 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:30:16.111 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.110+0000 7fc236e87740 1 -- 172.21.15.62:0/2035620386 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649c23c7900 con 0x5649c25712f0 2024-09-15T23:30:16.118 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.118+0000 7fc231daa640 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5649c25712f0 0x5649c2573710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.124+0000 7fc231daa640 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5649c25712f0 0x5649c2573710 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=995388e9913778d0 server_cookie=30b81f004d16eceb in_seq=0 out_seq=0 2024-09-15T23:30:16.127 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.124+0000 7fc2157fa640 10 client.18690 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.157+0000 7fc2157fa640 1 -- 172.21.15.62:0/2035620386 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649c23c7900 con 0x5649c25712f0 2024-09-15T23:30:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.157+0000 7fc2157fa640 10 client.18690 handle_command_reply: tid=0 2024-09-15T23:30:16.158 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.157+0000 7fc236e87740 1 client.18690 shutdown 2024-09-15T23:30:16.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 -- 172.21.15.62:0/2035620386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24e5370 msgr2=0x5649c2506e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24e5370 0x5649c2506e50 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fc21c018d40 tx=0x7fc21c018d70 comp rx=0 tx=0).stop 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 -- 172.21.15.62:0/2035620386 shutdown_connections 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5649c25712f0 0x5649c2573710 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5649c24ef630 0x5649c2509e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5649c24e5370 0x5649c2506e50 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 --2- 172.21.15.62:0/2035620386 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5649c24df9a0 0x5649c2506910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 -- 172.21.15.62:0/2035620386 >> 172.21.15.62:0/2035620386 conn(0x5649c24db010 msgr2=0x5649c24e92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc2157fa640 0 client.18690 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 -- 172.21.15.62:0/2035620386 shutdown_connections 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 1 -- 172.21.15.62:0/2035620386 wait complete. 2024-09-15T23:30:16.160 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.158+0000 7fc236e87740 20 client.18690 trim_cache size 0 max 16384 2024-09-15T23:30:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:30:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:30:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:30:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:30:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 -- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2180abd30 msgr2=0x7fc2180ae1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2180abd30 0x7fc2180ae1f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc228005ab0 tx=0x7fc22800c5f0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 -- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc22c1025f0 msgr2=0x7fc22c1b6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc22c1025f0 0x7fc22c1b6950 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fc224009b30 tx=0x7fc224030c70 comp rx=0 tx=0).stop 2024-09-15T23:30:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 -- 172.21.15.62:0/3500578921 shutdown_connections 2024-09-15T23:30:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc2180abd30 0x7fc2180ae1f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc22c1025f0 0x7fc22c1b6950 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc22c101c30 0x7fc22c108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 --2- 172.21.15.62:0/3500578921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc22c100a40 0x7fc22c107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.159+0000 7fc23400e640 1 -- 172.21.15.62:0/3500578921 >> 172.21.15.62:0/3500578921 conn(0x7fc22c0fc1e0 msgr2=0x7fc22c070210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.160+0000 7fc23400e640 1 -- 172.21.15.62:0/3500578921 shutdown_connections 2024-09-15T23:30:16.163 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:16.160+0000 7fc23400e640 1 -- 172.21.15.62:0/3500578921 wait complete. 2024-09-15T23:30:16.175 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:15 smithi062 ceph-mon[30949]: pgmap v789: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 597 op/s 2024-09-15T23:30:16.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:15 smithi067 ceph-mon[35665]: pgmap v789: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 597 op/s 2024-09-15T23:30:16.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:15 smithi148 ceph-mon[35610]: pgmap v789: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 597 op/s 2024-09-15T23:30:16.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.553+0000 7fecab1d3640 1 -- 172.21.15.62:0/274962144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca411b140 msgr2=0x7feca411d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.553+0000 7fecab1d3640 1 --2- 172.21.15.62:0/274962144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca411b140 0x7feca411d530 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fec9c009860 tx=0x7fec9c02a380 comp rx=0 tx=0).stop 2024-09-15T23:30:16.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.559+0000 7fecab1d3640 1 -- 172.21.15.62:0/274962144 shutdown_connections 2024-09-15T23:30:16.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.559+0000 7fecab1d3640 1 --2- 172.21.15.62:0/274962144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca411b140 0x7feca411d530 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.559+0000 7fecab1d3640 1 --2- 172.21.15.62:0/274962144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feca4109830 0x7feca4109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.559+0000 7fecab1d3640 1 --2- 172.21.15.62:0/274962144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feca4109450 0x7feca4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.559+0000 7fecab1d3640 1 -- 172.21.15.62:0/274962144 >> 172.21.15.62:0/274962144 conn(0x7feca406e140 msgr2=0x7feca406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.562+0000 7fecab1d3640 1 -- 172.21.15.62:0/274962144 shutdown_connections 2024-09-15T23:30:16.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.562+0000 7fecab1d3640 1 -- 172.21.15.62:0/274962144 wait complete. 2024-09-15T23:30:16.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.564+0000 7fecab1d3640 1 Processor -- start 2024-09-15T23:30:16.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.564+0000 7fecab1d3640 1 -- start start 2024-09-15T23:30:16.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.564+0000 7fecab1d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feca4109450 0x7feca411a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.565+0000 7fecab1d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca4109830 0x7feca41168d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.565+0000 7feca8f6f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feca4109450 0x7feca411a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.565+0000 7feca8f6f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feca4109450 0x7feca411a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:35972/0 (socket says 172.21.15.62:35972) 2024-09-15T23:30:16.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.565+0000 7feca3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca4109830 0x7feca41168d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.565+0000 7feca3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca4109830 0x7feca41168d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:59062/0 (socket says 172.21.15.62:59062) 2024-09-15T23:30:16.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.565+0000 7fecab1d3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feca411b140 0x7feca4116e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.566+0000 7feca8f6f640 1 -- 172.21.15.62:0/1354053967 learned_addr learned my addr 172.21.15.62:0/1354053967 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.566+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feca41132d0 con 0x7feca411b140 2024-09-15T23:30:16.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.566+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feca4113150 con 0x7feca4109830 2024-09-15T23:30:16.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.566+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7feca4113450 con 0x7feca4109450 2024-09-15T23:30:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.567+0000 7feca8f6f640 1 -- 172.21.15.62:0/1354053967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca4109830 msgr2=0x7feca41168d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.567+0000 7feca8f6f640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca4109830 0x7feca41168d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.567+0000 7feca8f6f640 1 -- 172.21.15.62:0/1354053967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feca411b140 msgr2=0x7feca4116e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.567+0000 7feca8f6f640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feca411b140 0x7feca4116e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.567+0000 7feca8f6f640 1 -- 172.21.15.62:0/1354053967 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca4117540 con 0x7feca4109450 2024-09-15T23:30:16.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.568+0000 7feca8f6f640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feca4109450 0x7feca411a850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec9401cd20 tx=0x7fec9401b6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.569+0000 7feca1ffb640 1 -- 172.21.15.62:0/1354053967 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec94013020 con 0x7feca4109450 2024-09-15T23:30:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.569+0000 7feca1ffb640 1 -- 172.21.15.62:0/1354053967 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec9400fb60 con 0x7feca4109450 2024-09-15T23:30:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.569+0000 7feca1ffb640 1 -- 172.21.15.62:0/1354053967 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9400eb70 con 0x7feca4109450 2024-09-15T23:30:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.569+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca41cd180 con 0x7feca4109450 2024-09-15T23:30:16.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.569+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca41cd580 con 0x7feca4109450 2024-09-15T23:30:16.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.575+0000 7feca1ffb640 1 -- 172.21.15.62:0/1354053967 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fec94016020 con 0x7feca4109450 2024-09-15T23:30:16.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.576+0000 7feca1ffb640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fec900a33d0 0x7fec900a5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.577+0000 7feca1ffb640 1 -- 172.21.15.62:0/1354053967 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fec9409b820 con 0x7feca4109450 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.578+0000 7fecad5ac740 1 Processor -- start 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.578+0000 7fecad5ac740 1 -- start start 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.578+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 0x561cbeb6a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.578+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbeb75140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.578+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbeb77a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.578+0000 7fecad5ac740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561cbead02e0 con 0x561cbeb75680 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.578+0000 7fecad5ac740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561cbea77ce0 con 0x561cbeb6adc0 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.578+0000 7fecad5ac740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561cbeacc2a0 con 0x561cbeb6a4a0 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.579+0000 7feca9770640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbeb77a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.579+0000 7feca9770640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbeb77a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55936/0 (socket says 172.21.15.62:55936) 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.579+0000 7feca9770640 1 -- 172.21.15.62:0/2639547886 learned_addr learned my addr 172.21.15.62:0/2639547886 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.579+0000 7feca8f6f640 1 --2- 172.21.15.62:0/2639547886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 0x561cbeb6a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.580+0000 7feca9770640 1 -- 172.21.15.62:0/2639547886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 msgr2=0x561cbeb6a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.580+0000 7feca9770640 1 --2- 172.21.15.62:0/2639547886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 0x561cbeb6a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.580+0000 7feca9770640 1 -- 172.21.15.62:0/2639547886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 msgr2=0x561cbeb75140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:30:16.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.580+0000 7feca9770640 1 --2- 172.21.15.62:0/2639547886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbeb75140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.580+0000 7feca9770640 1 -- 172.21.15.62:0/2639547886 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cbea5c440 con 0x561cbeb75680 2024-09-15T23:30:16.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.580+0000 7feca9770640 1 --2- 172.21.15.62:0/2639547886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbeb77a70 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fec9c005b60 tx=0x7fec9c0304b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19280b52c35f1a73 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.582+0000 7feca3fff640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fec900a33d0 0x7fec900a5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.583+0000 7feca3fff640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fec900a33d0 0x7fec900a5890 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x561cbeb67890 tx=0x7fec8c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.583+0000 7fec9affd640 1 -- 172.21.15.62:0/2639547886 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c037040 con 0x561cbeb75680 2024-09-15T23:30:16.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.584+0000 7fec9affd640 1 -- 172.21.15.62:0/2639547886 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec9c040070 con 0x561cbeb75680 2024-09-15T23:30:16.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.584+0000 7fec9affd640 1 -- 172.21.15.62:0/2639547886 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c003850 con 0x561cbeb75680 2024-09-15T23:30:16.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.585+0000 7fecad5ac740 1 -- 172.21.15.62:0/2639547886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 msgr2=0x561cbeb77a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.585+0000 7fecad5ac740 1 --2- 172.21.15.62:0/2639547886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbeb77a70 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fec9c005b60 tx=0x7fec9c0304b0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.590+0000 7fecad5ac740 1 -- 172.21.15.62:0/2639547886 shutdown_connections 2024-09-15T23:30:16.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.590+0000 7fecad5ac740 1 --2- 172.21.15.62:0/2639547886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbeb77a70 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.590+0000 7fecad5ac740 1 --2- 172.21.15.62:0/2639547886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbeb75140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.590+0000 7fecad5ac740 1 --2- 172.21.15.62:0/2639547886 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 0x561cbeb6a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.590+0000 7fecad5ac740 1 -- 172.21.15.62:0/2639547886 >> 172.21.15.62:0/2639547886 conn(0x561cbeb66a30 msgr2=0x561cbeb66e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.590+0000 7fecad5ac740 1 -- 172.21.15.62:0/2639547886 shutdown_connections 2024-09-15T23:30:16.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.590+0000 7fecad5ac740 1 -- 172.21.15.62:0/2639547886 wait complete. 2024-09-15T23:30:16.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.591+0000 7fecad5ac740 1 -- start start 2024-09-15T23:30:16.592 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.591+0000 7fecad5ac740 1 Processor -- start 2024-09-15T23:30:16.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.592+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 0x561cbeb74f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.592+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6adc0 0x561cbeb72230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.592+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb75680 0x561cbeb72770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.592+0000 7fecad5ac740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x561cbeb9a2b0 con 0x561cbeb6a4a0 2024-09-15T23:30:16.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.593+0000 7feca9770640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb75680 0x561cbeb72770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.593+0000 7feca9770640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb75680 0x561cbeb72770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:59074/0 (socket says 172.21.15.62:59074) 2024-09-15T23:30:16.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.593+0000 7feca3fff640 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6adc0 0x561cbeb72230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.593+0000 7feca9770640 1 -- 172.21.15.62:0/728078984 learned_addr learned my addr 172.21.15.62:0/728078984 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.593+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561cbea8b050 con 0x561cbeb75680 2024-09-15T23:30:16.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.593+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x561cbeb9a430 con 0x561cbeb6adc0 2024-09-15T23:30:16.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.593+0000 7fecad5ac740 10 client.0 _ll_register_callbacks cb 0x561cbeacb160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:16.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.595+0000 7feca8f6f640 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 0x561cbeb74f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.595+0000 7feca9770640 1 -- 172.21.15.62:0/728078984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6adc0 msgr2=0x561cbeb72230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.595+0000 7feca9770640 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6adc0 0x561cbeb72230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.595+0000 7feca9770640 1 -- 172.21.15.62:0/728078984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 msgr2=0x561cbeb74f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.595+0000 7feca9770640 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 0x561cbeb74f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.595+0000 7feca9770640 1 -- 172.21.15.62:0/728078984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cbe8413c0 con 0x561cbeb75680 2024-09-15T23:30:16.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.596+0000 7fec83fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:30:16.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.596+0000 7feca9770640 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb75680 0x561cbeb72770 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fec9c02a970 tx=0x7fec9c0328b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5aba2cc97ebd7268 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.596+0000 7fec83fff640 1 -- 172.21.15.62:0/728078984 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9c037040 con 0x561cbeb75680 2024-09-15T23:30:16.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.596+0000 7fec83fff640 1 -- 172.21.15.62:0/728078984 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fec9c0305b0 con 0x561cbeb75680 2024-09-15T23:30:16.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.597+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec9c040070 con 0x561cbeb75680 2024-09-15T23:30:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.597+0000 7fec83fff640 1 -- 172.21.15.62:0/728078984 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fec9c037040 con 0x561cbeb75680 2024-09-15T23:30:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.597+0000 7fecad5ac740 10 client.28211 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:16.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.597+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561cbeba5ca0 con 0x561cbeb75680 2024-09-15T23:30:16.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.599+0000 7fec83fff640 1 -- 172.21.15.62:0/728078984 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec9c040070 con 0x561cbeb75680 2024-09-15T23:30:16.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.599+0000 7fec83fff640 1 -- 172.21.15.62:0/728078984 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fec9c03a070 con 0x561cbeb75680 2024-09-15T23:30:16.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.600+0000 7fecad5ac740 10 client.28211 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:16.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.600+0000 7fecad5ac740 10 client.28211 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:16.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.600+0000 7fecad5ac740 20 client.28211 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:16.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.600+0000 7fecad5ac740 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561cbeba6ff0 0x561cbeba93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.600+0000 7fecad5ac740 4 client.28211 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:16.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.600+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cbeadd850 con 0x561cbeba6ff0 2024-09-15T23:30:16.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.600+0000 7feca8f6f640 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561cbeba6ff0 0x561cbeba93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.601+0000 7fec83fff640 10 client.28211 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:16.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.601+0000 7feca8f6f640 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561cbeba6ff0 0x561cbeba93e0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22c632a87937852d server_cookie=9fc9ca8872fad467 in_seq=0 out_seq=0 2024-09-15T23:30:16.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.602+0000 7fec83fff640 1 -- 172.21.15.62:0/728078984 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561cbeadd850 con 0x561cbeba6ff0 2024-09-15T23:30:16.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.602+0000 7fec83fff640 10 client.28211 handle_command_reply: tid=0 2024-09-15T23:30:16.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.602+0000 7fecad5ac740 1 client.28211 shutdown 2024-09-15T23:30:16.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb75680 msgr2=0x561cbeb72770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb75680 0x561cbeb72770 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fec9c02a970 tx=0x7fec9c0328b0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 shutdown_connections 2024-09-15T23:30:16.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561cbeba6ff0 0x561cbeba93e0 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fec83fff640 0 client.28211 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb75680 0x561cbeb72770 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6adc0 0x561cbeb72230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 --2- 172.21.15.62:0/728078984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 0x561cbeb74f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 >> 172.21.15.62:0/728078984 conn(0x561cbeb66a30 msgr2=0x561cbeb6c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 shutdown_connections 2024-09-15T23:30:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.603+0000 7fecad5ac740 1 -- 172.21.15.62:0/728078984 wait complete. 2024-09-15T23:30:16.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.604+0000 7fecad5ac740 20 client.28211 trim_cache size 0 max 16384 2024-09-15T23:30:16.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.634+0000 7fecad5ac740 1 Processor -- start 2024-09-15T23:30:16.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.634+0000 7fecad5ac740 1 -- start start 2024-09-15T23:30:16.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.634+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 0x561cbeb6d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.634+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebac400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.634+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb75680 0x561cbebac940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.634+0000 7fecad5ac740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feca40807a0 con 0x561cbeb6a4a0 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.634+0000 7fecad5ac740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feca407e040 con 0x561cbeb6adc0 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.634+0000 7fecad5ac740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7feca407be30 con 0x561cbeb75680 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca8f6f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 0x561cbeb6d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebac400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:59084/0 (socket says 172.21.15.62:59084) 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca8f6f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 0x561cbeb6d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55948/0 (socket says 172.21.15.62:55948) 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca3fff640 1 -- 172.21.15.62:0/4197627474 learned_addr learned my addr 172.21.15.62:0/4197627474 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca9770640 1 --2- 172.21.15.62:0/4197627474 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb75680 0x561cbebac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca3fff640 1 -- 172.21.15.62:0/4197627474 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb75680 msgr2=0x561cbebac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca3fff640 1 --2- 172.21.15.62:0/4197627474 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb75680 0x561cbebac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca3fff640 1 -- 172.21.15.62:0/4197627474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 msgr2=0x561cbeb6d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca3fff640 1 --2- 172.21.15.62:0/4197627474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 0x561cbeb6d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca3fff640 1 -- 172.21.15.62:0/4197627474 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cbeb9a300 con 0x561cbeb6adc0 2024-09-15T23:30:16.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.635+0000 7feca3fff640 1 --2- 172.21.15.62:0/4197627474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebac400 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00d800 tx=0x7fec8c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8650583a7061c5ba server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.636+0000 7fec9a7fc640 1 -- 172.21.15.62:0/4197627474 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c0149b0 con 0x561cbeb6adc0 2024-09-15T23:30:16.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.636+0000 7fec9a7fc640 1 -- 172.21.15.62:0/4197627474 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec8c025070 con 0x561cbeb6adc0 2024-09-15T23:30:16.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.636+0000 7fec9a7fc640 1 -- 172.21.15.62:0/4197627474 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c0123e0 con 0x561cbeb6adc0 2024-09-15T23:30:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.637+0000 7fecad5ac740 1 -- 172.21.15.62:0/4197627474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 msgr2=0x561cbebac400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.637+0000 7fecad5ac740 1 --2- 172.21.15.62:0/4197627474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebac400 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00d800 tx=0x7fec8c013000 comp rx=0 tx=0).stop 2024-09-15T23:30:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.639+0000 7fecad5ac740 1 -- 172.21.15.62:0/4197627474 shutdown_connections 2024-09-15T23:30:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.639+0000 7fecad5ac740 1 --2- 172.21.15.62:0/4197627474 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb75680 0x561cbebac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.639+0000 7fecad5ac740 1 --2- 172.21.15.62:0/4197627474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebac400 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.639+0000 7fecad5ac740 1 --2- 172.21.15.62:0/4197627474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb6a4a0 0x561cbeb6d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.639+0000 7fecad5ac740 1 -- 172.21.15.62:0/4197627474 >> 172.21.15.62:0/4197627474 conn(0x561cbeb66a30 msgr2=0x561cbeb6d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.640+0000 7fecad5ac740 1 -- 172.21.15.62:0/4197627474 shutdown_connections 2024-09-15T23:30:16.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.640+0000 7fecad5ac740 1 -- 172.21.15.62:0/4197627474 wait complete. 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.640+0000 7fecad5ac740 1 -- start start 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.640+0000 7fecad5ac740 1 Processor -- start 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 0x561cbebaa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebaa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7fecad5ac740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbebaada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7fecad5ac740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7feca407c500 con 0x561cbeb75680 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7fecad5ac740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feca407d970 con 0x561cbeb6adc0 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7fecad5ac740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7feca407cbd0 con 0x561cbeb6a4a0 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7fecad5ac740 10 client.0 _ll_register_callbacks cb 0x7fec9c035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebaa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca3fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebaa860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:59086/0 (socket says 172.21.15.62:59086) 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca3fff640 1 -- 172.21.15.62:0/1363297616 learned_addr learned my addr 172.21.15.62:0/1363297616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca8f6f640 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 0x561cbebaa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca9770640 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbebaada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca8f6f640 1 -- 172.21.15.62:0/1363297616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 msgr2=0x561cbebaa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca8f6f640 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebaa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca8f6f640 1 -- 172.21.15.62:0/1363297616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 msgr2=0x561cbebaada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca8f6f640 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbebaada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca8f6f640 1 -- 172.21.15.62:0/1363297616 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cbec10ad0 con 0x561cbeb6a4a0 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.641+0000 7feca3fff640 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebaa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.642+0000 7feca8f6f640 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 0x561cbebaa320 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fec940cec30 tx=0x7fec940a5ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cb01bfee93cfa62 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:16.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.642+0000 7fec83fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.642+0000 7fec83fff640 1 -- 172.21.15.62:0/1363297616 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec9409a1b0 con 0x561cbeb6a4a0 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.642+0000 7fec83fff640 1 -- 172.21.15.62:0/1363297616 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fec9409b820 con 0x561cbeb6a4a0 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.642+0000 7fecad5ac740 1 -- 172.21.15.62:0/1363297616 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec8c025070 con 0x561cbeb6a4a0 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.642+0000 7fec83fff640 1 -- 172.21.15.62:0/1363297616 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fec9409a350 con 0x561cbeb6a4a0 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fecad5ac740 10 client.27886 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fecad5ac740 1 -- 172.21.15.62:0/1363297616 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561cbec11dc0 con 0x561cbeb6a4a0 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fec83fff640 1 -- 172.21.15.62:0/1363297616 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec8c025070 con 0x561cbeb6a4a0 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fec83fff640 1 -- 172.21.15.62:0/1363297616 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fec94099040 con 0x561cbeb6a4a0 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fecad5ac740 10 client.27886 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fecad5ac740 10 client.27886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:16.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fecad5ac740 20 client.27886 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fecad5ac740 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561cbec12450 0x561cbec14870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fecad5ac740 4 client.27886 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.643+0000 7fecad5ac740 1 -- 172.21.15.62:0/1363297616 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561cbead7d60 con 0x561cbec12450 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.644+0000 7feca3fff640 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561cbec12450 0x561cbec14870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.644+0000 7fec83fff640 10 client.27886 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.644+0000 7feca3fff640 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561cbec12450 0x561cbec14870 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10c68113ed6edea4 server_cookie=8f362db21f0ef23 in_seq=0 out_seq=0 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.644+0000 7fec83fff640 1 -- 172.21.15.62:0/1363297616 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561cbead7d60 con 0x561cbec12450 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.644+0000 7fec83fff640 10 client.27886 handle_command_reply: tid=0 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.644+0000 7fecad5ac740 1 client.27886 shutdown 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 -- 172.21.15.62:0/1363297616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 msgr2=0x561cbebaa320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 0x561cbebaa320 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fec940cec30 tx=0x7fec940a5ef0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 -- 172.21.15.62:0/1363297616 shutdown_connections 2024-09-15T23:30:16.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x561cbec12450 0x561cbec14870 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fec83fff640 0 client.27886 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561cbeb75680 0x561cbebaada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561cbeb6adc0 0x561cbebaa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 --2- 172.21.15.62:0/1363297616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x561cbeb6a4a0 0x561cbebaa320 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 -- 172.21.15.62:0/1363297616 >> 172.21.15.62:0/1363297616 conn(0x561cbeb66a30 msgr2=0x561cbeb742b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 -- 172.21.15.62:0/1363297616 shutdown_connections 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 1 -- 172.21.15.62:0/1363297616 wait complete. 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.645+0000 7fecad5ac740 20 client.27886 trim_cache size 0 max 16384 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:30:16.649 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:30:16.650 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:30:16.650 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fec900a33d0 msgr2=0x7fec900a5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fec900a33d0 0x7fec900a5890 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x561cbeb67890 tx=0x7fec8c008040 comp rx=0 tx=0).stop 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feca4109450 msgr2=0x7feca411a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feca4109450 0x7feca411a850 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec9401cd20 tx=0x7fec9401b6d0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 shutdown_connections 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fec900a33d0 0x7fec900a5890 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feca411b140 0x7feca4116e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feca4109830 0x7feca41168d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 --2- 172.21.15.62:0/1354053967 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7feca4109450 0x7feca411a850 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:16.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.646+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 >> 172.21.15.62:0/1354053967 conn(0x7feca406e140 msgr2=0x7feca4070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:16.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.647+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 shutdown_connections 2024-09-15T23:30:16.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:16.647+0000 7fecab1d3640 1 -- 172.21.15.62:0/1354053967 wait complete. 2024-09-15T23:30:17.549 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:30:17.549 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:30:18.105 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:18.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:17 smithi062 ceph-mon[30949]: pgmap v790: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.1 MiB/s wr, 618 op/s 2024-09-15T23:30:18.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:17 smithi067 ceph-mon[35665]: pgmap v790: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.1 MiB/s wr, 618 op/s 2024-09-15T23:30:18.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:17 smithi148 ceph-mon[35610]: pgmap v790: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.1 MiB/s wr, 618 op/s 2024-09-15T23:30:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.729+0000 7f16a61ba640 1 -- 172.21.15.62:0/4010498321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a0102570 msgr2=0x7f16a01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.729+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4010498321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a0102570 0x7f16a01029b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1688009900 tx=0x7f168802a480 comp rx=0 tx=0).stop 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.729+0000 7f16a61ba640 1 -- 172.21.15.62:0/4010498321 shutdown_connections 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.729+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4010498321 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a0102ef0 0x7f16a010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.729+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4010498321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a0102570 0x7f16a01029b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.729+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4010498321 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a0100e20 0x7f16a0101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.729+0000 7f16a61ba640 1 -- 172.21.15.62:0/4010498321 >> 172.21.15.62:0/4010498321 conn(0x7f16a00fc7d0 msgr2=0x7f16a00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.730+0000 7f16a61ba640 1 -- 172.21.15.62:0/4010498321 shutdown_connections 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.730+0000 7f16a61ba640 1 -- 172.21.15.62:0/4010498321 wait complete. 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.730+0000 7f16a61ba640 1 Processor -- start 2024-09-15T23:30:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f16a61ba640 1 -- start start 2024-09-15T23:30:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f16a61ba640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a0100e20 0x7f16a01162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f16a61ba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a0102570 0x7f16a0116820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f16a61ba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a0102ef0 0x7f16a011c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f16a61ba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f16a010fe80 con 0x7f16a0102ef0 2024-09-15T23:30:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f16a61ba640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f16a010fd00 con 0x7f16a0100e20 2024-09-15T23:30:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f16a61ba640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f16a0110000 con 0x7f16a0102570 2024-09-15T23:30:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f1696ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a0102570 0x7f16a0116820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f1696ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a0102570 0x7f16a0116820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36042/0 (socket says 172.21.15.62:36042) 2024-09-15T23:30:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f1696ffd640 1 -- 172.21.15.62:0/4001138166 learned_addr learned my addr 172.21.15.62:0/4001138166 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f169ffff640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a0102ef0 0x7f16a011c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.731+0000 7f169f7fe640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a0100e20 0x7f16a01162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f169ffff640 1 -- 172.21.15.62:0/4001138166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a0102570 msgr2=0x7f16a0116820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f169ffff640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a0102570 0x7f16a0116820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f169ffff640 1 -- 172.21.15.62:0/4001138166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a0100e20 msgr2=0x7f16a01162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f169ffff640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a0100e20 0x7f16a01162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f169ffff640 1 -- 172.21.15.62:0/4001138166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a011c8c0 con 0x7f16a0102ef0 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f1696ffd640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a0102570 0x7f16a0116820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f169ffff640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a0102ef0 0x7f16a011c000 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f169800e970 tx=0x7f169800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f169d7fa640 1 -- 172.21.15.62:0/4001138166 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169800ccc0 con 0x7f16a0102ef0 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a011cbb0 con 0x7f16a0102ef0 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f169d7fa640 1 -- 172.21.15.62:0/4001138166 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1698005020 con 0x7f16a0102ef0 2024-09-15T23:30:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.732+0000 7f169d7fa640 1 -- 172.21.15.62:0/4001138166 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698004a00 con 0x7f16a0102ef0 2024-09-15T23:30:18.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.733+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a01117e0 con 0x7f16a0102ef0 2024-09-15T23:30:18.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.734+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a01122c0 con 0x7f16a0102ef0 2024-09-15T23:30:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.736+0000 7f169d7fa640 1 -- 172.21.15.62:0/4001138166 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f16980051d0 con 0x7f16a0102ef0 2024-09-15T23:30:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.737+0000 7f169d7fa640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f168c0a3340 0x7f168c0a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.737+0000 7f169d7fa640 1 -- 172.21.15.62:0/4001138166 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f16980cfb90 con 0x7f16a0102ef0 2024-09-15T23:30:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.737+0000 7f169f7fe640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f168c0a3340 0x7f168c0a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.738+0000 7f169f7fe640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f168c0a3340 0x7f168c0a5800 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1690005fd0 tx=0x7f1690009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:18.742+0000 7f169d7fa640 1 -- 172.21.15.62:0/4001138166 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f169809c4d0 con 0x7f16a0102ef0 2024-09-15T23:30:19.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.131+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16a0112d90 con 0x7f16a0102ef0 2024-09-15T23:30:19.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.132+0000 7f169d7fa640 1 -- 172.21.15.62:0/4001138166 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7f169809bdd0 con 0x7f16a0102ef0 2024-09-15T23:30:19.134 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:30:19.134 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:30:19.134 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.135+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f168c0a3340 msgr2=0x7f168c0a5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.135+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f168c0a3340 0x7f168c0a5800 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1690005fd0 tx=0x7f1690009500 comp rx=0 tx=0).stop 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.135+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a0102ef0 msgr2=0x7f16a011c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.135+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a0102ef0 0x7f16a011c000 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f169800e970 tx=0x7f169800ee30 comp rx=0 tx=0).stop 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.136+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 shutdown_connections 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.136+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f168c0a3340 0x7f168c0a5800 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.136+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16a0102ef0 0x7f16a011c000 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.136+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f16a0102570 0x7f16a0116820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.136+0000 7f16a61ba640 1 --2- 172.21.15.62:0/4001138166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16a0100e20 0x7f16a01162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.136+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 >> 172.21.15.62:0/4001138166 conn(0x7f16a00fc7d0 msgr2=0x7f16a0104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.136+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 shutdown_connections 2024-09-15T23:30:19.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:19.136+0000 7f16a61ba640 1 -- 172.21.15.62:0/4001138166 wait complete. 2024-09-15T23:30:19.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 840.0 s, 502.4 tps, lat 201.707 ms stddev 238.834 2024-09-15T23:30:19.941 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:30:20.091 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:19 smithi148 ceph-mon[35610]: pgmap v791: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.3 MiB/s wr, 570 op/s 2024-09-15T23:30:20.091 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:19 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/4001138166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:20.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:19 smithi062 ceph-mon[30949]: pgmap v791: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.3 MiB/s wr, 570 op/s 2024-09-15T23:30:20.233 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:19 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/4001138166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:20.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:19 smithi067 ceph-mon[35665]: pgmap v791: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.3 MiB/s wr, 570 op/s 2024-09-15T23:30:20.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:19 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/4001138166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:20.435 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:21.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.228+0000 7f273ffff640 1 -- 172.21.15.62:0/787570091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2740101270 msgr2=0x7f274010d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:21.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.228+0000 7f273ffff640 1 --2- 172.21.15.62:0/787570091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2740101270 0x7f274010d7a0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f273801ace0 tx=0x7f273803b990 comp rx=0 tx=0).stop 2024-09-15T23:30:21.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.229+0000 7f273ffff640 1 -- 172.21.15.62:0/787570091 shutdown_connections 2024-09-15T23:30:21.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.229+0000 7f273ffff640 1 --2- 172.21.15.62:0/787570091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2740101270 0x7f274010d7a0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.229+0000 7f273ffff640 1 --2- 172.21.15.62:0/787570091 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27401008d0 0x7f2740100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.229+0000 7f273ffff640 1 --2- 172.21.15.62:0/787570091 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f27401068d0 0x7f2740106cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.229+0000 7f273ffff640 1 -- 172.21.15.62:0/787570091 >> 172.21.15.62:0/787570091 conn(0x7f27400fc610 msgr2=0x7f27400fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.230+0000 7f273ffff640 1 -- 172.21.15.62:0/787570091 shutdown_connections 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.230+0000 7f273ffff640 1 -- 172.21.15.62:0/787570091 wait complete. 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.231+0000 7f273ffff640 1 Processor -- start 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.231+0000 7f273ffff640 1 -- start start 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.231+0000 7f273ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27401008d0 0x7f274010d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.231+0000 7f273ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2740101270 0x7f2740108350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.231+0000 7f273ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27401068d0 0x7f2740108890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.231+0000 7f273ffff640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f274006e660 con 0x7f27401068d0 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.231+0000 7f273ffff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f274006e4e0 con 0x7f27401008d0 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.231+0000 7f273ffff640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f274006e7e0 con 0x7f2740101270 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.231+0000 7f273f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27401068d0 0x7f2740108890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.232+0000 7f273f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27401068d0 0x7f2740108890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55994/0 (socket says 172.21.15.62:55994) 2024-09-15T23:30:21.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.232+0000 7f273f7fe640 1 -- 172.21.15.62:0/3972879105 learned_addr learned my addr 172.21.15.62:0/3972879105 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.232+0000 7f273f7fe640 1 -- 172.21.15.62:0/3972879105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2740101270 msgr2=0x7f2740108350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.232+0000 7f273f7fe640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2740101270 0x7f2740108350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.232+0000 7f273f7fe640 1 -- 172.21.15.62:0/3972879105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27401008d0 msgr2=0x7f274010d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.232+0000 7f273f7fe640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27401008d0 0x7f274010d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.232+0000 7f273f7fe640 1 -- 172.21.15.62:0/3972879105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2740108e00 con 0x7f27401068d0 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.232+0000 7f273f7fe640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27401068d0 0x7f2740108890 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f2738004af0 tx=0x7f2738041e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.233+0000 7f2744941640 1 -- 172.21.15.62:0/3972879105 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738045070 con 0x7f27401068d0 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.233+0000 7f2744941640 1 -- 172.21.15.62:0/3972879105 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2738040be0 con 0x7f27401068d0 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.233+0000 7f2744941640 1 -- 172.21.15.62:0/3972879105 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273804a690 con 0x7f27401068d0 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.234+0000 7f273ffff640 1 -- 172.21.15.62:0/3972879105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2740109090 con 0x7f27401068d0 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.234+0000 7f273ffff640 1 -- 172.21.15.62:0/3972879105 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f274019a2b0 con 0x7f27401068d0 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.241+0000 7f2747d1c740 1 Processor -- start 2024-09-15T23:30:21.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.242+0000 7f2747d1c740 1 -- start start 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.242+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e88cf0 0x5602d1e87240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.242+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 0x5602d1e87bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.242+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e92520 0x5602d1e94910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.242+0000 7f2747d1c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5602d1ded2f0 con 0x5602d1e87780 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.242+0000 7f2747d1c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5602d1d94cf0 con 0x5602d1e88cf0 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.242+0000 7f2747d1c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5602d1de92b0 con 0x5602d1e92520 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.242+0000 7f2744941640 1 -- 172.21.15.62:0/3972879105 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2738048020 con 0x7f27401068d0 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.243+0000 7f2744941640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27280a3380 0x7f27280a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.244+0000 7f273f7fe640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27280a3380 0x7f27280a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.244+0000 7f273effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 0x5602d1e87bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.244+0000 7f273effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 0x5602d1e87bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56004/0 (socket says 172.21.15.62:56004) 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.244+0000 7f273effd640 1 -- 172.21.15.62:0/1618403427 learned_addr learned my addr 172.21.15.62:0/1618403427 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:21.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.245+0000 7f273f7fe640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27280a3380 0x7f27280a5840 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2738041eb0 tx=0x7f273803b990 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.245+0000 7f2744941640 1 -- 172.21.15.62:0/3972879105 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f27381027f0 con 0x7f27401068d0 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.247+0000 7f273effd640 1 -- 172.21.15.62:0/1618403427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e92520 msgr2=0x5602d1e94910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.247+0000 7f273effd640 1 --2- 172.21.15.62:0/1618403427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e92520 0x5602d1e94910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.247+0000 7f273effd640 1 -- 172.21.15.62:0/1618403427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e88cf0 msgr2=0x5602d1e87240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.247+0000 7f273effd640 1 --2- 172.21.15.62:0/1618403427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e88cf0 0x5602d1e87240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.247+0000 7f273effd640 1 -- 172.21.15.62:0/1618403427 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d1d793a0 con 0x5602d1e87780 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.264+0000 7f273effd640 1 --2- 172.21.15.62:0/1618403427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 0x5602d1e87bc0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2730009770 tx=0x7f2730009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfeed673f2cacfb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.269+0000 7f2725ffb640 1 -- 172.21.15.62:0/1618403427 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730015020 con 0x5602d1e87780 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.269+0000 7f2725ffb640 1 -- 172.21.15.62:0/1618403427 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2730011420 con 0x5602d1e87780 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.269+0000 7f2725ffb640 1 -- 172.21.15.62:0/1618403427 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27300105e0 con 0x5602d1e87780 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.270+0000 7f2747d1c740 1 -- 172.21.15.62:0/1618403427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 msgr2=0x5602d1e87bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.270+0000 7f2747d1c740 1 --2- 172.21.15.62:0/1618403427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 0x5602d1e87bc0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f2730009770 tx=0x7f2730009c30 comp rx=0 tx=0).stop 2024-09-15T23:30:21.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.270+0000 7f2747d1c740 1 -- 172.21.15.62:0/1618403427 shutdown_connections 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.270+0000 7f2747d1c740 1 --2- 172.21.15.62:0/1618403427 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e92520 0x5602d1e94910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.270+0000 7f2747d1c740 1 --2- 172.21.15.62:0/1618403427 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 0x5602d1e87bc0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.270+0000 7f2747d1c740 1 --2- 172.21.15.62:0/1618403427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e88cf0 0x5602d1e87240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.270+0000 7f2747d1c740 1 -- 172.21.15.62:0/1618403427 >> 172.21.15.62:0/1618403427 conn(0x5602d1e50310 msgr2=0x5602d1e83e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.270+0000 7f2747d1c740 1 -- 172.21.15.62:0/1618403427 shutdown_connections 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.273+0000 7f2747d1c740 1 -- 172.21.15.62:0/1618403427 wait complete. 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.273+0000 7f2747d1c740 1 -- start start 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.273+0000 7f2747d1c740 1 Processor -- start 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.278+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 0x5602d1e91f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.278+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e88cf0 0x5602d1e8f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.278+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e92520 0x5602d1e8f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.278+0000 7f2747d1c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5602d1e95220 con 0x5602d1e92520 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.278+0000 7f2747d1c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5602d1da8060 con 0x5602d1e88cf0 2024-09-15T23:30:21.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.278+0000 7f2747d1c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5602d1e953a0 con 0x5602d1e87780 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.278+0000 7f2747d1c740 10 client.0 _ll_register_callbacks cb 0x5602d1afb8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.278+0000 7f273e7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e92520 0x5602d1e8f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.279+0000 7f273e7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e92520 0x5602d1e8f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56020/0 (socket says 172.21.15.62:56020) 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.279+0000 7f273e7fc640 1 -- 172.21.15.62:0/2203902655 learned_addr learned my addr 172.21.15.62:0/2203902655 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.279+0000 7f273e7fc640 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 0x5602d1e91f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.279+0000 7f273e7fc640 1 -- 172.21.15.62:0/2203902655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 msgr2=0x5602d1e91f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.279+0000 7f273e7fc640 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 0x5602d1e91f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.279+0000 7f273e7fc640 1 -- 172.21.15.62:0/2203902655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e88cf0 msgr2=0x5602d1e8f220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.279+0000 7f273e7fc640 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e88cf0 0x5602d1e8f220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.279+0000 7f273e7fc640 1 -- 172.21.15.62:0/2203902655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d1b5e3c0 con 0x5602d1e92520 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.283+0000 7f273e7fc640 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e92520 0x5602d1e8f9b0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2734015a40 tx=0x7f2734009dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520cb5e138c05272 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.284+0000 7f271effd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:30:21.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.284+0000 7f271effd640 1 -- 172.21.15.62:0/2203902655 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734012040 con 0x5602d1e92520 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.284+0000 7f2747d1c740 1 -- 172.21.15.62:0/2203902655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2730011420 con 0x5602d1e92520 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.284+0000 7f271effd640 1 -- 172.21.15.62:0/2203902655 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2734007500 con 0x5602d1e92520 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.287+0000 7f271effd640 1 -- 172.21.15.62:0/2203902655 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2734022050 con 0x5602d1e92520 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.291+0000 7f2747d1c740 10 client.18723 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.291+0000 7f2747d1c740 1 -- 172.21.15.62:0/2203902655 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5602d1ea0e50 con 0x5602d1e92520 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.295+0000 7f271effd640 1 -- 172.21.15.62:0/2203902655 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2730011420 con 0x5602d1e92520 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.295+0000 7f271effd640 1 -- 172.21.15.62:0/2203902655 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f2734009030 con 0x5602d1e92520 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.296+0000 7f2747d1c740 10 client.18723 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.296+0000 7f2747d1c740 10 client.18723 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.296+0000 7f2747d1c740 20 client.18723 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.296+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5602d1ea2310 0x5602d1ea4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.296+0000 7f2747d1c740 4 client.18723 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:21.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.296+0000 7f2747d1c740 1 -- 172.21.15.62:0/2203902655 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602d1dfa860 con 0x5602d1ea2310 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.297+0000 7f273effd640 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5602d1ea2310 0x5602d1ea4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.310+0000 7f273effd640 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5602d1ea2310 0x5602d1ea4740 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3408f4339fb6703 server_cookie=6edbe82d1361ddf7 in_seq=0 out_seq=0 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.310+0000 7f271effd640 10 client.18723 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.314+0000 7f271effd640 1 -- 172.21.15.62:0/2203902655 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5602d1dfa860 con 0x5602d1ea2310 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.314+0000 7f271effd640 10 client.18723 handle_command_reply: tid=0 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.316+0000 7f2747d1c740 1 client.18723 shutdown 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.317+0000 7f2747d1c740 1 -- 172.21.15.62:0/2203902655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e92520 msgr2=0x5602d1e8f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.317+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e92520 0x5602d1e8f9b0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f2734015a40 tx=0x7f2734009dd0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.319+0000 7f2747d1c740 1 -- 172.21.15.62:0/2203902655 shutdown_connections 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.319+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5602d1ea2310 0x5602d1ea4740 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.319+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e92520 0x5602d1e8f9b0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.319+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e88cf0 0x5602d1e8f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.319+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2203902655 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 0x5602d1e91f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.320+0000 7f2747d1c740 1 -- 172.21.15.62:0/2203902655 >> 172.21.15.62:0/2203902655 conn(0x5602d1e50310 msgr2=0x5602d1e89ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.320+0000 7f271effd640 0 client.18723 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.320+0000 7f2747d1c740 1 -- 172.21.15.62:0/2203902655 shutdown_connections 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.320+0000 7f2747d1c740 1 -- 172.21.15.62:0/2203902655 wait complete. 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.323+0000 7f2747d1c740 20 client.18723 trim_cache size 0 max 16384 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.352+0000 7f2747d1c740 1 Processor -- start 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f2747d1c740 1 -- start start 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 0x5602d1e8f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e88cf0 0x5602d1e8a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 0x5602d1e8a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f273e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 0x5602d1e8f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f273e7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 0x5602d1e8f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36058/0 (socket says 172.21.15.62:36058) 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f273effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e88cf0 0x5602d1e8a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f273effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e88cf0 0x5602d1e8a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56022/0 (socket says 172.21.15.62:56022) 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f273e7fc640 1 -- 172.21.15.62:0/299122248 learned_addr learned my addr 172.21.15.62:0/299122248 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f2747d1c740 1 -- 172.21.15.62:0/299122248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f272c012530 con 0x5602d1e88cf0 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f2747d1c740 1 -- 172.21.15.62:0/299122248 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00fdd0 con 0x5602d1e92520 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f2747d1c740 1 -- 172.21.15.62:0/299122248 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00dc20 con 0x5602d1e87780 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.357+0000 7f273e7fc640 1 --2- 172.21.15.62:0/299122248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 0x5602d1e8a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.362+0000 7f273effd640 1 -- 172.21.15.62:0/299122248 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 msgr2=0x5602d1e8f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.362+0000 7f273effd640 1 --2- 172.21.15.62:0/299122248 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 0x5602d1e8f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.362+0000 7f273effd640 1 -- 172.21.15.62:0/299122248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 msgr2=0x5602d1e8a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.362+0000 7f273effd640 1 --2- 172.21.15.62:0/299122248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 0x5602d1e8a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.362+0000 7f273effd640 1 -- 172.21.15.62:0/299122248 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602d1e8d770 con 0x5602d1e88cf0 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.371+0000 7f273effd640 1 --2- 172.21.15.62:0/299122248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e88cf0 0x5602d1e8a2d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x5602d1e90860 tx=0x7f2730009110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7db42d8f899bb38 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.372+0000 7f27257fa640 1 -- 172.21.15.62:0/299122248 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730029070 con 0x5602d1e88cf0 2024-09-15T23:30:21.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.372+0000 7f27257fa640 1 -- 172.21.15.62:0/299122248 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f273000bdb0 con 0x5602d1e88cf0 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.372+0000 7f27257fa640 1 -- 172.21.15.62:0/299122248 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730015040 con 0x5602d1e88cf0 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.374+0000 7f2747d1c740 1 -- 172.21.15.62:0/299122248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e88cf0 msgr2=0x5602d1e8a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.374+0000 7f2747d1c740 1 --2- 172.21.15.62:0/299122248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e88cf0 0x5602d1e8a2d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x5602d1e90860 tx=0x7f2730009110 comp rx=0 tx=0).stop 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.377+0000 7f2747d1c740 1 -- 172.21.15.62:0/299122248 shutdown_connections 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.377+0000 7f2747d1c740 1 --2- 172.21.15.62:0/299122248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 0x5602d1e8a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.377+0000 7f2747d1c740 1 --2- 172.21.15.62:0/299122248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e88cf0 0x5602d1e8a2d0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.377+0000 7f2747d1c740 1 --2- 172.21.15.62:0/299122248 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e87780 0x5602d1e8f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.377+0000 7f2747d1c740 1 -- 172.21.15.62:0/299122248 >> 172.21.15.62:0/299122248 conn(0x5602d1e50310 msgr2=0x5602d1e8eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.378+0000 7f2747d1c740 1 -- 172.21.15.62:0/299122248 shutdown_connections 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.378+0000 7f2747d1c740 1 -- 172.21.15.62:0/299122248 wait complete. 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.379+0000 7f2747d1c740 1 -- start start 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.379+0000 7f2747d1c740 1 Processor -- start 2024-09-15T23:30:21.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.380+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 0x5602d1e956a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.380+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e88cf0 0x5602d1e95be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.380+0000 7f2747d1c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 0x5602d1e96120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.380+0000 7f2747d1c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00e290 con 0x5602d1e87780 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.380+0000 7f2747d1c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00f700 con 0x5602d1e92520 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.380+0000 7f2747d1c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f272c00e960 con 0x5602d1e88cf0 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.380+0000 7f2747d1c740 10 client.0 _ll_register_callbacks cb 0x7f2734001310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.380+0000 7f273effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e88cf0 0x5602d1e95be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.381+0000 7f273effd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e88cf0 0x5602d1e95be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:36070/0 (socket says 172.21.15.62:36070) 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.381+0000 7f273effd640 1 -- 172.21.15.62:0/2189766342 learned_addr learned my addr 172.21.15.62:0/2189766342 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.381+0000 7f273e7fc640 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 0x5602d1e96120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.381+0000 7f273effd640 1 -- 172.21.15.62:0/2189766342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 msgr2=0x5602d1e96120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.381+0000 7f273effd640 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 0x5602d1e96120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.381+0000 7f273effd640 1 -- 172.21.15.62:0/2189766342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 msgr2=0x5602d1e956a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:21.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.381+0000 7f273effd640 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 0x5602d1e956a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.381+0000 7f273effd640 1 -- 172.21.15.62:0/2189766342 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602d1f0b230 con 0x5602d1e88cf0 2024-09-15T23:30:21.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.382+0000 7f273effd640 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e88cf0 0x5602d1e95be0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2730011e00 tx=0x7f2730009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df33f3ea9ad9385f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:21.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.383+0000 7f271effd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:21.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.388+0000 7f271effd640 1 -- 172.21.15.62:0/2189766342 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730029040 con 0x5602d1e88cf0 2024-09-15T23:30:21.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.388+0000 7f271effd640 1 -- 172.21.15.62:0/2189766342 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f27300118c0 con 0x5602d1e88cf0 2024-09-15T23:30:21.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.389+0000 7f2747d1c740 1 -- 172.21.15.62:0/2189766342 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f273000bdb0 con 0x5602d1e88cf0 2024-09-15T23:30:21.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.395+0000 7f271effd640 1 -- 172.21.15.62:0/2189766342 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2730011ae0 con 0x5602d1e88cf0 2024-09-15T23:30:21.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.397+0000 7f2747d1c740 10 client.27901 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:21.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.397+0000 7f2747d1c740 1 -- 172.21.15.62:0/2189766342 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5602d1f0c530 con 0x5602d1e88cf0 2024-09-15T23:30:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.404+0000 7f271effd640 1 -- 172.21.15.62:0/2189766342 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f273000bdb0 con 0x5602d1e88cf0 2024-09-15T23:30:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.404+0000 7f271effd640 1 -- 172.21.15.62:0/2189766342 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f27300036b0 con 0x5602d1e88cf0 2024-09-15T23:30:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.405+0000 7f2747d1c740 10 client.27901 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.405+0000 7f2747d1c740 10 client.27901 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:30:21.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.405+0000 7f2747d1c740 20 client.27901 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:21.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.405+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5602d1f0c910 0x5602d1f0ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:21.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.405+0000 7f2747d1c740 4 client.27901 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:30:21.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.405+0000 7f2747d1c740 1 -- 172.21.15.62:0/2189766342 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602d1df4d70 con 0x5602d1f0c910 2024-09-15T23:30:21.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.405+0000 7f273e7fc640 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5602d1f0c910 0x5602d1f0ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:21.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.406+0000 7f271effd640 10 client.27901 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:30:21.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.407+0000 7f273e7fc640 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5602d1f0c910 0x5602d1f0ed30 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19be05972dca805a server_cookie=56203c7e64479702 in_seq=0 out_seq=0 2024-09-15T23:30:21.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.408+0000 7f271effd640 1 -- 172.21.15.62:0/2189766342 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5602d1df4d70 con 0x5602d1f0c910 2024-09-15T23:30:21.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.408+0000 7f271effd640 10 client.27901 handle_command_reply: tid=0 2024-09-15T23:30:21.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.408+0000 7f2747d1c740 1 client.27901 shutdown 2024-09-15T23:30:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.409+0000 7f2747d1c740 1 -- 172.21.15.62:0/2189766342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e88cf0 msgr2=0x5602d1e95be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:21.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.409+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e88cf0 0x5602d1e95be0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2730011e00 tx=0x7f2730009700 comp rx=0 tx=0).stop 2024-09-15T23:30:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.410+0000 7f2747d1c740 1 -- 172.21.15.62:0/2189766342 shutdown_connections 2024-09-15T23:30:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.410+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x5602d1f0c910 0x5602d1f0ed30 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.410+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5602d1e92520 0x5602d1e96120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.410+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5602d1e88cf0 0x5602d1e95be0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.410+0000 7f2747d1c740 1 --2- 172.21.15.62:0/2189766342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5602d1e87780 0x5602d1e956a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.410+0000 7f2747d1c740 1 -- 172.21.15.62:0/2189766342 >> 172.21.15.62:0/2189766342 conn(0x5602d1e50310 msgr2=0x5602d1e90a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.410+0000 7f271effd640 0 client.27901 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:30:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.410+0000 7f2747d1c740 1 -- 172.21.15.62:0/2189766342 shutdown_connections 2024-09-15T23:30:21.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.410+0000 7f2747d1c740 1 -- 172.21.15.62:0/2189766342 wait complete. 2024-09-15T23:30:21.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.411+0000 7f2747d1c740 20 client.27901 trim_cache size 0 max 16384 2024-09-15T23:30:21.414 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:30:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 -- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27280a3380 msgr2=0x7f27280a5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27280a3380 0x7f27280a5840 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2738041eb0 tx=0x7f273803b990 comp rx=0 tx=0).stop 2024-09-15T23:30:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 -- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27401068d0 msgr2=0x7f2740108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27401068d0 0x7f2740108890 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f2738004af0 tx=0x7f2738041e40 comp rx=0 tx=0).stop 2024-09-15T23:30:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 -- 172.21.15.62:0/3972879105 shutdown_connections 2024-09-15T23:30:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27280a3380 0x7f27280a5840 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27401068d0 0x7f2740108890 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2740101270 0x7f2740108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 --2- 172.21.15.62:0/3972879105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27401008d0 0x7f274010d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.412+0000 7f273ffff640 1 -- 172.21.15.62:0/3972879105 >> 172.21.15.62:0/3972879105 conn(0x7f27400fc610 msgr2=0x7f27400fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.413+0000 7f273ffff640 1 -- 172.21.15.62:0/3972879105 shutdown_connections 2024-09-15T23:30:21.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:21.413+0000 7f273ffff640 1 -- 172.21.15.62:0/3972879105 wait complete. 2024-09-15T23:30:22.076 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:21 smithi067 ceph-mon[35665]: pgmap v792: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 5.3 MiB/s wr, 569 op/s 2024-09-15T23:30:22.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:21 smithi062 ceph-mon[30949]: pgmap v792: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 5.3 MiB/s wr, 569 op/s 2024-09-15T23:30:22.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:21 smithi148 ceph-mon[35610]: pgmap v792: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 5.3 MiB/s wr, 569 op/s 2024-09-15T23:30:22.334 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:30:22.842 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:23.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.432+0000 7f9f1d079640 1 -- 172.21.15.62:0/75852711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 msgr2=0x7f9f1811d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:23.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.432+0000 7f9f1d079640 1 --2- 172.21.15.62:0/75852711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 0x7f9f1811d530 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9f1001acb0 tx=0x7f9f1003b920 comp rx=0 tx=0).stop 2024-09-15T23:30:23.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.432+0000 7f9f1d079640 1 -- 172.21.15.62:0/75852711 shutdown_connections 2024-09-15T23:30:23.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.432+0000 7f9f1d079640 1 --2- 172.21.15.62:0/75852711 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 0x7f9f1811d530 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.432+0000 7f9f1d079640 1 --2- 172.21.15.62:0/75852711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18109830 0x7f9f18109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.432+0000 7f9f1d079640 1 --2- 172.21.15.62:0/75852711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f18109450 0x7f9f18110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.432+0000 7f9f1d079640 1 -- 172.21.15.62:0/75852711 >> 172.21.15.62:0/75852711 conn(0x7f9f1806e140 msgr2=0x7f9f1806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.433+0000 7f9f1d079640 1 -- 172.21.15.62:0/75852711 shutdown_connections 2024-09-15T23:30:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.433+0000 7f9f1d079640 1 -- 172.21.15.62:0/75852711 wait complete. 2024-09-15T23:30:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.434+0000 7f9f1d079640 1 Processor -- start 2024-09-15T23:30:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.434+0000 7f9f1d079640 1 -- start start 2024-09-15T23:30:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.434+0000 7f9f1d079640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18109450 0x7f9f1811a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.434+0000 7f9f1d079640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f18109830 0x7f9f18116830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.434+0000 7f9f1d079640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 0x7f9f18116d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.434+0000 7f9f1d079640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9f181132b0 con 0x7f9f18109450 2024-09-15T23:30:23.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.434+0000 7f9f1d079640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f18113130 con 0x7f9f18109830 2024-09-15T23:30:23.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.434+0000 7f9f1d079640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9f18113430 con 0x7f9f1811b140 2024-09-15T23:30:23.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.436+0000 7f9f1759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 0x7f9f18116d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.436+0000 7f9f1759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 0x7f9f18116d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38058/0 (socket says 172.21.15.62:38058) 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.436+0000 7f9f1759e640 1 -- 172.21.15.62:0/1338550398 learned_addr learned my addr 172.21.15.62:0/1338550398 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.436+0000 7f9f1759e640 1 -- 172.21.15.62:0/1338550398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f18109830 msgr2=0x7f9f18116830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.436+0000 7f9f1759e640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f18109830 0x7f9f18116830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.436+0000 7f9f1759e640 1 -- 172.21.15.62:0/1338550398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18109450 msgr2=0x7f9f1811a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.436+0000 7f9f1759e640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18109450 0x7f9f1811a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.436+0000 7f9f1759e640 1 -- 172.21.15.62:0/1338550398 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f18117360 con 0x7f9f1811b140 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.436+0000 7f9f1759e640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 0x7f9f18116d70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9f10040a90 tx=0x7f9f10040ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.437+0000 7f9efffff640 1 -- 172.21.15.62:0/1338550398 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f10045070 con 0x7f9f1811b140 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.437+0000 7f9f1d079640 1 -- 172.21.15.62:0/1338550398 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f181175f0 con 0x7f9f1811b140 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.437+0000 7f9f1d079640 1 -- 172.21.15.62:0/1338550398 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f1807a6e0 con 0x7f9f1811b140 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.437+0000 7f9efffff640 1 -- 172.21.15.62:0/1338550398 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f10040be0 con 0x7f9f1811b140 2024-09-15T23:30:23.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.437+0000 7f9efffff640 1 -- 172.21.15.62:0/1338550398 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1004a690 con 0x7f9f1811b140 2024-09-15T23:30:23.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.443+0000 7f9f1f452740 1 Processor -- start 2024-09-15T23:30:23.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.444+0000 7f9f1f452740 1 -- start start 2024-09-15T23:30:23.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.445+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad6508090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.445+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad65085d0 0x563ad6508a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.445+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6513370 0x563ad6515760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.445+0000 7f9f1f452740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563ad646e2f0 con 0x563ad6513370 2024-09-15T23:30:23.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.445+0000 7f9f1f452740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563ad6415cf0 con 0x563ad65085d0 2024-09-15T23:30:23.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.445+0000 7f9f1f452740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563ad646a2b0 con 0x563ad6509b40 2024-09-15T23:30:23.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.445+0000 7f9efffff640 1 -- 172.21.15.62:0/1338550398 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f9f10048020 con 0x7f9f1811b140 2024-09-15T23:30:23.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.446+0000 7f9efffff640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9f000a3450 0x7f9f000a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.446+0000 7f9f1759e640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9f000a3450 0x7f9f000a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.447+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6513370 0x563ad6515760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.447+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6513370 0x563ad6515760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33770/0 (socket says 172.21.15.62:33770) 2024-09-15T23:30:23.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.447+0000 7f9f1659c640 1 -- 172.21.15.62:0/795254282 learned_addr learned my addr 172.21.15.62:0/795254282 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:23.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.447+0000 7f9f1659c640 1 --2- 172.21.15.62:0/795254282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad6508090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.447+0000 7f9efffff640 1 -- 172.21.15.62:0/1338550398 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9f10101a30 con 0x7f9f1811b140 2024-09-15T23:30:23.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.449+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/795254282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad65085d0 0x563ad6508a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.452+0000 7f9f1759e640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9f000a3450 0x7f9f000a5910 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9f100cdc00 tx=0x7f9f100ca000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.453+0000 7f9f16d9d640 1 -- 172.21.15.62:0/795254282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 msgr2=0x563ad6508090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.453+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/795254282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad6508090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.453+0000 7f9f16d9d640 1 -- 172.21.15.62:0/795254282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6513370 msgr2=0x563ad6515760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.453+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/795254282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6513370 0x563ad6515760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.453+0000 7f9f16d9d640 1 -- 172.21.15.62:0/795254282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ad63fa3a0 con 0x563ad65085d0 2024-09-15T23:30:23.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.454+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/795254282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad65085d0 0x563ad6508a10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00ba40 tx=0x7f9f0c00bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a71f9cc17c74e66 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:23.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.455+0000 7f9efd7fa640 1 -- 172.21.15.62:0/795254282 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c00dc40 con 0x563ad65085d0 2024-09-15T23:30:23.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.455+0000 7f9efd7fa640 1 -- 172.21.15.62:0/795254282 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f0c00dde0 con 0x563ad65085d0 2024-09-15T23:30:23.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.455+0000 7f9efd7fa640 1 -- 172.21.15.62:0/795254282 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c004a00 con 0x563ad65085d0 2024-09-15T23:30:23.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.456+0000 7f9f1f452740 1 -- 172.21.15.62:0/795254282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad65085d0 msgr2=0x563ad6508a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:23.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.456+0000 7f9f1f452740 1 --2- 172.21.15.62:0/795254282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad65085d0 0x563ad6508a10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00ba40 tx=0x7f9f0c00bf00 comp rx=0 tx=0).stop 2024-09-15T23:30:23.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.459+0000 7f9f1f452740 1 -- 172.21.15.62:0/795254282 shutdown_connections 2024-09-15T23:30:23.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.459+0000 7f9f1f452740 1 --2- 172.21.15.62:0/795254282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6513370 0x563ad6515760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.459+0000 7f9f1f452740 1 --2- 172.21.15.62:0/795254282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad65085d0 0x563ad6508a10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.459+0000 7f9f1f452740 1 --2- 172.21.15.62:0/795254282 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad6508090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.461 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.459+0000 7f9f1f452740 1 -- 172.21.15.62:0/795254282 >> 172.21.15.62:0/795254282 conn(0x563ad64d1310 msgr2=0x563ad6504c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:23.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.462+0000 7f9f1f452740 1 -- 172.21.15.62:0/795254282 shutdown_connections 2024-09-15T23:30:23.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.466+0000 7f9f1f452740 1 -- 172.21.15.62:0/795254282 wait complete. 2024-09-15T23:30:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.466+0000 7f9f1f452740 1 -- start start 2024-09-15T23:30:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.466+0000 7f9f1f452740 1 Processor -- start 2024-09-15T23:30:23.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.467+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 0x563ad6512d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.467+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad650ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.467+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad6510560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.467+0000 7f9f1f452740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563ad6516070 con 0x563ad65085d0 2024-09-15T23:30:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.467+0000 7f9f1f452740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563ad6429060 con 0x563ad6513370 2024-09-15T23:30:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f1f452740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563ad65161f0 con 0x563ad6509b40 2024-09-15T23:30:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f1f452740 10 client.0 _ll_register_callbacks cb 0x563ad617c8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad650ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f16d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad650ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38082/0 (socket says 172.21.15.62:38082) 2024-09-15T23:30:23.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f16d9d640 1 -- 172.21.15.62:0/3577560168 learned_addr learned my addr 172.21.15.62:0/3577560168 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f16d9d640 1 -- 172.21.15.62:0/3577560168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 msgr2=0x563ad6510560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:30:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad6510560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f16d9d640 1 -- 172.21.15.62:0/3577560168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 msgr2=0x563ad6512d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 0x563ad6512d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.468+0000 7f9f16d9d640 1 -- 172.21.15.62:0/3577560168 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ad61df3c0 con 0x563ad6509b40 2024-09-15T23:30:23.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.469+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad650ffc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c005550 tx=0x7f9f0c00dc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffeaa4398be0ff69 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:23.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.470+0000 7f9efaff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:23.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.470+0000 7f9efaff5640 1 -- 172.21.15.62:0/3577560168 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c004450 con 0x563ad6509b40 2024-09-15T23:30:23.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.470+0000 7f9f1f452740 1 -- 172.21.15.62:0/3577560168 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f0c00dde0 con 0x563ad6509b40 2024-09-15T23:30:23.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.470+0000 7f9efaff5640 1 -- 172.21.15.62:0/3577560168 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9f0c0035f0 con 0x563ad6509b40 2024-09-15T23:30:23.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.471+0000 7f9efaff5640 1 -- 172.21.15.62:0/3577560168 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9f0c02d050 con 0x563ad6509b40 2024-09-15T23:30:23.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.472+0000 7f9f1f452740 10 client.27913 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:23.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.472+0000 7f9f1f452740 1 -- 172.21.15.62:0/3577560168 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563ad6521b50 con 0x563ad6509b40 2024-09-15T23:30:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.473+0000 7f9efaff5640 1 -- 172.21.15.62:0/3577560168 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f0c00dde0 con 0x563ad6509b40 2024-09-15T23:30:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.473+0000 7f9efaff5640 1 -- 172.21.15.62:0/3577560168 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f9f0c022020 con 0x563ad6509b40 2024-09-15T23:30:23.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.473+0000 7f9f1f452740 10 client.27913 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.473+0000 7f9f1f452740 10 client.27913 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:30:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.473+0000 7f9f1f452740 20 client.27913 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.473+0000 7f9f1f452740 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x563ad6523370 0x563ad6525830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.473+0000 7f9f1f452740 4 client.27913 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.473+0000 7f9f1f452740 1 -- 172.21.15.62:0/3577560168 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ad647b860 con 0x563ad6523370 2024-09-15T23:30:23.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.474+0000 7f9f1659c640 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x563ad6523370 0x563ad6525830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.474+0000 7f9f1659c640 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x563ad6523370 0x563ad6525830 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cd2168eee45e04fd server_cookie=6ca616b1e5a15d25 in_seq=0 out_seq=0 2024-09-15T23:30:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.475+0000 7f9efaff5640 10 client.27913 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:30:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.475+0000 7f9efaff5640 1 -- 172.21.15.62:0/3577560168 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563ad647b860 con 0x563ad6523370 2024-09-15T23:30:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.475+0000 7f9efaff5640 10 client.27913 handle_command_reply: tid=0 2024-09-15T23:30:23.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.475+0000 7f9f1f452740 1 client.27913 shutdown 2024-09-15T23:30:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 -- 172.21.15.62:0/3577560168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 msgr2=0x563ad650ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad650ffc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c005550 tx=0x7f9f0c00dc70 comp rx=0 tx=0).stop 2024-09-15T23:30:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 -- 172.21.15.62:0/3577560168 shutdown_connections 2024-09-15T23:30:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x563ad6523370 0x563ad6525830 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad6510560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad650ffc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 --2- 172.21.15.62:0/3577560168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 0x563ad6512d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 -- 172.21.15.62:0/3577560168 >> 172.21.15.62:0/3577560168 conn(0x563ad64d1310 msgr2=0x563ad650a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9efaff5640 0 client.27913 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:30:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 -- 172.21.15.62:0/3577560168 shutdown_connections 2024-09-15T23:30:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 1 -- 172.21.15.62:0/3577560168 wait complete. 2024-09-15T23:30:23.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.476+0000 7f9f1f452740 20 client.27913 trim_cache size 0 max 16384 2024-09-15T23:30:23.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.501+0000 7f9f1f452740 1 Processor -- start 2024-09-15T23:30:23.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.501+0000 7f9f1f452740 1 -- start start 2024-09-15T23:30:23.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.502+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 0x563ad650ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.502+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad651f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.502+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 0x563ad650ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.502+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 0x563ad650ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33780/0 (socket says 172.21.15.62:33780) 2024-09-15T23:30:23.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.503+0000 7f9f1659c640 1 -- 172.21.15.62:0/2366738850 learned_addr learned my addr 172.21.15.62:0/2366738850 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:23.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.503+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad650f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.503+0000 7f9f1f452740 1 -- 172.21.15.62:0/2366738850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9f04012520 con 0x563ad65085d0 2024-09-15T23:30:23.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.503+0000 7f9f1f452740 1 -- 172.21.15.62:0/2366738850 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0400fdc0 con 0x563ad6513370 2024-09-15T23:30:23.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.503+0000 7f9f1f452740 1 -- 172.21.15.62:0/2366738850 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0400dc10 con 0x563ad6509b40 2024-09-15T23:30:23.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.503+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad651f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.504+0000 7f9f1659c640 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad650f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.504+0000 7f9f1659c640 1 -- 172.21.15.62:0/2366738850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 msgr2=0x563ad651f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:23.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.504+0000 7f9f1659c640 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad651f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.504+0000 7f9f1659c640 1 -- 172.21.15.62:0/2366738850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 msgr2=0x563ad650f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:23.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.504+0000 7f9f1659c640 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad650f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.504+0000 7f9f1659c640 1 -- 172.21.15.62:0/2366738850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ad6513080 con 0x563ad65085d0 2024-09-15T23:30:23.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.504+0000 7f9f1659c640 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 0x563ad650ff80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800d790 tx=0x7f9f08005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881c11bfb5347703 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:23.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.505+0000 7f9efcff9640 1 -- 172.21.15.62:0/2366738850 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08011800 con 0x563ad65085d0 2024-09-15T23:30:23.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.505+0000 7f9efcff9640 1 -- 172.21.15.62:0/2366738850 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f08011e80 con 0x563ad65085d0 2024-09-15T23:30:23.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.505+0000 7f9efcff9640 1 -- 172.21.15.62:0/2366738850 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f080103d0 con 0x563ad65085d0 2024-09-15T23:30:23.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.506+0000 7f9f1f452740 1 -- 172.21.15.62:0/2366738850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 msgr2=0x563ad650ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:23.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.506+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 0x563ad650ff80 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800d790 tx=0x7f9f08005850 comp rx=0 tx=0).stop 2024-09-15T23:30:23.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.506+0000 7f9f1f452740 1 -- 172.21.15.62:0/2366738850 shutdown_connections 2024-09-15T23:30:23.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.506+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad650f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.506+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad6509b40 0x563ad651f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.506+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2366738850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad65085d0 0x563ad650ff80 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.507+0000 7f9f1f452740 1 -- 172.21.15.62:0/2366738850 >> 172.21.15.62:0/2366738850 conn(0x563ad64d1310 msgr2=0x563ad650b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:23.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.507+0000 7f9f1f452740 1 -- 172.21.15.62:0/2366738850 shutdown_connections 2024-09-15T23:30:23.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.507+0000 7f9f1f452740 1 -- 172.21.15.62:0/2366738850 wait complete. 2024-09-15T23:30:23.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.508+0000 7f9f1f452740 1 -- start start 2024-09-15T23:30:23.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.508+0000 7f9f1f452740 1 Processor -- start 2024-09-15T23:30:23.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.509+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad65085d0 0x563ad652b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.509+0000 7f9f1f452740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6509b40 0x563ad651b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.509+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad65085d0 0x563ad652b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.509+0000 7f9f1659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad65085d0 0x563ad652b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38098/0 (socket says 172.21.15.62:38098) 2024-09-15T23:30:23.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.509+0000 7f9f16d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6509b40 0x563ad651b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.509+0000 7f9f16d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6509b40 0x563ad651b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33794/0 (socket says 172.21.15.62:33794) 2024-09-15T23:30:23.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.509+0000 7f9f16d9d640 1 -- 172.21.15.62:0/2182949691 learned_addr learned my addr 172.21.15.62:0/2182949691 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:23.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.510+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad651ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.510+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0400e280 con 0x563ad6509b40 2024-09-15T23:30:23.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.510+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0400f6f0 con 0x563ad6513370 2024-09-15T23:30:23.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.510+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0400e950 con 0x563ad65085d0 2024-09-15T23:30:23.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.510+0000 7f9f1f452740 10 client.0 _ll_register_callbacks cb 0x7f9f0800dcc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:23.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.511+0000 7f9f1659c640 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad651ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.511+0000 7f9f1659c640 1 -- 172.21.15.62:0/2182949691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 msgr2=0x563ad651ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:23.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.511+0000 7f9f1659c640 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad651ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.511+0000 7f9f1659c640 1 -- 172.21.15.62:0/2182949691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6509b40 msgr2=0x563ad651b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:23.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.511+0000 7f9f1659c640 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6509b40 0x563ad651b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.511+0000 7f9f1659c640 1 -- 172.21.15.62:0/2182949691 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ad651c0c0 con 0x563ad65085d0 2024-09-15T23:30:23.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.511+0000 7f9f1659c640 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad65085d0 0x563ad652b6f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800d760 tx=0x7f9f0801e620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=129e683c7122f5b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:23.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.511+0000 7f9efaff5640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:23.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.512+0000 7f9efaff5640 1 -- 172.21.15.62:0/2182949691 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08004220 con 0x563ad65085d0 2024-09-15T23:30:23.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.512+0000 7f9efaff5640 1 -- 172.21.15.62:0/2182949691 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f9f080043c0 con 0x563ad65085d0 2024-09-15T23:30:23.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.512+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563ad6475d70 con 0x563ad65085d0 2024-09-15T23:30:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.513+0000 7f9efaff5640 1 -- 172.21.15.62:0/2182949691 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9f0803f050 con 0x563ad65085d0 2024-09-15T23:30:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.513+0000 7f9f1f452740 10 client.27916 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:23.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.513+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563ad652a150 con 0x563ad65085d0 2024-09-15T23:30:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.513+0000 7f9efaff5640 1 -- 172.21.15.62:0/2182949691 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563ad6475d70 con 0x563ad65085d0 2024-09-15T23:30:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.513+0000 7f9efaff5640 1 -- 172.21.15.62:0/2182949691 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f9f08037020 con 0x563ad65085d0 2024-09-15T23:30:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.514+0000 7f9f1f452740 10 client.27916 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.514+0000 7f9f1f452740 10 client.27916 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:30:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.514+0000 7f9f1f452740 20 client.27916 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.514+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x563ad652a720 0x563ad652ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.514+0000 7f9f1f452740 4 client.27916 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:30:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.514+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ad63f2170 con 0x563ad652a720 2024-09-15T23:30:23.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.515+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x563ad652a720 0x563ad652ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:23.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.515+0000 7f9f16d9d640 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x563ad652a720 0x563ad652ab00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea16cbaf54d52fd6 server_cookie=2b017e90b23336ae in_seq=0 out_seq=0 2024-09-15T23:30:23.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.515+0000 7f9efaff5640 10 client.27916 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:30:23.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.515+0000 7f9efaff5640 1 -- 172.21.15.62:0/2182949691 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ad63f2170 con 0x563ad652a720 2024-09-15T23:30:23.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.515+0000 7f9efaff5640 10 client.27916 handle_command_reply: tid=0 2024-09-15T23:30:23.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.515+0000 7f9f1f452740 1 client.27916 shutdown 2024-09-15T23:30:23.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.516+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad65085d0 msgr2=0x563ad652b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.516+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad65085d0 0x563ad652b6f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800d760 tx=0x7f9f0801e620 comp rx=0 tx=0).stop 2024-09-15T23:30:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.517+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 shutdown_connections 2024-09-15T23:30:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.517+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x563ad652a720 0x563ad652ab00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.517+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563ad6513370 0x563ad651ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.517+0000 7f9efaff5640 0 client.27916 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:30:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.517+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563ad6509b40 0x563ad651b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.517+0000 7f9f1f452740 1 --2- 172.21.15.62:0/2182949691 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563ad65085d0 0x563ad652b6f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.517+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 >> 172.21.15.62:0/2182949691 conn(0x563ad64d1310 msgr2=0x563ad650e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:23.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.522+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 shutdown_connections 2024-09-15T23:30:23.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.522+0000 7f9f1f452740 1 -- 172.21.15.62:0/2182949691 wait complete. 2024-09-15T23:30:23.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.523+0000 7f9f1f452740 20 client.27916 trim_cache size 0 max 16384 2024-09-15T23:30:23.527 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:30:23.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 -- 172.21.15.62:0/1338550398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9f000a3450 msgr2=0x7f9f000a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:23.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9f000a3450 0x7f9f000a5910 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9f100cdc00 tx=0x7f9f100ca000 comp rx=0 tx=0).stop 2024-09-15T23:30:23.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 -- 172.21.15.62:0/1338550398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 msgr2=0x7f9f18116d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:23.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 0x7f9f18116d70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9f10040a90 tx=0x7f9f10040ac0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 -- 172.21.15.62:0/1338550398 shutdown_connections 2024-09-15T23:30:23.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f9f000a3450 0x7f9f000a5910 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f9f1811b140 0x7f9f18116d70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f18109830 0x7f9f18116830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 --2- 172.21.15.62:0/1338550398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f18109450 0x7f9f1811a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:23.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.529+0000 7f9f1d079640 1 -- 172.21.15.62:0/1338550398 >> 172.21.15.62:0/1338550398 conn(0x7f9f1806e140 msgr2=0x7f9f180722d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:23.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.530+0000 7f9f1d079640 1 -- 172.21.15.62:0/1338550398 shutdown_connections 2024-09-15T23:30:23.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:23.530+0000 7f9f1d079640 1 -- 172.21.15.62:0/1338550398 wait complete. 2024-09-15T23:30:24.105 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:23 smithi148 ceph-mon[35610]: pgmap v793: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 5.3 MiB/s wr, 556 op/s 2024-09-15T23:30:24.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:23 smithi062 ceph-mon[30949]: pgmap v793: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 5.3 MiB/s wr, 556 op/s 2024-09-15T23:30:24.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:23 smithi067 ceph-mon[35665]: pgmap v793: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 5.3 MiB/s wr, 556 op/s 2024-09-15T23:30:24.359 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:30:24.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 845.0 s, 512.2 tps, lat 194.160 ms stddev 223.336 2024-09-15T23:30:24.914 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:25.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.488+0000 7fcff1c2d640 1 -- 172.21.15.62:0/2957541922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1009a0 msgr2=0x7fcfec100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:25.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.488+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/2957541922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1009a0 0x7fcfec100d80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009900 tx=0x7fcfdc02a480 comp rx=0 tx=0).stop 2024-09-15T23:30:25.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.495+0000 7fcff1c2d640 1 -- 172.21.15.62:0/2957541922 shutdown_connections 2024-09-15T23:30:25.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.495+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/2957541922 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcfec105db0 0x7fcfec106240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.495+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/2957541922 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfec1012c0 0x7fcfec105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.495+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/2957541922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1009a0 0x7fcfec100d80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.495+0000 7fcff1c2d640 1 -- 172.21.15.62:0/2957541922 >> 172.21.15.62:0/2957541922 conn(0x7fcfec0fc630 msgr2=0x7fcfec0fea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:25.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.496+0000 7fcff1c2d640 1 -- 172.21.15.62:0/2957541922 shutdown_connections 2024-09-15T23:30:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.496+0000 7fcff1c2d640 1 -- 172.21.15.62:0/2957541922 wait complete. 2024-09-15T23:30:25.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.497+0000 7fcff1c2d640 1 Processor -- start 2024-09-15T23:30:25.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.497+0000 7fcff1c2d640 1 -- start start 2024-09-15T23:30:25.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.497+0000 7fcff1c2d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfec1009a0 0x7fcfec073950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.497+0000 7fcff1c2d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1012c0 0x7fcfec073e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.497+0000 7fcff1c2d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcfec105db0 0x7fcfec06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.497+0000 7fcff1c2d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec118420 con 0x7fcfec1009a0 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.497+0000 7fcff1c2d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1182a0 con 0x7fcfec1012c0 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.497+0000 7fcff1c2d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1185a0 con 0x7fcfec105db0 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.498+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1012c0 0x7fcfec073e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.498+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1012c0 0x7fcfec073e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40256/0 (socket says 172.21.15.62:40256) 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.498+0000 7fcfeaffd640 1 -- 172.21.15.62:0/3985099461 learned_addr learned my addr 172.21.15.62:0/3985099461 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.498+0000 7fcfeaffd640 1 -- 172.21.15.62:0/3985099461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcfec105db0 msgr2=0x7fcfec06cb70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.498+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcfec105db0 0x7fcfec06cb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.498+0000 7fcfeaffd640 1 -- 172.21.15.62:0/3985099461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfec1009a0 msgr2=0x7fcfec073950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.498+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfec1009a0 0x7fcfec073950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.498+0000 7fcfeaffd640 1 -- 172.21.15.62:0/3985099461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfec06d430 con 0x7fcfec1012c0 2024-09-15T23:30:25.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.499+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1012c0 0x7fcfec073e90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400b9e0 tx=0x7fcfe400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.500+0000 7fcfe8ff9640 1 -- 172.21.15.62:0/3985099461 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4019070 con 0x7fcfec1012c0 2024-09-15T23:30:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.500+0000 7fcfe8ff9640 1 -- 172.21.15.62:0/3985099461 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe4014440 con 0x7fcfec1012c0 2024-09-15T23:30:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.500+0000 7fcfe8ff9640 1 -- 172.21.15.62:0/3985099461 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4007650 con 0x7fcfec1012c0 2024-09-15T23:30:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.500+0000 7fcff1c2d640 1 -- 172.21.15.62:0/3985099461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfec06d720 con 0x7fcfec1012c0 2024-09-15T23:30:25.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.501+0000 7fcff1c2d640 1 -- 172.21.15.62:0/3985099461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfec0716b0 con 0x7fcfec1012c0 2024-09-15T23:30:25.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.505+0000 7fcff4006740 1 Processor -- start 2024-09-15T23:30:25.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.506+0000 7fcff4006740 1 -- start start 2024-09-15T23:30:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.506+0000 7fcff4006740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 0x560790715240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.506+0000 7fcff4006740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x560790715bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.506+0000 7fcff4006740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 0x560790722910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.506+0000 7fcff4006740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56079067b2f0 con 0x560790716cf0 2024-09-15T23:30:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.506+0000 7fcff4006740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560790622cf0 con 0x560790720520 2024-09-15T23:30:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.506+0000 7fcff4006740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5607906772b0 con 0x560790715780 2024-09-15T23:30:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.507+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x560790715bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.507+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x560790715bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38128/0 (socket says 172.21.15.62:38128) 2024-09-15T23:30:25.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.507+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/2432057216 learned_addr learned my addr 172.21.15.62:0/2432057216 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:25.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.507+0000 7fcfebfff640 1 --2- 172.21.15.62:0/2432057216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 0x560790715240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.507+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/2432057216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 0x560790722910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.508+0000 7fcfe8ff9640 1 -- 172.21.15.62:0/3985099461 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fcfe401c070 con 0x7fcfec1012c0 2024-09-15T23:30:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.508+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/2432057216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 msgr2=0x560790722910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.508+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/2432057216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 0x560790722910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.508+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/2432057216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 msgr2=0x560790715240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.508+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/2432057216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 0x560790715240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.508+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/2432057216 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607906073a0 con 0x560790715780 2024-09-15T23:30:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.509+0000 7fcfe8ff9640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcfd40a3450 0x7fcfd40a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.509+0000 7fcfe8ff9640 1 -- 172.21.15.62:0/3985099461 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcfe40cf210 con 0x7fcfec1012c0 2024-09-15T23:30:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.510+0000 7fcfebfff640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcfd40a3450 0x7fcfd40a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.510+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/2432057216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x560790715bc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005c20 tx=0x7fcfdc031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33fb546a22d95d42 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:25.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.510+0000 7fcfd1ffb640 1 -- 172.21.15.62:0/2432057216 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc040070 con 0x560790715780 2024-09-15T23:30:25.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.510+0000 7fcfd1ffb640 1 -- 172.21.15.62:0/2432057216 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc004040 con 0x560790715780 2024-09-15T23:30:25.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.510+0000 7fcfd1ffb640 1 -- 172.21.15.62:0/2432057216 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc035040 con 0x560790715780 2024-09-15T23:30:25.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.511+0000 7fcff4006740 1 -- 172.21.15.62:0/2432057216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 msgr2=0x560790715bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:25.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.511+0000 7fcff4006740 1 --2- 172.21.15.62:0/2432057216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x560790715bc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc005c20 tx=0x7fcfdc031400 comp rx=0 tx=0).stop 2024-09-15T23:30:25.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.511+0000 7fcfebfff640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcfd40a3450 0x7fcfd40a5910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009b60 tx=0x7fcfe0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:25.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.512+0000 7fcff4006740 1 -- 172.21.15.62:0/2432057216 shutdown_connections 2024-09-15T23:30:25.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.512+0000 7fcff4006740 1 --2- 172.21.15.62:0/2432057216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 0x560790722910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.512+0000 7fcff4006740 1 --2- 172.21.15.62:0/2432057216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x560790715bc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.512+0000 7fcff4006740 1 --2- 172.21.15.62:0/2432057216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 0x560790715240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.512+0000 7fcff4006740 1 -- 172.21.15.62:0/2432057216 >> 172.21.15.62:0/2432057216 conn(0x5607906de310 msgr2=0x560790711e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.512+0000 7fcff4006740 1 -- 172.21.15.62:0/2432057216 shutdown_connections 2024-09-15T23:30:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.512+0000 7fcff4006740 1 -- 172.21.15.62:0/2432057216 wait complete. 2024-09-15T23:30:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.513+0000 7fcff4006740 1 -- start start 2024-09-15T23:30:25.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.513+0000 7fcff4006740 1 Processor -- start 2024-09-15T23:30:25.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcff4006740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079071feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcff4006740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 0x56079071d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcff4006740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 0x56079071d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcff4006740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5607907231c0 con 0x560790716cf0 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcff4006740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x560790636060 con 0x560790720520 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcff4006740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x560790723340 con 0x560790715780 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcff4006740 10 client.0 _ll_register_callbacks cb 0x5607903898c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079071feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079071feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38134/0 (socket says 172.21.15.62:38134) 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/55017581 learned_addr learned my addr 172.21.15.62:0/55017581 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.514+0000 7fcfebfff640 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 0x56079071d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 0x56079071d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/55017581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 msgr2=0x56079071d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 0x56079071d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/55017581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 msgr2=0x56079071d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 0x56079071d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/55017581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607903ec3c0 con 0x560790715780 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079071feb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0057c0 tx=0x7fcfdc032ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c256658fe2f521cf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcfcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcfcaffd640 1 -- 172.21.15.62:0/55017581 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc040050 con 0x560790715780 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.515+0000 7fcff4006740 1 -- 172.21.15.62:0/55017581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfdc004040 con 0x560790715780 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcfcaffd640 1 -- 172.21.15.62:0/55017581 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcfdc0315b0 con 0x560790715780 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcfcaffd640 1 -- 172.21.15.62:0/55017581 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcfdc056050 con 0x560790715780 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcff4006740 10 client.27925 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcff4006740 1 -- 172.21.15.62:0/55017581 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560790723210 con 0x560790715780 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcfcaffd640 1 -- 172.21.15.62:0/55017581 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc004040 con 0x560790715780 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcfcaffd640 1 -- 172.21.15.62:0/55017581 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fcfdc045700 con 0x560790715780 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcff4006740 10 client.27925 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcff4006740 10 client.27925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:25.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcff4006740 20 client.27925 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:25.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcff4006740 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56079072fd70 0x560790732160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcff4006740 4 client.27925 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:25.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.516+0000 7fcff4006740 1 -- 172.21.15.62:0/55017581 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560790688860 con 0x56079072fd70 2024-09-15T23:30:25.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.517+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56079072fd70 0x560790732160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.517+0000 7fcfcaffd640 10 client.27925 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.517+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56079072fd70 0x560790732160 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eeeaa98d24db49 server_cookie=de66283a902146ac in_seq=0 out_seq=0 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcfcaffd640 1 -- 172.21.15.62:0/55017581 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560790688860 con 0x56079072fd70 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcfcaffd640 10 client.27925 handle_command_reply: tid=0 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 client.27925 shutdown 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 -- 172.21.15.62:0/55017581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 msgr2=0x56079071feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079071feb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc0057c0 tx=0x7fcfdc032ed0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 -- 172.21.15.62:0/55017581 shutdown_connections 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56079072fd70 0x560790732160 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790720520 0x56079071d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790716cf0 0x56079071d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 --2- 172.21.15.62:0/55017581 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079071feb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 -- 172.21.15.62:0/55017581 >> 172.21.15.62:0/55017581 conn(0x5607906de310 msgr2=0x560790717b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcfcaffd640 0 client.27925 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:25.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 -- 172.21.15.62:0/55017581 shutdown_connections 2024-09-15T23:30:25.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 1 -- 172.21.15.62:0/55017581 wait complete. 2024-09-15T23:30:25.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.518+0000 7fcff4006740 20 client.27925 trim_cache size 0 max 16384 2024-09-15T23:30:25.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.541+0000 7fcff4006740 1 Processor -- start 2024-09-15T23:30:25.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.541+0000 7fcff4006740 1 -- start start 2024-09-15T23:30:25.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.541+0000 7fcff4006740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079072c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.541+0000 7fcff4006740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790716cf0 0x56079071c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.541+0000 7fcff4006740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790721910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.541+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790716cf0 0x56079071c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790716cf0 0x56079071c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40286/0 (socket says 172.21.15.62:40286) 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079072c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfeb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079072c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38150/0 (socket says 172.21.15.62:38150) 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfebfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790721910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcff4006740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1c8930 con 0x560790720520 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcff4006740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1c61d0 con 0x560790716cf0 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfebfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790721910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33822/0 (socket says 172.21.15.62:33822) 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcff4006740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10edd0 con 0x560790715780 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfebfff640 1 -- 172.21.15.62:0/3139284191 learned_addr learned my addr 172.21.15.62:0/3139284191 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfebfff640 1 -- 172.21.15.62:0/3139284191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 msgr2=0x56079072c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:25.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfebfff640 1 --2- 172.21.15.62:0/3139284191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079072c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfebfff640 1 -- 172.21.15.62:0/3139284191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790716cf0 msgr2=0x56079071c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfebfff640 1 --2- 172.21.15.62:0/3139284191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790716cf0 0x56079071c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfebfff640 1 -- 172.21.15.62:0/3139284191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560790712b10 con 0x560790720520 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/3139284191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079072c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/3139284191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790716cf0 0x56079071c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfebfff640 1 --2- 172.21.15.62:0/3139284191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790721910 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0011e80 tx=0x7fcfe0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d7e4820857e4e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.542+0000 7fcfd17fa640 1 -- 172.21.15.62:0/3139284191 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0004020 con 0x560790720520 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcfd17fa640 1 -- 172.21.15.62:0/3139284191 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe00041c0 con 0x560790720520 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcfd17fa640 1 -- 172.21.15.62:0/3139284191 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe001f790 con 0x560790720520 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcff4006740 1 -- 172.21.15.62:0/3139284191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 msgr2=0x560790721910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcff4006740 1 --2- 172.21.15.62:0/3139284191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790721910 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0011e80 tx=0x7fcfe0014000 comp rx=0 tx=0).stop 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcff4006740 1 -- 172.21.15.62:0/3139284191 shutdown_connections 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcff4006740 1 --2- 172.21.15.62:0/3139284191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790721910 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcff4006740 1 --2- 172.21.15.62:0/3139284191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790716cf0 0x56079071c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcff4006740 1 --2- 172.21.15.62:0/3139284191 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790715780 0x56079072c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcff4006740 1 -- 172.21.15.62:0/3139284191 >> 172.21.15.62:0/3139284191 conn(0x5607906de310 msgr2=0x56079071ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcff4006740 1 -- 172.21.15.62:0/3139284191 shutdown_connections 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.543+0000 7fcff4006740 1 -- 172.21.15.62:0/3139284191 wait complete. 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcff4006740 1 -- start start 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcff4006740 1 Processor -- start 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcff4006740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790715780 0x5607907289e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcff4006740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790716cf0 0x560790736e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcff4006740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790737370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcff4006740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10f440 con 0x560790720520 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcff4006740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec1c5b00 con 0x560790715780 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcff4006740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fcfec10fb10 con 0x560790716cf0 2024-09-15T23:30:25.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcff4006740 10 client.0 _ll_register_callbacks cb 0x7fcfe40d2c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790716cf0 0x560790736e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcfeaffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790716cf0 0x560790736e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38160/0 (socket says 172.21.15.62:38160) 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcfeaffd640 1 -- 172.21.15.62:0/1631438264 learned_addr learned my addr 172.21.15.62:0/1631438264 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcfebfff640 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790737370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.544+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790715780 0x5607907289e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/1631438264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790716cf0 msgr2=0x560790736e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790716cf0 0x560790736e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/1631438264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 msgr2=0x560790737370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790737370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfeb7fe640 1 -- 172.21.15.62:0/1631438264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607907378b0 con 0x560790715780 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfeb7fe640 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790715780 0x5607907289e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc045ad0 tx=0x7fcfdc004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a166dd914b1ff1a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:30:25.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfcaffd640 1 -- 172.21.15.62:0/1631438264 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc040050 con 0x560790715780 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcff4006740 1 -- 172.21.15.62:0/1631438264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfe00041c0 con 0x560790715780 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfcaffd640 1 -- 172.21.15.62:0/1631438264 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fcfdc0317c0 con 0x560790715780 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcfcaffd640 1 -- 172.21.15.62:0/1631438264 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcfdc067050 con 0x560790715780 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcff4006740 10 client.28259 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.545+0000 7fcff4006740 1 -- 172.21.15.62:0/1631438264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560790727270 con 0x560790715780 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.546+0000 7fcfcaffd640 1 -- 172.21.15.62:0/1631438264 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfe00041c0 con 0x560790715780 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.546+0000 7fcfcaffd640 1 -- 172.21.15.62:0/1631438264 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fcfdc062030 con 0x560790715780 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.546+0000 7fcff4006740 10 client.28259 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.546+0000 7fcff4006740 10 client.28259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.546+0000 7fcff4006740 20 client.28259 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.546+0000 7fcff4006740 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56079079ac30 0x56079079d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.546+0000 7fcff4006740 4 client.28259 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.546+0000 7fcff4006740 1 -- 172.21.15.62:0/1631438264 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560790682d70 con 0x56079079ac30 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.546+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56079079ac30 0x56079079d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:25.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcfeaffd640 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56079079ac30 0x56079079d020 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=515e38a4bc3a28c2 server_cookie=156eded06e117168 in_seq=0 out_seq=0 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcfcaffd640 10 client.28259 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcfcaffd640 1 -- 172.21.15.62:0/1631438264 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560790682d70 con 0x56079079ac30 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcfcaffd640 10 client.28259 handle_command_reply: tid=0 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 client.28259 shutdown 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 -- 172.21.15.62:0/1631438264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790715780 msgr2=0x5607907289e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790715780 0x5607907289e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc045ad0 tx=0x7fcfdc004610 comp rx=0 tx=0).stop 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 -- 172.21.15.62:0/1631438264 shutdown_connections 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56079079ac30 0x56079079d020 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcfcaffd640 0 client.28259 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560790720520 0x560790737370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x560790716cf0 0x560790736e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 --2- 172.21.15.62:0/1631438264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560790715780 0x5607907289e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 -- 172.21.15.62:0/1631438264 >> 172.21.15.62:0/1631438264 conn(0x5607906de310 msgr2=0x56079071c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:25.556 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 -- 172.21.15.62:0/1631438264 shutdown_connections 2024-09-15T23:30:25.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.547+0000 7fcff4006740 1 -- 172.21.15.62:0/1631438264 wait complete. 2024-09-15T23:30:25.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff4006740 20 client.28259 trim_cache size 0 max 16384 2024-09-15T23:30:25.557 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:30:25.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 -- 172.21.15.62:0/3985099461 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcfd40a3450 msgr2=0x7fcfd40a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:25.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcfd40a3450 0x7fcfd40a5910 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009b60 tx=0x7fcfe0005f90 comp rx=0 tx=0).stop 2024-09-15T23:30:25.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 -- 172.21.15.62:0/3985099461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1012c0 msgr2=0x7fcfec073e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:25.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1012c0 0x7fcfec073e90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fcfe400b9e0 tx=0x7fcfe400bea0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 -- 172.21.15.62:0/3985099461 shutdown_connections 2024-09-15T23:30:25.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fcfd40a3450 0x7fcfd40a5910 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fcfec105db0 0x7fcfec06cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfec1012c0 0x7fcfec073e90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 --2- 172.21.15.62:0/3985099461 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfec1009a0 0x7fcfec073950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.548+0000 7fcff1c2d640 1 -- 172.21.15.62:0/3985099461 >> 172.21.15.62:0/3985099461 conn(0x7fcfec0fc630 msgr2=0x7fcfec103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.549+0000 7fcff1c2d640 1 -- 172.21.15.62:0/3985099461 shutdown_connections 2024-09-15T23:30:25.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:25.549+0000 7fcff1c2d640 1 -- 172.21.15.62:0/3985099461 wait complete. 2024-09-15T23:30:26.159 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:25 smithi062 ceph-mon[30949]: pgmap v794: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 4.8 MiB/s wr, 534 op/s 2024-09-15T23:30:26.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:25 smithi067 ceph-mon[35665]: pgmap v794: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 4.8 MiB/s wr, 534 op/s 2024-09-15T23:30:26.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:25 smithi148 ceph-mon[35610]: pgmap v794: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 4.8 MiB/s wr, 534 op/s 2024-09-15T23:30:27.209 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:30:27.209 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:30:27.687 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:28.143 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:27 smithi062 ceph-mon[30949]: pgmap v795: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 4.9 MiB/s wr, 534 op/s 2024-09-15T23:30:28.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:27 smithi067 ceph-mon[35665]: pgmap v795: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 4.9 MiB/s wr, 534 op/s 2024-09-15T23:30:28.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:27 smithi148 ceph-mon[35610]: pgmap v795: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 4.9 MiB/s wr, 534 op/s 2024-09-15T23:30:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.276+0000 7f5151318640 1 -- 172.21.15.62:0/3205062298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5144005780 msgr2=0x7f5144097ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.276+0000 7f5151318640 1 --2- 172.21.15.62:0/3205062298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5144005780 0x7f5144097ff0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f514c067960 tx=0x7f514c096d90 comp rx=0 tx=0).stop 2024-09-15T23:30:28.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.276+0000 7f5151318640 1 -- 172.21.15.62:0/3205062298 shutdown_connections 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.276+0000 7f5151318640 1 --2- 172.21.15.62:0/3205062298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5144005780 0x7f5144097ff0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.276+0000 7f5151318640 1 --2- 172.21.15.62:0/3205062298 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5144004cf0 0x7f5144005130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.276+0000 7f5151318640 1 --2- 172.21.15.62:0/3205062298 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f514409a4e0 0x7f514409a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.276+0000 7f5151318640 1 -- 172.21.15.62:0/3205062298 >> 172.21.15.62:0/3205062298 conn(0x7f514400a210 msgr2=0x7f514400a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.277+0000 7f5151318640 1 -- 172.21.15.62:0/3205062298 shutdown_connections 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.277+0000 7f5151318640 1 -- 172.21.15.62:0/3205062298 wait complete. 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.277+0000 7f5151318640 1 Processor -- start 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.277+0000 7f5151318640 1 -- start start 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f5151318640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5144004cf0 0x7f51440a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f5151318640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144005780 0x7f51440a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f5151318640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f514409a4e0 0x7f51440a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f5151318640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f51440abef0 con 0x7f514409a4e0 2024-09-15T23:30:28.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f5151318640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f51440abd70 con 0x7f5144005780 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f5151318640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f51440ac070 con 0x7f5144004cf0 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5144004cf0 0x7f51440a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5144004cf0 0x7f51440a8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38186/0 (socket says 172.21.15.62:38186) 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514affd640 1 -- 172.21.15.62:0/2294860381 learned_addr learned my addr 172.21.15.62:0/2294860381 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514a7fc640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144005780 0x7f51440a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514b7fe640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f514409a4e0 0x7f51440a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514affd640 1 -- 172.21.15.62:0/2294860381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144005780 msgr2=0x7f51440a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514affd640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144005780 0x7f51440a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514affd640 1 -- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f514409a4e0 msgr2=0x7f51440a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514affd640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f514409a4e0 0x7f51440a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514affd640 1 -- 172.21.15.62:0/2294860381 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51440a28c0 con 0x7f5144004cf0 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.278+0000 7f514b7fe640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f514409a4e0 0x7f51440a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:28.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.279+0000 7f514affd640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5144004cf0 0x7f51440a8d60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f513c00e9f0 tx=0x7f513c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.279+0000 7f5133fff640 1 -- 172.21.15.62:0/2294860381 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c00cd80 con 0x7f5144004cf0 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.279+0000 7f5151318640 1 -- 172.21.15.62:0/2294860381 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51440a2bb0 con 0x7f5144004cf0 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.279+0000 7f5133fff640 1 -- 172.21.15.62:0/2294860381 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c010910 con 0x7f5144004cf0 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.279+0000 7f5133fff640 1 -- 172.21.15.62:0/2294860381 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c018910 con 0x7f5144004cf0 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.279+0000 7f5151318640 1 -- 172.21.15.62:0/2294860381 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51440a6bf0 con 0x7f5144004cf0 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.281+0000 7f51536f1740 1 Processor -- start 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.281+0000 7f51536f1740 1 -- start start 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f5133fff640 1 -- 172.21.15.62:0/2294860381 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f513c010ac0 con 0x7f5144004cf0 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f51536f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598182150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f51536f1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b59818ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f51536f1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b59818f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f51536f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b5980e7280 con 0x55b598181d70 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f51536f1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b59808eb10 con 0x55b59818cf50 2024-09-15T23:30:28.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f51536f1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b5980e3240 con 0x55b598182690 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b59818ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598182150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514affd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b59818ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38194/0 (socket says 172.21.15.62:38194) 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514affd640 1 -- 172.21.15.62:0/2535978502 learned_addr learned my addr 172.21.15.62:0/2535978502 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514a7fc640 1 --2- 172.21.15.62:0/2535978502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b59818f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514affd640 1 -- 172.21.15.62:0/2535978502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 msgr2=0x55b59818f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514affd640 1 --2- 172.21.15.62:0/2535978502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b59818f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514affd640 1 -- 172.21.15.62:0/2535978502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 msgr2=0x55b598182150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514affd640 1 --2- 172.21.15.62:0/2535978502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598182150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.282+0000 7f514affd640 1 -- 172.21.15.62:0/2535978502 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b59809b1f0 con 0x55b598182690 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.283+0000 7f514affd640 1 --2- 172.21.15.62:0/2535978502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b59818ca10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f513c0d8190 tx=0x7f513c0d3f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79ae4821db4c04e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.283+0000 7f5133fff640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51340a3250 0x7f51340a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.283+0000 7f514b7fe640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51340a3250 0x7f51340a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.283+0000 7f5133fff640 1 -- 172.21.15.62:0/2294860381 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f513c0cffa0 con 0x7f5144004cf0 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.283+0000 7f514b7fe640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51340a3250 0x7f51340a5710 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f514c065360 tx=0x7f514c068410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.284+0000 7f51317fa640 1 -- 172.21.15.62:0/2535978502 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c0e3070 con 0x55b598182690 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.284+0000 7f51317fa640 1 -- 172.21.15.62:0/2535978502 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c0d8e10 con 0x55b598182690 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 -- 172.21.15.62:0/2535978502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 msgr2=0x55b59818ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 --2- 172.21.15.62:0/2535978502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b59818ca10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f513c0d8190 tx=0x7f513c0d3f90 comp rx=0 tx=0).stop 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51317fa640 1 -- 172.21.15.62:0/2535978502 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c09bbf0 con 0x55b598182690 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 -- 172.21.15.62:0/2535978502 shutdown_connections 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 --2- 172.21.15.62:0/2535978502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b59818f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 --2- 172.21.15.62:0/2535978502 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b59818ca10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 --2- 172.21.15.62:0/2535978502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598182150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 -- 172.21.15.62:0/2535978502 >> 172.21.15.62:0/2535978502 conn(0x55b59817e300 msgr2=0x55b59817e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 -- 172.21.15.62:0/2535978502 shutdown_connections 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 -- 172.21.15.62:0/2535978502 wait complete. 2024-09-15T23:30:28.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 -- start start 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.285+0000 7f51536f1740 1 Processor -- start 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f51536f1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b598181d70 0x55b59818c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f51536f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598182690 0x55b598189d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f51536f1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b59818cf50 0x55b59818a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f51536f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55b59818fb70 con 0x55b598182690 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f51536f1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b5980a1f10 con 0x55b598181d70 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f51536f1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55b59818fcf0 con 0x55b59818cf50 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f51536f1740 10 client.0 _ll_register_callbacks cb 0x55b59806aa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598182690 0x55b598189d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598182690 0x55b598189d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33862/0 (socket says 172.21.15.62:33862) 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514a7fc640 1 -- 172.21.15.62:0/3901058140 learned_addr learned my addr 172.21.15.62:0/3901058140 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b598181d70 0x55b59818c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514b7fe640 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b59818cf50 0x55b59818a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514a7fc640 1 -- 172.21.15.62:0/3901058140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b59818cf50 msgr2=0x55b59818a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514a7fc640 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b59818cf50 0x55b59818a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514a7fc640 1 -- 172.21.15.62:0/3901058140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b598181d70 msgr2=0x55b59818c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514a7fc640 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b598181d70 0x55b59818c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514a7fc640 1 -- 172.21.15.62:0/3901058140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b597e586a0 con 0x55b598182690 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.286+0000 7f514b7fe640 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b59818cf50 0x55b59818a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f514a7fc640 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598182690 0x55b598189d10 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f514000bad0 tx=0x7f514000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822b7afc6082d1e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f512eff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f512eff5640 1 -- 172.21.15.62:0/3901058140 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5140003e20 con 0x55b598182690 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f51536f1740 1 -- 172.21.15.62:0/3901058140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513c0d8e10 con 0x55b598182690 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f512eff5640 1 -- 172.21.15.62:0/3901058140 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5140011440 con 0x55b598182690 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f512eff5640 1 -- 172.21.15.62:0/3901058140 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5140011640 con 0x55b598182690 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f51536f1740 10 client.18762 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:28.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f51536f1740 1 -- 172.21.15.62:0/3901058140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b59819b790 con 0x55b598182690 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f512eff5640 1 -- 172.21.15.62:0/3901058140 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c0d8e10 con 0x55b598182690 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.287+0000 7f512eff5640 1 -- 172.21.15.62:0/3901058140 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f514001b9b0 con 0x55b598182690 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.288+0000 7f51536f1740 10 client.18762 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.288+0000 7f51536f1740 10 client.18762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.288+0000 7f51536f1740 20 client.18762 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.288+0000 7f51536f1740 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b59819ccf0 0x55b59819f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.288+0000 7f51536f1740 4 client.18762 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.288+0000 7f51536f1740 1 -- 172.21.15.62:0/3901058140 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5980f47f0 con 0x55b59819ccf0 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.288+0000 7f514affd640 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b59819ccf0 0x55b59819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.292+0000 7f514affd640 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b59819ccf0 0x55b59819f110 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8317156e34a5cdce server_cookie=a04833b4c7570146 in_seq=0 out_seq=0 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.292+0000 7f512eff5640 10 client.18762 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f512eff5640 1 -- 172.21.15.62:0/3901058140 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b5980f47f0 con 0x55b59819ccf0 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f512eff5640 10 client.18762 handle_command_reply: tid=0 2024-09-15T23:30:28.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 client.18762 shutdown 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 -- 172.21.15.62:0/3901058140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598182690 msgr2=0x55b598189d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598182690 0x55b598189d10 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f514000bad0 tx=0x7f514000bf90 comp rx=0 tx=0).stop 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 -- 172.21.15.62:0/3901058140 shutdown_connections 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b59819ccf0 0x55b59819f110 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b59818cf50 0x55b59818a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598182690 0x55b598189d10 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f512eff5640 0 client.18762 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 --2- 172.21.15.62:0/3901058140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b598181d70 0x55b59818c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 -- 172.21.15.62:0/3901058140 >> 172.21.15.62:0/3901058140 conn(0x55b59817e300 msgr2=0x55b598183fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 -- 172.21.15.62:0/3901058140 shutdown_connections 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 1 -- 172.21.15.62:0/3901058140 wait complete. 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.293+0000 7f51536f1740 20 client.18762 trim_cache size 0 max 16384 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f51536f1740 1 Processor -- start 2024-09-15T23:30:28.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f51536f1740 1 -- start start 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f51536f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598189c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f51536f1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b598198e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f51536f1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b5981a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f51536f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5144153e00 con 0x55b598181d70 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f51536f1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f51441516a0 con 0x55b59818cf50 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f51536f1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f514414f4f0 con 0x55b598182690 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f514affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598189c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f514affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598189c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33870/0 (socket says 172.21.15.62:33870) 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f514affd640 1 -- 172.21.15.62:0/3621885542 learned_addr learned my addr 172.21.15.62:0/3621885542 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f514a7fc640 1 --2- 172.21.15.62:0/3621885542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b598198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.311+0000 7f514b7fe640 1 --2- 172.21.15.62:0/3621885542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b5981a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.312+0000 7f514affd640 1 -- 172.21.15.62:0/3621885542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 msgr2=0x55b598198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.312+0000 7f514affd640 1 --2- 172.21.15.62:0/3621885542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b598198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.312+0000 7f514affd640 1 -- 172.21.15.62:0/3621885542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 msgr2=0x55b5981a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.312+0000 7f514affd640 1 --2- 172.21.15.62:0/3621885542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b5981a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.312+0000 7f514affd640 1 -- 172.21.15.62:0/3621885542 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5981824c0 con 0x55b598181d70 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.312+0000 7f514affd640 1 --2- 172.21.15.62:0/3621885542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598189c40 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f513c0986f0 tx=0x7f513c0d39e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92dd80d9ccfa778 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.312+0000 7f5130ff9640 1 -- 172.21.15.62:0/3621885542 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c09c060 con 0x55b598181d70 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.312+0000 7f5130ff9640 1 -- 172.21.15.62:0/3621885542 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c09c290 con 0x55b598181d70 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.312+0000 7f5130ff9640 1 -- 172.21.15.62:0/3621885542 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c09bdd0 con 0x55b598181d70 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 -- 172.21.15.62:0/3621885542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 msgr2=0x55b598189c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 --2- 172.21.15.62:0/3621885542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598189c40 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f513c0986f0 tx=0x7f513c0d39e0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 -- 172.21.15.62:0/3621885542 shutdown_connections 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 --2- 172.21.15.62:0/3621885542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b5981a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 --2- 172.21.15.62:0/3621885542 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b598198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 --2- 172.21.15.62:0/3621885542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598189c40 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 -- 172.21.15.62:0/3621885542 >> 172.21.15.62:0/3621885542 conn(0x55b59817e300 msgr2=0x55b59818bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 -- 172.21.15.62:0/3621885542 shutdown_connections 2024-09-15T23:30:28.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 -- 172.21.15.62:0/3621885542 wait complete. 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 -- start start 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 Processor -- start 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598186ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b598187420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b598187960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f514414fb60 con 0x55b598181d70 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5144150fd0 con 0x55b59818cf50 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5144150230 con 0x55b598182690 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f51536f1740 10 client.0 _ll_register_callbacks cb 0x7f5140004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f514affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598186ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f514affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598186ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33880/0 (socket says 172.21.15.62:33880) 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.313+0000 7f514affd640 1 -- 172.21.15.62:0/1395062470 learned_addr learned my addr 172.21.15.62:0/1395062470 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f514a7fc640 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b598187420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f514affd640 1 -- 172.21.15.62:0/1395062470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 msgr2=0x55b598187420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f514affd640 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b598187420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f514affd640 1 -- 172.21.15.62:0/1395062470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 msgr2=0x55b598187960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f514affd640 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b598187960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f514affd640 1 -- 172.21.15.62:0/1395062470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b598193930 con 0x55b598181d70 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f514a7fc640 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b598187420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f514affd640 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598186ee0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f513c0d8ae0 tx=0x7f513c0d38b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10233c0931e847ce server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f512eff5640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f512eff5640 1 -- 172.21.15.62:0/1395062470 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c09c060 con 0x55b598181d70 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f512eff5640 1 -- 172.21.15.62:0/1395062470 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f513c0d3af0 con 0x55b598181d70 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f51536f1740 1 -- 172.21.15.62:0/1395062470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513c09c290 con 0x55b598181d70 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f512eff5640 1 -- 172.21.15.62:0/1395062470 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f513c0a0590 con 0x55b598181d70 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f51536f1740 10 client.18768 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f51536f1740 1 -- 172.21.15.62:0/1395062470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b598207e60 con 0x55b598181d70 2024-09-15T23:30:28.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.314+0000 7f512eff5640 1 -- 172.21.15.62:0/1395062470 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c09c290 con 0x55b598181d70 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f512eff5640 1 -- 172.21.15.62:0/1395062470 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f513c098900 con 0x55b598181d70 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f51536f1740 10 client.18768 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f51536f1740 10 client.18768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f51536f1740 20 client.18768 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f51536f1740 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5982080e0 0x55b59820a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f51536f1740 4 client.18768 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f51536f1740 1 -- 172.21.15.62:0/1395062470 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b5980eed00 con 0x55b5982080e0 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f514a7fc640 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5982080e0 0x55b59820a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f514a7fc640 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5982080e0 0x55b59820a4d0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd06b147d0aa8a05 server_cookie=41f373d824a84eaf in_seq=0 out_seq=0 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.315+0000 7f512eff5640 10 client.18768 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.316+0000 7f512eff5640 1 -- 172.21.15.62:0/1395062470 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b5980eed00 con 0x55b5982080e0 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.316+0000 7f512eff5640 10 client.18768 handle_command_reply: tid=0 2024-09-15T23:30:28.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 client.18768 shutdown 2024-09-15T23:30:28.798 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:30:28.798 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:30:28.798 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "0297da35-b0b2-4cbc-a5be-aaeb09781952", 2024-09-15T23:30:28.798 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:30:28.798 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:30:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 -- 172.21.15.62:0/1395062470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 msgr2=0x55b598186ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598186ee0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f513c0d8ae0 tx=0x7f513c0d38b0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 -- 172.21.15.62:0/1395062470 shutdown_connections 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55b5982080e0 0x55b59820a4d0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b59818cf50 0x55b598187960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55b598182690 0x55b598187420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f512eff5640 0 client.18768 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 --2- 172.21.15.62:0/1395062470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b598181d70 0x55b598186ee0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 -- 172.21.15.62:0/1395062470 >> 172.21.15.62:0/1395062470 conn(0x55b59817e300 msgr2=0x55b59818b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 -- 172.21.15.62:0/1395062470 shutdown_connections 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 1 -- 172.21.15.62:0/1395062470 wait complete. 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f51536f1740 20 client.18768 trim_cache size 0 max 16384 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 -- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51340a3250 msgr2=0x7f51340a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51340a3250 0x7f51340a5710 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f514c065360 tx=0x7f514c068410 comp rx=0 tx=0).stop 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 -- 172.21.15.62:0/2294860381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5144004cf0 msgr2=0x7f51440a8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:28.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5144004cf0 0x7f51440a8d60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f513c00e9f0 tx=0x7f513c00eeb0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 -- 172.21.15.62:0/2294860381 shutdown_connections 2024-09-15T23:30:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f51340a3250 0x7f51340a5710 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f514409a4e0 0x7f51440a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144005780 0x7f51440a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 --2- 172.21.15.62:0/2294860381 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5144004cf0 0x7f51440a8d60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.317+0000 7f5151318640 1 -- 172.21.15.62:0/2294860381 >> 172.21.15.62:0/2294860381 conn(0x7f514400a210 msgr2=0x7f5144096fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.318+0000 7f5151318640 1 -- 172.21.15.62:0/2294860381 shutdown_connections 2024-09-15T23:30:28.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:28.318+0000 7f5151318640 1 -- 172.21.15.62:0/2294860381 wait complete. 2024-09-15T23:30:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:28 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:30:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:28 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:29.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:28 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:30:29.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:28 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:30:29.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:28 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:28 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:30:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:28 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:28 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:30:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:28 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:30:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:28 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:28 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:30:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:28 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:28 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:30:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:28 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:30:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:28 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:29.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.288+0000 7fc463303640 1 -- 172.21.15.62:0/827241546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc45c101c30 msgr2=0x7fc45c1020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:29.292 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.288+0000 7fc463303640 1 --2- 172.21.15.62:0/827241546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc45c101c30 0x7fc45c1020b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc44c01a910 tx=0x7fc44c03b470 comp rx=0 tx=0).stop 2024-09-15T23:30:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.289+0000 7fc463303640 1 -- 172.21.15.62:0/827241546 shutdown_connections 2024-09-15T23:30:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.289+0000 7fc463303640 1 --2- 172.21.15.62:0/827241546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45c1025f0 0x7fc45c108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.289+0000 7fc463303640 1 --2- 172.21.15.62:0/827241546 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc45c101c30 0x7fc45c1020b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.289+0000 7fc463303640 1 --2- 172.21.15.62:0/827241546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45c100a40 0x7fc45c100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.289+0000 7fc463303640 1 -- 172.21.15.62:0/827241546 >> 172.21.15.62:0/827241546 conn(0x7fc45c0fc1e0 msgr2=0x7fc45c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.289+0000 7fc463303640 1 -- 172.21.15.62:0/827241546 shutdown_connections 2024-09-15T23:30:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.289+0000 7fc463303640 1 -- 172.21.15.62:0/827241546 wait complete. 2024-09-15T23:30:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.290+0000 7fc463303640 1 Processor -- start 2024-09-15T23:30:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.290+0000 7fc463303640 1 -- start start 2024-09-15T23:30:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.295+0000 7fc463303640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45c100a40 0x7fc45c108070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.295+0000 7fc463303640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc45c101c30 0x7fc45c1085b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.295+0000 7fc463303640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45c1025f0 0x7fc45c10fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.295+0000 7fc463303640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c06d7c0 con 0x7fc45c100a40 2024-09-15T23:30:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.295+0000 7fc463303640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c06d640 con 0x7fc45c1025f0 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.295+0000 7fc463303640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c06d940 con 0x7fc45c101c30 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.295+0000 7fc46109f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45c100a40 0x7fc45c108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.295+0000 7fc46109f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45c100a40 0x7fc45c108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33894/0 (socket says 172.21.15.62:33894) 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.296+0000 7fc46109f640 1 -- 172.21.15.62:0/831584602 learned_addr learned my addr 172.21.15.62:0/831584602 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.296+0000 7fc4618a0640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45c1025f0 0x7fc45c10fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.297+0000 7fc4618a0640 1 -- 172.21.15.62:0/831584602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc45c101c30 msgr2=0x7fc45c1085b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.297+0000 7fc4618a0640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc45c101c30 0x7fc45c1085b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.297+0000 7fc4618a0640 1 -- 172.21.15.62:0/831584602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45c100a40 msgr2=0x7fc45c108070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.297+0000 7fc4618a0640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45c100a40 0x7fc45c108070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.297+0000 7fc4618a0640 1 -- 172.21.15.62:0/831584602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45c1103a0 con 0x7fc45c1025f0 2024-09-15T23:30:29.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.315+0000 7fc4618a0640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45c1025f0 0x7fc45c10fd00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc45400ca30 tx=0x7fc45400cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.320+0000 7fc4527fc640 1 -- 172.21.15.62:0/831584602 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454007d90 con 0x7fc45c1025f0 2024-09-15T23:30:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.320+0000 7fc4527fc640 1 -- 172.21.15.62:0/831584602 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc454005020 con 0x7fc45c1025f0 2024-09-15T23:30:29.323 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.320+0000 7fc4527fc640 1 -- 172.21.15.62:0/831584602 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45400f780 con 0x7fc45c1025f0 2024-09-15T23:30:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.322+0000 7fc463303640 1 -- 172.21.15.62:0/831584602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc45c110690 con 0x7fc45c1025f0 2024-09-15T23:30:29.324 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.322+0000 7fc463303640 1 -- 172.21.15.62:0/831584602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc45c110be0 con 0x7fc45c1025f0 2024-09-15T23:30:29.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.329+0000 7fc46609a740 1 Processor -- start 2024-09-15T23:30:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.331+0000 7fc46609a740 1 -- start start 2024-09-15T23:30:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.331+0000 7fc46609a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f1fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.331+0000 7fc46609a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f202e0 0x564cf6f24da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.331+0000 7fc46609a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f25370 0x564cf6f27830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.331+0000 7fc46609a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564cf6e832f0 con 0x564cf6f25370 2024-09-15T23:30:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.331+0000 7fc46609a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564cf6e2b270 con 0x564cf6f1f9a0 2024-09-15T23:30:29.334 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.331+0000 7fc46609a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564cf6e7f2b0 con 0x564cf6f202e0 2024-09-15T23:30:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.332+0000 7fc4527fc640 1 -- 172.21.15.62:0/831584602 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fc4540051d0 con 0x7fc45c1025f0 2024-09-15T23:30:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.333+0000 7fc4527fc640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc4480a34e0 0x7fc4480a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.333+0000 7fc46109f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f202e0 0x564cf6f24da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.333+0000 7fc46109f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f202e0 0x564cf6f24da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38248/0 (socket says 172.21.15.62:38248) 2024-09-15T23:30:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.333+0000 7fc46109f640 1 -- 172.21.15.62:0/2987608448 learned_addr learned my addr 172.21.15.62:0/2987608448 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:29.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.333+0000 7fc4527fc640 1 -- 172.21.15.62:0/831584602 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc4540d0220 con 0x7fc45c1025f0 2024-09-15T23:30:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.334+0000 7fc46109f640 1 -- 172.21.15.62:0/2987608448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 msgr2=0x564cf6f1fda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:30:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.334+0000 7fc46109f640 1 --2- 172.21.15.62:0/2987608448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f1fda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.334+0000 7fc46109f640 1 -- 172.21.15.62:0/2987608448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f25370 msgr2=0x564cf6f27830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:30:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.334+0000 7fc46109f640 1 --2- 172.21.15.62:0/2987608448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f25370 0x564cf6f27830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.334+0000 7fc46109f640 1 -- 172.21.15.62:0/2987608448 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cf6e2a340 con 0x564cf6f202e0 2024-09-15T23:30:29.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.334+0000 7fc46109f640 1 --2- 172.21.15.62:0/2987608448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f202e0 0x564cf6f24da0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc458009aa0 tx=0x7fc458009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daa2d3a68562d262 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.337+0000 7fc4618a0640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc4480a34e0 0x7fc4480a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.338+0000 7fc4477fe640 1 -- 172.21.15.62:0/2987608448 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc458017070 con 0x564cf6f202e0 2024-09-15T23:30:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.338+0000 7fc4477fe640 1 -- 172.21.15.62:0/2987608448 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc45800eb70 con 0x564cf6f202e0 2024-09-15T23:30:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.338+0000 7fc4477fe640 1 -- 172.21.15.62:0/2987608448 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc458012630 con 0x564cf6f202e0 2024-09-15T23:30:29.342 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.339+0000 7fc46609a740 1 -- 172.21.15.62:0/2987608448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f202e0 msgr2=0x564cf6f24da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.339+0000 7fc46609a740 1 --2- 172.21.15.62:0/2987608448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f202e0 0x564cf6f24da0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc458009aa0 tx=0x7fc458009f60 comp rx=0 tx=0).stop 2024-09-15T23:30:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.340+0000 7fc46609a740 1 -- 172.21.15.62:0/2987608448 shutdown_connections 2024-09-15T23:30:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.340+0000 7fc46609a740 1 --2- 172.21.15.62:0/2987608448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f25370 0x564cf6f27830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.340+0000 7fc46609a740 1 --2- 172.21.15.62:0/2987608448 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f202e0 0x564cf6f24da0 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc458009aa0 tx=0x7fc458009f60 comp rx=0 tx=0).stop 2024-09-15T23:30:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.340+0000 7fc46609a740 1 --2- 172.21.15.62:0/2987608448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f1fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.343 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.340+0000 7fc46609a740 1 -- 172.21.15.62:0/2987608448 >> 172.21.15.62:0/2987608448 conn(0x564cf6f1b010 msgr2=0x564cf6f1b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.345+0000 7fc46609a740 1 -- 172.21.15.62:0/2987608448 shutdown_connections 2024-09-15T23:30:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.345+0000 7fc4618a0640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc4480a34e0 0x7fc4480a59a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc454098e20 tx=0x7fc454005b40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.349+0000 7fc46609a740 1 -- 172.21.15.62:0/2987608448 wait complete. 2024-09-15T23:30:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.350+0000 7fc46609a740 1 -- start start 2024-09-15T23:30:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.350+0000 7fc46609a740 1 Processor -- start 2024-09-15T23:30:29.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.351+0000 7fc46609a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.351+0000 7fc46609a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f25370 0x564cf6f2a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.351+0000 7fc46609a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f317f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.351+0000 7fc46609a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564cf6f28340 con 0x564cf6f2f400 2024-09-15T23:30:29.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.351+0000 7fc46609a740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564cf6e0c290 con 0x564cf6f1f9a0 2024-09-15T23:30:29.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.351+0000 7fc46609a740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564cf6f284c0 con 0x564cf6f25370 2024-09-15T23:30:29.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.351+0000 7fc46609a740 10 client.0 _ll_register_callbacks cb 0x564cf6e3d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:29.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.351+0000 7fc46089e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.354 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.352+0000 7fc46089e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40376/0 (socket says 172.21.15.62:40376) 2024-09-15T23:30:29.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.352+0000 7fc46089e640 1 -- 172.21.15.62:0/1833860890 learned_addr learned my addr 172.21.15.62:0/1833860890 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:29.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.352+0000 7fc46089e640 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.352+0000 7fc46109f640 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f25370 0x564cf6f2a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.352+0000 7fc46089e640 1 -- 172.21.15.62:0/1833860890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f25370 msgr2=0x564cf6f2a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:29.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.352+0000 7fc46089e640 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f25370 0x564cf6f2a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.352+0000 7fc46089e640 1 -- 172.21.15.62:0/1833860890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 msgr2=0x564cf6f317f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:29.355 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.352+0000 7fc46089e640 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f317f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.356 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.352+0000 7fc46089e640 1 -- 172.21.15.62:0/1833860890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cf6b96210 con 0x564cf6f1f9a0 2024-09-15T23:30:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.358+0000 7fc46089e640 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29e90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc44c005ec0 tx=0x7fc44c004aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93164381e8fd6bff server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.359+0000 7fc444ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:30:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.359+0000 7fc444ff9640 1 -- 172.21.15.62:0/1833860890 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c0047b0 con 0x564cf6f1f9a0 2024-09-15T23:30:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.359+0000 7fc444ff9640 1 -- 172.21.15.62:0/1833860890 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc44c019030 con 0x564cf6f1f9a0 2024-09-15T23:30:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.360+0000 7fc46609a740 1 -- 172.21.15.62:0/1833860890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc45800eb70 con 0x564cf6f1f9a0 2024-09-15T23:30:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.363+0000 7fc444ff9640 1 -- 172.21.15.62:0/1833860890 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc44c051450 con 0x564cf6f1f9a0 2024-09-15T23:30:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.364+0000 7fc46609a740 10 client.28280 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.364+0000 7fc46609a740 1 -- 172.21.15.62:0/1833860890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564cf6c62620 con 0x564cf6f1f9a0 2024-09-15T23:30:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.366+0000 7fc444ff9640 1 -- 172.21.15.62:0/1833860890 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc45800eb70 con 0x564cf6f1f9a0 2024-09-15T23:30:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.367+0000 7fc444ff9640 1 -- 172.21.15.62:0/1833860890 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fc44c0504b0 con 0x564cf6f1f9a0 2024-09-15T23:30:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.367+0000 7fc46609a740 10 client.28280 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.367+0000 7fc46609a740 10 client.28280 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.367+0000 7fc46609a740 20 client.28280 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.367+0000 7fc46609a740 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cf6f36270 0x564cf6f386e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.367+0000 7fc46609a740 4 client.28280 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.367+0000 7fc46609a740 1 -- 172.21.15.62:0/1833860890 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564cf6e90a20 con 0x564cf6f36270 2024-09-15T23:30:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.371+0000 7fc46109f640 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cf6f36270 0x564cf6f386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.371+0000 7fc46109f640 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cf6f36270 0x564cf6f386e0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f24e01cb757b7201 server_cookie=fd4e9df7f39367a7 in_seq=0 out_seq=0 2024-09-15T23:30:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.371+0000 7fc444ff9640 10 client.28280 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:29.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.372+0000 7fc444ff9640 1 -- 172.21.15.62:0/1833860890 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564cf6e90a20 con 0x564cf6f36270 2024-09-15T23:30:29.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.372+0000 7fc444ff9640 10 client.28280 handle_command_reply: tid=0 2024-09-15T23:30:29.374 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.372+0000 7fc46609a740 1 client.28280 shutdown 2024-09-15T23:30:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.373+0000 7fc46609a740 1 -- 172.21.15.62:0/1833860890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 msgr2=0x564cf6f29e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.373+0000 7fc46609a740 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29e90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fc44c005ec0 tx=0x7fc44c004aa0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.373+0000 7fc46609a740 1 -- 172.21.15.62:0/1833860890 shutdown_connections 2024-09-15T23:30:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.373+0000 7fc46609a740 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cf6f36270 0x564cf6f386e0 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.373+0000 7fc46609a740 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f317f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.373+0000 7fc46609a740 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f25370 0x564cf6f2a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.373+0000 7fc46609a740 1 --2- 172.21.15.62:0/1833860890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29e90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.373+0000 7fc46609a740 1 -- 172.21.15.62:0/1833860890 >> 172.21.15.62:0/1833860890 conn(0x564cf6f1b010 msgr2=0x564cf6f1c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.373+0000 7fc444ff9640 0 client.28280 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.374+0000 7fc46609a740 1 -- 172.21.15.62:0/1833860890 shutdown_connections 2024-09-15T23:30:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.375+0000 7fc46609a740 1 -- 172.21.15.62:0/1833860890 wait complete. 2024-09-15T23:30:29.378 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.376+0000 7fc46609a740 20 client.28280 trim_cache size 0 max 16384 2024-09-15T23:30:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.415+0000 7fc46609a740 1 Processor -- start 2024-09-15T23:30:29.421 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.420+0000 7fc46609a740 1 -- start start 2024-09-15T23:30:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.420+0000 7fc46609a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.420+0000 7fc46609a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f25370 0x564cf6f2a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.420+0000 7fc46609a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f1be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.420+0000 7fc46089e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46089e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38270/0 (socket says 172.21.15.62:38270) 2024-09-15T23:30:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46089e640 1 -- 172.21.15.62:0/2798348014 learned_addr learned my addr 172.21.15.62:0/2798348014 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46609a740 1 -- 172.21.15.62:0/2798348014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c11d8c0 con 0x564cf6f2f400 2024-09-15T23:30:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46609a740 1 -- 172.21.15.62:0/2798348014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c11b160 con 0x564cf6f25370 2024-09-15T23:30:29.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46609a740 1 -- 172.21.15.62:0/2798348014 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c119060 con 0x564cf6f1f9a0 2024-09-15T23:30:29.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46089e640 1 --2- 172.21.15.62:0/2798348014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f1be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46089e640 1 -- 172.21.15.62:0/2798348014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f25370 msgr2=0x564cf6f2a180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:29.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46089e640 1 --2- 172.21.15.62:0/2798348014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f25370 0x564cf6f2a180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46089e640 1 -- 172.21.15.62:0/2798348014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 msgr2=0x564cf6f1be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46089e640 1 --2- 172.21.15.62:0/2798348014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f1be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.421+0000 7fc46089e640 1 -- 172.21.15.62:0/2798348014 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cf6f1ae30 con 0x564cf6f1f9a0 2024-09-15T23:30:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.422+0000 7fc46089e640 1 --2- 172.21.15.62:0/2798348014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29c40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc44c03b990 tx=0x7fc44c03b470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e3a2e83229f2d95 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.422+0000 7fc446ffd640 1 -- 172.21.15.62:0/2798348014 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c050aa0 con 0x564cf6f1f9a0 2024-09-15T23:30:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.422+0000 7fc446ffd640 1 -- 172.21.15.62:0/2798348014 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44c050c40 con 0x564cf6f1f9a0 2024-09-15T23:30:29.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.422+0000 7fc446ffd640 1 -- 172.21.15.62:0/2798348014 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c051cf0 con 0x564cf6f1f9a0 2024-09-15T23:30:29.426 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.424+0000 7fc46609a740 1 -- 172.21.15.62:0/2798348014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f1f9a0 msgr2=0x564cf6f29c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:29.426 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.424+0000 7fc46609a740 1 --2- 172.21.15.62:0/2798348014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29c40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc44c03b990 tx=0x7fc44c03b470 comp rx=0 tx=0).stop 2024-09-15T23:30:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.428+0000 7fc46609a740 1 -- 172.21.15.62:0/2798348014 shutdown_connections 2024-09-15T23:30:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.428+0000 7fc46609a740 1 --2- 172.21.15.62:0/2798348014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f1be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.428+0000 7fc46609a740 1 --2- 172.21.15.62:0/2798348014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f25370 0x564cf6f2a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.428+0000 7fc46609a740 1 --2- 172.21.15.62:0/2798348014 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f1f9a0 0x564cf6f29c40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.428+0000 7fc46609a740 1 -- 172.21.15.62:0/2798348014 >> 172.21.15.62:0/2798348014 conn(0x564cf6f1b010 msgr2=0x564cf6f21660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.431+0000 7fc46609a740 1 -- 172.21.15.62:0/2798348014 shutdown_connections 2024-09-15T23:30:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.431+0000 7fc46609a740 1 -- 172.21.15.62:0/2798348014 wait complete. 2024-09-15T23:30:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.432+0000 7fc46609a740 1 -- start start 2024-09-15T23:30:29.434 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.432+0000 7fc46609a740 1 Processor -- start 2024-09-15T23:30:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.433+0000 7fc46609a740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f478d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.433+0000 7fc46609a740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f25370 0x564cf6f47e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.433+0000 7fc46089e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.434+0000 7fc46089e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f478d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40392/0 (socket says 172.21.15.62:40392) 2024-09-15T23:30:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.434+0000 7fc46089e640 1 -- 172.21.15.62:0/3747577320 learned_addr learned my addr 172.21.15.62:0/3747577320 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.434+0000 7fc46609a740 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f48e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.434+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c119620 con 0x564cf6f2f400 2024-09-15T23:30:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.434+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c11aa90 con 0x564cf6f1f9a0 2024-09-15T23:30:29.436 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.434+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fc45c119cf0 con 0x564cf6f25370 2024-09-15T23:30:29.437 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.434+0000 7fc46609a740 10 client.0 _ll_register_callbacks cb 0x7fc44c01a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:29.438 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.436+0000 7fc46089e640 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f48e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.436+0000 7fc46089e640 1 -- 172.21.15.62:0/3747577320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f25370 msgr2=0x564cf6f47e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.436+0000 7fc46089e640 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f25370 0x564cf6f47e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.436+0000 7fc46089e640 1 -- 172.21.15.62:0/3747577320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 msgr2=0x564cf6f48e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.436+0000 7fc46089e640 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f48e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.439 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.436+0000 7fc46089e640 1 -- 172.21.15.62:0/3747577320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cf6f49430 con 0x564cf6f1f9a0 2024-09-15T23:30:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.436+0000 7fc46089e640 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f478d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc44c054000 tx=0x7fc44c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba47f304aa2a9ab7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.436+0000 7fc444ff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:30:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.437+0000 7fc444ff9640 1 -- 172.21.15.62:0/3747577320 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44c005d50 con 0x564cf6f1f9a0 2024-09-15T23:30:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.437+0000 7fc444ff9640 1 -- 172.21.15.62:0/3747577320 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fc44c019040 con 0x564cf6f1f9a0 2024-09-15T23:30:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.437+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc44c050c40 con 0x564cf6f1f9a0 2024-09-15T23:30:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.437+0000 7fc444ff9640 1 -- 172.21.15.62:0/3747577320 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc44c074050 con 0x564cf6f1f9a0 2024-09-15T23:30:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.437+0000 7fc46609a740 10 client.28283 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.437+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564cf6f4a750 con 0x564cf6f1f9a0 2024-09-15T23:30:29.440 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.437+0000 7fc444ff9640 1 -- 172.21.15.62:0/3747577320 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44c050c40 con 0x564cf6f1f9a0 2024-09-15T23:30:29.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.438+0000 7fc444ff9640 1 -- 172.21.15.62:0/3747577320 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fc44c042440 con 0x564cf6f1f9a0 2024-09-15T23:30:29.441 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.438+0000 7fc46609a740 10 client.28283 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.438+0000 7fc46609a740 10 client.28283 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.438+0000 7fc46609a740 20 client.28283 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.439+0000 7fc46609a740 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cf6fa1340 0x564cf6fa3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.439+0000 7fc46609a740 4 client.28283 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:30:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.439+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564cf6e8ad70 con 0x564cf6fa1340 2024-09-15T23:30:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.439+0000 7fc46109f640 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cf6fa1340 0x564cf6fa3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.439+0000 7fc46109f640 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cf6fa1340 0x564cf6fa3760 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=714ea3ddfe97b4a4 server_cookie=496730d385d1a987 in_seq=0 out_seq=0 2024-09-15T23:30:29.442 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.440+0000 7fc444ff9640 10 client.28283 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.466+0000 7fc444ff9640 1 -- 172.21.15.62:0/3747577320 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564cf6e8ad70 con 0x564cf6fa1340 2024-09-15T23:30:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.467+0000 7fc444ff9640 10 client.28283 handle_command_reply: tid=0 2024-09-15T23:30:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.467+0000 7fc46609a740 1 client.28283 shutdown 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.467+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 msgr2=0x564cf6f478d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.467+0000 7fc46609a740 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f478d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc44c054000 tx=0x7fc44c005fd0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.467+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 shutdown_connections 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.467+0000 7fc46609a740 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x564cf6fa1340 0x564cf6fa3760 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.467+0000 7fc46609a740 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564cf6f2f400 0x564cf6f48e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.468+0000 7fc46609a740 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564cf6f25370 0x564cf6f47e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.468+0000 7fc46609a740 1 --2- 172.21.15.62:0/3747577320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cf6f1f9a0 0x564cf6f478d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.468+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 >> 172.21.15.62:0/3747577320 conn(0x564cf6f1b010 msgr2=0x564cf6f29c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.468+0000 7fc444ff9640 0 client.28283 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.468+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 shutdown_connections 2024-09-15T23:30:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.468+0000 7fc46609a740 1 -- 172.21.15.62:0/3747577320 wait complete. 2024-09-15T23:30:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.468+0000 7fc46609a740 20 client.28283 trim_cache size 0 max 16384 2024-09-15T23:30:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:30:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:30:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:30:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:30:29.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 -- 172.21.15.62:0/831584602 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc4480a34e0 msgr2=0x7fc4480a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:29.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc4480a34e0 0x7fc4480a59a0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fc454098e20 tx=0x7fc454005b40 comp rx=0 tx=0).stop 2024-09-15T23:30:29.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 -- 172.21.15.62:0/831584602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45c1025f0 msgr2=0x7fc45c10fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:29.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45c1025f0 0x7fc45c10fd00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fc45400ca30 tx=0x7fc45400cef0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 -- 172.21.15.62:0/831584602 shutdown_connections 2024-09-15T23:30:29.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fc4480a34e0 0x7fc4480a59a0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.473 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc45c1025f0 0x7fc45c10fd00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fc45c101c30 0x7fc45c1085b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 --2- 172.21.15.62:0/831584602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc45c100a40 0x7fc45c108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 -- 172.21.15.62:0/831584602 >> 172.21.15.62:0/831584602 conn(0x7fc45c0fc1e0 msgr2=0x7fc45c070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 -- 172.21.15.62:0/831584602 shutdown_connections 2024-09-15T23:30:29.474 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:29.470+0000 7fc463303640 1 -- 172.21.15.62:0/831584602 wait complete. 2024-09-15T23:30:29.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 850.0 s, 433.8 tps, lat 218.306 ms stddev 265.162 2024-09-15T23:30:30.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:29 smithi062 ceph-mon[30949]: pgmap v796: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 4.4 MiB/s wr, 495 op/s 2024-09-15T23:30:30.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:29 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:30:30.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:29 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:30:30.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:29 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:30.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:29 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:30.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:29 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:30.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:29 smithi067 ceph-mon[35665]: pgmap v796: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 4.4 MiB/s wr, 495 op/s 2024-09-15T23:30:30.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:29 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:30:30.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:29 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:30:30.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:29 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:30.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:29 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:30.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:29 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:29 smithi148 ceph-mon[35610]: pgmap v796: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 4.4 MiB/s wr, 495 op/s 2024-09-15T23:30:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:29 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:30:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:29 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:30:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:29 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:30.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:29 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:30.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:29 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:30:30.385 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:30:30.894 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:31.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.506+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/3741902863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c2810e970 msgr2=0x7f4c28115310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:31.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.506+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/3741902863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c2810e970 0x7f4c28115310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4c20005fd0 tx=0x7f4c2002b430 comp rx=0 tx=0).stop 2024-09-15T23:30:31.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.507+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/3741902863 shutdown_connections 2024-09-15T23:30:31.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.507+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/3741902863 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c2810e970 0x7f4c28115310 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.507+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/3741902863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c28109e00 0x7f4c2810e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.507+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/3741902863 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28109450 0x7f4c28109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.507+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/3741902863 >> 172.21.15.62:0/3741902863 conn(0x7f4c28073180 msgr2=0x7f4c28073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:31.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.507+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/3741902863 shutdown_connections 2024-09-15T23:30:31.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.507+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/3741902863 wait complete. 2024-09-15T23:30:31.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.508+0000 7f4c2d8b9640 1 Processor -- start 2024-09-15T23:30:31.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.509+0000 7f4c2d8b9640 1 -- start start 2024-09-15T23:30:31.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.509+0000 7f4c2d8b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28109450 0x7f4c281a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.509+0000 7f4c2d8b9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c28109e00 0x7f4c281a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.509+0000 7f4c2d8b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c2810e970 0x7f4c2819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.509+0000 7f4c2c8b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28109450 0x7f4c281a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.510+0000 7f4c2c8b7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28109450 0x7f4c281a4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40398/0 (socket says 172.21.15.62:40398) 2024-09-15T23:30:31.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.510+0000 7f4c2c8b7640 1 -- 172.21.15.62:0/4098304255 learned_addr learned my addr 172.21.15.62:0/4098304255 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:31.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.510+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4c28118410 con 0x7f4c28109e00 2024-09-15T23:30:31.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.510+0000 7f4c2d0b8640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c2810e970 0x7f4c2819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.510+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c28118290 con 0x7f4c28109450 2024-09-15T23:30:31.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.510+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4c28118590 con 0x7f4c2810e970 2024-09-15T23:30:31.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.511+0000 7f4c27fff640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c28109e00 0x7f4c281a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.511+0000 7f4c2d0b8640 1 -- 172.21.15.62:0/4098304255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28109450 msgr2=0x7f4c281a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:31.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.511+0000 7f4c2d0b8640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28109450 0x7f4c281a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.511+0000 7f4c2d0b8640 1 -- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c28109e00 msgr2=0x7f4c281a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:31.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.511+0000 7f4c2d0b8640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c28109e00 0x7f4c281a4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.511+0000 7f4c2d0b8640 1 -- 172.21.15.62:0/4098304255 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2819ef70 con 0x7f4c2810e970 2024-09-15T23:30:31.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.512+0000 7f4c2d0b8640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c2810e970 0x7f4c2819e700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4c20035040 tx=0x7f4c2000a990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:31.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.512+0000 7f4c25ffb640 1 -- 172.21.15.62:0/4098304255 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c20042070 con 0x7f4c2810e970 2024-09-15T23:30:31.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.512+0000 7f4c25ffb640 1 -- 172.21.15.62:0/4098304255 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c2000a2d0 con 0x7f4c2810e970 2024-09-15T23:30:31.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.513+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2819f200 con 0x7f4c2810e970 2024-09-15T23:30:31.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.513+0000 7f4c25ffb640 1 -- 172.21.15.62:0/4098304255 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c20037040 con 0x7f4c2810e970 2024-09-15T23:30:31.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.513+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c281ab3e0 con 0x7f4c2810e970 2024-09-15T23:30:31.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.517+0000 7f4c30ef4740 1 Processor -- start 2024-09-15T23:30:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.518+0000 7f4c30ef4740 1 -- start start 2024-09-15T23:30:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.518+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.518+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b51c7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.518+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.518+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5599b51232e0 con 0x5599b51cc8b0 2024-09-15T23:30:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.518+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5599b50cace0 con 0x5599b51c7870 2024-09-15T23:30:31.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.518+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5599b511f2a0 con 0x5599b51cd1d0 2024-09-15T23:30:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.521+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.522+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.522+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33928/0 (socket says 172.21.15.62:33928) 2024-09-15T23:30:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.522+0000 7f4c27fff640 1 -- 172.21.15.62:0/2146315721 learned_addr learned my addr 172.21.15.62:0/2146315721 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.522+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/2146315721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b51c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.522+0000 7f4c25ffb640 1 -- 172.21.15.62:0/4098304255 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4c200327a0 con 0x7f4c2810e970 2024-09-15T23:30:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.523+0000 7f4c25ffb640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c180a31a0 0x7f4c180a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.524+0000 7f4c27fff640 1 -- 172.21.15.62:0/2146315721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 msgr2=0x5599b51c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:31.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.524+0000 7f4c27fff640 1 --2- 172.21.15.62:0/2146315721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b51c7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.524+0000 7f4c27fff640 1 -- 172.21.15.62:0/2146315721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 msgr2=0x5599b51ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.524+0000 7f4c27fff640 1 --2- 172.21.15.62:0/2146315721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.524+0000 7f4c27fff640 1 -- 172.21.15.62:0/2146315721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599b50af440 con 0x5599b51c7870 2024-09-15T23:30:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.525+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/2146315721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b51c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.525+0000 7f4c27fff640 1 --2- 172.21.15.62:0/2146315721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7cf0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009fc0 tx=0x7f4c14005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a38c6e4823096dc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.525+0000 7f4c2d0b8640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c180a31a0 0x7f4c180a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.526+0000 7f4c25ffb640 1 -- 172.21.15.62:0/4098304255 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4c200ef070 con 0x7f4c2810e970 2024-09-15T23:30:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.526+0000 7f4c12ffd640 1 -- 172.21.15.62:0/2146315721 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1401a070 con 0x5599b51c7870 2024-09-15T23:30:31.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.526+0000 7f4c12ffd640 1 -- 172.21.15.62:0/2146315721 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c14003610 con 0x5599b51c7870 2024-09-15T23:30:31.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.526+0000 7f4c12ffd640 1 -- 172.21.15.62:0/2146315721 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14010660 con 0x5599b51c7870 2024-09-15T23:30:31.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.527+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2146315721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 msgr2=0x5599b51c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:31.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.527+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/2146315721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7cf0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009fc0 tx=0x7f4c14005850 comp rx=0 tx=0).stop 2024-09-15T23:30:31.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.529+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2146315721 shutdown_connections 2024-09-15T23:30:31.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.529+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/2146315721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7cf0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.529+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/2146315721 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b51c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.529+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/2146315721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.529+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2146315721 >> 172.21.15.62:0/2146315721 conn(0x5599b51b96b0 msgr2=0x5599b51b9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.531+0000 7f4c2d0b8640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c180a31a0 0x7f4c180a5660 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4c200bb240 tx=0x7f4c200f2740 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:31.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.531+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2146315721 shutdown_connections 2024-09-15T23:30:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.533+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2146315721 wait complete. 2024-09-15T23:30:31.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.533+0000 7f4c30ef4740 1 -- start start 2024-09-15T23:30:31.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.533+0000 7f4c30ef4740 1 Processor -- start 2024-09-15T23:30:31.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.534+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51c7870 0x5599b51d3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.534+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.534+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51cd1d0 0x5599b51d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.534+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5599b51d3c40 con 0x5599b51cc8b0 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5599b50de050 con 0x5599b51cd1d0 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5599b51d3dc0 con 0x5599b51c7870 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c2c8b7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c2c8b7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33932/0 (socket says 172.21.15.62:33932) 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c2c8b7640 1 -- 172.21.15.62:0/3503950456 learned_addr learned my addr 172.21.15.62:0/3503950456 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c30ef4740 10 client.0 _ll_register_callbacks cb 0x5599b511e160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c2c8b7640 1 -- 172.21.15.62:0/3503950456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51c7870 msgr2=0x5599b51d3120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51c7870 0x5599b51d3120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c2c8b7640 1 -- 172.21.15.62:0/3503950456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51cd1d0 msgr2=0x5599b51d09c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:31.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51cd1d0 0x5599b51d09c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c2c8b7640 1 -- 172.21.15.62:0/3503950456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599b4e943c0 con 0x5599b51cc8b0 2024-09-15T23:30:31.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.535+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51d03d0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009fd0 tx=0x7f4c1c0116f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe988d9f28c7eab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:31.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.536+0000 7f4c07fff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:30:31.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.539+0000 7f4c07fff640 1 -- 172.21.15.62:0/3503950456 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c019070 con 0x5599b51cc8b0 2024-09-15T23:30:31.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.539+0000 7f4c07fff640 1 -- 172.21.15.62:0/3503950456 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4c1c015870 con 0x5599b51cc8b0 2024-09-15T23:30:31.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.539+0000 7f4c30ef4740 1 -- 172.21.15.62:0/3503950456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c14003610 con 0x5599b51cc8b0 2024-09-15T23:30:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.543+0000 7f4c07fff640 1 -- 172.21.15.62:0/3503950456 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4c1c023050 con 0x5599b51cc8b0 2024-09-15T23:30:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.543+0000 7f4c30ef4740 10 client.18789 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:31.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.543+0000 7f4c30ef4740 1 -- 172.21.15.62:0/3503950456 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5599b51df770 con 0x5599b51cc8b0 2024-09-15T23:30:31.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.544+0000 7f4c07fff640 1 -- 172.21.15.62:0/3503950456 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c14003610 con 0x5599b51cc8b0 2024-09-15T23:30:31.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.544+0000 7f4c07fff640 1 -- 172.21.15.62:0/3503950456 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f4c1c01ec90 con 0x5599b51cc8b0 2024-09-15T23:30:31.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.545+0000 7f4c30ef4740 10 client.18789 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:31.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.545+0000 7f4c30ef4740 10 client.18789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:31.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.545+0000 7f4c30ef4740 20 client.18789 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:31.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.545+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5599b51e0c30 0x5599b51e3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.545+0000 7f4c30ef4740 4 client.18789 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:31.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.545+0000 7f4c30ef4740 1 -- 172.21.15.62:0/3503950456 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599b5130850 con 0x5599b51e0c30 2024-09-15T23:30:31.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.546+0000 7f4c27fff640 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5599b51e0c30 0x5599b51e3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.547+0000 7f4c07fff640 10 client.18789 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:31.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.547+0000 7f4c27fff640 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5599b51e0c30 0x5599b51e3060 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=500c90c311b6fa13 server_cookie=fd5d10395c37f325 in_seq=0 out_seq=0 2024-09-15T23:30:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.548+0000 7f4c07fff640 1 -- 172.21.15.62:0/3503950456 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5599b5130850 con 0x5599b51e0c30 2024-09-15T23:30:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.548+0000 7f4c07fff640 10 client.18789 handle_command_reply: tid=0 2024-09-15T23:30:31.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.548+0000 7f4c30ef4740 1 client.18789 shutdown 2024-09-15T23:30:31.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.549+0000 7f4c30ef4740 1 -- 172.21.15.62:0/3503950456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 msgr2=0x5599b51d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:31.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.549+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51d03d0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009fd0 tx=0x7f4c1c0116f0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.549+0000 7f4c30ef4740 1 -- 172.21.15.62:0/3503950456 shutdown_connections 2024-09-15T23:30:31.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.550+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5599b51e0c30 0x5599b51e3060 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.550+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51cd1d0 0x5599b51d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.550+0000 7f4c07fff640 0 client.18789 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:31.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.550+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cc8b0 0x5599b51d03d0 secure :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009fd0 tx=0x7f4c1c0116f0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.550+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/3503950456 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51c7870 0x5599b51d3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.550+0000 7f4c30ef4740 1 -- 172.21.15.62:0/3503950456 >> 172.21.15.62:0/3503950456 conn(0x5599b51b96b0 msgr2=0x5599b51b8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:31.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.551+0000 7f4c30ef4740 1 -- 172.21.15.62:0/3503950456 shutdown_connections 2024-09-15T23:30:31.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.552+0000 7f4c30ef4740 1 -- 172.21.15.62:0/3503950456 wait complete. 2024-09-15T23:30:31.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.552+0000 7f4c30ef4740 20 client.18789 trim_cache size 0 max 16384 2024-09-15T23:30:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.586+0000 7f4c30ef4740 1 Processor -- start 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.586+0000 7f4c30ef4740 1 -- start start 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.586+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.586+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cd1d0 0x5599b51dcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.586+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cbdc0 0x5599b51d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.586+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4c281b77d0 con 0x5599b51cd1d0 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.586+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c281b5070 con 0x5599b51c7870 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.586+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4c281b2ec0 con 0x5599b51cbdc0 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.590+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.590+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cbdc0 0x5599b51d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.590+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40418/0 (socket says 172.21.15.62:40418) 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.590+0000 7f4c27fff640 1 -- 172.21.15.62:0/2612351628 learned_addr learned my addr 172.21.15.62:0/2612351628 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:31.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.591+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/2612351628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cd1d0 0x5599b51dcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.591+0000 7f4c27fff640 1 -- 172.21.15.62:0/2612351628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cbdc0 msgr2=0x5599b51d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:31.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.591+0000 7f4c27fff640 1 --2- 172.21.15.62:0/2612351628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cbdc0 0x5599b51d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.591+0000 7f4c27fff640 1 -- 172.21.15.62:0/2612351628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cd1d0 msgr2=0x5599b51dcf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:31.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.591+0000 7f4c27fff640 1 --2- 172.21.15.62:0/2612351628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cd1d0 0x5599b51dcf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.591+0000 7f4c27fff640 1 -- 172.21.15.62:0/2612351628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599b51cc4f0 con 0x5599b51c7870 2024-09-15T23:30:31.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.595+0000 7f4c27fff640 1 --2- 172.21.15.62:0/2612351628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7200 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4c14014e30 tx=0x7f4c14009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=113d953c7f2b6c53 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:31.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.595+0000 7f4c127fc640 1 -- 172.21.15.62:0/2612351628 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1401a040 con 0x5599b51c7870 2024-09-15T23:30:31.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.595+0000 7f4c127fc640 1 -- 172.21.15.62:0/2612351628 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5599b5130850 con 0x5599b51c7870 2024-09-15T23:30:31.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.595+0000 7f4c127fc640 1 -- 172.21.15.62:0/2612351628 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14010d70 con 0x5599b51c7870 2024-09-15T23:30:31.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.596+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/2612351628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cd1d0 0x5599b51dcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:31.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.598+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2612351628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 msgr2=0x5599b51c7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:31.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.598+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/2612351628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7200 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4c14014e30 tx=0x7f4c14009f90 comp rx=0 tx=0).stop 2024-09-15T23:30:31.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.598+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2612351628 shutdown_connections 2024-09-15T23:30:31.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.598+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/2612351628 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cbdc0 0x5599b51d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.598+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/2612351628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51cd1d0 0x5599b51dcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.598+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/2612351628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51c7870 0x5599b51c7200 secure :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4c14014e30 tx=0x7f4c14009f90 comp rx=0 tx=0).stop 2024-09-15T23:30:31.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.598+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2612351628 >> 172.21.15.62:0/2612351628 conn(0x5599b51b96b0 msgr2=0x5599b51d1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:31.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.608+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2612351628 shutdown_connections 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.610+0000 7f4c30ef4740 1 -- 172.21.15.62:0/2612351628 wait complete. 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.610+0000 7f4c30ef4740 1 -- start start 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.610+0000 7f4c30ef4740 1 Processor -- start 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.611+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51cbdc0 0x5599b52460a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.611+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b52465e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.611+0000 7f4c30ef4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51d8100 0x5599b51d8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.611+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4c281b3530 con 0x5599b51d8100 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.611+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4c281b49a0 con 0x5599b51cbdc0 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.611+0000 7f4c30ef4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4c281b3c00 con 0x5599b51cd1d0 2024-09-15T23:30:31.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.611+0000 7f4c30ef4740 10 client.0 _ll_register_callbacks cb 0x7f4c1c009500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.612+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51d8100 0x5599b51d8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.612+0000 7f4c27fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51d8100 0x5599b51d8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:33948/0 (socket says 172.21.15.62:33948) 2024-09-15T23:30:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.612+0000 7f4c27fff640 1 -- 172.21.15.62:0/60188929 learned_addr learned my addr 172.21.15.62:0/60188929 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.622+0000 7f4c27fff640 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51cbdc0 0x5599b52460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.623+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b52465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.623+0000 7f4c2c8b7640 1 -- 172.21.15.62:0/60188929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51cbdc0 msgr2=0x5599b52460a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.623+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51cbdc0 0x5599b52460a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.623+0000 7f4c2c8b7640 1 -- 172.21.15.62:0/60188929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51d8100 msgr2=0x5599b51d8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:31.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.623+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51d8100 0x5599b51d8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.623+0000 7f4c2c8b7640 1 -- 172.21.15.62:0/60188929 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599b51d8b40 con 0x5599b51cd1d0 2024-09-15T23:30:31.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.624+0000 7f4c2c8b7640 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b52465e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x5599b51d14c0 tx=0x7f4c1c014c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72e0d5a042509ffa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:31.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.624+0000 7f4c07fff640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:31.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.624+0000 7f4c07fff640 1 -- 172.21.15.62:0/60188929 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c019070 con 0x5599b51cd1d0 2024-09-15T23:30:31.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.624+0000 7f4c30ef4740 1 -- 172.21.15.62:0/60188929 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5599b5130850 con 0x5599b51cd1d0 2024-09-15T23:30:31.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.627+0000 7f4c07fff640 1 -- 172.21.15.62:0/60188929 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4c1c005b70 con 0x5599b51cd1d0 2024-09-15T23:30:31.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.631+0000 7f4c07fff640 1 -- 172.21.15.62:0/60188929 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4c1c036050 con 0x5599b51cd1d0 2024-09-15T23:30:31.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.633+0000 7f4c30ef4740 10 client.27970 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:31.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.633+0000 7f4c30ef4740 1 -- 172.21.15.62:0/60188929 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5599b51d9d80 con 0x5599b51cd1d0 2024-09-15T23:30:31.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.634+0000 7f4c07fff640 1 -- 172.21.15.62:0/60188929 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5599b5130850 con 0x5599b51cd1d0 2024-09-15T23:30:31.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.634+0000 7f4c07fff640 1 -- 172.21.15.62:0/60188929 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f4c1c003870 con 0x5599b51cd1d0 2024-09-15T23:30:31.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.636+0000 7f4c30ef4740 10 client.27970 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:31.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.636+0000 7f4c30ef4740 10 client.27970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:31.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.636+0000 7f4c30ef4740 20 client.27970 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:31.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.636+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5599b51c3fe0 0x5599b51c6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:31.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.636+0000 7f4c30ef4740 4 client.27970 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:30:31.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.636+0000 7f4c30ef4740 1 -- 172.21.15.62:0/60188929 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5599b512ad60 con 0x5599b51c3fe0 2024-09-15T23:30:31.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.639+0000 7f4c27fff640 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5599b51c3fe0 0x5599b51c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:31.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.642+0000 7f4c27fff640 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5599b51c3fe0 0x5599b51c6400 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44216380c792d120 server_cookie=74776c72c7073a4 in_seq=0 out_seq=0 2024-09-15T23:30:31.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.643+0000 7f4c07fff640 10 client.27970 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:31.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.643+0000 7f4c07fff640 1 -- 172.21.15.62:0/60188929 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5599b512ad60 con 0x5599b51c3fe0 2024-09-15T23:30:31.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.643+0000 7f4c07fff640 10 client.27970 handle_command_reply: tid=0 2024-09-15T23:30:31.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.643+0000 7f4c30ef4740 1 client.27970 shutdown 2024-09-15T23:30:31.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.644+0000 7f4c30ef4740 1 -- 172.21.15.62:0/60188929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 msgr2=0x5599b52465e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.644+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b52465e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x5599b51d14c0 tx=0x7f4c1c014c50 comp rx=0 tx=0).stop 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.644+0000 7f4c30ef4740 1 -- 172.21.15.62:0/60188929 shutdown_connections 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.644+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5599b51c3fe0 0x5599b51c6400 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.644+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599b51d8100 0x5599b51d8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.644+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5599b51cd1d0 0x5599b52465e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.644+0000 7f4c07fff640 0 client.27970 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.644+0000 7f4c30ef4740 1 --2- 172.21.15.62:0/60188929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5599b51cbdc0 0x5599b52460a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.644+0000 7f4c30ef4740 1 -- 172.21.15.62:0/60188929 >> 172.21.15.62:0/60188929 conn(0x5599b51b96b0 msgr2=0x5599b51d2450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.645+0000 7f4c30ef4740 1 -- 172.21.15.62:0/60188929 shutdown_connections 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.645+0000 7f4c30ef4740 1 -- 172.21.15.62:0/60188929 wait complete. 2024-09-15T23:30:31.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.645+0000 7f4c30ef4740 20 client.27970 trim_cache size 0 max 16384 2024-09-15T23:30:31.649 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:30:31.649 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:30:31.649 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:30:31.649 INFO:teuthology.orchestra.run.smithi062.stdout: "0297da35-b0b2-4cbc-a5be-aaeb09781952": { 2024-09-15T23:30:31.649 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:30:31.649 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "0297da35-b0b2-4cbc-a5be-aaeb09781952", 2024-09-15T23:30:31.650 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:30:31.650 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:30:31.650 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:30:31.650 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:30:31.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.647+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c180a31a0 msgr2=0x7f4c180a5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:31.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.647+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c180a31a0 0x7f4c180a5660 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4c200bb240 tx=0x7f4c200f2740 comp rx=0 tx=0).stop 2024-09-15T23:30:31.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.647+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c2810e970 msgr2=0x7f4c2819e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:31.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.648+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c2810e970 0x7f4c2819e700 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4c20035040 tx=0x7f4c2000a990 comp rx=0 tx=0).stop 2024-09-15T23:30:31.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.648+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 shutdown_connections 2024-09-15T23:30:31.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.648+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f4c180a31a0 0x7f4c180a5660 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.648+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f4c2810e970 0x7f4c2819e700 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.648+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4c28109e00 0x7f4c281a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.648+0000 7f4c2d8b9640 1 --2- 172.21.15.62:0/4098304255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c28109450 0x7f4c281a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:31.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.648+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 >> 172.21.15.62:0/4098304255 conn(0x7f4c28073180 msgr2=0x7f4c2810c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:31.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.650+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 shutdown_connections 2024-09-15T23:30:31.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:31.650+0000 7f4c2d8b9640 1 -- 172.21.15.62:0/4098304255 wait complete. 2024-09-15T23:30:31.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:31 smithi062 ceph-mon[30949]: pgmap v797: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.1 MiB/s wr, 551 op/s 2024-09-15T23:30:32.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:31 smithi067 ceph-mon[35665]: pgmap v797: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.1 MiB/s wr, 551 op/s 2024-09-15T23:30:32.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:31 smithi148 ceph-mon[35610]: pgmap v797: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 5.1 MiB/s wr, 551 op/s 2024-09-15T23:30:32.458 INFO:tasks.cephfs.filesystem:scrub status for tag:0297da35-b0b2-4cbc-a5be-aaeb09781952 - {'path': '/', 'tag': '0297da35-b0b2-4cbc-a5be-aaeb09781952', 'options': 'recursive,force'} 2024-09-15T23:30:34.212 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:33 smithi148 ceph-mon[35610]: pgmap v798: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 4.9 MiB/s wr, 554 op/s 2024-09-15T23:30:34.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:33 smithi062 ceph-mon[30949]: pgmap v798: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 4.9 MiB/s wr, 554 op/s 2024-09-15T23:30:34.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:33 smithi067 ceph-mon[35665]: pgmap v798: 129 pgs: 129 active+clean; 1.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 4.9 MiB/s wr, 554 op/s 2024-09-15T23:30:34.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 855.0 s, 557.0 tps, lat 187.468 ms stddev 248.215 2024-09-15T23:30:36.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:35 smithi062 ceph-mon[30949]: pgmap v799: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 4.7 MiB/s wr, 575 op/s 2024-09-15T23:30:36.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:35 smithi067 ceph-mon[35665]: pgmap v799: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 4.7 MiB/s wr, 575 op/s 2024-09-15T23:30:36.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:35 smithi148 ceph-mon[35610]: pgmap v799: 129 pgs: 129 active+clean; 1.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 4.7 MiB/s wr, 575 op/s 2024-09-15T23:30:38.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:37 smithi062 ceph-mon[30949]: pgmap v800: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 4.9 MiB/s wr, 588 op/s 2024-09-15T23:30:38.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:37 smithi067 ceph-mon[35665]: pgmap v800: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 4.9 MiB/s wr, 588 op/s 2024-09-15T23:30:38.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:37 smithi148 ceph-mon[35610]: pgmap v800: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 4.9 MiB/s wr, 588 op/s 2024-09-15T23:30:39.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 860.0 s, 614.0 tps, lat 164.326 ms stddev 194.211 2024-09-15T23:30:40.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:39 smithi062 ceph-mon[30949]: pgmap v801: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.6 MiB/s wr, 573 op/s 2024-09-15T23:30:40.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:39 smithi067 ceph-mon[35665]: pgmap v801: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.6 MiB/s wr, 573 op/s 2024-09-15T23:30:40.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:39 smithi148 ceph-mon[35610]: pgmap v801: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 4.6 MiB/s wr, 573 op/s 2024-09-15T23:30:42.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:41 smithi062 ceph-mon[30949]: pgmap v802: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 5.2 MiB/s wr, 621 op/s 2024-09-15T23:30:42.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:41 smithi067 ceph-mon[35665]: pgmap v802: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 5.2 MiB/s wr, 621 op/s 2024-09-15T23:30:42.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:41 smithi148 ceph-mon[35610]: pgmap v802: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 5.2 MiB/s wr, 621 op/s 2024-09-15T23:30:42.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.621+0000 7f55c455f640 1 -- 172.21.15.62:0/2097641811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc100a50 msgr2=0x7f55bc100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.623 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.621+0000 7f55c455f640 1 --2- 172.21.15.62:0/2097641811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc100a50 0x7f55bc100e50 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f55b800fb10 tx=0x7f55b802e590 comp rx=0 tx=0).stop 2024-09-15T23:30:42.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.622+0000 7f55c455f640 1 -- 172.21.15.62:0/2097641811 shutdown_connections 2024-09-15T23:30:42.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.622+0000 7f55c455f640 1 --2- 172.21.15.62:0/2097641811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55bc102600 0x7f55bc108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.622+0000 7f55c455f640 1 --2- 172.21.15.62:0/2097641811 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55bc101c40 0x7f55bc1020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.622+0000 7f55c455f640 1 --2- 172.21.15.62:0/2097641811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc100a50 0x7f55bc100e50 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.622+0000 7f55c455f640 1 -- 172.21.15.62:0/2097641811 >> 172.21.15.62:0/2097641811 conn(0x7f55bc0fc1b0 msgr2=0x7f55bc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.622+0000 7f55c455f640 1 -- 172.21.15.62:0/2097641811 shutdown_connections 2024-09-15T23:30:42.624 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.622+0000 7f55c455f640 1 -- 172.21.15.62:0/2097641811 wait complete. 2024-09-15T23:30:42.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.623+0000 7f55c455f640 1 Processor -- start 2024-09-15T23:30:42.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.624+0000 7f55c455f640 1 -- start start 2024-09-15T23:30:42.625 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.624+0000 7f55c455f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc100a50 0x7f55bc1959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.624+0000 7f55c455f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55bc101c40 0x7f55bc195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.624+0000 7f55c455f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55bc102600 0x7f55bc19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c1afa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55bc101c40 0x7f55bc195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c1afa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55bc101c40 0x7f55bc195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38830/0 (socket says 172.21.15.62:38830) 2024-09-15T23:30:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc10b610 con 0x7f55bc100a50 2024-09-15T23:30:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc10b490 con 0x7f55bc102600 2024-09-15T23:30:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc10b790 con 0x7f55bc101c40 2024-09-15T23:30:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c22fb640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc100a50 0x7f55bc1959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c2afc640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55bc102600 0x7f55bc19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c1afa640 1 -- 172.21.15.62:0/2397339435 learned_addr learned my addr 172.21.15.62:0/2397339435 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c2afc640 1 -- 172.21.15.62:0/2397339435 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55bc101c40 msgr2=0x7f55bc195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c2afc640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55bc101c40 0x7f55bc195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c2afc640 1 -- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc100a50 msgr2=0x7f55bc1959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c2afc640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc100a50 0x7f55bc1959f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.627 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.625+0000 7f55c2afc640 1 -- 172.21.15.62:0/2397339435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc19d610 con 0x7f55bc102600 2024-09-15T23:30:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.626+0000 7f55c2afc640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55bc102600 0x7f55bc19cfa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f55b400ec50 tx=0x7f55b400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.627+0000 7f55b37fe640 1 -- 172.21.15.62:0/2397339435 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4003b70 con 0x7f55bc102600 2024-09-15T23:30:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.627+0000 7f55b37fe640 1 -- 172.21.15.62:0/2397339435 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b4003d10 con 0x7f55bc102600 2024-09-15T23:30:42.628 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.627+0000 7f55b37fe640 1 -- 172.21.15.62:0/2397339435 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4010970 con 0x7f55bc102600 2024-09-15T23:30:42.629 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.629+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc19d900 con 0x7f55bc102600 2024-09-15T23:30:42.630 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.629+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc19dd70 con 0x7f55bc102600 2024-09-15T23:30:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.633+0000 7f55c73d4740 1 Processor -- start 2024-09-15T23:30:42.635 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.634+0000 7f55b37fe640 1 -- 172.21.15.62:0/2397339435 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f55b4003ec0 con 0x7f55bc102600 2024-09-15T23:30:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.634+0000 7f55c73d4740 1 -- start start 2024-09-15T23:30:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.635+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 0x563e6e6beda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.635+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 0x563e6e6c3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.635+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6c6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.635+0000 7f55c73d4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563e6e6222f0 con 0x563e6e6c4370 2024-09-15T23:30:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.635+0000 7f55c73d4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563e6e5ca270 con 0x563e6e6bf2e0 2024-09-15T23:30:42.636 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.635+0000 7f55c73d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563e6e61e2b0 con 0x563e6e6be9a0 2024-09-15T23:30:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.636+0000 7f55c22fb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 0x563e6e6c3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.636+0000 7f55c22fb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 0x563e6e6c3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49066/0 (socket says 172.21.15.62:49066) 2024-09-15T23:30:42.637 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.636+0000 7f55c22fb640 1 -- 172.21.15.62:0/652981896 learned_addr learned my addr 172.21.15.62:0/652981896 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.636+0000 7f55b37fe640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55a80a3370 0x7f55a80a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.636+0000 7f55b37fe640 1 -- 172.21.15.62:0/2397339435 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f55b40d0180 con 0x7f55bc102600 2024-09-15T23:30:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.637+0000 7f55c22fb640 1 -- 172.21.15.62:0/652981896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 msgr2=0x563e6e6beda0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.637+0000 7f55c22fb640 1 --2- 172.21.15.62:0/652981896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 0x563e6e6beda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.637+0000 7f55c22fb640 1 -- 172.21.15.62:0/652981896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 msgr2=0x563e6e6c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.637+0000 7f55c22fb640 1 --2- 172.21.15.62:0/652981896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.637+0000 7f55c22fb640 1 -- 172.21.15.62:0/652981896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e6e5c9340 con 0x563e6e6bf2e0 2024-09-15T23:30:42.638 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.637+0000 7f55c22fb640 1 --2- 172.21.15.62:0/652981896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 0x563e6e6c3da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f55bc101aa0 tx=0x7f55b8005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1288eb9d6c5e736 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.638+0000 7f55c2afc640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55a80a3370 0x7f55a80a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.638+0000 7f55b0bfe640 1 -- 172.21.15.62:0/652981896 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b800ee10 con 0x563e6e6bf2e0 2024-09-15T23:30:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.638+0000 7f55b0bfe640 1 -- 172.21.15.62:0/652981896 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b800e660 con 0x563e6e6bf2e0 2024-09-15T23:30:42.639 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.638+0000 7f55b0bfe640 1 -- 172.21.15.62:0/652981896 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8004310 con 0x563e6e6bf2e0 2024-09-15T23:30:42.640 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.639+0000 7f55c73d4740 1 -- 172.21.15.62:0/652981896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 msgr2=0x563e6e6c3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.639+0000 7f55c73d4740 1 --2- 172.21.15.62:0/652981896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 0x563e6e6c3da0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f55bc101aa0 tx=0x7f55b8005c00 comp rx=0 tx=0).stop 2024-09-15T23:30:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.639+0000 7f55c2afc640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55a80a3370 0x7f55a80a5830 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f55b400f010 tx=0x7f55b40047d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.639+0000 7f55c73d4740 1 -- 172.21.15.62:0/652981896 shutdown_connections 2024-09-15T23:30:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.639+0000 7f55c73d4740 1 --2- 172.21.15.62:0/652981896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.639+0000 7f55c73d4740 1 --2- 172.21.15.62:0/652981896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 0x563e6e6c3da0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.639+0000 7f55c73d4740 1 --2- 172.21.15.62:0/652981896 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 0x563e6e6beda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.641 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.639+0000 7f55c73d4740 1 -- 172.21.15.62:0/652981896 >> 172.21.15.62:0/652981896 conn(0x563e6e6ba010 msgr2=0x563e6e6ba410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.640+0000 7f55c73d4740 1 -- 172.21.15.62:0/652981896 shutdown_connections 2024-09-15T23:30:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.640+0000 7f55c73d4740 1 -- 172.21.15.62:0/652981896 wait complete. 2024-09-15T23:30:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.640+0000 7f55c73d4740 1 -- start start 2024-09-15T23:30:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.640+0000 7f55c73d4740 1 Processor -- start 2024-09-15T23:30:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6be9a0 0x563e6e6d8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6d92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.642 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6e02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c73d4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x563e6e6d82f0 con 0x563e6e6c4370 2024-09-15T23:30:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c73d4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x563e6e5ab290 con 0x563e6e6be9a0 2024-09-15T23:30:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c73d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x563e6e6d8470 con 0x563e6e6bf2e0 2024-09-15T23:30:42.643 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c73d4740 10 client.0 _ll_register_callbacks cb 0x563e6e5dc140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:42.644 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c22fb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6d92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c22fb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6d92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38840/0 (socket says 172.21.15.62:38840) 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c22fb640 1 -- 172.21.15.62:0/3013763606 learned_addr learned my addr 172.21.15.62:0/3013763606 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.641+0000 7f55c1afa640 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6be9a0 0x563e6e6d8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.642+0000 7f55c1afa640 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6e02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.642+0000 7f55c22fb640 1 -- 172.21.15.62:0/3013763606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6be9a0 msgr2=0x563e6e6d8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.642+0000 7f55c22fb640 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6be9a0 0x563e6e6d8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.642+0000 7f55c22fb640 1 -- 172.21.15.62:0/3013763606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 msgr2=0x563e6e6e02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.642+0000 7f55c22fb640 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6e02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.642+0000 7f55c22fb640 1 -- 172.21.15.62:0/3013763606 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e6e335210 con 0x563e6e6bf2e0 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.643+0000 7f55c22fb640 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6d92b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f55b802e5d0 tx=0x7f55b800e970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f05192c935dbcab server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.645 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.643+0000 7f55a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.645+0000 7f55a5ffb640 1 -- 172.21.15.62:0/3013763606 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b800ec10 con 0x563e6e6bf2e0 2024-09-15T23:30:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.645+0000 7f55c73d4740 1 -- 172.21.15.62:0/3013763606 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b800e660 con 0x563e6e6bf2e0 2024-09-15T23:30:42.649 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.645+0000 7f55a5ffb640 1 -- 172.21.15.62:0/3013763606 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f55b800edb0 con 0x563e6e6bf2e0 2024-09-15T23:30:42.651 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.648+0000 7f55a5ffb640 1 -- 172.21.15.62:0/3013763606 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f55b8059050 con 0x563e6e6bf2e0 2024-09-15T23:30:42.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.652+0000 7f55c73d4740 10 client.27979 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:42.655 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.652+0000 7f55c73d4740 1 -- 172.21.15.62:0/3013763606 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563e6e401620 con 0x563e6e6bf2e0 2024-09-15T23:30:42.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.656+0000 7f55a5ffb640 1 -- 172.21.15.62:0/3013763606 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b800e660 con 0x563e6e6bf2e0 2024-09-15T23:30:42.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.656+0000 7f55a5ffb640 1 -- 172.21.15.62:0/3013763606 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f55b8003800 con 0x563e6e6bf2e0 2024-09-15T23:30:42.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.656+0000 7f55c73d4740 10 client.27979 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:42.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.656+0000 7f55c73d4740 10 client.27979 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:42.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.656+0000 7f55c73d4740 20 client.27979 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:42.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.656+0000 7f55c73d4740 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563e6e6e4e90 0x563e6e6e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.656+0000 7f55c73d4740 4 client.27979 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:42.657 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.656+0000 7f55c73d4740 1 -- 172.21.15.62:0/3013763606 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e6e62fa20 con 0x563e6e6e4e90 2024-09-15T23:30:42.659 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.658+0000 7f55c1afa640 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563e6e6e4e90 0x563e6e6e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.669 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.668+0000 7f55c1afa640 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563e6e6e4e90 0x563e6e6e72b0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30a52e7516860204 server_cookie=6d7e442e4364f7da in_seq=0 out_seq=0 2024-09-15T23:30:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.669+0000 7f55a5ffb640 10 client.27979 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.669+0000 7f55a5ffb640 1 -- 172.21.15.62:0/3013763606 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563e6e62fa20 con 0x563e6e6e4e90 2024-09-15T23:30:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.669+0000 7f55a5ffb640 10 client.27979 handle_command_reply: tid=0 2024-09-15T23:30:42.670 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.669+0000 7f55c73d4740 1 client.27979 shutdown 2024-09-15T23:30:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.675+0000 7f55c73d4740 1 -- 172.21.15.62:0/3013763606 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 msgr2=0x563e6e6d92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.676 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.675+0000 7f55c73d4740 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6d92b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f55b802e5d0 tx=0x7f55b800e970 comp rx=0 tx=0).stop 2024-09-15T23:30:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.677+0000 7f55c73d4740 1 -- 172.21.15.62:0/3013763606 shutdown_connections 2024-09-15T23:30:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.677+0000 7f55c73d4740 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563e6e6e4e90 0x563e6e6e72b0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.677+0000 7f55c73d4740 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6e02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.677+0000 7f55c73d4740 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6d92b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.677+0000 7f55c73d4740 1 --2- 172.21.15.62:0/3013763606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6be9a0 0x563e6e6d8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.678 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.677+0000 7f55c73d4740 1 -- 172.21.15.62:0/3013763606 >> 172.21.15.62:0/3013763606 conn(0x563e6e6ba010 msgr2=0x563e6e6bb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.680 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.678+0000 7f55a5ffb640 0 client.27979 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:42.682 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.681+0000 7f55c73d4740 1 -- 172.21.15.62:0/3013763606 shutdown_connections 2024-09-15T23:30:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.682+0000 7f55c73d4740 1 -- 172.21.15.62:0/3013763606 wait complete. 2024-09-15T23:30:42.683 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.682+0000 7f55c73d4740 20 client.27979 trim_cache size 0 max 16384 2024-09-15T23:30:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.723+0000 7f55c73d4740 1 Processor -- start 2024-09-15T23:30:42.724 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.723+0000 7f55c73d4740 1 -- start start 2024-09-15T23:30:42.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.724+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6be9a0 0x563e6e6f2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.724+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6e02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.725 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.724+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6c4370 0x563e6e6b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.724+0000 7f55c73d4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc1a25d0 con 0x563e6e6be9a0 2024-09-15T23:30:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.724+0000 7f55c73d4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc19fe70 con 0x563e6e6c4370 2024-09-15T23:30:42.726 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.724+0000 7f55c73d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc0740a0 con 0x563e6e6bf2e0 2024-09-15T23:30:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.727+0000 7f55c22fb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6e02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.727+0000 7f55c22fb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6e02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38850/0 (socket says 172.21.15.62:38850) 2024-09-15T23:30:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.727+0000 7f55c22fb640 1 -- 172.21.15.62:0/423516730 learned_addr learned my addr 172.21.15.62:0/423516730 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:42.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.729+0000 7f55c22fb640 1 -- 172.21.15.62:0/423516730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6c4370 msgr2=0x563e6e6b90b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:42.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.729+0000 7f55c22fb640 1 --2- 172.21.15.62:0/423516730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6c4370 0x563e6e6b90b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.729+0000 7f55c22fb640 1 -- 172.21.15.62:0/423516730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6be9a0 msgr2=0x563e6e6f2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.729+0000 7f55c22fb640 1 --2- 172.21.15.62:0/423516730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6be9a0 0x563e6e6f2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.730 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.729+0000 7f55c22fb640 1 -- 172.21.15.62:0/423516730 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e6e6d8870 con 0x563e6e6bf2e0 2024-09-15T23:30:42.731 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.730+0000 7f55c22fb640 1 --2- 172.21.15.62:0/423516730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6e02b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009230 tx=0x7f55b800e770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db639c6a63d927d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.733 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.733+0000 7f55a7fff640 1 -- 172.21.15.62:0/423516730 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8063070 con 0x563e6e6bf2e0 2024-09-15T23:30:42.734 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.733+0000 7f55a7fff640 1 -- 172.21.15.62:0/423516730 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b800b070 con 0x563e6e6bf2e0 2024-09-15T23:30:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.734+0000 7f55c73d4740 1 -- 172.21.15.62:0/423516730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 msgr2=0x563e6e6e02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.734+0000 7f55c73d4740 1 --2- 172.21.15.62:0/423516730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6e02b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009230 tx=0x7f55b800e770 comp rx=0 tx=0).stop 2024-09-15T23:30:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.736+0000 7f55c73d4740 1 -- 172.21.15.62:0/423516730 shutdown_connections 2024-09-15T23:30:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.736+0000 7f55c73d4740 1 --2- 172.21.15.62:0/423516730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6c4370 0x563e6e6b90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.736+0000 7f55c73d4740 1 --2- 172.21.15.62:0/423516730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6bf2e0 0x563e6e6e02b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.736+0000 7f55c73d4740 1 --2- 172.21.15.62:0/423516730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6be9a0 0x563e6e6f2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.738 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.736+0000 7f55c73d4740 1 -- 172.21.15.62:0/423516730 >> 172.21.15.62:0/423516730 conn(0x563e6e6ba010 msgr2=0x563e6e6b89a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.741 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.740+0000 7f55c73d4740 1 -- 172.21.15.62:0/423516730 shutdown_connections 2024-09-15T23:30:42.742 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.742+0000 7f55c73d4740 1 -- 172.21.15.62:0/423516730 wait complete. 2024-09-15T23:30:42.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.742+0000 7f55c73d4740 1 -- start start 2024-09-15T23:30:42.743 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.742+0000 7f55c73d4740 1 Processor -- start 2024-09-15T23:30:42.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.743+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 0x563e6e6ddf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.743+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 0x563e6e6de470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.744 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.743+0000 7f55c73d4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6df480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.743+0000 7f55c73d4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc19e330 con 0x563e6e6c4370 2024-09-15T23:30:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.743+0000 7f55c73d4740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc19f7a0 con 0x563e6e6bf2e0 2024-09-15T23:30:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.743+0000 7f55c73d4740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc19ea00 con 0x563e6e6be9a0 2024-09-15T23:30:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.743+0000 7f55c73d4740 10 client.0 _ll_register_callbacks cb 0x7f55b800f990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.744+0000 7f55c1afa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6df480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.744+0000 7f55c1afa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6df480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41154/0 (socket says 172.21.15.62:41154) 2024-09-15T23:30:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.744+0000 7f55c1afa640 1 -- 172.21.15.62:0/1440067476 learned_addr learned my addr 172.21.15.62:0/1440067476 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:42.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.744+0000 7f55c1afa640 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 0x563e6e6ddf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.744+0000 7f55c1afa640 1 -- 172.21.15.62:0/1440067476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 msgr2=0x563e6e6de470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.744+0000 7f55c1afa640 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 0x563e6e6de470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.744+0000 7f55c1afa640 1 -- 172.21.15.62:0/1440067476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 msgr2=0x563e6e6df480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.744+0000 7f55c1afa640 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6df480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.744+0000 7f55c1afa640 1 -- 172.21.15.62:0/1440067476 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e6e6dfa90 con 0x563e6e6be9a0 2024-09-15T23:30:42.746 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.745+0000 7f55c1afa640 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 0x563e6e6ddf30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f55ac000c00 tx=0x7f55ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c75783a567b0f41 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:42.750 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.749+0000 7f55a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:42.750 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.749+0000 7f55a5ffb640 1 -- 172.21.15.62:0/1440067476 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac012790 con 0x563e6e6be9a0 2024-09-15T23:30:42.750 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.749+0000 7f55a5ffb640 1 -- 172.21.15.62:0/1440067476 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f55ac012930 con 0x563e6e6be9a0 2024-09-15T23:30:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.750+0000 7f55c73d4740 1 -- 172.21.15.62:0/1440067476 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b800b070 con 0x563e6e6be9a0 2024-09-15T23:30:42.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55a5ffb640 1 -- 172.21.15.62:0/1440067476 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f55ac012b30 con 0x563e6e6be9a0 2024-09-15T23:30:42.753 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55c73d4740 10 client.27985 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55c73d4740 1 -- 172.21.15.62:0/1440067476 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563e6e7722a0 con 0x563e6e6be9a0 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55a5ffb640 1 -- 172.21.15.62:0/1440067476 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b800b070 con 0x563e6e6be9a0 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55a5ffb640 1 -- 172.21.15.62:0/1440067476 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f55ac00f040 con 0x563e6e6be9a0 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55c73d4740 10 client.27985 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55c73d4740 10 client.27985 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55c73d4740 20 client.27985 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55c73d4740 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563e6e772be0 0x563e6e775000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55c73d4740 4 client.27985 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.752+0000 7f55c73d4740 1 -- 172.21.15.62:0/1440067476 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e6e629d70 con 0x563e6e772be0 2024-09-15T23:30:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.753+0000 7f55c22fb640 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563e6e772be0 0x563e6e775000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.758+0000 7f55c22fb640 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563e6e772be0 0x563e6e775000 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1a2c6dd04d44c77 server_cookie=273fb4a0f1d94519 in_seq=0 out_seq=0 2024-09-15T23:30:42.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.758+0000 7f55a5ffb640 10 client.27985 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.774+0000 7f55a5ffb640 1 -- 172.21.15.62:0/1440067476 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e6e629d70 con 0x563e6e772be0 2024-09-15T23:30:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.774+0000 7f55a5ffb640 10 client.27985 handle_command_reply: tid=0 2024-09-15T23:30:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.774+0000 7f55c73d4740 1 client.27985 shutdown 2024-09-15T23:30:42.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 -- 172.21.15.62:0/1440067476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 msgr2=0x563e6e6ddf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 0x563e6e6ddf30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f55ac000c00 tx=0x7f55ac0046c0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 -- 172.21.15.62:0/1440067476 shutdown_connections 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x563e6e772be0 0x563e6e775000 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563e6e6c4370 0x563e6e6df480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e6e6bf2e0 0x563e6e6de470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 --2- 172.21.15.62:0/1440067476 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x563e6e6be9a0 0x563e6e6ddf30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 -- 172.21.15.62:0/1440067476 >> 172.21.15.62:0/1440067476 conn(0x563e6e6ba010 msgr2=0x563e6e6b91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55a5ffb640 0 client.27985 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 -- 172.21.15.62:0/1440067476 shutdown_connections 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 1 -- 172.21.15.62:0/1440067476 wait complete. 2024-09-15T23:30:42.777 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.775+0000 7f55c73d4740 20 client.27985 trim_cache size 0 max 16384 2024-09-15T23:30:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:30:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:30:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:30:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:30:42.778 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55a80a3370 msgr2=0x7f55a80a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55a80a3370 0x7f55a80a5830 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f55b400f010 tx=0x7f55b40047d0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55bc102600 msgr2=0x7f55bc19cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55bc102600 0x7f55bc19cfa0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f55b400ec50 tx=0x7f55b400c6a0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 shutdown_connections 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f55a80a3370 0x7f55a80a5830 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55bc102600 0x7f55bc19cfa0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f55bc101c40 0x7f55bc195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 --2- 172.21.15.62:0/2397339435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55bc100a50 0x7f55bc1959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 >> 172.21.15.62:0/2397339435 conn(0x7f55bc0fc1b0 msgr2=0x7f55bc104e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 shutdown_connections 2024-09-15T23:30:42.779 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:42.776+0000 7f55c455f640 1 -- 172.21.15.62:0/2397339435 wait complete. 2024-09-15T23:30:43.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:42 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:30:43.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:42 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:30:43.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:42 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:30:43.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:42 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:30:43.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:42 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:30:43.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:42 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:30:44.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:43 smithi062 ceph-mon[30949]: pgmap v803: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 4.9 MiB/s wr, 577 op/s 2024-09-15T23:30:44.231 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:43 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:30:44.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:43 smithi067 ceph-mon[35665]: pgmap v803: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 4.9 MiB/s wr, 577 op/s 2024-09-15T23:30:44.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:43 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:30:44.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:43 smithi148 ceph-mon[35610]: pgmap v803: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 4.9 MiB/s wr, 577 op/s 2024-09-15T23:30:44.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:43 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:30:44.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 865.0 s, 603.2 tps, lat 165.121 ms stddev 199.486 2024-09-15T23:30:45.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:44 smithi062 ceph-mon[30949]: pgmap v804: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 4.9 MiB/s wr, 541 op/s 2024-09-15T23:30:45.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:44 smithi067 ceph-mon[35665]: pgmap v804: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 4.9 MiB/s wr, 541 op/s 2024-09-15T23:30:45.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:44 smithi148 ceph-mon[35610]: pgmap v804: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 4.9 MiB/s wr, 541 op/s 2024-09-15T23:30:45.940 INFO:tasks.quiescer.fs.[cephfs]:Total duration (35.4) is longer than `split_if_longer` (35.0), will split into two consecutive quiesces 2024-09-15T23:30:45.940 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.7 2024-09-15T23:30:45.940 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:30:46.502 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.087+0000 7fd3f9fe2640 1 -- 172.21.15.62:0/3731562606 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 msgr2=0x7fd3f4109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.087+0000 7fd3f9fe2640 1 --2- 172.21.15.62:0/3731562606 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 0x7fd3f4109c90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009860 tx=0x7fd3dc02a360 comp rx=0 tx=0).stop 2024-09-15T23:30:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.089+0000 7fd3f9fe2640 1 -- 172.21.15.62:0/3731562606 shutdown_connections 2024-09-15T23:30:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.089+0000 7fd3f9fe2640 1 --2- 172.21.15.62:0/3731562606 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f411b140 0x7fd3f411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.089+0000 7fd3f9fe2640 1 --2- 172.21.15.62:0/3731562606 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 0x7fd3f4109c90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.089+0000 7fd3f9fe2640 1 --2- 172.21.15.62:0/3731562606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f4109450 0x7fd3f4110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.089+0000 7fd3f9fe2640 1 -- 172.21.15.62:0/3731562606 >> 172.21.15.62:0/3731562606 conn(0x7fd3f406e140 msgr2=0x7fd3f406e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.091+0000 7fd3f9fe2640 1 -- 172.21.15.62:0/3731562606 shutdown_connections 2024-09-15T23:30:47.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.092+0000 7fd3f9fe2640 1 -- 172.21.15.62:0/3731562606 wait complete. 2024-09-15T23:30:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.093+0000 7fd3f9fe2640 1 Processor -- start 2024-09-15T23:30:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.093+0000 7fd3f9fe2640 1 -- start start 2024-09-15T23:30:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.094+0000 7fd3f9fe2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f4109450 0x7fd3f41a72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.094+0000 7fd3f9fe2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 0x7fd3f41a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.094+0000 7fd3f9fe2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f411b140 0x7fd3f41abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.094+0000 7fd3f9fe2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f4120df0 con 0x7fd3f411b140 2024-09-15T23:30:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.094+0000 7fd3f9fe2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f4120c70 con 0x7fd3f4109450 2024-09-15T23:30:47.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.094+0000 7fd3f9fe2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f4120f70 con 0x7fd3f4109830 2024-09-15T23:30:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.096+0000 7fd3f2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 0x7fd3f41a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.096+0000 7fd3f2ffd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 0x7fd3f41a7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:38880/0 (socket says 172.21.15.62:38880) 2024-09-15T23:30:47.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.096+0000 7fd3f2ffd640 1 -- 172.21.15.62:0/327974983 learned_addr learned my addr 172.21.15.62:0/327974983 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.099+0000 7fd3f2ffd640 1 -- 172.21.15.62:0/327974983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f4109450 msgr2=0x7fd3f41a72f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-15T23:30:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.099+0000 7fd3f2ffd640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f4109450 0x7fd3f41a72f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.099+0000 7fd3f2ffd640 1 -- 172.21.15.62:0/327974983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f411b140 msgr2=0x7fd3f41abbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.099+0000 7fd3f2ffd640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f411b140 0x7fd3f41abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.099+0000 7fd3f2ffd640 1 -- 172.21.15.62:0/327974983 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f41ac2a0 con 0x7fd3f4109830 2024-09-15T23:30:47.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.101+0000 7fd3f2ffd640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 0x7fd3f41a7830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc035040 tx=0x7fd3dc030ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:47.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.101+0000 7fd3f0ff9640 1 -- 172.21.15.62:0/327974983 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc037040 con 0x7fd3f4109830 2024-09-15T23:30:47.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.102+0000 7fd3f9fe2640 1 -- 172.21.15.62:0/327974983 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3f41ac530 con 0x7fd3f4109830 2024-09-15T23:30:47.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.102+0000 7fd3f9fe2640 1 -- 172.21.15.62:0/327974983 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f41b3e70 con 0x7fd3f4109830 2024-09-15T23:30:47.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.102+0000 7fd3f0ff9640 1 -- 172.21.15.62:0/327974983 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc040070 con 0x7fd3f4109830 2024-09-15T23:30:47.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.102+0000 7fd3f0ff9640 1 -- 172.21.15.62:0/327974983 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc003850 con 0x7fd3f4109830 2024-09-15T23:30:47.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.104+0000 7fd3f9fe2640 1 -- 172.21.15.62:0/327974983 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3d4005290 con 0x7fd3f4109830 2024-09-15T23:30:47.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.107+0000 7fd3f0ff9640 1 -- 172.21.15.62:0/327974983 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd3dc004050 con 0x7fd3f4109830 2024-09-15T23:30:47.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.109+0000 7fd3f0ff9640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3e00a3450 0x7fd3e00a5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:47.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.109+0000 7fd3f0ff9640 1 -- 172.21.15.62:0/327974983 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd3dc0ef170 con 0x7fd3f4109830 2024-09-15T23:30:47.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.109+0000 7fd3f37fe640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3e00a3450 0x7fd3e00a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:47.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.111+0000 7fd3f37fe640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3e00a3450 0x7fd3e00a5910 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009a60 tx=0x7fd3ec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:47.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.112+0000 7fd3f0ff9640 1 -- 172.21.15.62:0/327974983 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd3dc049020 con 0x7fd3f4109830 2024-09-15T23:30:47.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:47 smithi062 ceph-mon[30949]: pgmap v805: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.6 MiB/s wr, 504 op/s 2024-09-15T23:30:47.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.516+0000 7fd3f9fe2640 1 -- 172.21.15.62:0/327974983 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3d4005580 con 0x7fd3f4109830 2024-09-15T23:30:47.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.521+0000 7fd3f0ff9640 1 -- 172.21.15.62:0/327974983 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7fd3dc0bbab0 con 0x7fd3f4109830 2024-09-15T23:30:47.523 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:30:47.523 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:30:47.523 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:30:47.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:47 smithi067 ceph-mon[35665]: pgmap v805: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.6 MiB/s wr, 504 op/s 2024-09-15T23:30:47.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.528+0000 7fd3ea7fc640 1 -- 172.21.15.62:0/327974983 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3e00a3450 msgr2=0x7fd3e00a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.528+0000 7fd3ea7fc640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3e00a3450 0x7fd3e00a5910 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009a60 tx=0x7fd3ec005f90 comp rx=0 tx=0).stop 2024-09-15T23:30:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.528+0000 7fd3ea7fc640 1 -- 172.21.15.62:0/327974983 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 msgr2=0x7fd3f41a7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:47.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.528+0000 7fd3ea7fc640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 0x7fd3f41a7830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc035040 tx=0x7fd3dc030ee0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.529+0000 7fd3ea7fc640 1 -- 172.21.15.62:0/327974983 shutdown_connections 2024-09-15T23:30:47.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.529+0000 7fd3ea7fc640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd3e00a3450 0x7fd3e00a5910 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.529+0000 7fd3ea7fc640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3f411b140 0x7fd3f41abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.529+0000 7fd3ea7fc640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd3f4109830 0x7fd3f41a7830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.529+0000 7fd3ea7fc640 1 --2- 172.21.15.62:0/327974983 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3f4109450 0x7fd3f41a72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:47.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.529+0000 7fd3ea7fc640 1 -- 172.21.15.62:0/327974983 >> 172.21.15.62:0/327974983 conn(0x7fd3f406e140 msgr2=0x7fd3f411c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:47.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.531+0000 7fd3ea7fc640 1 -- 172.21.15.62:0/327974983 shutdown_connections 2024-09-15T23:30:47.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:47.532+0000 7fd3ea7fc640 1 -- 172.21.15.62:0/327974983 wait complete. 2024-09-15T23:30:47.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:47 smithi148 ceph-mon[35610]: pgmap v805: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.6 MiB/s wr, 504 op/s 2024-09-15T23:30:48.236 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0' 2024-09-15T23:30:48.236 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0 2024-09-15T23:30:48.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:48 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/327974983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:48.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:48 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/327974983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:48.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:48 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/327974983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:30:48.760 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:30:49.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.365+0000 7f7916a8e640 1 -- 172.21.15.62:0/1683728589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7910110190 msgr2=0x7f791006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.365+0000 7f7916a8e640 1 --2- 172.21.15.62:0/1683728589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7910110190 0x7f791006e4f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f7900009be0 tx=0x7f790002a6f0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.365+0000 7f7916a8e640 1 -- 172.21.15.62:0/1683728589 shutdown_connections 2024-09-15T23:30:49.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.365+0000 7f7916a8e640 1 --2- 172.21.15.62:0/1683728589 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f791011b140 0x7f791011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.365+0000 7f7916a8e640 1 --2- 172.21.15.62:0/1683728589 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f791006ea30 0x7f791006ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.365+0000 7f7916a8e640 1 --2- 172.21.15.62:0/1683728589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7910110190 0x7f791006e4f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.365+0000 7f7916a8e640 1 -- 172.21.15.62:0/1683728589 >> 172.21.15.62:0/1683728589 conn(0x7f7910073180 msgr2=0x7f7910073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.366+0000 7f7916a8e640 1 -- 172.21.15.62:0/1683728589 shutdown_connections 2024-09-15T23:30:49.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.367+0000 7f7916a8e640 1 -- 172.21.15.62:0/1683728589 wait complete. 2024-09-15T23:30:49.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.368+0000 7f7916a8e640 1 Processor -- start 2024-09-15T23:30:49.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f7916a8e640 1 -- start start 2024-09-15T23:30:49.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f7916a8e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f791006ea30 0x7f7910119710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f7916a8e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7910110190 0x7f79101127f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f7916a8e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f791011b140 0x7f7910112d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f7916a8e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7910120c10 con 0x7f791006ea30 2024-09-15T23:30:49.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f7916a8e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7910120a90 con 0x7f791011b140 2024-09-15T23:30:49.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f7916a8e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7910120d90 con 0x7f7910110190 2024-09-15T23:30:49.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f791628d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f791011b140 0x7f7910112d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f791628d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f791011b140 0x7f7910112d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49128/0 (socket says 172.21.15.62:49128) 2024-09-15T23:30:49.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.369+0000 7f791628d640 1 -- 172.21.15.62:0/3535303216 learned_addr learned my addr 172.21.15.62:0/3535303216 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:49.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.371+0000 7f7915a8c640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f791006ea30 0x7f7910119710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.372+0000 7f7915a8c640 1 -- 172.21.15.62:0/3535303216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7910110190 msgr2=0x7f79101127f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.372+0000 7f7915a8c640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7910110190 0x7f79101127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.372+0000 7f7915a8c640 1 -- 172.21.15.62:0/3535303216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f791011b140 msgr2=0x7f7910112d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.372+0000 7f7915a8c640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f791011b140 0x7f7910112d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.372+0000 7f7915a8c640 1 -- 172.21.15.62:0/3535303216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79101132f0 con 0x7f791006ea30 2024-09-15T23:30:49.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.372+0000 7f7915a8c640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f791006ea30 0x7f7910119710 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f790002ac10 tx=0x7f7900032f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.374+0000 7f790effd640 1 -- 172.21.15.62:0/3535303216 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900041070 con 0x7f791006ea30 2024-09-15T23:30:49.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.374+0000 7f790effd640 1 -- 172.21.15.62:0/3535303216 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7900003e50 con 0x7f791006ea30 2024-09-15T23:30:49.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.374+0000 7f790effd640 1 -- 172.21.15.62:0/3535303216 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900036070 con 0x7f791006ea30 2024-09-15T23:30:49.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.377+0000 7f7916a8e640 1 -- 172.21.15.62:0/3535303216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7910113580 con 0x7f791006ea30 2024-09-15T23:30:49.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.377+0000 7f7916a8e640 1 -- 172.21.15.62:0/3535303216 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7910117470 con 0x7f791006ea30 2024-09-15T23:30:49.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.382+0000 7f790effd640 1 -- 172.21.15.62:0/3535303216 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7900044070 con 0x7f791006ea30 2024-09-15T23:30:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.384+0000 7f790effd640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78f80a3410 0x7f78f80a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.385+0000 7f790effd640 1 -- 172.21.15.62:0/3535303216 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f79000efd20 con 0x7f791006ea30 2024-09-15T23:30:49.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.385+0000 7f791528b640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78f80a3410 0x7f78f80a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.386+0000 7f791a0c9740 1 Processor -- start 2024-09-15T23:30:49.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.389+0000 7f791528b640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78f80a3410 0x7f78f80a58d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7910113f60 tx=0x7f7904006c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.394+0000 7f791a0c9740 1 -- start start 2024-09-15T23:30:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.394+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6788900 0x5618b6788ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.394+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6789220 0x5618b67935a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.394+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b6795ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.395+0000 7f791a0c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5618b66ee2a0 con 0x5618b6793ae0 2024-09-15T23:30:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.395+0000 7f791a0c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5618b6695b10 con 0x5618b6789220 2024-09-15T23:30:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.395+0000 7f791a0c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5618b66ea260 con 0x5618b6788900 2024-09-15T23:30:49.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.396+0000 7f791528b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6789220 0x5618b67935a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.396+0000 7f791528b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6789220 0x5618b67935a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49142/0 (socket says 172.21.15.62:49142) 2024-09-15T23:30:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.396+0000 7f791528b640 1 -- 172.21.15.62:0/2384219027 learned_addr learned my addr 172.21.15.62:0/2384219027 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:49.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.396+0000 7f7915a8c640 1 --2- 172.21.15.62:0/2384219027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6788900 0x5618b6788ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.397+0000 7f791528b640 1 -- 172.21.15.62:0/2384219027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6788900 msgr2=0x5618b6788ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.397+0000 7f791528b640 1 --2- 172.21.15.62:0/2384219027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6788900 0x5618b6788ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.397+0000 7f791528b640 1 -- 172.21.15.62:0/2384219027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 msgr2=0x5618b6795ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.397+0000 7f791528b640 1 --2- 172.21.15.62:0/2384219027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b6795ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.397+0000 7f791528b640 1 -- 172.21.15.62:0/2384219027 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618b645fb90 con 0x5618b6789220 2024-09-15T23:30:49.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.398+0000 7f791528b640 1 --2- 172.21.15.62:0/2384219027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6789220 0x5618b67935a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7904015890 tx=0x7f790400e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67775cc27cfa0a19 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.401+0000 7f78f7fff640 1 -- 172.21.15.62:0/2384219027 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904023070 con 0x5618b6789220 2024-09-15T23:30:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.401+0000 7f78f7fff640 1 -- 172.21.15.62:0/2384219027 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7904015e10 con 0x5618b6789220 2024-09-15T23:30:49.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.401+0000 7f78f7fff640 1 -- 172.21.15.62:0/2384219027 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790401ec90 con 0x5618b6789220 2024-09-15T23:30:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.404+0000 7f791a0c9740 1 -- 172.21.15.62:0/2384219027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6789220 msgr2=0x5618b67935a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.404+0000 7f791a0c9740 1 --2- 172.21.15.62:0/2384219027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6789220 0x5618b67935a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7904015890 tx=0x7f790400e000 comp rx=0 tx=0).stop 2024-09-15T23:30:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.406+0000 7f791a0c9740 1 -- 172.21.15.62:0/2384219027 shutdown_connections 2024-09-15T23:30:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.406+0000 7f791a0c9740 1 --2- 172.21.15.62:0/2384219027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b6795ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.406+0000 7f791a0c9740 1 --2- 172.21.15.62:0/2384219027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6789220 0x5618b67935a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.406+0000 7f791a0c9740 1 --2- 172.21.15.62:0/2384219027 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6788900 0x5618b6788ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.406+0000 7f791a0c9740 1 -- 172.21.15.62:0/2384219027 >> 172.21.15.62:0/2384219027 conn(0x5618b6784e90 msgr2=0x5618b6785270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.407+0000 7f791a0c9740 1 -- 172.21.15.62:0/2384219027 shutdown_connections 2024-09-15T23:30:49.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.407+0000 7f791a0c9740 1 -- 172.21.15.62:0/2384219027 wait complete. 2024-09-15T23:30:49.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.407+0000 7f791a0c9740 1 -- start start 2024-09-15T23:30:49.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.408+0000 7f791a0c9740 1 Processor -- start 2024-09-15T23:30:49.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.408+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b6790b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.408+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b6791070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.408+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b67915b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.408+0000 7f791a0c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5618b67966b0 con 0x5618b6793ae0 2024-09-15T23:30:49.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.408+0000 7f791a0c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5618b66766e0 con 0x5618b6788900 2024-09-15T23:30:49.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.408+0000 7f791a0c9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5618b6796830 con 0x5618b6789220 2024-09-15T23:30:49.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.408+0000 7f791a0c9740 10 client.0 _ll_register_callbacks cb 0x5618b669aa80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:49.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.409+0000 7f791628d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b67915b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.410+0000 7f791628d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b67915b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41194/0 (socket says 172.21.15.62:41194) 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.410+0000 7f791628d640 1 -- 172.21.15.62:0/1173023633 learned_addr learned my addr 172.21.15.62:0/1173023633 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.410+0000 7f7915a8c640 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b6790b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.410+0000 7f791628d640 1 -- 172.21.15.62:0/1173023633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 msgr2=0x5618b6791070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.410+0000 7f791628d640 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b6791070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.410+0000 7f791628d640 1 -- 172.21.15.62:0/1173023633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 msgr2=0x5618b6790b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.410+0000 7f791628d640 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b6790b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.410+0000 7f791628d640 1 -- 172.21.15.62:0/1173023633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618b679fd20 con 0x5618b6793ae0 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.411+0000 7f791628d640 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b67915b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f790800be60 tx=0x7f79080107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ebeae63b689fe0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.411+0000 7f78f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:30:49.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.412+0000 7f78f57fa640 1 -- 172.21.15.62:0/1173023633 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908011850 con 0x5618b6793ae0 2024-09-15T23:30:49.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.412+0000 7f791a0c9740 1 -- 172.21.15.62:0/1173023633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7904015e10 con 0x5618b6793ae0 2024-09-15T23:30:49.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.412+0000 7f78f57fa640 1 -- 172.21.15.62:0/1173023633 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f79080119f0 con 0x5618b6793ae0 2024-09-15T23:30:49.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.412+0000 7f78f57fa640 1 -- 172.21.15.62:0/1173023633 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7908025050 con 0x5618b6793ae0 2024-09-15T23:30:49.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.415+0000 7f791a0c9740 10 client.18816 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:49.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.415+0000 7f791a0c9740 1 -- 172.21.15.62:0/1173023633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5618b67a26f0 con 0x5618b6793ae0 2024-09-15T23:30:49.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.416+0000 7f78f57fa640 1 -- 172.21.15.62:0/1173023633 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7904015e10 con 0x5618b6793ae0 2024-09-15T23:30:49.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.417+0000 7f78f57fa640 1 -- 172.21.15.62:0/1173023633 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f790801a380 con 0x5618b6793ae0 2024-09-15T23:30:49.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.417+0000 7f791a0c9740 10 client.18816 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:49.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.417+0000 7f791a0c9740 10 client.18816 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:30:49.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.417+0000 7f791a0c9740 20 client.18816 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:49.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.417+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5618b67a3fa0 0x5618b67a6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.417+0000 7f791a0c9740 4 client.18816 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:49.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.417+0000 7f791a0c9740 1 -- 172.21.15.62:0/1173023633 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618b66fb9d0 con 0x5618b67a3fa0 2024-09-15T23:30:49.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.418+0000 7f7915a8c640 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5618b67a3fa0 0x5618b67a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.419+0000 7f7915a8c640 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5618b67a3fa0 0x5618b67a6420 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8508e6d402b38949 server_cookie=fd30edf939efcfd in_seq=0 out_seq=0 2024-09-15T23:30:49.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.419+0000 7f78f57fa640 10 client.18816 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:49.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.420+0000 7f78f57fa640 1 -- 172.21.15.62:0/1173023633 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5618b66fb9d0 con 0x5618b67a3fa0 2024-09-15T23:30:49.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.420+0000 7f78f57fa640 10 client.18816 handle_command_reply: tid=0 2024-09-15T23:30:49.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.420+0000 7f791a0c9740 1 client.18816 shutdown 2024-09-15T23:30:49.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.421+0000 7f791a0c9740 1 -- 172.21.15.62:0/1173023633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 msgr2=0x5618b67915b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.421+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b67915b0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f790800be60 tx=0x7f79080107a0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.423+0000 7f791a0c9740 1 -- 172.21.15.62:0/1173023633 shutdown_connections 2024-09-15T23:30:49.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.423+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5618b67a3fa0 0x5618b67a6420 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.423+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b67915b0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.423+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b6791070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.423+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1173023633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b6790b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.423+0000 7f791a0c9740 1 -- 172.21.15.62:0/1173023633 >> 172.21.15.62:0/1173023633 conn(0x5618b6784e90 msgr2=0x5618b6784000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.424+0000 7f78f57fa640 0 client.18816 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:49.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.424+0000 7f791a0c9740 1 -- 172.21.15.62:0/1173023633 shutdown_connections 2024-09-15T23:30:49.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.425+0000 7f791a0c9740 1 -- 172.21.15.62:0/1173023633 wait complete. 2024-09-15T23:30:49.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.425+0000 7f791a0c9740 20 client.18816 trim_cache size 0 max 16384 2024-09-15T23:30:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.462+0000 7f791a0c9740 1 Processor -- start 2024-09-15T23:30:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.463+0000 7f791a0c9740 1 -- start start 2024-09-15T23:30:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.463+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6788900 0x5618b67abd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.463+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b67ac270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.463+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6793ae0 0x5618b6792390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.463+0000 7f791a0c9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f79101c8b50 con 0x5618b6788900 2024-09-15T23:30:49.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.463+0000 7f791a0c9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79101c63f0 con 0x5618b6793ae0 2024-09-15T23:30:49.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.463+0000 7f791628d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6793ae0 0x5618b6792390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.463+0000 7f791628d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6793ae0 0x5618b6792390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49162/0 (socket says 172.21.15.62:49162) 2024-09-15T23:30:49.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.463+0000 7f791628d640 1 -- 172.21.15.62:0/1859970287 learned_addr learned my addr 172.21.15.62:0/1859970287 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:49.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.464+0000 7f791a0c9740 1 -- 172.21.15.62:0/1859970287 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f79101c4240 con 0x5618b6789220 2024-09-15T23:30:49.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.464+0000 7f7915a8c640 1 --2- 172.21.15.62:0/1859970287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6788900 0x5618b67abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.466 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:49 smithi062 ceph-mon[30949]: pgmap v806: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 MiB/s wr, 503 op/s 2024-09-15T23:30:49.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.464+0000 7f791628d640 1 -- 172.21.15.62:0/1859970287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 msgr2=0x5618b67ac270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.464+0000 7f791628d640 1 --2- 172.21.15.62:0/1859970287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b67ac270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.464+0000 7f791628d640 1 -- 172.21.15.62:0/1859970287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6788900 msgr2=0x5618b67abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.464+0000 7f791628d640 1 --2- 172.21.15.62:0/1859970287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6788900 0x5618b67abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.464+0000 7f791628d640 1 -- 172.21.15.62:0/1859970287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618b678f0c0 con 0x5618b6793ae0 2024-09-15T23:30:49.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.465+0000 7f791628d640 1 --2- 172.21.15.62:0/1859970287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6793ae0 0x5618b6792390 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f790801a870 tx=0x7f790800b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=343e471df865a18f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.466+0000 7f78f77fe640 1 -- 172.21.15.62:0/1859970287 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790802d070 con 0x5618b6793ae0 2024-09-15T23:30:49.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.466+0000 7f78f77fe640 1 -- 172.21.15.62:0/1859970287 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7908005510 con 0x5618b6793ae0 2024-09-15T23:30:49.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.466+0000 7f78f77fe640 1 -- 172.21.15.62:0/1859970287 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908022040 con 0x5618b6793ae0 2024-09-15T23:30:49.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.467+0000 7f791a0c9740 1 -- 172.21.15.62:0/1859970287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6793ae0 msgr2=0x5618b6792390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.467+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1859970287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6793ae0 0x5618b6792390 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f790801a870 tx=0x7f790800b9a0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.468+0000 7f791a0c9740 1 -- 172.21.15.62:0/1859970287 shutdown_connections 2024-09-15T23:30:49.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.468+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1859970287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6793ae0 0x5618b6792390 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.468+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1859970287 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b67ac270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.468+0000 7f791a0c9740 1 --2- 172.21.15.62:0/1859970287 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6788900 0x5618b67abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.469 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.468+0000 7f791a0c9740 1 -- 172.21.15.62:0/1859970287 >> 172.21.15.62:0/1859970287 conn(0x5618b6784e90 msgr2=0x5618b678be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.471+0000 7f791a0c9740 1 -- 172.21.15.62:0/1859970287 shutdown_connections 2024-09-15T23:30:49.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.471+0000 7f791a0c9740 1 -- 172.21.15.62:0/1859970287 wait complete. 2024-09-15T23:30:49.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.472+0000 7f791a0c9740 1 -- start start 2024-09-15T23:30:49.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.472+0000 7f791a0c9740 1 Processor -- start 2024-09-15T23:30:49.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.473+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b678db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.473+0000 7f791a0c9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b678e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.473+0000 7f7915a8c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b678db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.473+0000 7f7915a8c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b678db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:49178/0 (socket says 172.21.15.62:49178) 2024-09-15T23:30:49.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.473+0000 7f7915a8c640 1 -- 172.21.15.62:0/3634801120 learned_addr learned my addr 172.21.15.62:0/3634801120 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:49.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.473+0000 7f791a0c9740 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b678e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.473+0000 7f791528b640 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b678e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f79101c48b0 con 0x5618b6793ae0 2024-09-15T23:30:49.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f79101c5d20 con 0x5618b6788900 2024-09-15T23:30:49.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f79101c4f80 con 0x5618b6789220 2024-09-15T23:30:49.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f791a0c9740 10 client.0 _ll_register_callbacks cb 0x7f790800baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:49.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f791528b640 1 -- 172.21.15.62:0/3634801120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 msgr2=0x5618b678db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f791528b640 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b678db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f791528b640 1 -- 172.21.15.62:0/3634801120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 msgr2=0x5618b678e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:49.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f791528b640 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b678e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f791528b640 1 -- 172.21.15.62:0/3634801120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618b680dd00 con 0x5618b6789220 2024-09-15T23:30:49.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.474+0000 7f7915a8c640 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b678db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:30:49.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.475+0000 7f791528b640 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b678e060 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x5618b6792170 tx=0x7f790401e640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c985e29e43adbadf server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:49.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.476+0000 7f78f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:30:49.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.476+0000 7f78f57fa640 1 -- 172.21.15.62:0/3634801120 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904023040 con 0x5618b6789220 2024-09-15T23:30:49.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.476+0000 7f78f57fa640 1 -- 172.21.15.62:0/3634801120 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f790401e730 con 0x5618b6789220 2024-09-15T23:30:49.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.478+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7908005510 con 0x5618b6789220 2024-09-15T23:30:49.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.479+0000 7f78f57fa640 1 -- 172.21.15.62:0/3634801120 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7904015590 con 0x5618b6789220 2024-09-15T23:30:49.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.481+0000 7f791a0c9740 10 client.27994 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:49.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.481+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5618b680ef20 con 0x5618b6789220 2024-09-15T23:30:49.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.484+0000 7f78f57fa640 1 -- 172.21.15.62:0/3634801120 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7908005510 con 0x5618b6789220 2024-09-15T23:30:49.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.485+0000 7f78f57fa640 1 -- 172.21.15.62:0/3634801120 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f7904026070 con 0x5618b6789220 2024-09-15T23:30:49.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.485+0000 7f791a0c9740 10 client.27994 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:49.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.485+0000 7f791a0c9740 10 client.27994 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:30:49.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.485+0000 7f791a0c9740 20 client.27994 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:49.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.485+0000 7f791a0c9740 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5618b680f380 0x5618b68117a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:49.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.485+0000 7f791a0c9740 4 client.27994 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}] 2024-09-15T23:30:49.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.485+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}) -- 0x5618b66f5d20 con 0x5618b680f380 2024-09-15T23:30:49.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.485+0000 7f7915a8c640 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5618b680f380 0x5618b68117a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:49.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.489+0000 7f7915a8c640 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5618b680f380 0x5618b68117a0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51a336029dd23e8f server_cookie=e5fb1c1e79fd7a5b in_seq=0 out_seq=0 2024-09-15T23:30:49.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.489+0000 7f78f57fa640 10 client.27994 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:49.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:49 smithi067 ceph-mon[35665]: pgmap v806: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 MiB/s wr, 503 op/s 2024-09-15T23:30:49.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:49 smithi148 ceph-mon[35610]: pgmap v806: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.7 MiB/s wr, 503 op/s 2024-09-15T23:30:49.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 870.0 s, 610.4 tps, lat 166.281 ms stddev 234.873 2024-09-15T23:30:49.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.853+0000 7f78f57fa640 1 -- 172.21.15.62:0/3634801120 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x5618b66f5d20 con 0x5618b680f380 2024-09-15T23:30:49.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.854+0000 7f78f57fa640 10 client.27994 handle_command_reply: tid=0 2024-09-15T23:30:49.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.854+0000 7f791a0c9740 1 client.27994 shutdown 2024-09-15T23:30:49.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.854+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 msgr2=0x5618b678e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.854+0000 7f791a0c9740 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b678e060 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x5618b6792170 tx=0x7f790401e640 comp rx=0 tx=0).stop 2024-09-15T23:30:49.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.854+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 shutdown_connections 2024-09-15T23:30:49.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.854+0000 7f791a0c9740 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5618b680f380 0x5618b68117a0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.854+0000 7f791a0c9740 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5618b6793ae0 0x5618b678e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.854+0000 7f791a0c9740 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5618b6789220 0x5618b678e060 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.855+0000 7f791a0c9740 1 --2- 172.21.15.62:0/3634801120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5618b6788900 0x5618b678db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.855+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 >> 172.21.15.62:0/3634801120 conn(0x5618b6784e90 msgr2=0x5618b6790ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.855+0000 7f78f57fa640 0 client.27994 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:49.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.855+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 shutdown_connections 2024-09-15T23:30:49.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.855+0000 7f791a0c9740 1 -- 172.21.15.62:0/3634801120 wait complete. 2024-09-15T23:30:49.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.855+0000 7f791a0c9740 20 client.27994 trim_cache size 0 max 16384 2024-09-15T23:30:49.857 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:30:49.857 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 36, 2024-09-15T23:30:49.857 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:30:49.857 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 54, 2024-09-15T23:30:49.857 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "2bc220ac": { 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 54, 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 137.7, 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:30:49.858 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:30:49.859 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:30:49.859 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:30:49.859 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:30:49.859 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:30:49.859 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:30:49.859 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:30:49.859 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:30:49.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.860+0000 7f7916a8e640 1 -- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78f80a3410 msgr2=0x7f78f80a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.860+0000 7f7916a8e640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78f80a3410 0x7f78f80a58d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7910113f60 tx=0x7f7904006c40 comp rx=0 tx=0).stop 2024-09-15T23:30:49.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.860+0000 7f7916a8e640 1 -- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f791006ea30 msgr2=0x7f7910119710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:49.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.860+0000 7f7916a8e640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f791006ea30 0x7f7910119710 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f790002ac10 tx=0x7f7900032f10 comp rx=0 tx=0).stop 2024-09-15T23:30:49.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.860+0000 7f7916a8e640 1 -- 172.21.15.62:0/3535303216 shutdown_connections 2024-09-15T23:30:49.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.860+0000 7f7916a8e640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f78f80a3410 0x7f78f80a58d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.860+0000 7f7916a8e640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f791011b140 0x7f7910112d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.860+0000 7f7916a8e640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7910110190 0x7f79101127f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.860+0000 7f7916a8e640 1 --2- 172.21.15.62:0/3535303216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f791006ea30 0x7f7910119710 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:49.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.861+0000 7f7916a8e640 1 -- 172.21.15.62:0/3535303216 >> 172.21.15.62:0/3535303216 conn(0x7f7910073180 msgr2=0x7f791006b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:49.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.861+0000 7f7916a8e640 1 -- 172.21.15.62:0/3535303216 shutdown_connections 2024-09-15T23:30:49.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:30:49.861+0000 7f7916a8e640 1 -- 172.21.15.62:0/3535303216 wait complete. 2024-09-15T23:30:51.467 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:51 smithi062 ceph-mon[30949]: pgmap v807: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 531 op/s 2024-09-15T23:30:51.503 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '2bc220ac', quiesce took 5.6 seconds. Will release after: 12.1 2024-09-15T23:30:51.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:51 smithi067 ceph-mon[35665]: pgmap v807: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 531 op/s 2024-09-15T23:30:51.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:51 smithi148 ceph-mon[35610]: pgmap v807: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.9 MiB/s wr, 531 op/s 2024-09-15T23:30:53.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:53 smithi062 ceph-mon[30949]: pgmap v808: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 505 op/s 2024-09-15T23:30:53.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:53 smithi067 ceph-mon[35665]: pgmap v808: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 505 op/s 2024-09-15T23:30:53.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:53 smithi148 ceph-mon[35610]: pgmap v808: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 4.6 MiB/s wr, 505 op/s 2024-09-15T23:30:54.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 875.0 s, 36.8 tps, lat 155.874 ms stddev 122.005 2024-09-15T23:30:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:55 smithi062 ceph-mon[30949]: pgmap v809: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 MiB/s wr, 484 op/s 2024-09-15T23:30:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:30:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:30:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:30:55.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:30:55.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:55 smithi067 ceph-mon[35665]: pgmap v809: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 MiB/s wr, 484 op/s 2024-09-15T23:30:55.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:30:55.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:30:55.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:30:55.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:30:55.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:55 smithi148 ceph-mon[35610]: pgmap v809: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.3 MiB/s wr, 484 op/s 2024-09-15T23:30:55.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:30:55.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:30:55.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:30:55.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:30:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.870+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1368734385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 msgr2=0x7fef18100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.870+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1368734385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 0x7fef18100e60 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fef08009900 tx=0x7fef0802a410 comp rx=0 tx=0).stop 2024-09-15T23:30:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1368734385 shutdown_connections 2024-09-15T23:30:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1368734385 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef18102610 0x7fef18108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1368734385 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef18101c50 0x7fef181020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1368734385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 0x7fef18100e60 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1368734385 >> 172.21.15.62:0/1368734385 conn(0x7fef180fc1e0 msgr2=0x7fef180fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1368734385 shutdown_connections 2024-09-15T23:30:55.874 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1368734385 wait complete. 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 Processor -- start 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 -- start start 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 0x7fef18197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.871+0000 7fef1ce5c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef18101c50 0x7fef181982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1ce5c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef18102610 0x7fef1819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1ce5c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef1810d830 con 0x7fef18100a60 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1ce5c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef1810d6b0 con 0x7fef18102610 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1ce5c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fef1810d9b0 con 0x7fef18101c50 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 0x7fef18197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 0x7fef18197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50042/0 (socket says 172.21.15.62:50042) 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1659c640 1 -- 172.21.15.62:0/1429552192 learned_addr learned my addr 172.21.15.62:0/1429552192 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef15d9b640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef18101c50 0x7fef181982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef16d9d640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef18102610 0x7fef1819f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1659c640 1 -- 172.21.15.62:0/1429552192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef18101c50 msgr2=0x7fef181982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1659c640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef18101c50 0x7fef181982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1659c640 1 -- 172.21.15.62:0/1429552192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef18102610 msgr2=0x7fef1819f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1659c640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef18102610 0x7fef1819f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1659c640 1 -- 172.21.15.62:0/1429552192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1819f9c0 con 0x7fef18100a60 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1659c640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 0x7fef18197db0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fef08031440 tx=0x7fef08032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef077fe640 1 -- 172.21.15.62:0/1429552192 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0803f030 con 0x7fef18100a60 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef077fe640 1 -- 172.21.15.62:0/1429552192 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef08003ce0 con 0x7fef18100a60 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1429552192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef1819fc50 con 0x7fef18100a60 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef077fe640 1 -- 172.21.15.62:0/1429552192 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08035070 con 0x7fef18100a60 2024-09-15T23:30:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.872+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1429552192 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef181a0190 con 0x7fef18100a60 2024-09-15T23:30:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.875+0000 7fef077fe640 1 -- 172.21.15.62:0/1429552192 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fef08042030 con 0x7fef18100a60 2024-09-15T23:30:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.875+0000 7fef1fbf3740 1 Processor -- start 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.875+0000 7fef1fbf3740 1 -- start start 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 0x558f4d5efda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d5f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 0x558f4d5f7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558f4d5532f0 con 0x558f4d5f02e0 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558f4d4fb270 con 0x558f4d5f5370 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558f4d54f2b0 con 0x558f4d5ef9a0 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d5f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef077fe640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef000a3370 0x7fef000a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef15d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 0x558f4d5f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d5f4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50058/0 (socket says 172.21.15.62:50058) 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef1659c640 1 -- 172.21.15.62:0/3321624749 learned_addr learned my addr 172.21.15.62:0/3321624749 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef16d9d640 1 --2- 172.21.15.62:0/3321624749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 0x558f4d5efda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef16d9d640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef000a3370 0x7fef000a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef077fe640 1 -- 172.21.15.62:0/1429552192 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fef080f0350 con 0x7fef18100a60 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef15d9b640 1 -- 172.21.15.62:0/3321624749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 msgr2=0x558f4d5efda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef15d9b640 1 --2- 172.21.15.62:0/3321624749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 0x558f4d5efda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef15d9b640 1 -- 172.21.15.62:0/3321624749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 msgr2=0x558f4d5f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef15d9b640 1 --2- 172.21.15.62:0/3321624749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d5f4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef15d9b640 1 -- 172.21.15.62:0/3321624749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f4d4fa340 con 0x558f4d5f5370 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef15d9b640 1 --2- 172.21.15.62:0/3321624749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 0x558f4d5f7830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fef0c004610 tx=0x7fef0c00bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82247a8d4943e4cb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.876+0000 7fef16d9d640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef000a3370 0x7fef000a5830 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fef10013b90 tx=0x7fef10004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef04ff9640 1 -- 172.21.15.62:0/3321624749 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c003bd0 con 0x558f4d5f5370 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef04ff9640 1 -- 172.21.15.62:0/3321624749 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef0c015070 con 0x558f4d5f5370 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef04ff9640 1 -- 172.21.15.62:0/3321624749 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c004d90 con 0x558f4d5f5370 2024-09-15T23:30:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef1fbf3740 1 -- 172.21.15.62:0/3321624749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 msgr2=0x558f4d5f7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/3321624749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 0x558f4d5f7830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fef0c004610 tx=0x7fef0c00bb20 comp rx=0 tx=0).stop 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef1fbf3740 1 -- 172.21.15.62:0/3321624749 shutdown_connections 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/3321624749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 0x558f4d5f7830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/3321624749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d5f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/3321624749 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 0x558f4d5efda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef1fbf3740 1 -- 172.21.15.62:0/3321624749 >> 172.21.15.62:0/3321624749 conn(0x558f4d5eb010 msgr2=0x558f4d5eb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.877+0000 7fef1fbf3740 1 -- 172.21.15.62:0/3321624749 shutdown_connections 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 1 -- 172.21.15.62:0/3321624749 wait complete. 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 1 -- start start 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 1 Processor -- start 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5f9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 0x558f4d5fa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5f5370 0x558f4d601110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x558f4d5f8340 con 0x558f4d5ef9a0 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x558f4d4dc290 con 0x558f4d5f02e0 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x558f4d5f84c0 con 0x558f4d5f5370 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1fbf3740 10 client.0 _ll_register_callbacks cb 0x558f4d50d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5f9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5f9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50068/0 (socket says 172.21.15.62:50068) 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1659c640 1 -- 172.21.15.62:0/789163378 learned_addr learned my addr 172.21.15.62:0/789163378 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef15d9b640 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 0x558f4d5fa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef16d9d640 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5f5370 0x558f4d601110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1659c640 1 -- 172.21.15.62:0/789163378 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5f5370 msgr2=0x558f4d601110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1659c640 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5f5370 0x558f4d601110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.878+0000 7fef1659c640 1 -- 172.21.15.62:0/789163378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 msgr2=0x558f4d5fa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1659c640 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 0x558f4d5fa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1659c640 1 -- 172.21.15.62:0/789163378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f4d266210 con 0x558f4d5ef9a0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1659c640 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5f9b90 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fef080b8f30 tx=0x7fef080b8140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=780985f79253374 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7feefdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7feefdffb640 1 -- 172.21.15.62:0/789163378 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08049040 con 0x558f4d5ef9a0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7feefdffb640 1 -- 172.21.15.62:0/789163378 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fef080bc850 con 0x558f4d5ef9a0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1fbf3740 1 -- 172.21.15.62:0/789163378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef0c015070 con 0x558f4d5ef9a0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7feefdffb640 1 -- 172.21.15.62:0/789163378 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fef080bbf50 con 0x558f4d5ef9a0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1fbf3740 10 client.18831 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1fbf3740 1 -- 172.21.15.62:0/789163378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558f4d332620 con 0x558f4d5ef9a0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7feefdffb640 1 -- 172.21.15.62:0/789163378 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef0c015070 con 0x558f4d5ef9a0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7feefdffb640 1 -- 172.21.15.62:0/789163378 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fef080c4dd0 con 0x558f4d5ef9a0 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1fbf3740 10 client.18831 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1fbf3740 10 client.18831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1fbf3740 20 client.18831 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558f4d605750 0x558f4d607bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1fbf3740 4 client.18831 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.879+0000 7fef1fbf3740 1 -- 172.21.15.62:0/789163378 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f4d560a20 con 0x558f4d605750 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.880+0000 7fef15d9b640 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558f4d605750 0x558f4d607bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.880+0000 7feefdffb640 10 client.18831 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.880+0000 7fef15d9b640 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558f4d605750 0x558f4d607bd0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c0c0da9667d73d server_cookie=344563309dd1366f in_seq=0 out_seq=0 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7feefdffb640 1 -- 172.21.15.62:0/789163378 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558f4d560a20 con 0x558f4d605750 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7feefdffb640 10 client.18831 handle_command_reply: tid=0 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 client.18831 shutdown 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 -- 172.21.15.62:0/789163378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 msgr2=0x558f4d5f9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5f9b90 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fef080b8f30 tx=0x7fef080b8140 comp rx=0 tx=0).stop 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 -- 172.21.15.62:0/789163378 shutdown_connections 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558f4d605750 0x558f4d607bd0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5f5370 0x558f4d601110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 0x558f4d5fa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7feefdffb640 0 client.18831 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/789163378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5f9b90 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 -- 172.21.15.62:0/789163378 >> 172.21.15.62:0/789163378 conn(0x558f4d5eb010 msgr2=0x558f4d5ec820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 -- 172.21.15.62:0/789163378 shutdown_connections 2024-09-15T23:30:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 1 -- 172.21.15.62:0/789163378 wait complete. 2024-09-15T23:30:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.881+0000 7fef1fbf3740 20 client.18831 trim_cache size 0 max 16384 2024-09-15T23:30:55.901 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.895+0000 7fef1fbf3740 1 Processor -- start 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.895+0000 7fef1fbf3740 1 -- start start 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.895+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5fa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.895+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 0x558f4d5ea0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5f5370 0x558f4d60c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef181acec0 con 0x558f4d5ef9a0 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef181aa760 con 0x558f4d5f02e0 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fef181a85b0 con 0x558f4d5f5370 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5fa160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50070/0 (socket says 172.21.15.62:50070) 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1659c640 1 -- 172.21.15.62:0/1975118643 learned_addr learned my addr 172.21.15.62:0/1975118643 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1659c640 1 -- 172.21.15.62:0/1975118643 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5f5370 msgr2=0x558f4d60c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1659c640 1 --2- 172.21.15.62:0/1975118643 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5f5370 0x558f4d60c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1659c640 1 -- 172.21.15.62:0/1975118643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 msgr2=0x558f4d5ea0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef15d9b640 1 --2- 172.21.15.62:0/1975118643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 0x558f4d5ea0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1659c640 1 --2- 172.21.15.62:0/1975118643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 0x558f4d5ea0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1659c640 1 -- 172.21.15.62:0/1975118643 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f4d5f07c0 con 0x558f4d5ef9a0 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef15d9b640 1 --2- 172.21.15.62:0/1975118643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 0x558f4d5ea0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7fef1659c640 1 --2- 172.21.15.62:0/1975118643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5fa160 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fef080bc580 tx=0x7fef0802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21d53277e0b0a67 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7feefffff640 1 -- 172.21.15.62:0/1975118643 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08049040 con 0x558f4d5ef9a0 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7feefffff640 1 -- 172.21.15.62:0/1975118643 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef080bc5f0 con 0x558f4d5ef9a0 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.896+0000 7feefffff640 1 -- 172.21.15.62:0/1975118643 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef080f01b0 con 0x558f4d5ef9a0 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 -- 172.21.15.62:0/1975118643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 msgr2=0x558f4d5fa160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/1975118643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5fa160 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fef080bc580 tx=0x7fef0802a410 comp rx=0 tx=0).stop 2024-09-15T23:30:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 -- 172.21.15.62:0/1975118643 shutdown_connections 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/1975118643 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5f5370 0x558f4d60c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/1975118643 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f02e0 0x558f4d5ea0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/1975118643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5ef9a0 0x558f4d5fa160 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 -- 172.21.15.62:0/1975118643 >> 172.21.15.62:0/1975118643 conn(0x558f4d5eb010 msgr2=0x558f4d5e99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 -- 172.21.15.62:0/1975118643 shutdown_connections 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 -- 172.21.15.62:0/1975118643 wait complete. 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 -- start start 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 Processor -- start 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 0x558f4d66bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d66c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 0x558f4d673440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef181a8c20 con 0x558f4d5f02e0 2024-09-15T23:30:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fef181aa090 con 0x558f4d5f5370 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.897+0000 7fef1fbf3740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fef181a92f0 con 0x558f4d5ef9a0 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef1fbf3740 10 client.0 _ll_register_callbacks cb 0x7fef08009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef15d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d66c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef15d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d66c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50082/0 (socket says 172.21.15.62:50082) 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef15d9b640 1 -- 172.21.15.62:0/2292579404 learned_addr learned my addr 172.21.15.62:0/2292579404 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef1659c640 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 0x558f4d66bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef16d9d640 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 0x558f4d673440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef15d9b640 1 -- 172.21.15.62:0/2292579404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 msgr2=0x558f4d66bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef15d9b640 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 0x558f4d66bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef15d9b640 1 -- 172.21.15.62:0/2292579404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 msgr2=0x558f4d673440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef15d9b640 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 0x558f4d673440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef15d9b640 1 -- 172.21.15.62:0/2292579404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f4d673a50 con 0x558f4d5f02e0 2024-09-15T23:30:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef1659c640 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 0x558f4d66bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef15d9b640 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d66c4e0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00b620 tx=0x7fef0c005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656dafd872b24e5e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7feefdffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7feefdffb640 1 -- 172.21.15.62:0/2292579404 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0c003e50 con 0x558f4d5f02e0 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7feefdffb640 1 -- 172.21.15.62:0/2292579404 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fef0c011440 con 0x558f4d5f02e0 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef1fbf3740 1 -- 172.21.15.62:0/2292579404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef080bc5f0 con 0x558f4d5f02e0 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7feefdffb640 1 -- 172.21.15.62:0/2292579404 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fef0c011660 con 0x558f4d5f02e0 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef1fbf3740 10 client.18837 fetch_fsmap learned FSMap version 36 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7fef1fbf3740 1 -- 172.21.15.62:0/2292579404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558f4d674d40 con 0x558f4d5f02e0 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.898+0000 7feefdffb640 1 -- 172.21.15.62:0/2292579404 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef080bc5f0 con 0x558f4d5f02e0 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7feefdffb640 1 -- 172.21.15.62:0/2292579404 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fef0c0257c0 con 0x558f4d5f02e0 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7fef1fbf3740 10 client.18837 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7fef1fbf3740 10 client.18837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:30:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7fef1fbf3740 20 client.18837 populate_metadata read hostname 'smithi062' 2024-09-15T23:30:55.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558f4d6755e0 0x558f4d677a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:30:55.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7fef1fbf3740 4 client.18837 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:30:55.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7fef1fbf3740 1 -- 172.21.15.62:0/2292579404 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558f4d55ad70 con 0x558f4d6755e0 2024-09-15T23:30:55.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7fef1659c640 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558f4d6755e0 0x558f4d677a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:30:55.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7fef1659c640 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558f4d6755e0 0x558f4d677a00 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ada286525bc5d82 server_cookie=23776bd4dcb3909a in_seq=0 out_seq=0 2024-09-15T23:30:55.907 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.899+0000 7feefdffb640 10 client.18837 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.910+0000 7feefdffb640 1 -- 172.21.15.62:0/2292579404 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558f4d55ad70 con 0x558f4d6755e0 2024-09-15T23:30:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.910+0000 7feefdffb640 10 client.18837 handle_command_reply: tid=0 2024-09-15T23:30:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.910+0000 7fef1fbf3740 1 client.18837 shutdown 2024-09-15T23:30:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 -- 172.21.15.62:0/2292579404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 msgr2=0x558f4d66c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d66c4e0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fef0c00b620 tx=0x7fef0c005e70 comp rx=0 tx=0).stop 2024-09-15T23:30:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 -- 172.21.15.62:0/2292579404 shutdown_connections 2024-09-15T23:30:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x558f4d6755e0 0x558f4d677a00 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558f4d5f5370 0x558f4d673440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7feefdffb640 0 client.18837 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:30:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x558f4d5f02e0 0x558f4d66c4e0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 --2- 172.21.15.62:0/2292579404 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x558f4d5ef9a0 0x558f4d66bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 -- 172.21.15.62:0/2292579404 >> 172.21.15.62:0/2292579404 conn(0x558f4d5eb010 msgr2=0x558f4d5ea1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 -- 172.21.15.62:0/2292579404 shutdown_connections 2024-09-15T23:30:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 1 -- 172.21.15.62:0/2292579404 wait complete. 2024-09-15T23:30:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1fbf3740 20 client.18837 trim_cache size 0 max 16384 2024-09-15T23:30:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:30:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:30:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:30:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:30:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef000a3370 msgr2=0x7fef000a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef000a3370 0x7fef000a5830 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fef10013b90 tx=0x7fef10004a40 comp rx=0 tx=0).stop 2024-09-15T23:30:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 msgr2=0x7fef18197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:30:55.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 0x7fef18197db0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fef08031440 tx=0x7fef08032e20 comp rx=0 tx=0).stop 2024-09-15T23:30:55.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1429552192 shutdown_connections 2024-09-15T23:30:55.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fef000a3370 0x7fef000a5830 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef18102610 0x7fef1819f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fef18101c50 0x7fef181982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 --2- 172.21.15.62:0/1429552192 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef18100a60 0x7fef18197db0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:30:55.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.911+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1429552192 >> 172.21.15.62:0/1429552192 conn(0x7fef180fc1e0 msgr2=0x7fef18104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:30:55.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.912+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1429552192 shutdown_connections 2024-09-15T23:30:55.916 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:30:55.912+0000 7fef1ce5c640 1 -- 172.21.15.62:0/1429552192 wait complete. 2024-09-15T23:30:57.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:57 smithi062 ceph-mon[30949]: pgmap v810: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-09-15T23:30:57.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:57 smithi067 ceph-mon[35665]: pgmap v810: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-09-15T23:30:57.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:57 smithi148 ceph-mon[35610]: pgmap v810: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 438 op/s 2024-09-15T23:30:59.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:30:59 smithi062 ceph-mon[30949]: pgmap v811: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.3 MiB/s wr, 365 op/s 2024-09-15T23:30:59.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:30:59 smithi067 ceph-mon[35665]: pgmap v811: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.3 MiB/s wr, 365 op/s 2024-09-15T23:30:59.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:30:59 smithi148 ceph-mon[35610]: pgmap v811: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.3 MiB/s wr, 365 op/s 2024-09-15T23:30:59.575 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 880.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:31:01.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:01 smithi062 ceph-mon[30949]: pgmap v812: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2024-09-15T23:31:01.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:01 smithi067 ceph-mon[35665]: pgmap v812: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2024-09-15T23:31:01.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:01 smithi148 ceph-mon[35610]: pgmap v812: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2024-09-15T23:31:02.460 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:31:02.734 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:03.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.187+0000 7f657cee2640 1 -- 172.21.15.62:0/2505755694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578103660 msgr2=0x7f657810bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:03.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.187+0000 7f657cee2640 1 --2- 172.21.15.62:0/2505755694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578103660 0x7f657810bb50 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f6568009900 tx=0x7f656802a480 comp rx=0 tx=0).stop 2024-09-15T23:31:03.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.187+0000 7f657cee2640 1 -- 172.21.15.62:0/2505755694 shutdown_connections 2024-09-15T23:31:03.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.187+0000 7f657cee2640 1 --2- 172.21.15.62:0/2505755694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f657810c170 0x7f657810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.187+0000 7f657cee2640 1 --2- 172.21.15.62:0/2505755694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578103660 0x7f657810bb50 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.187+0000 7f657cee2640 1 --2- 172.21.15.62:0/2505755694 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6578102d40 0x7f6578103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.187+0000 7f657cee2640 1 -- 172.21.15.62:0/2505755694 >> 172.21.15.62:0/2505755694 conn(0x7f65780fc7a0 msgr2=0x7f65780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.187+0000 7f657cee2640 1 -- 172.21.15.62:0/2505755694 shutdown_connections 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.187+0000 7f657cee2640 1 -- 172.21.15.62:0/2505755694 wait complete. 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657cee2640 1 Processor -- start 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657cee2640 1 -- start start 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657cee2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578102d40 0x7f6578198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657cee2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6578103660 0x7f6578198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657cee2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f657810c170 0x7f657819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657cee2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6578111fb0 con 0x7f6578102d40 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657cee2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6578111e30 con 0x7f657810c170 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657cee2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f6578112130 con 0x7f6578103660 2024-09-15T23:31:03.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578102d40 0x7f6578198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578102d40 0x7f6578198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45396/0 (socket says 172.21.15.62:45396) 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657659c640 1 -- 172.21.15.62:0/4141281730 learned_addr learned my addr 172.21.15.62:0/4141281730 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f6575d9b640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6578103660 0x7f6578198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f6576d9d640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f657810c170 0x7f657819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.188+0000 7f657659c640 1 -- 172.21.15.62:0/4141281730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6578103660 msgr2=0x7f6578198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f657659c640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6578103660 0x7f6578198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f657659c640 1 -- 172.21.15.62:0/4141281730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f657810c170 msgr2=0x7f657819ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f657659c640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f657810c170 0x7f657819ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f657659c640 1 -- 172.21.15.62:0/4141281730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657819d520 con 0x7f6578102d40 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f6576d9d640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f657810c170 0x7f657819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f657659c640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578102d40 0x7f6578198570 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f656000b4a0 tx=0x7f656000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f656f7fe640 1 -- 172.21.15.62:0/4141281730 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560014070 con 0x7f6578102d40 2024-09-15T23:31:03.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f656f7fe640 1 -- 172.21.15.62:0/4141281730 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6560010af0 con 0x7f6578102d40 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f656f7fe640 1 -- 172.21.15.62:0/4141281730 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560017d10 con 0x7f6578102d40 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f657cee2640 1 -- 172.21.15.62:0/4141281730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657819d810 con 0x7f6578102d40 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.189+0000 7f657cee2640 1 -- 172.21.15.62:0/4141281730 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65781a50f0 con 0x7f6578102d40 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.191+0000 7f656f7fe640 1 -- 172.21.15.62:0/4141281730 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f6560017070 con 0x7f6578102d40 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f657f2bb740 1 Processor -- start 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f657f2bb740 1 -- start start 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc7710a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710acc0 0x55cc77119550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc77119a90 0x55cc7711be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f657f2bb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55cc770702e0 con 0x55cc77119a90 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f657f2bb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55cc77017ce0 con 0x55cc7710a3a0 2024-09-15T23:31:03.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f657f2bb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55cc7706c2a0 con 0x55cc7710acc0 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f6575d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc77119a90 0x55cc7711be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f6575d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc77119a90 0x55cc7711be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45412/0 (socket says 172.21.15.62:45412) 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f6575d9b640 1 -- 172.21.15.62:0/1891836563 learned_addr learned my addr 172.21.15.62:0/1891836563 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f657659c640 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710acc0 0x55cc77119550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f6576d9d640 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc7710a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.192+0000 7f656f7fe640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65640a3240 0x7f65640a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f6575d9b640 1 -- 172.21.15.62:0/1891836563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710acc0 msgr2=0x55cc77119550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f6575d9b640 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710acc0 0x55cc77119550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f6575d9b640 1 -- 172.21.15.62:0/1891836563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 msgr2=0x55cc7710a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f6575d9b640 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc7710a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f6575d9b640 1 -- 172.21.15.62:0/1891836563 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc76ffc440 con 0x55cc77119a90 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f6576d9d640 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc7710a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:03.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f6576d9d640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65640a3240 0x7f65640a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f656f7fe640 1 -- 172.21.15.62:0/4141281730 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f65600cf180 con 0x7f6578102d40 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f6575d9b640 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc77119a90 0x55cc7711be80 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6568009900 tx=0x7f6568032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298890c5e38fd923 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f656cff9640 1 -- 172.21.15.62:0/1891836563 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656803f070 con 0x55cc77119a90 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f656cff9640 1 -- 172.21.15.62:0/1891836563 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6568032e20 con 0x55cc77119a90 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.193+0000 7f656cff9640 1 -- 172.21.15.62:0/1891836563 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6568035070 con 0x55cc77119a90 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 -- 172.21.15.62:0/1891836563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc77119a90 msgr2=0x55cc7711be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f6576d9d640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65640a3240 0x7f65640a5700 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6570005fd0 tx=0x7f6570004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc77119a90 0x55cc7711be80 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f6568009900 tx=0x7f6568032c30 comp rx=0 tx=0).stop 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 -- 172.21.15.62:0/1891836563 shutdown_connections 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc77119a90 0x55cc7711be80 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710acc0 0x55cc77119550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 --2- 172.21.15.62:0/1891836563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc7710a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 -- 172.21.15.62:0/1891836563 >> 172.21.15.62:0/1891836563 conn(0x55cc77106a30 msgr2=0x55cc77106e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:03.203 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 -- 172.21.15.62:0/1891836563 shutdown_connections 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 -- 172.21.15.62:0/1891836563 wait complete. 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 -- start start 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.194+0000 7f657f2bb740 1 Processor -- start 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710a3a0 0x55cc77119460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77116710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc77119a90 0x55cc77116d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f657f2bb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55cc7711c660 con 0x55cc7710acc0 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f657f2bb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55cc7702b050 con 0x55cc77119a90 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f657f2bb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55cc7711c7e0 con 0x55cc7710a3a0 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f657f2bb740 10 client.0 _ll_register_callbacks cb 0x55cc7706b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6575d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77116710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6575d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77116710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45420/0 (socket says 172.21.15.62:45420) 2024-09-15T23:31:03.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6575d9b640 1 -- 172.21.15.62:0/2560503481 learned_addr learned my addr 172.21.15.62:0/2560503481 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6576d9d640 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc77119a90 0x55cc77116d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f657659c640 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710a3a0 0x55cc77119460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6575d9b640 1 -- 172.21.15.62:0/2560503481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710a3a0 msgr2=0x55cc77119460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6575d9b640 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710a3a0 0x55cc77119460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6575d9b640 1 -- 172.21.15.62:0/2560503481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc77119a90 msgr2=0x55cc77116d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6575d9b640 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc77119a90 0x55cc77116d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6575d9b640 1 -- 172.21.15.62:0/2560503481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc76de13c0 con 0x55cc7710acc0 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f657659c640 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710a3a0 0x55cc77119460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f6575d9b640 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77116710 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f6568031550 tx=0x7f65680318b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbcaa8257ccd2ad server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.195+0000 7f655dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:03.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f655dffb640 1 -- 172.21.15.62:0/2560503481 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656804b060 con 0x55cc7710acc0 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f655dffb640 1 -- 172.21.15.62:0/2560503481 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f6568004440 con 0x55cc7710acc0 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f657f2bb740 1 -- 172.21.15.62:0/2560503481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6568032e20 con 0x55cc7710acc0 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f655dffb640 1 -- 172.21.15.62:0/2560503481 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6568052050 con 0x55cc7710acc0 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f657f2bb740 10 client.18849 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f657f2bb740 1 -- 172.21.15.62:0/2560503481 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55cc771280d0 con 0x55cc7710acc0 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f655dffb640 1 -- 172.21.15.62:0/2560503481 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6568032e20 con 0x55cc7710acc0 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f655dffb640 1 -- 172.21.15.62:0/2560503481 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f6568004aa0 con 0x55cc7710acc0 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f657f2bb740 10 client.18849 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f657f2bb740 10 client.18849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f657f2bb740 20 client.18849 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cc77129420 0x55cc7712b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f657f2bb740 4 client.18849 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.196+0000 7f657f2bb740 1 -- 172.21.15.62:0/2560503481 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc7707d850 con 0x55cc77129420 2024-09-15T23:31:03.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.197+0000 7f657659c640 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cc77129420 0x55cc7712b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.197+0000 7f657659c640 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cc77129420 0x55cc7712b810 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a30c72c78057f89b server_cookie=cd18cf73fb57b666 in_seq=0 out_seq=0 2024-09-15T23:31:03.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.197+0000 7f655dffb640 10 client.18849 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f655dffb640 1 -- 172.21.15.62:0/2560503481 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55cc7707d850 con 0x55cc77129420 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f655dffb640 10 client.18849 handle_command_reply: tid=0 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 client.18849 shutdown 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 -- 172.21.15.62:0/2560503481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 msgr2=0x55cc77116710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77116710 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f6568031550 tx=0x7f65680318b0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 -- 172.21.15.62:0/2560503481 shutdown_connections 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cc77129420 0x55cc7712b810 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc77119a90 0x55cc77116d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77116710 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2560503481 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc7710a3a0 0x55cc77119460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f655dffb640 0 client.18849 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:03.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 -- 172.21.15.62:0/2560503481 >> 172.21.15.62:0/2560503481 conn(0x55cc77106a30 msgr2=0x55cc7710c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:03.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 -- 172.21.15.62:0/2560503481 shutdown_connections 2024-09-15T23:31:03.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 1 -- 172.21.15.62:0/2560503481 wait complete. 2024-09-15T23:31:03.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.198+0000 7f657f2bb740 20 client.18849 trim_cache size 0 max 16384 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657f2bb740 1 Processor -- start 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657f2bb740 1 -- start start 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710a3a0 0x55cc77125900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710acc0 0x55cc77130e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 0x55cc77131370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657f2bb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f65781af370 con 0x55cc7710a3a0 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657f2bb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65781acc10 con 0x55cc7710acc0 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657f2bb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f65781aaa60 con 0x55cc77119a90 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710a3a0 0x55cc77125900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710a3a0 0x55cc77125900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45434/0 (socket says 172.21.15.62:45434) 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657659c640 1 -- 172.21.15.62:0/2547894682 learned_addr learned my addr 172.21.15.62:0/2547894682 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:03.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f6576d9d640 1 --2- 172.21.15.62:0/2547894682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 0x55cc77131370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f6575d9b640 1 --2- 172.21.15.62:0/2547894682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710acc0 0x55cc77130e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657659c640 1 -- 172.21.15.62:0/2547894682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 msgr2=0x55cc77131370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657659c640 1 --2- 172.21.15.62:0/2547894682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 0x55cc77131370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657659c640 1 -- 172.21.15.62:0/2547894682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710acc0 msgr2=0x55cc77130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657659c640 1 --2- 172.21.15.62:0/2547894682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710acc0 0x55cc77130e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657659c640 1 -- 172.21.15.62:0/2547894682 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc7710aaf0 con 0x55cc7710a3a0 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.216+0000 7f657659c640 1 --2- 172.21.15.62:0/2547894682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710a3a0 0x55cc77125900 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f65600ccc70 tx=0x7f6560095f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f36e5b41d8139c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f655ffff640 1 -- 172.21.15.62:0/2547894682 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656001e040 con 0x55cc7710a3a0 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f655ffff640 1 -- 172.21.15.62:0/2547894682 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656009b420 con 0x55cc7710a3a0 2024-09-15T23:31:03.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f655ffff640 1 -- 172.21.15.62:0/2547894682 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6560017360 con 0x55cc7710a3a0 2024-09-15T23:31:03.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f657f2bb740 1 -- 172.21.15.62:0/2547894682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710a3a0 msgr2=0x55cc77125900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:03.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2547894682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710a3a0 0x55cc77125900 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f65600ccc70 tx=0x7f6560095f90 comp rx=0 tx=0).stop 2024-09-15T23:31:03.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f657f2bb740 1 -- 172.21.15.62:0/2547894682 shutdown_connections 2024-09-15T23:31:03.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2547894682 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 0x55cc77131370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2547894682 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710acc0 0x55cc77130e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f657f2bb740 1 --2- 172.21.15.62:0/2547894682 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710a3a0 0x55cc77125900 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f657f2bb740 1 -- 172.21.15.62:0/2547894682 >> 172.21.15.62:0/2547894682 conn(0x55cc77106a30 msgr2=0x55cc77111d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f657f2bb740 1 -- 172.21.15.62:0/2547894682 shutdown_connections 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.217+0000 7f657f2bb740 1 -- 172.21.15.62:0/2547894682 wait complete. 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657f2bb740 1 -- start start 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657f2bb740 1 Processor -- start 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc77121a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77121fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657f2bb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 0x55cc7711ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657f2bb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f65781ab0d0 con 0x55cc7710acc0 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657f2bb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65781ac540 con 0x55cc7710a3a0 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657f2bb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f65781ab7a0 con 0x55cc77119a90 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657f2bb740 10 client.0 _ll_register_callbacks cb 0x7f65600974f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f6575d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77121fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f6575d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77121fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45450/0 (socket says 172.21.15.62:45450) 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f6575d9b640 1 -- 172.21.15.62:0/3436856631 learned_addr learned my addr 172.21.15.62:0/3436856631 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.218+0000 7f657659c640 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc77121a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f6576d9d640 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 0x55cc7711ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f6575d9b640 1 -- 172.21.15.62:0/3436856631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 msgr2=0x55cc7711ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f6575d9b640 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 0x55cc7711ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f6575d9b640 1 -- 172.21.15.62:0/3436856631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 msgr2=0x55cc77121a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f6575d9b640 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc77121a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f6575d9b640 1 -- 172.21.15.62:0/3436856631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc7711ef80 con 0x55cc7710acc0 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f6576d9d640 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 0x55cc7711ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f657659c640 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc77121a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f6575d9b640 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77121fb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6568005010 tx=0x7f6568009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcda3c00c405d9d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f655dffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:03.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f655dffb640 1 -- 172.21.15.62:0/3436856631 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656805b060 con 0x55cc7710acc0 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f655dffb640 1 -- 172.21.15.62:0/3436856631 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f65680046c0 con 0x55cc7710acc0 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f657f2bb740 1 -- 172.21.15.62:0/3436856631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f656009b420 con 0x55cc7710acc0 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f655dffb640 1 -- 172.21.15.62:0/3436856631 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6568062050 con 0x55cc7710acc0 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f657f2bb740 10 client.18855 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.219+0000 7f657f2bb740 1 -- 172.21.15.62:0/3436856631 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55cc771945c0 con 0x55cc7710acc0 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f655dffb640 1 -- 172.21.15.62:0/3436856631 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656009b420 con 0x55cc7710acc0 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f655dffb640 1 -- 172.21.15.62:0/3436856631 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f6568003990 con 0x55cc7710acc0 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f657f2bb740 10 client.18855 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f657f2bb740 10 client.18855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f657f2bb740 20 client.18855 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f657f2bb740 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cc77194c50 0x55cc77197070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f657f2bb740 4 client.18855 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:31:03.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f657f2bb740 1 -- 172.21.15.62:0/3436856631 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cc77077d60 con 0x55cc77194c50 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f657659c640 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cc77194c50 0x55cc77197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f657659c640 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cc77194c50 0x55cc77197070 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8420ab28e943fe2 server_cookie=4fe51397d13fca7d in_seq=0 out_seq=0 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.220+0000 7f655dffb640 10 client.18855 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f655dffb640 1 -- 172.21.15.62:0/3436856631 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cc77077d60 con 0x55cc77194c50 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f655dffb640 10 client.18855 handle_command_reply: tid=0 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 client.18855 shutdown 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 -- 172.21.15.62:0/3436856631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 msgr2=0x55cc77121fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77121fb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6568005010 tx=0x7f6568009650 comp rx=0 tx=0).stop 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 -- 172.21.15.62:0/3436856631 shutdown_connections 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55cc77194c50 0x55cc77197070 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55cc77119a90 0x55cc7711ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cc7710acc0 0x55cc77121fb0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 --2- 172.21.15.62:0/3436856631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc7710a3a0 0x55cc77121a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 -- 172.21.15.62:0/3436856631 >> 172.21.15.62:0/3436856631 conn(0x55cc77106a30 msgr2=0x55cc77117f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:03.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f655dffb640 0 client.18855 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:03.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 -- 172.21.15.62:0/3436856631 shutdown_connections 2024-09-15T23:31:03.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 1 -- 172.21.15.62:0/3436856631 wait complete. 2024-09-15T23:31:03.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657f2bb740 20 client.18855 trim_cache size 0 max 16384 2024-09-15T23:31:03.228 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:31:03.228 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:31:03.228 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:31:03.228 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657cee2640 1 -- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65640a3240 msgr2=0x7f65640a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657cee2640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65640a3240 0x7f65640a5700 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6570005fd0 tx=0x7f6570004a40 comp rx=0 tx=0).stop 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657cee2640 1 -- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578102d40 msgr2=0x7f6578198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.221+0000 7f657cee2640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578102d40 0x7f6578198570 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f656000b4a0 tx=0x7f656000b960 comp rx=0 tx=0).stop 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.222+0000 7f657cee2640 1 -- 172.21.15.62:0/4141281730 shutdown_connections 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.222+0000 7f657cee2640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f65640a3240 0x7f65640a5700 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.222+0000 7f657cee2640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f657810c170 0x7f657819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.222+0000 7f657cee2640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6578103660 0x7f6578198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.222+0000 7f657cee2640 1 --2- 172.21.15.62:0/4141281730 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6578102d40 0x7f6578198570 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:03.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.222+0000 7f657cee2640 1 -- 172.21.15.62:0/4141281730 >> 172.21.15.62:0/4141281730 conn(0x7f65780fc7a0 msgr2=0x7f65780fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.222+0000 7f657cee2640 1 -- 172.21.15.62:0/4141281730 shutdown_connections 2024-09-15T23:31:03.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:03.222+0000 7f657cee2640 1 -- 172.21.15.62:0/4141281730 wait complete. 2024-09-15T23:31:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:03 smithi062 ceph-mon[30949]: pgmap v813: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-09-15T23:31:03.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:03 smithi067 ceph-mon[35665]: pgmap v813: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-09-15T23:31:03.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:03 smithi148 ceph-mon[35610]: pgmap v813: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 1.7 MiB/s wr, 192 op/s 2024-09-15T23:31:03.604 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:31:04.020 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:31:04.020 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:31:04.096 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:04.573 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 885.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2114801158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf41082a0 msgr2=0x7f0bf410ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2114801158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf41082a0 0x7f0bf410ec40 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f0be4009cd0 tx=0x7f0be402a8e0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2114801158 shutdown_connections 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2114801158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf41082a0 0x7f0bf410ec40 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2114801158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf41036e0 0x7f0bf4107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2114801158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf4102d30 0x7f0bf4103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2114801158 >> 172.21.15.62:0/2114801158 conn(0x7f0bf40fc7d0 msgr2=0x7f0bf40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2114801158 shutdown_connections 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2114801158 wait complete. 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 Processor -- start 2024-09-15T23:31:04.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.560+0000 7f0bfabdb640 1 -- start start 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bfabdb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf4102d30 0x7f0bf4198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bfabdb640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf41036e0 0x7f0bf4198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bfabdb640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf41082a0 0x7f0bf419ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bfabdb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf41120a0 con 0x7f0bf4102d30 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bfabdb640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf4111f20 con 0x7f0bf41036e0 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bfabdb640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf4112220 con 0x7f0bf41082a0 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf8977640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf4102d30 0x7f0bf4198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf8977640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf4102d30 0x7f0bf4198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45476/0 (socket says 172.21.15.62:45476) 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf8977640 1 -- 172.21.15.62:0/2501764909 learned_addr learned my addr 172.21.15.62:0/2501764909 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf3fff640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf41036e0 0x7f0bf4198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf9178640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf41082a0 0x7f0bf419ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf8977640 1 -- 172.21.15.62:0/2501764909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf41082a0 msgr2=0x7f0bf419ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:04.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf8977640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf41082a0 0x7f0bf419ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf8977640 1 -- 172.21.15.62:0/2501764909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf41036e0 msgr2=0x7f0bf4198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf8977640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf41036e0 0x7f0bf4198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf8977640 1 -- 172.21.15.62:0/2501764909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf419d520 con 0x7f0bf4102d30 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf8977640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf4102d30 0x7f0bf4198570 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00e9f0 tx=0x7f0bec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.561+0000 7f0bf1ffb640 1 -- 172.21.15.62:0/2501764909 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec00cd80 con 0x7f0bf4102d30 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.562+0000 7f0bf1ffb640 1 -- 172.21.15.62:0/2501764909 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bec005500 con 0x7f0bf4102d30 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.562+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf419d810 con 0x7f0bf4102d30 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.562+0000 7f0bf1ffb640 1 -- 172.21.15.62:0/2501764909 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec010690 con 0x7f0bf4102d30 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.562+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf41a50f0 con 0x7f0bf4102d30 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.563+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf4103110 con 0x7f0bf4102d30 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.564+0000 7f0bf1ffb640 1 -- 172.21.15.62:0/2501764909 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0bec0046c0 con 0x7f0bf4102d30 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.565+0000 7f0bf1ffb640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0be00a3240 0x7f0be00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.565+0000 7f0bf3fff640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0be00a3240 0x7f0be00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:04.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.565+0000 7f0bf1ffb640 1 -- 172.21.15.62:0/2501764909 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0bec0d03b0 con 0x7f0bf4102d30 2024-09-15T23:31:04.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.565+0000 7f0bf3fff640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0be00a3240 0x7f0be00a5700 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40fede0 tx=0x7f0bdc00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:04.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.568+0000 7f0bf1ffb640 1 -- 172.21.15.62:0/2501764909 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f0bec0541b0 con 0x7f0bf4102d30 2024-09-15T23:31:04.660 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:04.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.794+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0bf41998d0 con 0x7f0bf4102d30 2024-09-15T23:31:04.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.795+0000 7f0bf1ffb640 1 -- 172.21.15.62:0/2501764909 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7f0bec09ccf0 con 0x7f0bf4102d30 2024-09-15T23:31:04.796 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:04.796 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:31:04.797 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:31:04.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0be00a3240 msgr2=0x7f0be00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0be00a3240 0x7f0be00a5700 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40fede0 tx=0x7f0bdc00b390 comp rx=0 tx=0).stop 2024-09-15T23:31:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf4102d30 msgr2=0x7f0bf4198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf4102d30 0x7f0bf4198570 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00e9f0 tx=0x7f0bec00edf0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 shutdown_connections 2024-09-15T23:31:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0be00a3240 0x7f0be00a5700 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0bf41082a0 0x7f0bf419ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0bf41036e0 0x7f0bf4198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 --2- 172.21.15.62:0/2501764909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bf4102d30 0x7f0bf4198570 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.797+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 >> 172.21.15.62:0/2501764909 conn(0x7f0bf40fc7d0 msgr2=0x7f0bf4105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.798+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 shutdown_connections 2024-09-15T23:31:04.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:04.798+0000 7f0bfabdb640 1 -- 172.21.15.62:0/2501764909 wait complete. 2024-09-15T23:31:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.284+0000 7fbf53227640 1 -- 172.21.15.62:0/2667267919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c101de0 msgr2=0x7fbf4c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.284+0000 7fbf53227640 1 --2- 172.21.15.62:0/2667267919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c101de0 0x7fbf4c1021c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009900 tx=0x7fbf4002a3f0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.284+0000 7fbf53227640 1 -- 172.21.15.62:0/2667267919 shutdown_connections 2024-09-15T23:31:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.284+0000 7fbf53227640 1 --2- 172.21.15.62:0/2667267919 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf4c106f10 0x7fbf4c10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.284+0000 7fbf53227640 1 --2- 172.21.15.62:0/2667267919 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf4c102790 0x7fbf4c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.284+0000 7fbf53227640 1 --2- 172.21.15.62:0/2667267919 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c101de0 0x7fbf4c1021c0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.284+0000 7fbf53227640 1 -- 172.21.15.62:0/2667267919 >> 172.21.15.62:0/2667267919 conn(0x7fbf4c0fd8c0 msgr2=0x7fbf4c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.284+0000 7fbf53227640 1 -- 172.21.15.62:0/2667267919 shutdown_connections 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.284+0000 7fbf53227640 1 -- 172.21.15.62:0/2667267919 wait complete. 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf53227640 1 Processor -- start 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf53227640 1 -- start start 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf53227640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf4c101de0 0x7fbf4c1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf53227640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf4c102790 0x7fbf4c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf53227640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c106f10 0x7fbf4c19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf53227640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c111f80 con 0x7fbf4c106f10 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf53227640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c111e00 con 0x7fbf4c101de0 2024-09-15T23:31:05.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf53227640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c112100 con 0x7fbf4c102790 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf517c4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c106f10 0x7fbf4c19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf517c4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c106f10 0x7fbf4c19cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45486/0 (socket says 172.21.15.62:45486) 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf517c4640 1 -- 172.21.15.62:0/10334229 learned_addr learned my addr 172.21.15.62:0/10334229 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf50fc3640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf4c101de0 0x7fbf4c1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf517c4640 1 -- 172.21.15.62:0/10334229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf4c102790 msgr2=0x7fbf4c1988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf517c4640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf4c102790 0x7fbf4c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf517c4640 1 -- 172.21.15.62:0/10334229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf4c101de0 msgr2=0x7fbf4c1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf517c4640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf4c101de0 0x7fbf4c1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf517c4640 1 -- 172.21.15.62:0/10334229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf4c19d350 con 0x7fbf4c106f10 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.285+0000 7fbf50fc3640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf4c101de0 0x7fbf4c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.286+0000 7fbf517c4640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c106f10 0x7fbf4c19cc70 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009fd0 tx=0x7fbf4400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.286+0000 7fbf49ffb640 1 -- 172.21.15.62:0/10334229 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf44017070 con 0x7fbf4c106f10 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.286+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf4c19d640 con 0x7fbf4c106f10 2024-09-15T23:31:05.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.286+0000 7fbf49ffb640 1 -- 172.21.15.62:0/10334229 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf44003e50 con 0x7fbf4c106f10 2024-09-15T23:31:05.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.286+0000 7fbf49ffb640 1 -- 172.21.15.62:0/10334229 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf440106c0 con 0x7fbf4c106f10 2024-09-15T23:31:05.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.286+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf4c1a4f20 con 0x7fbf4c106f10 2024-09-15T23:31:05.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.287+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf4c1021c0 con 0x7fbf4c106f10 2024-09-15T23:31:05.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.288+0000 7fbf49ffb640 1 -- 172.21.15.62:0/10334229 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fbf44003970 con 0x7fbf4c106f10 2024-09-15T23:31:05.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.289+0000 7fbf49ffb640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbf3c0a3240 0x7fbf3c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:05.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.289+0000 7fbf50fc3640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbf3c0a3240 0x7fbf3c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:05.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.289+0000 7fbf49ffb640 1 -- 172.21.15.62:0/10334229 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fbf440cfab0 con 0x7fbf4c106f10 2024-09-15T23:31:05.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.290+0000 7fbf50fc3640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbf3c0a3240 0x7fbf3c0a5700 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbf40005ab0 tx=0x7fbf40000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:05.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.293+0000 7fbf49ffb640 1 -- 172.21.15.62:0/10334229 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fbf4409c450 con 0x7fbf4c106f10 2024-09-15T23:31:05.338 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:05 smithi062 ceph-mon[30949]: pgmap v814: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 937 KiB/s wr, 106 op/s 2024-09-15T23:31:05.338 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:05 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2501764909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:05.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.506+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf4c10a120 con 0x7fbf4c106f10 2024-09-15T23:31:05.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.507+0000 7fbf49ffb640 1 -- 172.21.15.62:0/10334229 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7fbf4409bd50 con 0x7fbf4c106f10 2024-09-15T23:31:05.509 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:05.509 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:31:05.509 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbf3c0a3240 msgr2=0x7fbf3c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbf3c0a3240 0x7fbf3c0a5700 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fbf40005ab0 tx=0x7fbf40000bc0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c106f10 msgr2=0x7fbf4c19cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c106f10 0x7fbf4c19cc70 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009fd0 tx=0x7fbf4400b260 comp rx=0 tx=0).stop 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 shutdown_connections 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fbf3c0a3240 0x7fbf3c0a5700 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbf4c106f10 0x7fbf4c19cc70 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fbf4c102790 0x7fbf4c1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 --2- 172.21.15.62:0/10334229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf4c101de0 0x7fbf4c1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 >> 172.21.15.62:0/10334229 conn(0x7fbf4c0fd8c0 msgr2=0x7fbf4c0feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 shutdown_connections 2024-09-15T23:31:05.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:05.509+0000 7fbf53227640 1 -- 172.21.15.62:0/10334229 wait complete. 2024-09-15T23:31:05.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:05 smithi067 ceph-mon[35665]: pgmap v814: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 937 KiB/s wr, 106 op/s 2024-09-15T23:31:05.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:05 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2501764909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:05.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:05 smithi148 ceph-mon[35610]: pgmap v814: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 937 KiB/s wr, 106 op/s 2024-09-15T23:31:05.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:05 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2501764909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:06.174 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 2bc220ac --release --await' 2024-09-15T23:31:06.175 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 2bc220ac --release --await 2024-09-15T23:31:06.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:06 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/10334229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:06.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:06 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/10334229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:06.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:06 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/10334229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:06.630 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:31:06.770 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:07.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.270+0000 7f27859d4640 1 -- 172.21.15.62:0/1962784532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 msgr2=0x7f27801021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:07.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.270+0000 7f27859d4640 1 --2- 172.21.15.62:0/1962784532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 0x7f27801021c0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f2770009900 tx=0x7f277002a410 comp rx=0 tx=0).stop 2024-09-15T23:31:07.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.270+0000 7f27859d4640 1 -- 172.21.15.62:0/1962784532 shutdown_connections 2024-09-15T23:31:07.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.270+0000 7f27859d4640 1 --2- 172.21.15.62:0/1962784532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2780106f10 0x7f278010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.270+0000 7f27859d4640 1 --2- 172.21.15.62:0/1962784532 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2780102790 0x7f27801067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.270+0000 7f27859d4640 1 --2- 172.21.15.62:0/1962784532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 0x7f27801021c0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.270+0000 7f27859d4640 1 -- 172.21.15.62:0/1962784532 >> 172.21.15.62:0/1962784532 conn(0x7f27800fd8c0 msgr2=0x7f27800ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.270+0000 7f27859d4640 1 -- 172.21.15.62:0/1962784532 shutdown_connections 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.270+0000 7f27859d4640 1 -- 172.21.15.62:0/1962784532 wait complete. 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f27859d4640 1 Processor -- start 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f27859d4640 1 -- start start 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f27859d4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 0x7f2780198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f27859d4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2780102790 0x7f2780198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f27859d4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2780106f10 0x7f278019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f27859d4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2780112010 con 0x7f2780101de0 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f27859d4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2780111e90 con 0x7f2780106f10 2024-09-15T23:31:07.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f27859d4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2780112190 con 0x7f2780102790 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f277effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 0x7f2780198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f277effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 0x7f2780198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45506/0 (socket says 172.21.15.62:45506) 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f277effd640 1 -- 172.21.15.62:0/3032354823 learned_addr learned my addr 172.21.15.62:0/3032354823 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f277f7fe640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2780106f10 0x7f278019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.271+0000 7f277e7fc640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2780102790 0x7f2780198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f277effd640 1 -- 172.21.15.62:0/3032354823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2780102790 msgr2=0x7f2780198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f277effd640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2780102790 0x7f2780198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f277effd640 1 -- 172.21.15.62:0/3032354823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2780106f10 msgr2=0x7f278019cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f277effd640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2780106f10 0x7f278019cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f277effd640 1 -- 172.21.15.62:0/3032354823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278019d630 con 0x7f2780101de0 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f277effd640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 0x7f2780198680 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2770005ab0 tx=0x7f27700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f27849d2640 1 -- 172.21.15.62:0/3032354823 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277003f030 con 0x7f2780101de0 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f27849d2640 1 -- 172.21.15.62:0/3032354823 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2770003ce0 con 0x7f2780101de0 2024-09-15T23:31:07.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f27859d4640 1 -- 172.21.15.62:0/3032354823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f278019d8c0 con 0x7f2780101de0 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f27849d2640 1 -- 172.21.15.62:0/3032354823 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2770035070 con 0x7f2780101de0 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.272+0000 7f27859d4640 1 -- 172.21.15.62:0/3032354823 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27801a5200 con 0x7f2780101de0 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.274+0000 7f27849d2640 1 -- 172.21.15.62:0/3032354823 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2770042030 con 0x7f2780101de0 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f2787dad740 1 Processor -- start 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f2787dad740 1 -- start start 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f2787dad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d060 0x55c19332d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f2787dad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c193337d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f2787dad740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c193338240 0x55c19333a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f2787dad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c193292620 con 0x55c19332d060 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f2787dad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c193239e50 con 0x55c19332d980 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f2787dad740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c19328e5e0 con 0x55c193338240 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d060 0x55c19332d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d060 0x55c19332d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45516/0 (socket says 172.21.15.62:45516) 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277f7fe640 1 -- 172.21.15.62:0/1812677155 learned_addr learned my addr 172.21.15.62:0/1812677155 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277effd640 1 --2- 172.21.15.62:0/1812677155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c193337d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f27849d2640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27680a32c0 0x7f27680a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277e7fc640 1 --2- 172.21.15.62:0/1812677155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c193338240 0x55c19333a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277f7fe640 1 -- 172.21.15.62:0/1812677155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c193338240 msgr2=0x55c19333a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277f7fe640 1 --2- 172.21.15.62:0/1812677155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c193338240 0x55c19333a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277f7fe640 1 -- 172.21.15.62:0/1812677155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 msgr2=0x55c193337d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277f7fe640 1 --2- 172.21.15.62:0/1812677155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c193337d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.275+0000 7f277f7fe640 1 -- 172.21.15.62:0/1812677155 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c19322ad40 con 0x55c19332d060 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.276+0000 7f27849d2640 1 -- 172.21.15.62:0/3032354823 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f27700f0190 con 0x7f2780101de0 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.276+0000 7f277f7fe640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27680a32c0 0x7f27680a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.276+0000 7f277f7fe640 1 --2- 172.21.15.62:0/1812677155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d060 0x55c19332d440 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f277800a5d0 tx=0x7f277800aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d986b4e14f831d4d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:07.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.276+0000 7f2765ffb640 1 -- 172.21.15.62:0/1812677155 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277801a040 con 0x55c19332d060 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.276+0000 7f2765ffb640 1 -- 172.21.15.62:0/1812677155 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2778021ba0 con 0x55c19332d060 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.276+0000 7f2765ffb640 1 -- 172.21.15.62:0/1812677155 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277801f350 con 0x55c19332d060 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.276+0000 7f277f7fe640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27680a32c0 0x7f27680a5780 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2778020000 tx=0x7f2778021000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 -- 172.21.15.62:0/1812677155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d060 msgr2=0x55c19332d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 --2- 172.21.15.62:0/1812677155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d060 0x55c19332d440 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f277800a5d0 tx=0x7f277800aa90 comp rx=0 tx=0).stop 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 -- 172.21.15.62:0/1812677155 shutdown_connections 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 --2- 172.21.15.62:0/1812677155 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c193338240 0x55c19333a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 --2- 172.21.15.62:0/1812677155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c193337d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 --2- 172.21.15.62:0/1812677155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d060 0x55c19332d440 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 -- 172.21.15.62:0/1812677155 >> 172.21.15.62:0/1812677155 conn(0x55c1933295f0 msgr2=0x55c1933299d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 -- 172.21.15.62:0/1812677155 shutdown_connections 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 -- 172.21.15.62:0/1812677155 wait complete. 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 -- start start 2024-09-15T23:31:07.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.277+0000 7f2787dad740 1 Processor -- start 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f2787dad740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 0x55c1933350f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f2787dad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c193335630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f2787dad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c193335b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f2787dad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55c19333ae10 con 0x55c193338240 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f2787dad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55c19321af10 con 0x55c19332d980 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f2787dad740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55c19333af90 con 0x55c19332d060 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f2787dad740 10 client.0 _ll_register_callbacks cb 0x55c19323edb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c193335b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c193335b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45524/0 (socket says 172.21.15.62:45524) 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277f7fe640 1 -- 172.21.15.62:0/2637411739 learned_addr learned my addr 172.21.15.62:0/2637411739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277e7fc640 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c193335630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277effd640 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 0x55c1933350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277f7fe640 1 -- 172.21.15.62:0/2637411739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 msgr2=0x55c1933350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277f7fe640 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 0x55c1933350f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277f7fe640 1 -- 172.21.15.62:0/2637411739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 msgr2=0x55c193335630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277f7fe640 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c193335630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277f7fe640 1 -- 172.21.15.62:0/2637411739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c192fa1580 con 0x55c193338240 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277effd640 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 0x55c1933350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f277f7fe640 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c193335b70 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f27780098f0 tx=0x7f2778009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a51bac24e3294e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.278+0000 7f275effd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f275effd640 1 -- 172.21.15.62:0/2637411739 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277802d070 con 0x55c193338240 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f275effd640 1 -- 172.21.15.62:0/2637411739 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2778020c50 con 0x55c193338240 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f2787dad740 1 -- 172.21.15.62:0/2637411739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2778021ba0 con 0x55c193338240 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f275effd640 1 -- 172.21.15.62:0/2637411739 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2778039050 con 0x55c193338240 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f2787dad740 10 client.18879 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f2787dad740 1 -- 172.21.15.62:0/2637411739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c19306b960 con 0x55c193338240 2024-09-15T23:31:07.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f275effd640 1 -- 172.21.15.62:0/2637411739 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2778021ba0 con 0x55c193338240 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f275effd640 1 -- 172.21.15.62:0/2637411739 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f2778028040 con 0x55c193338240 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f2787dad740 10 client.18879 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f2787dad740 10 client.18879 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f2787dad740 20 client.18879 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f2787dad740 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c193347c70 0x55c19334a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f2787dad740 4 client.18879 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f2787dad740 1 -- 172.21.15.62:0/2637411739 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c19329fb90 con 0x55c193347c70 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.279+0000 7f277effd640 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c193347c70 0x55c19334a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.280+0000 7f277effd640 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c193347c70 0x55c19334a060 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b99b4c1e9edaa98a server_cookie=e1cd899c72d854d8 in_seq=0 out_seq=0 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.280+0000 7f275effd640 10 client.18879 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f275effd640 1 -- 172.21.15.62:0/2637411739 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c19329fb90 con 0x55c193347c70 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f275effd640 10 client.18879 handle_command_reply: tid=0 2024-09-15T23:31:07.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 client.18879 shutdown 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 -- 172.21.15.62:0/2637411739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 msgr2=0x55c193335b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c193335b70 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f27780098f0 tx=0x7f2778009b80 comp rx=0 tx=0).stop 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 -- 172.21.15.62:0/2637411739 shutdown_connections 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c193347c70 0x55c19334a060 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c193335b70 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c193335630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 --2- 172.21.15.62:0/2637411739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 0x55c1933350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 -- 172.21.15.62:0/2637411739 >> 172.21.15.62:0/2637411739 conn(0x55c1933295f0 msgr2=0x55c193328750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f275effd640 0 client.18879 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 -- 172.21.15.62:0/2637411739 shutdown_connections 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 1 -- 172.21.15.62:0/2637411739 wait complete. 2024-09-15T23:31:07.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.281+0000 7f2787dad740 20 client.18879 trim_cache size 0 max 16384 2024-09-15T23:31:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f2787dad740 1 Processor -- start 2024-09-15T23:31:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f2787dad740 1 -- start start 2024-09-15T23:31:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f2787dad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d060 0x55c19332fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f2787dad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d980 0x55c193328640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f2787dad740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c193338240 0x55c19334f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f2787dad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f27801b15c0 con 0x55c19332d980 2024-09-15T23:31:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f2787dad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27801aee60 con 0x55c19332d060 2024-09-15T23:31:07.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f2787dad740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f27801accb0 con 0x55c193338240 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f277e7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d980 0x55c193328640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f277e7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d980 0x55c193328640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45528/0 (socket says 172.21.15.62:45528) 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.298+0000 7f277e7fc640 1 -- 172.21.15.62:0/1066009472 learned_addr learned my addr 172.21.15.62:0/1066009472 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f277effd640 1 --2- 172.21.15.62:0/1066009472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d060 0x55c19332fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f277e7fc640 1 -- 172.21.15.62:0/1066009472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c193338240 msgr2=0x55c19334f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f277e7fc640 1 --2- 172.21.15.62:0/1066009472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c193338240 0x55c19334f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f277e7fc640 1 -- 172.21.15.62:0/1066009472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d060 msgr2=0x55c19332fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f277e7fc640 1 --2- 172.21.15.62:0/1066009472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d060 0x55c19332fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f277e7fc640 1 -- 172.21.15.62:0/1066009472 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c193330cd0 con 0x55c19332d980 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f277e7fc640 1 --2- 172.21.15.62:0/1066009472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d980 0x55c193328640 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f277400b8a0 tx=0x7f277400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c8e1966d83ef39 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f27657fa640 1 -- 172.21.15.62:0/1066009472 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2774013ab0 con 0x55c19332d980 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f27657fa640 1 -- 172.21.15.62:0/1066009472 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2774013c50 con 0x55c19332d980 2024-09-15T23:31:07.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.299+0000 7f27657fa640 1 -- 172.21.15.62:0/1066009472 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2774014730 con 0x55c19332d980 2024-09-15T23:31:07.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 -- 172.21.15.62:0/1066009472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d980 msgr2=0x55c193328640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:07.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 --2- 172.21.15.62:0/1066009472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d980 0x55c193328640 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f277400b8a0 tx=0x7f277400bd60 comp rx=0 tx=0).stop 2024-09-15T23:31:07.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 -- 172.21.15.62:0/1066009472 shutdown_connections 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 --2- 172.21.15.62:0/1066009472 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c193338240 0x55c19334f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 --2- 172.21.15.62:0/1066009472 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c19332d980 0x55c193328640 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 --2- 172.21.15.62:0/1066009472 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d060 0x55c19332fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 -- 172.21.15.62:0/1066009472 >> 172.21.15.62:0/1066009472 conn(0x55c1933295f0 msgr2=0x55c193327f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 -- 172.21.15.62:0/1066009472 shutdown_connections 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 -- 172.21.15.62:0/1066009472 wait complete. 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 -- start start 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.300+0000 7f2787dad740 1 Processor -- start 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f2787dad740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 0x55c19333c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f2787dad740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c19333c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f2787dad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c1933b1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f2787dad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f27801ad320 con 0x55c193338240 2024-09-15T23:31:07.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f2787dad740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27801ae790 con 0x55c19332d980 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f2787dad740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f27801ad9f0 con 0x55c19332d060 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f2787dad740 10 client.0 _ll_register_callbacks cb 0x7f277800a400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c1933b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c1933b1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45536/0 (socket says 172.21.15.62:45536) 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277f7fe640 1 -- 172.21.15.62:0/4241371996 learned_addr learned my addr 172.21.15.62:0/4241371996 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277effd640 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 0x55c19333c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277e7fc640 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c19333c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277f7fe640 1 -- 172.21.15.62:0/4241371996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 msgr2=0x55c19333c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277f7fe640 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 0x55c19333c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277f7fe640 1 -- 172.21.15.62:0/4241371996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 msgr2=0x55c19333c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277f7fe640 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c19333c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277f7fe640 1 -- 172.21.15.62:0/4241371996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c19333d0d0 con 0x55c193338240 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f277f7fe640 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c1933b1880 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f2778004a40 tx=0x7f2778009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c2acd270b80533 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:07.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f275effd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f275effd640 1 -- 172.21.15.62:0/4241371996 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277802d070 con 0x55c193338240 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f275effd640 1 -- 172.21.15.62:0/4241371996 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f27780205b0 con 0x55c193338240 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.301+0000 7f2787dad740 1 -- 172.21.15.62:0/4241371996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2774013c50 con 0x55c193338240 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f275effd640 1 -- 172.21.15.62:0/4241371996 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2778048050 con 0x55c193338240 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f2787dad740 10 client.18885 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f2787dad740 1 -- 172.21.15.62:0/4241371996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c1933b2f10 con 0x55c193338240 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f275effd640 1 -- 172.21.15.62:0/4241371996 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2774013c50 con 0x55c193338240 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f275effd640 1 -- 172.21.15.62:0/4241371996 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f2778028070 con 0x55c193338240 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f2787dad740 10 client.18885 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f2787dad740 10 client.18885 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f2787dad740 20 client.18885 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f2787dad740 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1933b35c0 0x55c1933b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f2787dad740 4 client.18885 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "2bc220ac", "release": true, "await": true}] 2024-09-15T23:31:07.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f2787dad740 1 -- 172.21.15.62:0/4241371996 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "2bc220ac", "release": true, "await": true}) -- 0x55c19329a0a0 con 0x55c1933b35c0 2024-09-15T23:31:07.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.302+0000 7f277effd640 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1933b35c0 0x55c1933b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:07.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.303+0000 7f277effd640 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1933b35c0 0x55c1933b59b0 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59f328596df3282 server_cookie=9cc2d7e104ccf661 in_seq=0 out_seq=0 2024-09-15T23:31:07.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.303+0000 7f275effd640 10 client.18885 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:07.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.325+0000 7f275effd640 1 -- 172.21.15.62:0/4241371996 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55c19329a0a0 con 0x55c1933b35c0 2024-09-15T23:31:07.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.325+0000 7f275effd640 10 client.18885 handle_command_reply: tid=0 2024-09-15T23:31:07.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.325+0000 7f2787dad740 1 client.18885 shutdown 2024-09-15T23:31:07.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.326+0000 7f2787dad740 1 -- 172.21.15.62:0/4241371996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 msgr2=0x55c1933b1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:07.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.326+0000 7f2787dad740 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c1933b1880 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f2778004a40 tx=0x7f2778009650 comp rx=0 tx=0).stop 2024-09-15T23:31:07.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.328+0000 7f2787dad740 1 -- 172.21.15.62:0/4241371996 shutdown_connections 2024-09-15T23:31:07.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.328+0000 7f2787dad740 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55c1933b35c0 0x55c1933b59b0 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.329+0000 7f2787dad740 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c193338240 0x55c1933b1880 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.329+0000 7f2787dad740 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c19332d980 0x55c19333c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.329+0000 7f2787dad740 1 --2- 172.21.15.62:0/4241371996 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55c19332d060 0x55c19333c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.329+0000 7f275effd640 0 client.18885 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:07.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.329+0000 7f2787dad740 1 -- 172.21.15.62:0/4241371996 >> 172.21.15.62:0/4241371996 conn(0x55c1933295f0 msgr2=0x55c193327f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:07.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.336+0000 7f2787dad740 1 -- 172.21.15.62:0/4241371996 shutdown_connections 2024-09-15T23:31:07.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.336+0000 7f2787dad740 1 -- 172.21.15.62:0/4241371996 wait complete. 2024-09-15T23:31:07.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.340+0000 7f2787dad740 20 client.18885 trim_cache size 0 max 16384 2024-09-15T23:31:07.343 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:31:07.343 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 36, 2024-09-15T23:31:07.343 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:31:07.343 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 56, 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "2bc220ac": { 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 56, 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 137.7, 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:31:07.344 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:31:07.345 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:31:07.345 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:31:07.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:07.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:07.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:07.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:07.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:07.345 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:31:07.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.348+0000 7f27859d4640 1 -- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27680a32c0 msgr2=0x7f27680a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:07.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.348+0000 7f27859d4640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27680a32c0 0x7f27680a5780 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f2778020000 tx=0x7f2778021000 comp rx=0 tx=0).stop 2024-09-15T23:31:07.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.348+0000 7f27859d4640 1 -- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 msgr2=0x7f2780198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:07.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.348+0000 7f27859d4640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 0x7f2780198680 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f2770005ab0 tx=0x7f27700042f0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.349+0000 7f27859d4640 1 -- 172.21.15.62:0/3032354823 shutdown_connections 2024-09-15T23:31:07.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.349+0000 7f27859d4640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27680a32c0 0x7f27680a5780 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.349+0000 7f27859d4640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2780106f10 0x7f278019cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.349+0000 7f27859d4640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2780102790 0x7f2780198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.349+0000 7f27859d4640 1 --2- 172.21.15.62:0/3032354823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2780101de0 0x7f2780198680 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:07.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.349+0000 7f27859d4640 1 -- 172.21.15.62:0/3032354823 >> 172.21.15.62:0/3032354823 conn(0x7f27800fd8c0 msgr2=0x7f27800fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:07.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.351+0000 7f27859d4640 1 -- 172.21.15.62:0/3032354823 shutdown_connections 2024-09-15T23:31:07.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:07.351+0000 7f27859d4640 1 -- 172.21.15.62:0/3032354823 wait complete. 2024-09-15T23:31:07.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:07 smithi062 ceph-mon[30949]: pgmap v815: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 220 KiB/s wr, 35 op/s 2024-09-15T23:31:07.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:07 smithi067 ceph-mon[35665]: pgmap v815: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 220 KiB/s wr, 35 op/s 2024-09-15T23:31:07.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:07 smithi148 ceph-mon[35610]: pgmap v815: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 220 KiB/s wr, 35 op/s 2024-09-15T23:31:07.659 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:08.151 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '2bc220ac', total seconds elapsed: 22.2 2024-09-15T23:31:08.151 DEBUG:tasks.quiescer.fs.[cephfs]:Going to quiesce for duration: 17.7 2024-09-15T23:31:08.151 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:31:08.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.846+0000 7fd5f9faa640 1 -- 172.21.15.62:0/4236750170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5f4102570 msgr2=0x7fd5f41029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.847+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/4236750170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5f4102570 0x7fd5f41029b0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc009900 tx=0x7fd5dc02a480 comp rx=0 tx=0).stop 2024-09-15T23:31:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.847+0000 7fd5f9faa640 1 -- 172.21.15.62:0/4236750170 shutdown_connections 2024-09-15T23:31:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.847+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/4236750170 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5f4102ef0 0x7fd5f410da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.847+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/4236750170 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5f4102570 0x7fd5f41029b0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.847+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/4236750170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5f4100e20 0x7fd5f4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.847+0000 7fd5f9faa640 1 -- 172.21.15.62:0/4236750170 >> 172.21.15.62:0/4236750170 conn(0x7fd5f40fc7d0 msgr2=0x7fd5f40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.847+0000 7fd5f9faa640 1 -- 172.21.15.62:0/4236750170 shutdown_connections 2024-09-15T23:31:08.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.847+0000 7fd5f9faa640 1 -- 172.21.15.62:0/4236750170 wait complete. 2024-09-15T23:31:08.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.849+0000 7fd5f9faa640 1 Processor -- start 2024-09-15T23:31:08.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.849+0000 7fd5f9faa640 1 -- start start 2024-09-15T23:31:08.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.849+0000 7fd5f9faa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5f4100e20 0x7fd5f41973e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.849+0000 7fd5f9faa640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5f4102570 0x7fd5f4197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.849+0000 7fd5f9faa640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5f4102ef0 0x7fd5f419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.849+0000 7fd5f9faa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f4110f80 con 0x7fd5f4100e20 2024-09-15T23:31:08.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.849+0000 7fd5f9faa640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f4110e00 con 0x7fd5f4102570 2024-09-15T23:31:08.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.849+0000 7fd5f9faa640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f4111100 con 0x7fd5f4102ef0 2024-09-15T23:31:08.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5f4102ef0 0x7fd5f419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5ebfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5f4102570 0x7fd5f4197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5f4102ef0 0x7fd5f419bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:50934/0 (socket says 172.21.15.62:50934) 2024-09-15T23:31:08.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5f3fff640 1 -- 172.21.15.62:0/2033812284 learned_addr learned my addr 172.21.15.62:0/2033812284 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:08.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5f37fe640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5f4100e20 0x7fd5f41973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5f3fff640 1 -- 172.21.15.62:0/2033812284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5f4102570 msgr2=0x7fd5f4197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5f4102570 0x7fd5f4197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5f3fff640 1 -- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5f4100e20 msgr2=0x7fd5f41973e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5f4100e20 0x7fd5f41973e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.850+0000 7fd5f3fff640 1 -- 172.21.15.62:0/2033812284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f419c390 con 0x7fd5f4102ef0 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.851+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5f4102ef0 0x7fd5f419bcb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00eaa0 tx=0x7fd5ec00ef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.851+0000 7fd5f17fa640 1 -- 172.21.15.62:0/2033812284 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec00cdb0 con 0x7fd5f4102ef0 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.851+0000 7fd5f17fa640 1 -- 172.21.15.62:0/2033812284 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec005020 con 0x7fd5f4102ef0 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.851+0000 7fd5f17fa640 1 -- 172.21.15.62:0/2033812284 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0116d0 con 0x7fd5f4102ef0 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.851+0000 7fd5f9faa640 1 -- 172.21.15.62:0/2033812284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f419c680 con 0x7fd5f4102ef0 2024-09-15T23:31:08.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.851+0000 7fd5f9faa640 1 -- 172.21.15.62:0/2033812284 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f4069f70 con 0x7fd5f4102ef0 2024-09-15T23:31:08.855 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.854+0000 7fd5f17fa640 1 -- 172.21.15.62:0/2033812284 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd5ec005540 con 0x7fd5f4102ef0 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.854+0000 7fd5fc383740 1 Processor -- start 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5fc383740 1 -- start start 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5f17fa640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd5e00a3390 0x7fd5e00a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 0x555d28ebe8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28ebf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28ecbf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5f17fa640 1 -- 172.21.15.62:0/2033812284 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd5ec0cfdf0 con 0x7fd5f4102ef0 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5fc383740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555d28e1c2f0 con 0x555d28ec9b80 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5fc383740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555d28dc3cf0 con 0x555d28ebede0 2024-09-15T23:31:08.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5fc383740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555d28e182b0 con 0x555d28ec0350 2024-09-15T23:31:08.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28ecbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28ecbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45554/0 (socket says 172.21.15.62:45554) 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5f3fff640 1 -- 172.21.15.62:0/367789709 learned_addr learned my addr 172.21.15.62:0/367789709 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5ebfff640 1 --2- 172.21.15.62:0/367789709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28ebf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5ebfff640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd5e00a3390 0x7fd5e00a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.855+0000 7fd5f37fe640 1 --2- 172.21.15.62:0/367789709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 0x555d28ebe8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.856+0000 7fd5f3fff640 1 -- 172.21.15.62:0/367789709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 msgr2=0x555d28ebe8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.856+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/367789709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 0x555d28ebe8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.856+0000 7fd5f3fff640 1 -- 172.21.15.62:0/367789709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 msgr2=0x555d28ebf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.856+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/367789709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28ebf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.856+0000 7fd5f3fff640 1 -- 172.21.15.62:0/367789709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d28da83a0 con 0x555d28ec9b80 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.856+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/367789709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28ecbf70 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0984f0 tx=0x7fd5ec09c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50bdc282f1649c49 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.856+0000 7fd5ebfff640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd5e00a3390 0x7fd5e00a5850 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc009230 tx=0x7fd5dc0058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.856+0000 7fd5e9ffb640 1 -- 172.21.15.62:0/367789709 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec01f040 con 0x555d28ec9b80 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.857+0000 7fd5e9ffb640 1 -- 172.21.15.62:0/367789709 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec09a390 con 0x555d28ec9b80 2024-09-15T23:31:08.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.857+0000 7fd5e9ffb640 1 -- 172.21.15.62:0/367789709 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec01f040 con 0x555d28ec9b80 2024-09-15T23:31:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.857+0000 7fd5fc383740 1 -- 172.21.15.62:0/367789709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 msgr2=0x555d28ecbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.857+0000 7fd5fc383740 1 --2- 172.21.15.62:0/367789709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28ecbf70 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0984f0 tx=0x7fd5ec09c8e0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.857+0000 7fd5fc383740 1 -- 172.21.15.62:0/367789709 shutdown_connections 2024-09-15T23:31:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.857+0000 7fd5fc383740 1 --2- 172.21.15.62:0/367789709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28ecbf70 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.857+0000 7fd5fc383740 1 --2- 172.21.15.62:0/367789709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28ebf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.857+0000 7fd5fc383740 1 --2- 172.21.15.62:0/367789709 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 0x555d28ebe8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.857+0000 7fd5fc383740 1 -- 172.21.15.62:0/367789709 >> 172.21.15.62:0/367789709 conn(0x555d28e7f310 msgr2=0x555d28ebb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.858+0000 7fd5fc383740 1 -- 172.21.15.62:0/367789709 shutdown_connections 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.858+0000 7fd5fc383740 1 -- 172.21.15.62:0/367789709 wait complete. 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.858+0000 7fd5fc383740 1 -- start start 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.858+0000 7fd5fc383740 1 Processor -- start 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ebede0 0x555d28ec9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28ec68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ec9b80 0x555d28ec7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5fc383740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x555d28ecc820 con 0x555d28ec0350 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5fc383740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x555d28dd7060 con 0x555d28ec9b80 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5fc383740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x555d28ecc9a0 con 0x555d28ebede0 2024-09-15T23:31:08.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5fc383740 10 client.0 _ll_register_callbacks cb 0x555d28b2a8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:08.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5ebfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28ec68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5ebfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28ec68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45560/0 (socket says 172.21.15.62:45560) 2024-09-15T23:31:08.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5ebfff640 1 -- 172.21.15.62:0/1698521915 learned_addr learned my addr 172.21.15.62:0/1698521915 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:08.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ec9b80 0x555d28ec7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.859+0000 7fd5f37fe640 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ebede0 0x555d28ec9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.860+0000 7fd5f3fff640 1 -- 172.21.15.62:0/1698521915 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ebede0 msgr2=0x555d28ec9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.860+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ebede0 0x555d28ec9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.860+0000 7fd5f3fff640 1 -- 172.21.15.62:0/1698521915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 msgr2=0x555d28ec68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.860+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28ec68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.860+0000 7fd5f3fff640 1 -- 172.21.15.62:0/1698521915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d28b8d3c0 con 0x555d28ec9b80 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.860+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ec9b80 0x555d28ec7080 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0cd5c0 tx=0x7fd5ec0a05c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bc976af5ff8dddb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.860+0000 7fd5d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.860+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/1698521915 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0b4070 con 0x555d28ec9b80 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.860+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/1698521915 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd5ec0a4870 con 0x555d28ec9b80 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.861+0000 7fd5fc383740 1 -- 172.21.15.62:0/1698521915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5ec09a390 con 0x555d28ec9b80 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.861+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/1698521915 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd5ec01f040 con 0x555d28ec9b80 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.861+0000 7fd5fc383740 10 client.28385 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.861+0000 7fd5fc383740 1 -- 172.21.15.62:0/1698521915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555d28ed8390 con 0x555d28ec9b80 2024-09-15T23:31:08.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.862+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/1698521915 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec09a1b0 con 0x555d28ec9b80 2024-09-15T23:31:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.862+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/1698521915 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fd5ec00f040 con 0x555d28ec9b80 2024-09-15T23:31:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.862+0000 7fd5fc383740 10 client.28385 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.862+0000 7fd5fc383740 10 client.28385 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:31:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.862+0000 7fd5fc383740 20 client.28385 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.862+0000 7fd5fc383740 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x555d28ed96e0 0x555d28edbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.862+0000 7fd5fc383740 4 client.28385 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.863+0000 7fd5fc383740 1 -- 172.21.15.62:0/1698521915 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd5ec09a390 con 0x555d28ed96e0 2024-09-15T23:31:08.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.864+0000 7fd5f37fe640 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x555d28ed96e0 0x555d28edbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.864+0000 7fd5d2ffd640 10 client.28385 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:31:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.864+0000 7fd5f37fe640 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x555d28ed96e0 0x555d28edbad0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8deaa62818df2b9 server_cookie=1ae8e2b0735235c6 in_seq=0 out_seq=0 2024-09-15T23:31:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.865+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/1698521915 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fd5ec09a390 con 0x555d28ed96e0 2024-09-15T23:31:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.865+0000 7fd5d2ffd640 10 client.28385 handle_command_reply: tid=0 2024-09-15T23:31:08.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.865+0000 7fd5fc383740 1 client.28385 shutdown 2024-09-15T23:31:08.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 -- 172.21.15.62:0/1698521915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ec9b80 msgr2=0x555d28ec7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ec9b80 0x555d28ec7080 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec0cd5c0 tx=0x7fd5ec0a05c0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 -- 172.21.15.62:0/1698521915 shutdown_connections 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x555d28ed96e0 0x555d28edbad0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ec9b80 0x555d28ec7080 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5d2ffd640 0 client.28385 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28ec68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 --2- 172.21.15.62:0/1698521915 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ebede0 0x555d28ec9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 -- 172.21.15.62:0/1698521915 >> 172.21.15.62:0/1698521915 conn(0x555d28e7f310 msgr2=0x555d28ec11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 -- 172.21.15.62:0/1698521915 shutdown_connections 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 1 -- 172.21.15.62:0/1698521915 wait complete. 2024-09-15T23:31:08.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.866+0000 7fd5fc383740 20 client.28385 trim_cache size 0 max 16384 2024-09-15T23:31:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.889+0000 7fd5fc383740 1 Processor -- start 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.889+0000 7fd5fc383740 1 -- start start 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.890+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28ebb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.890+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 0x555d28ec58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.890+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28edeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.890+0000 7fd5ebfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 0x555d28ec58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.890+0000 7fd5ebfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 0x555d28ec58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:50972/0 (socket says 172.21.15.62:50972) 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.890+0000 7fd5ebfff640 1 -- 172.21.15.62:0/703707195 learned_addr learned my addr 172.21.15.62:0/703707195 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.890+0000 7fd5fc383740 1 -- 172.21.15.62:0/703707195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41af3f0 con 0x555d28ec9b80 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.890+0000 7fd5fc383740 1 -- 172.21.15.62:0/703707195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41acc90 con 0x555d28ebede0 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.890+0000 7fd5fc383740 1 -- 172.21.15.62:0/703707195 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41aaae0 con 0x555d28ec0350 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.891+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/703707195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28edeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.891+0000 7fd5f3fff640 1 -- 172.21.15.62:0/703707195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 msgr2=0x555d28ec58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.891+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/703707195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 0x555d28ec58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.891+0000 7fd5f3fff640 1 -- 172.21.15.62:0/703707195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 msgr2=0x555d28ebb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.891+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/703707195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28ebb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.891+0000 7fd5f3fff640 1 -- 172.21.15.62:0/703707195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d28ec99b0 con 0x555d28ec9b80 2024-09-15T23:31:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.891+0000 7fd5f3fff640 1 --2- 172.21.15.62:0/703707195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28edeaf0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec098a10 tx=0x7fd5ec09c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c688ce70e5329a27 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.896+0000 7fd5e97fa640 1 -- 172.21.15.62:0/703707195 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0a8070 con 0x555d28ec9b80 2024-09-15T23:31:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.896+0000 7fd5e97fa640 1 -- 172.21.15.62:0/703707195 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec09a390 con 0x555d28ec9b80 2024-09-15T23:31:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.896+0000 7fd5e97fa640 1 -- 172.21.15.62:0/703707195 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0b4030 con 0x555d28ec9b80 2024-09-15T23:31:08.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.898+0000 7fd5fc383740 1 -- 172.21.15.62:0/703707195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 msgr2=0x555d28edeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:08.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.898+0000 7fd5fc383740 1 --2- 172.21.15.62:0/703707195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28edeaf0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec098a10 tx=0x7fd5ec09c210 comp rx=0 tx=0).stop 2024-09-15T23:31:08.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.898+0000 7fd5fc383740 1 -- 172.21.15.62:0/703707195 shutdown_connections 2024-09-15T23:31:08.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.898+0000 7fd5fc383740 1 --2- 172.21.15.62:0/703707195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec9b80 0x555d28edeaf0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.898+0000 7fd5fc383740 1 --2- 172.21.15.62:0/703707195 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec0350 0x555d28ec58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.898+0000 7fd5fc383740 1 --2- 172.21.15.62:0/703707195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28ebb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.898+0000 7fd5fc383740 1 -- 172.21.15.62:0/703707195 >> 172.21.15.62:0/703707195 conn(0x555d28e7f310 msgr2=0x555d28ec51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:08.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.901+0000 7fd5fc383740 1 -- 172.21.15.62:0/703707195 shutdown_connections 2024-09-15T23:31:08.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.901+0000 7fd5fc383740 1 -- 172.21.15.62:0/703707195 wait complete. 2024-09-15T23:31:08.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.901+0000 7fd5fc383740 1 -- start start 2024-09-15T23:31:08.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.902+0000 7fd5fc383740 1 Processor -- start 2024-09-15T23:31:08.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.902+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28edef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.903+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28edc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.903+0000 7fd5fc383740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec9b80 0x555d28edc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.903+0000 7fd5fc383740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41ab150 con 0x555d28ec0350 2024-09-15T23:31:08.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.903+0000 7fd5fc383740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41ac5c0 con 0x555d28ebede0 2024-09-15T23:31:08.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.903+0000 7fd5fc383740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f41ab820 con 0x555d28ec9b80 2024-09-15T23:31:08.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.903+0000 7fd5fc383740 10 client.0 _ll_register_callbacks cb 0x7fd5dc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:08.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.908+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec9b80 0x555d28edc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.908+0000 7fd5f37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28edef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.908+0000 7fd5f3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec9b80 0x555d28edc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:50976/0 (socket says 172.21.15.62:50976) 2024-09-15T23:31:08.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.908+0000 7fd5f3fff640 1 -- 172.21.15.62:0/2213865518 learned_addr learned my addr 172.21.15.62:0/2213865518 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:08.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.909+0000 7fd5ebfff640 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28edc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.916+0000 7fd5ebfff640 1 -- 172.21.15.62:0/2213865518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec9b80 msgr2=0x555d28edc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:08.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.916+0000 7fd5ebfff640 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec9b80 0x555d28edc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.916+0000 7fd5ebfff640 1 -- 172.21.15.62:0/2213865518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 msgr2=0x555d28edef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:08.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.916+0000 7fd5ebfff640 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28edef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.916+0000 7fd5ebfff640 1 -- 172.21.15.62:0/2213865518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d28edcd40 con 0x555d28ec0350 2024-09-15T23:31:08.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.923+0000 7fd5ebfff640 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28edc2c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc006010 tx=0x7fd5dc02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf0f15d17289543a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:08.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.923+0000 7fd5d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:08.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.927+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/2213865518 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc035040 con 0x555d28ec0350 2024-09-15T23:31:08.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.927+0000 7fd5fc383740 1 -- 172.21.15.62:0/2213865518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5ec09a390 con 0x555d28ec0350 2024-09-15T23:31:08.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.927+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/2213865518 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd5dc005020 con 0x555d28ec0350 2024-09-15T23:31:08.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.928+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/2213865518 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd5dc005220 con 0x555d28ec0350 2024-09-15T23:31:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.929+0000 7fd5fc383740 10 client.18903 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:08.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.929+0000 7fd5fc383740 1 -- 172.21.15.62:0/2213865518 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555d28f43d30 con 0x555d28ec0350 2024-09-15T23:31:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.929+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/2213865518 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5dc074070 con 0x555d28ec0350 2024-09-15T23:31:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.930+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/2213865518 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fd5dc06d040 con 0x555d28ec0350 2024-09-15T23:31:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.930+0000 7fd5fc383740 10 client.18903 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.930+0000 7fd5fc383740 10 client.18903 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:31:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.930+0000 7fd5fc383740 20 client.18903 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.930+0000 7fd5fc383740 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x555d28ed96e0 0x555d28f46500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.930+0000 7fd5fc383740 4 client.18903 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:08.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.930+0000 7fd5fc383740 1 -- 172.21.15.62:0/2213865518 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd5ec09a390 con 0x555d28ed96e0 2024-09-15T23:31:08.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.932+0000 7fd5f37fe640 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x555d28ed96e0 0x555d28f46500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:08.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.939+0000 7fd5d2ffd640 10 client.18903 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:31:08.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.939+0000 7fd5f37fe640 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x555d28ed96e0 0x555d28f46500 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f509a31cc36976a server_cookie=c8935b51e256b9ef in_seq=0 out_seq=0 2024-09-15T23:31:08.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.940+0000 7fd5d2ffd640 1 -- 172.21.15.62:0/2213865518 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd5ec09a390 con 0x555d28ed96e0 2024-09-15T23:31:08.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.940+0000 7fd5d2ffd640 10 client.18903 handle_command_reply: tid=0 2024-09-15T23:31:08.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.940+0000 7fd5fc383740 1 client.18903 shutdown 2024-09-15T23:31:08.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.941+0000 7fd5fc383740 1 -- 172.21.15.62:0/2213865518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 msgr2=0x555d28edc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:08.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.941+0000 7fd5fc383740 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28edc2c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc006010 tx=0x7fd5dc02a480 comp rx=0 tx=0).stop 2024-09-15T23:31:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.945+0000 7fd5fc383740 1 -- 172.21.15.62:0/2213865518 shutdown_connections 2024-09-15T23:31:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.945+0000 7fd5fc383740 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x555d28ed96e0 0x555d28f46500 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.945+0000 7fd5fc383740 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x555d28ec9b80 0x555d28edc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.945+0000 7fd5d2ffd640 0 client.18903 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:31:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.945+0000 7fd5fc383740 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555d28ec0350 0x555d28edc2c0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.945+0000 7fd5fc383740 1 --2- 172.21.15.62:0/2213865518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555d28ebede0 0x555d28edef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.945+0000 7fd5fc383740 1 -- 172.21.15.62:0/2213865518 >> 172.21.15.62:0/2213865518 conn(0x555d28e7f310 msgr2=0x555d28ed4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.946+0000 7fd5fc383740 1 -- 172.21.15.62:0/2213865518 shutdown_connections 2024-09-15T23:31:08.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.946+0000 7fd5fc383740 1 -- 172.21.15.62:0/2213865518 wait complete. 2024-09-15T23:31:08.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.946+0000 7fd5fc383740 20 client.18903 trim_cache size 0 max 16384 2024-09-15T23:31:08.948 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:31:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 -- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd5e00a3390 msgr2=0x7fd5e00a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd5e00a3390 0x7fd5e00a5850 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc009230 tx=0x7fd5dc0058d0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 -- 172.21.15.62:0/2033812284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5f4102ef0 msgr2=0x7fd5f419bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5f4102ef0 0x7fd5f419bcb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00eaa0 tx=0x7fd5ec00ef60 comp rx=0 tx=0).stop 2024-09-15T23:31:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 -- 172.21.15.62:0/2033812284 shutdown_connections 2024-09-15T23:31:08.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd5e00a3390 0x7fd5e00a5850 secure :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc009230 tx=0x7fd5dc0058d0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd5f4102ef0 0x7fd5f419bcb0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5f4102570 0x7fd5f4197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 --2- 172.21.15.62:0/2033812284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5f4100e20 0x7fd5f41973e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:08.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.953+0000 7fd5f9faa640 1 -- 172.21.15.62:0/2033812284 >> 172.21.15.62:0/2033812284 conn(0x7fd5f40fc7d0 msgr2=0x7fd5f41068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:08.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.954+0000 7fd5f9faa640 1 -- 172.21.15.62:0/2033812284 shutdown_connections 2024-09-15T23:31:08.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:08.954+0000 7fd5f9faa640 1 -- 172.21.15.62:0/2033812284 wait complete. 2024-09-15T23:31:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.129+0000 7f008c2c3640 1 -- 172.21.15.62:0/1163600094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0084072d20 msgr2=0x7f0084073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:09.132 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.129+0000 7f008c2c3640 1 --2- 172.21.15.62:0/1163600094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0084072d20 0x7f0084073180 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f007c009be0 tx=0x7f007c02a710 comp rx=0 tx=0).stop 2024-09-15T23:31:09.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.131+0000 7f008c2c3640 1 -- 172.21.15.62:0/1163600094 shutdown_connections 2024-09-15T23:31:09.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.131+0000 7f008c2c3640 1 --2- 172.21.15.62:0/1163600094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f00840736c0 0x7f0084119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.131+0000 7f008c2c3640 1 --2- 172.21.15.62:0/1163600094 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0084072d20 0x7f0084073180 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.131+0000 7f008c2c3640 1 --2- 172.21.15.62:0/1163600094 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0084109700 0x7f0084109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.133 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.131+0000 7f008c2c3640 1 -- 172.21.15.62:0/1163600094 >> 172.21.15.62:0/1163600094 conn(0x7f008406ece0 msgr2=0x7f0084071120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.141+0000 7f008c2c3640 1 -- 172.21.15.62:0/1163600094 shutdown_connections 2024-09-15T23:31:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.143+0000 7f008c2c3640 1 -- 172.21.15.62:0/1163600094 wait complete. 2024-09-15T23:31:09.145 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.144+0000 7f008c2c3640 1 Processor -- start 2024-09-15T23:31:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.146+0000 7f008c2c3640 1 -- start start 2024-09-15T23:31:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.146+0000 7f008c2c3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0084072d20 0x7f00841158a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.146+0000 7f008c2c3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00840736c0 0x7f0084115de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.146+0000 7f008c2c3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0084109700 0x7f00841b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.147 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.146+0000 7f008c2c3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f008411c5e0 con 0x7f0084109700 2024-09-15T23:31:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.146+0000 7f008c2c3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f008411c460 con 0x7f0084072d20 2024-09-15T23:31:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.146+0000 7f008c2c3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f008411c760 con 0x7f00840736c0 2024-09-15T23:31:09.148 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.147+0000 7f008a860640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0084109700 0x7f00841b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.147+0000 7f008a860640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0084109700 0x7f00841b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45582/0 (socket says 172.21.15.62:45582) 2024-09-15T23:31:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.147+0000 7f008a860640 1 -- 172.21.15.62:0/905873687 learned_addr learned my addr 172.21.15.62:0/905873687 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.147+0000 7f008985e640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00840736c0 0x7f0084115de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.149 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.147+0000 7f008a05f640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0084072d20 0x7f00841158a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.150 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.150+0000 7f008a05f640 1 -- 172.21.15.62:0/905873687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00840736c0 msgr2=0x7f0084115de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.153+0000 7f008a05f640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00840736c0 0x7f0084115de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.153+0000 7f008a05f640 1 -- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0084109700 msgr2=0x7f00841b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.153+0000 7f008a05f640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0084109700 0x7f00841b6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.153+0000 7f008a05f640 1 -- 172.21.15.62:0/905873687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00841164c0 con 0x7f0084072d20 2024-09-15T23:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.154+0000 7f008a860640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0084109700 0x7f00841b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.154+0000 7f008985e640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00840736c0 0x7f0084115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:09.159 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.158+0000 7f008a05f640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0084072d20 0x7f00841158a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f008000e790 tx=0x7f008000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.162+0000 7f007b7fe640 1 -- 172.21.15.62:0/905873687 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008000eea0 con 0x7f0084072d20 2024-09-15T23:31:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.162+0000 7f007b7fe640 1 -- 172.21.15.62:0/905873687 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008000ccc0 con 0x7f0084072d20 2024-09-15T23:31:09.164 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.162+0000 7f007b7fe640 1 -- 172.21.15.62:0/905873687 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080018780 con 0x7f0084072d20 2024-09-15T23:31:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.163+0000 7f008c2c3640 1 -- 172.21.15.62:0/905873687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00841b7350 con 0x7f0084072d20 2024-09-15T23:31:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.163+0000 7f008c2c3640 1 -- 172.21.15.62:0/905873687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00841b7820 con 0x7f0084072d20 2024-09-15T23:31:09.169 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.167+0000 7f008f05d740 1 Processor -- start 2024-09-15T23:31:09.173 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.172+0000 7f008f05d740 1 -- start start 2024-09-15T23:31:09.173 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.172+0000 7f008f05d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da04772da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.173 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.172+0000 7f008f05d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da04777da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.172+0000 7f008f05d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.172+0000 7f008f05d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55da046d62f0 con 0x55da047729a0 2024-09-15T23:31:09.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.172+0000 7f008f05d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55da0467e270 con 0x55da04778370 2024-09-15T23:31:09.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.172+0000 7f008f05d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55da046d22b0 con 0x55da047732e0 2024-09-15T23:31:09.174 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.174+0000 7f008985e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.174+0000 7f008a860640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da04772da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.174+0000 7f008a860640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da04772da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45594/0 (socket says 172.21.15.62:45594) 2024-09-15T23:31:09.175 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.174+0000 7f008a860640 1 -- 172.21.15.62:0/1118864443 learned_addr learned my addr 172.21.15.62:0/1118864443 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:09.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.176+0000 7f007b7fe640 1 -- 172.21.15.62:0/905873687 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0080010430 con 0x7f0084072d20 2024-09-15T23:31:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.178+0000 7f007b7fe640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00700a3530 0x7f00700a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.180+0000 7f008a05f640 1 --2- 172.21.15.62:0/1118864443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da04777da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.182+0000 7f007b7fe640 1 -- 172.21.15.62:0/905873687 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f00800cfcb0 con 0x7f0084072d20 2024-09-15T23:31:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.182+0000 7f008985e640 1 -- 172.21.15.62:0/1118864443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 msgr2=0x55da04777da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.182+0000 7f008985e640 1 --2- 172.21.15.62:0/1118864443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da04777da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.182+0000 7f008985e640 1 -- 172.21.15.62:0/1118864443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 msgr2=0x55da04772da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.182+0000 7f008985e640 1 --2- 172.21.15.62:0/1118864443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da04772da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.182+0000 7f008985e640 1 -- 172.21.15.62:0/1118864443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0467d340 con 0x55da04778370 2024-09-15T23:31:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.182+0000 7f008a860640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00700a3530 0x7f00700a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.183+0000 7f008985e640 1 --2- 172.21.15.62:0/1118864443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477a830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f007c034040 tx=0x7f007c005ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c390a8688e0180cc server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:09.186 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.186+0000 7f008a860640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00700a3530 0x7f00700a59f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0074009fd0 tx=0x7f0074004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:09.187 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.186+0000 7f0078bfe640 1 -- 172.21.15.62:0/1118864443 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c041070 con 0x55da04778370 2024-09-15T23:31:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.187+0000 7f0078bfe640 1 -- 172.21.15.62:0/1118864443 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007c0305a0 con 0x55da04778370 2024-09-15T23:31:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.187+0000 7f0078bfe640 1 -- 172.21.15.62:0/1118864443 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c036040 con 0x55da04778370 2024-09-15T23:31:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.190+0000 7f008f05d740 1 -- 172.21.15.62:0/1118864443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 msgr2=0x55da0477a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.190+0000 7f008f05d740 1 --2- 172.21.15.62:0/1118864443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477a830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f007c034040 tx=0x7f007c005ec0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.192+0000 7f008f05d740 1 -- 172.21.15.62:0/1118864443 shutdown_connections 2024-09-15T23:31:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.192+0000 7f008f05d740 1 --2- 172.21.15.62:0/1118864443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477a830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.192+0000 7f008f05d740 1 --2- 172.21.15.62:0/1118864443 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da04777da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.192+0000 7f008f05d740 1 --2- 172.21.15.62:0/1118864443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da04772da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.192+0000 7f008f05d740 1 -- 172.21.15.62:0/1118864443 >> 172.21.15.62:0/1118864443 conn(0x55da0476e010 msgr2=0x55da0476e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.198+0000 7f008f05d740 1 -- 172.21.15.62:0/1118864443 shutdown_connections 2024-09-15T23:31:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.199+0000 7f008f05d740 1 -- 172.21.15.62:0/1118864443 wait complete. 2024-09-15T23:31:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.199+0000 7f008f05d740 1 -- start start 2024-09-15T23:31:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.200+0000 7f008f05d740 1 Processor -- start 2024-09-15T23:31:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.200+0000 7f008f05d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da0477cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.200+0000 7f008f05d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da0477d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.200+0000 7f008f05d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da047840c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.200+0000 7f008f05d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55da0477b2e0 con 0x55da047729a0 2024-09-15T23:31:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.200+0000 7f008f05d740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55da0465f290 con 0x55da04778370 2024-09-15T23:31:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.200+0000 7f008f05d740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55da0477b460 con 0x55da047732e0 2024-09-15T23:31:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.200+0000 7f008f05d740 10 client.0 _ll_register_callbacks cb 0x55da04690140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.201+0000 7f008a860640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da047840c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.201+0000 7f008a860640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da047840c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40028/0 (socket says 172.21.15.62:40028) 2024-09-15T23:31:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.201+0000 7f008a860640 1 -- 172.21.15.62:0/311671809 learned_addr learned my addr 172.21.15.62:0/311671809 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.202+0000 7f008a860640 1 -- 172.21.15.62:0/311671809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 msgr2=0x55da0477d080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:31:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.202+0000 7f008a860640 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da0477d080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.202+0000 7f008a860640 1 -- 172.21.15.62:0/311671809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 msgr2=0x55da0477cb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:31:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.202+0000 7f008a860640 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da0477cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.202+0000 7f008a860640 1 -- 172.21.15.62:0/311671809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da043e9210 con 0x55da04778370 2024-09-15T23:31:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.207+0000 7f008a860640 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da047840c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f007400ff00 tx=0x7f00740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa587f7d06d48d0e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.207+0000 7f006dffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:31:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.207+0000 7f006dffb640 1 -- 172.21.15.62:0/311671809 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007402e060 con 0x55da04778370 2024-09-15T23:31:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.207+0000 7f008f05d740 1 -- 172.21.15.62:0/311671809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f007c0305a0 con 0x55da04778370 2024-09-15T23:31:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.208+0000 7f006dffb640 1 -- 172.21.15.62:0/311671809 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f007401e600 con 0x55da04778370 2024-09-15T23:31:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.212+0000 7f006dffb640 1 -- 172.21.15.62:0/311671809 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0074035050 con 0x55da04778370 2024-09-15T23:31:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.217+0000 7f008f05d740 10 client.28400 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.218+0000 7f008f05d740 1 -- 172.21.15.62:0/311671809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55da044b5620 con 0x55da04778370 2024-09-15T23:31:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.221+0000 7f006dffb640 1 -- 172.21.15.62:0/311671809 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007c0305a0 con 0x55da04778370 2024-09-15T23:31:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.222+0000 7f006dffb640 1 -- 172.21.15.62:0/311671809 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f00740299f0 con 0x55da04778370 2024-09-15T23:31:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.222+0000 7f008f05d740 10 client.28400 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.222+0000 7f008f05d740 10 client.28400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.222+0000 7f008f05d740 20 client.28400 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.222+0000 7f008f05d740 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55da04788d70 0x55da0478b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.222+0000 7f008f05d740 4 client.28400 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.222+0000 7f008f05d740 1 -- 172.21.15.62:0/311671809 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da046e3a20 con 0x55da04788d70 2024-09-15T23:31:09.227 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:09.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:09 smithi062 ceph-mon[30949]: pgmap v816: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 76 KiB/s wr, 14 op/s 2024-09-15T23:31:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.231+0000 7f008a05f640 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55da04788d70 0x55da0478b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.248 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.246+0000 7f006dffb640 10 client.28400 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:09.248 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.246+0000 7f008a05f640 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55da04788d70 0x55da0478b230 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84b9780c460d2b27 server_cookie=45011ada8310e00a in_seq=0 out_seq=0 2024-09-15T23:31:09.252 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.251+0000 7f006dffb640 1 -- 172.21.15.62:0/311671809 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55da046e3a20 con 0x55da04788d70 2024-09-15T23:31:09.252 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.251+0000 7f006dffb640 10 client.28400 handle_command_reply: tid=0 2024-09-15T23:31:09.252 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.251+0000 7f008f05d740 1 client.28400 shutdown 2024-09-15T23:31:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.256+0000 7f008f05d740 1 -- 172.21.15.62:0/311671809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 msgr2=0x55da047840c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:09.257 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.256+0000 7f008f05d740 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da047840c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f007400ff00 tx=0x7f00740046c0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.258+0000 7f008f05d740 1 -- 172.21.15.62:0/311671809 shutdown_connections 2024-09-15T23:31:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.258+0000 7f008f05d740 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55da04788d70 0x55da0478b230 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.258+0000 7f008f05d740 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da047840c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.258+0000 7f008f05d740 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da0477d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.258+0000 7f008f05d740 1 --2- 172.21.15.62:0/311671809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da0477cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.260 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.258+0000 7f008f05d740 1 -- 172.21.15.62:0/311671809 >> 172.21.15.62:0/311671809 conn(0x55da0476e010 msgr2=0x55da0476f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.260+0000 7f006dffb640 0 client.28400 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.260+0000 7f008f05d740 1 -- 172.21.15.62:0/311671809 shutdown_connections 2024-09-15T23:31:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.260+0000 7f008f05d740 1 -- 172.21.15.62:0/311671809 wait complete. 2024-09-15T23:31:09.261 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.260+0000 7f008f05d740 20 client.28400 trim_cache size 0 max 16384 2024-09-15T23:31:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.308+0000 7f008f05d740 1 Processor -- start 2024-09-15T23:31:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.313+0000 7f008f05d740 1 -- start start 2024-09-15T23:31:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.313+0000 7f008f05d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047729a0 0x55da04772d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.313+0000 7f008f05d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047732e0 0x55da04784080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.314+0000 7f008f05d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.314+0000 7f008f05d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0068012480 con 0x55da047732e0 2024-09-15T23:31:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.314+0000 7f008a860640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.314+0000 7f008a860640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40034/0 (socket says 172.21.15.62:40034) 2024-09-15T23:31:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.314+0000 7f008a860640 1 -- 172.21.15.62:0/2319452739 learned_addr learned my addr 172.21.15.62:0/2319452739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.314+0000 7f008f05d740 1 -- 172.21.15.62:0/2319452739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f006800fd20 con 0x55da04778370 2024-09-15T23:31:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.314+0000 7f008a05f640 1 --2- 172.21.15.62:0/2319452739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047729a0 0x55da04772d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.314+0000 7f008f05d740 1 -- 172.21.15.62:0/2319452739 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f006800db70 con 0x55da047729a0 2024-09-15T23:31:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.315+0000 7f008a860640 1 -- 172.21.15.62:0/2319452739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047729a0 msgr2=0x55da04772d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.315+0000 7f008a860640 1 --2- 172.21.15.62:0/2319452739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047729a0 0x55da04772d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.315+0000 7f008a860640 1 -- 172.21.15.62:0/2319452739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047732e0 msgr2=0x55da04784080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.315+0000 7f008a860640 1 --2- 172.21.15.62:0/2319452739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047732e0 0x55da04784080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.315+0000 7f008a860640 1 -- 172.21.15.62:0/2319452739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0476eca0 con 0x55da04778370 2024-09-15T23:31:09.317 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.317+0000 7f008a860640 1 --2- 172.21.15.62:0/2319452739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477cb40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f007400f8f0 tx=0x7f0074011520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3009f1f9f9b394d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.317+0000 7f006ffff640 1 -- 172.21.15.62:0/2319452739 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007401a040 con 0x55da04778370 2024-09-15T23:31:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.317+0000 7f006ffff640 1 -- 172.21.15.62:0/2319452739 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f007401d610 con 0x55da04778370 2024-09-15T23:31:09.318 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.317+0000 7f006ffff640 1 -- 172.21.15.62:0/2319452739 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00740047a0 con 0x55da04778370 2024-09-15T23:31:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.318+0000 7f008f05d740 1 -- 172.21.15.62:0/2319452739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 msgr2=0x55da0477cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.318+0000 7f008f05d740 1 --2- 172.21.15.62:0/2319452739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477cb40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f007400f8f0 tx=0x7f0074011520 comp rx=0 tx=0).stop 2024-09-15T23:31:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.318+0000 7f008f05d740 1 -- 172.21.15.62:0/2319452739 shutdown_connections 2024-09-15T23:31:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.319+0000 7f008f05d740 1 --2- 172.21.15.62:0/2319452739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04778370 0x55da0477cb40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.319+0000 7f008f05d740 1 --2- 172.21.15.62:0/2319452739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047732e0 0x55da04784080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.319+0000 7f008f05d740 1 --2- 172.21.15.62:0/2319452739 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047729a0 0x55da04772d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.320 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.319+0000 7f008f05d740 1 -- 172.21.15.62:0/2319452739 >> 172.21.15.62:0/2319452739 conn(0x55da0476e010 msgr2=0x55da04774480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.326+0000 7f008f05d740 1 -- 172.21.15.62:0/2319452739 shutdown_connections 2024-09-15T23:31:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.326+0000 7f008f05d740 1 -- 172.21.15.62:0/2319452739 wait complete. 2024-09-15T23:31:09.327 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.326+0000 7f008f05d740 1 -- start start 2024-09-15T23:31:09.328 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.327+0000 7f008f05d740 1 Processor -- start 2024-09-15T23:31:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.327+0000 7f008f05d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da047812b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.328+0000 7f008f05d740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da047817f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.329 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.328+0000 7f008f05d740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04782800 0x55da04782c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.328+0000 7f008985e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da047817f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.328+0000 7f008985e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da047817f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:51012/0 (socket says 172.21.15.62:51012) 2024-09-15T23:31:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.328+0000 7f008985e640 1 -- 172.21.15.62:0/1362826160 learned_addr learned my addr 172.21.15.62:0/1362826160 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.328+0000 7f008a860640 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04782800 0x55da04782c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.328+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f006800e1e0 con 0x55da047729a0 2024-09-15T23:31:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.328+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f006800f650 con 0x55da04782800 2024-09-15T23:31:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.328+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f006800e8b0 con 0x55da047732e0 2024-09-15T23:31:09.330 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.329+0000 7f008f05d740 10 client.0 _ll_register_callbacks cb 0x7f0074005c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.329+0000 7f008985e640 1 -- 172.21.15.62:0/1362826160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04782800 msgr2=0x55da04782c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.329+0000 7f008985e640 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04782800 0x55da04782c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.329+0000 7f008985e640 1 -- 172.21.15.62:0/1362826160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 msgr2=0x55da047812b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.329+0000 7f008985e640 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da047812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.329+0000 7f008985e640 1 -- 172.21.15.62:0/1362826160 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da047831f0 con 0x55da047732e0 2024-09-15T23:31:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.329+0000 7f008985e640 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da047817f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f007c005f00 tx=0x7f007c009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfd3f269885374aa server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:09.331 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.330+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f007401d610 con 0x55da047732e0 2024-09-15T23:31:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.330+0000 7f006dffb640 10 client.28063 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:31:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.330+0000 7f006dffb640 1 -- 172.21.15.62:0/1362826160 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f007c04a020 con 0x55da047732e0 2024-09-15T23:31:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.331+0000 7f006dffb640 1 -- 172.21.15.62:0/1362826160 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f007c031a90 con 0x55da047732e0 2024-09-15T23:31:09.332 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.331+0000 7f006dffb640 1 -- 172.21.15.62:0/1362826160 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f007c036070 con 0x55da047732e0 2024-09-15T23:31:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.331+0000 7f008f05d740 10 client.28063 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.331+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({fsmap=36}) -- 0x55da047843b0 con 0x55da047732e0 2024-09-15T23:31:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.334+0000 7f006dffb640 1 -- 172.21.15.62:0/1362826160 <== mon.1 v2:172.21.15.148:3300/0 4 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f007c031c90 con 0x55da047732e0 2024-09-15T23:31:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.334+0000 7f008f05d740 10 client.28063 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.334+0000 7f008f05d740 10 client.28063 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.334+0000 7f008f05d740 20 client.28063 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.334+0000 7f008f05d740 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55da047f42c0 0x55da047f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.334+0000 7f008f05d740 4 client.28063 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:31:09.335 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.334+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f007401d610 con 0x55da047f42c0 2024-09-15T23:31:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.335+0000 7f008a05f640 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55da047f42c0 0x55da047f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.336+0000 7f008a05f640 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55da047f42c0 0x55da047f66e0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a53f3d8ea0ca6d62 server_cookie=efbb2711111ac5f4 in_seq=0 out_seq=0 2024-09-15T23:31:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.338+0000 7f006dffb640 10 client.28063 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:09.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f006dffb640 1 -- 172.21.15.62:0/1362826160 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f007401d610 con 0x55da047f42c0 2024-09-15T23:31:09.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f006dffb640 10 client.28063 handle_command_reply: tid=0 2024-09-15T23:31:09.422 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f008f05d740 1 client.28063 shutdown 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 msgr2=0x55da047817f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f008f05d740 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da047817f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f007c005f00 tx=0x7f007c009800 comp rx=0 tx=0).stop 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 shutdown_connections 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f008f05d740 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55da047f42c0 0x55da047f66e0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f008f05d740 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55da04782800 0x55da04782c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f008f05d740 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55da047732e0 0x55da047817f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f006dffb640 0 client.28063 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f008f05d740 1 --2- 172.21.15.62:0/1362826160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55da047729a0 0x55da047812b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.420+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 >> 172.21.15.62:0/1362826160 conn(0x55da0476e010 msgr2=0x55da0477cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.421+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 shutdown_connections 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.421+0000 7f008f05d740 1 -- 172.21.15.62:0/1362826160 wait complete. 2024-09-15T23:31:09.423 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.421+0000 7f008f05d740 20 client.28063 trim_cache size 0 max 16384 2024-09-15T23:31:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:31:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:31:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:31:09.424 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 -- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00700a3530 msgr2=0x7f00700a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00700a3530 0x7f00700a59f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0074009fd0 tx=0x7f0074004730 comp rx=0 tx=0).stop 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 -- 172.21.15.62:0/905873687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0084072d20 msgr2=0x7f00841158a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0084072d20 0x7f00841158a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f008000e790 tx=0x7f008000ec50 comp rx=0 tx=0).stop 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 -- 172.21.15.62:0/905873687 shutdown_connections 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f00700a3530 0x7f00700a59f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0084109700 0x7f00841b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f00840736c0 0x7f0084115de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 --2- 172.21.15.62:0/905873687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0084072d20 0x7f00841158a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 -- 172.21.15.62:0/905873687 >> 172.21.15.62:0/905873687 conn(0x7f008406ece0 msgr2=0x7f00840717a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 -- 172.21.15.62:0/905873687 shutdown_connections 2024-09-15T23:31:09.426 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:09.423+0000 7f008c2c3640 1 -- 172.21.15.62:0/905873687 wait complete. 2024-09-15T23:31:09.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:09 smithi067 ceph-mon[35665]: pgmap v816: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 76 KiB/s wr, 14 op/s 2024-09-15T23:31:09.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:09 smithi148 ceph-mon[35610]: pgmap v816: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 76 KiB/s wr, 14 op/s 2024-09-15T23:31:09.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 890.0 s, 355.3 tps, lat 1102.267 ms stddev 4047.576 2024-09-15T23:31:09.818 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:31:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.417+0000 7fe40f416640 1 -- 172.21.15.62:0/3749206399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe408109b10 msgr2=0x7fe408114570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.417+0000 7fe40f416640 1 --2- 172.21.15.62:0/3749206399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe408109b10 0x7fe408114570 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009860 tx=0x7fe3f802a3f0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.429+0000 7fe40f416640 1 -- 172.21.15.62:0/3749206399 shutdown_connections 2024-09-15T23:31:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.429+0000 7fe40f416640 1 --2- 172.21.15.62:0/3749206399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe408114ab0 0x7fe408116ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.429+0000 7fe40f416640 1 --2- 172.21.15.62:0/3749206399 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe408109b10 0x7fe408114570 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.429+0000 7fe40f416640 1 --2- 172.21.15.62:0/3749206399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe4081091f0 0x7fe4081095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.429+0000 7fe40f416640 1 -- 172.21.15.62:0/3749206399 >> 172.21.15.62:0/3749206399 conn(0x7fe40806e140 msgr2=0x7fe40806e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.436+0000 7fe40f416640 1 -- 172.21.15.62:0/3749206399 shutdown_connections 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.440+0000 7fe40f416640 1 -- 172.21.15.62:0/3749206399 wait complete. 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.440+0000 7fe40f416640 1 Processor -- start 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40f416640 1 -- start start 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40f416640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4081091f0 0x7fe4081a5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40f416640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe408109b10 0x7fe40819e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40f416640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe408114ab0 0x7fe40819ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40f416640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe408118610 con 0x7fe408114ab0 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40f416640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe408118490 con 0x7fe408109b10 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40f416640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe408118790 con 0x7fe4081091f0 2024-09-15T23:31:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40d9b3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe408114ab0 0x7fe40819ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40d9b3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe408114ab0 0x7fe40819ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45628/0 (socket says 172.21.15.62:45628) 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.441+0000 7fe40d9b3640 1 -- 172.21.15.62:0/3250432957 learned_addr learned my addr 172.21.15.62:0/3250432957 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.442+0000 7fe40d1b2640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4081091f0 0x7fe4081a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.446+0000 7fe40d9b3640 1 -- 172.21.15.62:0/3250432957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4081091f0 msgr2=0x7fe4081a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.446+0000 7fe40d9b3640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4081091f0 0x7fe4081a5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.446+0000 7fe40d9b3640 1 -- 172.21.15.62:0/3250432957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe408109b10 msgr2=0x7fe40819e8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.446+0000 7fe40d9b3640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe408109b10 0x7fe40819e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.446+0000 7fe40d9b3640 1 -- 172.21.15.62:0/3250432957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40819f3c0 con 0x7fe408114ab0 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.461+0000 7fe40d9b3640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe408114ab0 0x7fe40819ee30 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fe40000daa0 tx=0x7fe40000df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.461+0000 7fe3fe7fc640 1 -- 172.21.15.62:0/3250432957 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400015070 con 0x7fe408114ab0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.461+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40819f6b0 con 0x7fe408114ab0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.461+0000 7fe3fe7fc640 1 -- 172.21.15.62:0/3250432957 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe40000be00 con 0x7fe408114ab0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.461+0000 7fe3fe7fc640 1 -- 172.21.15.62:0/3250432957 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe40000f620 con 0x7fe408114ab0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.461+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4081a36a0 con 0x7fe408114ab0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.464+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe408073a40 con 0x7fe408114ab0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.475+0000 7fe3fe7fc640 1 -- 172.21.15.62:0/3250432957 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe4000052d0 con 0x7fe408114ab0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.476+0000 7fe3fe7fc640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe3f40a32c0 0x7fe3f40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.477+0000 7fe40d1b2640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe3f40a32c0 0x7fe3f40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.477+0000 7fe3fe7fc640 1 -- 172.21.15.62:0/3250432957 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fe4000cfae0 con 0x7fe408114ab0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.477+0000 7fe40d1b2640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe3f40a32c0 0x7fe3f40a5780 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe404005fd0 tx=0x7fe404009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:10.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.479+0000 7fe3fe7fc640 1 -- 172.21.15.62:0/3250432957 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe40009c510 con 0x7fe408114ab0 2024-09-15T23:31:10.612 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:10.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.891+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4081095d0 con 0x7fe408114ab0 2024-09-15T23:31:10.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.892+0000 7fe3fe7fc640 1 -- 172.21.15.62:0/3250432957 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7fe4081095d0 con 0x7fe408114ab0 2024-09-15T23:31:10.893 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:10.893 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:31:10.894 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.894+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe3f40a32c0 msgr2=0x7fe3f40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.894+0000 7fe40f416640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe3f40a32c0 0x7fe3f40a5780 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe404005fd0 tx=0x7fe404009260 comp rx=0 tx=0).stop 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.894+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe408114ab0 msgr2=0x7fe40819ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.894+0000 7fe40f416640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe408114ab0 0x7fe40819ee30 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fe40000daa0 tx=0x7fe40000df60 comp rx=0 tx=0).stop 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.894+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 shutdown_connections 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.894+0000 7fe40f416640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe3f40a32c0 0x7fe3f40a5780 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.895+0000 7fe40f416640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe408114ab0 0x7fe40819ee30 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.895+0000 7fe40f416640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe408109b10 0x7fe40819e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.895+0000 7fe40f416640 1 --2- 172.21.15.62:0/3250432957 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4081091f0 0x7fe4081a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:10.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.895+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 >> 172.21.15.62:0/3250432957 conn(0x7fe40806e140 msgr2=0x7fe408070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:10.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.896+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 shutdown_connections 2024-09-15T23:31:10.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:10.896+0000 7fe40f416640 1 -- 172.21.15.62:0/3250432957 wait complete. 2024-09-15T23:31:11.280 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:11 smithi062 ceph-mon[30949]: pgmap v817: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 388 KiB/s wr, 52 op/s 2024-09-15T23:31:11.280 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:11 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3250432957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.422+0000 7f8fba3d3640 1 -- 172.21.15.62:0/2205484114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8fb406ea30 msgr2=0x7f8fb406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.422+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/2205484114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8fb406ea30 0x7f8fb406ee70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009860 tx=0x7f8fa402a360 comp rx=0 tx=0).stop 2024-09-15T23:31:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.423+0000 7f8fba3d3640 1 -- 172.21.15.62:0/2205484114 shutdown_connections 2024-09-15T23:31:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.423+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/2205484114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fb411b140 0x7f8fb411d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.423+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/2205484114 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8fb406ea30 0x7f8fb406ee70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.423+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/2205484114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb4110190 0x7f8fb406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.423+0000 7f8fba3d3640 1 -- 172.21.15.62:0/2205484114 >> 172.21.15.62:0/2205484114 conn(0x7f8fb4073180 msgr2=0x7f8fb4073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:11.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.427+0000 7f8fba3d3640 1 -- 172.21.15.62:0/2205484114 shutdown_connections 2024-09-15T23:31:11.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.427+0000 7f8fba3d3640 1 -- 172.21.15.62:0/2205484114 wait complete. 2024-09-15T23:31:11.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.428+0000 7f8fba3d3640 1 Processor -- start 2024-09-15T23:31:11.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.433+0000 7f8fba3d3640 1 -- start start 2024-09-15T23:31:11.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.433+0000 7f8fba3d3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb406ea30 0x7f8fb41a7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.433+0000 7f8fba3d3640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8fb4110190 0x7f8fb41a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.433+0000 7f8fba3d3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fb411b140 0x7f8fb41aba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.434+0000 7f8fba3d3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb4120b90 con 0x7f8fb411b140 2024-09-15T23:31:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.434+0000 7f8fba3d3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb4120a10 con 0x7f8fb406ea30 2024-09-15T23:31:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.434+0000 7f8fba3d3640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb4120d10 con 0x7f8fb4110190 2024-09-15T23:31:11.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.434+0000 7f8fb9bd2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fb411b140 0x7f8fb41aba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.434+0000 7f8fb9bd2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fb411b140 0x7f8fb41aba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45642/0 (socket says 172.21.15.62:45642) 2024-09-15T23:31:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.434+0000 7f8fb9bd2640 1 -- 172.21.15.62:0/4244550570 learned_addr learned my addr 172.21.15.62:0/4244550570 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:11.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.435+0000 7f8fb9bd2640 1 -- 172.21.15.62:0/4244550570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8fb4110190 msgr2=0x7f8fb41a7690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T23:31:11.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.435+0000 7f8fb9bd2640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8fb4110190 0x7f8fb41a7690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.435+0000 7f8fb9bd2640 1 -- 172.21.15.62:0/4244550570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb406ea30 msgr2=0x7f8fb41a7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:11.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.435+0000 7f8fb9bd2640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb406ea30 0x7f8fb41a7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.435+0000 7f8fb9bd2640 1 -- 172.21.15.62:0/4244550570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb41ac100 con 0x7f8fb411b140 2024-09-15T23:31:11.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.436+0000 7f8fb9bd2640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fb411b140 0x7f8fb41aba20 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00e920 tx=0x7f8fac00ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.437+0000 7f8faa7fc640 1 -- 172.21.15.62:0/4244550570 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac004d20 con 0x7f8fb411b140 2024-09-15T23:31:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.437+0000 7f8faa7fc640 1 -- 172.21.15.62:0/4244550570 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fac005020 con 0x7f8fb411b140 2024-09-15T23:31:11.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.437+0000 7f8fba3d3640 1 -- 172.21.15.62:0/4244550570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb41ac3f0 con 0x7f8fb411b140 2024-09-15T23:31:11.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.438+0000 7f8fba3d3640 1 -- 172.21.15.62:0/4244550570 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb41b3cd0 con 0x7f8fb411b140 2024-09-15T23:31:11.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.440+0000 7f8faa7fc640 1 -- 172.21.15.62:0/4244550570 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac0116d0 con 0x7f8fb411b140 2024-09-15T23:31:11.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.448+0000 7f8fbda0e740 1 Processor -- start 2024-09-15T23:31:11.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.448+0000 7f8faa7fc640 1 -- 172.21.15.62:0/4244550570 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8fac011930 con 0x7f8fb411b140 2024-09-15T23:31:11.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.449+0000 7f8faa7fc640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8fa00a3390 0x7f8fa00a5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.449+0000 7f8faa7fc640 1 -- 172.21.15.62:0/4244550570 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8fac09b030 con 0x7f8fb411b140 2024-09-15T23:31:11.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.450+0000 7f8fbda0e740 1 -- start start 2024-09-15T23:31:11.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.450+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cb2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.450+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb27b0 0x557f18cb2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.450+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cbd550 0x557f18cbf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.450+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557f18c182f0 con 0x557f18cbd550 2024-09-15T23:31:11.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.450+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557f18bbfcf0 con 0x557f18cb3d20 2024-09-15T23:31:11.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.450+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557f18c142b0 con 0x557f18cb27b0 2024-09-15T23:31:11.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.451+0000 7f8fb9bd2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cbd550 0x557f18cbf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.451+0000 7f8fb9bd2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cbd550 0x557f18cbf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45658/0 (socket says 172.21.15.62:45658) 2024-09-15T23:31:11.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.451+0000 7f8fb9bd2640 1 -- 172.21.15.62:0/1876459485 learned_addr learned my addr 172.21.15.62:0/1876459485 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:11.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.451+0000 7f8fb93d1640 1 --2- 172.21.15.62:0/1876459485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cb2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.452+0000 7f8fb9bd2640 1 -- 172.21.15.62:0/1876459485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb27b0 msgr2=0x557f18cb2bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:31:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.452+0000 7f8fb9bd2640 1 --2- 172.21.15.62:0/1876459485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb27b0 0x557f18cb2bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.452+0000 7f8fb9bd2640 1 -- 172.21.15.62:0/1876459485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 msgr2=0x557f18cb2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.452+0000 7f8fb9bd2640 1 --2- 172.21.15.62:0/1876459485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cb2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.452+0000 7f8fb9bd2640 1 -- 172.21.15.62:0/1876459485 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f18ba43a0 con 0x557f18cbd550 2024-09-15T23:31:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.452+0000 7f8fb9bd2640 1 --2- 172.21.15.62:0/1876459485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cbd550 0x557f18cbf940 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8fac09a210 tx=0x7f8fac011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc518018a4258d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:11.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.452+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8fa00a3390 0x7f8fa00a5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.455+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8fa00a3390 0x7f8fa00a5850 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8fa402a950 tx=0x7f8fa4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:11.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.457+0000 7f8f9f7fe640 1 -- 172.21.15.62:0/1876459485 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac0999c0 con 0x557f18cbd550 2024-09-15T23:31:11.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.457+0000 7f8f9f7fe640 1 -- 172.21.15.62:0/1876459485 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fac0a5c00 con 0x557f18cbd550 2024-09-15T23:31:11.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.457+0000 7f8f9f7fe640 1 -- 172.21.15.62:0/1876459485 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac099b60 con 0x557f18cbd550 2024-09-15T23:31:11.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.458+0000 7f8fbda0e740 1 -- 172.21.15.62:0/1876459485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cbd550 msgr2=0x557f18cbf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:11.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.458+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/1876459485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cbd550 0x557f18cbf940 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8fac09a210 tx=0x7f8fac011000 comp rx=0 tx=0).stop 2024-09-15T23:31:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.464+0000 7f8fbda0e740 1 -- 172.21.15.62:0/1876459485 shutdown_connections 2024-09-15T23:31:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.464+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/1876459485 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cbd550 0x557f18cbf940 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.464+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/1876459485 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb27b0 0x557f18cb2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.464+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/1876459485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cb2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.464+0000 7f8fbda0e740 1 -- 172.21.15.62:0/1876459485 >> 172.21.15.62:0/1876459485 conn(0x557f18c7b310 msgr2=0x557f18caee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:11.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.469+0000 7f8fbda0e740 1 -- 172.21.15.62:0/1876459485 shutdown_connections 2024-09-15T23:31:11.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.470+0000 7f8fbda0e740 1 -- 172.21.15.62:0/1876459485 wait complete. 2024-09-15T23:31:11.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.470+0000 7f8fbda0e740 1 -- start start 2024-09-15T23:31:11.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.471+0000 7f8fbda0e740 1 Processor -- start 2024-09-15T23:31:11.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.476+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cbcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.476+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cba250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.476+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cbd550 0x557f18cba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.476+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557f18cc0100 con 0x557f18cb27b0 2024-09-15T23:31:11.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.476+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557f18bd3060 con 0x557f18cb3d20 2024-09-15T23:31:11.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.476+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557f18cc0280 con 0x557f18cbd550 2024-09-15T23:31:11.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.476+0000 7f8fbda0e740 10 client.0 _ll_register_callbacks cb 0x557f189268c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:11.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.478+0000 7f8fb8bd0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cba250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.478+0000 7f8fb8bd0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cba250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40084/0 (socket says 172.21.15.62:40084) 2024-09-15T23:31:11.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.478+0000 7f8fb8bd0640 1 -- 172.21.15.62:0/2514438379 learned_addr learned my addr 172.21.15.62:0/2514438379 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:11.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.479+0000 7f8fb8bd0640 1 -- 172.21.15.62:0/2514438379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cbd550 msgr2=0x557f18cba8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:31:11.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.479+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cbd550 0x557f18cba8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.479+0000 7f8fb8bd0640 1 -- 172.21.15.62:0/2514438379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 msgr2=0x557f18cbcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:11.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.479+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cbcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.479+0000 7f8fb8bd0640 1 -- 172.21.15.62:0/2514438379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f189893c0 con 0x557f18cb3d20 2024-09-15T23:31:11.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.480+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cba250 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009860 tx=0x7f8fa402a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47b620196d13a407 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:11.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.480+0000 7f8f9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:31:11.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.483+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2514438379 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa404d060 con 0x557f18cb3d20 2024-09-15T23:31:11.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.483+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2514438379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fac0a5c00 con 0x557f18cb3d20 2024-09-15T23:31:11.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.483+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2514438379 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8fa4030720 con 0x557f18cb3d20 2024-09-15T23:31:11.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.483+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2514438379 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8fa4054050 con 0x557f18cb3d20 2024-09-15T23:31:11.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.484+0000 7f8fbda0e740 10 client.28415 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:11.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.484+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2514438379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557f18ccbca0 con 0x557f18cb3d20 2024-09-15T23:31:11.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.484+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2514438379 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fac0a5c00 con 0x557f18cb3d20 2024-09-15T23:31:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.485+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2514438379 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f8fa403a030 con 0x557f18cb3d20 2024-09-15T23:31:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.485+0000 7f8fbda0e740 10 client.28415 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.485+0000 7f8fbda0e740 10 client.28415 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:31:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.485+0000 7f8fbda0e740 20 client.28415 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.485+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x557f18ccd5d0 0x557f18ccfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.485+0000 7f8fbda0e740 4 client.28415 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.485+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2514438379 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f18c25860 con 0x557f18ccd5d0 2024-09-15T23:31:11.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.486+0000 7f8fb93d1640 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x557f18ccd5d0 0x557f18ccfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.492+0000 7f8fb93d1640 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x557f18ccd5d0 0x557f18ccfa50 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8288876b18c5375e server_cookie=deff47d3978fb483 in_seq=0 out_seq=0 2024-09-15T23:31:11.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.492+0000 7f8f9cff9640 10 client.28415 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:31:11.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.493+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2514438379 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557f18c25860 con 0x557f18ccd5d0 2024-09-15T23:31:11.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.493+0000 7f8f9cff9640 10 client.28415 handle_command_reply: tid=0 2024-09-15T23:31:11.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8fbda0e740 1 client.28415 shutdown 2024-09-15T23:31:11.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2514438379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 msgr2=0x557f18cba250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cba250 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8fa4009860 tx=0x7f8fa402a360 comp rx=0 tx=0).stop 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2514438379 shutdown_connections 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x557f18ccd5d0 0x557f18ccfa50 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cbd550 0x557f18cba8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cba250 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2514438379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cbcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2514438379 >> 172.21.15.62:0/2514438379 conn(0x557f18c7b310 msgr2=0x557f18cb4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.494+0000 7f8f9cff9640 0 client.28415 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.495+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2514438379 shutdown_connections 2024-09-15T23:31:11.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.495+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2514438379 wait complete. 2024-09-15T23:31:11.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.496+0000 7f8fbda0e740 20 client.28415 trim_cache size 0 max 16384 2024-09-15T23:31:11.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.520+0000 7f8fbda0e740 1 Processor -- start 2024-09-15T23:31:11.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.520+0000 7f8fbda0e740 1 -- start start 2024-09-15T23:31:11.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cba220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb3d20 0x557f18cc9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cbd550 0x557f18cb5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8f980123a0 con 0x557f18cb27b0 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb93d1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cba220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9800fc40 con 0x557f18cbd550 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9800db40 con 0x557f18cb3d20 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb9bd2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cbd550 0x557f18cb5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb8bd0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb3d20 0x557f18cc9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb9bd2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cbd550 0x557f18cb5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:40100/0 (socket says 172.21.15.62:40100) 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb9bd2640 1 -- 172.21.15.62:0/4182866181 learned_addr learned my addr 172.21.15.62:0/4182866181 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb8bd0640 1 -- 172.21.15.62:0/4182866181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cbd550 msgr2=0x557f18cb5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/4182866181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cbd550 0x557f18cb5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb8bd0640 1 -- 172.21.15.62:0/4182866181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 msgr2=0x557f18cba220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/4182866181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cba220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.521+0000 7f8fb8bd0640 1 -- 172.21.15.62:0/4182866181 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f18cafb40 con 0x557f18cb3d20 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.522+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/4182866181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb3d20 0x557f18cc9430 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fa403d560 tx=0x7f8fa4005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=741d01c6e84d17df server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:11.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.522+0000 7f8f9effd640 1 -- 172.21.15.62:0/4182866181 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa404d070 con 0x557f18cb3d20 2024-09-15T23:31:11.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.522+0000 7f8f9effd640 1 -- 172.21.15.62:0/4182866181 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa4004490 con 0x557f18cb3d20 2024-09-15T23:31:11.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.522+0000 7f8f9effd640 1 -- 172.21.15.62:0/4182866181 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4037040 con 0x557f18cb3d20 2024-09-15T23:31:11.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:11 smithi067 ceph-mon[35665]: pgmap v817: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 388 KiB/s wr, 52 op/s 2024-09-15T23:31:11.525 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:11 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3250432957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:11.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.523+0000 7f8fbda0e740 1 -- 172.21.15.62:0/4182866181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb3d20 msgr2=0x557f18cc9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.523+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/4182866181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb3d20 0x557f18cc9430 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fa403d560 tx=0x7f8fa4005f70 comp rx=0 tx=0).stop 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.523+0000 7f8fbda0e740 1 -- 172.21.15.62:0/4182866181 shutdown_connections 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.523+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/4182866181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cbd550 0x557f18cb5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.523+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/4182866181 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cb3d20 0x557f18cc9430 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.523+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/4182866181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cba220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.523+0000 7f8fbda0e740 1 -- 172.21.15.62:0/4182866181 >> 172.21.15.62:0/4182866181 conn(0x557f18c7b310 msgr2=0x557f18cb9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.523+0000 7f8fbda0e740 1 -- 172.21.15.62:0/4182866181 shutdown_connections 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.523+0000 7f8fbda0e740 1 -- 172.21.15.62:0/4182866181 wait complete. 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fbda0e740 1 -- start start 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fbda0e740 1 Processor -- start 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cc55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cc5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fbda0e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cbd550 0x557f18cb74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fb93d1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cc55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fbda0e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9800e100 con 0x557f18cb27b0 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fb93d1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cc55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:45698/0 (socket says 172.21.15.62:45698) 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fb93d1640 1 -- 172.21.15.62:0/2895603805 learned_addr learned my addr 172.21.15.62:0/2895603805 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cc5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9800f570 con 0x557f18cb3d20 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9800e7d0 con 0x557f18cbd550 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.524+0000 7f8fbda0e740 10 client.0 _ll_register_callbacks cb 0x7f8fa4035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.525+0000 7f8fb9bd2640 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cbd550 0x557f18cb74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.525+0000 7f8fb93d1640 1 -- 172.21.15.62:0/2895603805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cbd550 msgr2=0x557f18cb74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.525+0000 7f8fb93d1640 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cbd550 0x557f18cb74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.525+0000 7f8fb93d1640 1 -- 172.21.15.62:0/2895603805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 msgr2=0x557f18cc5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:11.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.525+0000 7f8fb93d1640 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cc5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.525+0000 7f8fb93d1640 1 -- 172.21.15.62:0/2895603805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f18cb7a10 con 0x557f18cb27b0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.525+0000 7f8fb93d1640 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cc55a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x557f18cd0300 tx=0x7f8fb0009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c027f5c7fe5577c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.525+0000 7f8f9cff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.525+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2895603805 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0005050 con 0x557f18cb27b0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa4004490 con 0x557f18cb27b0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2895603805 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8fb00051f0 con 0x557f18cb27b0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2895603805 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8fb003c050 con 0x557f18cb27b0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8fbda0e740 10 client.18933 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557f18cbe260 con 0x557f18cb27b0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2895603805 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fa4004490 con 0x557f18cb27b0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2895603805 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f8fb000f040 con 0x557f18cb27b0 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8fbda0e740 10 client.18933 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8fbda0e740 10 client.18933 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8fbda0e740 20 client.18933 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:11.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x557f18d38330 0x557f18d3a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8fbda0e740 4 client.18933 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.526+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f18c1fd70 con 0x557f18d38330 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.527+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x557f18d38330 0x557f18d3a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.527+0000 7f8fb8bd0640 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x557f18d38330 0x557f18d3a720 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58548ca04c682919 server_cookie=2f4826ed96c77cf3 in_seq=0 out_seq=0 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.527+0000 7f8f9cff9640 10 client.18933 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.527+0000 7f8f9cff9640 1 -- 172.21.15.62:0/2895603805 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f18c1fd70 con 0x557f18d38330 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.527+0000 7f8f9cff9640 10 client.18933 handle_command_reply: tid=0 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.527+0000 7f8fbda0e740 1 client.18933 shutdown 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 msgr2=0x557f18cc55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cc55a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x557f18cd0300 tx=0x7f8fb0009a50 comp rx=0 tx=0).stop 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 shutdown_connections 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x557f18d38330 0x557f18d3a720 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557f18cbd550 0x557f18cb74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557f18cb3d20 0x557f18cc5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 --2- 172.21.15.62:0/2895603805 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557f18cb27b0 0x557f18cc55a0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 >> 172.21.15.62:0/2895603805 conn(0x557f18c7b310 msgr2=0x557f18cad780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:11.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8f9cff9640 0 client.18933 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:31:11.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 shutdown_connections 2024-09-15T23:31:11.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 1 -- 172.21.15.62:0/2895603805 wait complete. 2024-09-15T23:31:11.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.528+0000 7f8fbda0e740 20 client.18933 trim_cache size 0 max 16384 2024-09-15T23:31:11.534 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:31:11.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 -- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8fa00a3390 msgr2=0x7f8fa00a5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:11.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8fa00a3390 0x7f8fa00a5850 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8fa402a950 tx=0x7f8fa4000bc0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 -- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fb411b140 msgr2=0x7f8fb41aba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:11.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fb411b140 0x7f8fb41aba20 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f8fac00e920 tx=0x7f8fac00ede0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 -- 172.21.15.62:0/4244550570 shutdown_connections 2024-09-15T23:31:11.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8fa00a3390 0x7f8fa00a5850 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fb411b140 0x7f8fb41aba20 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8fb4110190 0x7f8fb41a7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 --2- 172.21.15.62:0/4244550570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb406ea30 0x7f8fb41a7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:11.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 -- 172.21.15.62:0/4244550570 >> 172.21.15.62:0/4244550570 conn(0x7f8fb4073180 msgr2=0x7f8fb4119370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:11.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 -- 172.21.15.62:0/4244550570 shutdown_connections 2024-09-15T23:31:11.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:11.529+0000 7f8fba3d3640 1 -- 172.21.15.62:0/4244550570 wait complete. 2024-09-15T23:31:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:11 smithi148 ceph-mon[35610]: pgmap v817: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 388 KiB/s wr, 52 op/s 2024-09-15T23:31:11.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:11 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3250432957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:12.115 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0' 2024-09-15T23:31:12.115 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db / --timeout 90 --expiration 137.7 --await-for 45.0 2024-09-15T23:31:12.847 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:31:13.103 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:13.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:13 smithi067 ceph-mon[35665]: pgmap v818: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 940 KiB/s wr, 111 op/s 2024-09-15T23:31:13.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:13 smithi062 ceph-mon[30949]: pgmap v818: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 940 KiB/s wr, 111 op/s 2024-09-15T23:31:13.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:13 smithi148 ceph-mon[35610]: pgmap v818: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 940 KiB/s wr, 111 op/s 2024-09-15T23:31:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.726+0000 7f488f59e640 1 -- 172.21.15.62:0/69214400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489006ea30 msgr2=0x7f489006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.726+0000 7f488f59e640 1 --2- 172.21.15.62:0/69214400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489006ea30 0x7f489006ee70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f4878009be0 tx=0x7f487802a710 comp rx=0 tx=0).stop 2024-09-15T23:31:13.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.729+0000 7f488f59e640 1 -- 172.21.15.62:0/69214400 shutdown_connections 2024-09-15T23:31:13.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.729+0000 7f488f59e640 1 --2- 172.21.15.62:0/69214400 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f489011b140 0x7f489011d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.729+0000 7f488f59e640 1 --2- 172.21.15.62:0/69214400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489006ea30 0x7f489006ee70 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.729+0000 7f488f59e640 1 --2- 172.21.15.62:0/69214400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4890110190 0x7f489006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.729+0000 7f488f59e640 1 -- 172.21.15.62:0/69214400 >> 172.21.15.62:0/69214400 conn(0x7f4890073180 msgr2=0x7f4890073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:13.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.740+0000 7f488f59e640 1 -- 172.21.15.62:0/69214400 shutdown_connections 2024-09-15T23:31:13.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.740+0000 7f488f59e640 1 -- 172.21.15.62:0/69214400 wait complete. 2024-09-15T23:31:13.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.741+0000 7f488f59e640 1 Processor -- start 2024-09-15T23:31:13.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.746+0000 7f488f59e640 1 -- start start 2024-09-15T23:31:13.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.746+0000 7f488f59e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f489006ea30 0x7f489011a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.746+0000 7f488f59e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4890110190 0x7f489011a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.746+0000 7f488f59e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489011b140 0x7f4890114510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.746+0000 7f488f59e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4890110ef0 con 0x7f489011b140 2024-09-15T23:31:13.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.746+0000 7f488f59e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4890110d70 con 0x7f4890110190 2024-09-15T23:31:13.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.746+0000 7f488f59e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f4890111070 con 0x7f489006ea30 2024-09-15T23:31:13.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.749+0000 7f488dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4890110190 0x7f489011a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.749+0000 7f488dd9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4890110190 0x7f489011a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:58940/0 (socket says 172.21.15.62:58940) 2024-09-15T23:31:13.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.749+0000 7f488dd9b640 1 -- 172.21.15.62:0/2733609082 learned_addr learned my addr 172.21.15.62:0/2733609082 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:13.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.750+0000 7f488e59c640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f489006ea30 0x7f489011a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.751+0000 7f488e59c640 1 -- 172.21.15.62:0/2733609082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4890110190 msgr2=0x7f489011a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:13.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.751+0000 7f488e59c640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4890110190 0x7f489011a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.751+0000 7f488e59c640 1 -- 172.21.15.62:0/2733609082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489011b140 msgr2=0x7f4890114510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:13.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.751+0000 7f488e59c640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489011b140 0x7f4890114510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.751+0000 7f488e59c640 1 -- 172.21.15.62:0/2733609082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4890114dd0 con 0x7f489006ea30 2024-09-15T23:31:13.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.752+0000 7f488e59c640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f489006ea30 0x7f489011a470 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f488000e790 tx=0x7f488000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:13.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.760+0000 7f48877fe640 1 -- 172.21.15.62:0/2733609082 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488000eea0 con 0x7f489006ea30 2024-09-15T23:31:13.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.760+0000 7f488f59e640 1 -- 172.21.15.62:0/2733609082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48901150c0 con 0x7f489006ea30 2024-09-15T23:31:13.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.760+0000 7f488f59e640 1 -- 172.21.15.62:0/2733609082 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48901bc530 con 0x7f489006ea30 2024-09-15T23:31:13.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.760+0000 7f48877fe640 1 -- 172.21.15.62:0/2733609082 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f488000ccc0 con 0x7f489006ea30 2024-09-15T23:31:13.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.760+0000 7f48877fe640 1 -- 172.21.15.62:0/2733609082 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4880018780 con 0x7f489006ea30 2024-09-15T23:31:13.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.770+0000 7f4896cdc740 1 Processor -- start 2024-09-15T23:31:13.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.774+0000 7f4896cdc740 1 -- start start 2024-09-15T23:31:13.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.774+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a720ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.774+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a721220 0x55fa0a72b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.774+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a72bae0 0x55fa0a72ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.774+0000 7f4896cdc740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fa0a6862a0 con 0x55fa0a720900 2024-09-15T23:31:13.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.774+0000 7f4896cdc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fa0a62db10 con 0x55fa0a72bae0 2024-09-15T23:31:13.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.774+0000 7f4896cdc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55fa0a682260 con 0x55fa0a721220 2024-09-15T23:31:13.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.776+0000 7f48877fe640 1 -- 172.21.15.62:0/2733609082 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f4880018920 con 0x7f489006ea30 2024-09-15T23:31:13.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.777+0000 7f48877fe640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f487c0a3460 0x7f487c0a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.777+0000 7f488ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a720ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.777+0000 7f488ed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a720ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48292/0 (socket says 172.21.15.62:48292) 2024-09-15T23:31:13.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.777+0000 7f488ed9d640 1 -- 172.21.15.62:0/2525134904 learned_addr learned my addr 172.21.15.62:0/2525134904 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:13.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.777+0000 7f48877fe640 1 -- 172.21.15.62:0/2733609082 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f48800cfd10 con 0x7f489006ea30 2024-09-15T23:31:13.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.778+0000 7f488ed9d640 1 -- 172.21.15.62:0/2525134904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a721220 msgr2=0x55fa0a72b5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:31:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.778+0000 7f488ed9d640 1 --2- 172.21.15.62:0/2525134904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a721220 0x55fa0a72b5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.778+0000 7f488ed9d640 1 -- 172.21.15.62:0/2525134904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a72bae0 msgr2=0x55fa0a72ded0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:31:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.778+0000 7f488ed9d640 1 --2- 172.21.15.62:0/2525134904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a72bae0 0x55fa0a72ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.778+0000 7f488ed9d640 1 -- 172.21.15.62:0/2525134904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa0a3f7b90 con 0x55fa0a720900 2024-09-15T23:31:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.779+0000 7f488ed9d640 1 --2- 172.21.15.62:0/2525134904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a720ce0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4888009c10 tx=0x7f4888010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8975489cc0116b47 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:13.781 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.779+0000 7f4884ff9640 1 -- 172.21.15.62:0/2525134904 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888004020 con 0x55fa0a720900 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.779+0000 7f4884ff9640 1 -- 172.21.15.62:0/2525134904 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4888005500 con 0x55fa0a720900 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.779+0000 7f4884ff9640 1 -- 172.21.15.62:0/2525134904 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888004c70 con 0x55fa0a720900 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.784+0000 7f4896cdc740 1 -- 172.21.15.62:0/2525134904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 msgr2=0x55fa0a720ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.784+0000 7f4896cdc740 1 --2- 172.21.15.62:0/2525134904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a720ce0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f4888009c10 tx=0x7f4888010740 comp rx=0 tx=0).stop 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.784+0000 7f4896cdc740 1 -- 172.21.15.62:0/2525134904 shutdown_connections 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.784+0000 7f4896cdc740 1 --2- 172.21.15.62:0/2525134904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a72bae0 0x55fa0a72ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.784+0000 7f4896cdc740 1 --2- 172.21.15.62:0/2525134904 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a721220 0x55fa0a72b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.784+0000 7f4896cdc740 1 --2- 172.21.15.62:0/2525134904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a720ce0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.784+0000 7f4896cdc740 1 -- 172.21.15.62:0/2525134904 >> 172.21.15.62:0/2525134904 conn(0x55fa0a71ce90 msgr2=0x55fa0a71d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:13.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.786+0000 7f488ed9d640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f487c0a3460 0x7f487c0a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.788+0000 7f4896cdc740 1 -- 172.21.15.62:0/2525134904 shutdown_connections 2024-09-15T23:31:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.788+0000 7f4896cdc740 1 -- 172.21.15.62:0/2525134904 wait complete. 2024-09-15T23:31:13.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.788+0000 7f4896cdc740 1 -- start start 2024-09-15T23:31:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.789+0000 7f4896cdc740 1 Processor -- start 2024-09-15T23:31:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.789+0000 7f488ed9d640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f487c0a3460 0x7f487c0a5920 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4888006010 tx=0x7f48880096d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.789+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a728a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.789+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 0x55fa0a728f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.789+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a729490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.789+0000 7f4896cdc740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55fa0a72e6b0 con 0x55fa0a720900 2024-09-15T23:31:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.789+0000 7f4896cdc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fa0a60e6e0 con 0x55fa0a721220 2024-09-15T23:31:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.789+0000 7f4896cdc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55fa0a72e830 con 0x55fa0a72bae0 2024-09-15T23:31:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.789+0000 7f4896cdc740 10 client.0 _ll_register_callbacks cb 0x55fa0a632a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.790+0000 7f488ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a729490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.790+0000 7f488ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a729490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:42816/0 (socket says 172.21.15.62:42816) 2024-09-15T23:31:13.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.790+0000 7f488ed9d640 1 -- 172.21.15.62:0/3126797037 learned_addr learned my addr 172.21.15.62:0/3126797037 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:13.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.790+0000 7f488dd9b640 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 0x55fa0a728f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.793+0000 7f488ed9d640 1 -- 172.21.15.62:0/3126797037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 msgr2=0x55fa0a728f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:13.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.793+0000 7f488ed9d640 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 0x55fa0a728f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.793+0000 7f488ed9d640 1 -- 172.21.15.62:0/3126797037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 msgr2=0x55fa0a728a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:13.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.793+0000 7f488ed9d640 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a728a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.793+0000 7f488ed9d640 1 -- 172.21.15.62:0/3126797037 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa0a737c50 con 0x55fa0a72bae0 2024-09-15T23:31:13.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.797+0000 7f488ed9d640 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a729490 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f48880050f0 tx=0x7f4888009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55520152a062f776 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:13.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.797+0000 7f4875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.148:3300/0 2024-09-15T23:31:13.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.801+0000 7f4875ffb640 1 -- 172.21.15.62:0/3126797037 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888009880 con 0x55fa0a72bae0 2024-09-15T23:31:13.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.801+0000 7f4875ffb640 1 -- 172.21.15.62:0/3126797037 <== mon.1 v2:172.21.15.148:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f4888025110 con 0x55fa0a72bae0 2024-09-15T23:31:13.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.802+0000 7f4896cdc740 1 -- 172.21.15.62:0/3126797037 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4888005500 con 0x55fa0a72bae0 2024-09-15T23:31:13.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.806+0000 7f4875ffb640 1 -- 172.21.15.62:0/3126797037 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f48880259f0 con 0x55fa0a72bae0 2024-09-15T23:31:13.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.808+0000 7f4896cdc740 10 client.28087 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:13.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.808+0000 7f4896cdc740 1 -- 172.21.15.62:0/3126797037 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55fa0a73a5e0 con 0x55fa0a72bae0 2024-09-15T23:31:13.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.813+0000 7f4875ffb640 1 -- 172.21.15.62:0/3126797037 <== mon.1 v2:172.21.15.148:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4888005500 con 0x55fa0a72bae0 2024-09-15T23:31:13.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.813+0000 7f4875ffb640 1 -- 172.21.15.62:0/3126797037 <== mon.1 v2:172.21.15.148:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f4888012a40 con 0x55fa0a72bae0 2024-09-15T23:31:13.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.814+0000 7f4896cdc740 10 client.28087 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:13.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.814+0000 7f4896cdc740 10 client.28087 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:31:13.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.814+0000 7f4896cdc740 20 client.28087 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:13.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.814+0000 7f4896cdc740 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fa0a73ba40 0x55fa0a73de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.814+0000 7f4896cdc740 4 client.28087 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:13.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.814+0000 7f4896cdc740 1 -- 172.21.15.62:0/3126797037 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa0a6939d0 con 0x55fa0a73ba40 2024-09-15T23:31:13.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.817+0000 7f488e59c640 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fa0a73ba40 0x55fa0a73de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.824+0000 7f488e59c640 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fa0a73ba40 0x55fa0a73de30 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5495020e3321610d server_cookie=a211eafe0473aeb in_seq=0 out_seq=0 2024-09-15T23:31:13.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.824+0000 7f4875ffb640 10 client.28087 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:13.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.824+0000 7f4875ffb640 1 -- 172.21.15.62:0/3126797037 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fa0a6939d0 con 0x55fa0a73ba40 2024-09-15T23:31:13.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.824+0000 7f4875ffb640 10 client.28087 handle_command_reply: tid=0 2024-09-15T23:31:13.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.825+0000 7f4896cdc740 1 client.28087 shutdown 2024-09-15T23:31:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.828+0000 7f4896cdc740 1 -- 172.21.15.62:0/3126797037 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 msgr2=0x55fa0a729490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.828+0000 7f4896cdc740 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a729490 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f48880050f0 tx=0x7f4888009620 comp rx=0 tx=0).stop 2024-09-15T23:31:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.828+0000 7f4896cdc740 1 -- 172.21.15.62:0/3126797037 shutdown_connections 2024-09-15T23:31:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.828+0000 7f4896cdc740 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fa0a73ba40 0x55fa0a73de30 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.828+0000 7f4896cdc740 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a729490 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.828+0000 7f4896cdc740 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 0x55fa0a728f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.828+0000 7f4896cdc740 1 --2- 172.21.15.62:0/3126797037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a728a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.828+0000 7f4896cdc740 1 -- 172.21.15.62:0/3126797037 >> 172.21.15.62:0/3126797037 conn(0x55fa0a71ce90 msgr2=0x55fa0a722bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:13.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.828+0000 7f4875ffb640 0 client.28087 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:13.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.837+0000 7f4896cdc740 1 -- 172.21.15.62:0/3126797037 shutdown_connections 2024-09-15T23:31:13.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.837+0000 7f4896cdc740 1 -- 172.21.15.62:0/3126797037 wait complete. 2024-09-15T23:31:13.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.837+0000 7f4896cdc740 20 client.28087 trim_cache size 0 max 16384 2024-09-15T23:31:13.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.875+0000 7f4896cdc740 1 Processor -- start 2024-09-15T23:31:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.879+0000 7f4896cdc740 1 -- start start 2024-09-15T23:31:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.879+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a723360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.879+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 0x55fa0a743650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.879+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a743b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.879+0000 7f4896cdc740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4870012560 con 0x55fa0a720900 2024-09-15T23:31:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.879+0000 7f4896cdc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f487000fe00 con 0x55fa0a721220 2024-09-15T23:31:13.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.879+0000 7f4896cdc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f487000dc50 con 0x55fa0a72bae0 2024-09-15T23:31:13.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.886+0000 7f488e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a723360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.886+0000 7f488e59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a723360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48310/0 (socket says 172.21.15.62:48310) 2024-09-15T23:31:13.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.886+0000 7f488e59c640 1 -- 172.21.15.62:0/2773739221 learned_addr learned my addr 172.21.15.62:0/2773739221 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:13.894 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.886+0000 7f488dd9b640 1 --2- 172.21.15.62:0/2773739221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 0x55fa0a743650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.887+0000 7f488ed9d640 1 --2- 172.21.15.62:0/2773739221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a743b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.890+0000 7f488dd9b640 1 -- 172.21.15.62:0/2773739221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 msgr2=0x55fa0a743b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:13.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.890+0000 7f488dd9b640 1 --2- 172.21.15.62:0/2773739221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a743b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.890+0000 7f488dd9b640 1 -- 172.21.15.62:0/2773739221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 msgr2=0x55fa0a723360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:13.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.890+0000 7f488dd9b640 1 --2- 172.21.15.62:0/2773739221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a723360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.890+0000 7f488dd9b640 1 -- 172.21.15.62:0/2773739221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa0a71db20 con 0x55fa0a721220 2024-09-15T23:31:13.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.897+0000 7f488dd9b640 1 --2- 172.21.15.62:0/2773739221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 0x55fa0a743650 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4878006010 tx=0x7f4878005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d8c8758d3e40777 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:13.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.903+0000 7f4877fff640 1 -- 172.21.15.62:0/2773739221 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878036040 con 0x55fa0a721220 2024-09-15T23:31:13.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.903+0000 7f4877fff640 1 -- 172.21.15.62:0/2773739221 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4878003e50 con 0x55fa0a721220 2024-09-15T23:31:13.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.903+0000 7f4877fff640 1 -- 172.21.15.62:0/2773739221 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4878030620 con 0x55fa0a721220 2024-09-15T23:31:13.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.904+0000 7f4896cdc740 1 -- 172.21.15.62:0/2773739221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 msgr2=0x55fa0a743650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:13.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.904+0000 7f4896cdc740 1 --2- 172.21.15.62:0/2773739221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 0x55fa0a743650 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4878006010 tx=0x7f4878005df0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.913+0000 7f4896cdc740 1 -- 172.21.15.62:0/2773739221 shutdown_connections 2024-09-15T23:31:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.913+0000 7f4896cdc740 1 --2- 172.21.15.62:0/2773739221 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a743b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.913+0000 7f4896cdc740 1 --2- 172.21.15.62:0/2773739221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a721220 0x55fa0a743650 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.913+0000 7f4896cdc740 1 --2- 172.21.15.62:0/2773739221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a720900 0x55fa0a723360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.913+0000 7f4896cdc740 1 -- 172.21.15.62:0/2773739221 >> 172.21.15.62:0/2773739221 conn(0x55fa0a71ce90 msgr2=0x55fa0a72aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:13.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.921+0000 7f4896cdc740 1 -- 172.21.15.62:0/2773739221 shutdown_connections 2024-09-15T23:31:13.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.921+0000 7f4896cdc740 1 -- 172.21.15.62:0/2773739221 wait complete. 2024-09-15T23:31:13.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.922+0000 7f4896cdc740 1 -- start start 2024-09-15T23:31:13.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.922+0000 7f4896cdc740 1 Processor -- start 2024-09-15T23:31:13.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.926+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a720900 0x55fa0a72fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.926+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a721220 0x55fa0a7300a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.926+0000 7f4896cdc740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a7a5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.926+0000 7f4896cdc740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f487000e2c0 con 0x55fa0a721220 2024-09-15T23:31:13.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.926+0000 7f4896cdc740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f487000f730 con 0x55fa0a720900 2024-09-15T23:31:13.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.926+0000 7f4896cdc740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f487000e990 con 0x55fa0a72bae0 2024-09-15T23:31:13.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.926+0000 7f4896cdc740 10 client.0 _ll_register_callbacks cb 0x7f4888004d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:13.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.927+0000 7f488e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a720900 0x55fa0a72fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.927+0000 7f488ed9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a7a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.927+0000 7f488e59c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a720900 0x55fa0a72fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:58970/0 (socket says 172.21.15.62:58970) 2024-09-15T23:31:13.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.927+0000 7f488e59c640 1 -- 172.21.15.62:0/1077799519 learned_addr learned my addr 172.21.15.62:0/1077799519 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:13.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.927+0000 7f488e59c640 1 -- 172.21.15.62:0/1077799519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 msgr2=0x55fa0a7a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:13.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.927+0000 7f488e59c640 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a7a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.927+0000 7f488e59c640 1 -- 172.21.15.62:0/1077799519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a721220 msgr2=0x55fa0a7300a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:13.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.927+0000 7f488e59c640 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a721220 0x55fa0a7300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:13.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.927+0000 7f488e59c640 1 -- 172.21.15.62:0/1077799519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa0a730930 con 0x55fa0a720900 2024-09-15T23:31:13.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.937+0000 7f488e59c640 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a720900 0x55fa0a72fb60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f48800983b0 tx=0x7f48800092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e513c9ded808f2ed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:13.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.937+0000 7f4875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:31:13.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.937+0000 7f4875ffb640 1 -- 172.21.15.62:0/1077799519 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48800a0300 con 0x55fa0a720900 2024-09-15T23:31:13.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.937+0000 7f4875ffb640 1 -- 172.21.15.62:0/1077799519 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f48800985c0 con 0x55fa0a720900 2024-09-15T23:31:13.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.937+0000 7f4896cdc740 1 -- 172.21.15.62:0/1077799519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4878003e50 con 0x55fa0a720900 2024-09-15T23:31:13.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.937+0000 7f4875ffb640 1 -- 172.21.15.62:0/1077799519 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f48800a9050 con 0x55fa0a720900 2024-09-15T23:31:13.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.941+0000 7f4896cdc740 10 client.28436 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.941+0000 7f4896cdc740 1 -- 172.21.15.62:0/1077799519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55fa0a7a6c70 con 0x55fa0a720900 2024-09-15T23:31:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.943+0000 7f4875ffb640 1 -- 172.21.15.62:0/1077799519 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4878003e50 con 0x55fa0a720900 2024-09-15T23:31:13.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.943+0000 7f4875ffb640 1 -- 172.21.15.62:0/1077799519 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f488009b3a0 con 0x55fa0a720900 2024-09-15T23:31:13.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.945+0000 7f4896cdc740 10 client.28436 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:13.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.945+0000 7f4896cdc740 10 client.28436 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:31:13.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.945+0000 7f4896cdc740 20 client.28436 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:13.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.945+0000 7f4896cdc740 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fa0a7a7400 0x55fa0a7a9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:13.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.945+0000 7f4896cdc740 4 client.28436 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}] 2024-09-15T23:31:13.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.945+0000 7f4896cdc740 1 -- 172.21.15.62:0/1077799519 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "roots": ["/"], "timeout": 90.0, "expiration": 137.7, "await_for": 45.0}) -- 0x55fa0a68dd20 con 0x55fa0a7a7400 2024-09-15T23:31:13.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.949+0000 7f488dd9b640 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fa0a7a7400 0x55fa0a7a9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:13.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.956+0000 7f488dd9b640 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fa0a7a7400 0x55fa0a7a9830 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8524e6f8dede4172 server_cookie=d8efc89b68fafe76 in_seq=0 out_seq=0 2024-09-15T23:31:13.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:13.958+0000 7f4875ffb640 10 client.28436 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:14.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.282+0000 7f4875ffb640 1 -- 172.21.15.62:0/1077799519 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x55fa0a68dd20 con 0x55fa0a7a7400 2024-09-15T23:31:14.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.282+0000 7f4875ffb640 10 client.28436 handle_command_reply: tid=0 2024-09-15T23:31:14.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.282+0000 7f4896cdc740 1 client.28436 shutdown 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 -- 172.21.15.62:0/1077799519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a720900 msgr2=0x55fa0a72fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a720900 0x55fa0a72fb60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f48800983b0 tx=0x7f48800092d0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 -- 172.21.15.62:0/1077799519 shutdown_connections 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55fa0a7a7400 0x55fa0a7a9830 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55fa0a72bae0 0x55fa0a7a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4875ffb640 0 client.28436 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fa0a721220 0x55fa0a7300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 --2- 172.21.15.62:0/1077799519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fa0a720900 0x55fa0a72fb60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 -- 172.21.15.62:0/1077799519 >> 172.21.15.62:0/1077799519 conn(0x55fa0a71ce90 msgr2=0x55fa0a71b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 -- 172.21.15.62:0/1077799519 shutdown_connections 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 1 -- 172.21.15.62:0/1077799519 wait complete. 2024-09-15T23:31:14.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f4896cdc740 20 client.28436 trim_cache size 0 max 16384 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 36, 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 58, 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "7334f5cd": { 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 58, 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:31:14.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 137.7, 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "QUIESCED", 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:14.287 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:31:14.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f488f59e640 1 -- 172.21.15.62:0/2733609082 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f487c0a3460 msgr2=0x7f487c0a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:14.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f488f59e640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f487c0a3460 0x7f487c0a5920 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4888006010 tx=0x7f48880096d0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f488f59e640 1 -- 172.21.15.62:0/2733609082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f489006ea30 msgr2=0x7f489011a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:14.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.283+0000 7f488f59e640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f489006ea30 0x7f489011a470 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f488000e790 tx=0x7f488000ec50 comp rx=0 tx=0).stop 2024-09-15T23:31:14.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.284+0000 7f488f59e640 1 -- 172.21.15.62:0/2733609082 shutdown_connections 2024-09-15T23:31:14.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.284+0000 7f488f59e640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f487c0a3460 0x7f487c0a5920 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.284+0000 7f488f59e640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f489011b140 0x7f4890114510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.284+0000 7f488f59e640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4890110190 0x7f489011a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.284+0000 7f488f59e640 1 --2- 172.21.15.62:0/2733609082 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f489006ea30 0x7f489011a470 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:14.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.284+0000 7f488f59e640 1 -- 172.21.15.62:0/2733609082 >> 172.21.15.62:0/2733609082 conn(0x7f4890073180 msgr2=0x7f489011b6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:14.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.284+0000 7f488f59e640 1 -- 172.21.15.62:0/2733609082 shutdown_connections 2024-09-15T23:31:14.289 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.284+0000 7f488f59e640 1 -- 172.21.15.62:0/2733609082 wait complete. 2024-09-15T23:31:14.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 895.0 s, 457.4 tps, lat 206.998 ms stddev 259.571 2024-09-15T23:31:15.110 INFO:tasks.quiescer.fs.[cephfs]:Successfully quiesced set '7334f5cd', quiesce took 7.0 seconds. Will release after: 10.7 2024-09-15T23:31:15.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.380+0000 7ff9d2053640 1 -- 172.21.15.62:0/120202373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc074510 msgr2=0x7ff9cc0748f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.380+0000 7ff9d2053640 1 --2- 172.21.15.62:0/120202373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc074510 0x7ff9cc0748f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc009900 tx=0x7ff9bc02a480 comp rx=0 tx=0).stop 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.381+0000 7ff9d2053640 1 -- 172.21.15.62:0/120202373 shutdown_connections 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.381+0000 7ff9d2053640 1 --2- 172.21.15.62:0/120202373 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9cc10e440 0x7ff9cc110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.381+0000 7ff9d2053640 1 --2- 172.21.15.62:0/120202373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9cc074e30 0x7ff9cc10df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.381+0000 7ff9d2053640 1 --2- 172.21.15.62:0/120202373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc074510 0x7ff9cc0748f0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.381+0000 7ff9d2053640 1 -- 172.21.15.62:0/120202373 >> 172.21.15.62:0/120202373 conn(0x7ff9cc0fd8c0 msgr2=0x7ff9cc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.381+0000 7ff9d2053640 1 -- 172.21.15.62:0/120202373 shutdown_connections 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.381+0000 7ff9d2053640 1 -- 172.21.15.62:0/120202373 wait complete. 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.381+0000 7ff9d2053640 1 Processor -- start 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.381+0000 7ff9d2053640 1 -- start start 2024-09-15T23:31:15.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9d2053640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9cc074510 0x7ff9cc19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9d2053640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9cc074e30 0x7ff9cc19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9d2053640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc10e440 0x7ff9cc19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9d2053640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc114150 con 0x7ff9cc10e440 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9d2053640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc113fd0 con 0x7ff9cc074510 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9d2053640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc1142d0 con 0x7ff9cc074e30 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc10e440 0x7ff9cc19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc10e440 0x7ff9cc19efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48342/0 (socket says 172.21.15.62:48342) 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9cbfff640 1 -- 172.21.15.62:0/2728662368 learned_addr learned my addr 172.21.15.62:0/2728662368 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9cc074e30 0x7ff9cc19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9cc074510 0x7ff9cc19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9cbfff640 1 -- 172.21.15.62:0/2728662368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9cc074e30 msgr2=0x7ff9cc19ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9cbfff640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9cc074e30 0x7ff9cc19ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9cbfff640 1 -- 172.21.15.62:0/2728662368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9cc074510 msgr2=0x7ff9cc19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9cbfff640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9cc074510 0x7ff9cc19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9cbfff640 1 -- 172.21.15.62:0/2728662368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9cc19f6c0 con 0x7ff9cc10e440 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.382+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9cc074e30 0x7ff9cc19ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.383+0000 7ff9cbfff640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc10e440 0x7ff9cc19efe0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400e9d0 tx=0x7ff9c400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.383+0000 7ff9c97fa640 1 -- 172.21.15.62:0/2728662368 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c400cd80 con 0x7ff9cc10e440 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.383+0000 7ff9c97fa640 1 -- 172.21.15.62:0/2728662368 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9c4005020 con 0x7ff9cc10e440 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.383+0000 7ff9d2053640 1 -- 172.21.15.62:0/2728662368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9cc19f9b0 con 0x7ff9cc10e440 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.383+0000 7ff9c97fa640 1 -- 172.21.15.62:0/2728662368 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40116d0 con 0x7ff9cc10e440 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.383+0000 7ff9d2053640 1 -- 172.21.15.62:0/2728662368 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9cc1a7290 con 0x7ff9cc10e440 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9c97fa640 1 -- 172.21.15.62:0/2728662368 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff9c40051d0 con 0x7ff9cc10e440 2024-09-15T23:31:15.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9d442c740 1 Processor -- start 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9d442c740 1 -- start start 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6bfb40 0x5589de6be090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6bea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 0x5589de6cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9d442c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5589de6242f0 con 0x5589de6bfb40 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9d442c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5589de5cbcf0 con 0x5589de6c9370 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9d442c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5589de6202b0 con 0x5589de6be5d0 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9c2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6bfb40 0x5589de6be090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.386+0000 7ff9c2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6bfb40 0x5589de6be090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48356/0 (socket says 172.21.15.62:48356) 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c2ffd640 1 -- 172.21.15.62:0/1030418821 learned_addr learned my addr 172.21.15.62:0/1030418821 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:15.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/1030418821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6bea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1030418821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 0x5589de6cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c2ffd640 1 -- 172.21.15.62:0/1030418821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 msgr2=0x5589de6bea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1030418821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6bea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c2ffd640 1 -- 172.21.15.62:0/1030418821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 msgr2=0x5589de6cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1030418821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 0x5589de6cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c2ffd640 1 -- 172.21.15.62:0/1030418821 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589de5b03a0 con 0x5589de6bfb40 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c97fa640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9b80a32c0 0x7ff9b80a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1030418821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6bfb40 0x5589de6be090 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40135e0 tx=0x7ff9b40097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7c5ac27e6474a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9b80a32c0 0x7ff9b80a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c97fa640 1 -- 172.21.15.62:0/2728662368 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff9c40d0210 con 0x7ff9cc10e440 2024-09-15T23:31:15.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c1ffb640 1 -- 172.21.15.62:0/1030418821 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4011470 con 0x5589de6bfb40 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c1ffb640 1 -- 172.21.15.62:0/1030418821 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9b4011af0 con 0x5589de6bfb40 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.387+0000 7ff9c1ffb640 1 -- 172.21.15.62:0/1030418821 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b400fa70 con 0x5589de6bfb40 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9d442c740 1 -- 172.21.15.62:0/1030418821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6bfb40 msgr2=0x5589de6be090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9b80a32c0 0x7ff9b80a5780 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc005940 tx=0x7ff9bc0058b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1030418821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6bfb40 0x5589de6be090 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40135e0 tx=0x7ff9b40097d0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9d442c740 1 -- 172.21.15.62:0/1030418821 shutdown_connections 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1030418821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 0x5589de6cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1030418821 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6bea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1030418821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6bfb40 0x5589de6be090 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9d442c740 1 -- 172.21.15.62:0/1030418821 >> 172.21.15.62:0/1030418821 conn(0x5589de687310 msgr2=0x5589de6bac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9d442c740 1 -- 172.21.15.62:0/1030418821 shutdown_connections 2024-09-15T23:31:15.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.388+0000 7ff9d442c740 1 -- 172.21.15.62:0/1030418821 wait complete. 2024-09-15T23:31:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9d442c740 1 -- start start 2024-09-15T23:31:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9d442c740 1 Processor -- start 2024-09-15T23:31:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6be5d0 0x5589de6c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6bfb40 0x5589de6c6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 0x5589de6c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9d442c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5589de6cc070 con 0x5589de6be5d0 2024-09-15T23:31:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9d442c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5589de5df060 con 0x5589de6c9370 2024-09-15T23:31:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9d442c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5589de6cc1f0 con 0x5589de6bfb40 2024-09-15T23:31:15.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9d442c740 10 client.0 _ll_register_callbacks cb 0x5589de3328c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:15.248 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:31:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9cb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6be5d0 0x5589de6c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9cb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6be5d0 0x5589de6c8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48372/0 (socket says 172.21.15.62:48372) 2024-09-15T23:31:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9cb7fe640 1 -- 172.21.15.62:0/1415313616 learned_addr learned my addr 172.21.15.62:0/1415313616 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6bfb40 0x5589de6c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.389+0000 7ff9cbfff640 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 0x5589de6c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9cb7fe640 1 -- 172.21.15.62:0/1415313616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6bfb40 msgr2=0x5589de6c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6bfb40 0x5589de6c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9cb7fe640 1 -- 172.21.15.62:0/1415313616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 msgr2=0x5589de6c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 0x5589de6c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9cb7fe640 1 -- 172.21.15.62:0/1415313616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589de3953c0 con 0x5589de6be5d0 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6be5d0 0x5589de6c8db0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc033000 tx=0x7ff9bc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5fcb7c26d186d96 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9aaffd640 1 -- 172.21.15.62:0/1415313616 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc04c070 con 0x5589de6be5d0 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9aaffd640 1 -- 172.21.15.62:0/1415313616 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff9bc003cf0 con 0x5589de6be5d0 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9d442c740 1 -- 172.21.15.62:0/1415313616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9b4011af0 con 0x5589de6be5d0 2024-09-15T23:31:15.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9aaffd640 1 -- 172.21.15.62:0/1415313616 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff9bc058050 con 0x5589de6be5d0 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9d442c740 10 client.18954 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9d442c740 1 -- 172.21.15.62:0/1415313616 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5589de6d7b60 con 0x5589de6be5d0 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.390+0000 7ff9aaffd640 1 -- 172.21.15.62:0/1415313616 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9b4011af0 con 0x5589de6be5d0 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9aaffd640 1 -- 172.21.15.62:0/1415313616 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7ff9bc0049a0 con 0x5589de6be5d0 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9d442c740 10 client.18954 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9d442c740 10 client.18954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9d442c740 20 client.18954 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589de6d9030 0x5589de6db4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9d442c740 4 client.18954 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9d442c740 1 -- 172.21.15.62:0/1415313616 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589de631860 con 0x5589de6d9030 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589de6d9030 0x5589de6db4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.250 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9aaffd640 10 client.18954 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.391+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589de6d9030 0x5589de6db4d0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2c3f96ef54caba5 server_cookie=fcb388416aa2bcc7 in_seq=0 out_seq=0 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9aaffd640 1 -- 172.21.15.62:0/1415313616 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5589de631860 con 0x5589de6d9030 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9aaffd640 10 client.18954 handle_command_reply: tid=0 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 client.18954 shutdown 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 -- 172.21.15.62:0/1415313616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6be5d0 msgr2=0x5589de6c8db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6be5d0 0x5589de6c8db0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc033000 tx=0x7ff9bc009260 comp rx=0 tx=0).stop 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 -- 172.21.15.62:0/1415313616 shutdown_connections 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589de6d9030 0x5589de6db4d0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6c9370 0x5589de6c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6bfb40 0x5589de6c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1415313616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6be5d0 0x5589de6c8db0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9aaffd640 0 client.18954 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 -- 172.21.15.62:0/1415313616 >> 172.21.15.62:0/1415313616 conn(0x5589de687310 msgr2=0x5589de6c0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 -- 172.21.15.62:0/1415313616 shutdown_connections 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 1 -- 172.21.15.62:0/1415313616 wait complete. 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.392+0000 7ff9d442c740 20 client.18954 trim_cache size 0 max 16384 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.405+0000 7ff9d442c740 1 Processor -- start 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.405+0000 7ff9d442c740 1 -- start start 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6c1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9d442c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc1b3850 con 0x5589de6c9370 2024-09-15T23:31:15.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9d442c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc1b10f0 con 0x5589de6bfb40 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9d442c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc1aef40 con 0x5589de6be5d0 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6c1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9c2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6c1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48384/0 (socket says 172.21.15.62:48384) 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9cbfff640 1 -- 172.21.15.62:0/1198041557 learned_addr learned my addr 172.21.15.62:0/1198041557 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/1198041557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9c2ffd640 1 -- 172.21.15.62:0/1198041557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 msgr2=0x5589de6b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1198041557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9c2ffd640 1 -- 172.21.15.62:0/1198041557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 msgr2=0x5589de6c1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1198041557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6c1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9c2ffd640 1 -- 172.21.15.62:0/1198041557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589de6c45b0 con 0x5589de6bfb40 2024-09-15T23:31:15.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.406+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/1198041557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6c1120 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x5589de6c58f0 tx=0x7ff9b4005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81711740c9990a76 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9c17fa640 1 -- 172.21.15.62:0/1198041557 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b40130d0 con 0x5589de6bfb40 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9c17fa640 1 -- 172.21.15.62:0/1198041557 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9b4005520 con 0x5589de6bfb40 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9c17fa640 1 -- 172.21.15.62:0/1198041557 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b40035f0 con 0x5589de6bfb40 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9d442c740 1 -- 172.21.15.62:0/1198041557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 msgr2=0x5589de6c1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1198041557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6c1120 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x5589de6c58f0 tx=0x7ff9b4005fb0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9d442c740 1 -- 172.21.15.62:0/1198041557 shutdown_connections 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1198041557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6c1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1198041557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6c1120 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9d442c740 1 --2- 172.21.15.62:0/1198041557 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9d442c740 1 -- 172.21.15.62:0/1198041557 >> 172.21.15.62:0/1198041557 conn(0x5589de687310 msgr2=0x5589de6c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9d442c740 1 -- 172.21.15.62:0/1198041557 shutdown_connections 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.407+0000 7ff9d442c740 1 -- 172.21.15.62:0/1198041557 wait complete. 2024-09-15T23:31:15.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9d442c740 1 -- start start 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9d442c740 1 Processor -- start 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6cd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9d442c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9d442c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc1af5b0 con 0x5589de6c9370 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9d442c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc1b0a20 con 0x5589de6bfb40 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9d442c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff9cc1afc80 con 0x5589de6be5d0 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9d442c740 10 client.0 _ll_register_callbacks cb 0x7ff9bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9c2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6cdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48388/0 (socket says 172.21.15.62:48388) 2024-09-15T23:31:15.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9cb7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.408+0000 7ff9cbfff640 1 -- 172.21.15.62:0/2365560090 learned_addr learned my addr 172.21.15.62:0/2365560090 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9cbfff640 1 -- 172.21.15.62:0/2365560090 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 msgr2=0x5589de6cd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9cbfff640 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6cd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9cbfff640 1 -- 172.21.15.62:0/2365560090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 msgr2=0x5589de6cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9cbfff640 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9cbfff640 1 -- 172.21.15.62:0/2365560090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589de742dc0 con 0x5589de6c9370 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9c2ffd640 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9cbfff640 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6cdca0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400c640 tx=0x7ff9c409bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20be33d04b7f0f02 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:15.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9aaffd640 1 -- 172.21.15.62:0/2365560090 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c40b4070 con 0x5589de6c9370 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9aaffd640 1 -- 172.21.15.62:0/2365560090 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff9c40b5260 con 0x5589de6c9370 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9d442c740 1 -- 172.21.15.62:0/2365560090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9b4005520 con 0x5589de6c9370 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9aaffd640 1 -- 172.21.15.62:0/2365560090 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff9c401f040 con 0x5589de6c9370 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9d442c740 10 client.18960 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9d442c740 1 -- 172.21.15.62:0/2365560090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5589de7440a0 con 0x5589de6c9370 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9aaffd640 1 -- 172.21.15.62:0/2365560090 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9b4005520 con 0x5589de6c9370 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.409+0000 7ff9aaffd640 1 -- 172.21.15.62:0/2365560090 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7ff9c400f040 con 0x5589de6c9370 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9d442c740 10 client.18960 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9d442c740 10 client.18960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9d442c740 20 client.18960 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:15.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9d442c740 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589de7447f0 0x5589de746c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9d442c740 4 client.18960 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9d442c740 1 -- 172.21.15.62:0/2365560090 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5589de62bd70 con 0x5589de7447f0 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589de7447f0 0x5589de746c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9cb7fe640 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589de7447f0 0x5589de746c10 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa84d895453d962b server_cookie=5d707c4d1220fd59 in_seq=0 out_seq=0 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9aaffd640 10 client.18960 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9aaffd640 1 -- 172.21.15.62:0/2365560090 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5589de62bd70 con 0x5589de7447f0 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9aaffd640 10 client.18960 handle_command_reply: tid=0 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9d442c740 1 client.18960 shutdown 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9d442c740 1 -- 172.21.15.62:0/2365560090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 msgr2=0x5589de6cdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.410+0000 7ff9d442c740 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6cdca0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400c640 tx=0x7ff9c409bed0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d442c740 1 -- 172.21.15.62:0/2365560090 shutdown_connections 2024-09-15T23:31:15.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d442c740 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5589de7447f0 0x5589de746c10 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d442c740 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5589de6c9370 0x5589de6cdca0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d442c740 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5589de6bfb40 0x5589de6cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d442c740 1 --2- 172.21.15.62:0/2365560090 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5589de6be5d0 0x5589de6cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9aaffd640 0 client.18960 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d442c740 1 -- 172.21.15.62:0/2365560090 >> 172.21.15.62:0/2365560090 conn(0x5589de687310 msgr2=0x5589de6b94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d442c740 1 -- 172.21.15.62:0/2365560090 shutdown_connections 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d442c740 1 -- 172.21.15.62:0/2365560090 wait complete. 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d442c740 20 client.18960 trim_cache size 0 max 16384 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 -- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9b80a32c0 msgr2=0x7ff9b80a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9b80a32c0 0x7ff9b80a5780 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc005940 tx=0x7ff9bc0058b0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 -- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc10e440 msgr2=0x7ff9cc19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc10e440 0x7ff9cc19efe0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400e9d0 tx=0x7ff9c400ee90 comp rx=0 tx=0).stop 2024-09-15T23:31:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 -- 172.21.15.62:0/2728662368 shutdown_connections 2024-09-15T23:31:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff9b80a32c0 0x7ff9b80a5780 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff9cc10e440 0x7ff9cc19efe0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff9cc074e30 0x7ff9cc19ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 --2- 172.21.15.62:0/2728662368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9cc074510 0x7ff9cc19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 -- 172.21.15.62:0/2728662368 >> 172.21.15.62:0/2728662368 conn(0x7ff9cc0fd8c0 msgr2=0x7ff9cc0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 -- 172.21.15.62:0/2728662368 shutdown_connections 2024-09-15T23:31:15.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:14.411+0000 7ff9d2053640 1 -- 172.21.15.62:0/2728662368 wait complete. 2024-09-15T23:31:15.406 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:15 smithi062 ceph-mon[30949]: pgmap v819: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 MiB/s wr, 182 op/s 2024-09-15T23:31:15.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:15 smithi067 ceph-mon[35665]: pgmap v819: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 MiB/s wr, 182 op/s 2024-09-15T23:31:15.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:15 smithi148 ceph-mon[35610]: pgmap v819: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 MiB/s wr, 182 op/s 2024-09-15T23:31:16.857 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-09-15T23:31:16.857 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-09-15T23:31:17.128 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:17.524 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:17 smithi062 ceph-mon[30949]: pgmap v820: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 MiB/s wr, 258 op/s 2024-09-15T23:31:17.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:17 smithi067 ceph-mon[35665]: pgmap v820: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 MiB/s wr, 258 op/s 2024-09-15T23:31:17.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:17 smithi148 ceph-mon[35610]: pgmap v820: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.7 MiB/s wr, 258 op/s 2024-09-15T23:31:17.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.611+0000 7fa1c0e35640 1 -- 172.21.15.62:0/4230856266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc102570 msgr2=0x7fa1bc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:17.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.611+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/4230856266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc102570 0x7fa1bc1029b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009900 tx=0x7fa1b002a480 comp rx=0 tx=0).stop 2024-09-15T23:31:17.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.611+0000 7fa1c0e35640 1 -- 172.21.15.62:0/4230856266 shutdown_connections 2024-09-15T23:31:17.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.611+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/4230856266 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1bc102ef0 0x7fa1bc10da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.611+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/4230856266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc102570 0x7fa1bc1029b0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.611+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/4230856266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1bc100e20 0x7fa1bc101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.611+0000 7fa1c0e35640 1 -- 172.21.15.62:0/4230856266 >> 172.21.15.62:0/4230856266 conn(0x7fa1bc0fc7d0 msgr2=0x7fa1bc0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.611+0000 7fa1c0e35640 1 -- 172.21.15.62:0/4230856266 shutdown_connections 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.611+0000 7fa1c0e35640 1 -- 172.21.15.62:0/4230856266 wait complete. 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1c0e35640 1 Processor -- start 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1c0e35640 1 -- start start 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1c0e35640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc100e20 0x7fa1bc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1c0e35640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1bc102570 0x7fa1bc198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1c0e35640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1bc102ef0 0x7fa1bc19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1c0e35640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa1bc110fd0 con 0x7fa1bc100e20 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1c0e35640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1bc110e50 con 0x7fa1bc102ef0 2024-09-15T23:31:17.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1c0e35640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa1bc111150 con 0x7fa1bc102570 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1ba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc100e20 0x7fa1bc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1ba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc100e20 0x7fa1bc198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48404/0 (socket says 172.21.15.62:48404) 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.612+0000 7fa1ba59c640 1 -- 172.21.15.62:0/1856169247 learned_addr learned my addr 172.21.15.62:0/1856169247 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1bc102570 0x7fa1bc198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1bc102ef0 0x7fa1bc19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1ba59c640 1 -- 172.21.15.62:0/1856169247 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1bc102570 msgr2=0x7fa1bc198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1bc102570 0x7fa1bc198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1ba59c640 1 -- 172.21.15.62:0/1856169247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1bc102ef0 msgr2=0x7fa1bc19cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1bc102ef0 0x7fa1bc19cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1ba59c640 1 -- 172.21.15.62:0/1856169247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1bc19d3f0 con 0x7fa1bc100e20 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1bc102570 0x7fa1bc198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc100e20 0x7fa1bc198440 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00c920 tx=0x7fa1ac00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1ab7fe640 1 -- 172.21.15.62:0/1856169247 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac007c70 con 0x7fa1bc100e20 2024-09-15T23:31:17.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1c0e35640 1 -- 172.21.15.62:0/1856169247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1bc19d6e0 con 0x7fa1bc100e20 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1ab7fe640 1 -- 172.21.15.62:0/1856169247 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ac007e10 con 0x7fa1bc100e20 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1ab7fe640 1 -- 172.21.15.62:0/1856169247 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac018890 con 0x7fa1bc100e20 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.613+0000 7fa1c0e35640 1 -- 172.21.15.62:0/1856169247 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1bc1a4fc0 con 0x7fa1bc100e20 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.615+0000 7fa1ab7fe640 1 -- 172.21.15.62:0/1856169247 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa1ac018b30 con 0x7fa1bc100e20 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1c320e740 1 Processor -- start 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1c320e740 1 -- start start 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1ab7fe640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1a40a32c0 0x7fa1a40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa40e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 0x55777fa4fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 0x55777fa52540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1c320e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55777f9a6280 con 0x55777fa40a60 2024-09-15T23:31:17.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1c320e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55777f94db10 con 0x55777fa50150 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1c320e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55777f9a2240 con 0x55777fa41380 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1a40a32c0 0x7fa1a40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1ab7fe640 1 -- 172.21.15.62:0/1856169247 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa1ac0cfe60 con 0x7fa1bc100e20 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1ba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa40e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.616+0000 7fa1ba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa40e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48420/0 (socket says 172.21.15.62:48420) 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1ba59c640 1 -- 172.21.15.62:0/904173100 learned_addr learned my addr 172.21.15.62:0/904173100 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 0x55777fa4fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 0x55777fa52540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1ba59c640 1 -- 172.21.15.62:0/904173100 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 msgr2=0x55777fa4fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 0x55777fa4fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1ba59c640 1 -- 172.21.15.62:0/904173100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 msgr2=0x55777fa52540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 0x55777fa52540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1ba59c640 1 -- 172.21.15.62:0/904173100 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55777f95a1f0 con 0x55777fa40a60 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 0x55777fa4fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1a40a32c0 0x7fa1a40a5780 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4005fd0 tx=0x7fa1b4014ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa40e40 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0a4840 tx=0x7fa1ac00ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8d55205f6d9d27 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1a8ff9640 1 -- 172.21.15.62:0/904173100 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac09c800 con 0x55777fa40a60 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1a8ff9640 1 -- 172.21.15.62:0/904173100 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ac09c100 con 0x55777fa40a60 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.617+0000 7fa1a8ff9640 1 -- 172.21.15.62:0/904173100 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0bc110 con 0x55777fa40a60 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 -- 172.21.15.62:0/904173100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 msgr2=0x55777fa40e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa40e40 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0a4840 tx=0x7fa1ac00ff30 comp rx=0 tx=0).stop 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 -- 172.21.15.62:0/904173100 shutdown_connections 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 0x55777fa52540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 0x55777fa4fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 --2- 172.21.15.62:0/904173100 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa40e40 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 -- 172.21.15.62:0/904173100 >> 172.21.15.62:0/904173100 conn(0x55777fa3d0f0 msgr2=0x55777fa3d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:17.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 -- 172.21.15.62:0/904173100 shutdown_connections 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 -- 172.21.15.62:0/904173100 wait complete. 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.618+0000 7fa1c320e740 1 -- start start 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1c320e740 1 Processor -- start 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa4d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa41380 0x55777fa4d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa50150 0x55777fa4db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1c320e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55777fa52d80 con 0x55777fa40a60 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1c320e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55777f960f10 con 0x55777fa41380 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1c320e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x55777fa52f00 con 0x55777fa50150 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1c320e740 10 client.0 _ll_register_callbacks cb 0x55777f929a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1ba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa4d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1ba59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa4d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48422/0 (socket says 172.21.15.62:48422) 2024-09-15T23:31:17.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1ba59c640 1 -- 172.21.15.62:0/3358369257 learned_addr learned my addr 172.21.15.62:0/3358369257 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa41380 0x55777fa4d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa50150 0x55777fa4db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1ba59c640 1 -- 172.21.15.62:0/3358369257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa50150 msgr2=0x55777fa4db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa50150 0x55777fa4db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1ba59c640 1 -- 172.21.15.62:0/3358369257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa41380 msgr2=0x55777fa4d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa41380 0x55777fa4d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1ba59c640 1 -- 172.21.15.62:0/3358369257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55777f7176a0 con 0x55777fa40a60 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa50150 0x55777fa4db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa4d100 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0cd950 tx=0x7fa1ac09a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2737bec8de4163 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.619+0000 7fa1a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/3358369257 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0a4d70 con 0x55777fa40a60 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/3358369257 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa1ac0a0170 con 0x55777fa40a60 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1c320e740 1 -- 172.21.15.62:0/3358369257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1ac09c100 con 0x55777fa40a60 2024-09-15T23:31:17.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/3358369257 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa1ac09c800 con 0x55777fa40a60 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1c320e740 10 client.18972 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1c320e740 1 -- 172.21.15.62:0/3358369257 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55777fa52d80 con 0x55777fa40a60 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/3358369257 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ac09c100 con 0x55777fa40a60 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/3358369257 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fa1ac09afc0 con 0x55777fa40a60 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1c320e740 10 client.18972 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1c320e740 10 client.18972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1c320e740 20 client.18972 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55777fa5fc20 0x55777fa62010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1c320e740 4 client.18972 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.620+0000 7fa1c320e740 1 -- 172.21.15.62:0/3358369257 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55777f9b37f0 con 0x55777fa5fc20 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.621+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55777fa5fc20 0x55777fa62010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.621+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55777fa5fc20 0x55777fa62010 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36d725fcf2a22daf server_cookie=bb4dbb996b7a9430 in_seq=0 out_seq=0 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.621+0000 7fa1a1ffb640 10 client.18972 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:17.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/3358369257 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55777f9b37f0 con 0x55777fa5fc20 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1a1ffb640 10 client.18972 handle_command_reply: tid=0 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 client.18972 shutdown 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 -- 172.21.15.62:0/3358369257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 msgr2=0x55777fa4d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa4d100 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0cd950 tx=0x7fa1ac09a210 comp rx=0 tx=0).stop 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 -- 172.21.15.62:0/3358369257 shutdown_connections 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55777fa5fc20 0x55777fa62010 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa50150 0x55777fa4db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa41380 0x55777fa4d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1a1ffb640 0 client.18972 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3358369257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa40a60 0x55777fa4d100 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 -- 172.21.15.62:0/3358369257 >> 172.21.15.62:0/3358369257 conn(0x55777fa3d0f0 msgr2=0x55777fa42e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 -- 172.21.15.62:0/3358369257 shutdown_connections 2024-09-15T23:31:17.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 1 -- 172.21.15.62:0/3358369257 wait complete. 2024-09-15T23:31:17.642 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.622+0000 7fa1c320e740 20 client.18972 trim_cache size 0 max 16384 2024-09-15T23:31:17.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1c320e740 1 Processor -- start 2024-09-15T23:31:17.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1c320e740 1 -- start start 2024-09-15T23:31:17.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa40a60 0x55777fa43100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa41380 0x55777fa4d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 0x55777fa67780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1c320e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa1bc1af270 con 0x55777fa41380 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1c320e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1bc1acb10 con 0x55777fa50150 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1c320e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa1bc1aa960 con 0x55777fa40a60 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1b9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa41380 0x55777fa4d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1b9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa41380 0x55777fa4d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48424/0 (socket says 172.21.15.62:48424) 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1b9d9b640 1 -- 172.21.15.62:0/3009504498 learned_addr learned my addr 172.21.15.62:0/3009504498 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/3009504498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa40a60 0x55777fa43100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/3009504498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 0x55777fa67780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1b9d9b640 1 -- 172.21.15.62:0/3009504498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa40a60 msgr2=0x55777fa43100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.640+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/3009504498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa40a60 0x55777fa43100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.641+0000 7fa1b9d9b640 1 -- 172.21.15.62:0/3009504498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 msgr2=0x55777fa67780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.641+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/3009504498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 0x55777fa67780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.641+0000 7fa1b9d9b640 1 -- 172.21.15.62:0/3009504498 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55777fa411b0 con 0x55777fa41380 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.641+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/3009504498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa41380 0x55777fa4d9d0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0005ab0 tx=0x7fa1b0005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665fa9108b2e921 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.641+0000 7fa1a3fff640 1 -- 172.21.15.62:0/3009504498 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0035040 con 0x55777fa41380 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.641+0000 7fa1a3fff640 1 -- 172.21.15.62:0/3009504498 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1b0049070 con 0x55777fa41380 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.641+0000 7fa1a3fff640 1 -- 172.21.15.62:0/3009504498 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00038e0 con 0x55777fa41380 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.641+0000 7fa1c320e740 1 -- 172.21.15.62:0/3009504498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa41380 msgr2=0x55777fa4d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.641+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3009504498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa41380 0x55777fa4d9d0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0005ab0 tx=0x7fa1b0005a40 comp rx=0 tx=0).stop 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 -- 172.21.15.62:0/3009504498 shutdown_connections 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3009504498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa50150 0x55777fa67780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3009504498 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa41380 0x55777fa4d9d0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 --2- 172.21.15.62:0/3009504498 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa40a60 0x55777fa43100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 -- 172.21.15.62:0/3009504498 >> 172.21.15.62:0/3009504498 conn(0x55777fa3d0f0 msgr2=0x55777fa4f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 -- 172.21.15.62:0/3009504498 shutdown_connections 2024-09-15T23:31:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 -- 172.21.15.62:0/3009504498 wait complete. 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 -- start start 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 Processor -- start 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa40a60 0x55777fa54100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 0x55777fa54640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa50150 0x55777fa54b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa1bc1aafd0 con 0x55777fa50150 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1bc1ac440 con 0x55777fa40a60 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa1bc1ab6a0 con 0x55777fa41380 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1c320e740 10 client.0 _ll_register_callbacks cb 0x7fa1ac09bfa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1bad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa50150 0x55777fa54b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1bad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa50150 0x55777fa54b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48430/0 (socket says 172.21.15.62:48430) 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1bad9d640 1 -- 172.21.15.62:0/698060604 learned_addr learned my addr 172.21.15.62:0/698060604 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.642+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa40a60 0x55777fa54100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1b9d9b640 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 0x55777fa54640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1bad9d640 1 -- 172.21.15.62:0/698060604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 msgr2=0x55777fa54640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 0x55777fa54640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1bad9d640 1 -- 172.21.15.62:0/698060604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa40a60 msgr2=0x55777fa54100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa40a60 0x55777fa54100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1bad9d640 1 -- 172.21.15.62:0/698060604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55777fac9950 con 0x55777fa50150 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1bad9d640 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa50150 0x55777fa54b80 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4012850 tx=0x7fa1b401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0d918b4367d8d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/698060604 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4026070 con 0x55777fa50150 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1c320e740 1 -- 172.21.15.62:0/698060604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1b0049070 con 0x55777fa50150 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/698060604 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa1b4013b00 con 0x55777fa50150 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/698060604 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa1b4034050 con 0x55777fa50150 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1c320e740 10 client.18978 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.643+0000 7fa1c320e740 1 -- 172.21.15.62:0/698060604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55777facacc0 con 0x55777fa50150 2024-09-15T23:31:17.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.644+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/698060604 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1b0049070 con 0x55777fa50150 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.644+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/698060604 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7fa1b4021390 con 0x55777fa50150 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.644+0000 7fa1c320e740 10 client.18978 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.644+0000 7fa1c320e740 10 client.18978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.644+0000 7fa1c320e740 20 client.18978 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.644+0000 7fa1c320e740 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55777facb3c0 0x55777facd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.644+0000 7fa1c320e740 4 client.18978 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.644+0000 7fa1c320e740 1 -- 172.21.15.62:0/698060604 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55777f9add00 con 0x55777facb3c0 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.644+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55777facb3c0 0x55777facd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.645+0000 7fa1ba59c640 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55777facb3c0 0x55777facd7e0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=416eed752d94cc89 server_cookie=1186686fd84bd401 in_seq=0 out_seq=0 2024-09-15T23:31:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.645+0000 7fa1a1ffb640 10 client.18978 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:17.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1a1ffb640 1 -- 172.21.15.62:0/698060604 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55777f9add00 con 0x55777facb3c0 2024-09-15T23:31:17.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1a1ffb640 10 client.18978 handle_command_reply: tid=0 2024-09-15T23:31:17.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 client.18978 shutdown 2024-09-15T23:31:17.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 -- 172.21.15.62:0/698060604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa50150 msgr2=0x55777fa54b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:17.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa50150 0x55777fa54b80 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4012850 tx=0x7fa1b401a000 comp rx=0 tx=0).stop 2024-09-15T23:31:17.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 -- 172.21.15.62:0/698060604 shutdown_connections 2024-09-15T23:31:17.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x55777facb3c0 0x55777facd7e0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55777fa50150 0x55777fa54b80 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x55777fa41380 0x55777fa54640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 --2- 172.21.15.62:0/698060604 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55777fa40a60 0x55777fa54100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 -- 172.21.15.62:0/698060604 >> 172.21.15.62:0/698060604 conn(0x55777fa3d0f0 msgr2=0x55777fa3ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1a1ffb640 0 client.18978 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 -- 172.21.15.62:0/698060604 shutdown_connections 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 1 -- 172.21.15.62:0/698060604 wait complete. 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.646+0000 7fa1c320e740 20 client.18978 trim_cache size 0 max 16384 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:31:17.653 INFO:teuthology.orchestra.run.smithi062.stdout: "return_code": 0, 2024-09-15T23:31:17.654 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_tag": "21015795-fe3d-4418-a234-c746ddadeb0a", 2024-09-15T23:31:17.654 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": "asynchronous" 2024-09-15T23:31:17.654 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:31:17.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 -- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1a40a32c0 msgr2=0x7fa1a40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:17.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1a40a32c0 0x7fa1a40a5780 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4005fd0 tx=0x7fa1b4014ab0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 -- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc100e20 msgr2=0x7fa1bc198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:17.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc100e20 0x7fa1bc198440 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00c920 tx=0x7fa1ac00cde0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 -- 172.21.15.62:0/1856169247 shutdown_connections 2024-09-15T23:31:17.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1a40a32c0 0x7fa1a40a5780 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1bc102ef0 0x7fa1bc19cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1bc102570 0x7fa1bc198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 --2- 172.21.15.62:0/1856169247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1bc100e20 0x7fa1bc198440 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:17.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 -- 172.21.15.62:0/1856169247 >> 172.21.15.62:0/1856169247 conn(0x7fa1bc0fc7d0 msgr2=0x7fa1bc06a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:17.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 -- 172.21.15.62:0/1856169247 shutdown_connections 2024-09-15T23:31:17.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:17.647+0000 7fa1c0e35640 1 -- 172.21.15.62:0/1856169247 wait complete. 2024-09-15T23:31:18.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:18 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:31:18.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:18 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:18.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:18 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:31:18.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:18 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:18.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:18 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:31:18.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:18 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:18.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:31:19.254 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:19.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:19 smithi062 ceph-mon[30949]: scrub queued for path: / 2024-09-15T23:31:19.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:19 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:19 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:31:19.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:19 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:31:19.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:19 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:19 smithi062 ceph-mon[30949]: scrub summary: active paths [/] 2024-09-15T23:31:19.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:19 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:19 smithi062 ceph-mon[30949]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:19 smithi062 ceph-mon[30949]: pgmap v821: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.0 MiB/s wr, 297 op/s 2024-09-15T23:31:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:19 smithi067 ceph-mon[35665]: scrub queued for path: / 2024-09-15T23:31:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:19 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:19 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:31:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:19 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:31:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:19 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:19 smithi067 ceph-mon[35665]: scrub summary: active paths [/] 2024-09-15T23:31:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:19 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:19 smithi067 ceph-mon[35665]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:19 smithi067 ceph-mon[35665]: pgmap v821: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.0 MiB/s wr, 297 op/s 2024-09-15T23:31:19.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:19 smithi148 ceph-mon[35610]: scrub queued for path: / 2024-09-15T23:31:19.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:19 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:19 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:31:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:19 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:31:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:19 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:19 smithi148 ceph-mon[35610]: scrub summary: active paths [/] 2024-09-15T23:31:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:19 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:19 smithi148 ceph-mon[35610]: scrub summary: idle+waiting paths [/] 2024-09-15T23:31:19.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:19 smithi148 ceph-mon[35610]: pgmap v821: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 3.0 MiB/s wr, 297 op/s 2024-09-15T23:31:19.570 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 900.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:31:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.734+0000 7f0e17a5d640 1 -- 172.21.15.62:0/4208604121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 msgr2=0x7f0e101067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.734+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/4208604121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 0x7f0e101067f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c009900 tx=0x7f0e0c02a410 comp rx=0 tx=0).stop 2024-09-15T23:31:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.734+0000 7f0e17a5d640 1 -- 172.21.15.62:0/4208604121 shutdown_connections 2024-09-15T23:31:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.734+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/4208604121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10106f20 0x7f0e1010eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.734+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/4208604121 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 0x7f0e101067f0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.734+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/4208604121 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e10101df0 0x7f0e101021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.734+0000 7f0e17a5d640 1 -- 172.21.15.62:0/4208604121 >> 172.21.15.62:0/4208604121 conn(0x7f0e100fd8b0 msgr2=0x7f0e100ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:19.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.734+0000 7f0e17a5d640 1 -- 172.21.15.62:0/4208604121 shutdown_connections 2024-09-15T23:31:19.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.734+0000 7f0e17a5d640 1 -- 172.21.15.62:0/4208604121 wait complete. 2024-09-15T23:31:19.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e17a5d640 1 Processor -- start 2024-09-15T23:31:19.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e17a5d640 1 -- start start 2024-09-15T23:31:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e17a5d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10101df0 0x7f0e100744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e17a5d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 0x7f0e10074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e17a5d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e10106f20 0x7f0e10074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e17a5d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e10111f80 con 0x7f0e101027a0 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e17a5d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e10111e00 con 0x7f0e10101df0 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e17a5d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0e10112100 con 0x7f0e10106f20 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e14ff8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 0x7f0e10074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e14ff8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 0x7f0e10074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48446/0 (socket says 172.21.15.62:48446) 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e14ff8640 1 -- 172.21.15.62:0/1181786788 learned_addr learned my addr 172.21.15.62:0/1181786788 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e10106f20 0x7f0e10074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.735+0000 7f0e157f9640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10101df0 0x7f0e100744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e14ff8640 1 -- 172.21.15.62:0/1181786788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e10106f20 msgr2=0x7f0e10074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e10106f20 0x7f0e10074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e14ff8640 1 -- 172.21.15.62:0/1181786788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10101df0 msgr2=0x7f0e100744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10101df0 0x7f0e100744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e14ff8640 1 -- 172.21.15.62:0/1181786788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e101a6900 con 0x7f0e101027a0 2024-09-15T23:31:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e10106f20 0x7f0e10074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 0x7f0e10074a30 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c006010 tx=0x7f0e0c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e067fc640 1 -- 172.21.15.62:0/1181786788 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c03f030 con 0x7f0e101027a0 2024-09-15T23:31:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e067fc640 1 -- 172.21.15.62:0/1181786788 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e0c003ce0 con 0x7f0e101027a0 2024-09-15T23:31:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e067fc640 1 -- 172.21.15.62:0/1181786788 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c035070 con 0x7f0e101027a0 2024-09-15T23:31:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e17a5d640 1 -- 172.21.15.62:0/1181786788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e101a6b90 con 0x7f0e101027a0 2024-09-15T23:31:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.736+0000 7f0e17a5d640 1 -- 172.21.15.62:0/1181786788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e101a7070 con 0x7f0e101027a0 2024-09-15T23:31:19.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.738+0000 7f0e067fc640 1 -- 172.21.15.62:0/1181786788 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f0e0c042030 con 0x7f0e101027a0 2024-09-15T23:31:19.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e19e36740 1 Processor -- start 2024-09-15T23:31:19.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e19e36740 1 -- start start 2024-09-15T23:31:19.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafb4a0 0x559b0cafb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 0x559b0cb06140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb08a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e19e36740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559b0ca612e0 con 0x559b0cb06680 2024-09-15T23:31:19.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e19e36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559b0ca08ce0 con 0x559b0cafb4a0 2024-09-15T23:31:19.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e19e36740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559b0ca5d2a0 con 0x559b0cafbdc0 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e067fc640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0df80a3240 0x7f0df80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e14ff8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb08a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0df80a3240 0x7f0df80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e14ff8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb08a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48462/0 (socket says 172.21.15.62:48462) 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.739+0000 7f0e14ff8640 1 -- 172.21.15.62:0/756365465 learned_addr learned my addr 172.21.15.62:0/756365465 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/756365465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafb4a0 0x559b0cafb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e157f9640 1 --2- 172.21.15.62:0/756365465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 0x559b0cb06140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e067fc640 1 -- 172.21.15.62:0/1181786788 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0e0c0f0250 con 0x7f0e101027a0 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e14ff8640 1 -- 172.21.15.62:0/756365465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 msgr2=0x559b0cb06140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/756365465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 0x559b0cb06140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e14ff8640 1 -- 172.21.15.62:0/756365465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafb4a0 msgr2=0x559b0cafb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:19.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/756365465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafb4a0 0x559b0cafb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e14ff8640 1 -- 172.21.15.62:0/756365465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b0c9ed440 con 0x559b0cb06680 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0df80a3240 0x7f0df80a5700 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0e08005c40 tx=0x7f0e08009d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/756365465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb08a70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c0b89c0 tx=0x7f0e0c04b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75861807171047d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0df77fe640 1 -- 172.21.15.62:0/756365465 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c049070 con 0x559b0cb06680 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0df77fe640 1 -- 172.21.15.62:0/756365465 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e0c0bccb0 con 0x559b0cb06680 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.740+0000 7f0df77fe640 1 -- 172.21.15.62:0/756365465 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c0c5220 con 0x559b0cb06680 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 -- 172.21.15.62:0/756365465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 msgr2=0x559b0cb08a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 --2- 172.21.15.62:0/756365465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb08a70 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c0b89c0 tx=0x7f0e0c04b000 comp rx=0 tx=0).stop 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 -- 172.21.15.62:0/756365465 shutdown_connections 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 --2- 172.21.15.62:0/756365465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb08a70 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 --2- 172.21.15.62:0/756365465 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 0x559b0cb06140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 --2- 172.21.15.62:0/756365465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafb4a0 0x559b0cafb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 -- 172.21.15.62:0/756365465 >> 172.21.15.62:0/756365465 conn(0x559b0caf7a30 msgr2=0x559b0caf7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:19.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 -- 172.21.15.62:0/756365465 shutdown_connections 2024-09-15T23:31:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 -- 172.21.15.62:0/756365465 wait complete. 2024-09-15T23:31:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.741+0000 7f0e19e36740 1 -- start start 2024-09-15T23:31:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e19e36740 1 Processor -- start 2024-09-15T23:31:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb05fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 0x559b0cb03250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cb06680 0x559b0cb03790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e19e36740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x559b0cb09010 con 0x559b0cafb4a0 2024-09-15T23:31:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e19e36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x559b0ca1c050 con 0x559b0cb06680 2024-09-15T23:31:19.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e19e36740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x559b0cb09190 con 0x559b0cafbdc0 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e19e36740 10 client.0 _ll_register_callbacks cb 0x559b0ca5c160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e157f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb05fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e157f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb05fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48472/0 (socket says 172.21.15.62:48472) 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e157f9640 1 -- 172.21.15.62:0/263959688 learned_addr learned my addr 172.21.15.62:0/263959688 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 0x559b0cb03250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.742+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cb06680 0x559b0cb03790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0e157f9640 1 -- 172.21.15.62:0/263959688 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 msgr2=0x559b0cb03250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0e157f9640 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 0x559b0cb03250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0e157f9640 1 -- 172.21.15.62:0/263959688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cb06680 msgr2=0x559b0cb03790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0e157f9640 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cb06680 0x559b0cb03790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0e157f9640 1 -- 172.21.15.62:0/263959688 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b0c7d23c0 con 0x559b0cafb4a0 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0e157f9640 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb05fa0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f0e00011780 tx=0x7f0e00011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcee7880249c6c3a server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:19.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0df4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0df4ff9640 1 -- 172.21.15.62:0/263959688 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0001e060 con 0x559b0cafb4a0 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0df4ff9640 1 -- 172.21.15.62:0/263959688 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0e0000fdb0 con 0x559b0cafb4a0 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0e19e36740 1 -- 172.21.15.62:0/263959688 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e0c0bccb0 con 0x559b0cafb4a0 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0df4ff9640 1 -- 172.21.15.62:0/263959688 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0e00025050 con 0x559b0cafb4a0 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0e19e36740 10 client.18990 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0e19e36740 1 -- 172.21.15.62:0/263959688 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559b0cb148e0 con 0x559b0cafb4a0 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.743+0000 7f0df4ff9640 1 -- 172.21.15.62:0/263959688 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e0c0bccb0 con 0x559b0cafb4a0 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0df4ff9640 1 -- 172.21.15.62:0/263959688 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f0e00018030 con 0x559b0cafb4a0 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0e19e36740 10 client.18990 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0e19e36740 10 client.18990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:19.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0e19e36740 20 client.18990 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:19.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0e19e36740 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b0cb15da0 0x559b0cb18190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0e19e36740 4 client.18990 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:19.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0e19e36740 1 -- 172.21.15.62:0/263959688 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b0ca6e850 con 0x559b0cb15da0 2024-09-15T23:31:19.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b0cb15da0 0x559b0cb18190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0df4ff9640 10 client.18990 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.744+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b0cb15da0 0x559b0cb18190 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f52f7ecfada2e5c3 server_cookie=ec39183d7feecf4 in_seq=0 out_seq=0 2024-09-15T23:31:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0df4ff9640 1 -- 172.21.15.62:0/263959688 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559b0ca6e850 con 0x559b0cb15da0 2024-09-15T23:31:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0df4ff9640 10 client.18990 handle_command_reply: tid=0 2024-09-15T23:31:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 client.18990 shutdown 2024-09-15T23:31:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 -- 172.21.15.62:0/263959688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 msgr2=0x559b0cb05fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb05fa0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f0e00011780 tx=0x7f0e00011c40 comp rx=0 tx=0).stop 2024-09-15T23:31:19.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 -- 172.21.15.62:0/263959688 shutdown_connections 2024-09-15T23:31:19.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b0cb15da0 0x559b0cb18190 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cb06680 0x559b0cb03790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafbdc0 0x559b0cb03250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0df4ff9640 0 client.18990 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:19.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 --2- 172.21.15.62:0/263959688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb05fa0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 -- 172.21.15.62:0/263959688 >> 172.21.15.62:0/263959688 conn(0x559b0caf7a30 msgr2=0x559b0cafd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:19.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 -- 172.21.15.62:0/263959688 shutdown_connections 2024-09-15T23:31:19.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.745+0000 7f0e19e36740 1 -- 172.21.15.62:0/263959688 wait complete. 2024-09-15T23:31:19.753 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.746+0000 7f0e19e36740 20 client.18990 trim_cache size 0 max 16384 2024-09-15T23:31:19.768 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e19e36740 1 Processor -- start 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e19e36740 1 -- start start 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb12130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 0x559b0cb1da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cb06680 0x559b0cb1df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e19e36740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101b3700 con 0x559b0cafb4a0 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e19e36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101b0fa0 con 0x559b0cafbdc0 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e19e36740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101aedf0 con 0x559b0cb06680 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e157f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb12130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e157f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb12130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48488/0 (socket says 172.21.15.62:48488) 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e157f9640 1 -- 172.21.15.62:0/3358897457 learned_addr learned my addr 172.21.15.62:0/3358897457 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cb06680 0x559b0cb1df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 0x559b0cb1da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e157f9640 1 -- 172.21.15.62:0/3358897457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cb06680 msgr2=0x559b0cb1df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e157f9640 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cb06680 0x559b0cb1df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e157f9640 1 -- 172.21.15.62:0/3358897457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 msgr2=0x559b0cb1da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e157f9640 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 0x559b0cb1da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e157f9640 1 -- 172.21.15.62:0/3358897457 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b0cafb2d0 con 0x559b0cafb4a0 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.763+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 0x559b0cb1da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.764+0000 7f0e157f9640 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb12130 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0e00000c00 tx=0x7f0e00005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f75fa87f750726c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.764+0000 7f0df6ffd640 1 -- 172.21.15.62:0/3358897457 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0002d030 con 0x559b0cafb4a0 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.764+0000 7f0df6ffd640 1 -- 172.21.15.62:0/3358897457 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e0000a340 con 0x559b0cafb4a0 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.764+0000 7f0df6ffd640 1 -- 172.21.15.62:0/3358897457 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0000aa70 con 0x559b0cafb4a0 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.764+0000 7f0e19e36740 1 -- 172.21.15.62:0/3358897457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 msgr2=0x559b0cb12130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.764+0000 7f0e19e36740 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb12130 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0e00000c00 tx=0x7f0e00005670 comp rx=0 tx=0).stop 2024-09-15T23:31:19.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.765+0000 7f0e19e36740 1 -- 172.21.15.62:0/3358897457 shutdown_connections 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.765+0000 7f0e19e36740 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cb06680 0x559b0cb1df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.765+0000 7f0e19e36740 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 0x559b0cb1da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.765+0000 7f0e19e36740 1 --2- 172.21.15.62:0/3358897457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cafb4a0 0x559b0cb12130 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.765+0000 7f0e19e36740 1 -- 172.21.15.62:0/3358897457 >> 172.21.15.62:0/3358897457 conn(0x559b0caf7a30 msgr2=0x559b0cafe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.765+0000 7f0e19e36740 1 -- 172.21.15.62:0/3358897457 shutdown_connections 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.765+0000 7f0e19e36740 1 -- 172.21.15.62:0/3358897457 wait complete. 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.765+0000 7f0e19e36740 1 -- start start 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.765+0000 7f0e19e36740 1 Processor -- start 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafb4a0 0x559b0cb0a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 0x559b0cb0acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e19e36740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb7fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e19e36740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101af460 con 0x559b0cb06680 2024-09-15T23:31:19.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e19e36740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101b08d0 con 0x559b0cafbdc0 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e19e36740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0e101afb30 con 0x559b0cafb4a0 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e19e36740 10 client.0 _ll_register_callbacks cb 0x7f0e0000c690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e157f9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafb4a0 0x559b0cb0a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e15ffa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb7fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e157f9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafb4a0 0x559b0cb0a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:43022/0 (socket says 172.21.15.62:43022) 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e157f9640 1 -- 172.21.15.62:0/416915654 learned_addr learned my addr 172.21.15.62:0/416915654 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e14ff8640 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 0x559b0cb0acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e15ffa640 1 -- 172.21.15.62:0/416915654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafb4a0 msgr2=0x559b0cb0a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafb4a0 0x559b0cb0a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e15ffa640 1 -- 172.21.15.62:0/416915654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 msgr2=0x559b0cb0acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 0x559b0cb0acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e15ffa640 1 -- 172.21.15.62:0/416915654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b0cb80000 con 0x559b0cb06680 2024-09-15T23:31:19.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e157f9640 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafb4a0 0x559b0cb0a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0e15ffa640 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb7fa10 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x559b0caec4e0 tx=0x7f0e08005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482677be8a66692c server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0df4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.766+0000 7f0df4ff9640 1 -- 172.21.15.62:0/416915654 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e08004b20 con 0x559b0cb06680 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0df4ff9640 1 -- 172.21.15.62:0/416915654 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f0e0801b040 con 0x559b0cb06680 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0e19e36740 1 -- 172.21.15.62:0/416915654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e0000a340 con 0x559b0cb06680 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0df4ff9640 1 -- 172.21.15.62:0/416915654 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0e08034050 con 0x559b0cb06680 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0e19e36740 10 client.18996 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0e19e36740 1 -- 172.21.15.62:0/416915654 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559b0cb81220 con 0x559b0cb06680 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0df4ff9640 1 -- 172.21.15.62:0/416915654 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e0000a340 con 0x559b0cb06680 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0df4ff9640 1 -- 172.21.15.62:0/416915654 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f0e08029070 con 0x559b0cb06680 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0e19e36740 10 client.18996 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0e19e36740 10 client.18996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0e19e36740 20 client.18996 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:19.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0e19e36740 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b0cb819f0 0x559b0cb83e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:19.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0e19e36740 4 client.18996 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:31:19.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.767+0000 7f0e19e36740 1 -- 172.21.15.62:0/416915654 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559b0ca68d60 con 0x559b0cb819f0 2024-09-15T23:31:19.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e157f9640 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b0cb819f0 0x559b0cb83e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e157f9640 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b0cb819f0 0x559b0cb83e10 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfb5e0da1ca5f290 server_cookie=aa6abbf8610d3b54 in_seq=0 out_seq=0 2024-09-15T23:31:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0df4ff9640 10 client.18996 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0df4ff9640 1 -- 172.21.15.62:0/416915654 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559b0ca68d60 con 0x559b0cb819f0 2024-09-15T23:31:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0df4ff9640 10 client.18996 handle_command_reply: tid=0 2024-09-15T23:31:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e19e36740 1 client.18996 shutdown 2024-09-15T23:31:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e19e36740 1 -- 172.21.15.62:0/416915654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 msgr2=0x559b0cb7fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e19e36740 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb7fa10 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x559b0caec4e0 tx=0x7f0e08005bd0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e19e36740 1 -- 172.21.15.62:0/416915654 shutdown_connections 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e19e36740 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x559b0cb819f0 0x559b0cb83e10 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e19e36740 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559b0cb06680 0x559b0cb7fa10 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e19e36740 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559b0cafbdc0 0x559b0cb0acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.768+0000 7f0e19e36740 1 --2- 172.21.15.62:0/416915654 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x559b0cafb4a0 0x559b0cb0a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.769+0000 7f0e19e36740 1 -- 172.21.15.62:0/416915654 >> 172.21.15.62:0/416915654 conn(0x559b0caf7a30 msgr2=0x559b0cafcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.769+0000 7f0df4ff9640 0 client.18996 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.769+0000 7f0e19e36740 1 -- 172.21.15.62:0/416915654 shutdown_connections 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.769+0000 7f0e19e36740 1 -- 172.21.15.62:0/416915654 wait complete. 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.769+0000 7f0e19e36740 20 client.18996 trim_cache size 0 max 16384 2024-09-15T23:31:19.776 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:31:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-15T23:31:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": { 2024-09-15T23:31:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "21015795-fe3d-4418-a234-c746ddadeb0a": { 2024-09-15T23:31:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "path": "/", 2024-09-15T23:31:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "tag": "21015795-fe3d-4418-a234-c746ddadeb0a", 2024-09-15T23:31:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: "options": "recursive,force" 2024-09-15T23:31:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:19.777 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:19.777 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.769+0000 7f0e17a5d640 1 -- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0df80a3240 msgr2=0x7f0df80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.769+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0df80a3240 0x7f0df80a5700 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f0e08005c40 tx=0x7f0e08009d50 comp rx=0 tx=0).stop 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.769+0000 7f0e17a5d640 1 -- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 msgr2=0x7f0e10074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.769+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 0x7f0e10074a30 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c006010 tx=0x7f0e0c0042f0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.770+0000 7f0e17a5d640 1 -- 172.21.15.62:0/1181786788 shutdown_connections 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.770+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f0df80a3240 0x7f0df80a5700 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.770+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0e10106f20 0x7f0e10074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.770+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e101027a0 0x7f0e10074a30 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.770+0000 7f0e17a5d640 1 --2- 172.21.15.62:0/1181786788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e10101df0 0x7f0e100744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:19.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.770+0000 7f0e17a5d640 1 -- 172.21.15.62:0/1181786788 >> 172.21.15.62:0/1181786788 conn(0x7f0e100fd8b0 msgr2=0x7f0e100febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:19.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.770+0000 7f0e17a5d640 1 -- 172.21.15.62:0/1181786788 shutdown_connections 2024-09-15T23:31:19.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:19.770+0000 7f0e17a5d640 1 -- 172.21.15.62:0/1181786788 wait complete. 2024-09-15T23:31:20.454 INFO:tasks.cephfs.filesystem:scrub status for tag:21015795-fe3d-4418-a234-c746ddadeb0a - {'path': '/', 'tag': '21015795-fe3d-4418-a234-c746ddadeb0a', 'options': 'recursive,force'} 2024-09-15T23:31:21.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:21 smithi067 ceph-mon[35665]: pgmap v822: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 2.9 MiB/s wr, 337 op/s 2024-09-15T23:31:21.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:21 smithi148 ceph-mon[35610]: pgmap v822: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 2.9 MiB/s wr, 337 op/s 2024-09-15T23:31:21.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:21 smithi062 ceph-mon[30949]: pgmap v822: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 2.9 MiB/s wr, 337 op/s 2024-09-15T23:31:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.527+0000 7f7f8a16d640 1 -- 172.21.15.62:0/307705650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f84102d30 msgr2=0x7f7f841031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.527+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/307705650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f84102d30 0x7f7f841031b0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c009900 tx=0x7f7f6c02a410 comp rx=0 tx=0).stop 2024-09-15T23:31:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.527+0000 7f7f8a16d640 1 -- 172.21.15.62:0/307705650 shutdown_connections 2024-09-15T23:31:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.527+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/307705650 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f841036f0 0x7f7f84109f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.527+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/307705650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f84102d30 0x7f7f841031b0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.527+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/307705650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f84101b40 0x7f7f84101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.527+0000 7f7f8a16d640 1 -- 172.21.15.62:0/307705650 >> 172.21.15.62:0/307705650 conn(0x7f7f840fd2e0 msgr2=0x7f7f840ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.527+0000 7f7f8a16d640 1 -- 172.21.15.62:0/307705650 shutdown_connections 2024-09-15T23:31:22.530 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.527+0000 7f7f8a16d640 1 -- 172.21.15.62:0/307705650 wait complete. 2024-09-15T23:31:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f8a16d640 1 Processor -- start 2024-09-15T23:31:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f8a16d640 1 -- start start 2024-09-15T23:31:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f8a16d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f84101b40 0x7f7f84197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f8a16d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f84102d30 0x7f7f84198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f8a16d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f841036f0 0x7f7f8419f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f8a16d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8410d7f0 con 0x7f7f841036f0 2024-09-15T23:31:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f8a16d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8410d670 con 0x7f7f84101b40 2024-09-15T23:31:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f8a16d640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8410d970 con 0x7f7f84102d30 2024-09-15T23:31:22.531 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f83fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f841036f0 0x7f7f8419f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f83fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f841036f0 0x7f7f8419f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38940/0 (socket says 172.21.15.62:38940) 2024-09-15T23:31:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.528+0000 7f7f83fff640 1 -- 172.21.15.62:0/3264701006 learned_addr learned my addr 172.21.15.62:0/3264701006 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f837fe640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f84101b40 0x7f7f84197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f84102d30 0x7f7f84198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f83fff640 1 -- 172.21.15.62:0/3264701006 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f84102d30 msgr2=0x7f7f84198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f83fff640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f84102d30 0x7f7f84198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f83fff640 1 -- 172.21.15.62:0/3264701006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f84101b40 msgr2=0x7f7f84197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f83fff640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f84101b40 0x7f7f84197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.533 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f83fff640 1 -- 172.21.15.62:0/3264701006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f8419f970 con 0x7f7f841036f0 2024-09-15T23:31:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f83fff640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f841036f0 0x7f7f8419f300 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00cad0 tx=0x7f7f7c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f817fa640 1 -- 172.21.15.62:0/3264701006 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c00ae20 con 0x7f7f841036f0 2024-09-15T23:31:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f817fa640 1 -- 172.21.15.62:0/3264701006 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f7c005020 con 0x7f7f841036f0 2024-09-15T23:31:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f817fa640 1 -- 172.21.15.62:0/3264701006 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c00f6f0 con 0x7f7f841036f0 2024-09-15T23:31:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f8a16d640 1 -- 172.21.15.62:0/3264701006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f8419fc60 con 0x7f7f841036f0 2024-09-15T23:31:22.534 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.529+0000 7f7f8a16d640 1 -- 172.21.15.62:0/3264701006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f841a0130 con 0x7f7f841036f0 2024-09-15T23:31:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f817fa640 1 -- 172.21.15.62:0/3264701006 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7f7c0051d0 con 0x7f7f841036f0 2024-09-15T23:31:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f8cfeb740 1 Processor -- start 2024-09-15T23:31:22.535 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f8cfeb740 1 -- start start 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b097db0 0x56263b0981b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b0986f0 0x56263b09d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b09d780 0x56263b09fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56263aff72f0 con 0x56263b097db0 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56263af9f270 con 0x56263b09d780 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56263aff32b0 con 0x56263b0986f0 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f7affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b097db0 0x56263b0981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f7affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b097db0 0x56263b0981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38950/0 (socket says 172.21.15.62:38950) 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.532+0000 7f7f7affd640 1 -- 172.21.15.62:0/3815019199 learned_addr learned my addr 172.21.15.62:0/3815019199 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f837fe640 1 --2- 172.21.15.62:0/3815019199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b0986f0 0x56263b09d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.539 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3815019199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b09d780 0x56263b09fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f817fa640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7f700a32f0 0x7f7f700a57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f7affd640 1 -- 172.21.15.62:0/3815019199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b0986f0 msgr2=0x56263b09d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3815019199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b0986f0 0x56263b09d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f7affd640 1 -- 172.21.15.62:0/3815019199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b09d780 msgr2=0x56263b09fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3815019199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b09d780 0x56263b09fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f7affd640 1 -- 172.21.15.62:0/3815019199 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56263af9e340 con 0x56263b097db0 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f83fff640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7f700a32f0 0x7f7f700a57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3815019199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b097db0 0x56263b0981b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c005ab0 tx=0x7f7f6c005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9cf402023cf21f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f817fa640 1 -- 172.21.15.62:0/3264701006 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7f7c0cfe00 con 0x7f7f841036f0 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f79ffb640 1 -- 172.21.15.62:0/3815019199 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c043030 con 0x56263b097db0 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f79ffb640 1 -- 172.21.15.62:0/3815019199 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f6c005500 con 0x56263b097db0 2024-09-15T23:31:22.540 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.533+0000 7f7f79ffb640 1 -- 172.21.15.62:0/3815019199 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c035050 con 0x56263b097db0 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3815019199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b097db0 msgr2=0x56263b0981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f83fff640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7f700a32f0 0x7f7f700a57b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c09c030 tx=0x7f7f7c066a30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3815019199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b097db0 0x56263b0981b0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c005ab0 tx=0x7f7f6c005a20 comp rx=0 tx=0).stop 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3815019199 shutdown_connections 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3815019199 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b09d780 0x56263b09fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3815019199 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b0986f0 0x56263b09d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3815019199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b097db0 0x56263b0981b0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3815019199 >> 172.21.15.62:0/3815019199 conn(0x56263b093420 msgr2=0x56263b093820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3815019199 shutdown_connections 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.534+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3815019199 wait complete. 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f8cfeb740 1 -- start start 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f8cfeb740 1 Processor -- start 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0a2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.541 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56263b0a06f0 con 0x56263b09d780 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56263af80290 con 0x56263b0986f0 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56263b0a0870 con 0x56263b097db0 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f8cfeb740 10 client.0 _ll_register_callbacks cb 0x56263afb1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f7affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f7affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38952/0 (socket says 172.21.15.62:38952) 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f7affd640 1 -- 172.21.15.62:0/3761092836 learned_addr learned my addr 172.21.15.62:0/3761092836 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f837fe640 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.535+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0a2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f7affd640 1 -- 172.21.15.62:0/3761092836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 msgr2=0x56263b0a2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0a2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f7affd640 1 -- 172.21.15.62:0/3761092836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 msgr2=0x56263b0a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f7affd640 1 -- 172.21.15.62:0/3761092836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56263ad0a210 con 0x56263b09d780 2024-09-15T23:31:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a9510 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c031610 tx=0x7f7f6c03f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3a3469fba626be server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:22.543 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f62ffd640 1 -- 172.21.15.62:0/3761092836 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c043030 con 0x56263b09d780 2024-09-15T23:31:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f62ffd640 1 -- 172.21.15.62:0/3761092836 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7f6c0317f0 con 0x56263b09d780 2024-09-15T23:31:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3761092836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f6c005500 con 0x56263b09d780 2024-09-15T23:31:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f62ffd640 1 -- 172.21.15.62:0/3761092836 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7f6c05a050 con 0x56263b09d780 2024-09-15T23:31:22.545 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f8cfeb740 10 client.19008 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.536+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3761092836 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56263add6620 con 0x56263b09d780 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.537+0000 7f7f62ffd640 1 -- 172.21.15.62:0/3761092836 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f6c005500 con 0x56263b09d780 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.537+0000 7f7f62ffd640 1 -- 172.21.15.62:0/3761092836 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f7f6c003bc0 con 0x56263b09d780 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.537+0000 7f7f8cfeb740 10 client.19008 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.537+0000 7f7f8cfeb740 10 client.19008 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.537+0000 7f7f8cfeb740 20 client.19008 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.537+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56263b0ad810 0x56263b0afc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.537+0000 7f7f8cfeb740 4 client.19008 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.537+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3761092836 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56263b004a20 con 0x56263b0ad810 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.537+0000 7f7f837fe640 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56263b0ad810 0x56263b0afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.538+0000 7f7f62ffd640 10 client.19008 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:22.546 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.538+0000 7f7f837fe640 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56263b0ad810 0x56263b0afc30 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1be3d493a5d1742 server_cookie=bd0b16ba51d8d083 in_seq=0 out_seq=0 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.538+0000 7f7f62ffd640 1 -- 172.21.15.62:0/3761092836 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56263b004a20 con 0x56263b0ad810 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.538+0000 7f7f62ffd640 10 client.19008 handle_command_reply: tid=0 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.538+0000 7f7f8cfeb740 1 client.19008 shutdown 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.538+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3761092836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 msgr2=0x56263b0a9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.538+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a9510 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c031610 tx=0x7f7f6c03f040 comp rx=0 tx=0).stop 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.538+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3761092836 shutdown_connections 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.538+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56263b0ad810 0x56263b0afc30 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.539+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a9510 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.539+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.539+0000 7f7f62ffd640 0 client.19008 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.539+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3761092836 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.539+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3761092836 >> 172.21.15.62:0/3761092836 conn(0x56263b093420 msgr2=0x56263b08e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.539+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3761092836 shutdown_connections 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.539+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3761092836 wait complete. 2024-09-15T23:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.539+0000 7f7f8cfeb740 20 client.19008 trim_cache size 0 max 16384 2024-09-15T23:31:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.553+0000 7f7f8cfeb740 1 Processor -- start 2024-09-15T23:31:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.553+0000 7f7f8cfeb740 1 -- start start 2024-09-15T23:31:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0943b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0948f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7f841aabd0 con 0x56263b09d780 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f841a8470 con 0x56263b0986f0 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7f841a62c0 con 0x56263b097db0 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f7affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f7affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0948f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38962/0 (socket says 172.21.15.62:38962) 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f7affd640 1 -- 172.21.15.62:0/3488121044 learned_addr learned my addr 172.21.15.62:0/3488121044 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f837fe640 1 --2- 172.21.15.62:0/3488121044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f7affd640 1 -- 172.21.15.62:0/3488121044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 msgr2=0x56263b0943b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3488121044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0943b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f7affd640 1 -- 172.21.15.62:0/3488121044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 msgr2=0x56263b0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3488121044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f7affd640 1 -- 172.21.15.62:0/3488121044 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56263b093240 con 0x56263b09d780 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f837fe640 1 --2- 172.21.15.62:0/3488121044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f7affd640 1 --2- 172.21.15.62:0/3488121044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0948f0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c031440 tx=0x7f7f6c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2206146ff2379ced server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f797fa640 1 -- 172.21.15.62:0/3488121044 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c043040 con 0x56263b09d780 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f797fa640 1 -- 172.21.15.62:0/3488121044 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f6c004c50 con 0x56263b09d780 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.554+0000 7f7f797fa640 1 -- 172.21.15.62:0/3488121044 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c035050 con 0x56263b09d780 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3488121044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 msgr2=0x56263b0948f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3488121044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0948f0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c031440 tx=0x7f7f6c005e80 comp rx=0 tx=0).stop 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3488121044 shutdown_connections 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3488121044 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0948f0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3488121044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/3488121044 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0943b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3488121044 >> 172.21.15.62:0/3488121044 conn(0x56263b093420 msgr2=0x56263b0a9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3488121044 shutdown_connections 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/3488121044 wait complete. 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.555+0000 7f7f8cfeb740 1 -- start start 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f8cfeb740 1 Processor -- start 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f8cfeb740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7f841a6930 con 0x56263b09d780 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7f841a7da0 con 0x56263b0986f0 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f8cfeb740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7f841a7000 con 0x56263b097db0 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f8cfeb740 10 client.0 _ll_register_callbacks cb 0x7f7f6c004440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f7affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f837fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:41780/0 (socket says 172.21.15.62:41780) 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f837fe640 1 -- 172.21.15.62:0/1708720599 learned_addr learned my addr 172.21.15.62:0/1708720599 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f7affd640 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0a69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f837fe640 1 -- 172.21.15.62:0/1708720599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 msgr2=0x56263b0a69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f837fe640 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0a69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f837fe640 1 -- 172.21.15.62:0/1708720599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 msgr2=0x56263b0a7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f837fe640 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f837fe640 1 -- 172.21.15.62:0/1708720599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56263b0a8520 con 0x56263b0986f0 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.556+0000 7f7f7affd640 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f837fe640 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a6f00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x56263b0942a0 tx=0x7f7f74011b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed9005a65e899b72 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f62ffd640 1 -- 172.21.15.62:0/1708720599 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74010840 con 0x56263b0986f0 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/1708720599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f6c004c50 con 0x56263b0986f0 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f62ffd640 1 -- 172.21.15.62:0/1708720599 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f7f740109e0 con 0x56263b0986f0 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f62ffd640 1 -- 172.21.15.62:0/1708720599 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7f7402e050 con 0x56263b0986f0 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f8cfeb740 10 client.28496 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/1708720599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56263b1184b0 con 0x56263b0986f0 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f62ffd640 1 -- 172.21.15.62:0/1708720599 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f6c004c50 con 0x56263b0986f0 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.557+0000 7f7f62ffd640 1 -- 172.21.15.62:0/1708720599 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f7f7401c410 con 0x56263b0986f0 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.558+0000 7f7f8cfeb740 10 client.28496 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.558+0000 7f7f8cfeb740 10 client.28496 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.558+0000 7f7f8cfeb740 20 client.28496 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.558+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56263b118d20 0x56263b11b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.558+0000 7f7f8cfeb740 4 client.28496 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.558+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/1708720599 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56263affed70 con 0x56263b118d20 2024-09-15T23:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.558+0000 7f7f7affd640 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56263b118d20 0x56263b11b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.558+0000 7f7f62ffd640 10 client.28496 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.558+0000 7f7f7affd640 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56263b118d20 0x56263b11b140 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66f6443748198ec1 server_cookie=86118b933b80e68e in_seq=0 out_seq=0 2024-09-15T23:31:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f62ffd640 1 -- 172.21.15.62:0/1708720599 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56263affed70 con 0x56263b118d20 2024-09-15T23:31:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f62ffd640 10 client.28496 handle_command_reply: tid=0 2024-09-15T23:31:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 client.28496 shutdown 2024-09-15T23:31:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/1708720599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 msgr2=0x56263b0a6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a6f00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x56263b0942a0 tx=0x7f7f74011b00 comp rx=0 tx=0).stop 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/1708720599 shutdown_connections 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x56263b118d20 0x56263b11b140 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56263b09d780 0x56263b0a7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56263b0986f0 0x56263b0a6f00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 --2- 172.21.15.62:0/1708720599 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56263b097db0 0x56263b0a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/1708720599 >> 172.21.15.62:0/1708720599 conn(0x56263b093420 msgr2=0x56263b08d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f62ffd640 0 client.28496 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/1708720599 shutdown_connections 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 1 -- 172.21.15.62:0/1708720599 wait complete. 2024-09-15T23:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.572+0000 7f7f8cfeb740 20 client.28496 trim_cache size 0 max 16384 2024-09-15T23:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 -- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7f700a32f0 msgr2=0x7f7f700a57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7f700a32f0 0x7f7f700a57b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c09c030 tx=0x7f7f7c066a30 comp rx=0 tx=0).stop 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 -- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f841036f0 msgr2=0x7f7f8419f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f841036f0 0x7f7f8419f300 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c00cad0 tx=0x7f7f7c00cf90 comp rx=0 tx=0).stop 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 -- 172.21.15.62:0/3264701006 shutdown_connections 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f7f700a32f0 0x7f7f700a57b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7f841036f0 0x7f7f8419f300 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7f84102d30 0x7f7f84198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 --2- 172.21.15.62:0/3264701006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f84101b40 0x7f7f84197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 -- 172.21.15.62:0/3264701006 >> 172.21.15.62:0/3264701006 conn(0x7f7f840fd2e0 msgr2=0x7f7f84105fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 -- 172.21.15.62:0/3264701006 shutdown_connections 2024-09-15T23:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:22.573+0000 7f7f8a16d640 1 -- 172.21.15.62:0/3264701006 wait complete. 2024-09-15T23:31:23.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:23 smithi067 ceph-mon[35665]: pgmap v823: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 2.6 MiB/s wr, 335 op/s 2024-09-15T23:31:23.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:23 smithi148 ceph-mon[35610]: pgmap v823: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 2.6 MiB/s wr, 335 op/s 2024-09-15T23:31:23.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:23 smithi062 ceph-mon[30949]: pgmap v823: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 2.6 MiB/s wr, 335 op/s 2024-09-15T23:31:24.574 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 905.0 s, 0.0 tps, lat 0.000 ms stddev 0.000 2024-09-15T23:31:25.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:25 smithi067 ceph-mon[35665]: pgmap v824: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.1 MiB/s wr, 282 op/s 2024-09-15T23:31:25.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:25 smithi148 ceph-mon[35610]: pgmap v824: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.1 MiB/s wr, 282 op/s 2024-09-15T23:31:25.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:25 smithi062 ceph-mon[30949]: pgmap v824: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.1 MiB/s wr, 282 op/s 2024-09-15T23:31:25.812 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:31:26.085 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:26.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.583+0000 7f8686adf640 1 -- 172.21.15.62:0/1841216976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86801082a0 msgr2=0x7f868010ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.583+0000 7f8686adf640 1 --2- 172.21.15.62:0/1841216976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86801082a0 0x7f868010ec40 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f8670009930 tx=0x7f867002a420 comp rx=0 tx=0).stop 2024-09-15T23:31:26.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.583+0000 7f8686adf640 1 -- 172.21.15.62:0/1841216976 shutdown_connections 2024-09-15T23:31:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.583+0000 7f8686adf640 1 --2- 172.21.15.62:0/1841216976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86801082a0 0x7f868010ec40 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.583+0000 7f8686adf640 1 --2- 172.21.15.62:0/1841216976 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86801036e0 0x7f8680107b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.583+0000 7f8686adf640 1 --2- 172.21.15.62:0/1841216976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8680102d30 0x7f8680103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.583+0000 7f8686adf640 1 -- 172.21.15.62:0/1841216976 >> 172.21.15.62:0/1841216976 conn(0x7f86800fc7d0 msgr2=0x7f86800febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.583+0000 7f8686adf640 1 -- 172.21.15.62:0/1841216976 shutdown_connections 2024-09-15T23:31:26.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.583+0000 7f8686adf640 1 -- 172.21.15.62:0/1841216976 wait complete. 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.584+0000 7f8686adf640 1 Processor -- start 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.584+0000 7f8686adf640 1 -- start start 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.584+0000 7f8686adf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8680102d30 0x7f8680198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.584+0000 7f8686adf640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86801036e0 0x7f8680198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.584+0000 7f8686adf640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86801082a0 0x7f868019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.584+0000 7f8686adf640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f86801120a0 con 0x7f8680102d30 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.584+0000 7f8686adf640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8680111f20 con 0x7f86801082a0 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.584+0000 7f8686adf640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8680112220 con 0x7f86801036e0 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868487b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8680102d30 0x7f8680198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868487b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8680102d30 0x7f8680198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:38984/0 (socket says 172.21.15.62:38984) 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868487b640 1 -- 172.21.15.62:0/3968147842 learned_addr learned my addr 172.21.15.62:0/3968147842 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:26.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f867ffff640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86801036e0 0x7f8680198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868507c640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86801082a0 0x7f868019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868487b640 1 -- 172.21.15.62:0/3968147842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86801036e0 msgr2=0x7f8680198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868487b640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86801036e0 0x7f8680198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868487b640 1 -- 172.21.15.62:0/3968147842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86801082a0 msgr2=0x7f868019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868487b640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86801082a0 0x7f868019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868487b640 1 -- 172.21.15.62:0/3968147842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868019d510 con 0x7f8680102d30 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868507c640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86801082a0 0x7f868019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f868487b640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8680102d30 0x7f8680198560 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f867800d950 tx=0x7f867800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f867dffb640 1 -- 172.21.15.62:0/3968147842 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8678017030 con 0x7f8680102d30 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f868019d800 con 0x7f8680102d30 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.585+0000 7f867dffb640 1 -- 172.21.15.62:0/3968147842 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8678004020 con 0x7f8680102d30 2024-09-15T23:31:26.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.586+0000 7f867dffb640 1 -- 172.21.15.62:0/3968147842 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86780047a0 con 0x7f8680102d30 2024-09-15T23:31:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.586+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86801a50e0 con 0x7f8680102d30 2024-09-15T23:31:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.587+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8680103110 con 0x7f8680102d30 2024-09-15T23:31:26.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.588+0000 7f867dffb640 1 -- 172.21.15.62:0/3968147842 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f86780041d0 con 0x7f8680102d30 2024-09-15T23:31:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.588+0000 7f867dffb640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f866c0a3240 0x7f866c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.589+0000 7f867ffff640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f866c0a3240 0x7f866c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.589+0000 7f867dffb640 1 -- 172.21.15.62:0/3968147842 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f86780cf680 con 0x7f8680102d30 2024-09-15T23:31:26.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.589+0000 7f867ffff640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f866c0a3240 0x7f866c0a5700 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f86800fee00 tx=0x7f866800a570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:26.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.593+0000 7f867dffb640 1 -- 172.21.15.62:0/3968147842 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f867809c020 con 0x7f8680102d30 2024-09-15T23:31:26.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.804+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86801998c0 con 0x7f8680102d30 2024-09-15T23:31:26.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.805+0000 7f867dffb640 1 -- 172.21.15.62:0/3968147842 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7f867809b920 con 0x7f8680102d30 2024-09-15T23:31:26.806 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:26.806 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:31:26.807 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f866c0a3240 msgr2=0x7f866c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f866c0a3240 0x7f866c0a5700 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f86800fee00 tx=0x7f866800a570 comp rx=0 tx=0).stop 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8680102d30 msgr2=0x7f8680198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8680102d30 0x7f8680198560 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f867800d950 tx=0x7f867800de10 comp rx=0 tx=0).stop 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 shutdown_connections 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f866c0a3240 0x7f866c0a5700 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86801082a0 0x7f868019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f86801036e0 0x7f8680198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 --2- 172.21.15.62:0/3968147842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8680102d30 0x7f8680198560 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 >> 172.21.15.62:0/3968147842 conn(0x7f86800fc7d0 msgr2=0x7f8680105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.807+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 shutdown_connections 2024-09-15T23:31:26.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:26.808+0000 7f8686adf640 1 -- 172.21.15.62:0/3968147842 wait complete. 2024-09-15T23:31:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:27 smithi067 ceph-mon[35665]: pgmap v825: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.4 MiB/s wr, 212 op/s 2024-09-15T23:31:27.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:27 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3968147842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:27.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:27 smithi062 ceph-mon[30949]: pgmap v825: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.4 MiB/s wr, 212 op/s 2024-09-15T23:31:27.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:27 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3968147842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:27 smithi148 ceph-mon[35610]: pgmap v825: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.4 MiB/s wr, 212 op/s 2024-09-15T23:31:27.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:27 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3968147842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:27.571 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --set-id 7334f5cd --release --await' 2024-09-15T23:31:27.572 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --set-id 7334f5cd --release --await 2024-09-15T23:31:27.832 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2763531732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc101de0 msgr2=0x7ff6cc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2763531732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc101de0 0x7ff6cc1021c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009900 tx=0x7ff6bc02a410 comp rx=0 tx=0).stop 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2763531732 shutdown_connections 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2763531732 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6cc106f10 0x7ff6cc10ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2763531732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6cc102790 0x7ff6cc1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2763531732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc101de0 0x7ff6cc1021c0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2763531732 >> 172.21.15.62:0/2763531732 conn(0x7ff6cc0fd8c0 msgr2=0x7ff6cc0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2763531732 shutdown_connections 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2763531732 wait complete. 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.291+0000 7ff6d1df4640 1 Processor -- start 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6d1df4640 1 -- start start 2024-09-15T23:31:28.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6d1df4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6cc101de0 0x7ff6cc198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6d1df4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc102790 0x7ff6cc1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6d1df4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6cc106f10 0x7ff6cc19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6d1df4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc111f70 con 0x7ff6cc102790 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6d1df4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc111df0 con 0x7ff6cc101de0 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6d1df4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1120f0 con 0x7ff6cc106f10 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc102790 0x7ff6cc1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc102790 0x7ff6cc1989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39000/0 (socket says 172.21.15.62:39000) 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6c3fff640 1 -- 172.21.15.62:0/2668994842 learned_addr learned my addr 172.21.15.62:0/2668994842 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6cc101de0 0x7ff6cc198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6cbfff640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6cc106f10 0x7ff6cc19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6c3fff640 1 -- 172.21.15.62:0/2668994842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6cc106f10 msgr2=0x7ff6cc19cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6cc106f10 0x7ff6cc19cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6c3fff640 1 -- 172.21.15.62:0/2668994842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6cc101de0 msgr2=0x7ff6cc198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6cc101de0 0x7ff6cc198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6c3fff640 1 -- 172.21.15.62:0/2668994842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6cc19d420 con 0x7ff6cc102790 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.292+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc102790 0x7ff6cc1989b0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400d950 tx=0x7ff6b400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.293+0000 7ff6c97fa640 1 -- 172.21.15.62:0/2668994842 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4017070 con 0x7ff6cc102790 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.293+0000 7ff6c97fa640 1 -- 172.21.15.62:0/2668994842 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b4009c30 con 0x7ff6cc102790 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.293+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2668994842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6cc19d710 con 0x7ff6cc102790 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.293+0000 7ff6c97fa640 1 -- 172.21.15.62:0/2668994842 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b400f6b0 con 0x7ff6cc102790 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.293+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2668994842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6cc1a4ff0 con 0x7ff6cc102790 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.295+0000 7ff6c97fa640 1 -- 172.21.15.62:0/2668994842 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff6b40109d0 con 0x7ff6cc102790 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.295+0000 7ff6d41cd740 1 Processor -- start 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.295+0000 7ff6d41cd740 1 -- start start 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.295+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045f3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 0x556e045fdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045fe240 0x556e04600630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556e04558620 con 0x556e045f3060 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556e044ffe50 con 0x556e045f3980 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556e045545e0 con 0x556e045fe240 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045f3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39006/0 (socket says 172.21.15.62:39006) 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cbfff640 1 -- 172.21.15.62:0/1301118388 learned_addr learned my addr 172.21.15.62:0/1301118388 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 0x556e045fdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045fe240 0x556e04600630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6c97fa640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff6b80a3240 0x7ff6b80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/1301118388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045fe240 msgr2=0x556e04600630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045fe240 0x556e04600630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/1301118388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 msgr2=0x556e045f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/1301118388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e044f0d40 con 0x556e045f3980 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cbfff640 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cbfff640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff6b80a3240 0x7ff6b80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6c97fa640 1 -- 172.21.15.62:0/2668994842 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff6b40cf920 con 0x7ff6cc102790 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.296+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 0x556e045fdd00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc005940 tx=0x7ff6bc005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=857caa6ae14016c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6cbfff640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff6b80a3240 0x7ff6b80a5700 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009370 tx=0x7ff6c4004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6c1ffb640 1 -- 172.21.15.62:0/1301118388 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc040070 con 0x556e045f3980 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6c1ffb640 1 -- 172.21.15.62:0/1301118388 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc0319f0 con 0x556e045f3980 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6c1ffb640 1 -- 172.21.15.62:0/1301118388 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc035040 con 0x556e045f3980 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6d41cd740 1 -- 172.21.15.62:0/1301118388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 msgr2=0x556e045fdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 0x556e045fdd00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc005940 tx=0x7ff6bc005e80 comp rx=0 tx=0).stop 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6d41cd740 1 -- 172.21.15.62:0/1301118388 shutdown_connections 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045fe240 0x556e04600630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 0x556e045fdd00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/1301118388 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045f3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6d41cd740 1 -- 172.21.15.62:0/1301118388 >> 172.21.15.62:0/1301118388 conn(0x556e045ef5f0 msgr2=0x556e045ef9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.297+0000 7ff6d41cd740 1 -- 172.21.15.62:0/1301118388 shutdown_connections 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 1 -- 172.21.15.62:0/1301118388 wait complete. 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 1 -- start start 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 1 Processor -- start 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3060 0x556e045fb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3980 0x556e045fb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 0x556e045fbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x556e04600e10 con 0x556e045f3980 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x556e044e0f10 con 0x556e045fe240 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x556e04600f90 con 0x556e045f3060 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6d41cd740 10 client.0 _ll_register_callbacks cb 0x556e04504db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:28.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3980 0x556e045fb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.298+0000 7ff6c3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3980 0x556e045fb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39014/0 (socket says 172.21.15.62:39014) 2024-09-15T23:31:28.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6c3fff640 1 -- 172.21.15.62:0/2625447010 learned_addr learned my addr 172.21.15.62:0/2625447010 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:28.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6cbfff640 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 0x556e045fbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3060 0x556e045fb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6c3fff640 1 -- 172.21.15.62:0/2625447010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3060 msgr2=0x556e045fb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3060 0x556e045fb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6c3fff640 1 -- 172.21.15.62:0/2625447010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 msgr2=0x556e045fbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 0x556e045fbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6c3fff640 1 -- 172.21.15.62:0/2625447010 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e04267580 con 0x556e045f3980 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3980 0x556e045fb5f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4098590 tx=0x7ff6b409b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395a32703be2e426 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2625447010 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4020040 con 0x556e045f3980 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2625447010 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff6b40860d0 con 0x556e045f3980 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2625447010 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6bc0319f0 con 0x556e045f3980 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2625447010 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff6b4009de0 con 0x556e045f3980 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6d41cd740 10 client.19032 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.299+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2625447010 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556e04331960 con 0x556e045f3980 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.300+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2625447010 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc0319f0 con 0x556e045f3980 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.300+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2625447010 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7ff6b409b010 con 0x556e045f3980 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.300+0000 7ff6d41cd740 10 client.19032 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.300+0000 7ff6d41cd740 10 client.19032 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.300+0000 7ff6d41cd740 20 client.19032 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.300+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556e0460dd90 0x556e04610180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.300+0000 7ff6d41cd740 4 client.19032 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:28.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.300+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2625447010 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e04565b90 con 0x556e0460dd90 2024-09-15T23:31:28.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.300+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556e0460dd90 0x556e04610180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.301+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556e0460dd90 0x556e04610180 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62ad6b8bc42b2649 server_cookie=4c8b7b6200762571 in_seq=0 out_seq=0 2024-09-15T23:31:28.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.301+0000 7ff6aaffd640 10 client.19032 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.301+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2625447010 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556e04565b90 con 0x556e0460dd90 2024-09-15T23:31:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.301+0000 7ff6aaffd640 10 client.19032 handle_command_reply: tid=0 2024-09-15T23:31:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.301+0000 7ff6d41cd740 1 client.19032 shutdown 2024-09-15T23:31:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2625447010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3980 msgr2=0x556e045fb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3980 0x556e045fb5f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4098590 tx=0x7ff6b409b700 comp rx=0 tx=0).stop 2024-09-15T23:31:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2625447010 shutdown_connections 2024-09-15T23:31:28.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556e0460dd90 0x556e04610180 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 0x556e045fbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3980 0x556e045fb5f0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6aaffd640 0 client.19032 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2625447010 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3060 0x556e045fb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2625447010 >> 172.21.15.62:0/2625447010 conn(0x556e045ef5f0 msgr2=0x556e045ee750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2625447010 shutdown_connections 2024-09-15T23:31:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2625447010 wait complete. 2024-09-15T23:31:28.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.302+0000 7ff6d41cd740 20 client.19032 trim_cache size 0 max 16384 2024-09-15T23:31:28.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6d41cd740 1 Processor -- start 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6d41cd740 1 -- start start 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045ee640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 0x556e04615a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045fe240 0x556e04615f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1b15e0 con 0x556e045f3060 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1aee80 con 0x556e045f3980 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1accd0 con 0x556e045fe240 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6cb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6cb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045ee640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39022/0 (socket says 172.21.15.62:39022) 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/780405635 learned_addr learned my addr 172.21.15.62:0/780405635 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/780405635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 0x556e04615a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/780405635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045fe240 msgr2=0x556e04615f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/780405635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045fe240 0x556e04615f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/780405635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 msgr2=0x556e04615a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/780405635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 0x556e04615a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.319+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/780405635 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e04610800 con 0x556e045f3060 2024-09-15T23:31:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/780405635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045ee640 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x556e045f0450 tx=0x7ff6bc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68022b3eacc32e7f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6c17fa640 1 -- 172.21.15.62:0/780405635 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc040070 con 0x556e045f3060 2024-09-15T23:31:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6c17fa640 1 -- 172.21.15.62:0/780405635 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc0054e0 con 0x556e045f3060 2024-09-15T23:31:28.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6c17fa640 1 -- 172.21.15.62:0/780405635 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc035040 con 0x556e045f3060 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6d41cd740 1 -- 172.21.15.62:0/780405635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 msgr2=0x556e045ee640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/780405635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045ee640 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x556e045f0450 tx=0x7ff6bc000bc0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6d41cd740 1 -- 172.21.15.62:0/780405635 shutdown_connections 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/780405635 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045fe240 0x556e04615f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/780405635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045f3980 0x556e04615a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/780405635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e045ee640 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6d41cd740 1 -- 172.21.15.62:0/780405635 >> 172.21.15.62:0/780405635 conn(0x556e045ef5f0 msgr2=0x556e045edf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6d41cd740 1 -- 172.21.15.62:0/780405635 shutdown_connections 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.320+0000 7ff6d41cd740 1 -- 172.21.15.62:0/780405635 wait complete. 2024-09-15T23:31:28.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6d41cd740 1 -- start start 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6d41cd740 1 Processor -- start 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e046061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3980 0x556e04606730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6d41cd740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 0x556e046011a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1ad340 con 0x556e045f3060 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1ae7b0 con 0x556e045fe240 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6d41cd740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1ada10 con 0x556e045f3980 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6d41cd740 10 client.0 _ll_register_callbacks cb 0x7ff6b400df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6cb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e046061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6cb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e046061f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:39026/0 (socket says 172.21.15.62:39026) 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.321+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/2197383135 learned_addr learned my addr 172.21.15.62:0/2197383135 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6cbfff640 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 0x556e046011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3980 0x556e04606730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/2197383135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3980 msgr2=0x556e04606730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3980 0x556e04606730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/2197383135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 msgr2=0x556e046011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 0x556e046011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6cb7fe640 1 -- 172.21.15.62:0/2197383135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e04601790 con 0x556e045f3060 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6cb7fe640 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e046061f0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009900 tx=0x7ff6bc005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d33019eeab959d server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2197383135 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc035030 con 0x556e045f3060 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2197383135 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff6bc005000 con 0x556e045f3060 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2197383135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6bc0054e0 con 0x556e045f3060 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2197383135 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff6bc066050 con 0x556e045f3060 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6d41cd740 10 client.19038 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2197383135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556e04679340 con 0x556e045f3060 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.322+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2197383135 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc0054e0 con 0x556e045f3060 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2197383135 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7ff6bc0035f0 con 0x556e045f3060 2024-09-15T23:31:28.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6d41cd740 10 client.19038 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:28.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6d41cd740 10 client.19038 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:31:28.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6d41cd740 20 client.19038 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:28.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556e04679630 0x556e0467ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:28.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6d41cd740 4 client.19038 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "set_id": "7334f5cd", "release": true, "await": true}] 2024-09-15T23:31:28.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2197383135 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "set_id": "7334f5cd", "release": true, "await": true}) -- 0x556e045600a0 con 0x556e04679630 2024-09-15T23:31:28.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556e04679630 0x556e0467ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:28.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6c3fff640 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556e04679630 0x556e0467ba20 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=862471ab6cb5f337 server_cookie=47753b14a51fe152 in_seq=0 out_seq=0 2024-09-15T23:31:28.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.323+0000 7ff6aaffd640 10 client.19038 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:28.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.353+0000 7ff6aaffd640 1 -- 172.21.15.62:0/2197383135 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+591 (crc 0 0 0) 0x556e045600a0 con 0x556e04679630 2024-09-15T23:31:28.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.353+0000 7ff6aaffd640 10 client.19038 handle_command_reply: tid=0 2024-09-15T23:31:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.353+0000 7ff6d41cd740 1 client.19038 shutdown 2024-09-15T23:31:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.353+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2197383135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 msgr2=0x556e046061f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.353+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e046061f0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009900 tx=0x7ff6bc005b50 comp rx=0 tx=0).stop 2024-09-15T23:31:28.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2197383135 shutdown_connections 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x556e04679630 0x556e0467ba20 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e045fe240 0x556e046011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x556e045f3980 0x556e04606730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6d41cd740 1 --2- 172.21.15.62:0/2197383135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556e045f3060 0x556e046061f0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2197383135 >> 172.21.15.62:0/2197383135 conn(0x556e045ef5f0 msgr2=0x556e045fb890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6aaffd640 0 client.19038 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2197383135 shutdown_connections 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6d41cd740 1 -- 172.21.15.62:0/2197383135 wait complete. 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.355+0000 7ff6d41cd740 20 client.19038 trim_cache size 0 max 16384 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 36, 2024-09-15T23:31:28.358 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 60, 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": { 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "7334f5cd": { 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "version": 60, 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "age_ref": 0.0, 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "timeout": 90.0, 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "expiration": 137.7, 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "members": { 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "file:/": { 2024-09-15T23:31:28.359 INFO:teuthology.orchestra.run.smithi062.stdout: "excluded": false, 2024-09-15T23:31:28.360 INFO:teuthology.orchestra.run.smithi062.stdout: "state": { 2024-09-15T23:31:28.360 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RELEASED", 2024-09-15T23:31:28.360 INFO:teuthology.orchestra.run.smithi062.stdout: "age": 0.0 2024-09-15T23:31:28.360 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:28.360 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:28.360 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:28.360 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:28.360 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:31:28.360 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:31:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.356+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff6b80a3240 msgr2=0x7ff6b80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.356+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff6b80a3240 0x7ff6b80a5700 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009370 tx=0x7ff6c4004a40 comp rx=0 tx=0).stop 2024-09-15T23:31:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.356+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc102790 msgr2=0x7ff6cc1989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.356+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc102790 0x7ff6cc1989b0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400d950 tx=0x7ff6b400de10 comp rx=0 tx=0).stop 2024-09-15T23:31:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.357+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2668994842 shutdown_connections 2024-09-15T23:31:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.357+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff6b80a3240 0x7ff6b80a5700 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.357+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff6cc106f10 0x7ff6cc19cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.357+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6cc102790 0x7ff6cc1989b0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.357+0000 7ff6d1df4640 1 --2- 172.21.15.62:0/2668994842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6cc101de0 0x7ff6cc198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.357+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2668994842 >> 172.21.15.62:0/2668994842 conn(0x7ff6cc0fd8c0 msgr2=0x7ff6cc0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.358+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2668994842 shutdown_connections 2024-09-15T23:31:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:28.358+0000 7ff6d1df4640 1 -- 172.21.15.62:0/2668994842 wait complete. 2024-09-15T23:31:29.085 INFO:tasks.quiescer.fs.[cephfs]:Successfully released set '7334f5cd', total seconds elapsed: 20.9 2024-09-15T23:31:29.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:29 smithi062 ceph-mon[30949]: pgmap v826: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 314 KiB/s wr, 138 op/s 2024-09-15T23:31:29.524 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:29 smithi067 ceph-mon[35665]: pgmap v826: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 314 KiB/s wr, 138 op/s 2024-09-15T23:31:29.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:29 smithi148 ceph-mon[35610]: pgmap v826: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 314 KiB/s wr, 138 op/s 2024-09-15T23:31:29.569 INFO:teuthology.orchestra.run.smithi062.stderr:progress: 910.0 s, 18.6 tps, lat 14819.231 ms stddev 527.491 2024-09-15T23:31:29.815 INFO:teuthology.orchestra.run.smithi062.stdout:transaction type: 2024-09-15T23:31:29.815 INFO:teuthology.orchestra.run.smithi062.stdout:scaling factor: 32 2024-09-15T23:31:29.815 INFO:teuthology.orchestra.run.smithi062.stdout:query mode: simple 2024-09-15T23:31:29.815 INFO:teuthology.orchestra.run.smithi062.stdout:number of clients: 100 2024-09-15T23:31:29.815 INFO:teuthology.orchestra.run.smithi062.stdout:number of threads: 4 2024-09-15T23:31:29.815 INFO:teuthology.orchestra.run.smithi062.stdout:duration: 900 s 2024-09-15T23:31:29.815 INFO:teuthology.orchestra.run.smithi062.stdout:number of transactions actually processed: 770924 2024-09-15T23:31:29.816 INFO:teuthology.orchestra.run.smithi062.stdout:latency average = 117.943 ms 2024-09-15T23:31:29.816 INFO:teuthology.orchestra.run.smithi062.stdout:latency stddev = 862.217 ms 2024-09-15T23:31:29.816 INFO:teuthology.orchestra.run.smithi062.stdout:tps = 846.941357 (including connections establishing) 2024-09-15T23:31:29.816 INFO:teuthology.orchestra.run.smithi062.stdout:tps = 846.944039 (excluding connections establishing) 2024-09-15T23:31:29.818 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-09-15T23:31:31.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:31 smithi148 ceph-mon[35610]: pgmap v827: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 477 KiB/s wr, 143 op/s 2024-09-15T23:31:31.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:31 smithi062 ceph-mon[30949]: pgmap v827: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 477 KiB/s wr, 143 op/s 2024-09-15T23:31:31.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:31 smithi067 ceph-mon[35665]: pgmap v827: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 477 KiB/s wr, 143 op/s 2024-09-15T23:31:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:33 smithi148 ceph-mon[35610]: pgmap v828: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 739 KiB/s wr, 132 op/s 2024-09-15T23:31:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:33 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:31:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:33 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:31:33.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:33 smithi148 ceph-mon[35610]: scrub summary: idle 2024-09-15T23:31:33.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:33 smithi062 ceph-mon[30949]: pgmap v828: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 739 KiB/s wr, 132 op/s 2024-09-15T23:31:33.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:33 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:31:33.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:33 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:31:33.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:33 smithi062 ceph-mon[30949]: scrub summary: idle 2024-09-15T23:31:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:33 smithi067 ceph-mon[35665]: pgmap v828: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 739 KiB/s wr, 132 op/s 2024-09-15T23:31:33.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:33 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:31:33.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:33 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:31:33.775 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:33 smithi067 ceph-mon[35665]: scrub summary: idle 2024-09-15T23:31:35.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:35 smithi062 ceph-mon[30949]: pgmap v829: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-09-15T23:31:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.733+0000 7f276e911640 1 -- 172.21.15.62:0/1689466829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2768102600 msgr2=0x7f2768108e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:35.737 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.733+0000 7f276e911640 1 --2- 172.21.15.62:0/1689466829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2768102600 0x7f2768108e90 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f2760009ca0 tx=0x7f276002a8f0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.741+0000 7f276e911640 1 -- 172.21.15.62:0/1689466829 shutdown_connections 2024-09-15T23:31:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.741+0000 7f276e911640 1 --2- 172.21.15.62:0/1689466829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2768102600 0x7f2768108e90 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.741+0000 7f276e911640 1 --2- 172.21.15.62:0/1689466829 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2768101c40 0x7f27681020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.741+0000 7f276e911640 1 --2- 172.21.15.62:0/1689466829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2768100a50 0x7f2768100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.745 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.742+0000 7f276e911640 1 -- 172.21.15.62:0/1689466829 >> 172.21.15.62:0/1689466829 conn(0x7f27680fc1b0 msgr2=0x7f27680fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.747+0000 7f276e911640 1 -- 172.21.15.62:0/1689466829 shutdown_connections 2024-09-15T23:31:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.752+0000 7f276e911640 1 -- 172.21.15.62:0/1689466829 wait complete. 2024-09-15T23:31:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.752+0000 7f276e911640 1 Processor -- start 2024-09-15T23:31:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.756+0000 7f276e911640 1 -- start start 2024-09-15T23:31:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.757+0000 7f276e911640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2768100a50 0x7f2768110e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.757+0000 7f276e911640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2768101c40 0x7f27681113c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.757+0000 7f276e911640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2768102600 0x7f2768118430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.757+0000 7f276e911640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f276810b5a0 con 0x7f2768102600 2024-09-15T23:31:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.757+0000 7f276e911640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f276810b420 con 0x7f2768101c40 2024-09-15T23:31:35.760 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.757+0000 7f276e911640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f276810b720 con 0x7f2768100a50 2024-09-15T23:31:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.762+0000 7f27677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2768101c40 0x7f27681113c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.762+0000 7f27677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2768101c40 0x7f27681113c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50058/0 (socket says 172.21.15.62:50058) 2024-09-15T23:31:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.762+0000 7f27677fe640 1 -- 172.21.15.62:0/709284376 learned_addr learned my addr 172.21.15.62:0/709284376 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:35.768 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.766+0000 7f2767fff640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2768100a50 0x7f2768110e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.766+0000 7f27677fe640 1 -- 172.21.15.62:0/709284376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2768100a50 msgr2=0x7f2768110e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.766+0000 7f27677fe640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2768100a50 0x7f2768110e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.766+0000 7f27677fe640 1 -- 172.21.15.62:0/709284376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2768102600 msgr2=0x7f2768118430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.766+0000 7f27677fe640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2768102600 0x7f2768118430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.769 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.766+0000 7f27677fe640 1 -- 172.21.15.62:0/709284376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2768118aa0 con 0x7f2768101c40 2024-09-15T23:31:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.767+0000 7f27677fe640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2768101c40 0x7f27681113c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f275800c890 tx=0x7f275800cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:35.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:35 smithi067 ceph-mon[35665]: pgmap v829: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-09-15T23:31:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.772+0000 7f27657fa640 1 -- 172.21.15.62:0/709284376 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2758004220 con 0x7f2768101c40 2024-09-15T23:31:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.772+0000 7f27657fa640 1 -- 172.21.15.62:0/709284376 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27580043c0 con 0x7f2768101c40 2024-09-15T23:31:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.772+0000 7f27657fa640 1 -- 172.21.15.62:0/709284376 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27580049e0 con 0x7f2768101c40 2024-09-15T23:31:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.774+0000 7f276e911640 1 -- 172.21.15.62:0/709284376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2768118d90 con 0x7f2768101c40 2024-09-15T23:31:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.774+0000 7f276e911640 1 -- 172.21.15.62:0/709284376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27681aefa0 con 0x7f2768101c40 2024-09-15T23:31:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.779+0000 7f27716a9740 1 Processor -- start 2024-09-15T23:31:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.785+0000 7f27716a9740 1 -- start start 2024-09-15T23:31:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.785+0000 7f27716a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f779a0 0x557156f77da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.785+0000 7f27716a9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f7cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.785+0000 7f27716a9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 0x557156f7f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.785+0000 7f27716a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557156edb2f0 con 0x557156f779a0 2024-09-15T23:31:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.785+0000 7f27716a9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557156e83270 con 0x557156f782e0 2024-09-15T23:31:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.785+0000 7f27716a9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557156ed72b0 con 0x557156f7d370 2024-09-15T23:31:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.786+0000 7f27677fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 0x557156f7f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.786+0000 7f27677fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 0x557156f7f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:39888/0 (socket says 172.21.15.62:39888) 2024-09-15T23:31:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.786+0000 7f27677fe640 1 -- 172.21.15.62:0/326346729 learned_addr learned my addr 172.21.15.62:0/326346729 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:35.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:35 smithi148 ceph-mon[35610]: pgmap v829: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 1.6 MiB/s wr, 190 op/s 2024-09-15T23:31:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.787+0000 7f27657fa640 1 -- 172.21.15.62:0/709284376 <== mon.2 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2758004c40 con 0x7f2768101c40 2024-09-15T23:31:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.788+0000 7f2767fff640 1 --2- 172.21.15.62:0/326346729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f7cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.789+0000 7f27657fa640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27500a3530 0x7f27500a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.791 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.789+0000 7f27657fa640 1 -- 172.21.15.62:0/709284376 <== mon.2 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f27580cfc90 con 0x7f2768101c40 2024-09-15T23:31:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.790+0000 7f276ceae640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27500a3530 0x7f27500a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.791+0000 7f276ceae640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27500a3530 0x7f27500a59f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2760009bf0 tx=0x7f2760031ac0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.791+0000 7f27677fe640 1 -- 172.21.15.62:0/326346729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 msgr2=0x557156f7cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.791+0000 7f27677fe640 1 --2- 172.21.15.62:0/326346729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f7cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.791+0000 7f27677fe640 1 -- 172.21.15.62:0/326346729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f779a0 msgr2=0x557156f77da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.791+0000 7f27677fe640 1 --2- 172.21.15.62:0/326346729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f779a0 0x557156f77da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.791+0000 7f27677fe640 1 -- 172.21.15.62:0/326346729 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557156e82340 con 0x557156f7d370 2024-09-15T23:31:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.795+0000 7f27677fe640 1 --2- 172.21.15.62:0/326346729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 0x557156f7f830 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2758098920 tx=0x7f2758005020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a4771fd87fcfe91 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.796+0000 7f274e7fc640 1 -- 172.21.15.62:0/326346729 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275809bd20 con 0x557156f7d370 2024-09-15T23:31:35.801 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.796+0000 7f274e7fc640 1 -- 172.21.15.62:0/326346729 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f275809c650 con 0x557156f7d370 2024-09-15T23:31:35.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.796+0000 7f274e7fc640 1 -- 172.21.15.62:0/326346729 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275809b8b0 con 0x557156f7d370 2024-09-15T23:31:35.802 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.801+0000 7f27716a9740 1 -- 172.21.15.62:0/326346729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 msgr2=0x557156f7f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.801+0000 7f27716a9740 1 --2- 172.21.15.62:0/326346729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 0x557156f7f830 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2758098920 tx=0x7f2758005020 comp rx=0 tx=0).stop 2024-09-15T23:31:35.808 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.805+0000 7f27716a9740 1 -- 172.21.15.62:0/326346729 shutdown_connections 2024-09-15T23:31:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.805+0000 7f27716a9740 1 --2- 172.21.15.62:0/326346729 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 0x557156f7f830 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.805+0000 7f27716a9740 1 --2- 172.21.15.62:0/326346729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f7cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.805+0000 7f27716a9740 1 --2- 172.21.15.62:0/326346729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f779a0 0x557156f77da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.809 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.805+0000 7f27716a9740 1 -- 172.21.15.62:0/326346729 >> 172.21.15.62:0/326346729 conn(0x557156f73010 msgr2=0x557156f73410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:35.813 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.811+0000 7f27716a9740 1 -- 172.21.15.62:0/326346729 shutdown_connections 2024-09-15T23:31:35.818 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.815+0000 7f27716a9740 1 -- 172.21.15.62:0/326346729 wait complete. 2024-09-15T23:31:35.818 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.816+0000 7f27716a9740 1 -- start start 2024-09-15T23:31:35.819 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.816+0000 7f27716a9740 1 Processor -- start 2024-09-15T23:31:35.820 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.817+0000 7f27716a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f779a0 0x557156f81bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.821 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.817+0000 7f27716a9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f82130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.821 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.817+0000 7f27716a9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 0x557156f89170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.821 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.817+0000 7f27716a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x557156f802e0 con 0x557156f779a0 2024-09-15T23:31:35.821 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.817+0000 7f27716a9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557156e64290 con 0x557156f782e0 2024-09-15T23:31:35.821 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.817+0000 7f27716a9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x557156f80460 con 0x557156f7d370 2024-09-15T23:31:35.821 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.817+0000 7f27716a9740 10 client.0 _ll_register_callbacks cb 0x557156e95140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:35.822 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.820+0000 7f27677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f82130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.823 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.820+0000 7f27677fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f82130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:50082/0 (socket says 172.21.15.62:50082) 2024-09-15T23:31:35.823 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.821+0000 7f27677fe640 1 -- 172.21.15.62:0/1674410746 learned_addr learned my addr 172.21.15.62:0/1674410746 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:35.823 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.821+0000 7f27677fe640 1 -- 172.21.15.62:0/1674410746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 msgr2=0x557156f89170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:35.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.821+0000 7f27677fe640 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 0x557156f89170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.821+0000 7f27677fe640 1 -- 172.21.15.62:0/1674410746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f779a0 msgr2=0x557156f81bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T23:31:35.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.821+0000 7f27677fe640 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f779a0 0x557156f81bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.824 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.821+0000 7f27677fe640 1 -- 172.21.15.62:0/1674410746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557156bee210 con 0x557156f782e0 2024-09-15T23:31:35.827 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.825+0000 7f27677fe640 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f82130 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2758005fd0 tx=0x7f27580d3640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d253dd65b6affd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:35.829 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.827+0000 7f27477fe640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:31:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.827+0000 7f27477fe640 1 -- 172.21.15.62:0/1674410746 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275809ba90 con 0x557156f782e0 2024-09-15T23:31:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.827+0000 7f27477fe640 1 -- 172.21.15.62:0/1674410746 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f275809b3a0 con 0x557156f782e0 2024-09-15T23:31:35.830 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.828+0000 7f27716a9740 1 -- 172.21.15.62:0/1674410746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f275809c650 con 0x557156f782e0 2024-09-15T23:31:35.833 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.831+0000 7f27477fe640 1 -- 172.21.15.62:0/1674410746 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f275809bd20 con 0x557156f782e0 2024-09-15T23:31:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.833+0000 7f27716a9740 10 client.28526 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:35.836 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.833+0000 7f27716a9740 1 -- 172.21.15.62:0/1674410746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557156cba620 con 0x557156f782e0 2024-09-15T23:31:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.840+0000 7f27477fe640 1 -- 172.21.15.62:0/1674410746 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f275809c650 con 0x557156f782e0 2024-09-15T23:31:35.842 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.840+0000 7f27477fe640 1 -- 172.21.15.62:0/1674410746 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f275809ae10 con 0x557156f782e0 2024-09-15T23:31:35.845 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.841+0000 7f27716a9740 10 client.28526 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:35.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.841+0000 7f27716a9740 10 client.28526 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:35.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.841+0000 7f27716a9740 20 client.28526 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:35.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.841+0000 7f27716a9740 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557156f8d470 0x557156f8f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.841+0000 7f27716a9740 4 client.28526 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:35.846 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.841+0000 7f27716a9740 1 -- 172.21.15.62:0/1674410746 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557156ee8a20 con 0x557156f8d470 2024-09-15T23:31:35.848 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.846+0000 7f2767fff640 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557156f8d470 0x557156f8f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.853+0000 7f2767fff640 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557156f8d470 0x557156f8f860 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b10616af49612001 server_cookie=f9262a758cc6382a in_seq=0 out_seq=0 2024-09-15T23:31:35.855 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.853+0000 7f27477fe640 10 client.28526 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:35.860 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.855+0000 7f27477fe640 1 -- 172.21.15.62:0/1674410746 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557156ee8a20 con 0x557156f8d470 2024-09-15T23:31:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.855+0000 7f27477fe640 10 client.28526 handle_command_reply: tid=0 2024-09-15T23:31:35.861 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.857+0000 7f27716a9740 1 client.28526 shutdown 2024-09-15T23:31:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.867+0000 7f27716a9740 1 -- 172.21.15.62:0/1674410746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 msgr2=0x557156f82130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.867+0000 7f27716a9740 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f82130 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2758005fd0 tx=0x7f27580d3640 comp rx=0 tx=0).stop 2024-09-15T23:31:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.869+0000 7f27716a9740 1 -- 172.21.15.62:0/1674410746 shutdown_connections 2024-09-15T23:31:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.869+0000 7f27716a9740 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557156f8d470 0x557156f8f860 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.869+0000 7f27716a9740 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f7d370 0x557156f89170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.869+0000 7f27716a9740 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f782e0 0x557156f82130 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.869+0000 7f27716a9740 1 --2- 172.21.15.62:0/1674410746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f779a0 0x557156f81bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.869+0000 7f27716a9740 1 -- 172.21.15.62:0/1674410746 >> 172.21.15.62:0/1674410746 conn(0x557156f73010 msgr2=0x557156f74820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.869+0000 7f27477fe640 0 client.28526 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:35.871 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.869+0000 7f27716a9740 1 -- 172.21.15.62:0/1674410746 shutdown_connections 2024-09-15T23:31:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.873+0000 7f27716a9740 1 -- 172.21.15.62:0/1674410746 wait complete. 2024-09-15T23:31:35.877 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.876+0000 7f27716a9740 20 client.28526 trim_cache size 0 max 16384 2024-09-15T23:31:35.920 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.917+0000 7f27716a9740 1 Processor -- start 2024-09-15T23:31:35.920 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.917+0000 7f27716a9740 1 -- start start 2024-09-15T23:31:35.920 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.918+0000 7f27716a9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f779a0 0x557156f73ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.920 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.918+0000 7f27716a9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f782e0 0x557156f89130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.918+0000 7f27716a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f7d370 0x557156f74400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.918+0000 7f27716a9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f27681bbf00 con 0x557156f7d370 2024-09-15T23:31:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.918+0000 7f276ceae640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f7d370 0x557156f74400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.918+0000 7f27716a9740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27681b97a0 con 0x557156f779a0 2024-09-15T23:31:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.918+0000 7f27716a9740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f27681b75f0 con 0x557156f782e0 2024-09-15T23:31:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.920+0000 7f27677fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f782e0 0x557156f89130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.920+0000 7f27677fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f782e0 0x557156f89130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.62:39898/0 (socket says 172.21.15.62:39898) 2024-09-15T23:31:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.920+0000 7f27677fe640 1 -- 172.21.15.62:0/4142321347 learned_addr learned my addr 172.21.15.62:0/4142321347 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.920+0000 7f2767fff640 1 --2- 172.21.15.62:0/4142321347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f779a0 0x557156f73ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.921+0000 7f27677fe640 1 -- 172.21.15.62:0/4142321347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f779a0 msgr2=0x557156f73ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.921+0000 7f27677fe640 1 --2- 172.21.15.62:0/4142321347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f779a0 0x557156f73ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.921+0000 7f27677fe640 1 -- 172.21.15.62:0/4142321347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f7d370 msgr2=0x557156f74400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.921+0000 7f27677fe640 1 --2- 172.21.15.62:0/4142321347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f7d370 0x557156f74400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.923 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.921+0000 7f27677fe640 1 -- 172.21.15.62:0/4142321347 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557156f6e2d0 con 0x557156f782e0 2024-09-15T23:31:35.927 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.925+0000 7f27677fe640 1 --2- 172.21.15.62:0/4142321347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f782e0 0x557156f89130 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f27580983b0 tx=0x7f275809a460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49f39d7e0a0452e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:35.930 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.927+0000 7f274dffb640 1 -- 172.21.15.62:0/4142321347 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275809ba90 con 0x557156f782e0 2024-09-15T23:31:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.927+0000 7f274dffb640 1 -- 172.21.15.62:0/4142321347 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f275809c650 con 0x557156f782e0 2024-09-15T23:31:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.927+0000 7f274dffb640 1 -- 172.21.15.62:0/4142321347 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275809b3a0 con 0x557156f782e0 2024-09-15T23:31:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.928+0000 7f27716a9740 1 -- 172.21.15.62:0/4142321347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f782e0 msgr2=0x557156f89130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.928+0000 7f27716a9740 1 --2- 172.21.15.62:0/4142321347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f782e0 0x557156f89130 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f27580983b0 tx=0x7f275809a460 comp rx=0 tx=0).stop 2024-09-15T23:31:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.928+0000 7f27716a9740 1 -- 172.21.15.62:0/4142321347 shutdown_connections 2024-09-15T23:31:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.928+0000 7f27716a9740 1 --2- 172.21.15.62:0/4142321347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f7d370 0x557156f74400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.928+0000 7f27716a9740 1 --2- 172.21.15.62:0/4142321347 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f782e0 0x557156f89130 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.928+0000 7f27716a9740 1 --2- 172.21.15.62:0/4142321347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f779a0 0x557156f73ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.931 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.928+0000 7f27716a9740 1 -- 172.21.15.62:0/4142321347 >> 172.21.15.62:0/4142321347 conn(0x557156f73010 msgr2=0x557156f79520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.934+0000 7f27716a9740 1 -- 172.21.15.62:0/4142321347 shutdown_connections 2024-09-15T23:31:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.934+0000 7f27716a9740 1 -- 172.21.15.62:0/4142321347 wait complete. 2024-09-15T23:31:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.935+0000 7f27716a9740 1 -- start start 2024-09-15T23:31:35.937 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.935+0000 7f27716a9740 1 Processor -- start 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27716a9740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f779a0 0x557156f86480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27716a9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f782e0 0x557156f869c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27716a9740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f7d370 0x557156f87980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27677fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f782e0 0x557156f869c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27677fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f782e0 0x557156f869c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:55608/0 (socket says 172.21.15.62:55608) 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27677fe640 1 -- 172.21.15.62:0/542256160 learned_addr learned my addr 172.21.15.62:0/542256160 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f27681b7c60 con 0x557156f782e0 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f27681b90d0 con 0x557156f7d370 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f27681b8330 con 0x557156f779a0 2024-09-15T23:31:35.939 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.936+0000 7f27716a9740 10 client.0 _ll_register_callbacks cb 0x7f27580d3220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.938+0000 7f27677fe640 1 -- 172.21.15.62:0/542256160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f779a0 msgr2=0x557156f86480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:31:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.938+0000 7f27677fe640 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f779a0 0x557156f86480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.938+0000 7f27677fe640 1 -- 172.21.15.62:0/542256160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f7d370 msgr2=0x557156f87980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-15T23:31:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.938+0000 7f27677fe640 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f7d370 0x557156f87980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:35.940 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.938+0000 7f27677fe640 1 -- 172.21.15.62:0/542256160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557156f87f90 con 0x557156f782e0 2024-09-15T23:31:35.941 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.939+0000 7f27677fe640 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f782e0 0x557156f869c0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f275809b210 tx=0x7f275809b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba34c444d3bd9c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:35.942 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.940+0000 7f27477fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.940+0000 7f27477fe640 1 -- 172.21.15.62:0/542256160 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275809ba90 con 0x557156f782e0 2024-09-15T23:31:35.943 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.940+0000 7f27477fe640 1 -- 172.21.15.62:0/542256160 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2758096ac0 con 0x557156f782e0 2024-09-15T23:31:35.947 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.943+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f275809c650 con 0x557156f782e0 2024-09-15T23:31:35.948 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.947+0000 7f27477fe640 1 -- 172.21.15.62:0/542256160 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f27580b01f0 con 0x557156f782e0 2024-09-15T23:31:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.949+0000 7f27716a9740 10 client.19047 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.949+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557156f87f90 con 0x557156f782e0 2024-09-15T23:31:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.953+0000 7f27477fe640 1 -- 172.21.15.62:0/542256160 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27580a4bc0 con 0x557156f782e0 2024-09-15T23:31:35.955 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.953+0000 7f27477fe640 1 -- 172.21.15.62:0/542256160 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f27580079b0 con 0x557156f782e0 2024-09-15T23:31:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.954+0000 7f27716a9740 10 client.19047 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.954+0000 7f27716a9740 10 client.19047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.954+0000 7f27716a9740 20 client.19047 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.954+0000 7f27716a9740 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557156ff80c0 0x557156ffa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.954+0000 7f27716a9740 4 client.19047 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-15T23:31:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.954+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f275809c650 con 0x557156ff80c0 2024-09-15T23:31:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.955+0000 7f2767fff640 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557156ff80c0 0x557156ffa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.957+0000 7f2767fff640 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557156ff80c0 0x557156ffa4b0 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=757ceeec6461d69f server_cookie=1013cb2246abb19e in_seq=0 out_seq=0 2024-09-15T23:31:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:35.959+0000 7f27477fe640 10 client.19047 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:37.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:37 smithi062 ceph-mon[30949]: pgmap v830: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.9 MiB/s wr, 431 op/s 2024-09-15T23:31:37.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:37 smithi067 ceph-mon[35665]: pgmap v830: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.9 MiB/s wr, 431 op/s 2024-09-15T23:31:37.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:37 smithi148 ceph-mon[35610]: pgmap v830: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.9 MiB/s wr, 431 op/s 2024-09-15T23:31:39.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27477fe640 1 -- 172.21.15.62:0/542256160 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f275809c650 con 0x557156ff80c0 2024-09-15T23:31:39.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27477fe640 10 client.19047 handle_command_reply: tid=0 2024-09-15T23:31:39.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 client.19047 shutdown 2024-09-15T23:31:39.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f782e0 msgr2=0x557156f869c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:39.177 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f782e0 0x557156f869c0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f275809b210 tx=0x7f275809b5d0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 shutdown_connections 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x557156ff80c0 0x557156ffa4b0 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557156f7d370 0x557156f87980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27477fe640 0 client.19047 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557156f782e0 0x557156f869c0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 --2- 172.21.15.62:0/542256160 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x557156f779a0 0x557156f86480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 >> 172.21.15.62:0/542256160 conn(0x557156f73010 msgr2=0x557156f719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 shutdown_connections 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.174+0000 7f27716a9740 1 -- 172.21.15.62:0/542256160 wait complete. 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f27716a9740 20 client.19047 trim_cache size 0 max 16384 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:{ 2024-09-15T23:31:39.178 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "message": "", 2024-09-15T23:31:39.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout: "return_code": 0 2024-09-15T23:31:39.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stdout:} 2024-09-15T23:31:39.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 -- 172.21.15.62:0/709284376 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27500a3530 msgr2=0x7f27500a59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:39.179 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27500a3530 0x7f27500a59f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2760009bf0 tx=0x7f2760031ac0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 -- 172.21.15.62:0/709284376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2768101c40 msgr2=0x7f27681113c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2768101c40 0x7f27681113c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f275800c890 tx=0x7f275800cd50 comp rx=0 tx=0).stop 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 -- 172.21.15.62:0/709284376 shutdown_connections 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f27500a3530 0x7f27500a59f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2768102600 0x7f2768118430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2768101c40 0x7f27681113c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 --2- 172.21.15.62:0/709284376 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2768100a50 0x7f2768110e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 -- 172.21.15.62:0/709284376 >> 172.21.15.62:0/709284376 conn(0x7f27680fc1b0 msgr2=0x7f2768104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.175+0000 7f276e911640 1 -- 172.21.15.62:0/709284376 shutdown_connections 2024-09-15T23:31:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi062.front.sepia.ceph.com.smithi062.stderr:2024-09-15T23:31:39.176+0000 7f276e911640 1 -- 172.21.15.62:0/709284376 wait complete. 2024-09-15T23:31:39.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:39 smithi062 ceph-mon[30949]: pgmap v831: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.3 MiB/s wr, 584 op/s 2024-09-15T23:31:39.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:39 smithi067 ceph-mon[35665]: pgmap v831: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.3 MiB/s wr, 584 op/s 2024-09-15T23:31:39.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:39 smithi148 ceph-mon[35610]: pgmap v831: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 5.3 MiB/s wr, 584 op/s 2024-09-15T23:31:40.169 DEBUG:teuthology.orchestra.run.smithi062:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-09-15T23:31:40.237 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/: 2024-09-15T23:31:40.237 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.237 INFO:teuthology.orchestra.run.smithi062.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:31:40.237 INFO:teuthology.orchestra.run.smithi062.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 .. 2024-09-15T23:31:40.237 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 data 2024-09-15T23:31:40.237 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.237 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data: 2024-09-15T23:31:40.237 INFO:teuthology.orchestra.run.smithi062.stdout:total 36 2024-09-15T23:31:40.237 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 . 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 .. 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 15 23:16 base 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Sep 15 23:16 current_logfiles 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Sep 15 23:16 global 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 log 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_commit_ts 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_dynshmem 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Sep 15 23:16 pg_hba.conf 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Sep 15 23:16 pg_ident.conf 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:31 pg_logical 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 pg_multixact 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_notify 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_replslot 2024-09-15T23:31:40.238 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_serial 2024-09-15T23:31:40.239 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_snapshots 2024-09-15T23:31:40.239 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:31 pg_stat 2024-09-15T23:31:40.239 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:31 pg_stat_tmp 2024-09-15T23:31:40.239 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:31 pg_subtrans 2024-09-15T23:31:40.239 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_tblspc 2024-09-15T23:31:40.239 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pg_twophase 2024-09-15T23:31:40.239 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 PG_VERSION 2024-09-15T23:31:40.239 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 60 Sep 15 23:31 pg_wal 2024-09-15T23:31:40.240 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 pg_xact 2024-09-15T23:31:40.240 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Sep 15 23:16 postgresql.auto.conf 2024-09-15T23:31:40.240 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28074 Sep 15 23:16 postgresql.conf 2024-09-15T23:31:40.240 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Sep 15 23:16 postmaster.opts 2024-09-15T23:31:40.240 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.240 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/base: 2024-09-15T23:31:40.240 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.240 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 15 23:16 . 2024-09-15T23:31:40.240 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 1 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 13493 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Sep 15 23:22 13494 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 pgsql_tmp 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/base/1: 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:total 7777 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 . 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 15 23:16 .. 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 112 2024-09-15T23:31:40.241 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 113 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 1247 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1247_fsm 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1247_vm 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 15 23:16 1249 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1249_fsm 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1249_vm 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 15 23:16 1255 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1255_fsm 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1255_vm 2024-09-15T23:31:40.242 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 15 23:16 1259 2024-09-15T23:31:40.243 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1259_fsm 2024-09-15T23:31:40.243 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1259_vm 2024-09-15T23:31:40.243 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 15 23:16 13344 2024-09-15T23:31:40.243 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13344_fsm 2024-09-15T23:31:40.243 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13344_vm 2024-09-15T23:31:40.243 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13346 2024-09-15T23:31:40.243 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13348 2024-09-15T23:31:40.243 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13349_fsm 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349_vm 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13351 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13353 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13354_fsm 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354_vm 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13356 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13358 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13359_fsm 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359_vm 2024-09-15T23:31:40.244 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13361 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13363 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1417 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1418 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 174 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 175 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2187 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2224 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2328 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2336 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2337 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2579 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2600 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2600_fsm 2024-09-15T23:31:40.245 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2600_vm 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2601_fsm 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601_vm 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2602 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2602_fsm 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2602_vm 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2603 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2603_fsm 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2603_vm 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2604 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2605 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2605_fsm 2024-09-15T23:31:40.246 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2605_vm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2606_fsm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606_vm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2607 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2607_fsm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2607_vm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 15 23:16 2608 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2608_fsm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2608_vm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 15 23:16 2609 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2609_fsm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2609_vm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2610 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2610_fsm 2024-09-15T23:31:40.247 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2610_vm 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2611 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2612_fsm 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612_vm 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2613 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2615_fsm 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615_vm 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2616 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2616_fsm 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2616_vm 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 15 23:16 2617 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2617_fsm 2024-09-15T23:31:40.248 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2617_vm 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 15 23:16 2618 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2618_fsm 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2618_vm 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 15 23:16 2619 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2619_fsm 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2619_vm 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2620 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2650 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2651 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2652 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2653 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2654 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2655 2024-09-15T23:31:40.249 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2656 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2657 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 15 23:16 2658 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 2659 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2660 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2661 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2662 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2663 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2664 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2665 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2666 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2667 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2668 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2669 2024-09-15T23:31:40.250 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2670 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 15 23:16 2673 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 15 23:16 2674 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 15 23:16 2675 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2678 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2679 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2680 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2681 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2682 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2683 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2684 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2685 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2686 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2687 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2688 2024-09-15T23:31:40.251 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2689 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 15 23:16 2690 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 15 23:16 2691 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2692 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2693 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2696 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2699 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2701 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2702 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2703 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2704 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2753 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2753_fsm 2024-09-15T23:31:40.252 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2753_vm 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2754 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2755 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2756 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2757 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2830 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2831 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2832 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2833 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2834 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2835 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2836 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2837 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 15 23:16 2838 2024-09-15T23:31:40.253 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2838_fsm 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2838_vm 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2839 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840_fsm 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2840_vm 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2841 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2995 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2996 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3079_fsm 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079_vm 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3080 2024-09-15T23:31:40.254 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3081 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3085 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3118 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3119 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 15 23:16 3164 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3256 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3257 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3258 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3350 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3351 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3379 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3380 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3381 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3394 2024-09-15T23:31:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3394_fsm 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3394_vm 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3395 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3429 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3430 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3431 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3433 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3439 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3440 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3455 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 15 23:16 3456 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3456_fsm 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3456_vm 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3466 2024-09-15T23:31:40.256 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3467 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3468 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3501 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3502 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3503 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3534 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3541_fsm 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541_vm 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3542 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3574 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3575 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3576 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3596 2024-09-15T23:31:40.257 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3597 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3598 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3599 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3600_fsm 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600_vm 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3601_fsm 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601_vm 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3602_fsm 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602_vm 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603_fsm 2024-09-15T23:31:40.258 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3603_vm 2024-09-15T23:31:40.259 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3604 2024-09-15T23:31:40.259 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3605 2024-09-15T23:31:40.259 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3606 2024-09-15T23:31:40.259 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3607 2024-09-15T23:31:40.259 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3608 2024-09-15T23:31:40.259 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 3609 2024-09-15T23:31:40.259 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3712 2024-09-15T23:31:40.259 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764 2024-09-15T23:31:40.259 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3764_fsm 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764_vm 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3766 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3767 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3997 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4143 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4144 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4145 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4146 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4147 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4148 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4149 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4150 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4151 2024-09-15T23:31:40.260 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4152 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4153 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4154 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4155 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4156 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4157 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4158 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4159 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4160 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4161 2024-09-15T23:31:40.261 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4162 2024-09-15T23:31:40.262 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4163 2024-09-15T23:31:40.262 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4164 2024-09-15T23:31:40.262 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4165 2024-09-15T23:31:40.262 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4166 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4167 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4168 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4169 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4170 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4171 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4172 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4173 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4174 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 5002 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 548 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 549 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6102 2024-09-15T23:31:40.263 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6104 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6106 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6110 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6111 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6112 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6113 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6117 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 826 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 827 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 828 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 15 23:16 pg_filenode.map 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 PG_VERSION 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-09-15T23:31:40.264 INFO:teuthology.orchestra.run.smithi062.stdout:total 7777 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Sep 15 23:16 . 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 15 23:16 .. 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 112 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 113 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 1247 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1247_fsm 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1247_vm 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 15 23:16 1249 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1249_fsm 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1249_vm 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 15 23:16 1255 2024-09-15T23:31:40.265 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1255_fsm 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1255_vm 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 15 23:16 1259 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1259_fsm 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1259_vm 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 15 23:16 13344 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13344_fsm 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13344_vm 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13346 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13348 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13349_fsm 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349_vm 2024-09-15T23:31:40.266 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13351 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13353 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13354_fsm 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354_vm 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13356 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13358 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13359_fsm 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359_vm 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13361 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13363 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1417 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1418 2024-09-15T23:31:40.267 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 174 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 175 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2187 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2224 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2328 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2336 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2337 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2579 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2600 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2600_fsm 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2600_vm 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2601_fsm 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601_vm 2024-09-15T23:31:40.268 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2602 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2602_fsm 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2602_vm 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2603 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2603_fsm 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2603_vm 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2604 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2605 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2605_fsm 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2605_vm 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2606_fsm 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606_vm 2024-09-15T23:31:40.269 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2607 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2607_fsm 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2607_vm 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 15 23:16 2608 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2608_fsm 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2608_vm 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 15 23:16 2609 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2609_fsm 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2609_vm 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2610 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2610_fsm 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2610_vm 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2611 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612 2024-09-15T23:31:40.270 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2612_fsm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612_vm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2613 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2615_fsm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615_vm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2616 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2616_fsm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2616_vm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 15 23:16 2617 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2617_fsm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2617_vm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 15 23:16 2618 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2618_fsm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2618_vm 2024-09-15T23:31:40.271 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Sep 15 23:16 2619 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2619_fsm 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2619_vm 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2620 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2650 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2651 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2652 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2653 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2654 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2655 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2656 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2657 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 15 23:16 2658 2024-09-15T23:31:40.272 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 2659 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2660 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2661 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2662 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2663 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2664 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2665 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2666 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2667 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2668 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2669 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2670 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 15 23:16 2673 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 15 23:16 2674 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 15 23:16 2675 2024-09-15T23:31:40.273 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2678 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2679 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2680 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2681 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2682 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2683 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2684 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2685 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2686 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2687 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2688 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2689 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 15 23:16 2690 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 15 23:16 2691 2024-09-15T23:31:40.274 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2692 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2693 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2696 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2699 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2701 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2702 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2703 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2704 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2753 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2753_fsm 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2753_vm 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2754 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2755 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2756 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2757 2024-09-15T23:31:40.275 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2830 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2831 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2832 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2833 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2834 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2835 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2836 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2837 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 15 23:16 2838 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2838_fsm 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2838_vm 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2839 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840 2024-09-15T23:31:40.276 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840_fsm 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2840_vm 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2841 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2995 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2996 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3079_fsm 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079_vm 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3080 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3081 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3085 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3118 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3119 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 15 23:16 3164 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3256 2024-09-15T23:31:40.277 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3257 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3258 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3350 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3351 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3379 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3380 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3381 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3394 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3394_fsm 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3394_vm 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3395 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3429 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3430 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3431 2024-09-15T23:31:40.278 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3433 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3439 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3440 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3455 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 15 23:16 3456 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3456_fsm 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3456_vm 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3466 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3467 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3468 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3501 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3502 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3503 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3534 2024-09-15T23:31:40.279 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3541_fsm 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541_vm 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3542 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3574 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3575 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3576 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3596 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3597 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3598 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3599 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3600_fsm 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600_vm 2024-09-15T23:31:40.280 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3601_fsm 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601_vm 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3602_fsm 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602_vm 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603_fsm 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3603_vm 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3604 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3605 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3606 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3607 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3608 2024-09-15T23:31:40.281 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 3609 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3712 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3764_fsm 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764_vm 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3766 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3767 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3997 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4143 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4144 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4145 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4146 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4147 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4148 2024-09-15T23:31:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4149 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4150 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4151 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4152 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4153 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4154 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4155 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4156 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4157 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4158 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4159 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4160 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4161 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4162 2024-09-15T23:31:40.283 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4163 2024-09-15T23:31:40.284 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4164 2024-09-15T23:31:40.284 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4165 2024-09-15T23:31:40.284 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4166 2024-09-15T23:31:40.284 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4167 2024-09-15T23:31:40.284 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4168 2024-09-15T23:31:40.284 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4169 2024-09-15T23:31:40.284 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4170 2024-09-15T23:31:40.284 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4171 2024-09-15T23:31:40.284 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4172 2024-09-15T23:31:40.286 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4173 2024-09-15T23:31:40.286 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4174 2024-09-15T23:31:40.286 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 5002 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 548 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 549 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6102 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6104 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6106 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6110 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6111 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6112 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6113 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6117 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 826 2024-09-15T23:31:40.287 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 827 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 828 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 15 23:16 pg_filenode.map 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 PG_VERSION 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:total 549902 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Sep 15 23:22 . 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 15 23:16 .. 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 112 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 113 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 1247 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1247_fsm 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1247_vm 2024-09-15T23:31:40.288 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Sep 15 23:16 1249 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1249_fsm 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1249_vm 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Sep 15 23:16 1255 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1255_fsm 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1255_vm 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Sep 15 23:31 1259 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1259_fsm 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1259_vm 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Sep 15 23:16 13344 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13344_fsm 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13344_vm 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13346 2024-09-15T23:31:40.289 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13348 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13349_fsm 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13349_vm 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13351 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13353 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13354_fsm 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13354_vm 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13356 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13358 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 13359_fsm 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13359_vm 2024-09-15T23:31:40.290 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 13361 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 13363 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1417 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 1418 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 439894016 Sep 15 23:31 16396 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Sep 15 23:31 16396_fsm 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:26 16396_vm 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1024000 Sep 15 23:31 16397 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:31 16397_fsm 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:31 16397_vm 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1417216 Sep 15 23:31 16399 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:31 16399_fsm 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:31 16399_vm 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:31 16400 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 15 23:31 16402 2024-09-15T23:31:40.291 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Sep 15 23:31 16404 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40386560 Sep 15 23:31 16406 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:31 16406_fsm 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:27 16406_vm 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 174 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 175 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2187 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2224 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2328 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2336 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2337 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2579 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2600 2024-09-15T23:31:40.292 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2600_fsm 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2600_vm 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2601_fsm 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2601_vm 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2602 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2602_fsm 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2602_vm 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2603 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2603_fsm 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2603_vm 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2604 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2605 2024-09-15T23:31:40.293 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2605_fsm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2605_vm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2606_fsm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2606_vm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2607 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2607_fsm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2607_vm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Sep 15 23:16 2608 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2608_fsm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2608_vm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Sep 15 23:16 2609 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2609_fsm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2609_vm 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2610 2024-09-15T23:31:40.294 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2610_fsm 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2610_vm 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2611 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2612_fsm 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2612_vm 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2613 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2615_fsm 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2615_vm 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2616 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2616_fsm 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2616_vm 2024-09-15T23:31:40.295 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Sep 15 23:16 2617 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2617_fsm 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2617_vm 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Sep 15 23:16 2618 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2618_fsm 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2618_vm 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 163840 Sep 15 23:31 2619 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:19 2619_fsm 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2619_vm 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2620 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2650 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2651 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2652 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2653 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2654 2024-09-15T23:31:40.296 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2655 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2656 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2657 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Sep 15 23:16 2658 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Sep 15 23:16 2659 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2660 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2661 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2662 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2663 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2664 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2665 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2666 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2667 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2668 2024-09-15T23:31:40.297 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2669 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2670 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Sep 15 23:16 2673 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Sep 15 23:16 2674 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 15 23:16 2675 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2678 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2679 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2680 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2681 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2682 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2683 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2684 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2685 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2686 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2687 2024-09-15T23:31:40.298 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2688 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Sep 15 23:16 2689 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Sep 15 23:16 2690 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Sep 15 23:16 2691 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2692 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2693 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:20 2696 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2699 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2701 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2702 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2703 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 2704 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2753 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2753_fsm 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2753_vm 2024-09-15T23:31:40.299 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2754 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2755 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2756 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 2757 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2830 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2831 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2832 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2833 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2834 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2835 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2836 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2837 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Sep 15 23:16 2838 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2838_fsm 2024-09-15T23:31:40.300 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2838_vm 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2839 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2840_fsm 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2840_vm 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2841 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2995 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2996 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3079_fsm 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3079_vm 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3080 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3081 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3085 2024-09-15T23:31:40.301 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3118 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3119 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Sep 15 23:16 3164 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3256 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3257 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3258 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3350 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3351 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3379 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3380 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3381 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3394 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3394_fsm 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3394_vm 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3395 2024-09-15T23:31:40.302 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3429 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3430 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3431 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3433 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3439 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3440 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Sep 15 23:16 3455 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Sep 15 23:16 3456 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3456_fsm 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3456_vm 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3466 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3467 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3468 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3501 2024-09-15T23:31:40.303 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3502 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3503 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3534 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3541_fsm 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3541_vm 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3542 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3574 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3575 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3576 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3596 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3597 2024-09-15T23:31:40.304 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3598 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3599 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3600_fsm 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3600_vm 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3601_fsm 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3601_vm 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3602_fsm 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3602_vm 2024-09-15T23:31:40.305 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3603_fsm 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3603_vm 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3604 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3605 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3606 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3607 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3608 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Sep 15 23:16 3609 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3712 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 3764_fsm 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3764_vm 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3766 2024-09-15T23:31:40.306 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 3767 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3997 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4143 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4144 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4145 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4146 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4147 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4148 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4149 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4150 2024-09-15T23:31:40.307 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4151 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4152 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4153 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4154 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4155 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4156 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4157 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4158 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4159 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4160 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4161 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4162 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4163 2024-09-15T23:31:40.308 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4164 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4165 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4166 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4167 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4168 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4169 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4170 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4171 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4172 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4173 2024-09-15T23:31:40.309 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4174 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 5002 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 548 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 549 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6102 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6104 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6106 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6110 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6111 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6112 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6113 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6117 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 826 2024-09-15T23:31:40.310 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 827 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 828 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 15 23:16 pg_filenode.map 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Sep 15 23:16 pg_internal.init 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:16 PG_VERSION 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Sep 15 23:16 .. 2024-09-15T23:31:40.311 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/global: 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:total 560 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Sep 15 23:16 . 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1213 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1213_fsm 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1213_vm 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1214 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1214_fsm 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1214_vm 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 1232 2024-09-15T23:31:40.312 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 1233 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1260 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1260_fsm 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1260_vm 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1261 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1261_fsm 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1261_vm 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1262 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 1262_fsm 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 1262_vm 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2396 2024-09-15T23:31:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Sep 15 23:16 2396_fsm 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2396_vm 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2397 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2671 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2672 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2676 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2677 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2694 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2695 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2697 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Sep 15 23:16 2698 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2846 2024-09-15T23:31:40.314 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2847 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2964 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2965 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 2966 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 2967 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 3592 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 3593 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4060 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4061 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4175 2024-09-15T23:31:40.315 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4176 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4177 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4178 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4181 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4182 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4183 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4184 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 4185 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 4186 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6000 2024-09-15T23:31:40.316 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6001 2024-09-15T23:31:40.317 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6002 2024-09-15T23:31:40.317 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 6100 2024-09-15T23:31:40.317 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6114 2024-09-15T23:31:40.317 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 6115 2024-09-15T23:31:40.317 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:31 pg_control 2024-09-15T23:31:40.317 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Sep 15 23:16 pg_filenode.map 2024-09-15T23:31:40.317 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Sep 15 23:16 pg_internal.init 2024-09-15T23:31:40.317 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.317 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/log: 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:total 2 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1747 Sep 15 23:31 postgresql-Sun.log 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.318 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:total 1 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:31 . 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 mappings 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Sep 15 23:31 replorigin_checkpoint 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 snapshots 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:31 .. 2024-09-15T23:31:40.319 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:31 .. 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 . 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 members 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 offsets 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:total 8 2024-09-15T23:31:40.320 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 .. 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:16 0000 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:total 8 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Sep 15 23:16 .. 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Sep 15 23:21 0000 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.321 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.322 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:total 15 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Sep 15 23:31 . 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Sep 15 23:31 db_0.stat 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Sep 15 23:31 db_13494.stat 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Sep 15 23:31 global.stat 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:31 . 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-09-15T23:31:40.323 INFO:teuthology.orchestra.run.smithi062.stdout:total 200 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:31 . 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 204800 Sep 15 23:31 000B 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.324 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:total 966656 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 60 Sep 15 23:31 . 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:31 000000010000000000000062 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 000000010000000000000063 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 000000010000000000000064 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 000000010000000000000065 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 000000010000000000000066 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:23 000000010000000000000067 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 000000010000000000000068 2024-09-15T23:31:40.325 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 000000010000000000000069 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 00000001000000000000006A 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 00000001000000000000006B 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:24 00000001000000000000006C 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:24 00000001000000000000006D 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 00000001000000000000006E 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 00000001000000000000006F 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:23 000000010000000000000070 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 000000010000000000000071 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:23 000000010000000000000072 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 000000010000000000000073 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 000000010000000000000074 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 000000010000000000000075 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 000000010000000000000076 2024-09-15T23:31:40.326 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 000000010000000000000077 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:25 000000010000000000000078 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 000000010000000000000079 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 00000001000000000000007A 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 00000001000000000000007B 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 00000001000000000000007C 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:23 00000001000000000000007D 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:22 00000001000000000000007E 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 00000001000000000000007F 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:21 000000010000000000000080 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:26 000000010000000000000081 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:28 000000010000000000000082 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:27 000000010000000000000083 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:28 000000010000000000000084 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:28 000000010000000000000085 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:26 000000010000000000000086 2024-09-15T23:31:40.327 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:27 000000010000000000000087 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:31 000000010000000000000088 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:29 000000010000000000000089 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:28 00000001000000000000008A 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:30 00000001000000000000008B 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:27 00000001000000000000008C 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:28 00000001000000000000008D 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:26 00000001000000000000008E 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:28 00000001000000000000008F 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:27 000000010000000000000090 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:28 000000010000000000000091 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:29 000000010000000000000092 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:26 000000010000000000000093 2024-09-15T23:31:40.328 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:26 000000010000000000000094 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:27 000000010000000000000095 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:27 000000010000000000000096 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:30 000000010000000000000097 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:27 000000010000000000000098 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:26 000000010000000000000099 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:26 00000001000000000000009A 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:26 00000001000000000000009B 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Sep 15 23:29 00000001000000000000009C 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 archive_status 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:total 0 2024-09-15T23:31:40.329 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Sep 15 23:16 . 2024-09-15T23:31:40.330 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 60 Sep 15 23:31 .. 2024-09-15T23:31:40.330 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:40.330 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-09-15T23:31:40.330 INFO:teuthology.orchestra.run.smithi062.stdout:total 192 2024-09-15T23:31:40.330 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Sep 15 23:16 . 2024-09-15T23:31:40.330 INFO:teuthology.orchestra.run.smithi062.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Sep 15 23:31 .. 2024-09-15T23:31:40.330 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Sep 15 23:31 0000 2024-09-15T23:31:40.331 DEBUG:teuthology.run_tasks:Unwinding manager quiescer 2024-09-15T23:31:40.345 INFO:tasks.quiescer:joining Quiescers 2024-09-15T23:31:40.345 WARNING:tasks.quiescer:The quiescer is requested to stop, running cancel all 2024-09-15T23:31:40.345 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:31:40.734 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:41.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:41 smithi062 ceph-mon[30949]: pgmap v832: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.8 MiB/s wr, 745 op/s 2024-09-15T23:31:41.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:41 smithi067 ceph-mon[35665]: pgmap v832: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.8 MiB/s wr, 745 op/s 2024-09-15T23:31:41.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:41 smithi148 ceph-mon[35610]: pgmap v832: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.8 MiB/s wr, 745 op/s 2024-09-15T23:31:43.417 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:43 smithi062 ceph-mon[30949]: pgmap v833: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.5 MiB/s wr, 718 op/s 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.406+0000 7f85618e5640 1 -- 172.21.15.62:0/1252296602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c106f10 msgr2=0x7f855c10ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.406+0000 7f85618e5640 1 --2- 172.21.15.62:0/1252296602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c106f10 0x7f855c10ea90 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f8554009900 tx=0x7f855402a410 comp rx=0 tx=0).stop 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.406+0000 7f85618e5640 1 -- 172.21.15.62:0/1252296602 shutdown_connections 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.406+0000 7f85618e5640 1 --2- 172.21.15.62:0/1252296602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c106f10 0x7f855c10ea90 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.406+0000 7f85618e5640 1 --2- 172.21.15.62:0/1252296602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f855c102790 0x7f855c1067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.406+0000 7f85618e5640 1 --2- 172.21.15.62:0/1252296602 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f855c101de0 0x7f855c1021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.406+0000 7f85618e5640 1 -- 172.21.15.62:0/1252296602 >> 172.21.15.62:0/1252296602 conn(0x7f855c0fd8c0 msgr2=0x7f855c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.406+0000 7f85618e5640 1 -- 172.21.15.62:0/1252296602 shutdown_connections 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.406+0000 7f85618e5640 1 -- 172.21.15.62:0/1252296602 wait complete. 2024-09-15T23:31:43.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f85618e5640 1 Processor -- start 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f85618e5640 1 -- start start 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f85618e5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f855c101de0 0x7f855c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f85618e5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c102790 0x7f855c198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f85618e5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f855c106f10 0x7f855c19ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f85618e5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f855c111f70 con 0x7f855c102790 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f85618e5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f855c111df0 con 0x7f855c101de0 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f85618e5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f855c1120f0 con 0x7f855c106f10 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f855a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c102790 0x7f855c198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f855a7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c102790 0x7f855c198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59592/0 (socket says 172.21.15.62:59592) 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f855a7fc640 1 -- 172.21.15.62:0/2005118047 learned_addr learned my addr 172.21.15.62:0/2005118047 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f855affd640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f855c101de0 0x7f855c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:43.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.407+0000 7f855b7fe640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f855c106f10 0x7f855c19ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f855a7fc640 1 -- 172.21.15.62:0/2005118047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f855c106f10 msgr2=0x7f855c19ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f855a7fc640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f855c106f10 0x7f855c19ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f855a7fc640 1 -- 172.21.15.62:0/2005118047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f855c101de0 msgr2=0x7f855c1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f855a7fc640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f855c101de0 0x7f855c1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f855a7fc640 1 -- 172.21.15.62:0/2005118047 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855c19d3a0 con 0x7f855c102790 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f855b7fe640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f855c106f10 0x7f855c19ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f855a7fc640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c102790 0x7f855c198930 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f854400d950 tx=0x7f854400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f85608e3640 1 -- 172.21.15.62:0/2005118047 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544017030 con 0x7f855c102790 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855c19d690 con 0x7f855c102790 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f85608e3640 1 -- 172.21.15.62:0/2005118047 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8544004020 con 0x7f855c102790 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f85608e3640 1 -- 172.21.15.62:0/2005118047 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85440047a0 con 0x7f855c102790 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.408+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f855c1a4f70 con 0x7f855c102790 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.409+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f855c1021c0 con 0x7f855c102790 2024-09-15T23:31:43.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.410+0000 7f85608e3640 1 -- 172.21.15.62:0/2005118047 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f85440041d0 con 0x7f855c102790 2024-09-15T23:31:43.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.411+0000 7f85608e3640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f85480a3240 0x7f85480a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:43.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.411+0000 7f855affd640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f85480a3240 0x7f85480a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:43.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.411+0000 7f85608e3640 1 -- 172.21.15.62:0/2005118047 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f85440cf680 con 0x7f855c102790 2024-09-15T23:31:43.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.412+0000 7f855affd640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f85480a3240 0x7f85480a5700 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f854c005fd0 tx=0x7f854c009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:43.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.415+0000 7f85608e3640 1 -- 172.21.15.62:0/2005118047 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f854409bfc0 con 0x7f855c102790 2024-09-15T23:31:43.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.626+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f855c02d7c0 con 0x7f855c102790 2024-09-15T23:31:43.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.627+0000 7f85608e3640 1 -- 172.21.15.62:0/2005118047 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7f854409b8c0 con 0x7f855c102790 2024-09-15T23:31:43.628 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:43.628 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:31:43.629 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:31:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f85480a3240 msgr2=0x7f85480a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f85480a3240 0x7f85480a5700 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f854c005fd0 tx=0x7f854c009680 comp rx=0 tx=0).stop 2024-09-15T23:31:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c102790 msgr2=0x7f855c198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:43.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c102790 0x7f855c198930 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f854400d950 tx=0x7f854400de10 comp rx=0 tx=0).stop 2024-09-15T23:31:43.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 shutdown_connections 2024-09-15T23:31:43.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f85480a3240 0x7f85480a5700 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:43.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f855c106f10 0x7f855c19ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:43.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f855c102790 0x7f855c198930 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:43.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 --2- 172.21.15.62:0/2005118047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f855c101de0 0x7f855c1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:43.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 >> 172.21.15.62:0/2005118047 conn(0x7f855c0fd8c0 msgr2=0x7f855c0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:43.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 shutdown_connections 2024-09-15T23:31:43.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:43.629+0000 7f85618e5640 1 -- 172.21.15.62:0/2005118047 wait complete. 2024-09-15T23:31:43.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:43 smithi067 ceph-mon[35665]: pgmap v833: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.5 MiB/s wr, 718 op/s 2024-09-15T23:31:43.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:43 smithi148 ceph-mon[35610]: pgmap v833: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.5 MiB/s wr, 718 op/s 2024-09-15T23:31:44.355 DEBUG:tasks.quiescer.fs.[cephfs]:Running ceph command: 'tell mds.24497 quiesce db --cancel --all' 2024-09-15T23:31:44.356 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- timeout 105 ceph tell mds.24497 quiesce db --cancel --all 2024-09-15T23:31:44.613 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:44.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:44 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2005118047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:44.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:44 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2005118047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:44.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:44 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2005118047' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:45.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.115+0000 7f6405fcc640 1 -- 172.21.15.62:0/2466590307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 msgr2=0x7f6400103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.115+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2466590307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 0x7f6400103110 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009900 tx=0x7f63f002a480 comp rx=0 tx=0).stop 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.115+0000 7f6405fcc640 1 -- 172.21.15.62:0/2466590307 shutdown_connections 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.115+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2466590307 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f640010c160 0x7f640010e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.115+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2466590307 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6400103650 0x7f640010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.115+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2466590307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 0x7f6400103110 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.115+0000 7f6405fcc640 1 -- 172.21.15.62:0/2466590307 >> 172.21.15.62:0/2466590307 conn(0x7f64000fc790 msgr2=0x7f64000febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.115+0000 7f6405fcc640 1 -- 172.21.15.62:0/2466590307 shutdown_connections 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.115+0000 7f6405fcc640 1 -- 172.21.15.62:0/2466590307 wait complete. 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f6405fcc640 1 Processor -- start 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f6405fcc640 1 -- start start 2024-09-15T23:31:45.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f6405fcc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 0x7f6400198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f6405fcc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6400103650 0x7f64001989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f6405fcc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640010c160 0x7f640019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f6405fcc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6400111f60 con 0x7f6400102d30 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f6405fcc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6400111de0 con 0x7f640010c160 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f6405fcc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f64001120e0 con 0x7f6400103650 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f63ff7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 0x7f6400198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f63ff7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 0x7f6400198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59606/0 (socket says 172.21.15.62:59606) 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f63ff7fe640 1 -- 172.21.15.62:0/2860925776 learned_addr learned my addr 172.21.15.62:0/2860925776 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f63feffd640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6400103650 0x7f64001989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.116+0000 7f63fffff640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640010c160 0x7f640019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f63ff7fe640 1 -- 172.21.15.62:0/2860925776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6400103650 msgr2=0x7f64001989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6400103650 0x7f64001989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f63ff7fe640 1 -- 172.21.15.62:0/2860925776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640010c160 msgr2=0x7f640019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640010c160 0x7f640019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f63ff7fe640 1 -- 172.21.15.62:0/2860925776 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640019d410 con 0x7f6400102d30 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 0x7f6400198460 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f63f0006010 tx=0x7f63f00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f63fcff9640 1 -- 172.21.15.62:0/2860925776 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f003f030 con 0x7f6400102d30 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f63fcff9640 1 -- 172.21.15.62:0/2860925776 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f0003ce0 con 0x7f6400102d30 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f6405fcc640 1 -- 172.21.15.62:0/2860925776 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f640019d6a0 con 0x7f6400102d30 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f63fcff9640 1 -- 172.21.15.62:0/2860925776 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f0035070 con 0x7f6400102d30 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.117+0000 7f6405fcc640 1 -- 172.21.15.62:0/2860925776 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64001a4fe0 con 0x7f6400102d30 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.119+0000 7f63fcff9640 1 -- 172.21.15.62:0/2860925776 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f63f0042030 con 0x7f6400102d30 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f64083a5740 1 Processor -- start 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f64083a5740 1 -- start start 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f64083a5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f64083a5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8adb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f64083a5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8adbbe0 0x5570e8addfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f64083a5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5570e8a35200 con 0x5570e8ad0a00 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f64083a5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5570e89dbdc0 con 0x5570e8adbbe0 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f64083a5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5570e8a311c0 con 0x5570e8ad1320 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f63fffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f63fffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59614/0 (socket says 172.21.15.62:59614) 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f63fffff640 1 -- 172.21.15.62:0/1266727954 learned_addr learned my addr 172.21.15.62:0/1266727954 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f63feffd640 1 --2- 172.21.15.62:0/1266727954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8adbbe0 0x5570e8addfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.120+0000 7f63fcff9640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f63e80a3240 0x7f63e80a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63fcff9640 1 -- 172.21.15.62:0/2860925776 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f63f00f01a0 con 0x7f6400102d30 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/1266727954 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8adb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63fffff640 1 -- 172.21.15.62:0/1266727954 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 msgr2=0x5570e8adb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63fffff640 1 --2- 172.21.15.62:0/1266727954 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8adb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63fffff640 1 -- 172.21.15.62:0/1266727954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8adbbe0 msgr2=0x5570e8addfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:45.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63fffff640 1 --2- 172.21.15.62:0/1266727954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8adbbe0 0x5570e8addfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63fffff640 1 -- 172.21.15.62:0/1266727954 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570e87a6620 con 0x5570e8ad0a00 2024-09-15T23:31:45.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63fffff640 1 --2- 172.21.15.62:0/1266727954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad0de0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009fc0 tx=0x7f63f8010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2189c9d7345566f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:45.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63fffff640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f63e80a3240 0x7f63e80a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63e5ffb640 1 -- 172.21.15.62:0/1266727954 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f801c770 con 0x5570e8ad0a00 2024-09-15T23:31:45.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63e5ffb640 1 -- 172.21.15.62:0/1266727954 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f801cc70 con 0x5570e8ad0a00 2024-09-15T23:31:45.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.121+0000 7f63e5ffb640 1 -- 172.21.15.62:0/1266727954 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f801b570 con 0x5570e8ad0a00 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f63fffff640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f63e80a3240 0x7f63e80a5700 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f63f8010fd0 tx=0x7f63f801bc90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f64083a5740 1 -- 172.21.15.62:0/1266727954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 msgr2=0x5570e8ad0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f64083a5740 1 --2- 172.21.15.62:0/1266727954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad0de0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009fc0 tx=0x7f63f8010800 comp rx=0 tx=0).stop 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f64083a5740 1 -- 172.21.15.62:0/1266727954 shutdown_connections 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f64083a5740 1 --2- 172.21.15.62:0/1266727954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8adbbe0 0x5570e8addfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f64083a5740 1 --2- 172.21.15.62:0/1266727954 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8adb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f64083a5740 1 --2- 172.21.15.62:0/1266727954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad0de0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f64083a5740 1 -- 172.21.15.62:0/1266727954 >> 172.21.15.62:0/1266727954 conn(0x5570e8accf90 msgr2=0x5570e8acd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f64083a5740 1 -- 172.21.15.62:0/1266727954 shutdown_connections 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.122+0000 7f64083a5740 1 -- 172.21.15.62:0/1266727954 wait complete. 2024-09-15T23:31:45.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f64083a5740 1 -- start start 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f64083a5740 1 Processor -- start 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f64083a5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8ad0a00 0x5570e8adb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f64083a5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8ad8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f64083a5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8adbbe0 0x5570e8ad8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f64083a5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5570e8ae2bc0 con 0x5570e8adbbe0 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f64083a5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5570e89cfc50 con 0x5570e8ad0a00 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f64083a5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5570e8ae2d40 con 0x5570e8ad1320 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f64083a5740 10 client.0 _ll_register_callbacks cb 0x5570e89e1ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.123+0000 7f63fffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8adbbe0 0x5570e8ad8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63fffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8adbbe0 0x5570e8ad8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59630/0 (socket says 172.21.15.62:59630) 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63fffff640 1 -- 172.21.15.62:0/509277299 learned_addr learned my addr 172.21.15.62:0/509277299 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:45.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8ad0a00 0x5570e8adb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63feffd640 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8ad8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63fffff640 1 -- 172.21.15.62:0/509277299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 msgr2=0x5570e8ad8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63fffff640 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8ad8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63fffff640 1 -- 172.21.15.62:0/509277299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8ad0a00 msgr2=0x5570e8adb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63fffff640 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8ad0a00 0x5570e8adb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63fffff640 1 -- 172.21.15.62:0/509277299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570e8ad93f0 con 0x5570e8adbbe0 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8ad0a00 0x5570e8adb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63fffff640 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8adbbe0 0x5570e8ad8bc0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f63f8010800 tx=0x7f63f8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bed3dd0b936922f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63deffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63deffd640 1 -- 172.21.15.62:0/509277299 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8009b10 con 0x5570e8adbbe0 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63deffd640 1 -- 172.21.15.62:0/509277299 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f63f801c450 con 0x5570e8adbbe0 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f64083a5740 1 -- 172.21.15.62:0/509277299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63f801cc70 con 0x5570e8adbbe0 2024-09-15T23:31:45.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f63deffd640 1 -- 172.21.15.62:0/509277299 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f63f8039050 con 0x5570e8adbbe0 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f64083a5740 10 client.19065 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.124+0000 7f64083a5740 1 -- 172.21.15.62:0/509277299 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5570e8aee5f0 con 0x5570e8adbbe0 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.125+0000 7f63deffd640 1 -- 172.21.15.62:0/509277299 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f801cc70 con 0x5570e8adbbe0 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.125+0000 7f63deffd640 1 -- 172.21.15.62:0/509277299 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f63f800f040 con 0x5570e8adbbe0 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.125+0000 7f64083a5740 10 client.19065 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.125+0000 7f64083a5740 10 client.19065 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.125+0000 7f64083a5740 20 client.19065 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.125+0000 7f64083a5740 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5570e8aefab0 0x5570e8af1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.125+0000 7f64083a5740 4 client.19065 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.125+0000 7f64083a5740 1 -- 172.21.15.62:0/509277299 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570e8a42770 con 0x5570e8aefab0 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.125+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5570e8aefab0 0x5570e8af1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.126+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5570e8aefab0 0x5570e8af1ea0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31aab9e0b87ba82 server_cookie=8f6d9f030e10de88 in_seq=0 out_seq=0 2024-09-15T23:31:45.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.126+0000 7f63deffd640 10 client.19065 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.126+0000 7f63deffd640 1 -- 172.21.15.62:0/509277299 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5570e8a42770 con 0x5570e8aefab0 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.126+0000 7f63deffd640 10 client.19065 handle_command_reply: tid=0 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.126+0000 7f64083a5740 1 client.19065 shutdown 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 -- 172.21.15.62:0/509277299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8adbbe0 msgr2=0x5570e8ad8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8adbbe0 0x5570e8ad8bc0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f63f8010800 tx=0x7f63f8018000 comp rx=0 tx=0).stop 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 -- 172.21.15.62:0/509277299 shutdown_connections 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5570e8aefab0 0x5570e8af1ea0 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8adbbe0 0x5570e8ad8bc0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8ad8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 --2- 172.21.15.62:0/509277299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8ad0a00 0x5570e8adb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 -- 172.21.15.62:0/509277299 >> 172.21.15.62:0/509277299 conn(0x5570e8accf90 msgr2=0x5570e8acc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f63deffd640 0 client.19065 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 -- 172.21.15.62:0/509277299 shutdown_connections 2024-09-15T23:31:45.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 1 -- 172.21.15.62:0/509277299 wait complete. 2024-09-15T23:31:45.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.127+0000 7f64083a5740 20 client.19065 trim_cache size 0 max 16384 2024-09-15T23:31:45.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f64083a5740 1 Processor -- start 2024-09-15T23:31:45.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f64083a5740 1 -- start start 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f64083a5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 0x5570e8acc020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f64083a5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f64083a5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8aebe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f64083a5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f64001af330 con 0x5570e8ad0a00 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f64083a5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64001acbd0 con 0x5570e8accf90 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f64083a5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f64001aaa20 con 0x5570e8ad1320 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f63feffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f63feffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59638/0 (socket says 172.21.15.62:59638) 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f63feffd640 1 -- 172.21.15.62:0/2236514144 learned_addr learned my addr 172.21.15.62:0/2236514144 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/2236514144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 0x5570e8acc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f63feffd640 1 -- 172.21.15.62:0/2236514144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 msgr2=0x5570e8aebe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f63feffd640 1 --2- 172.21.15.62:0/2236514144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8aebe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f63feffd640 1 -- 172.21.15.62:0/2236514144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 msgr2=0x5570e8acc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:45.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f63feffd640 1 --2- 172.21.15.62:0/2236514144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 0x5570e8acc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.144+0000 7f63feffd640 1 -- 172.21.15.62:0/2236514144 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570e8accdc0 con 0x5570e8ad0a00 2024-09-15T23:31:45.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f63feffd640 1 --2- 172.21.15.62:0/2236514144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad6f50 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f63f400ab10 tx=0x7f63f4010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a541655a3d29fb server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:45.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f63e57fa640 1 -- 172.21.15.62:0/2236514144 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005220 con 0x5570e8ad0a00 2024-09-15T23:31:45.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f63e57fa640 1 -- 172.21.15.62:0/2236514144 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f4004c80 con 0x5570e8ad0a00 2024-09-15T23:31:45.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f63e57fa640 1 -- 172.21.15.62:0/2236514144 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f401b7c0 con 0x5570e8ad0a00 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f64083a5740 1 -- 172.21.15.62:0/2236514144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 msgr2=0x5570e8ad6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f64083a5740 1 --2- 172.21.15.62:0/2236514144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad6f50 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f63f400ab10 tx=0x7f63f4010430 comp rx=0 tx=0).stop 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f64083a5740 1 -- 172.21.15.62:0/2236514144 shutdown_connections 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f64083a5740 1 --2- 172.21.15.62:0/2236514144 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8aebe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f64083a5740 1 --2- 172.21.15.62:0/2236514144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ad6f50 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f64083a5740 1 --2- 172.21.15.62:0/2236514144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 0x5570e8acc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.145+0000 7f64083a5740 1 -- 172.21.15.62:0/2236514144 >> 172.21.15.62:0/2236514144 conn(0x5570e8ad8d20 msgr2=0x5570e8ada5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 -- 172.21.15.62:0/2236514144 shutdown_connections 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 -- 172.21.15.62:0/2236514144 wait complete. 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 -- start start 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 Processor -- start 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 0x5570e8ae44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ae4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.151 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8b59530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f64001ab090 con 0x5570e8ad0a00 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f64001ac500 con 0x5570e8accf90 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f64001ab760 con 0x5570e8ad1320 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.146+0000 7f64083a5740 10 client.0 _ll_register_callbacks cb 0x7f63f8005c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63feffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ae4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63feffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ae4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59646/0 (socket says 172.21.15.62:59646) 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63feffd640 1 -- 172.21.15.62:0/1298046590 learned_addr learned my addr 172.21.15.62:0/1298046590 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 0x5570e8ae44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63fffff640 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8b59530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63feffd640 1 -- 172.21.15.62:0/1298046590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 msgr2=0x5570e8b59530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63feffd640 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8b59530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63feffd640 1 -- 172.21.15.62:0/1298046590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 msgr2=0x5570e8ae44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:45.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63feffd640 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 0x5570e8ae44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63feffd640 1 -- 172.21.15.62:0/1298046590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570e8b59b20 con 0x5570e8ad0a00 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63feffd640 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ae4a20 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f63f4005720 tx=0x7f63f4010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d7d788ebafc4c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63deffd640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63deffd640 1 -- 172.21.15.62:0/1298046590 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f402b020 con 0x5570e8ad0a00 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63deffd640 1 -- 172.21.15.62:0/1298046590 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f63f40048c0 con 0x5570e8ad0a00 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f64083a5740 1 -- 172.21.15.62:0/1298046590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63f4004c80 con 0x5570e8ad0a00 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.147+0000 7f63deffd640 1 -- 172.21.15.62:0/1298046590 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f63f4036050 con 0x5570e8ad0a00 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f64083a5740 10 client.19071 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f64083a5740 1 -- 172.21.15.62:0/1298046590 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5570e8b5adc0 con 0x5570e8ad0a00 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f63deffd640 1 -- 172.21.15.62:0/1298046590 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f4004c80 con 0x5570e8ad0a00 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f63deffd640 1 -- 172.21.15.62:0/1298046590 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f63f401b520 con 0x5570e8ad0a00 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f64083a5740 10 client.19071 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f64083a5740 10 client.19071 resolve_mds: validated gid 24497 aka daemon mds.b 2024-09-15T23:31:45.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f64083a5740 20 client.19071 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f64083a5740 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5570e8adbbe0 0x5570e8b5d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f64083a5740 4 client.19071 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "quiesce db", "cancel": true, "all": true}] 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f64083a5740 1 -- 172.21.15.62:0/1298046590 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "quiesce db", "cancel": true, "all": true}) -- 0x5570e8a3cc80 con 0x5570e8adbbe0 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.148+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5570e8adbbe0 0x5570e8b5d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f63ff7fe640 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5570e8adbbe0 0x5570e8b5d600 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f600ce9786102e5 server_cookie=4c9b22f987c16cb in_seq=0 out_seq=0 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f63deffd640 10 client.19071 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f63deffd640 1 -- 172.21.15.62:0/1298046590 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+80 (crc 0 0 0) 0x5570e8a3cc80 con 0x5570e8adbbe0 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f63deffd640 10 client.19071 handle_command_reply: tid=0 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 client.19071 shutdown 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 -- 172.21.15.62:0/1298046590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 msgr2=0x5570e8ae4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ae4a20 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f63f4005720 tx=0x7f63f4010f00 comp rx=0 tx=0).stop 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 -- 172.21.15.62:0/1298046590 shutdown_connections 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5570e8adbbe0 0x5570e8b5d600 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5570e8ad1320 0x5570e8b59530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f63deffd640 0 client.19071 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570e8ad0a00 0x5570e8ae4a20 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 --2- 172.21.15.62:0/1298046590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5570e8accf90 0x5570e8ae44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 -- 172.21.15.62:0/1298046590 >> 172.21.15.62:0/1298046590 conn(0x5570e8ad8d20 msgr2=0x5570e8ada430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 -- 172.21.15.62:0/1298046590 shutdown_connections 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 1 -- 172.21.15.62:0/1298046590 wait complete. 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.149+0000 7f64083a5740 20 client.19071 trim_cache size 0 max 16384 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 36, 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stdout: "leader": 24497, 2024-09-15T23:31:45.155 INFO:teuthology.orchestra.run.smithi062.stdout: "set_version": 60, 2024-09-15T23:31:45.156 INFO:teuthology.orchestra.run.smithi062.stdout: "sets": {} 2024-09-15T23:31:45.156 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:31:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 -- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f63e80a3240 msgr2=0x7f63e80a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f63e80a3240 0x7f63e80a5700 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f63f8010fd0 tx=0x7f63f801bc90 comp rx=0 tx=0).stop 2024-09-15T23:31:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 -- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 msgr2=0x7f6400198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 0x7f6400198460 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f63f0006010 tx=0x7f63f00042f0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 -- 172.21.15.62:0/2860925776 shutdown_connections 2024-09-15T23:31:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f63e80a3240 0x7f63e80a5700 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f640010c160 0x7f640019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f6400103650 0x7f64001989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 --2- 172.21.15.62:0/2860925776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6400102d30 0x7f6400198460 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 -- 172.21.15.62:0/2860925776 >> 172.21.15.62:0/2860925776 conn(0x7f64000fc790 msgr2=0x7f64000fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 -- 172.21.15.62:0/2860925776 shutdown_connections 2024-09-15T23:31:45.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:45.150+0000 7f6405fcc640 1 -- 172.21.15.62:0/2860925776 wait complete. 2024-09-15T23:31:45.732 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:45 smithi062 ceph-mon[30949]: pgmap v834: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.6 MiB/s wr, 727 op/s 2024-09-15T23:31:45.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:45 smithi067 ceph-mon[35665]: pgmap v834: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.6 MiB/s wr, 727 op/s 2024-09-15T23:31:45.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:45 smithi148 ceph-mon[35610]: pgmap v834: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.6 MiB/s wr, 727 op/s 2024-09-15T23:31:45.965 INFO:tasks.quiescer:done joining Quiescers 2024-09-15T23:31:45.965 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-09-15T23:31:45.975 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T23:31:45.984 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-09-15T23:31:46.023 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-09-15T23:31:47.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:47 smithi062 ceph-mon[30949]: pgmap v835: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 5.8 MiB/s wr, 646 op/s 2024-09-15T23:31:47.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:47 smithi067 ceph-mon[35665]: pgmap v835: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 5.8 MiB/s wr, 646 op/s 2024-09-15T23:31:47.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:47 smithi148 ceph-mon[35610]: pgmap v835: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 5.8 MiB/s wr, 646 op/s 2024-09-15T23:31:49.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:49 smithi062 ceph-mon[30949]: pgmap v836: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 3.5 MiB/s wr, 400 op/s 2024-09-15T23:31:49.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:49 smithi067 ceph-mon[35665]: pgmap v836: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 3.5 MiB/s wr, 400 op/s 2024-09-15T23:31:49.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:49 smithi148 ceph-mon[35610]: pgmap v836: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 3.5 MiB/s wr, 400 op/s 2024-09-15T23:31:50.455 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-09-15T23:31:50.730 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.215+0000 7f7783c76640 1 -- 172.21.15.62:0/3059352809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 msgr2=0x7f777c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.215+0000 7f7783c76640 1 --2- 172.21.15.62:0/3059352809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 0x7f777c106e60 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f7778009930 tx=0x7f777802a420 comp rx=0 tx=0).stop 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.215+0000 7f7783c76640 1 -- 172.21.15.62:0/3059352809 shutdown_connections 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.215+0000 7f7783c76640 1 --2- 172.21.15.62:0/3059352809 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f777c101420 0x7f777c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.215+0000 7f7783c76640 1 --2- 172.21.15.62:0/3059352809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777c100a80 0x7f777c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.215+0000 7f7783c76640 1 --2- 172.21.15.62:0/3059352809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 0x7f777c106e60 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.215+0000 7f7783c76640 1 -- 172.21.15.62:0/3059352809 >> 172.21.15.62:0/3059352809 conn(0x7f777c0fc7a0 msgr2=0x7f777c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 -- 172.21.15.62:0/3059352809 shutdown_connections 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 -- 172.21.15.62:0/3059352809 wait complete. 2024-09-15T23:31:51.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 Processor -- start 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 -- start start 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f777c100a80 0x7f777c199270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777c101420 0x7f777c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 0x7f777c1a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f777c110ea0 con 0x7f777c106a80 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f777c110d20 con 0x7f777c101420 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.216+0000 7f7783c76640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f777c111020 con 0x7f777c100a80 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7782213640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 0x7f777c1a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7782213640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 0x7f777c1a42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59664/0 (socket says 172.21.15.62:59664) 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7782213640 1 -- 172.21.15.62:0/987091259 learned_addr learned my addr 172.21.15.62:0/987091259 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7781211640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777c101420 0x7f777c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7781a12640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f777c100a80 0x7f777c199270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7782213640 1 -- 172.21.15.62:0/987091259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f777c100a80 msgr2=0x7f777c199270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7782213640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f777c100a80 0x7f777c199270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7782213640 1 -- 172.21.15.62:0/987091259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777c101420 msgr2=0x7f777c1997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7782213640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777c101420 0x7f777c1997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7782213640 1 -- 172.21.15.62:0/987091259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c1a4830 con 0x7f777c106a80 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7782213640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 0x7f777c1a42f0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f777400b970 tx=0x7f777400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7772ffd640 1 -- 172.21.15.62:0/987091259 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7774019070 con 0x7f777c106a80 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7783c76640 1 -- 172.21.15.62:0/987091259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777c1a4a00 con 0x7f777c106a80 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7772ffd640 1 -- 172.21.15.62:0/987091259 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7774014440 con 0x7f777c106a80 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7772ffd640 1 -- 172.21.15.62:0/987091259 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77740075d0 con 0x7f777c106a80 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.217+0000 7f7783c76640 1 -- 172.21.15.62:0/987091259 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f777c1a4f10 con 0x7f777c106a80 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.220+0000 7f7772ffd640 1 -- 172.21.15.62:0/987091259 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f777401c070 con 0x7f777c106a80 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.220+0000 7f778604f740 1 Processor -- start 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.220+0000 7f778604f740 1 -- start start 2024-09-15T23:31:51.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.220+0000 7f778604f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada12880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.220+0000 7f778604f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada12dc0 0x5616ada1d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.220+0000 7f778604f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada1d680 0x5616ada1fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.220+0000 7f778604f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5616ad9782e0 con 0x5616ada124a0 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f778604f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5616ad91fce0 con 0x5616ada12dc0 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f778604f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5616ad9742a0 con 0x5616ada1d680 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781211640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada12880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781a12640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada12dc0 0x5616ada1d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781211640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada12880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59674/0 (socket says 172.21.15.62:59674) 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781211640 1 -- 172.21.15.62:0/3664052876 learned_addr learned my addr 172.21.15.62:0/3664052876 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7772ffd640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77680a3240 0x7f77680a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781211640 1 --2- 172.21.15.62:0/3664052876 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada1d680 0x5616ada1fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7782213640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77680a3240 0x7f77680a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781a12640 1 -- 172.21.15.62:0/3664052876 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada1d680 msgr2=0x5616ada1fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781a12640 1 --2- 172.21.15.62:0/3664052876 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada1d680 0x5616ada1fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781a12640 1 -- 172.21.15.62:0/3664052876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 msgr2=0x5616ada12880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781a12640 1 --2- 172.21.15.62:0/3664052876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada12880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781a12640 1 -- 172.21.15.62:0/3664052876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ad904440 con 0x5616ada12dc0 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7772ffd640 1 -- 172.21.15.62:0/987091259 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f77740cf290 con 0x7f777c106a80 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.221+0000 7f7781a12640 1 --2- 172.21.15.62:0/3664052876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada12dc0 0x5616ada1d140 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7778009230 tx=0x7f77780316e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fcc5fbe86132ef4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f7767fff640 1 -- 172.21.15.62:0/3664052876 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778051070 con 0x5616ada12dc0 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f7782213640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77680a3240 0x7f77680a5700 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f777409b510 tx=0x7f7774004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f7767fff640 1 -- 172.21.15.62:0/3664052876 <== mon.2 v2:172.21.15.67:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7778005020 con 0x5616ada12dc0 2024-09-15T23:31:51.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f7767fff640 1 -- 172.21.15.62:0/3664052876 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7778035040 con 0x5616ada12dc0 2024-09-15T23:31:51.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f778604f740 1 -- 172.21.15.62:0/3664052876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada12dc0 msgr2=0x5616ada1d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:51.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f778604f740 1 --2- 172.21.15.62:0/3664052876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada12dc0 0x5616ada1d140 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7778009230 tx=0x7f77780316e0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f778604f740 1 -- 172.21.15.62:0/3664052876 shutdown_connections 2024-09-15T23:31:51.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f778604f740 1 --2- 172.21.15.62:0/3664052876 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada1d680 0x5616ada1fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f778604f740 1 --2- 172.21.15.62:0/3664052876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada12dc0 0x5616ada1d140 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f778604f740 1 --2- 172.21.15.62:0/3664052876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada12880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.222+0000 7f778604f740 1 -- 172.21.15.62:0/3664052876 >> 172.21.15.62:0/3664052876 conn(0x5616ada0ea30 msgr2=0x5616ada0ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 -- 172.21.15.62:0/3664052876 shutdown_connections 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 -- 172.21.15.62:0/3664052876 wait complete. 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 -- start start 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 Processor -- start 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada16540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada16a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 0x5616ada16fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5616ada20190 con 0x5616ada124a0 2024-09-15T23:31:51.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5616ad933050 con 0x5616ada1d680 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5616ada20310 con 0x5616ada12dc0 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.223+0000 7f778604f740 10 client.0 _ll_register_callbacks cb 0x5616ad973160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada16540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada16540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59686/0 (socket says 172.21.15.62:59686) 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 -- 172.21.15.62:0/3527890791 learned_addr learned my addr 172.21.15.62:0/3527890791 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781a12640 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada16a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 0x5616ada16fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 -- 172.21.15.62:0/3527890791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 msgr2=0x5616ada16a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada16a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 -- 172.21.15.62:0/3527890791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 msgr2=0x5616ada16fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 0x5616ada16fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 -- 172.21.15.62:0/3527890791 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ad6e93c0 con 0x5616ada124a0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f7781211640 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada16540 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f776c015840 tx=0x7f776c015d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77eeb83d534a7d8b server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f77657fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f77657fa640 1 -- 172.21.15.62:0/3527890791 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c015a80 con 0x5616ada124a0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f778604f740 1 -- 172.21.15.62:0/3527890791 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7778005020 con 0x5616ada124a0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f77657fa640 1 -- 172.21.15.62:0/3527890791 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f776c005020 con 0x5616ada124a0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f77657fa640 1 -- 172.21.15.62:0/3527890791 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f776c028050 con 0x5616ada124a0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f778604f740 10 client.19083 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.224+0000 7f778604f740 1 -- 172.21.15.62:0/3527890791 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5616ada2bce0 con 0x5616ada124a0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.225+0000 7f77657fa640 1 -- 172.21.15.62:0/3527890791 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7778005020 con 0x5616ada124a0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.225+0000 7f77657fa640 1 -- 172.21.15.62:0/3527890791 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f776c011440 con 0x5616ada124a0 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.225+0000 7f778604f740 10 client.19083 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:51.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.225+0000 7f778604f740 10 client.19083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.225+0000 7f778604f740 20 client.19083 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.225+0000 7f778604f740 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616ada2d0f0 0x5616ada2f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.225+0000 7f778604f740 4 client.19083 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.225+0000 7f778604f740 1 -- 172.21.15.62:0/3527890791 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616ad985850 con 0x5616ada2d0f0 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.225+0000 7f7781a12640 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616ada2d0f0 0x5616ada2f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.226+0000 7f7781a12640 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616ada2d0f0 0x5616ada2f520 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e343970d47ec47b server_cookie=f69e6f205cad9000 in_seq=0 out_seq=0 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.226+0000 7f77657fa640 10 client.19083 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.226+0000 7f77657fa640 1 -- 172.21.15.62:0/3527890791 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5616ad985850 con 0x5616ada2d0f0 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.226+0000 7f77657fa640 10 client.19083 handle_command_reply: tid=0 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.226+0000 7f778604f740 1 client.19083 shutdown 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.226+0000 7f778604f740 1 -- 172.21.15.62:0/3527890791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 msgr2=0x5616ada16540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:51.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.226+0000 7f778604f740 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada16540 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f776c015840 tx=0x7f776c015d20 comp rx=0 tx=0).stop 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f778604f740 1 -- 172.21.15.62:0/3527890791 shutdown_connections 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f778604f740 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616ada2d0f0 0x5616ada2f520 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f778604f740 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 0x5616ada16fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f778604f740 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada16a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f77657fa640 0 client.19083 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f778604f740 1 --2- 172.21.15.62:0/3527890791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada16540 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f778604f740 1 -- 172.21.15.62:0/3527890791 >> 172.21.15.62:0/3527890791 conn(0x5616ada0ea30 msgr2=0x5616ada14760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f778604f740 1 -- 172.21.15.62:0/3527890791 shutdown_connections 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f778604f740 1 -- 172.21.15.62:0/3527890791 wait complete. 2024-09-15T23:31:51.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.227+0000 7f778604f740 20 client.19083 trim_cache size 0 max 16384 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f778604f740 1 Processor -- start 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f778604f740 1 -- start start 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f778604f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada124a0 0x5616ada0de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f778604f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada34df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f778604f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada1d680 0x5616ada35330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f778604f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1af2e0 con 0x5616ada1d680 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f778604f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1acb80 con 0x5616ada124a0 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f778604f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1aa9d0 con 0x5616ada12dc0 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781211640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada1d680 0x5616ada35330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781211640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada1d680 0x5616ada35330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59690/0 (socket says 172.21.15.62:59690) 2024-09-15T23:31:51.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781211640 1 -- 172.21.15.62:0/3746670685 learned_addr learned my addr 172.21.15.62:0/3746670685 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781211640 1 --2- 172.21.15.62:0/3746670685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada124a0 0x5616ada0de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781211640 1 -- 172.21.15.62:0/3746670685 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 msgr2=0x5616ada34df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781a12640 1 --2- 172.21.15.62:0/3746670685 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada34df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781211640 1 --2- 172.21.15.62:0/3746670685 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada34df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781211640 1 -- 172.21.15.62:0/3746670685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada124a0 msgr2=0x5616ada0de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781211640 1 --2- 172.21.15.62:0/3746670685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada124a0 0x5616ada0de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.244+0000 7f7781211640 1 -- 172.21.15.62:0/3746670685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ada2fba0 con 0x5616ada1d680 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f7781211640 1 --2- 172.21.15.62:0/3746670685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada1d680 0x5616ada35330 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f776c005790 tx=0x7f776c015770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67291679d8625dae server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f77677fe640 1 -- 172.21.15.62:0/3746670685 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c00a400 con 0x5616ada1d680 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f77677fe640 1 -- 172.21.15.62:0/3746670685 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776c011440 con 0x5616ada1d680 2024-09-15T23:31:51.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f77677fe640 1 -- 172.21.15.62:0/3746670685 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c0046c0 con 0x5616ada1d680 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f778604f740 1 -- 172.21.15.62:0/3746670685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada1d680 msgr2=0x5616ada35330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f778604f740 1 --2- 172.21.15.62:0/3746670685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada1d680 0x5616ada35330 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f776c005790 tx=0x7f776c015770 comp rx=0 tx=0).stop 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f778604f740 1 -- 172.21.15.62:0/3746670685 shutdown_connections 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f778604f740 1 --2- 172.21.15.62:0/3746670685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada1d680 0x5616ada35330 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f778604f740 1 --2- 172.21.15.62:0/3746670685 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada34df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f778604f740 1 --2- 172.21.15.62:0/3746670685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada124a0 0x5616ada0de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.245+0000 7f778604f740 1 -- 172.21.15.62:0/3746670685 >> 172.21.15.62:0/3746670685 conn(0x5616ada0ea30 msgr2=0x5616ada20d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.246+0000 7f778604f740 1 -- 172.21.15.62:0/3746670685 shutdown_connections 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.246+0000 7f778604f740 1 -- 172.21.15.62:0/3746670685 wait complete. 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.246+0000 7f778604f740 1 -- start start 2024-09-15T23:31:51.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.246+0000 7f778604f740 1 Processor -- start 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f778604f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada29b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f778604f740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada26e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f778604f740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 0x5616ada273b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f778604f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1ab040 con 0x5616ada124a0 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f778604f740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1ac4b0 con 0x5616ada1d680 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f778604f740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f777c1ab710 con 0x5616ada12dc0 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f778604f740 10 client.0 _ll_register_callbacks cb 0x7f776c015dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada29b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada29b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:59698/0 (socket says 172.21.15.62:59698) 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 -- 172.21.15.62:0/3836377108 learned_addr learned my addr 172.21.15.62:0/3836377108 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781a12640 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada26e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 0x5616ada273b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 -- 172.21.15.62:0/3836377108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 msgr2=0x5616ada26e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:51.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada26e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 -- 172.21.15.62:0/3836377108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 msgr2=0x5616ada273b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:51.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 0x5616ada273b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 -- 172.21.15.62:0/3836377108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ada279a0 con 0x5616ada124a0 2024-09-15T23:31:51.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781a12640 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada26e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:51.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f7781211640 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada29b20 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f776c015d60 tx=0x7f776c015800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d782ddacc347088f server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:51.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f77657fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:51.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f77657fa640 1 -- 172.21.15.62:0/3836377108 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c011a80 con 0x5616ada124a0 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f77657fa640 1 -- 172.21.15.62:0/3836377108 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f776c011c20 con 0x5616ada124a0 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.247+0000 7f778604f740 1 -- 172.21.15.62:0/3836377108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f776c011440 con 0x5616ada124a0 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f77657fa640 1 -- 172.21.15.62:0/3836377108 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f776c048050 con 0x5616ada124a0 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f778604f740 10 client.19089 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f778604f740 1 -- 172.21.15.62:0/3836377108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5616ada97be0 con 0x5616ada124a0 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f77657fa640 1 -- 172.21.15.62:0/3836377108 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f776c011440 con 0x5616ada124a0 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f77657fa640 1 -- 172.21.15.62:0/3836377108 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f776c023420 con 0x5616ada124a0 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f778604f740 10 client.19089 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f778604f740 10 client.19089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:51.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f778604f740 20 client.19089 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f778604f740 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616ada98190 0x5616ada9a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f778604f740 4 client.19089 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f778604f740 1 -- 172.21.15.62:0/3836377108 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5616ad97fd60 con 0x5616ada98190 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.248+0000 7f7781a12640 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616ada98190 0x5616ada9a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f7781a12640 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616ada98190 0x5616ada9a580 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=404380051e06d8f6 server_cookie=1f92908fa057dbaa in_seq=0 out_seq=0 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f77657fa640 10 client.19089 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f77657fa640 1 -- 172.21.15.62:0/3836377108 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5616ad97fd60 con 0x5616ada98190 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f77657fa640 10 client.19089 handle_command_reply: tid=0 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 client.19089 shutdown 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 -- 172.21.15.62:0/3836377108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 msgr2=0x5616ada29b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada29b20 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f776c015d60 tx=0x7f776c015800 comp rx=0 tx=0).stop 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 -- 172.21.15.62:0/3836377108 shutdown_connections 2024-09-15T23:31:51.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5616ada98190 0x5616ada9a580 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616ada1d680 0x5616ada273b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5616ada12dc0 0x5616ada26e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 --2- 172.21.15.62:0/3836377108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5616ada124a0 0x5616ada29b20 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f77657fa640 0 client.19089 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:51.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 -- 172.21.15.62:0/3836377108 >> 172.21.15.62:0/3836377108 conn(0x5616ada0ea30 msgr2=0x5616ada13ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:51.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 -- 172.21.15.62:0/3836377108 shutdown_connections 2024-09-15T23:31:51.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.249+0000 7f778604f740 1 -- 172.21.15.62:0/3836377108 wait complete. 2024-09-15T23:31:51.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f778604f740 20 client.19089 trim_cache size 0 max 16384 2024-09-15T23:31:51.258 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:31:51.258 INFO:teuthology.orchestra.run.smithi062.stdout: "status": "no active scrubs running", 2024-09-15T23:31:51.258 INFO:teuthology.orchestra.run.smithi062.stdout: "scrubs": {} 2024-09-15T23:31:51.258 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:31:51.258 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 -- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77680a3240 msgr2=0x7f77680a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77680a3240 0x7f77680a5700 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f777409b510 tx=0x7f7774004a40 comp rx=0 tx=0).stop 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 -- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 msgr2=0x7f777c1a42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 0x7f777c1a42f0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f777400b970 tx=0x7f777400bd70 comp rx=0 tx=0).stop 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 -- 172.21.15.62:0/987091259 shutdown_connections 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77680a3240 0x7f77680a5700 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f777c106a80 0x7f777c1a42f0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777c101420 0x7f777c1997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 --2- 172.21.15.62:0/987091259 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f777c100a80 0x7f777c199270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 -- 172.21.15.62:0/987091259 >> 172.21.15.62:0/987091259 conn(0x7f777c0fc7a0 msgr2=0x7f777c0fce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 -- 172.21.15.62:0/987091259 shutdown_connections 2024-09-15T23:31:51.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:51.250+0000 7f7783c76640 1 -- 172.21.15.62:0/987091259 wait complete. 2024-09-15T23:31:51.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:51 smithi062 ceph-mon[30949]: pgmap v837: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 2.1 MiB/s wr, 246 op/s 2024-09-15T23:31:51.591 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:51 smithi148 ceph-mon[35610]: pgmap v837: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 2.1 MiB/s wr, 246 op/s 2024-09-15T23:31:51.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:51 smithi067 ceph-mon[35665]: pgmap v837: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 2.1 MiB/s wr, 246 op/s 2024-09-15T23:31:51.981 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-15T23:31:51.982 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:31:52.241 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.728+0000 7ff431d01640 1 -- 172.21.15.62:0/492207037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c100a80 msgr2=0x7ff42c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.728+0000 7ff431d01640 1 --2- 172.21.15.62:0/492207037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c100a80 0x7ff42c100ee0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7ff414009930 tx=0x7ff41402a420 comp rx=0 tx=0).stop 2024-09-15T23:31:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.728+0000 7ff431d01640 1 -- 172.21.15.62:0/492207037 shutdown_connections 2024-09-15T23:31:52.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.728+0000 7ff431d01640 1 --2- 172.21.15.62:0/492207037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c101420 0x7ff42c10d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.728+0000 7ff431d01640 1 --2- 172.21.15.62:0/492207037 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c100a80 0x7ff42c100ee0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.728+0000 7ff431d01640 1 --2- 172.21.15.62:0/492207037 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff42c106a80 0x7ff42c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.728+0000 7ff431d01640 1 -- 172.21.15.62:0/492207037 >> 172.21.15.62:0/492207037 conn(0x7ff42c0fc7a0 msgr2=0x7ff42c0febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.728+0000 7ff431d01640 1 -- 172.21.15.62:0/492207037 shutdown_connections 2024-09-15T23:31:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.728+0000 7ff431d01640 1 -- 172.21.15.62:0/492207037 wait complete. 2024-09-15T23:31:52.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff431d01640 1 Processor -- start 2024-09-15T23:31:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff431d01640 1 -- start start 2024-09-15T23:31:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff431d01640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff42c100a80 0x7ff42c1992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff431d01640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c101420 0x7ff42c199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff431d01640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106a80 0x7ff42c1a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff431d01640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c110e80 con 0x7ff42c101420 2024-09-15T23:31:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff431d01640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c110d00 con 0x7ff42c106a80 2024-09-15T23:31:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff431d01640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff42c111000 con 0x7ff42c100a80 2024-09-15T23:31:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff42affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c101420 0x7ff42c199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:52.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.729+0000 7ff42affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c101420 0x7ff42c199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51004/0 (socket says 172.21.15.62:51004) 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff42affd640 1 -- 172.21.15.62:0/2722659220 learned_addr learned my addr 172.21.15.62:0/2722659220 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff42b7fe640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff42c100a80 0x7ff42c1992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff42bfff640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106a80 0x7ff42c1a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff42affd640 1 -- 172.21.15.62:0/2722659220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff42c100a80 msgr2=0x7ff42c1992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff42affd640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff42c100a80 0x7ff42c1992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff42affd640 1 -- 172.21.15.62:0/2722659220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106a80 msgr2=0x7ff42c1a4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff42affd640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106a80 0x7ff42c1a4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff42affd640 1 -- 172.21.15.62:0/2722659220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff42c1a4890 con 0x7ff42c101420 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff42affd640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c101420 0x7ff42c199810 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7ff414009230 tx=0x7ff414004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff428ff9640 1 -- 172.21.15.62:0/2722659220 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff41403f030 con 0x7ff42c101420 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff428ff9640 1 -- 172.21.15.62:0/2722659220 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff414003ad0 con 0x7ff42c101420 2024-09-15T23:31:52.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff42c1a4a60 con 0x7ff42c101420 2024-09-15T23:31:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff428ff9640 1 -- 172.21.15.62:0/2722659220 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414035070 con 0x7ff42c101420 2024-09-15T23:31:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.730+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff42c1a4eb0 con 0x7ff42c101420 2024-09-15T23:31:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.731+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff42c100ee0 con 0x7ff42c101420 2024-09-15T23:31:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.733+0000 7ff428ff9640 1 -- 172.21.15.62:0/2722659220 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff414042030 con 0x7ff42c101420 2024-09-15T23:31:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.734+0000 7ff428ff9640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4180a3240 0x7ff4180a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.734+0000 7ff42b7fe640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4180a3240 0x7ff4180a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.734+0000 7ff428ff9640 1 -- 172.21.15.62:0/2722659220 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff4140f0670 con 0x7ff42c101420 2024-09-15T23:31:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.735+0000 7ff42b7fe640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4180a3240 0x7ff4180a5700 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff41c005d70 tx=0x7ff41c005d00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:52.752 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.737+0000 7ff428ff9640 1 -- 172.21.15.62:0/2722659220 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff4140bd010 con 0x7ff42c101420 2024-09-15T23:31:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.948+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff42c109010 con 0x7ff42c101420 2024-09-15T23:31:52.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.948+0000 7ff428ff9640 1 -- 172.21.15.62:0/2722659220 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11904 (secure 0 0 0) 0x7ff4140bc910 con 0x7ff42c101420 2024-09-15T23:31:52.950 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:52.950 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":36,"btime":"2024-09-15T23:29:33:499883+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:29:32.913070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:31:52.950 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 36 2024-09-15T23:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.950+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4180a3240 msgr2=0x7ff4180a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.950+0000 7ff431d01640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4180a3240 0x7ff4180a5700 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff41c005d70 tx=0x7ff41c005d00 comp rx=0 tx=0).stop 2024-09-15T23:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.950+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c101420 msgr2=0x7ff42c199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.950+0000 7ff431d01640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c101420 0x7ff42c199810 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7ff414009230 tx=0x7ff414004270 comp rx=0 tx=0).stop 2024-09-15T23:31:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.951+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 shutdown_connections 2024-09-15T23:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.951+0000 7ff431d01640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff4180a3240 0x7ff4180a5700 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.951+0000 7ff431d01640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff42c106a80 0x7ff42c1a4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.951+0000 7ff431d01640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff42c101420 0x7ff42c199810 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.951+0000 7ff431d01640 1 --2- 172.21.15.62:0/2722659220 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff42c100a80 0x7ff42c1992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.951+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 >> 172.21.15.62:0/2722659220 conn(0x7ff42c0fc7a0 msgr2=0x7ff42c0fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.951+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 shutdown_connections 2024-09-15T23:31:52.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:52.951+0000 7ff431d01640 1 -- 172.21.15.62:0/2722659220 wait complete. 2024-09-15T23:31:53.576 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:53 smithi062 ceph-mon[30949]: pgmap v838: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 672 KiB/s wr, 82 op/s 2024-09-15T23:31:53.577 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:53 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2722659220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:53.577 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:53 smithi067 ceph-mon[35665]: pgmap v838: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 672 KiB/s wr, 82 op/s 2024-09-15T23:31:53.577 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:53 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2722659220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:53.606 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-09-15T23:31:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:53 smithi148 ceph-mon[35610]: pgmap v838: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 672 KiB/s wr, 82 op/s 2024-09-15T23:31:53.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:53 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2722659220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:31:53.863 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:54.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.281+0000 7ff3c0077640 1 -- 172.21.15.62:0/1530880391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 msgr2=0x7ff3b810ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.281+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1530880391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 0x7ff3b810ea90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009ca0 tx=0x7ff3b002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.281+0000 7ff3c0077640 1 -- 172.21.15.62:0/1530880391 shutdown_connections 2024-09-15T23:31:54.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.281+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1530880391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 0x7ff3b810ea90 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.281+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1530880391 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3b8102790 0x7ff3b81067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.281+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1530880391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8101de0 0x7ff3b81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.281+0000 7ff3c0077640 1 -- 172.21.15.62:0/1530880391 >> 172.21.15.62:0/1530880391 conn(0x7ff3b80fd8c0 msgr2=0x7ff3b80ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.281+0000 7ff3c0077640 1 -- 172.21.15.62:0/1530880391 shutdown_connections 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.281+0000 7ff3c0077640 1 -- 172.21.15.62:0/1530880391 wait complete. 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3c0077640 1 Processor -- start 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3c0077640 1 -- start start 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3c0077640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3b8101de0 0x7ff3b8198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3c0077640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8102790 0x7ff3b8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3c0077640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 0x7ff3b819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3c0077640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b8111f90 con 0x7ff3b8106f10 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3c0077640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b8111e10 con 0x7ff3b8102790 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3c0077640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b8112110 con 0x7ff3b8101de0 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3be614640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 0x7ff3b819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3be614640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 0x7ff3b819cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51012/0 (socket says 172.21.15.62:51012) 2024-09-15T23:31:54.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3be614640 1 -- 172.21.15.62:0/1705909883 learned_addr learned my addr 172.21.15.62:0/1705909883 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.282+0000 7ff3bde13640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3b8101de0 0x7ff3b8198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3be614640 1 -- 172.21.15.62:0/1705909883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3b8101de0 msgr2=0x7ff3b8198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3be614640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3b8101de0 0x7ff3b8198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3be614640 1 -- 172.21.15.62:0/1705909883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8102790 msgr2=0x7ff3b8198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3be614640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8102790 0x7ff3b8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3be614640 1 -- 172.21.15.62:0/1705909883 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b819d3c0 con 0x7ff3b8106f10 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3be614640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 0x7ff3b819cce0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0031a60 tx=0x7ff3b0031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3aeffd640 1 -- 172.21.15.62:0/1705909883 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0003b70 con 0x7ff3b8106f10 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3aeffd640 1 -- 172.21.15.62:0/1705909883 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3b0003d10 con 0x7ff3b8106f10 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3c0077640 1 -- 172.21.15.62:0/1705909883 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b819d650 con 0x7ff3b8106f10 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3aeffd640 1 -- 172.21.15.62:0/1705909883 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0004770 con 0x7ff3b8106f10 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.283+0000 7ff3c0077640 1 -- 172.21.15.62:0/1705909883 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b81a4f90 con 0x7ff3b8106f10 2024-09-15T23:31:54.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.286+0000 7ff3aeffd640 1 -- 172.21.15.62:0/1705909883 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff3b002f790 con 0x7ff3b8106f10 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.286+0000 7ff3c2450740 1 Processor -- start 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.286+0000 7ff3c2450740 1 -- start start 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.286+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d0bb40 0x564d13d0a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.286+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0a5d0 0x564d13d0aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.286+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d15370 0x564d13d17760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.286+0000 7ff3c2450740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564d13c702f0 con 0x564d13d15370 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.286+0000 7ff3c2450740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564d13c17cf0 con 0x564d13d0a5d0 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.286+0000 7ff3c2450740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564d13c6c2b0 con 0x564d13d0bb40 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d15370 0x564d13d17760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d15370 0x564d13d17760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51014/0 (socket says 172.21.15.62:51014) 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 -- 172.21.15.62:0/609059706 learned_addr learned my addr 172.21.15.62:0/609059706 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 --2- 172.21.15.62:0/609059706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d0bb40 0x564d13d0a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3aeffd640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3a40a32c0 0x7ff3a40a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bde13640 1 --2- 172.21.15.62:0/609059706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0a5d0 0x564d13d0aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3be614640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3a40a32c0 0x7ff3a40a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 -- 172.21.15.62:0/609059706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d0bb40 msgr2=0x564d13d0a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 --2- 172.21.15.62:0/609059706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d0bb40 0x564d13d0a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 -- 172.21.15.62:0/609059706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0a5d0 msgr2=0x564d13d0aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 --2- 172.21.15.62:0/609059706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0a5d0 0x564d13d0aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 -- 172.21.15.62:0/609059706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d13bfc3a0 con 0x564d13d15370 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3aeffd640 1 -- 172.21.15.62:0/1705909883 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff3b00f0fe0 con 0x7ff3b8106f10 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3bd612640 1 --2- 172.21.15.62:0/609059706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d15370 0x564d13d17760 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8001cb0 tx=0x7ff3a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b18cebec719133 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3be614640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3a40a32c0 0x7ff3a40a5780 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff3b004c000 tx=0x7ff3b00095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3a3fff640 1 -- 172.21.15.62:0/609059706 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8012040 con 0x564d13d15370 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3a3fff640 1 -- 172.21.15.62:0/609059706 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3a80070f0 con 0x564d13d15370 2024-09-15T23:31:54.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.287+0000 7ff3a3fff640 1 -- 172.21.15.62:0/609059706 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8004b50 con 0x564d13d15370 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.288+0000 7ff3c2450740 1 -- 172.21.15.62:0/609059706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d15370 msgr2=0x564d13d17760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.288+0000 7ff3c2450740 1 --2- 172.21.15.62:0/609059706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d15370 0x564d13d17760 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8001cb0 tx=0x7ff3a8009e30 comp rx=0 tx=0).stop 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.288+0000 7ff3c2450740 1 -- 172.21.15.62:0/609059706 shutdown_connections 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.288+0000 7ff3c2450740 1 --2- 172.21.15.62:0/609059706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d15370 0x564d13d17760 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.288+0000 7ff3c2450740 1 --2- 172.21.15.62:0/609059706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0a5d0 0x564d13d0aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.288+0000 7ff3c2450740 1 --2- 172.21.15.62:0/609059706 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d0bb40 0x564d13d0a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.288+0000 7ff3c2450740 1 -- 172.21.15.62:0/609059706 >> 172.21.15.62:0/609059706 conn(0x564d13cd3310 msgr2=0x564d13d06c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.288+0000 7ff3c2450740 1 -- 172.21.15.62:0/609059706 shutdown_connections 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.288+0000 7ff3c2450740 1 -- 172.21.15.62:0/609059706 wait complete. 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3c2450740 1 -- start start 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3c2450740 1 Processor -- start 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d0a5d0 0x564d13d14d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0bb40 0x564d13d12020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d15370 0x564d13d12610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3c2450740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x564d13d18070 con 0x564d13d0bb40 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3c2450740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x564d13c2b060 con 0x564d13d15370 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3c2450740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x564d13d181f0 con 0x564d13d0a5d0 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3c2450740 10 client.0 _ll_register_callbacks cb 0x564d1397e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3bde13640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0bb40 0x564d13d12020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3bde13640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0bb40 0x564d13d12020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51024/0 (socket says 172.21.15.62:51024) 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.289+0000 7ff3bde13640 1 -- 172.21.15.62:0/1786897513 learned_addr learned my addr 172.21.15.62:0/1786897513 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3bde13640 1 -- 172.21.15.62:0/1786897513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d0a5d0 msgr2=0x564d13d14d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3bd612640 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d15370 0x564d13d12610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3bde13640 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d0a5d0 0x564d13d14d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3bde13640 1 -- 172.21.15.62:0/1786897513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d15370 msgr2=0x564d13d12610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3bde13640 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d15370 0x564d13d12610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3bde13640 1 -- 172.21.15.62:0/1786897513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d139e13c0 con 0x564d13d0bb40 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3bde13640 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0bb40 0x564d13d12020 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4010a50 tx=0x7ff3b4010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419c5322f5bbf95e server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1786897513 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b401e060 con 0x564d13d0bb40 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1786897513 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff3b400fbd0 con 0x564d13d0bb40 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3c2450740 1 -- 172.21.15.62:0/1786897513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3a80070f0 con 0x564d13d0bb40 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1786897513 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff3b4025050 con 0x564d13d0bb40 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3c2450740 10 client.19107 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3c2450740 1 -- 172.21.15.62:0/1786897513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564d13d23b70 con 0x564d13d0bb40 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1786897513 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3a80070f0 con 0x564d13d0bb40 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1786897513 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7ff3b4018370 con 0x564d13d0bb40 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3c2450740 10 client.19107 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3c2450740 10 client.19107 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:31:54.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.290+0000 7ff3c2450740 20 client.19107 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.291+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x564d13d250d0 0x564d13d27510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.291+0000 7ff3c2450740 4 client.19107 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.291+0000 7ff3c2450740 1 -- 172.21.15.62:0/1786897513 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d13c7d860 con 0x564d13d250d0 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.291+0000 7ff3bd612640 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x564d13d250d0 0x564d13d27510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.291+0000 7ff3bd612640 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x564d13d250d0 0x564d13d27510 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e8c2df7300b6171 server_cookie=890636c3c9d4371b in_seq=0 out_seq=0 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.291+0000 7ff3a17fa640 10 client.19107 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1786897513 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564d13c7d860 con 0x564d13d250d0 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3a17fa640 10 client.19107 handle_command_reply: tid=0 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 client.19107 shutdown 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 -- 172.21.15.62:0/1786897513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0bb40 msgr2=0x564d13d12020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0bb40 0x564d13d12020 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4010a50 tx=0x7ff3b4010f10 comp rx=0 tx=0).stop 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 -- 172.21.15.62:0/1786897513 shutdown_connections 2024-09-15T23:31:54.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x564d13d250d0 0x564d13d27510 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d15370 0x564d13d12610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0bb40 0x564d13d12020 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3a17fa640 0 client.19107 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1786897513 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d0a5d0 0x564d13d14d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 -- 172.21.15.62:0/1786897513 >> 172.21.15.62:0/1786897513 conn(0x564d13cd3310 msgr2=0x564d13d0c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 -- 172.21.15.62:0/1786897513 shutdown_connections 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 1 -- 172.21.15.62:0/1786897513 wait complete. 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.292+0000 7ff3c2450740 20 client.19107 trim_cache size 0 max 16384 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.305+0000 7ff3c2450740 1 Processor -- start 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.305+0000 7ff3c2450740 1 -- start start 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d06e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 0x564d13d118e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d15370 0x564d13d21310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3c2450740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b81b1480 con 0x564d13d0a5d0 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3c2450740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b81aed20 con 0x564d13d0bb40 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3c2450740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b81acb70 con 0x564d13d15370 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bd612640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d06e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bd612640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d06e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51034/0 (socket says 172.21.15.62:51034) 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bd612640 1 -- 172.21.15.62:0/3138319866 learned_addr learned my addr 172.21.15.62:0/3138319866 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bde13640 1 --2- 172.21.15.62:0/3138319866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 0x564d13d118e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bd612640 1 -- 172.21.15.62:0/3138319866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d15370 msgr2=0x564d13d21310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bd612640 1 --2- 172.21.15.62:0/3138319866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d15370 0x564d13d21310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bd612640 1 -- 172.21.15.62:0/3138319866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 msgr2=0x564d13d118e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bd612640 1 --2- 172.21.15.62:0/3138319866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 0x564d13d118e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bd612640 1 -- 172.21.15.62:0/3138319866 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d13d07960 con 0x564d13d0a5d0 2024-09-15T23:31:54.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bde13640 1 --2- 172.21.15.62:0/3138319866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 0x564d13d118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3bd612640 1 --2- 172.21.15.62:0/3138319866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d06e10 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8001c50 tx=0x7ff3a800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baaa41ea94f4d9f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3a37fe640 1 -- 172.21.15.62:0/3138319866 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8027070 con 0x564d13d0a5d0 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3a37fe640 1 -- 172.21.15.62:0/3138319866 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3a80099f0 con 0x564d13d0a5d0 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.306+0000 7ff3a37fe640 1 -- 172.21.15.62:0/3138319866 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8004830 con 0x564d13d0a5d0 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 -- 172.21.15.62:0/3138319866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 msgr2=0x564d13d06e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 --2- 172.21.15.62:0/3138319866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d06e10 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8001c50 tx=0x7ff3a800b040 comp rx=0 tx=0).stop 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 -- 172.21.15.62:0/3138319866 shutdown_connections 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 --2- 172.21.15.62:0/3138319866 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d15370 0x564d13d21310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 --2- 172.21.15.62:0/3138319866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 0x564d13d118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 --2- 172.21.15.62:0/3138319866 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d06e10 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 -- 172.21.15.62:0/3138319866 >> 172.21.15.62:0/3138319866 conn(0x564d13cd3310 msgr2=0x564d13d06700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 -- 172.21.15.62:0/3138319866 shutdown_connections 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 -- 172.21.15.62:0/3138319866 wait complete. 2024-09-15T23:31:54.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 -- start start 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.307+0000 7ff3c2450740 1 Processor -- start 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d1d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 0x564d13d1d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3c2450740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d15370 0x564d13d1a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3c2450740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b81ad1e0 con 0x564d13d0a5d0 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3c2450740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b81ae650 con 0x564d13d0bb40 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3c2450740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b81ad8b0 con 0x564d13d15370 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3c2450740 10 client.0 _ll_register_callbacks cb 0x7ff3b400a820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bd612640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d1d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bd612640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d1d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51042/0 (socket says 172.21.15.62:51042) 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bd612640 1 -- 172.21.15.62:0/1570545521 learned_addr learned my addr 172.21.15.62:0/1570545521 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bde13640 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 0x564d13d1d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bd612640 1 -- 172.21.15.62:0/1570545521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d15370 msgr2=0x564d13d1a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bd612640 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d15370 0x564d13d1a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bd612640 1 -- 172.21.15.62:0/1570545521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 msgr2=0x564d13d1d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bd612640 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 0x564d13d1d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bd612640 1 -- 172.21.15.62:0/1570545521 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d13d21850 con 0x564d13d0a5d0 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.308+0000 7ff3bd612640 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d1d480 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8007460 tx=0x7ff3a8013a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9ed8c444d9a179 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1570545521 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8032060 con 0x564d13d0a5d0 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1570545521 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff3a8013d90 con 0x564d13d0a5d0 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3c2450740 1 -- 172.21.15.62:0/1570545521 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564d13c77d70 con 0x564d13d0a5d0 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1570545521 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff3a8039050 con 0x564d13d0a5d0 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3c2450740 10 client.19113 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3c2450740 1 -- 172.21.15.62:0/1570545521 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564d13d90150 con 0x564d13d0a5d0 2024-09-15T23:31:54.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1570545521 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564d13c77d70 con 0x564d13d0a5d0 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1570545521 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7ff3a80047d0 con 0x564d13d0a5d0 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3c2450740 10 client.19113 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3c2450740 10 client.19113 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3c2450740 20 client.19113 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x564d13d90580 0x564d13d92970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3c2450740 4 client.19113 mds_command: new command op to 24466 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.309+0000 7ff3c2450740 1 -- 172.21.15.62:0/1570545521 --> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d13bf4170 con 0x564d13d90580 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.310+0000 7ff3bde13640 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x564d13d90580 0x564d13d92970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.310+0000 7ff3bde13640 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x564d13d90580 0x564d13d92970 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6026bf299f07295e server_cookie=676797f7e6b78800 in_seq=0 out_seq=0 2024-09-15T23:31:54.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.310+0000 7ff3a17fa640 10 client.19113 ms_handle_connect on v2:172.21.15.148:6838/2241485315 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.310+0000 7ff3a17fa640 1 -- 172.21.15.62:0/1570545521 <== mds.2 v2:172.21.15.148:6838/2241485315 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d13bf4170 con 0x564d13d90580 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.310+0000 7ff3a17fa640 10 client.19113 handle_command_reply: tid=0 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.310+0000 7ff3c2450740 1 client.19113 shutdown 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.310+0000 7ff3c2450740 1 -- 172.21.15.62:0/1570545521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 msgr2=0x564d13d1d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d1d480 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8007460 tx=0x7ff3a8013a70 comp rx=0 tx=0).stop 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 1 -- 172.21.15.62:0/1570545521 shutdown_connections 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.148:6838/2241485315,v1:172.21.15.148:6839/2241485315] conn(0x564d13d90580 0x564d13d92970 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x564d13d15370 0x564d13d1a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564d13d0bb40 0x564d13d1d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3a17fa640 0 client.19113 ms_handle_reset on v2:172.21.15.148:6838/2241485315 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 1 --2- 172.21.15.62:0/1570545521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564d13d0a5d0 0x564d13d1d480 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 1 -- 172.21.15.62:0/1570545521 >> 172.21.15.62:0/1570545521 conn(0x564d13cd3310 msgr2=0x564d13d0c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 1 -- 172.21.15.62:0/1570545521 shutdown_connections 2024-09-15T23:31:54.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 1 -- 172.21.15.62:0/1570545521 wait complete. 2024-09-15T23:31:54.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c2450740 20 client.19113 trim_cache size 0 max 16384 2024-09-15T23:31:54.327 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:31:54.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c0077640 1 -- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3a40a32c0 msgr2=0x7ff3a40a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3a40a32c0 0x7ff3a40a5780 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff3b004c000 tx=0x7ff3b00095c0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c0077640 1 -- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 msgr2=0x7ff3b819cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:54.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 0x7ff3b819cce0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0031a60 tx=0x7ff3b0031a90 comp rx=0 tx=0).stop 2024-09-15T23:31:54.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c0077640 1 -- 172.21.15.62:0/1705909883 shutdown_connections 2024-09-15T23:31:54.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff3a40a32c0 0x7ff3a40a5780 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff3b8106f10 0x7ff3b819cce0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff3b8102790 0x7ff3b8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.311+0000 7ff3c0077640 1 --2- 172.21.15.62:0/1705909883 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff3b8101de0 0x7ff3b8198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:54.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.312+0000 7ff3c0077640 1 -- 172.21.15.62:0/1705909883 >> 172.21.15.62:0/1705909883 conn(0x7ff3b80fd8c0 msgr2=0x7ff3b80ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:54.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.312+0000 7ff3c0077640 1 -- 172.21.15.62:0/1705909883 shutdown_connections 2024-09-15T23:31:54.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:54.312+0000 7ff3c0077640 1 -- 172.21.15.62:0/1705909883 wait complete. 2024-09-15T23:31:55.032 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-09-15T23:31:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:55.587 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:55 smithi062 ceph-mon[30949]: pgmap v839: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 543 KiB/s wr, 66 op/s 2024-09-15T23:31:55.588 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:55 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:31:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:55 smithi067 ceph-mon[35665]: pgmap v839: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 543 KiB/s wr, 66 op/s 2024-09-15T23:31:55.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:55 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:31:55.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 -- 172.21.15.62:0/1647110528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d4106f20 msgr2=0x7f77d410eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 --2- 172.21.15.62:0/1647110528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d4106f20 0x7f77d410eaa0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009900 tx=0x7f77bc02a410 comp rx=0 tx=0).stop 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 -- 172.21.15.62:0/1647110528 shutdown_connections 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 --2- 172.21.15.62:0/1647110528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d4106f20 0x7f77d410eaa0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 --2- 172.21.15.62:0/1647110528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41027a0 0x7f77d41067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 --2- 172.21.15.62:0/1647110528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77d4101df0 0x7f77d41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 -- 172.21.15.62:0/1647110528 >> 172.21.15.62:0/1647110528 conn(0x7f77d40fd8f0 msgr2=0x7f77d40ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 -- 172.21.15.62:0/1647110528 shutdown_connections 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 -- 172.21.15.62:0/1647110528 wait complete. 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 Processor -- start 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.771+0000 7f77daf14640 1 -- start start 2024-09-15T23:31:55.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77daf14640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d4101df0 0x7f77d40744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77daf14640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d41027a0 0x7f77d4074a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77daf14640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77d4106f20 0x7f77d4074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77daf14640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4111f80 con 0x7f77d41027a0 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77daf14640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4111e00 con 0x7f77d4101df0 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77daf14640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4112100 con 0x7f77d4106f20 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d41027a0 0x7f77d4074a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d41027a0 0x7f77d4074a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51064/0 (socket says 172.21.15.62:51064) 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d3fff640 1 -- 172.21.15.62:0/3701297253 learned_addr learned my addr 172.21.15.62:0/3701297253 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d4101df0 0x7f77d40744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d94b1640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77d4106f20 0x7f77d4074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d3fff640 1 -- 172.21.15.62:0/3701297253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77d4106f20 msgr2=0x7f77d4074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d3fff640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77d4106f20 0x7f77d4074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d3fff640 1 -- 172.21.15.62:0/3701297253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d4101df0 msgr2=0x7f77d40744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d3fff640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d4101df0 0x7f77d40744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d3fff640 1 -- 172.21.15.62:0/3701297253 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d41a68a0 con 0x7f77d41027a0 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d3fff640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d41027a0 0x7f77d4074a30 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f77c400cce0 tx=0x7f77c400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77d1ffb640 1 -- 172.21.15.62:0/3701297253 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c400f650 con 0x7f77d41027a0 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.772+0000 7f77daf14640 1 -- 172.21.15.62:0/3701297253 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d41a6b90 con 0x7f77d41027a0 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.773+0000 7f77d1ffb640 1 -- 172.21.15.62:0/3701297253 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c400fcd0 con 0x7f77d41027a0 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.773+0000 7f77d1ffb640 1 -- 172.21.15.62:0/3701297253 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c4018c70 con 0x7f77d41027a0 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.773+0000 7f77daf14640 1 -- 172.21.15.62:0/3701297253 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d41a7120 con 0x7f77d41027a0 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.775+0000 7f77d1ffb640 1 -- 172.21.15.62:0/3701297253 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f77c400f7f0 con 0x7f77d41027a0 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.775+0000 7f77dd2ed740 1 Processor -- start 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.775+0000 7f77dd2ed740 1 -- start start 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.775+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e623b40 0x56150e622090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.775+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e6225d0 0x56150e622a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.775+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e62f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.775+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56150e5882f0 con 0x56150e62d370 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.775+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56150e52fcf0 con 0x56150e623b40 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.775+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56150e5842b0 con 0x56150e6225d0 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e62f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d8cb0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e6225d0 0x56150e622a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e62f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51066/0 (socket says 172.21.15.62:51066) 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d3fff640 1 -- 172.21.15.62:0/3351551186 learned_addr learned my addr 172.21.15.62:0/3351551186 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d94b1640 1 --2- 172.21.15.62:0/3351551186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e623b40 0x56150e622090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d1ffb640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77c00a3240 0x7f77c00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d8cb0640 1 -- 172.21.15.62:0/3351551186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e623b40 msgr2=0x56150e622090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/3351551186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e623b40 0x56150e622090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d8cb0640 1 -- 172.21.15.62:0/3351551186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 msgr2=0x56150e62f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/3351551186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e62f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d8cb0640 1 -- 172.21.15.62:0/3351551186 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56150e5143a0 con 0x56150e6225d0 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d94b1640 1 --2- 172.21.15.62:0/3351551186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e623b40 0x56150e622090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d94b1640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77c00a3240 0x7f77c00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d1ffb640 1 -- 172.21.15.62:0/3701297253 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f77c40d0030 con 0x7f77d41027a0 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/3351551186 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e6225d0 0x56150e622a10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f77cc004630 tx=0x7f77cc00bb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=903c08d069df9d43 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77caffd640 1 -- 172.21.15.62:0/3351551186 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77cc003bd0 con 0x56150e6225d0 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77caffd640 1 -- 172.21.15.62:0/3351551186 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77cc015070 con 0x56150e6225d0 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.776+0000 7f77caffd640 1 -- 172.21.15.62:0/3351551186 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77cc004d90 con 0x56150e6225d0 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 -- 172.21.15.62:0/3351551186 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e6225d0 msgr2=0x56150e622a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77d94b1640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77c00a3240 0x7f77c00a5700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0096c0 tx=0x7f77bc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/3351551186 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e6225d0 0x56150e622a10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f77cc004630 tx=0x7f77cc00bb20 comp rx=0 tx=0).stop 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 -- 172.21.15.62:0/3351551186 shutdown_connections 2024-09-15T23:31:55.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/3351551186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e62f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/3351551186 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e6225d0 0x56150e622a10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/3351551186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e623b40 0x56150e622090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 -- 172.21.15.62:0/3351551186 >> 172.21.15.62:0/3351551186 conn(0x56150e5eb310 msgr2=0x56150e61ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 -- 172.21.15.62:0/3351551186 shutdown_connections 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 -- 172.21.15.62:0/3351551186 wait complete. 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.777+0000 7f77dd2ed740 1 -- start start 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77dd2ed740 1 Processor -- start 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e62cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e629fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e62a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56150e630070 con 0x56150e62d370 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x56150e543060 con 0x56150e6225d0 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x56150e6301f0 con 0x56150e623b40 2024-09-15T23:31:55.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77dd2ed740 10 client.0 _ll_register_callbacks cb 0x56150e2968c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77d94b1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e62a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77d8cb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e62cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77d8cb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e62cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.62:55998/0 (socket says 172.21.15.62:55998) 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77d8cb0640 1 -- 172.21.15.62:0/1463142158 learned_addr learned my addr 172.21.15.62:0/1463142158 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77d3fff640 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e629fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77d8cb0640 1 -- 172.21.15.62:0/1463142158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 msgr2=0x56150e629fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e629fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77d8cb0640 1 -- 172.21.15.62:0/1463142158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 msgr2=0x56150e62a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.778+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e62a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.779+0000 7f77d8cb0640 1 -- 172.21.15.62:0/1463142158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56150e2f93c0 con 0x56150e6225d0 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.779+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e62cd10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f77cc006010 tx=0x7f77cc00bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f76ea2e9a35ca595 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:55.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.779+0000 7f77b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.779+0000 7f77b3fff640 1 -- 172.21.15.62:0/1463142158 <== mon.2 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77cc028060 con 0x56150e6225d0 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.779+0000 7f77b3fff640 1 -- 172.21.15.62:0/1463142158 <== mon.2 v2:172.21.15.67:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f77cc011440 con 0x56150e6225d0 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.779+0000 7f77dd2ed740 1 -- 172.21.15.62:0/1463142158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77cc015070 con 0x56150e6225d0 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.779+0000 7f77b3fff640 1 -- 172.21.15.62:0/1463142158 <== mon.2 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f77cc02f050 con 0x56150e6225d0 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.779+0000 7f77dd2ed740 10 client.28586 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.779+0000 7f77dd2ed740 1 -- 172.21.15.62:0/1463142158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56150e63baf0 con 0x56150e6225d0 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77b3fff640 1 -- 172.21.15.62:0/1463142158 <== mon.2 v2:172.21.15.67:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77cc015070 con 0x56150e6225d0 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77b3fff640 1 -- 172.21.15.62:0/1463142158 <== mon.2 v2:172.21.15.67:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f77cc022070 con 0x56150e6225d0 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77dd2ed740 10 client.28586 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77dd2ed740 10 client.28586 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77dd2ed740 20 client.28586 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56150e63cef0 0x56150e63f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77dd2ed740 4 client.28586 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:55.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77dd2ed740 1 -- 172.21.15.62:0/1463142158 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56150e595860 con 0x56150e63cef0 2024-09-15T23:31:55.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77d3fff640 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56150e63cef0 0x56150e63f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77d3fff640 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56150e63cef0 0x56150e63f2e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3427549f33b0a63 server_cookie=7688b45ab7d07523 in_seq=0 out_seq=0 2024-09-15T23:31:55.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.780+0000 7f77b3fff640 10 client.28586 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:31:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:55 smithi148 ceph-mon[35610]: pgmap v839: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 543 KiB/s wr, 66 op/s 2024-09-15T23:31:55.790 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:55 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-15T23:31:55.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77b3fff640 1 -- 172.21.15.62:0/1463142158 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56150e595860 con 0x56150e63cef0 2024-09-15T23:31:55.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77b3fff640 10 client.28586 handle_command_reply: tid=0 2024-09-15T23:31:55.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77dd2ed740 1 client.28586 shutdown 2024-09-15T23:31:55.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77dd2ed740 1 -- 172.21.15.62:0/1463142158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 msgr2=0x56150e62cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e62cd10 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f77cc006010 tx=0x7f77cc00bfa0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77dd2ed740 1 -- 172.21.15.62:0/1463142158 shutdown_connections 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56150e63cef0 0x56150e63f2e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e62a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e629fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/1463142158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e62cd10 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77b3fff640 0 client.28586 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.781+0000 7f77dd2ed740 1 -- 172.21.15.62:0/1463142158 >> 172.21.15.62:0/1463142158 conn(0x56150e5eb310 msgr2=0x56150e624960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.782+0000 7f77dd2ed740 1 -- 172.21.15.62:0/1463142158 shutdown_connections 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.782+0000 7f77dd2ed740 1 -- 172.21.15.62:0/1463142158 wait complete. 2024-09-15T23:31:55.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.782+0000 7f77dd2ed740 20 client.28586 trim_cache size 0 max 16384 2024-09-15T23:31:55.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77dd2ed740 1 Processor -- start 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77dd2ed740 1 -- start start 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e6225d0 0x56150e639340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e628fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e62d370 0x56150e644880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f77d41b36a0 con 0x56150e6225d0 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77d41b0f40 con 0x56150e62d370 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f77d41aed90 con 0x56150e623b40 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d8cb0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e6225d0 0x56150e639340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d8cb0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e6225d0 0x56150e639340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51088/0 (socket says 172.21.15.62:51088) 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d8cb0640 1 -- 172.21.15.62:0/432751389 learned_addr learned my addr 172.21.15.62:0/432751389 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d3fff640 1 --2- 172.21.15.62:0/432751389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e628fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d94b1640 1 --2- 172.21.15.62:0/432751389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e62d370 0x56150e644880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d8cb0640 1 -- 172.21.15.62:0/432751389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 msgr2=0x56150e628fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:55.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/432751389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e628fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d8cb0640 1 -- 172.21.15.62:0/432751389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e62d370 msgr2=0x56150e644880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:55.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/432751389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e62d370 0x56150e644880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d8cb0640 1 -- 172.21.15.62:0/432751389 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56150e623740 con 0x56150e6225d0 2024-09-15T23:31:55.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.795+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/432751389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e6225d0 0x56150e639340 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00b620 tx=0x7f77cc00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef9b1d7fa30baed server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:55.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77ca7fc640 1 -- 172.21.15.62:0/432751389 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77cc028070 con 0x56150e6225d0 2024-09-15T23:31:55.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77ca7fc640 1 -- 172.21.15.62:0/432751389 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77cc015040 con 0x56150e6225d0 2024-09-15T23:31:55.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77ca7fc640 1 -- 172.21.15.62:0/432751389 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77cc004890 con 0x56150e6225d0 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77dd2ed740 1 -- 172.21.15.62:0/432751389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e6225d0 msgr2=0x56150e639340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/432751389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e6225d0 0x56150e639340 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f77cc00b620 tx=0x7f77cc00b5b0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77dd2ed740 1 -- 172.21.15.62:0/432751389 shutdown_connections 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/432751389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e62d370 0x56150e644880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/432751389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e628fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/432751389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e6225d0 0x56150e639340 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77dd2ed740 1 -- 172.21.15.62:0/432751389 >> 172.21.15.62:0/432751389 conn(0x56150e5eb310 msgr2=0x56150e628890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77dd2ed740 1 -- 172.21.15.62:0/432751389 shutdown_connections 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.796+0000 7f77dd2ed740 1 -- 172.21.15.62:0/432751389 wait complete. 2024-09-15T23:31:55.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77dd2ed740 1 -- start start 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77dd2ed740 1 Processor -- start 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e6358d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e626960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77dd2ed740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e626ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f77d41af400 con 0x56150e62d370 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77d41b0870 con 0x56150e6225d0 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77dd2ed740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f77d41afad0 con 0x56150e623b40 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77dd2ed740 10 client.0 _ll_register_callbacks cb 0x7f77c40985f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77d94b1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e626ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77d8cb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e6358d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77d94b1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e626ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51102/0 (socket says 172.21.15.62:51102) 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77d94b1640 1 -- 172.21.15.62:0/2061772942 learned_addr learned my addr 172.21.15.62:0/2061772942 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:55.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.797+0000 7f77d3fff640 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e626960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77d94b1640 1 -- 172.21.15.62:0/2061772942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 msgr2=0x56150e626960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77d94b1640 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e626960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77d94b1640 1 -- 172.21.15.62:0/2061772942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 msgr2=0x56150e6358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77d94b1640 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e6358d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77d94b1640 1 -- 172.21.15.62:0/2061772942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56150e6273e0 con 0x56150e62d370 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77d94b1640 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e626ea0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f77bc033000 tx=0x7f77bc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1262175e82507153 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77b3fff640 1 -- 172.21.15.62:0/2061772942 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc035040 con 0x56150e62d370 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77dd2ed740 1 -- 172.21.15.62:0/2061772942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77cc015040 con 0x56150e62d370 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77b3fff640 1 -- 172.21.15.62:0/2061772942 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f77bc004320 con 0x56150e62d370 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77b3fff640 1 -- 172.21.15.62:0/2061772942 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f77bc03d9c0 con 0x56150e62d370 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77dd2ed740 10 client.19131 fetch_fsmap learned FSMap version 36 2024-09-15T23:31:55.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77dd2ed740 1 -- 172.21.15.62:0/2061772942 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56150e6a7530 con 0x56150e62d370 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.798+0000 7f77b3fff640 1 -- 172.21.15.62:0/2061772942 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77cc015040 con 0x56150e62d370 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77b3fff640 1 -- 172.21.15.62:0/2061772942 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 36) ==== 8358+0+0 (secure 0 0 0) 0x7f77bc053420 con 0x56150e62d370 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77dd2ed740 10 client.19131 fetch_fsmap finished waiting for FSMap version 36 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77dd2ed740 10 client.19131 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77dd2ed740 20 client.19131 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56150e6a7c30 0x56150e6aa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77dd2ed740 4 client.19131 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77dd2ed740 1 -- 172.21.15.62:0/2061772942 --> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56150e58fd70 con 0x56150e6a7c30 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56150e6a7c30 0x56150e6aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77b3fff640 10 client.19131 ms_handle_connect on v2:172.21.15.67:6836/747270351 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77d8cb0640 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56150e6a7c30 0x56150e6aa050 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41736f557d5838f3 server_cookie=56698118f7930c0 in_seq=0 out_seq=0 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77b3fff640 1 -- 172.21.15.62:0/2061772942 <== mds.1 v2:172.21.15.67:6836/747270351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56150e58fd70 con 0x56150e6a7c30 2024-09-15T23:31:55.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.799+0000 7f77b3fff640 10 client.19131 handle_command_reply: tid=0 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 client.19131 shutdown 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 -- 172.21.15.62:0/2061772942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 msgr2=0x56150e626ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e626ea0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f77bc033000 tx=0x7f77bc02a410 comp rx=0 tx=0).stop 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 -- 172.21.15.62:0/2061772942 shutdown_connections 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.67:6836/747270351,v1:172.21.15.67:6837/747270351] conn(0x56150e6a7c30 0x56150e6aa050 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56150e62d370 0x56150e626ea0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x56150e623b40 0x56150e626960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 --2- 172.21.15.62:0/2061772942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56150e6225d0 0x56150e6358d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77b3fff640 0 client.19131 ms_handle_reset on v2:172.21.15.67:6836/747270351 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 -- 172.21.15.62:0/2061772942 >> 172.21.15.62:0/2061772942 conn(0x56150e5eb310 msgr2=0x56150e6290b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 -- 172.21.15.62:0/2061772942 shutdown_connections 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 1 -- 172.21.15.62:0/2061772942 wait complete. 2024-09-15T23:31:55.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.800+0000 7f77dd2ed740 20 client.19131 trim_cache size 0 max 16384 2024-09-15T23:31:55.807 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:31:55.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 -- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77c00a3240 msgr2=0x7f77c00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77c00a3240 0x7f77c00a5700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0096c0 tx=0x7f77bc009260 comp rx=0 tx=0).stop 2024-09-15T23:31:55.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 -- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d41027a0 msgr2=0x7f77d4074a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:55.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d41027a0 0x7f77d4074a30 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f77c400cce0 tx=0x7f77c400a620 comp rx=0 tx=0).stop 2024-09-15T23:31:55.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 -- 172.21.15.62:0/3701297253 shutdown_connections 2024-09-15T23:31:55.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f77c00a3240 0x7f77c00a5700 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77d4106f20 0x7f77d4074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77d41027a0 0x7f77d4074a30 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 --2- 172.21.15.62:0/3701297253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d4101df0 0x7f77d40744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:55.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 -- 172.21.15.62:0/3701297253 >> 172.21.15.62:0/3701297253 conn(0x7f77d40fd8f0 msgr2=0x7f77d40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:55.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 -- 172.21.15.62:0/3701297253 shutdown_connections 2024-09-15T23:31:55.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:55.801+0000 7f77daf14640 1 -- 172.21.15.62:0/3701297253 wait complete. 2024-09-15T23:31:56.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:31:56.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:31:56.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:56 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:31:56.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:31:56.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:31:56.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:56 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:31:56.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-15T23:31:56.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-15T23:31:56.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:56 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' 2024-09-15T23:31:56.822 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-09-15T23:31:57.080 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:31:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.558+0000 7fa1f7793640 1 -- 172.21.15.62:0/3167742182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 msgr2=0x7fa1f01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.558+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3167742182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 0x7fa1f01067e0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009900 tx=0x7fa1e002a3f0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.558+0000 7fa1f7793640 1 -- 172.21.15.62:0/3167742182 shutdown_connections 2024-09-15T23:31:57.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.558+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3167742182 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f0106f10 0x7fa1f010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.558+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3167742182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 0x7fa1f01067e0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.558+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3167742182 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1f0101de0 0x7fa1f01021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.558+0000 7fa1f7793640 1 -- 172.21.15.62:0/3167742182 >> 172.21.15.62:0/3167742182 conn(0x7fa1f00fd8c0 msgr2=0x7fa1f00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 -- 172.21.15.62:0/3167742182 shutdown_connections 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 -- 172.21.15.62:0/3167742182 wait complete. 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 Processor -- start 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 -- start start 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f0101de0 0x7fa1f0198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 0x7fa1f0198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1f0106f10 0x7fa1f019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f0111f30 con 0x7fa1f0102790 2024-09-15T23:31:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f0111db0 con 0x7fa1f0101de0 2024-09-15T23:31:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.559+0000 7fa1f7793640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f01120b0 con 0x7fa1f0106f10 2024-09-15T23:31:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f4d2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 0x7fa1f0198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f4d2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 0x7fa1f0198950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51132/0 (socket says 172.21.15.62:51132) 2024-09-15T23:31:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f4d2e640 1 -- 172.21.15.62:0/3804633075 learned_addr learned my addr 172.21.15.62:0/3804633075 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:57.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1f0106f10 0x7fa1f019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f552f640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f0101de0 0x7fa1f0198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f4d2e640 1 -- 172.21.15.62:0/3804633075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1f0106f10 msgr2=0x7fa1f019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1f0106f10 0x7fa1f019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f4d2e640 1 -- 172.21.15.62:0/3804633075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f0101de0 msgr2=0x7fa1f0198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f0101de0 0x7fa1f0198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f4d2e640 1 -- 172.21.15.62:0/3804633075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f019d3c0 con 0x7fa1f0102790 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f552f640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f0101de0 0x7fa1f0198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 0x7fa1f0198950 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0005ab0 tx=0x7fa1e00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1e67fc640 1 -- 172.21.15.62:0/3804633075 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e003f030 con 0x7fa1f0102790 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1e67fc640 1 -- 172.21.15.62:0/3804633075 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1e0003ce0 con 0x7fa1f0102790 2024-09-15T23:31:57.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1f7793640 1 -- 172.21.15.62:0/3804633075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f019d650 con 0x7fa1f0102790 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.560+0000 7fa1e67fc640 1 -- 172.21.15.62:0/3804633075 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0035070 con 0x7fa1f0102790 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.561+0000 7fa1f7793640 1 -- 172.21.15.62:0/3804633075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f01a4f90 con 0x7fa1f0102790 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.563+0000 7fa1e67fc640 1 -- 172.21.15.62:0/3804633075 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fa1e0042030 con 0x7fa1f0102790 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.563+0000 7fa1f9b6c740 1 Processor -- start 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.563+0000 7fa1f9b6c740 1 -- start start 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15acb40 0x5567f15ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 0x5567f15b8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5567f15112f0 con 0x5567f15ab5d0 2024-09-15T23:31:57.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5567f14b8cf0 con 0x5567f15acb40 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5567f150d2b0 con 0x5567f15b6370 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f552f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f552f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51148/0 (socket says 172.21.15.62:51148) 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f552f640 1 -- 172.21.15.62:0/2558776154 learned_addr learned my addr 172.21.15.62:0/2558776154 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15acb40 0x5567f15ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1e67fc640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1dc0a3240 0x7fa1dc0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 0x5567f15b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1dc0a3240 0x7fa1dc0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f552f640 1 -- 172.21.15.62:0/2558776154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 msgr2=0x5567f15b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f552f640 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 0x5567f15b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1e67fc640 1 -- 172.21.15.62:0/3804633075 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa1e00f0580 con 0x7fa1f0102790 2024-09-15T23:31:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f552f640 1 -- 172.21.15.62:0/2558776154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15acb40 msgr2=0x5567f15ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f552f640 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15acb40 0x5567f15ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f552f640 1 -- 172.21.15.62:0/2558776154 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f149d3a0 con 0x5567f15ab5d0 2024-09-15T23:31:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.564+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15acb40 0x5567f15ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.565+0000 7fa1f552f640 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15aba10 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec008c60 tx=0x7fa1ec00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7237db0e193f3973 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.565+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1dc0a3240 0x7fa1dc0a5700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800afd0 tx=0x7fa1e8004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.565+0000 7fa1db7fe640 1 -- 172.21.15.62:0/2558776154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec014070 con 0x5567f15ab5d0 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.565+0000 7fa1db7fe640 1 -- 172.21.15.62:0/2558776154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ec0047a0 con 0x5567f15ab5d0 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.565+0000 7fa1db7fe640 1 -- 172.21.15.62:0/2558776154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec010740 con 0x5567f15ab5d0 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/2558776154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 msgr2=0x5567f15aba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15aba10 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec008c60 tx=0x7fa1ec00e720 comp rx=0 tx=0).stop 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/2558776154 shutdown_connections 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 0x5567f15b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15aba10 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/2558776154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15acb40 0x5567f15ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/2558776154 >> 172.21.15.62:0/2558776154 conn(0x5567f1574310 msgr2=0x5567f15a7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/2558776154 shutdown_connections 2024-09-15T23:31:57.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/2558776154 wait complete. 2024-09-15T23:31:57.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 -- start start 2024-09-15T23:31:57.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.566+0000 7fa1f9b6c740 1 Processor -- start 2024-09-15T23:31:57.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 0x5567f15b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x5567f15b9070 con 0x5567f15ab5d0 2024-09-15T23:31:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x5567f14cc060 con 0x5567f15b6370 2024-09-15T23:31:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x5567f15b91f0 con 0x5567f15acb40 2024-09-15T23:31:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f9b6c740 10 client.0 _ll_register_callbacks cb 0x5567f121f8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:57.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f552f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f552f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51152/0 (socket says 172.21.15.62:51152) 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f552f640 1 -- 172.21.15.62:0/3136566028 learned_addr learned my addr 172.21.15.62:0/3136566028 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 0x5567f15b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f552f640 1 -- 172.21.15.62:0/3136566028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 msgr2=0x5567f15b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f552f640 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 0x5567f15b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f552f640 1 -- 172.21.15.62:0/3136566028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 msgr2=0x5567f15b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f552f640 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f552f640 1 -- 172.21.15.62:0/3136566028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f12823c0 con 0x5567f15ab5d0 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 0x5567f15b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:57.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1f552f640 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b5d00 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec000c00 tx=0x7fa1ec0089a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186b56e67c7f6cd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:57.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3136566028 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec027060 con 0x5567f15ab5d0 2024-09-15T23:31:57.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3136566028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1ec0047a0 con 0x5567f15ab5d0 2024-09-15T23:31:57.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.567+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3136566028 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa1ec005340 con 0x5567f15ab5d0 2024-09-15T23:31:57.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3136566028 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa1ec02e050 con 0x5567f15ab5d0 2024-09-15T23:31:57.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1f9b6c740 10 client.19143 fetch_fsmap learned FSMap version 37 2024-09-15T23:31:57.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3136566028 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5567f15c4aa0 con 0x5567f15ab5d0 2024-09-15T23:31:57.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3136566028 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ec0047a0 con 0x5567f15ab5d0 2024-09-15T23:31:57.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3136566028 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa1ec018530 con 0x5567f15ab5d0 2024-09-15T23:31:57.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1f9b6c740 10 client.19143 fetch_fsmap finished waiting for FSMap version 37 2024-09-15T23:31:57.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1f9b6c740 10 client.19143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:57.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1f9b6c740 20 client.19143 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:57.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5567f15c5ea0 0x5567f15c8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1f9b6c740 4 client.19143 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-15T23:31:57.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.568+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3136566028 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567f151e860 con 0x5567f15c5ea0 2024-09-15T23:31:57.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.569+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5567f15c5ea0 0x5567f15c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.569+0000 7fa1d8ff9640 10 client.19143 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:57.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.569+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5567f15c5ea0 0x5567f15c8290 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eccd63138d15632c server_cookie=5b845b132edeb29c in_seq=0 out_seq=0 2024-09-15T23:31:57.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3136566028 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5567f151e860 con 0x5567f15c5ea0 2024-09-15T23:31:57.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1d8ff9640 10 client.19143 handle_command_reply: tid=0 2024-09-15T23:31:57.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 client.19143 shutdown 2024-09-15T23:31:57.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3136566028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 msgr2=0x5567f15b5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b5d00 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec000c00 tx=0x7fa1ec0089a0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3136566028 shutdown_connections 2024-09-15T23:31:57.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5567f15c5ea0 0x5567f15c8290 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 0x5567f15b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3136566028 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b5d00 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3136566028 >> 172.21.15.62:0/3136566028 conn(0x5567f1574310 msgr2=0x5567f15ad930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1d8ff9640 0 client.19143 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:57.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3136566028 shutdown_connections 2024-09-15T23:31:57.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3136566028 wait complete. 2024-09-15T23:31:57.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.570+0000 7fa1f9b6c740 20 client.19143 trim_cache size 0 max 16384 2024-09-15T23:31:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.583+0000 7fa1f9b6c740 1 Processor -- start 2024-09-15T23:31:57.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.583+0000 7fa1f9b6c740 1 -- start start 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.583+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.583+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 0x5567f15ae120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.583+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15ae660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.583+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f01b14e0 con 0x5567f15ab5d0 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.583+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f01aed80 con 0x5567f15b6370 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.583+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f01acbd0 con 0x5567f15acb40 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f552f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f552f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51154/0 (socket says 172.21.15.62:51154) 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f552f640 1 -- 172.21.15.62:0/4045211426 learned_addr learned my addr 172.21.15.62:0/4045211426 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f552f640 1 -- 172.21.15.62:0/4045211426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 msgr2=0x5567f15ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 0x5567f15ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f552f640 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 0x5567f15ae120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f552f640 1 -- 172.21.15.62:0/4045211426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 msgr2=0x5567f15ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f552f640 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15ae660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f552f640 1 -- 172.21.15.62:0/4045211426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f15b6070 con 0x5567f15ab5d0 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1f552f640 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b4f20 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec005fa0 tx=0x7fa1ec018aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9f6438ca5f87e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1daffd640 1 -- 172.21.15.62:0/4045211426 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec027070 con 0x5567f15ab5d0 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1daffd640 1 -- 172.21.15.62:0/4045211426 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ec004e10 con 0x5567f15ab5d0 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.584+0000 7fa1daffd640 1 -- 172.21.15.62:0/4045211426 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec014070 con 0x5567f15ab5d0 2024-09-15T23:31:57.595 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/4045211426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 msgr2=0x5567f15b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b4f20 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec005fa0 tx=0x7fa1ec018aa0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/4045211426 shutdown_connections 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15b6370 0x5567f15ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15acb40 0x5567f15ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/4045211426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15ab5d0 0x5567f15b4f20 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/4045211426 >> 172.21.15.62:0/4045211426 conn(0x5567f1574310 msgr2=0x5567f15b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/4045211426 shutdown_connections 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/4045211426 wait complete. 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 -- start start 2024-09-15T23:31:57.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.585+0000 7fa1f9b6c740 1 Processor -- start 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15ab5d0 0x5567f15bebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15acb40 0x5567f15b9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f9b6c740 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 0x5567f15ba420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f01ad240 con 0x5567f15acb40 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f01ae6b0 con 0x5567f15ab5d0 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f9b6c740 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f01ad910 con 0x5567f15b6370 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f9b6c740 10 client.0 _ll_register_callbacks cb 0x7fa1ec008ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f4d2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15acb40 0x5567f15b9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f4d2e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15acb40 0x5567f15b9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51158/0 (socket says 172.21.15.62:51158) 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f4d2e640 1 -- 172.21.15.62:0/3913780250 learned_addr learned my addr 172.21.15.62:0/3913780250 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:57.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f552f640 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15ab5d0 0x5567f15bebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f5d30640 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 0x5567f15ba420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f4d2e640 1 -- 172.21.15.62:0/3913780250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 msgr2=0x5567f15ba420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 0x5567f15ba420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f4d2e640 1 -- 172.21.15.62:0/3913780250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15ab5d0 msgr2=0x5567f15bebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15ab5d0 0x5567f15bebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.586+0000 7fa1f4d2e640 1 -- 172.21.15.62:0/3913780250 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f15bac50 con 0x5567f15acb40 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1f4d2e640 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15acb40 0x5567f15b9ee0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00bc7b0 tx=0x7fa1e00058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1862ec52ab8723a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1d8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3913780250 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0049040 con 0x5567f15acb40 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3913780250 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fa1e003bd70 con 0x5567f15acb40 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3913780250 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1ec004e10 con 0x5567f15acb40 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3913780250 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fa1e00bc820 con 0x5567f15acb40 2024-09-15T23:31:57.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1f9b6c740 10 client.19149 fetch_fsmap learned FSMap version 37 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3913780250 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5567f1630810 con 0x5567f15acb40 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3913780250 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ec004e10 con 0x5567f15acb40 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3913780250 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fa1e003b460 con 0x5567f15acb40 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1f9b6c740 10 client.19149 fetch_fsmap finished waiting for FSMap version 37 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1f9b6c740 10 client.19149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1f9b6c740 20 client.19149 populate_metadata read hostname 'smithi062' 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.587+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5567f1630f70 0x5567f1633390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1f9b6c740 4 client.19149 mds_command: new command op to 24497 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3913780250 --> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5567f1518d70 con 0x5567f1630f70 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1f552f640 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5567f1630f70 0x5567f1633390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1f552f640 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5567f1630f70 0x5567f1633390 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e86674221b0eb859 server_cookie=f1e8be5206aec3fa in_seq=0 out_seq=0 2024-09-15T23:31:57.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1d8ff9640 10 client.19149 ms_handle_connect on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1d8ff9640 1 -- 172.21.15.62:0/3913780250 <== mds.0 v2:172.21.15.67:6838/818760439 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5567f1518d70 con 0x5567f1630f70 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1d8ff9640 10 client.19149 handle_command_reply: tid=0 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1f9b6c740 1 client.19149 shutdown 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3913780250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15acb40 msgr2=0x5567f15b9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15acb40 0x5567f15b9ee0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00bc7b0 tx=0x7fa1e00058b0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3913780250 shutdown_connections 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.588+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.67:6838/818760439,v1:172.21.15.67:6839/818760439] conn(0x5567f1630f70 0x5567f1633390 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x5567f15b6370 0x5567f15ba420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5567f15acb40 0x5567f15b9ee0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f9b6c740 1 --2- 172.21.15.62:0/3913780250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5567f15ab5d0 0x5567f15bebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3913780250 >> 172.21.15.62:0/3913780250 conn(0x5567f1574310 msgr2=0x5567f15b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1d8ff9640 0 client.19149 ms_handle_reset on v2:172.21.15.67:6838/818760439 2024-09-15T23:31:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3913780250 shutdown_connections 2024-09-15T23:31:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f9b6c740 1 -- 172.21.15.62:0/3913780250 wait complete. 2024-09-15T23:31:57.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f9b6c740 20 client.19149 trim_cache size 0 max 16384 2024-09-15T23:31:57.601 INFO:teuthology.orchestra.run.smithi062.stdout:[] 2024-09-15T23:31:57.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 -- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1dc0a3240 msgr2=0x7fa1dc0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1dc0a3240 0x7fa1dc0a5700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa1e800afd0 tx=0x7fa1e8004ab0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 -- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 msgr2=0x7fa1f0198950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:57.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 0x7fa1f0198950 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0005ab0 tx=0x7fa1e00042f0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 -- 172.21.15.62:0/3804633075 shutdown_connections 2024-09-15T23:31:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fa1dc0a3240 0x7fa1dc0a5700 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa1f0106f10 0x7fa1f019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f0102790 0x7fa1f0198950 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 --2- 172.21.15.62:0/3804633075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1f0101de0 0x7fa1f0198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.589+0000 7fa1f7793640 1 -- 172.21.15.62:0/3804633075 >> 172.21.15.62:0/3804633075 conn(0x7fa1f00fd8c0 msgr2=0x7fa1f00fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.590+0000 7fa1f7793640 1 -- 172.21.15.62:0/3804633075 shutdown_connections 2024-09-15T23:31:57.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:57.590+0000 7fa1f7793640 1 -- 172.21.15.62:0/3804633075 wait complete. 2024-09-15T23:31:57.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:57 smithi062 ceph-mon[30949]: pgmap v840: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 102 KiB/s wr, 24 op/s 2024-09-15T23:31:57.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:57 smithi067 ceph-mon[35665]: pgmap v840: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 102 KiB/s wr, 24 op/s 2024-09-15T23:31:57.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:57 smithi148 ceph-mon[35610]: pgmap v840: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 102 KiB/s wr, 24 op/s 2024-09-15T23:31:58.367 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/fwd_scrub.py", line 35, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/fwd_scrub.py", line 49, in do_scrub self.sleep_unless_stopped(self.sleep_between_iterations) File "/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks/thrasher.py", line 47, in sleep_unless_stopped raise self.Stopped() tasks.thrasher.ThrasherGreenlet.Stopped 2024-09-15T23:31:58.368 INFO:tasks.fwd_scrub:done joining 2024-09-15T23:31:58.368 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-09-15T23:31:58.607 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T23:31:58.632 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:58 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:31:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.726+0000 7fd4c54e4640 1 -- 172.21.15.62:0/993907704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 msgr2=0x7fd4c01072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.726+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/993907704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 0x7fd4c01072d0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009900 tx=0x7fd4a802a410 comp rx=0 tx=0).stop 2024-09-15T23:31:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.726+0000 7fd4c54e4640 1 -- 172.21.15.62:0/993907704 shutdown_connections 2024-09-15T23:31:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.726+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/993907704 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c0107810 0x7fd4c0109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.726+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/993907704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 0x7fd4c01072d0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.726+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/993907704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c01025b0 0x7fd4c01049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.726+0000 7fd4c54e4640 1 -- 172.21.15.62:0/993907704 >> 172.21.15.62:0/993907704 conn(0x7fd4c00fc1b0 msgr2=0x7fd4c00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.726+0000 7fd4c54e4640 1 -- 172.21.15.62:0/993907704 shutdown_connections 2024-09-15T23:31:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.726+0000 7fd4c54e4640 1 -- 172.21.15.62:0/993907704 wait complete. 2024-09-15T23:31:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.727+0000 7fd4c54e4640 1 Processor -- start 2024-09-15T23:31:58.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.727+0000 7fd4c54e4640 1 -- start start 2024-09-15T23:31:58.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.727+0000 7fd4c54e4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c01025b0 0x7fd4c0197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:58.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.727+0000 7fd4c54e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 0x7fd4c0198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:58.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.727+0000 7fd4c54e4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c0107810 0x7fd4c019f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:58.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4c54e4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c010d840 con 0x7fd4c0104ee0 2024-09-15T23:31:58.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4c54e4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c010d6c0 con 0x7fd4c0107810 2024-09-15T23:31:58.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4c54e4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c010d9c0 con 0x7fd4c01025b0 2024-09-15T23:31:58.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4b67fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 0x7fd4c0198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4b67fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 0x7fd4c0198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51174/0 (socket says 172.21.15.62:51174) 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4b67fc640 1 -- 172.21.15.62:0/1291469120 learned_addr learned my addr 172.21.15.62:0/1291469120 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4beffd640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c01025b0 0x7fd4c0197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4bf7fe640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c0107810 0x7fd4c019f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4b67fc640 1 -- 172.21.15.62:0/1291469120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c01025b0 msgr2=0x7fd4c0197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4b67fc640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c01025b0 0x7fd4c0197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4b67fc640 1 -- 172.21.15.62:0/1291469120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c0107810 msgr2=0x7fd4c019f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4b67fc640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c0107810 0x7fd4c019f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4b67fc640 1 -- 172.21.15.62:0/1291469120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c019fa50 con 0x7fd4c0104ee0 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4bf7fe640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c0107810 0x7fd4c019f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4b67fc640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 0x7fd4c0198370 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8031440 tx=0x7fd4a8032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4bcff9640 1 -- 172.21.15.62:0/1291469120 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a803f030 con 0x7fd4c0104ee0 2024-09-15T23:31:58.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4bcff9640 1 -- 172.21.15.62:0/1291469120 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4a8003ce0 con 0x7fd4c0104ee0 2024-09-15T23:31:58.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c019fce0 con 0x7fd4c0104ee0 2024-09-15T23:31:58.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4bcff9640 1 -- 172.21.15.62:0/1291469120 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4a8035070 con 0x7fd4c0104ee0 2024-09-15T23:31:58.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.728+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c01a01d0 con 0x7fd4c0104ee0 2024-09-15T23:31:58.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.729+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c00fdca0 con 0x7fd4c0104ee0 2024-09-15T23:31:58.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.731+0000 7fd4bcff9640 1 -- 172.21.15.62:0/1291469120 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fd4a8042030 con 0x7fd4c0104ee0 2024-09-15T23:31:58.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.732+0000 7fd4bcff9640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3250 0x7fd4ac0a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:58.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.732+0000 7fd4bcff9640 1 -- 172.21.15.62:0/1291469120 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fd4a80f0670 con 0x7fd4c0104ee0 2024-09-15T23:31:58.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.732+0000 7fd4beffd640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3250 0x7fd4ac0a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:58.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.733+0000 7fd4beffd640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3250 0x7fd4ac0a5710 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000abe0 tx=0x7fd4b000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:58.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.735+0000 7fd4bcff9640 1 -- 172.21.15.62:0/1291469120 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fd4a80bd010 con 0x7fd4c0104ee0 2024-09-15T23:31:58.774 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:58 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:31:58.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:58 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:31:58.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.913+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4c002d9c0 con 0x7fd4c0104ee0 2024-09-15T23:31:58.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.914+0000 7fd4bcff9640 1 -- 172.21.15.62:0/1291469120 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24777 (secure 0 0 0) 0x7fd4a80bc910 con 0x7fd4c0104ee0 2024-09-15T23:31:58.916 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:31:58.916 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":92,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:15:42.605470+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3250 msgr2=0x7fd4ac0a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3250 0x7fd4ac0a5710 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd4b000abe0 tx=0x7fd4b000a2d0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 msgr2=0x7fd4c0198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 0x7fd4c0198370 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8031440 tx=0x7fd4a8032e20 comp rx=0 tx=0).stop 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 shutdown_connections 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fd4ac0a3250 0x7fd4ac0a5710 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4c0107810 0x7fd4c019f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd4c0104ee0 0x7fd4c0198370 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 --2- 172.21.15.62:0/1291469120 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fd4c01025b0 0x7fd4c0197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.916+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 >> 172.21.15.62:0/1291469120 conn(0x7fd4c00fc1b0 msgr2=0x7fd4c0102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.917+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 shutdown_connections 2024-09-15T23:31:58.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:58.917+0000 7fd4c54e4640 1 -- 172.21.15.62:0/1291469120 wait complete. 2024-09-15T23:31:58.929 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T23:08:54.129326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-15T23:13:29.172210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-15T23:13:31.408446+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-15T23:13:37.961318+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-15T23:31:58.929 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T23:31:59.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.075+0000 7f80ccdfd640 1 -- 172.21.15.62:0/3594646466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 msgr2=0x7f80c8109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.075+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/3594646466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 0x7f80c8109ca0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009ca0 tx=0x7f80c002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.075+0000 7f80ccdfd640 1 -- 172.21.15.62:0/3594646466 shutdown_connections 2024-09-15T23:31:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.075+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/3594646466 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 0x7f80c8109ca0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.075+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/3594646466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c8104f80 0x7f80c8107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.075+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/3594646466 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80c8102650 0x7f80c8104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.075+0000 7f80ccdfd640 1 -- 172.21.15.62:0/3594646466 >> 172.21.15.62:0/3594646466 conn(0x7f80c80fc1b0 msgr2=0x7f80c80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.075+0000 7f80ccdfd640 1 -- 172.21.15.62:0/3594646466 shutdown_connections 2024-09-15T23:31:59.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.075+0000 7f80ccdfd640 1 -- 172.21.15.62:0/3594646466 wait complete. 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80ccdfd640 1 Processor -- start 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80ccdfd640 1 -- start start 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80ccdfd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c8102650 0x7f80c8101fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80ccdfd640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80c8104f80 0x7f80c8100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80ccdfd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 0x7f80c8100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80ccdfd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f80c810d7d0 con 0x7f80c81078b0 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80ccdfd640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f80c810d650 con 0x7f80c8102650 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80ccdfd640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f80c810d950 con 0x7f80c8104f80 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80c6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 0x7f80c8100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80c6d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 0x7f80c8100b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51184/0 (socket says 172.21.15.62:51184) 2024-09-15T23:31:59.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80c6d9d640 1 -- 172.21.15.62:0/1374306379 learned_addr learned my addr 172.21.15.62:0/1374306379 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80c659c640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c8102650 0x7f80c8101fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80c5d9b640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80c8104f80 0x7f80c8100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80c6d9d640 1 -- 172.21.15.62:0/1374306379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80c8104f80 msgr2=0x7f80c8100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80c6d9d640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80c8104f80 0x7f80c8100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.076+0000 7f80c6d9d640 1 -- 172.21.15.62:0/1374306379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c8102650 msgr2=0x7f80c8101fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.077+0000 7f80c6d9d640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c8102650 0x7f80c8101fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.077+0000 7f80c6d9d640 1 -- 172.21.15.62:0/1374306379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c81013d0 con 0x7f80c81078b0 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.077+0000 7f80c6d9d640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 0x7f80c8100b40 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009bf0 tx=0x7f80c0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.077+0000 7f80b77fe640 1 -- 172.21.15.62:0/1374306379 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c0003b70 con 0x7f80c81078b0 2024-09-15T23:31:59.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.077+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c81a1f70 con 0x7f80c81078b0 2024-09-15T23:31:59.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.077+0000 7f80b77fe640 1 -- 172.21.15.62:0/1374306379 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80c0003d10 con 0x7f80c81078b0 2024-09-15T23:31:59.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.077+0000 7f80b77fe640 1 -- 172.21.15.62:0/1374306379 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c002fb80 con 0x7f80c81078b0 2024-09-15T23:31:59.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.077+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c81a23d0 con 0x7f80c81078b0 2024-09-15T23:31:59.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.078+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c80fdca0 con 0x7f80c81078b0 2024-09-15T23:31:59.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.079+0000 7f80b77fe640 1 -- 172.21.15.62:0/1374306379 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f80c002fd20 con 0x7f80c81078b0 2024-09-15T23:31:59.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.080+0000 7f80b77fe640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80b00a3250 0x7f80b00a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.080+0000 7f80b77fe640 1 -- 172.21.15.62:0/1374306379 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f80c00f0f10 con 0x7f80c81078b0 2024-09-15T23:31:59.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.080+0000 7f80c659c640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80b00a3250 0x7f80b00a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.080+0000 7f80c659c640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80b00a3250 0x7f80b00a5710 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f80b800abe0 tx=0x7f80b800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:59.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.083+0000 7f80b77fe640 1 -- 172.21.15.62:0/1374306379 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f80c00bd8b0 con 0x7f80c81078b0 2024-09-15T23:31:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.261+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f80c8062e90 con 0x7f80c81078b0 2024-09-15T23:31:59.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.261+0000 7f80b77fe640 1 -- 172.21.15.62:0/1374306379 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f80c00bd1b0 con 0x7f80c81078b0 2024-09-15T23:31:59.263 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80b00a3250 msgr2=0x7f80b00a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80b00a3250 0x7f80b00a5710 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f80b800abe0 tx=0x7f80b800a2d0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 msgr2=0x7f80c8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 0x7f80c8100b40 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f80c0009bf0 tx=0x7f80c0004ab0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 shutdown_connections 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f80b00a3250 0x7f80b00a5710 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80c81078b0 0x7f80c8100b40 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f80c8104f80 0x7f80c8100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 --2- 172.21.15.62:0/1374306379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80c8102650 0x7f80c8101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 >> 172.21.15.62:0/1374306379 conn(0x7f80c80fc1b0 msgr2=0x7f80c8102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:59.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 shutdown_connections 2024-09-15T23:31:59.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.263+0000 7f80ccdfd640 1 -- 172.21.15.62:0/1374306379 wait complete. 2024-09-15T23:31:59.275 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-15T23:31:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.381+0000 7f2dd9978640 1 -- 172.21.15.62:0/3522360768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd41036d0 msgr2=0x7f2dd4109f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.381+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3522360768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd41036d0 0x7f2dd4109f60 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009ca0 tx=0x7f2dcc02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.381+0000 7f2dd9978640 1 -- 172.21.15.62:0/3522360768 shutdown_connections 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.381+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3522360768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd41036d0 0x7f2dd4109f60 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.381+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3522360768 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dd4102d10 0x7f2dd4103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.381+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3522360768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd4101b20 0x7f2dd4101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.381+0000 7f2dd9978640 1 -- 172.21.15.62:0/3522360768 >> 172.21.15.62:0/3522360768 conn(0x7f2dd40fd2a0 msgr2=0x7f2dd40ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.381+0000 7f2dd9978640 1 -- 172.21.15.62:0/3522360768 shutdown_connections 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.381+0000 7f2dd9978640 1 -- 172.21.15.62:0/3522360768 wait complete. 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd9978640 1 Processor -- start 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd9978640 1 -- start start 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd9978640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd4101b20 0x7f2dd4197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd9978640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd4102d10 0x7f2dd4198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd9978640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dd41036d0 0x7f2dd419f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd9978640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd410d780 con 0x7f2dd4101b20 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd9978640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd410d600 con 0x7f2dd4102d10 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd9978640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd410d900 con 0x7f2dd41036d0 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd4101b20 0x7f2dd4197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd4101b20 0x7f2dd4197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51200/0 (socket says 172.21.15.62:51200) 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd2ffd640 1 -- 172.21.15.62:0/3559894071 learned_addr learned my addr 172.21.15.62:0/3559894071 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd27fc640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd4102d10 0x7f2dd4198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd2ffd640 1 -- 172.21.15.62:0/3559894071 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dd41036d0 msgr2=0x7f2dd419f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd2ffd640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dd41036d0 0x7f2dd419f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd2ffd640 1 -- 172.21.15.62:0/3559894071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd4102d10 msgr2=0x7f2dd4198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd2ffd640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd4102d10 0x7f2dd4198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd2ffd640 1 -- 172.21.15.62:0/3559894071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd419f9f0 con 0x7f2dd4101b20 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.382+0000 7f2dd27fc640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd4102d10 0x7f2dd4198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:31:59.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.383+0000 7f2dd2ffd640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd4101b20 0x7f2dd4197dd0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400d770 tx=0x7f2dc400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:59.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.383+0000 7f2dd8976640 1 -- 172.21.15.62:0/3559894071 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4017070 con 0x7f2dd4101b20 2024-09-15T23:31:59.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.383+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd419fce0 con 0x7f2dd4101b20 2024-09-15T23:31:59.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.383+0000 7f2dd8976640 1 -- 172.21.15.62:0/3559894071 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dc4009c30 con 0x7f2dd4101b20 2024-09-15T23:31:59.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.383+0000 7f2dd8976640 1 -- 172.21.15.62:0/3559894071 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc4003dd0 con 0x7f2dd4101b20 2024-09-15T23:31:59.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.383+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd41a01d0 con 0x7f2dd4101b20 2024-09-15T23:31:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.384+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd4108f60 con 0x7f2dd4101b20 2024-09-15T23:31:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.385+0000 7f2dd8976640 1 -- 172.21.15.62:0/3559894071 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2dc4003f70 con 0x7f2dd4101b20 2024-09-15T23:31:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.386+0000 7f2dd8976640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2dbc0a3250 0x7f2dbc0a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.386+0000 7f2dd27fc640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2dbc0a3250 0x7f2dbc0a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.386+0000 7f2dd8976640 1 -- 172.21.15.62:0/3559894071 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f2dc40cfac0 con 0x7f2dd4101b20 2024-09-15T23:31:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.386+0000 7f2dd27fc640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2dbc0a3250 0x7f2dbc0a5710 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80058c0 tx=0x7f2dc8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:59.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.389+0000 7f2dd8976640 1 -- 172.21.15.62:0/3559894071 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2dc4009de0 con 0x7f2dd4101b20 2024-09-15T23:31:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.563+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2dd402d9e0 con 0x7f2dd4101b20 2024-09-15T23:31:59.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.564+0000 7f2dd8976640 1 -- 172.21.15.62:0/3559894071 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f2dc409c460 con 0x7f2dd4101b20 2024-09-15T23:31:59.565 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-09-15T23:31:59.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2dbc0a3250 msgr2=0x7f2dbc0a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2dbc0a3250 0x7f2dbc0a5710 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2dc80058c0 tx=0x7f2dc8005850 comp rx=0 tx=0).stop 2024-09-15T23:31:59.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd4101b20 msgr2=0x7f2dd4197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd4101b20 0x7f2dd4197dd0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f2dc400d770 tx=0x7f2dc400dc30 comp rx=0 tx=0).stop 2024-09-15T23:31:59.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 shutdown_connections 2024-09-15T23:31:59.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f2dbc0a3250 0x7f2dbc0a5710 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2dd41036d0 0x7f2dd419f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dd4102d10 0x7f2dd4198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 --2- 172.21.15.62:0/3559894071 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2dd4101b20 0x7f2dd4197dd0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 >> 172.21.15.62:0/3559894071 conn(0x7f2dd40fd2a0 msgr2=0x7f2dd4106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:59.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 shutdown_connections 2024-09-15T23:31:59.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.566+0000 7f2dd9978640 1 -- 172.21.15.62:0/3559894071 wait complete. 2024-09-15T23:31:59.578 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-15T23:31:59.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.683+0000 7ff14c6d9640 1 -- 172.21.15.62:0/2819929032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 msgr2=0x7ff144109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.683+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/2819929032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 0x7ff144109c60 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7ff13c009ca0 tx=0x7ff13c02a8c0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 -- 172.21.15.62:0/2819929032 shutdown_connections 2024-09-15T23:31:59.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/2819929032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 0x7ff144109c60 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/2819929032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144104ec0 0x7ff1441072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/2819929032 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff144102590 0x7ff144104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 -- 172.21.15.62:0/2819929032 >> 172.21.15.62:0/2819929032 conn(0x7ff1440fc190 msgr2=0x7ff1440fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:59.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 -- 172.21.15.62:0/2819929032 shutdown_connections 2024-09-15T23:31:59.687 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 -- 172.21.15.62:0/2819929032 wait complete. 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 Processor -- start 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 -- start start 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144102590 0x7ff144197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff144104ec0 0x7ff1441983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 0x7ff14419f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410d7d0 con 0x7ff1441077f0 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410d650 con 0x7ff144102590 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14c6d9640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff14410d950 con 0x7ff144104ec0 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14ac76640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 0x7ff14419f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14ac76640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 0x7ff14419f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51218/0 (socket says 172.21.15.62:51218) 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.684+0000 7ff14ac76640 1 -- 172.21.15.62:0/3129760564 learned_addr learned my addr 172.21.15.62:0/3129760564 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff14a475640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144102590 0x7ff144197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff149c74640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff144104ec0 0x7ff1441983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff14ac76640 1 -- 172.21.15.62:0/3129760564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff144104ec0 msgr2=0x7ff1441983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff14ac76640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff144104ec0 0x7ff1441983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff14ac76640 1 -- 172.21.15.62:0/3129760564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144102590 msgr2=0x7ff144197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff14ac76640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144102590 0x7ff144197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff14ac76640 1 -- 172.21.15.62:0/3129760564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14419fad0 con 0x7ff1441077f0 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff14ac76640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 0x7ff14419f460 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7ff13c02f540 tx=0x7ff13c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff13b7fe640 1 -- 172.21.15.62:0/3129760564 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c003c30 con 0x7ff1441077f0 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff13b7fe640 1 -- 172.21.15.62:0/3129760564 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff13c003dd0 con 0x7ff1441077f0 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff14419fd60 con 0x7ff1441077f0 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff13b7fe640 1 -- 172.21.15.62:0/3129760564 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff13c02fd90 con 0x7ff1441077f0 2024-09-15T23:31:59.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.685+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1441a0250 con 0x7ff1441077f0 2024-09-15T23:31:59.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.686+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1440fdc80 con 0x7ff1441077f0 2024-09-15T23:31:59.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.687+0000 7ff13b7fe640 1 -- 172.21.15.62:0/3129760564 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff13c004050 con 0x7ff1441077f0 2024-09-15T23:31:59.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.688+0000 7ff13b7fe640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1300a3250 0x7ff1300a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.688+0000 7ff14a475640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1300a3250 0x7ff1300a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.688+0000 7ff13b7fe640 1 -- 172.21.15.62:0/3129760564 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7ff13c0f1030 con 0x7ff1441077f0 2024-09-15T23:31:59.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.689+0000 7ff14a475640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1300a3250 0x7ff1300a5710 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff140005fd0 tx=0x7ff14000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:59.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.691+0000 7ff13b7fe640 1 -- 172.21.15.62:0/3129760564 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff13c0bda60 con 0x7ff1441077f0 2024-09-15T23:31:59.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.870+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff144062df0 con 0x7ff1441077f0 2024-09-15T23:31:59.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.871+0000 7ff13b7fe640 1 -- 172.21.15.62:0/3129760564 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7ff13c0bd360 con 0x7ff1441077f0 2024-09-15T23:31:59.872 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-09-15T23:31:59.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1300a3250 msgr2=0x7ff1300a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1300a3250 0x7ff1300a5710 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff140005fd0 tx=0x7ff14000b390 comp rx=0 tx=0).stop 2024-09-15T23:31:59.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 msgr2=0x7ff14419f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 0x7ff14419f460 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7ff13c02f540 tx=0x7ff13c031ee0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.874 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 shutdown_connections 2024-09-15T23:31:59.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff1300a3250 0x7ff1300a5710 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1441077f0 0x7ff14419f460 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff144104ec0 0x7ff1441983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 --2- 172.21.15.62:0/3129760564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff144102590 0x7ff144197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 >> 172.21.15.62:0/3129760564 conn(0x7ff1440fc190 msgr2=0x7ff144102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:59.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 shutdown_connections 2024-09-15T23:31:59.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.873+0000 7ff14c6d9640 1 -- 172.21.15.62:0/3129760564 wait complete. 2024-09-15T23:31:59.884 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:59 smithi062 ceph-mon[30949]: pgmap v841: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 10 op/s 2024-09-15T23:31:59.884 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:59 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1291469120' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:31:59.884 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:59 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1374306379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:31:59.884 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:31:59 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3559894071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:31:59.885 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-15T23:31:59.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.990+0000 7f333d798640 1 -- 172.21.15.62:0/1355221725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33381025b0 msgr2=0x7f3338108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.990+0000 7f333d798640 1 --2- 172.21.15.62:0/1355221725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33381025b0 0x7f3338108e40 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f3328009ca0 tx=0x7f332802a8c0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.990+0000 7f333d798640 1 -- 172.21.15.62:0/1355221725 shutdown_connections 2024-09-15T23:31:59.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.990+0000 7f333d798640 1 --2- 172.21.15.62:0/1355221725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f33381025b0 0x7f3338108e40 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.991+0000 7f333d798640 1 --2- 172.21.15.62:0/1355221725 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3338101bf0 0x7f3338102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.991+0000 7f333d798640 1 --2- 172.21.15.62:0/1355221725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3338100a00 0x7f3338100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.991+0000 7f333d798640 1 -- 172.21.15.62:0/1355221725 >> 172.21.15.62:0/1355221725 conn(0x7f33380fc140 msgr2=0x7f33380fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.991+0000 7f333d798640 1 -- 172.21.15.62:0/1355221725 shutdown_connections 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.991+0000 7f333d798640 1 -- 172.21.15.62:0/1355221725 wait complete. 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.991+0000 7f333d798640 1 Processor -- start 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.991+0000 7f333d798640 1 -- start start 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f333d798640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3338100a00 0x7f3338104990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f333d798640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3338101bf0 0x7f3338102fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f333d798640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33381025b0 0x7f3338103510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f333d798640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f333810c6d0 con 0x7f3338100a00 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f333d798640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f333810c550 con 0x7f33381025b0 2024-09-15T23:31:59.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f333d798640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f333810c850 con 0x7f3338101bf0 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f3336ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3338100a00 0x7f3338104990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f3336ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3338100a00 0x7f3338104990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51236/0 (socket says 172.21.15.62:51236) 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f3336ffd640 1 -- 172.21.15.62:0/2255852519 learned_addr learned my addr 172.21.15.62:0/2255852519 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f33367fc640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3338101bf0 0x7f3338102fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f33377fe640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33381025b0 0x7f3338103510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f3336ffd640 1 -- 172.21.15.62:0/2255852519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3338101bf0 msgr2=0x7f3338102fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f3336ffd640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3338101bf0 0x7f3338102fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f3336ffd640 1 -- 172.21.15.62:0/2255852519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33381025b0 msgr2=0x7f3338103510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f3336ffd640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33381025b0 0x7f3338103510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f3336ffd640 1 -- 172.21.15.62:0/2255852519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3338103da0 con 0x7f3338100a00 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f3336ffd640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3338100a00 0x7f3338104990 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f333000d950 tx=0x7f333000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f332ffff640 1 -- 172.21.15.62:0/2255852519 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3330017020 con 0x7f3338100a00 2024-09-15T23:31:59.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f332ffff640 1 -- 172.21.15.62:0/2255852519 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3330009c30 con 0x7f3338100a00 2024-09-15T23:31:59.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33381a1f50 con 0x7f3338100a00 2024-09-15T23:31:59.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f332ffff640 1 -- 172.21.15.62:0/2255852519 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f333000f740 con 0x7f3338100a00 2024-09-15T23:31:59.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.992+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33381a2360 con 0x7f3338100a00 2024-09-15T23:31:59.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.993+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33381a2650 con 0x7f3338100a00 2024-09-15T23:31:59.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.995+0000 7f332ffff640 1 -- 172.21.15.62:0/2255852519 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f333001a020 con 0x7f3338100a00 2024-09-15T23:31:59.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.995+0000 7f332ffff640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33240a3250 0x7f33240a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:31:59.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.995+0000 7f33367fc640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33240a3250 0x7f33240a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:31:59.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.995+0000 7f332ffff640 1 -- 172.21.15.62:0/2255852519 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f33300cf710 con 0x7f3338100a00 2024-09-15T23:31:59.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.996+0000 7f33367fc640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33240a3250 0x7f33240a5710 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3338104700 tx=0x7f3320005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:00.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:31:59.999+0000 7f332ffff640 1 -- 172.21.15.62:0/2255852519 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f3330021070 con 0x7f3338100a00 2024-09-15T23:32:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:59 smithi067 ceph-mon[35665]: pgmap v841: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 10 op/s 2024-09-15T23:32:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:59 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1291469120' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:32:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:59 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1374306379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:32:00.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:31:59 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3559894071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:32:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:59 smithi148 ceph-mon[35610]: pgmap v841: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.8 KiB/s wr, 10 op/s 2024-09-15T23:32:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:59 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1291469120' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:32:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:59 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1374306379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-15T23:32:00.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:31:59 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3559894071' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-09-15T23:32:00.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.172+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f33381a2f80 con 0x7f3338100a00 2024-09-15T23:32:00.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.173+0000 7f332ffff640 1 -- 172.21.15.62:0/2255852519 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f3330009de0 con 0x7f3338100a00 2024-09-15T23:32:00.174 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-09-15T23:32:00.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33240a3250 msgr2=0x7f33240a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:00.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33240a3250 0x7f33240a5710 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f3338104700 tx=0x7f3320005f90 comp rx=0 tx=0).stop 2024-09-15T23:32:00.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3338100a00 msgr2=0x7f3338104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:00.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3338100a00 0x7f3338104990 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f333000d950 tx=0x7f333000de10 comp rx=0 tx=0).stop 2024-09-15T23:32:00.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 shutdown_connections 2024-09-15T23:32:00.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f33240a3250 0x7f33240a5710 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:00.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33381025b0 0x7f3338103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:00.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3338101bf0 0x7f3338102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:00.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 --2- 172.21.15.62:0/2255852519 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3338100a00 0x7f3338104990 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:00.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 >> 172.21.15.62:0/2255852519 conn(0x7f33380fc140 msgr2=0x7f3338069470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:00.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 shutdown_connections 2024-09-15T23:32:00.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.175+0000 7f333d798640 1 -- 172.21.15.62:0/2255852519 wait complete. 2024-09-15T23:32:00.187 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-09-15T23:32:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.289+0000 7f5b6c617640 1 -- 172.21.15.62:0/2511578833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64101c20 msgr2=0x7f5b641020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.289+0000 7f5b6c617640 1 --2- 172.21.15.62:0/2511578833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64101c20 0x7f5b641020a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f5b54009900 tx=0x7f5b5402a410 comp rx=0 tx=0).stop 2024-09-15T23:32:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.289+0000 7f5b6c617640 1 -- 172.21.15.62:0/2511578833 shutdown_connections 2024-09-15T23:32:00.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.289+0000 7f5b6c617640 1 --2- 172.21.15.62:0/2511578833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b641025e0 0x7f5b64108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.289+0000 7f5b6c617640 1 --2- 172.21.15.62:0/2511578833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64101c20 0x7f5b641020a0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.289+0000 7f5b6c617640 1 --2- 172.21.15.62:0/2511578833 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b64100a30 0x7f5b64100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.289+0000 7f5b6c617640 1 -- 172.21.15.62:0/2511578833 >> 172.21.15.62:0/2511578833 conn(0x7f5b640fc1b0 msgr2=0x7f5b640fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.289+0000 7f5b6c617640 1 -- 172.21.15.62:0/2511578833 shutdown_connections 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.289+0000 7f5b6c617640 1 -- 172.21.15.62:0/2511578833 wait complete. 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6c617640 1 Processor -- start 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6c617640 1 -- start start 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6c617640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64100a30 0x7f5b64197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6c617640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b64101c20 0x7f5b64198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:00.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6c617640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b641025e0 0x7f5b6419f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6c617640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6410d800 con 0x7f5b64100a30 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6c617640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6410d680 con 0x7f5b64101c20 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6c617640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6410d980 con 0x7f5b641025e0 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6a3b3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64100a30 0x7f5b64197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6a3b3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64100a30 0x7f5b64197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:51252/0 (socket says 172.21.15.62:51252) 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b6a3b3640 1 -- 172.21.15.62:0/296679196 learned_addr learned my addr 172.21.15.62:0/296679196 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.290+0000 7f5b69bb2640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b64101c20 0x7f5b64198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6a3b3640 1 -- 172.21.15.62:0/296679196 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b641025e0 msgr2=0x7f5b6419f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6abb4640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b641025e0 0x7f5b6419f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6a3b3640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b641025e0 0x7f5b6419f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6a3b3640 1 -- 172.21.15.62:0/296679196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b64101c20 msgr2=0x7f5b64198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6a3b3640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b64101c20 0x7f5b64198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6a3b3640 1 -- 172.21.15.62:0/296679196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6419fa60 con 0x7f5b64100a30 2024-09-15T23:32:00.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b69bb2640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b64101c20 0x7f5b64198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6abb4640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b641025e0 0x7f5b6419f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6a3b3640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64100a30 0x7f5b64197e40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000c970 tx=0x7f5b6000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b5b7fe640 1 -- 172.21.15.62:0/296679196 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b60004220 con 0x7f5b64100a30 2024-09-15T23:32:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b5b7fe640 1 -- 172.21.15.62:0/296679196 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b600043c0 con 0x7f5b64100a30 2024-09-15T23:32:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b6419fd50 con 0x7f5b64100a30 2024-09-15T23:32:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b5b7fe640 1 -- 172.21.15.62:0/296679196 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b60004a00 con 0x7f5b64100a30 2024-09-15T23:32:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.291+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b641a01c0 con 0x7f5b64100a30 2024-09-15T23:32:00.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.292+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b64107e90 con 0x7f5b64100a30 2024-09-15T23:32:00.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.293+0000 7f5b5b7fe640 1 -- 172.21.15.62:0/296679196 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f5b600079b0 con 0x7f5b64100a30 2024-09-15T23:32:00.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.294+0000 7f5b5b7fe640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b500a32d0 0x7f5b500a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:00.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.294+0000 7f5b5b7fe640 1 -- 172.21.15.62:0/296679196 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f5b600cfe60 con 0x7f5b64100a30 2024-09-15T23:32:00.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.294+0000 7f5b69bb2640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b500a32d0 0x7f5b500a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:00.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.295+0000 7f5b69bb2640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b500a32d0 0x7f5b500a5790 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5b54000c00 tx=0x7f5b540095f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:00.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.297+0000 7f5b5b7fe640 1 -- 172.21.15.62:0/296679196 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f5b60014030 con 0x7f5b64100a30 2024-09-15T23:32:00.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.520+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f5b6402d7c0 con 0x7f5b64100a30 2024-09-15T23:32:00.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.527+0000 7f5b5b7fe640 1 -- 172.21.15.62:0/296679196 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+742337 (secure 0 0 0) 0x7f5b6009c880 con 0x7f5b64100a30 2024-09-15T23:32:00.539 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:32:00.539 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:32:00.539 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 18, 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: "active_gid": 14150, 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: "active_name": "x", 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: "active_addrs": { 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6800", 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 4035090192 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.540 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:6801", 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 4035090192 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "active_addr": "172.21.15.62:6801/4035090192", 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "active_change": "2024-09-15T23:06:39.607997+0000", 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "active_mgr_features": 4540701547738038271, 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "available": true, 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "standbys": [ 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "gid": 24109, 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "z", 2024-09-15T23:32:00.541 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_features": 4540701547738038271, 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "available_modules": [ 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "alerts", 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:00.542 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How frequently to reexamine health status", 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.543 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.544 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.545 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.546 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.547 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.548 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_destination": { 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_destination", 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Email address to send alerts to", 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.549 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_from_name": { 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_from_name", 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "Ceph", 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Email From: name", 2024-09-15T23:32:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_host": { 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_host", 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SMTP server", 2024-09-15T23:32:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_password": { 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_password", 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Password to authenticate with", 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_port": { 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_port", 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "465", 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SMTP port", 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_sender": { 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_sender", 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SMTP envelope sender", 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_ssl": { 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_ssl", 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_user": { 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_user", 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "User to authenticate as", 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "balancer", 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "active": { 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "active", 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "automatically balance PGs across cluster", 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "begin_time": { 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "begin_time", 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0000", 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "beginning time of day to automatically balance", 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "begin_weekday": { 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "begin_weekday", 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:00.560 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "0", 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "6", 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_compat_max_iterations": { 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crush_compat_max_iterations", 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "25", 2024-09-15T23:32:00.561 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "250", 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_compat_metrics": { 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crush_compat_metrics", 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "pgs,objects,bytes", 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.562 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_compat_step": { 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crush_compat_step", 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0.5", 2024-09-15T23:32:00.563 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "0.001", 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "0.999", 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "aggressiveness of optimization", 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "end_time": { 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "end_time", 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "2359", 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.564 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "ending time of day to automatically balance", 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "end_weekday": { 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "end_weekday", 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "0", 2024-09-15T23:32:00.565 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "6", 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.566 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.567 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.568 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.569 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.570 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "min_score": { 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "min_score", 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.571 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": { 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mode", 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.572 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "upmap", 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "crush-compat", 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "none", 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "read", 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap", 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap-read" 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Balancer mode", 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.573 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "pool_ids": { 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pool_ids", 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.574 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.575 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap_max_deviation": { 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "upmap_max_deviation", 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-15T23:32:00.576 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap_max_optimizations": { 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "upmap_max_optimizations", 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-15T23:32:00.577 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cephadm", 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "agent_down_multiplier": { 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "agent_down_multiplier", 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:00.578 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "3.0", 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "agent_refresh_rate": { 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "agent_refresh_rate", 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.579 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "20", 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "agent_starting_port": { 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "agent_starting_port", 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.580 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "4721", 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "allow_ptrace": { 2024-09-15T23:32:00.581 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "allow_ptrace", 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "autotune_interval": { 2024-09-15T23:32:00.582 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "autotune_interval", 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.583 INFO:teuthology.orchestra.run.smithi062.stdout: "autotune_memory_target_ratio": { 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "autotune_memory_target_ratio", 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0.7", 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.584 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm_log_destination": { 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cephadm_log_destination", 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "file", 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "file,syslog", 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "syslog" 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.585 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "cgroups_split": { 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cgroups_split", 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.586 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "config_checks_enabled": { 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "config_checks_enabled", 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.587 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "config_dashboard": { 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "config_dashboard", 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.588 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_alertmanager": { 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_alertmanager", 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-15T23:32:00.589 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.590 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.590 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.590 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:00.590 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.590 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.590 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.590 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_base": { 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_base", 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Container image name, without the tag", 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.591 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_elasticsearch": { 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_elasticsearch", 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "elasticsearch container image", 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.592 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_grafana": { 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_grafana", 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.593 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_haproxy": { 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_haproxy", 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.594 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.595 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.595 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "HAproxy container image", 2024-09-15T23:32:00.595 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.595 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.595 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.595 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.595 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_jaeger_agent": { 2024-09-15T23:32:00.595 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_jaeger_agent", 2024-09-15T23:32:00.595 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Jaeger agent container image", 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_jaeger_collector": { 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_jaeger_collector", 2024-09-15T23:32:00.596 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Jaeger collector container image", 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_jaeger_query": { 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_jaeger_query", 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Jaeger query container image", 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.598 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_keepalived": { 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_keepalived", 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Keepalived container image", 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.599 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.600 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.600 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_loki": { 2024-09-15T23:32:00.600 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_loki", 2024-09-15T23:32:00.600 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.600 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.600 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.600 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-15T23:32:00.600 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Loki container image", 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_node_exporter": { 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_node_exporter", 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.601 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.602 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_nvmeof": { 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_nvmeof", 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Nvme-of container image", 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.603 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_prometheus": { 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_prometheus", 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_promtail": { 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_promtail", 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Promtail container image", 2024-09-15T23:32:00.605 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_samba": { 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_samba", 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.606 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Samba/SMB container image", 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_snmp_gateway": { 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_snmp_gateway", 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SNMP Gateway container image", 2024-09-15T23:32:00.607 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "container_init": { 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_init", 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.608 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Run podman/docker with `--init`", 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "daemon_cache_timeout": { 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "daemon_cache_timeout", 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.609 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "default_cephadm_command_timeout": { 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "default_cephadm_command_timeout", 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "900", 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.610 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "default_registry": { 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "default_registry", 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io", 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.611 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "device_cache_timeout": { 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "device_cache_timeout", 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "1800", 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.612 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "seconds to cache device inventory", 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "device_enhanced_scan": { 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "device_enhanced_scan", 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.613 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "facts_cache_timeout": { 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "facts_cache_timeout", 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:00.614 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "seconds to cache host facts data", 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "grafana_dashboards_path": { 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "grafana_dashboards_path", 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-15T23:32:00.615 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "host_check_interval": { 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "host_check_interval", 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.616 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to perform a host check", 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "hw_monitoring": { 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "hw_monitoring", 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.617 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "inventory_list_all": { 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "inventory_list_all", 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.618 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.620 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "log_refresh_metadata": { 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_refresh_metadata", 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.621 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.622 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.623 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.624 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-15T23:32:00.625 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-15T23:32:00.626 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "*", 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.627 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "max_count_per_host": { 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_count_per_host", 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "max number of daemons per service per host", 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.628 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "max_osd_draining_count": { 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_osd_draining_count", 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.629 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "migration_current": { 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "migration_current", 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "internal - do not modify", 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.630 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": { 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mode", 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "root", 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.631 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm-package", 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "root" 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "mode for remote execution of cephadm", 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "oob_default_addr": { 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "oob_default_addr", 2024-09-15T23:32:00.632 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "169.254.1.1", 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "prometheus_alerts_path": { 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "prometheus_alerts_path", 2024-09-15T23:32:00.633 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_insecure": { 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_insecure", 2024-09-15T23:32:00.634 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_password": { 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_password", 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.635 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_url": { 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_url", 2024-09-15T23:32:00.636 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_username": { 2024-09-15T23:32:00.637 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_username", 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "secure_monitoring_stack": { 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "secure_monitoring_stack", 2024-09-15T23:32:00.638 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "service_discovery_port": { 2024-09-15T23:32:00.639 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "service_discovery_port", 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8765", 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "cephadm service discovery port", 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "ssh_config_file": { 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssh_config_file", 2024-09-15T23:32:00.640 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.641 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.641 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.641 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.641 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.641 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.641 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.641 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-15T23:32:00.641 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "use_agent": { 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "use_agent", 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.642 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "use_repo_digest": { 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "use_repo_digest", 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.643 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_on_failed_host_check": { 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_on_failed_host_check", 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.644 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise a health warning if the host check fails", 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_on_stray_daemons": { 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_on_stray_daemons", 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.645 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_on_stray_hosts": { 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_on_stray_hosts", 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.646 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crash", 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.647 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.648 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.649 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.650 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.651 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.652 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "retain_interval": { 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "retain_interval", 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "31536000", 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.653 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_recent_interval": { 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_recent_interval", 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "1209600", 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.654 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "dashboard", 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.655 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "ALERTMANAGER_API_HOST": { 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.657 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "AUDIT_API_ENABLED": { 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "AUDIT_API_ENABLED", 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.658 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.659 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "ENABLE_BROWSABLE_API": { 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.660 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.661 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.662 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.663 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.664 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.665 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.666 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.667 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.668 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_PASSWORD": { 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.669 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "admin", 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.670 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_URL": { 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_URL", 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.671 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_USERNAME": { 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.672 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "admin", 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.673 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.674 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.675 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.676 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.677 INFO:teuthology.orchestra.run.smithi062.stdout: "PROMETHEUS_API_HOST": { 2024-09-15T23:32:00.678 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-15T23:32:00.678 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.678 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.678 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.678 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.678 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.678 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.679 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.679 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.679 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.679 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.679 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.679 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.679 INFO:teuthology.orchestra.run.smithi062.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-15T23:32:00.680 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-15T23:32:00.680 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.680 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.680 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.680 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.680 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.681 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.681 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.681 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.681 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.681 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.682 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.682 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.682 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-15T23:32:00.682 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-15T23:32:00.682 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.682 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.683 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.683 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.683 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.683 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.683 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.683 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.684 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.684 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.684 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.684 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.684 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-15T23:32:00.684 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-15T23:32:00.685 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.685 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.685 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.685 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.685 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.685 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.685 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.686 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.686 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.686 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.686 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.686 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.686 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-15T23:32:00.687 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-15T23:32:00.687 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.687 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.687 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.687 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.687 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.688 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.688 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.688 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.688 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.688 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.689 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.689 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.689 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-15T23:32:00.689 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-15T23:32:00.689 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.689 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.690 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.690 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.690 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.690 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.690 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.691 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.691 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.691 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.691 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.691 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.691 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-15T23:32:00.692 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-15T23:32:00.692 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.692 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.692 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.692 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.692 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.693 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.693 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.693 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.693 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.693 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.693 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.694 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.694 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-15T23:32:00.694 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-15T23:32:00.694 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.694 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.694 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.695 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.695 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.695 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.695 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.695 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.695 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.696 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.696 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.696 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.696 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-15T23:32:00.696 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-15T23:32:00.697 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.697 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.697 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.697 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.697 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.697 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.698 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.698 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.698 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.698 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.698 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.698 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.699 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_ENABLED": { 2024-09-15T23:32:00.699 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-15T23:32:00.699 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.699 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.699 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.699 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.699 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.700 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.700 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.700 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.700 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.700 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.700 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.701 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.702 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8", 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.703 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "45", 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.704 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_ACCESS_KEY": { 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.705 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.706 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "admin", 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_SECRET_KEY": { 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.707 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_SSL_VERIFY": { 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.708 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.709 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.710 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.711 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "cross_origin_url": { 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cross_origin_url", 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.712 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "crt_file": { 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crt_file", 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.713 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "debug": { 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "debug", 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.714 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Enable/disable debug options", 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "jwt_token_ttl": { 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "jwt_token_ttl", 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.715 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "28800", 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "key_file": { 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "key_file", 2024-09-15T23:32:00.716 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.717 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.718 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.719 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.720 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.721 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "motd": { 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "motd", 2024-09-15T23:32:00.722 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "The message of the day", 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "redirect_resolve_ip_addr": { 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "redirect_resolve_ip_addr", 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.723 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "server_addr": { 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_addr", 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.724 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "::", 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "server_port": { 2024-09-15T23:32:00.725 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_port", 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8080", 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "ssl": { 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssl", 2024-09-15T23:32:00.726 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "ssl_server_port": { 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssl_server_port", 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.727 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8443", 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_behaviour": { 2024-09-15T23:32:00.728 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_behaviour", 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "redirect", 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "redirect" 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.729 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_error_status_code": { 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_error_status_code", 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "500", 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "400", 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "599", 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.730 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "url_prefix": { 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "url_prefix", 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.731 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "devicehealth", 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: "enable_monitoring": { 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enable_monitoring", 2024-09-15T23:32:00.732 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "monitor device health metrics", 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.733 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.734 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.735 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.736 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.737 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.738 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "mark_out_threshold": { 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mark_out_threshold", 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "2419200", 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.739 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "pool_name": { 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pool_name", 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "device_health_metrics", 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-15T23:32:00.740 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "retention_period": { 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "retention_period", 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "15552000", 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how long to retain device health metrics", 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "scrape_frequency": { 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "scrape_frequency", 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "86400", 2024-09-15T23:32:00.742 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "self_heal": { 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "self_heal", 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.743 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.744 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to wake up and check device health", 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_threshold": { 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_threshold", 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.745 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "7257600", 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.746 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "diskprediction_local", 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.747 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.748 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.749 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.750 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.750 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.750 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.750 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.750 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.750 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.750 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.751 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.751 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.751 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.751 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.751 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.751 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.752 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.752 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.752 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.752 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.752 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.752 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.753 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.753 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.753 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.753 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.753 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.753 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.754 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.754 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.754 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.754 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.754 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.754 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.754 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.755 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.755 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.755 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.755 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.755 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.755 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.756 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.756 INFO:teuthology.orchestra.run.smithi062.stdout: "predict_interval": { 2024-09-15T23:32:00.756 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "predict_interval", 2024-09-15T23:32:00.756 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.756 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.756 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.757 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "86400", 2024-09-15T23:32:00.757 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.757 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.757 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.757 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.757 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.758 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.758 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.758 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.758 INFO:teuthology.orchestra.run.smithi062.stdout: "predictor_model": { 2024-09-15T23:32:00.758 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "predictor_model", 2024-09-15T23:32:00.758 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.759 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.759 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.759 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "prophetstor", 2024-09-15T23:32:00.759 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.759 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.759 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.760 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.760 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.760 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.760 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.760 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.760 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:00.761 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:00.761 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.761 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.761 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.761 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:00.761 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.762 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.762 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.762 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.762 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.762 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.762 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.763 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.763 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.763 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.763 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "influx", 2024-09-15T23:32:00.763 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": false, 2024-09-15T23:32:00.764 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "influxdb python module not found", 2024-09-15T23:32:00.764 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.764 INFO:teuthology.orchestra.run.smithi062.stdout: "batch_size": { 2024-09-15T23:32:00.764 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "batch_size", 2024-09-15T23:32:00.764 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.764 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.765 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.765 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5000", 2024-09-15T23:32:00.765 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.765 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.765 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.765 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-15T23:32:00.765 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.766 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.766 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.766 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.766 INFO:teuthology.orchestra.run.smithi062.stdout: "database": { 2024-09-15T23:32:00.766 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "database", 2024-09-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "ceph", 2024-09-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-15T23:32:00.767 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": { 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "hostname", 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.768 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "InfluxDB server hostname", 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "30", 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "5", 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.769 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.770 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.771 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.772 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.773 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "password": { 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "password", 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.775 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "password of InfluxDB server user", 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "port": { 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "port", 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8086", 2024-09-15T23:32:00.776 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "InfluxDB server port", 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "ssl": { 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssl", 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "false", 2024-09-15T23:32:00.777 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "threads": { 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "threads", 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:00.778 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "32", 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "username": { 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "username", 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.779 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "username of InfluxDB server user", 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "verify_ssl": { 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "verify_ssl", 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "true", 2024-09-15T23:32:00.780 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.781 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "insights", 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.783 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.784 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.785 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.786 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.787 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "iostat", 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.788 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.789 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.790 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.791 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.792 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "k8sevents", 2024-09-15T23:32:00.793 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_event_retention_days": { 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ceph_event_retention_days", 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "7", 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.794 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "config_check_secs": { 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "config_check_secs", 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "10", 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-15T23:32:00.795 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.796 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.797 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.799 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.800 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "localpool", 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "failure_domain": { 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "failure_domain", 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "host", 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.802 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "failure domain for any created local pool", 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.803 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.804 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.805 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.806 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.807 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "min_size": { 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "min_size", 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.808 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "default min_size for any created local pool", 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "num_rep": { 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "num_rep", 2024-09-15T23:32:00.809 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "3", 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "default replica count for any created local pool", 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.810 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "pg_num": { 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pg_num", 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "128", 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "default pg_num for any created local pool", 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.811 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "prefix": { 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "prefix", 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "name prefix for any created local pool", 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.812 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "subtree": { 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "subtree", 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "rack", 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.813 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mds_autoscaler", 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.814 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.815 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.816 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.817 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.818 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.819 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mirroring", 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.820 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.821 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.822 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.823 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.824 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "nfs", 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.825 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.826 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.827 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.828 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.829 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.830 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "orchestrator", 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "fail_fs": { 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "fail_fs", 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.831 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.832 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.833 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.834 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.835 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.836 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "orchestrator": { 2024-09-15T23:32:00.837 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "orchestrator", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "rook", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "test_orchestrator" 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Orchestrator backend", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.838 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "osd_perf_query", 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.839 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.840 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.841 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.842 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.843 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.844 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "osd_support", 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.845 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.846 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.847 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.849 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pg_autoscaler", 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.850 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.851 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.852 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.853 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.853 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.853 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.853 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.853 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.853 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.853 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.853 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.853 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.854 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.855 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "threshold": { 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "threshold", 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.856 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "3.0", 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1.0", 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "scaling threshold", 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "progress", 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.857 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "allow_pg_recovery_event": { 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "allow_pg_recovery_event", 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.858 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": { 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enabled", 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.859 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.860 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.861 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.862 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.863 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.864 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "max_completed_events": { 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_completed_events", 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "50", 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.865 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "number of past completed events to remember", 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.866 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how long the module is going to sleep", 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "prometheus", 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "cache": { 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cache", 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.867 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "exclude_perf_counters": { 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "exclude_perf_counters", 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.868 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.869 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.870 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.871 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.872 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.873 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_stats_pools": { 2024-09-15T23:32:00.874 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_stats_pools", 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-15T23:32:00.875 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "300", 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "scrape_interval": { 2024-09-15T23:32:00.876 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "scrape_interval", 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "15.0", 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.877 INFO:teuthology.orchestra.run.smithi062.stdout: "server_addr": { 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_addr", 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "::", 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.878 INFO:teuthology.orchestra.run.smithi062.stdout: "server_port": { 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_port", 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "9283", 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.879 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.880 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.880 INFO:teuthology.orchestra.run.smithi062.stdout: "stale_cache_strategy": { 2024-09-15T23:32:00.880 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "stale_cache_strategy", 2024-09-15T23:32:00.880 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.880 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.880 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.880 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "log", 2024-09-15T23:32:00.880 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.880 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.881 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.881 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.881 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.881 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.881 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.881 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.881 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_behaviour": { 2024-09-15T23:32:00.881 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_behaviour", 2024-09-15T23:32:00.881 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "default", 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: "default", 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: "error" 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.882 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_error_status_code": { 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_error_status_code", 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "500", 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "400", 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "599", 2024-09-15T23:32:00.883 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_support", 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.884 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.885 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.886 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.887 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.888 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.889 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.890 INFO:teuthology.orchestra.run.smithi062.stdout: "max_concurrent_snap_create": { 2024-09-15T23:32:00.890 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_concurrent_snap_create", 2024-09-15T23:32:00.890 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.890 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.890 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.890 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:00.890 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.890 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.890 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "mirror_snapshot_schedule": { 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mirror_snapshot_schedule", 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.891 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "trash_purge_schedule": { 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "trash_purge_schedule", 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.892 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "restful", 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.893 INFO:teuthology.orchestra.run.smithi062.stdout: "enable_auth": { 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enable_auth", 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.894 INFO:teuthology.orchestra.run.smithi062.stdout: "key_file": { 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "key_file", 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.895 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.896 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.897 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.898 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.899 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.900 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.901 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.901 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.901 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.901 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.901 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.901 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.901 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.901 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.901 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "server_addr": { 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_addr", 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.902 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "server_port": { 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_port", 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.903 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rgw", 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.904 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.905 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.906 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.907 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.908 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.909 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "secondary_zone_period_retry_limit": { 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.910 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rook", 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.911 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.912 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.912 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.912 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.912 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.912 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.912 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.912 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.912 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.912 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.913 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.914 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.915 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.916 INFO:teuthology.orchestra.run.smithi062.stdout: "storage_class": { 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "storage_class", 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "local", 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.917 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "selftest", 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.918 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.919 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.920 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.921 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.922 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "roption1": { 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "roption1", 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.923 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "roption2": { 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "roption2", 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.924 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "xyz", 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption1": { 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption1", 2024-09-15T23:32:00.925 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption2": { 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption2", 2024-09-15T23:32:00.926 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption3": { 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption3", 2024-09-15T23:32:00.927 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption4": { 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption4", 2024-09-15T23:32:00.928 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.929 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption5": { 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption5", 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.930 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption6": { 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption6", 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.931 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption7": { 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption7", 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "42", 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.932 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.933 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.933 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.933 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.933 INFO:teuthology.orchestra.run.smithi062.stdout: "testkey": { 2024-09-15T23:32:00.933 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "testkey", 2024-09-15T23:32:00.933 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.933 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.933 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.933 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "testlkey": { 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "testlkey", 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.934 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "testnewline": { 2024-09-15T23:32:00.935 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "testnewline", 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.936 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "snap_schedule", 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "allow_m_granularity": { 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "allow_m_granularity", 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.937 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "allow minute scheduled snapshots", 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "dump_on_update": { 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "dump_on_update", 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.938 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "dump database to debug log on update", 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.939 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.940 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.941 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.942 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.943 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.944 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "stats", 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.945 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.946 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.947 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.948 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.948 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.948 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.948 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.948 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.948 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.949 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.949 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.949 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.949 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.949 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.949 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.949 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.950 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.950 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.950 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.950 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.950 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.950 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.951 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.951 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.951 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.951 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.951 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.951 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.952 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.952 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.952 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.952 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.952 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.952 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.952 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.953 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.953 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.953 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.953 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.953 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.953 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.954 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.954 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "status", 2024-09-15T23:32:00.954 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.954 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.954 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.954 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.955 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.955 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.955 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.955 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.955 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.955 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.956 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.956 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.956 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.956 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.956 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.956 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.957 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.957 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.957 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.957 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.957 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.957 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.958 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.958 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.958 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.958 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.958 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.958 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.959 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.959 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.959 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.959 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.959 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.959 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.960 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.960 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.960 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.960 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.960 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.960 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.961 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.961 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.961 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.961 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.961 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.962 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.962 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.962 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.962 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.962 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.962 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.962 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.963 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.963 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.963 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.963 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.963 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.963 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.963 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.964 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.964 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.964 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.964 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.964 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.964 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.964 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.965 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.965 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.965 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.965 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.965 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.965 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.965 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.965 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.966 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.966 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.966 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "telegraf", 2024-09-15T23:32:00.966 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.966 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.966 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.967 INFO:teuthology.orchestra.run.smithi062.stdout: "address": { 2024-09-15T23:32:00.967 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "address", 2024-09-15T23:32:00.967 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.967 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.967 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.967 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-15T23:32:00.967 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.968 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.968 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.968 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.968 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.968 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.968 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.969 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.969 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:00.969 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:00.969 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:00.969 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.969 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.969 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "15", 2024-09-15T23:32:00.970 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.970 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.970 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.970 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.970 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.970 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.971 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.971 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.971 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.971 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.971 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.971 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.972 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.972 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.972 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.972 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.972 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.972 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.973 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.973 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.973 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.973 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.973 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.974 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.974 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.974 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.974 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.974 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.974 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.975 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:00.975 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:00.975 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.975 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.975 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.975 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.976 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.976 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.976 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.976 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.976 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.976 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.977 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.977 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.977 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:00.977 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:00.977 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.978 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.978 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.978 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:00.978 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.978 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.978 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.979 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.979 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.979 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.979 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.979 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.979 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.980 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.980 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.980 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.980 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.980 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.980 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.980 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:00.980 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:00.981 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.981 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.981 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.981 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.981 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.981 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.982 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.982 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.982 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.982 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.982 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "telemetry", 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_basic": { 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_basic", 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.983 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_crash": { 2024-09-15T23:32:00.984 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_crash", 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.985 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_device": { 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_device", 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.986 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_ident": { 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_ident", 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_perf": { 2024-09-15T23:32:00.987 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_perf", 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.988 INFO:teuthology.orchestra.run.smithi062.stdout: "contact": { 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "contact", 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.989 INFO:teuthology.orchestra.run.smithi062.stdout: "description": { 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "description", 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.990 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "device_url": { 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "device_url", 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.991 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": { 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enabled", 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.992 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "24", 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "8", 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.993 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "last_opt_revision": { 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "last_opt_revision", 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "1", 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.994 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.995 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.995 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.995 INFO:teuthology.orchestra.run.smithi062.stdout: "leaderboard": { 2024-09-15T23:32:00.995 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "leaderboard", 2024-09-15T23:32:00.995 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:00.995 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.995 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.995 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "leaderboard_description": { 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "leaderboard_description", 2024-09-15T23:32:00.996 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.997 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.997 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:00.997 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.997 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.997 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.997 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:00.997 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.997 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:00.998 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:00.999 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.000 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.001 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.002 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "organization": { 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "organization", 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.003 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "proxy": { 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "proxy", 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.004 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "url": { 2024-09-15T23:32:01.005 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "url", 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.006 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "test_orchestrator", 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.007 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.008 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.009 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.010 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.010 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.010 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.010 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.010 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.010 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.011 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.012 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "volumes", 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.013 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.014 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.015 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.016 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.016 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.016 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.016 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.016 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.016 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.016 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.016 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.016 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.017 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.017 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.017 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.017 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.017 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.017 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.017 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.017 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.018 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.018 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.018 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.018 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.018 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.018 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.018 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.018 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.019 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.019 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.019 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.019 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.019 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.019 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.019 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.019 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.019 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.020 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.020 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.020 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.020 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.020 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.020 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.020 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.020 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.020 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.021 INFO:teuthology.orchestra.run.smithi062.stdout: "max_concurrent_clones": { 2024-09-15T23:32:01.021 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_concurrent_clones", 2024-09-15T23:32:01.021 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.021 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.021 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.021 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "4", 2024-09-15T23:32:01.021 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.021 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.021 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.022 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-15T23:32:01.022 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.022 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.022 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.022 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.022 INFO:teuthology.orchestra.run.smithi062.stdout: "periodic_async_work": { 2024-09-15T23:32:01.022 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "periodic_async_work", 2024-09-15T23:32:01.022 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.023 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.023 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.023 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.023 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.023 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.023 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.023 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Periodically check for async work", 2024-09-15T23:32:01.023 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.023 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.024 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.024 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.024 INFO:teuthology.orchestra.run.smithi062.stdout: "snapshot_clone_delay": { 2024-09-15T23:32:01.024 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "snapshot_clone_delay", 2024-09-15T23:32:01.024 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.024 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.024 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.024 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.025 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.025 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.025 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.025 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-15T23:32:01.025 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.025 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.025 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.025 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.026 INFO:teuthology.orchestra.run.smithi062.stdout: "snapshot_clone_no_wait": { 2024-09-15T23:32:01.026 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "snapshot_clone_no_wait", 2024-09-15T23:32:01.026 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.026 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.026 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.026 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.026 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.026 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.026 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.027 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-15T23:32:01.027 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.027 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.027 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.027 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.027 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.027 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.027 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.028 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix", 2024-09-15T23:32:01.028 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.028 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.028 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.028 INFO:teuthology.orchestra.run.smithi062.stdout: "discovery_interval": { 2024-09-15T23:32:01.028 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "discovery_interval", 2024-09-15T23:32:01.028 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.028 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.028 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.029 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "100", 2024-09-15T23:32:01.029 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.029 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.029 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.029 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.029 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.029 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.029 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "identifier": { 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "identifier", 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.030 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.031 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.032 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.033 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.034 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.035 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.036 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "zabbix_host": { 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix_host", 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.037 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "zabbix_port": { 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix_port", 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10051", 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.038 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "zabbix_sender": { 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix_sender", 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.039 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: "gid": 24113, 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "y", 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr_features": 4540701547738038271, 2024-09-15T23:32:01.040 INFO:teuthology.orchestra.run.smithi062.stdout: "available_modules": [ 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "alerts", 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.041 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How frequently to reexamine health status", 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.042 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.043 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.044 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.045 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.046 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_destination": { 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_destination", 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.047 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Email address to send alerts to", 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_from_name": { 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_from_name", 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "Ceph", 2024-09-15T23:32:01.048 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Email From: name", 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_host": { 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_host", 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.049 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SMTP server", 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_password": { 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_password", 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.050 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Password to authenticate with", 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_port": { 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_port", 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "465", 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.051 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SMTP port", 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_sender": { 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_sender", 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.052 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SMTP envelope sender", 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_ssl": { 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_ssl", 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.053 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_user": { 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_user", 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.054 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "User to authenticate as", 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "balancer", 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "active": { 2024-09-15T23:32:01.055 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "active", 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "automatically balance PGs across cluster", 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "begin_time": { 2024-09-15T23:32:01.056 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "begin_time", 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0000", 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "beginning time of day to automatically balance", 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.057 INFO:teuthology.orchestra.run.smithi062.stdout: "begin_weekday": { 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "begin_weekday", 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "0", 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "6", 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.058 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_compat_max_iterations": { 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crush_compat_max_iterations", 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "25", 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "250", 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.059 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_compat_metrics": { 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crush_compat_metrics", 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "pgs,objects,bytes", 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.060 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_compat_step": { 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crush_compat_step", 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0.5", 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "0.001", 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "0.999", 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "aggressiveness of optimization", 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.061 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "end_time": { 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "end_time", 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "2359", 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "ending time of day to automatically balance", 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.062 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "end_weekday": { 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "end_weekday", 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "0", 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "6", 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.063 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.064 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.065 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.066 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.067 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.068 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "min_score": { 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "min_score", 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.069 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": { 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mode", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "upmap", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "crush-compat", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "none", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "read", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap", 2024-09-15T23:32:01.070 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap-read" 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Balancer mode", 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "pool_ids": { 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pool_ids", 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.071 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.072 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap_max_deviation": { 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "upmap_max_deviation", 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.073 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap_max_optimizations": { 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "upmap_max_optimizations", 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.074 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cephadm", 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "agent_down_multiplier": { 2024-09-15T23:32:01.075 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "agent_down_multiplier", 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "3.0", 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "agent_refresh_rate": { 2024-09-15T23:32:01.076 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "agent_refresh_rate", 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "20", 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "agent_starting_port": { 2024-09-15T23:32:01.077 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "agent_starting_port", 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "4721", 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.078 INFO:teuthology.orchestra.run.smithi062.stdout: "allow_ptrace": { 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "allow_ptrace", 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.079 INFO:teuthology.orchestra.run.smithi062.stdout: "autotune_interval": { 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "autotune_interval", 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.080 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "autotune_memory_target_ratio": { 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "autotune_memory_target_ratio", 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0.7", 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm_log_destination": { 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cephadm_log_destination", 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "file", 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "file,syslog", 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: "syslog" 2024-09-15T23:32:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "cgroups_split": { 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cgroups_split", 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.083 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "config_checks_enabled": { 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "config_checks_enabled", 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.084 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "config_dashboard": { 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "config_dashboard", 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.085 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_alertmanager": { 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_alertmanager", 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-15T23:32:01.086 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_base": { 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_base", 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-15T23:32:01.087 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Container image name, without the tag", 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_elasticsearch": { 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_elasticsearch", 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-15T23:32:01.088 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "elasticsearch container image", 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_grafana": { 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_grafana", 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.089 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_haproxy": { 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_haproxy", 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-15T23:32:01.090 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "HAproxy container image", 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_jaeger_agent": { 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_jaeger_agent", 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-15T23:32:01.091 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Jaeger agent container image", 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_jaeger_collector": { 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_jaeger_collector", 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-15T23:32:01.092 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Jaeger collector container image", 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_jaeger_query": { 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_jaeger_query", 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-15T23:32:01.093 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Jaeger query container image", 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_keepalived": { 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_keepalived", 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.094 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Keepalived container image", 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_loki": { 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_loki", 2024-09-15T23:32:01.095 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Loki container image", 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.096 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_node_exporter": { 2024-09-15T23:32:01.097 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_node_exporter", 2024-09-15T23:32:01.097 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.097 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.097 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.097 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-15T23:32:01.097 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.097 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.097 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_nvmeof": { 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_nvmeof", 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.098 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Nvme-of container image", 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_prometheus": { 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_prometheus", 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.099 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_promtail": { 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_promtail", 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.100 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Promtail container image", 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_samba": { 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_samba", 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.101 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Samba/SMB container image", 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_snmp_gateway": { 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_snmp_gateway", 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.102 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SNMP Gateway container image", 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "container_init": { 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_init", 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.103 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Run podman/docker with `--init`", 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "daemon_cache_timeout": { 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "daemon_cache_timeout", 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.104 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "default_cephadm_command_timeout": { 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "default_cephadm_command_timeout", 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.105 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "900", 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "default_registry": { 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "default_registry", 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.106 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io", 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "device_cache_timeout": { 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "device_cache_timeout", 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.107 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "1800", 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "seconds to cache device inventory", 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "device_enhanced_scan": { 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "device_enhanced_scan", 2024-09-15T23:32:01.108 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "facts_cache_timeout": { 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "facts_cache_timeout", 2024-09-15T23:32:01.109 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "seconds to cache host facts data", 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "grafana_dashboards_path": { 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "grafana_dashboards_path", 2024-09-15T23:32:01.110 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.111 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.112 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.112 INFO:teuthology.orchestra.run.smithi062.stdout: "host_check_interval": { 2024-09-15T23:32:01.112 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "host_check_interval", 2024-09-15T23:32:01.112 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.112 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.112 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.113 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.113 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.113 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.113 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.113 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to perform a host check", 2024-09-15T23:32:01.113 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.114 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.114 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.114 INFO:teuthology.orchestra.run.smithi062.stdout: "hw_monitoring": { 2024-09-15T23:32:01.114 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "hw_monitoring", 2024-09-15T23:32:01.114 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.114 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.114 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.115 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.115 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.115 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.115 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.115 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-15T23:32:01.115 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.115 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.116 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.116 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.116 INFO:teuthology.orchestra.run.smithi062.stdout: "inventory_list_all": { 2024-09-15T23:32:01.116 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "inventory_list_all", 2024-09-15T23:32:01.116 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.116 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.116 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.116 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.117 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.117 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.117 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.117 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-15T23:32:01.117 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.117 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.117 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.118 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.118 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.118 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.118 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.118 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.118 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.119 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.119 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.119 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.119 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.119 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.119 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.119 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.120 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.120 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.120 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.120 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.120 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.120 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.120 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "log_refresh_metadata": { 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_refresh_metadata", 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.121 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.122 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.123 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.124 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-15T23:32:01.125 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-15T23:32:01.126 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "*", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "max_count_per_host": { 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_count_per_host", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.127 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "max number of daemons per service per host", 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "max_osd_draining_count": { 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_osd_draining_count", 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.128 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "migration_current": { 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "migration_current", 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.129 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "internal - do not modify", 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": { 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mode", 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.130 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "root", 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm-package", 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "root" 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "mode for remote execution of cephadm", 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.131 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "oob_default_addr": { 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "oob_default_addr", 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "169.254.1.1", 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.132 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "prometheus_alerts_path": { 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "prometheus_alerts_path", 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-15T23:32:01.133 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_insecure": { 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_insecure", 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.134 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_password": { 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_password", 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.135 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_url": { 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_url", 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.136 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_username": { 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_username", 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.137 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "secure_monitoring_stack": { 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "secure_monitoring_stack", 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.138 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "service_discovery_port": { 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "service_discovery_port", 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.139 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8765", 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "cephadm service discovery port", 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "ssh_config_file": { 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssh_config_file", 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.140 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "use_agent": { 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "use_agent", 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.141 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "use_repo_digest": { 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "use_repo_digest", 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.142 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_on_failed_host_check": { 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_on_failed_host_check", 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.143 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise a health warning if the host check fails", 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_on_stray_daemons": { 2024-09-15T23:32:01.144 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_on_stray_daemons", 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.145 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.146 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.146 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_on_stray_hosts": { 2024-09-15T23:32:01.146 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_on_stray_hosts", 2024-09-15T23:32:01.146 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.146 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.146 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.146 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.147 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.147 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.147 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.147 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-15T23:32:01.147 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.147 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.147 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.148 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.148 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.148 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.148 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.148 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crash", 2024-09-15T23:32:01.148 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.149 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.149 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.149 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.149 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.149 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.149 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.150 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.150 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.150 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.150 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.150 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.150 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.151 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.151 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.151 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.151 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.151 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.151 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.152 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.152 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.152 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.152 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.152 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.152 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.153 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.153 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.153 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.153 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.153 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.153 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.153 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.154 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.154 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.154 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.154 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.154 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.154 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.155 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.155 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.155 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.155 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.155 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.155 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.156 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.156 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.156 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.156 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.156 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.156 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.157 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.157 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.157 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.157 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.157 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.157 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.158 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.158 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.158 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.158 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.158 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.158 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.159 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.159 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.159 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.159 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.159 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.159 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.159 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.160 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.160 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.160 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.160 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.160 INFO:teuthology.orchestra.run.smithi062.stdout: "retain_interval": { 2024-09-15T23:32:01.160 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "retain_interval", 2024-09-15T23:32:01.160 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.161 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.161 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.161 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "31536000", 2024-09-15T23:32:01.161 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.161 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.161 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.162 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-15T23:32:01.162 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.162 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.162 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.162 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.162 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_recent_interval": { 2024-09-15T23:32:01.162 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_recent_interval", 2024-09-15T23:32:01.162 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "1209600", 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.163 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "dashboard", 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.164 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "ALERTMANAGER_API_HOST": { 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.165 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.166 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "AUDIT_API_ENABLED": { 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "AUDIT_API_ENABLED", 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.167 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.168 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "ENABLE_BROWSABLE_API": { 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.169 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.170 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.171 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.172 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.173 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.174 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.175 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.176 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_PASSWORD": { 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-15T23:32:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "admin", 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-15T23:32:01.179 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_URL": { 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_URL", 2024-09-15T23:32:01.180 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_USERNAME": { 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.181 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "admin", 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.182 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-15T23:32:01.183 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.184 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.185 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.186 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "PROMETHEUS_API_HOST": { 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.187 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.188 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.189 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.190 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.191 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.192 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.193 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.194 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.194 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.194 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.194 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.194 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.194 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-15T23:32:01.194 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-15T23:32:01.194 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.194 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.195 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.195 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.195 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.195 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.195 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.195 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.195 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.196 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_ENABLED": { 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.197 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.198 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.199 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.200 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8", 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.201 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "45", 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_ACCESS_KEY": { 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.202 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-15T23:32:01.203 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "admin", 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_SECRET_KEY": { 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-15T23:32:01.204 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_SSL_VERIFY": { 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-15T23:32:01.205 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-15T23:32:01.206 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-15T23:32:01.207 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-15T23:32:01.208 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "cross_origin_url": { 2024-09-15T23:32:01.209 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cross_origin_url", 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "crt_file": { 2024-09-15T23:32:01.210 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crt_file", 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.211 INFO:teuthology.orchestra.run.smithi062.stdout: "debug": { 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "debug", 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Enable/disable debug options", 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.212 INFO:teuthology.orchestra.run.smithi062.stdout: "jwt_token_ttl": { 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "jwt_token_ttl", 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "28800", 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.213 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "key_file": { 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "key_file", 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.214 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.215 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.216 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.217 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.218 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.219 INFO:teuthology.orchestra.run.smithi062.stdout: "motd": { 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "motd", 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "The message of the day", 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "redirect_resolve_ip_addr": { 2024-09-15T23:32:01.220 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "redirect_resolve_ip_addr", 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "server_addr": { 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_addr", 2024-09-15T23:32:01.221 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "::", 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "server_port": { 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_port", 2024-09-15T23:32:01.222 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8080", 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "ssl": { 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssl", 2024-09-15T23:32:01.223 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.224 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "ssl_server_port": { 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssl_server_port", 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8443", 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.225 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_behaviour": { 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_behaviour", 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "redirect", 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.226 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "redirect" 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_error_status_code": { 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_error_status_code", 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.227 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "500", 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "400", 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "599", 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "url_prefix": { 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "url_prefix", 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.228 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.229 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "devicehealth", 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "enable_monitoring": { 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enable_monitoring", 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.230 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "monitor device health metrics", 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.231 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.232 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.234 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.235 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.236 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "mark_out_threshold": { 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mark_out_threshold", 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "2419200", 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.237 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "pool_name": { 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pool_name", 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "device_health_metrics", 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.238 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "retention_period": { 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "retention_period", 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "15552000", 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.239 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how long to retain device health metrics", 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "scrape_frequency": { 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "scrape_frequency", 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "86400", 2024-09-15T23:32:01.240 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "self_heal": { 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "self_heal", 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.241 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.242 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to wake up and check device health", 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_threshold": { 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_threshold", 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "7257600", 2024-09-15T23:32:01.243 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "diskprediction_local", 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.244 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.245 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.246 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.247 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.248 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.249 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "predict_interval": { 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "predict_interval", 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "86400", 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.250 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "predictor_model": { 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "predictor_model", 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "prophetstor", 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.251 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.252 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "influx", 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": false, 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "influxdb python module not found", 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "batch_size": { 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "batch_size", 2024-09-15T23:32:01.253 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5000", 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "database": { 2024-09-15T23:32:01.254 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "database", 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "ceph", 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": { 2024-09-15T23:32:01.255 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "hostname", 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "InfluxDB server hostname", 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.256 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "30", 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "5", 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.257 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.258 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.259 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.260 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.261 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "password": { 2024-09-15T23:32:01.262 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "password", 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "password of InfluxDB server user", 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "port": { 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "port", 2024-09-15T23:32:01.263 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8086", 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "InfluxDB server port", 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "ssl": { 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssl", 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.264 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "false", 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "threads": { 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "threads", 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.265 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "32", 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "username": { 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "username", 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.266 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "username of InfluxDB server user", 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "verify_ssl": { 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "verify_ssl", 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.267 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "true", 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.268 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "insights", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.269 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.270 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.271 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.272 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.273 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "iostat", 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.274 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.276 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.277 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.278 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "k8sevents", 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_event_retention_days": { 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ceph_event_retention_days", 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.279 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "7", 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "config_check_secs": { 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "config_check_secs", 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.280 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "10", 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.281 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.282 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.283 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.284 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.285 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "localpool", 2024-09-15T23:32:01.286 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "failure_domain": { 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "failure_domain", 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "host", 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "failure domain for any created local pool", 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.287 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.288 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.289 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.290 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.291 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.292 INFO:teuthology.orchestra.run.smithi062.stdout: "min_size": { 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "min_size", 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "default min_size for any created local pool", 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.293 INFO:teuthology.orchestra.run.smithi062.stdout: "num_rep": { 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "num_rep", 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "3", 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "default replica count for any created local pool", 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.294 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "pg_num": { 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pg_num", 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "128", 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "default pg_num for any created local pool", 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.295 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "prefix": { 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "prefix", 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "name prefix for any created local pool", 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.296 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "subtree": { 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "subtree", 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "rack", 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-15T23:32:01.297 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mds_autoscaler", 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.298 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.299 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.300 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.301 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.302 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.303 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mirroring", 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.304 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.305 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.306 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.307 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.308 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "nfs", 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.309 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.310 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.311 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.312 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.313 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.314 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "orchestrator", 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "fail_fs": { 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "fail_fs", 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-15T23:32:01.315 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.316 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.317 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.318 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.319 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.320 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "orchestrator": { 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "orchestrator", 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.321 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm", 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "rook", 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "test_orchestrator" 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Orchestrator backend", 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.322 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "osd_perf_query", 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.323 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.324 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.325 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.326 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.327 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "osd_support", 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.328 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.329 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.330 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.331 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.332 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.333 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pg_autoscaler", 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.334 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.336 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.337 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.338 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.339 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "threshold": { 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "threshold", 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "3.0", 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1.0", 2024-09-15T23:32:01.340 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "scaling threshold", 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "progress", 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.341 INFO:teuthology.orchestra.run.smithi062.stdout: "allow_pg_recovery_event": { 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "allow_pg_recovery_event", 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": { 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enabled", 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.344 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.345 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.346 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.347 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.348 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "max_completed_events": { 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_completed_events", 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "50", 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "number of past completed events to remember", 2024-09-15T23:32:01.349 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.350 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how long the module is going to sleep", 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "prometheus", 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "cache": { 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cache", 2024-09-15T23:32:01.351 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "exclude_perf_counters": { 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "exclude_perf_counters", 2024-09-15T23:32:01.352 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.353 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.354 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.355 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.356 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.357 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_stats_pools": { 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_stats_pools", 2024-09-15T23:32:01.358 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-15T23:32:01.359 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "300", 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "scrape_interval": { 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "scrape_interval", 2024-09-15T23:32:01.360 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "15.0", 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "server_addr": { 2024-09-15T23:32:01.361 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_addr", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "::", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "server_port": { 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_port", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.362 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "9283", 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "stale_cache_strategy": { 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "stale_cache_strategy", 2024-09-15T23:32:01.363 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "log", 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_behaviour": { 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_behaviour", 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.364 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "default", 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "default", 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "error" 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.365 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_error_status_code": { 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_error_status_code", 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "500", 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "400", 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "599", 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.366 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_support", 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.367 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.368 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.369 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.370 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.371 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.372 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "max_concurrent_snap_create": { 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_concurrent_snap_create", 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.373 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "mirror_snapshot_schedule": { 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mirror_snapshot_schedule", 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.374 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "trash_purge_schedule": { 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "trash_purge_schedule", 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.375 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "restful", 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "enable_auth": { 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enable_auth", 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.376 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "key_file": { 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "key_file", 2024-09-15T23:32:01.377 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.378 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.379 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.380 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.381 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.382 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "server_addr": { 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_addr", 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.383 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "server_port": { 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_port", 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.384 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.385 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rgw", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.386 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.387 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.388 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.389 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.390 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "secondary_zone_period_retry_limit": { 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.391 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rook", 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.392 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.393 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.394 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.395 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "storage_class": { 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "storage_class", 2024-09-15T23:32:01.397 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "local", 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.398 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "selftest", 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.399 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.400 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.401 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.402 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.403 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "roption1": { 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "roption1", 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.404 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "roption2": { 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "roption2", 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.405 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "xyz", 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption1": { 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption1", 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.406 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption2": { 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption2", 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.407 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption3": { 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption3", 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.408 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption4": { 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption4", 2024-09-15T23:32:01.409 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption5": { 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption5", 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.410 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption6": { 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption6", 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.411 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption7": { 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption7", 2024-09-15T23:32:01.412 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "42", 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "testkey": { 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "testkey", 2024-09-15T23:32:01.413 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "testlkey": { 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "testlkey", 2024-09-15T23:32:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: "testnewline": { 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "testnewline", 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.416 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "snap_schedule", 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "allow_m_granularity": { 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "allow_m_granularity", 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.417 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "allow minute scheduled snapshots", 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "dump_on_update": { 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "dump_on_update", 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.418 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "dump database to debug log on update", 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.419 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.420 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.421 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.422 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.423 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "stats", 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.424 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.425 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.426 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.427 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.428 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.429 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "status", 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.430 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.431 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.432 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.433 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.434 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "telegraf", 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.435 INFO:teuthology.orchestra.run.smithi062.stdout: "address": { 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "address", 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.436 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "15", 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.437 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.438 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.439 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.440 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.441 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.442 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "telemetry", 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_basic": { 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_basic", 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.443 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_crash": { 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_crash", 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.444 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_device": { 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_device", 2024-09-15T23:32:01.445 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_ident": { 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_ident", 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.446 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_perf": { 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_perf", 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.447 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "contact": { 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "contact", 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.448 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "description": { 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "description", 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.449 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "device_url": { 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "device_url", 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.450 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": { 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enabled", 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.451 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.452 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "24", 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "8", 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "last_opt_revision": { 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "last_opt_revision", 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.453 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "1", 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "leaderboard": { 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "leaderboard", 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.454 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "leaderboard_description": { 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "leaderboard_description", 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.455 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.456 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.457 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.458 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.459 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.460 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "organization": { 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "organization", 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.461 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "proxy": { 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "proxy", 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.462 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "url": { 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "url", 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.463 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.464 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "test_orchestrator", 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.465 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.466 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.467 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.468 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.469 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "volumes", 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.470 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.471 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.472 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.473 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.474 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "max_concurrent_clones": { 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_concurrent_clones", 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.475 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "4", 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "periodic_async_work": { 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "periodic_async_work", 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.476 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Periodically check for async work", 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "snapshot_clone_delay": { 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "snapshot_clone_delay", 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.477 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "snapshot_clone_no_wait": { 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "snapshot_clone_no_wait", 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.478 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.479 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix", 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "discovery_interval": { 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "discovery_interval", 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "100", 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.480 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "identifier": { 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "identifier", 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.481 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.482 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.483 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.484 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.485 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.486 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.487 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.488 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.489 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.490 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.490 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.490 INFO:teuthology.orchestra.run.smithi062.stdout: "zabbix_host": { 2024-09-15T23:32:01.490 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix_host", 2024-09-15T23:32:01.490 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.490 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.490 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.490 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.490 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: "zabbix_port": { 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix_port", 2024-09-15T23:32:01.491 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10051", 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.492 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "zabbix_sender": { 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix_sender", 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.493 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: "modules": [ 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm", 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: "dashboard", 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: "iostat", 2024-09-15T23:32:01.494 INFO:teuthology.orchestra.run.smithi062.stdout: "nfs", 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "restful" 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "available_modules": [ 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "alerts", 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.495 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How frequently to reexamine health status", 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.496 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.497 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.498 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.499 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.500 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_destination": { 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_destination", 2024-09-15T23:32:01.501 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Email address to send alerts to", 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_from_name": { 2024-09-15T23:32:01.502 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_from_name", 2024-09-15T23:32:01.504 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.504 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.504 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.504 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "Ceph", 2024-09-15T23:32:01.504 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.504 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.504 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Email From: name", 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_host": { 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_host", 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.505 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SMTP server", 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_password": { 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_password", 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.506 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Password to authenticate with", 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_port": { 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_port", 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "465", 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.507 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SMTP port", 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_sender": { 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_sender", 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.508 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SMTP envelope sender", 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_ssl": { 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_ssl", 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.509 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use SSL to connect to SMTP server", 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "smtp_user": { 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "smtp_user", 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.510 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "User to authenticate as", 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "balancer", 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.511 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "active": { 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "active", 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "automatically balance PGs across cluster", 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.512 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "begin_time": { 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "begin_time", 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0000", 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "beginning time of day to automatically balance", 2024-09-15T23:32:01.513 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "begin_weekday": { 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "begin_weekday", 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "0", 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "6", 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.514 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_compat_max_iterations": { 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crush_compat_max_iterations", 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "25", 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "250", 2024-09-15T23:32:01.515 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_compat_metrics": { 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crush_compat_metrics", 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "pgs,objects,bytes", 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.516 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "crush_compat_step": { 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crush_compat_step", 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0.5", 2024-09-15T23:32:01.517 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "0.001", 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "0.999", 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "aggressiveness of optimization", 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "end_time": { 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "end_time", 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.518 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "2359", 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "ending time of day to automatically balance", 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "end_weekday": { 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "end_weekday", 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.519 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "0", 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "6", 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.520 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.521 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.522 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.523 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.524 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "min_score": { 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "min_score", 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.525 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": { 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mode", 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.526 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "upmap", 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "crush-compat", 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "none", 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "read", 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap", 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap-read" 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Balancer mode", 2024-09-15T23:32:01.527 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "pool_ids": { 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pool_ids", 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.528 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.529 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap_max_deviation": { 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "upmap_max_deviation", 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.530 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "deviation below which no optimization is attempted", 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "upmap_max_optimizations": { 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "upmap_max_optimizations", 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.531 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cephadm", 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.532 INFO:teuthology.orchestra.run.smithi062.stdout: "agent_down_multiplier": { 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "agent_down_multiplier", 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "3.0", 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.533 INFO:teuthology.orchestra.run.smithi062.stdout: "agent_refresh_rate": { 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "agent_refresh_rate", 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "20", 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.534 INFO:teuthology.orchestra.run.smithi062.stdout: "agent_starting_port": { 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "agent_starting_port", 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "4721", 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.535 INFO:teuthology.orchestra.run.smithi062.stdout: "allow_ptrace": { 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "allow_ptrace", 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.536 INFO:teuthology.orchestra.run.smithi062.stdout: "autotune_interval": { 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "autotune_interval", 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to autotune daemon memory", 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.537 INFO:teuthology.orchestra.run.smithi062.stdout: "autotune_memory_target_ratio": { 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "autotune_memory_target_ratio", 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0.7", 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm_log_destination": { 2024-09-15T23:32:01.538 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cephadm_log_destination", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "file", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "file,syslog", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "syslog" 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.539 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "cgroups_split": { 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cgroups_split", 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.540 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "config_checks_enabled": { 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "config_checks_enabled", 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.541 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "config_dashboard": { 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "config_dashboard", 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-09-15T23:32:01.542 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_alertmanager": { 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_alertmanager", 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.543 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_base": { 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_base", 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/ceph", 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Container image name, without the tag", 2024-09-15T23:32:01.544 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_elasticsearch": { 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_elasticsearch", 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "elasticsearch container image", 2024-09-15T23:32:01.545 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_grafana": { 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_grafana", 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.546 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_haproxy": { 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_haproxy", 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.547 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "HAproxy container image", 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_jaeger_agent": { 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_jaeger_agent", 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.548 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Jaeger agent container image", 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_jaeger_collector": { 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_jaeger_collector", 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.549 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Jaeger collector container image", 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_jaeger_query": { 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_jaeger_query", 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.550 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Jaeger query container image", 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_keepalived": { 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_keepalived", 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-09-15T23:32:01.551 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Keepalived container image", 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_loki": { 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_loki", 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.552 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io/grafana/loki:3.0.0", 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Loki container image", 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_node_exporter": { 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_node_exporter", 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.553 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_nvmeof": { 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_nvmeof", 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.554 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.5", 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Nvme-of container image", 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_prometheus": { 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_prometheus", 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-09-15T23:32:01.555 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Prometheus container image", 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_promtail": { 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_promtail", 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.556 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io/grafana/promtail:3.0.0", 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Promtail container image", 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_samba": { 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_samba", 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.557 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Samba/SMB container image", 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "container_image_snmp_gateway": { 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_image_snmp_gateway", 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.558 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "SNMP Gateway container image", 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "container_init": { 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "container_init", 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.559 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Run podman/docker with `--init`", 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "daemon_cache_timeout": { 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "daemon_cache_timeout", 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.560 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "default_cephadm_command_timeout": { 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "default_cephadm_command_timeout", 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "900", 2024-09-15T23:32:01.561 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "default_registry": { 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "default_registry", 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "docker.io", 2024-09-15T23:32:01.562 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "device_cache_timeout": { 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "device_cache_timeout", 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "1800", 2024-09-15T23:32:01.563 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "seconds to cache device inventory", 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "device_enhanced_scan": { 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "device_enhanced_scan", 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.564 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use libstoragemgmt during device scans", 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "facts_cache_timeout": { 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "facts_cache_timeout", 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.565 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "seconds to cache host facts data", 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "grafana_dashboards_path": { 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "grafana_dashboards_path", 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.566 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "host_check_interval": { 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "host_check_interval", 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.567 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to perform a host check", 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "hw_monitoring": { 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "hw_monitoring", 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.568 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "inventory_list_all": { 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "inventory_list_all", 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.569 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.570 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.571 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "log_refresh_metadata": { 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_refresh_metadata", 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.572 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-09-15T23:32:01.573 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.574 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.575 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "manage_etc_ceph_ceph_conf": { 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-09-15T23:32:01.576 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-09-15T23:32:01.577 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "*", 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "max_count_per_host": { 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_count_per_host", 2024-09-15T23:32:01.578 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "max number of daemons per service per host", 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "max_osd_draining_count": { 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_osd_draining_count", 2024-09-15T23:32:01.579 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "migration_current": { 2024-09-15T23:32:01.580 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "migration_current", 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "internal - do not modify", 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "mode": { 2024-09-15T23:32:01.581 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mode", 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "root", 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm-package", 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "root" 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "mode for remote execution of cephadm", 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.582 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "oob_default_addr": { 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "oob_default_addr", 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "169.254.1.1", 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Default address for RedFish API (oob management).", 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.583 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "prometheus_alerts_path": { 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "prometheus_alerts_path", 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.584 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_insecure": { 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_insecure", 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.585 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_password": { 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_password", 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.586 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_url": { 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_url", 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.587 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "registry_username": { 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "registry_username", 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-09-15T23:32:01.588 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "secure_monitoring_stack": { 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "secure_monitoring_stack", 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.589 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "service_discovery_port": { 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "service_discovery_port", 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8765", 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.590 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "cephadm service discovery port", 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "ssh_config_file": { 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssh_config_file", 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.591 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "use_agent": { 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "use_agent", 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.592 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "use_repo_digest": { 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "use_repo_digest", 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.593 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_on_failed_host_check": { 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_on_failed_host_check", 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.594 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise a health warning if the host check fails", 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_on_stray_daemons": { 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_on_stray_daemons", 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.595 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_on_stray_hosts": { 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_on_stray_hosts", 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.596 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crash", 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.597 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.598 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.599 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.600 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.601 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.602 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "retain_interval": { 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "retain_interval", 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "31536000", 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how long to retain crashes before pruning them", 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.603 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_recent_interval": { 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_recent_interval", 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "1209600", 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "time interval in which to warn about recent crashes", 2024-09-15T23:32:01.604 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "dashboard", 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-09-15T23:32:01.605 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "ALERTMANAGER_API_HOST": { 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ALERTMANAGER_API_HOST", 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.606 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.607 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "AUDIT_API_ENABLED": { 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "AUDIT_API_ENABLED", 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.608 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-09-15T23:32:01.609 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "ENABLE_BROWSABLE_API": { 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ENABLE_BROWSABLE_API", 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.610 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.611 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.612 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.613 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.614 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_NFS": { 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.615 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_RBD": { 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.616 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "FEATURE_TOGGLE_RGW": { 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.617 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.618 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_PASSWORD": { 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_PASSWORD", 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.619 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "admin", 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.620 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_URL": { 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_URL", 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.621 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_API_USERNAME": { 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_API_USERNAME", 2024-09-15T23:32:01.622 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "admin", 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-09-15T23:32:01.623 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-09-15T23:32:01.624 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-09-15T23:32:01.625 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "ISSUE_TRACKER_API_KEY": { 2024-09-15T23:32:01.626 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "PROMETHEUS_API_HOST": { 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PROMETHEUS_API_HOST", 2024-09-15T23:32:01.627 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-09-15T23:32:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-09-15T23:32:01.630 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-09-15T23:32:01.631 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-09-15T23:32:01.632 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-09-15T23:32:01.633 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-09-15T23:32:01.634 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-09-15T23:32:01.635 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_ENABLED": { 2024-09-15T23:32:01.636 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_ENABLED", 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.637 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.638 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.639 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8", 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.640 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "REST_REQUESTS_TIMEOUT": { 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "45", 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.641 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_ACCESS_KEY": { 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_ACCESS_KEY", 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.642 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "admin", 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.643 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_SECRET_KEY": { 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_SECRET_KEY", 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.644 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "RGW_API_SSL_VERIFY": { 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "RGW_API_SSL_VERIFY", 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.645 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.646 INFO:teuthology.orchestra.run.smithi062.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.647 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.648 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "cross_origin_url": { 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cross_origin_url", 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.649 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "crt_file": { 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "crt_file", 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.650 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "debug": { 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "debug", 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Enable/disable debug options", 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.651 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "jwt_token_ttl": { 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "jwt_token_ttl", 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "28800", 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.652 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "key_file": { 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "key_file", 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.653 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.654 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.655 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.656 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.657 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.658 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "motd": { 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "motd", 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.659 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "The message of the day", 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "redirect_resolve_ip_addr": { 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "redirect_resolve_ip_addr", 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.660 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "server_addr": { 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_addr", 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "::", 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.661 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "server_port": { 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_port", 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8080", 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.662 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "ssl": { 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssl", 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.663 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "ssl_server_port": { 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssl_server_port", 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8443", 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.664 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_behaviour": { 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_behaviour", 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "redirect", 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.665 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "redirect" 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_error_status_code": { 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_error_status_code", 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "500", 2024-09-15T23:32:01.666 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "400", 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "599", 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "url_prefix": { 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "url_prefix", 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.667 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "devicehealth", 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.668 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "enable_monitoring": { 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enable_monitoring", 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "monitor device health metrics", 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.669 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.670 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.671 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.672 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.673 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.674 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "mark_out_threshold": { 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mark_out_threshold", 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "2419200", 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.675 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "pool_name": { 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pool_name", 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "device_health_metrics", 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "name of pool in which to store device health metrics", 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.676 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "retention_period": { 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "retention_period", 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "15552000", 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how long to retain device health metrics", 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.677 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "scrape_frequency": { 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "scrape_frequency", 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "86400", 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to scrape device health metrics", 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.678 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "self_heal": { 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "self_heal", 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.679 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how frequently to wake up and check device health", 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.680 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "warn_threshold": { 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "warn_threshold", 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "7257600", 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.681 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "diskprediction_local", 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.682 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.683 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.684 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.685 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.686 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.687 INFO:teuthology.orchestra.run.smithi062.stdout: "predict_interval": { 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "predict_interval", 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "86400", 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.688 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "predictor_model": { 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "predictor_model", 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "prophetstor", 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.689 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "600", 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.690 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "influx", 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": false, 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "influxdb python module not found", 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "batch_size": { 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "batch_size", 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.691 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5000", 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "database": { 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "database", 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.692 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "ceph", 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": { 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "hostname", 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.693 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "InfluxDB server hostname", 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.694 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "30", 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "5", 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.695 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.696 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.697 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.698 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.699 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "password": { 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "password", 2024-09-15T23:32:01.700 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "password of InfluxDB server user", 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "port": { 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "port", 2024-09-15T23:32:01.701 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "8086", 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "InfluxDB server port", 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "ssl": { 2024-09-15T23:32:01.702 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ssl", 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "false", 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.703 INFO:teuthology.orchestra.run.smithi062.stdout: "threads": { 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "threads", 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "32", 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.704 INFO:teuthology.orchestra.run.smithi062.stdout: "username": { 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "username", 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "username of InfluxDB server user", 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.705 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "verify_ssl": { 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "verify_ssl", 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "true", 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.706 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "insights", 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.707 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.708 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.709 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.710 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.711 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "iostat", 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.712 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.713 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.714 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.715 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.716 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.717 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "k8sevents", 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_event_retention_days": { 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "ceph_event_retention_days", 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "7", 2024-09-15T23:32:01.718 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Days to hold ceph event information within local cache", 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "config_check_secs": { 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "config_check_secs", 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.719 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "10", 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.720 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.721 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.722 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.723 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "localpool", 2024-09-15T23:32:01.725 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "failure_domain": { 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "failure_domain", 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "host", 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "failure domain for any created local pool", 2024-09-15T23:32:01.726 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.727 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.728 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.729 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.730 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.731 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "min_size": { 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "min_size", 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "default min_size for any created local pool", 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.732 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "num_rep": { 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "num_rep", 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "3", 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "default replica count for any created local pool", 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.733 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "pg_num": { 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pg_num", 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "128", 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "default pg_num for any created local pool", 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.734 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "prefix": { 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "prefix", 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "name prefix for any created local pool", 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.735 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "subtree": { 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "subtree", 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "rack", 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "CRUSH level for which to create a local pool", 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-09-15T23:32:01.736 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mds_autoscaler", 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.737 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.738 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.739 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.740 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.741 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.742 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mirroring", 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.743 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.744 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.745 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.746 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.747 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "nfs", 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.748 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.749 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.750 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.751 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.752 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.753 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "orchestrator", 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "fail_fs": { 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "fail_fs", 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.754 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.755 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.756 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.757 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.758 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.759 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "orchestrator": { 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "orchestrator", 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.760 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: "cephadm", 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: "rook", 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: "test_orchestrator" 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Orchestrator backend", 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.761 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "osd_perf_query", 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.762 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.763 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.764 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.765 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.766 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "osd_support", 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.767 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.768 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.769 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.770 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.771 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.772 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "pg_autoscaler", 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.773 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.774 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.775 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.776 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.777 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.778 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "threshold": { 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "threshold", 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "3.0", 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1.0", 2024-09-15T23:32:01.779 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "scaling threshold", 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "progress", 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.780 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "allow_pg_recovery_event": { 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "allow_pg_recovery_event", 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "allow the module to show pg recovery progress", 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.781 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": { 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enabled", 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.782 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.783 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.784 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.785 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.786 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.787 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "max_completed_events": { 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_completed_events", 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "50", 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "number of past completed events to remember", 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.788 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "sleep_interval": { 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "sleep_interval", 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "how long the module is going to sleep", 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.789 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "prometheus", 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "cache": { 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "cache", 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.790 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "exclude_perf_counters": { 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "exclude_perf_counters", 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.791 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.792 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.793 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.794 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.795 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.796 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_stats_pools": { 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_stats_pools", 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.797 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_stats_pools_refresh_interval": { 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.798 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "300", 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "scrape_interval": { 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "scrape_interval", 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.799 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "15.0", 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "server_addr": { 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_addr", 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.800 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "::", 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "server_port": { 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_port", 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.801 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "9283", 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "stale_cache_strategy": { 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "stale_cache_strategy", 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.802 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "log", 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_behaviour": { 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_behaviour", 2024-09-15T23:32:01.803 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "default", 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "default", 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "error" 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.804 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "standby_error_status_code": { 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "standby_error_status_code", 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "500", 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "400", 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "599", 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.805 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_support", 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.806 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.807 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.808 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.809 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.810 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "max_concurrent_snap_create": { 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_concurrent_snap_create", 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.811 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10", 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "mirror_snapshot_schedule": { 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "mirror_snapshot_schedule", 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.812 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "trash_purge_schedule": { 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "trash_purge_schedule", 2024-09-15T23:32:01.813 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.814 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "restful", 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "enable_auth": { 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enable_auth", 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.815 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "key_file": { 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "key_file", 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.816 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.817 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.818 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.819 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.820 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.821 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "server_addr": { 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_addr", 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.822 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "server_port": { 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "server_port", 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.823 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rgw", 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.824 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.825 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.826 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.827 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.828 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.829 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "secondary_zone_period_retry_limit": { 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "secondary_zone_period_retry_limit", 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "5", 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "RGW module period update retry limit for secondary site", 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.830 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rook", 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.831 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.832 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.833 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.834 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.835 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "storage_class": { 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "storage_class", 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "local", 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.836 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "selftest", 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.837 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.838 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.839 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.840 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.841 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.842 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "roption1": { 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "roption1", 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.843 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "roption2": { 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "roption2", 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "xyz", 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.844 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption1": { 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption1", 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.845 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption2": { 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption2", 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.846 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption3": { 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption3", 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "float", 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.847 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption4": { 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption4", 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.848 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption5": { 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption5", 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.849 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption6": { 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption6", 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.850 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "rwoption7": { 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rwoption7", 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "1", 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "42", 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.851 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "testkey": { 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "testkey", 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.852 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "testlkey": { 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "testlkey", 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.853 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "testnewline": { 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "testnewline", 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.854 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "snap_schedule", 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "allow_m_granularity": { 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "allow_m_granularity", 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.855 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "allow minute scheduled snapshots", 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "dump_on_update": { 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "dump_on_update", 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.856 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "dump database to debug log on update", 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.857 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.858 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.859 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.860 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.861 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.862 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "stats", 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.863 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.864 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.865 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.866 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.867 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "status", 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.868 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.869 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.870 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.871 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.872 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.873 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "telegraf", 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: "address": { 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "address", 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.874 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.875 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "15", 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.876 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.877 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.878 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.879 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.880 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.881 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "telemetry", 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_basic": { 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_basic", 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.882 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share basic cluster information (size, version)", 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_crash": { 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_crash", 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.883 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_device": { 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_device", 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.884 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_ident": { 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_ident", 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.885 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "channel_perf": { 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "channel_perf", 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.886 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Share various performance metrics of a cluster", 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "contact": { 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "contact", 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.887 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "description": { 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "description", 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.888 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "device_url": { 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "device_url", 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.889 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": { 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "enabled", 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.890 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "24", 2024-09-15T23:32:01.891 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "8", 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "last_opt_revision": { 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "last_opt_revision", 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.892 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "1", 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "leaderboard": { 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "leaderboard", 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.893 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "leaderboard_description": { 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "leaderboard_description", 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.894 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.895 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.896 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.897 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.898 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.899 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "organization": { 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "organization", 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.900 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "proxy": { 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "proxy", 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.901 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "url": { 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "url", 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "test_orchestrator", 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.904 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.905 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.906 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.907 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.908 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "volumes", 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.909 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.910 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.911 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.912 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.913 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.914 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "max_concurrent_clones": { 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "max_concurrent_clones", 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "4", 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Number of asynchronous cloner threads", 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.915 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "periodic_async_work": { 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "periodic_async_work", 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Periodically check for async work", 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.916 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "snapshot_clone_delay": { 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "snapshot_clone_delay", 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "0", 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.917 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "snapshot_clone_no_wait": { 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "snapshot_clone_no_wait", 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "True", 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.918 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix", 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "can_run": true, 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "error_string": "", 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "module_options": { 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "discovery_interval": { 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "discovery_interval", 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "uint", 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.919 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "100", 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "identifier": { 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "identifier", 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.920 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "interval": { 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "interval", 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "secs", 2024-09-15T23:32:01.921 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "60", 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "log_level": { 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_level", 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.922 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.923 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster": { 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster", 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.924 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_cluster_level": { 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_cluster_level", 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "info", 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [ 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "", 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "critical", 2024-09-15T23:32:01.925 INFO:teuthology.orchestra.run.smithi062.stdout: "debug", 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "error", 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "info", 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "warning" 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "log_to_file": { 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "log_to_file", 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "bool", 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.926 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 1, 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "False", 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "zabbix_host": { 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix_host", 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "", 2024-09-15T23:32:01.927 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "zabbix_port": { 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix_port", 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "int", 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.928 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "10051", 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "zabbix_sender": { 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "zabbix_sender", 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "str", 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "level": "advanced", 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "flags": 0, 2024-09-15T23:32:01.929 INFO:teuthology.orchestra.run.smithi062.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "min": "", 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "max": "", 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "enum_allowed": [], 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "desc": "", 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "long_desc": "", 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "see_also": [] 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "services": { 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "dashboard": "https://172.21.15.62:8443/" 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.930 INFO:teuthology.orchestra.run.smithi062.stdout: "always_on_modules": { 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "octopus": [ 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "balancer", 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "devicehealth", 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "orchestrator", 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "pg_autoscaler", 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "progress", 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_support", 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "status", 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "telemetry", 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "volumes" 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "pacific": [ 2024-09-15T23:32:01.931 INFO:teuthology.orchestra.run.smithi062.stdout: "balancer", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "devicehealth", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "orchestrator", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "pg_autoscaler", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "progress", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_support", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "status", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "telemetry", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "volumes" 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "quincy": [ 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "balancer", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-09-15T23:32:01.932 INFO:teuthology.orchestra.run.smithi062.stdout: "devicehealth", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "orchestrator", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "pg_autoscaler", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "progress", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_support", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "status", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "telemetry", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "volumes" 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "reef": [ 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "balancer", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "devicehealth", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "orchestrator", 2024-09-15T23:32:01.933 INFO:teuthology.orchestra.run.smithi062.stdout: "pg_autoscaler", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "progress", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_support", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "status", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "telemetry", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "volumes" 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "squid": [ 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "balancer", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "crash", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "devicehealth", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "orchestrator", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "pg_autoscaler", 2024-09-15T23:32:01.934 INFO:teuthology.orchestra.run.smithi062.stdout: "progress", 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "rbd_support", 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "status", 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "telemetry", 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "volumes" 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "last_failure_osd_epoch": 3, 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "active_clients": [ 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "devicehealth", 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-09-15T23:32:01.935 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:0", 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 467758453 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "libcephsqlite", 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:0", 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1420105809 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:32:01.936 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "rbd_support", 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:0", 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1108354 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "volumes", 2024-09-15T23:32:01.937 INFO:teuthology.orchestra.run.smithi062.stdout: "addrvec": [ 2024-09-15T23:32:01.938 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-09-15T23:32:01.938 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v2", 2024-09-15T23:32:01.938 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "172.21.15.62:0", 2024-09-15T23:32:01.938 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 2629754523 2024-09-15T23:32:01.938 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.938 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:32:01.938 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:01.938 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-09-15T23:32:01.938 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:32:01.939 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:00 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3129760564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:32:01.940 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:00 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2255852519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:32:01.940 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:00 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/296679196' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-15T23:32:01.940 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:00 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3129760564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:32:01.940 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:00 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2255852519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:32:01.940 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:00 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/296679196' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b500a32d0 msgr2=0x7f5b500a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b500a32d0 0x7f5b500a5790 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5b54000c00 tx=0x7f5b540095f0 comp rx=0 tx=0).stop 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64100a30 msgr2=0x7f5b64197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64100a30 0x7f5b64197e40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f5b6000c970 tx=0x7f5b6000ce30 comp rx=0 tx=0).stop 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 shutdown_connections 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f5b500a32d0 0x7f5b500a5790 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f5b641025e0 0x7f5b6419f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b64101c20 0x7f5b64198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 --2- 172.21.15.62:0/296679196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b64100a30 0x7f5b64197e40 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 >> 172.21.15.62:0/296679196 conn(0x7f5b640fc1b0 msgr2=0x7f5b64105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:01.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 shutdown_connections 2024-09-15T23:32:01.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:00.533+0000 7f5b6c617640 1 -- 172.21.15.62:0/296679196 wait complete. 2024-09-15T23:32:01.997 DEBUG:tasks.check_counter:Getting stats from a 2024-09-15T23:32:01.997 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.a perf dump 2024-09-15T23:32:01.997 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:00 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3129760564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-09-15T23:32:01.998 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:00 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2255852519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-09-15T23:32:01.998 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:00 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/296679196' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-09-15T23:32:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:01 smithi067 ceph-mon[35665]: pgmap v842: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-09-15T23:32:02.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:01 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:32:02.025 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:01 smithi062 ceph-mon[30949]: pgmap v842: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-09-15T23:32:02.025 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:01 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:32:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:01 smithi148 ceph-mon[35610]: pgmap v842: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 7.5 KiB/s wr, 8 op/s 2024-09-15T23:32:02.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:01 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:32:02.270 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:02.746 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:32:02.746 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:02.746 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 0, 2024-09-15T23:32:02.746 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 2, 2024-09-15T23:32:02.746 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 0, 2024-09-15T23:32:02.746 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 0, 2024-09-15T23:32:02.746 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 0, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.001389585, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.000000000, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.000270469, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:02.747 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 307, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 302, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 507072, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 185328, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 2, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 1, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.047719188, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.020210449, 2024-09-15T23:32:02.748 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.028748431, 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 302, 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.032709386, 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000108309 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 507072, 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 185328 2024-09-15T23:32:02.749 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 2, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 229, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 320, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 197616, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.019434890, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.014417586, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.002953367, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 227, 2024-09-15T23:32:02.750 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.016241276, 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000071547 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 197616 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "cct": { 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "total_workers": 0, 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: "unhealthy_workers": 0 2024-09-15T23:32:02.751 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "mempool": { 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:02.752 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:02.753 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_items": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_bytes": 148704, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_items": 29, 2024-09-15T23:32:02.754 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bytes": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_items": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_bytes": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_items": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_items": 0, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_bytes": 928, 2024-09-15T23:32:02.755 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_items": 2, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_items": 0, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_items": 0 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 305, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 461242, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 305, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-09-15T23:32:02.756 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-09-15T23:32:02.757 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 305, 2024-09-15T23:32:02.757 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 461242, 2024-09-15T23:32:02.757 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-09-15T23:32:02.757 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:02.757 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:02.757 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:02.757 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:02.757 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:02.757 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:32:03.536 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-09-15T23:32:03.536 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-09-15T23:32:03.536 DEBUG:tasks.check_counter:Getting stats from d 2024-09-15T23:32:03.536 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.d perf dump 2024-09-15T23:32:03.837 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:04.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:03 smithi148 ceph-mon[35610]: pgmap v843: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2024-09-15T23:32:04.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:03 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:32:04.058 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:03 smithi062 ceph-mon[30949]: pgmap v843: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2024-09-15T23:32:04.058 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:03 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:32:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:03 smithi067 ceph-mon[35665]: pgmap v843: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2024-09-15T23:32:04.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:03 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:32:04.297 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:32:04.297 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 309, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 306, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 505712, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 188720, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 0, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.050545553, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.022427176, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.031688715, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 305, 2024-09-15T23:32:04.298 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.030450601, 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000099838 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 505712, 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 188720 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 2, 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 233, 2024-09-15T23:32:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 320, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 198320, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.020374422, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.013823271, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.001790388, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 231, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.022715460, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000098335 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 198320 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 4, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 4, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 2992, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 448, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 2, 2024-09-15T23:32:04.301 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 1, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.002356238, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.000041720, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.002302838, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 3, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.004231233, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.001410411 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:04.302 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "cct": { 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "total_workers": 0, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "unhealthy_workers": 0 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "mempool": { 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:04.303 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:04.304 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_items": 0, 2024-09-15T23:32:04.305 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_bytes": 148682, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_items": 29, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bytes": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_items": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:04.306 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_bytes": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_items": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_items": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_bytes": 928, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_items": 2, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_items": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_items": 0 2024-09-15T23:32:04.307 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 311, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 462022, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 311, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 311, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 462022, 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-09-15T23:32:04.308 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:04.309 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:04.309 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:04.309 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:04.309 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:04.309 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:32:05.097 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-09-15T23:32:05.098 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-09-15T23:32:05.098 DEBUG:tasks.check_counter:Getting stats from g 2024-09-15T23:32:05.098 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.g perf dump 2024-09-15T23:32:05.355 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:05.914 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:32:05.914 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:05.914 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 0, 2024-09-15T23:32:05.914 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 2, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 0, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 0, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 0, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.001474287, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.000000000, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.000261077, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 6, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 232, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 3312, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 198240, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:05.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.022689200, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.015856997, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.004312619, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 231, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.017744340, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000076815 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:05.917 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 198240 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 305, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 302, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 504624, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 186160, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 2, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 1, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.049460471, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.022002507, 2024-09-15T23:32:05.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.031512758, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 301, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.020738456, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000068898 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 504624, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 186160 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.919 INFO:teuthology.orchestra.run.smithi062.stdout: "cct": { 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: "total_workers": 0, 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: "unhealthy_workers": 0 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: "mempool": { 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:05.920 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:05.921 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:05.922 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_items": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_bytes": 148704, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_items": 29, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bytes": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_items": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:05.923 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_bytes": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_items": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_items": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_bytes": 928, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_items": 2, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_items": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:05.924 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_items": 0 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 307, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 461502, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 307, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 307, 2024-09-15T23:32:05.925 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 461502, 2024-09-15T23:32:05.926 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-09-15T23:32:05.926 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:05.926 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:05.926 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:05.926 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:05.926 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:05.926 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:32:06.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:05 smithi148 ceph-mon[35610]: pgmap v844: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.4 KiB/s wr, 8 op/s 2024-09-15T23:32:06.229 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:05 smithi062 ceph-mon[30949]: pgmap v844: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.4 KiB/s wr, 8 op/s 2024-09-15T23:32:06.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:05 smithi067 ceph-mon[35665]: pgmap v844: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.4 KiB/s wr, 8 op/s 2024-09-15T23:32:06.721 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-09-15T23:32:06.721 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-09-15T23:32:06.721 DEBUG:tasks.check_counter:Getting stats from j 2024-09-15T23:32:06.722 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.j perf dump 2024-09-15T23:32:06.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:06 smithi062 ceph-mon[30949]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:32:06.982 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:07.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:06 smithi067 ceph-mon[35665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:32:07.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:06 smithi148 ceph-mon[35610]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-09-15T23:32:07.449 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 0, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 2, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 0, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 0, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 0, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.001289476, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.000000000, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.000307912, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:07.450 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:07.451 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 310, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 307, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 505984, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 189360, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 2, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 1, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.046895578, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.021446733, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.029310722, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 306, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.020123504, 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000065763 2024-09-15T23:32:07.452 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 505984, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 189360 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 6, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 236, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 3312, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 198944, 2024-09-15T23:32:07.453 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.024218596, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.017905716, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.004049760, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 235, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.015765295, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000067086 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:07.454 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 198944 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "cct": { 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "total_workers": 0, 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "unhealthy_workers": 0 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "mempool": { 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:07.455 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:07.456 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:07.457 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_items": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_bytes": 148721, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_items": 30, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bytes": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_items": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:07.458 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_bytes": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_items": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_items": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_bytes": 928, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_items": 2, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_items": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:07.459 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_items": 0 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 312, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 462152, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 312, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 312, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 462152, 2024-09-15T23:32:07.460 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-09-15T23:32:07.461 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-09-15T23:32:07.461 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-09-15T23:32:07.461 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-09-15T23:32:07.461 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:07.461 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-09-15T23:32:07.461 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-09-15T23:32:08.127 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:07 smithi062 ceph-mon[30949]: pgmap v845: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.9 KiB/s wr, 8 op/s 2024-09-15T23:32:08.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:07 smithi067 ceph-mon[35665]: pgmap v845: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.9 KiB/s wr, 8 op/s 2024-09-15T23:32:08.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:07 smithi148 ceph-mon[35610]: pgmap v845: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.9 KiB/s wr, 8 op/s 2024-09-15T23:32:08.420 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-09-15T23:32:08.702 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-09-15T23:32:08.702 DEBUG:tasks.check_counter:Getting stats from b 2024-09-15T23:32:08.703 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.b perf dump 2024-09-15T23:32:09.022 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:32:09.627 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 6450, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 7035, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 2774917, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 12137887, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 372, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 6, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.621400218, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.203860929, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.332799932, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.065791379, 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:09.628 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 7037, 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.945230711, 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000134322 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 5488, 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.001153656, 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000210 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 624016, 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 219616 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.629 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 5886, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 6802, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 1941466, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 7987887, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 10, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 5, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.426494690, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.153168941, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.202973751, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.078701262, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6803, 2024-09-15T23:32:09.630 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.116307910, 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000164090 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6413, 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.001388704, 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000216 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 494320 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 12186, 2024-09-15T23:32:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 14521, 2024-09-15T23:32:09.632 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 4639415, 2024-09-15T23:32:09.632 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 14460878, 2024-09-15T23:32:09.632 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 18, 2024-09-15T23:32:09.632 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 8, 2024-09-15T23:32:09.632 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.854842773, 2024-09-15T23:32:09.632 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.320817155, 2024-09-15T23:32:09.632 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.503595062, 2024-09-15T23:32:09.632 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.160681892, 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 14514, 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.143643221, 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000078795 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 16095, 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.002305364, 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000143 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 2989776, 2024-09-15T23:32:09.633 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 5761648 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: "cct": { 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: "total_workers": 1, 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: "unhealthy_workers": 0 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-MDSRank": { 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 7940, 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 4.307227199, 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000542471 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:09.634 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-PurgeQueue": { 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 127, 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.005970298, 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000047010 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "request": 3223, 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "reply": 3125, 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "reply_latency": { 2024-09-15T23:32:09.635 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3125, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 6.516885116, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002085403 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_reply": 0, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "forward": 98, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_complete": 14, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_keys": 5, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_commit": 737, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_split": 3, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_merge": 0, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes": 166, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_top": 0, 2024-09-15T23:32:09.636 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_bottom": 6, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pin_tail": 160, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pinned": 166, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_expired": 1106, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_with_caps": 90, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": 90, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "subtrees": 11, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse": 5378, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_hit": 4150, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_forward": 97, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_discover": 4, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_dir_fetch": 2, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_remote_ino": 0, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_lock": 274, 2024-09-15T23:32:09.637 INFO:teuthology.orchestra.run.smithi067.stdout: "load_cent": 1543, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "q": 0, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "exported": 36, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "exported_inodes": 1064, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "imported": 6, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "imported_inodes": 83, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_dir_fetch": 0, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_backtrace_fetch": 3009, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_peer_discover": 0, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rfiles": 1020, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rbytes": 1569920835, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rsnaps": 0, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_fetch": 3009, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_set_tag": 0, 2024-09-15T23:32:09.638 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_inodes": 558, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_base_inodes": 20, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dirfrag_rstats": 538, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_file_inodes": 2451, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_inode_file_caps": 32, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_revoke": 1797, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_grant": 2314, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:32:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:32:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:32:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps": 3349, 2024-09-15T23:32:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps_dirty": 1405, 2024-09-15T23:32:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_cap_release": 64, 2024-09-15T23:32:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: "process_request_cap_release": 589 2024-09-15T23:32:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache": { 2024-09-15T23:32:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update": 0, 2024-09-15T23:32:09.640 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update_receipt": 0, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_try_discover": 0, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_send_discover": 4, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_handle_discover": 38, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays": 0, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_delayed": 0, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_created": 196, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_enqueued": 40, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_reintegrated": 156, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_migrated": 0, 2024-09-15T23:32:09.641 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_processing": 0, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_started": 0, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_completed": 0, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_quiesce_path": 15, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_quiesce_inode": 2567, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_enqueue_scrub": 20, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_exportdir": 36, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_flush": 0, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragmentdir": 3, 2024-09-15T23:32:09.642 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragstats": 0, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_inodestats": 0 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log": { 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "evlrg": 0, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "evadd": 5182, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "evex": 4990, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "evtrm": 4990, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "ev": 192, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "evexg": 0, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "evexd": 15, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "segadd": 78, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "segex": 77, 2024-09-15T23:32:09.643 INFO:teuthology.orchestra.run.smithi067.stdout: "segtrm": 77, 2024-09-15T23:32:09.644 INFO:teuthology.orchestra.run.smithi067.stdout: "seg": 1, 2024-09-15T23:32:09.644 INFO:teuthology.orchestra.run.smithi067.stdout: "segmjr": 1, 2024-09-15T23:32:09.644 INFO:teuthology.orchestra.run.smithi067.stdout: "segexg": 0, 2024-09-15T23:32:09.644 INFO:teuthology.orchestra.run.smithi067.stdout: "segexd": 1, 2024-09-15T23:32:09.644 INFO:teuthology.orchestra.run.smithi067.stdout: "expos": 17565808, 2024-09-15T23:32:09.644 INFO:teuthology.orchestra.run.smithi067.stdout: "wrpos": 18133047, 2024-09-15T23:32:09.644 INFO:teuthology.orchestra.run.smithi067.stdout: "rdpos": 0, 2024-09-15T23:32:09.644 INFO:teuthology.orchestra.run.smithi067.stdout: "jlat": { 2024-09-15T23:32:09.645 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3195, 2024-09-15T23:32:09.645 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 69.103598330, 2024-09-15T23:32:09.645 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.021628669 2024-09-15T23:32:09.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.645 INFO:teuthology.orchestra.run.smithi067.stdout: "replayed": 0 2024-09-15T23:32:09.645 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.645 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mem": { 2024-09-15T23:32:09.645 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 106, 2024-09-15T23:32:09.645 INFO:teuthology.orchestra.run.smithi067.stdout: "ino+": 1222, 2024-09-15T23:32:09.646 INFO:teuthology.orchestra.run.smithi067.stdout: "ino-": 1116, 2024-09-15T23:32:09.646 INFO:teuthology.orchestra.run.smithi067.stdout: "dir": 41, 2024-09-15T23:32:09.646 INFO:teuthology.orchestra.run.smithi067.stdout: "dir+": 81, 2024-09-15T23:32:09.646 INFO:teuthology.orchestra.run.smithi067.stdout: "dir-": 40, 2024-09-15T23:32:09.646 INFO:teuthology.orchestra.run.smithi067.stdout: "dn": 166, 2024-09-15T23:32:09.646 INFO:teuthology.orchestra.run.smithi067.stdout: "dn+": 1808, 2024-09-15T23:32:09.646 INFO:teuthology.orchestra.run.smithi067.stdout: "dn-": 1642, 2024-09-15T23:32:09.646 INFO:teuthology.orchestra.run.smithi067.stdout: "cap": 90, 2024-09-15T23:32:09.646 INFO:teuthology.orchestra.run.smithi067.stdout: "cap+": 1429, 2024-09-15T23:32:09.647 INFO:teuthology.orchestra.run.smithi067.stdout: "cap-": 1339, 2024-09-15T23:32:09.647 INFO:teuthology.orchestra.run.smithi067.stdout: "rss": 73812, 2024-09-15T23:32:09.647 INFO:teuthology.orchestra.run.smithi067.stdout: "heap": 231712 2024-09-15T23:32:09.647 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.647 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_server": { 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_client_request": 3897, 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_server_request": 398, 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_request": 3223, 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_session": 349, 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_peer_request": 796, 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "req_create_latency": { 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1071, 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 3.142111217, 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002933810 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getattr_latency": { 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 35, 2024-09-15T23:32:09.648 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.135994574, 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.003885559 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getfilelock_latency": { 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: "req_link_latency": { 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 156, 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.310039435, 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.001987432 2024-09-15T23:32:09.649 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookup_latency": { 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 891, 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.788671376, 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000885153 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookuphash_latency": { 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupino_latency": { 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000178116, 2024-09-15T23:32:09.650 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000178116 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupname_latency": { 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupparent_latency": { 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupsnap_latency": { 2024-09-15T23:32:09.651 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lssnap_latency": { 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mkdir_latency": { 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 37, 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.480721248, 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.012992466 2024-09-15T23:32:09.652 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mknod_latency": { 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mksnap_latency": { 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "req_open_latency": { 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 507, 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.152859320, 2024-09-15T23:32:09.653 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000301497 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "req_readdir_latency": { 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 51, 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.107922525, 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002116127 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rename_latency": { 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 196, 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.078237644, 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.005501212 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "req_renamesnap_latency": { 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.654 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.655 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.655 INFO:teuthology.orchestra.run.smithi067.stdout: "req_snapdiff_latency": { 2024-09-15T23:32:09.655 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.655 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.655 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.655 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmdir_latency": { 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2, 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.010286014, 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.005143007 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmsnap_latency": { 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmxattr_latency": { 2024-09-15T23:32:09.656 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setattr_latency": { 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2, 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.001239065, 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000619532 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setdirlayout_latency": { 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setfilelock_latency": { 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setlayout_latency": { 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setxattr_latency": { 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8, 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.020032537, 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002504067 2024-09-15T23:32:09.658 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "req_symlink_latency": { 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "req_unlink_latency": { 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 168, 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.288592045, 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.001717809 2024-09-15T23:32:09.659 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getvxattr_latency": { 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessions": { 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "session_count": 1, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "session_add": 3, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "session_remove": 2, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_open": 1, 2024-09-15T23:32:09.660 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_stale": 0, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "total_load": 492, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "average_load": 492, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "avg_session_uptime": 984, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "mdthresh_evicted": 0 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool": { 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_bytes": 240, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_items": 240, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:09.661 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:09.662 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:09.663 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_items": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_bytes": 289935, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_items": 185, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bytes": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_items": 0, 2024-09-15T23:32:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_bytes": 8768, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_items": 165, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_items": 0, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_bytes": 482686, 2024-09-15T23:32:09.665 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_items": 13601, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_items": 0, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_items": 0 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter": { 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "op_active": 0, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "op_laggy": 0, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send": 10428, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send_bytes": 17243038, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "op_resend": 32, 2024-09-15T23:32:09.669 INFO:teuthology.orchestra.run.smithi067.stdout: "op_reply": 10428, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "op_latency": { 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 10428, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 423.798203628, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.040640410 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "op_inflight": 0, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "oplen_avg": { 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 10428, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 15474 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "op": 10428, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "op_r": 3088, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "op_w": 7340, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "op_rmw": 0, 2024-09-15T23:32:09.670 INFO:teuthology.orchestra.run.smithi067.stdout: "op_pg": 0, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_stat": 665, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_create": 2216, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_read": 31, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_write": 3206, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writefull": 262, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writesame": 0, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_append": 0, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_zero": 0, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_truncate": 3, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_delete": 610, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_mapext": 0, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_sparse_read": 0, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_clonerange": 0, 2024-09-15T23:32:09.671 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_getxattr": 3047, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_setxattr": 3391, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_rmxattr": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_call": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_watch": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_notify": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_other": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_active": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_send": 0, 2024-09-15T23:32:09.672 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_resend": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_ping": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_active": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_send": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_resend": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_active": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_send": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_resend": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_active": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_send": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_resend": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "command_active": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "command_send": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "command_resend": 0, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "map_epoch": 92, 2024-09-15T23:32:09.673 INFO:teuthology.orchestra.run.smithi067.stdout: "map_full": 0, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "map_inc": 10, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_sessions": 12, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_open": 12, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_close": 0, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_laggy": 0, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_wr": 1868, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_rd": 47, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_del": 128 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "oft": { 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_objs": 1, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_updates": 0, 2024-09-15T23:32:09.674 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_removes": 0 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "osd-slow-ops": { 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_ops_count": 2388 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "purge_queue": { 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops": 0, 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops_high_water": 24, 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing": 0, 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_high_water": 8, 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed_ops": 92, 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed": 40, 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_item_in_journal": 0 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:09.675 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 24518, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 6733153, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 24518, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 24518, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 6733153, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.676 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_bytes": { 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 10428, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 17246870, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 10428, 2024-09-15T23:32:09.677 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 17246870, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_ops": { 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 1024, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:09.678 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 10428, 2024-09-15T23:32:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 10428, 2024-09-15T23:32:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 10428, 2024-09-15T23:32:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 10428, 2024-09-15T23:32:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.679 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 40, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 4344, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 40, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:09.680 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 11, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 4344, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle-0x55598f102960": { 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 5182, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 13938743, 2024-09-15T23:32:09.681 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 5182, 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 3195, 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 13938743, 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:09.682 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T23:32:10.177 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:09 smithi067 ceph-mon[35665]: pgmap v846: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-09-15T23:32:10.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:09 smithi062 ceph-mon[30949]: pgmap v846: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-09-15T23:32:10.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:09 smithi148 ceph-mon[35610]: pgmap v846: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 10 op/s 2024-09-15T23:32:10.597 INFO:tasks.check_counter:Daemon mds.b mds.exported=36 2024-09-15T23:32:10.597 INFO:tasks.check_counter:Daemon mds.b mds.imported=6 2024-09-15T23:32:10.597 DEBUG:tasks.check_counter:Getting stats from e 2024-09-15T23:32:10.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.e perf dump 2024-09-15T23:32:10.864 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:32:11.513 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 3331, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 3333, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 3308018, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 1059265, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 6, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.255139865, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.071253340, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.067569862, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.099319335, 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:11.514 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3331, 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.137067047, 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000041148 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:11.515 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 3, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 235, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 716, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 478522, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 3, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.012516263, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.009138902, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.001328364, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.000057295, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 233, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.018801466, 2024-09-15T23:32:11.516 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000080692 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 477824 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 341, 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 319, 2024-09-15T23:32:11.517 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 599196, 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 191662, 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.042821215, 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.018689574, 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.029513819, 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.000018470, 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 319, 2024-09-15T23:32:11.518 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.019551193, 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000061289 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 598928, 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 191344 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.519 INFO:teuthology.orchestra.run.smithi067.stdout: "cct": { 2024-09-15T23:32:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "total_workers": 1, 2024-09-15T23:32:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "unhealthy_workers": 0 2024-09-15T23:32:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-MDSRank": { 2024-09-15T23:32:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-09-15T23:32:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-09-15T23:32:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 5611, 2024-09-15T23:32:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.133186269, 2024-09-15T23:32:11.520 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000023736 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-PurgeQueue": { 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:11.521 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "request": 0, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "reply": 0, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "reply_latency": { 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_reply": 0, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "forward": 0, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_complete": 0, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_keys": 0, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_commit": 0, 2024-09-15T23:32:11.522 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_split": 0, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_merge": 0, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes": 523, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_top": 296, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_bottom": 227, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pin_tail": 0, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pinned": 99, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_expired": 0, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_with_caps": 0, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": 0, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "subtrees": 21, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse": 0, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_hit": 0, 2024-09-15T23:32:11.523 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_forward": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_discover": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_remote_ino": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_lock": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "load_cent": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "q": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "exported": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "exported_inodes": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "imported": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "imported_inodes": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_dir_fetch": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_peer_discover": 0, 2024-09-15T23:32:11.524 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rfiles": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rbytes": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rsnaps": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_set_tag": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_file_inodes": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:32:11.525 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_cap_release": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "process_request_cap_release": 0 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache": { 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update_receipt": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_try_discover": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_send_discover": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_handle_discover": 0, 2024-09-15T23:32:11.526 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_delayed": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_created": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_enqueued": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_reintegrated": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_migrated": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_processing": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_started": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_completed": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_quiesce_path": 0, 2024-09-15T23:32:11.527 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_quiesce_inode": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_exportdir": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_flush": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragstats": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_inodestats": 0 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log": { 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "evlrg": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "evadd": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "evex": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "evtrm": 0, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "ev": 1000, 2024-09-15T23:32:11.528 INFO:teuthology.orchestra.run.smithi067.stdout: "evexg": 0, 2024-09-15T23:32:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "evexd": 0, 2024-09-15T23:32:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "segadd": 0, 2024-09-15T23:32:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "segex": 0, 2024-09-15T23:32:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "segtrm": 0, 2024-09-15T23:32:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "seg": 2, 2024-09-15T23:32:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "segmjr": 2, 2024-09-15T23:32:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "segexg": 0, 2024-09-15T23:32:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "segexd": 0, 2024-09-15T23:32:11.529 INFO:teuthology.orchestra.run.smithi067.stdout: "expos": 4194304, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "wrpos": 6568641, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "rdpos": 6568067, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "jlat": { 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "replayed": 1000 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mem": { 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 484, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "ino+": 494, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "ino-": 10, 2024-09-15T23:32:11.530 INFO:teuthology.orchestra.run.smithi067.stdout: "dir": 46, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "dir+": 47, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "dir-": 1, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "dn": 523, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "dn+": 530, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "dn-": 7, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "cap": 0, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "cap+": 0, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "cap-": 0, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "rss": 52228, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "heap": 215328 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_server": { 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_client_request": 0, 2024-09-15T23:32:11.531 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_server_request": 0, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_request": 0, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_session": 0, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_peer_request": 0, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "req_create_latency": { 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getattr_latency": { 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.532 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getfilelock_latency": { 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "req_link_latency": { 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookup_latency": { 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.533 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookuphash_latency": { 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupino_latency": { 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupname_latency": { 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.534 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupparent_latency": { 2024-09-15T23:32:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupsnap_latency": { 2024-09-15T23:32:11.535 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lssnap_latency": { 2024-09-15T23:32:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.536 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mkdir_latency": { 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mknod_latency": { 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mksnap_latency": { 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.537 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "req_open_latency": { 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "req_readdir_latency": { 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.538 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rename_latency": { 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "req_renamesnap_latency": { 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "req_snapdiff_latency": { 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.539 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmdir_latency": { 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmsnap_latency": { 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmxattr_latency": { 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.540 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setattr_latency": { 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setdirlayout_latency": { 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setfilelock_latency": { 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.541 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setlayout_latency": { 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setxattr_latency": { 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "req_symlink_latency": { 2024-09-15T23:32:11.542 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "req_unlink_latency": { 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getvxattr_latency": { 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.543 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessions": { 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "session_count": 1, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "session_add": 2, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "session_remove": 1, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_open": 1, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_stale": 0, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "total_load": 0, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "average_load": 0, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "avg_session_uptime": 2348, 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: "mdthresh_evicted": 0 2024-09-15T23:32:11.544 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool": { 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:11.545 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:11.546 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_items": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_bytes": 199566, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_items": 521, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:11.547 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bytes": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_items": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_bytes": 8768, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_items": 165, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_items": 0, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_bytes": 1541956, 2024-09-15T23:32:11.548 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_items": 12258, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_items": 0, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_items": 0 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter": { 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "op_active": 0, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "op_laggy": 0, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send": 3334, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send_bytes": 21, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "op_resend": 3, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "op_reply": 3334, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "op_latency": { 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3334, 2024-09-15T23:32:11.549 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 20.529582538, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.006157643 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "op_inflight": 0, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "oplen_avg": { 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3334, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 3335 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "op": 3334, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "op_r": 3334, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "op_w": 0, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "op_rmw": 0, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "op_pg": 0, 2024-09-15T23:32:11.550 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_stat": 2052, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_create": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_read": 1280, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_write": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writefull": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writesame": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_append": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_zero": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_truncate": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_delete": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_mapext": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_sparse_read": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_clonerange": 0, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_getxattr": 1, 2024-09-15T23:32:11.551 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_setxattr": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_rmxattr": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_call": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_watch": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_notify": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_other": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_active": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_send": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_resend": 0, 2024-09-15T23:32:11.552 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_ping": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_active": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_send": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_resend": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_active": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_send": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_resend": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_active": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_send": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_resend": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "command_active": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "command_send": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "command_resend": 0, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "map_epoch": 92, 2024-09-15T23:32:11.553 INFO:teuthology.orchestra.run.smithi067.stdout: "map_full": 0, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "map_inc": 4, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_sessions": 4, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_open": 5, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_close": 1, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_laggy": 0, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_wr": 0, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_rd": 2, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_del": 0 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "oft": { 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_objs": 0, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_updates": 0, 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_removes": 0 2024-09-15T23:32:11.554 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "osd-slow-ops": { 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_ops_count": 0 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "purge_queue": { 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops": 0, 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing": 0, 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_high_water": 0, 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed_ops": 0, 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed": 0, 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_item_in_journal": 0 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:11.555 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 3671, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 3557342, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 3671, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 3671, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 3557342, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.556 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_bytes": { 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 3334, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 2374342, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 3334, 2024-09-15T23:32:11.557 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 2374342, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_ops": { 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 1024, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:11.558 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 3334, 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 3334, 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 3334, 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 3334, 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:32:11.559 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 0, 2024-09-15T23:32:11.560 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 0, 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle-0x559415abeb40": { 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-09-15T23:32:11.561 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 0, 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 0, 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:11.562 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T23:32:12.127 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:11 smithi067 ceph-mon[35665]: pgmap v847: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-09-15T23:32:12.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:11 smithi062 ceph-mon[30949]: pgmap v847: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-09-15T23:32:12.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:11 smithi148 ceph-mon[35610]: pgmap v847: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-09-15T23:32:12.447 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-09-15T23:32:12.448 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-09-15T23:32:12.448 DEBUG:tasks.check_counter:Getting stats from h 2024-09-15T23:32:12.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.h perf dump 2024-09-15T23:32:12.705 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:32:13.200 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T23:32:13.200 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:13.200 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 6552, 2024-09-15T23:32:13.200 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 5926, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 3472790, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 4974285, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 46, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 5, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.481424168, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.195727555, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.230566852, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.071914914, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 5925, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.310714040, 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000052441 2024-09-15T23:32:13.201 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 5216, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.001163898, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000223 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 612656, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 215744 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 5742, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 7012, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 2310634, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 4155354, 2024-09-15T23:32:13.202 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 16, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 6, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.358781199, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.154073605, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.219454971, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.033261315, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 7008, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.317257587, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000045270 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 7044, 2024-09-15T23:32:13.203 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.001482856, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000210 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 1315232, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 3545248 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 4713, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 4901, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 2719727, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 1903913, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 12, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 5, 2024-09-15T23:32:13.204 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.297000351, 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.097013894, 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.097191965, 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.087325788, 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 4902, 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.194006843, 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000039577 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 903, 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000198565, 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000219 2024-09-15T23:32:13.205 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "cct": { 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "total_workers": 1, 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "unhealthy_workers": 0 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-MDSRank": { 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8117, 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.386318924, 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000170792 2024-09-15T23:32:13.206 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-PurgeQueue": { 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 42, 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.001663744, 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000039612 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "request": 2761, 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "reply": 2739, 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "reply_latency": { 2024-09-15T23:32:13.207 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2739, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 58.718034054, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.021437763 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_reply": 0, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "forward": 22, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_complete": 0, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_keys": 0, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_commit": 11, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_split": 0, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_merge": 0, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes": 500, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_top": 0, 2024-09-15T23:32:13.208 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_bottom": 1, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pin_tail": 499, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pinned": 499, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_expired": 9, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_with_caps": 470, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": 470, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "subtrees": 21, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse": 3260, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_hit": 3211, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_forward": 11, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_discover": 12, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_remote_ino": 0, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_lock": 4, 2024-09-15T23:32:13.209 INFO:teuthology.orchestra.run.smithi067.stdout: "load_cent": 1859, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "q": 0, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "exported": 1, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "exported_inodes": 6, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "imported": 17, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "imported_inodes": 504, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_dir_fetch": 0, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_backtrace_fetch": 8235, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_peer_discover": 0, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rfiles": 1023, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rbytes": 1569920835, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rsnaps": 0, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_fetch": 8235, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_set_tag": 0, 2024-09-15T23:32:13.210 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_inodes": 133, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dirfrag_rstats": 133, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_file_inodes": 8102, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_inode_file_caps": 22, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_revoke": 664, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_grant": 1194, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps": 1222, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps_dirty": 366, 2024-09-15T23:32:13.211 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_cap_release": 20, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "process_request_cap_release": 59 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache": { 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update": 0, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update_receipt": 0, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_try_discover": 0, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_send_discover": 36, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_handle_discover": 4, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays": 0, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_delayed": 0, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_created": 9, 2024-09-15T23:32:13.212 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_enqueued": 9, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_reintegrated": 0, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_migrated": 0, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_processing": 0, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_started": 0, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_completed": 0, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_quiesce_path": 15, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_quiesce_inode": 6622, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_exportdir": 1, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_flush": 0, 2024-09-15T23:32:13.213 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragstats": 0, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_inodestats": 0 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log": { 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "evlrg": 0, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "evadd": 1001, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "evex": 0, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "evtrm": 0, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "ev": 1001, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "evexg": 0, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "evexd": 0, 2024-09-15T23:32:13.214 INFO:teuthology.orchestra.run.smithi067.stdout: "segadd": 2, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "segex": 0, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "segtrm": 0, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "seg": 2, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "segmjr": 2, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "segexg": 0, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "segexd": 0, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "expos": 4194304, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "wrpos": 6569215, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "rdpos": 0, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "jlat": { 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 652, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 9.322533711, 2024-09-15T23:32:13.215 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.014298364 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "replayed": 0 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mem": { 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 493, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "ino+": 503, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "ino-": 10, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "dir": 56, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "dir+": 57, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "dir-": 1, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "dn": 500, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "dn+": 539, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "dn-": 39, 2024-09-15T23:32:13.216 INFO:teuthology.orchestra.run.smithi067.stdout: "cap": 470, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "cap+": 681, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "cap-": 211, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "rss": 61908, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "heap": 223520 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_server": { 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_client_request": 2923, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_server_request": 0, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_request": 2761, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_session": 24, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_peer_request": 9, 2024-09-15T23:32:13.217 INFO:teuthology.orchestra.run.smithi067.stdout: "req_create_latency": { 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 5, 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.007391917, 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.001478383 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getattr_latency": { 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2202, 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 57.421710620, 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.026077071 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getfilelock_latency": { 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.218 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "req_link_latency": { 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookup_latency": { 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 299, 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.089585329, 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.003644098 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookuphash_latency": { 2024-09-15T23:32:13.219 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupino_latency": { 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupname_latency": { 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.220 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupparent_latency": { 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupsnap_latency": { 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lssnap_latency": { 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.221 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mkdir_latency": { 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mknod_latency": { 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.222 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mksnap_latency": { 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "req_open_latency": { 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 206, 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.172685695, 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000838280 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "req_readdir_latency": { 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 18, 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.011160433, 2024-09-15T23:32:13.223 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000620024 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rename_latency": { 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 4, 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.008695273, 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002173818 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: "req_renamesnap_latency": { 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.224 INFO:teuthology.orchestra.run.smithi067.stdout: "req_snapdiff_latency": { 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmdir_latency": { 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmsnap_latency": { 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.225 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmxattr_latency": { 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setattr_latency": { 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setdirlayout_latency": { 2024-09-15T23:32:13.226 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setfilelock_latency": { 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setlayout_latency": { 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.227 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setxattr_latency": { 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "req_symlink_latency": { 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "req_unlink_latency": { 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 5, 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.006804787, 2024-09-15T23:32:13.228 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.001360957 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getvxattr_latency": { 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessions": { 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "session_count": 1, 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "session_add": 2, 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "session_remove": 1, 2024-09-15T23:32:13.229 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_open": 1, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_stale": 0, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "total_load": 332, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "average_load": 332, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "avg_session_uptime": 2350, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "mdthresh_evicted": 0 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool": { 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_bytes": 120, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_items": 120, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:13.230 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:13.231 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:13.232 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_items": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_bytes": 325569, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_items": 577, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bytes": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_items": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:13.233 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_bytes": 8768, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_items": 165, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_items": 0, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_bytes": 1767456, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_items": 17367, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_items": 0, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_items": 0 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.234 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter": { 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "op_active": 0, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "op_laggy": 0, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send": 9097, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send_bytes": 2466851, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "op_resend": 28, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "op_reply": 9097, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "op_latency": { 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 9097, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 23.745514448, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002610257 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "op_inflight": 0, 2024-09-15T23:32:13.235 INFO:teuthology.orchestra.run.smithi067.stdout: "oplen_avg": { 2024-09-15T23:32:13.236 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 9097, 2024-09-15T23:32:13.236 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 9207 2024-09-15T23:32:13.236 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.236 INFO:teuthology.orchestra.run.smithi067.stdout: "op": 9097, 2024-09-15T23:32:13.236 INFO:teuthology.orchestra.run.smithi067.stdout: "op_r": 8239, 2024-09-15T23:32:13.236 INFO:teuthology.orchestra.run.smithi067.stdout: "op_w": 858, 2024-09-15T23:32:13.236 INFO:teuthology.orchestra.run.smithi067.stdout: "op_rmw": 0, 2024-09-15T23:32:13.236 INFO:teuthology.orchestra.run.smithi067.stdout: "op_pg": 0, 2024-09-15T23:32:13.236 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_stat": 0, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_create": 10, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_read": 4, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_write": 656, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writefull": 63, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writesame": 0, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_append": 0, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_zero": 0, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_truncate": 0, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_delete": 30, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_mapext": 0, 2024-09-15T23:32:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_sparse_read": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_clonerange": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_getxattr": 8235, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_setxattr": 20, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_rmxattr": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_call": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_watch": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_notify": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:32:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_other": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_active": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_send": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_resend": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_ping": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_active": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_send": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_resend": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_active": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_send": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_resend": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_active": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_send": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_resend": 0, 2024-09-15T23:32:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: "command_active": 0, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "command_send": 0, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "command_resend": 0, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "map_epoch": 92, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "map_full": 0, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "map_inc": 4, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_sessions": 12, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_open": 12, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_close": 0, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_laggy": 0, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_wr": 155, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_rd": 0, 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_del": 34 2024-09-15T23:32:13.240 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "oft": { 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_objs": 1, 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_updates": 0, 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_removes": 0 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "osd-slow-ops": { 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_ops_count": 5691 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "purge_queue": { 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops": 0, 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops_high_water": 12, 2024-09-15T23:32:13.241 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing": 0, 2024-09-15T23:32:13.242 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_high_water": 6, 2024-09-15T23:32:13.242 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed_ops": 18, 2024-09-15T23:32:13.242 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed": 9, 2024-09-15T23:32:13.242 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_item_in_journal": 0 2024-09-15T23:32:13.242 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.242 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:13.242 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:13.242 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-09-15T23:32:13.242 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 17003, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 6768346, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 17003, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 17003, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 6768346, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.243 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_bytes": { 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 9097, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 2467832, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 9097, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 2467832, 2024-09-15T23:32:13.244 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_ops": { 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 1024, 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:13.245 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 9097, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 9097, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 9097, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 9097, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-09-15T23:32:13.246 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 9, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 981, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 9, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 4, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 981, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.247 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:13.248 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:13.248 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle-0x55947cc00960": { 2024-09-15T23:32:13.248 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 574, 2024-09-15T23:32:13.248 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-09-15T23:32:13.248 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:13.248 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 1001, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 2374911, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 1001, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 652, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 2374337, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:13.249 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T23:32:13.914 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:13 smithi067 ceph-mon[35665]: pgmap v848: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 KiB/s wr, 8 op/s 2024-09-15T23:32:13.942 INFO:tasks.check_counter:Daemon mds.h mds.exported=1 2024-09-15T23:32:13.942 INFO:tasks.check_counter:Daemon mds.h mds.imported=17 2024-09-15T23:32:13.942 DEBUG:tasks.check_counter:Getting stats from k 2024-09-15T23:32:13.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.k perf dump 2024-09-15T23:32:14.195 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.b/config 2024-09-15T23:32:14.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:13 smithi062 ceph-mon[30949]: pgmap v848: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 KiB/s wr, 8 op/s 2024-09-15T23:32:14.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:13 smithi148 ceph-mon[35610]: pgmap v848: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.5 KiB/s wr, 8 op/s 2024-09-15T23:32:14.813 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-09-15T23:32:14.813 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:14.813 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 0, 2024-09-15T23:32:14.813 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 2, 2024-09-15T23:32:14.813 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 0, 2024-09-15T23:32:14.813 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 0, 2024-09-15T23:32:14.813 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:14.813 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 0, 2024-09-15T23:32:14.813 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.000933646, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.000000000, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.000102662, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.814 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 313, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 310, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 506800, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 191280, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 2, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 1, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.040602598, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.019553920, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.025497796, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:14.815 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 309, 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.016379234, 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000053007 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 506800, 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 191280 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.816 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 6, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 239, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 3312, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 199472, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.015291367, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.009793577, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.003273817, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 238, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.018795704, 2024-09-15T23:32:14.817 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000078973 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 199472 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "cct": { 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "total_workers": 0, 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "unhealthy_workers": 0 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.818 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool": { 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:14.819 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:14.820 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_items": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_bytes": 148682, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_items": 29, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bytes": 0, 2024-09-15T23:32:14.821 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_items": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_bytes": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_items": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_items": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_bytes": 928, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_items": 2, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:14.822 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_items": 0, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_items": 0 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 315, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 462542, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 315, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-09-15T23:32:14.823 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 315, 2024-09-15T23:32:14.824 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 462542, 2024-09-15T23:32:14.824 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-09-15T23:32:14.824 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-09-15T23:32:14.824 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-09-15T23:32:14.824 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-09-15T23:32:14.824 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:14.824 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-09-15T23:32:14.824 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-09-15T23:32:15.492 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-09-15T23:32:15.493 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-09-15T23:32:15.493 DEBUG:tasks.check_counter:Getting stats from c 2024-09-15T23:32:15.493 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.c perf dump 2024-09-15T23:32:15.811 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:32:15.932 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:15 smithi148 ceph-mon[35610]: pgmap v849: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 KiB/s wr, 9 op/s 2024-09-15T23:32:16.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:15 smithi062 ceph-mon[30949]: pgmap v849: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 KiB/s wr, 9 op/s 2024-09-15T23:32:16.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:15 smithi067 ceph-mon[35665]: pgmap v849: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 KiB/s wr, 9 op/s 2024-09-15T23:32:16.561 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-15T23:32:16.561 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:16.561 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 1371, 2024-09-15T23:32:16.561 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 1371, 2024-09-15T23:32:16.561 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 4629969, 2024-09-15T23:32:16.561 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 435164, 2024-09-15T23:32:16.561 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 6, 2024-09-15T23:32:16.561 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.141392051, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.040627554, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.040887488, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.046735351, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1370, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.080774178, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000058959 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.562 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 2040, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 2020, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 6583989, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 732676, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 3, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.183909366, 2024-09-15T23:32:16.563 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.059692474, 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.069382621, 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.051388761, 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 2019, 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.105951039, 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000052476 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.564 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 596096, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 191056 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 452, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 684, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 4309235, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 710829, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 5, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 3, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.050958692, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.022756029, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.012112612, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.010786874, 2024-09-15T23:32:16.565 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:16.566 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 682, 2024-09-15T23:32:16.566 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.033842351, 2024-09-15T23:32:16.566 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000049622 2024-09-15T23:32:16.566 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.566 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:16.566 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 567728 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: "cct": { 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: "total_workers": 1, 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: "unhealthy_workers": 0 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.567 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-MDSRank": { 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 5945, 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.179151113, 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000030134 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-PurgeQueue": { 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.568 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "mds": { 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "request": 0, 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "reply": 0, 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "reply_latency": { 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_reply": 0, 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "forward": 0, 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_complete": 0, 2024-09-15T23:32:16.569 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_keys": 0, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_commit": 0, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_split": 0, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_merge": 0, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes": 428, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_top": 214, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_bottom": 214, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pin_tail": 0, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pinned": 121, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_expired": 0, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_with_caps": 0, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "caps": 0, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "subtrees": 11, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse": 0, 2024-09-15T23:32:16.570 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_hit": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_forward": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_discover": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_remote_ino": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_lock": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "load_cent": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "q": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "exported": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "exported_inodes": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "imported": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "imported_inodes": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_dir_fetch": 0, 2024-09-15T23:32:16.571 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_peer_discover": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rfiles": 1020, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rbytes": 1569920962, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rsnaps": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_set_tag": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_file_inodes": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:32:16.572 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_cap_release": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "process_request_cap_release": 0 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache": { 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update_receipt": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_try_discover": 0, 2024-09-15T23:32:16.573 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_send_discover": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_handle_discover": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_delayed": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_created": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_enqueued": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_reintegrated": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_migrated": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_processing": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_started": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_completed": 0, 2024-09-15T23:32:16.574 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_path": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_inode": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_exportdir": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_flush": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragstats": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_inodestats": 0 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log": { 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "evlrg": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "evadd": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "evex": 0, 2024-09-15T23:32:16.575 INFO:teuthology.orchestra.run.smithi148.stdout: "evtrm": 0, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "ev": 5183, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "evexg": 0, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "evexd": 0, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "segadd": 0, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "segex": 0, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "segtrm": 0, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "seg": 2, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "segmjr": 73, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "segexg": 0, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "segexd": 0, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "expos": 17565808, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "wrpos": 18140075, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "rdpos": 18133047, 2024-09-15T23:32:16.576 INFO:teuthology.orchestra.run.smithi148.stdout: "jlat": { 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "replayed": 5183 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_mem": { 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "ino": 108, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "ino+": 1176, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "ino-": 1068, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "dir": 38, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "dir+": 73, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "dir-": 35, 2024-09-15T23:32:16.577 INFO:teuthology.orchestra.run.smithi148.stdout: "dn": 428, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "dn+": 1493, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "dn-": 1065, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "cap": 0, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "cap+": 0, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "cap-": 0, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "rss": 51892, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "heap": 215328 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_server": { 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_client_request": 0, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_server_request": 0, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_request": 0, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_session": 0, 2024-09-15T23:32:16.578 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_peer_request": 0, 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "req_create_latency": { 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getattr_latency": { 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getfilelock_latency": { 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.579 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "req_link_latency": { 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookup_latency": { 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookuphash_latency": { 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.580 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupino_latency": { 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupname_latency": { 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupparent_latency": { 2024-09-15T23:32:16.581 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupsnap_latency": { 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lssnap_latency": { 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.582 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.583 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mkdir_latency": { 2024-09-15T23:32:16.583 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.583 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.583 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.583 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mknod_latency": { 2024-09-15T23:32:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mksnap_latency": { 2024-09-15T23:32:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.584 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "req_open_latency": { 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "req_readdir_latency": { 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rename_latency": { 2024-09-15T23:32:16.585 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "req_renamesnap_latency": { 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "req_snapdiff_latency": { 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.586 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmdir_latency": { 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmsnap_latency": { 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmxattr_latency": { 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.587 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setattr_latency": { 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setdirlayout_latency": { 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setfilelock_latency": { 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.588 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setlayout_latency": { 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setxattr_latency": { 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.589 INFO:teuthology.orchestra.run.smithi148.stdout: "req_symlink_latency": { 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "req_unlink_latency": { 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getvxattr_latency": { 2024-09-15T23:32:16.590 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_sessions": { 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "session_count": 1, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "session_add": 3, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "session_remove": 2, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_open": 1, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_stale": 0, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "total_load": 0, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "average_load": 0, 2024-09-15T23:32:16.591 INFO:teuthology.orchestra.run.smithi148.stdout: "avg_session_uptime": 2354, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "mdthresh_evicted": 0 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "mempool": { 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:16.592 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:16.593 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_items": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_bytes": 193410, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_items": 139, 2024-09-15T23:32:16.594 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bytes": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_items": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_bytes": 8768, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_items": 165, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:16.595 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_items": 0, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_bytes": 612879, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_items": 14028, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_items": 0, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_items": 0 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter": { 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "op_active": 0, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "op_laggy": 0, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send": 3520, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send_bytes": 26, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "op_resend": 4, 2024-09-15T23:32:16.596 INFO:teuthology.orchestra.run.smithi148.stdout: "op_reply": 3520, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "op_latency": { 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3520, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 51.082891789, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.014512185 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "op_inflight": 0, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "oplen_avg": { 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3520, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 3521 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "op": 3520, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "op_r": 3520, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "op_w": 0, 2024-09-15T23:32:16.597 INFO:teuthology.orchestra.run.smithi148.stdout: "op_rmw": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "op_pg": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_stat": 2082, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_create": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_read": 1435, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_write": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writefull": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writesame": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_append": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_zero": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_truncate": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_delete": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_mapext": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_sparse_read": 0, 2024-09-15T23:32:16.598 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_clonerange": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_getxattr": 2, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_setxattr": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_rmxattr": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_call": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_watch": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_notify": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_other": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_active": 0, 2024-09-15T23:32:16.599 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_send": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_resend": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_ping": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_active": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_send": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_resend": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_active": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_send": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_resend": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_active": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_send": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_resend": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "command_active": 0, 2024-09-15T23:32:16.600 INFO:teuthology.orchestra.run.smithi148.stdout: "command_send": 0, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "command_resend": 0, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "map_epoch": 92, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "map_full": 0, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "map_inc": 4, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_sessions": 5, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_open": 5, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_close": 0, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_laggy": 0, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_wr": 0, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_rd": 2, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_del": 0 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "oft": { 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_objs": 0, 2024-09-15T23:32:16.601 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_updates": 0, 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_removes": 0 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "osd-slow-ops": { 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_ops_count": 0 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "purge_queue": { 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops": 0, 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing": 0, 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_high_water": 0, 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed_ops": 0, 2024-09-15T23:32:16.602 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed": 0, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_item_in_journal": 0 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 3859, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 15155596, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 3859, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3859, 2024-09-15T23:32:16.603 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 15155596, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_bytes": { 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:16.604 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 3520, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 13945781, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3520, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 13945781, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_ops": { 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 1024, 2024-09-15T23:32:16.605 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 3520, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 3520, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3520, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 3520, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.606 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:16.607 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 0, 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 0, 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle-0x5627586beb40": { 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:16.608 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 0, 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 0, 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:16.609 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-15T23:32:17.505 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-09-15T23:32:17.505 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-09-15T23:32:17.505 DEBUG:tasks.check_counter:Getting stats from f 2024-09-15T23:32:17.505 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.f perf dump 2024-09-15T23:32:17.756 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:32:18.161 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:17 smithi148 ceph-mon[35610]: pgmap v850: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-09-15T23:32:18.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:17 smithi062 ceph-mon[30949]: pgmap v850: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-09-15T23:32:18.265 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-15T23:32:18.265 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:18.265 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 1288, 2024-09-15T23:32:18.265 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 1290, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 2815183, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 409648, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 7, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.110202117, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.031186124, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.033300692, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.034329952, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1288, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.060628765, 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000047072 2024-09-15T23:32:18.266 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 0 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 1305, 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 1283, 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 836366, 2024-09-15T23:32:18.267 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 498541, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 3, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.130323068, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.040654663, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.050994830, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.036245572, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1283, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.081067857, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000063186 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.268 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 599200, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 191984 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 1080, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 1313, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 362704, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 824354, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 3, 2024-09-15T23:32:18.269 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.129676535, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.044802848, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.032556118, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.039567993, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1311, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.077890283, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000059412 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.270 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 481232 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "cct": { 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "total_workers": 1, 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "unhealthy_workers": 0 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-MDSRank": { 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 5620, 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.164799811, 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000029323 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.271 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-PurgeQueue": { 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "mds": { 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "request": 0, 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "reply": 0, 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "reply_latency": { 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.272 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_reply": 0, 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: "forward": 0, 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_complete": 0, 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_keys": 0, 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_commit": 0, 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_split": 0, 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_merge": 0, 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes": 700, 2024-09-15T23:32:18.273 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_top": 333, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_bottom": 367, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pin_tail": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pinned": 224, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_expired": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_with_caps": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "caps": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "subtrees": 16, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_hit": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_forward": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_discover": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_remote_ino": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_lock": 0, 2024-09-15T23:32:18.274 INFO:teuthology.orchestra.run.smithi148.stdout: "load_cent": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "q": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "exported": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "exported_inodes": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "imported": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "imported_inodes": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_dir_fetch": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_backtrace_fetch": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_peer_discover": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rfiles": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rbytes": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rsnaps": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_fetch": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_set_tag": 0, 2024-09-15T23:32:18.275 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_file_inodes": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_revoke": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_grant": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps": 0, 2024-09-15T23:32:18.276 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps_dirty": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_cap_release": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "process_request_cap_release": 0 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache": { 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update_receipt": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_try_discover": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_send_discover": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_handle_discover": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_delayed": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_created": 0, 2024-09-15T23:32:18.277 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_enqueued": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_reintegrated": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_migrated": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_processing": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_started": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_completed": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_path": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_inode": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_exportdir": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_flush": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragstats": 0, 2024-09-15T23:32:18.278 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_inodestats": 0 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log": { 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "evlrg": 0, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "evadd": 0, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "evex": 0, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "evtrm": 0, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "ev": 1491, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "evexg": 0, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "evexd": 0, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "segadd": 0, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "segex": 0, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "segtrm": 0, 2024-09-15T23:32:18.279 INFO:teuthology.orchestra.run.smithi148.stdout: "seg": 3, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "segmjr": 2, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "segexg": 0, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "segexd": 0, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "expos": 4194304, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "wrpos": 6674688, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "rdpos": 6674627, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "jlat": { 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "replayed": 1491 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.280 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_mem": { 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "ino": 502, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "ino+": 762, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "ino-": 260, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "dir": 25, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "dir+": 31, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "dir-": 6, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "dn": 700, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "dn+": 783, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "dn-": 83, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "cap": 0, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "cap+": 0, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "cap-": 0, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "rss": 51608, 2024-09-15T23:32:18.281 INFO:teuthology.orchestra.run.smithi148.stdout: "heap": 215328 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_server": { 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_client_request": 0, 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_server_request": 0, 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_request": 0, 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_session": 0, 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_peer_request": 0, 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "req_create_latency": { 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getattr_latency": { 2024-09-15T23:32:18.282 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getfilelock_latency": { 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "req_link_latency": { 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.283 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.284 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookup_latency": { 2024-09-15T23:32:18.284 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.284 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.284 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.284 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.284 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookuphash_latency": { 2024-09-15T23:32:18.284 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.284 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.284 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupino_latency": { 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupname_latency": { 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.285 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupparent_latency": { 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupsnap_latency": { 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.286 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.287 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.287 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lssnap_latency": { 2024-09-15T23:32:18.287 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.287 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.287 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.287 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.287 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mkdir_latency": { 2024-09-15T23:32:18.288 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:17 smithi067 ceph-mon[35665]: pgmap v850: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.2 KiB/s wr, 8 op/s 2024-09-15T23:32:18.288 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.288 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.288 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.288 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.288 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mknod_latency": { 2024-09-15T23:32:18.288 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mksnap_latency": { 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "req_open_latency": { 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "req_readdir_latency": { 2024-09-15T23:32:18.289 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rename_latency": { 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "req_renamesnap_latency": { 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.290 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "req_snapdiff_latency": { 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmdir_latency": { 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmsnap_latency": { 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.291 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmxattr_latency": { 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setattr_latency": { 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setdirlayout_latency": { 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.292 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setfilelock_latency": { 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setlayout_latency": { 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setxattr_latency": { 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.293 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "req_symlink_latency": { 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "req_unlink_latency": { 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:32:18.294 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getvxattr_latency": { 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_sessions": { 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "session_count": 1, 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "session_add": 1, 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "session_remove": 0, 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_open": 1, 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_stale": 0, 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "total_load": 0, 2024-09-15T23:32:18.295 INFO:teuthology.orchestra.run.smithi148.stdout: "average_load": 0, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "avg_session_uptime": 2356, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "mdthresh_evicted": 0 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "mempool": { 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:18.296 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:18.297 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_items": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_bytes": 209226, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_items": 549, 2024-09-15T23:32:18.298 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bytes": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_items": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_bytes": 8768, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_items": 165, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_items": 0, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_bytes": 1677139, 2024-09-15T23:32:18.299 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_items": 12929, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_items": 0, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_items": 0 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter": { 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "op_active": 0, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "op_laggy": 0, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send": 3331, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send_bytes": 21, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "op_resend": 3, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "op_reply": 3331, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "op_latency": { 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3331, 2024-09-15T23:32:18.300 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 11.625747599, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.003490167 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "op_inflight": 0, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "oplen_avg": { 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3331, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 3332 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "op": 3331, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "op_r": 3331, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "op_w": 0, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "op_rmw": 0, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "op_pg": 0, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_stat": 2041, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_create": 0, 2024-09-15T23:32:18.301 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_read": 1288, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_write": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writefull": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writesame": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_append": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_zero": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_truncate": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_delete": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_mapext": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_sparse_read": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_clonerange": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_getxattr": 1, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_setxattr": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_rmxattr": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_call": 0, 2024-09-15T23:32:18.302 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_watch": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_notify": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_other": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_active": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_send": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_resend": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_ping": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_active": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_send": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_resend": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_active": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_send": 0, 2024-09-15T23:32:18.303 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_resend": 0, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_active": 0, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_send": 0, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_resend": 0, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "command_active": 0, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "command_send": 0, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "command_resend": 0, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "map_epoch": 92, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "map_full": 0, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "map_inc": 4, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_sessions": 5, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_open": 6, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_close": 1, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_laggy": 0, 2024-09-15T23:32:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_wr": 0, 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_rd": 2, 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_del": 0 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "oft": { 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_objs": 0, 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_updates": 0, 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_removes": 0 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "osd-slow-ops": { 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_ops_count": 0 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.305 INFO:teuthology.orchestra.run.smithi148.stdout: "purge_queue": { 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops": 0, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops_high_water": 0, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing": 0, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_high_water": 0, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed_ops": 0, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed": 0, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_item_in_journal": 0 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 3669, 2024-09-15T23:32:18.306 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 3663793, 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 3669, 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3669, 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 3663793, 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.307 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_bytes": { 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 3331, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 2480389, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3331, 2024-09-15T23:32:18.308 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 2480389, 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_ops": { 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 1024, 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:18.309 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 3331, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 3331, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 3331, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 3331, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.310 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:18.311 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 0, 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 0, 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle-0x55a9d55f8b40": { 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:18.312 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 0, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 0, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:18.313 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-15T23:32:19.037 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-09-15T23:32:19.037 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-09-15T23:32:19.037 DEBUG:tasks.check_counter:Getting stats from i 2024-09-15T23:32:19.037 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.i perf dump 2024-09-15T23:32:19.283 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:32:19.765 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-15T23:32:19.765 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:19.765 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 8658, 2024-09-15T23:32:19.765 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 9815, 2024-09-15T23:32:19.765 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 4191170, 2024-09-15T23:32:19.765 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 7203489, 2024-09-15T23:32:19.765 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 46, 2024-09-15T23:32:19.765 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 3, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.585382448, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.196606267, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.280811039, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.111769591, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 9813, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.658861226, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000067141 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 5663, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000997889, 2024-09-15T23:32:19.766 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000176 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 967872, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 2360352 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 6699, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 5994, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 3573551, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 2040916, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 7, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 6, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.427534526, 2024-09-15T23:32:19.767 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.160365332, 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.227942704, 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.058057372, 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 5993, 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.277366086, 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000046281 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 6426, 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.001182012, 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000183 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.768 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 608784, 2024-09-15T23:32:19.769 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 211200 2024-09-15T23:32:19.769 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.769 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:19.769 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 2237, 2024-09-15T23:32:19.769 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 2277, 2024-09-15T23:32:19.769 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 1287437, 2024-09-15T23:32:19.769 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 1450399, 2024-09-15T23:32:19.769 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 20, 2024-09-15T23:32:19.769 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 5, 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.156904658, 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.058164334, 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.058998182, 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.033462820, 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 2276, 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.109109576, 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000047939 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 929, 2024-09-15T23:32:19.770 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000172875, 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000186 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 572704 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "cct": { 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "total_workers": 1, 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "unhealthy_workers": 0 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-MDSRank": { 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-15T23:32:19.771 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 8211, 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 1.421655502, 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000173140 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: "finisher-PurgeQueue": { 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: "queue_len": 0, 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: "complete_latency": { 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 680, 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.029577433, 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000043496 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.772 INFO:teuthology.orchestra.run.smithi148.stdout: "mds": { 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "request": 1811, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "reply": 1726, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "reply_latency": { 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1726, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 96.799186707, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.056082958 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_reply": 0, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "forward": 85, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_complete": 2, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_fetch_keys": 0, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_commit": 11, 2024-09-15T23:32:19.773 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_split": 0, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_merge": 0, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes": 516, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_top": 0, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_bottom": 66, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pin_tail": 450, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_pinned": 516, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_expired": 92, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "inodes_with_caps": 500, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "caps": 500, 2024-09-15T23:32:19.774 INFO:teuthology.orchestra.run.smithi148.stdout: "subtrees": 17, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse": 3011, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_hit": 2577, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_forward": 85, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_discover": 14, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_dir_fetch": 0, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_remote_ino": 0, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "traverse_lock": 65, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "load_cent": 1210, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "q": 0, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "exported": 5, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "exported_inodes": 77, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "imported": 19, 2024-09-15T23:32:19.775 INFO:teuthology.orchestra.run.smithi148.stdout: "imported_inodes": 560, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_dir_fetch": 0, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_backtrace_fetch": 8738, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "openino_peer_discover": 0, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rfiles": 1023, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rbytes": 1569920835, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "root_rsnaps": 0, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_fetch": 8738, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_set_tag": 0, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_backtrace_repaired": 0, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_inotable_repaired": 0, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_inodes": 0, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dir_base_inodes": 0, 2024-09-15T23:32:19.776 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_dirfrag_rstats": 0, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "scrub_file_inodes": 8738, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_inode_file_caps": 0, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_revoke": 370, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_grant": 996, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_trunc": 0, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps": 848, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_caps_dirty": 288, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_cap_release": 126, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "process_request_cap_release": 789 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_cache": { 2024-09-15T23:32:19.777 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_update_receipt": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_try_discover": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_send_discover": 14, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "dir_handle_discover": 12, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_delayed": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "num_strays_enqueuing": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_created": 194, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_enqueued": 194, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_reintegrated": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "strays_migrated": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_processing": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_enqueued": 0, 2024-09-15T23:32:19.778 INFO:teuthology.orchestra.run.smithi148.stdout: "num_recovering_prioritized": 0, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_started": 0, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "recovery_completed": 0, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_path": 15, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_quiesce_inode": 7041, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_enqueue_scrub": 0, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_exportdir": 5, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_flush": 0, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragmentdir": 0, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_fragstats": 0, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "ireq_inodestats": 0 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_log": { 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "evlrg": 0, 2024-09-15T23:32:19.779 INFO:teuthology.orchestra.run.smithi148.stdout: "evadd": 1491, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "evex": 0, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "evtrm": 0, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "ev": 1491, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "evexg": 0, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "evexd": 0, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "segadd": 3, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "segex": 0, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "segtrm": 0, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "seg": 3, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "segmjr": 2, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "segexg": 0, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "segexd": 0, 2024-09-15T23:32:19.780 INFO:teuthology.orchestra.run.smithi148.stdout: "expos": 4194304, 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "wrpos": 6674688, 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "rdpos": 0, 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "jlat": { 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 471, 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 5.005319450, 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.010627005 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "replayed": 0 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_mem": { 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "ino": 519, 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "ino+": 788, 2024-09-15T23:32:19.781 INFO:teuthology.orchestra.run.smithi148.stdout: "ino-": 269, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "dir": 34, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "dir+": 39, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "dir-": 5, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "dn": 516, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "dn+": 809, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "dn-": 293, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "cap": 500, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "cap+": 832, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "cap-": 332, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "rss": 65116, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "heap": 223520 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_server": { 2024-09-15T23:32:19.782 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_client_request": 2387, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "dispatch_server_request": 0, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_request": 1811, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_client_session": 15, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "handle_peer_request": 389, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "req_create_latency": { 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 194, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 53.018358385, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.273290507 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getattr_latency": { 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 891, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 34.140982386, 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.038317600 2024-09-15T23:32:19.783 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getfilelock_latency": { 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "req_link_latency": { 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookup_latency": { 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 235, 2024-09-15T23:32:19.784 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 9.035484608, 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.038448870 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookuphash_latency": { 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupino_latency": { 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 1, 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000147847, 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000147847 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupname_latency": { 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupparent_latency": { 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lookupsnap_latency": { 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.786 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.787 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.787 INFO:teuthology.orchestra.run.smithi148.stdout: "req_lssnap_latency": { 2024-09-15T23:32:19.787 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.787 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.787 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.787 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mkdir_latency": { 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mknod_latency": { 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "req_mksnap_latency": { 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.788 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "req_open_latency": { 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 197, 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.103312781, 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000524430 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "req_readdir_latency": { 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 14, 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.010513295, 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000750949 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.789 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rename_latency": { 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 181, 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.457359747, 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.002526849 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "req_renamesnap_latency": { 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "req_snapdiff_latency": { 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.790 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmdir_latency": { 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmsnap_latency": { 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "req_rmxattr_latency": { 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.791 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setattr_latency": { 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setdirlayout_latency": { 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setfilelock_latency": { 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.792 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setlayout_latency": { 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "req_setxattr_latency": { 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.793 INFO:teuthology.orchestra.run.smithi148.stdout: "req_symlink_latency": { 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "req_unlink_latency": { 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 13, 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.033027658, 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.002540589 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_revoke_eviction": 0, 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "cap_acquisition_throttle": 0, 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "req_getvxattr_latency": { 2024-09-15T23:32:19.794 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_sessions": { 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "session_count": 1, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "session_add": 1, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "session_remove": 0, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_open": 1, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "sessions_stale": 0, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "total_load": 165, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "average_load": 165, 2024-09-15T23:32:19.795 INFO:teuthology.orchestra.run.smithi148.stdout: "avg_session_uptime": 972, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "mdthresh_evicted": 0 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "mempool": { 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:19.796 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:19.797 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_items": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:19.798 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_bytes": 314773, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_items": 603, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_bytes": 88, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_items": 1, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bytes": 0, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_items": 0, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_bytes": 8768, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_items": 165, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:19.799 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_items": 0, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_bytes": 1811508, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_items": 17086, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_items": 0, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_items": 0 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "objecter": { 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "op_active": 0, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "op_laggy": 0, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send": 10079, 2024-09-15T23:32:19.800 INFO:teuthology.orchestra.run.smithi148.stdout: "op_send_bytes": 2617089, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "op_resend": 28, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "op_reply": 10079, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "op_latency": { 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 10079, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 47.225975062, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.004685581 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "op_inflight": 0, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "oplen_avg": { 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 10079, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 10269 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "op": 10079, 2024-09-15T23:32:19.801 INFO:teuthology.orchestra.run.smithi148.stdout: "op_r": 8830, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "op_w": 1249, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "op_rmw": 0, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "op_pg": 0, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_stat": 0, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_create": 10, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_read": 83, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_write": 555, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writefull": 121, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_writesame": 0, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_append": 0, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_zero": 0, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_truncate": 0, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_delete": 400, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_mapext": 0, 2024-09-15T23:32:19.802 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_sparse_read": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_clonerange": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_getxattr": 8738, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_setxattr": 20, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_cmpxattr": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_rmxattr": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_resetxattrs": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_call": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_watch": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_notify": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_src_cmpxattr": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_pgls_filter": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "osdop_other": 0, 2024-09-15T23:32:19.803 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_active": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_send": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_resend": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "linger_ping": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_active": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_send": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "poolop_resend": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_active": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_send": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "poolstat_resend": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_active": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_send": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "statfs_resend": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "command_active": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "command_send": 0, 2024-09-15T23:32:19.804 INFO:teuthology.orchestra.run.smithi148.stdout: "command_resend": 0, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "map_epoch": 92, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "map_full": 0, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "map_inc": 4, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_sessions": 12, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_open": 12, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_session_close": 0, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_laggy": 0, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_wr": 250, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_rd": 11, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_del": 81 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "oft": { 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_objs": 1, 2024-09-15T23:32:19.805 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_kv_pairs": 0, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_updates": 0, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "omap_total_removes": 0 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "osd-slow-ops": { 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "slow_ops_count": 6025 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "purge_queue": { 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops": 0, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_ops_high_water": 16, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing": 0, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executing_high_water": 8, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed_ops": 388, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_executed": 194, 2024-09-15T23:32:19.806 INFO:teuthology.orchestra.run.smithi148.stdout: "pq_item_in_journal": 0 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 17590, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 7314485, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 17590, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 17590, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 7314485, 2024-09-15T23:32:19.807 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_bytes": { 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:19.808 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 10079, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 2638024, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 10079, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 2638024, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-objecter_ops": { 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 1024, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:19.809 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 0, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 0, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 0, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 10079, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 10079, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 10079, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 10079, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.810 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle": { 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 194, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 21146, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 194, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 84, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 21146, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.811 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-write_buf_throttle-0x55e48c596960": { 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 3758096384, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 1491, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 2480384, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 1491, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 471, 2024-09-15T23:32:19.812 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 2480384, 2024-09-15T23:32:19.813 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:19.813 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:19.813 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:19.813 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:19.813 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:19.813 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:19.813 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-15T23:32:20.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:19 smithi148 ceph-mon[35610]: pgmap v851: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.4 KiB/s wr, 8 op/s 2024-09-15T23:32:20.230 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:19 smithi062 ceph-mon[30949]: pgmap v851: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.4 KiB/s wr, 8 op/s 2024-09-15T23:32:20.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:19 smithi067 ceph-mon[35665]: pgmap v851: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.4 KiB/s wr, 8 op/s 2024-09-15T23:32:20.584 INFO:tasks.check_counter:Daemon mds.i mds.exported=5 2024-09-15T23:32:20.584 INFO:tasks.check_counter:Daemon mds.i mds.imported=19 2024-09-15T23:32:20.584 DEBUG:tasks.check_counter:Getting stats from l 2024-09-15T23:32:20.585 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph daemon mds.l perf dump 2024-09-15T23:32:20.837 INFO:teuthology.orchestra.run.smithi148.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.c/config 2024-09-15T23:32:21.500 INFO:teuthology.orchestra.run.smithi148.stdout:{ 2024-09-15T23:32:21.500 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-0": { 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 4, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 4, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 2992, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 448, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 0, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.003130762, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.000044414, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.002299292, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:21.501 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 3, 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.004198319, 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.001399439 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 448 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-1": { 2024-09-15T23:32:21.502 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 2, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 239, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 320, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 199376, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 4, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 2, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.013921725, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.010156485, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.001282000, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 237, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.014048351, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000059275 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.503 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 320, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 199376 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "AsyncMessenger::Worker-2": { 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_messages": 315, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages": 312, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_bytes": 507344, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_bytes": 192560, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_created_connections": 2, 2024-09-15T23:32:21.504 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_active_connections": 1, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_total_time": 0.043021466, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_send_time": 0.019556177, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_recv_time": 0.029028527, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_messages_queue_lat": { 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 311, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.018228061, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000058611 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_handle_ack_lat": { 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:21.505 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_recv_encrypted_bytes": 507344, 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "msgr_send_encrypted_bytes": 192560 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "cct": { 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "total_workers": 0, 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "unhealthy_workers": 0 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "mempool": { 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_bytes": 0, 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "bloom_filter_items": 0, 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_bytes": 0, 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_alloc_items": 0, 2024-09-15T23:32:21.506 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_bytes": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_data_items": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_onode_items": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_meta_items": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_bytes": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_other_items": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_cache_buffer_items": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_bytes": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_extent_items": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_bytes": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_blob_items": 0, 2024-09-15T23:32:21.507 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_shared_blob_items": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_inline_bl_items": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_bytes": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_fsck_items": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_bytes": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_txc_items": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_deferred_items": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_bytes": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluestore_writing_items": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_bytes": 0, 2024-09-15T23:32:21.508 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_items": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_bytes": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_reader_items": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_bytes": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "bluefs_file_writer_items": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_bytes": 148721, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_anon_items": 30, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_bytes": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "buffer_meta_items": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_bytes": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_items": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_bytes": 0, 2024-09-15T23:32:21.509 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_mapbl_items": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_bytes": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "osd_pglog_items": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_bytes": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_items": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_bytes": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "osdmap_mapping_items": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_bytes": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "pgmap_items": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_bytes": 928, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "mds_co_items": 2, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_bytes": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_1_items": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_bytes": 0, 2024-09-15T23:32:21.510 INFO:teuthology.orchestra.run.smithi148.stdout: "unittest_2_items": 0 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: }, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "val": 0, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "max": 104857600, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "get_started": 0, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "get": 317, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "get_sum": 462802, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_fail": 0, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "get_or_fail_success": 317, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "take": 0, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "take_sum": 0, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "put": 317, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "put_sum": 462802, 2024-09-15T23:32:21.511 INFO:teuthology.orchestra.run.smithi148.stdout: "wait": { 2024-09-15T23:32:21.512 INFO:teuthology.orchestra.run.smithi148.stdout: "avgcount": 0, 2024-09-15T23:32:21.512 INFO:teuthology.orchestra.run.smithi148.stdout: "sum": 0.000000000, 2024-09-15T23:32:21.512 INFO:teuthology.orchestra.run.smithi148.stdout: "avgtime": 0.000000000 2024-09-15T23:32:21.512 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:21.512 INFO:teuthology.orchestra.run.smithi148.stdout: } 2024-09-15T23:32:21.512 INFO:teuthology.orchestra.run.smithi148.stdout:} 2024-09-15T23:32:22.040 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:21 smithi148 ceph-mon[35610]: pgmap v852: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-09-15T23:32:22.224 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-09-15T23:32:22.224 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-09-15T23:32:22.224 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-15T23:32:22.234 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-15T23:32:22.234 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:32:22.234 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:32:22.235 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:21 smithi062 ceph-mon[30949]: pgmap v852: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-09-15T23:32:22.256 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:21 smithi067 ceph-mon[35665]: pgmap v852: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.5 KiB/s wr, 6 op/s 2024-09-15T23:32:22.267 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-09-15T23:32:22.267 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-15T23:32:22.328 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd blocklist ls 2024-09-15T23:32:22.636 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.112+0000 7f26eecea640 1 -- 172.21.15.62:0/2843622840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8106a80 msgr2=0x7f26e8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.112+0000 7f26eecea640 1 --2- 172.21.15.62:0/2843622840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8106a80 0x7f26e8106e60 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f26d8009cd0 tx=0x7f26d802a8e0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.112+0000 7f26eecea640 1 -- 172.21.15.62:0/2843622840 shutdown_connections 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.112+0000 7f26eecea640 1 --2- 172.21.15.62:0/2843622840 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26e8101420 0x7f26e810d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.112+0000 7f26eecea640 1 --2- 172.21.15.62:0/2843622840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8100a80 0x7f26e8100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.112+0000 7f26eecea640 1 --2- 172.21.15.62:0/2843622840 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8106a80 0x7f26e8106e60 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.112+0000 7f26eecea640 1 -- 172.21.15.62:0/2843622840 >> 172.21.15.62:0/2843622840 conn(0x7f26e80fc7a0 msgr2=0x7f26e80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.112+0000 7f26eecea640 1 -- 172.21.15.62:0/2843622840 shutdown_connections 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.112+0000 7f26eecea640 1 -- 172.21.15.62:0/2843622840 wait complete. 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26eecea640 1 Processor -- start 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26eecea640 1 -- start start 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26eecea640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8100a80 0x7f26e8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26eecea640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8101420 0x7f26e8198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26eecea640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26e8106a80 0x7f26e819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26eecea640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f26e8110e80 con 0x7f26e8101420 2024-09-15T23:32:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26eecea640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f26e8110d00 con 0x7f26e8100a80 2024-09-15T23:32:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26eecea640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f26e8111000 con 0x7f26e8106a80 2024-09-15T23:32:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8101420 0x7f26e8198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8101420 0x7f26e8198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48900/0 (socket says 172.21.15.62:48900) 2024-09-15T23:32:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.113+0000 7f26e7fff640 1 -- 172.21.15.62:0/1472579975 learned_addr learned my addr 172.21.15.62:0/1472579975 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26ed287640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26e8106a80 0x7f26e819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26eca86640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8100a80 0x7f26e8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26e7fff640 1 -- 172.21.15.62:0/1472579975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26e8106a80 msgr2=0x7f26e819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26e7fff640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26e8106a80 0x7f26e819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26e7fff640 1 -- 172.21.15.62:0/1472579975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8100a80 msgr2=0x7f26e8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26e7fff640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8100a80 0x7f26e8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26e7fff640 1 -- 172.21.15.62:0/1472579975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e819d400 con 0x7f26e8101420 2024-09-15T23:32:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26eca86640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8100a80 0x7f26e8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26ed287640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26e8106a80 0x7f26e819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26e7fff640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8101420 0x7f26e8198990 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00c890 tx=0x7f26dc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26e5ffb640 1 -- 172.21.15.62:0/1472579975 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc004220 con 0x7f26e8101420 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26e5ffb640 1 -- 172.21.15.62:0/1472579975 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26dc0043c0 con 0x7f26e8101420 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26e819d6f0 con 0x7f26e8101420 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26e5ffb640 1 -- 172.21.15.62:0/1472579975 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0049e0 con 0x7f26e8101420 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.114+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26e81a4fd0 con 0x7f26e8101420 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.115+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26e8100ee0 con 0x7f26e8101420 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.116+0000 7f26e5ffb640 1 -- 172.21.15.62:0/1472579975 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f26dc0079b0 con 0x7f26e8101420 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.117+0000 7f26e5ffb640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26d00a3240 0x7f26d00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.117+0000 7f26eca86640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26d00a3240 0x7f26d00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.117+0000 7f26e5ffb640 1 -- 172.21.15.62:0/1472579975 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f26dc0cfdc0 con 0x7f26e8101420 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.118+0000 7f26eca86640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26d00a3240 0x7f26d00a5700 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000f80 tx=0x7f26d8031510 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.122+0000 7f26e5ffb640 1 -- 172.21.15.62:0/1472579975 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f26dc014030 con 0x7f26e8101420 2024-09-15T23:32:23.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.310+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f26e81a5340 con 0x7f26e8101420 2024-09-15T23:32:23.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.310+0000 7f26e5ffb640 1 -- 172.21.15.62:0/1472579975 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v92) ==== 81+0+589 (secure 0 0 0) 0x7f26dc09c760 con 0x7f26e8101420 2024-09-15T23:32:23.311 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/1303697831 2024-09-16T23:06:39.607848+0000 2024-09-15T23:32:23.312 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/2614061737 2024-09-16T23:06:39.607848+0000 2024-09-15T23:32:23.312 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/574363632 2024-09-16T23:06:15.815154+0000 2024-09-15T23:32:23.312 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/1277095060 2024-09-16T23:06:15.815154+0000 2024-09-15T23:32:23.312 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6801/2614061737 2024-09-16T23:06:39.607848+0000 2024-09-15T23:32:23.312 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:6800/1277095060 2024-09-16T23:06:15.815154+0000 2024-09-15T23:32:23.312 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/50968825 2024-09-16T23:06:39.607848+0000 2024-09-15T23:32:23.312 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/2637169275 2024-09-16T23:06:39.607848+0000 2024-09-15T23:32:23.312 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3526091028 2024-09-16T23:06:15.815154+0000 2024-09-15T23:32:23.313 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62:0/3293859590 2024-09-16T23:06:15.815154+0000 2024-09-15T23:32:23.313 INFO:teuthology.orchestra.run.smithi062.stderr:listed 10 entries 2024-09-15T23:32:23.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26d00a3240 msgr2=0x7f26d00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:23.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26d00a3240 0x7f26d00a5700 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f26d8000f80 tx=0x7f26d8031510 comp rx=0 tx=0).stop 2024-09-15T23:32:23.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8101420 msgr2=0x7f26e8198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:23.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8101420 0x7f26e8198990 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f26dc00c890 tx=0x7f26dc00cd50 comp rx=0 tx=0).stop 2024-09-15T23:32:23.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 shutdown_connections 2024-09-15T23:32:23.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f26d00a3240 0x7f26d00a5700 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f26e8106a80 0x7f26e819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26e8101420 0x7f26e8198990 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 --2- 172.21.15.62:0/1472579975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e8100a80 0x7f26e8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:23.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 >> 172.21.15.62:0/1472579975 conn(0x7f26e80fc7a0 msgr2=0x7f26e8104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:23.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 shutdown_connections 2024-09-15T23:32:23.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:23.314+0000 7f26eecea640 1 -- 172.21.15.62:0/1472579975 wait complete. 2024-09-15T23:32:24.011 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-15T23:32:24.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:32:24.012 DEBUG:teuthology.orchestra.run.smithi062:> sudo umount /tmp/cephfs 2024-09-15T23:32:24.037 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:23 smithi062 ceph-mon[30949]: pgmap v853: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-15T23:32:24.037 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:23 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1472579975' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:32:24.108 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T23:32:24.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:32:24.108 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-09-15T23:32:24.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:32:24.189 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-09-15T23:32:24.269 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-09-15T23:32:24.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:32:24.269 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-09-15T23:32:24.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:23 smithi067 ceph-mon[35665]: pgmap v853: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-15T23:32:24.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:23 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1472579975' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:32:24.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:23 smithi148 ceph-mon[35610]: pgmap v853: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-15T23:32:24.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:23 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1472579975' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-15T23:32:24.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-15T23:32:24.336 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link delete ceph-brx 2024-09-15T23:32:24.438 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-15T23:32:24.449 INFO:tasks.cephadm:Teardown begin 2024-09-15T23:32:24.449 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-15T23:32:24.449 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:32:24.709 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:25.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.179+0000 7f2861597640 1 -- 172.21.15.62:0/1277993528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100de0 msgr2=0x7f285c104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.179+0000 7f2861597640 1 --2- 172.21.15.62:0/1277993528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100de0 0x7f285c104e30 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f2850009ca0 tx=0x7f285002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.179+0000 7f2861597640 1 -- 172.21.15.62:0/1277993528 shutdown_connections 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.179+0000 7f2861597640 1 --2- 172.21.15.62:0/1277993528 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c105560 0x7f285c10d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.179+0000 7f2861597640 1 --2- 172.21.15.62:0/1277993528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100de0 0x7f285c104e30 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.179+0000 7f2861597640 1 --2- 172.21.15.62:0/1277993528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f285c100430 0x7f285c100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.179+0000 7f2861597640 1 -- 172.21.15.62:0/1277993528 >> 172.21.15.62:0/1277993528 conn(0x7f285c077230 msgr2=0x7f285c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.179+0000 7f2861597640 1 -- 172.21.15.62:0/1277993528 shutdown_connections 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.179+0000 7f2861597640 1 -- 172.21.15.62:0/1277993528 wait complete. 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f2861597640 1 Processor -- start 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f2861597640 1 -- start start 2024-09-15T23:32:25.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f2861597640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100430 0x7f285c1962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f2861597640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f285c100de0 0x7f285c196800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f2861597640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c105560 0x7f285c19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f2861597640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f285c10fd40 con 0x7f285c100430 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f2861597640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f285c10fbc0 con 0x7f285c105560 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f2861597640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f285c10fec0 con 0x7f285c100de0 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100430 0x7f285c1962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100430 0x7f285c1962c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48908/0 (socket says 172.21.15.62:48908) 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285affd640 1 -- 172.21.15.62:0/377846907 learned_addr learned my addr 172.21.15.62:0/377846907 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285b7fe640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c105560 0x7f285c19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285affd640 1 -- 172.21.15.62:0/377846907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f285c100de0 msgr2=0x7f285c196800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285affd640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f285c100de0 0x7f285c196800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285affd640 1 -- 172.21.15.62:0/377846907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c105560 msgr2=0x7f285c19ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285affd640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c105560 0x7f285c19ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285affd640 1 -- 172.21.15.62:0/377846907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c19b270 con 0x7f285c100430 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.180+0000 7f285b7fe640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c105560 0x7f285c19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.181+0000 7f285affd640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100430 0x7f285c1962c0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f285400c970 tx=0x7f285400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.181+0000 7f2843fff640 1 -- 172.21.15.62:0/377846907 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854007bb0 con 0x7f285c100430 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.181+0000 7f2843fff640 1 -- 172.21.15.62:0/377846907 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2854007d50 con 0x7f285c100430 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.181+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285c19b560 con 0x7f285c100430 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.181+0000 7f2843fff640 1 -- 172.21.15.62:0/377846907 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285400f450 con 0x7f285c100430 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.181+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f285c1a2e40 con 0x7f285c100430 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.182+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f285c100810 con 0x7f285c100430 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.183+0000 7f2843fff640 1 -- 172.21.15.62:0/377846907 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f2854005020 con 0x7f285c100430 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.184+0000 7f2843fff640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28440a3240 0x7f28440a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.184+0000 7f285a7fc640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28440a3240 0x7f28440a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.184+0000 7f2843fff640 1 -- 172.21.15.62:0/377846907 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f28540cfb20 con 0x7f285c100430 2024-09-15T23:32:25.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.184+0000 7f285a7fc640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28440a3240 0x7f28440a5700 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2850004af0 tx=0x7f2850009360 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:25.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.188+0000 7f2843fff640 1 -- 172.21.15.62:0/377846907 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f2854014030 con 0x7f285c100430 2024-09-15T23:32:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.398+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f285c1087a0 con 0x7f285c100430 2024-09-15T23:32:25.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.399+0000 7f2843fff640 1 -- 172.21.15.62:0/377846907 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11910 (secure 0 0 0) 0x7f285409c4c0 con 0x7f285c100430 2024-09-15T23:32:25.401 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:32:25.401 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":40,"btime":"2024-09-15T23:32:05:782827+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:32:05.309548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:32:25.401 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 40 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28440a3240 msgr2=0x7f28440a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28440a3240 0x7f28440a5700 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2850004af0 tx=0x7f2850009360 comp rx=0 tx=0).stop 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100430 msgr2=0x7f285c1962c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100430 0x7f285c1962c0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f285400c970 tx=0x7f285400ce30 comp rx=0 tx=0).stop 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 shutdown_connections 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f28440a3240 0x7f28440a5700 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f285c105560 0x7f285c19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f285c100de0 0x7f285c196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 --2- 172.21.15.62:0/377846907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c100430 0x7f285c1962c0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.402+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 >> 172.21.15.62:0/377846907 conn(0x7f285c077230 msgr2=0x7f285c0fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:25.404 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.403+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 shutdown_connections 2024-09-15T23:32:25.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:25.403+0000 7f2861597640 1 -- 172.21.15.62:0/377846907 wait complete. 2024-09-15T23:32:25.967 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:25 smithi062 ceph-mon[30949]: pgmap v854: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-15T23:32:25.967 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:25 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/377846907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:25.998 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs dump --format=json 2024-09-15T23:32:26.257 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:26.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:25 smithi067 ceph-mon[35665]: pgmap v854: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-15T23:32:26.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:25 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/377846907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:26.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:25 smithi148 ceph-mon[35610]: pgmap v854: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 853 B/s wr, 6 op/s 2024-09-15T23:32:26.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:25 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/377846907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.711+0000 7f8a6e60c640 1 -- 172.21.15.62:0/3997811585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6810e4d0 msgr2=0x7f8a681108e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.711+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/3997811585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6810e4d0 0x7f8a681108e0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009ca0 tx=0x7f8a6002a8c0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.711+0000 7f8a6e60c640 1 -- 172.21.15.62:0/3997811585 shutdown_connections 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.711+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/3997811585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a6810e4d0 0x7f8a681108e0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.711+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/3997811585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a68101360 0x7f8a6810df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.711+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/3997811585 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a681009b0 0x7f8a68100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.711+0000 7f8a6e60c640 1 -- 172.21.15.62:0/3997811585 >> 172.21.15.62:0/3997811585 conn(0x7f8a680fc7d0 msgr2=0x7f8a680febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.711+0000 7f8a6e60c640 1 -- 172.21.15.62:0/3997811585 shutdown_connections 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.711+0000 7f8a6e60c640 1 -- 172.21.15.62:0/3997811585 wait complete. 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a6e60c640 1 Processor -- start 2024-09-15T23:32:26.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a6e60c640 1 -- start start 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a6e60c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a681009b0 0x7f8a681a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a6e60c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a68101360 0x7f8a681a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a6e60c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6810e4d0 0x7f8a6819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a6e60c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681142a0 con 0x7f8a681009b0 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a6e60c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a68114120 con 0x7f8a6810e4d0 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a6e60c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f8a68114420 con 0x7f8a68101360 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a67fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a681009b0 0x7f8a681a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a67fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a681009b0 0x7f8a681a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48940/0 (socket says 172.21.15.62:48940) 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.712+0000 7f8a67fff640 1 -- 172.21.15.62:0/1511284274 learned_addr learned my addr 172.21.15.62:0/1511284274 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a677fe640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a68101360 0x7f8a681a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a6cba9640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6810e4d0 0x7f8a6819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a67fff640 1 -- 172.21.15.62:0/1511284274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a68101360 msgr2=0x7f8a681a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a67fff640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a68101360 0x7f8a681a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a67fff640 1 -- 172.21.15.62:0/1511284274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6810e4d0 msgr2=0x7f8a6819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a67fff640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6810e4d0 0x7f8a6819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a67fff640 1 -- 172.21.15.62:0/1511284274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6819ace0 con 0x7f8a681009b0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a6cba9640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6810e4d0 0x7f8a6819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a67fff640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a681009b0 0x7f8a681a03f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800d770 tx=0x7f8a5800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a657fa640 1 -- 172.21.15.62:0/1511284274 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58017070 con 0x7f8a681009b0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6819afd0 con 0x7f8a681009b0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a657fa640 1 -- 172.21.15.62:0/1511284274 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a58005500 con 0x7f8a681009b0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a657fa640 1 -- 172.21.15.62:0/1511284274 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58004e00 con 0x7f8a681009b0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.713+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a681a72c0 con 0x7f8a681009b0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.714+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a68100d90 con 0x7f8a681009b0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.715+0000 7f8a657fa640 1 -- 172.21.15.62:0/1511284274 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f8a58004fa0 con 0x7f8a681009b0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.716+0000 7f8a657fa640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8a500a3240 0x7f8a500a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.716+0000 7f8a677fe640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8a500a3240 0x7f8a500a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.716+0000 7f8a657fa640 1 -- 172.21.15.62:0/1511284274 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7f8a580cf7d0 con 0x7f8a681009b0 2024-09-15T23:32:26.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.717+0000 7f8a677fe640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8a500a3240 0x7f8a500a5700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8a68102340 tx=0x7f8a5c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:26.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.721+0000 7f8a657fa640 1 -- 172.21.15.62:0/1511284274 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f8a5809c110 con 0x7f8a681009b0 2024-09-15T23:32:26.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.936+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a68109660 con 0x7f8a681009b0 2024-09-15T23:32:26.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.937+0000 7f8a657fa640 1 -- 172.21.15.62:0/1511284274 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11910 (secure 0 0 0) 0x7f8a5809ba10 con 0x7f8a681009b0 2024-09-15T23:32:26.938 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:32:26.939 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":40,"btime":"2024-09-15T23:32:05:782827+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/3727539953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":3727539953},{"type":"v1","addr":"172.21.15.62:6835","nonce":3727539953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/862835307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":862835307},{"type":"v1","addr":"172.21.15.62:6837","nonce":862835307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14604,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/2389718073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":2389718073},{"type":"v1","addr":"172.21.15.62:6839","nonce":2389718073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14616,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6841/3484718619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6840","nonce":3484718619},{"type":"v1","addr":"172.21.15.62:6841","nonce":3484718619}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/2970459675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":2970459675},{"type":"v1","addr":"172.21.15.148:6833","nonce":2970459675}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24425,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/190864103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":190864103},{"type":"v1","addr":"172.21.15.67:6833","nonce":190864103}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-15T23:13:33.200855+0000","modified":"2024-09-15T23:32:05.309548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24497,"mds_1":24470,"mds_2":24466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6835/1239577031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":1239577031},{"type":"v1","addr":"172.21.15.148:6835","nonce":1239577031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.67:6835/3132447542","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":3132447542},{"type":"v1","addr":"172.21.15.67:6835","nonce":3132447542}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24454":{"gid":24454,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.148:6837/870913016","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":870913016},{"type":"v1","addr":"172.21.15.148:6837","nonce":870913016}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":15,"addr":"172.21.15.148:6839/2241485315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6838","nonce":2241485315},{"type":"v1","addr":"172.21.15.148:6839","nonce":2241485315}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6837/747270351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":747270351},{"type":"v1","addr":"172.21.15.67:6837","nonce":747270351}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24497":{"gid":24497,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6839/818760439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":818760439},{"type":"v1","addr":"172.21.15.67:6839","nonce":818760439}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24497,"qdb_cluster":[24497,24470,24466]},"id":1}]} 2024-09-15T23:32:26.939 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 40 2024-09-15T23:32:26.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8a500a3240 msgr2=0x7f8a500a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:26.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8a500a3240 0x7f8a500a5700 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8a68102340 tx=0x7f8a5c008040 comp rx=0 tx=0).stop 2024-09-15T23:32:26.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a681009b0 msgr2=0x7f8a681a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:26.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a681009b0 0x7f8a681a03f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800d770 tx=0x7f8a5800dc30 comp rx=0 tx=0).stop 2024-09-15T23:32:26.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 shutdown_connections 2024-09-15T23:32:26.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f8a500a3240 0x7f8a500a5700 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a6810e4d0 0x7f8a6819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f8a68101360 0x7f8a681a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 --2- 172.21.15.62:0/1511284274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a681009b0 0x7f8a681a03f0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:26.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 >> 172.21.15.62:0/1511284274 conn(0x7f8a680fc7d0 msgr2=0x7f8a680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:26.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 shutdown_connections 2024-09-15T23:32:26.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:26.940+0000 7f8a6e60c640 1 -- 172.21.15.62:0/1511284274 wait complete. 2024-09-15T23:32:27.702 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-15T23:32:27.965 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:27.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:27 smithi062 ceph-mon[30949]: pgmap v855: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.1 KiB/s wr, 6 op/s 2024-09-15T23:32:27.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:27 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1511284274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:28.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:27 smithi067 ceph-mon[35665]: pgmap v855: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.1 KiB/s wr, 6 op/s 2024-09-15T23:32:28.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:27 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1511284274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:28.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:27 smithi148 ceph-mon[35610]: pgmap v855: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.1 KiB/s wr, 6 op/s 2024-09-15T23:32:28.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:27 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1511284274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-15T23:32:28.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.613+0000 7fac4455a640 1 -- 172.21.15.62:0/2177427198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102ef0 msgr2=0x7fac3c10da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:28.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.613+0000 7fac4455a640 1 --2- 172.21.15.62:0/2177427198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102ef0 0x7fac3c10da40 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fac34009ca0 tx=0x7fac3402a950 comp rx=0 tx=0).stop 2024-09-15T23:32:28.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.613+0000 7fac4455a640 1 -- 172.21.15.62:0/2177427198 shutdown_connections 2024-09-15T23:32:28.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.613+0000 7fac4455a640 1 --2- 172.21.15.62:0/2177427198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102ef0 0x7fac3c10da40 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.613+0000 7fac4455a640 1 --2- 172.21.15.62:0/2177427198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac3c102570 0x7fac3c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.613+0000 7fac4455a640 1 --2- 172.21.15.62:0/2177427198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac3c100e20 0x7fac3c101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.613+0000 7fac4455a640 1 -- 172.21.15.62:0/2177427198 >> 172.21.15.62:0/2177427198 conn(0x7fac3c0fc7d0 msgr2=0x7fac3c0febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.613+0000 7fac4455a640 1 -- 172.21.15.62:0/2177427198 shutdown_connections 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.613+0000 7fac4455a640 1 -- 172.21.15.62:0/2177427198 wait complete. 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac4455a640 1 Processor -- start 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac4455a640 1 -- start start 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac4455a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac3c100e20 0x7fac3c0fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac4455a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102570 0x7fac3c0fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:28.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac4455a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac3c102ef0 0x7fac3c0fd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac4455a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c110f80 con 0x7fac3c102570 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac4455a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c110e00 con 0x7fac3c102ef0 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac4455a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c111100 con 0x7fac3c100e20 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac41af5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102570 0x7fac3c0fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac41af5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102570 0x7fac3c0fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48952/0 (socket says 172.21.15.62:48952) 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac41af5640 1 -- 172.21.15.62:0/358497865 learned_addr learned my addr 172.21.15.62:0/358497865 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac422f6640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac3c100e20 0x7fac3c0fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.614+0000 7fac42af7640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac3c102ef0 0x7fac3c0fd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac41af5640 1 -- 172.21.15.62:0/358497865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac3c100e20 msgr2=0x7fac3c0fcc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac41af5640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac3c100e20 0x7fac3c0fcc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:28.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac41af5640 1 -- 172.21.15.62:0/358497865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac3c102ef0 msgr2=0x7fac3c0fd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac41af5640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac3c102ef0 0x7fac3c0fd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac41af5640 1 -- 172.21.15.62:0/358497865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3c1a6980 con 0x7fac3c102570 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac41af5640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102570 0x7fac3c0fd1c0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00d950 tx=0x7fac2c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac337fe640 1 -- 172.21.15.62:0/358497865 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c017070 con 0x7fac3c102570 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac337fe640 1 -- 172.21.15.62:0/358497865 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac2c009c30 con 0x7fac3c102570 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac337fe640 1 -- 172.21.15.62:0/358497865 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00f6b0 con 0x7fac3c102570 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3c1a6c70 con 0x7fac3c102570 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.615+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac3c1a7160 con 0x7fac3c102570 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.616+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac3c101200 con 0x7fac3c102570 2024-09-15T23:32:28.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.617+0000 7fac337fe640 1 -- 172.21.15.62:0/358497865 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fac2c0109d0 con 0x7fac3c102570 2024-09-15T23:32:28.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.618+0000 7fac337fe640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fac280a32c0 0x7fac280a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:28.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.618+0000 7fac337fe640 1 -- 172.21.15.62:0/358497865 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fac2c0cf720 con 0x7fac3c102570 2024-09-15T23:32:28.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.618+0000 7fac422f6640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fac280a32c0 0x7fac280a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:28.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.619+0000 7fac422f6640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fac280a32c0 0x7fac280a5780 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fac38009bd0 tx=0x7fac38005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:28.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.621+0000 7fac337fe640 1 -- 172.21.15.62:0/358497865 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fac2c009de0 con 0x7fac3c102570 2024-09-15T23:32:28.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.802+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac3c10a0a0 con 0x7fac3c102570 2024-09-15T23:32:28.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.803+0000 7fac337fe640 1 -- 172.21.15.62:0/358497865 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24777 (secure 0 0 0) 0x7fac2c09baf0 con 0x7fac3c102570 2024-09-15T23:32:28.805 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-09-15T23:32:28.805 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":92,"fsid":"ee7f888e-73b6-11ef-bceb-c7b262605968","created":"2024-09-15T23:05:50.747034+0000","modified":"2024-09-15T23:15:42.605470+0000","last_up_change":"2024-09-15T23:12:42.334694+0000","last_in_change":"2024-09-15T23:12:19.285025+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T23:08:54.129326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-15T23:13:29.172210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-15T23:13:31.408446+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-15T23:13:37.961318+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f2a0a912-0888-41cb-a20e-745c2c97e6df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6803","nonce":2006762506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6805","nonce":2006762506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6809","nonce":2006762506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2006762506},{"type":"v1","addr":"172.21.15.62:6807","nonce":2006762506}]},"public_addr":"172.21.15.62:6803/2006762506","cluster_addr":"172.21.15.62:6805/2006762506","heartbeat_back_addr":"172.21.15.62:6809/2006762506","heartbeat_front_addr":"172.21.15.62:6807/2006762506","state":["exists","up"]},{"osd":1,"uuid":"ebfe65d3-c394-48c5-8063-078760acf36c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6801","nonce":1021530759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6803","nonce":1021530759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6807","nonce":1021530759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1021530759},{"type":"v1","addr":"172.21.15.67:6805","nonce":1021530759}]},"public_addr":"172.21.15.67:6801/1021530759","cluster_addr":"172.21.15.67:6803/1021530759","heartbeat_back_addr":"172.21.15.67:6807/1021530759","heartbeat_front_addr":"172.21.15.67:6805/1021530759","state":["exists","up"]},{"osd":2,"uuid":"51482605-4595-4d0b-b982-8613d1e73969","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6801","nonce":1833942254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6803","nonce":1833942254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6807","nonce":1833942254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":1833942254},{"type":"v1","addr":"172.21.15.148:6805","nonce":1833942254}]},"public_addr":"172.21.15.148:6801/1833942254","cluster_addr":"172.21.15.148:6803/1833942254","heartbeat_back_addr":"172.21.15.148:6807/1833942254","heartbeat_front_addr":"172.21.15.148:6805/1833942254","state":["exists","up"]},{"osd":3,"uuid":"7cbd02ce-8ea0-479b-812d-5540bfd4fbd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6811","nonce":2959993284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6813","nonce":2959993284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6817","nonce":2959993284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":2959993284},{"type":"v1","addr":"172.21.15.62:6815","nonce":2959993284}]},"public_addr":"172.21.15.62:6811/2959993284","cluster_addr":"172.21.15.62:6813/2959993284","heartbeat_back_addr":"172.21.15.62:6817/2959993284","heartbeat_front_addr":"172.21.15.62:6815/2959993284","state":["exists","up"]},{"osd":4,"uuid":"f6441578-9a31-46f2-a84e-0bfa41cd1c49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6809","nonce":3506283519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6811","nonce":3506283519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6815","nonce":3506283519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":3506283519},{"type":"v1","addr":"172.21.15.67:6813","nonce":3506283519}]},"public_addr":"172.21.15.67:6809/3506283519","cluster_addr":"172.21.15.67:6811/3506283519","heartbeat_back_addr":"172.21.15.67:6815/3506283519","heartbeat_front_addr":"172.21.15.67:6813/3506283519","state":["exists","up"]},{"osd":5,"uuid":"5d4f2bbc-07fb-40de-88f0-ecf68cdb2579","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6809","nonce":1859303200}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6810","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6811","nonce":1859303200}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6814","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6815","nonce":1859303200}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":1859303200},{"type":"v1","addr":"172.21.15.148:6813","nonce":1859303200}]},"public_addr":"172.21.15.148:6809/1859303200","cluster_addr":"172.21.15.148:6811/1859303200","heartbeat_back_addr":"172.21.15.148:6815/1859303200","heartbeat_front_addr":"172.21.15.148:6813/1859303200","state":["exists","up"]},{"osd":6,"uuid":"75629f41-6236-4d44-82fd-0f1d5e0df841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6819","nonce":2615321035}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6821","nonce":2615321035}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6825","nonce":2615321035}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":2615321035},{"type":"v1","addr":"172.21.15.62:6823","nonce":2615321035}]},"public_addr":"172.21.15.62:6819/2615321035","cluster_addr":"172.21.15.62:6821/2615321035","heartbeat_back_addr":"172.21.15.62:6825/2615321035","heartbeat_front_addr":"172.21.15.62:6823/2615321035","state":["exists","up"]},{"osd":7,"uuid":"f605b7f7-4cee-43c8-aae4-f197c5664f49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6817","nonce":2000628718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6819","nonce":2000628718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6823","nonce":2000628718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2000628718},{"type":"v1","addr":"172.21.15.67:6821","nonce":2000628718}]},"public_addr":"172.21.15.67:6817/2000628718","cluster_addr":"172.21.15.67:6819/2000628718","heartbeat_back_addr":"172.21.15.67:6823/2000628718","heartbeat_front_addr":"172.21.15.67:6821/2000628718","state":["exists","up"]},{"osd":8,"uuid":"8a81c2c5-9444-460b-b7a4-403598e61ca3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6817","nonce":381356822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6818","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6819","nonce":381356822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6822","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6823","nonce":381356822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":381356822},{"type":"v1","addr":"172.21.15.148:6821","nonce":381356822}]},"public_addr":"172.21.15.148:6817/381356822","cluster_addr":"172.21.15.148:6819/381356822","heartbeat_back_addr":"172.21.15.148:6823/381356822","heartbeat_front_addr":"172.21.15.148:6821/381356822","state":["exists","up"]},{"osd":9,"uuid":"2220ce09-96e7-4fcf-af2a-2c7fa4081679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6827","nonce":3552755977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6829","nonce":3552755977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6833","nonce":3552755977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3552755977},{"type":"v1","addr":"172.21.15.62:6831","nonce":3552755977}]},"public_addr":"172.21.15.62:6827/3552755977","cluster_addr":"172.21.15.62:6829/3552755977","heartbeat_back_addr":"172.21.15.62:6833/3552755977","heartbeat_front_addr":"172.21.15.62:6831/3552755977","state":["exists","up"]},{"osd":10,"uuid":"0bb9d76c-f64e-4792-b7c4-9e873a5837c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6825","nonce":3693654334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6827","nonce":3693654334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6831","nonce":3693654334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3693654334},{"type":"v1","addr":"172.21.15.67:6829","nonce":3693654334}]},"public_addr":"172.21.15.67:6825/3693654334","cluster_addr":"172.21.15.67:6827/3693654334","heartbeat_back_addr":"172.21.15.67:6831/3693654334","heartbeat_front_addr":"172.21.15.67:6829/3693654334","state":["exists","up"]},{"osd":11,"uuid":"c08c325c-1728-416e-893e-35954d5d8cd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6825","nonce":759280989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6827","nonce":759280989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6831","nonce":759280989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":759280989},{"type":"v1","addr":"172.21.15.148:6829","nonce":759280989}]},"public_addr":"172.21.15.148:6825/759280989","cluster_addr":"172.21.15.148:6827/759280989","heartbeat_back_addr":"172.21.15.148:6831/759280989","heartbeat_front_addr":"172.21.15.148:6829/759280989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:01.087036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:26.135737+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:08:50.950556+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:16.782199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:09:43.221589+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:08.988797+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:33.275943+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:10:58.869263+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:23.971802+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:11:50.310139+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:15.240115+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T23:12:41.275657+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.a","mappings":[{"from":5,"to":10}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:0/1303697831":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/574363632":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:6801/2614061737":"2024-09-16T23:06:39.607848+0000","172.21.15.62:6800/1277095060":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/50968825":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/2637169275":"2024-09-16T23:06:39.607848+0000","172.21.15.62:0/3526091028":"2024-09-16T23:06:15.815154+0000","172.21.15.62:0/3293859590":"2024-09-16T23:06:15.815154+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T23:32:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.805+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fac280a32c0 msgr2=0x7fac280a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.805+0000 7fac4455a640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fac280a32c0 0x7fac280a5780 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fac38009bd0 tx=0x7fac38005f90 comp rx=0 tx=0).stop 2024-09-15T23:32:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.805+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102570 msgr2=0x7fac3c0fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.805+0000 7fac4455a640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102570 0x7fac3c0fd1c0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00d950 tx=0x7fac2c00de10 comp rx=0 tx=0).stop 2024-09-15T23:32:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.805+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 shutdown_connections 2024-09-15T23:32:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.806+0000 7fac4455a640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fac280a32c0 0x7fac280a5780 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.806+0000 7fac4455a640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac3c102ef0 0x7fac3c0fd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:28.807 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.806+0000 7fac4455a640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac3c102570 0x7fac3c0fd1c0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.806+0000 7fac4455a640 1 --2- 172.21.15.62:0/358497865 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fac3c100e20 0x7fac3c0fcc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.806+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 >> 172.21.15.62:0/358497865 conn(0x7fac3c0fc7d0 msgr2=0x7fac3c06a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.806+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 shutdown_connections 2024-09-15T23:32:28.808 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:28.806+0000 7fac4455a640 1 -- 172.21.15.62:0/358497865 wait complete. 2024-09-15T23:32:29.117 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:28 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/358497865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:32:29.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:28 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/358497865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:32:29.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:28 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/358497865' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-15T23:32:29.678 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs fail cephfs 2024-09-15T23:32:29.943 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:29.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:29 smithi062 ceph-mon[30949]: pgmap v856: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-09-15T23:32:30.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:29 smithi067 ceph-mon[35665]: pgmap v856: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-09-15T23:32:30.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:29 smithi148 ceph-mon[35610]: pgmap v856: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 8 op/s 2024-09-15T23:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.437+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/2218998678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0101de0 msgr2=0x7fb0c01021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.437+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/2218998678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0101de0 0x7fb0c01021c0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009900 tx=0x7fb0b402a3f0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.437+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/2218998678 shutdown_connections 2024-09-15T23:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.437+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/2218998678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0106f10 0x7fb0c010ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.437+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/2218998678 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0c0102790 0x7fb0c01067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.437+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/2218998678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0101de0 0x7fb0c01021c0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.437+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/2218998678 >> 172.21.15.62:0/2218998678 conn(0x7fb0c00fd8c0 msgr2=0x7fb0c00ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.437+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/2218998678 shutdown_connections 2024-09-15T23:32:30.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.437+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/2218998678 wait complete. 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c6ee4640 1 Processor -- start 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c6ee4640 1 -- start start 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c6ee4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0101de0 0x7fb0c01982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c6ee4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0c0102790 0x7fb0c0198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c6ee4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0106f10 0x7fb0c019cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c6ee4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c0111f70 con 0x7fb0c0106f10 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c6ee4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c0111df0 con 0x7fb0c0101de0 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c6ee4640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c01120f0 con 0x7fb0c0102790 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c5481640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0106f10 0x7fb0c019cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c5481640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0106f10 0x7fb0c019cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48968/0 (socket says 172.21.15.62:48968) 2024-09-15T23:32:30.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.438+0000 7fb0c5481640 1 -- 172.21.15.62:0/1565267000 learned_addr learned my addr 172.21.15.62:0/1565267000 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0bffff640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0c0102790 0x7fb0c0198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c4c80640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0101de0 0x7fb0c01982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c5481640 1 -- 172.21.15.62:0/1565267000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0c0102790 msgr2=0x7fb0c0198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c5481640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0c0102790 0x7fb0c0198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c5481640 1 -- 172.21.15.62:0/1565267000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0101de0 msgr2=0x7fb0c01982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c5481640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0101de0 0x7fb0c01982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c5481640 1 -- 172.21.15.62:0/1565267000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c019d2a0 con 0x7fb0c0106f10 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c4c80640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0101de0 0x7fb0c01982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c5481640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0106f10 0x7fb0c019cbc0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800cce0 tx=0x7fb0b800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0bdffb640 1 -- 172.21.15.62:0/1565267000 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b800f450 con 0x7fb0c0106f10 2024-09-15T23:32:30.450 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0bdffb640 1 -- 172.21.15.62:0/1565267000 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b800fad0 con 0x7fb0c0106f10 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c019d590 con 0x7fb0c0106f10 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0bdffb640 1 -- 172.21.15.62:0/1565267000 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8018c10 con 0x7fb0c0106f10 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.439+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c01a4e70 con 0x7fb0c0106f10 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.440+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c01021c0 con 0x7fb0c0106f10 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.441+0000 7fb0bdffb640 1 -- 172.21.15.62:0/1565267000 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb0b800f5f0 con 0x7fb0c0106f10 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.442+0000 7fb0bdffb640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0b00a3240 0x7fb0b00a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.442+0000 7fb0c4c80640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0b00a3240 0x7fb0b00a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.442+0000 7fb0bdffb640 1 -- 172.21.15.62:0/1565267000 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9129+0+0 (secure 0 0 0) 0x7fb0b80d0340 con 0x7fb0c0106f10 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.443+0000 7fb0c4c80640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0b00a3240 0x7fb0b00a5700 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4000c00 tx=0x7fb0b4032d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:30.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.446+0000 7fb0bdffb640 1 -- 172.21.15.62:0/1565267000 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb0b8014030 con 0x7fb0c0106f10 2024-09-15T23:32:30.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.660+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb0c01a5110 con 0x7fb0c0106f10 2024-09-15T23:32:30.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.662+0000 7fb0bdffb640 1 -- 172.21.15.62:0/1565267000 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) ==== 161+0+0 (secure 0 0 0) 0x7fb0b809cd80 con 0x7fb0c0106f10 2024-09-15T23:32:30.663 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-15T23:32:30.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0b00a3240 msgr2=0x7fb0b00a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:30.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0b00a3240 0x7fb0b00a5700 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4000c00 tx=0x7fb0b4032d50 comp rx=0 tx=0).stop 2024-09-15T23:32:30.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0106f10 msgr2=0x7fb0c019cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:30.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0106f10 0x7fb0c019cbc0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800cce0 tx=0x7fb0b800a620 comp rx=0 tx=0).stop 2024-09-15T23:32:30.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 shutdown_connections 2024-09-15T23:32:30.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb0b00a3240 0x7fb0b00a5700 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0c0106f10 0x7fb0c019cbc0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0c0102790 0x7fb0c0198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 --2- 172.21.15.62:0/1565267000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0101de0 0x7fb0c01982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:30.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 >> 172.21.15.62:0/1565267000 conn(0x7fb0c00fd8c0 msgr2=0x7fb0c00ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:30.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 shutdown_connections 2024-09-15T23:32:30.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:30.665+0000 7fb0c6ee4640 1 -- 172.21.15.62:0/1565267000 wait complete. 2024-09-15T23:32:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:30 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a[30945]: 2024-09-15T23:32:30.660+0000 7f19de132640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T23:32:30.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:30 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1565267000' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-15T23:32:30.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:30 smithi062 ceph-mon[30949]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-15T23:32:30.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:30 smithi062 ceph-mon[30949]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T23:32:30.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:30 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1565267000' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-15T23:32:30.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:30 smithi062 ceph-mon[30949]: osdmap e93: 12 total, 12 up, 12 in 2024-09-15T23:32:30.981 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:30 smithi062 ceph-mon[30949]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-15T23:32:31.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1565267000' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-15T23:32:31.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-mon[35665]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-15T23:32:31.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-mon[35665]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T23:32:31.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1565267000' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-15T23:32:31.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-mon[35665]: osdmap e93: 12 total, 12 up, 12 in 2024-09-15T23:32:31.025 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-mon[35665]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-15T23:32:31.025 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: ignoring --setuser ceph since I am not root 2024-09-15T23:32:31.025 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: ignoring --setgroup ceph since I am not root 2024-09-15T23:32:31.025 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-mds[65714]: -- 172.21.15.67:0/2760467654 <== mon.2 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ec12b0d860 con 0x55ec12aeb400 2024-09-15T23:32:31.025 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: starting mds.b at 2024-09-15T23:32:31.025 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e[65099]: ignoring --setuser ceph since I am not root 2024-09-15T23:32:31.026 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e[65099]: ignoring --setgroup ceph since I am not root 2024-09-15T23:32:31.026 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-mds[65103]: -- 172.21.15.67:0/969115827 <== mon.1 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dbfa50dc20 con 0x55dbfa4eb400 2024-09-15T23:32:31.026 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e[65099]: starting mds.e at 2024-09-15T23:32:31.026 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h[65398]: ignoring --setuser ceph since I am not root 2024-09-15T23:32:31.026 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h[65398]: ignoring --setgroup ceph since I am not root 2024-09-15T23:32:31.026 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-mds[65402]: -- 172.21.15.67:0/2296256877 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b2b39bd860 con 0x55b2b399a800 2024-09-15T23:32:31.026 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:32:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h[65398]: starting mds.h at 2024-09-15T23:32:31.289 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f[66248]: ignoring --setuser ceph since I am not root 2024-09-15T23:32:31.289 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f[66248]: ignoring --setgroup ceph since I am not root 2024-09-15T23:32:31.290 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-mds[66252]: -- 172.21.15.148:0/230101933 <== mon.1 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e58aa15c20 con 0x55e58a9f3400 2024-09-15T23:32:31.290 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f[66248]: starting mds.f at 2024-09-15T23:32:31.290 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i[66848]: ignoring --setuser ceph since I am not root 2024-09-15T23:32:31.290 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i[66848]: ignoring --setgroup ceph since I am not root 2024-09-15T23:32:31.290 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-mds[66852]: -- 172.21.15.148:0/3933659197 <== mon.1 v2:172.21.15.148:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555b42ba5860 con 0x555b42b83000 2024-09-15T23:32:31.290 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i[66848]: starting mds.i at 2024-09-15T23:32:31.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1565267000' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-09-15T23:32:31.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-mon[35610]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-15T23:32:31.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-mon[35610]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-15T23:32:31.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1565267000' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-09-15T23:32:31.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-mon[35610]: osdmap e93: 12 total, 12 up, 12 in 2024-09-15T23:32:31.291 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-mon[35610]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-09-15T23:32:31.291 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c[66550]: ignoring --setuser ceph since I am not root 2024-09-15T23:32:31.291 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c[66550]: ignoring --setgroup ceph since I am not root 2024-09-15T23:32:31.291 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-mds[66554]: -- 172.21.15.148:0/3775334710 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed7a481860 con 0x55ed7a45e800 2024-09-15T23:32:31.291 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:32:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c[66550]: starting mds.c at 2024-09-15T23:32:31.330 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-09-15T23:32:31.588 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:31.972 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: pgmap v857: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.67:6834/1761796122,v1:172.21.15.67:6835/1761796122] up:boot 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.67:6837/2127831380,v1:172.21.15.67:6839/2127831380] up:boot 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.67:6836/940222280,v1:172.21.15.67:6838/940222280] up:boot 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.148:6834/3504658501,v1:172.21.15.148:6835/3504658501] up:boot 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.148:6836/3529798347,v1:172.21.15.148:6838/3529798347] up:boot 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: mds.? [v2:172.21.15.148:6837/3624575991,v1:172.21.15.148:6839/3624575991] up:boot 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:32:31.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:31 smithi062 ceph-mon[30949]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:32:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.055+0000 7fb359144640 1 -- 172.21.15.62:0/3630683862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354106a80 msgr2=0x7fb354106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.055+0000 7fb359144640 1 --2- 172.21.15.62:0/3630683862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354106a80 0x7fb354106e60 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fb34c009cd0 tx=0x7fb34c02a8e0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.055+0000 7fb359144640 1 -- 172.21.15.62:0/3630683862 shutdown_connections 2024-09-15T23:32:32.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.055+0000 7fb359144640 1 --2- 172.21.15.62:0/3630683862 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb354101420 0x7fb35410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.055+0000 7fb359144640 1 --2- 172.21.15.62:0/3630683862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354100a80 0x7fb354100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.055+0000 7fb359144640 1 --2- 172.21.15.62:0/3630683862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354106a80 0x7fb354106e60 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.055+0000 7fb359144640 1 -- 172.21.15.62:0/3630683862 >> 172.21.15.62:0/3630683862 conn(0x7fb3540fc7a0 msgr2=0x7fb3540febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.055+0000 7fb359144640 1 -- 172.21.15.62:0/3630683862 shutdown_connections 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.055+0000 7fb359144640 1 -- 172.21.15.62:0/3630683862 wait complete. 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb359144640 1 Processor -- start 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb359144640 1 -- start start 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb359144640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354100a80 0x7fb354197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb359144640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb354101420 0x7fb354197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb359144640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354106a80 0x7fb354197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb359144640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb354110e80 con 0x7fb354100a80 2024-09-15T23:32:32.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb359144640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb354110d00 con 0x7fb354106a80 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb359144640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb354111000 con 0x7fb354101420 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb352d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354100a80 0x7fb354197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb352d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354100a80 0x7fb354197020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49000/0 (socket says 172.21.15.62:49000) 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb352d9d640 1 -- 172.21.15.62:0/574567171 learned_addr learned my addr 172.21.15.62:0/574567171 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb35259c640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb354101420 0x7fb354197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb35359e640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354106a80 0x7fb354197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb352d9d640 1 -- 172.21.15.62:0/574567171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb354101420 msgr2=0x7fb354197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb352d9d640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb354101420 0x7fb354197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb352d9d640 1 -- 172.21.15.62:0/574567171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354106a80 msgr2=0x7fb354197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.056+0000 7fb352d9d640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354106a80 0x7fb354197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.057+0000 7fb352d9d640 1 -- 172.21.15.62:0/574567171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3541a6890 con 0x7fb354100a80 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.057+0000 7fb35359e640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354106a80 0x7fb354197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:32.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.057+0000 7fb352d9d640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354100a80 0x7fb354197020 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fb34c02ae00 tx=0x7fb34c003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.057+0000 7fb34bfff640 1 -- 172.21.15.62:0/574567171 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c004040 con 0x7fb354100a80 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.057+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3541a6b20 con 0x7fb354100a80 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.057+0000 7fb34bfff640 1 -- 172.21.15.62:0/574567171 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb34c0046c0 con 0x7fb354100a80 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.057+0000 7fb34bfff640 1 -- 172.21.15.62:0/574567171 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34c046480 con 0x7fb354100a80 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.057+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3541a7060 con 0x7fb354100a80 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.058+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb354100ee0 con 0x7fb354100a80 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.059+0000 7fb34bfff640 1 -- 172.21.15.62:0/574567171 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fb34c0305a0 con 0x7fb354100a80 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.060+0000 7fb34bfff640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3400a3240 0x7fb3400a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.061+0000 7fb35259c640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3400a3240 0x7fb3400a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.061+0000 7fb34bfff640 1 -- 172.21.15.62:0/574567171 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9387+0+0 (secure 0 0 0) 0x7fb34c02e030 con 0x7fb354100a80 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.061+0000 7fb35259c640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3400a3240 0x7fb3400a5700 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb354199180 tx=0x7fb33c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:32.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.064+0000 7fb34bfff640 1 -- 172.21.15.62:0/574567171 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fb34c04b060 con 0x7fb354100a80 2024-09-15T23:32:32.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: pgmap v857: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2024-09-15T23:32:32.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.67:6834/1761796122,v1:172.21.15.67:6835/1761796122] up:boot 2024-09-15T23:32:32.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.67:6837/2127831380,v1:172.21.15.67:6839/2127831380] up:boot 2024-09-15T23:32:32.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.67:6836/940222280,v1:172.21.15.67:6838/940222280] up:boot 2024-09-15T23:32:32.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.148:6834/3504658501,v1:172.21.15.148:6835/3504658501] up:boot 2024-09-15T23:32:32.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.148:6836/3529798347,v1:172.21.15.148:6838/3529798347] up:boot 2024-09-15T23:32:32.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: mds.? [v2:172.21.15.148:6837/3624575991,v1:172.21.15.148:6839/3624575991] up:boot 2024-09-15T23:32:32.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-15T23:32:32.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:32:32.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:32:32.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:32:32.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:32:32.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:32:32.275 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:31 smithi067 ceph-mon[35665]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:32:32.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.276+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb35402d7c0 con 0x7fb354100a80 2024-09-15T23:32:32.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: pgmap v857: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2024-09-15T23:32:32.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.67:6834/1761796122,v1:172.21.15.67:6835/1761796122] up:boot 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.67:6837/2127831380,v1:172.21.15.67:6839/2127831380] up:boot 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.67:6836/940222280,v1:172.21.15.67:6838/940222280] up:boot 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.148:6834/3504658501,v1:172.21.15.148:6835/3504658501] up:boot 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.148:6836/3529798347,v1:172.21.15.148:6838/3529798347] up:boot 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: mds.? [v2:172.21.15.148:6837/3624575991,v1:172.21.15.148:6839/3624575991] up:boot 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-09-15T23:32:32.290 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:31 smithi148 ceph-mon[35610]: from='mgr.14150 172.21.15.62:0/2350811718' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-09-15T23:32:32.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.847+0000 7fb34bfff640 1 -- 172.21.15.62:0/574567171 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) ==== 104+0+0 (secure 0 0 0) 0x7fb34c030890 con 0x7fb354100a80 2024-09-15T23:32:32.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3400a3240 msgr2=0x7fb3400a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:32.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3400a3240 0x7fb3400a5700 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb354199180 tx=0x7fb33c00b390 comp rx=0 tx=0).stop 2024-09-15T23:32:32.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354100a80 msgr2=0x7fb354197020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:32.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354100a80 0x7fb354197020 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fb34c02ae00 tx=0x7fb34c003970 comp rx=0 tx=0).stop 2024-09-15T23:32:32.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 shutdown_connections 2024-09-15T23:32:32.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fb3400a3240 0x7fb3400a5700 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb354106a80 0x7fb354197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb354101420 0x7fb354197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 --2- 172.21.15.62:0/574567171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb354100a80 0x7fb354197020 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:32.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 >> 172.21.15.62:0/574567171 conn(0x7fb3540fc7a0 msgr2=0x7fb3540fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:32.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 shutdown_connections 2024-09-15T23:32:32.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:32.850+0000 7fb359144640 1 -- 172.21.15.62:0/574567171 wait complete. 2024-09-15T23:32:33.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:32 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/574567171' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:32:33.160 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:32 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/574567171' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:32:33.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:32 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/574567171' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-09-15T23:32:33.686 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-15T23:32:33.948 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:33 smithi062 ceph-mon[30949]: pgmap v859: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-09-15T23:32:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:33 smithi062 ceph-mon[30949]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-15T23:32:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:33 smithi062 ceph-mon[30949]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-15T23:32:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:33 smithi062 ceph-mon[30949]: Cluster is now healthy 2024-09-15T23:32:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:33 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/574567171' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:32:33.980 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:33 smithi062 ceph-mon[30949]: fsmap 12 up:standby 2024-09-15T23:32:34.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:33 smithi067 ceph-mon[35665]: pgmap v859: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-09-15T23:32:34.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:33 smithi067 ceph-mon[35665]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-15T23:32:34.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:33 smithi067 ceph-mon[35665]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-15T23:32:34.277 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:33 smithi067 ceph-mon[35665]: Cluster is now healthy 2024-09-15T23:32:34.278 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:33 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/574567171' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:32:34.278 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:33 smithi067 ceph-mon[35665]: fsmap 12 up:standby 2024-09-15T23:32:34.309 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:33 smithi148 ceph-mon[35610]: pgmap v859: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-09-15T23:32:34.309 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:33 smithi148 ceph-mon[35610]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-15T23:32:34.309 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:33 smithi148 ceph-mon[35610]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-15T23:32:34.309 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:33 smithi148 ceph-mon[35610]: Cluster is now healthy 2024-09-15T23:32:34.309 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:33 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/574567171' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-09-15T23:32:34.309 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:33 smithi148 ceph-mon[35610]: fsmap 12 up:standby 2024-09-15T23:32:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 -- 172.21.15.62:0/2004273508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424100a80 msgr2=0x7fe424100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2004273508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424100a80 0x7fe424100ee0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fe40c009930 tx=0x7fe40c02a420 comp rx=0 tx=0).stop 2024-09-15T23:32:34.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 -- 172.21.15.62:0/2004273508 shutdown_connections 2024-09-15T23:32:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2004273508 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe424101420 0x7fe42410d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2004273508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424100a80 0x7fe424100ee0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2004273508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424106a80 0x7fe424106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 -- 172.21.15.62:0/2004273508 >> 172.21.15.62:0/2004273508 conn(0x7fe4240fc7a0 msgr2=0x7fe4240febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 -- 172.21.15.62:0/2004273508 shutdown_connections 2024-09-15T23:32:34.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 -- 172.21.15.62:0/2004273508 wait complete. 2024-09-15T23:32:34.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 Processor -- start 2024-09-15T23:32:34.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.417+0000 7fe42a2de640 1 -- start start 2024-09-15T23:32:34.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42a2de640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe424100a80 0x7fe4240fcd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42a2de640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424101420 0x7fe4240fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42a2de640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424106a80 0x7fe4240fd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42a2de640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe424110ea0 con 0x7fe424106a80 2024-09-15T23:32:34.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42a2de640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe424110d20 con 0x7fe424101420 2024-09-15T23:32:34.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42a2de640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe424111020 con 0x7fe424100a80 2024-09-15T23:32:34.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42887b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424106a80 0x7fe4240fd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42887b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424106a80 0x7fe4240fd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50834/0 (socket says 172.21.15.62:50834) 2024-09-15T23:32:34.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42887b640 1 -- 172.21.15.62:0/2626922881 learned_addr learned my addr 172.21.15.62:0/2626922881 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:34.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe4237fe640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424101420 0x7fe4240fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe423fff640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe424100a80 0x7fe4240fcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42887b640 1 -- 172.21.15.62:0/2626922881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe424100a80 msgr2=0x7fe4240fcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42887b640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe424100a80 0x7fe4240fcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42887b640 1 -- 172.21.15.62:0/2626922881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424101420 msgr2=0x7fe4240fd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42887b640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424101420 0x7fe4240fd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.418+0000 7fe42887b640 1 -- 172.21.15.62:0/2626922881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4241a6920 con 0x7fe424106a80 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.419+0000 7fe42887b640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424106a80 0x7fe4240fd810 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00cce0 tx=0x7fe41c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.419+0000 7fe4217fa640 1 -- 172.21.15.62:0/2626922881 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c00f450 con 0x7fe424106a80 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.419+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4241a6c10 con 0x7fe424106a80 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.419+0000 7fe4217fa640 1 -- 172.21.15.62:0/2626922881 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe41c00fad0 con 0x7fe424106a80 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.419+0000 7fe4217fa640 1 -- 172.21.15.62:0/2626922881 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c018c10 con 0x7fe424106a80 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.419+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4241a7120 con 0x7fe424106a80 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.420+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe424100ee0 con 0x7fe424106a80 2024-09-15T23:32:34.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.421+0000 7fe4217fa640 1 -- 172.21.15.62:0/2626922881 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7fe41c00f5f0 con 0x7fe424106a80 2024-09-15T23:32:34.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.422+0000 7fe4217fa640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4100a32c0 0x7fe4100a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:34.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.422+0000 7fe423fff640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4100a32c0 0x7fe4100a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:34.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.422+0000 7fe4217fa640 1 -- 172.21.15.62:0/2626922881 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9387+0+0 (secure 0 0 0) 0x7fe41c0d0400 con 0x7fe424106a80 2024-09-15T23:32:34.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.422+0000 7fe423fff640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4100a32c0 0x7fe4100a5780 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe414009890 tx=0x7fe414008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:34.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.426+0000 7fe4217fa640 1 -- 172.21.15.62:0/2626922881 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7fe41c014030 con 0x7fe424106a80 2024-09-15T23:32:34.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.602+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe42410a020 con 0x7fe424106a80 2024-09-15T23:32:34.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.847+0000 7fe4217fa640 1 -- 172.21.15.62:0/2626922881 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v94) ==== 181+0+0 (secure 0 0 0) 0x7fe41c09cca0 con 0x7fe424106a80 2024-09-15T23:32:34.848 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4100a32c0 msgr2=0x7fe4100a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4100a32c0 0x7fe4100a5780 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe414009890 tx=0x7fe414008040 comp rx=0 tx=0).stop 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424106a80 msgr2=0x7fe4240fd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424106a80 0x7fe4240fd810 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fe41c00cce0 tx=0x7fe41c00a620 comp rx=0 tx=0).stop 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 shutdown_connections 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7fe4100a32c0 0x7fe4100a5780 secure :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe414009890 tx=0x7fe414008040 comp rx=0 tx=0).stop 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe424106a80 0x7fe4240fd810 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe424101420 0x7fe4240fd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 --2- 172.21.15.62:0/2626922881 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe424100a80 0x7fe4240fcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.850+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 >> 172.21.15.62:0/2626922881 conn(0x7fe4240fc7a0 msgr2=0x7fe42406a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:34.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.851+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 shutdown_connections 2024-09-15T23:32:34.862 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:34.851+0000 7fe42a2de640 1 -- 172.21.15.62:0/2626922881 wait complete. 2024-09-15T23:32:35.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:34 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2626922881' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:32:35.208 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:34 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2626922881' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:32:35.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:34 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2626922881' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:32:35.507 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-15T23:32:35.764 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:36.140 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:35 smithi062 ceph-mon[30949]: pgmap v860: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-09-15T23:32:36.141 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:35 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/2626922881' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:32:36.141 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:35 smithi062 ceph-mon[30949]: osdmap e94: 12 total, 12 up, 12 in 2024-09-15T23:32:36.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 -- 172.21.15.62:0/3716116324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278101420 msgr2=0x7ff27810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:36.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3716116324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278101420 0x7ff27810d950 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7ff270009cd0 tx=0x7ff27002a8e0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 -- 172.21.15.62:0/3716116324 shutdown_connections 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3716116324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278101420 0x7ff27810d950 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3716116324 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff278100a80 0x7ff278100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3716116324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278106a80 0x7ff278106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 -- 172.21.15.62:0/3716116324 >> 172.21.15.62:0/3716116324 conn(0x7ff2780fc7a0 msgr2=0x7ff2780febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 -- 172.21.15.62:0/3716116324 shutdown_connections 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 -- 172.21.15.62:0/3716116324 wait complete. 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 Processor -- start 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.224+0000 7ff27edf2640 1 -- start start 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27edf2640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff278100a80 0x7ff27819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27edf2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278101420 0x7ff27819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27edf2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278106a80 0x7ff27819ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27edf2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff278110e80 con 0x7ff278106a80 2024-09-15T23:32:36.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27edf2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff278110d00 con 0x7ff278101420 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27edf2640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff278111000 con 0x7ff278100a80 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27d38f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278106a80 0x7ff27819ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27d38f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278106a80 0x7ff27819ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50856/0 (socket says 172.21.15.62:50856) 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27d38f640 1 -- 172.21.15.62:0/3863391469 learned_addr learned my addr 172.21.15.62:0/3863391469 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27cb8e640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff278100a80 0x7ff27819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff277fff640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278101420 0x7ff27819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27d38f640 1 -- 172.21.15.62:0/3863391469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff278100a80 msgr2=0x7ff27819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27d38f640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff278100a80 0x7ff27819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27d38f640 1 -- 172.21.15.62:0/3863391469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278101420 msgr2=0x7ff27819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27d38f640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278101420 0x7ff27819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27d38f640 1 -- 172.21.15.62:0/3863391469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27819f540 con 0x7ff278106a80 2024-09-15T23:32:36.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff277fff640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278101420 0x7ff27819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.225+0000 7ff27d38f640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278106a80 0x7ff27819ee60 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7ff270006010 tx=0x7ff270003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.226+0000 7ff275ffb640 1 -- 172.21.15.62:0/3863391469 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff270004450 con 0x7ff278106a80 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.226+0000 7ff275ffb640 1 -- 172.21.15.62:0/3863391469 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff270003e10 con 0x7ff278106a80 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.226+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff27819f7d0 con 0x7ff278106a80 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.226+0000 7ff275ffb640 1 -- 172.21.15.62:0/3863391469 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2700466e0 con 0x7ff278106a80 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.226+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2781a7110 con 0x7ff278106a80 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.227+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff278100ee0 con 0x7ff278106a80 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.228+0000 7ff275ffb640 1 -- 172.21.15.62:0/3863391469 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7ff2700305a0 con 0x7ff278106a80 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.229+0000 7ff275ffb640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2600a32c0 0x7ff2600a5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.229+0000 7ff27cb8e640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2600a32c0 0x7ff2600a5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.229+0000 7ff275ffb640 1 -- 172.21.15.62:0/3863391469 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8854+0+0 (secure 0 0 0) 0x7ff27002e030 con 0x7ff278106a80 2024-09-15T23:32:36.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.230+0000 7ff27cb8e640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2600a32c0 0x7ff2600a5780 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff2680058c0 tx=0x7ff26800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:36.236 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.233+0000 7ff275ffb640 1 -- 172.21.15.62:0/3863391469 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7ff27004b060 con 0x7ff278106a80 2024-09-15T23:32:36.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:35 smithi067 ceph-mon[35665]: pgmap v860: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-09-15T23:32:36.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:35 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/2626922881' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:32:36.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:35 smithi067 ceph-mon[35665]: osdmap e94: 12 total, 12 up, 12 in 2024-09-15T23:32:36.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:35 smithi148 ceph-mon[35610]: pgmap v860: 129 pgs: 129 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2024-09-15T23:32:36.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:35 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/2626922881' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:32:36.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:35 smithi148 ceph-mon[35610]: osdmap e94: 12 total, 12 up, 12 in 2024-09-15T23:32:36.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.413+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff2781a73b0 con 0x7ff278106a80 2024-09-15T23:32:36.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.857+0000 7ff275ffb640 1 -- 172.21.15.62:0/3863391469 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v95) ==== 169+0+0 (secure 0 0 0) 0x7ff2700bdf70 con 0x7ff278106a80 2024-09-15T23:32:36.859 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.861+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2600a32c0 msgr2=0x7ff2600a5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.861+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2600a32c0 0x7ff2600a5780 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff2680058c0 tx=0x7ff26800a2d0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.861+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278106a80 msgr2=0x7ff27819ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.861+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278106a80 0x7ff27819ee60 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7ff270006010 tx=0x7ff270003c20 comp rx=0 tx=0).stop 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.862+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 shutdown_connections 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.862+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7ff2600a32c0 0x7ff2600a5780 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.862+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff278106a80 0x7ff27819ee60 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.862+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff278101420 0x7ff27819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.862+0000 7ff27edf2640 1 --2- 172.21.15.62:0/3863391469 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff278100a80 0x7ff27819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:36.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.862+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 >> 172.21.15.62:0/3863391469 conn(0x7ff2780fc7a0 msgr2=0x7ff27806a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:36.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.862+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 shutdown_connections 2024-09-15T23:32:36.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:36.862+0000 7ff27edf2640 1 -- 172.21.15.62:0/3863391469 wait complete. 2024-09-15T23:32:37.178 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:36 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3863391469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:32:37.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:36 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3863391469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:32:37.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:36 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3863391469' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:32:37.537 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc shell --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 -- ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-15T23:32:37.807 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/mon.a/config 2024-09-15T23:32:38.120 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:37 smithi062 ceph-mon[30949]: pgmap v862: 97 pgs: 97 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:32:38.120 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:37 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/3863391469' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:32:38.120 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:37 smithi062 ceph-mon[30949]: osdmap e95: 12 total, 12 up, 12 in 2024-09-15T23:32:38.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:37 smithi067 ceph-mon[35665]: pgmap v862: 97 pgs: 97 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:32:38.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:37 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/3863391469' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:32:38.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:37 smithi067 ceph-mon[35665]: osdmap e95: 12 total, 12 up, 12 in 2024-09-15T23:32:38.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:37 smithi148 ceph-mon[35610]: pgmap v862: 97 pgs: 97 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:32:38.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:37 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/3863391469' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:32:38.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:37 smithi148 ceph-mon[35610]: osdmap e95: 12 total, 12 up, 12 in 2024-09-15T23:32:38.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.316+0000 7f7255560640 1 -- 172.21.15.62:0/1355414478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250103660 msgr2=0x7f725010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.316+0000 7f7255560640 1 --2- 172.21.15.62:0/1355414478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250103660 0x7f725010bb50 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f7238009900 tx=0x7f723802a480 comp rx=0 tx=0).stop 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.316+0000 7f7255560640 1 -- 172.21.15.62:0/1355414478 shutdown_connections 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.316+0000 7f7255560640 1 --2- 172.21.15.62:0/1355414478 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f725010c170 0x7f725010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.316+0000 7f7255560640 1 --2- 172.21.15.62:0/1355414478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250103660 0x7f725010bb50 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.316+0000 7f7255560640 1 --2- 172.21.15.62:0/1355414478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7250102d40 0x7f7250103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.316+0000 7f7255560640 1 -- 172.21.15.62:0/1355414478 >> 172.21.15.62:0/1355414478 conn(0x7f72500fc7a0 msgr2=0x7f72500febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.316+0000 7f7255560640 1 -- 172.21.15.62:0/1355414478 shutdown_connections 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.316+0000 7f7255560640 1 -- 172.21.15.62:0/1355414478 wait complete. 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.317+0000 7f7255560640 1 Processor -- start 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.317+0000 7f7255560640 1 -- start start 2024-09-15T23:32:38.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.317+0000 7f7255560640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250102d40 0x7f7250198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.317+0000 7f7255560640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7250103660 0x7f7250198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.317+0000 7f7255560640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f725010c170 0x7f725019cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.317+0000 7f7255560640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7250111fb0 con 0x7f7250102d40 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.317+0000 7f7255560640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7250111e30 con 0x7f725010c170 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.317+0000 7f7255560640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7250112130 con 0x7f7250103660 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250102d40 0x7f7250198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250102d40 0x7f7250198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:50882/0 (socket says 172.21.15.62:50882) 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724effd640 1 -- 172.21.15.62:0/1768175426 learned_addr learned my addr 172.21.15.62:0/1768175426 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724f7fe640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f725010c170 0x7f725019cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724e7fc640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7250103660 0x7f7250198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724effd640 1 -- 172.21.15.62:0/1768175426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7250103660 msgr2=0x7f7250198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:38.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724effd640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7250103660 0x7f7250198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724effd640 1 -- 172.21.15.62:0/1768175426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f725010c170 msgr2=0x7f725019cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724effd640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f725010c170 0x7f725019cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724effd640 1 -- 172.21.15.62:0/1768175426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725019d3f0 con 0x7f7250102d40 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f724effd640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250102d40 0x7f7250198440 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f724800c920 tx=0x7f724800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f7247fff640 1 -- 172.21.15.62:0/1768175426 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248007c70 con 0x7f7250102d40 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f725019d6e0 con 0x7f7250102d40 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f7247fff640 1 -- 172.21.15.62:0/1768175426 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7248007e10 con 0x7f7250102d40 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f7247fff640 1 -- 172.21.15.62:0/1768175426 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7248018890 con 0x7f7250102d40 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.318+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72501a4fc0 con 0x7f7250102d40 2024-09-15T23:32:38.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.319+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f725006b070 con 0x7f7250102d40 2024-09-15T23:32:38.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.321+0000 7f7247fff640 1 -- 172.21.15.62:0/1768175426 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) ==== 138355+0+0 (secure 0 0 0) 0x7f7248018a50 con 0x7f7250102d40 2024-09-15T23:32:38.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.321+0000 7f7247fff640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f723c0a3240 0x7f723c0a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-15T23:32:38.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.321+0000 7f724e7fc640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f723c0a3240 0x7f723c0a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-15T23:32:38.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.321+0000 7f7247fff640 1 -- 172.21.15.62:0/1768175426 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8324+0+0 (secure 0 0 0) 0x7f72480cfa30 con 0x7f7250102d40 2024-09-15T23:32:38.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.322+0000 7f724e7fc640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f723c0a3240 0x7f723c0a5700 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7238000c00 tx=0x7f72380058d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-15T23:32:38.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.325+0000 7f7247fff640 1 -- 172.21.15.62:0/1768175426 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188821 (secure 0 0 0) 0x7f7248014030 con 0x7f7250102d40 2024-09-15T23:32:38.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.500+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f725002d7c0 con 0x7f7250102d40 2024-09-15T23:32:38.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.864+0000 7f7247fff640 1 -- 172.21.15.62:0/1768175426 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v96) ==== 178+0+0 (secure 0 0 0) 0x7f724809c6f0 con 0x7f7250102d40 2024-09-15T23:32:38.866 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data_ec' removed 2024-09-15T23:32:38.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f723c0a3240 msgr2=0x7f723c0a5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:38.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f723c0a3240 0x7f723c0a5700 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7238000c00 tx=0x7f72380058d0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250102d40 msgr2=0x7f7250198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T23:32:38.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250102d40 0x7f7250198440 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f724800c920 tx=0x7f724800cde0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 shutdown_connections 2024-09-15T23:32:38.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:6800/4035090192,v1:172.21.15.62:6801/4035090192] conn(0x7f723c0a3240 0x7f723c0a5700 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f725010c170 0x7f725019cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7250103660 0x7f7250198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 --2- 172.21.15.62:0/1768175426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7250102d40 0x7f7250198440 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-15T23:32:38.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.867+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 >> 172.21.15.62:0/1768175426 conn(0x7f72500fc7a0 msgr2=0x7f72500fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T23:32:38.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.868+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 shutdown_connections 2024-09-15T23:32:38.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-09-15T23:32:38.868+0000 7f7255560640 1 -- 172.21.15.62:0/1768175426 wait complete. 2024-09-15T23:32:39.203 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:38 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1768175426' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:32:39.274 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:38 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1768175426' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:32:39.289 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:38 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1768175426' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-15T23:32:39.548 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:32:39.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:32:39.624 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:32:39.654 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-15T23:32:39.654 DEBUG:teuthology.orchestra.run.smithi062:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-15T23:32:39.695 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-15T23:32:39.711 DEBUG:teuthology.orchestra.run.smithi148:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-15T23:32:39.727 INFO:tasks.cephadm:Stopping all daemons... 2024-09-15T23:32:39.727 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-15T23:32:39.727 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.a 2024-09-15T23:32:40.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:39 smithi067 ceph-mon[35665]: pgmap v864: 33 pgs: 33 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:32:40.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:39 smithi067 ceph-mon[35665]: from='client.? 172.21.15.62:0/1768175426' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:32:40.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:39 smithi067 ceph-mon[35665]: osdmap e96: 12 total, 12 up, 12 in 2024-09-15T23:32:40.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:39 smithi148 ceph-mon[35610]: pgmap v864: 33 pgs: 33 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:32:40.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:39 smithi148 ceph-mon[35610]: from='client.? 172.21.15.62:0/1768175426' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:32:40.039 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:39 smithi148 ceph-mon[35610]: osdmap e96: 12 total, 12 up, 12 in 2024-09-15T23:32:40.082 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:39 smithi062 systemd[1]: Stopping Ceph mon.a for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:32:40.082 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:39 smithi062 ceph-mon[30949]: pgmap v864: 33 pgs: 33 active+clean; 1.5 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-15T23:32:40.082 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:39 smithi062 ceph-mon[30949]: from='client.? 172.21.15.62:0/1768175426' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-09-15T23:32:40.082 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:39 smithi062 ceph-mon[30949]: osdmap e96: 12 total, 12 up, 12 in 2024-09-15T23:32:40.446 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:40 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a[30945]: 2024-09-15T23:32:40.083+0000 7f19e393d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:32:40.446 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:40 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a[30945]: 2024-09-15T23:32:40.083+0000 7f19e393d640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-15T23:32:40.446 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:40 smithi062 podman[118473]: 2024-09-15 23:32:40.15619957 +0000 UTC m=+0.217731413 container died 293eb3210083980e5630f0dfe4069d6ff633cdc31fd5c2673f86edd4071ceba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-15T23:32:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:40 smithi062 podman[118473]: 2024-09-15 23:32:40.53304173 +0000 UTC m=+0.594573577 container cleanup 293eb3210083980e5630f0dfe4069d6ff633cdc31fd5c2673f86edd4071ceba0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e) 2024-09-15T23:32:40.730 INFO:journalctl@ceph.mon.a.smithi062.stdout:Sep 15 23:32:40 smithi062 bash[118473]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-a 2024-09-15T23:32:41.027 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.a.service' 2024-09-15T23:32:41.088 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:32:41.088 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-15T23:32:41.088 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-15T23:32:41.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.b 2024-09-15T23:32:41.444 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:41 smithi067 systemd[1]: Stopping Ceph mon.b for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:32:41.754 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-b[35661]: 2024-09-15T23:32:41.443+0000 7fc9fc335640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:32:41.754 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-b[35661]: 2024-09-15T23:32:41.443+0000 7fc9fc335640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-15T23:32:41.754 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:41 smithi067 podman[70828]: 2024-09-15 23:32:41.464229985 +0000 UTC m=+0.147856505 container died 58bf2b6c8495a2f5ea796fc895dbba9be6c1c8cdea5a03f0e054b7f538199d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-15T23:32:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:41 smithi067 podman[70828]: 2024-09-15 23:32:41.969821674 +0000 UTC m=+0.653448186 container cleanup 58bf2b6c8495a2f5ea796fc895dbba9be6c1c8cdea5a03f0e054b7f538199d03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0) 2024-09-15T23:32:42.024 INFO:journalctl@ceph.mon.b.smithi067.stdout:Sep 15 23:32:41 smithi067 bash[70828]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-b 2024-09-15T23:32:42.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.b.service' 2024-09-15T23:32:42.484 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:32:42.484 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-15T23:32:42.484 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-15T23:32:42.484 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.c 2024-09-15T23:32:42.789 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:42 smithi148 systemd[1]: Stopping Ceph mon.c for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:32:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:43 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-c[35606]: 2024-09-15T23:32:43.080+0000 7fe824ed0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:32:43.539 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:43 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-c[35606]: 2024-09-15T23:32:43.080+0000 7fe824ed0640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-15T23:32:43.540 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:43 smithi148 podman[72460]: 2024-09-15 23:32:43.173730549 +0000 UTC m=+0.316599719 container died 2e8816ac8b2ae38766f783fff8ce673e48e7d6f390918202b6935c722e5ef53d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:32:44.134 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:43 smithi148 podman[72460]: 2024-09-15 23:32:43.907649029 +0000 UTC m=+1.050518242 container cleanup 2e8816ac8b2ae38766f783fff8ce673e48e7d6f390918202b6935c722e5ef53d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-15T23:32:44.134 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:43 smithi148 bash[72460]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-c 2024-09-15T23:32:44.385 INFO:journalctl@ceph.mon.c.smithi148.stdout:Sep 15 23:32:44 smithi148 podman[72474]: 2024-09-15 23:32:44.134030234 +0000 UTC m=+0.957470177 container remove 2e8816ac8b2ae38766f783fff8ce673e48e7d6f390918202b6935c722e5ef53d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mon-c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-15T23:32:44.389 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mon.c.service' 2024-09-15T23:32:44.435 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:32:44.436 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-15T23:32:44.436 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-15T23:32:44.436 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.x 2024-09-15T23:32:44.730 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:32:44 smithi062 systemd[1]: Stopping Ceph mgr.x for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:32:45.006 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:32:44 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x[31175]: 2024-09-15T23:32:44.778+0000 7f90ea36c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:32:45.006 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:32:44 smithi062 podman[118587]: 2024-09-15 23:32:44.810640167 +0000 UTC m=+0.171065155 container died 460390df780a825c934ea1c594a0d6aeaeddabff3663df6b2926bbedb1006eb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-15T23:32:45.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:32:45 smithi062 podman[118587]: 2024-09-15 23:32:45.780014232 +0000 UTC m=+1.140439214 container cleanup 460390df780a825c934ea1c594a0d6aeaeddabff3663df6b2926bbedb1006eb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8921c8ef947807b717db041d85f0761b7914d5fc, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-15T23:32:45.980 INFO:journalctl@ceph.mgr.x.smithi062.stdout:Sep 15 23:32:45 smithi062 bash[118587]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-x 2024-09-15T23:32:46.226 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-15T23:32:46.265 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:32:46.266 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-15T23:32:46.266 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-15T23:32:46.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.y 2024-09-15T23:32:46.612 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:32:46 smithi067 systemd[1]: Stopping Ceph mgr.y for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:32:46.910 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:32:46 smithi067 podman[70954]: 2024-09-15 23:32:46.611775567 +0000 UTC m=+0.146179015 container died 73e4e6a40fa40f2a4dd3080d97afd9af0031b933725fb998cd180f9c8b057d26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-8921c8e) 2024-09-15T23:32:47.274 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:32:47 smithi067 podman[70954]: 2024-09-15 23:32:47.125098787 +0000 UTC m=+0.659502233 container cleanup 73e4e6a40fa40f2a4dd3080d97afd9af0031b933725fb998cd180f9c8b057d26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:32:47.274 INFO:journalctl@ceph.mgr.y.smithi067.stdout:Sep 15 23:32:47 smithi067 bash[70954]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-y 2024-09-15T23:32:47.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-15T23:32:47.541 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:32:47.541 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-15T23:32:47.541 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-09-15T23:32:47.541 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.z 2024-09-15T23:32:47.892 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:32:47 smithi148 systemd[1]: Stopping Ceph mgr.z for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:32:48.289 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:32:47 smithi148 podman[72583]: 2024-09-15 23:32:47.891773394 +0000 UTC m=+0.149360948 container died 39d719a09112c909a3b4523a64d1eca55b79e56b18602266f01eb529b5bff67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, ceph=True) 2024-09-15T23:32:48.789 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:32:48 smithi148 podman[72583]: 2024-09-15 23:32:48.60492191 +0000 UTC m=+0.862509463 container cleanup 39d719a09112c909a3b4523a64d1eca55b79e56b18602266f01eb529b5bff67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-15T23:32:48.789 INFO:journalctl@ceph.mgr.z.smithi148.stdout:Sep 15 23:32:48 smithi148 bash[72583]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mgr-z 2024-09-15T23:32:49.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mgr.z.service' 2024-09-15T23:32:49.530 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:32:49.530 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-09-15T23:32:49.530 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-15T23:32:49.530 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.0 2024-09-15T23:32:49.980 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:49 smithi062 systemd[1]: Stopping Ceph osd.0 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:32:50.480 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:50 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0[42523]: 2024-09-15T23:32:50.101+0000 7fcfecb98640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:32:50.480 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:50 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0[42523]: 2024-09-15T23:32:50.101+0000 7fcfecb98640 -1 osd.0 96 *** Got signal Terminated *** 2024-09-15T23:32:50.480 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:50 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0[42523]: 2024-09-15T23:32:50.101+0000 7fcfecb98640 -1 osd.0 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:32:55.981 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:55 smithi062 podman[118720]: 2024-09-15 23:32:55.623508799 +0000 UTC m=+5.896603298 container died 169f9de9b0056df21151cdf6ad78c534f90c24f8b1faa7298a1af3881d2d3fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-15T23:32:56.671 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:56 smithi062 podman[118720]: 2024-09-15 23:32:56.360480671 +0000 UTC m=+6.633575163 container cleanup 169f9de9b0056df21151cdf6ad78c534f90c24f8b1faa7298a1af3881d2d3fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e) 2024-09-15T23:32:56.671 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:56 smithi062 bash[118720]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0 2024-09-15T23:32:56.980 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:56 smithi062 podman[118736]: 2024-09-15 23:32:56.670656769 +0000 UTC m=+0.996470841 container remove 169f9de9b0056df21151cdf6ad78c534f90c24f8b1faa7298a1af3881d2d3fc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-15T23:32:57.250 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:56 smithi062 podman[118824]: 2024-09-15 23:32:56.890993531 +0000 UTC m=+0.016671476 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:32:57.250 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:57 smithi062 podman[118824]: 2024-09-15 23:32:57.013420044 +0000 UTC m=+0.139097987 container create 676a4414f46e73e538325150669e25f39ebbbf72c766f598fe2d3ce8c0a88543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-15T23:32:57.730 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:57 smithi062 podman[118824]: 2024-09-15 23:32:57.249379496 +0000 UTC m=+0.375057437 container init 676a4414f46e73e538325150669e25f39ebbbf72c766f598fe2d3ce8c0a88543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-15T23:32:57.731 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:57 smithi062 podman[118824]: 2024-09-15 23:32:57.253436998 +0000 UTC m=+0.379114950 container start 676a4414f46e73e538325150669e25f39ebbbf72c766f598fe2d3ce8c0a88543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:32:57.731 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:57 smithi062 podman[118824]: 2024-09-15 23:32:57.357712496 +0000 UTC m=+0.483390450 container attach 676a4414f46e73e538325150669e25f39ebbbf72c766f598fe2d3ce8c0a88543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-15T23:32:57.731 INFO:journalctl@ceph.osd.0.smithi062.stdout:Sep 15 23:32:57 smithi062 podman[118824]: 2024-09-15 23:32:57.493926027 +0000 UTC m=+0.619603970 container died 676a4414f46e73e538325150669e25f39ebbbf72c766f598fe2d3ce8c0a88543 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:32:58.099 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.0.service' 2024-09-15T23:32:58.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:32:58.143 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-15T23:32:58.143 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-15T23:32:58.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.1 2024-09-15T23:32:58.465 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:32:58 smithi067 systemd[1]: Stopping Ceph osd.1 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:32:58.774 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:32:58 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1[40138]: 2024-09-15T23:32:58.464+0000 7ff977f07640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:32:58.774 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:32:58 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1[40138]: 2024-09-15T23:32:58.464+0000 7ff977f07640 -1 osd.1 96 *** Got signal Terminated *** 2024-09-15T23:32:58.774 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:32:58 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1[40138]: 2024-09-15T23:32:58.464+0000 7ff977f07640 -1 osd.1 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:33:04.424 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:04 smithi067 podman[71091]: 2024-09-15 23:33:04.035663396 +0000 UTC m=+5.691570317 container died e78cdad9887a0b6593e4d51c96cbbfd85bb2960b55a4f25aeb11ffe0e9f61099 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-15T23:33:04.774 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:04 smithi067 podman[71091]: 2024-09-15 23:33:04.489622949 +0000 UTC m=+6.145529912 container cleanup e78cdad9887a0b6593e4d51c96cbbfd85bb2960b55a4f25aeb11ffe0e9f61099 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-15T23:33:04.774 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:04 smithi067 bash[71091]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1 2024-09-15T23:33:05.159 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:04 smithi067 podman[71106]: 2024-09-15 23:33:04.799729515 +0000 UTC m=+0.761668336 container remove e78cdad9887a0b6593e4d51c96cbbfd85bb2960b55a4f25aeb11ffe0e9f61099 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-15T23:33:05.408 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:05 smithi067 podman[71194]: 2024-09-15 23:33:05.056256822 +0000 UTC m=+0.016022015 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:33:05.408 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:05 smithi067 podman[71194]: 2024-09-15 23:33:05.170638532 +0000 UTC m=+0.130403722 container create 2110df4ab329139141b4b5ad368e9d6ae1aaa91ae67449fdb6a827542639eaac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:33:05.408 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:05 smithi067 podman[71194]: 2024-09-15 23:33:05.403737895 +0000 UTC m=+0.363503103 container init 2110df4ab329139141b4b5ad368e9d6ae1aaa91ae67449fdb6a827542639eaac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-15T23:33:05.409 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:05 smithi067 podman[71194]: 2024-09-15 23:33:05.407125431 +0000 UTC m=+0.366890625 container start 2110df4ab329139141b4b5ad368e9d6ae1aaa91ae67449fdb6a827542639eaac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-15T23:33:05.774 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:05 smithi067 podman[71194]: 2024-09-15 23:33:05.503378039 +0000 UTC m=+0.463143252 container attach 2110df4ab329139141b4b5ad368e9d6ae1aaa91ae67449fdb6a827542639eaac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731) 2024-09-15T23:33:05.775 INFO:journalctl@ceph.osd.1.smithi067.stdout:Sep 15 23:33:05 smithi067 podman[71194]: 2024-09-15 23:33:05.640130284 +0000 UTC m=+0.599895478 container died 2110df4ab329139141b4b5ad368e9d6ae1aaa91ae67449fdb6a827542639eaac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-15T23:33:06.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.1.service' 2024-09-15T23:33:06.306 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:33:06.306 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-15T23:33:06.306 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-15T23:33:06.307 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.2 2024-09-15T23:33:06.653 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:06 smithi148 systemd[1]: Stopping Ceph osd.2 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:33:07.039 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:06 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2[40560]: 2024-09-15T23:33:06.651+0000 7f8457749640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:33:07.040 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:06 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2[40560]: 2024-09-15T23:33:06.651+0000 7f8457749640 -1 osd.2 96 *** Got signal Terminated *** 2024-09-15T23:33:07.040 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:06 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2[40560]: 2024-09-15T23:33:06.651+0000 7f8457749640 -1 osd.2 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:33:12.536 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:12 smithi148 podman[72717]: 2024-09-15 23:33:12.232120196 +0000 UTC m=+5.716464965 container died f9773566b554acee1cddee5adfb9ae57d5940bde767e14695ee9956c3ae543b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=) 2024-09-15T23:33:12.789 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:12 smithi148 podman[72717]: 2024-09-15 23:33:12.610093859 +0000 UTC m=+6.094438630 container cleanup f9773566b554acee1cddee5adfb9ae57d5940bde767e14695ee9956c3ae543b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:33:12.789 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:12 smithi148 bash[72717]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2 2024-09-15T23:33:13.209 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:12 smithi148 podman[72733]: 2024-09-15 23:33:12.895003632 +0000 UTC m=+0.659581969 container remove f9773566b554acee1cddee5adfb9ae57d5940bde767e14695ee9956c3ae543b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-15T23:33:13.524 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:13 smithi148 podman[72821]: 2024-09-15 23:33:13.110487952 +0000 UTC m=+0.016476858 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:33:13.524 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:13 smithi148 podman[72821]: 2024-09-15 23:33:13.245973329 +0000 UTC m=+0.151962270 container create d480ad53944f79d4ac13099f9c962273607098fec6ff8be3ded423cc22ddb87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:33:13.524 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:13 smithi148 podman[72821]: 2024-09-15 23:33:13.523612143 +0000 UTC m=+0.429601036 container init d480ad53944f79d4ac13099f9c962273607098fec6ff8be3ded423cc22ddb87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-15T23:33:13.788 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:13 smithi148 podman[72821]: 2024-09-15 23:33:13.527195415 +0000 UTC m=+0.433184308 container start d480ad53944f79d4ac13099f9c962273607098fec6ff8be3ded423cc22ddb87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-15T23:33:13.789 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:13 smithi148 podman[72821]: 2024-09-15 23:33:13.631965461 +0000 UTC m=+0.537954372 container attach d480ad53944f79d4ac13099f9c962273607098fec6ff8be3ded423cc22ddb87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-15T23:33:14.039 INFO:journalctl@ceph.osd.2.smithi148.stdout:Sep 15 23:33:13 smithi148 podman[72821]: 2024-09-15 23:33:13.78859578 +0000 UTC m=+0.694584683 container died d480ad53944f79d4ac13099f9c962273607098fec6ff8be3ded423cc22ddb87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-15T23:33:14.479 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.2.service' 2024-09-15T23:33:14.521 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:33:14.521 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-15T23:33:14.521 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-15T23:33:14.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.3 2024-09-15T23:33:14.832 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:14 smithi062 systemd[1]: Stopping Ceph osd.3 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:33:15.101 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3[49391]: 2024-09-15T23:33:14.830+0000 7f4afc75a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:33:15.102 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3[49391]: 2024-09-15T23:33:14.830+0000 7f4afc75a640 -1 osd.3 96 *** Got signal Terminated *** 2024-09-15T23:33:15.102 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:14 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3[49391]: 2024-09-15T23:33:14.830+0000 7f4afc75a640 -1 osd.3 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:33:16.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:16 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:16.520+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:17.480 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:17 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3[49391]: 2024-09-15T23:33:17.015+0000 7f4af8d70640 -1 osd.3 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.090209+0000 front 2024-09-15T23:32:52.090243+0000 (oldest deadline 2024-09-15T23:33:16.190400+0000) 2024-09-15T23:33:17.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:17 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:17.163+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:17.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:17 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:17.473+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:17.792 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:17 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:17.336+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:18.480 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3[49391]: 2024-09-15T23:33:18.060+0000 7f4af8d70640 -1 osd.3 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.090209+0000 front 2024-09-15T23:32:52.090243+0000 (oldest deadline 2024-09-15T23:33:16.190400+0000) 2024-09-15T23:33:18.481 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:18 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:18.293+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:18.501 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:18.316+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:18.501 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:18.120+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:18.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:18 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:18.196+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:18.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:18.500+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:19.480 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:19 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3[49391]: 2024-09-15T23:33:19.030+0000 7f4af8d70640 -1 osd.3 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.090209+0000 front 2024-09-15T23:32:52.090243+0000 (oldest deadline 2024-09-15T23:33:16.190400+0000) 2024-09-15T23:33:19.480 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:19 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:19.260+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:19.515 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:19.080+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:19.516 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:19.324+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:19.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:19 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:19.199+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:19.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:19.514+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:20.473 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:20.108+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:20.473 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:20.349+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:20.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:20 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:20.178+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:20.683 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:20 smithi062 podman[118937]: 2024-09-15 23:33:20.377447942 +0000 UTC m=+5.656740149 container died 39cf981d3a4cc981a2bd5819246b0947e062fac4b87e45b81a54edf93dd43ee3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-09-15T23:33:20.683 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:20 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:20.296+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:20.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:20.472+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:21.230 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:20 smithi062 podman[118937]: 2024-09-15 23:33:20.956204276 +0000 UTC m=+6.235496471 container cleanup 39cf981d3a4cc981a2bd5819246b0947e062fac4b87e45b81a54edf93dd43ee3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-15T23:33:21.230 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:20 smithi062 bash[118937]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3 2024-09-15T23:33:21.391 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:21.134+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:21.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:21 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:21.150+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:21.639 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:21 smithi062 podman[118964]: 2024-09-15 23:33:21.274431368 +0000 UTC m=+0.893902899 container remove 39cf981d3a4cc981a2bd5819246b0947e062fac4b87e45b81a54edf93dd43ee3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux ) 2024-09-15T23:33:21.640 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:21 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:21.332+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:21.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:21.514+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:21.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:21.391+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:21.895 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:21 smithi062 podman[119056]: 2024-09-15 23:33:21.542731773 +0000 UTC m=+0.016767960 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:33:21.895 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:21 smithi062 podman[119056]: 2024-09-15 23:33:21.663425223 +0000 UTC m=+0.137461406 container create ec0ff638f136cf490185f165b77e1d2fa2c936707c428a9b8f73337a4f96d38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-15T23:33:21.895 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:21 smithi062 podman[119056]: 2024-09-15 23:33:21.894884454 +0000 UTC m=+0.368920632 container init ec0ff638f136cf490185f165b77e1d2fa2c936707c428a9b8f73337a4f96d38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-15T23:33:22.230 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:21 smithi062 podman[119056]: 2024-09-15 23:33:21.898352302 +0000 UTC m=+0.372388480 container start ec0ff638f136cf490185f165b77e1d2fa2c936707c428a9b8f73337a4f96d38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:33:22.230 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:21 smithi062 podman[119056]: 2024-09-15 23:33:21.961544063 +0000 UTC m=+0.435580251 container attach ec0ff638f136cf490185f165b77e1d2fa2c936707c428a9b8f73337a4f96d38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-15T23:33:22.230 INFO:journalctl@ceph.osd.3.smithi062.stdout:Sep 15 23:33:22 smithi062 podman[119056]: 2024-09-15 23:33:22.086549849 +0000 UTC m=+0.560586026 container died ec0ff638f136cf490185f165b77e1d2fa2c936707c428a9b8f73337a4f96d38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-15T23:33:22.520 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:22.183+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:22.520 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:22.409+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:22.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:22 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:22.125+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:22.666 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:22 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:22.338+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:22.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:22.518+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:22.803 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.3.service' 2024-09-15T23:33:22.846 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:33:22.846 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-15T23:33:22.846 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-15T23:33:22.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.4 2024-09-15T23:33:23.160 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:22 smithi067 systemd[1]: Stopping Ceph osd.4 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:33:23.451 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:23.147+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:23.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:23 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:23.158+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:23.524 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:23 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4[47037]: 2024-09-15T23:33:23.169+0000 7f78ee831640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:33:23.524 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:23 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4[47037]: 2024-09-15T23:33:23.169+0000 7f78ee831640 -1 osd.4 96 *** Got signal Terminated *** 2024-09-15T23:33:23.525 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:23 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4[47037]: 2024-09-15T23:33:23.169+0000 7f78ee831640 -1 osd.4 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:33:23.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:23 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:23.350+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:23.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:23.511+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:23.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:23.450+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:24.462 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:24.191+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:24.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:24 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:24.196+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:24.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:24 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:24.399+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:24.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:24.461+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:24.791 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:24.530+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:24.792 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:24.530+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:25.374 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:25 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:25.018+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:25.374 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:25 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:25.373+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:25.419 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:25.167+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:25.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:25 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:25.221+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:25.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:25.494+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:25.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:25.494+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:25.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:25.418+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:26.230 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:25 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:25.974+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:26.433 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:26 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:26.154+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:26.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:26 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:26.262+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:26.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:26 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:26.377+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:26.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:26 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:26.431+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:26.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:26 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:26.466+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:26.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:26 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:26.466+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:27.403 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:26 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:26.987+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:27.454 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:27 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:27.175+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:27.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:27 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:27.218+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:27.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:27 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:27.218+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:27.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:27 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:27.401+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:27.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:27 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:27.455+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:27.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:27 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:27.452+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:27.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:27 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:27.452+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:28.211 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:27 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4[47037]: 2024-09-15T23:33:27.808+0000 7f78e6e3f640 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.772388+0000 front 2024-09-15T23:33:01.772350+0000 (oldest deadline 2024-09-15T23:33:27.072546+0000) 2024-09-15T23:33:28.359 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:28 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:28.023+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:28.419 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:28 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:28.150+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:28.419 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:28 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:28.150+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.029227+0000 front 2024-09-15T23:33:02.029175+0000 (oldest deadline 2024-09-15T23:33:27.329270+0000) 2024-09-15T23:33:28.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:28 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:28.210+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:28.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:28 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:28.210+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:28.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:28 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:28.359+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:28.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:28 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:28.417+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:28.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:28 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:28.421+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:28.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:28 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:28.421+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:29.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:28 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:28.663+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:29.024 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:28 smithi067 podman[71301]: 2024-09-15 23:33:28.811815168 +0000 UTC m=+5.759898215 container died ec3bf70361802df68a6857e9374ce3d706d1fe9d4a2ffe021bb406317f3e2dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-15T23:33:29.410 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:29 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:29.020+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:29.423 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:29 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:29.133+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:29.423 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:29 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:29.133+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.029227+0000 front 2024-09-15T23:33:02.029175+0000 (oldest deadline 2024-09-15T23:33:27.329270+0000) 2024-09-15T23:33:29.446 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:29 smithi067 podman[71301]: 2024-09-15 23:33:29.186212158 +0000 UTC m=+6.134295240 container cleanup ec3bf70361802df68a6857e9374ce3d706d1fe9d4a2ffe021bb406317f3e2dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-15T23:33:29.446 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:29 smithi067 bash[71301]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4 2024-09-15T23:33:29.446 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:29 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:29.182+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:29.447 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:29 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:29.182+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:29.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:29 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:29.408+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:29.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:29 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:29.408+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:29.774 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:29 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:29.678+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:29.774 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:29 smithi067 podman[71329]: 2024-09-15 23:33:29.445714151 +0000 UTC m=+0.633836060 container remove ec3bf70361802df68a6857e9374ce3d706d1fe9d4a2ffe021bb406317f3e2dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-15T23:33:29.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:29 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:29.456+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:29.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:29 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:29.456+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:29.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:29 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:29.422+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:29.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:29 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:29.422+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:30.050 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:29 smithi067 podman[71420]: 2024-09-15 23:33:29.699689561 +0000 UTC m=+0.016092004 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:33:30.050 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:29 smithi067 podman[71420]: 2024-09-15 23:33:29.822116315 +0000 UTC m=+0.138518779 container create 62ea00518b8be7b88ae918df46f99a7b06b66d5638e9534a368ef69aff396da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True) 2024-09-15T23:33:30.432 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:30.139+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:30.432 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:30.139+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.029227+0000 front 2024-09-15T23:33:02.029175+0000 (oldest deadline 2024-09-15T23:33:27.329270+0000) 2024-09-15T23:33:30.434 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:30 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:30.040+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:30.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:30.133+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:30.525 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:30.133+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:30.525 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:30 smithi067 podman[71420]: 2024-09-15 23:33:30.049808799 +0000 UTC m=+0.366211229 container init 62ea00518b8be7b88ae918df46f99a7b06b66d5638e9534a368ef69aff396da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-15T23:33:30.525 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:30 smithi067 podman[71420]: 2024-09-15 23:33:30.053350607 +0000 UTC m=+0.369753029 container start 62ea00518b8be7b88ae918df46f99a7b06b66d5638e9534a368ef69aff396da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-15T23:33:30.526 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:30 smithi067 podman[71420]: 2024-09-15 23:33:30.14949364 +0000 UTC m=+0.465896065 container attach 62ea00518b8be7b88ae918df46f99a7b06b66d5638e9534a368ef69aff396da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS) 2024-09-15T23:33:30.526 INFO:journalctl@ceph.osd.4.smithi067.stdout:Sep 15 23:33:30 smithi067 podman[71420]: 2024-09-15 23:33:30.260276925 +0000 UTC m=+0.576679357 container died 62ea00518b8be7b88ae918df46f99a7b06b66d5638e9534a368ef69aff396da0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-15T23:33:30.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:30 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:30.434+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:30.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:30 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:30.434+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:30.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:30.434+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:30.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:30.434+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:30.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:30.430+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:30.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:30 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:30.430+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:30.904 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:30 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:30.653+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:30.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.4.service' 2024-09-15T23:33:30.972 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:33:30.972 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-15T23:33:30.972 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-15T23:33:30.972 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.5 2024-09-15T23:33:31.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:31 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:31.085+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:31.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:31 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:31.085+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:31.289 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:31 smithi148 systemd[1]: Stopping Ceph osd.5 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:33:31.289 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:31 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:31.135+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:31.290 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:31 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:31.135+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.029227+0000 front 2024-09-15T23:33:02.029175+0000 (oldest deadline 2024-09-15T23:33:27.329270+0000) 2024-09-15T23:33:31.461 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:31 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:31.042+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:31.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:31 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:31.460+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:31.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:31 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:31.460+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:31.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:31 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:31.387+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:31.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:31 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:31.387+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:31.790 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:31 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:31.313+0000 7f26e1c46640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:33:31.790 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:31 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:31.313+0000 7f26e1c46640 -1 osd.5 96 *** Got signal Terminated *** 2024-09-15T23:33:31.790 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:31 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:31.313+0000 7f26e1c46640 -1 osd.5 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:33:31.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:31 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:31.442+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:31.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:31 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:31.442+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:32.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:31 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:31.611+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:32.449 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:32 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:32.157+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:32.449 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:32 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:32.157+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.029227+0000 front 2024-09-15T23:33:02.029175+0000 (oldest deadline 2024-09-15T23:33:27.329270+0000) 2024-09-15T23:33:32.449 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:32 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:32.365+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:32.449 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:32 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:32.365+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:32.480 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:32 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:32.072+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:32.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:32 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:32.066+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:32.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:32 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:32.066+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:32.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:32 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:32.447+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:32.804 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:32 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:32.447+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:32.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:32 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:32.490+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:32.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:32 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:32.490+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:33.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:32 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:32.606+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:33.420 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:33 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:33.328+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:33.421 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:33 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:33.328+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:33.421 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:33 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:33.149+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:33.421 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:33 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:33.149+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.029227+0000 front 2024-09-15T23:33:02.029175+0000 (oldest deadline 2024-09-15T23:33:27.329270+0000) 2024-09-15T23:33:33.421 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:33 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:33.149+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.430140+0000 front 2024-09-15T23:33:11.430183+0000 (oldest deadline 2024-09-15T23:33:32.530336+0000) 2024-09-15T23:33:33.459 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:33 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:33.049+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:33.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:33 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:33.079+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:33.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:33 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:33.079+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:33.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:33 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:33.458+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:33.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:33 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:33.458+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:33.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:33 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:33.418+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:33.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:33 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:33.418+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:34.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:33 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:33.649+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:34.444 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:34 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:34.112+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:34.445 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:34 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:34.112+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.029227+0000 front 2024-09-15T23:33:02.029175+0000 (oldest deadline 2024-09-15T23:33:27.329270+0000) 2024-09-15T23:33:34.445 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:34 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:34.112+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.430140+0000 front 2024-09-15T23:33:11.430183+0000 (oldest deadline 2024-09-15T23:33:32.530336+0000) 2024-09-15T23:33:34.445 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:34 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:34.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:34.445 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:34 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:34.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:34.445 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:34 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:34.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:34.451 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:34 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:34.062+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:34.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:34 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:34.037+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:34.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:34 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:34.037+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:34.525 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:34 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:34.037+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:34.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:34 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:34.450+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:34.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:34 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:34.450+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:34.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:34 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:34.443+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:34.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:34 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:34.443+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:35.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:34 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:34.655+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:35.445 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:35.314+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:35.445 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:35.314+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:35.445 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:35.314+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:35.445 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:35.157+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:35.446 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:35.157+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.029227+0000 front 2024-09-15T23:33:02.029175+0000 (oldest deadline 2024-09-15T23:33:27.329270+0000) 2024-09-15T23:33:35.446 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:35.157+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.430140+0000 front 2024-09-15T23:33:11.430183+0000 (oldest deadline 2024-09-15T23:33:32.530336+0000) 2024-09-15T23:33:35.460 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:35 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:35.102+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:35.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:35 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:35.038+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:35.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:35 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:35.038+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:35.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:35 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:35.038+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:35.719 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:35.443+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:35.719 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:35 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:35.443+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:35.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:35 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:35.459+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:35.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:35 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:35.459+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:36.009 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:35 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:35.647+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:36.275 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:36 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:36.008+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:36.275 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:36 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:36.008+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:36.275 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:36 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:36.008+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:36.468 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:36.141+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:52.628425+0000 front 2024-09-15T23:32:52.628211+0000 (oldest deadline 2024-09-15T23:33:17.928124+0000) 2024-09-15T23:33:36.468 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:36.141+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.029227+0000 front 2024-09-15T23:33:02.029175+0000 (oldest deadline 2024-09-15T23:33:27.329270+0000) 2024-09-15T23:33:36.468 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5[47458]: 2024-09-15T23:33:36.141+0000 7f26da254640 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.430140+0000 front 2024-09-15T23:33:11.430183+0000 (oldest deadline 2024-09-15T23:33:32.530336+0000) 2024-09-15T23:33:36.469 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:36.317+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:36.469 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:36.317+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:36.469 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:36.317+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:36.482 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:36 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:36.057+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:36.482 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:36 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:36.057+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.874040+0000 front 2024-09-15T23:33:09.874025+0000 (oldest deadline 2024-09-15T23:33:35.173770+0000) 2024-09-15T23:33:36.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:36.465+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:36.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:36 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:36.465+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:36.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:36 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:36.486+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:36.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:36 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:36.486+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:37.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:36 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:36.642+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:37.215 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:36 smithi148 podman[72942]: 2024-09-15 23:33:36.954808005 +0000 UTC m=+5.780126080 container died b3aaac7e021295cb898a8f378437abf2732b5e6a015344328ea4c7faea8c34e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-15T23:33:37.452 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:37.032+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:37.452 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:37.032+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.874040+0000 front 2024-09-15T23:33:09.874025+0000 (oldest deadline 2024-09-15T23:33:35.173770+0000) 2024-09-15T23:33:37.500 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:37 smithi148 podman[72942]: 2024-09-15 23:33:37.297091671 +0000 UTC m=+6.122409743 container cleanup b3aaac7e021295cb898a8f378437abf2732b5e6a015344328ea4c7faea8c34e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:33:37.500 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:37 smithi148 bash[72942]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5 2024-09-15T23:33:37.500 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:37 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:37.287+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:37.501 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:37 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:37.287+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:37.501 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:37 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:37.287+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:37.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:37 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:37.047+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:37.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:37 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:37.047+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:37.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:37 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:37.047+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:37.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:37.451+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:37.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:37.451+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:37.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:37.451+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:37.789 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:37 smithi148 podman[72958]: 2024-09-15 23:33:37.581892682 +0000 UTC m=+0.631373611 container remove b3aaac7e021295cb898a8f378437abf2732b5e6a015344328ea4c7faea8c34e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:33:37.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:37 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:37.499+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:37.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:37 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:37.499+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:38.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:37 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:37.613+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:38.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:37 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:37.613+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:38.161 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:37 smithi148 podman[73044]: 2024-09-15 23:33:37.808340991 +0000 UTC m=+0.015862485 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:33:38.162 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:37 smithi148 podman[73044]: 2024-09-15 23:33:37.924675645 +0000 UTC m=+0.132197139 container create c3ed74a4294e752afff585e22efe08b2dc961d50bbccc61be2a04b5e46177967 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-15T23:33:38.162 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:38 smithi148 podman[73044]: 2024-09-15 23:33:38.160946077 +0000 UTC m=+0.368467586 container init c3ed74a4294e752afff585e22efe08b2dc961d50bbccc61be2a04b5e46177967 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD) 2024-09-15T23:33:38.430 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:38.069+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:38.430 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:38.069+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.874040+0000 front 2024-09-15T23:33:09.874025+0000 (oldest deadline 2024-09-15T23:33:35.173770+0000) 2024-09-15T23:33:38.493 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:38.321+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:38.493 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:38.321+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:38.493 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:38.321+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:38.493 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:38 smithi148 podman[73044]: 2024-09-15 23:33:38.164978847 +0000 UTC m=+0.372500358 container start c3ed74a4294e752afff585e22efe08b2dc961d50bbccc61be2a04b5e46177967 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-15T23:33:38.493 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:38 smithi148 podman[73044]: 2024-09-15 23:33:38.235469582 +0000 UTC m=+0.442991090 container attach c3ed74a4294e752afff585e22efe08b2dc961d50bbccc61be2a04b5e46177967 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-09-15T23:33:38.493 INFO:journalctl@ceph.osd.5.smithi148.stdout:Sep 15 23:33:38 smithi148 podman[73044]: 2024-09-15 23:33:38.357144457 +0000 UTC m=+0.564665955 container died c3ed74a4294e752afff585e22efe08b2dc961d50bbccc61be2a04b5e46177967 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-15T23:33:38.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:38.076+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:38.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:38.076+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:38.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:38.076+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:38.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:38.430+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:38.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:38.430+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:38.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:38 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:38.430+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:38.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:38.491+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:38.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:38 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:38.491+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:38.970 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.5.service' 2024-09-15T23:33:39.012 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:33:39.012 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-15T23:33:39.012 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-15T23:33:39.012 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.6 2024-09-15T23:33:39.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:38.611+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:39.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:38 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:38.611+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:39.443 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:39 smithi062 systemd[1]: Stopping Ceph osd.6 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:33:39.444 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:39 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:39.106+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:39.444 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:39 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:39.106+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.874040+0000 front 2024-09-15T23:33:09.874025+0000 (oldest deadline 2024-09-15T23:33:35.173770+0000) 2024-09-15T23:33:39.444 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:39 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:39.442+0000 7fa998d5a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:33:39.444 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:39 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:39.442+0000 7fa998d5a640 -1 osd.6 96 *** Got signal Terminated *** 2024-09-15T23:33:39.444 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:39 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:39.442+0000 7fa998d5a640 -1 osd.6 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:33:39.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:39 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:39.097+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:39.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:39 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:39.097+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:39.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:39 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:39.097+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:39.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:39 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:39.450+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:39.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:39 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:39.450+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:39.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:39 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:39.450+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:39.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:39.460+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:39.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:39.460+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:39.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:39.318+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:39.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:39.318+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:39.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:39 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:39.318+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:40.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:39 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:39.620+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:40.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:39 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:39.620+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:40.480 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:40 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:40.111+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:40.480 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:40 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:40.111+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.874040+0000 front 2024-09-15T23:33:09.874025+0000 (oldest deadline 2024-09-15T23:33:35.173770+0000) 2024-09-15T23:33:40.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:40 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:40.095+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:40.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:40 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:40.095+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:40.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:40 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:40.095+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:40.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:40 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:40.500+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:40.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:40 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:40.500+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:40.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:40 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:40.500+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:40.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:40 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:40.318+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:40.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:40 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:40.318+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:40.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:40 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:40.318+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:40.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:40 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:40.487+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:40.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:40 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:40.487+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:40.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:40 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:40.487+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:41.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:40 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:40.663+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:41.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:40 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:40.663+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:41.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:40 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:40.663+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:41.480 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:41.072+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:41.480 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:41.072+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.874040+0000 front 2024-09-15T23:33:09.874025+0000 (oldest deadline 2024-09-15T23:33:35.173770+0000) 2024-09-15T23:33:41.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:41.061+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:41.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:41.061+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:41.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:41.061+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:41.791 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:41 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:41.535+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:41.791 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:41 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:41.535+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:41.791 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:41 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:41.535+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:41.791 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:41 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:41.351+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:41.791 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:41 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:41.351+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:41.792 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:41 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:41.351+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:41.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:41.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:41.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:41.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:41.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:41.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:42.021 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:41.693+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:42.021 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:41.693+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:42.021 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:41 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:41.693+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:42.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:42 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:42.020+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:42.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:42 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:42.020+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:42.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:42 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:42.020+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:42.480 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:42 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:42.063+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:42.480 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:42 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:42.063+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.874040+0000 front 2024-09-15T23:33:09.874025+0000 (oldest deadline 2024-09-15T23:33:35.173770+0000) 2024-09-15T23:33:42.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:42 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:42.509+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:42.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:42 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:42.509+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:42.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:42 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:42.509+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:42.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:42 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:42.325+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:42.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:42 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:42.325+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:42.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:42 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:42.325+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:42.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:42 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:42.481+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:42.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:42 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:42.481+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:42.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:42 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:42.481+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:43.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:42 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:42.721+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:43.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:42 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:42.721+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:43.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:42 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:42.721+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:43.463 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:43 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:43.102+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:43.464 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:43 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:43.102+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.874040+0000 front 2024-09-15T23:33:09.874025+0000 (oldest deadline 2024-09-15T23:33:35.173770+0000) 2024-09-15T23:33:43.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:43 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:43.037+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:43.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:43 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:43.037+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:43.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:43 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:43.037+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:43.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:43 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:43.463+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:43.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:43 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:43.463+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:43.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:43 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:43.463+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:43.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:43 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:43.463+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:43.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:43 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:43.518+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:43.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:43 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:43.518+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:43.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:43 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:43.518+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:43.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:43 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:43.321+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:43.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:43 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:43.321+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:43.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:43 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:43.321+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:44.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:43 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:43.709+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:44.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:43 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:43.709+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:44.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:43 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:43.709+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:44.476 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:44 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:44.145+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.471487+0000 front 2024-09-15T23:33:02.471479+0000 (oldest deadline 2024-09-15T23:33:24.771456+0000) 2024-09-15T23:33:44.476 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:44 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6[56318]: 2024-09-15T23:33:44.145+0000 7fa991368640 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.874040+0000 front 2024-09-15T23:33:09.874025+0000 (oldest deadline 2024-09-15T23:33:35.173770+0000) 2024-09-15T23:33:44.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:44 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:44.023+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:44.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:44 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:44.023+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:44.525 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:44 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:44.023+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:44.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:44 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:44.473+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:44.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:44 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:44.473+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:44.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:44 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:44.473+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:44.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:44 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:44.473+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:44.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:44 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:44.540+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:44.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:44 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:44.540+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:44.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:44 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:44.540+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:44.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:44 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:44.314+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:44.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:44 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:44.314+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:44.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:44 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:44.314+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:44.997 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:44 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:44.745+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:44.997 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:44 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:44.745+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:44.997 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:44 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:44.745+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:45.230 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:44 smithi062 podman[119158]: 2024-09-15 23:33:44.98636717 +0000 UTC m=+5.770087942 container died e002307370d8d19b46fd0e13c34f2e34615ae014f16a433ef3c553f7f2b8bdf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-15T23:33:45.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:44 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:44.996+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:45.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:44 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:44.996+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:45.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:44 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:44.996+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:45.730 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:45 smithi062 podman[119158]: 2024-09-15 23:33:45.493217635 +0000 UTC m=+6.276938410 container cleanup e002307370d8d19b46fd0e13c34f2e34615ae014f16a433ef3c553f7f2b8bdf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True) 2024-09-15T23:33:45.730 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:45 smithi062 bash[119158]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6 2024-09-15T23:33:45.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:45 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:45.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:45.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:45 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:45.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:45.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:45 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:45.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:45.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:45 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:45.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:45.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:45 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:45.579+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:45.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:45 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:45.579+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:45.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:45 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:45.579+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:45.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:45 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:45.340+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:45.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:45 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:45.340+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:45.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:45 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:45.340+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:46.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:45 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:45.745+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:46.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:45 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:45.745+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:46.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:45 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:45.745+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:46.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:45 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:45.983+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:46.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:45 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:45.983+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:46.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:45 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:45.983+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:46.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:45 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:45.983+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:46.230 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:45 smithi062 podman[119182]: 2024-09-15 23:33:45.886401047 +0000 UTC m=+0.886121585 container remove e002307370d8d19b46fd0e13c34f2e34615ae014f16a433ef3c553f7f2b8bdf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True) 2024-09-15T23:33:46.531 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:46 smithi062 podman[119273]: 2024-09-15 23:33:46.181002297 +0000 UTC m=+0.016849310 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:33:46.531 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:46 smithi062 podman[119273]: 2024-09-15 23:33:46.304188144 +0000 UTC m=+0.140035198 container create 94cd5ae835a34c8fe999d3e5e6cd327c1abf8cc6ea3b680ad7c1051a419d008a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-15T23:33:46.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:46 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:46.533+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:46.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:46 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:46.533+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:46.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:46 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:46.533+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:46.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:46 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:46.367+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:46.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:46 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:46.367+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:46.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:46 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:46.367+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:46.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:46 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:46.528+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:46.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:46 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:46.528+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:46.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:46 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:46.528+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:46.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:46 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:46.528+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:46.980 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:46 smithi062 podman[119273]: 2024-09-15 23:33:46.561630214 +0000 UTC m=+0.397477227 container init 94cd5ae835a34c8fe999d3e5e6cd327c1abf8cc6ea3b680ad7c1051a419d008a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:33:46.981 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:46 smithi062 podman[119273]: 2024-09-15 23:33:46.56505031 +0000 UTC m=+0.400897328 container start 94cd5ae835a34c8fe999d3e5e6cd327c1abf8cc6ea3b680ad7c1051a419d008a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-15T23:33:46.981 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:46 smithi062 podman[119273]: 2024-09-15 23:33:46.623511733 +0000 UTC m=+0.459358757 container attach 94cd5ae835a34c8fe999d3e5e6cd327c1abf8cc6ea3b680ad7c1051a419d008a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-15T23:33:46.981 INFO:journalctl@ceph.osd.6.smithi062.stdout:Sep 15 23:33:46 smithi062 podman[119273]: 2024-09-15 23:33:46.757571965 +0000 UTC m=+0.593418988 container died 94cd5ae835a34c8fe999d3e5e6cd327c1abf8cc6ea3b680ad7c1051a419d008a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-15T23:33:47.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:46 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:46.971+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:47.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:46 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:46.971+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:47.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:46 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:46.971+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:47.274 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:46 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:46.971+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:47.274 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:46 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:46.781+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:47.275 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:46 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:46.781+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:47.275 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:46 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:46.781+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:47.431 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.6.service' 2024-09-15T23:33:47.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:33:47.471 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-15T23:33:47.472 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-15T23:33:47.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.7 2024-09-15T23:33:47.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:47 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:47.514+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:47.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:47 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:47.514+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:47.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:47 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:47.514+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:47.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:47 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:47.514+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:47.759 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:47 smithi067 systemd[1]: Stopping Ceph osd.7 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:33:47.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:47 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:47.494+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:47.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:47 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:47.494+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:47.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:47 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:47.494+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:47.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:47 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:47.411+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:47.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:47 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:47.411+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:47.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:47 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:47.411+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:48.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:47.968+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:48.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:47.968+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:48.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:47.968+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:48.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:47.968+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:48.025 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:47.758+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:48.026 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:47.758+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:48.026 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:47.758+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:48.026 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:47.780+0000 7fbbf1541640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:33:48.026 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:47.780+0000 7fbbf1541640 -1 osd.7 96 *** Got signal Terminated *** 2024-09-15T23:33:48.026 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:47 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:47.780+0000 7fbbf1541640 -1 osd.7 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:33:48.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:48 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:48.467+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:48.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:48 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:48.467+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:48.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:48 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:48.467+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:48.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:48 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:48.467+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:48.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:48 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:48.456+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:48.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:48 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:48.456+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:48.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:48 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:48.456+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:48.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:48 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:48.401+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:48.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:48 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:48.401+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:48.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:48 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:48.401+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:49.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:48 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:48.749+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:49.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:48 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:48.749+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:49.025 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:48 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:48.749+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:49.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:48 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:48.940+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:49.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:48 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:48.940+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:49.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:48 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:48.940+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:49.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:48 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:48.940+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:49.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:49 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:49.463+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:49.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:49 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:49.463+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:49.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:49 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:49.463+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:49.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:49 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:49.436+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:49.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:49 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:49.436+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:49.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:49 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:49.436+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:49.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:49 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:49.436+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:49.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:49 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:49.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:49.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:49 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:49.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:49.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:49 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:49.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:49.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:49 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:49.508+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:50.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:49 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:49.739+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:50.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:49 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:49.739+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:50.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:49 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:49.739+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:50.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:49 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:49.903+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:50.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:49 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:49.903+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:50.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:49 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:49.903+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:50.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:49 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:49.903+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:50.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:50 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:50.435+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:50.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:50 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:50.435+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:50.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:50 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:50.435+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:50.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:50 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:50.404+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:50.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:50 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:50.404+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:50.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:50 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:50.404+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:50.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:50 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:50.404+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:50.982 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:50 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:50.484+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:51.116 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:50 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:50.484+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:51.116 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:50 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:50.484+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:51.116 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:50 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:50.484+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:51.117 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:50 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:50.723+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:51.117 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:50 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:50.723+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:51.117 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:50 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:50.723+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:51.117 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:50 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:50.868+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:51.117 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:50 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:50.868+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:51.117 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:50 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:50.868+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:51.117 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:50 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:50.868+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:51.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:51.436+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:51.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:51.436+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:51.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:51.436+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:51.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:51.436+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:33:51.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:51.363+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:51.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:51.363+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:51.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:51.363+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:51.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:51 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:51.363+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:51.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:51 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:51.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:51.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:51 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:51.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:51.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:51 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:51.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:51.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:51 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:51.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:52.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:51 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:51.725+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:52.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:51 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:51.725+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:52.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:51 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:51.725+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:52.025 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:51 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:51.725+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.914286+0000 front 2024-09-15T23:33:27.914315+0000 (oldest deadline 2024-09-15T23:33:50.814381+0000) 2024-09-15T23:33:52.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:51 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:51.848+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:52.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:51 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:51.848+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:52.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:51 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:51.848+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:52.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:51 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:51.848+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:52.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:51 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:51.848+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:33:52.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:52 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:52.481+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:52.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:52 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:52.481+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:52.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:52 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:52.481+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:52.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:52 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:52.481+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:33:52.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:52 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:52.358+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:52.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:52 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:52.358+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:52.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:52 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:52.358+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:52.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:52 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:52.358+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:52.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:52 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:52.531+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:52.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:52 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:52.531+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:52.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:52 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:52.531+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:52.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:52 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:52.531+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:52.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:52 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:52.531+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:33:53.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:52 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:52.727+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:01.809724+0000 front 2024-09-15T23:33:01.809847+0000 (oldest deadline 2024-09-15T23:33:27.709927+0000) 2024-09-15T23:33:53.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:52 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:52.727+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.311739+0000 front 2024-09-15T23:33:09.311561+0000 (oldest deadline 2024-09-15T23:33:37.512202+0000) 2024-09-15T23:33:53.025 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:52 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:52.727+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.512826+0000 front 2024-09-15T23:33:17.512894+0000 (oldest deadline 2024-09-15T23:33:40.412738+0000) 2024-09-15T23:33:53.025 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:52 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7[53546]: 2024-09-15T23:33:52.727+0000 7fbbe9b4f640 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.914286+0000 front 2024-09-15T23:33:27.914315+0000 (oldest deadline 2024-09-15T23:33:50.814381+0000) 2024-09-15T23:33:53.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:52 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:52.799+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:53.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:52 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:52.799+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:53.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:52 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:52.799+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:53.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:52 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:52.799+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:53.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:52 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:52.799+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:33:53.741 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:53 smithi067 podman[71523]: 2024-09-15 23:33:53.327007486 +0000 UTC m=+5.654287804 container died e150347fc53bfc0170e9c9283281339783374009ced5a80c888359da8e8d5948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-15T23:33:53.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:53 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:53.479+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:53.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:53 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:53.479+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:53.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:53 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:53.479+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:53.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:53 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:53.479+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:33:53.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:53 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:53.390+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:53.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:53 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:53.390+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:53.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:53 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:53.390+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:53.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:53 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:53.390+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:53.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:53 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:53.529+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:53.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:53 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:53.529+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:53.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:53 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:53.529+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:53.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:53 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:53.529+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:53.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:53 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:53.529+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:33:54.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:53 smithi067 podman[71523]: 2024-09-15 23:33:53.86481533 +0000 UTC m=+6.192095650 container cleanup e150347fc53bfc0170e9c9283281339783374009ced5a80c888359da8e8d5948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-15T23:33:54.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:53 smithi067 bash[71523]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7 2024-09-15T23:33:54.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:53 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:53.769+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:54.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:53 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:53.769+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:54.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:53 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:53.769+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:54.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:53 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:53.769+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:54.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:53 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:53.769+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:33:54.523 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:54 smithi067 podman[71538]: 2024-09-15 23:33:54.082815044 +0000 UTC m=+0.751682799 container remove e150347fc53bfc0170e9c9283281339783374009ced5a80c888359da8e8d5948 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-15T23:33:54.773 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:54 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:54.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:54.773 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:54 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:54.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:54.773 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:54 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:54.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:54.773 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:54 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:54.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:54.774 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:54 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:54.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:33:54.774 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:54 smithi067 podman[71624]: 2024-09-15 23:33:54.425472867 +0000 UTC m=+0.015973821 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:33:54.774 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:54 smithi067 podman[71624]: 2024-09-15 23:33:54.525165976 +0000 UTC m=+0.115666917 container create bba7f1491d111e5443a1fbeda9a031b4acaf0c7d97f82f356373803812e16c42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-15T23:33:54.774 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:54 smithi067 podman[71624]: 2024-09-15 23:33:54.769750028 +0000 UTC m=+0.360250966 container init bba7f1491d111e5443a1fbeda9a031b4acaf0c7d97f82f356373803812e16c42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:33:54.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:54 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:54.462+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:54.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:54 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:54.462+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:54.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:54 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:54.462+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:54.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:54 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:54.462+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:33:54.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:54 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:54.364+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:54.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:54 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:54.364+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:54.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:54 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:54.364+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:54.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:54 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:54.364+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:54.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:54 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:54.559+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:54.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:54 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:54.559+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:54.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:54 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:54.559+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:54.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:54 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:54.559+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:54.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:54 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:54.559+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:33:55.024 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:54 smithi067 podman[71624]: 2024-09-15 23:33:54.773050364 +0000 UTC m=+0.363551303 container start bba7f1491d111e5443a1fbeda9a031b4acaf0c7d97f82f356373803812e16c42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS) 2024-09-15T23:33:55.025 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:54 smithi067 podman[71624]: 2024-09-15 23:33:54.878228864 +0000 UTC m=+0.468729802 container attach bba7f1491d111e5443a1fbeda9a031b4acaf0c7d97f82f356373803812e16c42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-15T23:33:55.025 INFO:journalctl@ceph.osd.7.smithi067.stdout:Sep 15 23:33:54 smithi067 podman[71624]: 2024-09-15 23:33:54.964795096 +0000 UTC m=+0.555296035 container died bba7f1491d111e5443a1fbeda9a031b4acaf0c7d97f82f356373803812e16c42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-15T23:33:55.690 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:55 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:55.479+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:55.691 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:55 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:55.479+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:55.691 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:55 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:55.479+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:55.691 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:55 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:55.479+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:33:55.691 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:55 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:55.342+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:55.691 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:55 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:55.342+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:55.691 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:55 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:55.342+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:55.691 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:55 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:55.342+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:55.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.7.service' 2024-09-15T23:33:55.803 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:33:55.803 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-15T23:33:55.803 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-15T23:33:55.803 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.8 2024-09-15T23:33:55.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:55 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:55.556+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:55.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:55 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:55.556+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:55.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:55 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:55.556+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:55.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:55 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:55.556+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:55.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:55 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:55.556+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:33:55.996 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:55 smithi148 systemd[1]: Stopping Ceph osd.8 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:33:56.022 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:55 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:55.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:56.022 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:55 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:55.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:56.022 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:55 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:55.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:56.022 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:55 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:55.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:56.022 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:55 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:55.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:33:56.291 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:56.142+0000 7f596ece3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:33:56.291 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:56.142+0000 7f596ece3640 -1 osd.8 96 *** Got signal Terminated *** 2024-09-15T23:33:56.291 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:56.142+0000 7f596ece3640 -1 osd.8 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:33:56.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:56.432+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:56.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:56.432+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:56.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:56.432+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:56.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:56.432+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:33:56.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:56.301+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:56.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:56.301+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:56.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:56.301+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:56.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:56 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:56.301+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:56.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:56 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:56.536+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:56.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:56 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:56.536+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:56.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:56 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:56.536+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:56.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:56 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:56.536+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:56.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:56 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:56.536+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:33:57.000 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:56 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:56.721+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:57.001 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:56 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:56.721+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:57.001 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:56 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:56.721+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:57.001 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:56 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:56.721+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:57.001 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:56 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:56.721+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:33:57.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:57 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:57.412+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:57.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:57 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:57.412+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:57.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:57 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:57.412+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:57.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:57 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:57.412+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:33:57.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:57 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:57.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:57.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:57 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:57.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:57.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:57 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:57.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:57.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:57 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:57.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:57.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:57 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:57.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:33:57.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:57 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:57.533+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:57.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:57 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:57.533+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:57.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:57 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:57.533+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:57.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:57 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:57.533+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:57.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:57 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:57.533+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:33:57.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:57 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:57.533+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:33:58.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:57 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:57.708+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:58.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:57 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:57.708+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:58.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:57 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:57.708+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:58.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:57 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:57.708+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:58.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:57 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:57.708+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:33:58.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:58.423+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:58.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:58.423+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:58.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:58.423+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:58.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:58.423+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:33:58.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:58.423+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.504232+0000 front 2024-09-15T23:33:35.504275+0000 (oldest deadline 2024-09-15T23:33:58.404381+0000) 2024-09-15T23:33:58.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:58.353+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:58.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:58.353+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:58.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:58.353+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:58.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:58.353+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:58.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:58 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:58.353+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:33:58.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:58 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:58.517+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:58.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:58 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:58.517+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:58.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:58 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:58.517+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:58.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:58 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:58.517+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:58.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:58 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:58.517+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:33:58.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:58 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:58.517+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:33:59.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:58 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:58.681+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:33:59.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:58 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:58.681+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:33:59.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:58 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:58.681+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:33:59.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:58 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:58.681+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:33:59.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:58 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:58.681+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:33:59.791 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:59.451+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:33:59.791 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:59.451+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:33:59.791 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:59.451+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:33:59.791 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:59.451+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:33:59.791 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:33:59.451+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.504232+0000 front 2024-09-15T23:33:35.504275+0000 (oldest deadline 2024-09-15T23:33:58.404381+0000) 2024-09-15T23:33:59.791 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:59.374+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:33:59.791 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:59.374+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:33:59.792 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:59.374+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:33:59.792 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:59.374+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:33:59.792 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:33:59 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:33:59.374+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:33:59.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:59 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:59.488+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:33:59.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:59 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:59.488+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:33:59.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:59 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:59.488+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:33:59.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:59 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:59.488+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:33:59.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:59 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:59.488+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:33:59.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:33:59 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:33:59.488+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:00.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:59 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:59.695+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:00.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:59 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:59.696+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:00.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:59 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:59.696+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:00.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:59 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:59.696+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:00.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:33:59 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:33:59.696+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:00.789 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:34:00.481+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.496445+0000 front 2024-09-15T23:32:54.496282+0000 (oldest deadline 2024-09-15T23:33:16.196431+0000) 2024-09-15T23:34:00.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:34:00.481+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.498351+0000 front 2024-09-15T23:33:02.498387+0000 (oldest deadline 2024-09-15T23:33:24.198418+0000) 2024-09-15T23:34:00.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:34:00.481+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:17.501134+0000 front 2024-09-15T23:33:17.501231+0000 (oldest deadline 2024-09-15T23:33:40.401177+0000) 2024-09-15T23:34:00.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:34:00.481+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:26.202537+0000 front 2024-09-15T23:33:26.202579+0000 (oldest deadline 2024-09-15T23:33:50.902645+0000) 2024-09-15T23:34:00.790 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8[54416]: 2024-09-15T23:34:00.481+0000 7f596b2f9640 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.504232+0000 front 2024-09-15T23:33:35.504275+0000 (oldest deadline 2024-09-15T23:33:58.404381+0000) 2024-09-15T23:34:00.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:00.336+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:00.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:00.336+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:00.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:00.336+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:00.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:00.336+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:00.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:00 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:00.336+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:00.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:00.492+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:34:00.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:00.492+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:34:00.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:00.492+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:34:00.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:00.492+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:34:00.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:00.492+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:34:00.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:00 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:00.492+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:01.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:00 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:00.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:01.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:00 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:00.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:01.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:00 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:00.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:01.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:00 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:00.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:01.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:00 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:00.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:01.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:00 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:00.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:01.726 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:01 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:01.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:01.727 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:01 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:01.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:01.727 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:01 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:01.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:01.727 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:01 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:01.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:01.727 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:01 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:01.322+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:01.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:01.519+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:34:01.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:01.519+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:34:01.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:01.519+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:34:01.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:01.519+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:34:01.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:01.519+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:34:01.981 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:01 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:01.519+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:02.018 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:01 smithi148 podman[73146]: 2024-09-15 23:34:01.72609799 +0000 UTC m=+5.689064875 container died c28d748a6a1d5ec7f55df6b26cf453e30d6deb5e6981b3e69773ebcdc52ed8ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-15T23:34:02.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:01 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:01.756+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:02.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:01 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:01.756+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:02.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:01 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:01.756+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:02.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:01 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:01.756+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:02.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:01 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:01.756+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:02.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:01 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:01.756+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:02.289 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:02 smithi148 podman[73146]: 2024-09-15 23:34:02.125298741 +0000 UTC m=+6.088265616 container cleanup c28d748a6a1d5ec7f55df6b26cf453e30d6deb5e6981b3e69773ebcdc52ed8ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:34:02.289 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:02 smithi148 bash[73146]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8 2024-09-15T23:34:02.764 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:02 smithi148 podman[73173]: 2024-09-15 23:34:02.476503387 +0000 UTC m=+0.747033664 container remove c28d748a6a1d5ec7f55df6b26cf453e30d6deb5e6981b3e69773ebcdc52ed8ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-09-15T23:34:02.765 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:02 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:02.331+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:02.765 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:02 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:02.331+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:02.765 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:02 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:02.331+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:02.765 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:02 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:02.331+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:02.765 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:02 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:02.331+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:02.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:02 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:02.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:34:02.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:02 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:02.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:34:02.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:02 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:02.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:34:02.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:02 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:02.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:34:02.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:02 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:02.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:34:02.980 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:02 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:02.493+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:03.017 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:02 smithi148 podman[73261]: 2024-09-15 23:34:02.66695977 +0000 UTC m=+0.016409436 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:34:03.017 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:02 smithi148 podman[73261]: 2024-09-15 23:34:02.76878932 +0000 UTC m=+0.118238980 container create 5f5e7640fe32f7ea71ac17eae632952c873adc5940cfaf47068657f34b7230d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e) 2024-09-15T23:34:03.017 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:03 smithi148 podman[73261]: 2024-09-15 23:34:03.013189308 +0000 UTC m=+0.362638965 container init 5f5e7640fe32f7ea71ac17eae632952c873adc5940cfaf47068657f34b7230d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-15T23:34:03.017 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:03 smithi148 podman[73261]: 2024-09-15 23:34:03.016343191 +0000 UTC m=+0.365792851 container start 5f5e7640fe32f7ea71ac17eae632952c873adc5940cfaf47068657f34b7230d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-15T23:34:03.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:02 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:02.746+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:03.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:02 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:02.746+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:03.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:02 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:02.746+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:03.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:02 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:02.746+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:03.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:02 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:02.746+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:03.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:02 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:02.746+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:03.289 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:03 smithi148 podman[73261]: 2024-09-15 23:34:03.130213904 +0000 UTC m=+0.479663563 container attach 5f5e7640fe32f7ea71ac17eae632952c873adc5940cfaf47068657f34b7230d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-15T23:34:03.289 INFO:journalctl@ceph.osd.8.smithi148.stdout:Sep 15 23:34:03 smithi148 podman[73261]: 2024-09-15 23:34:03.207166368 +0000 UTC m=+0.556616028 container died 5f5e7640fe32f7ea71ac17eae632952c873adc5940cfaf47068657f34b7230d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-15T23:34:03.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:03 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:03.445+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:34:03.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:03 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:03.445+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:34:03.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:03 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:03.445+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:34:03.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:03 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:03.445+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:34:03.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:03 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:03.445+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:34:03.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:03 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:03.445+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:03.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:03 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:03.348+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:03.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:03 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:03.348+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:03.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:03 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:03.348+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:03.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:03 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:03.348+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:03.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:03 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:03.348+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:03.859 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.8.service' 2024-09-15T23:34:03.902 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:03.903 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-15T23:34:03.903 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-15T23:34:03.903 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.9 2024-09-15T23:34:04.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:03 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:03.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:04.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:03 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:03.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:04.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:03 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:03.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:04.026 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:03 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:03.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:04.026 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:03 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:03.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:04.026 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:03 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:03.730+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:04.214 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:03 smithi062 systemd[1]: Stopping Ceph osd.9 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:04.214 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:04.213+0000 7fcfb491f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:04.214 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:04.213+0000 7fcfb491f640 -1 osd.9 96 *** Got signal Terminated *** 2024-09-15T23:34:04.480 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:04.213+0000 7fcfb491f640 -1 osd.9 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:34:04.480 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:04.400+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:34:04.480 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:04.400+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:34:04.481 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:04.400+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:34:04.481 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:04.400+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:34:04.481 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:04.400+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:34:04.481 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:04 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:04.400+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:04.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:04 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:04.337+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:04.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:04 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:04.337+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:04.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:04 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:04.337+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:04.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:04 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:04.337+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:04.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:04 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:04.337+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:05.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:04 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:04.731+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:05.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:04 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:04.731+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:05.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:04 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:04.731+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:05.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:04 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:04.731+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:05.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:04 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:04.731+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:05.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:04 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:04.731+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:05.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:05 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:05.375+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:34:05.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:05 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:05.375+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:34:05.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:05 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:05.375+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:34:05.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:05 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:05.375+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:34:05.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:05 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:05.375+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:34:05.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:05 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:05.375+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:05.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:05 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:05.290+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:05.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:05 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:05.290+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:05.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:05 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:05.290+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:05.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:05 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:05.290+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:05.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:05 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:05.290+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:06.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:05 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:05.767+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:06.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:05 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:05.767+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:06.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:05 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:05.767+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:06.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:05 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:05.767+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:06.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:05 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:05.767+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:06.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:05 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:05.767+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:06.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:06 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:06.284+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:06.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:06 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:06.284+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:06.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:06 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:06.284+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:06.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:06 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:06.284+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:06.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:06 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:06.284+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:06.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:06 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:06.341+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:34:06.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:06 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:06.341+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:34:06.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:06 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:06.341+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:34:06.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:06 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:06.341+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:34:06.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:06 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:06.341+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:34:06.731 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:06 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:06.341+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:07.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:06 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:06.771+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:07.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:06 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:06.771+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:07.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:06 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:06.771+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:07.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:06 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:06.771+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:07.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:06 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:06.771+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:07.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:06 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:06.771+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:07.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:07 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:07.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:07.541 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:07 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:07.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:07.541 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:07 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:07.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:07.541 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:07 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:07.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:07.541 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:07 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:07.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:07.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:07 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:07.309+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:34:07.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:07 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:07.309+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:34:07.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:07 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:07.309+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:34:07.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:07 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:07.309+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:34:07.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:07 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:07.309+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:34:07.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:07 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:07.309+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:08.026 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:07 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:07.743+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:08.026 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:07 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:07.743+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:08.026 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:07 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:07.743+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:08.026 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:07 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:07.743+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:08.026 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:07 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:07.743+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:08.026 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:07 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:07.743+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:08.027 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:07 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:07.744+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:08.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:08 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:08.249+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:08.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:08 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:08.249+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:08.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:08 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:08.249+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:08.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:08 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:08.249+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:08.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:08 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:08.249+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:08.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:08 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:08.249+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:08.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:08 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:08.273+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:53.541187+0000 front 2024-09-15T23:32:53.541220+0000 (oldest deadline 2024-09-15T23:33:18.241318+0000) 2024-09-15T23:34:08.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:08 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:08.273+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.843269+0000 front 2024-09-15T23:33:02.843497+0000 (oldest deadline 2024-09-15T23:33:28.743308+0000) 2024-09-15T23:34:08.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:08 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:08.273+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:10.444530+0000 front 2024-09-15T23:33:10.444513+0000 (oldest deadline 2024-09-15T23:33:36.845367+0000) 2024-09-15T23:34:08.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:08 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:08.273+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:16.845815+0000 front 2024-09-15T23:33:16.845778+0000 (oldest deadline 2024-09-15T23:33:43.246646+0000) 2024-09-15T23:34:08.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:08 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:08.273+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:27.848654+0000 front 2024-09-15T23:33:27.848511+0000 (oldest deadline 2024-09-15T23:33:51.948639+0000) 2024-09-15T23:34:08.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:08 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9[62868]: 2024-09-15T23:34:08.273+0000 7fcfacf2d640 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:35.851077+0000 front 2024-09-15T23:33:35.851183+0000 (oldest deadline 2024-09-15T23:33:56.950902+0000) 2024-09-15T23:34:09.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:08 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:08.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:09.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:08 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:08.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:09.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:08 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:08.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:09.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:08 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:08.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:09.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:08 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:08.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:09.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:08 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:08.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:09.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:08 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:08.749+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:09.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:09 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:09.207+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:09.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:09 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:09.207+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:09.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:09 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:09.207+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:09.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:09 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:09.207+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:09.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:09 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:09.207+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:09.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:09 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:09.207+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:10.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:09 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:09.740+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:10.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:09 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:09.740+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:10.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:09 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:09.740+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:10.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:09 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:09.740+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:10.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:09 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:09.740+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:10.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:09 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:09.740+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:10.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:09 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:09.740+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:10.043 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:09 smithi062 podman[119378]: 2024-09-15 23:34:09.785707988 +0000 UTC m=+5.679627519 container died b628b8da1a7d8b755356bee7ffa300d9871d38090bd24fc74e0256563afac6cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:34:10.423 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:10 smithi062 podman[119378]: 2024-09-15 23:34:10.196515063 +0000 UTC m=+6.090434592 container cleanup b628b8da1a7d8b755356bee7ffa300d9871d38090bd24fc74e0256563afac6cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-15T23:34:10.424 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:10 smithi062 bash[119378]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9 2024-09-15T23:34:10.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:10 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:10.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:10.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:10 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:10.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:10.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:10 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:10.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:10.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:10 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:10.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:10.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:10 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:10.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:10.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:10 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:10.250+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:10.730 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:10 smithi062 podman[119395]: 2024-09-15 23:34:10.422880899 +0000 UTC m=+0.640173005 container remove b628b8da1a7d8b755356bee7ffa300d9871d38090bd24fc74e0256563afac6cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-15T23:34:11.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:10 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:10.713+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:11.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:10 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:10.713+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:11.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:10 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:10.713+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:11.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:10 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:10.713+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:11.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:10 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:10.713+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:11.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:10 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:10.713+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:11.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:10 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:10.713+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:11.169 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:10 smithi062 podman[119485]: 2024-09-15 23:34:10.733306052 +0000 UTC m=+0.016325909 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:34:11.169 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:10 smithi062 podman[119485]: 2024-09-15 23:34:10.832124634 +0000 UTC m=+0.115144488 container create f1b95cceda0bcd319ac6fdb4e4280cd1271da5973ece313756701ea5a61c6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:34:11.169 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:11 smithi062 podman[119485]: 2024-09-15 23:34:11.076503067 +0000 UTC m=+0.359522922 container init f1b95cceda0bcd319ac6fdb4e4280cd1271da5973ece313756701ea5a61c6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-15T23:34:11.169 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:11 smithi062 podman[119485]: 2024-09-15 23:34:11.079999641 +0000 UTC m=+0.363019503 container start f1b95cceda0bcd319ac6fdb4e4280cd1271da5973ece313756701ea5a61c6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-15T23:34:11.169 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:11 smithi062 podman[119485]: 2024-09-15 23:34:11.167757087 +0000 UTC m=+0.450776976 container attach f1b95cceda0bcd319ac6fdb4e4280cd1271da5973ece313756701ea5a61c6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9-deactivate, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-15T23:34:11.480 INFO:journalctl@ceph.osd.9.smithi062.stdout:Sep 15 23:34:11 smithi062 podman[119485]: 2024-09-15 23:34:11.270569533 +0000 UTC m=+0.553589396 container died f1b95cceda0bcd319ac6fdb4e4280cd1271da5973ece313756701ea5a61c6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-15T23:34:11.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:11 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:11.224+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:11.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:11 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:11.224+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:11.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:11 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:11.224+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:11.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:11 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:11.224+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:11.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:11 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:11.224+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:11.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:11 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:11.224+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:11.858 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.9.service' 2024-09-15T23:34:11.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:11.901 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-15T23:34:11.902 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-15T23:34:11.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.10 2024-09-15T23:34:12.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:11 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:11.714+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:12.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:11 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:11.714+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:12.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:11 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:11.714+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:12.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:11 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:11.714+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:12.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:11 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:11.714+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:12.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:11 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:11.714+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:12.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:11 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:11.714+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:12.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:11 smithi067 systemd[1]: Stopping Ceph osd.10 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:12.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:12 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:12.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:12.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:12 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:12.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:12.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:12 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:12.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:12.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:12 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:12.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:12.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:12 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:12.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:12.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:12 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:12.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:12.734 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.342+0000 7f8fc254b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:12.734 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.342+0000 7f8fc254b640 -1 osd.10 96 *** Got signal Terminated *** 2024-09-15T23:34:12.735 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.342+0000 7f8fc254b640 -1 osd.10 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:34:13.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.733+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:13.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.733+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:13.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.733+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:13.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.733+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:13.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.733+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:13.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.733+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:13.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:12 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:12.733+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:13.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:13 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:13.247+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:13.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:13 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:13.247+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:13.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:13 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:13.247+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:13.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:13 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:13.247+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:13.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:13 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:13.247+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:13.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:13 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:13.247+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:14.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:13 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:13.698+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:14.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:13 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:13.698+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:14.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:13 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:13.698+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:14.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:13 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:13.698+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:14.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:13 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:13.698+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:14.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:13 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:13.698+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:14.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:13 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:13.698+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:14.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:14 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:14.219+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:14.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:14 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:14.219+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:14.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:14 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:14.219+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:14.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:14 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:14.219+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:14.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:14 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:14.219+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:14.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:14 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:14.219+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:15.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:14 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:14.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:15.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:14 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:14.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:15.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:14 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:14.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:15.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:14 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:14.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:15.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:14 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:14.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:15.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:14 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:14.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:15.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:14 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:14.725+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:15.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:15 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:15.180+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:15.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:15 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:15.180+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:15.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:15 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:15.180+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:15.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:15 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:15.180+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:15.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:15 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:15.180+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:15.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:15 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:15.180+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:16.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:15 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:15.699+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:16.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:15 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:15.699+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:16.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:15 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:15.699+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:16.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:15 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:15.699+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:16.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:15 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:15.699+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:16.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:15 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:15.699+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:16.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:15 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:15.699+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:16.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:16 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:16.202+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:16.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:16 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:16.202+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:16.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:16 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:16.202+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:16.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:16 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:16.202+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:16.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:16 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:16.202+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:16.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:16 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:16.202+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:17.024 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:16 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:16.744+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:51.793832+0000 front 2024-09-15T23:32:51.793994+0000 (oldest deadline 2024-09-15T23:33:16.493950+0000) 2024-09-15T23:34:17.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:16 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:16.744+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:00.595007+0000 front 2024-09-15T23:33:00.595081+0000 (oldest deadline 2024-09-15T23:33:26.495120+0000) 2024-09-15T23:34:17.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:16 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:16.744+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:09.097893+0000 front 2024-09-15T23:33:09.097822+0000 (oldest deadline 2024-09-15T23:33:33.797754+0000) 2024-09-15T23:34:17.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:16 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:16.744+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6814 osd.3 since back 2024-09-15T23:33:19.698770+0000 front 2024-09-15T23:33:19.698850+0000 (oldest deadline 2024-09-15T23:33:45.598788+0000) 2024-09-15T23:34:17.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:16 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:16.744+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.599141+0000 front 2024-09-15T23:33:25.599108+0000 (oldest deadline 2024-09-15T23:33:51.499344+0000) 2024-09-15T23:34:17.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:16 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:16.744+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.400360+0000 front 2024-09-15T23:33:34.400381+0000 (oldest deadline 2024-09-15T23:33:59.700189+0000) 2024-09-15T23:34:17.025 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:16 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10[60066]: 2024-09-15T23:34:16.744+0000 7f8fbab59640 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:44.401298+0000 front 2024-09-15T23:33:44.401244+0000 (oldest deadline 2024-09-15T23:34:07.301006+0000) 2024-09-15T23:34:17.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:17 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:17.231+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:17.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:17 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:17.231+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:17.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:17 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:17.231+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:17.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:17 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:17.231+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:17.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:17 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:17.231+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:17.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:17 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:17.231+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:17.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:17 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:17.231+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6820 osd.7 since back 2024-09-15T23:33:50.976126+0000 front 2024-09-15T23:33:50.976115+0000 (oldest deadline 2024-09-15T23:34:16.876150+0000) 2024-09-15T23:34:18.060 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:17 smithi067 podman[71738]: 2024-09-15 23:34:17.805966645 +0000 UTC m=+5.702365985 container died de6e1146ab60717cfde3559779e5bcdf1129eb11c7766d574754dc55d6123355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:34:18.428 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:18 smithi067 podman[71738]: 2024-09-15 23:34:18.242872854 +0000 UTC m=+6.139272186 container cleanup de6e1146ab60717cfde3559779e5bcdf1129eb11c7766d574754dc55d6123355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, io.buildah.version=1.37.2) 2024-09-15T23:34:18.428 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:18 smithi067 bash[71738]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10 2024-09-15T23:34:18.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:18.255+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:18.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:18.255+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:18.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:18.255+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:18.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:18.255+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:18.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:18.255+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:18.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:18.255+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:18.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:18 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:18.255+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6820 osd.7 since back 2024-09-15T23:33:50.976126+0000 front 2024-09-15T23:33:50.976115+0000 (oldest deadline 2024-09-15T23:34:16.876150+0000) 2024-09-15T23:34:18.774 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:18 smithi067 podman[71753]: 2024-09-15 23:34:18.427739188 +0000 UTC m=+0.616431317 container remove de6e1146ab60717cfde3559779e5bcdf1129eb11c7766d574754dc55d6123355 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-15T23:34:19.172 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:18 smithi067 podman[71840]: 2024-09-15 23:34:18.761184412 +0000 UTC m=+0.016148121 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:34:19.172 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:18 smithi067 podman[71840]: 2024-09-15 23:34:18.878084311 +0000 UTC m=+0.133048014 container create 6edb29cacce4387194ced088dbbfa69f531c4649de72fe92fa1d28a7f94cea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0) 2024-09-15T23:34:19.173 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:19 smithi067 podman[71840]: 2024-09-15 23:34:19.096864781 +0000 UTC m=+0.351828481 container init 6edb29cacce4387194ced088dbbfa69f531c4649de72fe92fa1d28a7f94cea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:34:19.173 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:19 smithi067 podman[71840]: 2024-09-15 23:34:19.100043427 +0000 UTC m=+0.355007127 container start 6edb29cacce4387194ced088dbbfa69f531c4649de72fe92fa1d28a7f94cea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:34:19.173 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:19 smithi067 podman[71840]: 2024-09-15 23:34:19.171875348 +0000 UTC m=+0.426839048 container attach 6edb29cacce4387194ced088dbbfa69f531c4649de72fe92fa1d28a7f94cea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-15T23:34:19.524 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:19 smithi067 podman[71840]: 2024-09-15 23:34:19.289830122 +0000 UTC m=+0.544793830 container died 6edb29cacce4387194ced088dbbfa69f531c4649de72fe92fa1d28a7f94cea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10-deactivate, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:34:19.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:19.208+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:19.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:19.208+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:19.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:19.208+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:19.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:19.208+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:19.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:19.208+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:19.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:19.208+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:19.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:19 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:19.208+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6820 osd.7 since back 2024-09-15T23:33:50.976126+0000 front 2024-09-15T23:33:50.976115+0000 (oldest deadline 2024-09-15T23:34:16.876150+0000) 2024-09-15T23:34:20.102 INFO:journalctl@ceph.osd.10.smithi067.stdout:Sep 15 23:34:20 smithi067 podman[71863]: 2024-09-15 23:34:20.083524587 +0000 UTC m=+0.787731121 container remove 6edb29cacce4387194ced088dbbfa69f531c4649de72fe92fa1d28a7f94cea57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:34:20.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.10.service' 2024-09-15T23:34:20.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:20.149 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-15T23:34:20.149 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-15T23:34:20.149 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.11 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.189+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.189+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.189+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.189+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.189+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.189+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.189+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6820 osd.7 since back 2024-09-15T23:33:50.976126+0000 front 2024-09-15T23:33:50.976115+0000 (oldest deadline 2024-09-15T23:34:16.876150+0000) 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 systemd[1]: Stopping Ceph osd.11 for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.493+0000 7f4ca943d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:20.496 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.493+0000 7f4ca943d640 -1 osd.11 96 *** Got signal Terminated *** 2024-09-15T23:34:20.497 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:20 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:20.493+0000 7f4ca943d640 -1 osd.11 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-15T23:34:21.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:21.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:21.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:21.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:21.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:21.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:21.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:21.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:21.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:21.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:21.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:21.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:21.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:21 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:21.240+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6820 osd.7 since back 2024-09-15T23:33:50.976126+0000 front 2024-09-15T23:33:50.976115+0000 (oldest deadline 2024-09-15T23:34:16.876150+0000) 2024-09-15T23:34:22.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:22.279+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:22.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:22.279+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:22.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:22.279+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:22.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:22.279+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:22.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:22.279+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:22.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:22.279+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:22.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:22 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:22.279+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6820 osd.7 since back 2024-09-15T23:33:50.976126+0000 front 2024-09-15T23:33:50.976115+0000 (oldest deadline 2024-09-15T23:34:16.876150+0000) 2024-09-15T23:34:23.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:23.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:23.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:23.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:23.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:23.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:23.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:23.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:23.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:23.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:23.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:23.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:23.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:23.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6820 osd.7 since back 2024-09-15T23:33:50.976126+0000 front 2024-09-15T23:33:50.976115+0000 (oldest deadline 2024-09-15T23:34:16.876150+0000) 2024-09-15T23:34:23.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:23 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:23.286+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6820 osd.8 since back 2024-09-15T23:33:59.777060+0000 front 2024-09-15T23:33:59.777039+0000 (oldest deadline 2024-09-15T23:34:22.677073+0000) 2024-09-15T23:34:24.789 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:24.301+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:24.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:24.302+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:24.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:24.302+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:24.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:24.302+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:24.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:24.302+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:24.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:24.302+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:24.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:24.302+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6820 osd.7 since back 2024-09-15T23:33:50.976126+0000 front 2024-09-15T23:33:50.976115+0000 (oldest deadline 2024-09-15T23:34:16.876150+0000) 2024-09-15T23:34:24.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:24 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:24.302+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6820 osd.8 since back 2024-09-15T23:33:59.777060+0000 front 2024-09-15T23:33:59.777039+0000 (oldest deadline 2024-09-15T23:34:22.677073+0000) 2024-09-15T23:34:25.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:25.266+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-09-15T23:32:54.466759+0000 front 2024-09-15T23:32:54.466643+0000 (oldest deadline 2024-09-15T23:33:16.766658+0000) 2024-09-15T23:34:25.539 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:25.266+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6804 osd.1 since back 2024-09-15T23:33:02.568813+0000 front 2024-09-15T23:33:02.568531+0000 (oldest deadline 2024-09-15T23:33:28.468639+0000) 2024-09-15T23:34:25.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:25.266+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6804 osd.2 since back 2024-09-15T23:33:11.370027+0000 front 2024-09-15T23:33:11.370065+0000 (oldest deadline 2024-09-15T23:33:33.670094+0000) 2024-09-15T23:34:25.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:25.266+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6812 osd.4 since back 2024-09-15T23:33:25.972096+0000 front 2024-09-15T23:33:25.972064+0000 (oldest deadline 2024-09-15T23:33:48.872133+0000) 2024-09-15T23:34:25.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:25.266+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6812 osd.5 since back 2024-09-15T23:33:34.173167+0000 front 2024-09-15T23:33:34.173103+0000 (oldest deadline 2024-09-15T23:33:57.073188+0000) 2024-09-15T23:34:25.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:25.266+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.62:6822 osd.6 since back 2024-09-15T23:33:42.274938+0000 front 2024-09-15T23:33:42.275026+0000 (oldest deadline 2024-09-15T23:34:08.174817+0000) 2024-09-15T23:34:25.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:25.266+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.67:6820 osd.7 since back 2024-09-15T23:33:50.976126+0000 front 2024-09-15T23:33:50.976115+0000 (oldest deadline 2024-09-15T23:34:16.876150+0000) 2024-09-15T23:34:25.540 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:25 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11[61413]: 2024-09-15T23:34:25.266+0000 7f4ca5a53640 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.148:6820 osd.8 since back 2024-09-15T23:33:59.777060+0000 front 2024-09-15T23:33:59.777039+0000 (oldest deadline 2024-09-15T23:34:22.677073+0000) 2024-09-15T23:34:26.480 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:26 smithi148 podman[73363]: 2024-09-15 23:34:26.163744699 +0000 UTC m=+5.810826585 container stop 182856f661c0e2498a60bcee02b26ee1a5aadf8ee82ae5a83acb784e6449ecbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-15T23:34:26.481 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:26 smithi148 podman[73363]: 2024-09-15 23:34:26.189490251 +0000 UTC m=+5.836572132 container died 182856f661c0e2498a60bcee02b26ee1a5aadf8ee82ae5a83acb784e6449ecbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-15T23:34:26.781 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:26 smithi148 podman[73363]: 2024-09-15 23:34:26.579630822 +0000 UTC m=+6.226712745 container cleanup 182856f661c0e2498a60bcee02b26ee1a5aadf8ee82ae5a83acb784e6449ecbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:34:26.781 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:26 smithi148 bash[73363]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11 2024-09-15T23:34:27.044 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:26 smithi148 podman[73378]: 2024-09-15 23:34:26.780568089 +0000 UTC m=+0.610249670 container remove 182856f661c0e2498a60bcee02b26ee1a5aadf8ee82ae5a83acb784e6449ecbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:34:27.515 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:27 smithi148 podman[73470]: 2024-09-15 23:34:27.181390396 +0000 UTC m=+0.111493895 container create 57c7c4f3d4d3987e659df43d79565acd639360f5062671e93e67101735661721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-15T23:34:27.515 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:27 smithi148 podman[73470]: 2024-09-15 23:34:27.085928651 +0000 UTC m=+0.016032152 image pull f1455bd0219c0836c14d3880561c736ab957883ab2384892956003bb36edc4ee quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80 2024-09-15T23:34:27.515 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:27 smithi148 podman[73470]: 2024-09-15 23:34:27.408617584 +0000 UTC m=+0.338721082 container init 57c7c4f3d4d3987e659df43d79565acd639360f5062671e93e67101735661721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-8921c8e) 2024-09-15T23:34:27.515 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:27 smithi148 podman[73470]: 2024-09-15 23:34:27.411754666 +0000 UTC m=+0.341858165 container start 57c7c4f3d4d3987e659df43d79565acd639360f5062671e93e67101735661721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-09-15T23:34:27.515 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:27 smithi148 podman[73470]: 2024-09-15 23:34:27.5085072 +0000 UTC m=+0.438610713 container attach 57c7c4f3d4d3987e659df43d79565acd639360f5062671e93e67101735661721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:34:27.790 INFO:journalctl@ceph.osd.11.smithi148.stdout:Sep 15 23:34:27 smithi148 podman[73470]: 2024-09-15 23:34:27.599658188 +0000 UTC m=+0.529761705 container died 57c7c4f3d4d3987e659df43d79565acd639360f5062671e93e67101735661721 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e) 2024-09-15T23:34:28.165 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@osd.11.service' 2024-09-15T23:34:28.204 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:28.205 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-15T23:34:28.205 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-09-15T23:34:28.205 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.a 2024-09-15T23:34:28.533 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:34:28 smithi062 systemd[1]: Stopping Ceph mds.a for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:28.953 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:34:28 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-a[69474]: 2024-09-15T23:34:28.531+0000 7f887135c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:28.954 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:34:28 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-a[69474]: 2024-09-15T23:34:28.531+0000 7f887135c640 -1 mds.a *** got signal Terminated *** 2024-09-15T23:34:32.143 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:34:31 smithi062 podman[119586]: 2024-09-15 23:34:31.850367365 +0000 UTC m=+3.441823004 container died 334aab8fbdd1e62e8d3e52599cfdfb40536d9fb44372308f03d72535c42e66aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:34:32.480 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:34:32 smithi062 podman[119586]: 2024-09-15 23:34:32.31441522 +0000 UTC m=+3.905870829 container cleanup 334aab8fbdd1e62e8d3e52599cfdfb40536d9fb44372308f03d72535c42e66aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-15T23:34:32.480 INFO:journalctl@ceph.mds.a.smithi062.stdout:Sep 15 23:34:32 smithi062 bash[119586]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-a 2024-09-15T23:34:32.715 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.a.service' 2024-09-15T23:34:32.754 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:32.754 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-09-15T23:34:32.755 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-09-15T23:34:32.755 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.d 2024-09-15T23:34:33.086 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:34:32 smithi062 systemd[1]: Stopping Ceph mds.d for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:33.480 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:34:33 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-d[68289]: 2024-09-15T23:34:33.084+0000 7f0e04996640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:33.480 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:34:33 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-d[68289]: 2024-09-15T23:34:33.084+0000 7f0e04996640 -1 mds.d *** got signal Terminated *** 2024-09-15T23:34:36.229 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:34:35 smithi062 podman[119720]: 2024-09-15 23:34:35.970104718 +0000 UTC m=+2.991512410 container died 46315ff11dd3ad86d6e11aaf0d8cacbb95ce7fd03cc19c478aca59f3736d834d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-d, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-15T23:34:36.480 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:34:36 smithi062 podman[119720]: 2024-09-15 23:34:36.377112365 +0000 UTC m=+3.398520045 container cleanup 46315ff11dd3ad86d6e11aaf0d8cacbb95ce7fd03cc19c478aca59f3736d834d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-d, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-15T23:34:36.480 INFO:journalctl@ceph.mds.d.smithi062.stdout:Sep 15 23:34:36 smithi062 bash[119720]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-d 2024-09-15T23:34:36.750 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.d.service' 2024-09-15T23:34:36.787 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:36.787 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-09-15T23:34:36.787 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-09-15T23:34:36.788 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.g 2024-09-15T23:34:37.230 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:34:36 smithi062 systemd[1]: Stopping Ceph mds.g for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:37.230 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:34:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-g[69060]: 2024-09-15T23:34:37.114+0000 7fb528950640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:37.230 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:34:37 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-g[69060]: 2024-09-15T23:34:37.114+0000 7fb528950640 -1 mds.g *** got signal Terminated *** 2024-09-15T23:34:40.633 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:34:40 smithi062 podman[119841]: 2024-09-15 23:34:40.362329667 +0000 UTC m=+3.352375688 container died 7c00931df3a67b1c2f94ca312e13f3e94274d6cd44b3d0eba404506a00e2c9a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-15T23:34:40.980 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:34:40 smithi062 podman[119841]: 2024-09-15 23:34:40.814887352 +0000 UTC m=+3.804933448 container cleanup 7c00931df3a67b1c2f94ca312e13f3e94274d6cd44b3d0eba404506a00e2c9a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True) 2024-09-15T23:34:40.980 INFO:journalctl@ceph.mds.g.smithi062.stdout:Sep 15 23:34:40 smithi062 bash[119841]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-g 2024-09-15T23:34:41.255 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.g.service' 2024-09-15T23:34:41.290 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:41.290 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-09-15T23:34:41.290 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-09-15T23:34:41.290 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.j 2024-09-15T23:34:41.611 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:34:41 smithi062 systemd[1]: Stopping Ceph mds.j for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:41.980 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:34:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-j[67896]: 2024-09-15T23:34:41.610+0000 7fa5fb051640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:41.980 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:34:41 smithi062 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-j[67896]: 2024-09-15T23:34:41.610+0000 7fa5fb051640 -1 mds.j *** got signal Terminated *** 2024-09-15T23:34:44.980 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:34:44 smithi062 podman[119960]: 2024-09-15 23:34:44.647254588 +0000 UTC m=+3.126533167 container died 51f3dd19b664cc728dc92fb6e16be36049123b9baba2320c4eaedf468562df6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-j, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-15T23:34:45.312 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:34:45 smithi062 podman[119960]: 2024-09-15 23:34:45.00227451 +0000 UTC m=+3.481553044 container cleanup 51f3dd19b664cc728dc92fb6e16be36049123b9baba2320c4eaedf468562df6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-j, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:34:45.312 INFO:journalctl@ceph.mds.j.smithi062.stdout:Sep 15 23:34:45 smithi062 bash[119960]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-j 2024-09-15T23:34:45.446 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.j.service' 2024-09-15T23:34:45.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:45.480 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-09-15T23:34:45.480 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-09-15T23:34:45.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.b 2024-09-15T23:34:45.774 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:34:45 smithi067 systemd[1]: Stopping Ceph mds.b for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:46.274 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:34:45 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: 2024-09-15T23:34:45.795+0000 7fda91539640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:46.274 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:34:45 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b[65710]: 2024-09-15T23:34:45.795+0000 7fda91539640 -1 mds.b *** got signal Terminated *** 2024-09-15T23:34:50.070 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:34:49 smithi067 podman[71941]: 2024-09-15 23:34:49.809524367 +0000 UTC m=+4.131530929 container died c2da18fa89c1be1359be7ec9d2da06472591065ddab16b18dbf7caec8a633b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-15T23:34:50.419 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:34:50 smithi067 podman[71941]: 2024-09-15 23:34:50.159465329 +0000 UTC m=+4.481471897 container cleanup c2da18fa89c1be1359be7ec9d2da06472591065ddab16b18dbf7caec8a633b9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:34:50.419 INFO:journalctl@ceph.mds.b.smithi067.stdout:Sep 15 23:34:50 smithi067 bash[71941]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-b 2024-09-15T23:34:50.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.b.service' 2024-09-15T23:34:50.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:50.636 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-09-15T23:34:50.636 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-09-15T23:34:50.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.e 2024-09-15T23:34:50.964 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:34:50 smithi067 systemd[1]: Stopping Ceph mds.e for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:51.274 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:34:50 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e[65099]: 2024-09-15T23:34:50.962+0000 7f74e4a17640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:51.274 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:34:50 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e[65099]: 2024-09-15T23:34:50.963+0000 7f74e4a17640 -1 mds.e *** got signal Terminated *** 2024-09-15T23:34:55.024 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:34:54 smithi067 podman[72074]: 2024-09-15 23:34:54.742725139 +0000 UTC m=+3.886149523 container died 0e3456702d7a859fd56fcc5d3c1884c94dc223658c07e49040d0f46f03186a0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-15T23:34:55.440 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:34:55 smithi067 podman[72074]: 2024-09-15 23:34:55.180049938 +0000 UTC m=+4.323474321 container cleanup 0e3456702d7a859fd56fcc5d3c1884c94dc223658c07e49040d0f46f03186a0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731) 2024-09-15T23:34:55.440 INFO:journalctl@ceph.mds.e.smithi067.stdout:Sep 15 23:34:55 smithi067 bash[72074]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-e 2024-09-15T23:34:55.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.e.service' 2024-09-15T23:34:55.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:55.690 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-09-15T23:34:55.690 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-09-15T23:34:55.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.h 2024-09-15T23:34:56.024 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:34:55 smithi067 systemd[1]: Stopping Ceph mds.h for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:34:56.524 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:34:56 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h[65398]: 2024-09-15T23:34:56.050+0000 7fda56492640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:34:56.524 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:34:56 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h[65398]: 2024-09-15T23:34:56.050+0000 7fda56492640 -1 mds.h *** got signal Terminated *** 2024-09-15T23:34:59.010 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:34:58 smithi067 podman[72195]: 2024-09-15 23:34:58.742764593 +0000 UTC m=+2.835713197 container died 4273c63666aea844efcb20200d6f0b47eeaf6a16c29451e4a983c9a6b2d80ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h, RELEASE=squid-8921c8e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-15T23:34:59.274 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:34:59 smithi067 podman[72195]: 2024-09-15 23:34:59.083939334 +0000 UTC m=+3.176887933 container cleanup 4273c63666aea844efcb20200d6f0b47eeaf6a16c29451e4a983c9a6b2d80ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:34:59.274 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:34:59 smithi067 bash[72195]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h 2024-09-15T23:34:59.573 INFO:journalctl@ceph.mds.h.smithi067.stdout:Sep 15 23:34:59 smithi067 podman[72208]: 2024-09-15 23:34:59.318395215 +0000 UTC m=+0.577562789 container remove 4273c63666aea844efcb20200d6f0b47eeaf6a16c29451e4a983c9a6b2d80ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-15T23:34:59.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.h.service' 2024-09-15T23:34:59.609 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:34:59.610 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-09-15T23:34:59.610 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-09-15T23:34:59.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.k 2024-09-15T23:34:59.938 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:34:59 smithi067 systemd[1]: Stopping Ceph mds.k for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:35:00.274 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:34:59 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-k[64799]: 2024-09-15T23:34:59.937+0000 7f75c19f8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:35:00.274 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:34:59 smithi067 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-k[64799]: 2024-09-15T23:34:59.937+0000 7f75c19f8640 -1 mds.k *** got signal Terminated *** 2024-09-15T23:35:01.524 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:35:01 smithi067 podman[72314]: 2024-09-15 23:35:01.309344008 +0000 UTC m=+1.479694377 container died b7512f6e00c8dbc543cfffb16ce085feb9a1f4213442f58912d99f43904b5f78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-k, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-8921c8e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-15T23:35:01.923 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:35:01 smithi067 podman[72314]: 2024-09-15 23:35:01.671408731 +0000 UTC m=+1.841759107 container cleanup b7512f6e00c8dbc543cfffb16ce085feb9a1f4213442f58912d99f43904b5f78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-k, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:35:01.923 INFO:journalctl@ceph.mds.k.smithi067.stdout:Sep 15 23:35:01 smithi067 bash[72314]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-k 2024-09-15T23:35:02.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.k.service' 2024-09-15T23:35:02.094 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:35:02.094 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-09-15T23:35:02.094 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-09-15T23:35:02.094 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.c 2024-09-15T23:35:02.445 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:02 smithi148 systemd[1]: Stopping Ceph mds.c for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:35:02.704 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:02 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c[66550]: 2024-09-15T23:35:02.443+0000 7ff356f7e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:35:02.704 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:02 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c[66550]: 2024-09-15T23:35:02.443+0000 7ff356f7e640 -1 mds.c *** got signal Terminated *** 2024-09-15T23:35:06.789 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:06 smithi148 podman[73573]: 2024-09-15 23:35:06.460074828 +0000 UTC m=+4.173090722 container died adb39dd76ca19f009b1925b2b31b5cd8a70dc9d7314877dcd95862285d80a975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:35:07.087 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:06 smithi148 podman[73573]: 2024-09-15 23:35:06.894104835 +0000 UTC m=+4.607120717 container cleanup adb39dd76ca19f009b1925b2b31b5cd8a70dc9d7314877dcd95862285d80a975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8921c8e, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-15T23:35:07.087 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:06 smithi148 bash[73573]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c 2024-09-15T23:35:07.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.c.service' 2024-09-15T23:35:07.355 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:07 smithi148 podman[73598]: 2024-09-15 23:35:07.08700539 +0000 UTC m=+0.626352149 container remove adb39dd76ca19f009b1925b2b31b5cd8a70dc9d7314877dcd95862285d80a975 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-c, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-15T23:35:07.356 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:07 smithi148 systemd[1]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.c.service: Deactivated successfully. 2024-09-15T23:35:07.356 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:07 smithi148 systemd[1]: Stopped Ceph mds.c for ee7f888e-73b6-11ef-bceb-c7b262605968. 2024-09-15T23:35:07.356 INFO:journalctl@ceph.mds.c.smithi148.stdout:Sep 15 23:35:07 smithi148 systemd[1]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.c.service: Consumed 3.532s CPU time. 2024-09-15T23:35:07.369 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:35:07.369 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-09-15T23:35:07.369 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-09-15T23:35:07.370 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.f 2024-09-15T23:35:07.690 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:35:07 smithi148 systemd[1]: Stopping Ceph mds.f for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:35:08.039 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:35:07 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f[66248]: 2024-09-15T23:35:07.688+0000 7f413b3d3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:35:08.039 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:35:07 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f[66248]: 2024-09-15T23:35:07.688+0000 7f413b3d3640 -1 mds.f *** got signal Terminated *** 2024-09-15T23:35:11.139 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:35:10 smithi148 podman[73709]: 2024-09-15 23:35:10.821666662 +0000 UTC m=+3.230996873 container died 7463f5940306b8e9ec9f697b31654ddd546baa4d0824325dd29c32e76685389b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-8921c8e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:35:11.139 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:35:11 smithi148 podman[73709]: 2024-09-15 23:35:11.139110932 +0000 UTC m=+3.548441143 container cleanup 7463f5940306b8e9ec9f697b31654ddd546baa4d0824325dd29c32e76685389b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-8921c8e, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-15T23:35:11.539 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:35:11 smithi148 bash[73709]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f 2024-09-15T23:35:11.539 INFO:journalctl@ceph.mds.f.smithi148.stdout:Sep 15 23:35:11 smithi148 podman[73721]: 2024-09-15 23:35:11.357267153 +0000 UTC m=+0.532163031 container remove 7463f5940306b8e9ec9f697b31654ddd546baa4d0824325dd29c32e76685389b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-8921c8e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-15T23:35:11.581 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.f.service' 2024-09-15T23:35:11.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:35:11.615 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-09-15T23:35:11.615 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-09-15T23:35:11.615 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.i 2024-09-15T23:35:11.926 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:35:11 smithi148 systemd[1]: Stopping Ceph mds.i for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:35:12.289 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:35:11 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i[66848]: 2024-09-15T23:35:11.925+0000 7fd02da30640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:35:12.289 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:35:11 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i[66848]: 2024-09-15T23:35:11.925+0000 7fd02da30640 -1 mds.i *** got signal Terminated *** 2024-09-15T23:35:15.093 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:35:14 smithi148 podman[73830]: 2024-09-15 23:35:14.830467217 +0000 UTC m=+2.998844205 container died bc6ff9a3ba66ad6f927eaca78f4dcf86a0b221af58ec48b3c58c431f11aed998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-8921c8e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-09-15T23:35:15.461 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:35:15 smithi148 podman[73830]: 2024-09-15 23:35:15.175947688 +0000 UTC m=+3.344324669 container cleanup bc6ff9a3ba66ad6f927eaca78f4dcf86a0b221af58ec48b3c58c431f11aed998 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-8921c8e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-15T23:35:15.461 INFO:journalctl@ceph.mds.i.smithi148.stdout:Sep 15 23:35:15 smithi148 bash[73830]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-i 2024-09-15T23:35:15.615 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.i.service' 2024-09-15T23:35:15.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:35:15.647 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-09-15T23:35:15.648 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-09-15T23:35:15.648 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.l 2024-09-15T23:35:16.039 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:35:15 smithi148 systemd[1]: Stopping Ceph mds.l for ee7f888e-73b6-11ef-bceb-c7b262605968... 2024-09-15T23:35:16.039 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:35:15 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-l[65944]: 2024-09-15T23:35:15.971+0000 7f102edae640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-15T23:35:16.039 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:35:15 smithi148 ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-l[65944]: 2024-09-15T23:35:15.971+0000 7f102edae640 -1 mds.l *** got signal Terminated *** 2024-09-15T23:35:18.830 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:35:18 smithi148 podman[73950]: 2024-09-15 23:35:18.543399801 +0000 UTC m=+2.674900649 container died c31cf6fdb3874838cdb3633717e5c1654fef00f2ce9eeedea4b6b7df16bd460f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-l, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-8921c8e, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-15T23:35:19.197 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:35:18 smithi148 podman[73950]: 2024-09-15 23:35:18.90489797 +0000 UTC m=+3.036398818 container cleanup c31cf6fdb3874838cdb3633717e5c1654fef00f2ce9eeedea4b6b7df16bd460f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6b5c73953776cdbbe2c2650e2c9813ab5ac3fdead855fa68a3941507184c3a80, name=ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-l, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-8921c8e) 2024-09-15T23:35:19.198 INFO:journalctl@ceph.mds.l.smithi148.stdout:Sep 15 23:35:18 smithi148 bash[73950]: ceph-ee7f888e-73b6-11ef-bceb-c7b262605968-mds-l 2024-09-15T23:35:19.335 DEBUG:teuthology.orchestra.run.smithi148:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ee7f888e-73b6-11ef-bceb-c7b262605968@mds.l.service' 2024-09-15T23:35:19.367 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-15T23:35:19.368 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-09-15T23:35:19.368 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-15T23:35:19.580 INFO:teuthology.orchestra.run.smithi062.stdout:Deleting cluster with fsid: ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:35:22.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-15T23:35:22.399 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:35:24.899 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-15T23:35:25.100 INFO:teuthology.orchestra.run.smithi148.stdout:Deleting cluster with fsid: ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:35:27.509 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:35:27.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:35:27.577 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-15T23:35:27.604 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-15T23:35:27.607 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290/remote/smithi062/crash 2024-09-15T23:35:27.608 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/crash -- . 2024-09-15T23:35:27.652 INFO:teuthology.orchestra.run.smithi062.stderr:tar: /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-15T23:35:27.652 INFO:teuthology.orchestra.run.smithi062.stderr:tar: Error is not recoverable: exiting now 2024-09-15T23:35:27.654 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290/remote/smithi067/crash 2024-09-15T23:35:27.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/crash -- . 2024-09-15T23:35:27.696 INFO:teuthology.orchestra.run.smithi067.stderr:tar: /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-15T23:35:27.696 INFO:teuthology.orchestra.run.smithi067.stderr:tar: Error is not recoverable: exiting now 2024-09-15T23:35:27.698 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290/remote/smithi148/crash 2024-09-15T23:35:27.699 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/crash -- . 2024-09-15T23:35:27.734 INFO:teuthology.orchestra.run.smithi148.stderr:tar: /var/lib/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-15T23:35:27.734 INFO:teuthology.orchestra.run.smithi148.stderr:tar: Error is not recoverable: exiting now 2024-09-15T23:35:27.735 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-15T23:35:27.735 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-15T23:35:27.848 INFO:teuthology.orchestra.run.smithi062.stdout:2024-09-15T23:21:31.259648+0000 mds.b (mds.0) 39 : cluster [WRN] Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:35:27.849 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-15T23:35:27.849 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[SEC\]' /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-15T23:35:27.891 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]' /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-15T23:35:27.971 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[WRN\]' /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-09-15T23:35:28.050 INFO:teuthology.orchestra.run.smithi062.stdout:2024-09-15T23:21:31.259648+0000 mds.b (mds.0) 39 : cluster [WRN] Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details 2024-09-15T23:35:28.050 INFO:tasks.cephadm:Compressing logs... 2024-09-15T23:35:28.050 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T23:35:28.093 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T23:35:28.096 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T23:35:28.133 INFO:teuthology.orchestra.run.smithi062.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-15T23:35:28.133 INFO:teuthology.orchestra.run.smithi148.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-15T23:35:28.134 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-09-15T23:35:28.135 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-15T23:35:28.135 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-15T23:35:28.135 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.i.log 2024-09-15T23:35:28.135 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-15T23:35:28.142 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-15T23:35:28.142 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.11.log 2024-09-15T23:35:28.142 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-15T23:35:28.142 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-15T23:35:28.142 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log 2024-09-15T23:35:28.142 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.l.log 2024-09-15T23:35:28.143 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.8.log 2024-09-15T23:35:28.143 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mgr.z.log 2024-09-15T23:35:28.143 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.c.log 2024-09-15T23:35:28.143 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.f.log 2024-09-15T23:35:28.143 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-15T23:35:28.143 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.f.log: /var/log/ceph/cephadm.log: 94.7% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.l.log.gz 2024-09-15T23:35:28.144 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-15T23:35:28.144 INFO:teuthology.orchestra.run.smithi062.stderr:gzip/var/log/ceph/ceph-client.admin.log: -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.9.log 2024-09-15T23:35:28.144 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-15T23:35:28.144 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.a.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.a.log: /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.g.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.d.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.j.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-15T23:35:28.145 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 94.5% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.a.log.gz 2024-09-15T23:35:28.146 INFO:teuthology.orchestra.run.smithi062.stderr: 87.0% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-15T23:35:28.146 INFO:teuthology.orchestra.run.smithi062.stderr: 94.5% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.d.log.gz 2024-09-15T23:35:28.146 INFO:teuthology.orchestra.run.smithi062.stderr: 94.7% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.g.log.gz 2024-09-15T23:35:28.146 INFO:teuthology.orchestra.run.smithi062.stderr: 94.6% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.j.log.gz 2024-09-15T23:35:28.150 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-15T23:35:28.150 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.10.log 2024-09-15T23:35:28.150 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-15T23:35:28.150 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-15T23:35:28.151 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-15T23:35:28.151 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.b.log 2024-09-15T23:35:28.151 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.k.log 2024-09-15T23:35:28.151 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-15T23:35:28.151 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-15T23:35:28.151 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-15T23:35:28.151 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-15T23:35:28.152 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log 2024-09-15T23:35:28.152 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.h.log 2024-09-15T23:35:28.152 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-15T23:35:28.152 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.e.log 2024-09-15T23:35:28.152 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-15T23:35:28.199 INFO:teuthology.orchestra.run.smithi062.stderr: 82.8% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-15T23:35:28.223 INFO:teuthology.orchestra.run.smithi062.stderr: 94.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-15T23:35:28.237 INFO:teuthology.orchestra.run.smithi062.stderr: 90.2% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-15T23:35:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.e.log: /var/log/ceph/cephadm.log: 94.5% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.k.log.gz 2024-09-15T23:35:28.277 INFO:teuthology.orchestra.run.smithi148.stderr: 83.0% 87.0% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log.gz -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-15T23:35:28.277 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-15T23:35:28.279 INFO:teuthology.orchestra.run.smithi148.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-15T23:35:28.288 INFO:teuthology.orchestra.run.smithi067.stderr: 93.1% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-15T23:35:28.291 INFO:teuthology.orchestra.run.smithi067.stderr: 83.0% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-15T23:35:28.328 INFO:teuthology.orchestra.run.smithi148.stderr: 90.9% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-15T23:35:28.339 INFO:teuthology.orchestra.run.smithi067.stderr: 87.0% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-15T23:35:28.363 INFO:teuthology.orchestra.run.smithi148.stderr: 93.2% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mgr.z.log.gz 2024-09-15T23:35:28.370 INFO:teuthology.orchestra.run.smithi062.stderr: 90.9% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-15T23:35:28.380 INFO:teuthology.orchestra.run.smithi148.stderr: 90.4% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-15T23:35:28.394 INFO:teuthology.orchestra.run.smithi067.stderr: 90.4% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-15T23:35:28.424 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-15T23:35:28.426 INFO:teuthology.orchestra.run.smithi067.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-15T23:35:28.846 INFO:teuthology.orchestra.run.smithi148.stderr: 93.5% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.f.log.gz 2024-09-15T23:35:28.872 INFO:teuthology.orchestra.run.smithi062.stderr: 90.2% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-15T23:35:29.047 INFO:teuthology.orchestra.run.smithi148.stderr: 94.5% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.c.log.gz 2024-09-15T23:35:30.060 INFO:teuthology.orchestra.run.smithi067.stderr: 93.7% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.e.log.gz 2024-09-15T23:35:30.875 INFO:teuthology.orchestra.run.smithi067.stderr: 91.6% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-15T23:35:31.019 INFO:teuthology.orchestra.run.smithi148.stderr: 95.4% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.i.log.gz 2024-09-15T23:35:32.008 INFO:teuthology.orchestra.run.smithi148.stderr: 91.6% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-15T23:35:34.810 INFO:teuthology.orchestra.run.smithi067.stderr: 95.4% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.h.log.gz 2024-09-15T23:35:41.014 INFO:teuthology.orchestra.run.smithi062.stderr: 90.1% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-15T23:35:43.324 INFO:teuthology.orchestra.run.smithi067.stderr: 94.1% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-mds.b.log.gz 2024-09-15T23:36:49.490 INFO:teuthology.orchestra.run.smithi148.stderr: 90.5% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.8.log.gz 2024-09-15T23:36:53.427 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.10.log.gz 2024-09-15T23:36:53.852 INFO:teuthology.orchestra.run.smithi067.stderr: 90.6% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-15T23:37:17.175 INFO:teuthology.orchestra.run.smithi148.stderr: 90.6% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.11.log.gz 2024-09-15T23:37:42.555 INFO:teuthology.orchestra.run.smithi148.stderr: 90.9% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-15T23:37:46.454 INFO:teuthology.orchestra.run.smithi067.stderr: 90.6% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-15T23:37:53.020 INFO:teuthology.orchestra.run.smithi148.stderr: 90.6% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-15T23:37:53.022 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-09-15T23:37:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:real 2m24.911s 2024-09-15T23:37:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:user 4m46.806s 2024-09-15T23:37:53.022 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m9.593s 2024-09-15T23:38:15.847 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-15T23:38:15.849 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-09-15T23:38:15.849 INFO:teuthology.orchestra.run.smithi067.stderr:real 2m47.738s 2024-09-15T23:38:15.849 INFO:teuthology.orchestra.run.smithi067.stderr:user 4m34.577s 2024-09-15T23:38:15.849 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m8.606s 2024-09-15T23:40:01.250 INFO:teuthology.orchestra.run.smithi062.stderr: 90.5% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-15T23:40:14.846 INFO:teuthology.orchestra.run.smithi062.stderr: 90.6% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-15T23:41:31.480 INFO:teuthology.orchestra.run.smithi062.stderr: 90.6% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-15T23:42:10.693 INFO:teuthology.orchestra.run.smithi062.stderr: 90.7% -- replaced with /var/log/ceph/ee7f888e-73b6-11ef-bceb-c7b262605968/ceph-osd.9.log.gz 2024-09-15T23:42:10.696 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-09-15T23:42:10.696 INFO:teuthology.orchestra.run.smithi062.stderr:real 6m42.589s 2024-09-15T23:42:10.696 INFO:teuthology.orchestra.run.smithi062.stderr:user 5m49.488s 2024-09-15T23:42:10.696 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m12.146s 2024-09-15T23:42:10.696 INFO:tasks.cephadm:Archiving logs... 2024-09-15T23:42:10.697 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290/remote/smithi062/log 2024-09-15T23:42:10.698 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T23:43:10.092 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290/remote/smithi067/log 2024-09-15T23:43:10.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T23:43:58.632 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290/remote/smithi148/log 2024-09-15T23:43:58.633 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T23:44:53.245 INFO:tasks.cephadm:Removing cluster... 2024-09-15T23:44:53.245 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 --force 2024-09-15T23:44:53.460 INFO:teuthology.orchestra.run.smithi062.stdout:Deleting cluster with fsid: ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:44:54.204 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 --force 2024-09-15T23:44:54.413 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:44:55.031 DEBUG:teuthology.orchestra.run.smithi148:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ee7f888e-73b6-11ef-bceb-c7b262605968 --force 2024-09-15T23:44:55.233 INFO:teuthology.orchestra.run.smithi148.stdout:Deleting cluster with fsid: ee7f888e-73b6-11ef-bceb-c7b262605968 2024-09-15T23:44:55.884 INFO:tasks.cephadm:Removing cephadm ... 2024-09-15T23:44:55.884 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-15T23:44:55.905 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-15T23:44:55.925 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-15T23:44:55.941 INFO:tasks.cephadm:Teardown complete 2024-09-15T23:44:55.942 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T23:44:55.953 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T23:44:55.954 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T23:44:55.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T23:44:55.969 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T23:44:56.085 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T23:44:56.098 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T23:44:56.098 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:44:56.101 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:44:56.104 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T23:44:56.117 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:44:56.118 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:44:56.119 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-09-15T23:44:56.125 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:44:56.125 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-09-15T23:44:56.125 INFO:teuthology.orchestra.run.smithi148.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +466us[ +500us] +/- 57ms 2024-09-15T23:44:56.125 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1181us[-1148us] +/- 58ms 2024-09-15T23:44:56.125 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +884us[ +918us] +/- 67ms 2024-09-15T23:44:56.125 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:44:56.127 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:44:56.127 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-09-15T23:44:56.127 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 +459us[ +498us] +/- 57ms 2024-09-15T23:44:56.127 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 -1185us[-1147us] +/- 58ms 2024-09-15T23:44:56.127 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +860us[ +898us] +/- 67ms 2024-09-15T23:44:56.127 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:44:56.131 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T23:44:56.131 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-09-15T23:44:56.131 INFO:teuthology.orchestra.run.smithi062.stdout:^* hv01.front.sepia.ceph.com 3 6 377 36 +440us[ +415us] +/- 57ms 2024-09-15T23:44:56.131 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -1213us[-1213us] +/- 58ms 2024-09-15T23:44:56.132 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +837us[ +837us] +/- 67ms 2024-09-15T23:44:56.132 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T23:44:56.132 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T23:44:56.161 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T23:44:56.162 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T23:44:56.211 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T23:44:56.370 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T23:44:56.451 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T23:44:56.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T23:44:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:56.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T23:44:56.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:56.642 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-15T23:44:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:56.671 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T23:44:56.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T23:44:56.718 INFO:teuthology.task.internal:Duration was 2747.129879 seconds 2024-09-15T23:44:56.719 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T23:44:56.769 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T23:44:56.769 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T23:44:56.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T23:44:56.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T23:44:56.827 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T23:44:56.829 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T23:44:56.830 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T23:44:57.349 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T23:44:57.350 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-09-15T23:44:57.350 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T23:44:57.376 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-09-15T23:44:57.377 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T23:44:57.404 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-09-15T23:44:57.405 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T23:44:57.438 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T23:44:57.438 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T23:44:57.441 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T23:44:57.447 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T23:44:57.505 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T23:44:57.506 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T23:44:57.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T23:44:57.517 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T23:44:58.984 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T23:44:58.996 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T23:44:58.996 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T23:44:59.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T23:44:59.084 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T23:44:59.117 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T23:44:59.130 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:59.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:59.136 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:59.163 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-09-15T23:44:59.170 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-09-15T23:44:59.195 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-09-15T23:44:59.222 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:59.257 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:59.274 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T23:44:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T23:44:59.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T23:44:59.303 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T23:44:59.304 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290/remote/smithi062 2024-09-15T23:44:59.304 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T23:44:59.383 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290/remote/smithi067 2024-09-15T23:44:59.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T23:44:59.424 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:24:03-fs-squid-distro-default-smithi/7906290/remote/smithi148 2024-09-15T23:44:59.424 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T23:44:59.466 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T23:44:59.466 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T23:44:59.469 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T23:44:59.471 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T23:44:59.518 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T23:44:59.532 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T23:44:59.532 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T23:44:59.573 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T23:44:59.574 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T23:44:59.577 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T23:44:59.579 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T23:44:59.589 INFO:teuthology.orchestra.run.smithi062.stdout: 266064 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 23:44 /home/ubuntu/cephtest 2024-09-15T23:44:59.593 INFO:teuthology.orchestra.run.smithi067.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 23:44 /home/ubuntu/cephtest 2024-09-15T23:44:59.596 INFO:teuthology.orchestra.run.smithi148.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 23:44 /home/ubuntu/cephtest 2024-09-15T23:44:59.597 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T23:44:59.625 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T23:44:59.729 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/postgres}} duration: 2747.1298789978027 failure_reason: '"2024-09-15T23:21:31.259648+0000 mds.b (mds.0) 39 : cluster [WRN] Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/597b9bb6-f4c4-4e49-b195-c049d8e7226e/postgres/data/pg_subtrans) see mds.b log and `damage ls` output for details" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-09-15T23:44:59.729 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T23:44:59.836 INFO:teuthology.run:FAIL